# Better Call GPT, Comparing Large Language Models Against Lawyers

Lauren Martin, Nick Whitehouse, Stephanie Yiu, Lizzie Catterson, Rivindu
Perera  AI Center of Excellence, Onit Inc.AucklandNew Zealand
<EMAIL_ADDRESS>
(2023)

###### Abstract.

This paper presents a groundbreaking comparison between Large Language Models
(LLMs) and traditional legal contract reviewers—Junior Lawyers and Legal
Process Outsourcers (LPOs). We dissect whether LLMs can outperform humans in
accuracy, speed, and cost-efficiency during contract review. Our empirical
analysis benchmarks LLMs against a ground truth set by Senior Lawyers,
uncovering that advanced models match or exceed human accuracy in determining
legal issues. In speed, LLMs complete reviews in mere seconds, eclipsing the
hours required by their human counterparts. Cost-wise, LLMs operate at a
fraction of the price, offering a staggering 99.97 percent reduction in cost
over traditional methods. These results are not just statistics—they signal a
seismic shift in legal practice. LLMs stand poised to disrupt the legal
industry, enhancing accessibility and efficiency of legal services. Our
research asserts that the era of LLM dominance in legal contract review is
upon us, challenging the status quo and calling for a reimagined future of
legal workflows.

Generative AI, Large Language Models, LegalAI, NLP

††copyright: acmlicensed††ccs: Computing methodologies Natural language
generation††ccs: Computing methodologies Information extraction††ccs: Applied
computing Law††ccs: Applied computing Document searching

##  1\. Introduction

The integration of Artificial Intelligence (AI) into the legal sector has
opened a new frontier in legal services. However, as of the current state of
research, there appears to be a significant gap in exploratory and
experimental studies specifically addressing the capabilities of Generative AI
and Large Language Models (LLMs) in the context of determination and discovery
of legal issues. Such studies would be instrumental in understanding how these
advanced AI technologies manage the intricate task of accurately classifying
and pinpointing legal matters, a domain traditionally reliant on the deep,
contextual, and specialised knowledge of human legal experts.

To address the identified gap in the research landscape, this study proposes
an experimental and exploratory analysis of the performance of LLMs in the
legal domain. The research aims to evaluate the capabilities of LLMs
contrasting their performance against human legal practitioners on high volume
real-world legal tasks. These types of high volume legal tasks are frequently
outsourced or pushed to less experienced lawyers, and given the rapid
advancements made by LLMs, raises the question of whether LLMs have achieved a
level of legal comprehension that is comparable to the quality, accuracy and
efficiency of Junior Lawyers or outsourced legal practitioners on such tasks.

Specifically, we focus on three primary research questions:

Do LLMs outperform Junior Lawyers and Legal Process Outsourcers in
determination and location of legal issues in contracts?

This question aims to assess the precision and recall of LLMs compared to
human professionals in determining and locating legal issues.

Can LLMs review contracts faster than Junior Lawyers and Legal Process
Outsourcers?

In this question, our focus is on evaluating the efficiency (as measured on a
temporal perspective) of LLMs in processing and responding to legal queries
compared to the time taken by human lawyers.

Can LLMs review contracts cheaper than Junior Lawyers and Legal Process
Outsourcers?

This question focuses on evaluating the comparative costs between Lawyers and
LPOs to determine whether LLMs are more cost effective.

Through this research, we aim to contribute a comprehensive understanding of
the potential capabilities and limitations of LLMs in the legal domain,
providing valuable insights for Legal and AI practitioners.

##  2\. Related Work

The related work in this area encompasses a broad spectrum of technological
advancements in generative AI applied on legal contract reviews and generation
of legal contracts. In this section we will be exploring direct and also
indirect advancements towards application of Natural Language Understanding
(NLU) and Natural Language Generation (NLG) in legal domain giving the
priority to generative AI approaches.

Guha et al. (guha2023legalbench, ) introduce the LegalBench which is a
collaborative benchmark for measuring legal reasoning in LLMs. Their work
focuses on establishing a benchmark for evaluating the capabilities of LLMs in
legal reasoning. By employing the Issue, Rule, Application, Conclusion
framework (IRAC), which is a widely used method for organising legal analysis,
the authors have initiated a collaborative project aimed at developing a
comprehensive and open-source benchmark. This endeavour emphasises the
potential for computational tools to enhance legal practices, especially in
administrative and transactional settings. However, the primary objective of
LegalBench is not to replace legal professionals with computational systems,
but to assess the extent to which current models can support and augment legal
reasoning.

Our research, in contrast, delves into a more specialised commercial domain
within legal technology, focusing on the application of LLMs in determining
and locating legal issues in contracts. This specific focus on contract
analysis sets our work apart from the broader legal reasoning perspective
adopted in LegalBench. While Guha et al. emphasise the need for collaborative
efforts and in developing benchmarks for legal reasoning with open source
data, our study provides a detailed and practical examination of the
capabilities of LLMs compared directly with legal practitioners, including
Junior Lawyers and LPOs. This aspect of direct comparison is particularly
significant as it sheds light on the practical effectiveness and efficiency of
LLMs in real-world legal tasks, a perspective not deeply explored in
LegalBench.

The significance of our work is further accentuated by its emphasis on the
efficiency of LLMs in contract review, a crucial aspect in legal practice
where time is a critical factor. Our study not only benchmarks the performance
of LLMs against human practitioners but also evaluates their efficiency in
terms of time and cost, offering valuable insights into the practical
application of these models in legal settings. This focus on both performance
and efficiency provides a pragmatic perspective that has an immediate
applicability in legal tasks such as contract review, which are common and
often time intensive.

In conclusion, while LegalBench establishes a foundational framework for
evaluating LLMs in legal reasoning, our research offers a more focused and
application-oriented analysis within the legal field. By comparing the
capabilities and efficiency of LLMs with those of human legal practitioners in
the specific context of contract review, our study addresses a critical gap in
the current research landscape. This approach not only benchmarks the
effectiveness of LLMs in a practical legal task but also provides substantial
contributions to the evolving field of legal technology and its applications.

Tan et al. (tan2023chatgpt, ) bring a more practical dimension to the theory
oriented approach we noticed in the previous research. Although this study
does not focus on a deep statistical analysis, it provides some important
aspect of practical usage of existing tools. A key aspect of this research,
the comparative analysis between ChatGPT and JusticeBot, provides a nuanced
understanding of the strengths and limitations of AI-driven legal tools. The
study acknowledges that while ChatGPT may not always provide perfectly
accurate or reliable information, it offers a powerful and intuitive interface
for general public, a significant consideration in enhancing access to legal
information.

However, Tan et al. also identify potential areas for improvement,
particularly in the reliability and depth of the legal information provided by
ChatGPT. In the legal field, where incorrect information can have serious
consequences, these aspects are of utmost importance. The research suggests
the need for continuous improvement and testing of ChatGPT, especially in the
terms of keeping the model updated with the latest legal development and
training it on diverse legal scenarios. Although, Open AI claims that ChatGPT
currently performs in the level of a legal graduate, there is no evidence in
its performance in various legal scenarios. This is one of the key aspect
which we tried to address in our research.

Callister (callister2023generative, ) delves into the philosophical and
cognitive implications of AI in law, addressing concerns about the reliability
and consistency of AI, and highlighting potential issues such as information
hallucination. It also emphasises the transformative nature of AI in shifting
the traditional paradigms of legal research.

While Callister (callister2023generative, ) focus on the philosophical aspect
of AI and LLM’s ability to involve in the legal decision making, we tried to
offer a viewpoint of practical capabilities and efficiency of LLM’s in
specific legal tasks. We, however, agree with the cautionary perspective on
the broader implications of generative AI for legal epistemology brought to
light by Callister (callister2023generative, ) which needs to be accounted
when using generative AI widely in legal decision making.

Choi et al. (choi2023chatgpt, ) bring a new dimension to the LLMs
applicability in legal domain by focusing on ChatGPT’s applicability in legal
education. Although this research does not directly align with our study’s
interest in AI’s application in the legal domain, it brings some new
perspectives on how LLMs perform in the legal domain. One of the key findings
in this study is the analysis of limited ability of ChaptGPT to perform
complex legal analysis and issue spotting which we mainly focused on our
research. Our research also extends this dialogue by comparing LLMs’
performance not just to academic standards (as noticed in (guha2023legalbench,
)), but also against real world legal practitioners, particularly Junior
Lawyers and LPO professionals. This comparison offers a novel perspective on
the utility of LLMs in practical legal work, a dimension not fully explored in
any of the aforementioned research attempts.

##  3\. Methodology

This section focuses on the research methodology covering how data was
collected, analysed, prepared, and how the models were selected to be included
as part of this research.

###  3.1. Overall Design

The experiment was designed to compare Junior Lawyers and LPOs against LLMs
using Senior Lawyers as a benchmark, establishing ground truth for determining
and locating legal issues in contracts. This replicates the process lawyers go
through in contract reviews. The aforementioned benchmark formed the basis to
assess LLMs, LPOs and Junior Lawyers in their accuracy of making
determinations and locating the same legal issues.

###  3.2. Ethics Policy

This research adhered to the ethical standards defined by Onit Inc which
covered data collection, analysis, and participant involvement in carrying out
primary and secondary research.

All participants were fully informed about the research purpose, data usage,
and their rights, including the right to withdraw at any time. In addition, we
also ensured the anonymity of participants by replacing identifiable
information in the collected data.

All contract data used in the research was anonymised during the data analysis
phase to eliminate identifiable details of the involved parties from the
process. However, the participant data were de-identified to help further
analysis while maintaining the data privacy standards.

We also established ethical oversight and compliance through an involvement of
an ethics committee to ensure adherence to the organisation’s ethic policies.
This included audits of the research process and legal compliance to assure
the research activities complied with the applicable data protection and
privacy laws.

###  3.3. Data Collection and Analysis

This section will discuss the methods for preparing data for the use in this
research and how ground truth was established for the purpose of benchmarking.

####  3.3.1. Dataset Preparation

Data for this study were collected from ten procurement contracts, sourced
from real-world legal agreements and anonymised to preserve confidentiality.

The research used procurement contracts, as they are frequently among the most
voluminous types of contracts reviewed by legal practitioners, rivaling the
prevalence of Non-Disclosure Agreements (NDAs). The study deliberately
excluded NDAs from this analysis due to their typically concise format, which
does not provide a complex enough framework to carry out an in-depth
evaluation of a LLMs capabilities in determining and locating legal issues.

The jurisdictional scope of the selected contracts was deliberately balanced
between the United States (US) which is uniquely based on a combination of
statutory and common law, and New Zealand (NZ) which is based on common law.
This approach ensures that the findings of our research have relevance across
different legal systems, which increases the utility and applicability of the
research.

In the dataset of contracts utilised for this study, we took careful measures
to preserve confidentiality and data integrity. All references to party names
were substituted with pseudonyms, and any sections of the text that required
redaction were supplanted with plausible, yet fictitious, information. This
methodological step was essential to maintain the authenticity of the contract
structure while safeguarding sensitive information, thereby upholding ethical
research standards without compromising the quality of our analysis.

Our analysis was structured around a two-factor approach. The first factor
comprised a document scenario that provided context, including jurisdiction,
historical background, organisational size, and the product or service
involved for each party in the contract. The second factor was a set of
standardised checks using a contract review playbook, a common practice among
legal professionals. The review playbook employed in our study was derived
from real-world examples to ensure practical relevance.

Each contract was reviewed from the perspective of a third-party observer,
with an equal number of agreements analysed from the vantage points of the
supplier and the buyer, respectively. This balanced perspective is designed to
simulate the thoroughness and objectivity a legal practitioner would bring to
contract analysis, thereby testing the LLMs’ performance in a realistic
setting.

####  3.3.2. Ground Truth Establishment

Senior Lawyers were instructed to evaluate each contract for the purpose of
establishing ground truth data. Their task was to ascertain whether the
contract adhered to or deviated from the predefined standards. Additionally,
they were required to locate the specific sections of the contract that
influence their judgments. In instances where a standard was not met due to
the absence of relevant information in the contract, Senior Lawyers were
directed to record this explicitly at the conclusion of their assessment for
each contract.

This collected data was then aggregated to formulate benchmarks for each
evaluative criterion, relying on the majority consensus to determine both
compliance with the standard and the precise location of the relevant
contractual reference.

Furthermore, Senior Lawyers were requested to record the duration of each
contract review. This measure aimed to capture the average time expended by
legal professionals in reviewing contracts. This temporal data was intended
for subsequent comparison with the time taken by Junior Lawyers, LPOs and
LLMs.

###  3.4. Establishing hourly rates and LLM costs

The hourly rates for lawyers were based on inside counsel rates determined
through industry benchmark reports such as ACC’s 2023 Law Department
Compensation Survey (acc_compensation_survey, ) and external counsel rates
determined by market data held by Onit Inc. The costs for LLMs were determined
through commercial pricing provided by the service suppliers.

###  3.5. Model Analysis

The basis for model selection in this research was decided on several factors;
prominent models that have emerged in the LLM space, preliminary tests
demonstrated promising model capability in our research questions and the
context window limits of the models.

####  3.5.1. Preliminary Research

The advent of foundational LLMs in mainstream applications has led to the
entry of several major competitors in this domain, including OpenAI
(openai2023gpt, ), Google (anil2023palm, ), Anthropic (anthropic_claude, ),
Amazon (amazon_titan, ), and Meta (touvron2023llama, ). In selecting the
models for our analysis, we focused on those developed by these organisations,
as they represent the most prominent entities in this competitive landscape.

Preliminary evaluations were carried out on models from these identified
companies to assess their applicability and effectiveness within the legal
domain. These evaluations involved processing sample contracts through the
models, and conducting careful analysis of the outputs. The primary focus of
these tests was to analyse the models’ reasoning capabilities and their
ability to determine legal issues that existed in contracts, as well as their
proficiency in pinpointing the location of these issues within these
documents.

In developing our research methodology, we placed significant emphasis on
determining the optimal size of the context window to effectively address our
research questions. The context window’s size is a pivotal factor as it
directly impacts the ability of the language models to process and analyse the
contract documents comprehensively. To ensure a thorough review, we considered
the average length of procurement contracts, the complexity of document
scenarios required to provide a meaningful context, and the detail necessary
in the document checklists.

Our primary goal was to select models capable of handling an extensive amount
of contextual information to facilitate accurate and nuanced analysis. As a
result, we established a minimum threshold for the context window, and models
that could not support at least 16,000 tokens were considered inadequate for
our purposes and were subsequently excluded from our analysis.

During our preliminary testing, it was found that models with context windows
smaller than 16,000 tokens, such as LLaMA2 and Amazon Titan, required us to
split the documents into multiple parts. This segmentation led to a
significant portion of the context window being dedicated to repeatedly
referencing vital elements, including the document scenario, the review
playbook, and the contract’s definitions section. This repetition was
necessary to ensure that each segment was properly contextualised, but it was
also inefficient and detracted from the models’ ability to analyse the
contract as a whole.

Although there are various techniques, such as Retrieval Augmented Generation
(RAG), that mitigate the limitations of smaller context windows, our
preliminary evaluation indicated that these methods did not measure up to the
performance of models with larger context windows. The RAG approach, in
particular, was found to introduce discontinuities in the contextual
understanding of the contracts. These gaps were significant enough to
compromise the models’ performance, making them unsuitable for an accurate
comparison with the level of comprehension and analysis expected of a human
lawyer. Consequently, our study focused on models with large enough context
windows to ensure representative assessment of their capabilities in legal
contract analysis.

####  3.5.2. Model Settings

Table 1

Model | Temperature | Token Input | Training Data
---|---|---|---
GPT4 32k | 0.2 | 32,768 | Up to Sept 2021
GPT4-1106 | 0.2 | 128,000 | Up to Apr 2023
GPT3.5-turbo-16k | 0.2 | 16,385 | Up to Sept 2021
Claude 2.1 | 0.2 | 200,000 | Up to early 2023
Claude 2.0 | 0.2 | 100,000 | Up to early 2023
Palm2 Text-bison-32k-2 | 0.2 | 32,768 | Up to mid 2021
Table 1. Model Settings

###  3.6. Prompt Engineering

Each LLM had a different set of requirements for the prompts needed to
complete the contract review tasks. Prompts were structured with the following
elements; the role, the task and the context. The role required the LLMs to
adopt the persona of a lawyer in the task it was to undertake. The LLMs were
tasked with reviewing contracts against given standards by determining and
locating where those issues were in contracts. The context element was
designed to mirror the instructions typically provided to a Lawyer, LPO or
Contract Reviewer. Context included a range of factors such as the target
audience for the contract, pertinent background information regarding the
contracting parties, and the specific scenario under which the contract was
being negotiated. Examples of the prompts used for each LLM can be referred to
in Tables A.1 and A.1.

##  4\. Evaluation and Results

###  4.1. Legal practitioner comparison

####  4.1.1. Inter-annotator Agreement

To assess the level of agreement among the three groups of legal practitioners
(Senior Lawyers, Junior Lawyers, and LPOs), we conducted an inter-annotator
agreement analysis employing Cronbach’s Alpha (cronbach1951coefficient, ) as
the primary method of evaluation.

Figure 1. Level of agreement on issues by role

The inter-annotator agreement results as shown in Fig. 1 revealed noteworthy
findings in our research on the applicability of LLMs in the legal domain. The
overall agreement among all participants was remarkably high, with an Alpha
value of 0.923366, indicating a very strong consensus. Interestingly, the
agreement among Senior Lawyers alone was the lowest at 0.719308, suggesting a
more varied approach to issue identification in contracts among experience
practitioners. Junior lawyers demonstrated a slightly higher level of
agreement, with an Alpha value of 0.765058, potentially reflecting more
consistent training methodologies or adherence to established legal
frameworks. Notably, the agreement among LPO practitioners reached a perfect
Alpha of 1, indicating absolute uniformity in their responses.

###  4.2. Comparing LLMs against Junior Lawyers and LPOs

####  4.2.1. Performance Comparison

To address the research questions outlined in Section 1, our research centered
on conducting a comparative analysis of the accuracy of different LLMs (as
detailed in Section 3.5.1) in relation to Junior Lawyer and LPOs. This
comparison was conducted with the Senior Lawyers’ determinations serving as
the ground truth data against which those results were assessed.

| Precision | Recall | F-score | Loss
---|---|---|---|---
LPO | 0.933 | 0.823 | 0.874 | 0.126
GPT4-1106 | 0.835 | 0.910 | 0.871 | 0.129
Junior | 0.876 | 0.845 | 0.860 | 0.140
Claude 2.0 | 0.743 | 0.907 | 0.817 | 0.183
GPT4 -32k | 0.958 | 0.723 | 0.820 | 0.180
Claude 2.1 | 0.723 | 0.917 | 0.809 | 0.191
Palm2 text-bison | 0.617 | 0.831 | 0.708 | 0.292
GPT3.5 | 0.531 | 0.864 | 0.657 | 0.343
Table 2. Performance ranked by F-score in determining legal issues | Precision | Recall | F-score | Loss
---|---|---|---|---
LPO | 0.915 | 0.666 | 0.770 | 0.230
GPT4-32k | 0.847 | 0.660 | 0.740 | 0.260
Claude 2.1 | 0.911 | 0.569 | 0.701 | 0.299
GPT4-1106 | 0.857 | 0.575 | 0.686 | 0.312
Junior | 0.866 | 0.542 | 0.667 | 0.333
Claude 2.0 | 0.812 | 0.517 | 0.631 | 0.369
GPT 3.5 | 0.633 | 0.607 | 0.620 | 0.380
Palm2 text-bison | 0.563 | 0.378 | 0.452 | 0.548
Table 3. Performance ranked by F-score in locating legal issues

This comparative evaluation was carried out on two fronts similar to previous
evaluations: the ability to determine and locate legal issues in contracts.

In the issue determination aspect, both GPT4-1106 and LPO practitioners
emerged as the top performers with an F-score of 0.87, indicating a high level
of precision and reliability in this aspect. Junior lawyers achieved an
F-score of 0.86 suggesting that their expertise is slightly below the leading
LLMs in determining legal issues. Other models, such as Claude 2.0 and
GPT4-32k, also showed strong performance with F-scores of 0.82. Whereas GPT3.5
and Palm2 text-bison trailed with lower accuracy scores.

The proficiency in pinpointing legal issues within contracts presented a
different competitive scenario. Here, LPOs took the lead with an F-score of
0.77, edging out the top-performing LLM in this metric, GPT4-32k, which
achieved a F-score of 0.74. Notably, GPT4-1106, while excelling in issue
determination, fell short in localization accuracy, achieving an F-score of
0.69. This divergence underscores the distinct competencies and limitations
that are specific to each LLM. Models such as Claude and GPT3.5 exhibited
moderate localization capabilities. In stark contrast, Palm2 Text-bison’s
performance was markedly lower, particularly in issue location, with an
F-score of 0.46.

####  4.2.2. Time comparison

Alongside the performance comparison, a time analysis was conducted to
evaluate how Junior Lawyers and LPOs compared to LLMs in terms of time
efficiency when reviewing legal contracts.

The time efficiency analysis of each group yielded significant insights in our
study as shown in Table 4. Amongst the human practitioners, Senior Lawyers
demonstrated the most efficiency averaging 43.46 minutes per document, while
Junior Lawyers took slightly longer, averaging 56.17 minutes. Remarkably, LPOs
took substantially longer with an average of 201 minutes.

Conversely, all LLMs demonstrated markedly higher time efficiency. The longest
processing time was taken by GPT-1106 at an average of 4.7 minutes and the
shortest Palm2 text-bison at 0.73 minutes. These findings indicate that LLMs
significantly outperform both Junior Lawyers and LPOs in terms of time
efficiency for the task of reviewing legal contracts.

Role | Average Time per Document (Minutes)
---|---
Senior Lawyer | 43.46
Junior Lawyer | 56.17
LPO | 201.00
GPT 4-1106 | 4.70
GPT 4-32k | 2.11
GPT 3.5 | 1.44
Claude 2.1 | 2.05
Claude 2.0 | 1.63
Palm2 text-bison | 0.73
Table 4. Time Comparison

####  4.2.3. Cost comparison

As a part of the evaluation, we also conducted a detailed cost comparison
between lawyers, LPO practitioners and LLMs utilised in this research
(GPT4-1106, GPT4 32k, GPT 3.5, Claude 2.0, Claude 2.1 and Palm2 Text-bison).
This comparison is crucial to understand the economic implications of
employing LLMs in the legal domain, especially in the tasks involving the
determination and location of legal issues in contracts. The cost per contract
for human practitioners was calculated based on their average time spent on a
contact and their respective hourly rates. For LLMs, the cost was determined
by the average number of input and output tokens, and the respective cost per
token, except for Palm2 text-bison which was priced based on the number of
characters used.

The cost comparison, as shown in Table 5, illustrates a stark constrast in
cost efficiency between human practitioners and LLMs with LLMs offering a
significantly lower cost per document.

Role | Average Cost per Document ($USD)
---|---
Senior Lawyer | 75.92
Junior Lawyer | 74.26
LPO | 36.85
GPT 4-1106 | 0.25
GPT 4-32k | 1.24
GPT 3.5 | 0.05
Claude 2.1 | 0.02
Claude 2.0 | 0.02
Palm2 text-bison | 0.03
Table 5. Cost Comparison

##  5\. Discussion and Further work

The findings of this study provide a comprehensive analysis of the performance
of LLMs in comparison to Junior Lawyers and LPOs in the context of legal
contract review. Our discussion focuses on the three core questions that drive
this research.

###  5.1. Do LLMs outperform Junior Lawyers and LPOs in determination and
location of legal issues in contracts?

The results show LLMs demonstrate comparable, if not superior, performance in
determining legal issues when compared to Junior Lawyers and LPOs. However, a
LLMs ability to locate issues within contracts, particularly where a standard
is not present, is model-dependent and may not consistently outperform human
practitioners, highlighting the importance of selecting the right model for
the legal task.

Evidence from our results supports this view, with LLMs, specifically
GPT4-1106, achieving an F-score of 0.87 in issue determination, on par with
LPO results and slightly better than Junior Lawyers. However, in issue
location, GPT4-1106 only achieved an F-score of 0.69 which was lower than the
LPOs’ score of 0.77, by contrast, the top performing LLM in issue location,
GPT4-32k, achieved an F-score of 0.74.

Arguably, the lower performance by LLMs in locating issues could impact their
ability to accurately mark up contracts on their own, constraining their
effectiveness in eliminating the role of a Lawyer in contract review. However,
with the pace of LLM improvement and the many LegalTech software tools in
market utilising a variety of LLMs in combination with more traditional
software approaches, we believe the 0.03 F-score difference between LPOs and
LLMs is trivial, and we assert that when it comes to accuracy in contract
review, LLMs are now operating at the same level of quality as LPOs.

###  5.2. Can LLMs review contracts faster than Junior Lawyers and LPOs?

In evaluating the efficiency of contract review, our analysis, as detailed in
Table 4 indicates a substantial difference in the speed of contract review
between LLMs and human reviewers. Specifically, the fastest LLM, Palm2 text-
bison completed the contract review tasks in an average of 0.728 minutes. This
contrasts sharply with the average time of 56.17 minutes for a Junior Lawyer
and 201 minutes for an LPO. Such a disparity not only highlights the superior
processing capabilities of LLMs but also suggests a paradigm shift in how
contracts can be reviewed.

For instance, the fastest LLM outpaces LPOs by an astonishing 276-fold and
Junior Lawyers by 77-fold. Such metrics not only highlight the transformative
capabilities of LLMs in legal practice but also illustrate their potential to
revolutionize contract review processes. In the time it takes a human reviewer
to scrutinize a single contract, an LLM could analyze hundreds. Moreover, LLMs
possess the capacity to handle multiple requests concurrently, exponentially
increasing the volume of contracts they can assess. This level of efficiency
will significantly alter the landscape of legal services, offering
unprecedented scalability and speed.

A frequent skepticism surrounding the adoption of LLMs pertains to the initial
time investment required for their setup. Our research found that setting up,
testing, fine-tuning, and validating the LLMs’ system prompts took an average
of 16 hours. Notably, this investment is on par with the time dedicated to
instructing Junior Lawyers and LPOs for similar tasks. This equivalence in
preparatory time challenges the notion that LLMs’ speed advantage is offset by
their setup requirements, reinforcing the position that LLMs offer significant
time-saving benefits once operational.

###  5.3. Can LLMs review contracts cheaper than Junior Lawyers and LPOs?

Our cost analysis, outlined in Table 5, provides a stark economic comparison
between LLMs and traditional human contract review. The data reveal that the
cost per contract review for LLMs is significantly lower than that for human
reviewers. For instance, while a Junior Lawyer incurs an average cost of 74
dollars per contract review, the fastest LLM performed the same task for
approximately 2 cents. This equates to a cost reduction of more than 99.97
percent when compared to Junior Lawyers, and 99.94 percent compared to LPOs.

These figures not only reflect the raw efficiency of LLMs but also allude to
the potential for scaling legal services. The adoption of LLMs will enable
legal departments and law firms to substantially increase their contract
review throughput without a proportional increase in costs. This scalability
suggests a future where legal services could become more accessible to a wider
array of users or clients, revolutionizing the industry. Nevertheless, this
optimistic outlook should be tempered with careful consideration of the
quality of reviews and the need for ongoing supervision of LLM outputs.

###  5.4. Implications for the Legal Industry

The implications of this research on the legal industry are profound and
multifaceted. We believe LLMs are already at a point where they can
significantly disrupt LPOs. Our research shows LLMs are as accurate, yet
faster and more cost-effective at the tasks LPOs are typically engaged in.
There is little doubt that broad adoption of LLMs will lead to a significant
shift in the LPO business model, necessitating a need for LPOs to pivot from
providing manual legal process services, perhaps to becoming managers and
operators of LLM technology, focusing on quality control, interpretation, and
nuanced application of the insights generated by these models.

We anticipate that Junior Lawyers will also encounter disruption, though the
nature of these disruptions will differ. While our research does not explore a
broad enough spectrum of the capabilities performed by Junior Lawyers to
establish a one to one comparison, it does indicate that LLMs are already
outperforming Junior Lawyers in specific areas. We believe the partial
adoption of LLMs will begin to reduce the demand for entry-level legal
positions, as they are currently defined, over time. In contrast, established
Junior Lawyers will engage in more complex, high-value work earlier in their
careers, as routine tasks are gradually transitioned to LLMs.

The ramifications for legal departments and law firms are unmistakable: LLMs
present remarkable efficiency gains and cost savings. Early adopters of this
technology are poised to gain a significant competitive edge, likely
triggering an arms race in the legal sector. Despite the potential for
controversy and litigation, given the regulatory landscape of the legal
profession, genuine resistance to LLM adoption may be less about the validity
of the technology—as evidenced by our research—and more about protecting
established interests. Such opposition could be perceived as a strategic
manoeuvre to maintain the status quo rather than a genuine concern for the
practical or ethical shortcomings of AI in legal practice, this, of course,
would run counter to the industry wide pursuit for greater access to justice,
and potentially effect trust in the legal sector.

###  5.5. Further Work

To build on our findings, further research is needed to evaluate LLM
performance across a broader range of contract types and to expand the ground
truth dataset. Additionally, we are interested in exploring LLMs’ capabilities
in contract negotiation—a complex area requiring contextual understanding
beyond the contract text itself. This future work will help to map the full
potential of LLMs in the legal industry and address any limitations observed
in the current study.

##  6\. Conclusion

Our findings show that LLMs perform on par with LPOs and Junior Lawyers,
accurately determining legal issues within contracts. Notably, when it comes
to the speed of contract review, LLMs demonstrate a significant advantage due
to their computational efficiency, which enables them to process and analyse
text much faster than human practitioners. This advantage is a game-changer in
terms of productivity and turnaround times for contract review.

Moreover, the cost analysis show that LLMs offer a substantially cheaper
alternative for contract review when compared to the costs associated with
Junior Lawyers and LPOs. The combination of high accuracy, rapid processing,
and lower costs makes LLMs an attractive option for legal practitioners and
firms looking to optimise their contract review processes.

In conclusion, the evidence from this study supports the assertion that LLMs
are not only viable but are superior tools for legal contract review over
Junior Lawyers and LPOs. They can deliver accurate results at a fraction of
the time and cost required by traditional human-based review. As the
technology continues to advance, it is likely that LLMs will play an
increasingly crucial role in the legal industry, reshaping the landscape of
legal services and offering new opportunities for efficiency and scalability.

##  7\. Acknowledgments

This research was conducted by Onit’s Artificial Intelligence Center of
Excellence, with special mention to the principal investigators:

Lauren Martin, Legal Services Manager (PI), Stephanie Yiu, Legal Engineer (Co-
PI), Lizzie Catterson, Legal Engineer and Data Analyst (Co-PI), Dr. Rivindu
Perera, VP of AI and Data Science (Co-PI), Nick Whitehouse, Co-founder and
Managing Director (Co-PI).

## References

  * [1] Amazon.  Amazon titan: A large language model.  https://aws.amazon.com/bedrock/titan, 2023.  Accessed: 2023-12-19.
  * [2] R. Anil, A. M. Dai, O. Firat, M. Johnson, D. Lepikhin, A. Passos, S. Shakeri, E. Taropa, P. Bailey, Z. Chen, et al.  Palm 2 technical report.  arXiv preprint arXiv:2305.10403, 2023.
  * [3] P. D. Callister.  Generative ai and finding the law.  Available at SSRN 4608268, 2023.
  * [4] J. H. Choi, K. E. Hickman, A. Monahan, and D. Schwarcz.  Chatgpt goes to law school.  Available at SSRN, 2023.
  * [5] L. J. Cronbach.  Coefficient alpha and the internal structure of tests.  psychometrika, 16(3):297–334, 1951.
  * [6] N. Guha, J. Nyarko, D. E. Ho, C. Ré, A. Chilton, A. Narayana, A. Chohlas-Wood, A. Peters, B. Waldon, D. N. Rockmore, D. Zambrano, D. Talisman, E. Hoque, F. Surani, F. Fagan, G. Sarfaty, G. M. Dickinson, H. Porat, J. Hegland, J. Wu, J. Nudell, J. Niklaus, J. Nay, J. H. Choi, K. Tobia, M. Hagan, M. Ma, M. Livermore, N. Rasumov-Rahe, N. Holzenberger, N. Kolt, P. Henderson, S. Rehaag, S. Goel, S. Gao, S. Williams, S. Gandhi, T. Zur, V. Iyer, and Z. Li.  Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models, 2023.
  * [7] A. of Corporate Counsel.  2023 law department compensation survey.  https://www.acc.com/sites/default/files/2023-09/ACC_2023_Law_Dept_Compensation_Survey_Exec_Summary.pdf, 2023.  Accessed: 2023-12-01.
  * [8] OpenAI.  Gpt-4 technical report. arxiv 2303.08774.  Open AI, 2:13, 2023.
  * [9] J. Tan, H. Westermann, and K. Benyekhlef.  Chatgpt as an artificial lawyer?  Artificial Intelligence for Access to Justice (AI4AJ 2023), 2023.
  * [10] A. Team.  Claude: A large language model.  https://www.anthropic.com/claude, 2023.  Accessed: 2023-12-19.
  * [11] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  arXiv preprint arXiv:2307.09288, 2023.

##  Appendix A Appendix

###  A.1. System prompts used in research

For each LLM used in the research a system prompt was constructed to send with
all the relevant information for each contract. Referenced is the system
prompt used for GPT 4-1106, see Table A.1, and Claude 2.1, see Table A.1.

o X[1] X[2] Model | System Prompt
---|---
GPT 4-1106 | As an experienced lawyer, your task is to meticulously review contracts currently under negotiation from the perspective of the Supplier party. Your intended audience is internal counsel, and the contracts you are reviewing are standalone agreements between the parties. These contracts are a mix of US and NZ documents. Your goal is to identify any legal issues that could potentially affect your client’s legal rights or obligations such as language in the contract which does not meet the standard or requirement of the party you represent. These issues could be points of contention, ambiguities, potential risks, or compliance matters that could lead to a dispute or liability. Your analysis should be based on the following: \- The document scenario given for each contract, which provides background context on both parties involved in the agreement including the size, industry, and product/service of the parties and any specific areas of concern in the contract. \- The provided contract. \- The provided checklist for the client you represent, which lists the items they deem important in the contract. These items, or ’checks’, indicate the legal standard or requirement of the party you represent. For each section of the provided contract, carefully analyze it and identify ALL of the checks listed in the provided checklist. Format your answer as a JSON object that includes an array of JSON objects, each representing a different check from the checklist. Each of these JSON objects MUST include the following keys: 1\. ćhecklist_Numbering’: The numbering of the check from the checklist. Please STRICTLY adhere to the EXACT numbering format used in the checklist. 2\. ćontract_Lines’: An array of sentences or clauses in the contract that corresponds to the check. If the contract does not contain a specific sentence or clause related to the check, then leave this key blank. 3\. ’explanation’: A comprehensive and descriptive explanation of how the contract meets/fails to meet the legal standard in the check. 4\. ‘check_Met’: Indicate whether the check is ‘Met’/‘Not met’. If the contract_Lines key is ’null’, then indicate ’Not met’. 5\. ‘assumptions’: List any additional assumptions that were necessary to answer this check. Important: \- Provide a separate JSON object for EACH check. \- Ensure each JSON object is formatted in the following order: ’checklist_Numbering’, ’contract_Lines’, ’explanation’, ’check_Met’, ’assumptions’. \- Ensure to adhere strictly to the structure and naming of the keys as described above. \- Remember to thoroughly reread the contract and treat each check independently for accurate identification. \- The context provided in the document scenario should guide your review process.
Table 6. System Prompt: GPT 4-1106

o X[1] X[2] Model | System Prompt
---|---
Claude 2.1 | Here is a document scenario for you to use as background context for reviewing a contract: ¡document_scenario¿ XXX ¡/document_scenario¿ Here is a contract for you to review: ¡contract¿ XXX ¡/contract¿ Here is a checklist for you to reference for your task: ¡checklist¿ XXX ¡/checklist¿ [Task instructions] You are an experienced lawyer, your task is to meticulously review contracts currently under negotiation from the perspective of the Supplier party. Your intended audience is internal counsel, and the contracts you are reviewing are standalone agreements between the parties. These contracts are a mix of US and NZ documents. Your goal is to identify any legal issues that could potentially affect your client’s legal rights or obligations such as language in the contract which does not meet the standard or requirement of the party you represent. These issues could be points of contention, ambiguities, potential risks, or compliance matters that could lead to a dispute or liability. Your analysis should be based on the following: \- The document scenario given for each contract, which provides background context on both parties involved in the agreement including the size, industry, and product/service of the parties and any specific areas of concern in the contract. \- The provided contract. \- The provided checklist for the client you represent, which lists the items they deem important in the contract. These items, or ’checks’, indicate the legal standard or requirement of the party you represent. For each section of the provided contract, carefully analyze it and identify ALL of the checks listed in the provided checklist. You ALWAYS follow these guidelines when writing your response: ¡guidelines¿ Format your answer as a JSON object that includes an array of JSON objects, each representing a different check from the checklist. Each of these JSON objects MUST include the following keys: 1\. ’checklist_Numbering’: The numbering of the check from the checklist. Please STRICTLY adhere to the EXACT numbering format used in the checklist. 2\. ’contract_Lines’: An array of sentences or clauses in the contract that corresponds to the check. If the contract does not contain a specific sentence or clause related to the check, then leave this key blank. 3\. ’explanation’: A comprehensive and descriptive explanation of how the contract meets/fails to meet the legal standard in the check. 4\. ‘check_Met’: Indicate whether the check is ‘Met’/‘Not met’. If the contract_Lines key is ’null’, then indicate ’Not met’. 5\. ‘assumptions’: List any additional assumptions that were necessary to answer this check. Important: \- Provide a separate JSON object for EACH check. \- Ensure each JSON object is formatted in the following order: ’checklist_Numbering’, ’contract_Lines’, ’explanation’, ’check_Met’, ’assumptions’. \- Ensure to adhere strictly to the structure and naming of the keys as described above. \- Remember to thoroughly reread the contract and treat each check independently for accurate identification. \- The context provided in the document scenario should guide your review process. \- Provide the JSON response immediately without preamble. ¡/guidelines¿
Table 7. System Prompt: Claude 2.1

###  A.2. Sample of Incorrect LLM determinations

In the course of our research, we observed that emergent LLMs, particularly
GPT 4-1106, exhibited a level of accuracy in identifying legal issues that was
on par with that of Junior Lawyers and LPOs. A notable finding, as detailed in
Table A.2, was the trend across these groups showing a preference for
precision over recall. This tendency often led to an under-identification of
legal issues, marked by a frequent conclusion of non-existence of issues where
they actually were present.

This phenomenon was especially pronounced in the LLMs, where a significant
portion of inaccuracies was linked to the misinterpretation of specific
wordings. The examples in the aforementioned table illustrate these
inaccuracies, which were largely due to challenges in grasping the nuances of
legal language. In comparison, the Senior Lawyers, serving as the benchmark
for this study, showed exceptional care in interpreting subtle differences in
language, indicating a sophisticated understanding of semantic variations.
This juxtaposition between the performance of LLMs and human experts offers
valuable insights into the current capabilities and limitations of LLMs in
legal issue determination, highlighting areas for potential improvement and
further research.

o X[1] X[3] X[1] X[2] Required Check | Contract Reference | Ground Truth Response | LLMs Response
---|---|---|---
Immediate written notice is required in the event of data or security breach | 6.2 Notifications. Each party shall promptly notify the other party of any actual or suspected misuse or unauthorised disclosure of the other party’s Confidential Information. | Standard is not met as it does not specify the requirement for notice to be written. | The contract requires each party to promptly notify the other party of any actual or suspected misuse or unauthorized disclosure of the other party’s Confidential Information. - GPT 3.5 The contract requires immediate written notice in the event of a data or security breach. - text-bison
Publicity: Ensure written consent is required for any public announcements or use of logo or branding. | 15.3. Public Announcements. Neither Party shall issue or release any announcement, statement, press release, or other publicity or marketing materials relating to this Agreement or, unless expressly permitted under this Agreement, otherwise use the other Party’s trademarks, service marks, trade names, logos, domain names, or other indicia of source, association, or sponsorship, in each case, without the prior written consent of the other Party, which consent shall not be unreasonably withheld; provided, however, that we may, without your written consent, include or display your name, logo and other indicia in our lists of current or former customers in promotional and marketing materials. | Standard is not met as not all publicity requires written consent | Section 15.3 states written consent is required for any public announcements or use of logos/branding, meeting this requirement. - Claude 2.1
Table 8. Examples of Determination missed by LLMs


0 Research

# Beyond English: Centering Multilingualism in Data Visualization

Noëlle Rakotondravony 44footnotemark: 4
University of Konstanz e-mail<EMAIL_ADDRESS>University of
Antananarivo
Worcester Polytechnic Institute  Priya Dhawka 44footnotemark: 4
University of Calgary
University of Washington  e-mail<EMAIL_ADDRESS>Melanie Bancilhon
Washington University in St. Louis e-mail<EMAIL_ADDRESS>authors
contributed equally to this work.

###### Abstract

Information visualization and natural language are intricately linked.
However, the majority of research and relevant work in information and data
visualization (and human-computer interaction) involve English-speaking
populations as both researchers and participants, are published in English,
and are presented predominantly at English-speaking venues. Although several
solutions can be proposed such as translating English texts in visualization
to other languages, there is little research that looks at the intersection of
data visualization and different languages, and the implications that current
visualization practices have on non-English speaking communities. In this
position paper, we argue that linguistically diverse communities abound beyond
the English-speaking world and offer a richness of experiences for the
visualization research community to engage with. Through a case study of how
two non-English languages interplay with data visualization reasoning in
Madagascar, we describe how monolingualism in data visualization impacts the
experiences of underrepresented populations and emphasize potential harm to
these communities. Lastly, we raise several questions towards advocating for
more inclusive visualization practices that center the diverse experiences of
linguistically underrepresented populations.

Human-centered computingVisualizationVisualization design and evaluation
methods

##  1 Introduction

Visualizing data is fundamentally about communicating information to viewers.
To this end, designers and researchers often incorporate elements such as
descriptive text, and symbols to more effectively communicate the information
being represented. For instance, including human-readable text in natural
language to explain a data visualization is a nearly universal practice in
visualization research and design communities [15]. With the majority of
human-computer interaction researchers, including those working in information
visualization, being based in English-speaking WEIRD countries [19], English
is used extensively as the language of choice in data visualization research.
However, researchers and designers outside of predominantly English-speaking
countries have long been using data visualizations to communicate information
across the globe, in a variety of non-English speaking cultures.

As online and hybrid conferences have widened access to the visualization
research community in recent years, it has become more apparent that existing
practices within visualization research and design (such as monolingualism)
may contribute towards excluding underrepresented and under-served groups from
the visualization research community. Despite ongoing efforts to foreground
inclusion and equity in our research and design practices as a community, we
have often neglected language as a factor that may influence who gets to
participate in and benefit from visualization research.

Hence, in this paper, we ask:

  * •

What is the impact of monolingual, English-speaking visualization research
practices on underrepresented communities, such as individuals from non-
English speaking cultures?

  * •

How do monolingual visualization practices further the exclusion of already
underrepresented and under-served communities in visualization research?

We focus specifically on language used for descriptive text, human-readable
captions, and communication with data visualizations. We illustrate our
discussion with a case study on the use of two non-English languages and their
interactions with data visualization in Madagascar where multilingualism is a
result of colonization, imposing distinct languages for colloquial
communications and for formal instruction. We argue that simple measures, such
as translating English text in data visualizations to local languages, are
accessible but fundamentally ignore the broader issue at hand — that
monolingual research practices hide how the lived experiences of viewers may
influence the ways in which they interact with visualizations. We end with a
call to action to the visualization research community to examine our current
practices for exclusionary effects and provide suggestions for potential
research directions to increase multilingualism in visualization research.

##  2 Related Work

We describe previous research within information visualization and human-
computer interaction (HCI) with a focus on critical data visualization and
HCI.

###  2.1 Critical Data Visualization and HCI

Scholars working in critical data visualization and HCI advocate for equitable
research practices when working with marginalized and underrepresented
populations. Within the context of information visualization, Dörk et al. [12]
outline a number of ways in which researchers can question how their values
and assumptions pervade their existing research practices. Specifically, they
suggest research practices that prioritize disclosure, contingency, plurality
and empowerment [12].

On the HCI side, D’Ignazio and Klein propose a data feminism framework for
researchers and practitioners to question and disrupt inequitable research
practices when working with marginalized populations [11]. They urge
researchers to challenge assumptions about gender binaries, power structures,
and the objectivity of data while incorporating context and the diverse lived
experiences of the people behind the data [11]. In their call to action to the
HCI community, Ogbonnaya-Ogburu et al.propose ways to adapt critical race
theory learnings to HCI research. They elucidate how racism is pervasive in
existing research practices, and advocate for race-conscious research
practices [24].

Work within critical visualization has also started considering the ethics of
unquestioned research practices on audiences. For instance, Correll highlights
potential side-effects of standard visualizations such as viewers feeling
alienated from the people whose data is being represented [8]. Meanwhile, in
their study of attitudes towards data in rural Pennsylvania, Peck et al. [29]
found that one-size-fits-all approaches in visualization research tend to
neglect certain demographic populations and that individuals’ personal lived
experiences strongly influence how they relate to data visualizations.

##  3 The Integration of Visualization & Text Should Consider Language

When it comes to information communication, the language surrounding
visualization has been shown to be equally important as the chart, if not
more. Several studies have shown that integrating text and visualization can
improve recall [5]. People have also reported to prefer formats that contain
text [32, 3]. The integration of text and charts has been examined across a
number of applications, including high-stake tasks such as Bayesian reasoning,
which is notoriously difficult for most people.

Bancilhon et al.found evidence that integrating text and visualization reduced
cognitive load when solving a Bayesian task compared to when showing either
representation alone [4]. However, Ottley et al.has shown that text,
visualization and a combination of the two formats elicit the same speed and
accuracy in Bayesian tasks [27]. When examining eye tracking patterns during a
study task, they found that participants likely identified critical
information more effectively using visualization, but extracted information
more effectively from text [26]. These mixed findings show that the
integration of text and chart is not straightforward. While several other
applications have shown that integration techniques can improve recall and
accuracy [35], techniques such as linking and hovering have shown no
measurable improvement in the context of Bayesian reasoning [22]. More
research needs to be conducted to examine the best techniques to integrate the
two such that their respective benefits are optimized.

While format and layout can impact reasoning when integrating text and charts,
semantics can also play a role. Stokes et al.found that texts that describe
statistical or relational components of a chart lead to more takeaways than
texts that describe elemental or other components [33]. Other studies have
shown that the title of a visualization can influence its interpretation more
than the visualization itself [16].

A majority of the studies examining the effects of integrating chart, text,
and language have been conducted in English. A study by Rakotondravony et
al.showed an example of how the verbalization of quantitative probability
through visualizations can vary across French, Arabic, English, German, and
Mandarin [30]. However, further studies are still needed to build evidence on
how text interacts with visualization in under-represented languages.
Moreover, when evaluating reasoning using charts, there needs to be important
considerations for individual differences in visual literacy, which can be
influenced by language and culture. There lacks research on developing valid
and inclusive visualization literacy assessments. Pandey et al., who developed
a shortened version of the visualization literacy test, posit that more
research needs to be done to adopt the MINI-VLAT in different languages [28].

In emerging fields such as AI and NLP, there is an increasing amount of
research on text and visualization integration. Several researchers have
investigated ways to assist the visually impaired or people with low
visualization literacy via techniques such as chart summarization [23]. Other
researchers like Shen et al.have examined how to automate chart generation
using natural language and conducted a survey of natural language interfaces
for data visualization [31]. The lack of consideration for language persists
across these fields, where most of the development and evaluation is done in
English.

##  4 Madagascar: A case study of language, data visualization, in non-English

In this section, we illustrate the interplay between language and data
visualization in a non-English speaking context. We describe how the
affordance of language can impact the expressiveness of data visualizations,
and challenge the different sub-areas of visualization research. The case of
Madagascar that we highlight is likely common to countries and communities
speaking more than one official language, and for which the spoken colloquial
languages are different from that of instruction. In Madagascar, Malagasy is
the country’s national language. It is spoken and understood overall in its
different dialectical variations [6]. In the post colonial era, French was
instituted as language of instruction, and was reaffirmed as a de jure
official language by the country’s constitution in 2007. In public schools
through grade five, Malagasy is the language of instruction for all subjects,
whereas in high school it is for the subjects of history and Malagasy language
only. Instruction for science and the other subjects is delivered in French.

While the majority of people have been in contact with the French language in
their primary years, most rarely, if not never, practice it after school. As
of 2022, it is estimated that $26.5\%$ of Malagasy population older than 10
years old speak French at least at a colloquial fluency [20]. Beyond being the
predominant language of education in primary and higher education, French is
also used and referenced for most technology-related topics on the island111Le
Français langue d’enseignement, accessed: 2023-07-10.

###  4.1 Communicating With Data Visualization

Figure 1: Stimulus used in Andrianarivony et al.’s exploratory study
investigating how bilingual audiences use their native and secondary languages
with visualizations in Madagascar [1]. Image courtesy of the authors.

Data visualization today is about communicating (quantitative) information
through static or interactive visuals and graphs. In line with existing
observations of the performance of audiences from bilingual settings in
mathematical reasoning [2, 34] and in HCI [13, 25], the prevalence of French
in Madagascar, especially in science education and technology, can influence
the statistical or quantitative reasoning with technological instruments, such
as interactive interfaces, in Madagascar even among an highly educated
audience.

In an exploratory study of how French and Malagasy are used to discuss data
and data visualization in Madagascar, Andrianarivony et al. [1] asked native
speakers to describe their approach for completing basic visualization reading
tasks such as finding a specific value, and to verbalize their takeaways from
a data visualization in either Malagasy or French only. The stimulus combines
a line chart and a bar chart depicting the evolution of Madagascar’s GDP and
the recorded number of social conflicts in the country. Participants in the
study hold at least a higher education degree and were fluent in both
languages.

Results highlighted the easiness of using French for participants to talk
about how they interact with the charts, explicitly naming the different
elements of the visuals (such as barchart, linechart, …), the actions that
they took to explore it (such as clicking, zooming, …), and their note on the
data (such as increase, decrease, …). On the other hand, participants in
Malagasy showed difficulty to verbalize their analysis of both the chart and
its underlying data, essentially in finding the necessary terminologies which
they often already know in French. While the knowledge of French appears to be
helpful, the lack of linguistic tools for verbalizing data and visualizations
in their native language excludes approximately $74\%$ of the Malagasy
population who do not speak French in Madagascar from benefiting from visual
and graph-based communications. It also reflects the understudy of non-English
languages in visualization and interaction studies.

###  4.2 Data Visualization Literacy

Beyond challenging the use of data visualization for communication, a lack of
multilingualism in research also directly impacts visualization literacy. As
visualization literacy gains interest among researchers, visualization
literacy assessment tests constitute useful instruments to help develop future
design and methodologies that help audiences better read and make sense of
data visualizations. However, most approaches in the research literature are
in English, and assume underlying characteristics of the test takers that are
not inclusive, especially for those from non-WEIRD contexts.

For the majority of potential visualization users in Madagascar, knowledge of
English constitutes a limitation to access the off-the-shelf assessment tests.
Test items extensively refer to data, charts elements, and the use of
statistical analyses. They require fluency in charts naming conventions,
mathematics, and data reasoning, and their contents are sourced from materials
that do not necessarily align with other educational systems and curricula
outside that of the country in which they were developed. For example, for
VLAT [18], test-takers are expected to possess graph reading and
interpretation skills comparable to the K-12 curriculum. In Madagascar,
students are extensively exposed to graph reading at a much later stage in
their course of studies. This difference in curriculum required for data
visualization literacy and tests might therefore impact participants’
performance, potentially leading to incorrect conclusions.

While translation from English constitutes a good alternative to the lack of
multilingualism of data visualization, more aspects of data visualizations
beyond text translation still need to be considered and culturally adapted to
ensure adequate evaluations. As Peck et al. [29]’s study showed, the way
individuals relate to data and visualizations influence their engagement with
a visual representation. While we hypothesize that adapting the western-
centered visualization literacy test items to local contexts can increase test
takers’ interest in the data, more evidence and studies are still needed to
understand how such engagement may impact their visualization literacy scores.

In this illustrative case of Madagascar, we argued how a lack of
multilingualism in data and information visualization research can exclude
underrepresented communities from participating in research studies. Similar
to Madagascar, the described duality in language is shared among many other
countries and post colonial countries, where the inherited or imposed language
of instruction does not align with the colloquial language, creating among
other challenges, the unspoken divide between who can benefit from data and
information visualization and who cannot. While we acknowledge that this issue
is at core systemic, we argue that the information visualization research
community can actively contribute to closing the gap.

##  5 What is the impact of monolingualism and what can we do about it?

Data visualization and text, hence language, are intrinsically linked [15].
The interplay of visualizations along with integrated text are many times used
to communicate critical information that can impact decision-making across
different levels, and to promote social good. Reflecting on the case in
section 4, a direct harm that the lack of multilingualism in our research
practices can pose is the exclusion of marginalized populations.

###  5.1 The Exclusion of Marginalized Populations

Data about marginalized and underrepresented populations is often incomplete,
does not exist or requires additional care for the privacy and anonymity of
the people in the dataset. Available data often focus on their experiences of
exclusion and suffering. For instance, anthropographics (human-shaped
visualizations) of marginalized and underrepresented populations are widely
used within English-based information visualization research in charitable
giving settings or in “humanitarian” visualizations [7, 21]. However,
marginalized and underrepresented populations are rarely involved in
visualization research projects, even the ones directly impacting them [9]. In
the case of English-based anthropographics, individuals belonging to the
populations being represented (refugees, victims of violence) may not even be
able to meaningfully interact with these visualizations due to language
barriers. Hence, it is essential that visualization research projects about
marginalized and underrepresented populations find creative ways to involve
said groups of people in the research process [10].

Moreover, the groups that are historically underrepresented from research are
mostly from non-WEIRD countries or countries with under-developed resources
for research. When their particular needs are not reflected in research
carried out in WEIRD countries for use globally, any claim for
generalizability of results perpetuates the imposition of culturally non-
adapted approaches, misleadingly labelled as development efforts. For
instance, as data visualization literacy is a fundamental skill and the basis
of the development of methods for teaching data visualization, the exclusion
of marginalized groups’ lived experiences can result in materials that are
inadequate for local education systems beyond WEIRD countries. This inadequacy
is often more prevalent in countries where the languages of instructions are
not the native and spoken languages for communication [34].

###  5.2 Call to Action

We propose actions for the information visualization community towards
increasing the involvement of linguistically underrepresented populations in
research. We emphasize that our propositions are preliminary ideas intended to
start a community conversation around our existing research practices, and
that several of these ideas will require additional long-term collective
thinking and active support before implementation.

#### Participatory Design and Visualization

Although information visualization research projects lean heavy on the
interaction side, as a community, we often engage with research participants
through short studies and in laboratory settings. This is in part due to the
nature of our field of research being largely quantitative. However, as rich,
qualitative studies become more common in visualization research, we can
consider investigating how the lived experiences of viewers influence how they
interact with data visualizations. For instance, future work can explore the
(side) effects of using English in visualizations for underrepresented
communities by directly involving said communities in visualization design
workshops and investigating their interactions with multilingual
visualizations. Additional research opportunities could focus on data
visualization literacy in multiple languages as well as in countries with
specific linguistic cultures.

#### Leveraging Emerging ML Technologies

With the rapid development of ML technologies, the information visualization
has already started speculating about the potential of incorporating these
technologies within existing visualization workflows. In terms of natural
language, access to LLMs in a variety of languages, faster and on-demand
language translation services can support linguistically underrepresented
populations who may not have access to human translators or where there are
few alternatives to English-based visualizations. We imagine research
opportunities around automated visualization tools with LLM-powered
translation features. Similarly, we imagine visualization design and literacy
tools that provide human-readable descriptions and captions in languages other
than English, that combine multiple languages in a single interface as well as
ones that include expertise from experts outside of data visualization (such
as linguists) to address the challenges of learning additional languages.

#### Expanding Access to the Visualization Community

The COVID-19 pandemic saw the visualization community adapt to remote and
hybrid conferences, which opened up access to communities who previously could
not attend these conferences due to a number of financial, geographical and
border-crossing limitations. Despite the many lessons learnt running
successful remote and hybrid conferences during this time, a return to
“normal”, in-person only conferences can erase much of the progress around
conference accessibility made during the pandemic.

We envision smaller satellite conferences and remote workshops, in local
languages, in under-served countries to increase researcher diversity within
information visualization research and to support languistic diversity. For
instance, within the HCI community, several geographically-bound conferences
such as NordicCHI and AfriCHI already take place throughout the year. Although
the information visualization community is smaller, we imagine a future with
more remote opportunities to be in community with visualization researchers
from underrepresented countries through workshop or paper tracks for specific
geographic regions and dedicated mentorship networks.

Additionally, there has been recent interest within the information
visualization community to examine our publication practices. For instance,
Hao and colleagues [14] examined 32 years of publications and citations at
IEEE VIS venues and found that the majority of cited and referenced papers
were limited to the same subfields of Computer Science. Building on this work,
as a community, we may reflect on our citational practices, as Kumar and
Karusala [17] urge HCI researchers: Why do we cite the way we do in
information visualization research? Who do we cite (and not cite) and why?
Where does the majority of our knowledge in information visualization research
come from and why?

####

Lastly, several challenges may arise when implementing actions towards
inclusive visualization research that go beyond simple translations. For
instance, collaborations can be costly in both material and intellectual
resources. Conducting studies online requires participants to have access to
the necessary technologies. Some training and study configurations require in-
person collaboration for which long distance, international travels are
necessary. International mobility, beyond issues related to its environmental
sustainability, comes with many barriers to the collaborations between
researchers across countries but may particularly impact those from the Global
South with limited border-crossing privileges.

Moreover, conducting data visualization research in languages other than
English requires peer-reviewers who understand the language. While there are
certainly experienced reviewers fluent in non-English languages, the community
still needs to reflect on and discuss the best practices for including under-
represented languages in research while being mindful of the potential of
creating isolated findings that cannot be communicated widely.

Despite these challenges, we strongly advocate for the visualization community
to start this important conversation on increasing linguistic diversity in our
research practices.

##  6 Conclusion

We discussed how monolingual practices in the data visualization community
creates a divide between who gets included in or benefits from visualization
research. We emphasize the importance of text and language in data
visualization by highlighting how their interplay helps in promoting the
access to existing data visualizations, and the inclusion of traditionally
underrepresented populations in data visualization research. Reflecting from
an exploratory study in Madagascar, where the lack of linguistic tools for
data and visualizations impacts how people interact with charts and verbalize
their takeaways, we highlight the potential harms that the exclusion of under-
studied languages can cause — especially when communicating data about and to
marginalized and underrepresented populations. We conclude this position paper
by calling the community’s attention to several discussion avenues towards
increasing the involvement of underrepresented populations in visualization
research through efforts such as participatory design, leveraging emerging ML
technologies and expanding global access to the visualization research
community. Our hope with this work is that data visualizations can be created,
designed, used and accessed by all, especially those who make up the majority
of the global population.

## References

  * [1] H. S. Andrianarivony, T. D. Raharison, I. T. Rakotoniaina, M. T. Ramarokoto, N. Rakotondravony, and L. Harrison.  POSTER: Investigating the Use of Native and Secondary Language with Data Visualization in Madagascar.  IEEE VIS, 2022.  https://osf.io/e4cwz/ \- accessed on 2023-07-10.
  * [2] C. Baker.  Foundations of Bilingual Education and Bilingualism.  Multilingual matters, 2011.
  * [3] M. Bancilhon, R. J. Crouser, and A. Ottley.  Communicating Intel to Decision-Makers: Toward the Integration Text and Charts in Reports.  arXiv preprint arXiv:2302.10885, 2023.
  * [4] M. Bancilhon, A. Wright, S. Ha, R. J. Crouser, and A. Ottley.  Why Combining Text and Visualization Could Improve Bayesian Reasoning: A Cognitive Load Perspective.  In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pp. 1–15, 2023.
  * [5] M. A. Borkin, Z. Bylinskii, N. W. Kim, C. M. Bainbridge, C. S. Yeh, D. Borkin, H. Pfister, and A. Oliva.  Beyond Memorability: Visualization Recognition and Recall.  IEEE transactions on visualization and computer graphics, 22(1):519–528, 2015.
  * [6] L. Bouwer.  Towards a Sociolinguistic Profile of Madagascar: Issues of Diversity.  2005\.
  * [7] J. Boy, A. V. Pandey, J. Emerson, M. Satterthwaite, O. Nov, and E. Bertini.  Showing People Behind Data: Does Anthropomorphizing Visualizations Elicit More Empathy for Human Rights Data?  In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, CHI ’17, p. 5462–5474. Association for Computing Machinery, New York, NY, USA, 2017. doi: 10 . 1145/3025453 . 3025512
  * [8] M. Correll.  Ethical Dimensions of Visualization Research.  In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pp. 1–13. ACM, Glasgow Scotland Uk, May 2019\. doi: 10 . 1145/3290605 . 3300418
  * [9] P. Dhawka, H. A. He, and W. Willett.  Representing marginalized populations: Challenges in anthropographics.  arXiv preprint arXiv:2210.02660, 2022.
  * [10] P. Dhawka, H. A. He, and W. Willett.  We Are the Data: Challenges and Opportunities for Creating Demographically Diverse Anthropographics.  In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI ’23. Association for Computing Machinery, New York, NY, USA, 2023. doi: 10 . 1145/3544548 . 3581086
  * [11] C. D’Ignazio and L. F. Klein.  1\. the power chapter.  In Data Feminism. The MIT Press, 2020.
  * [12] M. Dörk, P. Feng, C. Collins, and S. Carpendale.  Critical InfoVis: Exploring the Politics of Visualization.  In CHI ’13 Extended Abstracts on Human Factors in Computing Systems, CHI EA ’13, p. 2189–2198. Association for Computing Machinery, New York, NY, USA, 2013. doi: 10 . 1145/2468356 . 2468739
  * [13] V. Evers.  Cross-Cultural Understanding of Metaphors in Interface Design.  Proc. Cultural Attitudes towards Technology and Communication, 1998\.
  * [14] H. Hao, Y. Cui, Z. Wang, and Y.-S. Kim.  Thirty-two years of ieee vis: Authors, fields of study and citations.  IEEE Transactions on Visualization and Computer Graphics, 29(1):1016–1025, 2022.
  * [15] M. A. Hearst.  Show It or Tell It? Text, Visualization, and Their Combination.  2023\.
  * [16] H.-K. Kong, Z. Liu, and K. Karahalios.  Frames and Slants in Titles of Visualizations on Controversial Topics.  In Proceedings of the 2018 CHI conference on human factors in computing systems, pp. 1–12, 2018.
  * [17] N. Kumar and N. Karusala.  Braving Citational Justice in Human-Computer Interaction.  In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, CHI EA ’21. Association for Computing Machinery, New York, NY, USA, 2021. doi: 10 . 1145/3411763 . 3450389
  * [18] S. Lee, S.-H. Kim, and B. C. Kwon.  Vlat: Development of a Visualization Literacy Assessment Test.  IEEE transactions on visualization and computer graphics, 23(1):551–560, 2016.
  * [19] S. Linxen, C. Sturm, F. Brühlmann, V. Cassau, K. Opwis, and K. Reinecke.  How WEIRD is CHI?  In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, CHI ’21. Association for Computing Machinery, New York, NY, USA, 2021. doi: 10 . 1145/3411764 . 3445488
  * [20] R. Marcoux, L. Richard, and A. Wolff.  Estimation des Populations Francophones dans le Monde en 2022.  Observatoire Démographique et Statistique de l’Espace Francophone, 2022.
  * [21] L. Morais, Y. Jansen, N. Andrade, and P. Dragicevic.  Can Anthropographics Promote Prosociality?A Review and Large-Sample Study.  In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, CHI ’21. Association for Computing Machinery, New York, NY, USA, 2021. doi: 10 . 1145/3411764 . 3445637
  * [22] A. Mosca, A. Ottley, and R. Chang.  Does Interaction Improve Bayesian Reasoning with Visualization?  In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1–14, 2021.
  * [23] J. Obeid and E. Hoque.  Chart-to-Text: Generating Natural Language Descriptions for Charts by Adapting the Transformer Model.  CoRR, abs/2010.09142, 2020.
  * [24] I. F. Ogbonnaya-Ogburu, A. D. Smith, A. To, and K. Toyama.  Critical Race Theory for HCI.  In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, CHI ’20, p. 1–16. Association for Computing Machinery, New York, NY, USA, 2020. doi: 10 . 1145/3313831 . 3376392
  * [25] E. Onibere, S. Morgan, E. Busang, and D. Mpoeleng.  Human-Computer Interface Design Issues for a Multi-Cultural and Multilingual English Speaking Country—Botswana.  Interacting with computers, 13(4):497–512, 2001.
  * [26] A. Ottley, A. Kaszowska, R. J. Crouser, and E. M. Peck.  The Curious Case of Combining Text and Visualization.  EuroVis 2019-Short Papers, 2019.
  * [27] A. Ottley, E. M. Peck, L. T. Harrison, D. Afergan, C. Ziemkiewicz, H. A. Taylor, P. K. Han, and R. Chang.  Improving Bayesian Reasoning: The Effects of Phrasing, Visualization, and Spatial Ability.  IEEE transactions on visualization and computer graphics, 22(1):529–538, 2015.
  * [28] S. Pandey and A. Ottley.  Mini-vlat: A short and effective measure of visualization literacy.  arXiv preprint arXiv:2304.07905, 2023.
  * [29] E. M. Peck, S. E. Ayuso, and O. El-Etr.  Data is Personal: Attitudes and Perceptions of Data Visualization in Rural Pennsylvania.  In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pp. 1–12, 2019.
  * [30] N. Rakotondravony, Y. Ding, and L. Harrison.  Probablement, Wahrscheinlich, Likely? A Cross-Language Study of How People Verbalize Probabilities in Icon Array Visualizations.  IEEE Transactions on Visualization and Computer Graphics, 2022.
  * [31] L. Shen, E. Shen, Y. Luo, X. Yang, X. Hu, X. Zhang, Z. Tai, and J. Wang.  Towards Natural Language Interfaces for Data Visualization: a Survey.  IEEE transactions on visualization and computer graphics, 2022.
  * [32] C. Stokes and M. Hearst.  Why More Text is (Often) Better: Themes from Reader Preferences for Integration of Charts and Text.  arXiv preprint arXiv:2209.10789, 2022.
  * [33] C. Stokes, V. Setlur, B. Cogley, A. Satyanarayan, and M. A. Hearst.  Striking a Balance: Reader Takeaways and Preferences when Integrating Text and Charts.  IEEE Transactions on Visualization and Computer Graphics, 29(1):1233–1243, 2022.
  * [34] UNESCO - Global Education Monitoring Report Team.  If You Don’t Understand, How Can You Learn?  Technical report, 2016.
  * [35] Q. Zhi, A. Ottley, and R. Metoyer.  Linking and Layout: Exploring the Integration of Text and Visualization in Storytelling.  In Computer Graphics Forum, vol. 38, pp. 675–685. Wiley Online Library, 2019.