longtable Ginwidth=,height=,keepaspectratio

# SurveyLM: a platform to explore emerging value perspectives in Augmented
Language Models’ behaviors

Steve J. Bickley, Ho Fai Chan, Bang Dao, Benno Torgler, Son Tran

Panalogy Lab
Brisbane, 4000
<EMAIL_ADDRESS>
(01.08.2023)

###### Abstract

This white paper presents our work on SurveyLM, a platform for analyzing
augmented language models’ (ALMs) emergent alignment behaviors through their
dynamically evolving attitude and value perspectives in complex social
contexts. Social Artificial Intelligence (AI) systems, like ALMs, often
function within nuanced social scenarios where there is no singular correct
response, or where an answer is heavily dependent on contextual factors, thus
necessitating an in-depth understanding of their alignment dynamics. To
address this, we apply survey and experimental methodologies, traditionally
used in studying social behaviors, to evaluate ALMs systematically, thus
providing unprecedented insights into their alignment and emergent behaviors.
Moreover, the SurveyLM platform leverages the ALMs’ own feedback to enhance
survey and experiment designs, exploiting an underutilized aspect of ALMs,
which accelerates the development and testing of high-quality survey
frameworks while conserving resources. Through SurveyLM, we aim to shed light
on factors influencing ALMs’ emergent behaviors, facilitate their alignment
with human intentions and expectations, and thereby contribute to the
responsible development and deployment of advanced social AI systems. This
white paper underscores the platform’s potential to deliver robust results,
highlighting its significance to alignment research and its implications for
future social AI systems.

_K_ eywords Social Science • Social AI • Large Language Models • Augmented
Language Models • Alignment Research • Survey Method • Experimental Method

##  1 Motivation

We built SurveyLM with the following motivations in mind:

  * •

Artificial Intelligence (AI) alignment in complex social context is
_important_ , especially when AI systems make decisions or assist people in
making decisions in complex social settings where there may be no true “right”
answer or where an answer is heavily dependent on contextual factors and
future uncertainties (specific information in real-world situations).

  * •

The application of survey and experimental methodologies, which are
extensively utilized to study social behaviors, can greatly enhance the
effectiveness of language model evaluation frameworks. These approaches can
provide valuable insights into the behavior and alignment of the models,
particularly when implemented in a _systematic_ manner.

  * •

Augmented language models (ALMs) are AI systems that currently have the most
advanced general reasoning capability and are increasingly functioning within
diverse and intricate social scenarios, exhibiting emergent behaviors that
were previously unexpected. Therefore, it is desirable to apply survey and
experimental methodologies to not only unravel the factors influencing these
emergent behaviors, but also to ensure these ALMs are adapting and aligning
with human intentions and expectations, thus contributing to the responsible
development and deployment of advanced AI systems.

  * •

Being able to learn from ALM’s feedback to _improve_ survey/experiment designs
is a great and underexplored application of ALMs, thus helping researchers to
construct high quality survey frameworks at a fraction of the resources and
time that would be otherwise required (past approach where humans have been
doing everything).

  * •

SurveyLM aims to be the first platform to deliver robust results for the above
value propositions.

##  2 Introduction

Our new platform, SurveyLM, aims to explore emerging value perspectives in
ALMs (Mialon et al., 2023) through decision premises and contextuality in
complex social settings. We chose to focus on complex social context because
these environments present a wide array of situations and (social) dilemmas
that help assessing the multifaceted and nuanced value judgments that ALMs
have to make in the real world. As a result, we can better understand their
decision-making mechanisms and the underlying value systems they rely on.

Since our focus is on complex social decision situations that have no obvious
binary classification of being totally right or wrong, we adopt an approach
that enables social science researchers to systematically evaluate ALMs’
decision dynamics in a diverse range of social contexts using different
measures, instead of benchmarking ALMs capabilities against a simplified set
of sequential decision making tasks often used in the field of AI (Shridhar et
al., 2021). While these tasks provide a clear context to evaluate whether an
ALM is right or wrong in making some decisions, or in carrying out some
specific tasks, they are far from being adequate to capture real-world
decision processes that social science researchers are concerned with.

Our approach to exploring ALM’s behaviors, and their value perspectives, is
based on presenting the models with complex sets of survey questions and
experimental settings, and then analyzing their responses in various social
contexts. We opted for a survey-based approach as it has been proven to be an
effective method for probing and value elicitation of ALMs (Arora et al.,
2023; Binz & Schulz, 2023). Human value elicitation has always been an
important research avenue (Haerpfer et al., 2021; Hofstede, 2005), as an
individual’s identity and values often manifest in the choices and decisions
they make. Given the increasing role of ALMs as agents interacting with humans
who act as principals (i.e., principal-agent relationship in nature), it
becomes crucial to comprehend the underlying values of these models and their
implications for recommendations and decision-making processes, which may or
may not align with those of human principals. By allowing the models to
respond to a series of contextualized questions, we can gain valuable insights
into how they navigate ethical scenarios and prioritize differing values for
different principal profiles and characteristics (e.g., age, gender,
ethnicity, country of residence, etc.).

The first key aspect we address is the question of why understanding ALM’s
value perspectives matters in alignment research. By examining the context-
dependent values that drive ALMs’ behaviors, we gain valuable insights into
the ethical and moral frameworks and traces that shape their decision-making
processes (as well as our own). These insights can help us align ALMs with
(context-specific) human values and promote responsible AI development (Tamkin
et al., 2021).

Defining measures of values is another crucial aspect of our platform,
SurveyLM. We recognize that values are multifaceted and subjective, making
their quantification challenging. Human values, as well as AI alignment to
these values, evolve over time and iterations, necessitating a continuously
adaptive approach. Through rigorous research and analysis, we aim to develop
and translate robust methodologies to define and measure values in the context
of ALMs, and to discern trends and shifts in ALMs value systems and alignment.
This will provide a solid foundation for studying and understanding their
behavior patterns in complex social settings.

To effectively explore emerging value perspectives in ALMs’ behaviors, we
employ state-of-the-art ALM simulation techniques. By simulating ALMs in
realistic scenarios, we can observe their decision-making processes and
identify any evolving biases or value systems. This knowledge is crucial for
staying ahead of potential risks and ensuring the responsible development of
AI technologies.

The potential applications of our platform extend beyond the fields of
behavioral economics, cognitive psychology, management or market research. We
envision leveraging the insights gained to help drive social AI alignment and
design in research, industry and community alike. By aligning AI systems with
human values, we can (co-)create social AI systems that complement and enhance
lives, benefiting society as a whole.

Overall, our framework proposed provides an effective and user-friendly
platform for researchers to interact with GPT models. It upholds crucial
principles such as user-first orientation, systematic and consistent
operations, privacy respect, and safety in worst-case scenarios. Furthermore,
it features robust components including a simple front-end interface, a real-
time metrics monitor, secure databases, efficient agents, and a smart requests
manager, all engineered to make complex survey research a breeze. We hope that
this platform holds the potential to revolutionize how researchers engage with
GPT models, offering a unique blend of flexibility, efficiency, and
reliability.

In conclusion, our platform SurveyLM offers a unique opportunity to explore
and understand the emerging value perspectives from ALMs’ behaviors. By
addressing the questions of why values matter, defining measures of values,
and simulating ALMs in complex social settings, we can generate insights that
will help shape the future of AI development. With a focus on social AI
alignment and design, we aim to create long-lived, complex social AI agents
and systems that align with human values, promoting a responsible and
beneficial integration of AI into industry and society.

##  3 Platform Design

The design of our platform is guided by several key principles centered around
efficiency, flexibility, and user privacy.

Put Researchers First: This principle is the cornerstone of our design
framework. The goal is to make the researcher’s interaction with the platform
as intuitive and seamless as possible, reducing the complexity of the
underlying operations. From parameter configuration to document and data
processing, we aim to create an environment that empowers the user to focus
solely on their research questions and outcomes. To achieve this, complex
prompt engineering and API request handling will be handled internally by the
platform, abstracting the process and reducing the technical knowledge
necessary to operate the tool effectively.

Respect Research Logic: We design the platform in such a way that, despite
layers of operational complexity, keeps the research logic intact by
translating parameter configurations into prompt structures that ensure LLMs’
responses would be shaped by only those configurations. This is achieved by
giving researchers freedom in defining transparently what comes into and what
comes out of a simulation process. Whereas, the platform focuses on building
optimal prompt structure and request handling by using researchers’ inputs
with minimal additional content needed to interface consistently with LLMs.

Be Systematic Yet Flexible: A systematic approach to parameter configuration
is crucial to accommodate a diverse range of research scenarios. With the
flexibility to modify parameters as per the needs of the study, the platform
allows researchers to experiment and customize the models’ responses, thereby
facilitating a broader spectrum of research.

Embrace Consistency and Dynamism: Consistency is paramount for any research
work. As such, the platform will ensure that the GPT models produce consistent
responses to survey questions in terms of answer formats and compliance with
other instructions. At the same time, we recognize the evolving nature of AI,
incorporating the dynamic aspect of learning and growth in AI models to our
consistency ethos. Balancing these elements enhances not just the
reproducibility, but also the adaptive validity and reliability of our
research findings.

Respect Privacy and Data Control: With the rise of data breaches, privacy
protection is a critical concern for users. On our platform, we uphold the
principle of data control, retaining user data exclusively until such time as
the user decides to delete it. Upon deletion, all data will be purged except
for the basic user credentials. This ensures that users’ privacy and data
security are upheld without compromising the platform’s functionality.

Safe Worse Case: The platform will provide robustness against system errors,
particularly those caused by API rate limits. Should an error occur, the
platform will ensure that all simulation data leading up to the error are
preserved. Additionally, any uncompleted operations will also be recorded and
sent to the user for further decision making. This principle allows for
seamless recovery, ensuring users’ work is never lost and fostering trust in
the platform’s reliability.

Based on these principles, SurveyLM was built with several components as shown
in Figure 1.

Figure 1: Key components of the platform design

The platform’s architecture consists of several key components designed to
enhance user interaction and facilitate research.

Frontend Interface: The interface serves as the gateway for users to interact
with the platform. It allows for easy parameter configurations, document
uploads, and provides real-time monitoring metrics and a data viewer. The
interface is designed to be user-friendly and intuitive, minimizing the
learning curve for new users.

Databases: Two types of databases are incorporated: one for storing user
credentials and another for per-user simulation data. This separation is
designed to enhance data security and privacy, as well as to provide a smooth
user experience by keeping all relevant data at hand.

Backend interface: This is where a request is started and ended. The backend
interface is responsible for authenticating users, processing simulation input
data and sending survey data to the frontend listener. It executes these tasks
through appropriate interface calls to the user credential database and the
simulation component.

Simulation: This is where a survey process starts and ends. The simulation
component is responsible for processing uploaded documents, constructing,
creating request data, initialize the concurrent request handler, and update
metrics data for the frontend metrics listener. Each agent possesses a profile
constructed from parameter configurations and a set of interfaces for
interaction with the metrics monitor, databases, and GPT models. The
simulation component serves as a conduit, ensuring efficient and error-free
data flow within the platform. It also internalizes the complexity of prompt
engineering and related operations needed to ensure that the LLMs behave
consistently in term of the structure of their response.

Request data: This component stores agent data and request data in an optimal
format created by the simulation component. It also serves as a streaming data
source to avoid unnecessary, and costly, memory consumption that would take
place while concurrent api calls are made for each agent’s request.

Request Handler: This component manages the complexity of prompt engineering
and concurrent API calls. It allows the simulation process to communicate with
the GPT models seamlessly and ensures all prompts are correctly structured and
successfully sent. It also manages API calls to reduce the chance of hitting
rate limits frequently while ensuring that a simulation on large scale (e.g.
one with 1000 agents) can run as fast as it could. Another important feature
of this component is the handling of retry failures due to rate limit and API
request errors. In the face of these failures, the handler will automatically
process the last safe survey results and return them in proper formats for the
agents to process further.

##  4 Applications

With the emergent capabilities of Large Language Models (LLMs) and the ongoing
development of Augmented Language Models (ALMs), understanding the underlying
premises that govern their behaviors and decision-making processes becomes
increasingly crucial, especially when they operate in complex social settings
and have real-world impact. This understanding is essential to ensure the
alignment of social AI with human values.

ALMs, which are essentially built on top of pre-trained LLMs like OpenAI’s
gpt-4, incorporate various elements such as retrieval plug-ins, different
learning techniques (few-shot), diverse prompting methods (such as chain-of-
thought, self-model, and contextual prompts), functional coding, and
integration with other modalities like voice, vision, and sound (Mialon et
al., 2023). Additionally, future iterations of ALMs are expected to
incorporate different AI techniques such as reinforcement learning and
symbolic logic, enhancing their knowledge organisation, reasoning, and
learning capabilities.

Researchers have recognized the potential of LLMs as valuable tools to study
and probe the human mind and society (Arora et al., 2023; Binz & Schulz, 2023;
Korinek, 2023; Miotto et al., 2022), given their training on vast amounts of
human data and their ability to generate human-like text. Scholars have also
discussed their potential in simulating human subjects (Aher et al., 2023;
Horton, 2023; Park et al., 2022). Consequently, researchers from various
disciplines such as behavioral economics, cognitive psychology, social
psychology, linguistics have now started to investigate LLMs’ behavior and
decision-making processes and its use as a scientific tool. However, the
procedures and tuning of LLMs (e.g., temperature, context window, prompt
context and structure) for judgment and evaluation of alignment are not yet
standardized or consistently applied across studies. Moreover, digital
literacy and programming skills continue to present significant obstacles for
many researchers to implement such research robustly at scale, particularly
those in behavioral economics and the social sciences.

Considering the fast-paced nature of research and development in AI at the
moment, it is essential to also extend our focus beyond pre-trained LLMs and
consider the emergent capabilities and value systems of ALMs within various
different social contexts. ALMs are increasingly augmented with additional
tools and various prompting techniques, spanning different context windows and
incorporating other modalities. Furthermore, these ALMs are now actively
performing real-world actions. Calling a tool in the context of ALMs often
involves having an impact on the virtual or physical world and observing the
resulting effects, which are typically integrated into the ALM’s ongoing
context. Moreover, ALMs are increasingly engaging in delegate actions such as
carrying out transactions on our behalf or responding to customer queries and
emails in human-like ways.

By acknowledging the advancements in ALMs and the complex nature of their
interactions with the world, we can gain a comprehensive understanding of the
premises underlying their behaviors and decision-making processes by
benchmarking through survey and experimental methods. This knowledge is
crucial for ensuring the development of responsible and aligned social AI
systems that reflect human values for the benefit of all humankind.

SurveyLM facilitates this exploration in an easy and intuitive manner. It
empowers researchers to investigate the behaviors and decision-making of LLMs
and ALMs in a robust and systematic way, using an easy-to-use, click-and-play
online interface.

The SurveyLM platform is highly versatile and adaptable to a multitude of
decision-making scenarios and experimental settings. Here are just some
potential applications:

  1. 1.

Survey Data Generation: Simulate agents to answer an array of survey
questions, creating a rich dataset that can mimic diverse, human-like
responses to these questions. This can be particularly useful in preliminary
research phases, hypothesis testing, or for enhancing existing datasets.

  2. 2.

Allocation Games: Simulate scenarios where agents need to make decisions about
resource allocation. This could involve public goods games, bargaining games,
prisoner’s dilemma scenarios, or other economic games that explore
cooperation, competition, and negotiation.

  3. 3.

Cognitive Psychology Experiments and (Multi-Stage) Scenarios: Simulate
cognitive tasks and adaptation processes to understand decision-making
processes, memory, attention, perception, and problem-solving strategies.

  4. 4.

Social Interaction Simulations: Model and simulate complex social interactions
within groups, examining phenomena like group dynamics, communication
patterns, social influence, and conformity.

  5. 5.

Consumer Behavior Analysis: Simulate buying decisions of agents to understand
patterns in consumer behaviour, product preferences, and purchase rationales.

  6. 6.

Policy Impact Analysis: Simulate reactions to new policies or regulations to
gauge potential public response and impact (i.e., to understand emergent macro
behaviors).

  7. 7.

Risk-taking (Multi-Stage) Scenarios: Study decision-making under uncertainty
or risk, such as in gambling or investment scenarios.

  8. 8.

Health-related Decision-making: Explore choices related to health behaviors,
preventative measures, treatment options, etc.

  9. 9.

Educational Settings: Understand learning behavior, knowledge acquisition, and
responses to different teaching methods.

  10. 10.

Environmental Decision-making: Simulate agent decisions about resource usage,
conservation behaviors, and responses to environmental policies.

By simulating decision-making across a large spectrum of randomized agent
demographic attributes (e.g., age, gender, education level, personality,
etc.), SurveyLM provides a unique platform to investigate even the most
sensitive, challenging, or otherwise taboo subjects/topics that are typically
difficult to broach with human research participants, such as sexuality, drug
use or life-event shocks. By probing these areas in a simulated environment,
we leverage the potential of ALMs to explore sensitive topics (e.g., health,
social, economic, ethical, etc) in a safe and ethical environment.

SurveyLM’s potential is vast when it comes to exploring, e.g., sensitive,
heated, challenging or taboo topics. Here are a few examples:

  1. 1.

Mental Health: Explore attitudes and behaviors around mental health issues,
which are often stigmatized or misunderstood.

  2. 2.

Addiction: Understand the complex dynamics of substance use and addiction, and
the social attitudes towards these subjects.

  3. 3.

Sexuality: Explore attitudes towards various sexual orientations, gender
identities, or sexual behaviors.

  4. 4.

Religion and Faith: Assess how individuals interact with religious beliefs and
practices, including perceptions of other religions.

  5. 5.

Political Extremism: Investigate the drivers of extreme political views,
intolerance, or radicalisation.

  6. 6.

Race and Ethnicity: Examine attitudes towards different races and ethnicities,
including instances of bias, discrimination, and prejudice.

  7. 7.

Immigration: Assess perceptions and misconceptions about immigration and
immigrants.

  8. 8.

Body Image: Explore attitudes and pressures around body image and physical
appearance.

  9. 9.

(Economic) Inequality: Understand perspectives on wealth distribution,
poverty, and economic disparity or inequalities in general.

  10. 10.

Climate Change and Natural Disasters: Investigate attitudes and beliefs about
climate change and natural disasters, environmental responsibility, and
sustainability.

These topics are typically difficult to discuss openly, but SurveyLM provides
a secure and confidential platform for exploring them in depth.

We are confident that as SurveyLM evolves and adapts over time, it will
uncover numerous other promising areas of application. The examples provided
here represent just a fraction of the platform’s potential, highlighting only
the possibilities we, and others in the field, have identified to date. With
the rapidly advancing landscape of social science research, there is no doubt
that the scope of SurveyLM’s application will continue to expand, revealing
even more groundbreaking possibilities in the future.

##  5 Future Developments

Despite its advanced design and capabilities, our research platform faces
certain limitations that we are actively seeking to address. These hurdles
present opportunities for enhancement and refinement, contributing to the
platform’s ongoing evolution.

Simplistic Agent Profile Configuration: A significant limitation lies in the
current mechanistic profile configuration for each agent (e.g., you are <AGE>,
your personality is <BIG 5 PROFILE>, and reside in <LOCATION>). We do not draw
on or attempt to simulate human subjects from demographic backgrounds of past
survey respondents, as in e.g., (Argyle et al., 2023). Standard profile
constructs often used in survey studies underpin this design, providing a
simplified interaction model for users. However, these constructs’ rudimentary
nature restricts the context within which the GPT models function, sometimes
compromising the depth and richness of their responses. To capture the
nuanced, multifaceted nature of human contexts, we need a more sophisticated
approach. The solution we are developing and testing is a custom profile
prompt feature, which will enable users to create intricate, context-sensitive
profiles for their agents (e.g., profile construction by story telling). By
broadening the contextual basis of agent profiles, we anticipate an
enhancement in the model responses’ relevance and applicability.

OpenAI API Rate Limit Constraints: The rate limits imposed by OpenAI’s API can
influence the stability of output and latency, creating potential bottlenecks
for users requiring high-volume, real-time access to the models while being
able to receive responses to their complex request in expected formats. A good
solution depends on three things. First, users must be able to obtain API keys
that have the desirable rate limits. Second, OpenAI, and other LLM providers,
will increase or phase out rate limits. Third, the platform’s batching and
request mechanisms must be robust. We are currently enhancing our concurrent
request handler to optimize request scheduling and execution, thus maximizing
throughput within rate limits for a given API keys. Additionally, we are in
discussions with OpenAI to explore ways to improve throughput and latency.

Model Diversity: Our platform currently supports only OpenAI’s models, chosen
for their leading-edge capabilities and robust API access. This model-specific
dependency could limit the platform’s flexibility, as different models may
offer unique strengths and capabilities that could be beneficial in diverse
research scenarios. We are therefore actively testing other open source and
commercial AI models to potentially integrate into our platform (Bai et al.,
2022; Touvron et al., 2023), thus expanding its versatility and research
applicability. It should be noted that we avoid models that essentially add
new prompt-engineering layers on top of base ALMs to improve decision
performance in specific tasks (Shinn et al., 2023; Yao et al., 2023).

Realistic Condition Profiles: A minor drawback of our platform is the
occasional generation of unrealistic agent profiles due to the randomness of
profile construction. This approach also means that sometimes we end up with
“interesting” agent profile combinations that may seldom present in the real
world (e.g., a male lesbian). While rare, these cases can disrupt the research
process and lead to unrealistic model responses. One solution lies in
conditional profile construction, where agent attributes are selected based on
real-world prevalence and correlations. However, it is important to note that
this randomness can sometimes yield unique case studies that might not have
been otherwise considered, offering unexpected insights and interesting
research avenues.

Multi-Agent Games and Interaction: Currently, SurveyLM allows for the
simulation of an agent’s participation in games with other players only when
the other players and interaction rules are hard-coded into the uploaded
questions and answer instruction. However, we are yet to develop the capacity
for interactions between different agents within the same simulated
population. To achieve this, a series of sophisticated enhancements would be
necessary. Key among these improvements is the ability to form and manage
groups of agents. These groups function as independent social entities,
engaging in intricate social interactions within their own set boundaries. In
this envisioned application, users could define group sizes, roles within
these groups (e.g., the proposer and responder roles in the ultimatum game),
and any variations thereof. Furthermore, we would offer the ability to set
randomisation parameters for both roles and variations, adding yet another
layer of complexity and realism to the simulations. Another captivating idea
under this future avenue is the introduction of a chat function between
different agents. This feature would allow the observation of direct
communication patterns and language use within and across agent groups,
offering researchers another dimension to their social agent studies.

In summary, while our platform faces certain limitations, we view these as
opportunities for growth and enhancement. Our commitment to continuous
development and user satisfaction drives us to persistently explore innovative
solutions to these challenges. This means that we are open for feedback and
suggestions. As we progress on this journey, we look forward to unlocking
further potential in facilitating complex research through advanced ALM
models.

For beta access and further details about the SurveyLM platform, please
contact the corresponding author of this paper.

## References

reAher, G., Arriaga, R. I., & Kalai, A. T. (2023). _Using large language
models to simulate multiple humans and replicate human subject studies_
(arXiv:2208.10264). arXiv. http://arxiv.org/abs/2208.10264

preArgyle, L. P., Busby, E., Gubler, J., Bail, C., Howe, T., Rytting, C., &
Wingate, D. (2023). _AI chat assistants can improve conversations about
divisive topics_ (arXiv:2302.07268). arXiv. http://arxiv.org/abs/2302.07268

preArora, A., Kaffee, L.-A., & Augenstein, I. (2023). _Probing pre-trained
language models for cross-cultural differences in values_ (arXiv:2203.13722).
arXiv. http://arxiv.org/abs/2203.13722

preBai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen,
A., Goldie, A., Mirhoseini, A., McKinnon, C., Chen, C., Olsson, C., Olah, C.,
Hernandez, D., Drain, D., Ganguli, D., Li, D., Tran-Johnson, E., Perez, E., …
Kaplan, J. (2022). _Constitutional AI: Harmlessness from AI feedback_
(arXiv:2212.08073). arXiv. https://doi.org/10.48550/arXiv.2212.08073

preBinz, M., & Schulz, E. (2023). Using cognitive psychology to understand
GPT-3. _Proceedings of the National Academy of Sciences_ , _120_(6),
e2218523120. https://doi.org/10.1073/pnas.2218523120

preHaerpfer, C., Inglehart, R., Moreno, A., Welzel, C., Kizilova, K., Diez-
Medrano, J., & Puranen. (2021). _World values survey: Round seven–country-
pooled datafile._ JD Systems Institute & WVSA Secretariat, 7, 2021.

preHofstede, G. (2005). Culture’s recent consequences. _Proceedings of the
Seventh International Workshop on Internationalisation of Products and
Systems_ , 3–4.

preHorton, J. J. (2023). _Large language models as simulated economic agents:
What can we learn from homo silicus?_ (arXiv:2301.07543). arXiv.
http://arxiv.org/abs/2301.07543

preKorinek, A. (2023). _Language models and cognitive automation for economic
research_ (30957). National Bureau of Economic Research.
https://doi.org/10.3386/w30957

preMialon, G., Dessì, R., Lomeli, M., Nalmpantis, C., Pasunuru, R., Raileanu,
R., Rozière, B., Schick, T., Dwivedi-Yu, J., Celikyilmaz, A., Grave, E.,
LeCun, Y., & Scialom, T. (2023). _Augmented language models: A survey_
(arXiv:2302.07842). arXiv. https://doi.org/10.48550/arXiv.2302.07842

preMiotto, M., Rossberg, N., & Kleinberg, B. (2022). _Who is GPT-3? An
exploration of personality, values and demographics_ (arXiv:2209.14338).
arXiv. http://arxiv.org/abs/2209.14338

prePark, J. S., Popowski, L., Cai, C. J., Morris, M. R., Liang, P., &
Bernstein, M. S. (2022). _Social simulacra: Creating populated prototypes for
social computing systems_ (arXiv:2208.04024). arXiv.
http://arxiv.org/abs/2208.04024

preShinn, N., Cassano, F., Labash, B., Gopinath, A., Narasimhan, K., & Yao, S.
(2023). _Reflexion: Language agents with verbal reinforcement learning_
(arXiv:2303.11366). arXiv. https://doi.org/10.48550/arXiv.2303.11366

preShridhar, M., Yuan, X., Côté, M.-A., Bisk, Y., Trischler, A., & Hausknecht,
M. (2021). _ALFWorld: Aligning text and embodied environments for interactive
learning_ (arXiv:2010.03768). arXiv. https://doi.org/10.48550/arXiv.2010.03768

preTamkin, A., Brundage, M., Clark, J., & Ganguli, D. (2021). _Understanding
the capabilities, limitations, and societal impact of large language models_
(arXiv:2102.02503). arXiv. https://doi.org/10.48550/arXiv.2102.02503

preTouvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y.,
Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L.,
Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu,
W., … Scialom, T. (2023). _Llama 2: Open foundation and fine-tuned chat
models_ (arXiv:2307.09288). arXiv. https://doi.org/10.48550/arXiv.2307.09288

preYao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y.
(2023). _ReAct: Synergizing reasoning and acting in language models_
(arXiv:2210.03629). arXiv. https://doi.org/10.48550/arXiv.2210.03629

p


# ProxyLM: Predicting Language Model Performance on Multilingual
Tasks via Proxy Models

David Anugraha1, Genta Indra Winata2 , Chenyue Li1,
Patrick Amadeus Irawan3†, En-Shiun Annie Lee1,4
1University of Toronto 2Capital One
3Institut Teknologi Bandung 4Ontario Tech University
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>The work was conducted
outside Capital One.

###### Abstract

Performance prediction is a method to estimate the performance of Language
Models (LMs) on various Natural Language Processing (NLP) tasks, mitigating
computational costs associated with model capacity and data for fine-tuning.
Our paper introduces ProxyLM, a scalable framework for predicting LM
performance using proxy models in multilingual tasks. These proxy models act
as surrogates, approximating the performance of the LM of interest. By
leveraging proxy models, ProxyLM significantly reduces computational overhead
on task evaluations, achieving up to a 37.08$\times$ speedup compared to
traditional methods, even with our smallest proxy models. Additionally, our
methodology showcases adaptability to previously unseen languages in pre-
trained LMs, outperforming the state-of-the-art performance by 1.89$\times$ as
measured by root-mean-square error (RMSE). This framework streamlines model
selection, enabling efficient deployment and iterative LM enhancements without
extensive computational resources.

##  1 Introduction

Language Models (LMs) have become increasingly valuable for assessing Natural
Language Processing (NLP) tasks Raffel et al. (2020); Brown et al. (2020);
Costa-jussà et al. (2022b); Touvron et al. (2023a, b); Le Scao et al. (2023).
However, fine-tuning and evaluating these models are resource-intensive
processes in terms of both computation and time. These costs escalate with
model size, especially when experimenting across multiple datasets. As
highlighted in Kaplan et al. (2020), there is a scaling law that applies to
both model and dataset sizes, and computational demands, indicating that
larger models and broader datasets require increased computational resources.
Modeling low-resource languages (LRLs) in multilingual contexts presents a
range of challenges. One significant challenge is the limited data
availability, which hampers effective fine-tuning processes Gu et al. (2018);
Adilazuarda et al. (2024), making model adaptation through fine-tuning a
challenging task Zoph et al. (2016); Liu et al. (2021). Another critical issue
is the lack of pre-training data for numerous regional languages, such as
Southeast Asian languages Winata et al. (2022, 2024); Yong et al. (2024), with
many languages being omitted during the pre-training phase of multilingual
LMs.

Figure 1: ProxyLM framework for LM performance prediction. (Top) The
evaluation metric is computed on the test set using a proxy model
$\mathcal{M}_{p}^{i}$. (Bottom) The regressor $g$ is trained using proxy model
scores as well as dataset and language features by minimizing the RMSE
difference of $y_{\mathcal{M}}$ and $\hat{y}_{\mathcal{M}}$.

Given the limited academic computational resources for LM fine-tuning and
inadequate datasets for LRLs, performance prediction is an efficient method to
alleviate the dependency on extensive resources by leveraging past performance
records on an NLP task. While linear regression and gradient-boosting hold
promise in performance prediction Birch et al. (2008a); Srinivasan et al.
(2021); Xia et al. (2020a); Ye et al. (2021a); Schram et al. (2023); Khiu et
al. (2024), existing solutions primarily focus on homogeneous data settings
and prioritize high-resource languages using Transformer models Vaswani et al.
(2017). Khiu et al. (2024) examine diverse datasets and LRLs but encounter
limitations in the number of experiments, language diversity, and model scope,
focusing solely on mBART Liu et al. (2020). Recent advancements in larger
multilingual models, like NLLB Costa-jussà et al. (2022a) and M2M100 Fan et
al. (2021), have significantly improved machine translation capabilities,
exceeding those of mBART.

In this paper, we propose ProxyLM, 111We release our code at
https://github.com/davidanugraha/proxylm a framework to predict LMs
performance by utilizing proxy models on LRLs. Proxy models are defined as
substitute models, wherein the performance of these substitute models are used
to estimate the performance of another LM. This other model can be
significantly larger than our proxy models. For optimizing the prediction, we
utilize much smaller LMs as proxy models and off-the-shelf models without
further tuning. This approach is very scalable to multiple proxy models and
task-agnostic to any modalities, thus it can be applied to any downstream
tasks. This study focuses on machine translation tasks and our approach
outperforms the existing work from Xia et al. (2020a); Ye et al. (2021a);
Schram et al. (2023); Khiu et al. (2024), which opens a new avenue to employ
LMs for model performance prediction. Therefore the contribution of our paper
can be summarized in three-fold:

  1. 1.

We introduce ProxyLM, an efficient and scalable framework designed to predict
the performance of LMs. This framework significantly reduces the computational
costs associated with fine-tuning and inference during model selection.

  2. 2.

We demonstrate the effectiveness and robustness of ProxyLM across 21 dataset
sources and 50 languages on two estimated LM architectures. Our framework
substantially outperforms all existing baselines in English-centric, many-to-
many languages, and cross-dataset settings, including scenarios involving
extremely LRLs that remain unseen by pre-trained LMs, surpassing the state-of-
the-art performance measured with root-mean-square error (RMSE) by
1.89$\times$.

  3. 3.

We also provide a time analysis comparing the fine-tuning duration of proxy
models to direct model fine-tuning. Our results indicate that, with our
smallest proxy models, we can achieve up to a 37.08$\times$ speedup on task
evaluation compared to the traditional approach, highlighting the efficiency
of our approach.

##  2 Methodology

In this section, we formally define the LM performance prediction problem and
our proposal to improve performance prediction.

###  2.1 ProxyLM

Recall that performance prediction is a task of estimating a system’s
performance based on the model and its training strategy, training and test
dataset, and language used. Formally, let LM $\mathcal{M}$ be our estimated
model. $\mathcal{M}$ is trained over a training dataset $\mathcal{D}$ with
source language $\mathcal{L}_{s}$ and target language $\mathcal{L}_{t}$, and
then tested using dataset $\mathcal{D}^{\prime}$. $\mathcal{M}$’s performance,
denoted $y_{\mathcal{M}}$, can be formulated under function $f$ that relates
between these variables:

$y_{\mathcal{M}}=f(\mathcal{M},\mathcal{D},\mathcal{D}^{\prime},\mathcal{L}_{s},\mathcal{L}_{t}).$
(1)

We can approximate $f$ by transforming Equation 1 into a regression task with
a regressor function $g$, which will be trained on past performance records.
The regressor takes dataset features $\Phi(\mathcal{D},\mathcal{D}^{\prime})$
to identify the characteristics of the training and test datasets, as well as
the distribution shift between them. It also takes language features
$\Psi(\mathcal{L}_{s},\mathcal{L}_{t})$ to measure the dissimilarities between
the source and target languages. This can be formulated as follows:

$\hat{y_{\mathcal{M}}}=g(\Phi(\mathcal{D},\mathcal{D}^{\prime});\Psi(\mathcal{L}_{s},\mathcal{L}_{t})).$
(2)

We present ProxyLM, a framework that leverages the past performance of other
models, referred to as proxy models, as additional context for our regressor.
Intuitively, proxy models can provide valuable insights that assist in
predicting the performance of the estimated model $\mathcal{M}$. Formally, let
$\mathcal{M}_{p}=[\mathcal{M}_{p}^{1},\dots,\mathcal{M}_{p}^{N}]$ be a set of
$N$ proxy models. To integrate the information from these proxy models, we
modify Equation 2 as follows:

$\hat{y_{\mathcal{M}}}=g(\hat{y}_{\mathcal{M}_{p}};\Phi(\mathcal{D},\mathcal{D}^{\prime});\Psi(\mathcal{L}_{s},\mathcal{L}_{t})),$
(3)

where
$y_{\mathcal{M}_{p}}=[y_{\mathcal{M}_{p}^{1}},\dots,y_{\mathcal{M}_{p}^{N}}]$
represents the performance records of $N$ proxy models. The advantage of using
proxy models arises from their faster fine-tuning and evaluation compared to
the estimated model $\mathcal{M}$. This also means that off-the-shelf models
can be used directly without additional tuning if they already perform the
task adequately, further enhancing efficiency.

###  2.2 ProxyLM Features

#### Language Features

We use URIEL Typological Database Littell et al. (2017) similar to Xia et al.
(2020a) including geographic, genetic, inventory, syntactic, phonological, and
featural distance. The language features are useful to provide a language-
specific representation to the regressor.

#### Dataset Features

We extract 6 features from the dataset, including train size, vocab size,
average sentence length, word overlap, Type-Token Ratio (TTR), and TTR
distance from $\mathcal{D}$ and $\mathcal{D^{\prime}}$ based on Xia et al.
(2020a). We will refer to these features and language features combined as
NLPerf features. Furthermore, we incorporate the distribution shift
information between the training and test datasets using Jensen-Shannon
Divergence (JSD) as described by Khiu et al. (2024). In addition, we include
term frequency-inverse document frequency (TF-IDF) and sentence similarity
with Sentence-BERT Reimers and Gurevych (2019).

#### Proxy Models Features

We leverage the performance data from proxy models, derived by averaging
results from multiple fine-tuning and evaluation iterations on identical
datasets and languages. Moreover, we retain the flexibility to adjust the
number of proxy models employed, facilitating efficient and adaptable
performance estimation.

##  3 Experimental Setup

In this section, we describe the datasets and LMs used to obtain LMs’
performance records. These records are then used to train various regressor
models under different experimental settings to investigate our approach to
performance predictions. The details of the hyper-parameters for both the LMs
and the regressors are provided in A.3.

###  3.1 Datasets

We evaluate our approach using two types of datasets: English-centric and
Many-to-Many Languages. The English-centric dataset involves English serving
as either the source or target language. Our English-centric dataset comes
from the MT560 Gowda et al. (2021) dataset, where we curated 19 datasets and
selected 44 languages out of 500 for evaluation. Furthermore, we use the
FLoRes-200 dataset Costa-jussà et al. (2022b) for additional validation and
test sets. In contrast, the Many-to-Many Languages dataset allows any language
to act as the source or target. We use the NusaTranslation dataset Cahyawijaya
et al. (2023) as our Many-to-Many Languages dataset, which comprises parallel
texts in 12 Indonesian regional languages. As many of these languages are
absent in pre-trained multilingual models, we analyze 8 out of the 12
languages due to limited data in the remaining 4. Both datasets encompass 50
languages across various domains such as economics, technology, and medicine.
Detailed language insights are available in Section A.1 in the Appendix.

###  3.2 Models

#### Estimated LMs

We employ two estimated LMs: M2M100 1.2B Fan et al. (2021) and NLLB 1.3B
Costa-jussà et al. (2022b). Each estimated model is fine-tuned using a
standard next token prediction objective on the training set.

#### Proxy Models

We utilize four different transformer-based models: an encoder-decoder random
initialized Transformers (100M) Vaswani et al. (2017), SMaLL-100 (330M)
Mohammadshahi et al. (2022), M2M100 Fan et al. (2021), and NLLB Costa-jussà et
al. (2022b). For M2M100 and NLLB, we use the models without any additional
tuning (No FT) in a zero-shot fashion. Model details are provided in Section
A.2 in the Appendix. The evaluation is done using SentencePiece BLEU (spBLEU)
Goyal et al. (2022), as it has been demonstrated to be a fair metric in
multilingual settings, particularly in low-resource settings. For simplicity,
the term “fine-tuning" will be used throughout this paper to refer to both the
process of training from scratch (as in the case of the Transformer (100M)
model) and the process of fine-tuning pre-trained LMs.

#### Regressors

We utilize XGBoost Chen and Guestrin (2016), LGBM Ye et al. (2021a), Poly2
Khiu et al. (2024), and Poly3 Khiu et al. (2024) as our regressors. In most of
our experiments, we apply XGBoost as our default regressor because we find it
to be the best-performing model, while the other regressors serve as
baselines. Specifically for the Many-to-Many Languages setting, Matrix
Factorization with context features (MF) is used as an additional baseline
Schram et al. (2023). We do not apply MF to our English-centric setting
because MF requires the performance records to be structured in two
dimensions—one for the source language and one for the target language. In the
English-centric setting, this would result in a sparse matrix with only one
fully populated row or column, corresponding to English, making MF impractical
for this setup.

Models | Random | LOLO | Unseen |
---|---|---|---|---
| M2M100$\downarrow$ | NLLB$\downarrow$ | M2M100$\downarrow$ | NLLB$\downarrow$ | M2M100$\downarrow$ | Avg.
NLPerf Xia et al. (2020a) with different regressors |  |
XGBoost | 8.09 $\pm$ 0.44 | 8.40 $\pm$ 0.41 | 8.81 $\pm$ 0.39 | 12.35 $\pm$ 0.52 | 8.22 $\pm$ 0.46 | 9.17
Poly2 Khiu et al. (2024) | 10.00 $\pm$ 0.46 | 12.30 $\pm$ 0.48 | 10.54 $\pm$ 0.00 | 13.28 $\pm$ 0.00 | 9.91 $\pm$ 0.00 | 11.21
Poly3 Khiu et al. (2024) | 9.93 $\pm$ 0.52 | 12.33 $\pm$ 0.46 | 10.66 $\pm$ 0.00 | 14.06 $\pm$ 0.00 | 10.57 $\pm$ 0.00 | 11.51
LGBM Ye et al. (2021a) | 8.18 $\pm$ 0.34 | 8.59 $\pm$ 0.33 | 9.00 $\pm$ 0.49 | 12.41 $\pm$ 0.53 | 8.59 $\pm$ 0.69 | 9.35
ProxyLM (Ours)‡ with different proxy models |  |
Transformer | 5.29 $\pm$ 0.38 | 7.65 $\pm$ 0.47 | 6.25 $\pm$ 0.29 | 11.33 $\pm$ 0.50 | 6.54 $\pm$ 0.30 | 7.41
SMaLL-100 |  4.26 $\pm$ 0.35 | 7.02 $\pm$ 0.44 |  4.33 $\pm$ 0.24 | 9.70 $\pm$ 0.37 |  4.54 $\pm$ 0.41 | 5.97
SMaLL-100 (No FT) | 5.93 $\pm$ 0.23 | 6.73 $\pm$ 0.49 | 6.33 $\pm$ 0.28 | 10.66 $\pm$ 0.44 | 5.77 $\pm$ 6.06 | 7.08
Estimated Model (No FT) | 5.76 $\pm$ 0.19 |  4.29 $\pm$ 0.31 | 6.23 $\pm$ 0.34 |  4.59 $\pm$ 0.22 | 5.67 $\pm$ 0.41 | 5.31
Ensemble† |  3.79 $\pm$ 0.30 |  3.79 $\pm$ 0.25 |  3.90 $\pm$ 0.22 |  4.14 $\pm$ 0.24 |  4.36 $\pm$ 0.22 | 4.00

Table 1: English-centric test results in average RMSE $\pm$ standard deviation
(lower is better). Bold numbers indicate the best performance, while
underlined numbers represent the second-best performance. The columns show the
setting and estimated model. “No FT" denotes “no fine-tuning" and the model
inference is done in a zero-shot fashion. We only show M2M100 results for the
Unseen setting since NLLB covers all languages in the English-centric dataset.
Avg represents the average of the results across the row. ‡The reported
results use XGBoost as the regressor. †Ensemble denotes combining all four
proxy models, the detailed breakdown of this result can be seen in Section B
in the Appendix.

Models | M2M100 | NLLB
---|---|---
| LRL | MRL | LRL | MRL
Transformer | 6.14 | 6.67 | 11.60 | 10.26
SMaLL-100 | 4.28 | 4.55 | 10.18 | 7.83
SMaLL-100 (No FT) | 6.17 | 6.96 | 7.90 | 11.38
Estimated Model (No FT) | 6.15 | 6.57 | 4.66 | 4.32
Ensemble | 3.93 | 3.80 | 4.10 | 4.15

Table 2: English-centric LOLO test results in average RMSE by language
vitality (lower is better). “No FT" denotes “no fine-tuning". “LRL" and “MRL"
denote low-resource languages and medium-resource languages, based on Joshi et
al. (2021). The full mapping is in Table 8 & 9 in Section A.1 in the Appendix.

Models | Random | LOLO |
---|---|---|---
| M2M100$\downarrow$ | NLLB$\downarrow$ | M2M100$\downarrow$ | NLLB$\downarrow$ | Avg.
NLPerf Xia et al. (2020a) with different regressors |  |  |
XGBoost | 2.45 $\pm$ 0.30 |  1.21 $\pm$ 0.29 | 7.83 $\pm$ 0.23 | 8.28 $\pm$ 0.31 | 4.94
Poly2 Khiu et al. (2024) | 4.70 $\pm$ 0.40 | 4.68 $\pm$ 0.51 | 7.07 $\pm$ 0.00 | 7.90 $\pm$ 0.00 | 6.09
Poly3 Khiu et al. (2024) | 4.60 $\pm$ 0.41 | 4.64 $\pm$ 0.49 | 7.26 $\pm$ 0.00 | 8.01 $\pm$ 0.00 | 6.13
LGBM Ye et al. (2021a) | 2.66 $\pm$ 0.22 | 1.76 $\pm$ 0.29 | 7.91 $\pm$ 0.01 | 8.06 $\pm$ 0.00 | 5.10
MF Schram et al. (2023) | 3.65 $\pm$ 0.26 | 2.60 $\pm$ 0.39 | 7.08 $\pm$ 0.23 | 7.14 $\pm$ 0.22 | 5.12
ProxyLM‡ (Ours) with different proxy models |  |  |
Transformer | 2.56 $\pm$ 0.43 | 1.70 $\pm$ 0.20 | 5.65 $\pm$ 0.23 | 6.24 $\pm$ 0.34 | 4.04
SMaLL-100 | 2.56 $\pm$ 0.33 | 1.65 $\pm$ 0.44 |  4.85 $\pm$ 0.36 |  5.14 $\pm$ 0.46 | 3.55
SMaLL-100 (No FT) | 2.44 $\pm$ 0.21 | 1.34 $\pm$ 0.38 | 6.93 $\pm$ 0.34 | 7.25 $\pm$ 0.37 | 4.49
Estimated Model (No FT) |  2.38 $\pm$ 0.36 |  1.27 $\pm$ 0.30 | 5.10 $\pm$ 0.28 | 5.50 $\pm$ 0.26 | 3.56
Ensemble† |  2.41 $\pm$ 0.28 | 1.56 $\pm$ 0.35 |  3.73 $\pm$ 0.23 |  3.79 $\pm$ 0.19 | 2.90

Table 3: Many-to-Many Languages test results in average RMSE $\pm$ standard
deviation (lower is better). Bold numbers indicate the best performance, while
underlined numbers represent the second-best performance. The columns show the
setting and estimated model. “No FT" denotes “no fine-tuning". We do not cover
the Unseen setting as most of the languages are not covered by both M2M100 and
NLLB in the Many-to-Many Languages dataset. Avg represents the average of the
results across the row. ‡The reported results are experiments using XGBoost
regressor. †Ensemble denotes combining all four proxy models.

###  3.3 Experimental Settings

Each regressor is evaluated using RMSE as our performance metric and evaluated
5 times. We set our experiment settings as follows:

  * •

Random: We randomly sample the performance records into training and test sets
with a ratio of 7:3. Then, we run 10-fold cross-validation on the training set
to find the best hyper-parameters for each regressor. The best-performing
regressor would subsequently be evaluated on the test set.

  * •

Leave-One-Language-Out (LOLO): We select one language as the test set, which
is not encountered during training.

  * •

Unseen: The performance records can be divided into two categories: (1)
records with “seen" languages and (2) records with “unseen" languages.
“Unseen" languages refer to languages that are not present in the pre-training
LM data, while “seen" languages denote those that are present. In this
setting, the regressor is trained using records of “seen" languages and tested
using records of “unseen" languages.

  * •

Cross-Dataset: We train the regressor using performance records from the
English-centric dataset and test it using records from the Many-to-Many
Languages dataset. We opt not to reverse this setup as the dataset exhibits no
domain shift and contains fewer performance records.

##  4 Results and Analysis

In this section, we present the results of the performance predictions for
ProxyLM and baselines under the specified settings. Further, we discuss the
robustness, effectiveness, and efficiency of ProxyLM in the context of
performance prediction.

###  4.1 English-centric Results

Table 1 shows the overall results on the English-centric dataset. ProxyLM
remarkably outperforms all existing baselines. We find that incorporating all
proxy models (Ensemble) is the most effective for prediction, leading to a
2.29$\times$ averaged reduction in RMSE across all experimental settings
compared to the best baseline. We observe that using the No FT estimated model
to predict the performance of their fine-tuned models is surprisingly useful
in all settings, especially for NLLB, where the model already has decent
machine translation quality on LRLs. This observation is supported by our
findings within the XGBoost model that the NLLB No FT feature has the highest
importance score among all features, as shown in Figure 16 in the Appendix.
Further, using SMaLL-100 fine-tuned performance provides useful estimations
for settings involving M2M100 as the estimated model. This may indicate that
the performance of a model with similar architecture can be a good estimator
for the performance of the larger estimated model. In other words, the choice
of proxy model to help prediction matters. Feature importance analysis from
the XGBoost model supports this, revealing that the SMaLL-100 fine-tuned
feature has the highest importance score among all features, as shown in
Figure 15 in the Appendix.

Our analysis also indicates that XGBoost outperforms other regression models,
across all evaluated settings. Both XGBoost and LGBM, which are gradient-
boosting and tree-based learning methods, demonstrate superior performance
metrics across all settings. Their robustness and efficiency as non-linear
models are evident when compared to linear models, such as Poly2 and Poly3.
Poly2 and Poly3 regressors, which employ second-degree and third-degree
polynomial regression approaches respectively, tend to generate lower scores.
This diminished performance is largely attributed to their limitations in
capturing the nonlinear relationships inherent in the data, leading to
suboptimal results. We further present the results by the language vitality in
Table 2. The overall difference in RMSE between LRLs and medium-resource
languages (MRLs) is relatively small, except when using SMaLL-100 as proxy
models. An interesting observation here on SMaLL-100 in predicting the NLLB
model is that the No FT model can predict LRLs much better than the fine-tuned
counterpart, and the fine-tuned model better predicts MRLs.

Models | M2M100$\downarrow$ | NLLB$\downarrow$
---|---|---
NLPerf Xia et al. (2020a) with different regressors
XGBoost | 9.97 $\pm$ 0.35 | 10.52 $\pm$ 0.73
Poly2 Khiu et al. (2024) | 9.27 $\pm$ 0.00 | 21.43 $\pm$ 0.00
Poly3 Khiu et al. (2024) | 8.58 $\pm$ 0.00 | 19.81 $\pm$ 0.00
LGBM Ye et al. (2021a) | 9.24 $\pm$ 0.24 | 15.19 $\pm$ 1.00
ProxyLM‡ (Ours) with different proxy models
Transformer | 6.65 $\pm$ 0.26 | 10.69 $\pm$ 0.90
SMaLL-100 |  5.53 $\pm$ 0.98 | 7.68 $\pm$ 2.11
SMaLL-100 (No FT) | 9.08 $\pm$ 0.63 | 16.66 $\pm$ 1.08
Estimated Model (No FT) | 9.45 $\pm$ 0.78 |  7.10 $\pm$ 0.69
Ensemble† |  4.30 $\pm$ 0.17 |  6.21 $\pm$ 0.50

Table 4: Cross-Dataset test results in average RMSE $\pm$ standard deviation
(lower is better). Bold numbers indicate the best performance, while
underlined numbers represent the second-best performance. The columns show the
setting and estimated model. “No FT" denotes “no fine-tuning". ‡The reported
results are experiments using the LGBM regressor. †Ensemble denotes combining
all four proxy models.

###  4.2 Many-to-Many Languages Results

Table 3 presents the performance of different models on the Many-to-Many
Languages dataset. The results reveal that the Ensemble model achieves the
lowest RMSE, with a 1.70$\times$ averaged reduction in RMSE across all
experimental settings compared to the best baseline, indicating superior
accuracy in performance predictions. An exception occurs in the random NLLB
setting, where the model utilizing only NLPerf features outperforms the
ensemble model, achieving the best performance. Note that no domain shift
occurs within the dataset.

A comparative analysis shows that predicting the performance of the M2M100
model in the random setting presents a greater challenge compared to
predicting the NLLB model. This discrepancy suggests that the complexity of
performance prediction can vary substantially depending on the specific LM and
the conditions under which it is evaluated. A particularly noteworthy finding
is the effectiveness of using No FT models for estimating LM performance. The
No FT models, which do not require any additional fine-tuning, demonstrate
high accuracy in their performance predictions. This method offers substantial
efficiency benefits, as it eliminates the need for extensive computational
resources typically required for model training. In contrast, we find similar
results between the LOLO setting for Many-to-Many Languages and English-
centric results, where ProxyLM using Ensemble remarkably outperforms all
existing baselines. In addition, we find that using SMaLL-100 fine-tuned
performance results in better predictions compared to those of the No FT
estimated model.

###  4.3 Cross-Dataset Results

Table 4 illustrates model performance in the Cross-Dataset setup, showcasing
the superior performance of ProxyLM with LGBM. We use LGBM instead of XGBoost
because LGBM performs better. The results highlight that ProxyLM with Ensemble
significantly reduces RMSE compared to the best baseline by 2$\times$ and
1.69$\times$ for M2M100 and NLLB, respectively. This displays consistent
performance across datasets and languages that are not encountered during the
regressor’s training, including unseen languages for the pre-trained LMs.
However, the No FT models exhibit inconsistency compared to other proxy
models. We also find varied performance between M2M100 and NLLB, which may be
attributed to the English-centric dataset containing only seen languages for
NLLB and lacking unseen language examples for the regressor. This highlights
the significance of incorporating unseen language instances in training for
more robust predictions.

(a) English-centric test results.

(b) Many-to-Many Languages test results.

Figure 2: Ablation study on the LOLO setting with XGBoost on English-centric
and Many-to-Many Languages datasets. Proxy Models here indicates Ensemble,
which is a combination of all proxy models. Proxy Models significantly reduce
RMSE across all scenarios. Figure 3: RMSE decreases as XGBoost’s training set
size increases. Training set size refers to the number of English-centric past
performance records.

###  4.4 Ablation Study

Figure 2 highlights the impact of features used in ProxyLM in the LOLO setting
with XGBoost. Utilizing proxy models as features leads to a significant
reduction in RMSE across all scenarios, showcasing their importance compared
to other features. For the English-centric dataset, including language and
dataset features alongside proxy models enhances performance. Dataset features
alone show better improvement than language features alone, but the
combination of both yields the best performance. On the other hand, for the
Many-to-Many Languages dataset, the benefits of incorporating dataset and
language features are less pronounced, especially for the M2M100 model, and
there may even be a performance dip for the NLLB model due to the dataset’s
lack of domain shift.

###  4.5 Diminishing Returns with Increasing Training Set Size

In Figure 3, we examine the training of the XGBoost regressor using different
numbers of English-centric past performance records as the training dataset.
While the regressor’s performance shows enhancement with an expanding training
size, the incremental benefits start diminishing once the training set
surpasses about 400 past performance records. This observation implies that,
across datasets, there exists a threshold where the advantages of
incorporating additional past performance records begin to exhibit diminishing
returns.

Datasets | Inference | Fine-tuning
---|---|---
|  | English-centric | Many-to-Many Langs.
Estimated models |  |  |  |  |
M2M100 | 338 s | 3.94 | hrs (4.77$\times$) | 1.42 | hrs (7.32$\times$)
NLLB | 492 s | 9.4 | hrs (11.48$\times$) | 7.21 | hrs (37.08$\times$)
Proxy models |  |  |  |  |
SMaLL-100 | 201 s | 2.5 | hrs (3.03$\times$) | 1.03 | hrs (5.29$\times$)
Transformer | 60 s | 0.83 | hrs (1$\times$) | 0.19 | hrs (1$\times$)

Table 5: Comparison of LMs’ inference time (in seconds) and fine-tuning time
(in hours) for one experimental run. The multiplier of fine-tuning time is
relative to the Transformer model. All times were calculated using the
interquartile mean to ignore outliers. Since the inference time for English-
centric and Many-to-Many Languages datasets are roughly the same, only one
column is provided.

Regressors | English-centric | Many-to-Many Langs.
---|---|---
XGBoost | 129.29 s | 69.81 s
Poly2 | 0.06 s | 0.03 s
Poly3 | 0.07 s | 0.06 s
LGBM | 702.12 s | 173.17 s
MF | N/A | 465.37 s

Table 6: Regressor models training time comparison in seconds (s).

(a)

(b)

(c)

Figure 4: Detailed results of XGBoost with ProxyLM Ensemble on M2M100 model
under the LOLO setting using the English-centric dataset from Table 1. The
results are grouped by (a) Joshi Class and (b) language family that follows
the mapping which is provided in Appendix A.1; (c) shows the scatter plot
illustrating the correlation of spBLEU scores between the ProxyLM’s prediction
and estimated LM, with the light gray dashed line representing the line of
equality ($y=x$) with $R^{2}=0.88$ and black dashed line representing Locally
Weighted Scatterplot Smoothing (LOWESS) curve to represent the trend.

###  4.6 Time Efficiency

Table 5 compares the fine-tuning and inference times required for the
estimated and proxy models. The results demonstrate that fine-tuning proxy
models or direct inference from any model is remarkably faster than fine-
tuning all estimated models. Table 6 further illustrates this point, showing
only a minimal trade-off in the time needed to train the regressor models.
This additional training time is relatively negligible, highlighting the
efficiency of using proxy models.

###  4.7 Performance by Language Categories

In Figure 4, we present detailed XGBoost results with ProxyLM Ensemble on the
M2M100 model under the English-centric LOLO experiment, grouped by language
categories. Based on the Locally Weighted Scatterplot Smoothing (LOWESS)
Cleveland (1979) curve depicted in Figure 4(c), our method consistently
maintains unbiased predictions for spBLEU scores below 40 across various
language types. However, as the spBLEU score increases, the availability of
data points diminishes, leading to our method under-predicting the performance
compared to the true spBLEU score. Outliers observed in Kartvelian languages
and Indo-European languages with Joshi class 3 may have contributed to this
discrepancy in prediction. These observations suggest that increasing the
number of data points covering higher spBLEU scores may help mitigate the bias
in prediction. Further experiment details are available in Appendix B.

##  5 Related Work

The prediction performance of machine learning algorithms has been mainly
explored in two research directions: (1) predict the model performance during
the training runtime, and (2) predict the model performance by providing
extracted features from the dataset Xia et al. (2020a).

#### Performance Prediction During the Training Runtime

The former aims to infer and extrapolate the learning curve to approximate
training results using evaluation metric measurements Kolachina et al. (2012).
Domhan et al. (2015) study the quick detection of poor hyper-parameters in
probabilistic models after a few steps of Stochastic Gradient Descent (SGD).
Adriaensen et al. (2024) extrapolate learning curves from a parametric prior
using Markov Chain Monte Carlo (MCMC).

#### Performance Prediction Using Extracted Features

The latter aims to predict the model performance by learning a correlation
between input features and the final evaluation metric. Birch et al. (2008a)
identify strong predictive features such as the amount of reordering, the
morphological complexity of the target language, and the historical
relatedness of the two languages. Xia et al. (2020a) leverage extracted
dataset features and typological database language representations. Ye et al.
(2021a) introduce the use of confidence intervals and calibration with various
regressor algorithms for reliable performance prediction. Schram et al. (2023)
apply Bayesian matrix factorization for performance prediction on multilingual
NLP tasks. In this work, we focus on exploring the latter. Existing approaches
have shown promise using linear regression and gradient-boosting trees Birch
et al. (2008b); Xia et al. (2020b); Srinivasan et al. (2021); Ye et al.
(2021b). These studies have considered data size, typological features, and
language similarity as factors contributing to the model performance.

##  6 Conclusion

In this paper, we introduce ProxyLM, a novel framework designed to predict the
performance of LMs by leveraging proxy models including for LRLs. By utilizing
proxy models as substitutes to estimate the performance of the target model,
we strategically employ smaller LMs and off-the-shelf models without
additional fine-tuning. This framework is highly scalable to multiple proxy
models and is task-agnostic, making it applicable to a wide range of
downstream NLP tasks. Our streamlined approach showcases substantial
advancements in prediction accuracy compared to standard baselines and
exhibits strong generalization capabilities across varied scenarios.

## Limitations

This paper focuses exclusively on two estimated models: M2M100 and NLLB, to
evaluate our proposed framework. For demonstration purposes, we concentrate on
the usage of specific models, namely the Transformer model, the SMaLL-100
model, and the No FT models, to illustrate the effectiveness of our proxy
models. The M2M100 and NLLB models were selected due to their prominence and
relevance in the field of multilingual translation tasks. These models serve
as robust benchmarks for assessing the performance and reliability of our
proxy-based framework. By using these well-regarded models, we aim to provide
compelling evidence of the capabilities and advantages of ProxyLM. While our
proposed framework is evaluated solely within the context of machine
translation, it is not confined to this application alone. The framework is
designed to be versatile and can be extended to a variety of other downstream
tasks. We plan to explore these additional applications in future work.

Some other possible avenues for future work could involve a deeper
investigation into which proxy models are more effective for enhancing
performance prediction in specific settings. Our findings suggest that one
proxy model can outperform another in different scenarios, making it crucial
to carefully select the most relevant proxy models to maximize the benefits of
our approach. Additionally, developing methodologies for collecting relevant
past performance records could provide better insights and improve the
generalization and accuracy of our framework. Past performance records may
provide better information gain than others, potentially minimizing the number
of performance records required for a more robust and accurate predictor.

## Acknowledgements

We extend our sincere gratitude to Viktoria Schram for providing assistance to
reproduce baselines.

## References

  * Adilazuarda et al. (2024) Muhammad Farid Adilazuarda, Samuel Cahyawijaya, Alham Fikri Aji, Genta Indra Winata, and Ayu Purwarianti. 2024.  Lingualchemy: Fusing typological and geographical elements for unseen language generalization.  _arXiv preprint arXiv:2401.06034_.
  * Adriaensen et al. (2024) Steven Adriaensen, Herilalaina Rakotoarison, Samuel Müller, and Frank Hutter. 2024.  Efficient bayesian learning curve extrapolation using prior-data fitted networks.  _Advances in Neural Information Processing Systems_ , 36.
  * Birch et al. (2008a) Alexandra Birch, Miles Osborne, and Philipp Koehn. 2008a.  Predicting success in machine translation.  In _Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing_ , pages 745–754.
  * Birch et al. (2008b) Alexandra Birch, Miles Osborne, and Philipp Koehn. 2008b.  Predicting success in machine translation.  In _Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing_ , pages 745–754, Honolulu, Hawaii. Association for Computational Linguistics.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Cahyawijaya et al. (2023) Samuel Cahyawijaya, Holy Lovenia, Fajri Koto, Dea Adhista, Emmanuel Dave, Sarah Oktavianti, Salsabil Akbar, Jhonson Lee, Nuur Shadieq, Tjeng Wawan Cenggoro, et al. 2023.  Nusawrites: Constructing high-quality corpora for underrepresented and extremely low-resource languages.  In _Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 921–945.
  * Chen and Guestrin (2016) Tianqi Chen and Carlos Guestrin. 2016.  Xgboost: A scalable tree boosting system.  In _Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining_ , pages 785–794.
  * Christodouloupoulos and Steedman (2015) Christos Christodouloupoulos and Mark Steedman. 2015.  A massively parallel corpus: the bible in 100 languages.  _Language resources and evaluation_ , 49:375–395.
  * Cleveland (1979) William S Cleveland. 1979.  Robust locally weighted regression and smoothing scatterplots.  _J. Am. Stat. Assoc._ , 74(368):829–836.
  * Costa-jussà et al. (2022a) Marta R. Costa-jussà, Christine Basta, and Gerard I. Gállego. 2022a.  Evaluating gender bias in speech translation.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pages 2141–2147, Marseille, France. European Language Resources Association.
  * Costa-jussà et al. (2022b) Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022b.  No language left behind: Scaling human-centered machine translation.  _arXiv preprint arXiv:2207.04672_.
  * Domhan et al. (2015) Tobias Domhan, Jost Tobias Springenberg, and Frank Hutter. 2015.  Speeding up automatic hyperparameter optimization of deep neural networks by extrapolation of learning curves.  In _Twenty-fourth international joint conference on artificial intelligence_.
  * Fan et al. (2021) Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. 2021.  Beyond english-centric multilingual machine translation.  _Journal of Machine Learning Research_ , 22(107):1–48.
  * Gowda et al. (2021) Thamme Gowda, Zhao Zhang, Chris Mattmann, and Jonathan May. 2021.  Many-to-english machine translation tools, data, and pretrained models.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations_ , pages 306–316.
  * Goyal et al. (2022) Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzmán, and Angela Fan. 2022.  The Flores-101 evaluation benchmark for low-resource and multilingual machine translation.  _Transactions of the Association for Computational Linguistics_ , 10:522–538.
  * Gu et al. (2018) J Gu, H Hassan, J Devlin, and VOK Li. 2018.  Universal neural machine translation for extremely low resource languages.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_. Association for Computational Linguistic.
  * Haddow and Kirefu (2020) Barry Haddow and Faheem Kirefu. 2020.  Pmindia–a collection of parallel corpora of languages of india.  _arXiv preprint arXiv:2001.09907_.
  * Joshi et al. (2021) Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2021.  The state and fate of linguistic diversity and inclusion in the nlp world.
  * Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020.  Scaling laws for neural language models.  _arXiv preprint arXiv:2001.08361_.
  * Khiu et al. (2024) Eric Khiu, Hasti Toossi, David Anugraha, Jinyu Liu, Jiaxu Li, Juan Armando Parra Flores, Leandro Acros Roman, A Seza Doğruöz, and En-Shiun Annie Lee. 2024.  Predicting machine translation performance on low-resource languages: The role of domain similarity.  _arXiv preprint arXiv:2402.02633_.
  * Kolachina et al. (2012) Prasanth Kolachina, Nicola Cancedda, Marc Dymetman, and Sriram Venkatapathy. 2012.  Prediction of learning curves in machine translation.  In _Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 22–30.
  * Le Scao et al. (2023) Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2023.  Bloom: A 176b-parameter open-access multilingual language model.
  * Littell et al. (2017) Patrick Littell, David R Mortensen, Ke Lin, Katherine Kairis, Carlisle Turner, and Lori Levin. 2017.  Uriel and lang2vec: Representing languages as typological, geographical, and phylogenetic vectors.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_ , pages 8–14.
  * Liu et al. (2020) Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020.  Multilingual denoising pre-training for neural machine translation.  _Transactions of the Association for Computational Linguistics_ , 8:726–742.
  * Liu et al. (2021) Zihan Liu, Genta Indra Winata, and Pascale Fung. 2021.  Continual mixed-language pre-training for extremely low-resource neural machine translation.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 2706–2718.
  * Mohammadshahi et al. (2022) Alireza Mohammadshahi, Vassilina Nikoulina, Alexandre Bérard, Caroline Brun, James Henderson, and Laurent Besacier. 2022.  Small-100: Introducing shallow multilingual machine translation model for low-resource languages.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 8348–8359.
  * Qi et al. (2018) Ye Qi, Devendra Sachan, Matthieu Felix, Sarguna Padmanabhan, and Graham Neubig. 2018.  When and why are pre-trained word embeddings useful for neural machine translation?  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 529–535.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of machine learning research_ , 21(140):1–67.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.
  * Schram et al. (2023) Viktoria Schram, Daniel Beck, and Trevor Cohn. 2023.  Performance prediction via bayesian matrix factorisation for multilingual natural language processing tasks.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 1790–1801.
  * Schwenk et al. (2021) Holger Schwenk, Vishrav Chaudhary, Shuo Sun, Hongyu Gong, and Francisco Guzmán. 2021.  Wikimatrix: Mining 135m parallel sentences in 1620 language pairs from wikipedia.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 1351–1361.
  * Srinivasan et al. (2021) Anirudh Srinivasan, Sunayana Sitaram, Tanuja Ganu, Sandipan Dandapat, Kalika Bali, and Monojit Choudhury. 2021.  Predicting the performance of multilingual nlp models.  _arXiv preprint arXiv:2110.08875_.
  * Tiedemann (2012) Jörg Tiedemann. 2012.  Parallel data, tools and interfaces in opus.  In _Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12)_ , pages 2214–2218.
  * Touvron et al. (2023a) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Winata et al. (2022) Genta Winata, Shijie Wu, Mayank Kulkarni, Thamar Solorio, and Daniel Preoţiuc-Pietro. 2022.  Cross-lingual few-shot learning on unseen languages.  In _Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 777–791.
  * Winata et al. (2024) Genta Indra Winata, Ruochen Zhang, and David Ifeoluwa Adelani. 2024.  Miners: Multilingual language models as semantic retrievers.  _arXiv preprint arXiv:2406.07424_.
  * Xia et al. (2020a) Mengzhou Xia, Antonios Anastasopoulos, Ruochen Xu, Yiming Yang, and Graham Neubig. 2020a.  Predicting performance for natural language processing tasks.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8625–8646.
  * Xia et al. (2020b) Mengzhou Xia, Antonios Anastasopoulos, Ruochen Xu, Yiming Yang, and Graham Neubig. 2020b.  Predicting performance for natural language processing tasks.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8625–8646, Online. Association for Computational Linguistics.
  * Ye et al. (2021a) Zihuiwen Ye, Pengfei Liu, Jinlan Fu, and Graham Neubig. 2021a.  Towards more fine-grained and reliable nlp performance prediction.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 3703–3714.
  * Ye et al. (2021b) Zihuiwen Ye, Pengfei Liu, Jinlan Fu, and Graham Neubig. 2021b.  Towards more fine-grained and reliable NLP performance prediction.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 3703–3714, Online. Association for Computational Linguistics.
  * Yong et al. (2024) Zheng-Xin Yong, Cristina Menghini, and Stephen H Bach. 2024.  Lexc-gen: Generating data for extremely low-resource languages with large language models and bilingual lexicons.  _arXiv preprint arXiv:2402.14086_.
  * Zhang et al. (2020) Biao Zhang, Philip Williams, Ivan Titov, and Rico Sennrich. 2020.  Improving massively multilingual neural machine translation and zero-shot translation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1628–1639.
  * Zoph et al. (2016) Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. 2016.  Transfer learning for low-resource neural machine translation.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 1568–1575.

##  Appendix A Experimental Details

Datasets | Languages Under Study | Domain
---|---|---
English-centric Dataset
FLoRes-200 Costa-jussà et al. (2022b) | afr, amh, arz, bak, bel, ceb, cym, dik, ewe, fao, | Multi-domain
| guj, hau, hne, hye, ibo, ind, jav, kan, kat, kaz, |
| khm, kin, kir, lmo, ltz, mar, mri, mya, oci, pan, |
| plt, sin, sna, snd, som, ssw, tam, tat, tgl, tuk, |
| wol, xho, yor, zul |
MT560 Gowda et al. (2021) |  |
Joshua Indian Corpus | tam | Wiki
Neulab TED Talk Qi et al. (2018) | bel, hye, kat, kaz, mar, mya, tam | TED
News Commentary | kaz | Economics
OPUS100 Zhang et al. (2020) | afr, amh, bel, cym, guj, hau, hye, ibo, ind, kan, | Multi-domain
| kat, kaz, khm, kin, kir, mar, mya, oci, pan, sin, |
| tam, tat, tuk, xho, yor, zul |
OPUS Bible | afr, amh, ceb, dik, ewe, guj, hye, ind, kan, mar, | Religion
Christodouloupoulos and Steedman (2015) | mri, mya, plt, sin, sna, som, ssw, tam, tgl, wol, |
| xho, zul |
OPUS OpenSubtitles | afr, hye, kat, sin, tgl | Movies
OPUS Tatoeba Tiedemann (2012) | afr, amh, arz, bak, bel, ceb, fao, hau, hye, ibo, | Conversational
| ind, jav, kan, kat, kaz, khm, kin, kir, lmo, ltz, |
| mar, mri, mya, pan, sna, tat, tgl, tuk, yor, zul |
OPUS Tanzil Tiedemann (2012) | amh, hau, ind, snd, som, tat | Religion
OPUS Gnome Tiedemann (2012) | fao, mri, som | Technology
OPUS GlobalVoices Tiedemann (2012) | amh, mya | News
OPUS EU Bookshop Tiedemann (2012) | cym | Government
OPUS SPC | afr | Government
OPUS Memat | xho | Medicine
OPUS XhosaNavy | xho | Government
OPUS KDE4 | hne | Technology
OPUS infopankki | som | Immigration
PMIndia Haddow and Kirefu (2020) | guj, kan, mar, pan, sin, tam | Government
Wiki Titles | guj, kaz, tam | Wiki
WikiMatrix Schwenk et al. (2021) | arz, bak, bel, ceb, fao, ind, jav, kat, kaz, lmo, | Wiki
| ltz, mar, oci, sin, tam, tat, tgl |
Many-to-Many Languages Dataset
NusaTranslation Cahyawijaya et al. (2023) | bew, btk, ind, jav, mad, mak, min, sun | Social Media

Table 7: List of datasets under study covering 50 different languages. We only
opt for 44 out of 500 languages available in the MT560 dataset, 44 out of 200
languages available in the FLoRes-200 dataset, and 8 out of 12 languages
available in the NusaTranslation dataset.

###  A.1 Languages Under Study

We list all the languages used in the training from the MT560 Gowda et al.
(2021) and NusaTranslation Cahyawijaya et al. (2023) datasets in Table 8 and
Table 9, respectively. The language code follows ∗ISO639-3 coding. All
languages are also complemented by their †rarity taxonomy based on Joshi et
al. (2021) into two vitality classes: 0-2→low resource language (LRL), and
3→mid resource language (MRL). We also provide information about whether the
language was part of the pre-trained M2M100 model dataset to highlight the
model knowledge coverage.

###  A.2 Models

Here are the details on the proxy LMs we use in the experiments as follows:

  * •

Transformer (100M) Vaswani et al. (2017): a standard encoder-decoder
transformer-based model with 6 encoder layers and 6 decoder layers with an
embedding dimension of 512. We train the model from randomly initialized
parameters with the training set.

  * •

SMaLL-100 (330M) Mohammadshahi et al. (2022):222SMaLL-100 (330M) is taken from
https://github.com/alirezamshi/small100. a distilled version of the M2M100
(12B) model. We utilize the model in two ways: fine-tuned on training data and
zero-shot inference.

  * •

M2M100 (No FT) Fan et al. (2021):333M2M100 (1.2B) is taken from
https://github.com/facebookresearch/fairseq/tree/main/examples/m2m_100. a pre-
trained estimated model of M2M100 (1.2B) without any fine-tuning. We run the
model in a zero-shot fashion.

  * •

NLLB (No FT) Costa-jussà et al. (2022b):444NLLB (1.3B) is taken from
https://github.com/facebookresearch/fairseq/tree/nllb. a pre-trained estimated
model of NLLB-200 Distilled (1.3B) without any fine-tuning. We run the model
in a zero-shot fashion.

Language | Language Code∗ | Family | Joshi Class† | Vitality† | Seen by M2M100
---|---|---|---|---|---
Afrikaans | afr | indo-european | 3 | MRL | ✓
Amharic | amh | afro-asiatic | 2 | LRL | ✓
Armenian | hye | indo-european | 1 | LRL | ✓
Bashkir | bak | turkic | 1 | LRL | ✓
Belarusian | bel | indo-european | 3 | MRL | ✓
Burmese | mya | sino-tibetan | 1 | LRL | ✓
Cebuano | ceb | austronesian | 3 | MRL | ✓
Chhattisgarhi | hne | indo-european | 0 | LRL | ✗
Dinka | dik | nilo-saharan | 1 | LRL | ✗
Egyptian Arabic | arz | afro-asiatic | 3 | MRL | ✓
Ewe | ewe | niger-congo | 1 | LRL | ✗
Faroese | fao | indo-european | 1 | LRL | ✗
Georgian | kat | kartvelian | 3 | MRL | ✓
Gujarati | guj | indo-european | 1 | LRL | ✓
Hausa | hau | afro-asiatic | 2 | LRL | ✓
Igbo | ibo | niger-congo | 1 | LRL | ✓
Indonesian | ind | austronesian | 3 | MRL | ✓
Javanese | jav | austronesian | 1 | LRL | ✓
Kannada | kan | dravidian | 1 | LRL | ✓
Kazakh | kaz | turkic | 3 | MRL | ✓
Khmer | khm | austro-asiatic | 1 | LRL | ✓
Kirghiz | kir | turkic | 1 | LRL | ✗
Kinyarwanda | kin | niger-congo | 1 | LRL | ✗
Lombard | lmo | indo-european | 1 | LRL | ✗
Luxembourgish | ltz | indo-european | 1 | LRL | ✓
Malagasy | plt | austronesian | 1 | LRL | ✓
Maori | mri | austronesian | 1 | LRL | ✗
Marathi | mar | indo-european | 2 | LRL | ✓
Occitan | oci | indo-european | 1 | LRL | ✓
Punjabi | pan | indo-european | 2 | LRL | ✓
Shona | sna | niger-congo | 1 | LRL | ✗
Sindhi | snd | indo-european | 1 | LRL | ✓
Sinhala | sin | indo-european | 0 | LRL | ✓
Somali | som | afro-asiatic | 1 | LRL | ✓
Swati | ssw | niger-congo | 1 | LRL | ✓
Tagalog | tgl | austronesian | 3 | MRL | ✓
Tamil | tam | dravidian | 3 | MRL | ✓
Tatar | tat | turkic | 1 | LRL | ✗
Turkmen | tuk | turkic | 1 | LRL | ✗
Welsh | cym | indo-european | 1 | LRL | ✓
Wolof | wol | niger-congo | 2 | LRL | ✓
Xhosa | xho | niger-congo | 2 | LRL | ✓
Yoruba | yor | niger-congo | 2 | LRL | ✓
Zulu | zul | niger-congo | 2 | LRL | ✓

Table 8: List of languages from the English-centric dataset, including their
rarity category mapping and an indication of whether they are involved in the
pre-training process for M2M100. Note that all languages are involved in the
pre-training process for NLLB.

###  A.3 Hyper-parameters

#### LM

Each fine-tuning and evaluation for LMs is done with an NVIDIA Tesla V100 32GB
GPU. The hyper-parameters used during fine-tuning from the English-centric and
Many-to-Many Languages datasets are listed in Table 10, 11, 12, and 13 for
SMaLL100, M2M100, NLLB, and Transformer models, respectively.

#### Regressor

Each regressor is trained on an AMD Ryzen Threadripper 2990WX with 128 GB of
RAM and 16 threads. Regressors’ hyper-parameters used are provided in Table
14, 15, 16, and 17 for XGBoost, Poly2/Poly3, LGBM, and MF, respectively. These
hyper-parameters were obtained based on the best cross-validation RMSE score
using 10 folds.

Language | Language Code∗ | Family | Joshi Class† | Vitality† | Seen by M2M100 | Seen by NLLB
---|---|---|---|---|---|---
Indonesian | ind | austronesian | 3 | MRL | ✓ | ✓
Javanese | jav | austronesian | 1 | LRL | ✓ | ✓
Betawi | bew | creole | 0 | LRL | ✗ | ✗
Batak | btk | austronesian | 0 | LRL | ✗ | ✗
Madurese | mad | austronesian | 0 | LRL | ✗ | ✗
Makassarese | mak | austronesian | 0 | LRL | ✗ | ✗
Minangkabau | min | austronesian | 0 | LRL | ✗ | ✓
Sundanese | sun | austronesian | 1 | LRL | ✓ | ✓

Table 9: List of languages from the Many-to-Many Languages dataset along with
their rarity category mapping and an indication of whether they are included
in the pre-training process for each respective model.

Hyper-parameter | English-centric | Many-to-Many Langs.
---|---|---
Encoder Layers | 12 | 12
Decoder Layers | 3 | 3
Encoder Embed Dim | 1024 | 1024
Decoder Embed Dim | 1024 | 1024
Encoder FFN Embed Dim | 4096 | 4096
Decoder FFN Embed Dim | 4096 | 4096
Encoder Attention Heads | 16 | 16
Decoder Attention Heads | 16 | 16
Encoder Layerdrop | 0.05 | 0.05
Decoder Layerdrop | 0.05 | 0.05
Optimizer | Adam | Adam
Adam Eps | 1e-6 | 1e-6
Adam Betas | (0.9, 0.98) | (0.9, 0.98)
Patience | 6 | 6
Batch Size | 16 | 16
Dropout | 0.1 | 0.1
Attention Dropout | 0.1 | 0.1
ReLU Dropout | 0.1 | 0.1
Weight Decay | 0.0 | 0.0
Label Smoothing | 0.1 | 0.1
Clip Norm | 1.0 | 1.0
Learning Rate | 0.0001 | 0.0003
Max Tokens (per GPU) | 1,000 | 1,000

Table 10: List of hyper-parameters used for SMaLL100 with English-centric and
Many-to-Many Languages datasets.

Hyper-parameter | English-centric | Many-to-Many Langs.
---|---|---
Encoder Layers | 24 | 24
Decoder Layers | 24 | 24
Encoder Embed Dim | 1024 | 1024
Decoder Embed Dim | 1024 | 1024
Encoder FFN Embed Dim | 8192 | 8192
Decoder FFN Embed Dim | 8192 | 8192
Encoder Attention Heads | 16 | 16
Decoder Attention Heads | 16 | 16
Encoder Layerdrop | 0.05 | 0.05
Decoder Layerdrop | 0.05 | 0.05
Optimizer | Adam | Adam
Adam Eps | 1e-6 | 1e-6
Adam Betas | (0.9, 0.98) | (0.9, 0.98)
Patience | 6 | 6
Batch Size | 32 | 32
Dropout | 0.1 | 0.1
Attention Dropout | 0.1 | 0.1
ReLU Dropout | 0.1 | 0.1
Weight Decay | 0.0 | 0.0
Label Smoothing | 0.1 | 0.1
Clip Norm | 0.0 | 0.0
Learning Rate | 0.0002 | 0.0002
Max Tokens (per GPU) | 1,792 | 1,792

Table 11: List of hyper-parameters used for M2M100 with English-centric and
Many-to-Many Languages datasets.

###  A.4 Regressor Dataset Sizes

We provide the details of the regressor’s training and test set size in Table
18.

##  Appendix B More Detailed Results

We provide detailed visualizations of the results of XGBoost with ProxyLM
Ensemble based on multiple language groupings in Figures 5-10 for English-
centric result, and Figures 11-14 for Many-to-Many Languages result. Each
language groupings plot comprises multiple subplots, including (a) vitality
class, (b) Joshi class, (c) language family, and (d) individual languages. The
mapping of vitality, Joshi class, and language family follows the
classifications in Table 8 and 9.

##  Appendix C Feature Importances

We provide feature importance scores of XGBoost with ProxyLM Ensemble for the
random English-centric experiment in Figure 15 and 16. Each combination
consists of one most influential feature followed by others with marginal
contributions to the model, each with an importance score of 0.12 or less. We
observe that proxy models are always the most influential features in
prediction.

##  Appendix D License for Artifacts

We discuss the license or terms for the use of any artifacts we use in Table
19.

Hyper-parameter | English-centric | Many-to-Many Langs.
---|---|---
Encoder Layers | 24 | 24
Decoder Layers | 24 | 24
Encoder Embed Dim | 1024 | 1024
Decoder Embed Dim | 1024 | 1024
Encoder FFN Embed Dim | 8192 | 8192
Decoder FFN Embed Dim | 8192 | 8192
Encoder Attention Heads | 16 | 16
Decoder Attention Heads | 16 | 16
Encoder Layerdrop | 0.05 | 0.05
Decoder Layerdrop | 0.05 | 0.05
Optimizer | Adam | Adam
Adam Eps | 1e-6 | 1e-6
Adam Betas | (0.9, 0.98) | (0.9, 0.98)
Patience | 6 | 6
Batch Size | 32 | 32
Dropout | 0.1 | 0.1
Attention Dropout | 0.1 | 0.1
ReLU Dropout | 0.0 | 0.0
Weight Decay | 0.01 | 0.01
Label Smoothing | 0.1 | 0.1
Clip Norm | 1.0 | 1.0
Learning Rate | 0.00002 | 0.0001
Max Tokens (per GPU) | 1,000 | 1,000

Table 12: List of hyper-parameters used for NLLB with English-centric and
Many-to-Many Languages datasets.

Hyper-parameter | English-centric | Many-to-Many Langs.
---|---|---
Encoder Layers | 6 | 6
Decoder Layers | 6 | 6
Encoder Embed Dim | 512 | 512
Decoder Embed Dim | 512 | 512
Encoder FFN Embed Dim | 2048 | 2048
Decoder FFN Embed Dim | 2048 | 2048
Encoder Attention Heads | 8 | 8
Decoder Attention Heads | 8 | 8
Encoder Layerdrop | 0.05 | 0.05
Decoder Layerdrop | 0.05 | 0.05
Optimizer | Adam | Adam
Adam Eps | 1e-6 | 1e-6
Adam Betas | (0.9, 0.98) | (0.9, 0.98)
Patience | 6 | 6
Batch Size | 32 | 32
Dropout | 0.1 | 0.1
Attention Dropout | 0.1 | 0.1
ReLU Dropout | 0.1 | 0.1
Weight Decay | 0.0001 | 0.0001
Label Smoothing | 0.1 | 0.1
Clip Norm | 0 | 0
Learning Rate | 0.001 | 0.0005
Max Tokens (per GPU) | 1,000 | 1,000

Table 13: List of hyper-parameters used for Transformer with English-centric
and Many-to-Many Languages datasets.

Hyper-parameter | Many-to-Many Langs. | English-centric
---|---|---
| M2M100 | NLLB | M2M100 | NLLB
max n_estimators | 2000 | 2000 | 5000 | 5000
max eta | 0.1 | 0.1 | 0.1 | 0.1
min_child_weight | 5 | 2.5 | 3 | 3.5
max_depth | 3 | 3 | 5 | 4
gamma | 0 | 0 | 0.2 | 0.1
subsample | 0.7 | 0.9 | 0.95 | 0.8
colsample_bytree | 0.6 | 0.6 | 0.85 | 1.0
reg_alpha | 0 | 0 | 0.1 | 0.1
reg_lambda | 0.35 | 0.15 | 0.1 | 0.3

Table 14: List of hyper-parameters used for XGBoost Regressor with M2M100 and
NLLB models trained with English-centric and Many-to-Many Languages datasets.

Hyper-parameter | Value
---|---
alpha | 0.1
l1_ratio | 0.9

Table 15: List of hyper-parameters used for Poly2/Poly3 Regressor with M2M100
and NLLB models trained with English-centric and Many-to-Many Languages
datasets.

Hyper-parameter | Value
---|---
max learning_rate | 0.3
max num_leaves | 64
n_estimators | 100
max_bin | 200000
max_depth | 10
min_child_weight | 0.001
min_child_samples | 20
min_split_gain | 0.0
colsample_bytree | 1.0
subsample | 1.0
reg_alpha | 0.1
reg_lambda | 0.1

Table 16: List of hyper-parameters used for LGBM Regressor with M2M100 and NLLB models trained with English-centric and Many-to-Many Languages datasets. “Max" indicates the maximum value set for the hyper-parameter during the hyper-parameter search. Hyper-parameter | Specification
---|---
max alpha | 0.01
beta_w | 0.1
beta_h | 0.1
beta_z | 0.01
beta_s | 0.01
beta_t | 0.01
lr_decay | 0.001
iterations | 2000
Table 17: List of hyper-parameters used for MF Regressor with M2M100 and NLLB
models trained with English-centric and Many-to-Many Languages datasets. “Max"
indicates the maximum value set for the hyper-parameter during the hyper-
parameter search.

Experimental Settings | Train Size | Test Size
---|---|---
Random (English-centric) | 829 | 356
Random (Many-to-Many Langs.) | 156 | 68
Unseen | 1,084 | 101
Cross-Dataset | 1,185 | 224

Table 18: Regressor’s training and test set size on different experimental
settings. The total experimental records for English-centric and Many-to-Many
Languages datasets are 1,185 and 224, respectively.

Datasets | URL Link | License
---|---|---
MT560 Gowda et al. (2021) | https://opus.nlpl.eu/MT560 | Unknown
FLoRes Costa-jussà et al. (2022b) | Muennighoff/flores200 | CC-BY-SA 4.0
NusaTranslation Cahyawijaya et al. (2023) | https://huggingface.co/datasets/indonlp/nusatranslation_mt | Apache 2.0

Table 19: List of datasets under study with their licenses.

(a)

(b)

(c)

Figure 5: Detailed results of XGBoost with ProxyLM Ensemble on NLLB model
under the LOLO setting using the English-centric dataset from Table 1. The
results are grouped by (a) Joshi Class and (b) language family that follows
the mapping which is provided in Appendix A.1; (c) shows the scatter plot
illustrating the correlation of spBLEU scores between the ProxyLM’s prediction
and estimated LM, with the gray dashed line representing the line of equality
($y=x$) with $R^{2}=0.89$ and black dashed line representing Locally Weighted
Scatterplot Smoothing (LOWESS) curve to represent the trend.

(a)

(b)

Figure 6: Detailed results of XGBoost with ProxyLM Ensemble under the LOLO
setting using the English-centric dataset from Table 1, grouped by vitality,
for (a) M2M100 and (b) NLLB model. Vitality is outlined in Table 8. Figure 7:
Detailed results of XGBoost with ProxyLM Ensemble on the M2M100 model under
the LOLO setting using the English-centric dataset from Table 1, categorized
by language. Figure 8: Performance heatmap of XGBoost with ProxyLM Ensemble on
the M2M100 model under the LOLO setting using the English-centric dataset from
Table 1. The heatmap showcases the performance based on vitality level and
language family, as outlined in Table 8. Figure 9: Detailed results of XGBoost
with ProxyLM Ensemble on the NLLB model under the LOLO setting using the
English-centric dataset from Table 1, categorized by language. Figure 10:
Performance heatmap of XGBoost with ProxyLM Ensemble on the NLLB model under
the LOLO setting using the English-centric dataset from Table 1. The heatmap
illustrates the model’s performance based on vitality levels and language
family classifications as specified in Table 8.

(a)

(b)

(c)

Figure 11: Detailed results of XGBoost with ProxyLM Ensemble on the M2M100
model under the LOLO setting using the Many-to-Many Languages dataset from
Table 3, organized by language groupings. The mapping of (a) Vitality, (b)
Joshi Class, and (c) language family align with the classifications provided
in Table 9. Figure 12: Detailed results of XGBoost with ProxyLM Ensemble on
the M2M100 model under the LOLO setting using the Many-to-Many Languages
dataset from Table 3, categorized by language.

(a)

(b)

(c)

Figure 13: Detailed results of XGBoost with ProxyLM Ensemble on the NLLB model
under the LOLO setting using the Many-to-Many Languages dataset from Table 3,
organized by language groups. The mapping of (a) Vitality, (b) Joshi Class,
and (c) language family corresponds to the classifications provided in Table
9. Figure 14: Detailed results of XGBoost with ProxyLM Ensemble on the NLLB
model under the LOLO setting using the Many-to-Many Languages dataset from
Table 3, categorized by language. Figure 15: Feature importance analysis of
XGBoost with ProxyLM Ensemble on the M2M100 model using the English-centric
dataset. Figure 16: Feature importance analysis of XGBoost with ProxyLM
Ensemble on the NLLB model using the English-centric dataset.