# PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning
Optimization

Yidong Wang1,2 Equal contribution. Yidong did this work during his internship
at Westlake University.  Zhuohao Yu1∗ Zhengran Zeng1 Linyi Yang2 Cunxiang
Wang2 Hao Chen3 Chaoya Jiang1 Rui Xie1 Jindong Wang3 Xing Xie3 Wei Ye1
Corresponding to<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Shikun Zhang1† Yue Zhang2†
1Peking University 2Westlake University 3Microsoft Research Asia

###### Abstract

Instruction tuning large language models (LLMs) remains a challenging task,
owing to the complexity of hyperparameter selection and the difficulty
involved in evaluating the tuned models. To determine the optimal
hyperparameters, an automatic, robust, and reliable evaluation benchmark is
essential. However, establishing such a benchmark is not a trivial task due to
the challenges associated with evaluation accuracy and privacy protection. In
response to these challenges, we introduce a judge large language model, named
PandaLM, which is trained to distinguish the superior model given several
LLMs. PandaLM’s focus extends beyond just the objective correctness of
responses, which is the main focus of traditional evaluation datasets. It
addresses vital subjective factors such as relative conciseness, clarity,
adherence to instructions, comprehensiveness, and formality. To ensure the
reliability of PandaLM, we collect a diverse human-annotated test dataset,
where all contexts are generated by humans and labels are aligned with human
preferences. Our results indicate that PandaLM-7B achieves 93.75% of GPT-3.5’s
evaluation ability and 88.28% of GPT-4’s in terms of F1-score on our test
dataset. PandaLM enables the evaluation of LLM to be fairer but with less
cost, evidenced by significant improvements achieved by models tuned through
PandaLM compared to their counterparts trained with default Alpaca’s
hyperparameters. In addition, PandaLM does not depend on API-based
evaluations, thus avoiding potential data leakage. All resources of PandaLM
are released at https://github.com/WeOpenML/PandaLM.

##  1 Introduction

Large language models (LLMs) have attracted increasing attention in the field
of artificial intelligence [1, 2, 3, 4, 5, 6], with various applications from
question answering [7, 8], machine translation [9, 10] to content creation
[11, 12]. The Alpaca project [13] has been a pioneering effort in instruction
tuning of LLaMA [14], setting a precedent for instruction tuning LLMs,
followed by Vicunna [15]. Subsequent research [16, 17, 18] have typically
adopted Alpaca’s hyperparameters as a standard for training their LLMs. Given
the necessity of instruction tuning for these pre-trained models to
effectively understand and follow natural language instructions [19, 13, 20],
optimizing their tuning hyperparameters is crucial for peak performance.
Critical factors such as optimizer selection, learning rate, number of
training epochs, and quality and size of training data significantly influence
the model’s performance [21, 22]. However, a research gap remains in the area
of hyperparameter optimization specifically designed for instruction tuning
LLMs. To address this issue, we aim to construct an automated, reliable, and
robust evaluation method, which can be integrated into any open-sourced LLMs
and used as the judging basis for hyperparameter optimization.

The development of such an evaluation method presents its own challenges,
including ensuring evaluation reliability and privacy protection. Current
methods often involve either crowd-sourcing work or API usage, which could be
costly, and time-consuming. Besides, these methods face challenges in terms of
consistency and reproducibility. This is primarily due to the lack of
transparency regarding language model change logs and the inherent
subjectivity of human annotations. Note that utilizing API-based evaluations
carries the risk of potentially high costs associated with addressing data
leaks. Although open-sourced LLMs can be alternative evaluators, they are not
specifically designed for assessment, thus making it difficult to deploy them
directly as evaluators.

On the other hand, the labels of previous evaluation methods [23, 24] simply
definite answers and fail to consider the language complexity in practice. The
evaluation metrics of these procedures are typically accuracy and F1-score,
without considering the subjective evaluation metrics that autoregressive
generative language models should pay attention to, thus does not reflect the
potential of such models to generate contextually relevant text. The
appropriate subjective evaluation metrics can be relative conciseness,
clarity, adherence to instructions, comprehensiveness, formality, and context
relevance.

To tackle these challenges, we introduce a judge language model, aiming for
Reproducible and Automated Language Model Assessment (PandaLM). Tuned from
LLaMA-7B, PandaLM is used to distinguish the most superior model among various
candidates, each fine-tuned with different hyperparameters, and is also
capable of providing the rationale behind its choice based on the reference
response for the context. PandaLM surpasses the limitations of traditional
evaluation methods and focuses on more subjective aspects, such as relative
conciseness, clarity, comprehensiveness, formality, and adherence to
instructions. Furthermore, the robustness of PandaLM is strengthened by its
ability to identify and rectify problems such as logical fallacies,
unnecessary repetitions, grammatical inaccuracies, and context irrelevance. By
considering these diverse aspects, we leverage PandaLM’s ability to
distinguish the most superior model among candidates on the validation set and
then provide insights for facilitating hyperparameter optimization of
instruction tuning.

In practice, we generate paired responses from a diverse set of similarly
sized foundation models including LLaMA-7B [14], Bloom-7B [25], Cerebras-
GPT-6.7B [26], OPT-7B [27], and Pythia-6.9B [28]. Each of these models is
fine-tuned using the same data and hyperparameters as Alpaca [13]. The paired
responses from these tuned LLMs constitute the input of training data for
PandaLM. The most straightforward approach to generate the corresponding
target of training data is through human annotation, but this method can be
costly and time-consuming [29]. Considering that GPT-3.5 has the ability to
provide reliable evaluation to some extent, to reduce costs, we follow self-
instruct [19] to distill data from GPT-3.5 and apply heuristic data filtering
strategies to mitigate noise.

To ensure the reliability of PandaLM, we develop a test dataset that aligns
with human preference and covers a wide range of tasks and contexts. The
instructions and inputs of test data are sampled from the human evaluation
dataset of self-instruct [19], with responses generated by different LLMs and
each label independently provided by three different human evaluators. Samples
with significant divergences are excluded to ensure the Inter Annotator
Agreement (IAA) of each annotator remains larger than 0.85. PandaLM-7B
demonstrates highly competitive performance, achieving 93.75% of GPT-3.5’s
evaluation ability and 88.28% of GPT4’s in terms of F1-score on our diverse
human-annotated test dataset.

(a) Comparison Results of GPT-3.5.

(b) Comparison Results of GPT-4.

(c) Comparison Results of Human.

Figure 1: The models are evaluated and compared using both GPT-3.5, GPT-4 and
human annotators. The ‘Win’ count represents the number of responses where
models fine-tuned with PandaLM-selected optimal hyperparameters outperform
models using Alpaca’s hyperparameters. Conversely, the ‘Lose’ count represents
the number of responses where models utilizing Alpaca’s hyperparameters
produce superior responses compared with those fine-tuned with the optimal
hyperparameters determined by PandaLM. Note that the overall test set
comprises 170 instances, and ‘Tie’ scenarios are not considered in this
illustration.

Moreover, as illustrated in Figure 1, adopting PandaLM’s selected optimal
hyperparameters covering optimizer selection, learning rate, number of
training epochs, and learning rate scheduler brings noteworthy improvements.
When assessed using GPT-4 with a set of 170 instructions, a group of five open
language models, tuned with optimal hyperparameters selected by PandaLM,
achieves an average of 47.0 superior responses and 26.2 inferior responses,
outperforming those trained using Alpaca’s hyperparameters. Note that the
training data remains the same for conducting fair comparisons. Moreover, when
these LLMs are evaluated by human experts, using the same set of 170
instructions, they exhibit an average of 79.8 superior responses and 25.2
inferior responses, once again surpassing the performance of models trained
with Alpaca’s hyperparameters. The experimental results underline the
effectiveness of PandaLM in determining optimal hyperparameters for choosing
the best LLMs. In addition, when the fine-tuned LLMs are assessed using tasks
from the lm-eval[24], a unified framework to test LLM on a large number of
different traditional evaluation tasks, the results further reinforce the
superiority of LLMs optimized by PandaLM.

In conclusion, our work delivers three key contributions:

  * •

We introduce PandaLM, a privacy-protected judge language model for evaluating
and optimizing hyperparameters for LLMs.

  * •

We create a reliable human-annotated dataset, essential for validating
PandaLM’s performance and further research.

  * •

We utilize PandaLM to optimize the hyperparameters of a series of open-sourced
LLMs. Tuning models with PandaLM-selected hyperparameters yields substantial
performance enhancements.

By open-sourcing PandaLM with the associated resources at
https://github.com/WeOpenML/PandaLM, we hope to facilitate further research
and inspire new advancements in this area.

##  2 Related Work

This section reviews the relevant literature on the topic of hyperparameter
optimization and the evaluation of language models.

Hyperparameter Optimization The importance of hyperparameter optimization in
machine learning [30, 31, 32, 33, 34, 35], particularly in the context of
fine-tuning deep learning language models such as BERT [36] and GPT [37],
cannot be ignored. For these models, the choice of hyperparameters like the
learning rate, batch size, or the number of training epochs can significantly
influence their performance [38, 39, 40]. This selection process becomes even
more critical when fine-tuning these models on domain-specific tasks, where
the optimal set of hyperparameters can vary significantly among different
domains [41, 39].

Evaluation of Language Models Accurate evaluation of language models is
crucial in determining optimal hyperparameters, thus improving the models’
overall performance [39, 38]. Conventional objective metrics like perplexity
[42] and accuracy [43, 44, 45, 46] on downstream tasks[24] provide valuable
insights, but they may not effectively guide the choice of hyperparameters to
enhance LLMs [47] because evaluating LLMs requires other subjective metrics.
Advanced language models, such as GPT-4 [1] and Bard [2], incorporate human
evaluations as part of their testing method for LLMs, aiming to better align
with human judgements [29]. Although human-based evaluation methods offer
considerable insight into a model’s performance, they are costly and labor-
intensive, making it less feasible for iterative hyperparameter optimization
processes.

Subjective qualitative analysis of a model’s outputs, such as its ability to
handle ambiguous instructions and provide contextually appropriate responses,
is increasingly being recognized as a valuable metric for evaluating models
[23]. Optimizing hyperparameters with considerations towards these qualitative
measures could lead to models that perform more robustly in diverse real-world
scenarios. The previous qualitative analysis can be achieved either through
human evaluators or through APIs of advanced language models, which is
different from our motivation.

Figure 2: The pipeline of instruction tuning LLMs.

##  3 Methodology

As shown in Figure 2, the process of instruction tuning begins with a
foundation model, which is then fine-tuned using instructions. The performance
of each tuned model is evaluated to determine the best output. This involves
exploring numerous models, each tuned with different hyperparameters, to
identify the optimal one. To facilitate this pipeline, a reliable and
automated language model assessment system is essential. To address this, we
introduce PandaLM - a judge LLM specifically designed to assess the
performance of LLMs fine-tuned with various parameters. Our goal is to
identify the superior model from a pool of candidates accurately.

###  3.1 Train Data Collection and Preprocessing

The training data collection aims to create a rich dataset that allows the
model to evaluate different responses in a given context and generate an
evaluation reason and a reference response using the same context. As
demonstrated in Figure 3, each training data instance consists of an input
tuple (instruction, input, response1, response2) and an output tuple
(evaluation result, evaluation reason, reference response). The instructions
and inputs in the input tuple are sampled from the Alpaca 52K dataset [13].
The response pairs are produced by various instruction-tuned models: LLaMA-7B
[14], Bloom-7B [25], Cerebras-GPT-6.7B [26], OPT-7B [27], and Pythia-6.9B
[28]. These models are selected due to their comparable sizes and the public
availability of their model weights. Each is fine-tuned using the same
instruction data and hyperparameters following Alpaca [13]. The corresponding
output tuple includes an evaluation result, a brief explanation for the
evaluation, and a reference response. The evaluation result would be either
‘1’ or ‘2’, indicating that response 1 or response 2 is better, and ‘Tie’
indicates that two responses are similar in quality. As it is impractical to
source millions of output tuples from human annotators, and given that GPT-3.5
is capable of evaluating LLMs to some degree, we follow self-instruct [19] to
generate output tuples using GPT-3.5. As illustrated in Figure 4, we design
prompts carefully to guide the generation of training data for PandaLM. The
goal is to ensure PandaLM not only prioritizes objective response correctness
but also emphasizes critical subjective aspects such as relative conciseness,
clarity, comprehensiveness, formality, and adherence to instructions. Besides,
we encourage PandaLM to identify and rectify issues like logical fallacies,
unnecessary repetitions, grammatical inaccuracies, and the absence of context
relevance. A heuristic data filtering strategy is then applied to remove noisy
data. Specifically, to address the observed inherent bias in GPT-3.5 regarding
the order of input responses even with carefully designed prompts, samples
from the training dataset are removed if their evaluation results conflict
when the orders of input responses are swapped. We finally obtain a filtered
dataset containing 300K samples. The training data and self-instruct prompts
are open-sourced at https://github.com/WeOpenML/PandaLM.

Figure 3: A training data example for PandaLM-7B. Figure 4: The top 16 words
used in the PandaLM-7B evaluation reasons from randomly sampled 80k evaluation
outputs. An example of evaluation reason and evaluation outputs can be found
in Figure 3. Stop words are filtered.

###  3.2 PandaLM-7B Training

In this subsection, we provide details about the training procedure for
PandaLM. The backbone of PandaLM is a 7B parameter variant of the LLaMA[14]
model, as it exhibits strong performance on multiple complicated NLP
tasks[48].

We train PandaLM with the DeepSpeed[49] library, and Zero Redundancy Optimizer
(ZeRO)[50, 51] Stage 2, on 8 NVIDIA A100-SXM4-80GB GPUs. We use the bfloat16
(BF16) computation precision option to further optimize the model’s speed and
efficiency. Regarding the training hyperparameters, we apply the AdamW[52]
optimizer with a learning rate of 2e-5 and a cosine learning rate scheduler.
The model is trained for 2 epochs. The training process utilizes a warmup
ratio of 0.03 to avoid large gradients at the beginning of training. We use a
batch size of 2 per GPU with all inputs truncated to a maximum of 1024 tokens
and employ a gradient accumulation strategy with 8 steps.

(a) DAG of GPT-3.5.

(b) DAG of GPT-4.

(c) DAG of humans.

(d) DAG of PandaLM.

Figure 5: Comparative Visualization of Model Performance. The instruction-
tuned models use the same training data and hyperparameters. A directed edge
from node A to B indicates model A’s significant superiority over B, while a
dashed undirected edge indicates the two models are similar in performance.
The number associated with the directed edge (A, B) represents the difference
between the number of wins and losses for model A compared to model B. The
absence of a number on the dashed undirected edge indicates that the
difference between the number of wins and losses for the models is smaller
than 5. We swap the order of two responses to perform inference twice on each
data. The conflicting evaluation results are then modified to ‘Tie’.

##  4 Reliability Evaluation of PandaLM-7B

To ensure the reliability of PandaLM-7B, we create a test dataset that is
labeled by humans and designed to align with human preferences for responses.
Each instance of this test dataset consists of one instruction and input, and
two responses produced by different instruction-tuned LLMs. The paired
responses are provided by LLaMA-7B, Bloom-7B, Cerebras-GPT-6.7B, OPT-7B, and
Pythia-6.9B, all instruction tuned using the same instruction data and
hyperparameters following Alpaca [13]. The test data is sampled from the
diverse human evaluation dataset of self-instruct [19], which includes data
from Grammarly, Wikipedia, National Geographic and nearly one hundred apps or
websites. The inputs and labels are solely human-generated and include a range
of tasks and contents. Three different human evaluators independently annotate
the labels indicating the preferred response. Samples with significant
divergences are excluded to ensure the Inter Annotator Agreement (IAA) of each
annotator remains larger than 0.85. This is because such samples demand
additional knowledge or hard-to-obtain information, making them challenging
for humans to evaluate. The filtered test dataset contains 1K samples, while
the original unfiltered dataset has 2.5K samples.

To maintain high-quality crowdsourcing work, we involve three experts to
annotate the same data point concurrently during the annotation process. These
experts receive specialized training that goes beyond evaluating response
correctness, enabling them to emphasize other crucial aspects like relative
conciseness, clarity, comprehensiveness, formality, and adherence to
instructions. Furthermore, we guide these annotators in identifying and
addressing issues such as logical fallacies, unnecessary repetitions,
grammatical inaccuracies, and a lack of contextual relevance. After the trial
phase of data annotation, we eliminate some low-quality labeled data. The
final IAA amongst the three annotators, as measured by Cohen’s Kappa [53],
yields average scores of 0.85, 0.86, and 0.88 respectively, indicating a
relatively high level of reliability for our test dataset. The distribution of
the test data comprises 105 instances of ties, 422 instances where Response 1
wins, and 472 instances where Response 2 takes the lead. Note that the human-
generated dataset has no personally identifiable information or offensive
content, and all annotators receive redundant labor fees.

Table 1: Comparative analysis of evaluation results from various annotation
models. The tuple in the table means (#win,#lose,#tie). Specifically,
(72,28,11) in the first line of the table indicates that LLaMA-7B outperforms
Bloom-7B in 72 responses, underperforms in 28, and matches the quality in 11
responses. The ‘Judged By’ column represents different methods of response
evaluation. ‘Human’ indicates that humans evaluate the result, and ‘PandaLM’
indicates that our proposed PandaLM model evaluates the result.

Judged By Base Model LLaMA-7B Bloom-7B Cerebras-6.7B OPT-7B Pythia-6.9B Human
LLaMA-7B / (72,28,11) (80,24,6) (71,24,11) (58,27,9) Bloom-7B (28,72,11) /
(59,30,11) (43,35,11) (47,49,11) Cerebras-6.7B (24,80,6) (30,59,11) /
(33,49,9) (27,53,11) OPT-7B (24,71,11) (35,43,11) (49,33,9) / (32,53,15)
Pythia-6.9B (27,58,9) (49,47,11) (53,27,11) (53,32,15) / GPT-3.5 LLaMA-7B /
(59,19,33) (71,13,26) (58,17,31) (49,16,29) Bloom-7B (19,59,33) / (40,19,41)
(36,30,23) (33,34,40) Cerebras-6.7B (13,71,26) (19,40,41) / (24,38,29)
(22,43,26) OPT-7B (17,58,31) (30,36,23) (38,24,29) / (30,30,40) Pythia-6.9B
(16,49,29) (34,33,40) (43,22,26) (30,30,40) / GPT-4 LLaMA-7B / (58,15,38)
(69,9,32) (58,14,34) (52,17,25) Bloom-7B (15,58,38) / (47,16,37) (35,31,23)
(32,33,42) Cerebras-6.7B (9,69,32) (16,47,37) / (23,40,28) (17,41,33) OPT-7B
(14,58,34) (31,35,23) (40,23,28) / (25,37,38) Pythia-6.9B (17,52,25)
(33,32,42) (41,17,33) (37,25,38) / PandaLM-7B LLaMA-7B / (46,29,36) (68,18,24)
(52,26,28) (35,28,31) Bloom-7B (29,46,36) / (50,18,32) (36,30,23) (36,31,40)
Cerebras-6.7B (18,68,24) (18,50,32) / (28,39,24) (24,46,21) OPT-7B (26,52,28)
(30,36,23) (39,28,24) / (30,32,38) Pythia-6.9B (28,35,31) (31,36,40)
(46,24,21) (32,30,38) /

Table 2: Comparison between Human Annotation results and Judged Model
evaluation results.

Judged Model Accuracy Precision Recall F1 GPT-3.5 0.6296 0.6195 0.6359 0.5820
GPT-4 0.6647 0.6620 0.6815 0.6180 PandaLM-7B 0.5926 0.5728 0.5923 0.5456

After obtaining the human-labeled test dataset, we can assess and compare the
evaluation performances of GPT-3.5, GPT-4, and PandaLM-7B. An interesting
observation from Table 1 is the shared similar partial order graph between
GPT-3.5, GPT-4, PandaLM-7B, and humans. Furthermore, Figure 5 illustrates
directed orders of model superiority (if model A outperforms model B, a
directed edge from A to B is drawn; if model A and model B perform similarly,
a dashed line from A to B is drawn.), and provides a visual representation of
comparative model effectiveness. The experimental results indicate
similarities in the preferences of GPT-3.5, GPT-4, PandaLM-7B, and humans.
Note that for PandaLM, GPT-3.5, and GPT-4, we swap the input response order
and infer twice to procure the final evaluation output. The conflicting
evaluation results are revised to ‘Tie’.

As shown in Table 2, we conduct a statistical analysis comparing the accuracy,
precision, recall, and F1-score of GPT-3.5, GPT-4, and PandaLM-7B against
human annotations. GPT-4 demonstrated superior performance, recording the
highest scores across all assessed metrics. Despite PandaLM-7B having the
lowest F1-score, it still demonstrates a notable performance, achieving 93.75%
of GPT-3.5’s evaluation ability and 88.28% of GPT-4’s in terms of F1-score.
Moreover, we are committed to continuously training larger-sized versions of
PandaLM to enhance its evaluation performance further.

In addition, beyond performance metrics, PandaLM-7B introduces unique
advantages that are not present in models like GPT-3.5 and GPT-4. It offers
open-source availability, enabling reproducibility, and protecting data
privacy. Furthermore, it provides unlimited access, removing any restrictions
that might hinder comprehensive evaluation and application.

##  5 Using PandaLM-7B to Instruction Tune LLMs

Table 3: Evaluation of the effectiveness of PandaLM’s selected hyperparameters
and Alpaca’s hyperparameters. The tuple in the table means (#win,#lose,#tie).
Specifically, (45,26,99) in the first line of the table indicates that
PandaLM’s hyperparameter-tuned LLaMA-7B outperforms Alpaca’s version in 45
responses, underperforms in 26, and matches the quality in 99 instances. The
‘Judged By’ column represents different methods of response evaluation.

Judge Model LLaMA-7B Bloom-7B Cerebras-6.7B OPT-7B Pythia-6.9B GPT-3.5
(45,26,99) (48,24,98) (58,21,91) (48,34,88) (59,20,91) GPT-4 (40,17,113)
(44,34,92) (60,20,90) (39,30,101) (52,30,88) Human (82,21,67) (79,23,68)
(88,25,57) (68,26,76) (82,31,57)

To highlight the effectiveness of using PandaLM-7B for instruction tuning
LLMs, we compare the performance of models tuned with PandaLM’s selected
optimal hyperparameters against those tuned with Alpaca’s parameters using
GPT-3.5, GPT-4, and human experts. This comparison evaluates multiple tuned
LLMs: LLaMA-7B, Bloom-7B, Cerebras-GPT-6.7B, OPT-7B, and Pythia-6.9B. The
assessment is conducted on a validation set comprising 170 distinct
instructions and inputs obtained from our 1K test set introduced in Section 4.
Alpaca’s tuning protocol involves training for three epochs with the final
iteration’s checkpoints being used. It uses the AdamW [52] optimizer with a
learning rate of 2e-5 and a cosine learning rate scheduler. We perform a wider
range of hyperparamters to tune LLMs using PandaLM-7B. Specifically, we
explore checkpoints from each epoch (ranging from epoch 1 to epoch 5), four
different learning rates (2e-6, 1e-5, 2e-5, 2e-4), two types of optimizers
(SGD[54] and AdamW), and two learning rate schedulers (cosine and linear). In
total, this creates a configuration space of 80 different possibilities per
model.

We search for optimal hyperparameters among the 80 configurations. These are
divided into four blocks, each containing 20 configurations. Sequential
comparisons identify the best configuration in each block. The top
configurations from each block are then compared to determine the overall best
configuration. We repeat each comparison twice for robustness and carry out
800 comparisons in total. The conflicting evaluation results are modified to
‘Tie’. Key insights from our tuning process include: Bloom-7B performs best
with SGD, a learning rate of 2e-5, and a cosine schedule over 5 epochs.
Cerebras-GPT-6.7B also favors SGD with the same learning rate but with a
linear schedule. LLaMA-7B prefers AdamW, a learning rate of 1e-5, and a linear
schedule over 4 epochs. OPT-6.7B achieves top results with AdamW, a learning
rate of 2e-5, and a linear scheduler over 5 epochs. Pythia-6.9B prefers SGD, a
learning rate of 1e-5, a cosine schedule, and 5 epochs. This highlights the
importance of customized hyperparameter tuning for different models to achieve
peak performance. We also provide the analysis on data size and quality and
LoRA when instruction tuning LLMs in Appendix B and Appedix C.

As illustrated in Table 3, for GPT-3.5, GPT-4, and human, all base models
achieve superior performance when tuned with PandaLM’s selected
hyperparameters compared to Alpaca’s hyperparameters. Note that the procedure
of switching the order of input responses, as applied for PandaLM, is also
implemented for GPT-3.5 and GPT-4 to acquire more robust evaluation results.
This outcome not only supports the claim that PandaLM-7B can enhance the
performance of models but also highlights its potential to further improve
various large language models. In addition, as shown in Appendix A, based on
PandaLM’s evaluation, the model demonstrating superior performance is LLaMA-
PandaLM. It leads the ranking, followed by LLaMA-Alpaca, Bloom-PandaLM,
Pythia-PandaLM, OPT-PandaLM, Cerebras-PandaLM, OPT-Alpaca, Bloom-Alpaca,
Pythia-Alpaca, and Cerebras-Alpaca. This order emphasizes the efficacy of
PandaLM’s approach in choosing hyperparameters, resulting in better model
performance. Models tuned using PandaLM’s hyperparameters tend to consistently
surpass those optimized with Alpaca’s hyperparameters in a hybrid ranking
scenario, reinforcing the effectiveness of PandaLM. However, the base
foundation model also plays a vital role, as demonstrated by LLaMA claiming
both the first and second positions in performance.

Moreover, Table 4 compares fine-tuned LLMs on various traditional tasks with
lm-eval[24]. We select classic yet challenging datasets that require strong
reasoning ability or real-world knowledge, as well as popular datasets from
existing LLM leaderboards[48]. The results show that models fine-tuned with
PandaLM consistently outperform those optimized with Alpaca across most tasks.
Specifically, the LLaMA-PandaLM model achieves the highest scores in most
tasks, demonstrating the effectiveness of PandaLM’s approach in model fine-
tuning. Even in other models like Bloom, Cerebras, OPT, and Pythia, we observe
a noticeable improvement in performance when PandaLM is used for optimization.

Table 4: Comparison on several downstream tasks using lm-eval[24] between
foundation models fine-tuned on Alpaca’s hyperparameters, and foundation
models fine-tuned with PandaLM. Note that all experiments are conducted zero-
shot on all datasets with the same prompts and settings.

LLMs ARC Challenge[55] CB[56] COQA[57] HellaSwag[58] SQuAD 2.0[59] WSC[60]
Accuracy Accuracy F1 Accuracy F1 Accuracy LLaMA-Alpaca 0.4206 0.5179 0.7335
0.7244 0.2239 0.3654 LLaMA-PandaLM 0.4249 0.5357 0.7420 0.7343 0.1807 0.4327
Bloom-Alpaca 0.3549 0.4464 0.0000 0.5985 0.0832 0.3654 Bloom-PandaLM 0.3515
0.4286 0.0002 0.5997 0.1137 0.3654 Cerebras-Alpaca 0.3063 0.1071 0.5565 0.5493
0.1163 0.3654 Cerebras-PandaLM 0.3174 0.3929 0.5665 0.5528 0.1319 0.3654 OPT-
Alpaca 0.3413 0.0893 0.6535 0.6488 0.1096 0.4135 OPT-PandaLM 0.3422 0.0893
0.6442 0.6503 0.1304 0.4904 Pythia-Alpaca 0.3387 0.3929 0.5859 0.6025 0.1443
0.3654 Pythia-PandaLM 0.3481 0.4464 0.6045 0.6260 0.1545 0.3654

##  6 Limitations

While the outcomes of our study are encouraging, we discuss several
limitations here. Firstly, the selected range of hyperparameters used in this
work is based on common practice and prior literature, and thus may not
encompass the absolute optimal hyperparameters. While extending the search
bond will inevitably increase the computational cost. Another limitation
pertains to the size of PandaLM. Currently, we only support a 7B version.
However, we are committed to continuously updating PandaLM to support larger
sizes, including 13B and 65B versions in the future.

##  7 Conclusion

In our exploration of hyperparameter optimization, we apply PandaLM-7B: an
automatic and reliable judge model for the tuning of LLMs. Our findings
demonstrate that the use of PandaLM-7B is feasible and consistently produces
models of superior performance compared to those tuned with Alpaca’s default
parameters. We are dedicated to continually enhancing PandaLM by expanding its
capacity to support larger models and analyzing its intrinsic features,
thereby developing increasingly robust versions of the judging model in the
future.

## References

  * [1] OpenAI.  Gpt-4 technical report.  2023\.
  * [2] Google.  Bard.  2023\.
  * [3] Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al.  Glm-130b: An open bilingual pre-trained model.  arXiv preprint arXiv:2210.02414, 2022.
  * [4] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  Advances in neural information processing systems, 33:1877–1901, 2020.
  * [5] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al.  Palm: Scaling language modeling with pathways.  arXiv preprint arXiv:2204.02311, 2022.
  * [6] Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al.  Palm 2 technical report.  arXiv preprint arXiv:2305.10403, 2023.
  * [7] Lynette Hirschman and Robert Gaizauskas.  Natural language question answering: the view from here.  natural language engineering, 7(4):275–300, 2001.
  * [8] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al.  Natural questions: a benchmark for question answering research.  Transactions of the Association for Computational Linguistics, 7:453–466, 2019.
  * [9] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [10] Felix Stahlberg.  Neural machine translation: A review.  Journal of Artificial Intelligence Research, 69:343–418, 2020.
  * [11] Som Biswas.  Chatgpt and the future of medical writing, 2023.
  * [12] Donnie Adams and Kee-Man Chuah.  Artificial intelligence-based tools in research writing: Current trends and future potentials.  Artificial Intelligence in Higher Education, pages 169–184, 2022\.
  * [13] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto.  Stanford alpaca: An instruction-following llama model.  https://github.com/tatsu-lab/stanford_alpaca, 2023.
  * [14] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.  Llama: Open and efficient foundation language models.  arXiv preprint arXiv:2302.13971, 2023.
  * [15] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.  See https://vicuna. lmsys. org (accessed 14 April 2023), 2023.
  * [16] Shizhe Diao, Rui Pan, Hanze Dong, KaShun Shum, Jipeng Zhang, Wei Xiong, and Tong Zhang.  Lmflow: An extensible toolkit for finetuning and inference of large foundation models.  https://optimalscale.github.io/LMFlow/, 2023.
  * [17] Yunjie Ji, Yong Deng, Yiping Peng Yan Gong, Qiang Niu, Lei Zhang, Baochang Ma, and Xiangang Li.  Exploring the impact of instruction data scaling on large language models: An empirical study on real-world use cases.  arXiv preprint arXiv:2303.14742, 2023.
  * [18] Sahil Chaudhary.  Code alpaca: An instruction-following llama model for code generation.  https://github.com/sahil280114/codealpaca, 2023.
  * [19] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi.  Self-instruct: Aligning language model with self generated instructions.  arXiv preprint arXiv:2212.10560, 2022.
  * [20] Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao.  Instruction tuning with gpt-4.  arXiv preprint arXiv:2304.03277, 2023.
  * [21] Richard Liaw, Eric Liang, Robert Nishihara, Philipp Moritz, Joseph E Gonzalez, and Ion Stoica.  Tune: A research platform for distributed model selection and training.  arXiv preprint arXiv:1807.05118, 2018.
  * [22] Mingxing Tan and Quoc Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  In International conference on machine learning, pages 6105–6114. PMLR, 2019.
  * [23] Lianmin Zheng, Ying Sheng, Wei-Lin Chiang, Hao Zhang, Joseph Gonzalez, E., and Ion Stoica.  Chatbot arena: Benchmarking llms in the wild with elo ratings.  GitHub repository, 2023.
  * [24] Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou.  A framework for few-shot language model evaluation, September 2021.
  * [25] Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al.  Bloom: A 176b-parameter open-access multilingual language model.  arXiv preprint arXiv:2211.05100, 2022.
  * [26] Nolan Dey, Gurpreet Gosal, Hemant Khachane, William Marshall, Ribhu Pathria, Marvin Tom, Joel Hestness, et al.  Cerebras-gpt: Open compute-optimal language models trained on the cerebras wafer-scale cluster.  arXiv preprint arXiv:2304.03208, 2023.
  * [27] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al.  Opt: Open pre-trained transformer language models.  arXiv preprint arXiv:2205.01068, 2022.
  * [28] Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al.  Pythia: A suite for analyzing large language models across training and scaling.  arXiv preprint arXiv:2304.01373, 2023.
  * [29] Yiming Wang, Zhuosheng Zhang, and Rui Wang.  Element-aware summarization with large language models: Expert-aligned evaluation and chain-of-thought method.  arXiv preprint arXiv:2305.13412, 2023.
  * [30] Tong Yu and Hong Zhu.  Hyper-parameter optimization: A review of algorithms and applications.  arXiv preprint arXiv:2003.05689, 2020.
  * [31] Stefan Falkner, Aaron Klein, and Frank Hutter.  Bohb: Robust and efficient hyperparameter optimization at scale.  In International Conference on Machine Learning, pages 1437–1446. PMLR, 2018.
  * [32] Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar.  Hyperband: A novel bandit-based approach to hyperparameter optimization.  The Journal of Machine Learning Research, 18(1):6765–6816, 2017\.
  * [33] Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley.  Baize: An open-source chat model with parameter-efficient tuning on self-chat data.  arXiv preprint arXiv:2304.01196, 2023.
  * [34] Chi Wang, Susan Xueqing Liu, and Ahmed H Awadallah.  Cost-effective hyperparameter optimization for large language model generation inference.  arXiv preprint arXiv:2303.04673, 2023.
  * [35] Jia Wu, Xiu-Yun Chen, Hao Zhang, Li-Dong Xiong, Hang Lei, and Si-Hao Deng.  Hyperparameter optimization for machine learning models based on bayesian optimization.  Journal of Electronic Science and Technology, 17(1):26–40, 2019\.
  * [36] Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In Proceedings of NAACL-HLT, pages 4171–4186, 2019.
  * [37] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al.  Improving language understanding by generative pre-training.
  * [38] Varun Godbole, George E. Dahl, Justin Gilmer, Christopher J. Shallue, and Zachary Nado.  Deep learning tuning playbook, 2023.  Version 1.0.
  * [39] Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang.  How to fine-tune bert for text classification?  In Chinese Computational Linguistics: 18th China National Conference, CCL 2019, Kunming, China, October 18–20, 2019, Proceedings 18, pages 194–206. Springer, 2019.
  * [40] Lewis Tunstall, Leandro Von Werra, and Thomas Wolf.  Natural language processing with transformers.  " O’Reilly Media, Inc.", 2022.
  * [41] Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith.  Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping.  arXiv preprint arXiv:2002.06305, 2020.
  * [42] Carlo A Mallio, Andrea C Sertorio, Caterina Bernetti, and Bruno Beomonte Zobel.  Large language models for structured reporting in radiology: performance of gpt-4, chatgpt-3.5, perplexity and bing.  La radiologia medica, pages 1–5, 2023.
  * [43] Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, et al.  Clue: A chinese language understanding evaluation benchmark.  In Proceedings of the 28th International Conference on Computational Linguistics, pages 4762–4772, 2020.
  * [44] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman.  Glue: A multi-task benchmark and analysis platform for natural language understanding.  In International Conference on Learning Representations.
  * [45] Linyi Yang, Shuibai Zhang, Libo Qin, Yafu Li, Yidong Wang, Hanmeng Liu, Jindong Wang, Xing Xie, and Yue Zhang.  Glue-x: Evaluating natural language understanding models from an out-of-distribution generalization perspective.  arXiv preprint arXiv:2211.08073, 2022.
  * [46] Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, and Dacheng Tao.  Can chatgpt understand too? a comparative study on chatgpt and fine-tuned bert.  arXiv preprint arXiv:2302.10198, 2023.
  * [47] Anna Rogers, Olga Kovaleva, and Anna Rumshisky.  A primer in bertology: What we know about how bert works.  Transactions of the Association for Computational Linguistics, 8:842–866, 2021.
  * [48] Edward Beeching, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf.  Open llm leaderboard.  https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard, 2023.
  * [49] Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He.  Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters.  In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 3505–3506, 2020.
  * [50] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He.  Zero: Memory optimizations toward training trillion parameter models.  In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–16. IEEE, 2020.
  * [51] Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, and Yuxiong He.  Zero-offload: Democratizing billion-scale model training.  In USENIX Annual Technical Conference, pages 551–564, 2021.
  * [52] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  arXiv preprint arXiv:1711.05101, 2017.
  * [53] Jacob Cohen.  Kappa: Coefficient of concordance.  Educ Psych Measurement, 20(37):37–46, 1960.
  * [54] Ian Goodfellow, Yoshua Bengio, and Aaron Courville.  Deep Learning.  MIT Press, 2016.
  * [55] Peter Clark, Isaac Cowhey, Oren Etzioni, Daniel Khashabi, Ashish Sabharwal, Carissa Spaulding, Oyvind Tafjord, and Peter D. Turney.  Think you have solved question answering? try arc, the ai2 reasoning challenge.  arXiv preprint arXiv:1803.05457, 2018.
  * [56] Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser.  The commitmentbank: Investigating projection in naturally occurring discourse.  In proceedings of Sinn und Bedeutung, volume 23, pages 107–124, 2019.
  * [57] Siva Reddy, Danqi Chen, and Christopher D Manning.  Coqa: A conversational question answering challenge.  In Transactions of the Association for Computational Linguistics, volume 7, pages 249–266, 2019.
  * [58] Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi.  Hellaswag: Can a machine really finish your sentence?  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, 2019.
  * [59] Pranav Rajpurkar, Robin Jia, and Percy Liang.  Know what you don’t know: Unanswerable questions for squad.  arXiv preprint arXiv:1806.03822, 2018.
  * [60] Hector J Levesque, Ernest Davis, and Leora Morgenstern.  The winograd schema challenge.  2011\.
  * [61] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al.  Lora: Low-rank adaptation of large language models.  In International Conference on Learning Representations.

##  Appendix A Directed Acyclic Graph depicting the mixture ranking of models
trained using both Alpaca’s and PandaLM’s hyperparameters.

A directed acyclic graph (DAG) is presented in Figure 6, illustrating the
relative rankings of various models fine-tuned with different sets of
hyperparameters. Notably, this ranking differs from those in Figure5, due to
the variance in the test data: the test data for 6 is a sampled subset from
that used in Figure5 which is deliberately chosen to ensure a high Inter-
Annotator Agreement (IAA). A discernible pattern emerges from the rankings:
models fine-tuned using PandaLM’s hyperparameters consistently outshine their
counterparts fine-tuned with Alpaca’s. The top-rated model is PandaLM-LLaMA,
followed by Alpaca-LLaMA, PandaLM-Bloom, PandaLM-Pythia, PandaLM-OPT, PandaLM-
Cerebras-GPT, Alpaca-OPT, Alpaca-Bloom, Alpaca-Pythia, and Alpaca-Cerebras-
GPT, in descending order of performance. This juxtaposition accentuates the
effectiveness of PandaLM’s hyperparameter selection in improving model
performance, as models optimized with PandaLM consistently rank higher than
those using Alpaca’s hyperparameters in the hybrid ranking. These findings
underscore the potential of PandaLM as a powerful tool in enhancing the
performance of large language models, further supporting the assertion of its
efficacy.

Figure 6: Directed Acyclic Graph depicting the mixture ranking of models
trained using both Alpaca’s and PandaLM’s hyperparameters. The models are
ranked from strongest to weakest in the following order: PandaLM-LLaMA,
Alpaca-LLaMA, PandaLM-Bloom, PandaLM-Pythia, PandaLM-OPT, PandaLM-Cerebras-
GPT, Alpaca-OPT, Alpaca-Bloom, Alpaca-Pythia, Alpaca-Cerebras-GPT.

##  Appendix B Data Size and Quality Analysis in Instruction Tuning

We conduct an ablation study to investigate the impact of training data size
(up to 1,344,000) on the performance of the model, given optimal
hyperparameters. Importantly, a relationship exists between the size and
quality of training data. Thus, we focus on an ablation study of data size
here, but conducting a similar experiment on data quality is feasible. We
derive the results from PandaLM-7B. The objective is to discern how much
training data is required to reach each model’s peak performance. Table 5
reveals the optimal quantity of training data varies among models. More
training data typically enhances model performance. However, an optimal point
exists for each model, beyond which further data doesn’t improve performance.
For example, the OPT model peaks at 992,000 data points, indicating additional
data does not enhance the model’s performance.

Table 5: Optimal training data size for each model.

Model Bloom Cerebras-GPT LLaMA OPT Pythia Optimal Training Data Size 1,216,000
1,344,000 11,520,000 992,000 1,344,000

##  Appendix C LoRA Analysis in Instruction Tuning

We further aim to evaluate the efficacy of Low-Rank Adaptation (LoRA) [61]
compared to full fine-tuning across various models, utilizing optimal
hyperparameters. The results are also obtained from PandaLM-7B. Our analysis
seeks to provide a comparative understanding of these tuning methodologies. As
shown in Table 6, the results for the Bloom model reveal a distinct advantage
for full fine-tuning, which triumphs over LoRA in 66 instances as opposed to
LoRA’s 35. Notably, they tie in 69 instances. In the case of the Cerebras
model, full fine-tuning again proves superior, leading in 59 cases compared to
LoRA’s 40, despite drawing even 71 times. The trend of full fine-tuning
superiority is consistent in the LLaMA model. Out of 170 instances, full fine-
tuning results in better performance in 48 instances, whereas LoRA emerges
victorious in only 28 instances. The majority of the results are tied,
amounting to 94 instances. In the OPT model, full fine-tuning once more
showcases its advantage with 64 instances of superior performance compared to
LoRA’s 33, while recording a tie in 73 instances. Lastly, for the Pythia
model, full fine-tuning leads the race with 71 instances of better performance
against LoRA’s 21, and a tie occurring in 78 instances. These results
underscore that full fine-tuning generally yields more favorable results
compared to the use of LoRA, though the outcomes can vary depending on the
model. Despite the considerable number of ties, full fine-tuning holds the
upper hand in most models, thereby highlighting its effectiveness. This
suggests that while LoRA may provide comparable results in some instances, a
strategy of full fine-tuning often proves to be the more beneficial approach
in enhancing model performance.

Table 6: Comparison of LoRA and Full Fine-tuning. Model | LoRA Wins | Full Fine-tuning Wins | Ties
---|---|---|---
Bloom | 35 | 66 | 69
Cerebras-GPT | 40 | 59 | 71
LLaMA | 28 | 48 | 94
OPT | 33 | 64 | 73
Pythia | 21 | 71 | 78

##  Appendix D Data License and Maintenance Plan

The test data we create is open sourced at
https://github.com/WeOpenML/PandaLM. The test data is under Apache License
2.0. For model weights of PandaLM, we follow LLaMA [14] license. We plan to
collect more multilingual test data from more apps and websites and open
source it for future research.