Large language models (LLMs) have revolutionized the field of artificial intelligence, enabling natural language processing tasks that were previously thought to be exclusive to humans. In this work, we introduce [ is a moniker of Qianwen, which means “thousands of prompts” in Chinese. The pronunciation of “” can vary depending on the context and the individual speaking it. Here is one possible way to pronounce it: /kwEn/.], the first installment of our large language model series.  is a comprehensive language model series that encompasses distinct models with varying parameter counts. It  includes , the base pretrained language models, and , the chat models finetuned with human alignment techniques.
The base language models consistently demonstrate superior performance across a multitude of downstream tasks, and the chat models, particularly those trained using Reinforcement Learning from Human Feedback (RLHF), are highly competitive.
The chat models possess advanced tool-use and planning capabilities for creating agent applications, showcasing impressive performance even when compared to bigger models on complex tasks like utilizing a code interpreter.
Furthermore, we have developed coding-specialized models,  and , as well as mathematics-focused models, , which are built upon base language models.
These models demonstrate significantly improved performance in comparison with open-source models, and slightly fall behind the proprietary models.

§ PRETRAINING

The pretraining stage involves learning vast amount of data to acquire a comprehensive understanding of the world and its various complexities. This includes not only basic language capabilities but also advanced skills such as arithmetic, coding, and logical reasoning.
In this section, we introduce the data, the model design and scaling, as well as the comprehensive evaluation results on benchmark datasets.

§.§ Data

The size of data has proven to be a crucial factor in developing a robust large language model, as highlighted in previous research [Hoffmann et al., 2022, Touvron et al., 2023]. To create an effective pretraining dataset, it is essential to ensure that the data are diverse and cover a wide range of types, domains, and tasks. Our dataset is designed to meet these requirements and includes public web documents, encyclopedia, books, codes, etc. Additionally, our dataset is multilingual, with a significant portion of the data being in English and Chinese.

To ensure the quality of our pretraining data, we have developed a comprehensive data preprocessing procedure. For public web data, we extract text from HTML and use language identification tools to determine the language. To increase the diversity of our data, we employ deduplication techniques, including exact-match deduplication after normalization and fuzzy deduplication using MinHash and LSH algorithms.
To filter out low-quality data, we employ a combination of rule-based and machine-learning-based methods. Specifically, we use multiple models to score the content, including language models, text-quality scoring models, and models for identifying potentially offensive or inappropriate content. We also manually sample texts from various sources and review them to ensure their quality.
To further enhance the quality of our data, we selectively up-sample data from certain sources, to ensure that our models are trained on a diverse range of high-quality content.
In recent studies [Zeng et al., 2022, Aribandi et al., 2021, Raffel et al., 2020], it has been demonstrated that pretraining language models with multi-task instructions can enhance their zero-shot and few-shot performance. To further enhance the performance of our model, we have incorporated high-quality instruction data into our pretraining process.
To safeguard the integrity of our benchmark assessment, we have adopted a similar approach as Brown et al., 2020 and meticulously eliminated any instruction samples that exhibit a 13-gram overlap with any data present in the test sets utilized in our evaluation.
Given the large number of downstream tasks, it is not feasible to repeat this filtering process for all tasks. Instead, we have made sure that the instruction data for the reported tasks have undergone our filtering process to ensure their accuracy and reliability.
Finally, we have built a dataset of up to 3 trillion tokens.

§.§ Tokenization

The design of vocabulary significantly impacts the training efficiency and the downstream task performance.
In this study, we utilize byte pair encoding (BPE) as our tokenization method, following GPT-3.5 and GPT-4. We start with the open-source fast BPE tokenizer, tiktoken [Jain, 2022], and select the vocabulary cl100k base as our starting point.
To enhance the performance of our model on multilingual downstream tasks, particularly in Chinese, we augment the vocabulary with commonly used Chinese characters and words, as well as those in other languages.
Also, following Touvron et al., 2023, Touvron et al., 2023, we have split numbers into single digits.
The final vocabulary size is approximately 152K.

The performance of the  tokenizer in terms of compression is depicted in Figure <ref>. In this comparison, we have evaluated  against several other tokenizers, including XLM-R [Conneau et al., 2019], LLaMA [Touvron et al., 2023], Baichuan [Inc., 2023], and InternLM [InternLM Team, 2023]. Our findings reveal that  achieves higher compression efficiency than its competitors in most languages. This implies that the cost of serving can be significantly reduced since a smaller number of tokens from  can convey more information than its competitors.
Furthermore, we have conducted preliminary experiments to ensure that scaling the vocabulary size of  does not negatively impact the downstream performance of the pretrained model. Despite the increase in vocabulary size, our experiments have shown that  maintains its performance levels in downstream evaluation.

Encoding compression rates of different models. We randomly selected 1 million document corpora of each language to test and compare the encoding compression rates of different models (with XLM-R [Conneau et al., 2019], which supports 100 languages, as the base value 1, not shown in the figure). As can be seen, while ensuring the efficient decoding of Chinese, English, and code,  also achieves a high compression rate for many other languages (such as th, he, ar, ko, vi, ja, tr, id, pl, ru, nl, pt, it, de, es, fr, etc.), equipping the model with strong scalability as well as high training and inference efficiency in these languages.

§.§ Architecture

is designed using a modified version of the Transformer architecture. Specifically, we have adopted the recent open-source approach of training large language models, LLaMA [Touvron et al., 2023], which is widely regarded as the top open-source LLM. Our modifications to the architecture include:

* Embedding and output projection. Based on preliminary experimental findings, we have opted for the untied embedding approach instead of tying the weights of input embedding and output projection. This decision was made in order to achieve better performance with the price of memory costs.

* Positional embedding. We have chosen RoPE (Rotary Positional Embedding) [Su et al., 2021] as our preferred option for incorporating positional information into our model. RoPE has been widely adopted and has demonstrated success in contemporary large language models, notably PaLM [Chowdhery et al., 2022, Anil et al., 2023] and LLaMA [Touvron et al., 2023, Touvron et al., 2023]. In particular, we have opted to use FP32 precision for the inverse frequency matrix, rather than BF16 or FP16, in order to prioritize model performance and achieve higher accuracy.

* Bias. For most layers, we remove biases following Chowdhery et al., 2022, but we add biases in the QKV layer of attention to enhance the extrapolation ability of the model [Su, 2023].

* Pre-Norm & RMSNorm. In modern Transformer models, pre-normalization is the most widely used approach, which has been shown to improve training stability compared to post-normalization. Recent research has suggested alternative methods for better training stability, which we plan to explore in future versions of our model. Additionally, we have replaced the traditional layer normalization technique described in [Ba et al., 2016] with RMSNorm [Jiang et al., 2023]. This change has resulted in equivalent performance while also improving efficiency.

* Activation function. We have selected SwiGLU [Shazeer, 2020] as our activation function, a combination of Swish [Ramachandran et al., 2017] and Gated Linear Unit [Dauphin et al., 2017]. Our initial experiments have shown that activation functions based on GLU generally outperform other baseline options, such as GeLU [Hendrycks & Gimpel, 2016]. As is common practice in previous research, we have reduced the dimension of the feed-forward network (FFN) from 4 times the hidden size to $\frac{8}{3}$ of the hidden size.

Model sizes, architectures, and optimization hyper-parameters.

# of Params     Hidden size     Heads     Layers     Learning rate     Batch size     Training tokens

1.8B         2048            16          24           $3.0 \times 10 ^ {-4} $         4M             2.2T

7B           4096            32          32           $3.0 \times 10 ^ {-4} $          4M             2.4T

14B          5120            40          40           $3.0 \times 10 ^ {-4} $          4M             3.0T

§.§ Training

To train , we follow the standard approach of autoregressive language modeling, as described in Radford et al., 2018. This involves training the model to predict the next token based on the context provided by the previous tokens.
We train models with context lengths of 2048. To create batches of data, we shuffle and merge the documents, and then truncate them to the specified context lengths.
To improve computational efficiency and reduce memory usage, we employ Flash Attention in the attention modules [Dao et al., 2022].
We adopt the standard optimizer AdamW [Kingma & Ba, 2014, Loshchilov & Hutter, 2017] for pretraining optimization. We set the hyperparameters $\beta_1=0.9$, $\beta_2=0.95$, and $\epsilon=10^{-8}$. We use a cosine learning rate schedule with a specified peak learning rate for each model size.  The learning rate is decayed to a minimum learning rate of $10\%$ of the peak learning rate.
All the models are trained with BFloat16 mixed precision for training stability.

§.§ Context Length Extension

Transformer models have a significant limitation in terms of the context length for their attention mechanism. As the context length increases, the quadratic-complexity computation leads to a drastic increase in both computation and memory costs. In this work, we have implemented simple training-free techniques that are solely applied during inference to extend the context length of the model. One of the key techniques we have used is NTK-aware interpolation [bloc97, 2023]. Unlike position interpolation (PI) [Chen et al., 2023] which scales each dimension of RoPE equally, NTK-aware interpolation adjusts the base of RoPE to prevent the loss of high-frequency information in a training-free manner. To further improve performance, we have also implemented a trivial extension called dynamic NTK-aware interpolation, which is later formally discussed in [Peng et al., 2023]. It dynamically changes the scale by chunks, avoiding severe performance degradation. These techniques allow us to effectively extend the context length of Transformer models without compromising their computational efficiency or accuracy.

additionally incorporates two attention mechanisms: LogN-Scaling [Chiang & Cholak, 2022, Su, 2023] and window attention [Beltagy et al., 2020]. LogN-Scaling rescales the dot product of the query and value by a factor that depends on the ratio of the context length to the training length, ensuring that the entropy of the attention value remains stable as the context length grows.
Window attention restricts the attention to a limited context window, preventing the model from attending to tokens that are too far away.

We also observed that the long-context modeling ability of our model varies across layers, with lower layers being more sensitive in context length extension compared to the higher layers.
To leverage this observation, we assign different window sizes to each layer, using shorter windows for lower layers and longer windows for higher layers.

Overall performance on widely-used benchmarks compared to open-source base models. Our largest  model with 14 billion parameters outperforms previous 13B SoTA models on all datasets.

blackModel      Params       MMLU          C-Eval      GSM8K      MATH       HumanEval      MBPP         BBH

5-shot        5-shot           8-shot     4-shot       0-shot         3-shot       3-shot
2*MPT     7B     30.8     23.5     9.1     3.0     18.3     22.8     35.6

30B     47.9     -     15.2     3.1     25.0     32.8     38.0

lightgray2*Falcon     7B     27.8     -     6.8     2.3     -     11.2     28.0

40B     57.0     -      19.6     5.5      -     29.8     37.1

lightgrayChatGLM2     6B         47.9          51.7       32.4      6.5        -            -         33.7

lightgray2*InternLM     7B         51.0          53.4          31.2     6.3          10.4           14.0         37.0

20B         62.1          58.8          52.6       7.9       25.6           35.6         52.5
lightgray2*Baichuan2     7B        54.7          56.3        24.6      5.6         18.3           24.2         41.6

13B     59.5          59.0         52.8     10.1          17.1           30.2         49.0
4*LLaMA     7B           35.6          27.3        11.0     2.9         12.8           17.7         33.5

13B          47.7          31.8          20.3     4.2         15.8           22.0         37.9

33B          58.7          37.5           42.3      7.1        21.7           30.2         50.0

65B          63.7          40.4        54.4      10.6        23.7            37.7         58.4

4*     7B           46.8          32.5          16.7       3.3       12.8           20.8         38.2

13B          55.0          41.4         29.6      5.0        18.9           30.3         45.6

34B          62.6          -         42.2      6.2        22.6           33.0         44.1

70B          69.8          50.1        63.3      13.5        29.9            45.0          64.9
StableBeluga2     70B        68.6       51.4       69.6       14.6           28.0           11.4         69.3
3*     1.8B       44.6          54.7          21.2      5.6        17.1           14.8            28.2

7B         58.2          63.5        51.7       11.6       29.9           31.6         45.0

14B         66.3      72.1      61.3     24.8     32.3       40.8      53.4

§.§ Experimental Results

To evaluate the zero-shot and few-shot learning capabilities of our models, we conduct a thorough benchmark assessment using a series of datasets. We compare  with the most recent open-source base models, including LLaMA [Touvron et al., 2023],  [Touvron et al., 2023], MPT [Mosaic ML, 2023], Falcon [Almazrouei et al., 2023], Baichuan2 [Yang et al., 2023], ChatGLM2 [ChatGLM2 Team, 2023], InternLM [InternLM Team, 2023], XVERSE [Inc., 2023], and StableBeluga2 [Stability AI, 2023]. Our evaluation covers a total of 7 popular benchmarks, which are MMLU (5-shot) [Hendrycks et al., 2020], C-Eval (5-shot) [Huang et al., 2023], GSM8K (8-shot) [Cobbe et al., 2021], MATH (4-shot) [Hendrycks et al., 2021], HumanEval (0-shot) [Chen et al., 2021], MBPP (0-shot) [Austin et al., 2021], and BBH (Big Bench Hard) (3 shot) [Suzgun et al., 2022]. We aim to provide a comprehensive summary of the overall performance of our models across these benchmarks.

In this evaluation, we focus on the base language models without alignment and collect the baselines' best scores from their official results and OpenCompass [OpenCompass Team, 2023].
The results are presented in Table <ref>.

Our experimental results demonstrate that the three  models exhibit exceptional performance across all downstream tasks. It is worth noting that even the larger models, such as LLaMA2-70B, are outperformed by -14B in 3 tasks.
-7B also performs admirably, surpassing LLaMA2-13B and achieving comparable results to Baichuan2-13B. Notably, despite having a relatively small number of parameters, -1.8B is capable of competitive performance on certain tasks and even outperforms larger models in some instances.
The findings highlight the impressive capabilities of the  models, particularly -14B, and suggest that smaller models, such as -1.8B, can still achieve strong performance in certain applications.

Results of  on long-context inference using various techniques. Our experimental findings reveal that the application of our crucial techniques enables the model to consistently achieve low perplexity as the context length increases. This suggests that these techniques play a significant role in enhancing the model's ability to comprehend and generate lengthy texts.

2*[-]Model                  5cSequence Length
1c1024     1c2048     1c4096     1c8192     1c16384
-7B                                 4.23     3.78     39.35     469.81     2645.09

+ dynamic_ntk                          4.23     3.78     3.59      3.66       5.71

+ dynamic_ntk + logn                   4.23     3.78     3.58      3.56       4.62

+ dynamic_ntk + logn + window_attn     4.23     3.78     3.58      3.49       4.32
-14B     -     3.46     22.79     334.65     3168.35

+ dynamic_ntk + logn + window_attn     -     3.46     3.29     3.18       3.42

To evaluate the effectiveness of context length extension, Table <ref> presents the test results on arXiv[The dataset contains academic papers from <https://arxiv.org>.] in terms of perplexity (PPL).
These results demonstrate that by combining NTK-aware interpolation, LogN-Scaling, and layer-wise window assignment, we can effectively maintain the performance of our models in the context of over 8192 tokens.

§ ALIGNMENT

Pretrained large language models have been found to be not aligned with human behavior, making them unsuitable for serving as AI assistants in most cases. Recent research has shown that the use of alignment techniques, such as supervised finetuning (SFT) and reinforcement learning from human feedback (RLHF), can significantly improve the ability of language models to engage in natural conversation. In this section, we will delve into the details of how  models have been trained using SFT and RLHF, and evaluate their performance in the context of chat-based assistance.

§.§ Supervised Finetuning

To gain an understanding of human behavior, the initial step is to carry out SFT, which finetunes a pretrained LLM on chat-style data, including both queries and responses. In the following sections, we will delve into the details of data construction and training methods.

§.§.§ Data

To enhance the capabilities of our supervised finetuning datasets, we have annotated conversations in multiple styles. While conventional datasets [Wei et al., 2022] contain a vast amount of data prompted with questions, instructions, and answers in natural language, our approach takes it a step further by annotating human-style conversations. This practice, inspired by Ouyang et al., 2022, aims at improving the model's helpfulness by focusing on natural language generation for diverse tasks.
To ensure the model's ability to generalize to a wide range of scenarios, we specifically excluded data formatted in prompt templates that could potentially limit its capabilities. Furthermore, we have prioritized the safety of the language model by annotating data related to safety concerns such as violence, bias, and pornography.

In addition to data quality, we have observed that the training method can significantly impact the final performance of the model. To achieve this, we utilized the ChatML-style format [OpenAI, 2022], which is a versatile meta language capable of describing both the metadata (such as roles) and the content of a turn. This format enables the model to effectively distinguish between various types of information, including system setup, user inputs, and assistant outputs, among others. By leveraging this approach, we can enhance the model's ability to accurately process and analyze complex conversational data.

§.§.§ Training

Consistent with pretraining, we also apply next-token prediction as the training task for SFT. We apply the loss masks for the system and user inputs. More details are demonstrated in Section <ref>.

The model's training process utilizes the AdamW optimizer, with the following hyperparameters: $\beta_1$ set to 0.9, $\beta_2$ set to 0.95,  and $\epsilon$ set to $10^{-8}$.
The sequence length is limited to 2048, and the batch size is 128.
The model undergoes a total of 4000 steps,  with the learning rate gradually increased over the first 1430 steps, reaching a peak of $2 \times 10^{-6}$.
To prevent overfitting, weight decay is applied with a value of 0.1, dropout is set to 0.1, and gradient clipping is enforced with a limit of 1.0.

§.§ Reinforcement Learning from Human Feedback

While SFT has proven to be effective, we acknowledge that its generalization and creativity capabilities may be limited, and it is prone to overfitting. To address this issue, we have implemented Reinforcement Learning from Human Feedback (RLHF) to further align SFT models with human preferences, following the approaches of Ouyang et al., 2022, Christiano et al., 2017. This process involves training a reward model and using Proximal Policy Optimization (PPO) [Schulman et al., 2017] to conduct policy training.

§.§.§ Reward Model

To create a successful reward model, like building a large language model (LLM), it is crucial to first undergo pretraining and then finetuning. This pretraining process, also known as preference model pretraining (PMP) [Bai et al., 2022], necessitates a vast dataset of comparison data. This dataset consists of sample pairs, each containing two distinct responses for a single query and their corresponding preferences. Similarly, finetuning is also conducted on this type of comparison data, but with a higher quality due to the presence of quality annotations.

During the fine-tuning phase, we gather a variety of prompts and adjust the reward model based on human feedback for responses from the  models. To ensure the diversity and complexity of user prompts are properly taken into account, we have created a classification system with around 6600 detailed tags and implemented a balanced sampling algorithm that considers both diversity and complexity when selecting prompts for annotation by the reward model [Lu et al., 2023]. To generate a wide range of responses, we have utilized  models of different sizes and sampling strategies, as diverse responses can help reduce annotation difficulties and enhance the performance of the reward model. These responses are then evaluated by annotators following a standard annotation guideline, and comparison pairs are formed based on their scores.

In creating the reward model, we utilize the same-sized pre-trained language model  to initiate the process. It is important to mention that we have incorporated a pooling layer into the original  model to extract the reward for a sentence based on a specific end token.
The learning rate for this process has been set to a constant value of $3 \times 10^{-6}$,  and the batch size is 64. Additionally, the sequence length is set to 2048, and the training process lasts for a single epoch.

We adopted the accuracy on the test dataset as an important but not exclusive evaluation metric for the reward model. In Table <ref>, we report the test pairwise accuracy of PMP and reward models on diverse human preference benchmark datasets [Bai et al., 2022, Stiennon et al., 2020, Ethayarajh et al., 2022, Lightman et al., 2023]. Specifically,  Helpful-base and  Helpful-online are our proprietary datasets. The responses in  Helpful-base are generated from  without RLHF, whereas  Helpful-online includes responses from  with RLHF. The results show that the PMP model demonstrates high generalization capabilities on out-of-distribution data, and the reward model demonstrates significant improvement on our  reward datasets.

Test Accuracy of  preference
model pretraining (PMP) and reward model (RM) on diverse human preference benchmark datasets.

2*Dataset                   Anthropic      Anthropic      OpenAI      Stanford     OpenAI

Helpful-base      Helpful-online      Helpful-base      Helpful-online      Summ.      SHP     PRM800K
PMP            62.68                      61.62                        76.52                         65.43         69.60               60.05             70.59

RM             74.78                      69.71                        73.98                         64.57         69.99               60.10             70.52

§.§.§ Reinforcement Learning

Our Proximal Policy Optimization (PPO) process involves four models: the policy model, value model, reference model, and reward model. Before starting the PPO procedure, we pause the policy model's updates and focus solely on updating the value model for 50 steps. This approach ensures that the value model can adapt to different reward models effectively.

During the PPO operation, we use a strategy of sampling two responses for each query simultaneously. This strategy has proven to be more effective based on our internal benchmarking evaluations. We set the KL divergence coefficient to 0.04 and normalize the reward based on the running mean.

The policy and value models have learning rates of $1 \times 10^{-6}$ and $5 \times 10^{-6}$, respectively. To enhance training stability, we utilize value loss clipping with a clip value of 0.15. For inference, the policy top-p is set to 0.9. Our findings indicate that although the entropy is slightly lower than when top-p is set to 1.0, there is a faster increase in reward, ultimately resulting in consistently higher evaluation rewards under similar conditions.

Additionally, we have implemented a pretrained gradient to mitigate the alignment tax.
Empirical findings indicate that, with this specific reward model, the KL penalty is adequately robust to counteract the alignment tax in benchmarks that are not strictly code or math in nature, such as those that test common sense knowledge and reading comprehension. It is imperative to utilize a significantly larger volume of the pretrained data in comparison to the PPO data to ensure the effectiveness of the pretrained gradient. Additionally, our empirical study suggests that an overly large value for this coefficient can considerably impede the alignment to the reward model, eventually compromising the ultimate alignment, while an overly small value would only have a marginal effect on alignment tax reduction.

§.§ Automatic and Human Evaluation of Aligned Models

To showcase the effectiveness of our aligned models, we conduct a comparison with other aligned models on well-established benchmarks, including MMLU [Hendrycks et al., 2020], C-Eval [Huang et al., 2023], GSM8K [Cobbe et al., 2021], HumanEval [Chen et al., 2021], and BBH [Suzgun et al., 2022].
Besides the widely used few-shot setting, we test our aligned models in the zero-shot setting to demonstrate how well the models follow instructions. The prompt in a zero-shot setting consists of an instruction and a question without any previous examples in the context. The results of the baselines are collected from their official reports and OpenCompass [OpenCompass Team, 2023].

The results in Table <ref> demonstrate the effectiveness of our aligned models in understanding human instructions and generating appropriate responses. -14B-Chat outperforms all other models except ChatGPT [OpenAI, 2022] and -70B [Touvron et al., 2023] in all datasets, including MMLU [Hendrycks et al., 2020], C-Eval [Huang et al., 2023], GSM8K [Cobbe et al., 2021], HumanEval [Chen et al., 2021], and BBH [Suzgun et al., 2022]. In particular, 's performance in HumanEval, which measures the quality of generated codes, is significantly higher than that of other open-source models.

Moreover, 's performance is consistently better than that of open-source models of similar size, such as LLaMA2 [Touvron et al., 2023], ChatGLM2 [ChatGLM2 Team, 2023], InternLM [InternLM Team, 2023], and Baichuan2 [Yang et al., 2023]. This suggests that our alignment approach, which involves fine-tuning the model on a large dataset of human conversations, has been effective in improving the model's ability to understand and generate human-like language.

Performance of aligned models on widely-used benchmarks. We report both zero-shot and few-shot performance of the models.

Model      Params              2cMMLU          2cC-Eval        2cGSM8K         HumanEval          2cBBH

0-shot     5-shot         0-shot     5-shot       0-shot     8-shot         0-shot       0-shot     3-shot
11cProprietary models

blackGPT-3.5     -       -     69.1         -     52.5       -       78.2          73.2          -       70.1

GPT-4     -     -     83.0       -     69.9       -      91.4         86.6          -       86.7

black11cOpen-source models

black1*ChatGLM2       6B     45.5      46.0        50.1     52.6      -       28.8          11.0        -         32.7

lightgrayInternLM-Chat      7B      -     51.1       -       53.6       -       33.0          14.6           -      32.5
2*Baichuan2-Chat      7B      -     52.9       -       55.6        -      32.8          13.4           -        35.8

13B     -     57.3        -      56.7         -     55.3          17.7          -        49.9
3*     7B      -     46.2          -     31.9         -     26.3          12.2          -        35.6

13B       -      54.6          -     36.2        -      37.1          18.9         -        40.1

70B       -     63.8         -     44.3       -       59.3          32.3          -        60.8

black3*      1.8B     42.4     43.9          50.7     50.3          27.8     19.5       14.6             27.1     25.0

7B       55.8     57.0          59.7     59.3          50.3     54.1          37.2               39.6     46.7

14B       64.6     66.5          69.8     71.7          60.1     59.3      43.9               46.9     58.7

Results of the human evaluation for chat models. We compare Qwen-7B (SFT), Qwen-14B (SFT), Qwen-14B (RLHF), as well as GPT-4 against GPT-3.5. Each bar segment represents the percentage of wins, ties, and losses, from bottom to top. On average, the RLHF model outperforms the SFT model. The dataset consists of 300 Chinese instructions.

Despite this, we have reservations about the ability of traditional benchmark evaluation to accurately measure the performance and potential of chat models trained with alignment techniques in today's landscape. The results mentioned earlier provide some evidence of our competitive standing, but we believe that it is crucial to develop new evaluation methods specifically tailored to aligned models.

We believe that human evaluation is crucial, which is why we have created a carefully curated dataset for this purpose. Our process involved collecting 300 instructions in Chinese that covered a wide range of topics, including knowledge, language understanding, creative writing, coding, and mathematics.
To evaluate the performance of different models, we chose the SFT version of -7B and the SFT and RLHF versions of -14B, and added two strong baselines, GPT-3.5 and GPT-4[To obtain the results from the models, we use the OpenAI APIs of GPT-3.5-turbo-0613 and GPT-4-0613.], for comparison. For each instruction, we asked three annotators to rank the model responses by the overall score of helpfulness, informativeness, validity, and other relevant factors.
Our dataset and evaluation methodology provides a comprehensive and rigorous assessment of the capabilities of different language models in various domains.

Figure <ref> illustrates the win rates of the various models. For each model, we report the percentage of wins, ties, and losses against GPT-3.5, with the segments of each bar from bottom to top representing these statistics.
The experimental results clearly demonstrate that the RLHF model outperforms the SFT models by significant margins, indicating that RLHF can encourage the model to generate responses that are more preferred by humans. In terms of overall performance, we find that the RLHF model significantly outperforms the SFT models, falling behind GPT-4.
This indicates the effectiveness of RLHF for aligning to human preference.
To provide a more comprehensive understanding of the models' performance, we include a case study with examples from different models in Appendix <ref>.
Nonetheless, it remains difficult to accurately capture the gap between our models and the proprietary models. As such, a more extensive and rigorous assessment is required for the chat models.

§.§ Tool Use, Code Interpreter, and Agent

Performance of  on the in-house Chinese benchmark that evaluates its ability to use unseen tools via ReAct prompting.

Model       Params         Tool Selection (Acc.↑)     Tool Input (Rouge-L↑)     False Positive Error (%)↓
GPT-4        -        95                       90                  15.0

lightgrayGPT-3.5      -        85                      88                      75.0

black3*     1.8B      92     89     19.3

7B      98                   91                      07.3

14B      98     93     02.4

The proportion of code generated by  that is executable on the in-house evaluation benchmark for Code Interpreter. This benchmark examines 's coding proficiency in math problem solving, data visualization, and general purposes.  underperforms on visualization tasks because it hallucinates non-existent columns solely based on CSV file names (see Figure <ref>).

2*[-]Model     2*[-]Params      3cCategory
Math (%)     Visualization (%)      General (%)     All (%)
GPT-4        -     91.9     85.9     82.8     86.8

lightgrayGPT-3.5      -     89.2     65.0     74.1     72.9

lightgray2*      7B     41.9     33.1     24.1     33.6

13B     50.0     40.5     48.3     44.4

lightgray2*-Instruct      7B     85.1     54.0     70.7     65.1

13B     93.2     55.8     74.1     68.8

lightgray2*InternLM-Chat     7B v1.1     78.4     44.2     62.1     56.3

20B     70.3     44.2     65.5     54.9

black3*     1.8B     33.8     30.1     8.6     26.8

7B              82.4     64.4     67.2     70.2

14B              89.2     84.1     65.5     81.7

Correctness of the final response on the in-house evaluation benchmark for Code Interpreter. Visualization-Hard tasks involve planning multiple steps, while Visualization-Easy tasks do not. Visualization-All measures both types of tasks.
excels in performing Visualization-Easy tasks but tends to underperform in Visualization-Hard tasks, due to its inclination to hallucinate non-existent columns based on the name of a CSV file (see Figure <ref>).

2*[-]Model     2*[-]Params     4cCategory
Math (%)     Vis.-Hard (%)     Vis.-Easy (%)     Vis.-All (%)
GPT-4        -     82.8     66.7     60.8     63.8

lightgrayGPT-3.5      -     47.3     33.3     55.7     44.2

lightgray2*     7B     3.9     14.3     39.2     26.4

13B     8.3     8.3     40.5     23.9

lightgray2*-Instruct      7B     14.3     26.2     60.8     42.9

13B     28.2     27.4     62.0     44.2

lightgray2*InternLM-Chat     7B v1.1     28.5     4.8     40.5     22.1

20B     34.6     21.4     45.6     33.1

black3*     1.8B     14.7     3.6     20.3     11.7

7B     41.9     40.5     54.4     47.2

14B     58.4     53.6     59.5     56.4

Results of -Chat on the Hugging Face Agent benchmark.

2*[-]Task     2*[-]Model     2*[-]Params     3cMetric
Tool Selection $\uparrow $     Tool Used  $\uparrow $     Code Correctness $\uparrow $

black7*Run Mode     GPT-4      -       100                       100                    97.4

GPT-3.5      -       95.4                       96.3                    87.0

Starcoder-Base     15B     86.1     87.0     68.9

Starcoder     15B     87.0     88.0     68.9

3*     1.8B     85.2     84.3     61.1

7B     87.0     87.0     71.5

14B     93.5     94.4     87.0

black7*Chat Mode     GPT-4      -       97.9     97.9     98.5

GPT-3.5      -       97.3     96.8      89.6

Starcoder-Base     15B     97.9     97.9     91.1

Starcoder     15B     97.9     97.9     89.6

3*     1.8B     93.6     93.6     73.2

7B     94.7     94.7     85.1

14B     97.9     97.9     95.5

The  models, which are designed to be versatile, have the remarkable ability to assist with (semi-)automating daily tasks by leveraging their skills in tool-use and planning. As such, they can serve as agents or copilots to help streamline various tasks. We explore 's proficiency in the following areas:

* Utilizing unseen tools through ReAct prompting [Yao et al., 2022] (see Table <ref>).

* Using a Python code interpreter to enhance math reasoning, data analysis, and more (see Table <ref> and Table <ref>).

* Functioning as an agent that accesses Hugging Face's extensive collection of multimodal models while engaging with humans (see Table <ref>).

To enhance 's capabilities as an agent or copilot, we employ the self-instruct [Wang et al., 2023] strategy for SFT. Specifically, we utilize the in-context learning capability of  for self-instruction. By providing a few examples, we can prompt  to generate more relevant queries and generate outputs that follow a specific format, such as ReAct [Yao et al., 2022]. We then apply rules and involve human annotators to filter out any noisy samples.
Afterwards, the samples are incorporated into 's training data, resulting in an updated version of  that is more dependable for self-instruction.
We iterate through this process multiple times until we gather an ample number of samples that possess both exceptional quality and a wide range of diversity. As a result, our final collection consists of around 2000 high-quality samples.

During the finetuning process, we mix these high-quality samples with all the other general-purpose SFT samples, rather than introducing an additional training stage. By doing so, we are able to retain essential general-purpose capabilities that are also pertinent for constructing agent applications.

Using Tools via ReAct Prompting

We have created and made publicly available a benchmark for evaluating 's ability to call plugins, tools, functions, or APIs using ReAct Prompting <cit.>.
To ensure fair evaluation, we have excluded any plugins that were included in 's training set from the evaluation set. The benchmark assesses the model's accuracy in selecting the correct plugin from a pool of up to five candidates, as well as the plausibility of the parameters passed into the plugin and the frequency of false positives. In this evaluation, a false positive occurs when the model incorrectly invokes a plugin in response to a query, despite not being required to do so.

The results presented in Table <ref> demonstrate that  consistently achieves higher accuracy in identifying the relevance of a query to the available tools as the model size increases. However, the table also highlights that beyond a certain point, there is little improvement in performance when it comes to selecting the appropriate tool and providing relevant arguments. This suggests that the current preliminary benchmark may be relatively easy and may require further enhancement in future iterations.
It is worth noting that GPT-3.5 stands out as an exception, displaying suboptimal performance on this particular benchmark. This could potentially be attributed to the fact that the benchmark primarily focuses on the Chinese language, which may not align well with GPT-3.5's capabilities. Additionally, we observe that GPT-3.5 tends to attempt to use at least one tool, even if the query cannot be effectively addressed by the provided tools.

Using Code Interpreter for Math Reasoning and Data Analysis

The Python code interpreter is widely regarded as a powerful tool for augmenting the capabilities of an LLM agent. It is worth investigating whether  can harness the full potential of this interpreter to enhance its performance in diverse domains, such as mathematical reasoning and data analysis. To facilitate this exploration, we have developed and made publicly available a benchmark that is specifically tailored for this purpose <cit.>.

The benchmark encompasses three primary categories of tasks: math problem-solving, data visualization, and other general-purpose tasks like file post-processing and web crawling. Within the visualization tasks, we differentiate between two levels of difficulty. The easier level can be achieved by simply writing and executing a single code snippet without the need for advanced planning skills. However, the more challenging level requires strategic planning and executing multiple code snippets in a sequential manner. This is because the subsequent code must be written based on the output of the previous code. For example, an agent may need to examine the structure of a CSV file using one code snippet before proceeding to write and execute additional code to create a plot.

Regarding evaluation metrics, we consider both the executability and correctness of the generated code. To elaborate on the correctness metrics, for math problems, we measure accuracy by verifying if the ground truth numerical answer is present in both the code execution result and the final response. When it comes to data visualization, we assess accuracy by utilizing -VL [Bai et al., 2023], a powerful multimodal language model. -VL is capable of answering text questions paired with images, and we rely on it to confirm whether the image generated by the code fulfills the user's request.

The results regarding executability and correctness are presented in Table <ref> and Table <ref>, respectively. It is evident that  generally outperforms , its generalist counterpart, which is not surprising since this benchmark specifically requires coding skills.
However, it is worth noting that specialist models that are optimized for code synthesis do not necessarily outperform generalist models. This is due to the fact that this benchmark encompasses various skills beyond coding, such as abstracting math problems into equations, understanding language-specified constraints, and responding in the specified format such as ReAct.
Notably,  and  surpass all other open-source alternatives of similar scale significantly, despite being generalist models.

Serving as a Hugging Face Agent

Hugging Face provides a framework called the Hugging Face Agent or Transformers Agent [Hugging Face, 2023], which empowers LLM agents with a curated set of multimodal tools, including speech recognition and image synthesis. This framework allows an LLM agent to interact with humans, interpret natural language commands, and employ the provided tools as needed.

To evaluate 's effectiveness as a Hugging Face agent, we utilized the evaluation benchmarks offered by Hugging Face. The results are presented in Table <ref>. The evaluation results reveal that  performs quite well in comparison to other open-source alternatives, only slightly behind the proprietary GPT-4, demonstrating 's competitive capabilities.

§ : SPECIALIZED MODEL FOR CODING

Training on domain-specific data has been shown to be highly effective, particularly in the case of code pretraining and finetuning. A language model that has been reinforced with training on code data can serve as a valuable tool for coding, debugging, and interpretation, among other tasks. In this work, we have developed a series of generalist models using pretraining and alignment techniques. Building on this foundation, we have created domain-specific models for coding by leveraging the base language models of , including continued pretrained model,  and supervised finetuned model, . Both models have 14 billion and 7 billion parameters versions.

§.§ Code Pretraining

We believe that relying solely on code data for pretraining can result in a significant loss of the ability to function as a versatile assistant. Unlike previous approaches that focused solely on pretraining on code data [Li et al., 2022, Li et al., 2023], we take a different approach [Rozière et al., 2023] by starting with our base models  trained on a combination of text and code data, and then continuing to pretrain on the code data. We continue to pretrain the models on a total of around 90 billion tokens.
During the pre-training phase, we initialize the model using the base language models .
Many applications that rely on specialized models for coding may encounter lengthy contextual scenarios, such as tool usage and code interpretation, as mentioned in Section <ref>. To address this issue, we train our models with context lengths of up to 8192.
Similar to base model training in Section <ref>, we employ Flash Attention [Dao et al., 2022] in the attention modules, and adopt the standard optimizer AdamW [Kingma & Ba, 2014, Loshchilov & Hutter, 2017], setting  $\beta_{1} = 0.9$, $\beta_{2} = 0.95$, and $\epsilon = 10^{-8}$. We set the learning rate as $6.0 \times 10^{-5}$ for   and $3.0 \times 10^{-5}$ for , with $3\%$ warm up iterations and no learning rate decays.

§.§ Code Supervised Fine-Tuning

After conducting a series of empirical experiments, we have determined that the multi-stage SFT strategy yields the best performance compared to other methods.
In the supervised fine-tuning stage, the model  initialized by the code foundation model  are optimized by the AdamW [Kingma & Ba, 2014, Loshchilov & Hutter, 2017] optimizer ($\beta_{1} = 0.9$, $\beta_{2} = 0.95$, $\epsilon = 10^{-8}$) with a learning rate of $2.0 \times 10^{-6}$ and $1.0 \times 10^{-5}$ for the 14B and 7B model respectively. The learning rate increases to the peaking value with the cosine learning rate schedule ($3\%$ warm-up steps) and then remains constant.

§.§ Evaluation

Results of pass@1 (%) on HumanEval and MBPP. Most scores are retrieved from the papers of StarCoder [Li et al., 2023], CodeT5+ [Wang et al., 2023], WizardCoder [Luo et al., 2023] and  [Rozière et al., 2023].

Model      Params     HumanEval     MBPP

black4cProprietary models

blackPaLM     540B     26.2     36.8

lightgrayPaLM-Coder     540B     36.0     47.0

lightgrayPaLM 2-S     -     37.6     50.0

lightgrayCode-Cushman-001     -     33.5     45.9

lightgrayCode-Davinci-002     -     47.0     58.1

lightgrayGPT-3.5     -     73.2     -

lightgrayGPT-4     -     86.6     -

black4cOpen-source models

black4*     7B     12.2     20.8

13B     20.1     27.6

34B     22.6     33.8

70B     30.5     45.4

lightgrayCodeGen-Multi     16B     18.3     20.9

lightgrayCodeGen-Mono     16B     29.3     35.3

lightgrayCodeGeeX2     6B     35.9     -

lightgrayStarCoder-Prompted     15B     40.8     49.5

lightgrayCodeT5+     16B     30.9     -

lightgrayInstructCodeT5+     16B     35.0     -

lightgray3*     7B     33.5     41.4

13B     36.0     47.0

34B     48.8     55.0

lightgray3*Code Llama-Instruct     7B     34.8     44.4

13B     42.7     49.4

34B     41.5     57.0

lightgray3*Code Llama-Python    7B     38.4     47.6

13B     43.3     49.0

34B     53.7     56.2

lightgrayUnnatural Code Llama     34B     62.2     61.2

lightgray2*WizardCoder-Python	     13B     64.0     55.6

34B     73.2	     61.2

black2*       7B     37.2     35.8

14B     43.9     46.4

lightgray2*         7B     40.2      41.8

14B     45.1      51.4

lightgray2*     7B     43.3      44.2

14B       66.4         52.4

Zero-shot pass@1 (%) performance on the HumanEvalPack (synthesize) benchmark. The baseline results are partly from OctoPack [Muennighoff et al., 2023].

2*[-]Model     2*[-]Params     6cProgramming Language
Python     JavaScript       Java        Go          C++      Rust     Avg.

9cProprietary models

GPT-4      -      86.6     82.9    81.7    72.6    78.7    67.1       78.3

9cOpen-source models

InstructCodeT5+       16B     37.0    18.9    17.4    9.5    19.8    0.3    17.1

lightgrayStarChat-$\beta$      15B      33.5    31.4    26.7    25.5    26.6    14.0    26.3

lightgrayStarCoder             15B      33.6    30.8    30.2    17.6    31.6    21.8    27.6

lightgrayCodeGeeX2             6B     35.9    32.2    30.8    22.5    29.3    18.1    28.1

lightgrayOctoGeeX             6B      44.7    33.8    36.9    21.9    32.3    15.7    30.9

lightgrayOctoCoder              15B     46.2    39.2    38.2    30.4    35.6    23.4    35.5

lightgrayWizardCoder            15B     59.8    49.5    36.1    36.4    40.9    20.2    40.5

black2*          7B      37.2      	23.2       32.9     20.7      22.0     9.1        24.2

14B       43.9     	38.4       42.7     34.1     24.4     18.9       33.7

lightgray2*            7B      40.2       40.4       40.2     26.2     20.7     15.8       30.6

14B      45.1       51.8       57.3     39.6    18.2     20.7       38.8

lightgray2*         7B      43.3       41.5       49.4     29.3     32.9     20.1      36.1

14B      66.4        58.5       56.1     47.6     54.2     28.7       51.9

Our  models have been compared with both proprietary and open-source language models, as shown in Tables <ref> and <ref>. These tables present the results of our evaluation on the test sets of Humaneval [Chen et al., 2021], MBPP [Austin et al., 2021], and the multi-lingual code generation benchmark HumanEvalPack [Muennighoff et al., 2023]. The comparison is based on the pass@1 performance of the models on these benchmark datasets. The results of this comparison are clearly demonstrated in Tables <ref> and <ref>.

Our analysis reveals that specialized models, specifically  and , significantly outperform previous baselines with similar parameter counts, such as OctoGeeX [Muennighoff et al., 2023], InstructCodeT5+ [Wang et al., 2023], and CodeGeeX2 [Zheng et al., 2023]. In fact, these models even rival the performance of larger models like Starcoder [Li et al., 2023].

When compared to some of the extremely large-scale closed-source models,  and  demonstrate clear advantages in terms of pass@1. However, it is important to note that these models fall behind the state-of-the-art methods, such as GPT-4, in general. Nonetheless, with the continued scaling of both model size and data size, we believe that this gap can be narrowed in the near future.

It is crucial to emphasize that the evaluations mentioned previously are insufficient for grasping the full extent of the strengths and weaknesses of the models. In our opinion, it is necessary to develop more rigorous tests to enable us to accurately assess our relative performance in comparison to GPT-4.

§ : SPECIALIZED MODEL FOR MATHEMATICS REASONING

Results of models on mathematical reasoning. We report the accuracy of  for all benchmarks using greedy decoding. For MATH, we are reporting 's performances on the test set from [Lightman et al., 2023].

Model     Params     GSM8K     MATH     Math401     Math23K

6cProprietary models

blackGPT-4     -      92.0     42.5     83.5      74.0

lightgrayGPT-3.5     -     80.8     34.1     75.1      60.0

lightgray3*Minerva     8B     16.2     14.1     -    -

62B     52.4     27.6     -     -

540B     58.8     33.6     -      -

black6cOpen-source models

black2*LLaMA-1 RFT     7B     46.5     5.2     -    -

13B     52.1     5.1     -     -

lightgray3*WizardMath     7B     54.9     10.7      -     -

13B     63.9     14.0      -     -

70B     81.6     22.7       -    -

lightgray3*GAIRMath-Abel     7B     59.7     13.0      -     -

13B     66.4     17.3       -    -

70B     83.6     28.3       -    -

black2*     7B     50.3     6.8     57.4      51.2

14B     60.1     18.4     70.1    67.0

lightgray2*     7B     62.5     17.2     80.8     75.4

14B     69.8      24.2      85.0      78.4

We have created a mathematics-specialized model series called , which is built on top of the  pretrained language models. Specifically, we have developed assistant models that are specifically designed to excel in arithmetic and mathematics and are aligned with human behavior.
We are releasing two versions of this model series,  and , which have 14 billion and 7 billion parameters, respectively.

§.§ Training

We carry out math SFT on our augmented math instructional dataset for mathematics reasoning, and therefore we obtain the chat model, , directly.
Owing to shorter average lengths of the math SFT data, we use a sequence length of 1024 for faster training.
Most user inputs in the math SFT dataset are examination questions, and it is easy for the model to predict the input format and it is meaningless for the model to predict the input condition and numbers which could be random.
Thus, we mask the inputs of the system and user to avoid loss computation on them and find masking them accelerates the convergence during our preliminary experiments.
For optimization, we use the AdamW optimizer with the same hyperparameters of SFT except that we use a peak learning rate of $2 \times 10^{-5}$ and a training step of 50000.

§.§ Evaluation

We evaluate models on the test sets of GSM8K (Grade school math) [Cobbe et al., 2021], MATH (Challenging competition math problems) [Hendrycks et al., 2021], Math401 (Arithmetic ability) [Yuan et al., 2023], and Math23K (Chinese grade school math) [Wang et al., 2017]. We compare  with proprietary models ChatGPT and Minerva [Lewkowycz et al., 2022] and open-sourced math-specialized model RFT [Yuan et al., 2023], WizardMath [Luo et al., 2023], and GAIRMath-Abel [Chern et al., 2023] in Table <ref>.
models show better math reasoning and arithmetic abilities compared to open-sourced models and  models of similar sizes.
Compared to proprietary models,  outperforms Minerva-8B in MATH.
is chasing Minerva-62B and GPT-3.5 in GSM8K and MATH and delivers better performance on arithmetic ability and Chinese math problems.

§ RELATED WORK

§.§ Large Language Models

The excitement of LLM began with the introduction of the Transformer architecture [Vaswani et al., 2017], which was then applied to pretraining large-scale data by researchers such as Radford et al., 2018, Devlin et al., 2018, Liu et al., 2019. These efforts led to significant success in transfer learning, with model sizes growing from 100 million to over 10 billion parameters [Raffel et al., 2020, Shoeybi et al., 2019].

In 2020, the release of GPT-3, a massive language model that is 10 times larger than T5, demonstrated the incredible potential of few-shot and zero-shot learning through prompt engineering and in-context learning, and later chain-of-thought prompting [Wei et al., 2022]. This success has led to a number of studies exploring the possibilities of further scaling these models [Scao et al., 2022, Zhang et al., 2022, Du et al., 2021, Zeng et al., 2022, Lepikhin et al., 2020, Fedus et al., 2022, Du et al., 2022, Black et al., 2022, Rae et al., 2021, Hoffmann et al., 2022, Chowdhery et al., 2022, Thoppilan et al., 2022]. As a result, the community has come to view these large language models as essential foundations for downstream models [Bommasani et al., 2021].

The birth of ChatGPT [OpenAI, 2022] and the subsequent launch of GPT-4 [OpenAI, 2023] marked two historic moments in the field of artificial intelligence, demonstrating that large language models (LLMs) can serve as effective AI assistants capable of communicating with humans. These events have sparked interests among researchers and developers in building language models that are aligned with human values and potentially even capable of achieving artificial general intelligence (AGI) [Anil et al., 2023, Anthropic, 2023, Anthropic, 2023].

One notable development in this area is the emergence of open-source LLMs, specifically LLaMA [Touvron et al., 2023] and  [Touvron et al., 2023], which have been recognized as the most powerful open-source language models ever created. This has led to a surge of activity in the open-source community [Wolf et al., 2019], with a series of large language models being developed collaboratively to build upon this progress [Mosaic ML, 2023, Almazrouei et al., 2023, ChatGLM2 Team, 2023, Yang et al., 2023, InternLM Team, 2023].

§.§ Alignment

The community was impressed by the surprising effectiveness of alignment on LLMs. Previously, LLMs without alignment often struggle with issues such as repetitive generation, hallucination, and deviation from human preferences. Since 2021, researchers have been diligently working on developing methods to enhance the performance of LLMs in downstream tasks [Wei et al., 2022, Sanh et al., 2021, Longpre et al., 2023, Chung et al., 2022, Muennighoff et al., 2022]. Furthermore, researchers have been actively exploring ways to align LLMs with human instructions [Ouyang et al., 2022, Askell et al., 2021, Bai et al., 2022, Bai et al., 2022].
One major challenge in alignment research is the difficulty of collecting data. While OpenAI has utilized its platform to gather human prompts or instructions, it is not feasible for others to collect such data.

However, there has been some progress in this area, such as the self-instruct approach proposed in <cit.>. This innovative work offers a potential solution to the data collection problem in alignment research. As a result, there has been a surge in open-source chat data, including Alpaca [Taori et al., 2023], MOSS [Sun et al., 2023], Dolly [Conover et al., 2023], Evol-Instruct [Xu et al., 2023], and others [Sun et al., 2023, Xu et al., 2023, Xu et al., 2023, Chen et al., 2023, Ding et al., 2023, Ji et al., 2023, Yang, 2023]. Similarly, there has been an increase in open-source chat models, such as Alpaca [Taori et al., 2023], Vicuna [Chiang et al., 2023], Guanaco [Dettmers et al., 2023], MOSS [Sun et al., 2023],  WizardLM [Xu et al., 2023], and others [Xu et al., 2023, Chen et al., 2023, Ding et al., 2023, Wang et al., 2023].

To train an effective chat model, available solutions are mostly based on SFT and RLHF [Ouyang et al., 2022].
While SFT is similar to pretraining, it focuses on instruction following using the aforementioned data. However, for many developers, the limited memory capacity is a major obstacle to further research in SFT. As a result, parameter-efficient tuning methods, such as LoRA [Hu et al., 2021] and Q-LoRA [Dettmers et al., 2023], have gained popularity in the community. LoRA tunes only low-rank adapters, while Q-LoRA builds on LoRA and utilizes 4-bit quantized LLMs and paged attention [Dettmers et al., 2022, Frantar et al., 2022, Kwon et al., 2023]. In terms of RLHF, recent methods such as PPO [Schulman et al., 2017, Touvron et al., 2023] have been adopted, but there are also alternative techniques aimed at addressing the complexity of optimization, such as RRHF [Yuan et al., 2023], DPO [Rafailov et al., 2023], and PRO [Song et al., 2023]. Despite the ongoing debate about the effectiveness of RLHF, more evidence is needed to understand how it enhances the intelligence of LLMs and what potential drawbacks it may have.

§.§ Tool Use and Agents

LLM's planning function allows for the invocation of tools, such as APIs or agent capabilities, through in-context learning, as demonstrated by Schick et al., 2023. Yao et al., 2022 introduced ReAct, a generation format that enables the model to generate thoughts on which tool to use, accept input from API observations, and generate a response. GPT-3.5 and GPT-4, when prompted with few shots, have shown consistent and impressive performance.
In addition to tool usage, LLMs can utilize external memory sources like knowledge bases [Hu et al., 2023, Zhong et al., 2023] or search engines [Nakano et al., 2021, Liu et al., 2023] to generate more accurate and informative answers. This has led to the popularity of frameworks like LangChain [LangChain, Inc., 2023].
The research on LLMs for tool use has also sparked interest in building agents with LLM capabilities, such as agents that can call different AI models [Shen et al., 2023, Li et al., 2023], embodied lifelong learning or multimodal agents [Wang et al., 2023, Driess et al., 2023], and multiple agents interacting with each other and even building a micro-society [Chen et al., 2023, Li et al., 2023, Xu et al., 2023, Hong et al., 2023].

§.§ LLM for Coding

Previous research has demonstrated that LLMs possess remarkable capabilities in code understanding and generation, particularly those with massive numbers of parameters [Chowdhery et al., 2022, Anil et al., 2023, Rae et al., 2021, Hoffmann et al., 2022].
Moreover, several LLMs have been pre-trained, continued pre-trained, or fine-tuned on coding-related data, which has resulted in significantly improved performance compared to general-purpose LLMs. These models include Codex [Chen et al., 2021], AlphaCode [Li et al., 2022], SantaCoder [Allal et al., 2023], Starcoder-Base [Li et al., 2023], InCoder [Fried et al., 2022], CodeT5 [Wang et al., 2021], CodeGeeX [Zheng et al., 2023], and  [Rozière et al., 2023].
In addition to these models, recent studies have focused on developing specialized alignment techniques for coding, such as Code Llama-Instruct [Rozière et al., 2023] and StarCoder [Li et al., 2023]. These models can assist developers in various code-related tasks, including code generation [Chen et al., 2021, Austin et al., 2021], code completion [Zhang et al., 2023], code translation [Szafraniec et al., 2023], bug fixing [Muennighoff et al., 2023], code refinement [Liu et al., 2023], and code question answering [Liu & Wan, 2021].
In a word, LLMs have the potential to revolutionize the field of coding by providing developers with powerful tools for code comprehension, generation, and related tasks.

§.§ LLM for Mathematics

LLMs with a certain model scale have been found to possess the ability to perform mathematical reasoning [Wei et al., 2022, Suzgun et al., 2022]. In order to encourage LLMs to achieve better performance on math-related tasks, researchers have employed techniques such as chain-of-thought prompting [Wei et al., 2022] and scratchpad [Nye et al., 2021], which have shown promising results. Additionally, self-consistency [Wang et al., 2022] and least-to-most prompting [Zhou et al., 2022] have further improved the performance of these models on these tasks.
However, prompt engineering is a time-consuming process that requires a lot of trial and error, and it is still difficult for LLMs to consistently perform well or achieve satisfactory results in solving mathematical problems. Moreover, simply scaling the data and model size is not an efficient way to improve a model's mathematical reasoning abilities.
Instead, pretraining on math-related corpora has been shown to consistently enhance these capabilities [Hendrycks et al., 2021, Lewkowycz et al., 2022, Taylor et al., 2022, Lightman et al., 2023]. Additionally, fine-tuning on math-related instruction-following datasets [Si et al., 2023, Yuan et al., 2023, Luo et al., 2023, Yue et al., 2023, Chern et al., 2023, Yu et al., 2023], has also been effective and more cost-effective than math-specific pretraining.
Despite their limitations in terms of accuracy, LLMs still have significant potential to assist users with practical mathematical problems. There is ample scope for further development in this area.

§ CONCLUSION

In this report, we present the  series of large language models, which showcase the latest advancements in natural language processing. With 14B, 7B, and 1.8B parameters, these models have been pre-trained on massive amounts of data, including trillions of tokens, and fine-tuned using cutting-edge techniques such as SFT and RLHF. Additionally, the  series includes specialized models for coding and mathematics, such as , , and , which have been trained on domain-specific data to excel in their respective fields.
Our results demonstrate that the  series is competitive with existing open-source models and even matches the performance of some proprietary models on comprehensive benchmarks and human evaluation.

We believe that the open access of  will foster collaboration and innovation within the community, enabling researchers and developers to build upon our work and push the boundaries of what is possible with language models. By providing these models to the public, we hope to inspire new research and applications that will further advance the field and contribute to our understanding of the variables and techniques introduced in realistic settings.
In a nutshell, the  series represents a major milestone in our development of large language models, and we are excited to see how it will be used to drive progress and innovation in the years to come.

[Allal et al., 2023]
Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki,
Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan
Dey, et al.
SantaCoder: Don't reach for the stars!
arXiv preprint arXiv:2301.03988, 2023.

[Almazrouei et al., 2023]
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli,
Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Heslow, Julien
Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme
Falcon-40B: An open large language model with state-of-the-art
performance, 2023.

[Anil et al., 2023]
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin,
Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen,
et al.
PaLM 2 technical report.
arXiv preprint arXiv:2305.10403, 2023.

[Anthropic, 2023]
Introducing Claude, 2023a.
URL <https://www.anthropic.com/index/introducing-claude>.

[Anthropic, 2023]
Claude 2.
Technical report, Anthropic, 2023b.

[Aribandi et al., 2021]
Vamsi Aribandi, Yi Tay, Tal Schuster, Jinfeng Rao, Huaixiu Steven Zheng,
Sanket Vaibhav Mehta, Honglei Zhuang, Vinh Q Tran, Dara Bahri, Jianmo Ni,
et al.
ExT5: Towards extreme multi-task scaling for transfer learning.
arXiv preprint arXiv:2111.10952, 2021.

[Askell et al., 2021]
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan,
Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al.
A general language assistant as a laboratory for alignment.
arXiv preprint arXiv:2112.00861, 2021.

[Austin et al., 2021]
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski,
David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al.
Program synthesis with large language models.
arXiv preprint arXiv:2108.07732, 2021.

[AutoGPT, 2023]
AutoGPT: The heart of the open-source agent ecosystem, 2023.
URL <https://github.com/Significant-Gravitas/Auto-GPT>.

[Ba et al., 2016]
Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton.
Layer normalization.
CoRR, abs/1607.06450, 2016.
URL <http://arxiv.org/abs/1607.06450>.

[Bai et al., 2022]
Jinze Bai, Rui Men, Hao Yang, Xuancheng Ren, Kai Dang, Yichang Zhang, Xiaohuan
Zhou, Peng Wang, Sinan Tan, An Yang andf Zeyu Cui, Yu Han, Shuai Bai, Wenbin
Ge, Jianxin Ma, Junyang Lin, Jingren Zhou, and Chang Zhou.
OFASys: A multi-modal multi-task learning system for building
generalist models.
CoRR, abs/2212.04408, 2022a.
URL <https://doi.org/10.48550/arXiv.2212.04408>.

[Bai et al., 2023]
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang
Lin, Chang Zhou, and Jingren Zhou.
Qwen-VL: A versatile vision-language model for understanding,
localization, text reading, and beyond.
CoRR, abs/2308.12966, 2023.
URL <https://doi.org/10.48550/arXiv.2308.12966>.

[Bai et al., 2022]
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma,
Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al.
Training a helpful and harmless assistant with reinforcement learning
from human feedback.
arXiv preprint arXiv:2204.05862, 2022b.

[Bai et al., 2022]
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion,
Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon,
et al.
Constitutional AI: Harmlessness from AI feedback.
arXiv preprint arXiv:2212.08073, 2022c.

[Beltagy et al., 2020]
Iz Beltagy, Matthew E Peters, and Arman Cohan.
Longformer: The long-document transformer.
arXiv preprint arXiv:2004.05150, 2020.

[Bisk et al., 2020]
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi.
PIQA: reasoning about physical commonsense in natural language.
In The Thirty-Fourth AAAI Conference on Artificial
Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of
Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium
on Educational Advances in Artificial Intelligence, EAAI 2020, New York,
NY, USA, February 7-12, 2020, pp.  7432–7439. AAAI Press, 2020.
URL <https://doi.org/10.1609/aaai.v34i05.6239>.

[Black et al., 2022]
Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence
Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al.
GPT-NeoX-20B: An open-source autoregressive language model.
arXiv preprint arXiv:2204.06745, 2022.

[bloc97, 2023]
NTK-aware scaled RoPE allows LLaMA models to have extended
(8k+) context size without any fine-tuning and minimal perplexity
degradation., 2023.

[Bommasani et al., 2021]
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney
von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma
Brunskill, et al.
On the opportunities and risks of foundation models.
arXiv preprint arXiv:2108.07258, 2021.

[Brown et al., 2020]
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla
Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell,
et al.
Language models are few-shot learners.
Advances in neural information processing systems,
33:0 1877–1901, 2020.

[ChatGLM2 Team, 2023]
ChatGLM2 Team.
ChatGLM2-6B: An open bilingual chat LLM, 2023.
URL <https://github.com/THUDM/ChatGLM2-6B>.

[Chen et al., 2021]
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé
de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas
Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov,
Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick
Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian,
Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings,
Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss,
William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor
Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher
Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa,
Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter
Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and
Wojciech Zaremba.
Evaluating large language models trained on code.
CoRR, abs/2107.03374, 2021.
URL <https://arxiv.org/abs/2107.03374>.

[Chen et al., 2023]
Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian.
Extending context window of large language models via positional
arXiv preprint arXiv:2306.15595, 2023a.

[Chen et al., 2023]
Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chen Qian,
Chi-Min Chan, Yujia Qin, Yaxi Lu, Ruobing Xie, et al.
Agentverse: Facilitating multi-agent collaboration and exploring
emergent behaviors in agents.
arXiv preprint arXiv:2308.10848, 2023b.

[Chen et al., 2023]
Zhihong Chen, Feng Jiang, Junying Chen, Tiannan Wang, Fei Yu, Guiming Chen,
Hongbo Zhang, Juhao Liang, Chen Zhang, Zhiyi Zhang, et al.
Phoenix: Democratizing ChatGPT across languages.
arXiv preprint arXiv:2304.10453, 2023c.

[Chern et al., 2023]
Ethan Chern, Haoyang Zou, Xuefeng Li, Jiewen Hu, Kehua Feng, Junlong Li, and
Pengfei Liu.
Generative ai for math: Abel.
<https://github.com/GAIR-NLP/abel>, 2023a.

[Chern et al., 2023]
I Chern, Steffi Chern, Shiqi Chen, Weizhe Yuan, Kehua Feng, Chunting Zhou,
Junxian He, Graham Neubig, Pengfei Liu, et al.
Factool: Factuality detection in generative ai–a tool augmented
framework for multi-task and multi-domain scenarios.
arXiv preprint arXiv:2307.13528, 2023b.

[Chiang & Cholak, 2022]
David Chiang and Peter Cholak.
Overcoming a theoretical limitation of self-attention.
In Proceedings of the 60th Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers), pp.  7654–7664,

[Chiang et al., 2023]
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin
Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and
Eric P. Xing.
Vicuna: An open-source chatbot impressing GPT-4 with 90%*
ChatGPT quality, March 2023.
URL <https://lmsys.org/blog/2023-03-30-vicuna/>.

[Chowdhery et al., 2022]
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra,
Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian
Gehrmann, et al.
PaLM: Scaling language modeling with pathways.
arXiv preprint arXiv:2204.02311, 2022.

[Christiano et al., 2017]
Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and
Dario Amodei.
Deep reinforcement learning from human preferences.
In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach,
Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances
in Neural Information Processing Systems 30: Annual Conference on Neural
Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA,
USA, pp.  4299–4307, 2017.

[Chung et al., 2022]
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus,
Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al.
Scaling instruction-finetuned language models.
arXiv preprint arXiv:2210.11416, 2022.

[Clark et al., 2019]
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael
Collins, and Kristina Toutanova.
Boolq: Exploring the surprising difficulty of natural yes/no
In Jill Burstein, Christy Doran, and Thamar Solorio (eds.),
Proceedings of the 2019 Conference of the North American Chapter of the
Association for Computational Linguistics: Human Language Technologies,
NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and
Short Papers), pp.  2924–2936. Association for Computational Linguistics,
URL <https://doi.org/10.18653/v1/n19-1300>.

[Clark et al., 2018]
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa
Schoenick, and Oyvind Tafjord.
Think you have solved question answering? try arc, the AI2
reasoning challenge.
CoRR, abs/1803.05457, 2018.
URL <http://arxiv.org/abs/1803.05457>.

[Cobbe et al., 2021]
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz
Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano,
et al.
Training verifiers to solve math word problems.
arXiv preprint arXiv:2110.14168, 2021.

[Conneau et al., 2019]
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume
Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and
Veselin Stoyanov.
Unsupervised cross-lingual representation learning at scale.
arXiv preprint arXiv:1911.02116, 2019.

[Conover et al., 2023]
Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali
Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin.
Free Dolly: Introducing the world's first truly open
instruction-tuned LLM, 2023.

[Dai et al., 2023]
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng, Huat Tiong, Junqi Zhao,
Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi.
InstructBLIP: Towards general-purpose vision-language models with
instruction tuning.
arXiv preprint arXiv:2305.06500, 2023.

[Dao et al., 2022]
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré.
FlashAttention: Fast and memory-efficient exact attention with
In NeurIPS, 2022.

[Dauphin et al., 2017]
Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier.
Language modeling with gated convolutional networks.
In International conference on machine learning, pp.  933–941. PMLR, 2017.

[Dettmers et al., 2022]
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer.
LLM.int8(): 8-bit matrix multiplication for transformers at scale.
arXiv preprint arXiv:2208.07339, 2022.

[Dettmers et al., 2023]
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer.
QLoRA: Efficient finetuning of quantized LLMs.
arXiv preprint arXiv:2305.14314, 2023.

[Devlin et al., 2018]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.
BERT: Pre-training of deep bidirectional transformers for language
arXiv preprint arXiv:1810.04805, 2018.

[Ding et al., 2023]
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan
Liu, Maosong Sun, and Bowen Zhou.
Enhancing chat language models by scaling high-quality instructional
arXiv preprint arXiv:2305.14233, 2023.

[Driess et al., 2023]
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery,
Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al.
Palm-e: An embodied multimodal language model.
arXiv preprint arXiv:2303.03378, 2023.

[Du et al., 2022]
Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu,
Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al.
GLaM: Efficient scaling of language models with mixture-of-experts.
In International Conference on Machine Learning, pp.  5547–5569. PMLR, 2022.

[Du et al., 2021]
Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and
Jie Tang.
GLM: General language model pretraining with autoregressive blank
arXiv preprint arXiv:2103.10360, 2021.

[Ethayarajh et al., 2022]
Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta.
Understanding dataset difficulty with $\mathcal{V}$-usable
In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari,
Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th
International Conference on Machine Learning, volume 162 of
Proceedings of Machine Learning Research, pp.  5988–6008. PMLR,
17–23 Jul 2022.

[Fedus et al., 2022]
William Fedus, Barret Zoph, and Noam Shazeer.
Switch transformers: Scaling to trillion parameter models with simple
and efficient sparsity.
The Journal of Machine Learning Research, 230
(1):0 5232–5270, 2022.

[Frantar et al., 2022]
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh.
GPTQ: Accurate post-training quantization for generative
pre-trained transformers.
arXiv preprint arXiv:2210.17323, 2022.

[Fried et al., 2022]
Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida I. Wang, Eric Wallace, Freda
Shi, Ruiqi Zhong, Wen tau Yih, Luke Zettlemoyer, and Mike Lewis.
Incoder: A generative model for code infilling and synthesis.
ArXiv, abs/2204.05999, 2022.

[Google, 2023]
An important next step on our AI journey, 2023.

[Hendrycks & Gimpel, 2016]
Dan Hendrycks and Kevin Gimpel.
Bridging nonlinearities and stochastic regularizers with Gaussian
error linear units.
CoRR, abs/1606.08415, 2016.
URL <http://arxiv.org/abs/1606.08415>.

[Hendrycks et al., 2020]
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn
Song, and Jacob Steinhardt.
Measuring massive multitask language understanding.
arXiv preprint arXiv:2009.03300, 2020.

[Hendrycks et al., 2021]
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric
Tang, Dawn Song, and Jacob Steinhardt.
Measuring mathematical problem solving with the math dataset.
arXiv preprint arXiv:2103.03874, 2021.

[Hoffmann et al., 2022]
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor
Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes
Welbl, Aidan Clark, et al.
Training compute-optimal large language models.
arXiv preprint arXiv:2203.15556, 2022.

[Hong et al., 2023]
Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Ceyao Zhang, Zili Wang,
Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, et al.
Metagpt: Meta programming for multi-agent collaborative framework.
arXiv preprint arXiv:2308.00352, 2023.

[Hu et al., 2023]
Chenxu Hu, Jie Fu, Chenzhuang Du, Simian Luo, Junbo Zhao, and Hang Zhao.
Chatdb: Augmenting llms with databases as their symbolic memory.
arXiv preprint arXiv:2306.03901, 2023.

[Hu et al., 2021]
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean
Wang, Lu Wang, and Weizhu Chen.
LoRA: Low-rank adaptation of large language models.
arXiv preprint arXiv:2106.09685, 2021.

[Hu et al., 2020]
Hai Hu, Kyle Richardson, Liang Xu, Lu Li, Sandra Kübler, and Lawrence S.
OCNLI: original chinese natural language inference.
In Trevor Cohn, Yulan He, and Yang Liu (eds.), Findings of the
Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20
November 2020, volume EMNLP 2020 of Findings of ACL, pp.  3512–3526. Association for Computational Linguistics, 2020.
URL <https://doi.org/10.18653/v1/2020.findings-emnlp.314>.

[Huang et al., 2023]
Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su,
Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, et al.
C-Eval: A multi-level multi-discipline chinese evaluation suite for
foundation models.
arXiv preprint arXiv:2305.08322, 2023.

[Hugging Face, 2023]
Hugging Face.
Transformers agents, 2023.

[Inc., 2023]
Baichuan Inc.
Baichuan-7B: A large-scale 7B pretraining language model
developed by BaiChuan-Inc, 2023a.
URL <https://github.com/baichuan-inc/Baichuan-7B>.

[Inc., 2023]
XVERSE Technology Inc.
XVERSE-13B: A multilingual large language model developed by
XVERSE Technology Inc., 2023b.
URL <https://github.com/xverse-ai/XVERSE-13B>.

[InternLM Team, 2023]
InternLM Team.
InternLM: A multilingual language model with progressively enhanced
capabilities, 2023.
URL <https://github.com/InternLM/InternLM>.

[Jain, 2022]
Shantanu Jain.
tiktoken: A fast BPE tokeniser for use with OpenAI's models,
URL <https://github.com/openai/tiktoken/>.

[Ji et al., 2023]
Yunjie Ji, Yong Deng, Yan Gong, Yiping Peng, Qiang Niu, Lei Zhang, Baochang Ma,
and Xiangang Li.
Exploring the impact of instruction data scaling on large language
models: An empirical study on real-world use cases.
arXiv preprint arXiv:2303.14742, 2023.

[Jiang et al., 2023]
Zixuan Jiang, Jiaqi Gu, Hanqing Zhu, and David Z. Pan.
Pre-RMSNorm and Pre-CRMSNorm transformers: Equivalent and
efficient pre-LN transformers.
CoRR, abs/2305.14858, 2023.
URL <https://doi.org/10.48550/arXiv.2305.14858>.

[Kingma & Ba, 2014]
Diederik P Kingma and Jimmy Ba.
Adam: A method for stochastic optimization.
arXiv preprint arXiv:1412.6980, 2014.

[Kwiatkowski et al., 2019]
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins,
Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob
Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey,
Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov.
Natural questions: a benchmark for question answering research.
Trans. Assoc. Comput. Linguistics, 7:0 452–466, 2019.
URL <https://doi.org/10.1162/tacl_a_00276>.

[Kwon et al., 2023]
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu,
Joseph E. Gonzalez, Hao Zhang, and Ion Stoica.
Efficient memory management for large language model serving with
In Proceedings of the ACM SIGOPS 29th Symposium on Operating
Systems Principles, 2023.

[LangChain, Inc., 2023]
LangChain, Inc.
LangChain: Building applications with LLMs through composability,
URL <https://python.langchain.com/>.

[Lepikhin et al., 2020]
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping
Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen.
GShard: Scaling giant models with conditional computation and
automatic sharding.
arXiv preprint arXiv:2006.16668, 2020.

[Lewkowycz et al., 2022]
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk
Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo
Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra.
Solving quantitative reasoning problems with language models, 2022.

[Li et al., 2023]
Chenliang Li, Hehong Chen, Ming Yan, Weizhou Shen, Haiyang Xu, Zhikai Wu,
Zhicheng Zhang, Wenmeng Zhou, Yingda Chen, Chen Cheng, et al.
ModelScope-Agent: Building your customizable agent system with
open-source large language models.
arXiv preprint arXiv:2309.00986, 2023a.

[Li et al., 2023]
Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and
Bernard Ghanem.
Camel: Communicative agents for “mind” exploration of large scale
language model society.
arXiv preprint arXiv:2303.17760, 2023b.

[Li et al., 2023]
Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan
Duan, and Timothy Baldwin.
CMMLU: Measuring massive multitask language understanding in
arXiv preprint arXiv:2306.09212, 2023c.

[Li et al., 2023]
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov,
Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu,
Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig
Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas
Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi,
Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy V,
Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan
Dey, Zhihan Zhang, Nour Moustafa-Fahmy, Urvashi Bhattacharyya, Wenhao Yu,
Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov,
Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger,
Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer
Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor,
Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite,
Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro
von Werra, and Harm de Vries.
StarCoder: May the source be with you!
CoRR, abs/2305.06161, 2023d.
URL <https://doi.org/10.48550/arXiv.2305.06161>.

[Li et al., 2022]
Yujia Li, David H. Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser,
Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal
Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d'Autume, Igor Babuschkin,
Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov,
James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli,
Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals.
Competition-level code generation with AlphaCode.
CoRR, abs/2203.07814, 2022.

[Lightman et al., 2023]
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy
Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe.
Let's verify step by step.
arXiv preprint arXiv:2305.20050, 2023.

[Liu & Wan, 2021]
Chenxiao Liu and Xiaojun Wan.
CodeQA: A question answering dataset for source code
In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and
Scott Wen-tau Yih (eds.), Findings of the Association for
Computational Linguistics: EMNLP 2021, Virtual Event / Punta Cana,
Dominican Republic, 16-20 November, 2021, pp.  2618–2632. Association for
Computational Linguistics, 2021.
URL <https://doi.org/10.18653/v1/2021.findings-emnlp.223>.

[Liu et al., 2023]
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.
Visual instruction tuning.
arXiv preprint arXiv:2304.08485, 2023a.

[Liu et al., 2023]
Xiao Liu, Hanyu Lai, Hao Yu, Yifan Xu, Aohan Zeng, Zhengxiao Du, Peng Zhang,
Yuxiao Dong, and Jie Tang.
WebGLM: Towards an efficient web-enhanced question answering system
with human preferences.
arXiv preprint arXiv:2306.07906, 2023b.

[Liu et al., 2019]
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer
Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.
RoBERTa: A robustly optimized BERT pretraining approach.
arXiv preprint arXiv:1907.11692, 2019.

[Liu et al., 2023]
Yue Liu, Thanh Le-Cong, Ratnadira Widyasari, Chakkrit Tantithamthavorn,
Li Li, Xuan-Bach Dinh Le, and David Lo.
Refining ChatGPT-generated code: Characterizing and mitigating code
quality issues.
CoRR, abs/2307.12596, 2023c.
URL <https://doi.org/10.48550/arXiv.2307.12596>.

[Longpre et al., 2023]
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny
Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al.
The Flan collection: Designing data and methods for effective
instruction tuning.
arXiv preprint arXiv:2301.13688, 2023.

[Loshchilov & Hutter, 2017]
Ilya Loshchilov and Frank Hutter.
Decoupled weight decay regularization.
arXiv preprint arXiv:1711.05101, 2017.

[Lu et al., 2023]
Keming Lu, Hongyi Yuan, Zheng Yuan, Runji Lin, Junyang Lin, Chuanqi Tan, Chang
Zhou, and Jingren Zhou.
#InsTag: Instruction tagging for analyzing supervised
fine-tuning of large language models.
CoRR, abs/2308.07074, 2023.
URL <https://doi.org/10.48550/arXiv.2308.07074>.

[Luo et al., 2023]
Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo
Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang.
WizardMath: Empowering mathematical reasoning for large language
models via reinforced evol-instruct.
arXiv preprint arXiv:2308.09583, 2023a.

[Luo et al., 2023]
Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang
Tao, Jing Ma, Qingwei Lin, and Daxin Jiang.
WizardCoder: Empowering code large language models with
arXiv preprint arXiv:2306.08568, 2023b.

[Mosaic ML, 2023]
Mosaic ML.
MPT-30B: Raising the bar for open-source foundation models, 2023.
URL <https://www.mosaicml.com/blog/mpt-30b>.

[Muennighoff et al., 2022]
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella
Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey
Schoelkopf, et al.
Crosslingual generalization through multitask finetuning.
arXiv preprint arXiv:2211.01786, 2022.

[Muennighoff et al., 2023]
Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui,
Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro von Werra, and Shayne
OctoPack: Instruction tuning code large language models.
CoRR, abs/2308.07124, 2023.

[Nakano et al., 2021]
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina
Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders,
et al.
WebGPT: Browser-assisted question-answering with human feedback.
arXiv preprint arXiv:2112.09332, 2021.

[Nye et al., 2021]
Maxwell Nye, Anders Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin,
David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan,
Charles Sutton, and Augustus Odena.
Show your work: Scratchpads for intermediate computation with
language models.
ArXiv, abs/2112.00114, 2021.

[OpenAI, 2022]
Introducing ChatGPT, 2022.
URL <https://openai.com/blog/chatgpt>.

[OpenAI, 2022]
ChatML, 2022.

[OpenAI, 2023]
GPT4 technical report.
arXiv preprint arXiv:2303.08774, 2023.

[OpenCompass Team, 2023]
OpenCompass Team.
OpenCompass: A universal evaluation platform for foundation models,
URL <https://opencompass.org.cn/leaderboard-llm>.

[Ouyang et al., 2022]
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela
Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John
Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda
Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe.
Training language models to follow instructions with human feedback.
In NeurIPS, 2022.

[Paperno et al., 2016]
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham,
Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel
The LAMBADA dataset: Word prediction requiring a broad discourse
In Proceedings of the 54th Annual Meeting of the Association
for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin,
Germany, Volume 1: Long Papers. The Association for Computer Linguistics,
URL <https://doi.org/10.18653/v1/p16-1144>.

[Peng et al., 2023]
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole.
YaRN: Efficient context window extension of large language models.
arXiv preprint arXiv:2309.00071, 2023a.

[Peng et al., 2023]
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and
Furu Wei.
Kosmos-2: Grounding multimodal large language models to the world.
arXiv preprint arXiv:2306.14824, 2023b.

[Qwen Team, Alibaba Group, 2023]
Qwen Team, Alibaba Group.
Evaluation benchmark for code intepreter, 2023a.
URL <https: //github.com/QwenLM/Qwen-Agent/tree/main/benchmark>.

[Qwen Team, Alibaba Group, 2023]
Qwen Team, Alibaba Group.
Evaluation benchmark for tool usage through ReAct prompting,
URL <https: //github.com/QwenLM/Qwen-7B/tree/main/eval>.

[Radford et al., 2018]
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al.
Improving language understanding by generative pre-training.
Technical report, OpenAI, 2018.

[Rae et al., 2021]
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann,
Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young,
et al.
Scaling language models: Methods, analysis & insights from training
arXiv preprint arXiv:2112.11446, 2021.

[Rafailov et al., 2023]
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D
Manning, and Chelsea Finn.
Direct preference optimization: Your language model is secretly a
reward model.
arXiv preprint arXiv:2305.18290, 2023.

[Raffel et al., 2020]
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael
Matena, Yanqi Zhou, Wei Li, and Peter J Liu.
Exploring the limits of transfer learning with a unified text-to-text
The Journal of Machine Learning Research, 210
(1):0 5485–5551, 2020.

[Ramachandran et al., 2017]
Prajit Ramachandran, Barret Zoph, and Quoc V Le.
Searching for activation functions.
arXiv preprint arXiv:1710.05941, 2017.

[Reed et al., 2022]
Scott E. Reed, Konrad Zolna, Emilio Parisotto, Sergio Gómez Colmenarejo,
Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie
Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley
Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar
Bordbar, and Nando de Freitas.
A generalist agent.
Trans. Mach. Learn. Res., 2022, 2022.
URL <https://openreview.net/forum?id=1ikK0kHjvj>.

[Rozière et al., 2023]
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat,
Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin,
et al.
Code Llama: Open foundation models for code.
arXiv preprint arXiv:2308.12950, 2023.

[Sanh et al., 2021]
Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika,
Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja,
et al.
Multitask prompted training enables zero-shot task generalization.
arXiv preprint arXiv:2110.08207, 2021.

[Sap et al., 2019]
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi.
SocialIQA: Commonsense reasoning about social interactions.
CoRR, abs/1904.09728, 2019.
URL <http://arxiv.org/abs/1904.09728>.

[Scao et al., 2022]
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić,
Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François
Yvon, Matthias Gallé, et al.
BLOOM: A 176B-parameter open-access multilingual language model.
arXiv preprint arXiv:2211.05100, 2022.

[Schick et al., 2023]
Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria
Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom.
Toolformer: Language models can teach themselves to use tools.
arXiv preprint arXiv:2302.04761, 2023.

[Schulman et al., 2017]
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.
Proximal policy optimization algorithms.
arXiv preprint arXiv:1707.06347, 2017.

[Shazeer, 2020]
Noam Shazeer.
GLU variants improve transformer.
arXiv preprint arXiv:2002.05202, 2020.

[Shen et al., 2023]
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting
HuggingGPT: Solving AI tasks with ChatGPT and its friends in
arXiv preprint arXiv:2303.17580, 2023.

[Shoeybi et al., 2019]
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper,
and Bryan Catanzaro.
Megatron-LM: Training multi-billion parameter language models using
model parallelism.
arXiv preprint arXiv:1909.08053, 2019.

[Si et al., 2023]
Qingyi Si, Tong Wang, Naibin Gu, Rui Liu, and Zheng Lin.
Alpaca-CoT: An instruction-tuning platform with unified interface
of instruction collection, parameter-efficient methods, and large language
models, 2023.
URL <https://github.com/PhoebusSi/alpaca-CoT>.

[Song et al., 2023]
Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and
Houfeng Wang.
Preference ranking optimization for human alignment.
arXiv preprint arXiv:2306.17492, 2023.

[Stability AI, 2023]
Stability AI.
StableBeluga2, 2023.
URL <https://huggingface.co/stabilityai/StableBeluga2>.

[Stiennon et al., 2020]
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea
Voss, Alec Radford, Dario Amodei, and Paul F Christiano.
Learning to summarize with human feedback.
Advances in Neural Information Processing Systems,
33:0 3008–3021, 2020.

[Su, 2023]
Jianlin Su.
Improving transformer: Length extrapolation ability and position
robustness, 2023a.
URL <https://spaces.ac.cn/archives/9444>.

[Su, 2023]
Jianlin Su.
The magical effect of the Bias term: RoPE + Bias = better
length extrapolation, 2023b.
URL <https://spaces.ac.cn/archives/9577>.

[Su et al., 2021]
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu.
Roformer: Enhanced transformer with rotary position embedding.
arXiv preprint arXiv:2104.09864, 2021.

[Sun et al., 2023]
Tianxiang Sun, Xiaotian Zhang, Zhengfu He, Peng Li, Qinyuan Cheng, Hang Yan,
Xiangyang Liu, Yunfan Shao, Qiong Tang, Xingjian Zhao, Ke Chen, Yining Zheng,
Zhejian Zhou, Ruixiao Li, Jun Zhan, Yunhua Zhou, Linyang Li, Xiaogui Yang,
Lingling Wu, Zhangyue Yin, Xuanjing Huang, and Xipeng Qiu.
MOSS: Training conversational language models from synthetic data,

[Sun et al., 2023]
Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David
Cox, Yiming Yang, and Chuang Gan.
Principle-driven self-alignment of language models from scratch with
minimal human supervision.
arXiv preprint arXiv:2305.03047, 2023b.

[Suzgun et al., 2022]
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay,
Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al.
Challenging big-bench tasks and whether chain-of-thought can solve
arXiv preprint arXiv:2210.09261, 2022.

[Szafraniec et al., 2023]
Marc Szafraniec, Baptiste Rozière, Hugh Leather, Patrick Labatut,
François Charton, and Gabriel Synnaeve.
Code translation with compiler representations.
In The Eleventh International Conference on Learning
Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net,
URL <https://openreview.net/pdf?id=XomEU3eNeSQ>.

[Talmor et al., 2019]
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant.
CommonsenseQA: A question answering challenge targeting
commonsense knowledge.
In Jill Burstein, Christy Doran, and Thamar Solorio (eds.),
Proceedings of the 2019 Conference of the North American Chapter of the
Association for Computational Linguistics: Human Language Technologies,
NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and
Short Papers), pp.  4149–4158. Association for Computational Linguistics,
URL <https://doi.org/10.18653/v1/n19-1421>.

[Taori et al., 2023]
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos
Guestrin, Percy Liang, and Tatsunori B. Hashimoto.
Stanford Alpaca: An instruction-following LLaMA model, 2023.
URL <https://github.com/tatsu-lab/stanford_alpaca>.

[Taylor et al., 2022]
Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony
Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic.
Galactica: A large language model for science, 2022.

[Thoppilan et al., 2022]
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv
Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du,
YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo
Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao
Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Yanqi Zhou,
Chung-Ching Chang, Igor Krivokon, Will Rusch, Marc Pickett, Kathleen S.
Meier-Hellstern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos
Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prabhakaran,
Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin
Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew
Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray
Kurzweil, Blaise Agüera y Arcas, Claire Cui, Marian Croak, Ed H. Chi,
and Quoc Le.
LaMDA: Language models for dialog applications.
CoRR, abs/2201.08239, 2022.
URL <https://arxiv.org/abs/2201.08239>.

[Touvron et al., 2023]
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne
Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric
Hambro, Faisal Azhar, et al.
LLaMA: Open and efficient foundation language models.
arXiv preprint arXiv:2302.13971, 2023a.

[Touvron et al., 2023]
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine
Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale,
Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem
Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,
Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar
Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa,
Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux,
Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier
Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew
Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan
Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang,
Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan
Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang,
Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom.
Llama 2: Open foundation and fine-tuned chat models.
CoRR, abs/2307.09288, 2023b.
URL <https://doi.org/10.48550/arXiv.2307.09288>.

[Vaswani et al., 2017]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,
Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.
Attention is all you need.
Advances in neural information processing systems, 30, 2017.

[Wang et al., 2023]
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu,
Linxi Fan, and Anima Anandkumar.
Voyager: An open-ended embodied agent with large language models.
arXiv preprint arXiv:2305.16291, 2023a.

[Wang et al., 2022]
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Huai hsin Chi, and Denny
Self-consistency improves chain of thought reasoning in language
ArXiv, abs/2203.11171, 2022.

[Wang et al., 2017]
Yan Wang, Xiaojiang Liu, and Shuming Shi.
Deep neural solver for math word problems.
In Conference on Empirical Methods in Natural Language
Processing, 2017.
URL <https://api.semanticscholar.org/CorpusID:910689>.

[Wang et al., 2023]
Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot,
Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith,
Iz Beltagy, et al.
How far can camels go? Exploring the state of instruction tuning on
open resources.
arXiv preprint arXiv:2306.04751, 2023b.

[Wang et al., 2023]
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel
Khashabi, and Hannaneh Hajishirzi.
Self-Instruct: Aligning language models with self-generated
In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.),
Proceedings of the 61st Annual Meeting of the Association for
Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto,
Canada, July 9-14, 2023, pp.  13484–13508. Association for Computational
Linguistics, 2023c.
URL <https://doi.org/10.18653/v1/2023.acl-long.754>.

[Wang et al., 2021]
Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi.
CodeT5: Identifier-aware unified pre-trained encoder-decoder models
for code understanding and generation.
arXiv preprint arXiv:2109.00859, 2021.

[Wang et al., 2023]
Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi D. Q. Bui, Junnan Li, and
Steven C. H. Hoi.
CodeT5+: Open code large language models for code understanding and
CoRR, abs/2305.07922, 2023d.
URL <https://doi.org/10.48550/arXiv.2305.07922>.

[Wei et al., 2022]
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian
Lester, Nan Du, Andrew M. Dai, and Quoc V. Le.
Finetuned language models are zero-shot learners.
In The Tenth International Conference on Learning
Representations, ICLR 2022, Virtual Event, April 25-29, 2022.
OpenReview.net, 2022a.
URL <https://openreview.net/forum?id=gEZrGCozdqR>.

[Wei et al., 2022]
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian
Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed Huai
hsin Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and
William Fedus.
Emergent abilities of large language models.
Trans. Mach. Learn. Res., 2022, 2022b.
URL <https://api.semanticscholar.org/CorpusID:249674500>.

[Wei et al., 2022]
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V
Le, Denny Zhou, et al.
Chain-of-thought prompting elicits reasoning in large language
Advances in Neural Information Processing Systems,
35:0 24824–24837, 2022c.

[Wolf et al., 2019]
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue,
Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz,
et al.
HuggingFace's transformers: State-of-the-art natural language
arXiv preprint arXiv:1910.03771, 2019.

[Xu et al., 2023]
Benfeng Xu, An Yang, Junyang Lin, Quan Wang, Chang Zhou, Yongdong Zhang, and
Zhendong Mao.
ExpertPrompting: Instructing large language models to be
distinguished experts.
arXiv preprint arXiv:2305.14688, 2023a.

[Xu et al., 2023]
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang
Tao, and Daxin Jiang.
WizardLM: Empowering large language models to follow complex
arXiv preprint arXiv:2304.12244, 2023b.

[Xu et al., 2023]
Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley.
Baize: An open-source chat model with parameter-efficient tuning on
self-chat data.
arXiv preprint arXiv:2304.01196, 2023c.

[Xu et al., 2023]
Yuzhuang Xu, Shuo Wang, Peng Li, Fuwen Luo, Xiaolong Wang, Weidong Liu, and
Yang Liu.
Exploring large language models for communication games: An empirical
study on werewolf.
arXiv preprint arXiv:2309.04658, 2023d.

[Yang et al., 2023]
Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Chao Yin, Chenxu Lv,
Da Pan, Dian Wang, Dong Yan, Fan Yang, Fei Deng, Feng Wang, Feng Liu,
Guangwei Ai, Guosheng Dong, Haizhou Zhao, Hang Xu, Haoze Sun, Hongda Zhang,
Hui Liu, Jiaming Ji, Jian Xie, Juntao Dai, Kun Fang, Lei Su, Liang Song,
Lifeng Liu, Liyun Ru, Luyao Ma, Mang Wang, Mickel Liu, MingAn Lin, Nuolan
Nie, Peidong Guo, Ruiyang Sun, Tao Zhang, Tianpeng Li, Tianyu Li, Wei Cheng,
Weipeng Chen, Xiangrong Zeng, Xiaochuan Wang, Xiaoxi Chen, Xin Men, Xin Yu,
Xuehai Pan, Yanjun Shen, Yiding Wang, Yiyu Li, Youxin Jiang, Yuchen Gao,
Yupeng Zhang, Zenan Zhou, and Zhiying Wu.
Baichuan 2: Open large-scale language models.
Technical report, Baichuan Inc., 2023.

[Yang, 2023]
Jianxin Yang.
<https://github.com/yangjianxin1/Firefly>, 2023.

[Yao et al., 2022]
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan,
and Yuan Cao.
ReAct: Synergizing reasoning and acting in language models.
arXiv preprint arXiv:2210.03629, 2022.

[Ye et al., 2023]
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang
Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al.
mPLUG-Owl: Modularization empowers large language models with
arXiv preprint arXiv:2304.14178, 2023.

[Yu et al., 2023]
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang,
James T. Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu.
Metamath: Bootstrap your own mathematical questions for large
language models, 2023.

[Yuan et al., 2023]
Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan,
Chang Zhou, and Jingren Zhou.
Scaling relationship on learning mathematical reasoning with large
language models, 2023a.

[Yuan et al., 2023]
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, and Songfang Huang.
How well do large language models perform in arithmetic tasks?
arXiv preprint arXiv:2304.02015, 2023b.

[Yuan et al., 2023]
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang.
RRHF: Rank responses to align language models with human feedback
without tears, 2023c.

[Yue et al., 2023]
Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and
Wenhu Chen.
MAmmoTH: Building math generalist models through hybrid instruction
arXiv preprint arXiv:2309.05653, 2023.

[Zellers et al., 2019]
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi.
HellaSwag: Can a machine really finish your sentence?
In Anna Korhonen, David R. Traum, and Lluís Màrquez
(eds.), Proceedings of the 57th Conference of the Association for
Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2,
2019, Volume 1: Long Papers, pp.  4791–4800. Association for Computational
Linguistics, 2019.
URL <https://doi.org/10.18653/v1/p19-1472>.

[Zeng et al., 2022]
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi
Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al.
GLM-130B: An open bilingual pre-trained model.
arXiv preprint arXiv:2210.02414, 2022.

[Zhang et al., 2023]
Fengji Zhang, Bei Chen, Yue Zhang, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang
Lou, and Weizhu Chen.
RepoCoder: Repository-level code completion through iterative
retrieval and generation.
CoRR, abs/2303.12570, 2023a.
URL <https://doi.org/10.48550/arXiv.2303.12570>.

[Zhang et al., 2022]
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui
Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al.
OPT: Open pre-trained transformer language models.
arXiv preprint arXiv:2205.01068, 2022.

[Zhang et al., 2023]
Xiaotian Zhang, Chunyang Li, Yi Zong, Zhengyu Ying, Liang He, and Xipeng Qiu.
Evaluating the performance of large language models on GAOKAO
CoRR, abs/2305.12474, 2023b.
URL <https://doi.org/10.48550/arXiv.2305.12474>.

[Zheng et al., 2023]
Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang,
Lei Shen, Andi Wang, Yang Li, Teng Su, Zhilin Yang, and Jie Tang.
CodeGeeX: A pre-trained model for code generation with
multilingual evaluations on humaneval-x.
CoRR, abs/2303.17568, 2023.
URL <https://doi.org/10.48550/arXiv.2303.17568>.

[Zhong et al., 2023]
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin
Saied, Weizhu Chen, and Nan Duan.
AGIEval: A human-centric benchmark for evaluating foundation
CoRR, abs/2304.06364, 2023a.
URL <https://doi.org/10.48550/arXiv.2304.06364>.

[Zhong et al., 2023]
Wanjun Zhong, Lianghong Guo, Qiqi Gao, and Yanlin Wang.
MemoryBank: Enhancing large language models with long-term memory.
arXiv preprint arXiv:2305.10250, 2023b.

[Zhou et al., 2022]
Denny Zhou, Nathanael Scharli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang,
Dale Schuurmans, Olivier Bousquet, Quoc Le, and Ed Huai hsin Chi.
Least-to-most prompting enables complex reasoning in large language
ArXiv, abs/2205.10625, 2022.

§ APPENDIX

§.§ More Training Details

§.§.§ Data Format for

Different from conventional pretraining based on autoregressive next-token prediction, despite using a similar training task, there should be a specially design data format for SFT and RLHF to build a conversational AI assistant model.
Common formats include “human-assistant” and ChatML formats.
As to our knowledge, one of the earliest examples of the human-assistant format comes from Anthropic [Bai et al., 2022], which adds a special phrase “” in front of the user input and “” in front of the assistant response. It is easy for the base language model to transfer to the pattern of conversational AI. However, as the specific phrases are common words, it might be hard for the model to disambiguate from these words in other contexts.

Instead, we turned to the ChatML format proposed by OpenAI.[https://github.com/openai/openai-python/blob/main/chatml.md] This format allows the use of special tokens, i.e., “” and “”, that do not appear in pretraining, and thus resolve the aforementioned problem. We demonstrate an example of the format below.

breakable, enhanced,
left*=10pt, right*=10pt,
top=0pt, bottom=0pt,
subtitle style=boxrule=0pt,colback=gray!50!white,

[title=ChatML Format]

You are a helpful assistant.<|im_end|>
Hello! How can I assist you today?<|im_end|>

§.§ Evaluation

§.§.§ Automatic Evaluation

To provide a whole picture of the performance of our model series , here in this section we illustrate the detailed performance of our models as well as the baselines in the comprehensive benchmark evaluation proposed by OpenCompass Team, 2023.
We report the results in multiple tables based on the officially provided categories, including examination, language, knowledge, understanding, and reasoning. In terms of the performance of the baseline models, we report the higher results between the reported ones and those on the leaderboard.

Examination Here we evaluate the models on a series of datasets relevant to the examination. The datasets include:

* MMLU [Hendrycks et al., 2020] Massive Multi-task Language Understanding is designed for measuring language understanding capabilities. We report 5-shot results.

* C-Eval [Huang et al., 2023] C-Eval is a Chinese evaluation dataset spanning 52 diverse disciplines. We report 5-shot results.

* CMMLU [Li et al., 2023] CMMLU is designed for assessing language understanding capabilities in Chinese.  We report 5-shot results.

* AGIEval [Zhong et al., 2023] This is a benchmark consisting of human-centric examinations, including college entrance exams, law school admission tests, math competitions, and lawyer qualification tests. We report zero-shot results.

* Gaokao-Bench [Zhang et al., 2023] This is a benchmark with Gaokao (Chinese college-entrance examination) questions.  We report zero-shot results.

* ARC [Clark et al., 2018] ARC is a dataset consisting of grade-school level, multiple-choice science questions. It includes an easy set and a challenge set, which are referred by ARC-e and ARC-c. We report zero-shot results.

Results on MMLU. All are tested with five-shot accuracy. We provide the reported results of the other models for comparison.

Model     Params
1cSocial Sciences

2*MPT      7B     26.8         25.3     27.1     26.7     28.2

30B     46.9         39.0     52.8     44.5     52.9

lightgray2*Falcon      7B     26.2     26.2     24.7     26.4     27.4

40B     55.4         45.5     65.4     49.3     65.0

lightgray2*ChatGLM2      6B     47.9         41.2     54.4     43.7     54.5

12B     56.2         48.2     65.1     52.6     60.9

lightgrayInternLM      7B     51.0         -        -        -        -

lightgray2*Baichuan2     7B     54.2         -        -        -        -

13B     59.2         -        -        -        -

lightgrayXVERSE       13B     55.1         44.5     64.4     50.5     62.9

lightgray4*LLaMA        7B     35.1         30.5     38.3     34.0     38.1

13B       46.9         35.8     53.8     45.0     53.3

33B       57.8         46.0     66.7     55.8     63.4

65B       63.4         51.7     72.9     61.8     67.4

lightgray4*       7B      45.3         36.4     51.2     42.9     52.2

13B     54.8         44.1     62.6     52.8     61.1

34B       62.6         52.1     71.8     59.4     69.2

70B       68.9         58.0     80.3     65.0     74.6

black3*     1.8B     44.6         39.6     50.0     40.4     51.0

7B     58.2         50.2     68.6     52.5     64.9

14B     66.3         59.4     76.2     60.9     71.8

Leaderboard results of C-Eval. We include the results of both proprietary models and open-source models. Note that there are a number of models on the leaderboard with very few details, in terms of proprietary models, we only report the results of GPT-3.5, GPT-4, InternLM and ChatGLM2.

Model     Params     Avg.     Avg. (Hard)     STEM     Social Sciences     Humanities     Others

8cProprietary models

GPT-3.5     -     54.4     41.4     52.9     61.8     50.9     53.6

lightgrayGPT-4     -     68.7     54.9     67.1     77.6     64.5     67.8

lightgrayInternLM     123B     68.8     50.0     63.5     81.4     72.7     63.0

lightgrayChatGLM2     -     71.1     50.0     64.4     81.6     73.7     71.3

black8cOpen-source models

blackChatGLM2     6B                 51.7         37.1     48.6     60.5     51.3     49.8

lightgrayInternLM     7B                 52.8         37.1     48.0     67.4     55.4     45.8

lightgray2*Baichuan2      7B        54.0         -     -     -     -     -

13B       58.1         -     -     -     -     -

lightgrayXVERSE         13B             54.7         33.5     45.6     66.2     58.3     56.9

black3*       1.8B            54.7         41.8     50.8     69.9     56.3     46.2

7B             63.5         46.4     57.7     78.1     66.6     57.8

14B            72.1         53.7     65.7     85.4     75.3     68.4

Results on the other datasets of examination. Specifically, we report the results on CMMLU, AGIEval, ARC-e, and ARC-c.

Model     Params

1*MPT      7B     25.9     21.3     19.8     70.2     42.6

lightgray1*Falcon      7B     -     -     -     70.0     42.4

lightgray1*ChatGLM2      6B     49.3       39.0     46.4     73.0     61.0

lightgray2*InternLM      7B     51.8         36.9        43.0        78.7        69.5

20B     59.0     44.6     45.5     86.1     81.7

lightgray2*Baichuan2     7B     57.1        42.7        47.5        54.7        32.5

13B     62.0         48.2        54.3        61.9        38.0

lightgray4*LLaMA        7B     26.8         20.6     21.3     72.8     47.6

13B       31.5         22.0     20.4     74.8     52.7

33B       36.0         33.5     18.9     80.0     67.5

65B       40.6         33.9     19.1     80.6     69.5

lightgray3*       7B      31.8         21.8     18.9     75.2     45.9

13B     38.4         30.9     18.2     77.3     60.3

70B       53.6         40.2     23.3     85.9     78.3

lightgrayStableBeluga2      70B     51.8     41.6     40.9     91.2     86.1

black3*     1.8B     49.3         36.9     44.9     71.6     53.2

7B     62.2         45.8     52.5     84.0     75.3

14B     71.0         52.3     61.9     90.3     84.4

In terms of MMLU, we report the detailed results in Table <ref>. In terms of C-Eval, we report the results in Table <ref>. For the rest of the datasets, we report the results in Table <ref>. Note that AGIEval includes the parts of Chinese and English, while  only reported the results in the English part, so we use the results on OpenCompass. Additionally, while CMMLU, AGIEval, and Gaokao-Bench are related to Chinese, and MPT, Falcon, and the LLaMA series were not optimized for Chinese, these models achieved low performance on the datasets.

Knowledge and Understanding Here we evaluate the models on a series of datasets relevant to knowledge and natural language understanding.
The datasets include

* BoolQ [Clark et al., 2019] This is a QA dataset, where the questions are about passages of Wikipedia, and the model should answer yes or no to the given possible answer. We report zero-shot results.

* CommonsenseQA [Talmor et al., 2019] This is a dataset of multiple-choice question answering that asseses the understanding of commonsense knowledge. We report 8-shot results.

* NaturalQuestions [Kwiatkowski et al., 2019] It is a dataset of QA where the questions are from users and the answers are verified by experts. We report zero-shot results.

* LAMBADA [Paperno et al., 2016] This is dataset to evaluate language understanding by word prediction. It consists of passages related to human subjects. We report zero-shot results.

Results on the datasets concerning knowledge and understanding. Specifically, we report the results on BoolQ, CommonsenseQA, NaturalQuestions, and LAMBADA.

Model     Params

2*MPT      7B     75.0     61.8     11.6     70.0

lightgray2*Falcon      7B     67.5     20.8     15.7     -

lightgray1*ChatGLM2      6B     79.0       65.4     9.7     54.3

lightgray2*InternLM      7B     64.1         59.8           8.9         67.0

20B     87.5     70.6     25.2     71.8

lightgrayXVERSE       13B     64.2         62.2     0.3      48.2

lightgray2*Baichuan2     7B     63.2        63.0         9.4        73.3

13B     67.0         65.6           16.3         74.0

lightgray4*LLaMA        7B     76.5         64.9      16.8     73.3

13B       78.7         67.4      20.2     75.2

33B       84.4         72.5      30.9     77.2

65B       86.6         74.1      33.4     77.7

lightgray3*       7B      77.4         66.5      19.1     73.3

13B     82.4        67.3      24.9     76.5

70B       87.7         78.5      34.2     78.9

lightgrayStableBeluga2      70B     89.4     72.6     25.1     71.3

black3*     1.8B     68.0         60.1      3.2     58.4

7B     76.4         66.8      17.4      67.9

14B     86.2         70.3      23.9     71.1

We report the results in Table <ref>.

Reasoning We report the evaluation results on the datasets concerning reasoning, focusing on natural language reasoning. For the others, such as mathematics and coding, as we have illustrated detailed results, here we do not report those results repeatedly. The datasets for evaluation include:

* HellaSwag [Zellers et al., 2019] This is a commonsense natural language inference (NLI) dataset, where the questions are easy for humans but struggling for previous language models. We report zero-shot results.

* PIQA [Bisk et al., 2020] This is an NLI dataset assessing the physical knowledge. We report zero-shot results.

* SIQA [Sap et al., 2019] This is an NLI dataset evaluating social commonsense intelligence. We report zero-shot results.

* OCNLI [Hu et al., 2020] This is an NLI dataset focusing on Chinese. We report zero-shot results.

Results on the datasets related to natural language reasoning. Specifically, we report the results on HellaSwag, PIQA, SIQA, and OCNLI.

Model     Params

1*MPT      7B     76.4     80.6     48.5     30.0

lightgray1*Falcon      7B     74.1     76.7     47.2     -

lightgray1*ChatGLM2      6B     57.0       69.6     64.3      33.1
lightgray2*InternLM      7B     70.6         77.9           60.5       37.5

20B     78.1     80.3     72.8     42.5

lightgray2*Baichuan2     7B     67.0        76.2         44.4     30.3

13B     70.8         78.1           44.3        30.0

lightgray4*LLaMA        7B     76.1         79.8      48.9      33.6

13B       79.2         80.1      52.5      32.1

33B       82.8         82.3      57.8     30.7

65B       84.2         82.8      61.2     44.9

lightgray3*       7B      77.2         78.8      48.5       32.1

13B     80.7        80.5       54.8       34.1

70B       85.3         82.8      64.8     46.5

lightgrayStableBeluga2     70B     84.1     83.3     78.1     48.3

black3*     1.8B     56.7         73.3      56.1     39.0

7B     75.1         77.9      69.9      47.4

14B     80.2        79.9      77.9      57.9

We report the results in Table <ref>.

§.§.§ Human Evaluation

In this section, we demonstrate the cases of human analysis.
In our self-constructed evaluation dataset, the instructions are either manually written data or manual revised from public datasets, such as CLiB[https://github.com/jeinlee1991/chinese-llm-benchmark], C-Eval [Huang et al., 2023], FacTool [Chern et al., 2023], LeetCode[https://leetcode.cn/]), etc.

In terms of each case, we demonstrate the responses and Elo ratings[https://en.wikipedia.org/wiki/Elo_rating_system] of all models for comparison. Specifically, as the data in our human evaluation are in Chinese, we also provide their translations in English.

§.§ Analysis of Code Interpreter

Here we provide a case of comparison between  and . This case demonstrates the advantages of  in processing tabular data and performing complex tasks.

Example showcasing 's ability in using a code interpreter via ReAct prompting. The ReAct instruction is omitted for clarity.  creates a two-step plan and first investigates the columns present in the CSV file before proceeding to draw the plot, as shown in the top-left figure. , however, attempts to draw the plot based on non-existent columns in its initial attempt, as seen in the bottom figure.  can only reliably perform the task if the columns are provided in the user query, as shown in the top-right figure.