# GECKO: Generative Language Model for English, Code and Korean

Sungwoo Oh
KIFAI
<EMAIL_ADDRESS>
&Donggyu Kim
KIFAI
<EMAIL_ADDRESS>
Korea Institute of Finance and Artificial Intelligence(KIFAI) is an open
community aiming to research AI technologies and share the findings to the
public.

###### Abstract

We introduce GECKO, a bilingual large language model (LLM) optimized for
Korean and English, along with programming languages. GECKO is pretrained on
the balanced, high-quality corpus of Korean and English employing LLaMA
architecture. In this report, we share the experiences of several efforts to
build a better data pipeline for the corpus and to train our model. GECKO
shows great efficiency in token generations for both Korean and English,
despite its small size of vocabulary. We measure the performance on the
representative benchmarks in terms of Korean, English and Code, and it
exhibits great performance on KMMLU (Korean MMLU) and modest performance in
English and Code, even with its smaller number of trained tokens compared to
English-focused LLMs. GECKO is available to the open-source community under a
permissive license. We hope our work offers a research baseline and practical
insights for Korean LLM research. The model can be found at:
https://huggingface.co/kifai/GECKO-7B

##  1 Introduction

Recent advances in artificial intelligence yield significant breakthroughs in
the development of large language models (LLMs). Many proprietary LLMs [1, 33,
44] demonstrate human-level performances across multiple languages and on a
wide range of real-world tasks [5, 31, 51]. In response to this, the open-
source community has released various open large language models [46, 17, 18,
8], striving to match the capabilities of proprietary models.

While these open-source models have been mainly trained on English [46, 17] or
designed for specific use-cases such as programming [29, 47, 13] and
mathematics [28, 30, 50], there has been increasing demand for models
proficient in other languages. This need has led to the emergence of open-
source language models that show strong understanding of non-english languages
such as Chinese [2, 49], Finnish [32], and Indonesian [41]. They achieve
impressive performance by leveraging language-specific datasets at the
pretraining phase [2, 49, 32, 41].

Several open-source models enhance their linguistic performance by employing
the following strategies: 1) language-specific continuous pretraining [37], 2)
vocabulary expansion [24]. These approaches efficiently improve the cross-
lingual capabilities of the monolingual models compared to the process of
pretraining models from scratch, which requires massive computational
resources and extensive data.

Despite the achievements of previous Korean language models [34, 26, 21, 22],
research on pretraining methods and applications for Korean LLM remains
limited. To address this, we initiate the development of GECKO, a language
model designed mainly for Korean, yet capable in English and programming
languages. GECKO is pretrained from scratch, utilizing terabytes of textual
data in both Korean and English to secure a strong bilingual proficiency. In
the remainder of this report, we share our efforts and contributions as
follows:

  * •

Data preprocessing and training methods maintaining the balance between Korean
and English

  * •

Demonstration of strong performance in Korean with only small amount of
pretraining resources

  * •

Open-sourcing our model under a permissive license to encourage further
researches and applications

##  2 Datasets

###  2.1 Sources

Low-resource languages, such as Korean, have far fewer public data sources
available, even if they contain data with copyright issues. In contrast,
resource-rich languages like English have large, accessible data sources for
training language models.

Figure 1: Distribution of pretraining data sources for bilingual language
models. The left pie chart illustrates the proportional composition of the
corpus by language, highlighting a balanced representation of 35% Korean, 28%
English, and 37% code to accommodate low-resource language challenges. The
right pie chart details the types of data utilized, with 36% web sources, 24%
from Wikipedia, 16% from news articles, 16% from books, 5% from patents, and
3% from translated texts. This distribution supports efforts to enhance model
performance by diversifying and balancing the training data across different
types and languages.

#### Balancing Korean and English

As shown in Figure 1, similar to other bilingual language models [2, 32], we
aim to strike a balance between English and Korean in our pretraining corpus
by down-sampling and up-sampling English and Korean corpus respectively.

#### High quality Korean corpus

Since abundant open-source corpora for languages such as English and code
already exist, and their refinement and processing significantly impact on the
performance of language models [36, 35], our focus has shifted more towards
improving methods of data cleansing methods. However, because high-quality
Korean corpora without licensing issues are extremely limited, we collect data
from Web.

#### Reasoning capability

Additionally, research findings [48, 11, 16] indicate that incorporating code
data in the pretraining phase enhances the reasoning ability of language
models, along with the academic perspective that treats code data as its own
language. This ultimately led us to utilize three main corpora for
pretraining: English, code, and Korean.

#### Language Alignment

There is research [6] on using translation datasets consisting of different
language pairs for the purpose of multilingual alignment in the pretraining
phase. Adopting this methodology, we train our model to align languages
between English and Korean.

Figure 2: Pipeline for cleansing corpus

###  2.2 Preprocessing

We curate and process terabytes of Korean corpus, and utilize large scale
open-source corpora for English and programming languages. A sophisticated
pipeline for deduplication and cleaning of raw text is implemented to obtain
high-quality data as shown in Figure 2. The primary objectives of this data
processing are as follows:

  * •

Mitigate harmful content: Preprocess and use selective data in order to remove
harmful, toxic, and biased content from the training corpus. [19, 42].

  * •

Minimize data memorization: The data deduplication improves robustness and
generalization of the models when exposed to new, unseen data, preventing it
from merely replicating patterns and generating training examples [27, 20].

  * •

Keep structure information: Utilizing structural corpus including tables and
lists plays a crucial role in increasing model performance.

The training corpus includes the processing and normalization of specialized
datasets such as wikis, programming code, mathematical expressions, and expert
contributions. This step focuses on leveraging the structural elements
inherent in these data types while carefully preserving tags and markdown
features as shown in Figure 3. These considerations allow the model to
interpret and generate contextually informed and syntactically coherent
outputs, significantly enhancing its utility across various applications.

Figure 3: Example of normalization for a wiki dataset: The left image displays
the original data, while the right image shows the preprocessed and normalized
data in markdown format.

##  3 Pretraining

###  3.1 Tokenizer

We train GECKO tokenizer on the balanced corpus of Korean, English, and Code.
Similar to other large language models [46, 17], we utilize the Byte Pair
Encoding (BPE) algorithm and train the tokenizer using Hugging Face’s
tokenizer. We treat all numbers as individual digits and segment unknown UTF-8
characters into bytes to avoid out-of-vocabulary issues. Additionally, we opt
not to use NFKC normalization [9], recently reporting performance degradation
on BPE-based tokenizers [15, 38, 25].

We set the total vocabulary size to 32,000, following research [12] on optimal
vocabulary size that aims to balance computational efficiency and performance
considering larger vocabularies demand more computational power during the
inference. We measure the efficiency of GECKO tokenizer compared to others
using the following formula:

$\text{Efficiency}=\left(\frac{\\#\text{ of tokens}_{\text{model}}}{\\#\text{
of tokens}_{\text{GECKO}}}\right)\times 100\%$ (1)

The metric evaluates the tokenization efficiency by comparing the total number
of tokens produced by GECKO tokenizer and others. Our tokenizer demonstrates
superior efficiency in processing Korean while maintaining comparable results
in English and Code, contrasting to the models primarily trained in English.
The result of efficiency comparison using C4 corpus [39] and The Stack [23] is
illustrated in Table 1 and Figure 4.

Table 1: Overall toeknizer efficiency with respect to GECKO. Tokenizer | GECKO | Polyglot-Ko | LLaMA-2 | Mistral | Gemma | GPT-4
---|---|---|---|---|---|---
Vocab. size | 32,000 | 30,080 | 32,000 | 32,000 | 256,000 | 100,277
Efficiency | 100% | 71% | 86% | 92% | 109% | 110%
KoreanEnglishCode$0$$20$$40$$60$$80$$100$$120$$140$$100$$100$$100$$101$$57$$56$$48$$111$$98$$62$$115$$100$$67$$130$$132$Token
Efficiency (%)GECKOPolyglot-KoLLaMA-2MistralGPT-4 Figure 4: Comparative
analysis of tokenizer efficiency across multiple language models. This graph
illustrates the performance of various tokenizers, including GECKO, Polyglot-
Ko, LLaMA-2, Mistral, and GPT-4, across Korean, English, and code text
corpora. The y-axis represents token efficiency as a percentage, with higher
values indicating superior encoding performance relative to the tokenizer of
GECKO. This analysis highlights the varying efficiency levels each model
exhibits, offering insights into how effectively each tokenizer encodes
multilingual and coding data. The dashed red line at 100% serves as a
benchmark for baseline efficiency.

###  3.2 Training Details

GECKO adopts the classical decoder-only Transformer architecture used in LLaMA
[46]. The AdamW optimizer is employed to train the model, setting $\beta_{1}$
at 0.9 and $\beta_{2}$ at 0.95. The optimizer is configured to warm up over
10,000 iterations with a linearly increasing learning rate that peaks at 3e-4
and then decays to 3e-5 according to a cosine schedule. The model is trained
with 200 billion tokens using BF16 mixed precision. Rotary positional
embedding is utilized to train longer context tokens up to 8192 in length,
allowing longer sequences to be understood during pretraining. We use sequence
packing [7] to assemble multiple training samples into a single sequence and
use end-of-sequence token to separate the document sequences.

###  3.3 Training Infrastructure

We train our model on Google Cloud Platform and used TPUv4 with 256 chips,
utilizing Fully Sharded Data Parallelism (FSDP) and model parallelism.
Leveraging JAX [3, 10], we implement the single controller programming
paradigm, which enables us to manage and parallelize our training efficiently
using just one Python command.

Table 2: Performance evaluations across different models and benchmarks | KMMLU | MMLU | HumanEval | MATH
---|---|---|---|---
Model | 5-shot | 5-shot | pass@1 | 4-shot
LLaMA-2 7B | 24.2 | 45.3 | 12.8 | 2.5
Mistral 7B | 21.0 | 62.5 | 26.2 | 12.7
Gemma 7B | 21.1 | 64.3 | 32.3 | 24.3
Polyglot-Ko 5.8B | 28.3 | 26.8 | 0.0 | 0.3
GECKO | 30.7 | 28.3 | 17.7 | 4.3

##  4 Evaluation

We evaluate several pretrained open-source large language models (LLMs)
released under permissive licenses. For performance assessment, we use
standard academic benchmarks to evaluate knowledge and reasoning abilities
[14], as well as coding [4] and mathematics [40]. For LLaMA-2 [46], Mistral
[17], and Gemma [45], we directly quote the scores as reported in the Gemma
technical report [45]. Additionally, for the Korean evaluation set KMMLU [43],
we conduct our own evaluation in the same environment with previous works. The
result is shown in Table 2. In terms of Korean understanding (KMMLU), GECKO
shows better performance compared to the evaluated models. Our model also
demonstrates moderate performance in coding and mathematics.

##  5 Conclusion

GECKO is an open-source Korean pretrained LLM released under a permissive
license. Our work can contribute to both academic research and the practical
development of the large Korean language model pretraining. Our immediate goal
is to release an improved version of the model with additional training
resources. We are also preparing for instruction fine-tuning to evaluate
GECKO’s instruction-following ability. We believe that open-sourcing
artificial intelligence technologies helps create safer products, accelerate
innovation, and expand markets.

## Acknowledgements

We deeply thank the TRC Team at Google Cloud for their dedication and support,
which significantly enhanced our research through provision of Cloud TPUs.

## References

  * [1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.  Gpt-4 technical report.  arXiv preprint arXiv:2303.08774, 2023.
  * [2] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al.  Qwen technical report.  arXiv preprint arXiv:2309.16609, 2023.
  * [3] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, et al.  Jax: composable transformations of python+ numpy programs.  2018\.
  * [4] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al.  Evaluating large language models trained on code.  arXiv preprint arXiv:2107.03374, 2021.
  * [5] Cheng-Han Chiang and Hung-yi Lee.  Can large language models be an alternative to human evaluations?  arXiv preprint arXiv:2305.01937, 2023.
  * [6] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al.  Palm: Scaling language modeling with pathways.  Journal of Machine Learning Research, 24(240):1–113, 2023.
  * [7] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al.  Scaling instruction-finetuned language models.  Journal of Machine Learning Research, 25(70):1–53, 2024.
  * [8] Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin.  Free dolly: Introducing the world’s first truly open instruction-tuned llm.  Company Blog of Databricks, 2023.
  * [9] Mark Davis and Martin Dürst.  Unicode normalization forms, 2001.
  * [10] Roy Frostig, Matthew James Johnson, and Chris Leary.  Compiling machine learning programs via high-level tracing.  Systems for Machine Learning, 4(9), 2018.
  * [11] Yao Fu, Hao Peng, and Tushar Khot.  How does gpt obtain its ability? tracing emergent abilities of language models to their sources.  Yao Fu’s Notion, 2022.
  * [12] Thamme Gowda and Jonathan May.  Finding the optimal vocabulary size for neural machine translation.  arXiv preprint arXiv:2004.02334, 2020.
  * [13] Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y Wu, YK Li, et al.  Deepseek-coder: When the large language model meets programming–the rise of code intelligence.  arXiv preprint arXiv:2401.14196, 2024.
  * [14] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.  Measuring massive multitask language understanding.  Proceedings of the International Conference on Learning Representations (ICLR), 2021.
  * [15] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al.  Training compute-optimal large language models.  arXiv preprint arXiv:2203.15556, 2022.
  * [16] Jie Huang and Kevin Chen-Chuan Chang.  Towards reasoning in large language models: A survey.  arXiv preprint arXiv:2212.10403, 2022.
  * [17] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al.  Mistral 7b.  arXiv preprint arXiv:2310.06825, 2023.
  * [18] Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al.  Mixtral of experts.  arXiv preprint arXiv:2401.04088, 2024.
  * [19] Nikhil Kandpal, Eric Wallace, and Colin Raffel.  Deduplicating training data mitigates privacy risks in language models.  In International Conference on Machine Learning, pages 10697–10707. PMLR, 2022.
  * [20] Aly Kassem, Omar Mahmoud, and Sherif Saad.  Preserving privacy through dememorization: An unlearning technique for mitigating memorization risks in language models.  In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4360–4379, 2023.
  * [21] Ildoo Kim, Gunsoo Han, Jiyeon Ham, and Woonhyuk Baek.  Kogpt: Kakaobrain korean(hangul) generative pre-trained transformer.  https://github.com/kakaobrain/kogpt, 2021.
  * [22] Hyunwoong Ko, Kichang Yang, Minho Ryu, Taekyoon Choi, Seungmu Yang, Sungho Park, et al.  A technical report for polyglot-ko: Open-source large-scale korean language models.  arXiv preprint arXiv:2306.02254, 2023.
  * [23] Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau, Leandro von Werra, and Harm de Vries.  The stack: 3 tb of permissively licensed source code.  Preprint, 2022.
  * [24] L. Junbum.  llama-2-ko-7b (revision 4a9993e), 2023.
  * [25] Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al.  Bloom: A 176b-parameter open-access multilingual language model.  2023\.
  * [26] Junbum Lee.  Kcbert: Korean comments bert.  In Annual Conference on Human and Language Technology, pages 437–440. Human and Language Technology, 2020.
  * [27] Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini.  Deduplicating training data makes language models better.  arXiv preprint arXiv:2107.06499, 2021.
  * [28] Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al.  Solving quantitative reasoning problems with language models.  Advances in Neural Information Processing Systems, 35:3843–3857, 2022.
  * [29] Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al.  Starcoder: may the source be with you!  arXiv preprint arXiv:2305.06161, 2023.
  * [30] Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang.  Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct.  arXiv preprint arXiv:2308.09583, 2023.
  * [31] Xiaoliang Luo, Akilles Rechardt, Guangzhi Sun, Kevin K Nejad, Felipe Yáñez, Bati Yilmaz, Kangjoo Lee, Alexandra O Cohen, Valentina Borghesani, Anton Pashkov, et al.  Large language models surpass human experts in predicting neuroscience results.  arXiv preprint arXiv:2403.03230, 2024.
  * [32] Risto Luukkonen, Jonathan Burdge, Elaine Zosa, Aarne Talman, Ville Komulainen, Väinö Hatanpää, Peter Sarlin, and Sampo Pyysalo.  Poro 34b and the blessing of multilinguality.  arXiv preprint arXiv:2404.01856, 2024.
  * [33] Claude Models.  Model card and evaluations for claude models, 2023.
  * [34] Jangwon Park.  Koelectra: Pretrained electra model for korean.  https://github.com/monologg/KoELECTRA, 2020.
  * [35] Guilherme Penedo, Hynek Kydlíček, Leandro von Werra, and Thomas Wolf.  Fineweb, 2024.
  * [36] Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay.  The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only.  arXiv preprint arXiv:2306.01116, 2023.
  * [37] Kunat Pipatanakul, Phatrasek Jirabovonvisut, Potsawee Manakul, Sittipong Sripaisarnmongkol, Ruangsak Patomwong, Pathomporn Chokchainant, and Kasima Tharnpipitchai.  Typhoon: Thai large language models.  arXiv preprint arXiv:2312.13951, 2023.
  * [38] Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al.  Scaling language models: Methods, analysis & insights from training gopher.  arXiv preprint arXiv:2112.11446, 2021.
  * [39] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  Journal of machine learning research, 21(140):1–67, 2020.
  * [40] Hill Saxton, Grefenstette and Kohli.  Analysing mathematical reasoning abilities of neural models.  arXiv:1904.01557, 2019.
  * [41] AI Singapore.  Sea-lion (southeast asian languages in one network): A family of large language models for southeast asia.  https://github.com/aisingapore/sealion, 2023.
  * [42] Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, et al.  Dolma: An open corpus of three trillion tokens for language model pretraining research.  arXiv preprint arXiv:2402.00159, 2024.
  * [43] Guijin Son, Hanwool Lee, Sungdong Kim, Seungone Kim, Niklas Muennighoff, Taekyoon Choi, Cheonbok Park, Kang Min Yoo, and Stella Biderman.  Kmmlu: Measuring massive multitask language understanding in korean.  arXiv preprint arXiv:2402.11548, 2024.
  * [44] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al.  Gemini: a family of highly capable multimodal models.  arXiv preprint arXiv:2312.11805, 2023.
  * [45] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al.  Gemma: Open models based on gemini research and technology.  arXiv preprint arXiv:2403.08295, 2024.
  * [46] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  arXiv preprint arXiv:2307.09288, 2023.
  * [47] Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi DQ Bui, Junnan Li, and Steven CH Hoi.  Codet5+: Open code large language models for code understanding and generation.  arXiv preprint arXiv:2305.07922, 2023.
  * [48] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al.  Chain-of-thought prompting elicits reasoning in large language models.  Advances in neural information processing systems, 35:24824–24837, 2022.
  * [49] Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al.  Yi: Open foundation models by 01. ai.  arXiv preprint arXiv:2403.04652, 2024.
  * [50] Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen.  Mammoth: Building math generalist models through hybrid instruction tuning.  arXiv preprint arXiv:2309.05653, 2023.
  * [51] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al.  Judging llm-as-a-judge with mt-bench and chatbot arena.  Advances in Neural Information Processing Systems, 36, 2024.