Understanding LLM Development Through Longitudinal Study: Insights from the Open Ko-LLM Leaderboard
Chanjun Park Hyeonwoo Kim

Transmit Beamforming Design for ISAC with Stacked Intelligent Metasurfaces
     
Shunyu Li, Student Member, IEEE, 
Fan Zhang, Student Member, IEEE,
Tianqi Mao, Member, IEEE, Rui Na, Zhaocheng Wang, Fellow, IEEE, and George K. Karagiannidis, Fellow, IEEE


        
This work was supported by National Natural Science Foundation of China under Grant No. 62088101. (Corresponding authors: Tianqi Mao, Rui Na.)
S. Li, T. Mao and R. Na are with State Key Laboratory of CNS/ATM, Beijing Institute of Technology, Beijing 100081, China. T. Mao is also with Beijing Institute of Technology (Zhuhai), Zhuhai 519088, China. R. Na is also with Yangtze Delta Region Academy of Beijing Institute of Technology (Jiaxing), Jiaxing 314019, China (e-mails: li.shunyu@bit.edu.cn, maotq@bit.edu.cn, narui@bit.edu.cn).
F. Zhang and Z. Wang are with Department of Electronic Engineering, Tsinghua University, Beijing 100084, China (e-mails: zf22@mails.tsinghua.edu.cn, zcwang@tsinghua.edu.cn).
G. K. Karagiannidis is with Department of Electrical and Computer Engineering, Aristotle University of Thessaloniki, Greece and also with Artificial Intelligence & Cyber Systems Research Center, Lebanese American University (LAU), Lebanon (geokarag@auth.gr).

    September 9, 2024
=====================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================

§ ABSTRACT 
^† Corresponding Author 
This paper conducts a longitudinal study over eleven months to address the limitations of prior research on the Open Ko-LLM Leaderboard, which have relied on empirical studies with restricted observation periods of only five months. By extending the analysis duration, we aim to provide a more comprehensive understanding of the progression in developing Korean large language models (LLMs). Our study is guided by three primary research questions: (1) What are the specific challenges in improving LLM performance across diverse tasks on the Open Ko-LLM Leaderboard over time? (2) How does model size impact task performance correlations across various benchmarks? (3) How have the patterns in leaderboard rankings shifted over time on the Open Ko-LLM Leaderboard?. By analyzing 1,769 models over this period, our research offers a comprehensive examination of the ongoing advancements in LLMs and the evolving nature of evaluation frameworks.




§ INTRODUCTION

The rapid advancement of large language models (LLMs) <cit.> has led to the creation of various leaderboards designed to evaluate their performance across a wide range of tasks <cit.>. Among these, the Open LLM Leaderboard <cit.> developed by Hugging Face <cit.> has achieved significant global recognition. In the context of Korean language models, the Open Ko-LLM Leaderboard <cit.> was established to specifically assess LLM performance within the Korean language environment.

While previous analyses of the Open Ko-LLM Leaderboard <cit.> have provided valuable insights into LLM performance, they have been constrained observation periods of only five months, limiting their ability to capture long-term trends. To better understand the ongoing evolution and inherent challenges in LLM development, a more comprehensive and extended analysis is required. This paper addresses this gap by conducting a detailed longitudinal study of the Open Ko-LLM Leaderboard, guided by three primary research questions:

First, we analyze the longitudinal changes in performance across five tasks monitored by the Open Ko-LLM Leaderboard. These tasks are designed to evaluate various capabilities of LLMs, including reasoning, natural language understanding, and common sense knowledge. By examining data collected over a eleven-month period, this study aims to identify which capabilities have presented the greatest challenges for LLM developers, which tasks have reached performance saturation rapidly, and which tasks continue to pose significant difficulties. This analysis will provide quantitative insights into performance trends across different tasks, thereby guiding targeted research efforts and highlighting key areas that require further advancement to push the boundaries of model development.

Second, we explore the correlations between different tasks based on model size. This aspect of the study examines how the performance across different tasks varies depending on the scale of the model. Understanding these correlations will provide insights into the interaction between model capacity and task performance, offering a deeper understanding of how scaling influences overall effectiveness across tasks.

Third, we examine the evolution of leaderboard dynamics from the initial stages to the present by focusing on three key aspects: the correlations between task performances in the early months compared to the entire eleven-month period, the temporal changes in performance based on model type, and the shifts in performance relative to model size. This comprehensive analysis offers insights into the evolving interplay among tasks and the influence of various model characteristics on LLM performance throughout different phases of development.






§ EMPIRICAL ANALYSIS




 §.§ Challenges in Enhancing Task Performance Over Time

What are the specific challenges in improving LLM performance across diverse tasks on the Open Ko-LLM Leaderboard over time?. To investigate this question, we conducted a comprehensive analysis of performance trends over a eleven-month period across all tasks on the Open Ko-LLM Leaderboard, including Ko-HellaSwag (commonsense reasoning)<cit.>, Ko-ARC (commonsense and scientific reasoning)<cit.>, Ko-MMLU (multitask language understanding and domain knowledge)<cit.>, Ko-CommonGEN V2 (commonsense generation)<cit.>, and TruthfulQA (truthfulness) <cit.>.

Figure <ref> and Table <ref> show the varying performance patterns of LLMs across these tasks over the eleven-month period. Certain tasks, such as Ko-HellaSwag and Ko-TruthfulQA, exhibit rapid improvements in performance and early saturation. Specifically, Ko-HellaSwag reached a score of 50 almost immediately and achieved 80 by week 26, while Ko-TruthfulQA showed comparable progress, reaching a score of 80 within 25 weeks. These trends indicate that current LLMs are particularly well-suited for tasks requiring straightforward commonsense reasoning and truthfulness, suggesting a relatively lower barrier to achieving performance enhancements in these domains.

Conversely, tasks such as Ko-MMLU and Ko-CommonGEN V2 show slower, more gradual improvements without clear signs of saturation, highlighting their increased complexity and the deeper understanding required from LLMs. Ko-MMLU took 13 weeks to reach a score of 50 and then stabilized around 60 after 26 weeks, indicating a limit to the current models capabilities. Similarly, Ko-CommonGEN V2, despite reaching a score of 50 relatively quickly, showed minimal progress beyond 60. These patterns highlight the significant challenges LLMs face in tasks that demand complex reasoning and specialized knowledge, suggesting these are important areas for further research.

The initial rapid gains in Ko-ARC, followed by minimal progress beyond a score of 60 after 17 weeks, indicate that while LLMs can quickly adapt to certain tasks, their progress is constrained by the need for more complex reasoning skills. This underscores the importance of developing more challenging benchmarks to better evaluate the limitations and capabilities of LLMs, especially in tasks that require more advanced forms of reasoning.

Overall, these findings emphasize the need to include a broad range of complex tasks to comprehensively assess LLM capabilities. While some tasks demonstrate rapid performance saturation, others present ongoing challenges, serving as essential benchmarks for guiding future advancements in LLM development.







 §.§ The Influence of Model Size on Task Performance Correlations

How does model size impact task performance correlations across various benchmarks?. To investigate this question, we analyze how model size affects performance improvements across different tasks, using a framework similar to previous studies <cit.>. For this analysis, models were divided into three size categories: under 3 billion parameters, 3 to 7 billion parameters, and 7 to 14 billion parameters. This categorization allows for a detailed examination of how scaling impacts task performance.

Figure <ref> illustrates distinct patterns in task performance correlations depending on model size. Smaller models (under 3 billion parameters) show low or even negative correlations between certain tasks, such as Ko-TruthfulQA and Ko-CommonGen V2, and other tasks. This suggests that smaller models struggle to improve consistently across multiple capabilities, indicating that advancements in one area do not necessarily lead to improvements in others. Consequently, these models tend to have a fragmented skill set, making them less suitable for a comprehensive evaluation of LLM performance.

In contrast, larger models demonstrate higher correlations across most tasks, suggesting that increasing model size results in a more effective integration of various capabilities. For example, models in the 7 to 14 billion parameter category exhibit stronger positive correlations across a majority of tasks, especially those requiring advanced reasoning. This trend indicates that scaling up model size not only enhances performance on individual tasks but also supports a more cohesive development of capabilities, enabling more consistent performance improvements across a wide range of tasks.

These findings highlight the importance of model size in achieving balanced performance across a range of tasks. Smaller models, with their inconsistent performance across tasks, suggest a limitation in their ability to generalize learning effectively. In contrast, the positive correlations observed in larger models imply that increasing model size fosters a more comprehensive understanding and transfer of knowledge across different domains. This insight is crucial for future LLM development, as it underscores the need to consider model size not just for boosting individual task performance, but also for promoting a more integrated and holistic enhancement of capabilities.



 §.§ Temporal Shifts in Leaderboard Ranking Patterns

How have the patterns in leaderboard rankings shifted over time on the Open Ko-LLM Leaderboard?. To investigate this question, we extended our analysis to an eleven-month period to see if the initial trends, defined as those observed during the initial five months in the previous study by <cit.>, remained consistent or if new patterns emerged over time. This longer timeframe allows us to capture shifts in model performance and ranking dynamics.









  
Task Correlations Over Time. Figure <ref> shows the correlation analysis between tasks during the initial phases of the leaderboard and over the full eleven-month period. A notable increase was observed in the correlation between Ko-Truthful QA and other tasks, especially Ko-Hellaswag. This correlation, initially very low at 0.01, rose significantly to 0.5 over time. This change suggests that as higher-performing models, particularly those with 7 billion parameters or more, were introduced, the alignment between tasks became stronger. For most other tasks, correlations remained relatively stable, reflecting their initial patterns.



  
Performance Trends by Model Type. Figure <ref> presents the performance trends over time for different model types. As noted in previous research <cit.>, improvements in instruction-tuned models typically lagged behind those of pretrained models by about one week. When a pretrained model showed a significant performance boost, instruction-tuned models followed with a similar increase roughly one week later. This pattern persisted throughout the entire period analyzed, indicating a reliance of instruction-tuned models on the advancements made by pretrained models. After April 2024, the performance of pretrained models stabilized, leading to a corresponding lack of progress in both instruction-tuned and RL-tuned models. This trend indicates the fundamental role of pretrained models in driving overall performance gains in LLMs and suggests that further improvements in pretrained models are necessary for advancing model capabilities.



  
Performance Trends Across Model Sizes. Figure <ref> shows performance variations by model size. Models in the 0-3B range exhibited minimal improvement throughout the leaderboard period, indicating inherent scalability limitations. Similarly, models in the 3-7B range initially demonstrated gains, but their progress stabilized around five months in (April 2024 to August 2024), revealing similar scalability constraints.

Larger models in the 7-14B range showed steady performance improvements during the early phase of the leaderboard, continuing throughout the entire analysis period. However, after April 2024, their performance also reached a saturation point. This stagnation is likely due to the absence of new, high-performing Korean pretrained models, a trend also evident in the analysis of different model types in Figure <ref>.

These findings emphasize that improving LLM performance largely depends on advancements in pretrained models. The leaderboard analysis indicates that, without new breakthroughs in pretrained models, further improvements are limited. This highlights the essential role of continuous innovation in pretrained models for advancing LLM performance.



§ CONCLUSION

This study provides a longitudinal analysis of the Open Ko-LLM Leaderboard, revealing key performance trends: smaller models face scalability limitations, while larger models experience saturation without advancements in pretrained models. These findings highlight the need for continuous innovation in pretrained models to enhance LLM capabilities. Additionally, the study shows that analyzing leaderboard data offers valuable insights into the evolving dynamics of LLM performance.



§ ACKNOWLEDGMENTS

We sincerely thank the National Information Society Agency (NIA), Korea Telecom (KT), and Flitto for their support. We also extend our gratitude to the Korea University NLP & AI Lab, particularly Professor Heuiseok Lim and Jaehyung Seo, for their valuable data contributions, which have greatly enhanced the robustness of the leaderboard. Our appreciation goes to the Hugging Face teams, especially Clémentine Fourrier, Lewis Tunstall, Omar Sanseviero, and Philipp Schmid, for their assistance. We would like to thank SeongHwan Cho for his contributions to the leaderboard development, and Sanghoon Kim for his contributions to the leaderboard infrastructure. Special thanks to Hyunbyung Park for his initial contributions to Ko-H5. 

We are also grateful to Professor Harksoo Kim from Konkuk University, Professor Hwanjo Yu from Pohang University of Science and Technology, Professor Sangkeun Jung from Chungnam National University, and Professor Alice Oh from KAIST for their insightful advice on the Open Ko-LLM Leaderboard. Finally, we deeply appreciate the open-source community for their invaluable feedback and contributions.

This work was supported by Institute of Information & Communications Technology Planning & Evaluation(IITP) grant funded by the Korea government(MSIT) (No. RS-2024-00338140, Development of learning and utilization technology to reflect sustainability of generative language models and up-to-dateness over time).



§ LIMITATIONS

While this study provides valuable insights into the evaluation of LLMs, several limitations should be acknowledged. First, our analysis is primarily based on data from the Open Ko-LLM Leaderboard. Although this leaderboard offers extensive coverage of various tasks, it may not fully represent the complete spectrum of challenges and scenarios relevant to LLM performance, particularly in specialized or emerging domains. 

Additionally, the focus on Korean language models may restrict the generalizability of our findings to other languages and cultural contexts. The linguistic and cultural nuances specific to Korean may not entirely translate to other languages, potentially limiting the applicability of our conclusions. 

Furthermore, our study predominantly examines the relationship between model size and performance but does not explore other factors, such as training data diversity or the impact of different fine-tuning techniques, which could also significantly influence model outcomes. Future research should aim to address these gaps by incorporating a broader range of tasks, languages, and evaluation metrics. Expanding the scope of analysis to include models trained in different linguistic and cultural settings, as well as exploring the impact of varied training methodologies, would enhance the robustness and applicability of the findings.



§ ETHICS STATEMENT

In conducting this research, we adhered to the highest ethical standards, ensuring that all data used in the evaluation was sourced responsibly and in compliance with relevant regulations. We are committed to transparency and integrity in our research practices, and we have made our methods and findings available to the community for further scrutiny and development. We also acknowledge the importance of considering the societal impacts of LLMs, particularly in ensuring that their development and deployment are aligned with ethical principles that promote fairness, inclusivity, and accountability.







§ OPEN KO-LLM LEADERBOARD

The Open Ko-LLM Leaderboard <cit.> is a pioneering platform designed to evaluate large language models (LLMs) specifically in the Korean language, addressing the limitations of predominantly English-focused benchmarks. This leaderboard mirrors the structure of the globally recognized Open LLM Leaderboard by Hugging Face <cit.>, ensuring consistency and comparability across languages. It is built on two key principles: alignment with the English leaderboard and the use of private test sets to avoid data contamination, thereby enhancing evaluation robustness.

The leaderboard employs the Ko-H5 benchmark, comprising five tasks that assess various aspects of language understanding and generation in Korean. These tasks are designed to comprehensively evaluate LLM capabilities. The first task, Ko-Hellaswag <cit.>, tests commonsense reasoning by requiring models to complete sentences contextually and logically. The second task, Ko-ARC <cit.>, adapted from the English ARC, evaluates both commonsense and scientific reasoning through multiple-choice questions. Ko-MMLU <cit.>, the third task, assesses multitask language understanding and domain knowledge across various subjects, requiring models to respond accurately to questions from different domains. The fourth task, Ko-CommonGen V2 <cit.>, focuses on commonsense generation, where models must create coherent sentences from given concepts, testing their ability to connect common knowledge meaningfully. Lastly, Ko-TruthfulQA <cit.> evaluates a model ability to provide truthful and accurate responses, crucial for assessing the factual integrity of LLMs in real-world scenarios.

Through the Ko-H5 benchmark, the Open Ko-LLM Leaderboard provides a robust framework for evaluating Korean LLMs and promotes linguistic diversity in LLM evaluation. By incorporating tasks that reflect Korean linguistic and cultural nuances, the leaderboard offers valuable insights into LLM performance beyond English, encouraging a more inclusive approach to language model evaluation.



§ ADDITIONAL ANALYSIS




Figure <ref> presents the monthly distribution of submissions across different model types on the Open Ko-LLM leaderboard. Initially, pretrained models constituted 37% of all submissions, but this proportion declined sharply over time, with no pretrained models submitted by August 2024. This trend signals a diminishing focus on pretrained models within the community, which is concerning given their foundational importance discussed in Section <ref>. Therefore, a renewed emphasis on fostering interest and engagement with pretrained models could help address this emerging gap.

On the other hand, instruction-tuned models, which started at 61%, consistently dominated the submissions, maintaining a steady presence of 70-80% each month. This trend suggests that the community perceives instruction-tuned models as highly effective or suitable for the tasks evaluated. Additionally, RL-tuned models, though initially making up only 2% of submissions, gradually increased to a peak of 29%, reflecting a growing interest in exploring reinforcement learning approaches within the leaderboard context. This variety indicates a healthy exploration of diverse model types, but also highlights areas where community focus could be broadened or rebalanced.

In addition, Table <ref> presents the monthly statistics for both the number of model submissions and the number of completed model evaluations. The Model Submissions Count refers to the total number of models submitted to the leaderboard each month. In contrast, the Model Evaluation Count represents the number of these submitted models that successfully completed the evaluation process.

The discrepancy between the Model Submissions Count and the Model Evaluation Count is due to instances where some models fail to complete the evaluation phase on the leaderboard. This failure can occur for several reasons, such as models being too large to be processed within the available computational resources or issues related to library support and compatibility. As a result, not all submitted models are evaluated successfully, highlighting potential challenges and areas for improvement in handling diverse model architectures on the leaderboard.








