Update README.md

README.md

@@ -1,7 +1,7 @@
 ---
 license: llama2
 datasets:
-- HiTZ/euscrawl
+- HiTZ/latxa-corpus-v1.1
 language:
 - eu
 - en
@@ -10,21 +10,111 @@ metrics:
 - f1
 - perplexity
 pipeline_tag: text-generation
+model-index:
+- name: Latxa-13b-v1.1
+  results:
+    - task:
+        type: multiple-choice
+      dataset:
+        name: xstory_cloze
+        type: XStory
+      metrics:
+        - name: Accuracy (0-shot)
+          type: Accuracy (0-shot)
+          value: 67.24
+      source:
+        name: Paper
+        url: https://paper-url.com
+    - task:
+        type: multiple-choice
+      dataset:
+        name: belebele
+        type: Belebele
+      metrics:
+        - name: Accuracy (5-shot)
+          type: Accuracy (5-shot)
+          value: 51.56
+      source:
+        name: Paper
+        url: https://paper-url.com
+    - task:
+        type: mix
+      dataset:
+        name: basque_glue
+        type: BasqueGLUE
+      metrics:
+        - name: Average scores (5-shot)
+          type: Average scores (5-shot)
+          value: 54.04
+      source:
+        name: Paper
+        url: https://paper-url.com
+    - task:
+        type: multiple_choice
+      dataset:
+        name: eus_proficiency
+        type: EusProficiency
+      metrics:
+        - name: Accuracy (5-shot)
+          type: Accuracy (5-shot)
+          value: 45.02
+      source:
+        name: Paper
+        url: https://paper-url.com
+    - task:
+        type: multiple_choice
+      dataset:
+        name: eus_reading
+        type: EusReading
+      metrics:
+        - name: Accuracy (5-shot)
+          type: Accuracy (5-shot)
+          value: 29.83
+      source:
+        name: Paper
+        url: https://paper-url.com
+    - task:
+        type: multiple_choice
+      dataset:
+        name: eus_trivia
+        type: EusTrivia
+      metrics:
+        - name: Accuracy (5-shot)
+          type: Accuracy (5-shot)
+          value: 56.44
+      source:
+        name: Paper
+        url: https://paper-url.com
+    - task:
+        type: multiple_choice
+      dataset:
+        name: eus_exams
+        type: EusExams
+      metrics:
+        - name: Accuracy (5-shot)
+          type: Accuracy (5-shot)
+          value: 43.18
+      source:
+        name: Paper
+        url: https://paper-url.com
 ---
 
 # **Model Card for Latxa 13b**
 
 ![Latxa](latxa.jpeg)
 
-Latxa is a collection of foundation models specifically tuned for Basque. Based on Meta’s LLaMA 2 model family, these models were further trained with Euscrawl, a highly curated Basque corpora ([Artetxe et al., 2022](https://aclanthology.org/2022.emnlp-main.499/)). Ranging from 7 billion to 70 billion parameters, these models are currently the biggest and best-performing LLMs built for Basque. This is the 13b repository, links to other models can be found in the [Latxa Collection](https://huggingface.co/collections/HiTZ/latxa-65a697e6838b3acc53677304).
+We introduce Latxa, a family of large language models for Basque ranging from 7 to 70 billion parameters. Latxa is based on Llama 2, which we continue pretraining on a new Basque corpus comprising 4.3M documents and 4.2B tokens. In our extensive evaluation, Latxa outperforms all previous open models we compare to by a large margin. In addition, it is competitive with GPT-4 Turbo in language proficiency and understanding, despite lagging behind in reading comprehension and knowledgeintensive tasks. Both the Latxa family of models, as well as our new pretraining corpora and evaluation datasets, are publicly available under open licenses. Our suite enables reproducible research on methods to build LLMs for low-resource languages
 
+- 📒 Blog Post: [Latxa: An Open Language Model and Evaluation Suite for Basque](https://www.hitz.eus/en/node/340)
+- 📖 Paper: [Latxa: An Open Language Model and Evaluation Suite for Basque](https://openreview.net/forum?id=mMqOvfqFS9)
+- 💻 Code: [hitz-zentroa/latxa](https://github.com/hitz-zentroa/latxa)
 
 # **Model Details**
 
 
 ## **Model Description**
 
-Latxa is a family of Large Language Models (LLM) based on Meta’s [LLaMA models](https://huggingface.co/meta-llama). Current LLMs exhibit incredible performance for high-resource languages such as English, but, in the case of Basque and other low-resource languages, their performance is close to a random guesser. These limitations widen the gap between high- and low-resource languages when it comes to digital development. We present Latxa to overcome these limitations and promote the development of LLM-based technology and research for the Basque language. Latxa models follow the same architecture as their original counterparts and were further trained in Euscrawl v1 ([Artetxe et al., 2022](https://aclanthology.org/2022.emnlp-main.499/)), a high-quality Basque corpora. 
+Latxa is a family of Large Language Models (LLM) based on Meta’s [LLaMA models](https://huggingface.co/meta-llama). Current LLMs exhibit incredible performance for high-resource languages such as English, but, in the case of Basque and other low-resource languages, their performance is close to a random guesser. These limitations widen the gap between high- and low-resource languages when it comes to digital development. We present Latxa to overcome these limitations and promote the development of LLM-based technology and research for the Basque language. Latxa models follow the same architecture as their original counterparts and were further trained in [Latxa Corpus v1.1](https://huggingface.co/datasets/HiTZ/latxa-corpus-v1.1), a high-quality Basque corpora. 
 
 The models are released in three sizes: 7B, 13B and 70B.
 
@@ -46,7 +136,7 @@ Use the code below to get started with the model.
 
 from transformers import pipeline
 
-pipe = pipeline("text-generation", model=”HiTZ/latxa-13b-v1”)
+pipe = pipeline("text-generation", model="HiTZ/latxa-13b-v1.1")
 
 text = "Euskara adimen artifizialera iritsi da!"
 
@@ -79,9 +169,9 @@ The model was not fine-tuned to follow instructions or to work as a chat assista
 
 # **Bias, Risks, and Limitations**
 
-In an effort to alleviate the potentially disturbing or harmful content, Latxa has been trained on carefully selected and processed data which comes mainly from local media, national/regional newspapers, encyclopedias and blogs (see Euscrawl below). Still, the model is based on LLaMA models and can potentially carry the same bias, risk and limitations. 
+In an effort to alleviate the potentially disturbing or harmful content, Latxa has been trained on carefully selected and processed data which comes mainly from local media, national/regional newspapers, encyclopedias and blogs (see Latxa-Corpus below). Still, the model is based on LLaMA models and can potentially carry the same bias, risk and limitations. 
 
-Please see the LLaMA’s _Ethical Considerations and Limitations _for further information.
+Please see the LLaMA’s _Ethical Considerations and Limitations_ for further information.
 
 
 # **Training Details**
@@ -89,109 +179,17 @@ Please see the LLaMA’s _Ethical Considerations and Limitations _for further in
 
 ## **Training Data**
 
-The models were trained on EusCrawl v1, a high-quality corpus for Basque comprising 1.72M documents, 288M words, totalling 2.1GiB of uncompressed text. EusCrawl was built using ad-hoc scrapers to extract text from 33 Basque websites with high-quality content, resulting in cleaner text compared to general-purpose approaches.	
+Our training corpus combines various existing datasets, as well as some new ones that we release with this work. We have prioritized quality over quantity when constructing our corpus, prioritizing high-quality data sources and applying a thorough deduplication and filtering process. In total, a 4.17B tokens corpus is used to train the model.
 
-See more details in the [EusCrawl](https://huggingface.co/datasets/HiTZ/euscrawl) dataset card. 
+See more details in the [Latxa Corpus](https://huggingface.co/datasets/HiTZ/latxa-corpus-v1.1) dataset card. 
 
-Additionally, 100K documents of English data randomly selected from the [Pile](https://huggingface.co/datasets/EleutherAI/pile) dataset were also included to avoid catastrophic forgetting.
+Additionally, 500K documents of English data randomly selected from the [Pile](https://huggingface.co/datasets/EleutherAI/pile) dataset were also included to avoid catastrophic forgetting.
 
 
 ## **Training Procedure**
 
-The models were trained using the GPT-Neox library on the HPC CINECA computing cluster. All the models were approximately trained with an effective batch size of 2M tokens for 1000 to 2000 steps. 
-
-
-<table>
-  <tr>
-   <td>Model
-   </td>
-   <td>Steps
-   </td>
-   <td>Sequence length
-   </td>
-   <td>Effective Batch size
-   </td>
-   <td>Total tokens
-   </td>
-   <td>GPU hours
-   </td>
-  </tr>
-  <tr>
-   <td>Latxa 7B
-   </td>
-   <td><p style="text-align: right">
-2000</p>
-
-   </td>
-   <td><p style="text-align: right">
-4096</p>
-
-   </td>
-   <td><p style="text-align: right">
-2M tokens/step</p>
-
-   </td>
-   <td><p style="text-align: right">
-4B</p>
-
-   </td>
-   <td><p style="text-align: right">
-359.2h</p>
-
-   </td>
-  </tr>
-  <tr>
-   <td>Latxa 13B
-   </td>
-   <td><p style="text-align: right">
-1000</p>
-
-   </td>
-   <td><p style="text-align: right">
-4096</p>
-
-   </td>
-   <td><p style="text-align: right">
-2M tokens/step</p>
-
-   </td>
-   <td><p style="text-align: right">
-2B</p>
-
-   </td>
-   <td><p style="text-align: right">
-468.8h</p>
-
-   </td>
-  </tr>
-  <tr>
-   <td>Latxa 70B
-   </td>
-   <td><p style="text-align: right">
-1680</p>
-
-   </td>
-   <td><p style="text-align: right">
-4096</p>
-
-   </td>
-   <td><p style="text-align: right">
-2M tokens/step</p>
-
-   </td>
-   <td><p style="text-align: right">
-3.4B</p>
-
-   </td>
-   <td><p style="text-align: right">
-*6475.52h</p>
-
-   </td>
-  </tr>
-</table>
-
-
-* indicates the time for the entire training process (2000 steps), however the weights of the step 1680 are shared as it is the best checkpoint according to validation loss.
+The training of Latxa was conducted using the [GPT-Neox](https://github.com/EleutherAI/gpt-neox) library. As infrastructure, we leveraged the CINECA HPC Leonardo computing cluster located in Italy, which is powered by 3456 nodes each containing 4x custom A100 64Gb GPUs. The models were trained for 10k steps with a sequence length of 4096 tokens and an effective batch size of 2M tokens, resulting in a total of 20B tokens (around 4 epochs). We used a cosine learning rate schedule, with a warm-up of 500 steps and decaying down to 3\% of the peak learning rate. We set up the peak learning rate to be 1e-4. All other hyperparameters follow ([Touvron et al., 2023](https://arxiv.org/abs/2307.09288)).
+
 
 
 # **Evaluation**
@@ -219,13 +217,19 @@ We evaluated the models on zero-shot and few-shot settings on generative, multip
         * **EpecKorrefBin**: Correference detection task similar to WSC.
         * **QNLIeu**: Q&A NLI built from the Basque Wikipedia.
         * **WiCeu**: Basque Word-in-Context task.
-
+* **EusProficiency** ([Etxaniz et al., 2024]()): EusProficiency comprises 5,169 exercises on different topics from past EGA exams, the official C1-level certificate of proficiency in Basque.
+    * Data card: [https://huggingface.co/datasets/HiTZ/EusProficiency](https://huggingface.co/datasets/HiTZ/EusProficiency)
+* **EusReading** ([Etxaniz et al., 2024]()): EusReading consists of 352 reading comprehension exercises (_irakurmena_) sourced from the same set of past EGA exams.
+    * Data card: [https://huggingface.co/datasets/HiTZ/EusReading](https://huggingface.co/datasets/HiTZ/EusReading)
+* **EusTrivia** ([Etxaniz et al., 2024]()): EusTrivia consists of 1,715 trivia questions from multiple online sources. 56.3\% of the questions are elementary level (grades 3-6), while the rest are considered challenging.
+    * Data card: [https://huggingface.co/datasets/HiTZ/EusTrivia](https://huggingface.co/datasets/HiTZ/EusTrivia)
+* **EusExams** ([Etxaniz et al., 2024]()): EusExams is a collection of tests designed to prepare individuals for Public Service examinations conducted by several Basque institutions, including the public health system Osakidetza, the Basque Government, the City Councils of Bilbao and Gasteiz, and the University of the Basque Country (UPV/EHU).
+    * Data card: [https://huggingface.co/datasets/HiTZ/EusExams](https://huggingface.co/datasets/HiTZ/EusExams)
 
 ### **Metrics**
 
+For most of the task we used Accuracy, as they are framed as Multiple Choice questions. For the rest, particularly task from BasqueGLUE benchmark, we have used the following:
 
-
-* **Accuracy**: Belebele, X-StoryCloze, EpecKorrefBin, QNLI-eu, and, WiC-eu
 * **Micro F1**: BEC2016-eu and BHTCv2
 * **Macro F1**: VaxxStance (favor & against)
 
@@ -235,244 +239,66 @@ We evaluated the models on zero-shot and few-shot settings on generative, multip
 The model was evaluated using the LM Evaluation harness library from Eleuther AI. In order to reproduce our results please refer to our [fork](https://github.com/naiarapm/lm-evaluation-harness/tree/basqueglue) that includes the implementation for the mentioned datasets.
 
 
-<table>
-  <tr>
-   <td><strong>Model</strong>
-   </td>
-   <td><strong>Belebele</strong>
-   </td>
-   <td><strong>X-StoryCloze</strong>
-   </td>
-   <td><strong>BEC</strong>
-   </td>
-   <td><strong>Vaxx</strong>
-   </td>
-   <td><strong>BHTC</strong>
-   </td>
-   <td><strong>coref</strong>
-   </td>
-   <td><strong>QNLI</strong>
-   </td>
-   <td><strong>WiC</strong>
-   </td>
-   <td><strong>Average</strong>
-   </td>
-  </tr>
-  <tr>
-   <td>Random
-   </td>
-   <td>25.00
-   </td>
-   <td>50.00
-   </td>
-   <td>33.33
-   </td>
-   <td>33.33
-   </td>
-   <td>8.33
-   </td>
-   <td>50.00
-   </td>
-   <td>50.00
-   </td>
-   <td>50.00
-   </td>
-   <td>37.50
-   </td>
-  </tr>
-  <tr>
-   <td>LLaMA 2 7B
-   </td>
-   <td>26.22
-   </td>
-   <td>50.43
-   </td>
-   <td>41.63
-   </td>
-   <td>18.60
-   </td>
-   <td>20.06
-   </td>
-   <td>50.94
-   </td>
-   <td>48.32
-   </td>
-   <td>49.64
-   </td>
-   <td>38.23
-   </td>
-  </tr>
-  <tr>
-   <td>LLaMA 2 13B
-   </td>
-   <td>32.00
-   </td>
-   <td>50.63
-   </td>
-   <td>41.09
-   </td>
-   <td>18.25
-   </td>
-   <td>27.35
-   </td>
-   <td>49.23
-   </td>
-   <td>48.74
-   </td>
-   <td>49.21
-   </td>
-   <td>39.56
-   </td>
-  </tr>
-  <tr>
-   <td>LLaMA 2 70B
-   </td>
-   <td>33.56
-   </td>
-   <td>51.62
-   </td>
-   <td>47.47
-   </td>
-   <td>21.01
-   </td>
-   <td>31.01
-   </td>
-   <td>52.98
-   </td>
-   <td>51.26
-   </td>
-   <td>51.57
-   </td>
-   <td>42.56
-   </td>
-  </tr>
-  <tr>
-   <td>BLOOM 7B
-   </td>
-   <td>27.00
-   </td>
-   <td>57.18
-   </td>
-   <td>37.94
-   </td>
-   <td>20.72
-   </td>
-   <td>39.10
-   </td>
-   <td>48.21
-   </td>
-   <td>47.48
-   </td>
-   <td>47.57
-   </td>
-   <td>40.65
-   </td>
-  </tr>
-  <tr>
-   <td>XGLM 7B
-   </td>
-   <td>23.88
-   </td>
-   <td>57.71
-   </td>
-   <td>39.94
-   </td>
-   <td>21.58
-   </td>
-   <td>36.73
-   </td>
-   <td>50.94
-   </td>
-   <td>50.42
-   </td>
-   <td>49.21
-   </td>
-   <td>41.30
-   </td>
-  </tr>
-  <tr>
-   <td><strong>Latxa 7B</strong>
-   </td>
-   <td>35.67
-   </td>
-   <td>63.13
-   </td>
-   <td>55.61
-   </td>
-   <td>45.93
-   </td>
-   <td>44.44
-   </td>
-   <td>50.43
-   </td>
-   <td>55.04
-   </td>
-   <td>50.14
-   </td>
-   <td>50.05
-   </td>
-  </tr>
-  <tr>
-   <td><strong>Latxa 13B</strong>
-   </td>
-   <td>53.56
-   </td>
-   <td>65.85
-   </td>
-   <td>53.23
-   </td>
-   <td>48.66
-   </td>
-   <td><strong>53.61</strong>
-   </td>
-   <td>62.52
-   </td>
-   <td>57.14
-   </td>
-   <td>54.21
-   </td>
-   <td>56.10
-   </td>
-  </tr>
-  <tr>
-   <td><strong>Latxa 70B</strong>
-   </td>
-   <td><strong>71.78</strong>
-   </td>
-   <td><strong>67.57</strong>
-   </td>
-   <td><strong>63.52</strong>
-   </td>
-   <td><strong>48.95</strong>
-   </td>
-   <td>49.51
-   </td>
-   <td><strong>79.90</strong>
-   </td>
-   <td><strong>58.82</strong>
-   </td>
-   <td><strong>55.50</strong>
-   </td>
-   <td><strong>61.94</strong>
-   </td>
-  </tr>
-</table>
 
+| Model            | Size | XStory | Belebele | BasGLUE | EusProf | EusRead | EusTrivia | EusExams | Avg   |
+|------------------|------|--------|----------|---------|---------|---------|-----------|----------|-------|
+| **Random**           |      | 50.00  | 25.00    | 37.50   | 25.00   | 25.83   | 26.55     | 25.00    | 30.70 |
+|
+| GPT 3.5 Turbo    | n/a  | --     | 57.33    | 48.62   | 31.24   | 36.65   | 46.71     | 42.42    | --    |
+| GPT 4 Turbo      | n/a  | --     | **90.67**| **62.90**| **56.70**| **75.85**| **73.12** | **70.22**| --    |
+|
+| XGLM             | 7B   | 57.71  | 23.88    | 41.47   | 22.96   | 24.43   | 26.53     | 24.59    | 32.51 |
+| BLOOM            | 7B   | 57.18  | 27.00    | 40.17   | 25.34   | 28.41   | 27.17     | 25.07    | 33.86 |
+| Mistral          | 7B   | 51.09  | **38.89**| 39.22   | 25.01   | 29.26   | 34.58     | 32.15    | 35.94 |
+| Llama 2          | 7B   | 50.43  | 26.22    | 38.20   | 24.09   | 27.27   | 29.50     | 28.84    | 32.51 |
+| **Latxa v1**     | 7B   | 63.13 | 35.67 | 50.26 | 28.19 | 27.27 | 40.17 | 34.18 | 39.84 |
+| **Latxa v1.1**   | 7B   | **65.72**| 36.89    | **51.78**| **32.44**| **30.40**| **44.37** | **34.20**| **42.26** |
+|
+| mGPT             | 13B  | 55.39  | 25.00    | 37.56   | 25.00   | 24.15   | 27.17     | 25.73    | 32.14 |
+| Llama 2          | 13B  | 50.63  | 32.00    | 38.98   | 25.90   | 28.98   | 33.53     | 29.66    | 34.36 |
+| **Latxa v1**     | 13B  | 65.85 | **53.56** | **54.49** | 41.19    | **40.06**| 51.14     | 42.92    | **49.95** |
+| **Latxa v1.1**   | 13B  | **67.24**| 51.56  | 54.04     | **45.02**| 29.83    | **56.44** | **43.18**| 49.62 |
+|
+| Mixtral          | 8x7B | 52.55  | 50.44    | 45.00   | 26.43   | 37.50   | 42.51     | 39.87    | 41.97 |
+| Yi               | 34B  | 52.22  | 54.56    | 43.90   | 27.30   | 34.66   | 42.57     | 39.68    | 42.05 |
+| Llama 2          | 70B  | 51.62  | 33.56    | 42.55   | 24.16   | 27.84   | 38.43     | 33.08    | 35.47 |
+| **Latxa v1**     | 70B  | 67.57  | **71.78**    | 59.37   | 48.19   | 49.72   | 57.84     | 51.68    | 58.02 |
+| **Latxa v1.1**   | 70B  | **69.76**| 64.89| **61.66**| **60.61**| **53.69**| **61.52** | **54.48**| **60.94** |
 
 
 # **Environmental Impact**
 
 Carbon emissions are estimated using the[ Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in[ Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
 
+| Model      | Size | Time (GPU Hours) | Carbon Emitted (kg CO2 eq)  |
+|------------|------|-------------------|----------------------------|
+| Latxa v1.1 | 7B   | 1,895.4h          | 247.69kg                   |
+| Latxa v1.1 | 13B  | 2,518.0h          | 329.06kg                   |
+| Latxa v1.1 | 70B  | 30,266.0h         | 3,955.17kg                 |
+| Total      |  -   | 34,679.4h         | 4,531.92kg                 |
 
 
 * **Hardware Type:** HPC Cluster, 4x A100 64Gb nodes
-* **Hours used:** 359.2h + 468.8h + 6475.52h = 7303.52h
+* **Hours used:** 34,679.4h
 * **Compute cluster:** CINECA HPC
 * **Compute Region:** Italy
-* **Carbon Emitted:** 673.75kg CO<sub>2</sub> eq
+* **Carbon Emitted:** 4,531.92kg CO<sub>2</sub> eq
 
 
 # **Acknowledgements**
 
-This work has been partially supported by the Basque Government (IKER-GAITU project). The models were trained on the Leonardo supercomputer at CINECA under the EuroHPC Joint Undertaking, project EHPC-EXT-2023E01-013.
+This work has been partially supported by the Basque Government (IKER-GAITU project). The models were trained on the Leonardo supercomputer at CINECA under the EuroHPC Joint Undertaking, project EHPC-EXT-2023E01-013.
+
+# **Citation**
+To cite our work, please use:
+```bibtex
+@misc{etxaniz2024latxa,
+      title={{L}atxa: An Open Language Model and Evaluation Suite for {B}asque}, 
+      author={Julen Etxaniz and Oscar Sainz and Naiara Perez and Itziar Aldabe and German Rigau and Eneko Agirre and Aitor Ormazabal and Mikel Artetxe and Aitor Soroa},
+      year={2024},
+      eprint={},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL}
+}
+
+```