Update README.md
Browse files
README.md
CHANGED
|
@@ -6,54 +6,36 @@ language:
|
|
| 6 |
- en
|
| 7 |
---
|
| 8 |
|
| 9 |
-
# <b>MgGPT</b>
|
| 10 |
|
| 11 |
-
MgGPT is a fully fine-tuned generative text model collection
|
| 12 |
-
This is the repository for the version
|
| 13 |
|
| 14 |
---
|
| 15 |
## Model Details
|
| 16 |
-
We have released the MgGPT family of large language models, which is a collection of fully fine-tuned generative text models,
|
|
|
|
|
|
|
| 17 |
## Variations
|
| 18 |
-
MgGPT families come in a range of parameter sizes ——
|
|
|
|
|
|
|
| 19 |
## Input
|
| 20 |
Models input text only.
|
| 21 |
## Output
|
| 22 |
Models output text only.
|
| 23 |
## Model Evaluation Results
|
| 24 |
|
| 25 |
-
|
| 26 |
-
|
| 27 |
-
|
|
| 28 |
-
|
|
| 29 |
-
|
|
| 30 |
-
|
|
| 31 |
-
|
|
| 32 |
-
|
|
| 33 |
-
|
|
| 34 |
-
|
|
| 35 |
-
| Qwen1.5-72B | 60.24 | 61.23 | 54.41 | 82.98 | <u>81.20</u> | 75.93 | 76.79 | 70.40 |
|
| 36 |
-
| **MgGPT-32B** | 58.71 | 65.67 | 52.74 | 82.66 | 81.04 | 80.46 | 71.69 | 70.42 |
|
| 37 |
-
| Llama3-70B | <u>65.16</u> | 65.67 | 54.78 | 83.48 | **82.92** | 74.84 | 77.30 | 72.02 |
|
| 38 |
-
| **MgGPT-70B** | **65.19** | <u>67.71</u> | <u>56.19</u> | **84.79** | 80.93 | <u>80.93</u> | <u>80.93</u> | <u>73.81</u> |
|
| 39 |
-
| GPT-4 | 65.06 | **72.50** | **57.76** | <u>84.06</u> | 79.43 | **90.04** | **85.67** | **76.36** |
|
| 40 |
-
|
| 41 |
-
Benchmarks for English and Chinese are conducted using the [OpenCompass](https://github.com/open-compass/OpenCompass/) framework.
|
| 42 |
-
|
| 43 |
-
| | MMLU | RACE | English Avg. | CMMLU | CEval | Chinese Avg. | Avg. |
|
| 44 |
-
| ----------------- | :------------: | :------------: | :------------: | :-------------: | :-------------: | :------------: | :------------: |
|
| 45 |
-
| Jais-30B-v3 | 42.53 | 30.96 | 36.75 | 25.26 | 22.17 | 23.72 |30.23|
|
| 46 |
-
| **MgGPT-8B** | 65.48 | 60.49 | 62.99 | 53.44 | 50.37 | 51.91 |57.45|
|
| 47 |
-
| Llama3-8B | 66.57 | 65.92 | 66.25 | 50.70 | 49.78 | 50.24 |58.24|
|
| 48 |
-
| ChatGPT 3.5 Turbo | 69.03 | 83.00 | 76.02 | 53.90 | 52.50 | 53.20 |64.60|
|
| 49 |
-
| Qwen1.5-7B | 62.15 | 82.19 | 72.17 | 71.79 | 73.61 | 72.70 |72.44|
|
| 50 |
-
| **MgGPT-70B** | 76.71 | 80.48 | 78.60 | 68.97 | 66.87 | 67.92 |73.26|
|
| 51 |
-
| GPT-4 | **83.00** | **91.00** | **87.00** | 71.00 | 69.90 | 70.45 |78.73|
|
| 52 |
-
| Llama3-70B | <u>79.34</u> | 84.76 | <u>82.05</u> | 68.29 | 67.21 | 67.75 |74.90|
|
| 53 |
-
| Qwen1.5-32B | 75.10 | 83.29 | 79.20 | **83.12** | <u>82.68</u> | <u>82.90</u> |81.05|
|
| 54 |
-
| **MgGPT-32B** | 74.52 | <u>88.68</u> | 81.60 | 81.36 | 82.41 | 81.89 |<u>81.74</u>|
|
| 55 |
-
| Qwen1.5-72B | 75.78 | 88.23 | 82.01 | <u>83.11</u> | **83.04** | **83.08** |**82.54**|
|
| 56 |
-
|
| 57 |
|
| 58 |
|
| 59 |
## Samples
|
|
|
|
| 6 |
- en
|
| 7 |
---
|
| 8 |
|
| 9 |
+
# <b>MgGPT-8B</b>
|
| 10 |
|
| 11 |
+
MgGPT-8B is a fully fine-tuned generative text model collection based on LlaMA3, particularly in the
|
| 12 |
+
Arabic language domain. This is the repository for the version of 8B pre-trained model.
|
| 13 |
|
| 14 |
---
|
| 15 |
## Model Details
|
| 16 |
+
We have released the MgGPT family of large language models, which is a collection of fully fine-tuned generative text models based on LlaMA2(MgGPT-7B, MgGPT-13B), LlaMA3(MgGPT-8B, MgGPT-70B), Qwen2(MgGPT-32B). Our models include two main categories: MgGPT and MgGPT-chat. MgGPT-chat is an optimized version specifically designed for dialogue applications. It is worth mentioning that our models have demonstrated superior performance compared to all currently available open-source Arabic dialogue models in multiple benchmark tests. Furthermore, in our human evaluations, our models have shown comparable satisfaction levels to some closed-source models, such as ChatGPT, in the Arabic language.
|
| 17 |
+
<!-- ## Model Developers
|
| 18 |
+
We are from the King Abdullah University of Science and Technology (KAUST), the Chinese University of Hong Kong, Shenzhen (CUHKSZ), the Shenzhen Research Institute of Big Data (SRIBD), and King AbdulAziz University (KAU). -->
|
| 19 |
## Variations
|
| 20 |
+
MgGPT families come in a range of parameter sizes —— 7B, 8B, 13B, 32B and 70B, each size of model has a base category and a -chat category.
|
| 21 |
+
<!-- ## Paper -->
|
| 22 |
+
<!-- The paper can be accessed at [link](https://huggingface.co/FreedomIntelligence/AceGPT-v1.5-13B-Chat/blob/main/Second_Language_(Arabic)_Acquisition_of_LLMs_via_Progressive_Vocabulary_Expansion.pdf). -->
|
| 23 |
## Input
|
| 24 |
Models input text only.
|
| 25 |
## Output
|
| 26 |
Models output text only.
|
| 27 |
## Model Evaluation Results
|
| 28 |
|
| 29 |
+
| Model | Avg. | [ArabicMMLU]((https://github.com/mbzuai-nlp/ArabicMMLU)) | [ArabicMMLU]((https://github.com/mbzuai-nlp/ArabicMMLU)) | ARC | EXAMs | ACVA (clean) | ACVA (all) |
|
| 30 |
+
|---------------|--------|----------------|-----------------------|-------|-------|--------------|------------|
|
| 31 |
+
| MgGPT-7B | 45.19 | 34.03 | 37.00 | 17.49 | 37.28 | 72.69 | 72.67 |
|
| 32 |
+
| **MgGPT-8B** | 58.94 | 48.41 | 50.17 | 49.91 | 46.15 | 80.14 | 78.84 |
|
| 33 |
+
| MgGPT-13B | 52.11 | 40.95 | 47.60 | 31.57 | 35.10 | 79.45 | 78.01 |
|
| 34 |
+
| MgGPT-32B | 68.75 | 58.71 | 65.67 | 71.69 | 52.74 | 82.66 | 81.04 |
|
| 35 |
+
| MgGPT-70B | 72.62 | 65.19 | 67.71 | 80.93 | 56.19 | 84.79 | 80.93 |
|
| 36 |
+
| Jais-30B-v3 | 57.02 | 43.42 | 44.47 | 45.56 | 45.70 | 83.39 | 79.51 |
|
| 37 |
+
| GPT-3.5 | 60.71 | 49.07 | 57.70 | 60.24 | 45.93 | 74.45 | 76.88 |
|
| 38 |
+
| GPT-4 | 74.08 | 65.06 | 72.50 | 85.67 | 57.76 | 84.06 | 79.43 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 39 |
|
| 40 |
|
| 41 |
## Samples
|