Update README.md
Browse files
README.md
CHANGED
|
@@ -43,13 +43,13 @@ mistral-chat $HOME/mistral_models/mamba-codestral-7B-v0.1 --instruct --max_toke
|
|
| 43 |
|
| 44 |
## Evaluation
|
| 45 |
We evaluate Codestral Mamba, Codestral and open-weight models of the similar size on industry-standard benchmarks.
|
| 46 |
-
| Benchmarks | HumanEval | MBPP | Spider | CruxE | HumanEval C++ | HumanEvalJava |HumanEvalJS |HumanEval Bash
|
| 47 |
-
|
|
| 48 |
| CodeGemma 1.1 7B | 61.0% | 67.7% | 46.3% | 50.4% | 49.1% | 41.8% | 52.2% | 9.4% |
|
| 49 |
| CodeLlama 7B | 31.1% | 48.2% | 29.3% | 50.1% | 31.7% | 29.7% | 31.7% | 11.4% |
|
| 50 |
-
| DeepSeek v1.5 7B | 65.9% | 70.8
|
| 51 |
-
| Mamba 7B | 75.0
|
| 52 |
-
|
| 53 |
|
| 54 |
## The Mistral AI Team
|
| 55 |
|
|
|
|
| 43 |
|
| 44 |
## Evaluation
|
| 45 |
We evaluate Codestral Mamba, Codestral and open-weight models of the similar size on industry-standard benchmarks.
|
| 46 |
+
| Benchmarks | HumanEval | MBPP | Spider | CruxE | HumanEval C++ | HumanEvalJava |HumanEvalJS |HumanEval Bash |
|
| 47 |
+
| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|
| 48 |
| CodeGemma 1.1 7B | 61.0% | 67.7% | 46.3% | 50.4% | 49.1% | 41.8% | 52.2% | 9.4% |
|
| 49 |
| CodeLlama 7B | 31.1% | 48.2% | 29.3% | 50.1% | 31.7% | 29.7% | 31.7% | 11.4% |
|
| 50 |
+
| DeepSeek v1.5 7B | 65.9% | **70.8%** | **61.2%** | 55.5% | 59.0% | **62.7%** | 60.9% | **33.5%** |
|
| 51 |
+
| **Codestral Mamba (7B)** | **75.0%** | 68.5% | 58.8% | **57.8%** | **59.8%** | 57.0% | **61.5%** | 31.1% |
|
| 52 |
+
|
| 53 |
|
| 54 |
## The Mistral AI Team
|
| 55 |
|