File size: 1,944 Bytes
85bc296 39140e1 85bc296 8f9a7d3 f1a1a2d 8f9a7d3 c33cb3b 85bc296 39140e1 85bc296 39140e1 4bc9c61 39140e1 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 |
---
library_name: transformers
tags:
- mergekit
- merge
pipeline_tag: text-generation
---
<a href="https://ibb.co/k9z9yyz"><img src="https://i.ibb.co/61G1ZZG/DALL-E-2024-08-08-05-39-49-Create-a-visually-captivating-image-for-a-model-card-representing-a-prune.webp" alt="DALL-E-2024-08-08-05-39-49-Create-a-visually-captivating-image-for-a-model-card-representing-a-prune" border="0"></a><br /><a target='_blank' href='https://usefulwebtool.com/fr/convertir-minuscules-majuscules'></a><br />
# Na0s/Llama-3.1-8b-Pruned-4-Layers
This is a merge of meta-llama/Meta-Llama-3.1-8B created using [mergekit](https://github.com/cg123/mergekit), with respect to the paper ["The Unreasonable Ineffectiveness of the Deeper Layers"](https://arxiv.org/pdf/2403.17887)
## Merge Details
### Merge Method
This model was merged using the passthrough merge method.
### Models Merged
The following models were included in the merge:
* [meta-llama/Meta-Llama-3.1-8B](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B)
### Configuration
The following YAML configuration was used to produce this model:
```yaml
dtype: bfloat16
merge_method: passthrough
slices:
- sources:
- layer_range: [0, 23]
model: meta-llama/Meta-Llama-3.1-8B
- sources:
- layer_range: [28, 32]
model: meta-llama/Meta-Llama-3.1-8B
```
## Evaluation
MMLU Pro 0-shot: 0.2642
#### Evaluation Data
<!-- This should link to a Dataset Card if possible. -->
[TIGER-AI-Lab/MMLU-Pro]
## Environmental Impact
<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app --> |