|
--- |
|
base_model: migtissera/Tess-70B-v1.6 |
|
inference: false |
|
license: llama2 |
|
metrics: |
|
- accuracy |
|
--- |
|
|
|
This is a BF16 and pruned version of [migtissera/Tess-70B-v1.6](https://huggingface.co/migtissera/Tess-70B-v1.6) . |
|
|
|
[migtissera/Tess-70B-v1.6](https://huggingface.co/migtissera/Tess-70B-v1.6) has 69 billion params and Covasna-0.1 has 41.6 billion (~60.3% param size) |
|
|
|
# Steps to replicate: |
|
|
|
Use [laserQlora.ipynb](https://github.com/cognitivecomputations/laserRMT/blob/main/laserQlora.ipynb) from [cognitivecomputations/laserRMT](https://github.com/cognitivecomputations/laserRMT) to determine which layers should be eliminated. |
|
|
|
Adapt the script for `migtissera/Tess-70B-v1.6` by replacing `model_name = "mistralai/Mistral-7B-v0.1"` with `model_name = "migtissera/Tess-70B-v1.6"` and `layer_numbers = list(range(31, -1, -1))` with `layer_numbers = list(range(79, -1, -1))`, [79 being the last recurrent layer index Tess-70B-v1.6 has](https://huggingface.co/migtissera/Tess-70B-v1.6?show_tensors=true). |
|
|
|
Then look for the layer indexes where self_attn.v_proj snr is Infinity and eliminate those layers using [mergekit](https://github.com/arcee-ai/mergekit). |
|
|
|
Here is the mergekit config: |
|
|
|
```yml |
|
slices: |
|
- sources: |
|
- model: "migtissera/Tess-70B-v1.6" |
|
layer_range: [0, 7] |
|
- sources: |
|
- model: "migtissera/Tess-70B-v1.6" |
|
layer_range: [8, 9] |
|
- sources: |
|
- model: "migtissera/Tess-70B-v1.6" |
|
layer_range: [12, 29] |
|
- sources: |
|
- model: "migtissera/Tess-70B-v1.6" |
|
layer_range: [31, 32] |
|
- sources: |
|
- model: "migtissera/Tess-70B-v1.6" |
|
layer_range: [33, 45] |
|
- sources: |
|
- model: "migtissera/Tess-70B-v1.6" |
|
layer_range: [50, 52] |
|
- sources: |
|
- model: "migtissera/Tess-70B-v1.6" |
|
layer_range: [60, 61] |
|
- sources: |
|
- model: "migtissera/Tess-70B-v1.6" |
|
layer_range: [67, 68] |
|
- sources: |
|
- model: "migtissera/Tess-70B-v1.6" |
|
layer_range: [74, 80] |
|
merge_method: passthrough |
|
dtype: bfloat16 |
|
``` |
|
|
|
GGUF: |
|
[Covasna-0.1-GGUF](https://huggingface.co/mradermacher/Covasna-0.1-GGUF) |
