tags: | |
- merge | |
- mergekit | |
- lazymergekit | |
# TB0-8B-sce | |
TB0-8B-sce is a merge of the following models using [LazyMergekit](https://colab.research.google.com/drive/1obulZ1ROXHjYLn6PPZJwRR6GzgQogxxb?usp=sharing): | |
## 🧩 Configuration | |
```yaml | |
models: | |
# Pivot model | |
- model: bunnycore/HyperLlama-3.1-8B | |
# Target models | |
- model: Quazim0t0/Katana-8b-sce | |
merge_method: sce | |
base_model: bunnycore/HyperLlama-3.1-8B | |
parameters: | |
select_topk: 1.5 | |
dtype: bfloat16 | |
``` | |
## 💻 Usage | |
```python | |
!pip install -qU transformers accelerate | |
from transformers import AutoTokenizer | |
import transformers | |
import torch | |
model = "Quazim0t0/TB0-8B-sce" | |
messages = [{"role": "user", "content": "What is a large language model?"}] | |
tokenizer = AutoTokenizer.from_pretrained(model) | |
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) | |
pipeline = transformers.pipeline( | |
"text-generation", | |
model=model, | |
torch_dtype=torch.float16, | |
device_map="auto", | |
) | |
outputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95) | |
print(outputs[0]["generated_text"]) | |
``` |