---
license: apache-2.0
language:
- en
tags:
- llm
- fine-tune
- qlora
- llama
- bitcoin
- finance
- cryptocurrency
pipeline_tag: text-generation
base_model: meta-llama/Llama-3.2-3B-Instruct
datasets:
- tahamajs/bitcoin-llm-finetuning-dataset
---

-----

### **Overview**

This model, `tahamajs/llama-3.2-3b-instruct-bitcoin-analyst_best`, is a specialized version of the **Llama-3.2-3B-Instruct** large language model. It has been fine-tuned for the domain of **Bitcoin and cryptocurrency analysis**. The fine-tuning process, which used **QLoRA** on the `tahamajs/bitcoin-llm-finetuning-dataset` dataset, was designed to train the model to act as a virtual Bitcoin analyst.

-----

### **Usage**

This model uses a PEFT (Parameter-Efficient Fine-Tuning) adapter. To use it, you must first load the base model and then attach the fine-tuned adapter. The adapter itself is very small, making the overall model efficient to load and use.

```python
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

# Define the model IDs
base_model_id = "meta-llama/Llama-3.2-3B-Instruct"
peft_model_id = "tahamajs/llama-3.2-3b-instruct-bitcoin-analyst_best"

# Load the tokenizer
tokenizer = AutoTokenizer.from_pretrained(base_model_id)

# Load the base model with 4-bit quantization
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    quantization_config=bnb_config,
    device_map="auto"
)

# Load the fine-tuned PEFT adapter on top of the base model
model = PeftModel.from_pretrained(base_model, peft_model_id)

# Example inference
prompt = "What are the key technical differences between Bitcoin and Ethereum?"
messages = [
    {"role": "user", "content": prompt}
]
input_ids = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt"
).to(model.device)

outputs = model.generate(input_ids=input_ids, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

-----

### **Training Details** ⚙️

This model was fine-tuned using Supervised Fine-Tuning (SFT) with the `trl.SFTTrainer`. A key aspect of the training was the use of a **custom data collator**, which ensures that the model's loss is calculated only on the assistant's response, not on the user's prompt or the special tokens used to format the conversation. This technique helps the model learn to generate better responses without overfitting to the input.

#### **Hyperparameters**

The following hyperparameters were used for fine-tuning:

| Hyperparameter               | Value           |
| :--------------------------- | :-------------- |
| `num_train_epochs`           | 1               |
| `per_device_train_batch_size`| 1               |
| `gradient_accumulation_steps`| 4               |
| `learning_rate`              | 2e-4            |
| `optim`                      | `paged_adamw_32bit` |
| `r` (LoRA rank)              | 32              |
| `lora_alpha`                 | 32              |

-----

### **Limitations and Biases** ⚠️

  * **Domain Specificity:** The model's expertise is concentrated on Bitcoin and cryptocurrency. Its performance on general knowledge tasks may be limited.
  * **Data Cutoff:** The model's knowledge is static and based on the training data. It will not be aware of recent events or market changes.
  * **Potential Biases:** Responses may reflect biases present in the training data.

-----

### **License** 📜

This model is licensed under the **Apache 2.0 license**, inherited from its base model.