---
license: llama3
language:
- en
library_name: transformers
tags:
- llama-3
- llama-3.2
- bitcoin
- finance
- instruction-following
- fine-tuning
- merged
base_model: meta-llama/Llama-3.2-3B-Instruct
datasets:
- tahamajs/bitcoin-llm-finetuning-dataset
---

# Llama-3.2-3B Instruct - Advanced Bitcoin Analyst

This repository contains a highly specialized version of `meta-llama/Llama-3.2-3B-Instruct`, expertly fine-tuned to function as a **Bitcoin and cryptocurrency market analyst**. This model is the result of a "continuation training" process, where an already specialized model was further refined on a targeted dataset.

This model excels at understanding and responding to complex instructions related to blockchain technology, financial markets, and technical/fundamental analysis of cryptocurrencies.

## 🧠 Training Procedure

The final model was created through a sophisticated multi-stage process designed to build upon and deepen existing knowledge.

### Stage 1: Initial Specialization (Adapter Merge)
The process began with the base `meta-llama/Llama-3.2-3B-Instruct` model. This base was then merged with a previously fine-tuned, high-performing LoRA adapter to create an initial specialized model.
- **Initial Adapter:** `tahamajs/llama-3.2-3b-instruct-bitcoin-analyst-perfect`

### Stage 2: Continued Fine-Tuning (New LoRA)
A new LoRA adapter was then trained on top of the already-merged model from Stage 1. This continuation training allowed the model to further refine its expertise using a specific dataset, improving its nuance and instruction-following on relevant topics.
- **Dataset:** `tahamajs/bitcoin-llm-finetuning-dataset`

### Stage 3: Final Merge
The final step was to merge the newly trained adapter from Stage 2 into the model. This repository hosts this **fully merged, standalone model**, which contains the cumulative knowledge of the base model, the first specialized adapter, and the second round of continuation training.

---

## 📊 Training Details

### Hyperparameters
The second stage of LoRA fine-tuning was performed with the following key hyperparameters:

| Parameter | Value |
| :--- | :--- |
| `learning_rate` | 1e-4 |
| `num_train_epochs` | 1 |
| `lora_r` | 16 |
| `lora_alpha` | 32 |
| `optimizer` | paged_adamw_32bit |
| `precision` | bf16 |

### Training Loss
The training loss shows a clear downward trend, indicating that the model was successfully learning from the dataset. The process started with a loss of ~2.18 and converged to a loss in the ~1.4-1.6 range, demonstrating effective knowledge acquisition. The fluctuations are normal during training and reflect the varying difficulty of the data in each batch.

![Training Loss](https://googleusercontent.com/file_content/0)

---

## 🚀 How to Use

This is a fully merged model and can be used directly with the `transformers` library. For best results, use the Llama 3 chat template to format your prompts.

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

# Use the ID of the repository where this model is hosted
model_id = "tahamajs/llama-3.2-3b-instruct-bitcoin-analyst-perfect_v2"

# Load the tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

# Use the Llama 3 chat template for instruction-following
messages = [
    {"role": "user", "content": "Analyze the current sentiment around Bitcoin based on the concept of the Fear & Greed Index. What does a high 'Greed' value typically imply for the short-term market?"},
]

# Apply the chat template and tokenize
input_ids = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt"
).to(model.device)

# Generate a response
outputs = model.generate(
    input_ids,
    max_new_tokens=512,
    do_sample=True,
    temperature=0.6,
    top_p=0.9,
)

# Decode and print the output
response = outputs[0][input_ids.shape[-1]:]
print(tokenizer.decode(response, skip_special_tokens=True))
````

## Disclaimer

This model is provided for informational and educational purposes only. It is **not financial advice**. The outputs are generated by an AI and may contain errors or inaccuracies. Always perform your own due diligence and consult with a qualified financial professional before making any investment decisions.