|
|
--- |
|
|
license: llama3 |
|
|
language: |
|
|
- en |
|
|
library_name: transformers |
|
|
tags: |
|
|
- llama-3 |
|
|
- llama-3.2 |
|
|
- bitcoin |
|
|
- finance |
|
|
- instruction-following |
|
|
- fine-tuning |
|
|
- merged |
|
|
base_model: meta-llama/Llama-3.2-3B-Instruct |
|
|
datasets: |
|
|
- tahamajs/bitcoin-llm-finetuning-dataset |
|
|
--- |
|
|
|
|
|
# Llama-3.2-3B Instruct - Advanced Bitcoin Analyst |
|
|
|
|
|
This repository contains a highly specialized version of `meta-llama/Llama-3.2-3B-Instruct`, expertly fine-tuned to function as a **Bitcoin and cryptocurrency market analyst**. This model is the result of a "continuation training" process, where an already specialized model was further refined on a targeted dataset. |
|
|
|
|
|
This model excels at understanding and responding to complex instructions related to blockchain technology, financial markets, and technical/fundamental analysis of cryptocurrencies. |
|
|
|
|
|
## 🧠 Training Procedure |
|
|
|
|
|
The final model was created through a sophisticated multi-stage process designed to build upon and deepen existing knowledge. |
|
|
|
|
|
### Stage 1: Initial Specialization (Adapter Merge) |
|
|
The process began with the base `meta-llama/Llama-3.2-3B-Instruct` model. This base was then merged with a previously fine-tuned, high-performing LoRA adapter to create an initial specialized model. |
|
|
- **Initial Adapter:** `tahamajs/llama-3.2-3b-instruct-bitcoin-analyst-perfect` |
|
|
|
|
|
### Stage 2: Continued Fine-Tuning (New LoRA) |
|
|
A new LoRA adapter was then trained on top of the already-merged model from Stage 1. This continuation training allowed the model to further refine its expertise using a specific dataset, improving its nuance and instruction-following on relevant topics. |
|
|
- **Dataset:** `tahamajs/bitcoin-llm-finetuning-dataset` |
|
|
|
|
|
### Stage 3: Final Merge |
|
|
The final step was to merge the newly trained adapter from Stage 2 into the model. This repository hosts this **fully merged, standalone model**, which contains the cumulative knowledge of the base model, the first specialized adapter, and the second round of continuation training. |
|
|
|
|
|
--- |
|
|
|
|
|
## 📊 Training Details |
|
|
|
|
|
### Hyperparameters |
|
|
The second stage of LoRA fine-tuning was performed with the following key hyperparameters: |
|
|
|
|
|
| Parameter | Value | |
|
|
| :--- | :--- | |
|
|
| `learning_rate` | 1e-4 | |
|
|
| `num_train_epochs` | 1 | |
|
|
| `lora_r` | 16 | |
|
|
| `lora_alpha` | 32 | |
|
|
| `optimizer` | paged_adamw_32bit | |
|
|
| `precision` | bf16 | |
|
|
|
|
|
### Training Loss |
|
|
The training loss shows a clear downward trend, indicating that the model was successfully learning from the dataset. The process started with a loss of ~2.18 and converged to a loss in the ~1.4-1.6 range, demonstrating effective knowledge acquisition. The fluctuations are normal during training and reflect the varying difficulty of the data in each batch. |
|
|
|
|
|
 |
|
|
|
|
|
--- |
|
|
|
|
|
## 🚀 How to Use |
|
|
|
|
|
This is a fully merged model and can be used directly with the `transformers` library. For best results, use the Llama 3 chat template to format your prompts. |
|
|
|
|
|
```python |
|
|
import torch |
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer |
|
|
|
|
|
# Use the ID of the repository where this model is hosted |
|
|
model_id = "tahamajs/llama-3.2-3b-instruct-bitcoin-analyst-perfect_v2" |
|
|
|
|
|
# Load the tokenizer and model |
|
|
tokenizer = AutoTokenizer.from_pretrained(model_id) |
|
|
model = AutoModelForCausalLM.from_pretrained( |
|
|
model_id, |
|
|
torch_dtype=torch.bfloat16, |
|
|
device_map="auto", |
|
|
) |
|
|
|
|
|
# Use the Llama 3 chat template for instruction-following |
|
|
messages = [ |
|
|
{"role": "user", "content": "Analyze the current sentiment around Bitcoin based on the concept of the Fear & Greed Index. What does a high 'Greed' value typically imply for the short-term market?"}, |
|
|
] |
|
|
|
|
|
# Apply the chat template and tokenize |
|
|
input_ids = tokenizer.apply_chat_template( |
|
|
messages, |
|
|
add_generation_prompt=True, |
|
|
return_tensors="pt" |
|
|
).to(model.device) |
|
|
|
|
|
# Generate a response |
|
|
outputs = model.generate( |
|
|
input_ids, |
|
|
max_new_tokens=512, |
|
|
do_sample=True, |
|
|
temperature=0.6, |
|
|
top_p=0.9, |
|
|
) |
|
|
|
|
|
# Decode and print the output |
|
|
response = outputs[0][input_ids.shape[-1]:] |
|
|
print(tokenizer.decode(response, skip_special_tokens=True)) |
|
|
```` |
|
|
|
|
|
## Disclaimer |
|
|
|
|
|
This model is provided for informational and educational purposes only. It is **not financial advice**. The outputs are generated by an AI and may contain errors or inaccuracies. Always perform your own due diligence and consult with a qualified financial professional before making any investment decisions. |
|
|
|
|
|
|
