File size: 4,407 Bytes
b471a20 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 |
---
license: llama3
language:
- en
library_name: transformers
tags:
- llama-3
- llama-3.2
- bitcoin
- finance
- instruction-following
- fine-tuning
- merged
base_model: meta-llama/Llama-3.2-3B-Instruct
datasets:
- tahamajs/bitcoin-llm-finetuning-dataset
---
# Llama-3.2-3B Instruct - Advanced Bitcoin Analyst
This repository contains a highly specialized version of `meta-llama/Llama-3.2-3B-Instruct`, expertly fine-tuned to function as a **Bitcoin and cryptocurrency market analyst**. This model is the result of a "continuation training" process, where an already specialized model was further refined on a targeted dataset.
This model excels at understanding and responding to complex instructions related to blockchain technology, financial markets, and technical/fundamental analysis of cryptocurrencies.
## 🧠 Training Procedure
The final model was created through a sophisticated multi-stage process designed to build upon and deepen existing knowledge.
### Stage 1: Initial Specialization (Adapter Merge)
The process began with the base `meta-llama/Llama-3.2-3B-Instruct` model. This base was then merged with a previously fine-tuned, high-performing LoRA adapter to create an initial specialized model.
- **Initial Adapter:** `tahamajs/llama-3.2-3b-instruct-bitcoin-analyst-perfect`
### Stage 2: Continued Fine-Tuning (New LoRA)
A new LoRA adapter was then trained on top of the already-merged model from Stage 1. This continuation training allowed the model to further refine its expertise using a specific dataset, improving its nuance and instruction-following on relevant topics.
- **Dataset:** `tahamajs/bitcoin-llm-finetuning-dataset`
### Stage 3: Final Merge
The final step was to merge the newly trained adapter from Stage 2 into the model. This repository hosts this **fully merged, standalone model**, which contains the cumulative knowledge of the base model, the first specialized adapter, and the second round of continuation training.
---
## 📊 Training Details
### Hyperparameters
The second stage of LoRA fine-tuning was performed with the following key hyperparameters:
| Parameter | Value |
| :--- | :--- |
| `learning_rate` | 1e-4 |
| `num_train_epochs` | 1 |
| `lora_r` | 16 |
| `lora_alpha` | 32 |
| `optimizer` | paged_adamw_32bit |
| `precision` | bf16 |
### Training Loss
The training loss shows a clear downward trend, indicating that the model was successfully learning from the dataset. The process started with a loss of ~2.18 and converged to a loss in the ~1.4-1.6 range, demonstrating effective knowledge acquisition. The fluctuations are normal during training and reflect the varying difficulty of the data in each batch.
---
## 🚀 How to Use
This is a fully merged model and can be used directly with the `transformers` library. For best results, use the Llama 3 chat template to format your prompts.
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
# Use the ID of the repository where this model is hosted
model_id = "tahamajs/llama-3.2-3b-instruct-bitcoin-analyst-perfect_v2"
# Load the tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype=torch.bfloat16,
device_map="auto",
)
# Use the Llama 3 chat template for instruction-following
messages = [
{"role": "user", "content": "Analyze the current sentiment around Bitcoin based on the concept of the Fear & Greed Index. What does a high 'Greed' value typically imply for the short-term market?"},
]
# Apply the chat template and tokenize
input_ids = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
return_tensors="pt"
).to(model.device)
# Generate a response
outputs = model.generate(
input_ids,
max_new_tokens=512,
do_sample=True,
temperature=0.6,
top_p=0.9,
)
# Decode and print the output
response = outputs[0][input_ids.shape[-1]:]
print(tokenizer.decode(response, skip_special_tokens=True))
````
## Disclaimer
This model is provided for informational and educational purposes only. It is **not financial advice**. The outputs are generated by an AI and may contain errors or inaccuracies. Always perform your own due diligence and consult with a qualified financial professional before making any investment decisions.
|