|
--- |
|
base_model: unsloth/gemma-7b-bnb-4bit |
|
library_name: peft |
|
--- |
|
|
|
# Gemma2 Fine-Tuned LoRA Model |
|
|
|
## Overview |
|
This is a **LoRA (Low-Rank Adaptation)** fine-tuned model based on the **`unsloth/gemma-7b-bnb-4bit`** base model. It has been adapted for a **tipification analysis task** similar to the Llama-3.2-3B-Instruct LoRA fine-tuning, where the model classifies text into categories such as **"ESTAFA," "ROBO," "HURTO,"** and their **"TENTATIVA DE"** variations. |
|
|
|
During fine-tuning, only specific adapter layers were trained (\~50 million parameters), while the rest of the base model was frozen. This approach allows parameter-efficient training, significantly reducing computational costs. |
|
|
|
--- |
|
|
|
## Key Features |
|
- **Base Model**: `unsloth/gemma-7b-bnb-4bit` |
|
- **Task Type**: Causal Language Modeling (`CAUSAL_LM`) |
|
- **LoRA Parameters**: |
|
- `r`: 16 |
|
- `lora_alpha`: 16 |
|
- `lora_dropout`: 0.0 |
|
- **Target Modules**: |
|
- `gate_proj`, `up_proj`, `down_proj`, `k_proj`, `q_proj`, `o_proj`, `v_proj` |
|
- **Number of Trainable Parameters**: **50,003,968** |
|
- **Training Loss & Validation Loss**: |
|
- Observed over **117 steps** (1 epoch). |
|
- See table below for detailed step-by-step values. |
|
|
|
--- |
|
|
|
## Dataset Distribution |
|
This model was fine-tuned on the **same dataset** as the Llama-3.2-3B-Instruct LoRA version, with the following category distribution: |
|
|
|
| **Category** | **Count** | **Percentage** | |
|
|--------------------------|-----------|----------------| |
|
| ESTAFA | 4610 | 47.3% | |
|
| ROBO | 2307 | 23.7% | |
|
| HURTO | 2141 | 22.0% | |
|
| TENTATIVA DE ESTAFA | 306 | 3.1% | |
|
| TENTATIVA DE ROBO | 272 | 2.8% | |
|
| TENTATIVA DE HURTO | 113 | 1.2% | |
|
| **Total** | 9749 | 100% | |
|
|
|
Although the dataset has nearly 10K examples in this summary table, the fine-tuning run used an extended version (\~15K examples) for this particular training session. |
|
|
|
--- |
|
|
|
## Training Details |
|
- **Hardware**: Single GPU A100 40Gb |
|
- **Num Examples**: ~15,000 |
|
- **Epochs**: 1 |
|
- **Batch Size per Device**: 32 |
|
- **Gradient Accumulation Steps**: 4 |
|
- **Effective Total Batch Size**: 128 |
|
- **Total Steps**: 117 |
|
- **Number of Trainable Parameters**: 50,003,968 |
|
|
|
### Training and Validation Loss |
|
Below is a snapshot of how training and validation loss evolved during the single epoch (117 steps): |
|
|
|
| **Step** | **Training Loss** | **Validation Loss** | |
|
|----------|-------------------|---------------------| |
|
| 10 | 2.974900 | 4.242294 | |
|
| 20 | 5.451000 | 4.526450 | |
|
| 30 | 4.150400 | 3.632928 | |
|
| 40 | 3.036100 | 2.615031 | |
|
| 50 | 2.492900 | 2.178700 | |
|
| 60 | 2.095400 | 1.886430 | |
|
| 70 | 2.099200 | 1.548187 | |
|
| 80 | 1.983100 | 2.104600 | |
|
| 90 | 2.020900 | 1.526225 | |
|
| 100 | 1.727700 | 1.699223 | |
|
| 110 | 1.868300 | 1.716561 | |
|
| ... | ... | ... | |
|
|
|
Final training concluded at **step 117**. |
|
We observe a steady decrease in both training and validation losses, indicating the model was converging throughout the single epoch. |
|
|
|
--- |
|
|
|
## Deployment Instructions |
|
You can use this LoRA fine-tuned model with the Hugging Face Transformers library. Below is an example of how to load and run the model for text generation or classification-like tasks: |
|
|
|
```python |
|
from transformers import AutoTokenizer, AutoModelForCausalLM |
|
|
|
tokenizer = AutoTokenizer.from_pretrained("Petermoyano/unsloth-gemma-7b-bnb-4bit-LoRA-Tipification-CausalLM-16R-16Alpha-1Epoch") |
|
model = AutoModelForCausalLM.from_pretrained("Petermoyano/unsloth-gemma-7b-bnb-4bit-LoRA-Tipification-CausalLM-16R-16Alpha-1Epoch") |
|
|
|
input_text = "TENTATIVA DE ESTAFA:" |
|
inputs = tokenizer(input_text, return_tensors="pt") |
|
outputs = model.generate(**inputs, max_length=50) |
|
print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |
|
|
