File size: 4,172 Bytes
79d64cd 517a614 79d64cd 517a614 79d64cd 517a614 79d64cd 517a614 79d64cd 517a614 79d64cd 517a614 79d64cd 517a614 79d64cd 517a614 79d64cd 517a614 79d64cd 517a614 79d64cd 517a614 79d64cd 517a614 79d64cd 517a614 79d64cd 517a614 79d64cd 517a614 79d64cd 517a614 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 |
---
base_model: unsloth/gemma-7b-bnb-4bit
library_name: peft
---
# Gemma2 Fine-Tuned LoRA Model
## Overview
This is a **LoRA (Low-Rank Adaptation)** fine-tuned model based on the **`unsloth/gemma-7b-bnb-4bit`** base model. It has been adapted for a **tipification analysis task** similar to the Llama-3.2-3B-Instruct LoRA fine-tuning, where the model classifies text into categories such as **"ESTAFA," "ROBO," "HURTO,"** and their **"TENTATIVA DE"** variations.
During fine-tuning, only specific adapter layers were trained (\~50 million parameters), while the rest of the base model was frozen. This approach allows parameter-efficient training, significantly reducing computational costs.
---
## Key Features
- **Base Model**: `unsloth/gemma-7b-bnb-4bit`
- **Task Type**: Causal Language Modeling (`CAUSAL_LM`)
- **LoRA Parameters**:
- `r`: 16
- `lora_alpha`: 16
- `lora_dropout`: 0.0
- **Target Modules**:
- `gate_proj`, `up_proj`, `down_proj`, `k_proj`, `q_proj`, `o_proj`, `v_proj`
- **Number of Trainable Parameters**: **50,003,968**
- **Training Loss & Validation Loss**:
- Observed over **117 steps** (1 epoch).
- See table below for detailed step-by-step values.
---
## Dataset Distribution
This model was fine-tuned on the **same dataset** as the Llama-3.2-3B-Instruct LoRA version, with the following category distribution:
| **Category** | **Count** | **Percentage** |
|--------------------------|-----------|----------------|
| ESTAFA | 4610 | 47.3% |
| ROBO | 2307 | 23.7% |
| HURTO | 2141 | 22.0% |
| TENTATIVA DE ESTAFA | 306 | 3.1% |
| TENTATIVA DE ROBO | 272 | 2.8% |
| TENTATIVA DE HURTO | 113 | 1.2% |
| **Total** | 9749 | 100% |
Although the dataset has nearly 10K examples in this summary table, the fine-tuning run used an extended version (\~15K examples) for this particular training session.
---
## Training Details
- **Hardware**: Single GPU A100 40Gb
- **Num Examples**: ~15,000
- **Epochs**: 1
- **Batch Size per Device**: 32
- **Gradient Accumulation Steps**: 4
- **Effective Total Batch Size**: 128
- **Total Steps**: 117
- **Number of Trainable Parameters**: 50,003,968
### Training and Validation Loss
Below is a snapshot of how training and validation loss evolved during the single epoch (117 steps):
| **Step** | **Training Loss** | **Validation Loss** |
|----------|-------------------|---------------------|
| 10 | 2.974900 | 4.242294 |
| 20 | 5.451000 | 4.526450 |
| 30 | 4.150400 | 3.632928 |
| 40 | 3.036100 | 2.615031 |
| 50 | 2.492900 | 2.178700 |
| 60 | 2.095400 | 1.886430 |
| 70 | 2.099200 | 1.548187 |
| 80 | 1.983100 | 2.104600 |
| 90 | 2.020900 | 1.526225 |
| 100 | 1.727700 | 1.699223 |
| 110 | 1.868300 | 1.716561 |
| ... | ... | ... |
Final training concluded at **step 117**.
We observe a steady decrease in both training and validation losses, indicating the model was converging throughout the single epoch.
---
## Deployment Instructions
You can use this LoRA fine-tuned model with the Hugging Face Transformers library. Below is an example of how to load and run the model for text generation or classification-like tasks:
```python
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("Petermoyano/unsloth-gemma-7b-bnb-4bit-LoRA-Tipification-CausalLM-16R-16Alpha-1Epoch")
model = AutoModelForCausalLM.from_pretrained("Petermoyano/unsloth-gemma-7b-bnb-4bit-LoRA-Tipification-CausalLM-16R-16Alpha-1Epoch")
input_text = "TENTATIVA DE ESTAFA:"
inputs = tokenizer(input_text, return_tensors="pt")
outputs = model.generate(**inputs, max_length=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
|