README.md

---
base_model: unsloth/llama-3.2-3b-instruct-unsloth-bnb-4bit
tags:
- text-generation-inference
- transformers
- unsloth
- llama
- trl
- sft
- sql
license: apache-2.0
language:
- en
datasets:
- b-mc2/sql-create-context
---

## **Model Card for Llama3.2-3B-SQL-Expert-1Epoch**

### **Model Details**
#### **Model Description**
Llama3.2-3B-SQL-Expert-1Epoch is a fine-tuned version of Meta’s Llama-3.1-3B, specifically optimized for generating SQL queries from natural language input. The model has been trained using **Unsloth** for efficient fine-tuning and inference.

- **Developed by:** Azzedine (GitHub: Azzedde)  
- **Funded by [optional]:** N/A  
- **Shared by [optional]:** Azzedde  
- **Model Type:** Large Language Model (LLM) optimized for SQL query generation  
- **Language(s) (NLP):** English  
- **License:** Apache 2.0  
- **Finetuned from model [optional]:** Meta-Llama-3.1-3B-Instruct  

---

### **Model Sources**
- **Repository:** [Hugging Face](https://huggingface.co/Azzedde/llama3.2-3b-sql-expert-1-epoch)  
- **Paper [optional]:** N/A  
- **Demo [optional]:** N/A  

---

### **Uses**
#### **Direct Use**
This model is designed for generating **SQL queries** based on natural language inputs and is useful for:

- **Database management and administration**
- **Automated query generation**
- **Data analytics pipelines**
- **SQL education and training**
- **Business intelligence applications**

#### **Downstream Use [optional]**
- Embedding into LLM-based database assistants  
- Automating SQL-based analytics  
- Assisting developers in writing optimized queries  

#### **Out-of-Scope Use**
- General NLP tasks unrelated to SQL query generation  
- Applications requiring strong factual accuracy outside SQL  

---

### **Bias, Risks, and Limitations**
- **Incorrect or suboptimal queries:** The model may generate queries that are syntactically correct but do not yield the intended results.  
- **Lack of query optimization:** The generated queries are not always optimized for performance; users should validate execution plans.  
- **English-only support:** The model primarily supports English-language inputs.  
- **Limited schema understanding:** The model does not validate database structures and may assume incorrect relationships between tables.  

### **Recommendations**
Users should:
- **Always validate generated SQL queries** before executing them.  
- **Use the model as an assistant**, not a replacement for SQL expertise.  
- **Fine-tune the model further** for domain-specific databases.  

---

### **How to Get Started with the Model**
Use the following code to load and use the model:

```python
from unsloth import FastLanguageModel
from transformers import AutoTokenizer

# Load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained("Azzedde/llama3.2-3b-sql-expert-1-epoch")
model = FastLanguageModel.from_pretrained("Azzedde/llama3.2-3b-sql-expert-1-epoch")

# Example inference
sql_prompt = """Below is a SQL database schema and a question. Generate an SQL query to answer the question.

### Schema:
{schema}

### Question:
{question}

### SQL Query:
"""
input_text = sql_prompt.format(
    schema="CREATE TABLE employees (id INT PRIMARY KEY, name VARCHAR, salary DECIMAL, department_id INT);",
    question="Find the average salary per department."
)

# Tokenize and generate query
inputs = tokenizer(input_text, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=64, use_cache=True)
print(tokenizer.decode(outputs[0]))
```

---

### **Training Details**
#### **Training Data**
- The model was fine-tuned on a **structured SQL dataset**, including a mix of **publicly available SQL benchmarks** and **synthetically generated SQL queries**.  

#### **Training Procedure**
- **Preprocessing:** Tokenized using standard **SQL syntax formatting**  
- **Training Hyperparameters:**  
  - `batch_size = 4`  
  - `gradient_accumulation_steps = 8`  
  - `num_train_epochs = 1`  
  - `learning_rate = 2e-4`  
  - `fp16 = True`  

---

### **Evaluation**
#### **Testing Data**
- The model was evaluated on a separate test set of **SQL queries** derived from real-world **database schemas**.  

#### **Evaluation Metrics**
- **Exact Match Accuracy**: Percentage of queries that exactly match ground-truth SQL  
- **Execution Success Rate**: Percentage of generated queries that execute without errors  

#### **Results**
- **High accuracy** for common SQL queries  
- **Some errors** in complex multi-table joins and nested queries  

---

### **Environmental Impact**
- **Hardware Type:** Tesla T4 (Google Colab)  
- **Training Duration:** ~1.5 hours  
- **Compute Region:** N/A  
- **Estimated Carbon Emissions:** Minimal  

---

### **Technical Specifications**
#### **Model Architecture and Objective**
- Based on **Llama-3.1 3B**, fine-tuned with **LoRA** for SQL generation.  

#### **Compute Infrastructure**
- **Fine-tuned using Unsloth** for efficient training and inference.  

#### **Hardware**
- **GPU:** Tesla T4  
- **Max Reserved Memory:** ~6.5 GB  

#### **Software**
- **Libraries Used:** `unsloth`, `transformers`, `TRL`, `datasets`  

---

### **Citation [optional]**
#### **BibTeX:**
```bibtex
@article{llama3.2-3B-SQL-Expert,
  author    = {Azzedde},
  title     = {Llama3.2-3B-SQL-Expert: An SQL Query Generation Model},
  year      = {2025},
  url       = {https://huggingface.co/Azzedde/llama3.2-3b-sql-expert-1-epoch}
}
```

#### **APA:**
Azzedde. (2025). *Llama3.2-3B-SQL-Expert: An SQL Query Generation Model.* Retrieved from Hugging Face  

---

### **More Information**
For questions, reach out via **Hugging Face discussions** or **GitHub issues**.  

---

### **Model Card Authors**
- **Azzedde** (GitHub: [Azzedde](https://github.com/Azzedde))  

---

### **Model Card Contact**
- **Contact:** [Hugging Face Profile](https://huggingface.co/Azzedde)