|
|
--- |
|
|
base_model: unsloth/Qwen3-1.7B |
|
|
tags: |
|
|
- text-generation-inference |
|
|
- transformers |
|
|
- unsloth |
|
|
- qwen3 |
|
|
license: apache-2.0 |
|
|
language: |
|
|
- en |
|
|
--- |
|
|
# Fine-tuning Qwen3-1.7B for Text-to-SQL Task |
|
|
This project demonstrates the fine-tuning of the `Qwen3-1.7B` language model using a combined and preprocessed dataset for Text-to-SQL generation. The goal is to train the model to generate SQL queries from natural language questions given database schemas. |
|
|
|
|
|
## Dataset |
|
|
We used the [fahmiaziz/text2sql-dataset](https://huggingface.co/datasets/fahmiaziz/text2sql-dataset), which merges examples from: |
|
|
- **Wikisql** |
|
|
- **Bird** |
|
|
- **Spider** |
|
|
- **Synthetic SQL samples** |
|
|
|
|
|
Before training, the dataset was **cleaned and filtered** by: |
|
|
- Removing DDL/DML examples (`INSERT`, `UPDATE`, `DELETE`, etc.) |
|
|
- Deduplicating examples based on **semantic hashing of both SQL and questions** |
|
|
- Filtering only SELECT-style analytical queries |
|
|
|
|
|
## Training Format |
|
|
Since Qwen3 models require a two-part output (`<think>` + final answer), and our dataset does not contain intermediate reasoning, we left the `<think>` section **empty** during fine-tuning. |
|
|
|
|
|
### Example Format: |
|
|
|
|
|
``` |
|
|
|
|
|
<|im_start|>system |
|
|
Given the database schema and the user question, generate the corresponding SQL query. |
|
|
<|im_end|> |
|
|
<|im_start|>user |
|
|
|
|
|
\[SCHEMA] |
|
|
CREATE TABLE Inclusive\_Housing (Property\_ID INT, Inclusive VARCHAR(10), Property\_Size INT); |
|
|
INSERT INTO Inclusive\_Housing (Property\_ID, Inclusive, Property\_Size) |
|
|
VALUES (1, 'Yes', 900), (2, 'No', 1100), (3, 'Yes', 800), (4, 'No', 1200); |
|
|
|
|
|
\[QUESTION] |
|
|
What is the average property size in inclusive housing areas? |
|
|
<|im_end|> |
|
|
<|im_start|>assistant |
|
|
<think> |
|
|
|
|
|
</think> |
|
|
SELECT AVG(Property\_Size) FROM Inclusive\_Housing WHERE Inclusive = 'Yes'; |
|
|
<|im_end|> |
|
|
|
|
|
```` |
|
|
|
|
|
## Training Configuration |
|
|
Due to hardware limitations, **full model training** was not possible. Instead, we applied **LoRA (Low-Rank Adaptation)** with the following configuration: |
|
|
- **LoRA rank (`r`)**: 128 |
|
|
- **LoRA alpha**: 256 |
|
|
- **Hardware**: Kaggle T4 x2 GPUs |
|
|
|
|
|
### Training Hyperparameters |
|
|
``` |
|
|
per_device_train_batch_size = 6, |
|
|
gradient_accumulation_steps = 2, |
|
|
warmup_steps = 5, |
|
|
max_steps = 500, |
|
|
num_train_epochs = 3, |
|
|
learning_rate = 1e-4, |
|
|
fp16 = not is_bf16_supported(), |
|
|
bf16 = is_bf16_supported(), |
|
|
logging_steps = 25, |
|
|
optim = "adamw_8bit", |
|
|
weight_decay = 0.01, |
|
|
lr_scheduler_type = "linear", |
|
|
seed = 3407, |
|
|
output_dir = "outputs_v4", |
|
|
dataset_text_field = "text", |
|
|
max_seq_length = 1024, |
|
|
```` |
|
|
|
|
|
## Training Results |
|
|
``` |
|
|
global_step=500, |
|
|
training_loss=0.5783241882324218 |
|
|
``` |
|
|
|
|
|
## Evaluation |
|
|
We evaluated the model using **Exact Match (EM)** score on a manually selected sample of 100 examples. We get score **50%** |
|
|
|
|
|
--- |
|
|
|
|
|
## Notes |
|
|
* In future iterations, we plan to: |
|
|
|
|
|
* Add complex/long context schema |
|
|
* Full Finetuning |
|
|
|
|
|
|
|
|
|
|
|
# Uploaded finetuned model |
|
|
|
|
|
- **Developed by:** fahmiaziz |
|
|
- **License:** apache-2.0 |
|
|
- **Finetuned from model :** unsloth/Qwen3-1.7B |
|
|
|
|
|
This qwen3 model was trained 2x faster with [Unsloth](https://github.com/unslothai/unsloth) and Huggingface's TRL library. |
|
|
|
|
|
[<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/>](https://github.com/unslothai/unsloth) |
|
|
|
