README.md

---
library_name: transformers
tags:
- government
- conversational
- question-answering
- dutch
- geitje
license: apache-2.0
datasets:
- Nelis5174473/Dutch-QA-Pairs-Rijksoverheid
language:
- nl
pipeline_tag: text-generation
---

<p align="center" style="margin:0;padding:0">
<img src="https://cdn-uploads.huggingface.co/production/uploads/65e04544f59f66e0e072dc5c/b-OsZLNJtPHMwzbgwmGlV.png" alt="GovLLM Ultra banner" width="800" style="margin-left:'auto' margin-right:'auto' display:'block'"/>
</p>

<div style="margin:auto; text-align:center">
<h1 style="margin-bottom: 0">GovLLM-7B-ultra</h1>
<em>A question answering model about the Dutch Government.</em>
</div>

## Model description

This model is a fine-tuned version of the Dutch conversational model [BramVanroy/GEITje-7B-ULTRA](https://huggingface.co/BramVanroy/GEITje-7B-ultra) on a [Dutch question-answer pair dataset](https://huggingface.co/datasets/Nelis5174473/Dutch-QA-Pairs-Rijksoverheid) of the Dutch Government. This is a Dutch question/answer model ultimately based on Mistral and fine-tuned with SFT and LoRA. The training with 3 epochs took almost 2 hours and was run on an Nvidia A100 (40GB VRAM).

# Usage with Inference Endpoints (Dedicated)

```python
import requests

API_URL = "https://your-own-endpoint.us-east-1.aws.endpoints.huggingface.cloud"
headers = {"Authorization": "Bearer hf_your_own_token"}

def query(payload):
	response = requests.post(API_URL, headers=headers, json=payload)
	return response.json()

output = query({
	"inputs": "Geeft de overheid subsidie aan bedrijven?"
})

# print generated answer
print(output[0]['generated_text'])
```

## Training hyperparameters

The following hyperparameters were used during training:
- block_size: 1024,
- model_max_length: 2048,
- padding: right,
- mixed_precision: fp16,
- learning rate (lr): 0.00003,
- epochs: 3,
- batch_size: 2,
- optimizer: adamw_torch,
- schedular: linear,
- quantization: int8,
- peft: true,
- lora_r: 16,
- lora_alpha: 16,
- lora_dropout: 0.05

### Training results

|  Epoch |  Loss    |  Grad_norm | learning_rate |  step    |
|:------:|---------:|:----------:|:-------------:|:--------:|
|  0.14  |  1.3183  |  0.6038    |  1.3888e-05   |  25/540  |
|  0.42  |  1.0220  |  0.4180    |  2.8765e-05   |  75/540  |
|  0.69  |  0.9251  |  0.4119    |  2.56793-05   |  125/540 |
|  0.97  |  0.9260  |  0.4682    |  2.2592e-05   |  175/540 |
|  1.25  |  0.8586  |  0.5338    |  1.9506e-05   |  225/540 |
|  1.53  |  0.8767  |  0.6359    |  1.6420e-05   |  275/540 |
|  1.80  |  0.8721  |  0.6137    |  1.3333e-05   |  325/540 |
|  2.08  |  0.8469  |  0.7310    |  1.0247e-05   |  375/540 |
|  2.36  |  0.8324  |  0.7945    |  7.1605e-05   |  425/540 |
|  2.64  |  0.8170  |  0.8522    |  4.0741e-05   |  475/540 |
|  2.91  |  0.8185  |  0.8562    |  9.8765e-05   |  525/540 |