File size: 9,680 Bytes
1786a04 e30a43a 1786a04 e30a43a 1786a04 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 |
---
license: llama3.1
language:
- de
- en
- es
- fr
- it
tags:
- text-generation-inference
base_model:
- meta-llama/Llama-3.1-8B-Instruct
---
# Model Card for LLM-wsd-TT-20000
## Model description
<!-- Provide a quick summary of what the model is/does. -->
**LLM-wsd-TT-20000** is a *Large Language Model (LLM)* instruction-tuned over **meta-llama/Meta-Llama-3.1-8B-Instruct**.
This model has been trained for the **WSD** task over a balanced training dataset (20000 instances per language), with machine-translation. It is capable of providing the definition of a word in a given sentence. Specifically, it can answer both:
1) **Open-ended questions**, where the model will generate the definition of the target word;
2) **Closed-ended questions**, where the model will generate the identifier of the correct option out of a list of alternatives.
More details regarding the training procedure (e.g. hyperparameters, dataset construction, and so on) can be found in Section 4.2 of the [paper](https://arxiv.org/abs/2503.08662).
- **Developed by:** Pierpaolo Basile, Lucia Siciliani, Elio Musacchio
- **Model type:** LLaMA 3.1 Instruct
- **Language(s) (NLP):** English, French, German, Italian and Spanish
- **License:** [LLAMA 3.1 COMMUNITY LICENSE AGREEMENT](https://huggingface.co/meta-llama/Llama-3.1-70B-Instruct/blob/main/LICENSE)
- **Finetuned from model:** [meta-llama/Meta-Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct)
## Prompt Format
The model has been trained using several instructions depending on language, task (open-ended or closed-ended) and number of occurences of target word in the sentence. In [Instructions](#instructions), we provide the instructions used for all cases. The following placeholder variables have to be replaced:
- {target_word}: the target word in the input to disambiguate;
- {options}: options to provide to the model for the closed-ended task only. The options should be newline separated and each option should be identified by a number. Refer to the [closed-ended example](#closed-ended) for an example of options formatting;
- {occurrence}: the ordinal number of the {target_word} occurrence (e.g. "second"). This is required only when the input sentence contains multiple occurrences of {target_word}.
Please note that the complete prompt also has the following string after the instruction:
```python
" Input: \"{sentence}\""
```
where {sentence} is the input sentence containing the word to disambiguate.
## How to Get Started with the Model
Below you can find two examples of model usage, for open-ended and closed-ended generation respectively.
### Open-ended
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers.trainer_utils import set_seed
target_word = "long"
instruction = f"Give a brief definition of the word \"{target_word}\" in the sentence given as input. Generate only the definition."
input_sentence = "How long has it been since you reviewed the objectives of your benefit and service program?"
model_id = "swap-uniba/LLM-wsd-TT-20000"
set_seed(42)
tokenizer = AutoTokenizer.from_pretrained(model_id, use_fast=False)
tokenizer.padding_side = "left"
model = AutoModelForCausalLM.from_pretrained(
model_id,
device_map='cuda',
torch_dtype=torch.bfloat16,
).eval()
terminators = [
tokenizer.eos_token_id,
tokenizer.convert_tokens_to_ids("<|eot_id|>")
]
messages = [
{"role": "user", "content": instruction + " Input: \"" + input_sentence + "\""},
]
input_ids = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
outputs = model.generate(
input_ids.to('cuda'),
max_new_tokens=512,
eos_token_id=terminators,
num_beams=1,
do_sample=False
)
print(tokenizer.decode(outputs[0][input_ids.shape[-1]:], skip_special_tokens=True))
```
### Closed-ended
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers.trainer_utils import set_seed
target_word = "hurry"
instruction = f"Given the word \"{target_word}\" in the input sentence, choose the correct meaning from the following:\n1) Move very fast\n2) Urge to an unnatural speed\n\nGenerate only the number of the selected option."
input_sentence = "If you hurry you might beat the headquarters boys."
model_id = "swap-uniba/LLM-wsd-TT-20000"
set_seed(42)
tokenizer = AutoTokenizer.from_pretrained(model_id, use_fast=False)
tokenizer.padding_side = "left"
model = AutoModelForCausalLM.from_pretrained(
model_id,
device_map='cuda',
torch_dtype=torch.bfloat16,
).eval()
terminators = [
tokenizer.eos_token_id,
tokenizer.convert_tokens_to_ids("<|eot_id|>")
]
messages = [
{"role": "user", "content": instruction + " Input: \"" + input_sentence + "\""},
]
input_ids = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
outputs = model.generate(
input_ids.to('cuda'),
max_new_tokens=512,
eos_token_id=terminators,
num_beams=1,
do_sample=False
)
print(tokenizer.decode(outputs[0][input_ids.shape[-1]:], skip_special_tokens=True))
```
## Citation
If you use this model in your research, please cite the following:
```bibtex
@misc{basile2025exploringwordsensedisambiguation,
title={Exploring the Word Sense Disambiguation Capabilities of Large Language Models},
author={Pierpaolo Basile and Lucia Siciliani and Elio Musacchio and Giovanni Semeraro},
year={2025},
eprint={2503.08662},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2503.08662},
}
```
## Instructions
### Single occurrence of target word (open-ended)
#### English
```python
"Give a brief definition of the word \"{target_word}\" in the sentence given as input. Generate only the definition."
```
#### French
```python
"Donnez une brève définition du mot \"{target_word}\" dans la phrase d’entrée donnée. Ne donnez que la définition."
```
#### German
```python
"Geben Sie eine kurze Definition des Wortes \"{target_word}\" in dem gegebenen Satz an. Erzeugen Sie nur die Definition."
```
#### Italian
```python
"Fornisci una breve definizione della parola \"{target_word}\" nella frase data in input. Genera solo la definizione."
```
#### Spanish
```python
"Proporciona una definición breve de la palabra \"{target_word}\" en la frase dada en entrada. Genera solo la definición."
```
### Multiple occurences of target word (open-ended)
#### English
```python
"Give a brief definition of the {occurrence} occurrence of the word \"{target_word}\" in the sentence given as input. Generate only the definition."
```
#### French
```python
"Donnez une brève définition de l'occurrence {occurrence} du mot \"{target_word}\" dans la phrase d’entrée donnée. Ne donnez que la définition."
```
#### German
```python
"Geben Sie eine kurze Definition des {occurrence} Vorkommens des Wortes \"{target_word}\" in dem gegebenen Eingabesatz an. Erzeugen Sie nur die Definition."
```
#### Italian
```python
"Fornisci una breve definizione della {occurrence} occorrenza della parola \"{target_word}\" nella frase data in input. Genera solo la definizione."
```
#### Spanish
```python
"Proporciona una definición breve de la {occurrence} ocurrencia de la palabra \"{target_word}\" en la frase dada en entrada. Genera solo la definición."
```
### Single occurrence of target word (closed-ended)
#### English
```python
"Given the word \"{target_word}\" in the input sentence, choose the correct meaning from the following:\n{options}\n\nGenerate only the number of the selected option."
```
#### French
```python
"Étant donné le mot \"{target_word}\" dans la phrase saisie, choisissez la signification correcte parmi les suivantes:\n{options}\n\nNe donnez que le numéro de l’option sélectionnée."
```
#### German
```python
"Wählen Sie für das Wort \"{target_word}\" im Eingabesatz die richtige Bedeutung aus den folgenden Angaben:\n{options}\n\nErzeugt nur die Nummer der ausgewählten Option"
```
#### Italian
```python
"Data la parola \"{target_word}\" nella frase in input, scegli il significato corretto tra i seguenti:\n{options}\n\nGenera solo il numero dell'opzione selezionata."
```
#### Spanish
```python
"Dada la palabra \"{target_word}\" en la frase de entrada, elija el significado correcto entre los siguientes:\n{options}\n\nGenera solo el número de la opción seleccionada."
```
### Multiple occurrences of target word (closed-ended)
#### English
```python
"Given the word \"{target_word}\" in the input sentence, choose the correct meaning from the following:\n{options}\n\nGenerate only the number of the selected option."
```
#### French
```python
"Étant donné l'occurrence {occurrence} du mot \"{target_word}\" dans la phrase d'entrée, choisissez la signification correcte parmi les suivantes:\n{options}\n\nNe donnez que le numéro de l’option sélectionnée."
```
#### German
```python
"Wählen Sie angesichts des {occurrence} Vorkommens des Wortes \"{target_word}\" im Eingabesatz die richtige Bedeutung aus der folgenden Liste aus:\n{options}\n\nErzeugt nur die Nummer der ausgewählten Option."
```
#### Italian
```python
"Data la {occurrence} occorrenza della parola \"{target_word}\" nella frase in input, scegli il significato corretto tra i seguenti:\n{options}\n\nGenera solo il numero dell'opzione selezionata."
```
#### Spanish
```python
"Dada la {occurrence} ocurrencia de la palabra \"{target_word}\" en la frase de entrada, elije el significado correcto entre los siguientes:\n{options}\n\nGenera solo el número de la opción seleccionada."
``` |