Safetensors
llama
text-generation-inference
File size: 9,680 Bytes
1786a04
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
e30a43a
1786a04
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
e30a43a
 
 
 
 
 
 
 
 
1786a04
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
---
license: llama3.1
language:
- de
- en
- es
- fr
- it
tags:
- text-generation-inference
base_model:
- meta-llama/Llama-3.1-8B-Instruct
---
# Model Card for LLM-wsd-TT-20000

## Model description

<!-- Provide a quick summary of what the model is/does. -->

**LLM-wsd-TT-20000** is a *Large Language Model (LLM)* instruction-tuned over **meta-llama/Meta-Llama-3.1-8B-Instruct**.
This model has been trained for the **WSD** task over a balanced training dataset (20000 instances per language), with machine-translation. It is capable of providing the definition of a word in a given sentence. Specifically, it can answer both:
1) **Open-ended questions**, where the model will generate the definition of the target word; 
2) **Closed-ended questions**, where the model will generate the identifier of the correct option out of a list of alternatives.

More details regarding the training procedure (e.g. hyperparameters, dataset construction, and so on) can be found in Section 4.2 of the [paper](https://arxiv.org/abs/2503.08662).

- **Developed by:** Pierpaolo Basile, Lucia Siciliani, Elio Musacchio
- **Model type:** LLaMA 3.1 Instruct
- **Language(s) (NLP):** English, French, German, Italian and Spanish
- **License:** [LLAMA 3.1 COMMUNITY LICENSE AGREEMENT](https://huggingface.co/meta-llama/Llama-3.1-70B-Instruct/blob/main/LICENSE)
- **Finetuned from model:** [meta-llama/Meta-Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct)

## Prompt Format

The model has been trained using several instructions depending on language, task (open-ended or closed-ended) and number of occurences of target word in the sentence. In [Instructions](#instructions), we provide the instructions used for all cases. The following placeholder variables have to be replaced:
- {target_word}: the target word in the input to disambiguate;
- {options}: options to provide to the model for the closed-ended task only. The options should be newline separated and each option should be identified by a number. Refer to the [closed-ended example](#closed-ended) for an example of options formatting;
- {occurrence}: the ordinal number of the {target_word} occurrence (e.g. "second"). This is required only when the input sentence contains multiple occurrences of {target_word}.

Please note that the complete prompt also has the following string after the instruction:

```python
" Input: \"{sentence}\""
```

where {sentence} is the input sentence containing the word to disambiguate.

## How to Get Started with the Model

Below you can find two examples of model usage, for open-ended and closed-ended generation respectively.

### Open-ended

```python
import torch

from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers.trainer_utils import set_seed

target_word = "long"
instruction = f"Give a brief definition of the word \"{target_word}\" in the sentence given as input. Generate only the definition."
input_sentence = "How long has it been since you reviewed the objectives of your benefit and service program?"

model_id = "swap-uniba/LLM-wsd-TT-20000"

set_seed(42)

tokenizer = AutoTokenizer.from_pretrained(model_id, use_fast=False)

tokenizer.padding_side = "left"

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map='cuda',
    torch_dtype=torch.bfloat16,
).eval()

terminators = [
    tokenizer.eos_token_id,
    tokenizer.convert_tokens_to_ids("<|eot_id|>")
]

messages = [
    {"role": "user", "content": instruction + " Input: \"" + input_sentence + "\""},
]

input_ids = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")

outputs = model.generate(
    input_ids.to('cuda'),
    max_new_tokens=512,
    eos_token_id=terminators,
    num_beams=1,
    do_sample=False
)

print(tokenizer.decode(outputs[0][input_ids.shape[-1]:], skip_special_tokens=True))
```

### Closed-ended

```python
import torch

from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers.trainer_utils import set_seed

target_word = "hurry"
instruction = f"Given the word \"{target_word}\" in the input sentence, choose the correct meaning from the following:\n1) Move very fast\n2) Urge to an unnatural speed\n\nGenerate only the number of the selected option."
input_sentence = "If you hurry you might beat the headquarters boys."

model_id = "swap-uniba/LLM-wsd-TT-20000"

set_seed(42)

tokenizer = AutoTokenizer.from_pretrained(model_id, use_fast=False)

tokenizer.padding_side = "left"

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map='cuda',
    torch_dtype=torch.bfloat16,
).eval()

terminators = [
    tokenizer.eos_token_id,
    tokenizer.convert_tokens_to_ids("<|eot_id|>")
]

messages = [
    {"role": "user", "content": instruction + " Input: \"" + input_sentence + "\""},
]

input_ids = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")

outputs = model.generate(
    input_ids.to('cuda'),
    max_new_tokens=512,
    eos_token_id=terminators,
    num_beams=1,
    do_sample=False
)

print(tokenizer.decode(outputs[0][input_ids.shape[-1]:], skip_special_tokens=True))
```

## Citation

If you use this model in your research, please cite the following:

```bibtex
@misc{basile2025exploringwordsensedisambiguation,
      title={Exploring the Word Sense Disambiguation Capabilities of Large Language Models}, 
      author={Pierpaolo Basile and Lucia Siciliani and Elio Musacchio and Giovanni Semeraro},
      year={2025},
      eprint={2503.08662},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2503.08662}, 
}
```

## Instructions

### Single occurrence of target word (open-ended)

#### English
```python
"Give a brief definition of the word \"{target_word}\" in the sentence given as input. Generate only the definition."
```

#### French
```python
"Donnez une brève définition du mot \"{target_word}\" dans la phrase d’entrée donnée. Ne donnez que la définition."
```

#### German
```python
"Geben Sie eine kurze Definition des Wortes \"{target_word}\" in dem gegebenen Satz an. Erzeugen Sie nur die Definition."
```

#### Italian
```python
"Fornisci una breve definizione della parola \"{target_word}\" nella frase data in input. Genera solo la definizione."
```

#### Spanish
```python
"Proporciona una definición breve de la palabra \"{target_word}\" en la frase dada en entrada. Genera solo la definición."
```

### Multiple occurences of target word (open-ended)

#### English
```python
"Give a brief definition of the {occurrence} occurrence of the word \"{target_word}\" in the sentence given as input. Generate only the definition."
```

#### French
```python
"Donnez une brève définition de l'occurrence {occurrence} du mot \"{target_word}\" dans la phrase d’entrée donnée. Ne donnez que la définition."
```

#### German
```python
"Geben Sie eine kurze Definition des {occurrence} Vorkommens des Wortes \"{target_word}\" in dem gegebenen Eingabesatz an. Erzeugen Sie nur die Definition."
```

#### Italian
```python
"Fornisci una breve definizione della {occurrence} occorrenza della parola \"{target_word}\" nella frase data in input. Genera solo la definizione."
```

#### Spanish
```python
"Proporciona una definición breve de la {occurrence} ocurrencia de la palabra \"{target_word}\" en la frase dada en entrada. Genera solo la definición."
```

### Single occurrence of target word (closed-ended)

#### English
```python
"Given the word \"{target_word}\" in the input sentence, choose the correct meaning from the following:\n{options}\n\nGenerate only the number of the selected option."
```

#### French
```python
"Étant donné le mot \"{target_word}\" dans la phrase saisie, choisissez la signification correcte parmi les suivantes:\n{options}\n\nNe donnez que le numéro de l’option sélectionnée."
```

#### German
```python
"Wählen Sie für das Wort \"{target_word}\" im Eingabesatz die richtige Bedeutung aus den folgenden Angaben:\n{options}\n\nErzeugt nur die Nummer der ausgewählten Option"
```

#### Italian
```python
"Data la parola \"{target_word}\" nella frase in input, scegli il significato corretto tra i seguenti:\n{options}\n\nGenera solo il numero dell'opzione selezionata."
```

#### Spanish
```python
"Dada la palabra \"{target_word}\" en la frase de entrada, elija el significado correcto entre los siguientes:\n{options}\n\nGenera solo el número de la opción seleccionada."
```

### Multiple occurrences of target word (closed-ended)

#### English
```python
"Given the word \"{target_word}\" in the input sentence, choose the correct meaning from the following:\n{options}\n\nGenerate only the number of the selected option."
```

#### French
```python
"Étant donné l'occurrence {occurrence} du mot \"{target_word}\" dans la phrase d'entrée, choisissez la signification correcte parmi les suivantes:\n{options}\n\nNe donnez que le numéro de l’option sélectionnée."
```

#### German
```python
"Wählen Sie angesichts des {occurrence} Vorkommens des Wortes \"{target_word}\" im Eingabesatz die richtige Bedeutung aus der folgenden Liste aus:\n{options}\n\nErzeugt nur die Nummer der ausgewählten Option."
```

#### Italian
```python
"Data la {occurrence} occorrenza della parola \"{target_word}\" nella frase in input, scegli il significato corretto tra i seguenti:\n{options}\n\nGenera solo il numero dell'opzione selezionata."
```

#### Spanish
```python
"Dada la {occurrence} ocurrencia de la palabra \"{target_word}\" en la frase de entrada, elije el significado correcto entre los siguientes:\n{options}\n\nGenera solo el número de la opción seleccionada."
```