Model Card for LLM-wsd-FT-10000

Model description

LLM-wsd-FT-10000 is a Large Language Model (LLM) instruction-tuned over meta-llama/Meta-Llama-3.1-8B-Instruct. This model has been trained for the WSD task over a balanced training dataset (10000 instances per language), without machine-translation. It is capable of providing the definition of a word in a given sentence. Specifically, it can answer both:

1. Open-ended questions, where the model will generate the definition of the target word;
2. Closed-ended questions, where the model will generate the identifier of the correct option out of a list of alternatives.

More details regarding the training procedure (e.g. hyperparameters, dataset construction, and so on) can be found in Section 4.2 of the paper.

• Developed by: Pierpaolo Basile, Lucia Siciliani, Elio Musacchio
• Model type: LLaMA 3.1 Instruct
• Language(s) (NLP): English, French, German, Italian and Spanish
• License: LLAMA 3.1 COMMUNITY LICENSE AGREEMENT
• Finetuned from model: meta-llama/Meta-Llama-3.1-8B-Instruct

Prompt Format

The model has been trained using several instructions depending on language, task (open-ended or closed-ended) and number of occurences of target word in the sentence. In Instructions, we provide the instructions used for all cases. The following placeholder variables have to be replaced:

• {target_word}: the target word in the input to disambiguate;
• {options}: options to provide to the model for the closed-ended task only. The options should be newline separated and each option should be identified by a number. Refer to the closed-ended example for an example of options formatting;
• {occurrence}: the ordinal number of the {target_word} occurrence (e.g. "second"). This is required only when the input sentence contains multiple occurrences of {target_word}.

Please note that the complete prompt also has the following string after the instruction:

" Input: \"{sentence}\""

where {sentence} is the input sentence containing the word to disambiguate.

How to Get Started with the Model

Below you can find two examples of model usage, for open-ended and closed-ended generation respectively.

Open-ended

import torch

from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers.trainer_utils import set_seed

target_word = "long"
instruction = f"Give a brief definition of the word \"{target_word}\" in the sentence given as input. Generate only the definition."
input_sentence = "How long has it been since you reviewed the objectives of your benefit and service program?"

model_id = "swap-uniba/LLM-wsd-FT-10000"

set_seed(42)

tokenizer = AutoTokenizer.from_pretrained(model_id, use_fast=False)

tokenizer.padding_side = "left"

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map='cuda',
    torch_dtype=torch.bfloat16,
).eval()

terminators = [
    tokenizer.eos_token_id,
    tokenizer.convert_tokens_to_ids("<|eot_id|>")
]

messages = [
    {"role": "user", "content": instruction + " Input: \"" + input_sentence + "\""},
]

input_ids = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")

outputs = model.generate(
    input_ids.to('cuda'),
    max_new_tokens=512,
    eos_token_id=terminators,
    num_beams=1,
    do_sample=False
)

print(tokenizer.decode(outputs[0][input_ids.shape[-1]:], skip_special_tokens=True))

Closed-ended

import torch

from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers.trainer_utils import set_seed

target_word = "hurry"
instruction = f"Given the word \"{target_word}\" in the input sentence, choose the correct meaning from the following:\n1) Move very fast\n2) Urge to an unnatural speed\n\nGenerate only the number of the selected option."
input_sentence = "If you hurry you might beat the headquarters boys."

model_id = "swap-uniba/LLM-wsd-FT-10000"

set_seed(42)

tokenizer = AutoTokenizer.from_pretrained(model_id, use_fast=False)

tokenizer.padding_side = "left"

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map='cuda',
    torch_dtype=torch.bfloat16,
).eval()

terminators = [
    tokenizer.eos_token_id,
    tokenizer.convert_tokens_to_ids("<|eot_id|>")
]

messages = [
    {"role": "user", "content": instruction + " Input: \"" + input_sentence + "\""},
]

input_ids = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")

outputs = model.generate(
    input_ids.to('cuda'),
    max_new_tokens=512,
    eos_token_id=terminators,
    num_beams=1,
    do_sample=False
)

print(tokenizer.decode(outputs[0][input_ids.shape[-1]:], skip_special_tokens=True))

Citation

If you use this model in your research, please cite the following:

@misc{basile2025exploringwordsensedisambiguation,
      title={Exploring the Word Sense Disambiguation Capabilities of Large Language Models}, 
      author={Pierpaolo Basile and Lucia Siciliani and Elio Musacchio and Giovanni Semeraro},
      year={2025},
      eprint={2503.08662},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2503.08662}, 
}

Instructions

Single occurrence of target word (open-ended)

English

"Give a brief definition of the word \"{target_word}\" in the sentence given as input. Generate only the definition."

French

"Donnez une brève définition du mot \"{target_word}\" dans la phrase d’entrée donnée. Ne donnez que la définition."

German

"Geben Sie eine kurze Definition des Wortes \"{target_word}\" in dem gegebenen Satz an. Erzeugen Sie nur die Definition."

Italian

"Fornisci una breve definizione della parola \"{target_word}\" nella frase data in input. Genera solo la definizione."

Spanish

"Proporciona una definición breve de la palabra \"{target_word}\" en la frase dada en entrada. Genera solo la definición."

Multiple occurences of target word (open-ended)

English

"Give a brief definition of the {occurrence} occurrence of the word \"{target_word}\" in the sentence given as input. Generate only the definition."

French

"Donnez une brève définition de l'occurrence {occurrence} du mot \"{target_word}\" dans la phrase d’entrée donnée. Ne donnez que la définition."

German

"Geben Sie eine kurze Definition des {occurrence} Vorkommens des Wortes \"{target_word}\" in dem gegebenen Eingabesatz an. Erzeugen Sie nur die Definition."

Italian

"Fornisci una breve definizione della {occurrence} occorrenza della parola \"{target_word}\" nella frase data in input. Genera solo la definizione."

Spanish

"Proporciona una definición breve de la {occurrence} ocurrencia de la palabra \"{target_word}\" en la frase dada en entrada. Genera solo la definición."

Single occurrence of target word (closed-ended)

English

"Given the word \"{target_word}\" in the input sentence, choose the correct meaning from the following:\n{options}\n\nGenerate only the number of the selected option."

French

"Étant donné le mot \"{target_word}\" dans la phrase saisie, choisissez la signification correcte parmi les suivantes:\n{options}\n\nNe donnez que le numéro de l’option sélectionnée."

German

"Wählen Sie für das Wort \"{target_word}\" im Eingabesatz die richtige Bedeutung aus den folgenden Angaben:\n{options}\n\nErzeugt nur die Nummer der ausgewählten Option"

Italian

"Data la parola \"{target_word}\" nella frase in input, scegli il significato corretto tra i seguenti:\n{options}\n\nGenera solo il numero dell'opzione selezionata."

Spanish

"Dada la palabra \"{target_word}\" en la frase de entrada, elija el significado correcto entre los siguientes:\n{options}\n\nGenera solo el número de la opción seleccionada."

Multiple occurrences of target word (closed-ended)

English

"Given the word \"{target_word}\" in the input sentence, choose the correct meaning from the following:\n{options}\n\nGenerate only the number of the selected option."

French

"Étant donné l'occurrence {occurrence} du mot \"{target_word}\" dans la phrase d'entrée, choisissez la signification correcte parmi les suivantes:\n{options}\n\nNe donnez que le numéro de l’option sélectionnée."

German

"Wählen Sie angesichts des {occurrence} Vorkommens des Wortes \"{target_word}\" im Eingabesatz die richtige Bedeutung aus der folgenden Liste aus:\n{options}\n\nErzeugt nur die Nummer der ausgewählten Option."

Italian

"Data la {occurrence} occorrenza della parola \"{target_word}\" nella frase in input, scegli il significato corretto tra i seguenti:\n{options}\n\nGenera solo il numero dell'opzione selezionata."

Spanish

"Dada la {occurrence} ocurrencia de la palabra \"{target_word}\" en la frase de entrada, elije el significado correcto entre los siguientes:\n{options}\n\nGenera solo el número de la opción seleccionada."