File size: 3,184 Bytes
41a7ee2 ec764cd 41a7ee2 7dfbbb5 41a7ee2 29d5001 41a7ee2 1f97423 41a7ee2 543eb13 41a7ee2 543eb13 41a7ee2 543eb13 41a7ee2 543eb13 41a7ee2 29d5001 41a7ee2 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 |
---
license: openrail
datasets:
- universal-dependencies/universal_dependencies
language:
- bg
- ca
- zh
- hr
- cs
- da
- nl
- en
- fi
- fr
- de
- hu
- id
- it
- ja
- ko
- 'no'
- pl
- pt
- ro
- ru
- sl
- es
- sv
- uk
- sr
base_model:
- meta-llama/Llama-2-13b-hf
library_name: peft
---
# Introduction
The paper explores the capabilities of Large Language Models (LLMs) like LLaMA in syntactic parsing tasks. We introduce U-DepPLLaMA, a novel architecture that treats Dependency Parsing as a sequence-to-sequence problem, achieving state-of-the-art results in 26 languages from the Universal Dependency Treebank. Our approach demonstrates that LLMs can handle dependency parsing without the need for specialized architectures, showing robust performance even with complex sentence structures. The paper is available [here](https://www.ai-lc.it/wp-content/uploads/2024/08/IJCOL_10_1_2_hromei_et_al.pdf).
For more details, please consult the associated [Github repository](https://github.com/crux82/u-deppllama).
This model comes in two sizes:
- [13 billion parameters](https://huggingface.co/sag-uniroma2/u-depp-llama-2-13b)
- [7 billion parameters](https://huggingface.co/sag-uniroma2/u-depp-llama-2-7b)
# How to use it
```Python
import transformers
import torch
from transformers import BitsAndBytesConfig, AutoModelForCausalLM, AutoTokenizer, GenerationConfig
from peft import PeftModel
quant_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_use_double_quant=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16
)
model = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-2-13b-hf",
load_in_4bit=True,
quantization_config=quant_config,
torch_dtype=torch.float16,
trust_remote_code=True,
device_map={"": 0},
)
model = PeftModel.from_pretrained(
model,
"sag-uniroma2/u-depp-llama-2-13b"
)
generation_config = GenerationConfig(
num_beams=4,
do_sample=False,
early_stopping=True,
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-13b-hf", trust_remote_code=True)
input_string = "He was most widely recognized for some of his books."
prompt = f"""
### Input:
{input_string}
### Answer:"""
inputs = tokenizer(prompt, return_tensors="pt", padding=True, truncation=True, max_length=512)
input_ids = inputs["input_ids"].to(model.device)
with torch.no_grad():
gen_outputs = model.generate(
input_ids=input_ids,
generation_config=generation_config,
return_dict_in_generate=True,
output_scores=True,
max_new_tokens=1024,
use_cache=True,
)
s = gen_outputs.sequences[0]
output = tokenizer.decode(s, skip_special_tokens=True)
response = output.split("### Answer:")[1].rstrip().lstrip()
print(response)
```
# Citation
```
@article{hromei2024udeppllama,
author = "Hromei, Claudiu Daniel and Croce, Danilo and Basili, Roberto",
title = "U-DepPLLaMA: Universal Dependency Parsing via Auto-regressive Large Language Models",
journal = "IJCoL",
year = 2024,
volume = "10",
number = "1",
pages = "21--38"
}
``` |