---
base_model: meta-llama/Llama-3.2-1B-Instruct
library_name: peft
license: apache-2.0
language:
- en
metrics:
- accuracy
tags:
- finance
- relation_extraction
- relation_types
---


## Model Description

<!-- Provide a longer summary of what this model is. -->


- **Finetuned from model:** Llama-3.2-1B-Instruct



## Downstream Use

Model for predicting relations between entities in the financial documents. 

### Relation Types
- no_relation
- title
- operations_in
- employee_of
- agreement_with
- formed_on
- member_of
- subsidiary_of
- shares_of
- revenue_of
- loss_of
- headquartered_in
- acquired_on
- founder_of
- formed_in

## How to Get Started with the Model

```python
import torch
from peft import AutoPeftModelForCausalLM
from transformers import AutoTokenizer, pipeline

# Load Model with PEFT adapter

finetune_name = 'Askinkaty/llama-finance-relations'

finetined_model = AutoPeftModelForCausalLM.from_pretrained(
    pretrained_model_name_or_path=finetune_name,
    torch_dtype=torch.float16,
    low_cpu_mem_usage=True,
)


base_model = "meta-llama/Llama-3.2-1B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_name)
base_model.config.pad_token_id = base_model.config.eos_token_id


pipeline = pipeline('text-generation', model=base_model, tokenizer=tokenizer)
pipeline.model = model.to(device) 


```


## Training Details

### Training Data

Samples from [ReFinD dataset](https://refind-re.github.io/). 100 examples for each relation type were used, least frequent relation types are omitted.


#### Preprocessing

Dataset is converted into a message format as in the code snippet below: 

```python
def batch_convert_to_messages(data):
        
    questions = data.apply(
        lambda x: f"Entity 1: {' '.join(x['token'][x['e1_start']:x['e1_end']])}. "
                  f"Entity 2: {' '.join(x['token'][x['e2_start']:x['e2_end']])}. "
                  f"Input sentence: {' '.join(x['token'])}",
        axis=1
    )

    relations = data['relation'].apply(lambda relation: relation.split(':')[-1])
    
    messages = [
        [
            {
                "role": "system",
                "content": "You are an expert in financial documentation and market analysis. Define relations between two specified entities: entity 1 [E1] and entity 2 [E2] in a sentence. Return a short response of the required format. "
            },
            {"role": "user", "content": question},
            {"role": "assistant", "content": relation},
        ]
        for question, relation in zip(questions, relations)
    ]
    
    return messages
```




#### Training Hyperparameters

SFT parameters:
- num_train_epochs=1
- per_device_train_batch_size=2
- gradient_accumulation_steps=2
- gradient_checkpointing=True
- optim="adamw_torch_fused"
- learning_rate=2e-4
- max_grad_norm=0.3
- warmup_ratio=0.01
- lr_scheduler_type="cosine"
- bf16=True

LORA parameters:
- rank_dimension = 6
- lora_alpha = 8
- lora_dropout = 0.05



## Evaluation

<!-- This section describes the evaluation protocols and provides the results. -->

### Testing Data, Factors & Metrics

#### Testing Data

Test set sampled from Samples from [ReFinD dataset](https://refind-re.github.io/).


#### Metrics

Accuracy. Other metrics: work in progress.

### Results

Accuracy before clearning the output: 0.41

Accuracy after clearning the output: 0.62


- PEFT 0.14.0