|
---
|
|
license: mit
|
|
language:
|
|
- en
|
|
pipeline_tag: sentence-similarity
|
|
datasets:
|
|
- darrow-ai/LegalLensNLI
|
|
metrics:
|
|
- f1
|
|
base_model:
|
|
- ynie/roberta-large-snli_mnli_fever_anli_R1_R2_R3-nli
|
|
library_name: transformers
|
|
---
|
|
# roberta_cnn_legal |
|
|
|
## Overview |
|
This repository hosts the uOttawa model developed for Subtask B (Legal Natural Language Inference) in the LegalLens-2024 shared task. The task focuses on classifying relationships between legal texts, such as determining if a premise (e.g., a summary of a legal complaint) entails, contradicts, or is neutral with respect to a hypothesis (e.g., an online review). |
|
## Model Details |
|
- **Model Type**: Transformer-based model combined with a Convolutional Neural Network (CNN) |
|
|
|
- **Framework**: PyTorch, Transformers library |
|
|
|
- **Training Data**: LegalLensNLI dataset provided by the LegalLens-2024 organizers |
|
|
|
- **Architecture**: Integration of RoBERTa (ynie/roberta-large-snli_mnli_fever_anli_R1_R2_R3-nli) with a custom CNN for keyword pattern detection |
|
|
|
- **Use Case**: Classifying relationships between legal documents for applications like legal case matching and automated reasoning |
|
|
|
## Model Architecture |
|
The model architecture consists of: |
|
|
|
- **RoBERTa model**: Responsible for capturing contextual information from the input text. |
|
|
|
- **CNN model**: Used for keyword detection, including an embedding layer and three convolutional layers with filter sizes (2, 3, 4). |
|
|
|
- **Fully connected layer**: Combines the outputs from RoBERTa and CNN for the final classification. |
|
|
|
## Installation |
|
To use this model, clone this repository and make sure to have the following installed: |
|
|
|
```bash |
|
pip install torch |
|
pip install transformers |
|
``` |
|
## Quick Start |
|
Load the model and run inference using the Hugging Face Transformers library: |
|
|
|
```code |
|
from transformers import AutoTokenizer, AutoModelForSequenceClassification |
|
|
|
# Load the model and tokenizer |
|
model = AutoModelForSequenceClassification.from_pretrained("nimamegh/roberta_cnn_legal") |
|
tokenizer = AutoTokenizer.from_pretrained("nimamegh/roberta_cnn_legal") |
|
|
|
# Example inputs |
|
premise = "The cat is on the mat." |
|
hypothesis = "The animal is on the mat." |
|
inputs = tokenizer(premise, hypothesis, return_tensors='pt') |
|
|
|
# Get predictions |
|
outputs = model(**inputs) |
|
predictions = outputs.logits.argmax(dim=-1) |
|
|
|
# Print the prediction result |
|
print("Predicted class:", predictions.item()) |
|
|
|
# Interpretation (optional) |
|
label_map = {0: "Entailment", 1: "Neutral", 2: "Contradiction"} |
|
print("Result:", label_map[predictions.item()]) |
|
``` |
|
|
|
## Training Configuration |
|
|
|
- Learning Rate: 2e-5 |
|
|
|
- Batch Size: 4 (train and evaluation) |
|
|
|
- Number of Epochs: 20 |
|
|
|
- Weight Decay: 0.01 |
|
|
|
- Optimizer: AdamW |
|
|
|
- Trainer Class: Used for fine-tuning with early stopping and warmup steps |
|
|
|
## Evaluation Metrics |
|
The model was evaluated using an F1-score across multiple domains in the validation set: |
|
|
|
- Average F1-score: 88.6% |
|
|
|
## Result |
|
|
|
- Performance on Hidden Test Set: F1-score of 0.724, achieving 5th place in the LegalLens-2024 competition. |
|
|
|
- Comparison: |
|
|
|
- Falcon 7B: 81.02% (average across domains) |
|
|
|
- RoBERTa base: 71.02% (average) |
|
|
|
- uOttawa Model: 88.6% (average on validation) |
|
|
|
## Citation |
|
|
|
```bibtex |
|
@misc{meghdadi2024uottawalegallens2024transformerbasedclassification, |
|
title={uOttawa at LegalLens-2024: Transformer-based Classification Experiments}, |
|
author={Nima Meghdadi and Diana Inkpen}, |
|
year={2024}, |
|
eprint={2410.21139}, |
|
archivePrefix={arXiv}, |
|
primaryClass={cs.CL}, |
|
url={https://arxiv.org/abs/2410.21139}, |
|
} |
|
``` |
