README.md

---
license: apache-2.0
datasets:
- Babelscape/multinerd
language:
- en
metrics:
- f1
- precision
- recall
pipeline_tag: token-classification
tags:
- ner
- named-entity-recognition
- token-classification
model-index:
- name: robert-base on MultiNERD by Jayant Yadav
  results:
  - task:
      type: named-entity-recognition-ner
      name: Named Entity Recognition
    dataset:
      type: Babelscape/multinerd
      name: MultiNERD (English)
      split: test
      revision: 2814b78e7af4b5a1f1886fe7ad49632de4d9dd25
      config: Babelscape/multinerd
      args:
        split: train[:50%]
    metrics:
    - type: f1
      value: 0.943
      name: F1
    - type: precision
      value: 0.939
      name: Precision
    - type: recall
      value: 0.947
      name: Recall
      config: seqeval
    paper: https://aclanthology.org/2022.findings-naacl.60.pdf
base_model: roberta-base
library_name: transformers
---
# Model Card for Model ID

[roBERTa-base](https://huggingface.co/roberta-base) model was fine-tuned on 50% training English only split of MultiNERD dataset and later evaluated on full test split of the same.   
The finetuning script can be fetched from [fintuning.py](https://github.com/jayant-yadav/RISE-NER/blob/main/finetuning.ipynb). 

Various other model were tested on the same selection of dataset and the best checkpoint was uploaded. The detailed configuration summary can be found in Appendix section of [report](https://github.com/jayant-yadav/RISE-NER/blob/main/MultiNERD_NER___RISE.pdf).


## Model Details

### Model Description

Head over to [github repo](https://github.com/jayant-yadav/RISE-NER) for all the scripts used to finetune and evalute token-classification model.
The model is ready to use on [Kaggle](https://www.kaggle.com/datasets/jayantyadav/multinerd-ner-models/) too!


- **Developed by:** Jayant Yadav

## Uses

Token-classification of the following entities are possible:  

| Class | Description | Examples |
|-------|-------------|----------|
PER (person) | People | Ray Charles, Jessica Alba, Leonardo DiCaprio, Roger Federer, Anna Massey. |
ORG (organization) | Associations, companies, agencies, institutions, nationalities and religious or political groups | University of Edinburgh, San Francisco Giants, Google, Democratic Party. |
LOC (location) | Physical locations (e.g. mountains, bodies of water), geopolitical entities (e.g. cities, states), and facilities (e.g. bridges, buildings, airports). | Rome, Lake Paiku, Chrysler Building, Mount Rushmore, Mississippi River. |
ANIM (animal) | Breeds of dogs, cats and other animals, including their scientific names. | Maine Coon, African Wild Dog, Great White Shark, New Zealand Bellbird. |
BIO (biological) | Genus of fungus, bacteria and protoctists, families of viruses, and other biological entities. | Herpes Simplex Virus, Escherichia Coli, Salmonella, Bacillus Anthracis. |
CEL (celestial) | Planets, stars, asteroids, comets, nebulae, galaxies and other astronomical objects. | Sun, Neptune, Asteroid 187 Lamberta, Proxima Centauri, V838 Monocerotis. |
DIS (disease) | Physical, mental, infectious, non-infectious, deficiency, inherited, degenerative, social and self-inflicted diseases. | Alzheimer’s Disease, Cystic Fibrosis, Dilated Cardiomyopathy, Arthritis. |
EVE (event) | Sport events, battles, wars and other events. | American Civil War, 2003 Wimbledon Championships, Cannes Film Festival. |
FOOD (food) | Foods and drinks. | Carbonara, Sangiovese, Cheddar Beer Fondue, Pizza Margherita. |
INST (instrument) | Technological instruments, mechanical instruments, musical instruments, and other tools. | Spitzer Space Telescope, Commodore 64, Skype, Apple Watch, Fender Stratocaster. |
MEDIA (media) | Titles of films, books, magazines, songs and albums, fictional characters and languages. | Forbes, American Psycho, Kiss Me Once, Twin Peaks, Disney Adventures. |
PLANT (plant) | Types of trees, flowers, and other plants, including their scientific names. | Salix, Quercus Petraea, Douglas Fir, Forsythia, Artemisia Maritima. |
MYTH (mythological) | Mythological and religious entities. | Apollo, Persephone, Aphrodite, Saint Peter, Pope Gregory I, Hercules. |
TIME (time) | Specific and well-defined time intervals, such as eras, historical periods, centuries, years and important days. No months and days of the week. | Renaissance, Middle Ages, Christmas, Great Depression, 17th Century, 2012. |
VEHI (vehicle) | Cars, motorcycles and other vehicles. | Ferrari Testarossa, Suzuki Jimny, Honda CR-X, Boeing 747, Fairey Fulmar.


## Bias, Risks, and Limitations

Only trained on English split of MultiNERD dataset. Therefore will not perform well on other languages.

## How to Get Started with the Model

Use the code below to get started with the model:

```py
from transformers import AutoTokenizer, AutoModelForTokenClassification
from transformers import pipeline

tokenizer = AutoTokenizer.from_pretrained("jayant-yadav/roberta-base-multinerd")
model = AutoModelForTokenClassification.from_pretrained("jayant-yadav/roberta-base-multinerd")

nlp = pipeline("ner", model=model, tokenizer=tokenizer)
example = "My name is Wolfgang and I live in Berlin"

ner_results = nlp(example)
print(ner_results)
```

## Training Details

### Training Data

50% of train split of MultiNERD dataset was used to finetune the model.

### Training Procedure 


#### Preprocessing 

English dataset was filterd out : ```train_dataset = train_dataset.filter(lambda x: x['lang'] == 'en')```


#### Training Hyperparameters

The following hyperparameters were used during training:  

learning_rate: 5e-05  
train_batch_size: 32  
eval_batch_size: 32  
seed: 42  
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08  
lr_scheduler_type: linear  
lr_scheduler_warmup_ratio: 0.1  
num_epochs: 1  


## Evaluation

Evaluation was perfored on 50% of evaluation split of MultiNERD dataset.

### Testing Data & Metrics


#### Testing Data

Tested on Full test split of MultiNERD dataset.


#### Metrics
Model versions and checkpoint were evaluated using F1, Precision and Recall.  
For this `seqeval` metric was used: ```metric = load_metric("seqeval")```.

### Results

|Entity | Precision | Recall | F1 score | Support |
|---|---|---|---|---|
|ANIM | 0.71 | 0.77 | 0.739 | 1604 |
|BIO | 0.5 | 0.125 | 0.2 | 8 |
|CEL | 0.738 | 0.756 | 0.746 | 41 |
|DIS | 0.737 | 0.772 | 0.754 | 759 |
|EVE | 0.952 | 0.968 | 0.960 | 352 |
|FOOD | 0.679 | 0.545 | 0.605 | 566 |
|INST | 0.75 | 0.75 | 0.75 | 12 |
|LOC | 0.994 | 0.991 | 0.993 | 12024 |
|MEDIA | 0.940 | 0.969 | 0.954 | 458 |
|ORG | 0.977 | 0.981 | 0.979 | 3309 |
|PER | 0.992 | 0.995 | 0.993 | 5265 |
|PLANT | 0.617 | 0.730 | 0.669 | 894 |
|MYTH | 0.647 | 0.687 | 0.666 | 32 |
|TIME | 0.825 | 0.820 | 0.822 | 289 |
|VEHI | 0.812 | 0.812 | 0.812 | 32 |
|**Overall** | **0.939** | **0.947** | **0.943** |

## Technical Specifications

### Model Architecture and Objective
Follows the same as RoBERTa-BASE

### Compute Infrastructure

#### Hardware

Kaggle - GPU T4x2  
Google Colab - GPU T4x1

#### Software
pandas==1.5.3  
numpy==1.23.5  
seqeval==1.2.2  
datasets==2.15.0  
huggingface_hub==0.19.4  
transformers[torch]==4.35.2  
evaluate==0.4.1  
matplotlib==3.7.1  
collections  
torch==2.0.0  

## Model Card Contact
[jayant-yadav](https://huggingface.co/jayant-yadav)