README.md

---
license: mit
language:
- fr
library_name: transformers
tags:
- Biomedical
- Medical
- French-Biomedical
Mask token:
- [MASK]
widget:
- text: "A l’admission, l’examen clinique mettait en évidence : - une hypotension artérielle avec une pression [MASK] à 6 mmHg."
  example_title: "Example 1"
- text: "Le patient a été diagnostiqué avec une [MASK] lobaire  aiguë et a été traité avec des antibiotiques appropriés"
  example_title: "Example 2"
- text: "En mars 2001, le malade fut opéré, mais vu le caractère hémorragique de la tumeur, une simple biopsie surrénalienne a été réalisée ayant montré l’aspect de [MASK] malin non Hodgkinien de haut grade de malignité."
  example_title: "Example 3"
- text: "La cytologie urinaire n’a mis en évidence que des cellules [MASK] normales et l’examen cyto-bactériologique des urines était stérile."
  example_title: "Example 4"
- text: "La prise de greffe a été systématiquement réalisée au niveau de la face interne de la [MASK] afin de limiter la plaie cicatricielle."
  example_title: "Example 5"
---

# quinten-datalab/AliBERT-7GB: AliBERT: is a pre-trained language model for French biomedical text. 


# Introduction

AliBERT: is a pre-trained language model for French biomedical text. It is trained with masked language model like RoBERTa.

Here are the main contributions of our work:
  <ul>
  <li>
    A French biomedical language model, a language-specific and domain-specific PLM, which can be used to represent French biomedical text for different downstream tasks.
  </li>
  <li>
    A normalization of a Unigram sub-word tokenization of French biomedical textual input which improves our vocabulary and overall performance of the models trained.
  </li>
  <li>
      It is a foundation model that achieved state-of-the-art results on French biomedical text.
  </li>
  </ul>

The Paper can be found here: https://aclanthology.org/2023.bionlp-1.19/ 

# Data
The pre-training corpus was gathered from different sub-corpora.It is composed of 7GB French biomedical textual documents. Here are the sources used.

|Dataset name| Quantity| Size |
|----|---|---|
|Drug leaflets (Base de données publique des médicament)| 23K| 550Mb |
|RCP (a French equivalent of Physician’s Desk Reference)| 35K| 2200Mb|
|Articles (biomedical articles from ScienceDirect)| 500K| 4300Mb |
|Thesis (Thesis manuscripts in French)| 300K|300Mb |
|Cochrane (articles from Cochrane database)| 7.6K| 27Mb|
*Table 1: Pretraining dataset*

# How to use alibert-quinten/Oncology-NER with HuggingFace

Load quinten-datalab/AliBERT-7GB fill-mask model and the tokenizer used to train AliBERT:

```python
from transformers import AutoTokenizer, AutoModelForTokenClassification,pipeline

tokenizer = AutoTokenizer.from_pretrained("quinten-datalab/AliBERT-7GB")

model = AutoModelForTokenMaskedLM.from_pretrained("quinten-datalab/AliBERT-7GB")

fill_mask=pipeline("fill-mask",model=model,tokenizer=tokenizer)
nlp_AliBERT=fill_mask("La prise de greffe a été systématiquement réalisée au niveau de la face interne de la [MASK] afin de limiter la plaie cicatricielle.")

[{'score': 0.7724128365516663,
  'token': 6749,
  'token_str': 'cuisse',
  'sequence': 'La prise de greffe a été systématiquement réalisée au niveau de la face interne de la cuisse afin de limiter la plaie cicatricielle.'},
 {'score': 0.09472355246543884,
  'token': 4915,
  'token_str': 'jambe',
  'sequence': 'La prise de greffe a été systématiquement réalisée au niveau de la face interne de la jambe afin de limiter la plaie cicatricielle.'},
 {'score': 0.03340734913945198,
  'token': 2050,
  'token_str': 'main',
  'sequence': 'La prise de greffe a été systématiquement réalisée au niveau de la face interne de la main afin de limiter la plaie cicatricielle.'},
 {'score': 0.030924487859010696,
  'token': 844,
  'token_str': 'face',
  'sequence': 'La prise de greffe a été systématiquement réalisée au niveau de la face interne de la face afin de limiter la plaie cicatricielle.'},
 {'score': 0.012518334202468395,
  'token': 3448,
  'token_str': 'joue',
  'sequence': 'La prise de greffe a été systématiquement réalisée au niveau de la face interne de la joue afin de limiter la plaie cicatricielle.'}]
```

# Metrics and results
The model has been evaluted in the following downstream tasks

## Biomedical Named Entity Recognition (NER)
The model is evaluated on two (CAS and QUAERO) publically available Frech biomedical text.
#### CAS dataset

<style type="text/css">
.tg  {border-collapse:collapse;border-spacing:0;}
.tg .tg-baqh{text-align:center;vertical-align:top}
.tg .tg-0lax{text-align:center;vertical-align:top}
</style>
<table class="tg">
<thead>
  <tr>
    <th>Models</th>
    <th class="tg-0lax" colspan="3">CamemBERT</th>
    <th class="tg-0lax" colspan="3">AliBERT</th>
    <th class="tg-0lax" colspan="3">DrBERT</th>
  </tr>
</thead>
<tbody>
  <tr>
    <td>Entities</td>
    <td>P<br></td>
    <td>R</td>
    <td>F1</td>
    <td>P<br></td>
    <td>R</td>
    <td>F1</td>
    <td>P<br></td>
    <td>R</td>
    <td>F1</td>
  </tr>
  <tr>
    <td>Substance</td>
    <td>0.96</td>
    <td>0.87</td>
    <td>0.91</td>
    <td>0.96</td>
    <td>0.91</td>
    <td>0.93</td>
    <td>0.95</td>
    <td>0.91</td>
    <td>0.93</td>
  </tr>
  <tr>
    <td>Symptom</td> <td>0.89</td> <td>0.91</td> <td>0.90</td> <td>0.96</td> <td>0.98</td> <td>0.97</td> <td>0.94</td> <td>0.98</td> <td>0.96</td>
  </tr>
  <tr>
    <td>Anatomy</td> <td>0.94</td> <td>0.91</td> <td>0.88</td> <td>0.97</td> <td>0.97</td> <td>0.98</td> <td>0.96</td> <td>0.97</td> <td>0.96 </td>
  </tr>
  <tr>
    <td>Value</td> <td>0.88</td> <td>0.46</td> <td>0.60</td> <td>0.98</td> <td>0.99</td> <td>0.98</td> <td>0.93</td> <td>0.93</td> <td>	0.93</td>
  </tr>
  <tr>
   <td> Pathology</td> <td>0.79</td> <td>0.70</td> <td>0.74</td> <td>0.81</td> <td>0.39</td> <td>0.52</td> <td>0.85	<td>0.57</td> <td>0.68</td>
  </tr>
  <tr>
    <td>Macro Avg</td> <td>0.89  </td> <td>0.79</td> <td>0.81</td> <td> 0.94</td> <td>0.85</td> <td>0.88</td> <td> 0.92</td> <td> 0.87</td> <td>0.89</td>  
  </tr>
</tbody>
</table>
Table 2: NER performances on CAS dataset

#### QUAERO dataset

<table class="tg">
<thead>
  <tr>
    <th>Models</th>
    <th class="tg-0lax" colspan="3">CamemBERT</th>
    <th class="tg-0lax" colspan="3">AliBERT</th>
    <th class="tg-0lax" colspan="3">DrBERT</th>
  </tr>
</thead>
<tbody>
<tr>
<td>Entity </td> <td>  P </td> <td> R </td> <td> F1 </td> <td> P </td> <td> R </td> <td> F1 </td> <td>  P </td> <td> R </td> <td> F1 </td>
</tr>
<tr>
<td>Anatomy </td> <td>  0.649 </td> <td> 0.641 </td> <td> 0.645 </td> <td>  0.795 </td> <td> 0.811 </td> <td> 0.803 </td> <td> 0.799 </td> <td>   0.801 </td> <td> 0.800  </td>
</tr>
<tr>
<td>Chemical </td> <td>  0.844 </td> <td> 0.847 </td> <td> 0.846 </td> <td>  0.878 </td> <td> 0.893 </td> <td> 0.885 </td> <td>  0.898 </td> <td>   0.818 </td> <td> 0.856  </td>
</tr>
<tr>
<td>Device </td> <td>  0.000 </td> <td> 0.000 </td> <td> 0.000 </td> <td>  0.506 </td> <td> 0.356 </td> <td> 0.418 </td> <td>  0.549 </td> <td>   0.338 </td> <td> 0.419}  </td>
</tr>
<tr>
<td>Disorder </td> <td>  0.772 </td> <td> 0.818 </td> <td> 0.794 </td> <td>  0.857 </td> <td> 0.843 </td> <td> 0.850 </td> <td>  0.883 </td> <td>   0.809 </td> <td> 0.845  </td>
</tr>
<tr>
<td>Procedure </td> <td>  0.880 </td> <td> 0.894 </td> <td> 0.887 </td> <td>  0.969 </td> <td> 0.967 </td> <td> 0.968 </td> <td>  0.944 </td> <td>  0.976 </td> <td> 0.960  </td>
</tr>
<tr>
<td>Macro Avg </td> <td>  0.655 </td> <td> 0.656 </td> <td> 0.655 </td> <td>  0.807 </td> <td> 0.783 </td> <td> 0.793 </td> <td>  0.818 </td> <td>  0.755 </td> <td> 0.782  </td>
</tr>
</tbody>
</table>
Table 3: NER performances on QUAERO dataset

##AliBERT: A Pre-trained Language Model for French Biomedical Text