README.md

---
license: mit
tags:
- generated_from_trainer
metrics:
- bleu
model-index:
- name: iva_mt_wslot-m2m100_418M-0.1.0
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# iva_mt_wslot-m2m100_418M-0.1.0 en-pl

This model is a fine-tuned version of [facebook/m2m100_418M](https://huggingface.co/facebook/m2m100_418M) on the [iva_mt_wslot](https://huggingface.co/datasets/cartesinus/iva_mt_wslot) dataset.


It achieves the following results:
1) On the test set (iva_mt):
- Bleu (plain text): 39.1560
- Bleu (with slots): 63.8767
- Baseline m2m100-418M (plain text): TBD

2) WMT20 (en2pl):
- Bleu (lowercased, tags removed): 15.0863 (for reference WMT20 submission systems in en-pl direction had between 25 and 30 bleu)
- Baseline m2m100-418M (plain text): TBD
   
4) On the evaluation set:
- Bleu: 61.6249

3) On the training set (to see how it adjusted to train):
- Bleu (plain text): 70.5597
- Bleu (with slots): 93.8200

Bleu was measured with (sacrebleu)[https://github.com/mjpost/sacrebleu] library.

## Model description, intended uses & limitations

Model is biased towards virtual assistant (IVA) sentences in prediction/translation. These sentences are short, most of them are short, imperatives. Both of this facts
can be observed in above results. WMT results are very low while in-domain test is very high. One thing that needs to be mentioned is that BLEU is not particulary good
metric to evaluate IVA sentences due to their length and it should be evalued with other metrices (e.g. [GLEU](https://aclanthology.org/P15-2097.pdf)).

This model will most probably force IVA translations on your text. As long as sentences that you are translating are more or less similar to massive and leyzer domains it
can work. If you will translate domains unseen in either of them results might drop significantly to the point where baseline m2m100-418M will be better than this model.

This model will generate tags in output even if there is not tag in input sentence. Frequency of this depends on input text. When testing IVA utterances this occurs 
between 3 and 5% of all cases. When WMT20 was translated it happened in % cases (input text was from News domain).
This is not very severe and can be fixed easily in post-processing (something like `sed 's/<[a-z]>//g'` should be enough in most cases).

Translations with slots very often differ from same sentences when slots are removed. This is quite frequent and it happens between 30 and 50% of translated utterances.
For example there will be a difference between "is it raining in barcelona" and "is it raining in <a>barcelona<a>". In second case model will more likely localize name of
city to some Polish name (here Lublin, because such city was given in Massive train set). This might be useful if you want to generate more variants.

## How to use

First please make sure to install `pip install transformers`. First download model: 

```python
from transformers import M2M100ForConditionalGeneration, M2M100Tokenizer
import torch

def translate(input_text, lang):
    input_ids = tokenizer(input_text, return_tensors="pt")
    generated_tokens = model.generate(**input_ids, forced_bos_token_id=tokenizer.get_lang_id(lang))
    return tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)

model_name = "cartesinus/iva_mt_wslot-m2m100_418M-0.1.0"
tokenizer = M2M100Tokenizer.from_pretrained(model_name, src_lang="en", tgt_lang="pl")
model = M2M100ForConditionalGeneration.from_pretrained(model_name)
```

Then you can translate either plan text like this:
```python
print(translate("set the temperature on my thermostat", "pl"))
```
or you can translate with slot annotations that will be restored in tgt language:
```python
print(translate("wake me up at <a>nine am<a> on <b>friday<b>", "pl")) #translation: obudź mnie o <a>piątej rano<a> <b>w tym tygodniu<b>
```
Limitations of translation with slot transfer:
1) Annotated words must be placed between semi-xml tags like this "this is <a>example<a>"
2) There is no closing tag for example "<\a>" in above example - this is done on purpose to ommit problems with backslash escape
3) If sentence consists of more than one slot then simply use next alphabet letter. For example "this is <a>example<a> with more than <b>one<b> slot"
4) Please do not add space before first or last annotated word because this particular model was trained this way and it most probably will lower it's results 


## Training and evaluation data

## Dataset Composition (en-pl)
| Corpus                                                               | Train  | Dev   | Test  |
|----------------------------------------------------------------------|--------|-------|-------|
| [Massive 1.1](https://huggingface.co/datasets/AmazonScience/massive) | 11514  | 2033  | 2974  |
| [Leyzer 0.2.0](https://github.com/cartesinus/leyzer/tree/0.2.0)      | 3974   | 701   | 1380  |
| [OpenSubtitles from OPUS](https://opus.nlpl.eu/OpenSubtitles-v1.php) | 2329   | 411   | 500   |
| [KDE from OPUS](https://opus.nlpl.eu/KDE4.php)                       | 1154   | 241   | 241   |
| [CCMatrix from Opus](https://opus.nlpl.eu/CCMatrix.php)              | 1096   | 232   | 237   |
| [Ubuntu from OPUS](https://opus.nlpl.eu/Ubuntu.php)                  | 281    | 60    | 59    |
| [Gnome from OPUS](https://opus.nlpl.eu/GNOME.php)                    | 14     | 3     | 3     |
| *total*                                                              | 20362  | 3681  | 5394  |

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 4
- eval_batch_size: 4
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 10
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step  | Validation Loss | Bleu    | Gen Len |
|:-------------:|:-----:|:-----:|:---------------:|:-------:|:-------:|
| 0.0178        | 1.0   | 5091  | 0.0171          | 57.4439 | 21.1396 |
| 0.013         | 2.0   | 10182 | 0.0159          | 58.886  | 21.2285 |
| 0.0091        | 3.0   | 15273 | 0.0157          | 60.159  | 21.1222 |
| 0.0073        | 4.0   | 20364 | 0.0159          | 60.5893 | 21.1212 |
| 0.0054        | 5.0   | 25455 | 0.0161          | 60.6484 | 21.0679 |
| 0.004         | 6.0   | 30546 | 0.0166          | 61.5283 | 21.0875 |
| 0.0031        | 7.0   | 35637 | 0.0169          | 61.0439 | 21.1562 |
| 0.0024        | 8.0   | 40728 | 0.0172          | 61.9427 | 21.2203 |
| 0.0018        | 9.0   | 45819 | 0.0175          | 61.7325 | 21.1478 |
| 0.0014        | 10.0  | 50910 | 0.0176          | 61.6249 | 21.157  |


### Framework versions

- Transformers 4.26.1
- Pytorch 1.13.1+cu116
- Datasets 2.10.1
- Tokenizers 0.13.2