|
--- |
|
language: ru |
|
tags: |
|
- text |
|
- PyTorch |
|
- Transformers |
|
license: apache-2.0 |
|
widget: |
|
- text: уласны в москве интерне только в большом году что лепровели |
|
pipeline_tag: text2text-generation |
|
--- |
|
|
|
# ruT5-ASR |
|
|
|
Model was trained by [bond005](https://research.nsu.ru/en/persons/ibondarenko) to correct errors in the ASR output (in particular, output of [Wav2Vec2-Large-Ru-Golos](https://huggingface.co/bond005/wav2vec2-large-ru-golos)). The model is based on [ruT5-base](https://huggingface.co/ai-forever/ruT5-base). |
|
|
|
## Usage |
|
|
|
To correct ASR outputs the model can be used as a standalone sequence-to-sequence model as follows: |
|
|
|
```python |
|
from transformers import T5ForConditionalGeneration, T5Tokenizer |
|
import torch |
|
|
|
|
|
def rescore(text: str, tokenizer: T5Tokenizer, |
|
model: T5ForConditionalGeneration) -> str: |
|
if len(text) == 0: # if an input text is empty, then we return an empty text too |
|
return '' |
|
ru_letters = set('аоуыэяеёюибвгдйжзклмнпрстфхцчшщьъ') |
|
punct = set('.,:/\\?!()[]{};"\'-') |
|
x = tokenizer(text, return_tensors='pt', padding=True).to(model.device) |
|
max_size = int(x.input_ids.shape[1] * 1.5 + 10) |
|
min_size = 3 |
|
if x.input_ids.shape[1] <= min_size: |
|
return text # we don't rescore a very short text |
|
out = model.generate(**x, do_sample=False, num_beams=5, |
|
max_length=max_size, min_length=min_size) |
|
res = tokenizer.decode(out[0], skip_special_tokens=True).lower().strip() |
|
res = ' '.join(res.split()) |
|
postprocessed = '' |
|
for cur in res: |
|
if cur.isspace() or (cur in punct): |
|
postprocessed += ' ' |
|
elif cur in ru_letters: |
|
postprocessed += cur |
|
return (' '.join(postprocessed.strip().split())).replace('ё', 'е') |
|
|
|
|
|
# load model and tokenizer |
|
tokenizer_for_rescoring = T5Tokenizer.from_pretrained('bond005/ruT5-ASR') |
|
model_for_rescoring = T5ForConditionalGeneration.from_pretrained('bond005/ruT5-ASR') |
|
if torch.cuda.is_available(): |
|
model_for_rescoring = model_for_rescoring.cuda() |
|
|
|
input_examples = [ |
|
'уласны в москве интерне только в большом году что лепровели', |
|
'мороз и солнце день чудесный', |
|
'нейро сети эта харошо', |
|
'да' |
|
] |
|
|
|
for src in input_examples: |
|
rescored = rescore(src, tokenizer_for_rescoring, model_for_rescoring) |
|
print(f'{src} -> {rescored}') |
|
``` |
|
|
|
```text |
|
уласны в москве интерне только в большом году что лепровели -> у нас в москве интернет только в прошлом году что ли провели |
|
мороз и солнце день чудесный -> мороз и солнце день чудесный |
|
нейро сети эта харошо -> нейросети это хорошо |
|
да -> да |
|
``` |
|
|
|
## Evaluation |
|
This model was evaluated on the test subsets of [SberDevices Golos](https://huggingface.co/datasets/SberDevices/Golos), [Common Voice 6.0](https://huggingface.co/datasets/common_voice) (Russian part), and [Russian Librispeech](https://huggingface.co/datasets/bond005/rulibrispeech), but it was trained on the training subset of SberDevices Golos only. You can see the evaluation script on other datasets, including Russian Librispeech and SOVA RuDevices, on my Kaggle web-page https://www.kaggle.com/code/bond005/wav2vec2-t5-ru-eval |
|
|
|
*Comparison with "pure" Wav2Vec2-Large-Ru-Golos (WER, %)*: |
|
|
|
| dataset name | pure ASR | ASR with rescoring | |
|
|---------------------|----------|--------------------| |
|
| Voxforge Ru | **27.08** | 40.48 | |
|
| Russian LibriSpeech | **21.87** | 23.77 | |
|
| Sova RuDevices | 25.41 | **20.13** | |
|
| Golos Crowd | 10.14 | **9.42** | |
|
| Golos Farfield | 20.35 | **17.99** | |
|
| CommonVoice Ru | 18.55 | **11.60** | |
