Update README.md

README.md

@@ -1,106 +1,106 @@
----
-license: cc-by-4.0
----
-# Whisper-Small-hindi
-
-This is a fine-tuned version of [openai/whisper-small](https://huggingface.co/openai/whisper-small), fine-tuned on the following datasets:
-- [Shrutilipi](https://ai4bharat.iitm.ac.in/datasets/shrutilipi) (AI4Bharat): Shrutilipi is a labelled ASR corpus obtained by mining parallel audio and text pairs at the document scale from All India Radio news bulletins for 12 Indian languages - Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Odia, Punjabi, Sanskrit, Tamil, Telugu, Urdu. The corpus has over 6400 hours of data across all languages. Out of which hindi is ~ 1600 hours
-- [IITM Madras SpringLab](https://asr.iitm.ac.in/dataset) (CC BY 4.0 License): This data was collected on payment basis using the following vendors -- Mediscribe India, Desicrew, and Crescendo. Preliminary checking of quality of transcriptions was done by our partners at KL University as well as by SPRING Lab members. The data consists mostly of mock conversations as well as monolgues on different topics.
-
-The model is trained on around 2500 hours of hindi speech & optimized for ASR tasks in hindi, with a particular focus on high-accuracy transcription.
-
-## How to use
-The Whisper model is intrinsically designed to work on audio samples of up to 30s in duration. However, by using a chunking algorithm, it can be used to transcribe audio samples of up to arbitrary length. This is possible through Transformers pipeline method. Chunking is enabled by setting chunk_length_s=30 when instantiating the pipeline. With chunking enabled, the pipeline can be run with batched inference. It can also be extended to predict sequence level timestamps by passing return_timestamps=True:
-
-```python
->>> import torch
->>> from transformers import pipeline
->>> from datasets import load_dataset
-
->>> device = "cuda:0" if torch.cuda.is_available() else "cpu"
-
->>> asr_pipe = pipe(
->>>     "automatic-speech-recognition",
->>>     model="collabora/whisper-small-hindi",
->>>     chunk_length_s=30,
->>>     device=device
->>> )
-
->>> ds = load_dataset("mozilla-foundation/common_voice_11_0", "hi", split="validation")
->>> sample = ds[0]["audio"]
->>> prediction = asr_pipe(sample.copy(), batch_size=8, return_timestamps=True)["text"]
-हमने उस उम्मीदवार को चुना।
-```
-
-## Intended Use
-- The model is designed for high quality transcription in Hindi.
-- And is suitable for academic use in ASR related tasks.
-
-## Limitations
-- May not perform well on noisy or low-quality audio.
-- Focused primarily on Hindi.
-
-### Model Performance
-Whisper Normalization is counter-productive for hindi since it takes the meaning out of a sentence for e.g. consider the hindi phrase:
-```
-'राजनीतिज्ञों ने कहा कि उन्होंने निर्णायक मत को अनावश्यक रूप से निर्धारित करने के लिए अफगान संविधान में काफी अस्पष्टता पाई थी'
-```
-
-After whisper normalization:
-```
-'र जन त जञ न कह क उनह न न रण यक मत क अन वशयक रप स न रध र त करन क ल ए अफग न सव ध न म क फ असपषटत प ई थ'
-```
-
-So, we use [indic-normalization](https://github.com/anoopkunchukuttan/indic_nlp_library/blob/4cead0ae6c78fe9a19a51ef679f586206df9c476/indicnlp/normalize/indic_normalize.py#L325) for evaluation. Indic-norm produces the below output:
-```
-'राजनीतिज्ञों ने कहा कि उन्होंने निर्णायक मत को अनावश्यक रूप से निर्धारित करने के लिए अफगान संविधान में काफी अस्पष्टता पाई थी'
-```
-
-`openai-whisper/small` baseline results on `google/fleurs -- hindi`:
-```
-Word Error Rate (WER) with whisper norm: 67.37 % 
-Word Error Rate (WER) with indic norm: 89.73 % 
-```
-
-The model achieves the following benchmarks on the held out test set `google/fleurs -- hindi`:
-```
-Word Error Rate (WER) with whisper norm: 10.11 % 
-Word Error Rate (WER) with indic norm: 17.35 % 
-```
-
-Indic normalization retains diacritics and complex characters in Hindi text, which can increase the Word Error Rate (WER) when compared to Whisper's default normalization but produces more semantically accurate transcriptions.
-
-### Acknowledgments
-
-We thank the contributors and organizations behind the datasets:
-
-- [AI4Bharat](https://ai4bharat.iitm.ac.in/datasets/shrutilipi) for the Shrutilipi dataset.
-
-- [IIT Madras SpringLab](https://asr.iitm.ac.in/dataset) for their springx-hindi dataset.
-
-- [IndicNLP Library](https://github.com/anoopkunchukuttan/indic_nlp_library) by Anoop Kunchukuttan for providing normalization tools that were crucial for evaluation.
-
-
-### BibTeX entry and citation info
-
-#### Model Citation
-```bibtex
-@misc{whisper-small-hindi,
-  title = {Whisper-Small Fine-Tuned on Hindi},
-  author = {Collabora Ltd.},
-  year = {2025},
-  publisher = {Hugging Face},
-  note = {Fine-tuned using Shrutilipi and IITM Madras SpringLab datasets},
-  url = {https://huggingface.co/collabora/whisper-small-hindi/}
-}
-```
-
-#### IndicNLP Library Citation
-```
-@misc{kunchukuttan2020indicnlp,
-author = "Anoop Kunchukuttan",
-title = "{The IndicNLP Library}",
-year = "2020",
-howpublished={\url{https://github.com/anoopkunchukuttan/indic_nlp_library/blob/master/docs/indicnlp.pdf}}
-}
-```
+---
+license: cc-by-4.0
+---
+# Whisper-Small-hindi
+
+This is a fine-tuned version of [openai/whisper-small](https://huggingface.co/openai/whisper-small), fine-tuned on the following datasets:
+- [Shrutilipi](https://ai4bharat.iitm.ac.in/datasets/shrutilipi) (AI4Bharat): Shrutilipi is a labelled ASR corpus obtained by mining parallel audio and text pairs at the document scale from All India Radio news bulletins for 12 Indian languages - Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Odia, Punjabi, Sanskrit, Tamil, Telugu, Urdu. The corpus has over 6400 hours of data across all languages. Out of which hindi is ~ 1600 hours
+- [IITM Madras SpringLab](https://asr.iitm.ac.in/dataset) (CC BY 4.0 License): This data was collected on payment basis using the following vendors -- Mediscribe India, Desicrew, and Crescendo. Preliminary checking of quality of transcriptions was done by our partners at KL University as well as by SPRING Lab members. The data consists mostly of mock conversations as well as monolgues on different topics.
+
+The model is trained on around 2500 hours of hindi speech & optimized for ASR tasks in hindi, with a particular focus on high-accuracy transcription.
+
+## How to use
+The Whisper model is intrinsically designed to work on audio samples of up to 30s in duration. However, by using a chunking algorithm, it can be used to transcribe audio samples of up to arbitrary length. This is possible through Transformers pipeline method. Chunking is enabled by setting chunk_length_s=30 when instantiating the pipeline. With chunking enabled, the pipeline can be run with batched inference. It can also be extended to predict sequence level timestamps by passing return_timestamps=True:
+
+```python
+>>> import torch
+>>> from transformers import pipeline
+>>> from datasets import load_dataset
+
+>>> device = "cuda:0" if torch.cuda.is_available() else "cpu"
+
+>>> asr_pipe = pipe(
+>>>     "automatic-speech-recognition",
+>>>     model="collabora/whisper-small-hindi",
+>>>     chunk_length_s=30,
+>>>     device=device
+>>> )
+
+>>> ds = load_dataset("mozilla-foundation/common_voice_11_0", "hi", split="validation")
+>>> sample = ds[0]["audio"]
+>>> prediction = asr_pipe(sample.copy(), batch_size=8, return_timestamps=True)["text"]
+हमने उस उम्मीदवार को चुना।
+```
+
+## Intended Use
+- The model is designed for high quality transcription in Hindi.
+- And is suitable for academic use in ASR related tasks.
+
+## Limitations
+- May not perform well on noisy or low-quality audio.
+- Focused primarily on Hindi.
+
+### Model Performance
+Whisper Normalization is counter-productive for hindi since it takes the meaning out of a sentence for e.g. consider the hindi phrase:
+```
+'राजनीतिज्ञों ने कहा कि उन्होंने निर्णायक मत को अनावश्यक रूप से निर्धारित करने के लिए अफगान संविधान में काफी अस्पष्टता पाई थी'
+```
+
+After whisper normalization:
+```
+'र जन त जञ न कह क उनह न न रण यक मत क अन वशयक रप स न रध र त करन क ल ए अफग न सव ध न म क फ असपषटत प ई थ'
+```
+
+So, we use [indic-normalization](https://github.com/anoopkunchukuttan/indic_nlp_library/blob/4cead0ae6c78fe9a19a51ef679f586206df9c476/indicnlp/normalize/indic_normalize.py#L325) for evaluation. Indic-norm produces the below output:
+```
+'राजनीतिज्ञों ने कहा कि उन्होंने निर्णायक मत को अनावश्यक रूप से निर्धारित करने के लिए अफगान संविधान में काफी अस्पष्टता पाई थी'
+```
+
+`openai-whisper/small` baseline results on `google/fleurs -- hindi`:
+```
+Word Error Rate (WER) with whisper norm: 67.37 % 
+Word Error Rate (WER) with indic norm: 89.73 % 
+```
+
+The model achieves the following benchmarks on the held out test set `google/fleurs -- hindi`:
+```
+Word Error Rate (WER) with whisper norm: 10.11 % 
+Word Error Rate (WER) with indic norm: 17.35 % 
+```
+
+Indic normalization retains diacritics and complex characters in Hindi text, which can increase the Word Error Rate (WER) when compared to Whisper's default normalization but produces more semantically accurate transcriptions.
+
+### Acknowledgments
+
+We thank the contributors and organizations behind the datasets:
+
+- [AI4Bharat](https://ai4bharat.iitm.ac.in/datasets/shrutilipi) for the Shrutilipi dataset.
+
+- [IIT Madras SpringLab](https://asr.iitm.ac.in/dataset) for their springx-hindi dataset.
+
+- [IndicNLP Library](https://github.com/anoopkunchukuttan/indic_nlp_library) by Anoop Kunchukuttan for providing normalization tools that were crucial for evaluation.
+
+
+### BibTeX entry and citation info
+
+#### Model Citation
+```bibtex
+@misc{whisper-small-hindi,
+  title = {Whisper-Small Fine-Tuned on Hindi},
+  author = {Collabora Ltd.},
+  year = {2025},
+  publisher = {Hugging Face},
+  note = {Fine-tuned using Shrutilipi and IITM Madras SpringLab datasets},
+  howpublished = {\url{https://huggingface.co/collabora/whisper-small-hindi/}},
+}
+```
+
+#### IndicNLP Library Citation
+```
+@misc{kunchukuttan2020indicnlp,
+author = "Anoop Kunchukuttan",
+title = "{The IndicNLP Library}",
+year = "2020",
+howpublished={\url{https://github.com/anoopkunchukuttan/indic_nlp_library/blob/master/docs/indicnlp.pdf}}
+}
+```