README.md

---
license: mit
datasets:
- vrclc/dakshina-lexicons-ml
- vrclc/Dakshina-romanized-ml
- vrclc/Aksharantar-ml
language:
- ml
metrics:
- cer
- wer
- bleu
pipeline_tag: text2text-generation
model-index:
- name: Malayalam Transliteration
  results:
  - task:
      type: text2text-generation
      name: Transliteration
    dataset:
      name: IndoNLP Test -1 
      type: vrclc/IndoNLP-1
      split: test
      args: ml
    metrics:
    - type: cer
      value: 7.4
      name: CER
  - task:
      type: text2text-generation
      name: Transliteration
    dataset:
      name: IndoNLP Test -1 
      type: vrclc/IndoNLP-1
      split: test
      args: ml
    metrics:
    - type: wer
      value: 34.5
      name: WER  
  - task:
      type: text2text-generation
      name: Transliteration
    dataset:
      name: IndoNLP Test -1 
      type: vrclc/IndoNLP-1
      split: test
      args: ml
    metrics:
    - type: bleu
      value: 32.7
      name: BLEU
  - task:
      type: text2text-generation
      name: Transliteration
    dataset:
      name: IndoNLP Test -2 
      type: vrclc/IndoNLP-2
      split: test
      args: ml
    metrics:
    - type: cer
      value: 22.7
      name: CER
  - task:
      type: text2text-generation
      name: Transliteration
    dataset:
      name: IndoNLP Test -2 
      type: vrclc/IndoNLP-2
      split: test
      args: ml
    metrics:
    - type: wer
      value: 66.9
      name: WER  
  - task:
      type: text2text-generation
      name: Transliteration
    dataset:
      name: IndoNLP Test -2 
      type: vrclc/IndoNLP-2
      split: test
      args: ml
    metrics:
    - type: bleu
      value: 7.5
      name: BLEU
---
# Model Card

Sequence to Sequence Model for Treansliterationg Romanised Malayalam (Manglish) to Native Script.

### Model Sources

- **Repository:** https://github.com/VRCLC-DUK/ml-en-transliteration 
- **Paper:** https://arxiv.org/abs/2412.09957
- **Demo:** https://huggingface.co/spaces/vrclc/en-ml-transliteration

### Model Description

- **Developed by:** [Bajiyo Baiju](https://huggingface.co/Bajiyo), [Kavya Manohar](https://huggingface.co/kavyamanohar), [Leena G Pillai](https://huggingface.co/leenag), [ELizabeth Sherly](https://huggingface.co/SherlyE)
- **Language(s) (NLP):** Malayalam
- **License:** MIT
- Developed as a shared task submission to [INDONLP Workshop](https://indonlp-workshop.github.io/IndoNLP-Workshop/) at [COLING 2025](https://coling2025.org//), Abu Dhabi.

## How to Get Started with the Model

The model needs to have an user defined tokenizers for source and target scripts. The model is trained on words. If your use case involves transliterating full sentences, split the sentences into words before passing to the model.

### Load Dependencies
```
import keras
import huggingface_hub
import tensorflow as tf
import numpy as np
from tensorflow.keras.preprocessing.text import Tokenizer
from tensorflow.keras.preprocessing.sequence import pad_sequences
from huggingface_hub import from_pretrained_keras
import re
```

## Load Model
```
model = from_pretrained_keras("vrclc/transliteration")
```

### Define Tokens and Input Sequence Length:

```
source_tokens = list('abcdefghijklmnopqrstuvwxyz ')
source_tokenizer = Tokenizer(char_level=True, filters='')
source_tokenizer.fit_on_texts(source_tokens)

target_tokens = [
    # Independent vowels
    'അ', 'ആ', 'ഇ', 'ഈ', 'ഉ', 'ഊ', 'ഋ', 'ൠ', 'ഌ', 'ൡ', 'എ', 'ഏ', 'ഐ', 'ഒ', 'ഓ', 'ഔ',
    # Consonants
    'ക', 'ഖ', 'ഗ', 'ഘ', 'ങ', 'ച', 'ഛ', 'ജ', 'ഝ', 'ഞ',
    'ട', 'ഠ', 'ഡ', 'ഢ', 'ണ', 'ത', 'ഥ', 'ദ', 'ധ', 'ന',
    'പ', 'ഫ', 'ബ', 'ഭ', 'മ', 'യ', 'ര', 'ല', 'വ', 'ശ',
    'ഷ', 'സ', 'ഹ', 'ള', 'ഴ', 'റ',
    # Chillu letters
    'ൺ', 'ൻ', 'ർ', 'ൽ', 'ൾ',
    # Additional characters
    'ം', 'ഃ', '്',
    # Vowel modifiers / Signs
    'ാ', 'ി', 'ീ', 'ു', 'ൂ', 'ൃ', 'ൄ', 'െ', 'േ', 'ൈ', 'ൊ', 'ോ', 'ൌ', 'ൗ', ' '
]
target_tokenizer = Tokenizer(char_level=True, filters='')
target_tokenizer.fit_on_texts(target_tokens)

max_seq_length = model.get_layer("encoder_input").input_shape[0][1]

```

### Wrapper script to split input sentences to words before passing to the model

```
def transliterate_with_split_tokens(input_text, model, source_tokenizer, target_tokenizer, max_seq_length):
    """
    Transliterates input text in roman script, retains all other characters (including punctuation, spaces, etc.)
    """
    # Regular expression to split the text into tokens and non-tokens
    tokens_and_non_tokens = re.findall(r"([a-zA-Z]+)|([^a-zA-Z]+)", input_text)

    transliterated_text = ""
    for token_or_non_token in tokens_and_non_tokens:
        token = token_or_non_token[0]
        non_token = token_or_non_token[1]

        if token:
            input_sequence = source_tokenizer.texts_to_sequences([token])[0]
            input_sequence_padded = pad_sequences([input_sequence], maxlen=max_seq_length, padding='post')
            predicted_sequence = model.predict(input_sequence_padded)
            predicted_indices = np.argmax(predicted_sequence, axis=-1)[0]
            transliterated_word = ''.join([target_tokenizer.index_word[idx] for idx in predicted_indices if idx != 0])
            transliterated_text += transliterated_word
        elif non_token:
            transliterated_text += non_token

    return transliterated_text
```

### Usage
```
input text = "ente veedu"
transliterated_text = transliterate_with_split_tokens(input_text, model, source_tokenizer, target_tokenizer, max_seq_length)

print(transliterated_text)
```

## Citation 

```
@article{baiju2024romanized,
  title={Romanized to Native Malayalam Script Transliteration Using an Encoder-Decoder Framework},
  author={Baiju, Bajiyo and Pillai, Leena G and Manohar, Kavya and Sherly, Elizabeth},
  journal={arXiv preprint  arXiv:2412.09957},
  year={2024}
}
```