English to Tagin translator v0.1.0 - initial commit

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+models/target.spm filter=lfs diff=lfs merge=lfs -text

app.py

@@ -0,0 +1,120 @@
+from transformers import pipeline
+from datasets import Dataset
+import streamlit as st
+
+# Set the background color and layout with set_page_config
+st.set_page_config(
+    page_title="English to Tagin Translator",
+    page_icon=":repeat:",
+    layout="wide",
+)
+
+# Streamlit app setup
+st.title(":repeat: English to Tagin Translator")
+st.markdown("Welcome to the English to Tagin Translator. :sparkles: Simply enter your text in English, and get the translation in Tagin instantly! :thumbsup:")
+
+# Text input
+if 'text_input' not in st.session_state:
+    st.session_state.text_input = ""
+text_input = st.text_area("Enter English text to translate", height=150, value=st.session_state.text_input)
+
+# Define your model directory
+model_directory = "models"
+
+# Initialize the pipeline for translation
+translation_pipeline = pipeline(
+    task="translation",
+    model=model_directory,
+    tokenizer=model_directory,
+    device=0  # Use GPU (if available)
+)
+
+# Translate button
+if st.button("Translate", key="translate_button"):
+    if text_input:
+        with st.spinner("Translating... Please wait"):
+            # Prepare data for translation
+            sentences = [text_input]
+            data = Dataset.from_dict({"text": sentences})
+
+            # Apply translation
+            try:
+                results = data.map(lambda x: {"translation": translation_pipeline(x["text"])})
+                result = results[0]["translation"][0]['translation_text']
+                
+                # Capitalize the first letter of the result
+                result = result.capitalize()
+
+                # Display translation result with custom styling
+                st.markdown("#### Translated text:")
+                st.markdown(f'<h2 class="result-text">{result}</2>', unsafe_allow_html=True)
+                # st.markdown(result)
+                
+            except Exception as e:
+                st.error(f"Translation error: {e}")
+    else:
+        st.warning("Please enter text to translate.")
+
+# Clear input button
+if st.button("Clear Input"):
+    st.session_state.text_input = ""
+
+st.markdown("<br>" * 3, unsafe_allow_html=True)
+st.markdown("""
+❗❗❗ **Please note:** The English-to-Tagin translator is still in its initial phase, so it may provide incorrect translations at times. Your understanding is appreciated!
+
+🤝 **For contributions or inquiries, feel free to contact me!**
+
+""")
+
+st.markdown(""" 
+### Tagin Language
+
+Tagin is a beautiful language spoken by the Tagin tribe and belongs to the Tani group of Sino-Tibetan languages. You'll mainly find Tagin speakers in the Upper Subansiri, Shiyomi, Kara Dadi, Kurung Kumey, and Papum Pare districts of Arunachal Pradesh, India. While about 63,000 (according to 2011 Census of India) people speak Tagin as their mother tongue, UNESCO has marked it as 'definitely endangered', which means it's at risk of disappearing. Unfortunately, very few written materials exist in Tagin, which makes it hard to study and preserve the language.
+
+As a small contribution to preserving this rich cultural heritage, I've developed this English-Tagin translator using the GinLish Corpus v0.1.0. By creating this digital tool, I hope to help keep the Tagin language alive and make it more accessible to both the Tagin community and language enthusiasts. This project is my way of giving back to society and helping protect an important piece of our cultural diversity.
+
+
+""")
+
+st.markdown("""
+### GinLish Corpus v0.1.0 (2024)
+
+I'm excited:satisfied: to share that I created the GinLish Corpus v0.1.0, which is actually the first-ever collection of matched Tagin and English sentences. The corpus contains 60,000 carefully paired sentences that captures how these languages relate to each other. To build this, I translated English sentences from the Tatoeba website into Tagin and included traditional Tagin folk stories too.
+What makes this special is that I made sure to keep the true essence of the Tagin language alive in the translations. This means including Tagin sayings, cultural elements, and the unique way Tagin people express themselves. All this careful attention to detail makes the corpus really valuable for building translation tools, studying the language, and helping people learn Tagin. It's not just a simple word-for-word translation - it's a bridge between these two languages that respects and preserves Tagin's cultural identity.
+
+Good news:smiley: for researchers and language enthusiasts - I plan to release this dataset for non-commercial use once I complete my PhD!🎓 This way, others can also contribute to preserving and studying this beautiful language.
+
+""")
+# Sidebar for About and Contact information
+st.sidebar.header("About the Developer")
+
+st.sidebar.markdown("""
+Hey there! 👋 
+
+I’m **Tungon Dugi**. 
+
+Right now, I’m doing my PhD in Computer Science and Engineering at NIT Arunachal Pradesh. 🎓 
+
+I’ve got a keen interest in Natural Language Processing (NLP), Machine Translation (MT), Deep Learning, and Linguistics. 
+
+💻✨ I love exploring how tech can help preserve and promote low-resource languages, especially my own language, **Tagin**! 🌍💬""")
+
+
+# Create some space between main sidebar content and footer
+st.sidebar.markdown("<br>" * 5, unsafe_allow_html=True)
+# detailed version using sidebar:
+st.sidebar.markdown("---")
+st.sidebar.caption("Made with ❤️ by Tungon Dugi")
+st.sidebar.caption("Contact: [email protected]")
+
+# Or using columns in sidebar:
+col1, col2 = st.sidebar.columns(2)
+with col1:
+    st.caption("© 2024")
+with col2:
+    st.caption("v0.1.0")
+
+# To push it to very bottom of sidebar, add more space:
+st.sidebar.markdown("<br>" * 10, unsafe_allow_html=True)
+st.sidebar.caption("Bottom Footer")

models/config.json

@@ -0,0 +1,56 @@
+{
+  "_name_or_path": "Helsinki-NLP/opus-mt-en-hi",
+  "activation_dropout": 0.0,
+  "activation_function": "swish",
+  "add_bias_logits": false,
+  "add_final_layer_norm": false,
+  "architectures": [
+    "MarianMTModel"
+  ],
+  "attention_dropout": 0.0,
+  "bos_token_id": 0,
+  "classif_dropout": 0.0,
+  "classifier_dropout": 0.0,
+  "d_model": 512,
+  "decoder_attention_heads": 8,
+  "decoder_ffn_dim": 2048,
+  "decoder_layerdrop": 0.0,
+  "decoder_layers": 6,
+  "decoder_start_token_id": 61949,
+  "decoder_vocab_size": 61950,
+  "dropout": 0.1,
+  "encoder_attention_heads": 8,
+  "encoder_ffn_dim": 2048,
+  "encoder_layerdrop": 0.0,
+  "encoder_layers": 6,
+  "eos_token_id": 0,
+  "extra_pos_embeddings": 61950,
+  "forced_eos_token_id": 0,
+  "id2label": {
+    "0": "LABEL_0",
+    "1": "LABEL_1",
+    "2": "LABEL_2"
+  },
+  "init_std": 0.02,
+  "is_encoder_decoder": true,
+  "label2id": {
+    "LABEL_0": 0,
+    "LABEL_1": 1,
+    "LABEL_2": 2
+  },
+  "max_length": null,
+  "max_position_embeddings": 512,
+  "model_type": "marian",
+  "normalize_before": false,
+  "normalize_embedding": false,
+  "num_beams": null,
+  "num_hidden_layers": 6,
+  "pad_token_id": 61949,
+  "scale_embedding": true,
+  "share_encoder_decoder_embeddings": true,
+  "static_position_embeddings": true,
+  "torch_dtype": "float32",
+  "transformers_version": "4.46.1",
+  "use_cache": true,
+  "vocab_size": 61950
+}

models/generation_config.json

@@ -0,0 +1,16 @@
+{
+  "bad_words_ids": [
+    [
+      61949
+    ]
+  ],
+  "bos_token_id": 0,
+  "decoder_start_token_id": 61949,
+  "eos_token_id": 0,
+  "forced_eos_token_id": 0,
+  "max_length": 512,
+  "num_beams": 4,
+  "pad_token_id": 61949,
+  "renormalize_logits": true,
+  "transformers_version": "4.46.1"
+}

models/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0021e11c864a4515bbc4d35203cef19270e127811ab827bf73151b2aa673200d
+size 303704440

models/special_tokens_map.json

@@ -0,0 +1,5 @@
+{
+  "eos_token": "</s>",
+  "pad_token": "<pad>",
+  "unk_token": "<unk>"
+}

models/target.spm

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f5529d3a72f8c1d5f7e357f1b6fd30e3cf58f6e1ba0401db135a118ac92f4a76
+size 1067935

models/tokenizer_config.json

@@ -0,0 +1,38 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "61949": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "model_max_length": 512,
+  "pad_token": "<pad>",
+  "separate_vocabs": false,
+  "source_lang": "eng",
+  "sp_model_kwargs": {},
+  "target_lang": "hin",
+  "tokenizer_class": "MarianTokenizer",
+  "unk_token": "<unk>"
+}

models/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7097011bb71f66fc1e7c1189e1a810f31a4375bb0f420b9cfa97c692271d5b9b
+size 5368