Create README.md

README.md

@@ -0,0 +1,64 @@
+---
+datasets:
+- ai4bharat/indic-instruct-data-v0.1
+language:
+- en
+- hi
+license: llama2
+tags:
+- multilingual
+- instruction-tuning
+- llama2
+---
+
+# Airavata
+
+This model is a 7B OpenHathi model finetuned on [IndicInstruct dataset](https://huggingface.co/datasets/ai4bharat/indic-instruct-data-v0.1) 
+which is a collection of instruction datasets (Anudesh, wikiHow, Flan v2, Dolly, Anthropic-HHH, OpenAssistant v1, and LymSys-Chat). 
+Please check the corresponding huggingface dataset card for more details.
+
+This was trained as part of the blog [Introducing Airavata: Hindi Instruction-tuned Chat Model](https://ai4bharat.github.io/airavata).
+The codebase used to train and evaluate this model can be found at [https://github.com/AI4Bharat/IndicInstruct](https://github.com/AI4Bharat/IndicInstruct).
+
+
+
+## Usage
+
+Clone [https://github.com/AI4Bharat/IndicInstruct](https://github.com/AI4Bharat/IndicInstruct) and install the required dependencies. Then download or clone this model to the same machine.
+
+## Input Format
+
+The model is trained to use the chat format similar to [Wang et al. 2023](https://arxiv.org/abs/2306.04751) ([code repository](https://github.com/allenai/open-instruct)) (note the newlines):
+```
+<|user|>
+Your message here!
+<|assistant|>
+```
+
+For best results, format all inputs in this manner. **Make sure to include a newline after `<|assistant|>`, this can affect generation quality quite a bit.**
+
+## Hyperparameters
+
+We fine-tune OpenHathi base model on the aforementioned IndicInstruct dataset with LoRA. The hyperparameters for the LoRA fine-tuning are listed below:
+- LoRA Rank: 16
+- LoRA alpha: 32
+- LoRA Dropout: 0.05
+- LoRA Target Modules: ["q_proj", "v_proj", "down_proj", "gate_proj", "up_proj", "k_proj"]
+- Epochs: 4
+- Learning rate: 5e-4
+- Batch Size: 128
+- Floating Point Precision: bfloat16
+
+We recommend the readers to check out [our official blog post](https://ai4bharat.github.io/airavata) for more details on the model training, ablations and evaluation results.
+
+## Citation
+
+```bibtex
+@misc{airavata2024,
+  title = {Introducing Airavata: Hindi Instruction-tuned Chat Model},
+  url = {https://ai4bharat.github.io/airavata},
+  author = {Jay Gala and Thanmay Jayakumar and Jaavid Aktar Husain and Aswanth Kumar and Mohammed Safi Ur Rahman Khan and Diptesh Kanojia and Ratish Puduppully and Mitesh Khapra and Raj Dabre and Rudra Murthy and Anoop Kunchukuttan},
+  month = {January},
+  year = {2024}
+}
+```