Update README.md

README.md

@@ -4,4 +4,166 @@ datasets:
 - audichandra/bitext_customer_support_llm_dataset_indonesian
 language:
 - id
----
+---
+
+![Gajah_7-B](https://huggingface.co/audichandra/Gajah-7B/blob/main/img/gajah_7b.jpg)
+
+## Quick Intro
+
+Gajah-7B is the 1st iteration of Indonesian AI chatbot with [Merak-7B](https://huggingface.co/Ichsan2895/Merak-7B-v4) as the base model that is trained with PEFT Qlora method and Indonesian version of [bitext](https://huggingface.co/datasets/audichandra/bitext_customer_support_llm_dataset_indonesian) customer support dataset for LLM.
+
+Gajah-7B is licensed under [MIT](https://opensource.org/license/mit) license to support the open source initiative and served as another example of how to finetune pre-trained model. 
+
+you can contact me through my [LinkedIn](www.linkedin.com/in/audichandra) or [Github](https://github.com/audichandra/Indonesian_AI_Chatbot_Customer_Support) about this model and its applications. 
+
+## Installation 
+
+We need at least Python 3.10 and PyTorch 2, and do a pip install of the requirements.txt along with some optional pip install features such as flash attention: 
+
+```bash 
+pip install flash-attn 
+```
+
+## GPU requirements
+
+**Training**: 8x A40
+**Loading**: 1x RTX A500 
+*notes: the author trains and loads the model on Cloud GPU platform such as runpods*
+
+## Scripts
+
+**Scripts for loading model using multiple GPU** 
+
+```bash
+import torch
+import time
+from transformers import pipeline, AutoTokenizer, AutoModelForCausalLM, AutoConfig, LlamaTokenizer, BitsAndBytesConfig
+from peft import PeftModel, PeftConfig
+ 
+#BNB_CONFIG = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=torch.bfloat16, bnb_4bit_use_double_quant=True, bnb_4bit_quant_type="nf4")
+model_chat = "audichandra/Gajah-7B"
+model1 = AutoModelForCausalLM.from_pretrained(model_chat
+                                              , torch_dtype=torch.bfloat16, device_map="auto", pad_token_id=0 
+                                              , attn_implementation="flash_attention_2"
+                                              , cache_dir="/workspace"
+                                              #, quantization_config=BNB_CONFIG
+                                             )
+
+tokenizer = LlamaTokenizer.from_pretrained(model_chat)
+
+def generate_response(question: str) -> str:
+    chat = [
+      {"role": "system", "content": "Ada yang bisa saya bantu?"},
+      {"role": "user", "content": question},
+    ]
+
+    prompt = tokenizer.apply_chat_template(chat, tokenize=False, add_generation_prompt=True)
+
+    inputs = tokenizer(prompt, return_tensors="pt", return_attention_mask=True)
+
+    with torch.no_grad():
+        outputs = model1.generate(input_ids=inputs["input_ids"].to("cuda"),
+                           attention_mask=inputs.attention_mask,
+                           eos_token_id=tokenizer.eos_token_id,
+                           pad_token_id=tokenizer.eos_token_id,
+                           max_new_tokens=512)
+        response = tokenizer.batch_decode(outputs.detach().cpu().numpy(), skip_special_tokens=True)[0]
+
+        assistant_start = f'''{question} \n assistant\n '''
+        response_start = response.find(assistant_start)
+        return response[response_start + len(assistant_start) :].strip()
+
+start_time = time.time()
+prompt = "bagaimana saya dapat membatalkan pembelian saya?"
+print(generate_response(prompt))
+
+end_time = time.time()
+elapsed_time = end_time - start_time 
+print(f"Elapsed time: {elapsed_time} seconds")
+```
+
+*you can uncomment the bnbconfig to do a 4-bit quantization to run it with lower VRAM but the results might suffer in terms of quality and time* 
+
+**Scripts for loading model using single GPU**
+
+```bash
+import torch
+import time
+from transformers import pipeline, AutoTokenizer, AutoModelForCausalLM, AutoConfig, LlamaTokenizer, BitsAndBytesConfig
+from peft import PeftModel, PeftConfig
+
+#BNB_CONFIG = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=torch.bfloat16, bnb_4bit_use_double_quant=True, bnb_4bit_quant_type="nf4")
+#model_save_path1 = "/workspace/axolotl/merged_model"
+model_chat = "audichandra/Gajah-7B"
+device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+model1 = AutoModelForCausalLM.from_pretrained(model_chat
+                                              , torch_dtype=torch.bfloat16
+                                              #, device_map="auto", pad_token_id=0 
+                                              #, attn_implementation="flash_attention_2"  
+                                              , cache_dir="/workspace"
+                                              #, quantization_config=BNB_CONFIG
+                                             ).to(device)
+tokenizer = LlamaTokenizer.from_pretrained(model_chat)
+
+def generate_response(question: str) -> str:
+    chat = [
+      {"role": "system", "content": "Ada yang bisa saya bantu?"},
+      {"role": "user", "content": question},
+    ]
+
+    prompt = tokenizer.apply_chat_template(chat, tokenize=False, add_generation_prompt=True)
+    inputs = tokenizer(prompt, return_tensors="pt", return_attention_mask=True)
+
+    inputs = inputs.to(device)  # Ensure inputs are on the same device as the model
+
+    with torch.no_grad():
+        outputs = model1.generate(**inputs, max_new_tokens=512)
+        response = tokenizer.batch_decode(outputs, skip_special_tokens=True)[0]
+
+        assistant_start = f'''{question} \n assistant\n '''
+        response_start = response.find(assistant_start)
+        return response[response_start + len(assistant_start) :].strip()
+
+
+# Use the functions together
+start_time = time.time()
+prompt = "bagaimana saya dapat membatalkan pembelian saya?"
+print(generate_response(prompt))
+
+end_time = time.time()
+elapsed_time = end_time - start_time 
+print(f"Elapsed time: {elapsed_time} seconds")
+
+```
+
+*some features such as flash attention might not work on single GPU* 
+
+[<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl)
+
+## Citation 
+
+```bash
+@article{Merak,
+  title={Merak-7B: The LLM for Bahasa Indonesia},
+  author={Muhammad Ichsan},
+  publisher={Hugging Face}
+  journal={Hugging Face Repository},
+  year={2023}
+}
+
+@article{dettmers2023qlora,
+  title   = {QLoRA: Efficient Finetuning of Quantized LLMs},
+  author  = {Dettmers, Tim and Pagnoni, Artidoro and Holtzman, Ari and Zettlemoyer, Luke},
+  journal = {arXiv preprint arXiv:2305.14314},
+  year    = {2023}
+}
+
+@article{axolotl,
+  author = {{OpenAccess AI Collective}},
+  title = {Axolotl: A Repository for AI Research and Development},
+  year = {2023},
+  publisher = {GitHub},
+  journal = {GitHub repository},
+  howpublished = {\url{https://github.com/OpenAccess-AI-Collective/axolotl}}
+}
+```