yokhal-md / README.md
seonglae's picture
Update README.md
e7561d0 verified
---
language:
- ko
- en
library_name: transformers
tags:
- trl
- sft
widget:
- text: 안녕
---
# Yokhal (욕쟁이 할머니)
<!-- Provide a quick summary of what the model is/does. -->
Korean Chatbot based on Google Gemma
## Model Details
### Model Description
<!-- Provide a longer summary of what this model is. -->
- **Fine-tuned by:** Seonglae Cho
- **Model type:** Gemma
- **Language(s) (NLP):** Korean, English
- **Finetuned from model:** [Gemma-2b-it](https://huggingface.co/google/gemma-2b-it)
### Model Sources
<!-- Provide the basic links for the model. -->
- **Repository:** https://github.com/seonglae/yokhal
- **Demo:** https://huggingface.co/spaces/seonglae/yokhal
## Uses
<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
### Direct Use
Korean Chatbot with Internet culture
### Recommendations
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
## How to Get Started with the Model
Use the code below to get started with the model.
```py
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16,
device_map="auto" if device is None else device,
attn_implementation="flash_attention_2") # if flash enabled
sys_prompt = '한국어로 대답해'
texts = ['안녕', '서울은 오늘 어때']
chats = list(map(lambda t: [{'role': 'user', 'content': f'{sys_prompt}\n{t}'}], texts)) # ChatML format
prompts = list(map(lambda p: tokenizer.apply_chat_template(p, tokenize=False, add_generation_prompt=True), chats))
input_ids = tokenizer(prompts, return_tensors="pt", padding=True).to("cuda" if device is None else device)
outputs = model.generate(**input_ids, max_new_tokens=100, repetition_penalty=1.05)
for output in outputs:
print(tokenizer.decode(output, skip_special_tokens=True), end='\n\n')
```
## Training Details
Trained on 2 x RTX3090
[More Information on Github source code](https://github.com/seonglae/yokhal/blob/master/train.py)
### Training Data
<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
[More Information Needed]
### Training Procedure
1. Weight Initialized from Internet comments dataset
2. Trained on Korean Namuwiki dataset until step 80000 (30000 step is on main branch because of repetition issue above there)
- `seq_length` 1024 with dataset packing
- `batch` 3 per device
- `lr` 1e-5
- `optim` adafactor
3. Instruction tuning on Korean Instruction Dataset using QLoRa (not on main)
- `seq_length` 2048
- `lr` 2e-4
#### Preprocessing [optional]
Gemma do not support explicit system prompt in ChatML, so I trained putting system prompt before user message like below
```py
if (chat[0]['role'] == 'system'):
chat[1]['content'] = f"{chat[0]['content']}\n{chat[1]['content']}"
chat = chat[1:]
try:
prompt = tokenizer.apply_chat_template(chat, tokenize=False)
```
[Source Code](https://github.com/seonglae/yokhal/blob/master/yokhal/adapt.py)
#### Training Hyperparameters
- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
#### Speeds, Sizes, Times [optional]
<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
[More Information Needed]
## Evaluation
<!-- This section describes the evaluation protocols and provides the results. -->
### Testing Data, Factors & Metrics
#### Testing Data
<!-- This should link to a Dataset Card if possible. -->
[More Information Needed]
#### Factors
<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
[More Information Needed]
#### Metrics
<!-- These are the evaluation metrics being used, ideally with a description of why. -->
[More Information Needed]
### Results
[More Information Needed]
#### Summary