Upload README_HF_SFT_1.3B.md

README_HF_SFT_1.3B.md

@@ -0,0 +1,74 @@
+---
+language:
+- en
+- ko
+pipeline_tag: text-generation
+inference: false
+tags:
+- pytorch
+- llama
+- causal-lm
+- 42dot-llm
+license: cc-by-nc-4.0
+---
+# 42dot-SFT 1.3B
+
+**42dot-SFT** is a large language model (LLM) developed by [**42dot**](https://42dot.ai/) which is trained to follow natural language instructions, and derived from **42dot-PLM** by supervised fine-tuning (SFT). This repository contains a 1.3B-parameter version.
+
+## Model Description
+
+### Hyperparameters
+As same as 42dot-PLM, the model is built upon a Transformer decoder architecture similar to the [LLaMA 2](https://ai.meta.com/research/publications/llama-2-open-foundation-and-fine-tuned-chat-models/) and its hyperparameters are listed below.
+
+| Params | Layers | Attention heads | Hidden size | FFN size | Max. length\* |
+| -- | -- | -- | -- | -- | -- |
+| 1.3B | 24 | 32 | 2,048 | 5,632 | 8,192 |
+
+(\* unit: tokens)
+### Supervised Fine-tuning
+
+Fine-tuning took about 4 hours using 8 * NVIDIA A100 GPUs. For training dataset, we manually constructed  (question/insturuction) and response pairs, which can either be single- or multi-turn. 
+
+### Evaluation
+Inspired by recent attempts like [Vicuna](https://lmsys.org/blog/2023-03-30-vicuna/#how-good-is-vicuna), we evaluate 42dot-PLM with other proprietary/open-sourced chatbots using GPT-4 for assesing various aspects of responses. The sample of evaluation dataset and prompt template can be downloaded from our [GitHub repo](https://github.com/42dot/42dot_LLM).
+
+- Baselines:
+  - [ChatGPT](https://chat.openai.com/) using GPT-3.5-turbo and GPT-4
+  - [Bard](https://bard.google.com/)
+  - [KORani-v2-13B](https://huggingface.co/KRAFTON/KORani-v1-13B)
+
+| Model | GPT-3.5 |  GPT-4   |   Bard   | KORani | 42dot-SFT |
+| :-- |:-------:|:--------:|:--------:|:------:|:---------:|
+| Params | Unknown | Unknown | Unknown |  13B   |   1.3B    |
+
+<figure align="center">
+<img src="https://huggingface.co/42dot/42dot-sft-1.3b/resolve/main/asset/Ko-Score.png" width="90%" height="70%"/>
+<figcaption><b>Response quality evaluation result</b></figcaption>
+</figure>
+
+<figure align="center">
+<img src="https://huggingface.co/42dot/42dot-sft-1.3b/resolve/main/asset/42dot-SFT-vs.png" width="90%" height="70%"/>
+<figcaption><b>Comparison between proprietary chatbots and 42dot-SFT</b></figcaption>
+</figure>
+
+
+## Limitations and Ethical Considerations
+42dot-SFT shares a number of well-known limitations of other LLMs. For example, it may generate false and misinformative contents since 42dot-SFT is also subject to [hallucination](https://en.wikipedia.org/wiki/Hallucination_(artificial_intelligence)). In addition, 42dot-SFT may generate toxic, harmful and biased contents due to use of web-available training corpus in pre-training phase. We strongly suggest that 42dot-SFT users should beware of those limitations and take necessary steps for mitigating those issues.
+
+## Disclaimer
+The contents generated by 42dot LLM series ("42dot LLMs") do not necessarily reflect the views or opinions of 42dot Inc. ("42dot"). 42dot disclaims any and all liability to any part for any direct, indirect, implied, punitive, special, incidental or other consequential damages arising any use of the 42dot LLMs and theirs generated contents.
+
+## License
+The 42dot-SFT is licensed under the Creative Commons Attribution-NonCommercial 4.0 (CC BY-NC 4.0) license.
+
+## Citation
+
+```
+@misc{42dot2023lm,
+      title={42dot LM: Instruction Tuned Large Language Model of 42dot},
+      author={Woo-Jong Ryu and Sang-Kil Park and Jinwoo Park and Sungmin Lee and Yongkeun Hwang},
+      year={2023},
+      url = {https://gitlab.42dot.ai/NLP/hyperai/ChatBaker},
+      version = {pre-release},
+}
+```