KINS-ai
/

ATOMIS

+---
+license: apache-2.0
+language:
+  - en
+  - ko
+tags:
+- gemma-2
+- KINS-ai
+base_model:
+- google/gemma-2-27b
+pipeline_tag: text-generation
+library_name: transformers
+---
+# **Introduction**
+### About the Model
+We introduce ATOMIS, developed by the Korea Institute of Nuclear Safety (KINS). This model is specifically designed for the nuclear field and is a large language model (LLM) with 32 billion parameters. It achieves state-of-the-art performance among its peers on Logickor, a real-world Korean task benchmark; NuclearQA, a nuclear-domain benchmark; and RAGEval, a RAG benchmark. Please refer to the evaluation results table for details.
+## Key Features
+- **Korean Real-World use cases:** The model can understand and generate Korean text with high accuracy, making it suitable for practical scenarios.
+- **Specialized in the Nuclear Domain:** The model has been specifically trained on a vast, specialized corpus of nuclear data.
+- **RAG:** The model delivers accurate answers based on real documents through its high RAG performance.
+### Pre-Training
+We created the base model by expanding layers using a passthrough method, building on the gemma-2-27b model. Additionally, we extended the context length to 32K with RoPE and performed continuous pretraining to restore the model’s performance.
+In particular, to train specialized knowledge in the nuclear domain, we included the following data.
+- Atomic Wiki (https://atomic.snu.ac.kr)
+- NText (https://paperswithcode.com/dataset/ntext)
+- in-house data from KINS (Korea Institute of Nuclear Safety)
+### Post-Training
+The fine-tuning data includes over 1M publicly available instruction datasets as well as high-quality synthetic data. We use this dataset to perform supervised fine-tuning (SFT) and direct preference optimization (DPO).
+# **How to use**
+```python
+# pip install transformers==4.43.4 or later
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+tokenizer = AutoTokenizer.from_pretrained("KINS-ai/ATOMIS")
+model = AutoModelForCausalLM.from_pretrained(
+    "KINS-ai/ATOMIS",
+    device_map="auto",
+    torch_dtype=torch.bfloat16,
+)
+messages = [
+    {"role": "user", "content": "안녕하세요?"},
+]
+input_ids = tokenizer.apply_chat_template(messages, return_tensors="pt", return_dict=True).to("cuda")
+outputs = model.generate(**input_ids, max_new_tokens=256)
+print(tokenizer.decode(outputs[0]))
+```
+# **Evaluation**
+### Overall
+| Model                                | LogicKor | NuclearQA | RAGEval | Avg |
+|--------------------------------------|----- |-----|-----|-----|
+| **c4ai-command-r-08-2024**           | 8.27 | 7.82 | 9.41 | 8.50 |
+| **gemma-2-27b-it**                   | 8.66 | 8.18 | 8.97 | 8.60 |
+| **Qwen2.5-32B-instruct**             | 8.93 | 8.61 | 9.36 | 8.97 |
+| **phi-4**                            | 8.62 | 8.67 | 9.55 | 8.95 |
+| **Mistral-Small-24B-Instruct-2501**  | 8.36 | 8.68 | 9.04 | 8.69 |
+| **Llama-3.3-70b-instruct**           | 7.94 | 8.42 | 9.25 | 8.54 |
+| **ATOMIS**                           | 9.00 | 8.72 | 9.65 | **9.12** |
+### LogicKor
+We evaluated the performance using the [LogicKor](https://github.com/instructkr/LogicKor) code. As the judge model, we employed the officially recommended GPT-4-1106-preview. These scores reflect only the default zero-shot evaluation.
+| Model                                | Math | Reasoning | Coding | Writing | Understanding | Grammar | Single-turn | Multi-turn | Avg |
+|--------------------------------------|----- |-----|-----|-----|-----|-----|-----|-----|-----|
+| **c4ai-command-r-08-2024**           | 6.14 | 7.36 | 9.43 | 9.64 | 9.21 | 7.86 | 8.05 | 8.52 | 8.27 |
+| **gemma-2-27b-it**                   | 8.93 | 8.29 | 8.43 | 9.29 | 9.43 | 7.57 | 8.43 | 8.88 | 8.66 |
+| **Qwen2.5-32B-instruct**             | 8.79 | 8.64 | 9.36 | 9.50 | 9.29 | 8.00 | 8.79 | 9.10 | 8.93 |
+| **phi-4**                            | 8.79 | 9.21 | 9.86 | 9.21 | 9.00 | 5.64 | 8.50 | 8.74 | 8.62 |
+| **Mistral-Small-24B-Instruct-2501**  | 8.00 | 8.14 | 9.36 | 9.43 | 8.50 | 6.71 | 8.29 | 8.43 | 8.36 |
+| **Llama-3.3-70b-instruct**           | 7.43 | 6.50 | 8.79 | 8.43 | 8.64 | 7.86 | 8.14 | 7.74 | 7.94 |
+| **ATOMIS**                           | 8.36 | 8.71 | 9.79 | 9.64 | 8.29 | 9.21 | 9.14 | 8.86 | **9.00** |
+### NuclearQA
+We employed NuclearQA [1], a human-made benchmark consisting of 100 questions designed by experts to evaluate language models in the nuclear domain.
+We then used this question set to assess the LLM’s responses in a manner similar to the Logickor benchmark.
+[1] Acharya, A., Munikoti, S., Hellinger, A., Smith, S., Wagle, S. and Horawalavithana, S., 2023. NuclearQA: A Human-Made Benchmark for Language Models for the Nuclear Domain. arXiv:2310.10920.
+| Model                                | Easy | Medium | Hard | General | Scientific | Numerical | Num+Sci | Avg |
+|--------------------------------------|----- |-----|-----|-----|-----|-----|-----|-----|
+| **c4ai-command-r-08-2024**           | 8.77 | 8.21 | 6.47 | 7.73 | 8.38 | 7.35 | 7.35 | 7.82 |
+| **gemma-2-27b-it**                   | 8.97 | 8.24 | 7.33 | 7.92 | 8.23 | 8.12 | 8.45 | 8.18 |
+| **Qwen2.5-32B-instruct**             | 8.97 | 8.42 | 8.38 | 8.54 | 8.15 | 8.76 | 9.03 | 8.61 |
+| **phi-4**                            | 8.94 | 8.97 | 8.11 | 8.46 | 8.73 | 9.00 | 8.50 | 8.67 |
+| **Mistral-Small-24B-Instruct-2501**  | 9.13 | 8.76 | 8.14 | 8.41 | 8.81 | 8.59 | 8.95 | 8.68 |
+| **Llama-3.3-70b-instruct**           | 9.29 | 8.58 | 7.44 | 8.22 | 8.62 | 8.47 | 8.35 | 8.42 |
+| **ATOMIS**                           | 9.10 | 8.64 | 8.31 | 8.16 | 9.00 | 8.71 | 9.10 | **8.72** |
+### RAGEval
+We used RAGEval [2], a benchmark designed to evaluate RAG performance in terms of factual accuracy, using three novel metrics: Completeness, Hallucination, and Irrelevance.
+We evaluated performance using the [RAGEval](https://github.com/OpenBMB/RAGEval) code. As the judge model, we employed the officially recommended gpt-4o. These scores reflect only the completeness metric of the single-document QA evaluation.
+[2] Zhu, K., Luo, Y., Xu, D., Wang, R., Yu, S., Wang, S., Yan, Y., Liu, Z., Han, X., Liu, Z. and Sun, M., 2024. RAGEval: Scenario Specific RAG Evaluation Dataset Generation Framework. arXiv:2408.01262.
+| Model                                | Factual | Summarization | Multi-hop Reasoning | Avg |
+|--------------------------------------|----- |-----|-----|-----|
+| **c4ai-command-r-08-2024**           | 1.000 | 0.913 | 0.908 | 0.941 |
+| **gemma-2-27b-it**                   | 0.987 | 0.890 | 0.814 | 0.897 |
+| **Qwen2.5-32B-instruct**             | 0.980 | 0.906 | 0.923 | 0.936 |
+| **phi-4**                            | 1.000 | 0.931 | 0.934 | 0.955 |
+| **Mistral-Small-24B-Instruct-2501**  | 0.980 | 0.951 | 0.781 | 0.904 |
+| **Llama-3.3-70b-instruct**           | 0.977 | 0.907 | 0.893 | 0.925 |
+| **ATOMIS**                           | 0.993 | 0.942 | 0.960 | **0.965** |