seonjeongh/Korean-Propositionalizer

Overview

This model is designed for the abstractive proposition segmentation task in Korean, as described in the paper Scalable and Domain-General Abstractive Proposition Segmentation. The model segments text into atomic and self-contained units (atomic facts).

Training Details

• Base Model: yanolja/EEVE-Korean-Instruct-10.8B-v1.0
• Fine-tuning Method: LoRA
• Dataset: RoSE
  • Translation: The dataset was translated into Korean using GPT-4o.
    • GPT-4o was prompted to translate propositions using the vocabulary in the text.
  • Data Split: The dataset was randomly split into training, validation, and test sets (1900:100:500) for fine-tuning.

Usage

Data Preprocessing

from konlpy.tag import Kkma

sent_start_token = "<sent>"
sent_end_token = "</sent>"
instruction = "I will provide a passage split into sentences by <s> and </s> markers. For each sentence, generate its list of propositions. Each proposition contains a single fact mentioned in the corresponding sentence written as briefly and clearly as possible.\n\n"

kkma = Kkma()

def get_input(text, tokenizer):
  sentences = kkma.sentences(text)
  prompt = instruction + "Passage: " + sent_start_token + f"{sent_end_token}{sent_start_token}".join(sentences) + sent_end_token + "\nPropositions:\n"
  messages = [{"role": "system", "content": "You are a helpful assistant."},
              {"role": "user", "content": prompt}]
  input_text = tokenizer.apply_chat_template(
                      messages,
                      tokenize=False,
                      add_generation_prompt=True)
  return input_text

def get_output(text):
  results = []
  group = []

  if text.startswith("Propositions:"):
      lines = text[len("Propositions:"):].strip().split("\n")
  else:
      lines = text.strip().split("\n")
      
  for line in lines:
    if line.strip() == sent_start_token:
      continue
    elif line.strip() == sent_end_token:
      results.append(group)
      group = []
    else:
      if not line.strip().startswith("-"):
        break
      line = line[1:].strip()
      group.append(line)

  return results

Loading Model and Tokenizer

import peft, torch
from transformers import AutoModelForCausalLM, AutoTokenizer

LORA_PATH = "seonjeongh/Korean-Propositionalizer"

lora_config = peft.PeftConfig.from_pretrained(LORA_PATH)
base_model = AutoModelForCausalLM.from_pretrained(lora_config.base_model_name_or_path,
                                                  torch_dtype=torch.float16,
                                                  device_map="auto")
model = peft.PeftModel.from_pretrained(base_model, LORA_PATH)
model = model.merge_and_unload(progressbar=True)
tokenizer = AutoTokenizer.from_pretrained(lora_config.base_model_name_or_path)

Inference Example

device = "cuda"

text = "옥스포드는 화요일 맨체스터 유나이티드와의 경기에서 3-2로 패한 경기에서 21세 이하 팀으로 득점했다. 그 골은 16세 선수의 1군 데뷔 주장을 강화할 것이다. 센터백은 이번 시즌 웨스트햄 1군과 함께 훈련했다. 웨스트햄 유나이티드의 최신 뉴스는 여기를 클릭하세요."
inputs = tokenizer([get_input(text, tokenizer)], return_tensors='pt').to(device)
output = model.generate(**inputs, max_new_tokens=512, pad_token_id = tokenizer.pad_token_id, eos_token_id = tokenizer.eos_token_id, use_cache=True)
response = tokenizer.batch_decode(output[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0]
results = get_output(response)
print(results)

Example output

[
   [
    "옥스포드는 21세 이하 팀으로 득점했다.",
    "옥스포드는 맨체스터 유나이티드와의 경기에서 3-2로 패했다.",
    "옥스포드는 화요일 경기를 했다.",
   ],
   [
    "그 골은 16세 선수의 주장을 강화할 것이다.",
    "그 골은 16세 선수의 1 군 데뷔 주장을 강화할 것이다.",
   ],
   [
    "센터 백은 웨스트 햄 1 군과 함께 훈련했다.",
    "센터 백은 이번 시즌 웨스트 햄 1 군과 함께 훈련했다.",
   ],
   [
    "웨스트햄 유나이티드의 최신 뉴스는 여기를 클릭하세요."
   ]
]

Inputs and Outputs

• Input: Text.
• Output: List of propositions for all the sentences in the text passage. The propositions for each sentence are grouped separately.

Evaluation Results

• Metric: Reference-less & reference-base metrics proposed in Scalable and Domain-General Abstractive Proposition Segmentation.
• Models:
  • Dynamic 10-shot models: For each test example, the most similar 10 examples were selected from the training set using BM25.
  • Translate-test models: google/gemma-7b-aps-it model + EN->KO, KO->EN translation using GPT-4o or GPT-4o-mini.
  • Translate-train models: LoRA fine-tuned sLLMs using the Korean RoSE dataset.

Reference-less metric

Model	Precision	Recall	F1
Gold	97.46	96.28	95.88
dynamic 10-shot (Qwen/Qwen2.5-72B-Instruct)	98.86	93.99	95.58
dynamic 10-shot GPT-4o	97.61	97.00	96.87
dynamic 10-shot GPT-4o-mini	98.51	97.12	97.17
Translate-Test (google/gemma-7b-aps-it & GPT-4o Translation)	97.38	96.93	96.52
Translate-Test (google/gemma-7b-aps-it & GPT-4o-mini Translation)	97.24	96.26	95.73
Translate-Train (Qwen/Qwen2.5-7B-Instruct)	94.66	92.81	92.08
Translate-Train (LGAI-EXAONE/EXAONE-3.5-7.8B-Instruct)	93.80	93.29	92.80
Translate-Train (yanolja/EEVE-Korean-Instruct-10.8B-v1.0)	97.41	96.02	95.93

Reference-base metric

Model	Precision	Recall	F1
Gold	100	100	100
dynamic 10-shot (Qwen/Qwen2.5-72B-Instruct)	48.49	40.27	42.99
dynamic 10-shot GPT-4o	49.16	44.72	46.05
dynamic 10-shot GPT-4o-mini	49.30	39.25	42.88
Translate-Test (google/gemma-7b-aps-it & GPT-4o Translation)	57.02	47.52	51.10
Translate-Test (google/gemma-7b-aps-it & GPT-4o-mini Translation)	57.19	47.68	51.26
Translate-Train (Qwen/Qwen2.5-7B-Instruct)	42.62	38.37	39.64
Translate-Train (LGAI-EXAONE/EXAONE-3.5-7.8B-Instruct)	46.82	43.08	44.02
Translate-Train (yanolja/EEVE-Korean-Instruct-10.8B-v1.0)	50.82	45.89	47.44