kl3m-004-correction-001 Model

kl3m-004-correction-001 is a small, ~500M parameter language model model designed to assist in the correction of common typing, spelling, OCR, and format issues in English text, especially in the financial and legal domains.

Notably, this model has been trained on the alea-institute/kl3m-004-char-8k-cased tokenizer, which is a BPE tokenizer trained with a 3-character maximum token constraint.

This model was originally trained 3 days on 1xRTX3090, and a large ~3B parameter MoE is pending release.

Getting Started

Simply prompt the model with the original text, followed by the <|sep|> token, and wait for stop token (<|end|>) generation. You can use pipeline to handle this for you.

Deterministic

In many situations, deterministic correction (i.e., most probable logit sequence) is fine.

from transformers import pipeline
p = pipeline('text-generation', 'alea-institute/kl3m-004-correction-001', device='cpu')

text = "Tne Uni+ed 5tates is nct responsib|e for 5uch pr0duction"

correction = p(text + "<|sep|>", max_new_tokens=512, return_full_text=False)[0]['generated_text']

# Output: The United States is not responsible for such production

Sampled with Frequency Weighting

In other situations, it can be useful to generate multiple corrections with a sampler and evaluate the distribution. For example:

• using a string or token-based distance metric to score or rank corrections
• showing multiple suggestions to a user with frequency-weighted order

from transformers import pipeline
from collections import Counter

p = pipeline('text-generation', 'alea-institute/kl3m-004-correction-001', device='cuda')

text = "Tne Uni+ed 5tates is nct responsib|e for 5uch pr0duction"

corrections = Counter(
  [
    g['generated_text']
    for g in p(
      text + "<|sep|>",
      max_new_tokens=512,
      return_full_text=False,
      temperature=0.5,
      # top_p, top_k, custom sampler, etc.
      do_sample=True,
      num_return_sequences=10
    )
  ]
).most_common(3)

# Output: [('The United States is not responsible for such production', 7), ('the United States is not responsible for such production', 3)]

Source

https://github.com/alea-institute/kl3m-model-research

Training Data

This model was trained on a dataset generated with the KL3M data collection and the https://github.com/alea-institute/alea-data-generator library, which can create realistic synthetic samples using traditional (non-generative) techniques.

The source code to retrieve and process this dataset is available here: https://github.com/alea-institute/kl3m-data

Some pre-tokenized subsets of the KL3M data collection are available on Hugging Face: https://huggingface.co/datasets?sort=most_rows&search=kl3m-data

Complete, raw data is available upon request at this time via S3 under a Requester Pays model. We are actively working on a zero-cost distribution model as soon as we can obtain additional support.

Model Details

Summary

• Architecture: LlamaForCausalLM
• Parameters: 478.2M
• Context Window: 512 tokens (no ROPE)
• Language(s): Primarily English
• Tokenizer: kl3m-004-char-8k-cased BPE tokenizer (8K tokens, between 1-3 characters each)
• Developed by: ALEA Institute
• License: CC-BY 4.0
• Hardware Requirements: Runs real-time in fp32 on CPU or consumer NV/AMD GPUs

Key Features

• Clean Training Data: Built on what was originally referred to as the Kelvin Legal DataPack, ensuring all training data is ethically sourced and legally permissible.
• Low Toxicity: Empirically lower toxicity and bias
• Enterprise Focus: Specifically designed for legal, regulatory, and financial workflows.
• Efficient Deployment: Optimized for real-time inference on consumer hardware.

Use Cases

• Correcting common typing or spelling errors
• Correcting common OCR errors
• Correcting common formatting errors

License

Model weights are released under the CC-BY 4.0 License.

Contact

The KL3M model family is now maintained by the ALEA Institute. For technical support, collaboration opportunities, or general inquiries:

• GitHub: https://github.com/alea-institute/kl3m-model-research
• Email: [email protected]
• Website: https://aleainstitute.ai

Citation

Tokenizer, dataset, and model publications are pending.

Contact

For any questions, please contact ALEA Institute at [email protected] or create an issue on this repository or GitHub.