See axolotl config

axolotl version: 0.6.0

# Original base model config
# base_model: Dans-DiscountModels/Meta-Llama-3.2-3B-ChatML
# Using smaller model instead
base_model: HuggingFaceTB/SmolLM2-360M

# Original tokenizer config
# tokenizer_config: Dans-DiscountModels/Meta-Llama-3.2-3B-ChatML
# Using matching tokenizer for smaller model
tokenizer_config: HuggingFaceTB/SmolLM2-360M

# Model loading configuration
load_in_8bit: false
load_in_4bit: false
strict: false

# Chat template configuration
chat_template: chatml

# Dataset configuration
datasets:
  - path: Emm9625/textwork-00-dedupe-0.75
    name: smol-constraints
    split: train
    type: chat_template
    field_messages: messages
    message_field_role: role
    message_field_content: content
    train_on_eos: turn
    # shards: 2
    # shard_idx: 0
    
  - path: Emm9625/textwork-00-dedupe-0.75
    name: smol-rewrite
    split: train
    type: chat_template
    field_messages: messages
    message_field_role: role
    message_field_content: content
    train_on_eos: turn
    # shards: 2
    # shard_idx: 0
    
  - path: Emm9625/textwork-00-dedupe-0.75
    name: smol-summarize
    split: train
    type: chat_template
    field_messages: messages
    message_field_role: role
    message_field_content: content
    train_on_eos: turn
    # shards: 2
    # shard_idx: 0
    

test_datasets:
  - path: Emm9625/textwork-00-dedupe-0.75
    name: smol-constraints
    split: test
    type: chat_template
    field_messages: messages
    message_field_role: role
    message_field_content: content
    train_on_eos: turn
    shards: 5
    shard_idx: 0
    
  - path: Emm9625/textwork-00-dedupe-0.75
    name: smol-rewrite
    split: test
    type: chat_template
    field_messages: messages
    message_field_role: role
    message_field_content: content
    train_on_eos: turn
    shards: 5
    shard_idx: 0

  - path: Emm9625/textwork-00-dedupe-0.75
    name: smol-summarize
    split: test
    type: chat_template
    field_messages: messages
    message_field_role: role
    message_field_content: content
    train_on_eos: turn
    shards: 5
    shard_idx: 0



dataset_prepared_path: /notebooks/last_run_prepared
output_dir: /tmp/meow/
hub_model_id: Emm9625/tw-350M-dedupe-0.75-overfit
hub_strategy: checkpoint
# Whether to use hf `use_auth_token` for loading datasets. Useful for fetching private datasets
# Required to be true when used in combination with `push_dataset_to_hub`
hf_use_auth_token: true

# Model configuration
sequence_len: 4096
sample_packing: true
eval_sample_packing: true
pad_to_sequence_len: true

adapter:
lora_model_dir:

lora_r: 16
lora_alpha: 32
lora_dropout: 0.05
lora_fan_in_fan_out:
lora_target_modules:
  - gate_proj
  - down_proj
  - up_proj
  - q_proj
  - v_proj
  - k_proj
  - o_proj


# Unsloth optimizations
unsloth_cross_entropy_loss: true
unsloth_rms_norm: true
unsloth_rope: true
#Lora Optimizations
# unsloth_lora_mlp: true
# unsloth_lora_qkv: true
# unsloth_lora_o: true



# Training configuration
gradient_accumulation_steps: 1
micro_batch_size: 32
num_epochs: 5
optimizer: adamw_8bit
lr_scheduler: cosine
learning_rate: 1e-5
torch_compile: auto

train_on_inputs: false
group_by_length: false
bf16: true
gradient_checkpointing: true
flash_attention: true

# Training monitoring
loss_watchdog_threshold: 5.0
loss_watchdog_patience: 3
warmup_ratio: 0.10
weight_decay: 0.00
saves_per_epoch: 1
evals_per_epoch: 5
save_safetensors: true
wandb_project: textwork-00-dedupe
logging_steps: 1

# Special tokens configuration
special_tokens:
  eos_token: "<|im_end|>"
  bos_token: "<|im_start|>"
  pad_token: "<|im_end|>"

fsdp:
fsdp_config:

tw-350M-dedupe-0.75-overfit

This model is a fine-tuned version of HuggingFaceTB/SmolLM2-360M on the Emm9625/textwork-00-dedupe-0.75, the Emm9625/textwork-00-dedupe-0.75 and the Emm9625/textwork-00-dedupe-0.75 datasets. It achieves the following results on the evaluation set:

• Loss: 1.2812

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 32
• eval_batch_size: 32
• seed: 42
• optimizer: Use OptimizerNames.ADAMW_8BIT with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 92
• num_epochs: 5

Training results

Training Loss	Epoch	Step	Validation Loss
1.5664	0.0054	1	1.5880
1.5605	0.2	37	1.5856
1.6023	0.4	74	1.5722
1.5213	0.6	111	1.5354
1.4658	0.8	148	1.4970
1.4403	1.0	185	1.4584
1.4088	1.1946	222	1.4166
1.3583	1.3946	259	1.3798
1.3049	1.5946	296	1.3495
1.3063	1.7946	333	1.3293
1.2535	1.9946	370	1.3154
1.2862	2.1892	407	1.3059
1.3075	2.3892	444	1.2983
1.26	2.5892	481	1.2935
1.241	2.7892	518	1.2896
1.2975	2.9892	555	1.2864
1.264	3.1838	592	1.2843
1.2527	3.3838	629	1.2832
1.2438	3.5838	666	1.2822
1.3144	3.7838	703	1.2814
1.2393	3.9838	740	1.2815
1.2786	4.1784	777	1.2809
1.2307	4.3784	814	1.2811
1.2784	4.5784	851	1.2811
1.3118	4.7784	888	1.2810
1.3135	4.9784	925	1.2812

Framework versions

• Transformers 4.47.1
• Pytorch 2.5.1+cu124
• Datasets 3.2.0
• Tokenizers 0.21.0