Built with Axolotl

ee5dcac6-f384-4e2d-ab77-871303131dd3

This model is a fine-tuned version of katuni4ka/tiny-random-falcon-40b on the None dataset. It achieves the following results on the evaluation set:

  • Loss: 10.5577

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.0001006
  • train_batch_size: 4
  • eval_batch_size: 4
  • seed: 42
  • gradient_accumulation_steps: 2
  • total_train_batch_size: 8
  • optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=adam_beta1=0.9,adam_beta2=0.95,adam_epsilon=1e-5
  • lr_scheduler_type: linear
  • lr_scheduler_warmup_steps: 10
  • training_steps: 200

Training results

Training Loss Epoch Step Validation Loss
22.2622 0.0001 1 11.1238
21.4032 0.0050 50 10.6575
21.3746 0.0100 100 10.5907
21.5424 0.0150 150 10.5663
21.4729 0.0201 200 10.5577

Framework versions

  • PEFT 0.13.2
  • Transformers 4.46.0
  • Pytorch 2.5.0+cu124
  • Datasets 3.0.1
  • Tokenizers 0.20.1
Downloads last month
0
Inference Providers NEW
This model is not currently available via any of the supported Inference Providers.
The model cannot be deployed to the HF Inference API: The model has no pipeline_tag.

Model tree for lesso/ee5dcac6-f384-4e2d-ab77-871303131dd3

Adapter
(283)
this model