---
library_name: peft
license: mit
base_model: fxmarty/really-tiny-falcon-testing
tags:
- axolotl
- generated_from_trainer
model-index:
- name: 418e6629-dcd3-424e-b3db-50c11d0d7c6b
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

[<img src="https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/axolotl-ai-cloud/axolotl)
<br>

# 418e6629-dcd3-424e-b3db-50c11d0d7c6b

This model is a fine-tuned version of [fxmarty/really-tiny-falcon-testing](https://huggingface.co/fxmarty/really-tiny-falcon-testing) on the None dataset.
It achieves the following results on the evaluation set:
- Loss: 10.9416

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.000214
- train_batch_size: 4
- eval_batch_size: 4
- seed: 140
- gradient_accumulation_steps: 8
- total_train_batch_size: 32
- optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 100
- training_steps: 25000

### Training results

| Training Loss | Epoch  | Step  | Validation Loss |
|:-------------:|:------:|:-----:|:---------------:|
| No log        | 0.0001 | 1     | 11.0842         |
| 87.9206       | 0.0563 | 500   | 10.9867         |
| 87.8458       | 0.1127 | 1000  | 10.9740         |
| 87.8012       | 0.1690 | 1500  | 10.9673         |
| 87.7685       | 0.2254 | 2000  | 10.9628         |
| 87.7475       | 0.2817 | 2500  | 10.9602         |
| 87.7401       | 0.3381 | 3000  | 10.9575         |
| 87.7205       | 0.3944 | 3500  | 10.9554         |
| 87.6986       | 0.4508 | 4000  | 10.9532         |
| 87.6961       | 0.5071 | 4500  | 10.9514         |
| 87.677        | 0.5635 | 5000  | 10.9498         |
| 87.6867       | 0.6198 | 5500  | 10.9488         |
| 87.6773       | 0.6762 | 6000  | 10.9481         |
| 87.6696       | 0.7325 | 6500  | 10.9472         |
| 87.664        | 0.7889 | 7000  | 10.9466         |
| 87.6684       | 0.8452 | 7500  | 10.9463         |
| 87.6553       | 0.9015 | 8000  | 10.9458         |
| 87.6592       | 0.9579 | 8500  | 10.9454         |
| 87.6622       | 1.0143 | 9000  | 10.9448         |
| 87.6536       | 1.0706 | 9500  | 10.9444         |
| 87.6519       | 1.1270 | 10000 | 10.9442         |
| 87.6443       | 1.1833 | 10500 | 10.9439         |
| 87.6468       | 1.2397 | 11000 | 10.9438         |
| 87.6477       | 1.2960 | 11500 | 10.9434         |
| 87.6381       | 1.3524 | 12000 | 10.9433         |
| 87.6359       | 1.4087 | 12500 | 10.9430         |
| 87.6432       | 1.4650 | 13000 | 10.9428         |
| 87.6428       | 1.5214 | 13500 | 10.9428         |
| 87.634        | 1.5777 | 14000 | 10.9426         |
| 87.6254       | 1.6341 | 14500 | 10.9425         |
| 87.6294       | 1.6904 | 15000 | 10.9424         |
| 87.6307       | 1.7468 | 15500 | 10.9422         |
| 87.6393       | 1.8031 | 16000 | 10.9422         |
| 87.6266       | 1.8595 | 16500 | 10.9421         |
| 87.6327       | 1.9158 | 17000 | 10.9419         |
| 87.6298       | 1.9722 | 17500 | 10.9419         |
| 87.6353       | 2.0285 | 18000 | 10.9420         |
| 87.6329       | 2.0849 | 18500 | 10.9418         |
| 87.6332       | 2.1412 | 19000 | 10.9418         |
| 87.6301       | 2.1976 | 19500 | 10.9417         |
| 87.6308       | 2.2539 | 20000 | 10.9417         |
| 87.6302       | 2.3103 | 20500 | 10.9417         |
| 87.6378       | 2.3666 | 21000 | 10.9416         |
| 87.6317       | 2.4230 | 21500 | 10.9416         |
| 87.6272       | 2.4793 | 22000 | 10.9416         |
| 87.6308       | 2.5357 | 22500 | 10.9416         |
| 87.6299       | 2.5920 | 23000 | 10.9416         |
| 87.6303       | 2.6484 | 23500 | 10.9416         |
| 87.6262       | 2.7047 | 24000 | 10.9415         |
| 87.6281       | 2.7610 | 24500 | 10.9415         |
| 87.6344       | 2.8174 | 25000 | 10.9416         |


### Framework versions

- PEFT 0.13.2
- Transformers 4.46.0
- Pytorch 2.5.0+cu124
- Datasets 3.0.1
- Tokenizers 0.20.1