You need to agree to share your contact information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

LLaDA-8B-Instruct

This model is a fine-tuned version of sengi/lladou-gsm8k-b32 on an unknown dataset. It achieves the following results on the evaluation set:

More information needed

More information needed

More information needed

The following hyperparameters were used during training:

learning_rate: 5e-05
train_batch_size: 10
eval_batch_size: 32
seed: 42
distributed_type: multi-GPU
gradient_accumulation_steps: 4
total_train_batch_size: 40
optimizer: Use adamw_torch_fused with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
lr_scheduler_type: cosine
lr_scheduler_warmup_steps: 200
training_steps: 10000

Training Loss	Epoch	Step	Validation Loss
12.5223	0.001	1000	19.8564
15.1946	0.002	2000	20.4100
-13.4932	0.3	3000	-12.4603
-12.5083	0.4	4000	-9.9804
-11.9398	0.5	5000	-9.1448
-15.111	1.0532	6000	-9.1216
-16.8137	1.1532	7000	-9.8617
-14.6685	1.2532	8000	-10.7716
-13.1242	1.3532	9000	-9.1976
-12.7972	1.4532	10000	-9.4135

Safetensors

Model size

8B params

Tensor type

BF16

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Base model

Finetuned

(1)

this model