ntyazh/llm-course-hw3-tinyllama-qlora

Model Card for Model ID

The model was created for the task of tweet classification with 3 classes: positive tweet, neutral or negative. It is an adapter for the default TinyLlama/TinyLlama-1.1B-Chat-v1.0 and unfortunately it degrades f1-score on the task from 0.18 to 0.12.

Training Details

Training Data

The model was trained on cardiffnlp/tweet_eval that was created exactly for the given task -- tweet classification.

Training Procedure

The model was trained with trl SFTTrainer for ~260 iterations. LR was cosine with start 5e-5, effective batch size was 32. The rank for the matricies was standard 8, alpha -- 16. AdamW was used as the optimizer.LoRA layers were adapted for v_proj and k_proj layers. Quant type was normal float4, type for the computations was bfloat16

Results

f1 score was degraded from 0.18 to 0.12. I have should to try another hyperparameters and use more epochs for training, I guess