McGill-DMaS
/

DMaS-LLaMa-Lite-step-3.3k

Model card Files Files and versions Community

MilesQLi commited on Dec 17, 2024

Commit

bb3947b

·

verified ·

1 Parent(s): 23e75e2

Update README.md

Files changed (1) hide show

README.md +1 -1

README.md CHANGED Viewed

@@ -17,7 +17,7 @@ This repository provides access to **DMaS-LLaMa-Lite-step-3.3k**, a 1.7-billion-
 - **Parameters**: 1.7B (36 layers, 32 attention heads, RMSNorm)
 - **Tokenizer**: GPT-2 tokenizer
 - **Training Data**: FineWeb-Edu subset (educational text)
-- **Training Steps**: 2,700
 - **Optimizer**: AdamW with linear warmup and decay
 - **Hardware**: Trained on 1-2 RTX A6000 GPUs with PyTorch DDP
 - **Dataset Source**: [FineWeb-Edu Dataset](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu)

 - **Parameters**: 1.7B (36 layers, 32 attention heads, RMSNorm)
 - **Tokenizer**: GPT-2 tokenizer
 - **Training Data**: FineWeb-Edu subset (educational text)
+- **Training Steps**: 3,300
 - **Optimizer**: AdamW with linear warmup and decay
 - **Hardware**: Trained on 1-2 RTX A6000 GPUs with PyTorch DDP
 - **Dataset Source**: [FineWeb-Edu Dataset](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu)