README.md

---
license: apache-2.0
datasets:
- HuggingFaceFW/fineweb-edu
- yahma/alpaca-cleaned
---

# DMaS-LLaMa-Lite-step-43.5k-instruct

This repository provides access to **DMaS-LLaMa-Lite-step-43.5k-instruct**, an instruction fine-tuned version of the DMaS-LLaMa-Lite model. Based on the **43,500-step checkpoint**, this model has been further fine-tuned on question-answering datasets to enable instruction-following and dialogue capabilities. 

It serves as a demonstration of the potential to fine-tune a pretrained LLaMa-based model for downstream tasks, such as question-answering, with improved alignment for human-like responses.

## Model Overview

- **Base Model**: [DMaS-LLaMa-Lite-step-43.5k](https://huggingface.co/McGill-DMaS/DMaS-LLaMa-Lite-step-43.5k)
- **Architecture**: LLaMa-based
- **Parameters**: 1.7B (36 layers, 32 attention heads, RMSNorm)
- **Tokenizer**: GPT-2 tokenizer
- **Fine-tuning Data**: [yahma/alpaca-cleaned](https://huggingface.co/datasets/yahma/alpaca-cleaned)
- **Chat Template**: Follows Vicuna 1.1 format
- **Response End**: Machine answers are terminated with `<|endoftext|>`
- **Objective**: Fine-tune the pretrained model to improve alignment with user instructions and dialogue-based question answering

This model showcases how a LLaMa-based pretrained checkpoint can be effectively adapted for instruction-following tasks, resulting in coherent, relevant, and contextually grounded completions.

## Key Features

- **Instruction Fine-tuning**: Demonstrates alignment to question-answering and dialogue tasks.
- **Chat Template Compatibility**: Compatible with Vicuna 1.1-style formatting, ensuring easy integration with chat-based applications.
- **Early Demonstration**: Highlights the feasibility of fine-tuning pretrained models with minimal downstream data.

## Chat Template

The model follows the Vicuna 1.1 chat format:

```text
A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions.

USER: {user input}
ASSISTANT:{machine response}<|endoftext|>
```

## Usage

You can load and use the model with Hugging Face's Transformers library:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "McGill-DMaS/DMaS-LLaMa-Lite-step-43.5k-instruct"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

# Define the prompt in Vicuna 1.1 format
prompt = "A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions.\n\nUSER: What are the Pyramids of Giza known for?\nASSISTANT:"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_length=100)

# Decode the machine response
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)
```

## Citation

If you use this model or its training insights in your work, please cite the following [paper](https://arxiv.org/abs/2412.13335):

```bibtex
@article{li2024effectiveness,
  title={Experience of Training a 1.7B-Parameter LLaMa Model From Scratch},
  author={Li, Miles Q and Fung, Benjamin and Huang, Shih-Chia},
  journal={arXiv preprint arXiv:2412.13335},
  year={2024}
}
```

## License

This model and code are released under the **Apache License 2.0**. Please check the respective repositories for detailed terms.

## Training Code

For details on pre-training processes, the scripts are available at:  
📄 **[DMaS-LLaMa-Lite Training Code](https://github.com/McGill-DMaS/DMaS-LLaMa-Lite-Training-Code)**