DAMO-NLP-SG
/

Mistral-7B-LongPO-512K-EXP

@@ -1,199 +1,150 @@
 ---
 library_name: transformers
-tags: []
 ---
-# Model Card for Model ID
-<!-- Provide a quick summary of what the model is/does. -->
-## Model Details
-### Model Description
-<!-- Provide a longer summary of what this model is. -->
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-### Model Sources [optional]
-<!-- Provide the basic links for the model. -->
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
-## Uses
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-### Direct Use
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-[More Information Needed]
-### Downstream Use [optional]
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-[More Information Needed]
-### Out-of-Scope Use
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-[More Information Needed]
-## Bias, Risks, and Limitations
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-[More Information Needed]
-### Recommendations
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-## How to Get Started with the Model
-Use the code below to get started with the model.
-[More Information Needed]
-## Training Details
-### Training Data
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-[More Information Needed]
-### Training Procedure
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-#### Preprocessing [optional]
-[More Information Needed]
-#### Training Hyperparameters
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-#### Speeds, Sizes, Times [optional]
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-[More Information Needed]
 ## Evaluation
-<!-- This section describes the evaluation protocols and provides the results. -->
-### Testing Data, Factors & Metrics
-#### Testing Data
-<!-- This should link to a Dataset Card if possible. -->
-[More Information Needed]
-#### Factors
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-[More Information Needed]
-#### Metrics
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-[More Information Needed]
-### Results
-[More Information Needed]
-#### Summary
-## Model Examination [optional]
-<!-- Relevant interpretability work for the model goes here -->
-[More Information Needed]
-## Environmental Impact
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-## Technical Specifications [optional]
-### Model Architecture and Objective
-[More Information Needed]
-### Compute Infrastructure
-[More Information Needed]
-#### Hardware
-[More Information Needed]
-#### Software
-[More Information Needed]
-## Citation [optional]
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-**BibTeX:**
-[More Information Needed]
-**APA:**
-[More Information Needed]
-## Glossary [optional]
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-[More Information Needed]
-## More Information [optional]
-[More Information Needed]
-## Model Card Authors [optional]
-[More Information Needed]
-## Model Card Contact
-[More Information Needed]

 ---
 library_name: transformers
+license: apache-2.0
+datasets:
+- DAMO-NLP-SG/Mistral-7B-LongPO-512K-tokenized
+base_model:
+- DAMO-NLP-SG/Mistral-7B-LongPO-128K
 ---
+# LongPO: Long Context Self-Evolution of Large Language Models through Short-to-Long Preference Optimization
+This repo provides the checkpoint of Mistral-7B-LongPO-512K in our paper "LongPO: Long Context Self-Evolution of Large Language Models through Short-to-Long Preference Optimization".
+(Note that it is an experimental an experimental version (for rebuttal purposes) that may have not been fully tuned or provided with sufficient data to achieve convergence.)
+## Highlights of LongPO
+- Self-evolving long-context alignment without human/superior LLMs annotations.
+- Extending context length while keeping aligned in one stage.
+- No degradation on short-context capabilities.
+## Models and Training Data
+| Models                                                       | Base Model               | Training Data                                                | # Data Samples |
+| ------------------------------------------------------------ | ------------------------ | ------------------------------------------------------------ | -------------- |
+| [Mistral-7B-LongPO-128K](https://huggingface.co/DAMO-NLP-SG/Mistral-7B-LongPO-128K) | Mistral-7B-Instruct-v0.2 | [HF Link](https://huggingface.co/datasets/DAMO-NLP-SG/Mistral-7B-LongPO-128K-tokenized) | 45K            |
+| [Qwen2.5-7B-LongPO-128K](https://huggingface.co/DAMO-NLP-SG/Qwen2.5-7B-LongPO-128K) | Qwen2.5-7B-Instruct      | [HF Link](https://huggingface.co/datasets/DAMO-NLP-SG/Qwen2.5-7B-LongPO-128K-tokenized) | 32K            |
+| [Mistral-7B-LongPO-256K-EXP](https://huggingface.co/DAMO-NLP-SG/Mistral-7B-LongPO-256K-EXP)* | Mistral-7B-LongPO-128K   | [HF Link](https://huggingface.co/datasets/DAMO-NLP-SG/Mistral-7B-LongPO-256K-tokenized) | 16K            |
+| [Mistral-7B-LongPO-512K-EXP](https://huggingface.co/DAMO-NLP-SG/Mistral-7B-LongPO-512K-EXP)* | Mistral-7B-LongPO-128K   | [HF Link](https://huggingface.co/datasets/DAMO-NLP-SG/Mistral-7B-LongPO-512K-tokenized) | 2.5K           |
+\* indicates an experimental version (for rebuttal purposes) that may have not been fully tuned or provided with sufficient data to achieve convergence.
+## Training Process:
+1. Prompt a short-context instruct LLM (e.g., Mistral-7B-Instruct-v0.2) to self-generate short-to-long preference data as illustrated in [data_prepare](data_prepare/readme.md).
+2. Replace the (Flash) Attention module into Ulyssess (Flash) Attn using monkey patch to apply sequence parallel.
+3. Using our custom LongPO Trainer: `LongPOMTLMUlyssesTrainer`
+4. Train Script (using Mistral-7B-Instruct-v0.2 as example):
+```
+export training_length=131072
+export gradient_accumulation_steps=8
+export batch_size=1
+accelerate launch \
+--config_file playground/accelerate_single_node_zero3.yaml \
+train/train_longpo.py \
+    --model_name_or_path mistralai/Mistral-7B-Instruct-v0.2 \
+    --ref_model_name_or_path mistralai/Mistral-7B-Instruct-v0.2 \
+    --data_path /path/to/data \
+    --bf16 True \
+    --run_name mistral_longpo \
+    --report_to wandb \
+    --output_dir path/to/save \
+    --num_train_epochs 1 \
+    --per_device_train_batch_size $batch_size \
+    --gradient_accumulation_steps $gradient_accumulation_steps \
+    --save_strategy "steps" \
+    --save_steps 500 \
+    --evaluation_strategy "no" \
+    --learning_rate 5e-7 \
+    --weight_decay 0. \
+    --warmup_ratio 0.1 \
+    --lr_scheduler_type "cosine" \
+    --optim "rmsprop" \
+    --logging_steps 1 \
+    --tf32 True \
+    --model_max_length $training_length \
+    --gradient_checkpointing True \
+    --do_train True \
+    --do_eval False \
+    --do_predict False \
+    --seed 42 \
+    --use_sequence_parallel True \
+    --dpo_beta 0.01 \
+    --dpo_lambda 0.01 \
+    --rope_theta 10000000
+```
 ## Evaluation
+### InfiniteBench
+| Model            | Train/Claimed Length | En.Sum | En.QA  | En.MC  | AVG.   |
+| ---------------- | -------------------- | ------ | ------ | ------ | ------ |
+| GPT-4-128K       | 128K                 | 14.73  | 22.44  | 67.25  | 34.81  |
+| Qwen2-72B        | 128K                 | 24.32ᵇ | 7.03ᵇ  | 72.05ᵇ | 34.47ᵇ |
+| LLaMA 3.1-70B    | 128K                 | 33.55ᵇ | 36.08ᵇ | 69.00ᵇ | 46.21ᵇ |
+| LLaMA 3.1-8B     | 128K                 | 28.06ᵇ | 30.47ᵇ | 58.08ᵇ | 38.87ᵇ |
+| GLM-4-9B         | 128K                 | 14.84ᵇ | 9.51ᵇ  | 67.25ᵇ | 30.53ᵇ |
+| GLM-4-9B-1M      | 1M                   | 28.3   | 9.7    | 68.6   | 35.53  |
+| LWM-7B-1M        | 1M                   | 4.33ᵇ  | 0.0ᵇ   | 3.06ᵇ  | 2.46ᵇ  |
+| YaRN-Mistral-7B  | 128K                 | 9.09   | 9.55   | 27.95  | 15.53  |
+| Mistral-7B       | 32K                  | 22.13  | 4.93   | 14.41  | 13.82  |
+| - SFT            | 128K                 | 23.44  | 13.45  | 53.21  | 30.03  |
+| - DPO            | 128K                 | 15.21  | 10.34  | 48.14  | 25.56  |
+| - LongPO (iter1) | 128K                 | 27.05  | 23.51  | 67.25  | 39.27  |
+| - LongPO (iter2) | 256K                 | 28.16  | 24.43  | 66.35  | 39.65  |
+| - LongPO (iter3) | 512K                 | 29.10  | 27.85  | 66.67  | 41.21  |
+| Qwen2.5-7B       | 128K                 | 22.89  | 6.08   | 52.4   | 27.12  |
+| - LongPO (iter1) | 128K                 | 32.06  | 17.32  | 72.05  | 40.48  |
+- Our results are evaluated with greedy decoding.
+- Baseline results marked with ᵇ are evaluated by us, while unmarked baseline results are sourced from their official report.
+### RULER
+| Model                    | NIAH  | VT    | AGG   | QA    | AVG (13 tasks) |
+| ------------------------ | ----- | ----- | ----- | ----- | -------------- |
+| Qwen2.5-7B-Instruct      | 82.10 | 80.09 | 74.50 | 54.30 | 76.50          |
+| Qwen2.5-7B-LongPO-128K   | 95.82 | 89.71 | 78.67 | 59.40 | 87.11          |
+| Mistral-7B-Instruct-v0.2 | 72.60 | 74.40 | 64.40 | 52.20 | 68.40          |
+| Mistral-7B-LongPO-128K   | 96.88 | 96.49 | 71.55 | 64.81 | 88.02          |
+| Mistral-7B-LongPO-256K   | 96.80 | 97.00 | 69.14 | 64.87 | 87.65          |
+| Mistral-7B-LongPO-512K   | 97.28 | 97.48 | 69.22 | 64.92 | 88.00          |
+### Short Context
+| Model | MMLU | ARC-C | Hellaswag | Winogrande | Avg |
+|-------|-------|--------|------------|-------------|-----|
+| Mistral-7B-Instruction-v0.2 | 59.15 | 59.26 | 83.2 | 78.4 | 70.00 |
+| Mistral-7B-LongPO-128K | 59.99 | 59.34 | 82.99 | 78.53 | 70.21 |
+| Mistral-7B-LongPO-256K-EXP | 59.47 | 60.28 | 83.14 | 78.14 | 70.26 |
+| Mistral-7B-LongPO-512K-EXP | 59.51 | 60.58 | 82.87 | 77.66 | 70.16 |
+| Qwen2.5-7B-Instruct | 74.28 | 67.15 | 81.41 | 74.66 | 74.38 |
+| Qwen2.5-7B-LongPO-128K | 73.64 | 65.70 | 80.82 | 74.98 | 73.79 |