Update README.md
Browse files
README.md
CHANGED
|
@@ -151,9 +151,11 @@ See the Falcon 180B model card for an example of this.
|
|
| 151 |
## Hyperparamters
|
| 152 |
|
| 153 |
DPO:
|
| 154 |
-
- **Learning Rate**: 5 × 10⁻⁷ (8B), 2.0e-7 (70B)
|
| 155 |
- **Learning Rate Schedule**: Linear
|
| 156 |
-
- **Batch Size (effective)**: 32 (8B), 128 (70B)
|
|
|
|
|
|
|
| 157 |
- **Max Sequence Length**: 2,048
|
| 158 |
- **Epochs**: 1
|
| 159 |
|
|
|
|
| 151 |
## Hyperparamters
|
| 152 |
|
| 153 |
DPO:
|
| 154 |
+
- **Learning Rate**: 5 × 10⁻⁷ (8B), 2.0e-7 (70B, 405B)
|
| 155 |
- **Learning Rate Schedule**: Linear
|
| 156 |
+
- **Batch Size (effective)**: 32 (8B), 128 (70B), 256(405B)
|
| 157 |
+
- **KL Penalty Coefficient**: 5
|
| 158 |
+
- **Warm-up Ratio**: 0.1
|
| 159 |
- **Max Sequence Length**: 2,048
|
| 160 |
- **Epochs**: 1
|
| 161 |
|