Create README.md

README.md

@@ -0,0 +1,19 @@
+| Model Name | Parameters | Class | Ratio | Tokens | Batch Size (Tokens) | Training Loss ↓ | hellaswag acc ↑ | lambada ppl ↓ | lambada acc ↑ | winogrande acc ↑
+| --- | --- | --- | --- | --- |  --- |  --- |   --- |   --- |   --- |   --- | 
+| [GerbilLab/GerbilBlender-A-15m](https://hf.co/GerbilLab/GerbilBlender-A-15m) | 15m | A-Class | 20 | 280M | 131k | 4.9642 | 
+
+"Blender" models, inspired by UL2 pretraining, are trained equally in fill-in-the-middle, causal modelling, and masked language modelling tasks. Special tokens for these models include:
+
+```
+'<fitm_start>', '<multiple_tok_mask>', '<fitm_result>', '<causal>', '<mlm_start>', '<single_tok_mask>', '<mlm_end>'
+
+# Example fill in the middle
+'<fitm_start> this is an <multiple_tok_mask> for fill-in-the-middle <fitm_result> example text <|endoftext|>'
+
+# Example causal language modelling
+'<causal> this is an example text for causal language modelling <|endoftext|>'
+
+# Example masked language modelling
+'<mlm_start> this is an <single_tok_mask> text for masked language modelling <mlm_end> example <|endoftext|>'
+
+```