guillaumephd
/

t5-french-base

@@ -10,8 +10,8 @@ library_name: transformers
 # T5-french-base Model
 ## Model Overview
-The T5-French-Base model is a ~250M params only T5 model trained solely on French data from the RedPajama 2 dataset.
-This model was trained for 85,000 steps and was only pre-trained without any supervised training.
 Therefore, this model has to be fine-tuned before it is useable on a downstream task.
 It is intended to serve as a foundation for further fine-tuning and as a starting point for downstream tasks in the French language.
 Since the training compute buget was very limited, the model is mainly useful for research only.
@@ -19,7 +19,7 @@ Since the training compute buget was very limited, the model is mainly useful fo
 ## Model Details
 - Model Architecture: T5 Base, version 1.1 (GEGLU activation in feed-forward hidden layer, rather than ReLU)
 - Training Dataset: RedPajama 2 dataset (French-only)
-- Training Steps: 85,000
 - Tokenizer: T5 Tokenizer
 ## Intended Use
@@ -33,12 +33,13 @@ It may be used as a starting point for fine-tuning on tasks such as:
 ## Limitations
 The T5-French-Base model may not be suitable for user-facing, or production applications.
 It is mainly meant for researchers only.
 The training budget was really limited (85k steps only, ~250M params only, for a final loss of ~1.1).
 The model is a base model that hasn't been fine-tuned yet. As such, it does NOT follow instructions.
 Additionally, the model was trained solely on French data and won't work for tasks that require cross-lingual understanding or multilingual capabilities.
 ## Ethical Considerations
-The T5-French-Base model was trained on publicly available data and does not contain any known biases or ethical concerns.
 However, researchers should be aware of potential biases in the RedPajama 2 training data and should carefully evaluate the model's outputs for any unintended consequences.
 ## Citation

 # T5-french-base Model
 ## Model Overview
+The T5-French-Base model is a ~250M params only T5 model trained (entirely from scratch) solely on French data from the RedPajama 2 dataset.
+This model was trained for 85,000 steps and was only pre-trained from scratch without any supervised training.
 Therefore, this model has to be fine-tuned before it is useable on a downstream task.
 It is intended to serve as a foundation for further fine-tuning and as a starting point for downstream tasks in the French language.
 Since the training compute buget was very limited, the model is mainly useful for research only.
 ## Model Details
 - Model Architecture: T5 Base, version 1.1 (GEGLU activation in feed-forward hidden layer, rather than ReLU)
 - Training Dataset: RedPajama 2 dataset (French-only)
+- Training Steps: 85,000 (from scratch)
 - Tokenizer: T5 Tokenizer
 ## Intended Use
 ## Limitations
 The T5-French-Base model may not be suitable for user-facing, or production applications.
 It is mainly meant for researchers only.
+It was trained entirely from scratch.
 The training budget was really limited (85k steps only, ~250M params only, for a final loss of ~1.1).
 The model is a base model that hasn't been fine-tuned yet. As such, it does NOT follow instructions.
 Additionally, the model was trained solely on French data and won't work for tasks that require cross-lingual understanding or multilingual capabilities.
 ## Ethical Considerations
+The T5-French-Base model was trained from scratch on publicly available data and does not contain any known biases or ethical concerns.
 However, researchers should be aware of potential biases in the RedPajama 2 training data and should carefully evaluate the model's outputs for any unintended consequences.
 ## Citation