|
--- |
|
library_name: transformers |
|
license: apache-2.0 |
|
base_model: distilbert/distilbert-base-uncased |
|
tags: |
|
- masked_language_modeling, |
|
- domain_addaption |
|
model-index: |
|
- name: distilbert-med-v2 |
|
results: [] |
|
datasets: |
|
- FreedomIntelligence/medical-o1-reasoning-SFT |
|
language: |
|
- en |
|
--- |
|
|
|
# Model Card: Fine-Tuned Language Model for Medical Reasoning |
|
|
|
## **Model Overview** |
|
This language model has been fine-tuned on the [FreedomIntelligence/medical-o1-reasoning-SFT](https://huggingface.co/datasets/FreedomIntelligence/medical-o1-reasoning-SFT). The fine-tuning process focused on enhancing the model's ability to reason and provide contextually accurate predictions within the medical domain. Domain adaptation was achieved through fine-tuning the pretrained model on the medical reasoning dataset using the Masked Language Modeling (MLM) objective. This approach allowed the model to better understand domain-specific terminology and context by predicting masked tokens based on surrounding words. This model can assist in tasks requiring reasoning over medical text, such as medical Q&A, report generation, or other domain-specific NLP applications. |
|
|
|
|
|
--- |
|
|
|
## **Intended Use** |
|
### **Primary Applications** |
|
- Medical reasoning tasks. |
|
- Text completion or generation in the medical domain. |
|
- Context-aware masked token prediction for medical content. |
|
|
|
### **Limitations** |
|
- This model is fine-tuned for **medical reasoning** and might not generalize well to other domains. |
|
- It is not a substitute for professional medical advice or diagnosis. |
|
|
|
### **Usage Warnings** |
|
- Outputs should always be reviewed by qualified professionals. |
|
- The model may produce incorrect or misleading results when provided with ambiguous or incomplete context. |
|
|
|
--- |
|
|
|
## **Fine-Tuning Details** |
|
### **Pretrained Model** |
|
- Base model:[ `distilbert/distilbert-base-uncased`](https://huggingface.co/distilbert/distilbert-base-uncased) |
|
|
|
### **Fine-Tuning Objective** |
|
The model was fine-tuned using the **Masked Language Modeling (MLM)** objective. Both **subword masking** and **whole word masking** strategies were experimented with, but no significant performance difference was observed between the two approaches. |
|
|
|
|
|
### **Dataset** |
|
- Dataset: [FreedomIntelligence/medical-o1-reasoning-SFT](https://huggingface.co/datasets/FreedomIntelligence/medical-o1-reasoning-SFT) |
|
- Dataset Description: This dataset is designed for reasoning tasks in the medical domain and includes diverse medical scenarios. |
|
|
|
### **Training Setup** |
|
- **Hardware**: T4 GPU (Google Colab) |
|
- **Software**: - Transformers 4.48.1 |
|
- Pytorch 2.5.1+cu121 |
|
- Datasets 3.2.0 |
|
- Tokenizers 0.21.0 |
|
|
|
--- |
|
|
|
## **Performance Metrics** |
|
- **Evaluation Metrics**: Cross-entropy loss and perplexity were used to evaluate model performance. |
|
- **Results**: |
|
- Cross-Entropy Loss: 1.2834 |
|
- Perplexity: 3.6 |
|
|
|
### **Observations** |
|
- Both subword masking and whole word masking yielded similar performance metrics, indicating no significant advantage of one masking strategy over the other. |
|
|
|
--- |
|
|
|
### **Limitations and Biases** |
|
- The model's performance is dependent on the quality and representativeness of the dataset. |
|
- Potential biases in the dataset may propagate into the model outputs. |
|
- The model might not handle rare or out-of-distribution medical terms effectively. |
|
|
|
|
|
|
|
|
|
## Training results |
|
|
|
| Training Loss | Epoch | Step | Validation Loss | |
|
|:-------------:|:-----:|:----:|:---------------:| |
|
| 1.4437 | 1.0 | 263 | 1.4178 | |
|
| 1.4236 | 2.0 | 526 | 1.3673 | |
|
| 1.4316 | 3.0 | 789 | 1.3320 | |
|
| 1.4174 | 4.0 | 1052 | 1.3123 | |
|
| 1.4015 | 5.0 | 1315 | 1.2988 | |
|
| 1.3922 | 6.0 | 1578 | 1.2834 | |
|
|
