---
library_name: transformers
license: apache-2.0
base_model: distilbert/distilbert-base-uncased
tags:
- masked_language_modeling,
- domain_addaption
model-index:
- name: distilbert-med-v2
  results: []
datasets:
- FreedomIntelligence/medical-o1-reasoning-SFT
language:
- en
---

# Model Card: Fine-Tuned Language Model for Medical Reasoning

## **Model Overview**
This language model has been fine-tuned on the [FreedomIntelligence/medical-o1-reasoning-SFT](https://huggingface.co/datasets/FreedomIntelligence/medical-o1-reasoning-SFT). The fine-tuning process focused on enhancing the model's ability to reason and provide contextually accurate predictions within the medical domain. Domain adaptation was achieved through fine-tuning the pretrained model on the medical reasoning dataset using the Masked Language Modeling (MLM) objective. This approach allowed the model to better understand domain-specific terminology and context by predicting masked tokens based on surrounding words. This model can assist in tasks requiring reasoning over medical text, such as medical Q&A, report generation, or other domain-specific NLP applications.


---

## **Intended Use**
### **Primary Applications**
- Medical reasoning tasks.
- Text completion or generation in the medical domain.
- Context-aware masked token prediction for medical content.

### **Limitations**
- This model is fine-tuned for **medical reasoning** and might not generalize well to other domains.
- It is not a substitute for professional medical advice or diagnosis.

### **Usage Warnings**
- Outputs should always be reviewed by qualified professionals.
- The model may produce incorrect or misleading results when provided with ambiguous or incomplete context.

---

## **Fine-Tuning Details**
### **Pretrained Model**
- Base model:[ `distilbert/distilbert-base-uncased`](https://huggingface.co/distilbert/distilbert-base-uncased)

### **Fine-Tuning Objective**
The model was fine-tuned using the **Masked Language Modeling (MLM)** objective. Both **subword masking** and **whole word masking** strategies were experimented with, but no significant performance difference was observed between the two approaches.


### **Dataset**
- Dataset: [FreedomIntelligence/medical-o1-reasoning-SFT](https://huggingface.co/datasets/FreedomIntelligence/medical-o1-reasoning-SFT)
- Dataset Description: This dataset is designed for reasoning tasks in the medical domain and includes diverse medical scenarios.

### **Training Setup**
- **Hardware**: T4 GPU (Google Colab)
- **Software**: - Transformers 4.48.1
                - Pytorch 2.5.1+cu121
                - Datasets 3.2.0
                - Tokenizers 0.21.0

---

## **Performance Metrics**
- **Evaluation Metrics**: Cross-entropy loss and perplexity were used to evaluate model performance.
- **Results**:
  - Cross-Entropy Loss: 1.2834 
  - Perplexity: 3.6

### **Observations**
- Both subword masking and whole word masking yielded similar performance metrics, indicating no significant advantage of one masking strategy over the other.

---

### **Limitations and Biases**
- The model's performance is dependent on the quality and representativeness of the dataset.
- Potential biases in the dataset may propagate into the model outputs.
- The model might not handle rare or out-of-distribution medical terms effectively.




## Training results

| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 1.4437        | 1.0   | 263  | 1.4178          |
| 1.4236        | 2.0   | 526  | 1.3673          |
| 1.4316        | 3.0   | 789  | 1.3320          |
| 1.4174        | 4.0   | 1052 | 1.3123          |
| 1.4015        | 5.0   | 1315 | 1.2988          |
| 1.3922        | 6.0   | 1578 | 1.2834          |