---
library_name: peft
license: apache-2.0
base_model: mistralai/Mistral-7B-v0.3
tags:
- trl
- sft
- medical
- diagnosis
- generated_from_trainer
model-index:
- name: Disease-Diagnosis-Mistral-7B-v0.3LoRA64
  results: []
datasets:
- sajjadhadi/disease-diagnosis-dataset
metrics:
- accuracy
language:
- en
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. -->

## Disclaimer

**Important Notice**: This fine-tuned disease diagnosis model is intended **STRICTLY** for research and educational purposes. It is **NOT CERTIFIED** for clinical use and should **NOT** be used for real medical diagnosis. The model may generate inaccurate or incomplete suggestions based on its training data.

**Medical Disclaimer**: This AI system is not a substitute for professional medical advice, diagnosis, or treatment. Always seek the guidance of qualified healthcare providers with any questions regarding medical conditions. The developers disclaim all responsibility for any decisions made based on this model's outputs.

## Model Description

This model is a LoRA-adapted version of Mistral-7B-v0.3, specifically fine-tuned for disease diagnosis research. Key features:

- **Architecture**: 7B parameter transformer with LoRA adapters (r=32, alpha=64)
- **Training Approach**: 4-bit quantized training with NF4 quantization
- **Capabilities**: Symptom analysis and differential diagnosis suggestion
- **Context Window**: 8k tokens for comprehensive case analysis


## Metrics
| Metric | Accuracy |
|-----------|-------|
| Top-1 | 87% |
| Top-3 | 92% |

Check out [this Notebook](https://www.kaggle.com/code/sajjadhadi/disease-diagnosis-test-fine-tuned-mistral7b-tk) for more information about test results.

## Training Configuration

### Hyperparameters

| Parameter | Value |
|-----------|-------|
| Learning Rate | 1e-4 |
| Batch Size | 64 |
| Sequence Length | 93 tokens |
| Optimizer | Paged AdamW 32-bit |
| Scheduler | Cosine with 3% Warmup |
| LoRA Rank | 32 |
| LoRA Alpha | 64 |
| Epochs | 1 |

### Technical Stack

- **PEFT**: 0.14.0
- **Transformers**: 4.49.0
- **PyTorch**: 2.6.0+cu124  
- **Datasets**: 3.3.2
- **Bitsandbytes**: 0.43.0
- **TRL**: 0.8.6

## Intended Use

**Research Applications**:
- Medical education simulations
- Symptom pattern analysis
- Diagnostic decision support systems research

**Prohibited Uses**:
- Clinical diagnosis
- Patient treatment decisions
- Self-diagnosis tools


## Ethical Considerations

- **Bias Mitigation**: Regularization techniques applied during training
- **Privacy**: No real patient data used in training
- **Transparency**: Full architecture details available in configuration