license: openrail
language:
- en
base_model:
- CompVis/stable-diffusion-v1-4
pipeline_tag: text-to-image
library_name: diffusers
tags:
- medical
- X-ray
- Diffusion
- Generation
- Text-to-image
- stable-diffusion
- lora
- fine_tune
widget:
- text: >-
Hey doc, I've been feeling really out of breath lately, especially when
I'm walking up a flight of stairs or doing some light exercise. It's like
my chest gets tight and I can't catch my breath. I've also been coughing
up some stuff that's not quite right, it's been a few weeks now. And I've
noticed a bit of weight loss, I'm not sure if that's related but it's been
on my mind. I've been to a few doctors already, but they haven't been able
to figure out what's going on. I'm hoping you can help.
output:
url: example.png
Symptom-to-Medical-Image Generator
This project introduces a text-to-image diffusion model fine-tuned using LoRA (Low-Rank Adaptation) on top of CompVis/stable-diffusion-v1-4 for the task of medical image generation. The model generates X-ray, CT, or MRI scans based on natural language descriptions of patient symptoms, offering a novel way to visualize potential diagnostic outcomes.
What Is This Model?
This is a domain-adapted diffusion model tailored to generate realistic medical scans conditioned on symptom prompts. The model was fine-tuned using LoRA, which allowed for:
- Efficient training without modifying the original model weights.
- Adaptation to a smaller, highly-specialized medical dataset.
- Retention of high-quality generative capabilities from the base model.
Key Features
- Symptom-to-scan generation: Input symptoms in plain English and receive a plausible X-ray, CT, or MRI image.
- Multi-modality support: Generate different types of scans (e.g., chest X-rays, brain MRIs) depending on the prompt context.
- High realism: Outputs are visually realistic and follow anatomical structure, trained using real medical datasets.
When Can You Use This Model?
Use Cases
| Application Area | Description |
|---|---|
| Medical Research | Generate datasets for hypothesis testing or model training without using real patient data. |
| Education & Training | Teach students about correlations between symptoms and imaging in an interactive way. |
| AI-Aided Prototyping | Test downstream diagnostic pipelines on synthetic but realistic image data. |
| Data Augmentation | Enrich datasets for training classification/segmentation models. |
| Prompt-Based Exploration | Investigate how changes in symptoms affect image generation (e.g., how “fever + cough” differs from “chest pain + shortness of breath”). |
Not for Use In:
- Real-world clinical diagnosis or decision-making
- Generating scans for real patients or influencing treatment
- Bypassing ethical or regulatory controls in medical AI
Example Usage
Input Prompt:
"I've been feeling really out of breath lately, especially when I'm walking up a flight of stairs or doing some light exercise. It's like my chest gets tight and I can't catch my breath. "
Output:
The model generates a chest X-ray image that corresponds to symptoms of a potential pulmonary issue.
Under the Hood
- Base Model:
CompVis/stable-diffusion-v1-4 - Fine-tuning Method: LoRA (efficient, parameter-light adaptation)
- Dataset: Custom dataset of symptom-to-image pairs, curated for medical imaging consistency
- Framework: PyTorch + 🤗 Diffusers + Hugging Face Spaces
Ethical & Legal Disclaimer
This model is strictly intended for research and educational use. It is not a substitute for professional medical judgment. Use of synthetic medical images should follow all local regulatory and ethical guidelines.