metadata
license: apache-2.0
pipeline_tag: image-text-to-text
library_name: transformers
GEM: Empowering MLLM for Grounded ECG Understanding with Time Series and Images
Introduction
GEM is a multimodal LLM unifying ECG time series, 12-lead ECG images and text for grounded and clinician-aligned ECG interpretation. GEM enables feature-grounded analysis, evidence-driven reasoning, and a clinician-like diagnostic process.
π₯ Updates
Paper: π Arxiv
Model: π€ GEM
Data: π€ ECG-Grounding
Setup
git clone https://github.com/lanxiang1017/GEM.git
bash GEM/setup.sh
Data Preparation
Please download required data:
ECG:
Images:
- ECG-Grounding-Images (mimic_gen)
- ECG-Bench
After downloading all of them, organize the data as follows in ./data
,
βββ ecg_timeseries
βββ champan-shaoxing
βββ code15
βββ cpsc2018
βββ ptbxl
βββ georgia
βββ mimic-iv
βββ ecg_images
βββ cod15_v4
βββ csn_aug_all_layout_papersize
βββ csn_ori_layout_papersize
βββ csn_part_noise_layout_papersize
βββ gen_images
βββ mimic_gen
βββ mimic
βββ mimic_v4
βββ ptb-xl
βββ ecg_bench
βββ images
βββ jsons
βββ ecg_jsons
βββ ECG_Grounding_30k.json
βββ ECG_Grounding_130k.json
βββ ecg_grounding_test_data
βββ ecg-grounding-test.json
Pretrained Model Preparation
Pretrained ECG Encoder:
- ECG-CoCa : place it in
GEM/ecg_coca/open_clip/checkpoint
Pretrained MLLMs:
Train
bash GEM/scripts/train_gem.sh
Evaluation
For ECG-Grounding:
- step 1. generate interpretations:
GEM/evaluation/gem_bench/bench_ecggrounding.sh
- step 2. process interpretations:
GEM/gem_evaluation/process_gem_outputs.ipynb
- step 3. generate GPT evaluation reports:
GEM/gem_evaluation/generate_gpt_eval.py
- step 4. process evaluation reports and get scores:
GEM/gem_evaluation/process_grounding_scores.ipynb
For ECG-Bench:
- step 1. generate results:
GEM/evaluation/gem_bench/bench_ecggrounding.sh
- step 2. evaluate results:
GEM/evaluation/evaluate_ecgbench.py
- step 3. evaluate reports:
GEM/evaluation/eval_report.py
Note: You'll need to specify result paths first in all evaluation scripts
Citation
If you find GEM helpful for your research and applications, please cite our paper:
@misc{lan2025gemempoweringmllmgrounded,
title={GEM: Empowering MLLM for Grounded ECG Understanding with Time Series and Images},
author={Xiang Lan and Feng Wu and Kai He and Qinghao Zhao and Shenda Hong and Mengling Feng},
year={2025},
eprint={2503.06073},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2503.06073},
}
Acknowledgement
We thank the authors of PULSE and ECG-Chat for their publicly released models, datasets, and training codes.