CT-ModernBERT-JPN

CT-ModernBERT-JPN is a Japanese ModernBERT-based model for multilabel classification of abnormal findings in radiology reports, fine-tuned on the CT-RATE-JPN dataset. This model offers superior tokenization efficiency and training speed compared to conventional BERT models while maintaining comparable classification performance.

Model Overview

This model is based on ModernBERT Japanese 130M, and has been fine-tuned on the CT-RATE-JPN dataset, which provides Japanese translations of radiology reports from the CT-RATE dataset. The training data consists of deduplicated radiology reports with corresponding abnormality labels for 18 common chest CT conditions.

Key Advantages of ModernBERT

• Superior Tokenization Efficiency: Requires 24.0% fewer tokens per document compared to BERT Base
• Faster Training: Completes training 39% faster than BERT Base
• Improved Processing Speed: 1.65× faster during training and 1.66× faster during inference
• Comparable Classification Performance: Achieves similar or slightly better performance across most conditions

How to Use

from transformers import AutoModelForSequenceClassification, AutoTokenizer
import torch

# Load the model and tokenizer from Hugging Face Hub
model_name = "YYama0/CT-ModernBERT-JPN"
model = AutoModelForSequenceClassification.from_pretrained(
    model_name,
    num_labels=18,
    problem_type="multi_label_classification"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Define the inference function
def infer(input_texts):
    inputs = tokenizer(input_texts, padding=True, truncation=True, return_tensors="pt")
    model.eval()
    with torch.no_grad():
        outputs = model(**inputs)
    logits = outputs.logits
    probs = torch.sigmoid(logits)
    return probs

# Run inference
input_texts = ["気管および両主気管支の内腔は開存しています。気管および両主気管支の内腔には閉塞病変は認められませんでした。縦隔内の主要血管構造、心臓の輪郭、サイズは正常です。胸部大動脈の直径は正常です。心嚢水、心膜肥厚は確認されませんでした。胸部食道径は正常であり、非造影検査の範囲内で有意な病的壁肥厚は認められませんでした。縦隔内の上下部気管傍、大動脈肺動脈窓の血管前領域、および気管分岐下において、最大短軸が7mmのリンパ節が認められました。肺野条件では、右側の胸膜葉の間に最大で8cmの厚さに達する広範な胸水が認められました。左側では、最も広い部分で26mmです。隣接する肺実質には、特に右側でびまん性の無気肺変化が認められました。加えて、両肺で小葉間隔壁の肥厚を伴うびまん性のすりガラス陰影の増加およびcrazy paving appearancesが認められました。これらの所見は感染症過程と一致している可能性があります。肺水腫も鑑別診断に考慮されるべきです。臨床および検査との対比を考慮すること、および治療後の管理が推奨されます。両肺にミリ単位の非特異的な実質性結節がいくつか認められました。検査範囲の上腹部では、有意な病変は認められませんでした。骨構造においても溶骨性破壊病変は認められませんでした。"]
probs = infer(input_texts)

Model Performance

The following table shows the F1 scores for each medical condition on the validation data (n=150) of CT-RATE-JPN, compared with the BERT Base model. For metrics requiring binarization, a threshold of 0.5 was used.

Findings	ModernBERT F1	BERT Base F1	Difference
Medical material	86.70%	87.50%	-0.80%
Arterial wall calcification	100.00%	98.00%	+2.00%
Cardiomegaly	95.80%	95.80%	±0%
Pericardial effusion	95.70%	100.00%	-4.30%
Coronary artery wall calcification	97.80%	97.80%	±0%
Hiatal hernia	100.00%	100.00%	±0%
Lymphadenopathy	97.30%	97.30%	±0%
Emphysema	96.90%	95.20%	+1.70%
Atelectasis	100.00%	99.00%	+1.00%
Lung nodule	97.50%	96.90%	+0.60%
Lung opacity	94.50%	93.70%	+0.80%
Pulmonary fibrotic sequela	89.70%	92.50%	-2.80%
Pleural effusion	97.40%	95.00%	+2.40%
Mosaic attenuation pattern	100.00%	100.00%	±0%
Peribronchial thickening	84.20%	83.30%	+0.90%
Consolidation	80.70%	82.80%	-2.10%
Bronchiectasis	95.20%	93.00%	+2.20%
Interlobular septal thickening	93.30%	93.30%	±0%

Overall Exact Match Accuracy: 74.67% (vs. 72.67% for BERT Base)

Efficiency Comparison

Metric	ModernBERT	BERT Base	Improvement
Mean tokens per document	258.1	339.6	24.0% reduction
Training time (seconds)	1877.67	3090.54	39.0% reduction
Training throughput (samples/sec)	38.82	23.58	1.65× faster
Inference throughput (samples/sec)	139.90	84.18	1.66× faster

Citation

Base ModernBERT Model:

Please cite the original ModernBERT Japanese model from sbintuitions/modernbert-ja-130m.

ModernBERT Architecture:

@misc{warner2024smarterbetterfasterlonger,
      title={Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference}, 
      author={Benjamin Warner and Antoine Chaffin and Benjamin Clavié and Orion Weller and Oskar Hallström and Said Taghadouini and Alexis Gallagher and Raja Biswas and Faisal Ladhak and Tom Aarsen and Nathan Cooper and Griffin Adams and Jeremy Howard and Iacopo Poli},
      year={2024},
      eprint={2412.13663},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2412.13663}, 
}

CT-RATE Dataset:

Please visit the original CT-RATE repository for the appropriate citation information.

CT-RATE-JPN and CT-ModernBERT-JPN:

@misc{yamagishi2024ctrep,
      title={Development of a Large-scale Dataset of Chest Computed Tomography Reports in Japanese and a High-performance Finding Classification Model}, 
      author={Yosuke Yamagishi and Yuta Nakamura and Tomohiro Kikuchi and Yuki Sonoda and Hiroshi Hirakawa and Shintaro Kano and Satoshi Nakamura and Shouhei Hanaoka and Takeharu Yoshikawa and Osamu Abe},
      year={2024},
      eprint={2412.15907},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2412.15907}, 
}

For the comparative study between BERT Base and ModernBERT on Japanese radiology reports (which produced this model), please cite:

@misc{yamagishi2025modernber,
      title={ModernBERT is More Efficient than Conventional BERT for Chest CT Findings Classification in Japanese Radiology Reports}, 
      author={Yosuke Yamagishi and Tomohiro Kikuchi and Shouhei Hanaoka and Takeharu Yoshikawa and Osamu Abe},
      year={2025},
      eprint={2503.05060},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2503.05060}, 
}

License

This model was trained using the CT-RATE-JPN dataset, which is released under the Creative Commons Attribution-NonCommercial-ShareAlike (CC BY-NC-SA) license.

Regarding model usage:

• The model outputs and weights can be used for non-commercial research purposes only
• When using the dataset, users must comply with the terms of the original CC BY-NC-SA license

Acknowledgments

• The original ModernBERT Japanese model developers (SB Intuitions)
• The CT-RATE dataset creators
• The CT-RATE-JPN dataset creators