Tissue Concepts: supervised foundation models in computational pathology
Till Nicke Jan Raphael Schaefer Henning Hoefener Friedrich Feuerhake Dorit Merhof Fabian Kiessling Johannes Lotz

preparation
    Euclid Collaboration: J. Adamek0000-0002-0723-6740julian.adamek@uzh.ch<ref>
B. Fiorini0000-0002-0092-4321<ref>
M. Baldi0000-0003-4145-1943<ref>,<ref>,<ref>
G. Brando0000-0003-0805-1905<ref>
M.-A. Breton<ref>,<ref>,<ref>
F. Hassani0000-0003-2640-4460<ref>
K. Koyama0000-0001-6727-6915<ref>
A. M. C. Le Brun0000-0002-0936-4594<ref>
G. Rácz0000-0003-3906-5699<ref>
H.-A. Winther0000-0002-6325-2710<ref>
A. Casalino0000-0001-6709-5292<ref>
C. Hernández-Aguayo0000-0001-9921-8832<ref>
B. Li0000-0002-1098-9188<ref>
D. Potter0000-0002-0757-5195<ref>
E. Altamura0000-0001-6973-1897<ref>
C. Carbone0000-0003-0125-3563<ref>
C. Giocoli0000-0002-9590-7961<ref>,<ref>
D. F. Mota0000-0003-3141-142X<ref>
A. Pourtsidou0000-0001-9110-5550<ref>,<ref>
Z. Sakr0000-0002-4823-3757<ref>,<ref>,<ref>
F. Vernizzi0000-0003-3426-2802<ref>
A. Amara<ref>
S. Andreon0000-0002-2041-8784<ref>
N. Auricchio0000-0003-4444-8651<ref>
C. Baccigalupi0000-0002-8211-1630<ref>,<ref>,<ref>,<ref>
S. Bardelli0000-0002-8900-0298<ref>
P. Battaglia0000-0002-7337-5909<ref>
D. Bonino0000-0002-3336-9977<ref>
E. Branchini0000-0002-0808-6908<ref>,<ref>,<ref>
M. Brescia0000-0001-9506-5680<ref>,<ref>,<ref>
J. Brinchmann0000-0003-4359-8797<ref>,<ref>
A. Caillat<ref>
S. Camera0000-0003-3399-3574<ref>,<ref>,<ref>
V. Capobianco0000-0002-3309-7692<ref>
V. F. Cardone<ref>,<ref>
J. Carretero0000-0002-3130-0204<ref>,<ref>
S. Casas0000-0002-4751-5138<ref>
F. J. Castander0000-0001-7316-4573<ref>,<ref>
M. Castellano0000-0001-9875-8263<ref>
G. Castignani0000-0001-6831-0687<ref>
S. Cavuoti0000-0002-3787-4196<ref>,<ref>
A. Cimatti<ref>
C. Colodro-Conde<ref>
G. Congedo0000-0003-2508-0046<ref>
C. J. Conselice0000-0003-1949-7638<ref>
L. Conversi0000-0002-6710-8476<ref>,<ref>
Y. Copin0000-0002-5317-7518<ref>
F. Courbin0000-0003-0758-6510<ref>,<ref>,<ref>
H. M. Courtois0000-0003-0509-1776<ref>
A. Da Silva0000-0002-6385-1609<ref>,<ref>
H. Degaudenzi0000-0002-5887-6799<ref>
G. De Lucia0000-0002-6220-9104<ref>
M. Douspis0000-0003-4203-3954<ref>
F. Dubath0000-0002-6533-2810<ref>
X. Dupac<ref>
S. Dusini0000-0002-1128-0664<ref>
M. Farina0000-0002-3089-7846<ref>
S. Farrens0000-0002-9594-9387<ref>
S. Ferriol<ref>
P. Fosalba0000-0002-1510-5214<ref>,<ref>
M. Frailis0000-0002-7400-2135<ref>
E. Franceschi0000-0002-0585-6591<ref>
M. Fumana0000-0001-6787-5950<ref>
S. Galeotta0000-0002-3748-5115<ref>
B. Gillis0000-0002-4478-1270<ref>
P. Gómez-Alvarez0000-0002-8594-5358<ref>,<ref>
A. Grazian0000-0002-5688-0663<ref>
F. Grupp<ref>,<ref>
L. Guzzo0000-0001-8264-5192<ref>,<ref>
S. V. H. Haugan0000-0001-9648-7260<ref>
W. Holmes<ref>
F. Hormuth<ref>
A. Hornstrup0000-0002-3363-0936<ref>,<ref>
S. Ilić0000-0003-4285-9086<ref>,<ref>
K. Jahnke0000-0003-3804-2137<ref>
M. Jhabvala<ref>
B. Joachimi0000-0001-7494-1303<ref>
E. Keihänen0000-0003-1804-7715<ref>
S. Kermiche0000-0002-0302-5735<ref>
A. Kiessling0000-0002-2590-1273<ref>
M. Kilbinger0000-0001-9513-7138<ref>
B. Kubik0009-0006-5823-4880<ref>
M. Kümmel0000-0003-2791-2117<ref>
M. Kunz0000-0002-3052-7394<ref>
H. Kurki-Suonio0000-0002-4618-3063<ref>,<ref>
S. Ligori0000-0003-4172-4606<ref>
P. B. Lilje0000-0003-4324-7794<ref>
V. Lindholm0000-0003-2317-5471<ref>,<ref>
I. Lloro<ref>
G. Mainetti0000-0003-2384-2377<ref>
E. Maiorano0000-0003-2593-4355<ref>
O. Mansutti0000-0001-5758-4658<ref>
O. Marggraf0000-0001-7242-3852<ref>
K. Markovic0000-0001-6764-073X<ref>
M. Martinelli0000-0002-6943-7732<ref>,<ref>
N. Martinet0000-0003-2786-7790<ref>
F. Marulli0000-0002-8850-0303<ref>,<ref>,<ref>
R. Massey0000-0002-6085-3780<ref>
E. Medinaceli0000-0002-4040-7783<ref>
S. Mei0000-0002-2849-559X<ref>
M. Melchior<ref>
Y. Mellier<ref>,<ref>
M. Meneghetti0000-0003-1225-7084<ref>,<ref>
E. Merlin0000-0001-6870-8900<ref>
G. Meylan<ref>
M. Moresco0000-0002-7616-7136<ref>,<ref>
L. Moscardini0000-0002-3473-6716<ref>,<ref>,<ref>
C. Neissner0000-0001-8524-4968<ref>,<ref>
S.-M. Niemi<ref>
C. Padilla0000-0001-7951-0166<ref>
S. Paltani0000-0002-8108-9179<ref>
F. Pasian0000-0002-4869-3227<ref>
K. Pedersen<ref>
W. J. Percival0000-0002-0644-5727<ref>,<ref>,<ref>
V. Pettorino<ref>
S. Pires0000-0002-0249-2104<ref>
G. Polenta0000-0003-4067-9196<ref>
M. Poncet<ref>
L. A. Popa<ref>
L. Pozzetti0000-0001-7085-0412<ref>
F. Raison0000-0002-7819-6918<ref>
A. Renzi0000-0001-9856-1970<ref>,<ref>
J. Rhodes0000-0002-4485-8549<ref>
G. Riccio<ref>
E. Romelli0000-0003-3069-9222<ref>
M. Roncarelli0000-0001-9587-7822<ref>
R. Saglia0000-0003-0378-7032<ref>,<ref>
A. G. Sánchez0000-0003-1198-831X<ref>
D. Sapone0000-0001-7089-4503<ref>
B. Sartoris0000-0003-1337-5269<ref>,<ref>
M. Schirmer0000-0003-2568-9994<ref>
T. Schrabback0000-0002-6987-7834<ref>
A. Secroun0000-0003-0505-3710<ref>
G. Seidel0000-0003-2907-353X<ref>
S. Serrano0000-0002-0211-2861<ref>,<ref>,<ref>
C. Sirignano0000-0002-0995-7146<ref>,<ref>
G. Sirri0000-0003-2626-2853<ref>
L. Stanco0000-0002-9706-5104<ref>
J. Steinwagner0000-0001-7443-1047<ref>
P. Tallada-Crespí0000-0002-1336-8328<ref>,<ref>
D. Tavagnacco0000-0001-7475-9894<ref>
I. Tereno<ref>,<ref>
R. Toledo-Moreo0000-0002-2997-4859<ref>
F. Torradeflot0000-0003-1160-1517<ref>,<ref>
I. Tutusaus0000-0002-3199-0399<ref>
E. A. Valentijn<ref>
L. Valenziano0000-0002-1170-0104<ref>,<ref>
T. Vassallo0000-0001-6512-6358<ref>,<ref>
G. Verdoes Kleijn0000-0001-5803-2580<ref>
A. Veropalumbo0000-0003-2387-1194<ref>,<ref>,<ref>
Y. Wang0000-0002-4749-2984<ref>
J. Weller0000-0002-8282-2010<ref>,<ref>
G. Zamorani0000-0002-2318-301X<ref>
E. Zucca0000-0002-5845-8132<ref>
A. Biviano0000-0002-0857-0732<ref>,<ref>
C. Burigana0000-0002-3005-5796<ref>,<ref>
M. Calabrese0000-0002-2637-2422<ref>,<ref>
D. Di Ferdinando<ref>
J. A. Escartin Vigo<ref>
G. Fabbian0000-0002-3255-4695<ref>,<ref>
F. Finelli0000-0002-6694-3269<ref>,<ref>
J. Gracia-Carpio<ref>
S. Matthew0000-0001-8448-1697<ref>
N. Mauri0000-0001-8196-1548<ref>,<ref>
A. Pezzotta0000-0003-0726-2268<ref>
M. Pöntinen0000-0001-5442-2530<ref>
V. Scottez<ref>,<ref>
M. Tenti0000-0002-4254-5901<ref>
M. Viel0000-0002-2642-5707<ref>,<ref>,<ref>,<ref>,<ref>
M. Wiesmann0009-0000-8199-5860<ref>
Y. Akrami0000-0002-2407-7956<ref>,<ref>
V. Allevato0000-0001-7232-5152<ref>
S. Anselmi0000-0002-3579-9583<ref>,<ref>,<ref>
M. Archidiacono0000-0003-4952-9012<ref>,<ref>
F. Atrio-Barandela0000-0002-2130-2513<ref>
A. Balaguera-Antolinez0000-0001-5028-3035<ref>,<ref>
M. Ballardini0000-0003-4481-3559<ref>,<ref>,<ref>
A. Blanchard0000-0001-8555-9003<ref>
L. Blot0000-0002-9622-7167<ref>,<ref>
H. Böhringer0000-0001-8241-4204<ref>,<ref>,<ref>
S. Borgani0000-0001-6151-6439<ref>,<ref>,<ref>,<ref>
S. Bruton0000-0002-6503-5218<ref>
R. Cabanac0000-0001-6679-2600<ref>
A. Calabro0000-0003-2536-1614<ref>
B. Camacho Quevedo0000-0002-8789-4232<ref>,<ref>
G. Cañas-Herrera0000-0003-2796-2149<ref>,<ref>
A. Cappi<ref>,<ref>
F. Caro<ref>
C. S. Carvalho<ref>
T. Castro0000-0002-6292-3228<ref>,<ref>,<ref>,<ref>
K. C. Chambers0000-0001-6965-7789<ref>
S. Contarini0000-0002-9843-723X<ref>
A. R. Cooray0000-0002-3892-0190<ref>
G. Desprez0000-0001-8325-1742<ref>
A. Díaz-Sánchez0000-0003-0748-4768<ref>
J. J. Diaz<ref>
S. Di Domizio0000-0003-2863-5895<ref>,<ref>
H. Dole0000-0002-9767-3839<ref>
S. Escoffier0000-0002-2847-7498<ref>
A. G. Ferrari0009-0005-5266-4110<ref>,<ref>
P. G. Ferreira0000-0002-3021-2851<ref>
I. Ferrero0000-0002-1295-1132<ref>
A. Finoguenov0000-0002-4606-5403<ref>
F. Fornari0000-0003-2979-6738<ref>
L. Gabarra0000-0002-8486-8856<ref>
K. Ganga0000-0001-8159-8208<ref>
J. García-Bellido0000-0002-9370-8360<ref>
T. Gasparetto0000-0002-7913-4866<ref>
V. Gautard<ref>
E. Gaztanaga0000-0001-9632-0815<ref>,<ref>,<ref>
F. Giacomini0000-0002-3129-2814<ref>
F. Gianotti0000-0003-4666-119X<ref>
G. Gozaliasl0000-0002-0236-919X<ref>
C. M. Gutierrez0000-0001-7854-783X<ref>
A. Hall0000-0002-3139-8651<ref>
H. Hildebrandt0000-0002-9814-3338<ref>
J. Hjorth0000-0002-4571-2306<ref>
A. Jimenez Muñoz0009-0004-5252-185X<ref>
S. Joudaki0000-0001-8820-673X<ref>
J. J. E. Kajava0000-0002-3010-8333<ref>,<ref>
V. Kansal0000-0002-4008-6078<ref>,<ref>
D. Karagiannis0000-0002-4927-0816<ref>,<ref>
C. C. Kirkpatrick<ref>
S. Kruk0000-0001-8010-8879<ref>
J. Le Graet0000-0001-6523-7971<ref>
L. Legrand0000-0003-0610-5252<ref>
J. Lesgourgues0000-0001-7627-353X<ref>
T. I. Liaudat0000-0002-9104-314X<ref>
A. Loureiro0000-0002-4371-0876<ref>,<ref>
G. Maggio0000-0003-4020-4836<ref>
M. Magliocchetti0000-0001-9158-4838<ref>
F. Mannucci0000-0002-4803-2381<ref>
R. Maoli0000-0002-6065-3025<ref>,<ref>
C. J. A. P. Martins0000-0002-4886-9261<ref>,<ref>
L. Maurin0000-0002-8406-0857<ref>
R. B. Metcalf0000-0003-3167-2574<ref>,<ref>
M. Migliaccio<ref>,<ref>
M. Miluzio<ref>,<ref>
P. Monaco0000-0003-2083-7564<ref>,<ref>,<ref>,<ref>
A. Montoro0000-0003-4730-8590<ref>,<ref>
A. Mora0000-0002-1922-8529<ref>
C. Moretti0000-0003-3314-8936<ref>,<ref>,<ref>,<ref>,<ref>
G. Morgante<ref>
S. Nadathur0000-0001-9070-3102<ref>
L. Patrizii<ref>
V. Popa0000-0002-9118-8330<ref>
P. Reimberg0000-0003-3410-0280<ref>
I. Risso0000-0003-2525-7761<ref>
P.-F. Rocci<ref>
M. Sahlén0000-0003-0973-4804<ref>
E. Sarpa0000-0002-1256-655X<ref>,<ref>,<ref>
A. Schneider0000-0001-7055-8104<ref>
M. Sereno0000-0003-0302-0325<ref>,<ref>
A. Silvestri0000-0001-6904-5061<ref>
A. Spurio Mancini0000-0001-5698-0990<ref>,<ref>
K. Tanidis<ref>
C. Tao0000-0001-7961-8177<ref>
N. Tessore0000-0002-9696-7931<ref>
G. Testera<ref>
R. Teyssier0000-0001-7689-0933<ref>
S. Toft0000-0003-3631-7176<ref>,<ref>
S. Tosi0000-0002-7275-9193<ref>,<ref>
A. Troja0000-0003-0239-4595<ref>,<ref>
M. Tucci<ref>
C. Valieri<ref>
J. Valiviita0000-0001-6225-3693<ref>,<ref>
D. Vergani0000-0003-0898-2216<ref>
G. Verza0000-0002-1886-8348<ref>,<ref>
P. Vielzeuf0000-0003-2035-9339<ref>
N. A. Walton0000-0003-3983-8778<ref>
    Received XXX; accepted ZZZ
=======================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================



§ ABSTRACT 
Due to the increasing workload of pathologists, the need for automation to support diagnostic tasks and quantitative biomarker evaluation is becoming more and more apparent.
Foundation models have the potential to improve generalizability within and across centers and serve as starting points for data efficient development of specialized yet robust AI models.
However, the training foundation models themselves is usually very expensive in terms of data, computation, and time.

This paper proposes a supervised training method that drastically reduces these expenses.
The proposed method is based on multi-task learning to train a joint encoder, by combining 16 different classification, segmentation, and detection tasks on a total of 912,000 patches.
Since the encoder is capable of capturing the properties of the samples, we term it the Tissue Concepts encoder.

To evaluate the performance and generalizability of the Tissue Concepts encoder across centers, classification of whole slide images from four of the most prevalent solid cancers - breast, colon, lung, and prostate - was used.
The experiments show that the Tissue Concepts model achieve comparable performance to models trained with self-supervision, while requiring only 6% of the amount of training patches.
Furthermore, the Tissue Concepts encoder outperforms an ImageNet pre-trained encoder on both in-domain and out-of-domain data.

The pre-trained models and will be made available under <https://github.com/FraunhoferMEVIS/MedicalMultitaskModeling>.









§ INTRODUCTION



The need for diagnostic systems to help pathologists manage the anticipated workload increases as cancer cancers worldwide are on the rise <cit.>. As <cit.> estimate, breast, colorectal, prostate, and lung cancers are among the six most common cancers types. Projections suggest that cases of these cancers will continue to increase, posing significant challenges due to time-consuming diagnosis, increased demand for tumor subtyping, and personalized treatment <cit.>. Deep learning (DL) has made significant progress in medical imaging, particularly in the field of computational pathology (CPath). Some studies have demonstrated that DL models even surpass human performance in certain tasks, making DL models effective tools to help pathologists cope with the increasing workload <cit.>. However, the unavailability of the required large data sets and the needed investment of time and effort limits the effectiveness and impact of DL models in pathology.

Recent advances in self-supervised learning have enabled the training of deep neural networks on large amounts of unlabeled medical data, resulting in the creation of foundation models in computer vision <cit.>. These models are pre-trained on a wide range of images, primarily using self-supervision through contrastive learning or masked image modeling. They have been shown to perform well in downstream tasks, including patch classification and weakly labeled whole slide image (WSI) classification <cit.>. Projects such as the Tissue Cancer Genome Atlas Program (TCGA) provide a data source of thousands of WSIs for training these networks on real-world data. This vast amount of data is necessary for self-supervised trained networks to reach their full potential <cit.>. However, the amount of resources required to create, train, and deploy such models has raised concerns among researchers about the environmental and other impacts <cit.>. In addition, extended training periods of several weeks impede development cycles and prolong research time.

Supervised learning, on the other hand, has been shown to outperform models trained on self-supervision in some tasks <cit.>. Although there are many annotated datasets available through challenges or other benchmarks, these datasets vary in size and contain annotations with varying degrees of detail. This variability between the datasets makes it challenging to condense the knowledge they contain into a single model. One approach to integrating all of these label types is to use multi-task learning (MTL) <cit.>. In <cit.>, we recently proposed a learning framework that combines the information contained in different labeling strategies, including detection, segmentation, and classification, and use it to train a single shared backbone model on a large corpus of images. In the study, images from different medical imaging domains, such as CT, X-ray, and microscopic images but also non-medical images were included.


This paper demonstrates that training a foundation model on supervised signals in CPath using MTL requires less data, time, and energy compared to models trained with self-supervision. At the same time, the measured performance is similar to that obtained from models trained on about 17 times more data without supervision. Following the MTL training scheme presented in <ref>a, this paper presents Tissue Concepts (TC), a robust encoder that is trained on a mixture of diverse annotations from small and medium-sized datasets in CPath to learn different concepts related to tissue. Considering the future prediction of cancer cases and clinical workflow, we evaluated the performance of the encoder on the four major cancer types breast, colon, lung, and prostate, for whole slide image classification, as shown in <ref>b. In addition, since models trained on one site are known to perform worse when evaluated on different sites, we test the performance of Tissue Concepts using a cross-center evaluation scheme <cit.>.

The paper's main contributions can be summarized as follows.

    
  * We show that diverse pre-training using MTL learns robust representations and drastically reduces the required amount of data compared to self-supervised approaches.
    
  * Our evaluation of the Tissue Concepts encoder on four of the most prevalent cancer types across multiple centers highlights the generalizability of our approach.




§ RELATED WORK


First approaches using MTL in CPath were presented by<cit.> and <cit.>. Mormont and colleagues converted different datasets into 22 classification tasks to train a shared network and contrasted the learned encoders against ImageNet-weights. A SVM was trained on the latent representations of the encoder. They found that the representations perform equal or better to the baseline ImageNet-weights. Graham et al. then used MTL on segmentation and classification tasks. This research focused on specific tasks that were present in the pre-training. 
However, the evaluation of the general-purpose encoder and the corresponding latent representations based on whole slide image classification combined with cross-center evaluation is still an unexplored area. In addition, general purpose encoders in the form of foundation models have not been considered by <cit.>.

In <cit.> we presented a first approach using MTL to train supervised foundation models. By utilized expert knowledge in the form of multi-task learning we trained a shared model, called UMedPT, which can be applied to various medical images. To achieve this, different imaging domains, such as CT, X-ray, and microscopic images were used to train a shared backbone on classification, segmentation, and detection tasks. Currently, the impact of tasks outside the histopathology domain remains unclear due to the diverse pre-training of the encoder. This impact on performance and robustness requires further investigation.

The following sections focus in more detail on two topics discussed in this paper. While foundation models are still largely unexplored in terms of their application and performance, some approaches are mentioned below.



 §.§ Foundation Models

A foundation model is broadly defined as being trained on a wide variety of data and being easily adaptable to many different downstream tasks <cit.>. 

<cit.> used data from the TCGA in combination with data from the pathology AI platform PAIP to train a modified swin transformer, called CTransPath (CTP), using self-supervision on 15 million patches. They presented an adapted contrastive loss, based on MoCo v3 <cit.>, which uses a memory bank to retrieve the top S semantically relevant entries. These entries were used as additional positive examples for the loss calculation. The authors evaluated their model using patch classification, image retrieval, and weakly labeled WSI classification. Due to the large number of training images and the slow convergence of self-supervised training, they reported a training time of 250 hours on 48 GPUs (12.000 GPU-hours). 

<cit.>, presented a comparable approach, training a tiny vision transformer (ViT) using standard DINO, and a ViT base model using standard masked autoencoding (MAE), both trained on about 3 billion patches. The authors report a training time of over 3000 GPU-hours for the models that were evaluated on a variety of tasks ranging from disease detection to outcome prediction. The evaluation also included images scanned at a different hospital than the training slides. 

<cit.> present a general purpose foundation model that leverages over 100 million patches from 100.000 WSI slides across 20 major cancer types. They train a large ViT on patches, that were collected in an internal dataset, using DINOv2 <cit.>. They evaluate the model on 34 Tasks and find that it surpasses the previous baselines on most of them. The model was trained on 24 80GB GPUs.

Overall, all of the presented models rely on large image databases and require long training times, which contributes to increased CO2 emissions. The presented TC encoder and MTL training aim to reduce the need for large amounts of data while maintaining the desired performance.
In addition cross-center evaluation is needed to accurately asses models' performances. 



 §.§ Weakly Labeled WSI Classification


Learning from WSIs that are only labeled on a case basis, or that have only one endpoint, is challenging because training on the entire image at once typically exceeds the GPU memory. In addition, since a WSI provides only one sample, many WSIs are needed to effectively train a deep learning model. Classification of such gigapixel images is therefore typically performed using multiple instance learning (MIL) <cit.>. Using MIL involves two parts: first, extracting features from patches of the WSI using a pre-trained encoder to convert them into their latent representations, and second, aggregating features from a WSI using a trainable MIL head to predict the given label <cit.>. Therefore, robust encoders are needed to obtain patch representations that facilitate the second step of MIL <cit.>. In this paper MIL is used as an evaluation procedure to test the representativeness of the encoder's features. The following presents the most commonly used approaches, that focus on solving the second stage of MIL using either attention or convolution-based methods.

<cit.> introduced CLAM, a clustering-constrained attention MIL algorithm. The authors trained an attention-based head on features extracted from patches of a WSI to classify the corresponding labels. The attention was then used to identify sub-regions of high diagnostic value, which in turn were used to classify the entire slide. In addition, instance-level clustering was applied over the representative regions to constrain and refine the feature space.

<cit.> proposed TransMIL, an attention-based correlation method for solving weakly labeled classification tasks. The method uses differently sized convolutional layers to apply additional pyramid position encoding information between the attention modules. This allows the attention layers to aggregate morphological features, while the Pyramid Position Encoding Generator (PPEG) encodes spatial information.

<cit.> proposed neural image compression to train on entire WSIs. The authors trained an autoencoder on patches and used the resulting encoder for feature extraction. The patches extracted in the image domain were encoded and their latent representations were placed in the same spatial location. This effectively compressed the entire WSI into a smaller latent image with more channels, while preserving the spatial relationship between the individual patches. A small CNN was then trained on the compressed WSIs to predict the label of the WSI. In a second version of this approach the same authors used multi-task learning on four classification tasks to train the feature extractor <cit.>. The effect of segmentation and detection tasks, as well as more diverse pre-training, remained a point of further investigation and are part of the research presented in this paper. 

All presented methods propose different aggregation methods to learn the desired label predictions based on the extracted features and thus work with the features extracted by the TC encoder. As an evaluation method, we adapted the convolution-based aggregation method presented by <cit.> and also applied an attention-based approach based on <cit.>, which are further described in Section <ref>. 



§ METHODS


Multi-task learning was used to train the Tissue Concepts encoder on 14 small and medium-sized datasets. The pre-training datasets and the procedure used during this phase are described in the following sections. In addition, the evaluation datasets, MIL head, and corresponding training are explained in detail. 




 §.§ Pre-training Datasets



To pre-train the shared encoder, a total of 14 data sources were collected and distributed over 16 tasks. All data sources with corresponding number of patches and WSIs, as well as their tasks are presented in <ref>. The NCT-CRC-HE 100k, Panda, TUH, Breakhis, and Arvaniti datasets have been designated as patch-level classification tasks by their respective curators. Data from the PANDA dataset was used for classification and segmentation tasks, where the patches in PANDA were 20 patches per WSI, sampled of a total of 4000 WSIs. To increase data diversity, the Breakhis dataset was used at both 40x and 100x magnification.
Conic and MiDoG were included as detection tasks, while Conic also served as a segmentation task. Other datasets used for segmentation were the SemiCOL training dataset, Arvaniti, Peso, Schoemig-Markiefka, PANDA, TIGER, CRAG, BCSS, and HubMap. Each of the mentioned datasets is described in more detail in the <ref>. In total, about 912,000 patches from around 7,000 WSIs were used during the training. The training data consisted of about 100,000 patches from colorectal, 600,000 patches from prostate, and 10,000 patches from breast tissue. In addition, a small number of spleen, liver, and skin tissue slides were included, which are marked as "various" in <ref>.

All patches were extracted or scaled to 224 × 224 pixels, resulting in a resolution of approximately 1 to 0.5 micron per pixel (MPP). In addition, standard augmentation, such as random rotation, distortion, blurring, brightness, contrast, and hue changes were applied during pre-training.



 §.§ Encoder Training


The shared backbone was trained using multi-task learning with all of the above data sources <cit.>. In the MTL pipeline, each task was solved by a shallow task-specific head (θ_t) that received input from larger, shared modules (θ_shared). <ref>a gives a brief overview of the different blocks in the architecture. The shared blocks are shown in the blue area, while the individual tasks are shown in green.
During training, all tasks were treated equally and were processed iteratively within the training loop. For each task the task-specific loss was computed and accumulated on the total loss. Formally, the total loss ℒ_total(X, Y) for a set of tasks X=(X_1,...,X_T) of task specific images  X_t=(x_t,1,...,x_t,N) and corresponding labels Y_t=(y_t,1,...,y_t,N) is computed as

    ℒ_total(X, Y) = ∑_t=1^Tℒ_t(θ_t(θ_shared(X_t)), Y_t)

where the loss of the task t is calculated based on the output of the task specific head θ_t that received its' input from a shared network structure θ_shared. X_t and Y_t describe the task-specific batch input images and the corresponding labels respectively. While the task-specific parameters θ_t in <ref> changed, depending on which task was being processed, the shared parameters θ_shared were task-independent. Since all tasks contributed equally to the total loss, multiple cycles through all tasks can be performed by performing gradient accumulation before an optimization step is performed.

The tasks themselves were divided into classification, segmentation, and detection. All tasks shared the same encoder, which yielded a feature pyramid with four feature maps at different perceptual levels. The feature pyramid was further processed depending on the task. 

Classification tasks received a globally average pooled version of the lowest feature pyramid level from the encoder. Each task-specific classification head consisted of 20% dropout followed by a linear layer, mapping from the dimension of the latent space to the number of classes. By minimizing the cross-entropy loss, each head learned to predict the task-specific classes. 

Segmentation tasks consisted of a shared u-net-style decoder that received features from all levels of the shared encoder. Each individual segmentation task was then processed by a 1 × 1 convolutional layer operating on the output of the last decoder level. A combination of dice and focal loss was minimized by each segmentation head. 

The detection tasks also consisted of a shared u-net style decoder that received features from all levels of the shared encoder to extract spatially relevant feature maps. Unlike the segmentation tasks head, the detection task head consisted of an FCOS detection head, which minimized the corresponding losses.  

Since transformer-based approaches are currently dominant in the literature we chose a tiny swin transformer as shared encoder <cit.> (further denoted as TC-swin). Additionally, we compared the attention-based backbone to a tiny ConvNeXt architecture that shares a similar number of parameters but follows a convolution-based approach <cit.> (further denoted as TC-conv). Both networks were initialized with corresponding ImageNet-1k weights which were imported from the torchvision library <cit.>. During pre-training, the networks were optimized for 5 million steps until convergence, where one step was defined as computing the loss for one batch of one task. The AdamW optimizer <cit.> was used with a constant learning rate of 10^-4, a weight decay of 0.01, and gradient accumulation of 128 steps. In total, the models were trained for 160 hours on one NVidia RTX A5000 GPU and the final models were used for evaluation.



§ EVALUATION

This section first establishes a comparison between UMedPT and TC using a sample efficiency experiment as an ablation study. Then, the evaluation datasets and corresponding tasks are presented, and the MIL head training procedure is described.



 §.§ Ablation Study


To determine whether a multi-domain encoder, UMedPT, or a histology specific encoder, TC-swin, performs better on unseen histology data, we measured the sample efficiency on an unseen, in-domain dataset. In this experiment, the frozen encoder generated latent representations for all patches in the downstream dataset. These latent representations were then used as input to a random forest model. Sample efficiency was measured by establishing of a fixed training and validation split. Different subsets of images per class were systematically sampled from the training split to serve as training examples for the random forest. Each model was consecutively evaluated on the designated test split. This training-validation process was repeated 10 times with different seed values for robustness in the analyses and the F1-score was reported. We compared TC-Swin to the base Swin transformer encoder presented in <cit.>. Additionally, ImageNet weights from torchvision were used as baseline for a tiny swin transformer.

The BACH challenge dataset <cit.> is a breast tumor classification dataset consisting of 400 images of 2048 × 1536 pixels at 0.25 MPP, which are equally divided into 4 classes. Images were center cropped to 1024 × 1024 pixels and downsampled to 224 × 224 pixels. The dataset was split 80/20 into a training and test subset. From the training split of the BACH dataset, sets of 1, 3, 5, 10, 25, and max 46 images per class were sampled using 10 different seeds. For each training, the random forest was evaluated on the left out test set.


Figure <ref> shows that the domain-specific encoder outperforms the multi-domain encoder and the pre-trained ImageNet encoder in terms of sample efficiency.
The TC-Swin encoder showed higher F1-scores compared to the multi-domain encoder and to ImageNet. With three images per class, TC-Swin's mean F1-score was at 0.531, while UMedPT and ImageNet were at 0.496 and 0.423, respectively. With 10 images per class, the TC-Swin encoder showed a mean F1-score of 0.673, while UmedPT and ImageNet showed values of 0.628 and 0.566, respectively. Using the maximum number of available training images per class, the differences in reported performance are similar. While TC-Swin achieved 0.816, UmedPT and ImageNet achieved mean F1-scores of 0.762 and 0.662, respectively.

The TC-Swin encoder outperforms the multi-domain medical encoder UMedPT even though the later was trained with more images overall. In particular, with 5 or more images per class, the features obtained from the TC encoder provide better performance than the ImageNet and UMedPT features. When considering 1 and 3 images per class, UMedPT shows similar performance to the TC encoder, but also shows a higher variance over the 10 seeds. Therefore, we excluded the UMedPT encoder from further evaluation and focused on the TC encoder instead.



 §.§ Evaluation datasets

We compared the TC encoder to the current state of the art for each organ, referred to as the external baseline, as well as to the results obtained using the features of the publicly available CTransPath encoder <cit.>. In addition, we use the Swin transformer model with ImageNet weights without further training as an additional baseline.

The learned representations of Tissue Concepts were evaluated using weakly labeled WSI classification tasks, colorectum, represent tissues within the training domain, lung cancer tissue was deliberately chosen to represent tissues outside the training domain. The architecture and learning schedule as described in Section <ref>, was used during the training of each MIL head. Where possible, multi-site datasets were split so that evaluation sites were not included in the training. The feature extractors were evaluated on the following datasets for each organ: 

Breast Cancer: The BReAst Cancer Subtyping (BRACS) dataset consists of 547 WSIs from 189 patients which are already divided into training (395), validation (87) and test (65) splits <cit.>. According to <cit.>, the WSIs were obtained at the Department of Pathology at the National Cancer Institute - IRCCS-Fondazione Pascale, Naples, Italy, and were scanned with an Aperio AT2 scanner at a resolution of 0.25 microns per pixel (mpp). Each WSI is assigned to one of 7 different classes: normal, benign, usual ductal hyperplasia (UDH), atypical ductal hyperplasia (ADH), flat epithelial atypia (FEA), ductal carcinoma in situ (DCIS), and invasive. In addition, <cit.> grouped the different types of breast carcinoma into four coarser classes: normal, non-cancerous, precancerous, and cancerous. Furthermore, binary classification between invasive and non-invasive breast cancer was also performed by <cit.>. The authors also presented a trained model, HACT-Net, and applied it to this dataset. They reported the weighted F1-score for all of the classification problems mentioned above which we used as an external baseline and also reported the weighted F1-score in the BRACS experiments. Additionally, following <cit.>, we reported accuracy as multi-class metric and area under the receiver operating curve (AUC) as multi-threshold metric. For the evaluation of the TC-encoder, three experiments were created form the dataset, similarly to the approach proposed by <cit.>. The first experiment consisted of differentiating between normal and cancerous tissue slides. The second experiment focused on more fine-grained classification between normal, non-cancerous, precancerous, and cancerous tissues. Finally, the third experiment for breast cancer aimed at predicting one of the seven sub-types provided.

Prostate Cancer: The Prostate cANcer graDe Assessment (PANDA) challenge provides 10616 openly available WSIs from two different centers, Karolinska and Radboud <cit.>. Each of the WSIs was assigned an ISUP score resulting from the primary and secondary Gleason pattern. The score ranges from 0 to 5, with 0 representing normal tissue and higher numbers representing more severe prostate cancer. This information was used to create binary classification task for evaluation, where the task consists of differentiating between normal (ISUP 0) and cancerous tissue (ISUP > 0). 
To test cross-center performance and overall performance, three different experiments were created from this dataset. First, the entire set of images obtained from the Radboud UMC was used as both the training and validation set, using a split ratio of 90% for training and 10% for validation. All images provided by the Karolinska Institute were used as the hold-out test set. In a subsequent experiment, the roles of training and test centers were swapped. In addition, a 5-fold cross-validation experiment involving both centers was conducted. We note that a small percentage of the dataset in the form of pre-selected patches were used during pre-training of the encoder, however, the task-specific head was discarded. <cit.> used this dataset in a k-fold cross-validation to evaluate the performance of their model and reported the accuracy and area under the curve for binary classification. Their reported results serve as an external baseline. <cit.> reported the accuracy and AUC of their algorithm on the dataset. To be comparable, we also report these metrics in our experiments. Additionally, we report the F1-score as per-class metric, as suggested by <cit.>.

Colorectal Cancer: During the SemiCOL challenge, a dataset of 499 WSIs from 4 different centers was provided as training dataset. The challenge task was to predict whether a given slide contained cancerous tissue or normal tissue. During the challenge, we used a modified model presented in <cit.> as an initial starting point and further fine-tuned on several colon and colorectal tasks. Using the learned encoder to extract features from the WSI, a MIL head was successively trained on these features. This resulted in an external test AUC of 0.99. In the experiments presented in this paper, the challenge training data was split by slide provider, effectively creating four smaller subsets of the original challenge dataset. For the evaluation presented in this paper, four different experiments were conducted. Images from one center were used as training examples (90/10), while images from the other three centers served as the hold out test set. By treating each center as training provider once, and using the other three centers as test centers, four different configurations were created We note that a small percentage of patch data of the dataset was used during pre-training of the encoder, however, the task-specific head was discarded. To the best of our knowledge, there is no other publication reporting results on this specific dataset, so we are unable to report an external baseline. Since the dataset is balanced with regard to the class distribution for each center in all splits, we reported the Accuracy, F1-Score and AUC as metrics, following <cit.>.

Lung Cancer: The TCGA-NSCLC dataset contains two subtypes of lung cancer which are lung squamous cell carcinoma (TCGA-LUSC <cit.>), and lung adenocarcinoma (TCGA-LUAD <cit.>) in 1006 (512 LUSC, 494 LUAD) slides from over 40 different sites. Two training and testing scenarios were created from the TCGA-NSCLC dataset. One experiment focused on corss-center evaluation. In this scenario, the three largest contributors (Johns Hopkins, International Genomics Consortium, and Asterand) were selected to serve as the training and validation subset providers. The remaining 39 sites were selected to serve as hold out test set. This split resulted in 340 training slides (90/10 split) and 666 test slides. In addition, a 5-fold cross-validation experiment was performed on the entire dataset to increase training set diversity, again using a 90/10 training and validation split. The objective of each experiment was to distinguish between slides containing lung adenocarcinoma and those containing squamous cell carcinoma (LUAD vs LUSC). The obtained results were compared to the results reported by <cit.> and <cit.>. To be comparable, the accuracy and AUC, as reported by the external baselines, were selected as metrics. Additionally, the F1-score was reported.



 §.§ Training of the MIL head




Following medical diagnoses, WSIs are often only labeled at the patient or slide level. To address this, we used MIL to train slide-level classification heads for the respective evaluation tasks. The WSIs were compressed into latent images, by the respective frozen encoder, as suggested by <cit.>. During this process, all patches of a WSI were extracted with a size of 224 × 224 at about 0.5 MPP.

Using the exact same architecture for aggregation as presented by <cit.>, we found that training the proposed MIL head left room for improvement in terms of training stability and convergence speed. The selection and adaptation were motivated by the prior performance on the SemiCOL challenge <cit.>. There, we fine-tuned UMedPT and trained the small MIL head on WSI classification. On an external, multi-center test set, this pipeline achieved an AUC of 0.998.

To aggregate the latent WSIs, a small CNN with global max pooling, was used. The initial layer utilized a 1x1 convolution, reducing the latent WSIs' 768 channel to 16. Following this, 3x3 convolutions with a stride of 2, and padding of 1 doubling in depth each time were applied. Instance normalization, leaky ReLU activation, and 10% 2D dropout were applied in the convolutional layers. The lowest feature pyramid of 64 channels was globally max pooled, followed by one hidden linear layer and a classification layer. This resulted in 19250 total trainable parameters. The convolutional layers were initialized by drawing from a Kaiming normal distribution with a fixed seed. Additionally, we tested an attention-based approach by replacing the max pooling layer with an attention pooling layer, as suggested by <cit.>. Overall, we found similar results with both approaches across all organs. The results using an ABMIL aggregation head can be found in the <ref>.)

The same architecture was used over all evaluation datasets. To avoid overfitting, a label smoothing of 0.1 was used while minimizing the cross-entropy loss. The MIL head was trained using the AdamW optimizer with a constant learning rate of 10^-4 and a weight decay of 0.01 for 100 epochs, while monitoring the validation loss. The best performing model was used for testing. During training, additional augmentations were applied to each latent WSI. Since each latent WSI was treated as a 768-channel image, random flipping, mirroring, and resizing with nearest neighbor interpolation were applied during training. This resulted in high-dimensional latent WSI representations with sizes ranging from 32 × 32 to 224 × 224.



§ RESULTS

An overview of the obtained results is given in <ref>, where all tasks are presented with corresponding test AUC scores over 5 runs. The results show that the TC encoder, although trained with less data, performs as well as, or better than the CTransPath encoder, given the selected tasks. The Swin transformer with ImageNet weights exhibit the lowest overall performance and the largest variance across all tasks. All reported results were obtained using the default parameters of the corresponding functions from the scitkit-learn library <cit.>. In the following, the results on the different organs are examined in more detail.



 §.§ Breast Cancer


<ref> shows the results for the three sub-problems of the BRACS classification challenge, including the weighted F1-score and the AUC score for each. For the binary classification problem, ImageNet weights achieve the highest weighted F1-score on average with 0.960, which is comparable to the external baseline of 0.96. CTP and TC-swin show similar performance with average wF1-scores of 0.954 and 0.951, respectively. The TC-Conv encoder performs almost three percentage points worse than the TC-swin encoder. Overall, all tested encoders result in similar AUC values.

Considering the more fine-grained classification of the given WSIs, none of the encoders matches the reported performance of the external baseline HACT-Net with a weighted F1-score of 0.84. The two TC encoders outperform the ImageNet weights on the four- and seven-class problem formulations. On the four-class problem, CTP achieves a mean weighted F1-score of 0.613, approximately 6 percentage points higher than the two TC encoders. However, when considering the AUC, all of the foundation models perform equally with a mean AUC of about 0.83. For the more detailed seven-class problem, the TC-swin encoder achieves the highest mean weighted F1 and AUC scores out of all the tested models with 0.40 and 0.76, respectively. 

Overall, the performance of all four encoders tested decreases significantly as the problem formulation becomes more fine-grained. Although the external baseline performance also declines, the decline is not as steep as that observed with the foundation models and ImageNet weights. 



 §.§ Prostate Cancer


<ref> shows the results obtained of five runs on the PANDA dataset with cross-center and mixed evaluation. The 'clinic' column indicates different settings, where "Karolinska → Radboud" refers to the MIL head being trained on data from the Karolinska Institute and evaluated on data from the Radboud Clinic. 

When comparing the results obtained in the 5-fold cross-validation with the external baseline, both TC encoders show performances that were close to the external baseline, with TC-conv slightly surpassing it with a mean accuracy of 0.957. The mean accuracies of CTP and ImageNet are 2 and 5 percentage points below the selected baseline, respectively. All foundation models outperform the pure ImageNet encoder by at least three percentage points. In terms of accuracy and AUC scores, both TC encoders slightly outperform the CTP weights by one percentage point with mean acc of 0.936 and mean auc of 0.979.

In the cross-center evaluation, the performance of all models decreases compared to an in-center evaluation. CTP achieves a mean accuracy of 0.819 when transferring decision boundaries from the Karolinska site to Radboud subset. The opposite scenario results in a decrease of 14 percentage points (0.678 mean acc). Furthermore, an even steeper drop in mean accuracy and AUC from 5-fold cross-validation to cross-center validation is observed when using the ImageNet weights (from 0.900 to 0.658). Both TC encoders are able to extract robust features. The TC-Swin encoder (from 0.942 to 0.846) shows more stable performance than the TC-Conv encoder (from 0.957 to 0.764) when compared to the metrics reported in the cross-center evaluation.
 


 §.§ Colorectal Cancer

 
<ref> presents the results per trained center on the SemiCOL challenge dataset. The 'clinic' column indicates the training center. The averaged results are shown at the bottom of the table for orientation. On average the TC-swin encoder showed higher mean accuracy (0.953) compared to the TC-conv (0.941) and CTP (0.944) encoders, as well as higher F1-scores. In terms of AUC, all three encoders show approximately the same performance on average.

When looking at the performance on individual clinics, the transfer from UBERN to other clinics seems to be the most challenging. ImageNet (0.734) and CTP (0.860) both showed a performance about 10 percentage points lower than the performance on other centers. However, both TC encoders were able to extract features that allowed the transfer of the learned classification to other centers (TC-swin: 0.943, TC-conv: 0.943). The performance when trained on data from the UBERN clinic is similar to that when trained on data from other clinics.

Looking at performance on other clinics, there is a slight discrepancy in mean accuracy and AUC score between CTP and TC. CTP consistently achieves slightly higher scores of approximately one percentage point over the TC-Swin encoder.




 §.§ Lung Cancer


<ref> shows the results of the 5-fold cross-validation and the cross-center evaluation on the TCGA-NSCLC dataset. Despite the absence of lung cancer images in the pre-training, both TC encoders outperform the ImageNet weights in the cross-center and 5-fold cross-validation tasks. The CTP encoder achieves the highest mean accuracy of 0.902, which is about 2 percentage points better than the TC-swin of 0.878. The CTP encoder was pretrained using the TCGA-NSCLC dataset and its performance achieved when evaluating the CTP in our pipeline is similar to the external baseline reported by <cit.> (0.902 vs. 0.912). The TC-swin encoder shows comparable performance earlier results reported by <cit.>, with an accuracy of 0.878 vs. 0.884 and an AUC of 0.949 vs. 0.961.  

As expected, both models TC-models show weaker performance when evaluated across centers, compared to mixed-center 5-fold cross-validation. Both TC encoders outperform the ImageNet-based encoder. The CTP encoder was pre-trained to extract features from this dataset and outperforms all other models in the cross-center evaluation.

As an additional experiment, the UNI <cit.> encoder was compared to the other models using the ABMIL aggregation head and showed similar results to the other encoders. The detailed results can be found in <ref>.



§ DISCUSSION

This paper demonstrates that MTL is an effective method for training foundation models on supervised signals in CPath. We tested the MTL-trained Tissue Concepts encoders on breast, prostate, colorectal, and lung cancers and found that the performance of the encoders is comparable to a self-supervised model. This comparable performance was achieved with only 6% of the data and resources: training TC required only 912,000 patches, compared to 15 million patches used in <cit.>'s self-supervised approach. Less data and shorter training enable faster development and research cycles.

Further development and scaling of the foundation models in CPath will inevitably contribute to extended CO2 emissions. The TC encoders were trained for 160 hours on a single Nvidia RTX A5000 in Europe. This corresponds to an estimated 18.91 kg of CO2 emissions <cit.>. Most self-supervised approaches exceed these emissions by orders of magnitude due to larger training datasets, longer convergence times, and the use of multiple GPUs. Training such a model for the same amount of time on 48 NVIDIA V100s, a common number of GPUs for training large foundation models, results in 2004 kg of CO2 during training. 
Overall, depending on the GPU, one training of an MTL-based model produces only 0.9% to 2.25% of the CO2 emissions of an SSL-based training while providing similar performance.

Even though, performance generally is comparable, there are two differences between TC and CTP in organ specific performance. One is the difference on the prostate dataset. While a small amount of patches of this dataset were present during the pre-training of TC, the vast amount of patches used during testing were unseen. However, a lot of prostate patches were used during the pre-training of TC, as mentioned in section <ref>. This dominant representation of prostate tissue during pre-training could potentially lead to the better performance on this organ.
This raises the question if organ-specific fine-tuning of TC encoders can lead to better performance on specific organs when considering cross-center evaluation and testing on unseen datasets. This question is further justified by the differences in organ specific performance between foundation models and external baselines.

Another clear difference in performance between CTransPath and Tissue Concepts was observed for lung cancer in the 5-fold cross-validation and cross-center validation. This difference most likely arises due to the out-of-domain nature of lung cancer tissue in TC pre-training. However, training on pathology images instead of real-world images improved performance when comparing Tissue Concepts weights to ImageNet weights. Scaling the Tissue Concepts pre-training with more data should be considered to overcome the current limitations. Importantly, balancing different tasks, tissue types, and magnifications during pre-training is important to effectively scale and apply multi-task learning in the CPath domain.

In the present study, only frozen encoders were considered.
While fine-tuning these foundation models can lead to better adapting to specific tasks, more research is needed on the balance between data specificity and variation, and the amount of data is required. In <cit.> we found that a fine-tuned foundation model encoder outperformed a fine-tuned ImageNet-base model on all tasks in terms of data efficiency and F1-score. Similar results can be expected for TC encoders and CPath specific problems, but this remains an open question to be investigated further. Additionally, this study focuses on solving MIL problems which occur frequently in diagnostic or prognostic tasks. While MIL tasks are clinically motivated, the individual performance of the segmentation, and detection branches in the multi-task trained model need to be considered in future evaluations.

Overall, the results show that the more domain-specific pre-training of Tissue Concepts encoders is advantageous for solving domain-specific tasks compared to the more general, multi-domain pre-trained encoders. A question that remains to be answered is the performance difference between foundation models and models trained on one specific organ. This will be part of future research.



§ CONCLUSION

In this paper, we propose to train a foundation model for CPath on supervised signals using multi-task learning to reduce the need for large corpora of data, computation time, and resources during training. The proposed method shows comparable results to a model trained using self-supervision while relying on a fraction of the training patches.

In addition, we found that, although trained on large amounts of data, existing models still exhibit a loss of performance when applied across centers. Better cross-center generalization is crucial to facilitate broader clinical application and needs to be further addressed in future research.

In order to effectively scale multi-task learning of foundation models, questions regarding the balance of tasks and tissue types need to be answered. Organ-specific fine-tuning might create robust and high-performing encoders for specific problems but needs to be further explored.



§ DATA AVAILABILITY

The trained models, scripts, and notebooks to reproduce the results will be made publicly available upon publication.

The pre-training framework to build the pipeline will be available as a python package.
The links to all datasets are mentioned in the appendix with a corresponding description.



§ DECLARATION OF COMPETING INTEREST

The authors declare that they have no known competing financial or personal relationships or interests that could have appeared to influence the work reported in this paper.



§ ACKNOWLEDGMENTS

This research was in part funded by the German ministry of education and research (BMBF) through the projects SynDICAD (01IS21067) and PROSurvival (01KD2213). The authors are responsible for the content of this publication.

The results show in this paper are in part based upon data generated by the TCGA Research Network: <https://www.cancer.gov/tcga>.









§ TRAINING DATASETS



NCT-CRC-HE 100k

The NCT-CRC-HE 100k dataset consists of 100.000 non-overlapping patches from 86 H&E stained human colorectal cancer and normal WSIs. The patches are of size 224 × 224 and have a resolution of 0.5 microns per pixel (MPP). Nine different tissue classes were present in this classification dataset. The dataset is available under <https://zenodo.org/records/1214456>.

Conic

This dataset is based in the Lizard Dataset <cit.>. It contains 4981 H&E stained, non-overlapping patches of colon tissue at 256 × 256 pixels. The segmentation masks were generated using the Hover-Net The dataset is available under <https://conic-challenge.grand-challenge.org/>.

CRAG

The CRAG dataset consists of 213 H&E stained patches from colon tissue of 38 WSIs. The patches are of size 1512 × 1512 at a resolution of 0.5 MPP. During preprocessing the patches and masks were further cropped into non overlapping 224 × 224 tiles with 1MPP resolution. The original 173/40 patch train/val split was maintained. The dataset is available under <https://warwick.ac.uk/fac/sci/dcs/research/tia/data/mildnet> with corresponding login.

SemiCol

The training data from the SemiCOL challenge is divided into two parts. One contains manual segmentation masks, which were used in the pre-training of the encoder. The other part contains weakly labeled data, which was described in <ref>. The 20 sparsly annotated slides were obtained from the university hospital in Cologne (Hamamatsu S360) and from the university hospital LMU in Munich (Leica GT450). Both sides scanned at 0.5 MPP and provided 10 slides each. Patches with annotations were extracted from the provided images and scaled to 1 MPP and 224 × 224 pixels.
More details about the data can be found here <https://www.semicol.org/data/>.

Arvaniti

The Arvaniti TMA dataset consists of prostate tissue microarrays, which were scanned at 0.23 MPP resolution at the University Hospital Zurich using a Hamamatsu NanoZoomer-XR Digital slide scanner. Patches of  1024 × 1024 were cut out from the foreground and scaled to 224 × 224. The major class of the mask was selected to serve as patch label.
The dataset is available under <https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/OCYCMP>.

Peso

The PESO dataset consists of 102 WSIs, which were scanned at 20x around 0.48 MPP with a 3DHistech Pannoramic Flash II 250 scanner. From the H&E stained prostate slides were, patches were extracted and scaled to 1 MPP of size 224  times 224. The corresponding masks were used as targets. 
The dataset is available under <https://zenodo.org/records/1485967>.

Schoemig-Markiefka

This prostate dataset contains 6 sub-datasets, each containing 120.000 patches. All of the patches were scanned at approximately 0.25 MPP with at least 4 different scanners. Each sub-dataset contains 50.000 patches with tumor tissue, 50.000 patches with non-neoplastic glandular prostate tissue and 20.000 patches with non-glandular tissue. In our training the patches were scaled accordingly. 5 of the sub-datasets were used for pre-training, while 1 was used for validation
The dataset is available under: <http://zenodo.org>, Deposits: 4789576 (Dataset 1–4) and 4904569 (Datasets 5–6).

Panda

This prostate dataset was already described in <ref>. From the slides, patches of 224 × 224 were extracted. The labels for the patches were generated from the corresponding masks.
The dataset is available under <https://www.kaggle.com/c/prostate-cancer-grade-assessment/data>.

TUH

The Temple University digital pathology corpus consists of over 3505 annotated images of breast tissue. The labels range from artifact and background annotations to specific breast cancer annotations like invasive ductal carcinoma. All WSIs of the 296 patients are scanned at 0.5 MPP with corresponding annotations. From the 3505 slides 136 exhibited quality annotations that were used in the training. Fixed 20 patches per WSI were sampled. 
Details about the download can be found here <https://isip.piconepress.com/projects/nsf_dpath/html/downloads.shtml>.

Tiger

This breast cancer dataset contains H&E stained patches from tumor infiltrating lymphocytes. The patches were extracted from 151 TCGA-BRCA slides at f0.25 MPP. Manual annotations for 7 different classes were used as masks. Patches and masks were scaled to 224 × 224 pixels.
The dataset is available under <https://tiger.grand-challenge.org/Data/>.

BCSS

The BCSS dataset was contained as part of the TIGER challenge. Annotations provided through the challenge were used as segmentations masks. These annotations were derived from the original BCSS dataset. All images were scaled same as patches from the Tiger dataset mentioned above. 124 slides from BCSS and NuCLS were annotated, where some of the annotations were grouped.
The dataset is available under <https://tiger.grand-challenge.org/Data/>.

BreakHis

The breakhis dataset contains H&E stained patches from 82 patients at different magnifications. The patches are of size 700  times 460 and were scaled accordingly to 1 MPP resolution. The classes for this dataset were derived from the original dataset. A two-class benign/malignant split is possible, however, the more fine-grained 8 class dataset was used.
This dataset is available under <https://web.inf.ufpr.br/vri/databases/breast-cancer-histopathological-database-breakhis/>.

MiDoG

The MIDOG 2022 challenge dataset contains 405 tumor cases across six different tumor types. During training the 44 lung cancer cases were excluded. The cases were patchified and scaled to 224  times 224 pixels with approximately 1 MPP.
The dataset is available under <https://zenodo.org/records/6547151>.

HubMap

The HubMap dataset contains 351 cases from different organs. Patches were extracted from the foreground containing annotation masks and scaled to 1 MPP of 224 × 224 pixels. Only large intestine was used during training.
The dataset can be found under <https://www.kaggle.com/datasets/dingyan/hubmap-data>.



§ MIL HEAD ARCHITECTURE


To aggregate the latent WSIs, as described in section <ref>, a small CNN with global max pooling, based on <cit.> was used. The initial layer utilized a 1x1 convolution, reducing the latent WSIs' 768 channel to 16. Following this, 3x3 convolutions with a stride of 2, and padding of 1 doubling in depth each time were applied. Instance normalization, leaky ReLU activation, and 10% 2D dropout were applied in the convolutional layers. The lowest feature pyramid of 64 channels was globally max pooled, followed by one hidden linear layer and a classification layer. This resulted in 19250 total trainable parameters. The convolutional layers were initialized by drawing from a Kaiming normal distribution with a fixed seed. 



 §.§ ABMIL Results and comparison to the UNI encoder


A different aggregation approach was proposed by <cit.> under the term "Attention-based Deep Multiple Instance Learning”. For comparison, the following tables present the results obtained by using an attention based multiple instance learning aggregation head instead of a maximum pooling aggregation head. The results were collected over three runs and averaged. Additionally, the experiments were repeated using the UNI <cit.> foundation model.

Overall, we observed similar results between the different encoders and aggregation techniques across all organs. 







 




