(eccv) Package eccv Warning: Package ‘hyperref’ is loaded with option
‘pagebackref’, which is *not* recommended for camera-ready version

11institutetext: University of Alberta

# Enhancing Weakly-Supervised Histopathology Image Segmentation with Knowledge
Distillation on MIL-Based Pseudo-Labels

Yinsheng He  Xingyu Li  Roger J. Zemp

###### Abstract

Segmenting tumors in histological images is vital for cancer diagnosis. While
fully supervised models excel with pixel-level annotations, creating such
annotations is labor-intensive and costly. Accurate histopathology image
segmentation under weakly-supervised conditions with coarse-grained image
labels is still a challenging problem. Although multiple instance learning
(MIL) has shown promise in segmentation tasks, surprisingly, no previous
pseudo-supervision methods have used MIL-based outputs as pseudo-masks for
training. We suspect this stems from concerns over noises in MIL results
affecting pseudo supervision quality. To explore the potential of leveraging
MIL-based segmentation for pseudo supervision, we propose a novel distillation
framework for histopathology image segmentation. This framework introduces a
iterative fusion-knowledge distillation strategy, enabling the student model
to learn directly from the teacher’s comprehensive outcomes. Through dynamic
role reversal between the fixed teacher and learnable student models and the
incorporation of weighted cross-entropy loss for model optimization, our
approach prevents performance deterioration and noise amplification during
knowledge distillation. Experimental results on public histopathology
datasets, Camelyon16 and Digestpath2019, demonstrate that our approach not
only complements various MIL-based segmentation methods but also significantly
enhances their performance. Additionally, our method achieves new SOTA in the
field. Codes are available at https://github.com/wollf2008/IKD-MIL.

###### Keywords:

Histopathology Image Image Segmentation Weakly-Supervised Knowledge
Distillation

##  1 Introduction

In the current medical system, the visual examination of histological images
is known as the gold standard for the identification of cancer. Accurate
identifying cancerous regions are a critical prerequisite for pathologists in
determining cancer invasiveness. Over the past decade, a multitude of deep
learning-based algorithms have been developed for this purpose. These methods
mainly fall into three categories: supervised [36] [21], weakly supervised [5]
[49], and unsupervised [30] [4]. In recent years, with significant
advancements in deep learning approaches, the performance of fully supervised
learning networks has approached, and in some aspects, even close to that of
human experts [37]. However, such success is largely attributable to the
availability of extensive, high-quality manually annotated data. To obtain
sufficiently accurate ground truth, pathologists typically spend considerable
time annotating histopathological images with ultra-fine granularity,
consuming substantial labor costs. On the other hand, unsupervised learning
strategies, which require no annotations whatsoever, are still a long way from
practical application due to lower accuracy. Weakly supervised learning, as a
strategy that necessitates only coarse annotations, achieves a better balance
in terms of cost and performance [41].

Weakly supervised segmentation allows for the utilization of coarse-grained,
image-level annotations as guidance. In this process, pathologists merely
indicate the presence or absence of cancerous tissues within training images;
the model then proceeds to autonomously conduct pixel-level segmentation on
new data. Multiple instance learning (MIL) [13] has been exploited for this
purpose. It consider pixels as instances within a bag (i.e. an image) and
infers the specific pixel-level labels from the bag label [43, 20, 32, 41,
35]. Alternatively, pseudo-supervision serves as another prevalent method for
addressing this task. It leverages coarse-grained annotation information to
generate pseudo-masks which are then used as targets for training a
segmentation model in a manner akin to supervised learning. In prior arts,
class Activation Mapping (CAM) is commonly used to produce these pseudo-masks
[18, 27, 34, 11, 15, 38, 29]. Nevertheless, CAM’s inability to accurately
define object boundaries often results in pseudo-masks that are both blurred
and imprecise [1]. Consequently, CAM-based pseudo-supervision may not
consistently apply identical labels to the same anatomical structures across
various histopathology images, highlighting a potential limitation in
achieving precise segmentation [8]. In theory, both CAM-generated and MIL-
derived segmentation results hold potential for use as pseudo-masks within the
pseudo-supervision framework for histopathology image segmentation. As
demonstrated in Fig. 1, segmentation outcomes from MIL have higher granularity
and more defined boundaries compared to the typically blurred pseudo-masks
produced by CAM-based methods, albeit potentially introducing more noise [32,
41]. Intriguingly, a review of existing literature reveals an absence of
studies employing MIL-based outcomes as pseudo-masks for training. This
observation leads us to hypothesize that the direct integration of these
noise-prone MIL-based segmentation results as pseudo-masks might adversely
affect the segmentation model’s accuracy over time due to pseudo supervision.

To verify our hypothesis and fill the existing research void, this study
investigates MIL-based pseudo-supervision for weakly supervised segmentation
in histopathology images. To mitigate the potential degradation in model
performance due to pseudo supervision, we introduce a novel knowledge
distillation approach designed to both stabilize and enhance the segmentation
model. Specifically, we innovate a iterative fusion-knowledge distillation
strategy to address the noise and label-ambiguity issue in MIL. We show that
this strategy can be seamlessly integrated with a variety of MIL-based
segmentation architectures, leading to notable improvements in performance. As
demonstrated in Fig. 2, diverging from the common knowledge distillation
methods which align the student model’s behavior with that of the teacher at
either the final output or intermediate representations, our fusion knowledge
distillation guides each block of the student model to directly assimilate the
comprehensive insights from teacher’s final outcome. Echoing the findings of
[50], our approach empowers even the shallow layers of the student model to
capture features typically reserved for deeper layers, enabling effective
segmentation of both local and global cancerous regions. Another unique aspect
of our knowledge distillation is the dynamic role reversal between teacher and
student models once the student outperforms the teacher, followed by a repeat
of the distillation cycle to refine segmentation accuracy further. During each
distillation cycle, the fixed (frozen) teacher model enables the student to
consistently learn and enhance its capabilities from the teacher’s pre-
established, static insights. To further counteract the potential
magnification of noise during knowledge distillation, we incorporate a
weighted cross-entropy loss [34] as a regularization mechanism in the
optimization of the student model. Our experimental evaluation across two
publicly available histopathology datasets confirms that our proposed
methodology sets a new benchmark for state-of-the-art segmentation
performance.

Figure 1: Pseudo-mask comparison between MIL method and CAM method. (a)
original image, (b) ground truth mask for the image, (c) pseudo-mask generated
by CAM method, (d) pseudo-mask generated by MIL method. The challenge of using
MIL-based segmentation as pseudo masks lies in how to prevent the segmentation
model from deteriorating over time under pseudo supervision. Figure 2:
Knowledge distillation structures, (a), (b) are common knowledge distillation
methods, and (c) is our fusion knowledge distillation. The comparison between
these methods is shown in Fig. 5 and Fig. 6.

  * •

We introduce a MIL-based pseudo-supervision framework for weakly supervised
segmentation with coarse-grained labels on histopathology images. This
framework can be seamlessly integrated with a variety of MIL-based
segmentation models for performance boost, opening a new avenue of leveraging
MIL-based results in the pseudo-supervision paradigm.

  * •

We innovate a fusion-knowledge distillation strategy. It empowers shallow
layers of the student model to capture abstract features typically associated
with deeper layers, enhancing segmentation performance for both local and
global regions of interest. Special designs such as dynamic teacher-student
switch help prevent model deterioration over time.

  * •

We conduct extensive experiments to validate the effectiveness of the proposed
method and report a new SOTA in the field.

##  2 Related Work

###  2.1 Histopathology Image Weakly Supervised Segmentation

Pseudo-supervision is a common strategy in Weakly Supervised Semantic
Segmentation (WSSS). In the field of histopathology imaging, pseudo-
supervision is often tied with Class Activation Mapping (CAM) [43, 20, 51, 32,
41, 35, 18, 27, 11, 15, 38, 29]. The basic idea is to train a classification
network and use corresponding CAM images as pseudo-masks, assisting another
segmentation network in fully supervised learning [9]. It should be noted that
the segmentation task aims to identify the complete objects; but CAM usually
highlights the most discriminative part, thereby compromising segmentation
results. Moreover, due to the high homogeneity of histopathological images,
CAM might confuse local activations caused by different representations within
an object [34]. To address these challenges, SCCAM [10] proposed to sub-
categorize objects by feature-level clustering before the pseudo supervision
procedure. Han _et al_. [18] introduced a Progressive Drop Attention (PDA)
method to deactivate highlighted regions gradually. Alternatively, PistoSeg
[15] trained a Synthesized Dataset Generation Module, partially replacing the
functionality of CAM. C-CAM [11] introduces a causality module to regularize
CAM maps by addressing organ co-occurrence in medical images.

Multiple Instance Learning is also widely adopted for various image
segmentation [33] [39] [45] [40] and classification tasks [3] [44]. It was
initially proposed by Dietterich _et al_. [13], where training is done on sets
of instances (referred to as "bags") instead of individual instances. In
binary classification settings, a bag is labeled positive if it contains at
least one positive instance, otherwise, it’s labeled negative. With the rise
of neural networks, particularly Convolutional Neural Networks (CNNs) [31] for
rich hierarchical feature learning, deep learning have been extensively
combined with MIL methods, achieving outstanding results [47]. In the field of
histopathological imaging, MIL was first applied to patch-level classification
tasks within Whole Slide Images (WSIs) [48]. In such tasks, each WSI is
treated as a bag, and each patch is regarded as an instance. This approach
allowed histopathologists to annotate only at the image level, significantly
reducing the labeling time and cost. Subsequently, Jia _et al_. further
proposed treating each labeled histopathological image as a bag and referring
image pixels as instances for pixel-level segmentation [26]. Since then, many
emerging techniques have been applied to MIL. For instance, Hashimoto _et al_.
[19] incorporated multi-scale feature learning into MIL for WSI segmentation
and Ilse _et al_. [25] introduced attention mechanisms into MIL to address its
blurred boundary issue.

This research introduces a fusion-knowledge distillation framework designed to
counteract potential degradation from noise in MIL outcomes, opening a new
avenue of leveraging MIL in the pseudo-supervision paradigm.

###  2.2 Knowledge Distillation

Knowledge distillation [23] is a technique that relies on teacher-student
(T-S) architecture and targets to transfer knowledge from one model (teacher
model) to another (student model). As shown in Fig. 2, beyond labels, the
student model can learn various feature representations from the teacher
model, such as prediction probabilities, feature vectors, and more [50]. Since
the student model usually exhibits superior performance over the teacher model
[16], methods like momentum update have been widely used to further improve
teacher’s performance by updating it through the student model [7].
Particularly, in iterative knowledge distillation, the student itself acts as
the teacher, leveraging its past predictions for more informative supervision
during training for generalization capability enhancement [28]. To prevent the
occurrence of model collapses in such self-supervised learning tasks, various
strategies including contrastive loss [46], clustering constraints [6],
predictor [17] or batch normalizations [42] are adopted in model optimization
process.

It’s important to highlight that, in contrast to the ensemble teacher
distillation approach for classification mentioned in [28], our knowledge
distillation method maintains the teacher model unchanged during the student’s
training phase. Once the student model outperforms the teacher, their roles
are reversed, and a new cycle of knowledge distillation begins.

##  3 Methodology

Problem formulation. Let
$X=\\{x_{1},x_{2},...,x_{n}|x_{i}\in\mathbb{R}^{3\times H\times W}\\}$ be the
training data with their image-level labels denoted as
$Y=\\{y_{1},y_{2},...,y_{n}|y_{i}\in\\{0,1\\}\\}$, where $y_{i}=0$ indicates
the corresponding image containing normal tissue only, whereas $y_{i}=1$
suggests that $x_{i}$ contains either a combination of normal and cancerous
tissues or solely cancerous tissue. With the coarse-grained supervision
$\\{X,Y\\}$, the goal of this study is to train a segmentation model that can
predict pixel-level masks for histopathology images.

Figure 3: An overview of our proposed weakly-supervised segmentation method.
Our method consists of two basic stages. In the first stage, we train a simple
teacher model through MIL with input image patches and patch labels. In the
second stage, we use iterative knowledge distillation to achieve precise
pixel-level segmentation. Note, in the knowledge distillation process, the
teacher model is frozen before parameter switch.

####  3.0.1 System Overview.

To explore MIL for pseudo-supervision in weakly supervised segmentation, a
critical hurdle to overcome is to prevent potential adversarial effects of
noise present in MIL-derived segmentation outputs. To tackle this challenge,
we introduce a iterative fusion-knowledge distillation framework to refine and
stabilize the pseudo supervision learning. As illustrated in Fig. 3, our MIL-
based knowledge distillation framework consists of two training stages.
Briefly, we first train a simplistic segmentation model following the MIL
paradigm for pixel-level pseudo segmentation masks. To refine the pseudo-mask,
we then construct a teacher-student framework and optimize the student
segmentation net by a novel self-supervised fusion-knowledge distillation
strategy. Particularly, this strategy encourages each block of the student
model to align with the teacher’s final output directly, facilitating
effective segmentation of local and global cancerous regions. It is worth
noting that though both of the teacher and student models in Fig. 3 adopt the
first three convolutional blocks of VGG16 [43] as the backbones, we show in
our experiments that our method can easily adapt to different MIL-based
backbones and boost their performance significantly. We will specify the two
training stages as follows.

###  3.1 MIL for Pseudo-Mask Generation

In our multiple instance learning, we take each image pixel as one instance
and an image as a bag. Following the convention, the initial mask
$l_{i}\in\mathbb{R}^{\times H\times W}$ of data $x_{i}$ for teacher model
training are naively set as follows: if $y_{i}=0$, $l_{i}=zeros(H,W)$;
Otherwise, $l_{i}=ones(H,W)$. This naively-initialized training set
$\\{X,L\\}=\\{(x_{i},l_{i})\\}$ is then exploited to train the segmentation
model $T$. Considering the cancerous regions can be any size, we follow the
feature pyramid paradigm and let each block generate an independent
segmentation probability map through a lightweight 1x1 convolution layer.
Then, the output is upscaled to the original image size through bilinear
interpolation. By minimizing the difference between each of these probability
maps and the actual labels, the system effectively discerns and learns the
unique characteristics differentiating normal tissues from cancerous ones.

Mathematically, given an image $x$, let $\\{f_{1}^{t},f_{2}^{t},f_{3}^{t}\\}$
be its multi-scale segmentation results by the model $T$. We define their
aggregated outcome $f_{w}^{t}=\sum_{i}w_{i}f_{i}^{t}$ with weight parameter
$w$. To determine the optimal value of $w$ and prevent overfitting, we set $w$
as a learnable parameter and conducted a separate training for 10 epochs to
ascertain its final value. Then, a compound loss is designed to optimize the
teacher segmentation net:

$L^{t}=Dice({f_{w}^{t}},\check{l})+\sum_{i=0}^{3}Dice({f_{i}^{t}},\check{l}),$
(1)

where $Dice$ refers to the dice loss. Notably, instead of directly using the
naive segmentation mask $l$ for model optimization, drawing on the
optimization approach in [22], we let $\check{f_{i}^{t}}=1-f_{i}^{t}$,
$\check{f_{w}^{t}}=1-f_{w}^{t}$ and $\check{l}=1-l$, when $y=0$; otherwise,
$\check{f_{i}^{t}}=f_{i}^{t}$, $\check{f_{w}^{t}}=f_{w}^{t}$ and
$\check{l}=l$. The reason is that normal samples contain only normal tissue
$y=0$; thus, the direct application of Dice loss would result in the Dice loss
for all positive samples consistently being 1. Consequently, the model would
tend to favor the class with a more substantial presence in the dataset,
predicting all areas as cancerous regions.

After the MIL training, the segmentation model $T$ is used as the teacher in
the second stage, i.e. knowledge distillation.

###  3.2 Pseudo Supervision with Iterative Fusion-Knowledge Distillation

Due to the uncertainty and label ambiguity in multiple instance learning, the
pseudo-mask generated by the teacher model $T$ may be noisy and inaccurate,
which is likely to receive excessive focus during certain epochs of the
training process [7]. Thus, the challenge of using the results of MIL-based
segmentation as pseudo masks lies in how to prevent the segmentation model
from deteriorating over time under pseudo supervision. To address this issue,
we propose a novel knowledge distillation procedure which is composed by three
basic elements: fusion-knowledge distillation strategy, (2) distillation
regularization, and (3) dynamic teacher-student switch.

####  3.2.1 Fusion-knowledge distillation

distinguishes itself from conventional knowledge distillation approaches by
targeting a distinct distillation objective. Its focus shifts from merely
mimicking individual blocks in the teacher model to a more direct and holistic
learning process. As illustrated in Fig. 2, our fusion knowledge distillation
method mandates that each block in the student model directly learns from the
final output of the teacher model. This unique strategy involves guiding each
block of the student model to acquire a balanced representation of both high-
level and low-level features. This innovation empowers even the shallow layers
of the student model to capture abstract features typically associated with
deeper neural networks. Thereby providing the student network with the
opportunity to achieve performance surpassing that of the teacher network.

In specific, for an input image $x$, the teacher model and student model will
each generate their multilevel results, denoted by
$\\{f_{1}^{t},f_{2}^{t},f_{3}^{t}\\}$ and
$\\{f_{1}^{s},f_{2}^{s},f_{3}^{s}\\}$ respectively. The student follows the
same manner to calculate the weighted average $f_{w}^{s}$. Then, the fusion-
knowledge distillation loss can be formulated as

$L_{kd}=Dice(\check{f_{w}^{s}},\check{f_{w}^{t}})+\sum_{i=0}^{3}Dice(\check{f_{i}^{s}},\check{f_{w}^{t}}).$
(2)

Note, that since the normal patches only contain normal tissue, we use this
ground-truth mask to optimize the student model. Following the process in
teacher optimization, when $y=0$, $\check{f_{i}^{t}}=1-f_{i}^{t}$,
$\check{f_{w}^{s}}=1-f_{w}^{s}$ and $\check{f_{w}^{t}}=\check{f_{i}^{t}}=0$.

####  3.2.2 Distillation regularization

is introduced to prevent performance deterioration in the knowledge
distillation process. As shown in Fig. 1, MIL-based pseudo-masks invariably
contain a substantial amount of noise. These noises may be intensified
throughout the self-distillation, thereby imposing a detriment far greater
than other methodologies. Specifically, we incorporate a weighted cross-
entropy term $L_{wse}$ [34] into our distillation process.

$L_{wce}=\sum_{i}ce_{i}*\frac{e^{-ce_{i}}}{\sum_{i}e^{-ce_{i}}},$ (3)

where $ce_{i}=-f_{w,i}^{t}\log f_{w,i}^{s}$ is the pixel-level cross-entropy
between segmentation predictions from the teacher and student models. This
regularization term brings two benefits: (1) it prevents the student model
from intensifying noise in pseudo masks during training, and (2) it encourages
Teacher-student block-wise feature alignment statistically.

In summary, our knowledge distillation objective function for student training
is

$L^{s}=L_{kd}+a*L_{wce},$ (4)

where $a$ is a scale factor for $L_{wce}$. If $a$ is small, it fails to
significantly impact the model’s learning process; conversely, when $a$ is
large, it causes the model to overlook smaller cancerous areas. In this study,
we used for $a$ is 0.25. Ablation on this hyperparameter is shown in Table 4.

####  3.2.3 Dynamic Teacher-student Switch

is explored in our iterative knowledge distillation framework. Instead of the
slow-paced momentum update on the teacher model during knowledge distillation,
we initially freeze the model $T$ as a static teacher for stable supervision.
This effectively counters noise amplification during student model updates by
providing a consistent reference, preventing error propagation in knowledge
distillation and performance deterioration over time. With the training
progress, the student gradually catches up, eventually outperforming the
teacher model. Then, we completely swap the role of the two models, turning
the original student model into a new teacher, and the original teacher model
into the student for the next round of knowledge distillation. This iterative
knowledge distillation cycle repeats several times to remove noise and
imperfection in the evolving pseudo-masks. We show in the ablations that
switching the roles of teacher and student models significantly boost the
segmentation performance.

##  4 Experiments

###  4.1 Datasets

We evaluate our approach on two public histopathology datasets as follows.

Camelyon16 [2] contains 398 annotated Hematoxylin and Eosin (H&E) stained
slides for detecting breast cancer metastasis in sentinel lymph nodes, among
which a total of 270 slides were allocated for the training set and the rest
128 slides were for testing. The training set comprised 111 tumor slides and
159 normal slides. The testing set included 80 normal slides and 48 tumor
slides. These slides are stored in a multi-resolution pyramid. At level 0,
each slide has a size of around 100,000 × 100,000 pixels, with x40
magnification (226 nm/pixel).

To prepare the datasets for our weakly supervised task, we crop image patches
from slides into 1280 * 1280 pixels at level 1 (x20 magnification) in our
experiment. Specifically, from the 111 tumor slides in the training set, we
dropped those patches with white backgrounds exceeding 80%, ultimately
retaining 6,570 positive (tumor) patches in our training set. We then cropped
the same number of image patches from normal slides and included them in our
training set, herein referred to as neg-patches.Similarly, we cropped 985 pos-
patches and an equivalent number of neg-patches from the test set. Notably,
when selecting the pos-patches for the testing set, we also dropped all
patches containing more than 90% background areas. The rationale behind this
is that including an excess of such patches in the test set could
unintentionally skew the weakly supervised models’ test results towards those
of fully supervised models, obscuring the true measure of performance.

DigestPath 2019 [12] is a Colonoscopy tissue segmentation dataset, which
contains 250 images of tissue from 93 WSIs, with pixel-level annotation. All
images were stained following the H&E protocol and scanned with a x20
magnification factor. In our experiment, we selected 225 images for training,
while the remaining 25 for testing. All images are randomly cropped into 1536
× 1536 pixels patches. Similar with the CAMELYON16 dataset, we cropped and
filtered 9,000 pos-patches that contained cancerous tissues and 18000 neg-
patches, cumulatively forming the training set. From 25 testing slides, 726
pos-patches and neg-patches are included in the testing set.

###  4.2 Evaluation Protocol

Implementation details: All the experiments were implemented in PyTorch
environment on a server equipped with 2 NVIDIA GeForce RTX 3090 GPUs that have
24G Memory. All image patches are resized to 256 × 256 when feeding into our
model. For both dataset, Adam optimizers are employed to train the model with
initial learning rate 5e-5, a weight decay 5e-4, and a batch size 16. In stage
1, the backbone would be trained for 30 epochs to get the teacher model. In
stage 2, $a=0.25$ and the teacher and student model would switch their
parameters every 30 epochs. All experiments are repeated three times to
calculate the mean and standard deviation. Note, in our main experiment, the
simplest VGG blocks is explored as the backbone. We will substitute VGG with
ResNet and other more advanced MIL-based methods in our ablation study to
demonstrate the generalization of the proposal knowledge distillation
framework.

Evaluation metrics: Following convention, F1-score (F1), Intersection over
Union (IOU) score [14], and Hausdorff Distance(HD) [24] are used to evaluate
the lesion segmentation performance. F1 score is the harmonic mean between
precision and recall. IOU is calculated as the area of overlap between the
predicted segmentation and the ground truth, divided by the area of union of
the predicted segmentation and the ground truth. The Hausdorff Distance
measures the spatial accuracy of the segmented regions, comparing the distance
between the borders of the predicted segmentation and the ground truth. For
each evaluation metric, we first calculate the value for each patch, and then
the average of all patches is taken as the final score.

Comparison baselines: We applied MIL on VGG as our baseline method. For
comparison, the state-of-the-art weakly supervised methods: VGG-MIL [43],
Resnet-MIL [20], SA-MIL [32], Swin-MIL [41], OAA [27], PistoSeg [15],OEEM [34]
are included. VGG-MIL, Resnet-MIL, SA-MIL and Swin-MIL are MIL based methods,
and OAA, OEEM and PistoSeg are CAM based pseudo supervision approaches. In
addition, we conduct the evaluation on a fully supervised method CAC-Unet [52]
for performance reference. For each method, we execute their officially-
released codes on our datasets for a fair comparison.

###  4.3 Results and Discussions

Table 1: Performance comparison to state-of-the-art weakly supervised methods, where BOLD highlights the best performance and$\pm$denotes the standard deviation. A full supervised approach is presented in the table for reference. Methods | Venues | Camelyon16 | DigestPath 2019
---|---|---|---
F1(%) $\uparrow$ |  IOU(%) $\uparrow$ |  HDPos $\downarrow$ |  F1(%) $\uparrow$ |  IOU(%) $\uparrow$ |  HDPos $\downarrow$
Weakly Supervised Methods
OAA [27] | CVPR’19 |  76.3$\pm$1.2 |  70.5 $\pm$1.0 |  44.5$\pm$1.1 |  77.7$\pm$0.7 |  70.9$\pm$1.4 |  33.5$\pm$2.8
OEEM [34] | MICCAI’22 |  81.9$\pm$0.5 |  77.8 $\pm$0.4 |  40.9$\pm$0.9 |  82.9$\pm$0.3 |  70.9$\pm$0.2 |  33.6$\pm$2.1
PistoSeg [15] | AAAI’23 |  81.2$\pm$0.3 |  72.8 $\pm$0.9 |  45.8$\pm$8.7 |  80.6$\pm$0.7 |  72.5 $\pm$0.4 |  29.3 $\pm$1.6
Resnet-MIL [20] | CVPR’16 |  78.6$\pm$0.1 |  72.2 $\pm$0.8 |  46.7$\pm$1.5 |  74.5$\pm$0.8 |  65.9 $\pm$0.8 |  51.0 $\pm$3.5
SA-MIL [32] | MIA’23 |  81.2$\pm$0.7 |  74.4$\pm$1.1 |  36.9$\pm$2.8 |  83.7$\pm$0.2 |  75.9$\pm$0.4 |  22.8$\pm$0.7
Swin-MIL [41] | MICCAI’23 |  82.1$\pm$1.1 |  77.4$\pm$1.0 |  30.8$\pm$2.3 |  81.8$\pm$0.2 |  75.2$\pm$0.6 |  23.6$\pm$2.0
VGG-MIL [43] | arVix’14 |  82.0$\pm$0.1 |  75.0$\pm$0.3 |  27.2$\pm$0.6 |  79.4$\pm$0.9 |  70.3$\pm$1.1 |  24.0$\pm$2.1
Ours | - | 85.6$\pm$0.1 | 80.0$\pm$0.1 | 19.0$\pm$0.4 | 85.4$\pm$0.1 | 77.7$\pm$0.1 | 14.0$\pm$0.8
Fully Supervised Method
CAC-Unet [52] | NeuCompt’21 |  88.3$\pm$0.1 |  84.5$\pm$0.1 |  16.7$\pm$0.4 |  90.1$\pm$0.1 |  85.5$\pm$0.1 |  9.6$\pm$0.4
Figure 4: Visualization of segmentation results on the Camelyon16. The cancer
tissues are shown in blue, and normal tissues are shown in green. OAA [27],
OEEM [34], and PistoSeg [15] follow the CAM-based pseudo-supervision paradigm,
and SA-MIL [32], Swin-MIL [41], Resnet-MIL [20], and our baseline VGG-MIL are
MIL-based approaches. For reference, we also include the results by a fully-
supervised approach CAC-Unet [52].

Main results: The quantitative evaluation results are presented in Table 1.
Our approach surpasses SOTA weakly-supervised learning methods across both
datasets. Particularly, compared to the VGG-MIL baseline, the proposed
iterative knowledge distillation strategy brings in significant performance
boosts. The qualitative evaluation are displayed in Fig. 4. Weakly-supervised
learning methods based on Multiple Instance Learning tend to generate
substantial noise in their predictions. In contrast, methods reliant on Class
Activation Mapping (CAM) and pseudo-labeling struggle to produce sufficiently
accurate segmentation maps. By integrating the strengths of both approaches,
our model has achieved significant improvements in terms of accuracy and
stability.

####  4.3.1 Compatible to MIL-based methods:

As we previously stated, the backbone of our model is not fixed and our
iterative knowledge distillation strategy can be easily adapted to different
MIL-based methods. To substantiate our claim, we substituted the model
backbone with Resnet, SA-MIL and Swin-MIL and conducted tests on both datasets
and report the experiment results in Table 2. Our iterative fusion-knowledge
distillation method boosts the performance of all 4 MIL-based segmentation
methods with considerable large margins.

Table 2: Applying our iterative fusion-knowledge distillation (IFKD) to different MIL-based segmentation methods. Substantial performance improvements is observed.  | Camelyon16 | Digestpath2019
---|---|---
Methods |  F1(%) $\uparrow$ |  IOU(%) $\uparrow$ |  HDPos $\downarrow$ |  F1(%) $\uparrow$ |  IOU(%) $\uparrow$ |  HDPos $\downarrow$
VGG-MIL |  82.0$\pm$0.1 |  75.0$\pm$0.3 |  27.2$\pm$0.6 |  79.4$\pm$0.9 |  70.3$\pm$1.1 |  24.0$\pm$2.1
\+ FKSD |  85.6$\pm$0.1 |  80.0$\pm$0.1 |  19.0$\pm$0.4 |  85.4$\pm$0.1 |  77.7$\pm$0.1 |  14.0$\pm$0.8
ResNet-MIL |  78.6$\pm$0.1 |  72.2$\pm$0.8 |  46.7$\pm$1.5 |  74.5$\pm$0.8 |  65.9$\pm$0.8 |  51.0$\pm$3.5
\+ FKSD |  83.4$\pm$0.1 |  77.8$\pm$0.1 |  33.6$\pm$0.3 |  83.2$\pm$0.2 |  75.5$\pm$0.2 |  22.6$\pm$0.5
Swin-MIL |  82.1$\pm$1.1 |  77.4$\pm$1.0 |  30.8$\pm$2.3 |  81.8$\pm$0.2 |  75.2$\pm$0.6 |  23.6$\pm$2.0
\+ FKSD |  84.4$\pm$0.2 |  78.0$\pm$0.2 |  23.5$\pm$0.4 |  83.5$\pm$0.3 |  76.5$\pm$0.2 |  23.5$\pm$0.6
SA-MIL |  81.2$\pm$0.7 |  74.4$\pm$1.1 |  36.9$\pm$2.8 |  83.7$\pm$0.2 |  75.9$\pm$0.4 |  22.8$\pm$0.7
\+ FKSD |  85.1$\pm$0.1 |  79.1$\pm$0.1 |  21.2$\pm$0.3 |  84.1$\pm$0.2 |  76.8$\pm$0.1 |  20.1$\pm$0.8
Figure 5: Ablation study for fusion-knowledge distillation on Camelyon16. Each
point represents the best result in the past 30 epochs, and the standard
deviation is marked by the error bars. The dashed line is the best score that
teacher model could achieve during MIL. Basic knowledge distillation (a) and
(b) correspond to the knowledge distillation structures shown in Fig. 2.
Figure 6: Ablation study for fusion-knowledge distillation on Digestpath2019.

###  4.4 Ablation Studies

Ablations are performed to assess the effectiveness of the building blocks in
our solution. Each test was repeated for 3 rounds, and mean and standard
deviation were reported. All ablation studies in this section takes VGG as the
backbone.

Figure 7: Ablation on teacher-student role switch in iterative fusion-knowledge distillation on Camelyon16. Each point represents the best result in the past 30 epochs, and the standard deviation is marked by the error bars. The dashed line is the best performance that teacher model could achieve during MIL. Figure 8: Ablation on teacher-student role switch in iterative fusion-knowledge distillation on Digestpath2019.  Table 3: Ablation on our iterative fusion-knowledge distillation objective function. |  | Camelyon16 | Digestpath2019
---|---|---|---
$L_{\text{kd}}$ | $L_{\text{wce}}$ | F1 | IOU | HD | F1 | IOU | HD
✓ |  |  85.1$\pm$0.2 |  78.9$\pm$0.2 |  22.1$\pm$1.0 |  84.5$\pm$0.5 |  76.4$\pm$0.4 |  15.5$\pm$1.4
✓ | ✓ |  85.6$\pm$0.1 |  80.0$\pm$0.1 |  19.0$\pm$0.4 |  85.4$\pm$0.1 |  77.1$\pm$0.1 |  14.0$\pm$0.8
Table 4: Ablation on the hyperparameter $a$ of the objective function on Camelyon16. $a$ | 0 | 0.1 | 0.25 | 0.5 | 1 | 5
---|---|---|---|---|---|---
F1 Score(%) |  85.1 |  85.5 |  85.6 |  85.2 |  84.2 |  81.0

Ablation on Fusion-knowledge Distillation: We compare our fusion-knowledge
distillation to conventional knowledge distillation paradigms shown in Fig. 1
and report their performance over 450 epochs on Camelyon16 and Digetpath2019
in Fig. 5 and 6, respectively. For basic knowledge distillation paradigms (a)
and (b), the shallow layers of the student model are prone to learning local
information from the teacher, while student’s deeper layers can only learn
global semantic information. Consequently, the improvement to the student
model’s performance is relatively limited. In contrast, with fusion knowledge
distillation, each block of the student model learns from the multi-
dimensional features extracted by the teacher model, which empowers student’s
segmentation capability. It is noteworthy that on the Digestpath2019 dataset,
the performance of the knowledge distillation method (a) falls below the
baseline of the VGG model. We speculate that the performance drop arises from
a lack of constraints on the early layers of the student model within the
knowledge distillation process.This also highlights the necessity of fusing
segmentation information from different dimensions.

Ablation on Teacher-Student Role Switch in Iterative Fusion-knowledge
Distillation: Our iterative fusion-knowledge distillation strategy proposes to
iteratively switch the role between the teacher and student models. That is,
the teacher model is also updated. In this ablation, we compare the
segmentation performance with and without the teacher update and show the
results in Fig. 7 and Fig. 8. With the same teacher-student framework, no
teacher update tends to stabilize in performance around 200 epochs, but our
iterative fusion-knowledge distillation continues to bolster overall
performance by updating the teacher model.

Ablation on Iterative Fusion-knowledge Distillation Objective Function and
Hyper-parameter: Our iterative fusion-knowledge distillation strategy is
harnessed by two loss terms: $L_{kd}$, $L_{wce}$. Table 3 shows their impacts
on segmentation results on both Camelyon16 and Digestpath2019. $L_{wce}$
substantially improves the network’s segmentation capability and stability.
The two loss terms are combined by a hyperparameter $a$ in our method. Table 4
quantifies sensitivity of the solution to its numerical setting. It shows that
the system is quite stable when $a<1$.

##  5 Conclusion & Limitation

In this study, we introduced a novel MIL-based iterative fusion-knowledge
distillation framework for weakly supervised semantic segmentation in
histopathology. The iterative fusion-knowledge distillation strategy
significantly enhances MIL-based segmentation by addressing noise and label
ambiguity, enabling a deeper and more comprehensive learning from the teacher
model’s outcomes. Extensive experiments on Camelyon16 and DigestPath 2019
datasets show that our approach surpasses SOTA methods by a large margin.

It should be noted that the process of iterative fusion-knowledge
distillation, especially with the dynamic role reversal and continuous
refinement of models, can be computationally intensive. This may require
significant computational resources and time, potentially limiting its
applicability in resource-constrained scenarios.

## References

  * [1] Ahn, J., Kwak, S.: Learning pixel-level semantic affinity with image-level supervision for weakly supervised semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4981–4990 (2018)
  * [2] Bejnordi, B.E., Veta, M., Van Diest, P.J., Van Ginneken, B., Karssemeijer, N., Litjens, G., Van Der Laak, J.A., Hermsen, M., Manson, Q.F., Balkenhol, M., et al.: Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. Jama 318(22), 2199–2210 (2017)
  * [3] Bontempo, G., Porrello, A., Bolelli, F., Calderara, S., Ficarra, E.: Das-mil: Distilling across scales for mil classification of histological wsis. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 248–258. Springer (2023)
  * [4] Bulten, W., Litjens, G.: Unsupervised prostate cancer detection on h&e using convolutional adversarial autoencoders. arXiv preprint arXiv:1804.07098 (2018)
  * [5] Campanella, G., Hanna, M.G., Geneslaw, L., Miraflor, A., Werneck Krauss Silva, V., Busam, K.J., Brogi, E., Reuter, V.E., Klimstra, D.S., Fuchs, T.J.: Clinical-grade computational pathology using weakly supervised deep learning on whole slide images. Nature medicine 25(8), 1301–1309 (2019)
  * [6] Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., Joulin, A.: Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems 33, 9912–9924 (2020)
  * [7] Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 9650–9660 (2021)
  * [8] Chaitanya, K., Erdil, E., Karani, N., Konukoglu, E.: Local contrastive loss with pseudo-label based self-training for semi-supervised medical image segmentation. Medical Image Analysis 87, 102792 (2023)
  * [9] Chan, L., Hosseini, M.S., Rowsell, C., Plataniotis, K.N., Damaskinos, S.: Histosegnet: Semantic segmentation of histological tissue type in whole slide images. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 10662–10671 (2019)
  * [10] Chang, Y.T., Wang, Q., Hung, W.C., Piramuthu, R., Tsai, Y.H., Yang, M.H.: Weakly-supervised semantic segmentation via sub-category exploration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 8991–9000 (2020)
  * [11] Chen, Z., Tian, Z., Zhu, J., Li, C., Du, S.: C-cam: Causal cam for weakly supervised semantic segmentation on medical image. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11676–11685 (2022)
  * [12] Da, Q., Huang, X., Li, Z., Zuo, Y., Zhang, C., Liu, J., Chen, W., Li, J., Xu, D., Hu, Z., et al.: Digestpath: A benchmark dataset with challenge review for the pathological detection and segmentation of digestive-system. Medical Image Analysis 80, 102485 (2022)
  * [13] Dietterich, T.G., Lathrop, R.H., Lozano-Pérez, T.: Solving the multiple instance problem with axis-parallel rectangles. Artificial intelligence 89(1-2), 31–71 (1997)
  * [14] Everingham, M., Eslami, S.A., Van Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The pascal visual object classes challenge: A retrospective. International journal of computer vision 111, 98–136 (2015)
  * [15] Fang, Z., Chen, Y., Wang, Y., Wang, Z., Ji, X., Zhang, Y.: Weakly-supervised semantic segmentation for histopathology images based on dataset synthesis and feature consistency constraint. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 37, pp. 606–613 (2023)
  * [16] Furlanello, T., Lipton, Z., Tschannen, M., Itti, L., Anandkumar, A.: Born again neural networks. In: International Conference on Machine Learning. pp. 1607–1616. PMLR (2018)
  * [17] Grill, J.B., Strub, F., Altché, F., Tallec, C., Richemond, P., Buchatskaya, E., Doersch, C., Avila Pires, B., Guo, Z., Gheshlaghi Azar, M., et al.: Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems 33, 21271–21284 (2020)
  * [18] Han, C., Lin, J., Mai, J., Wang, Y., Zhang, Q., Zhao, B., Chen, X., Pan, X., Shi, Z., Xu, Z., et al.: Multi-layer pseudo-supervision for histopathology tissue semantic segmentation using patch-level classification labels. Medical Image Analysis 80, 102487 (2022)
  * [19] Hashimoto, N., Fukushima, D., Koga, R., Takagi, Y., Ko, K., Kohno, K., Nakaguro, M., Nakamura, S., Hontani, H., Takeuchi, I.: Multi-scale domain-adversarial multiple-instance cnn for cancer subtype classification with unannotated histopathological images. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 3852–3861 (2020)
  * [20] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
  * [21] He, Y., Li, X.: Whole-slide-imaging cancer metastases detection and localization with limited tumorous data. arXiv preprint arXiv:2303.10342 (2023)
  * [22] He, Y., Li, X.: Whole-slide-imaging cancer metastases detection and localization with limited tumorous data. In: Medical Imaging with Deep Learning. pp. 877–887. PMLR (2024)
  * [23] Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015)
  * [24] Huttenlocher, D.P., Klanderman, G.A., Rucklidge, W.J.: Comparing images using the hausdorff distance. IEEE Transactions on pattern analysis and machine intelligence 15(9), 850–863 (1993)
  * [25] Ilse, M., Tomczak, J., Welling, M.: Attention-based deep multiple instance learning. In: International conference on machine learning. pp. 2127–2136. PMLR (2018)
  * [26] Jia, Z., Huang, X., Eric, I., Chang, C., Xu, Y.: Constrained deep weak supervision for histopathology image segmentation. IEEE transactions on medical imaging 36(11), 2376–2388 (2017)
  * [27] Jiang, P.T., Hou, Q., Cao, Y., Cheng, M.M., Wei, Y., Xiong, H.K.: Integral object mining via online attention accumulation. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 2070–2079 (2019)
  * [28] Kim, K., Ji, B., Yoon, D., Hwang, S.: Self-knowledge distillation: A simple way for better generalization. arXiv preprint arXiv:2006.12000 3, 1 (2020)
  * [29] Kuang, Z., Yan, Z., Yu, L.: Weakly supervised learning for multi-class medical image segmentation via feature decomposition. Computers in Biology and Medicine 171, 108228 (2024)
  * [30] Kumar, N., Verma, R., Sharma, S., Bhargava, S., Vahadane, A., Sethi, A.: A dataset and a technique for generalized nuclear segmentation for computational pathology. IEEE transactions on medical imaging 36(7), 1550–1560 (2017)
  * [31] LeCun, Y., Bengio, Y., et al.: Convolutional networks for images, speech, and time series. The handbook of brain theory and neural networks 3361(10), 1995 (1995)
  * [32] Li, K., Qian, Z., Han, Y., Eric, I., Chang, C., Wei, B., Lai, M., Liao, J., Fan, Y., Xu, Y.: Weakly supervised histopathology image segmentation with self-attention. Medical Image Analysis 86, 102791 (2023)
  * [33] Li, Q., Wu, J., Tu, Z.: Harvesting mid-level visual concepts from large-scale internet images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 851–858 (2013)
  * [34] Li, Y., Yu, Y., Zou, Y., Xiang, T., Li, X.: Online easy example mining for weakly-supervised gland segmentation from histology images. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 578–587. Springer (2022)
  * [35] Lin, T., Yu, Z., Hu, H., Xu, Y., Chen, C.W.: Interventional bag multi-instance learning on whole-slide pathological images. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 19830–19839 (2023)
  * [36] Muñoz-Aguirre, M., Ntasis, V.F., Rojas, S., Guigó, R.: Pyhist: a histological image segmentation tool. PLoS computational biology 16(10), e1008349 (2020)
  * [37] Nagpal, K., Foote, D., Liu, Y., Chen, P.H.C., Wulczyn, E., Tan, F., Olson, N., Smith, J.L., Mohtashamian, A., Wren, J.H., et al.: Development and validation of a deep learning algorithm for improving gleason scoring of prostate cancer. NPJ digital medicine 2(1), 48 (2019)
  * [38] Pan, L., Li, K., Liu, W., Xu, L., Feng, Z., Peng, S.: Cvfc: Attention-based cross-view feature consistency for weakly supervised semantic segmentation of pathology images. In: 2023 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). pp. 1374–1379. IEEE (2023)
  * [39] Papandreou, G., Chen, L.C., Murphy, K.P., Yuille, A.L.: Weakly-and semi-supervised learning of a deep convolutional network for semantic image segmentation. In: Proceedings of the IEEE international conference on computer vision. pp. 1742–1750 (2015)
  * [40] Pinheiro, P.O., Collobert, R.: From image-level to pixel-level labeling with convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1713–1721 (2015)
  * [41] Qian, Z., Li, K., Lai, M., Chang, E.I.C., Wei, B., Fan, Y., Xu, Y.: Transformer based multiple instance learning for weakly supervised histopathology image segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 160–170. Springer (2022)
  * [42] Richemond, P.H., Grill, J.B., Altché, F., Tallec, C., Strub, F., Brock, A., Smith, S., De, S., Pascanu, R., Piot, B., et al.: Byol works even without batch statistics. arXiv preprint arXiv:2010.10241 (2020)
  * [43] Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
  * [44] Wang, X., Tang, F., Chen, H., Luo, L., Tang, Z., Ran, A.R., Cheung, C.Y., Heng, P.A.: Ud-mil: uncertainty-driven deep multiple instance learning for oct image classification. IEEE journal of biomedical and health informatics 24(12), 3431–3442 (2020)
  * [45] Wang, X., Wang, B., Bai, X., Liu, W., Tu, Z.: Max-margin multiple-instance dictionary learning. In: International conference on machine learning. pp. 846–854. PMLR (2013)
  * [46] Wu, Z., Xiong, Y., Yu, S.X., Lin, D.: Unsupervised feature learning via non-parametric instance discrimination. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3733–3742 (2018)
  * [47] Xu, Y., Mo, T., Feng, Q., Zhong, P., Lai, M., Eric, I., Chang, C.: Deep learning of feature representation with multiple instance learning for medical image analysis. In: 2014 IEEE international conference on acoustics, speech and signal processing (ICASSP). pp. 1626–1630. IEEE (2014)
  * [48] Xu, Y., Zhang, J., Chang, E.I.C., Lai, M., Tu, Z.: Context-constrained multiple instance learning for histopathology image segmentation. In: Medical Image Computing and Computer-Assisted Intervention–MICCAI 2012: 15th International Conference, Nice, France, October 1-5, 2012, Proceedings, Part III 15. pp. 623–630. Springer (2012)
  * [49] Xu, Y., Zhu, J.Y., Eric, I., Chang, C., Lai, M., Tu, Z.: Weakly supervised histopathology cancer image segmentation and classification. Medical image analysis 18(3), 591–604 (2014)
  * [50] Zhang, L., Bao, C., Ma, K.: Self-distillation: Towards efficient and compact neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence 44(8), 4388–4403 (2021)
  * [51] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2921–2929 (2016)
  * [52] Zhu, C., Mei, K., Peng, T., Luo, Y., Liu, J., Wang, Y., Jin, M.: Multi-level colonoscopy malignant tissue detection with adversarial cac-unet. Neurocomputing 438, 165–183 (2021)