# IDRNet: Intervention-Driven Relation Network for Semantic Segmentation

Zhenchao Jin1, Xiaowei Hu2, Lingting Zhu1, Luchuan Song4, Li Yuan3, Lequan Yu1
1The University of Hong Kong 2Shanghai AI Laboratory
3Peking University 4University of Rochester
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>

###### Abstract

Co-occurrent visual patterns suggest that pixel relation modeling facilitates
dense prediction tasks, which inspires the development of numerous context
modeling paradigms, _e.g._ , multi-scale-driven and similarity-driven context
schemes. Despite the impressive results, these existing paradigms often suffer
from inadequate or ineffective contextual information aggregation due to
reliance on large amounts of predetermined priors. To alleviate the issues, we
propose a novel Intervention-Driven Relation Network (IDRNet), which leverages
a deletion diagnostics procedure to guide the modeling of contextual relations
among different pixels. Specifically, we first group pixel-level
representations into semantic-level representations with the guidance of
pseudo labels and further improve the distinguishability of the grouped
representations with a feature enhancement module. Next, a deletion
diagnostics procedure is conducted to model relations of these semantic-level
representations via perceiving the network outputs and the extracted relations
are utilized to guide the semantic-level representations to interact with each
other. Finally, the interacted representations are utilized to augment
original pixel-level representations for final predictions. Extensive
experiments are conducted to validate the effectiveness of IDRNet
quantitatively and qualitatively. Notably, our intervention-driven context
scheme brings consistent performance improvements to state-of-the-art
segmentation frameworks and achieves competitive results on popular benchmark
datasets, including ADE20K, COCO-Stuff, PASCAL-Context, LIP, and Cityscapes.
Code is available at https://github.com/SegmentationBLWX/sssegmentation.

Table 1: Performance comparison between existing context schemes and our
IDRNet.

| Multi-Scale-Driven | Similarity-Driven | Intervention-Driven
---|---|---|---
Benchmark |  ASPP [1] |  PPM [2] |  UPerNet [3] |  Non-Local [4] |  Ann [5] |  OCR [6] | IDRNet | IDRNet+
ADE20K [7] | 43.19 | 42.64 | 43.02 | 42.15 | 41.75 | 42.47 | 43.61 | 44.84
Cityscapes [8] | 79.62 | 79.05 | 79.08 | 78.34 | 78.36 | 79.40 | 79.91 | 80.81

##  1 Introduction

Semantic image segmentation, which aims at assigning a semantic category to
each pixel, is one of the most fundamental research areas in computer vision.
Recent years have witnessed great progress in semantic segmentation using deep
convolutional neural networks [9] and transformers [10]. The state-of-the-art
semantic segmentation approaches [1, 11, 12, 13, 14, 15, 16, 6, 17] mostly
follow the encoder-decoder paradigm [18] where the decoder is defined by
replacing the fully-connected layers in classification tasks with some
convolution layers.

Inspired by co-occurrent visual patterns [19, 20, 2], a popular class of
boosting the encoder-decoder architecture can be regarded as incorporating
various contextual modules [1, 14, 21, 4, 6, 2] into decoders. Consequently,
the segmentation performance improvements are conferred by enabling each pixel
to perceive information from other pixels in the input image. For instance,
ASPP [1] and PPM [2] exploited dilated convolutions or pyramid pooling layers
to aggregate pixel representations at different spatial positions. OCRNet [6]
proposed to augment pixel representations with object-level representations.
MCIBI++ [13, 21] differentiated the beyond-image contextual information from
the within-image contextual information and thus improved pixel
representations from the perspective of the whole dataset. To summarize, the
existing contextual modules can be grouped into two batches, _i.e._ , multi-
scale-driven [1, 3, 2] and similarity-driven [4, 6, 5] context schemes.
Despite these methods achieved astonishing results, both schemes suffer from
inadequate or ineffective contextual information aggregation as pixel
relations are modeled with masses of predetermined priors. Specific to
similarity-driven methods, contextual modules are designed to make pixels
aggregate semantically similar contextual representations, which is inadequate
for building the co-occurrent visual patterns. Also, some rigorous empirical
analysis [22, 23] demonstrates that the contexts of pixels modeled by
similarity-driven operations are indistinguishable for different query
positions within an input image. As for multi-scale-driven approaches, pixel
contexts are assumed to be fixed with certain geometric structures, which may
compel pixels to capture some uninformative feature representations. Besides,
previous studies [24, 16, 25] indicate that this fixed context assumption
prevents the generalization of one-scenario-trained segmentors to other
scenarios with unknown geometric transformations.

Figure 1:  Diagram of our intervention-driven relation network. Deletion
diagnostics is leveraged to build relations between semantic-level
representations. With the built relation matrix and semantic-level
representations, pixel representations can be augmented for pixel prediction.

Different from conventional pipelines of building contextual modules, this
paper investigates a more effective paradigm to build relationships among
pixels. The proposed intervention-driven paradigm utilizes deletion
diagnostics [26] to guide building pixel relations and thus aggregate more
meaningful contextual information. In practice, there are two main challenges
in constructing such a paradigm: (1) it is time-consuming to perform deletion
diagnostics on all pixels and (2) it is memory-consuming to maintain a pixel
relation matrix as one image contains plenty of pixels. Motivated by the
observation that pixel relationships are closely connected to object
relationships since a pixel semantic label is determined by which object
region the pixel belongs to, we propose to simplify pixel-level relations
modeling as object-level relations modeling to address the above-mentioned
challenges.

Figure 1 depicts the diagram of our proposed intervention-driven relation
network (IDRNet). We first group pixel representations into semantic-level
representations according to predicted pseudo labels. Then, a relation matrix
is constructed to make the semantic-level representations interact with each
other and thus enhance themselves, where the relation matrix is updated by our
deletion diagnostics mechanism. Finally, original pixel representations are
augmented with the interacted representations and used to produce pixel
labels. Compared with simply performing pixel-level deletion diagnostics, the
GPU memory and computational complexity of our method can be significantly
reduced. For example, on Cityscapes [8], we only need to maintain a relation
matrix with a size $19\times 19$ rather than a relation matrix with an input
image size of $1024\times 512$ during training. As the proposed deletion
diagnostics mechanism is designed for semantic-level representations, it is
also worth noting that our paradigm is general and can be easily extended to
broader applications in other vision tasks, _e.g._ , object detection [27] and
instance segmentation [11].

Table 1 compares the segmentation performance of various context modeling
paradigms on Cityscapes and ADE20K benchmark datasets under fair settings
(_e.g._ , backbone networks and training schedules). As observed, a simple FCN
segmentation framework equipped with our intervention-driven context module
(termed _IDRNet_) can achieve $43.61\%$ and $79.91\%$ mIoU on the datasets,
respectively, which outperforms both dominant similarity-driven and multi-
scale-driven context aggregation schemes. To further demonstrate the
superiority of our method, we also integrate the proposed context scheme into
UPerNet segmentation framework (termed _IDRNet+_), and it is observed that our
paradigm with ResNet-50 backbone yields more impressive results on both
datasets under single-scale testing.

In principle, our paradigm is fundamentally different from and would
complement most similarity-driven and multi-scale-driven contextual modules.
The main contribution can be summarized as,

  * •

To our best knowledge, for the first time, this paper presents an
intervention-driven paradigm, _i.e._ , deletion diagnostics mechanism, to help
model pixel relations.

  * •

This paper indicates that, in semantic segmentation task, pixel-level relation
modeling can be simplified as building semantic-level relations.

  * •

Our intervention-driven context modeling paradigm is conscientiously designed.
It can be seamlessly integrated into various state-of-the-art semantic
segmentation frameworks and achieve consistent performance improvements on
various benchmark datasets, including ADE20K, Cityscapes, COCO-Stuff, LIP and
PASCAL-Context.

We expect that IDRNet can provide insights for the future development of
semantic segmentation and other related vision tasks.

##  2 Related Work

###  2.1 Semantic Segmentation

Semantic segmentation is a fundamental computer vision task, which is an
indispensable component for many applications such as autopilot and medical
diagnosis. Modern semantic image segmentation methods (in deep learning era)
are believed to be derived from FCN [18] which formulates semantic
segmentation as an encoder-decoder architecture. After that, numerous efforts
have been devoted to improving the pipeline. For example, some tailored
encoders like Res2Net [28] and HRNet [29] are designed for semantic
segmentation to replace classification networks [30, 31]. In addition,
automating network engineering [32, 33, 34] is also one of the most efficient
ways to boost the encoder structure. Since transformer [10] attains growing
attention, many recent works [35, 36, 37, 12, 15, 38, 39, 16, 40, 41, 17] also
introduce transformer structure to help the encoder extract more effective
pixel representations. Furthermore, there are also plenty of studies focusing
on utilizing co-occurrent patterns to boost the decoder performance, _e.g._ ,
modeling long-range dependency [22, 42, 43, 14, 4, 23, 44] and enlarging the
receptive field [45, 1, 46, 47, 48, 2]. Besides, some researchers also find
that introducing context-aware optimization objectives [49, 50, 51] can
strengthen context cues. Apart from the above, some studies also choose to
break new ground, _e.g._ , introducing contrastive learning mechanism [52,
53], formulating the instance, panoptic and semantic segmentation tasks in a
unified framework [11, 54, 55], leveraging boundary information [56, 57, 58,
59, 60], to name a few.

This paper studies how to leverage co-occurrent patterns. The key differences
are that (1) we transform the pixel relation modeling problem to the semantic-
level relation modeling problem and (2) for the first time, we propose to
introduce deletion diagnostics mechanism to help model pixel relations.

###  2.2 Pixel Relation Modeling

As there exist co-occurrent patterns (_i.e._ , how likely two semantic classes
can exist in a same image) in semantic segmentation, pixel relation modeling
becomes a crucial component of recent state-of-the-art semantic segmentation
algorithms [22, 61, 14, 3, 4, 23, 48, 6, 2]. For example, non-local network
[4] built non-local blocks for capturing long-range dependencies. GCNet [22]
unified the non-local and squeeze-excitation operations into a general
framework for aggregating more efficacious global contextual information. DNL
[23] presented the disentangled non-local block to help learn more
discriminative pixel representations. ISA [61] was motivated to improve the
efficiency of the self-attention operations and proposed to factorize one
dense affinity matrix as the product of two sparse affinity matrices. Besides,
OCRNet [6] proposed to emphasize utilizing the contextual information in the
corresponding object class. ISNet [14] further improved OCRNet by
incorporating image-level and semantic-level contextual information.

Succinctly, these contextual modules utilize feature similarities or fixed
geometric structures to model pixel relations, which injects an unavoidable
human priors bias. Distinctively, this paper introduces deletion diagnostics
mechanism to guide pixel relation modeling.

###  2.3 Object Relation Modeling

In object detection, co-occurrence is first applied in DPM [62] to refine
object scores. After that, several works [63, 64, 65] were proposed to take
object position and size into account for better building object relations in
pre-deep learning era. In deep learning era, object relation modeling is also
one of the most popular techniques to augment object representations. For
instance, relation networks [66] extended original attention module to model
both object similarities and relative geometry as the relations between
objects. Motivated by relation networks, SGRN [67] took advantage of graph
convolutional networks [68] to better bridge object regions by defining them
as graph nodes. SMN [69] proposed to adopt a memory mechanism to perform
object-object context reasoning.

Considering that each semantic-level representation in our paradigm plays a
similar role to an object region in object detection, our intervention-driven
relation modeling paradigm can also be extended to object detection tasks to
help object relation modeling.

Figure 2:  Illustration of our intervention-driven relation network (IDRNet).
We first extract pixel representations $R_{p}$ using a backbone network
$\mathcal{F}_{B}$, _e.g._ , ResNet [30] or SwinTransformer [15]. Then, $R_{p}$
is grouped into semantic-level representations $R_{sl}$ based on a coarse
pixel prediction $P_{c}$. Next, we leverage a relation matrix
$\mathcal{M}_{r}$ to make $R_{sl}$ interact with each other and further obtain
enhanced semantic-level representations $R_{esl}$. Finally, $R_{p}$ is
augmented by $R_{esl}$ and utilized for final pixel predictions $P_{o}$. After
each training iteration, the deletion diagnostics is conducted to update
$\mathcal{M}_{r}$.

##  3 Methodology

###  3.1 Overview of IDRNet

Figure 2 illustrates our proposed intervention-driven relation network
(IDRNet). The initial pixel-wise representations are grouped into semantic-
level representations to facilitate relationship modeling. In detail, the
semantic-level representations are first processed with a feature enhancement
module and a feature interaction module with the guidance of a relation matrix
$\mathcal{M}_{r}$, where the relation matrix is learned with the proposed
deletion diagnostics paradigm. We subsequently augment the initial pixel
representations with the processed semantic-level representations for final
predictions.

Semantic-level Representation Generation. Following conventional FCN-based
architecture [18], we first adopt a backbone network $\mathcal{F}_{B}$ to
extract the pixel representations
$R_{p}\in\mathbb{R}^{Z\times\frac{H}{8}\times\frac{W}{8}}$ from an input image
$\mathcal{I}\in\mathbb{R}^{3\times H\times W}$, where $H$ and $W$ are height
and width of the image and $Z$ is the number of feature channels. To
facilitate the seamless integration of our paradigm with existing segmentation
frameworks, we formulate the pseudo-label generation process as follows,

$P_{c}=\mathcal{F}_{C}(\mathcal{C}_{e}(R_{p})),~{}Y_{c}=\text{argmax}(P_{c}),$
(1)

where $\mathcal{C}_{e}$ denotes an identity function (instantiated as FCN
[18]) or a previous similarity- / multi-scale-driven context module
(instantiated as OCRNet [6] and PSPNet [2]) and $\mathcal{F}_{C}$ is a
convolution layer to project pixel representations into probability space.
$P_{c}\in\mathbb{R}^{K\times\frac{H}{8}\times\frac{W}{8}}$ is the predicted
probabilities of $K$ classes and
$Y_{c}\in\mathbb{R}^{\frac{H}{8}\times\frac{W}{8}}$ is the pseudo label.

As it is computationally impracticable to direct model pixel-level
relationships with deletion diagnostics, we propose to simplify pixel-level
relations modeling as semantic-level relations modeling, _i.e._ , we group
$R_{p}$ into semantic-level representations $R_{sl}$ by aggregating the
corresponding input pixel representations with the same pseudo label,

$R_{sl}=\Big{\\{}\mathcal{F}_{M}\Big{(}\big{\\{}R_{p,[*,~{}i,~{}j]}~{}|~{}Y_{c,[i,~{}j]}=k\big{\\}}\Big{)},k\in[0,K)\Big{\\}},$
(2)

where $R_{sl}\in\mathbb{R}^{N_{e}\times Z}$ and $N_{e}$ is the number of
classes existed in $\mathcal{I}$. $\mathcal{F}_{M}$ is a weighted summation
operation to merge pixel representations with the corresponding probabilities
in $P_{c}$ as the weights.

Feature Enhancement. We further introduce a feature enhancement module to
improve the discrimination of $R_{sl}$ to help augment pixel representations
$R_{p}$. Specifically, we modify Eq (2) as

$R_{sl}=\Big{\\{}\mathcal{F}_{M}\Big{(}\big{\\{}R_{p,[*,~{}i,~{}j]}\oplus
R_{ie,[k,*]}~{}|~{}Y_{c,[i,~{}j]}=k\big{\\}}\Big{)},k\in[0,K)\Big{\\}},$ (3)

where $\oplus$ denotes a concatenation operation and
$R_{ie}\in\mathbb{R}^{K\times Z}$ denotes discriminative vectors for each
class in the training dataset. In our experiments, we investigate two
strategies to obtain $R_{ie}$. The first one is to yield a random orthogonal
matrix [70] to represent $R_{ie}$, while the second one is to learn a dataset-
level representation for each category to construct $R_{ie}$ [13]. For the
second strategy, we simplify $R_{ie}$ update strategy for class id $k$ as

$R_{ie,[k,*]}=R_{ie,[k,*]}\times(1-m)+R_{gt}\times m,$ (4)

where $m$ is an update momentum and set as $0.1$ by default. Before training,
we simply initialize $R_{ie}$ as a zero matrix and
$R_{gt}\in\mathbb{R}^{1\times Z}$ is calculated as

$R_{gt}=\Big{\\{}\text{Mean}(\\{R_{p,[*,~{}i,~{}j]}~{}|~{}GT_{[i,~{}j]}=k\\}),k\in[0,K)\Big{\\}},$
(5)

where $GT$ denotes the ground truth segmentation mask and $\text{Mean}(\cdot)$
is used to average the extracted pixel representations with the same ground
truth label.

Feature Interaction. After the feature enhancement module, we propose to make
$R_{sl}$ interact with each other to enrich themselves by using a semantic-
level relation matrix $\mathcal{M}_{r}\in\mathbb{R}^{K\times K}$, which is
updated by the proposed deletion diagnostics mechanism. The procedure of the
feature interaction module can be represented as

$R_{esl}=\text{Softmax}(\mathcal{T}(\mathcal{M}_{r}),dim=1)\otimes R_{sl},$
(6)

where we adopt $\otimes$ to denote matrix multiplication and
$R_{esl}\in\mathbb{R}^{N_{e}\times Z}$ is the enhanced semantic-level
representations. $\mathcal{T}$ is employed to transform $\mathcal{M}_{r}$ to
adapt the shape of $R_{sl}$, and it is formulated as

$\hat{\mathcal{M}}_{r}=\mathcal{T}(\mathcal{M}_{r})=\\{\mathcal{S}(\mathcal{M}_{r,[i,j]},t)~{}|~{}i,j\in\\{class~{}ids~{}in~{}R_{sl}\\}\\},$
(7)

where $\mathcal{S}$ is introduced to control that there are no interactions
between two semantic-level representations whose relationship in
$\mathcal{M}_{r}$ is weak. $t=0$ is a threshold utilized to identify those
weak relationships. In practice, we set $\mathcal{M}_{r,[i,j]}$ be negative
infinity in the calculation if $\mathcal{M}_{r,[i,j]}$ is smaller than $t$.
Subsequently, we re-arrange $R_{esl}$ into the original pixel representation
shape to augment $R_{p}$,

$R_{a,[*,i,j]}=R_{esl,[k^{idx},*]}~{}~{}if~{}~{}Y_{c,[i,j]}=k,~{}k\in\\{class~{}ids~{}in~{}R_{sl}\\},$
(8)

where $k^{idx}$ is the corresponding matrix index for class id $k$ in
$\mathcal{R}_{sl}$. $R_{a}\in\mathbb{R}^{Z\times\frac{H}{8}\times\frac{W}{8}}$
are the feature representations initialized as a zero matrix, and we fill it
with the representations in $R_{esl}$ according to the pseudo labels in each
pixel position.

Final Prediction. We use
$R_{a}\in\mathbb{R}^{Z\times\frac{H}{8}\times\frac{W}{8}}$ to augment $R_{p}$
for final predictions,

$\widetilde{R}=R_{a}\oplus\mathcal{C}_{e}(R_{p}),R=SA(\widetilde{R}),$ (9)

where we use the self-attention mechanism $SA$ [10] to further process the
augmented representations to balance the diversity and discriminative of the
pixels with the same class. Based on $R$, the final pixel predictions
$P_{o}\in\mathbb{R}^{K\times H\times W}$ is produced as

$P_{o}=\mathcal{U}_{8\times}(\mathcal{F}_{O}(R)),$ (10)

where $\mathcal{F}_{O}$ is implemented by a convolution layer, which is
utilized to produce pixel class probabilities and $\mathcal{U}$ is a bilinear
interpolation operation.

During training, the overall objective function for backpropagation algorithm
[71, 72] is defined as

$\mathcal{L}=\alpha\mathcal{L}_{c}+\mathcal{L}_{o},$ (11)

where $\mathcal{L}_{c}$ and $\mathcal{L}_{o}$ are the cross entropy losses
between $P_{c}$ and $GT$, and $P_{o}$ and $GT$, respectively. $\alpha$ is a
hyperparameter for balancing $\mathcal{L}_{c}$ and $\mathcal{L}_{o}$. We
empirically set it as $0.4$ in our experiments.

###  3.2 Deletion Diagnostics

Deletion diagnostics is proposed to update the relation matrix
$\mathcal{M}_{r}$, which is initialized as an identity matrix. As shown in
Figure 2, we randomly delete one semantic-level representation in $R_{sl}$
with a deletion distribution $Prob$ and obtain $R^{\prime}_{sl}$. The deletion
probability $Prob_{i}$ of the $i$-th semantic class representation is related
to the number of times (_i.e._ , $count_{i}$) that this category has been
deleted and can be represented as (termed balanced deletion in Figure 2)

$Prob_{i}=\frac{\frac{1}{count_{i}}}{\sum\frac{1}{count_{i}}},i\in\\{class~{}ids~{}in~{}R_{sl}\\}.$
(12)

We calculate the pixel predictions $P^{\prime}_{o}$ by re-conducting Eq (6) -
(10) with $R^{\prime}_{sl}$ as input and further calculate the pixel-wise
cross-entropy loss $l$ and $l^{\prime}$ for $P_{o}$ and $P^{\prime}_{o}$,
respectively. After that, we extract the loss values from $l$ with respect to
semantic class id $j$ as follows,

$l_{j}=\\{l_{[h,w]}|GT_{[h,w]}=j\\}.$ (13)

We also conduct the same operation on $l^{\prime}$ to obtain $l^{\prime}_{j}$.

Given the extracted two sets of loss values $l_{j}$ and $l^{\prime}_{j}$, we
can model the relationship between the deleted semantic class id $i$-th and
one reserved class id $j$ in $R_{sl}$. Specifically, we calculate the mean and
variance differences between $l_{j}$ and $l^{\prime}_{j}$,

$\begin{split}r^{i,j}_{mean}&=\frac{\sum l^{\prime}_{j}-\sum l_{j}}{K\times
H\times W},\\\
r^{i,j}_{var}&=\frac{\sum(l^{\prime}_{j,[\cdot]}-\overline{l}^{\prime}_{j})^{2}-\sum(l_{j,[\cdot]}-\overline{l}_{j})^{2}}{K\times
H\times W},\end{split}$ (14)

where $\overline{l}_{j}$ and $\overline{l}^{\prime}_{j}$ denote the mean of
$l_{j}$ and $l^{\prime}_{j}$, respectively. Eq (14) indicates that the built
semantic-level relations are supposed to help pixel recognition from the
perspective of the mean and variance of the loss variation. To utilize such
modeled relationship, we maintain two relation matrices
$\mathcal{M}_{r_{mean}}$ and $\mathcal{M}_{r_{var}}$ during training and
update them as

$\begin{split}\mathcal{M}_{r_{mean},[i,j]}&=m_{mean}\times
r^{i,j}_{mean}+(1-m_{mean})\times\mathcal{M}_{r_{mean},[i,j]},\\\
\mathcal{M}_{r_{var},[i,j]}&=m_{var}\times
r^{i,j}_{var}+(1-m_{var})\times\mathcal{M}_{r_{var},[i,j]},\end{split}$ (15)

where $m_{mean}$ and $m_{var}$ are the momentum for two matrices and they are
both empirically set as $0.1$. By re-conducting Eq (6)-(8) with
$\mathcal{M}_{r_{mean}}$ and $\mathcal{M}_{r_{var}}$ as the relation matrix,
we can obtain $R_{a,mean}$ and $R_{a,var}$, respectively. Finally, we combine
them to obtain $R_{a}$ in Eq (9): $R_{a}=R_{a,mean}\oplus R_{a,var}$.

###  3.3 Discussion of IDRNet

The procedure of deletion diagnostics looks a lot like Dropout [73], while
they are totally different mechanisms. Dropout is a mechanism to prevent deep
neural networks from overfitting, while our approach introduces deletion
diagnostics to help model pixel relations. In addition, our work also relates
closely to some recent segmentation approaches, which extract semantic-level
representations to augment pixel representations, _e.g._ , OCRNet [6] and
ISNet [14]. Whereas, these methods only use feature similarities to make
connections between pixels, which is distinct from our paradigm as it is
intervention-driven and without masses of predetermined priors.

Our paradigm simplifies pixel relations modeling as object relations modeling.
Benefiting from this simplification, deletion diagnostics enables the network
to directly utilize its objective function to examine whether one class $i$
can help recognize another class $j$. The key idea behind this mechanism is
that loss value of class $j$ should increase after deleting class $i$ if class
$i$ can facilitate the prediction of class $j$, and the increased value can be
adopted to measure the importance of class $i$ to $j$. Therefore, there is no
potential predetermined priors bias in the process of deletion diagnostics. As
a comparison, the geometric transformations of multi-scale-driven context
schemes are manually set, which tends to aggregate some ineffective
information and lacks of generalization. Similarity-driven context schemes are
assumed to aggregate semantically similar pixel representations, which results
in ignoring other dissimilar but effective pixel representations for building
co-occurrent patterns, _e.g._ , sky pixels for airplane pixels. In a nutshell,
our intervention-driven paradigm is more effective compared to the existing
similarity- / multi-scale-driven context modules.

##  4 Experiments

###  4.1 Experimental Setup

Table 2: Performance improvements on various benchmark datasets with different
segmentation frameworks after leveraging our intervention-driven paradigm to
model pixel relations.

Method | Backbone | Stride | ADE20K (_train / val_) | COCO-Stuff (_train / test_) | Cityscapes (_train / val_) | LIP (_train / val_)
---|---|---|---|---|---|---
FCN [18] | ResNet-50 | $8\times$ | 36.96 | 31.76 | 75.16 | 48.63
FCN+IDRNet (_ours_) | ResNet-50 | $8\times$ | 43.61 (+6.65) | 38.64 (+6.88) | 79.91 (+4.75) | 51.24 (+2.61)
PSPNet [2] | ResNet-50 | $8\times$ | 42.64 | 37.40 | 79.05 | 51.94
PSPNet+IDRNet (_ours_) | ResNet-50 | $8\times$ | 44.02 (+1.38) | 39.13 (+1.73) | 79.88 (+0.83) | 53.29(+1.35)
DeeplabV3 [1] | ResNet-50 | $8\times$ | 43.19 | 37.63 | 79.62 | 52.35
DeeplabV3+IDRNet (_ours_) | ResNet-50 | $8\times$ | 44.75 (+1.56) | 39.31 (+1.68) | 80.69 (+1.07) | 53.87 (+1.52)
UPerNet [3] | ResNet-50 | $8\times$ | 43.02 | 37.65 | 79.08 | 52.95
UPerNet+IDRNet (_ours_) | ResNet-50 | $8\times$ | 44.84 (+1.82) | 39.35 (+1.70) | 80.81 (+1.73) | 54.00 (+1.05)

Table 3:  Ablation study on the design of IDRNet. IE-Orthogonal: feature
enhancement with orthogonal matrix. IE-DL: feature enhancement with dataset-
level representations. BD: balanced deletion in deletion diagnostics. RD:
random deletion in deletion diagnostics.

IE-Orthogonal | IE-DL | $\mathcal{M}_{r_{mean}}$ | $\mathcal{M}_{r_{var}}$ | BD | RD | ADE20K (_train / val_) | PASCAL-Context (_train / val_) | COCO-Stuff (_train / test_)
---|---|---|---|---|---|---|---|---
✗ | ✗ | ✗ | ✗ | ✗ | ✗ | 36.96 | 45.48 | 31.76
✗ | ✗ | ✓ | ✓ | ✓ | ✗ | 42.82 | 52.65 | 37.79
✗ | ✓ | ✓ | ✓ | ✓ | ✗ | 43.61 | 52.97 | 38.64
✓ | ✗ | ✓ | ✓ | ✓ | ✗ | 42.99 | 52.80 | 38.86
✗ | ✓ | ✗ | ✓ | ✓ | ✗ | 42.74 | 52.86 | 38.06
✗ | ✓ | ✓ | ✗ | ✓ | ✗ | 43.13 | 52.66 | 37.89
✗ | ✓ | ✓ | ✓ | ✗ | ✓ | 42.90 | 52.92 | 37.59

Benchmark Datasets. Our approach is validated on five popular semantic
segmentation benchmark datasets, including ADE20K [7], COCO-Stuff [74],
Cityscapes [8], LIP [75] and PASCAL-Context [76]. In detail, ADE20K is one of
the most well-known datasets for scene parsing, which contains 150
stuff/object category labels. There are 20K/2K/3K images for training,
validation and test set, respectively in the dataset. COCO-Stuff is a
challenging dataset which provides rich annotations for 91 thing classes and
91 stuff classes. It consists of 9K/1K images in the training and test sets.
Cityscapes has 5K high-resolution annotated urban scene images, with
2,975/500/1,524 images for training/validation/testing. It covers 19
challenging categories, such as traffic sign, vegetation and rider. LIP mainly
focuses on single human parsing and contains 50K images with 19 semantic human
part classes and 1 background class. Its training, validation and test sets
separately involve 30K/10K/10K images. PASCAL-Context involves 59 foreground
classes and a background class for scene parsing. The dataset is divided into
4,998 and 5,105 images for training and validation.

Implementation Details. Our algorithm is implemented in PyTorch [77] and
SSSegmentation [78]. For backbone networks (_i.e._ , encoders), they are all
pretrained on ImageNet dataset [79]. Remaining layers (_i.e._ , decoders) are
initialized by the default initialization strategy in PyTorch. As for the data
augmentation, some common data augmentation techniques are utilized, including
random horizontal flipping, random cropping, color jitter and random scaling
(from 0.5 to 2). The learning rate is scheduled by polynomial annealing policy
with factor $(1-\frac{iter}{total\\_iter})^{0.9}$.

Figure 3:  Qualitative comparison with FCN, OCRNet, ISNet and MCIBI++ on
Cityscapes dataset.

Figure 4: Compare category sampling frequency of balanced deletion (BD) and
random deletion (RD) on Cityscapes (left) and LIP (right) benchmark dataset.

Update strategy of $\mathcal{M}_{r}$ | Backbone | mIoU on ADE20K
---|---|---
FCN | ResNet-50 | 36.96
FCN + BP-driven $\mathcal{M}_{r}$ | ResNet-50 | 40.35
FCN + DD-driven $\mathcal{M}_{r}$ | ResNet-50 | 43.61

Table 4: Performance comparison between utilizing back propagation algorithm
(_i.e._ , BP-driven $\mathcal{M}_{r}$) and using our deletion diagnostics
(_i.e._ , DD-driven $\mathcal{M}_{r}$) to update the relation matrix
$\mathcal{M}_{r}$.

Table 5:  Complexity comparison with existing context modules on a single RTX
3090 Ti GPU. The input feature map is with size $\left[1\times 2048\times
128\times 128\right]$. Excepted for mIoU column, all numbers are the smaller
the better.

Context Module | Parameters | FLOPS | Time | GPU Memory | mIoU on ADE20K (%)
---|---|---|---|---|---
OCR [6] | 15.12M | 242.48G | 16.58ms | 617.24M | 42.47
ASPP [1] | 42.21M | 674.47G | 41.98ms | 976.06M | 43.19
PPM [2] | 23.07M | 309.45G | 21.45ms | 960.63M | 42.64
UPerNet [3] | 34.75M | 500.76G | 36.51ms | 1429.18M | 43.02
ANN [5] | 22.42M | 369.62G | 26.58ms | 1445.75M | 41.75
CCNet [43] | 23.92M | 397.38G | 30.92ms | 986.28M | 42.48
DNL [23] | 24.12M | 395.25G | 51.38ms | 2381.04M | 43.50
IDRNet (_ours_) | 10.79M | 155.89G | 20.52ms | 365.66M | 43.61
PPM+IDRNet (_ours_) | 23.65M | 349.23G | 32.64ms | 1034.28M | 44.02

Table 6: Compare segmentation performance in cross domain.

|  Urban Scene Parsing (_Cityscapes train_) |  Human Parsing (_LIP train_)
---|---|---
Method | Dark Zurich val | Nighttime Driving test | CIHP val
FCN [18] | 10.66 | 17.90 | 27.20
FCN+IDRNet (_ours_) |  12.55 (+1.89) |  21.33 (+3.43) |  28.59 (+1.39)
DeeplabV3 [1] | 10.03 | 21.91 | 27.02
DeeplabV3+IDRNet (_ours_) |  13.66 (+3.63) |  23.85 (+1.94) |  27.93 (+0.91)
UPerNet [2] | 11.06 | 17.67 | 26.84
UPerNet+IDRNet (_ours_) |  11.42 (+0.36) |  21.13 (+3.46) |  27.46 (+0.62)

Specific to ADE20K, we set learning rate, crop size, batch size and training
epochs as $0.01$, $512\times 512$, $16$ and $130$, respectively. Specific to
COCO-Stuff, learning rate, crop size, batch size and training epochs are set
as $0.001$, $512\times 512$, $16$ and $110$, respectively. As for LIP, we set
learning rate, crop size, batch size and training epochs as $0.01$, $473\times
473$, $32$ and $150$, respectively. As for Cityscapes, learning rate, crop
size, batch size and training epochs are set as $0.01$, $512\times 1024$, $8$
and $220$, respectively. Specific to PASCAL-Context, we set learning rate,
crop size, batch size and training epochs as $0.004$, $480\times 480$, $16$
and $260$, respectively.

Evaluation Metric. Following conventions [45, 18], mean intersection-over-
union (mIoU) is utilized for evaluation.

###  4.2 Ablation Study

Performance Improvements. As seen in Table 2, it is observed that introducing
deletion diagnostics to help pixel relationship building can bring consistent
performance improvements on different segmentation approaches (_i.e._ , FCN,
PSPNet, DeeplabV3 and UPerNet) and benchmark datasets (_i.e._ , ADE20K, COCO-
Stuff, Cityscapes and LIP). By way of illustration, our method boosts FCN,
PSPNet, DeeplabV3 and UPerNet by 2.61%, 1.35%, 1.52%, 1.05% mIoU on LIP
dataset, respectively.

Feature Enhancement. Feature enhancement is designed to make pixel
representations with different classes more discriminative to help model
relationship between semantic-level representations $R_{sl}$. As indicated in
Table 3, we can observe that yielding random orthogonal matrix (IE-Orthogonal)
and generating dataset-level representations (IE-DL) for feature enhancement
are both promising strategies to boost segmentation performance. Specifically,
IE-Orthogonal brings $0.17\%$, $0.15\%$ and $1.07\%$ mIoU gains for ADE20K,
PASCAL-Context and COCO-Stuff, respectively. IE-DL achieves $0.79\%$, $0.32\%$
and $0.85\%$ mIoU improvements for the three datasets, respectively.

Relation Matrix. We also validate the effectiveness of the proposed relation
matrix in Table 3. It is observed that $\mathcal{M}_{r_{mean}}$ and
$\mathcal{M}_{r_{var}}$ both contribute to the performance improvements. In
detail, $\mathcal{M}_{r_{mean}}$ makes our segmentor outperform baseline
models (first row) by $6.17\%$, $7.18\%$ and $6.13\%$ on ADE20K, PASCAL-
Context and COCO-Stuff, respectively. $\mathcal{M}_{r_{var}}$ boosts baseline
models by $5.78\%$, $7.38\%$ and $6.30\%$ on the three datasets, respectively.
The combination of $\mathcal{M}_{r_{mean}}$ and $\mathcal{M}_{r_{var}}$
finally yields $43.61\%$, $52.97\%$ and $38.64\%$ mIoU on the three datasets,
respectively.

Deletion Diagnostics. In Table 4, we compare our proposed deletion diagnostics
procedure (DD-driven $\mathcal{M}_{r}$) with back-propagation updating
strategy (BP-driven $\mathcal{M}_{r}$). We can observe that adopting DD-driven
$\mathcal{M}_{r}$ outperforms using BP-driven $\mathcal{M}_{r}$ by 3.26% mIoU
on ADE20k. The result suggests that our deletion diagnostics strategy is more
effective than the back-propagation updating strategy.

Balanced Deletion. Balanced deletion is utilized to help improve the being-
diagnosed probabilities of uncommon categories in training dataset and
thereby, deletion diagnostics can be performed evenly. In Table 3, we can
observe that introducing the balanced deletion strategy can help improve the
segmentation performance by $0.71\%$, $0.05\%$ and $1.05\%$ for ADE20K,
PASCAL-Context and COCO-Stuff, respectively. Also, we count and compare the
sampling frequency of each category when using balanced deletion (BD) and
random deletion (RD) in Figure 4. We can observe that the sampling frequency
of uncommon categories have increased after utilizing BD as we expected.

Complexity Comparison. We also present analysis on complexity of our
intervention-driven context module following the settings of OCR [6]. In Table
5, it is observed that our IDRNet requires the least Parameters, FLOPS and GPU
Memory, while achieving the best mIoU on the ADE20k dataset. Moreover, our
IDRNet is complementary to other context modules. For example, when
incorporated with PPM (termed "PPM+IDRNet" in the table), the Parameters,
FLOPS, Time and GPU Memory only increase by 0.58 M, 39.78 G, 11.19 ms and
73.65 M, respectively, which shows that our proposed IDRNet is light and
efficient during network inference. Note that few parameters increase in
PPM+IDR as we adopt a shared $3\times 3$ convolution layer to reduce the
feature channels of the backbone outputs.

Qualitative Results. Figure 3 provides qualitative comparison of the proposed
approach against FCN [18], OCRNet [6], ISNet [14] and MCIBI++[21] on
representative examples in various benchmark datasets. It is observed that
IDRNet can output better segmentation results.

###  4.3 Cross Domain Segmentation Performance.

Table 6 compares the cross-domain segmentation performance. It can be observed
that IDRNet can also boost cross-domain segmentation performance, which
demonstrates the generalization of our intervention-driven pixel relation
modeling paradigm. For example, DeeplabV3+IDRNet trained on day-time domain
(_i.e._ , Cityscapes) can also bring $3.63\%$ and $1.94\%$ mIoU gains to Dark
Zurich [80] and Nighttime Driving [81] datasets which belong to night-time
domain. In addition, after training FCN+IDRNet on single-human parsing
scenarios (_i.e._ , LIP), it also boosts the segmentation performance on
multi-human parsing scenarios (_i.e._ , CIHP [82]) by $1.39\%$ mIoU.

Table 7: State-of-the-art results on ADE20K, COCO-Stuff, LIP and PASCAL-
Context dataset.

Method | Backbone | Stride | ADE20K (_train / val_) | COCO-Stuff (_train / test_) | PASCAL-Context (_train / val_) | LIP (_train / val_)
---|---|---|---|---|---|---
PSPNet [2] | ResNet-101 | $8\times$ | 43.29 | - | 47.80 | -
EMANet [83] | ResNet-101 | $8\times$ | - | 39.90 | 53.10 | -
OCNet [84] | ResNet-101 | $8\times$ | 45.45 | - | - | 54.72
ACNet [85] | ResNet-101 | $8\times$ | 45.90 | 40.10 | 54.10 | -
APCNet [42] | ResNet-101 | $8\times$ | 45.38 | - | 54.70 | -
OCRNet [6] | ResNet-101 | $8\times$ | 45.28 | 39.50 | 54.80 | 55.60
ISNet [14] | ResNet-101 | $8\times$ | 47.31 | 41.60 | - | 55.41
UPerNet+MCIBI++ [21] | ResNet-101 | $8\times$ | 47.93 | 41.84 | 56.82 | 56.32
OCRNet [6] | HRNetV2-W48 | $4\times$ | 45.66 | 40.50 | 56.20 | 56.65
SETR [17] | ViT-Large | $16\times$ | 50.28 | - | 55.83 | -
Segmenter [39] | ViT-Large | $16\times$ | 53.63 | - | 59.00 | -
UPerNet [3] | BEiT-Large | $16\times$ | 57.00 | - | - | -
UPerNet [3] | Swin-Large | $32\times$ | 53.50 | - | - | -
UPerNet+MCIBI++ [21] | Swin-Large | $32\times$ | 54.52 | 50.27 | 64.01 | 59.91
MaskFormer [54] | Swin-Large | $32\times$ | 55.60 | - | - | -
Mask2Former [11] | Swin-Large | $32\times$ | 57.30 | - | - | -
K-Net+UPerNet [55] | Swin-Large | $32\times$ | 54.30 | - | - | -
SegNeXt-L [12] | MSCAN-Large | $32\times$ | 52.10 | 47.20 | 60.30 | -
Segformer [40] | MiT-B5 | $32\times$ | 51.80 | 46.70 | - | -
UPerNet+IDRNet (_ours_) | Swin-Large | $32\times$ | 54.67 | 50.50 | 64.51 | 61.17
Mask2Former+IDRNet (_ours_) | Swin-Large | $32\times$ | 58.22 | 49.96 | 61.65 | 61.86

Table 8: Performance improvements on two popular object detection frameworks
and MS COCO benchmark dataset after leveraging deletion diagnostics to build
object relations.

Method | Backbone | Schedule | AP | AP50 | AP75 | APs | APm | APl
---|---|---|---|---|---|---|---|---
Faster R-CNN [86] | ResNet-50-FPN | $1\times$ | 37.4 | 58.1 | 40.4 | 21.2 | 41.0 | 48.1
Faster R-CNN+IDRNet (_ours_) | ResNet-50-FPN | $1\times$ | 38.2 (+0.8) | 59,7 | 41.3 | 22.0 | 42.0 | 49.2
Mask R-CNN [87] | ResNet-50-FPN | $1\times$ | 38.2 | 58.8 | 41.4 | 21.9 | 40.9 | 49.5
Mask R-CNN+IDRNet (_ours_) | ResNet-50-FPN | $1\times$ | 39.7 (+1.5) | 60.8 | 43.0 | 22.6 | 43.2 | 52.5

###  4.4 Comparison with State-of-the-art Methods

Table 7 compares the quantitative results on four challenging benchmark
datasets, including ADE20K _val_ , LIP _val_ , PASCAL-Context _val_ and COCO-
Stuff _test_. As for ADE20K _val_ , it is observed that prior to this paper,
Mask2Former [11] with Swin-Large backbone [21] achieves the state-of-the-art
performance hitting $57.30\%$ mIoU. By incorporating Mask2Former with IDRNet,
it yields $58.22\%$ mIoU which surpasses other state-of-the-art segmentation
methods. Specific to LIP _val_ , we can observe that among the previous SOTA
methods, UPerNet+MCIBI++ with Swin-Large backbone obtains the state-of-the-art
result, _i.e._ , $59.91\%$ mIoU. Our method, UPerNet+IDRNet with Swin-Large
backbone, outperforms it by $1.26\%$ and reports the new SOTA result,
$61.17\%$ mIoU. With regard to PASCAL-Context _val_ , our method,
UPerNet+IDRNet with Swin-Large backbone, surpasses all the competitors, _i.e._
, $0.50\%$ over UPerNet-MCIBI++, $5.51\%$ over Segmenter, $8.68\%$ over SETR,
$9.71\%$ over OCRNet, to name a few. On the subject of COCO-Stuff _test_ , we
can observe that our UPerNet+IDRNet with Swin-Large backbone reaches $50.50\%$
mIoU which is $0.23\%$ higher than previous state-of-the-art method
UPerNet+MCIBI++.

###  4.5 Extend Deletion Diagnostics to Object Detection

In this section, we apply deletion diagnostics mechanism to the object
detection frameworks to further demonstrate the effectiveness and
generalization of our method.

Implementation Details. Our experiments are conducted on MS COCO dataset [88].
All algorithm implementations are based on PyTorch [77] and MMDetection [89],
and we follow the default settings in MMDetection for model training and
testing.

Experimental Results. As illustrated in Table 8, the proposed deletion
diagnostics mechanism is integrated into two popular object detection
frameworks, _i.e._ , Faster R-CNN [86] and Mask R-CNN [87]. We can observe
that intervention-driven relation building for objects brings $0.8\%$ and
$1.5\%$ AP improvements for the two detection frameworks, respectively.

##  5 Conclusion

This work studies how to utilize deletion diagnostics to model pixel
relationships. The key contribution consists of two parts: (1) we simplify
pixel relation modeling as object relation modeling and (2) we introduce
deletion diagnostics to make the networks build object relationships by
perceiving changes in the outputs of the objective functions. Compared to
similarity-/multi-scale-driven context modeling paradigms, there are fewer
human priors in our context modeling paradigm. Experimental results
demonstrate the effectiveness of our proposed paradigm qualitatively and
quantitatively.

## Acknowledgement

The work described in this paper was partially supported by grants from the
National Natural Science Fund (62201483) and the Research Grants Council of
the Hong Kong Special Administrative Region, China (T45-401/22-N).

## References

  * Chen et al. [2017a] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam.  Rethinking atrous convolution for semantic image segmentation.  _arXiv preprint arXiv:1706.05587_ , 2017a.
  * Zhao et al. [2017] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia.  Pyramid scene parsing network.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2881–2890, 2017.
  * Xiao et al. [2018] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun.  Unified perceptual parsing for scene understanding.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 418–434, 2018.
  * Wang et al. [2018] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He.  Non-local neural networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 7794–7803, 2018.
  * Zhu et al. [2019a] Zhen Zhu, Mengde Xu, Song Bai, Tengteng Huang, and Xiang Bai.  Asymmetric non-local neural networks for semantic segmentation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 593–602, 2019a.
  * Yuan et al. [2020a] Yuhui Yuan, Xilin Chen, and Jingdong Wang.  Object-contextual representations for semantic segmentation.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part VI 16_ , pages 173–190. Springer, 2020a.
  * Zhou et al. [2017] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba.  Scene parsing through ade20k dataset.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 633–641, 2017.
  * Cordts et al. [2016] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The cityscapes dataset for semantic urban scene understanding.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3213–3223, 2016.
  * LeCun et al. [2015] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton.  Deep learning.  _nature_ , 521(7553):436–444, 2015.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Cheng et al. [2022] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar.  Masked-attention mask transformer for universal image segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1290–1299, 2022.
  * Guo et al. [2022] Meng-Hao Guo, Cheng-Ze Lu, Qibin Hou, Zhengning Liu, Ming-Ming Cheng, and Shi-Min Hu.  Segnext: Rethinking convolutional attention design for semantic segmentation.  _arXiv preprint arXiv:2209.08575_ , 2022.
  * Jin et al. [2021a] Zhenchao Jin, Tao Gong, Dongdong Yu, Qi Chu, Jian Wang, Changhu Wang, and Jie Shao.  Mining contextual information beyond image for semantic segmentation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 7231–7241, 2021a.
  * Jin et al. [2021b] Zhenchao Jin, Bin Liu, Qi Chu, and Nenghai Yu.  Isnet: Integrate image-level and semantic-level context for semantic segmentation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 7189–7198, 2021b.
  * Liu et al. [2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 10012–10022, 2021.
  * Wang et al. [2022a] Wenhai Wang, Jifeng Dai, Zhe Chen, Zhenhang Huang, Zhiqi Li, Xizhou Zhu, Xiaowei Hu, Tong Lu, Lewei Lu, Hongsheng Li, et al.  Internimage: Exploring large-scale vision foundation models with deformable convolutions.  _arXiv preprint arXiv:2211.05778_ , 2022a.
  * Zheng et al. [2021] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al.  Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 6881–6890, 2021.
  * Long et al. [2015] Jonathan Long, Evan Shelhamer, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3431–3440, 2015.
  * Galleguillos et al. [2008] Carolina Galleguillos, Andrew Rabinovich, and Serge Belongie.  Object categorization using co-occurrence, location and appearance.  In _2008 IEEE Conference on Computer Vision and Pattern Recognition_ , pages 1–8. IEEE, 2008.
  * Torralba et al. [2003] Antonio Torralba, Kevin P Murphy, William T Freeman, and Mark A Rubin.  Context-based vision system for place and object recognition.  In _Computer Vision, IEEE International Conference on_ , volume 2, pages 273–273. IEEE Computer Society, 2003.
  * Jin et al. [2022] Zhenchao Jin, Dongdong Yu, Zehuan Yuan, and Lequan Yu.  Mcibi++: Soft mining contextual information beyond image for semantic segmentation.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2022.
  * Cao et al. [2019] Yue Cao, Jiarui Xu, Stephen Lin, Fangyun Wei, and Han Hu.  Gcnet: Non-local networks meet squeeze-excitation networks and beyond.  In _Proceedings of the IEEE/CVF international conference on computer vision workshops_ , pages 0–0, 2019.
  * Yin et al. [2020] Minghao Yin, Zhuliang Yao, Yue Cao, Xiu Li, Zheng Zhang, Stephen Lin, and Han Hu.  Disentangled non-local neural networks.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XV 16_ , pages 191–207. Springer, 2020.
  * Dai et al. [2017] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei.  Deformable convolutional networks.  In _Proceedings of the IEEE international conference on computer vision_ , pages 764–773, 2017.
  * Zhu et al. [2019b] Xizhou Zhu, Han Hu, Stephen Lin, and Jifeng Dai.  Deformable convnets v2: More deformable, better results.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 9308–9316, 2019b.
  * Cook [1977] R Dennis Cook.  Detection of influential observation in linear regression.  _Technometrics_ , 19(1):15–18, 1977.
  * Girshick [2015] Ross Girshick.  Fast r-cnn.  In _Proceedings of the IEEE international conference on computer vision_ , pages 1440–1448, 2015.
  * Gao et al. [2019] Shang-Hua Gao, Ming-Ming Cheng, Kai Zhao, Xin-Yu Zhang, Ming-Hsuan Yang, and Philip Torr.  Res2net: A new multi-scale backbone architecture.  _IEEE transactions on pattern analysis and machine intelligence_ , 43(2):652–662, 2019.
  * Wang et al. [2020] Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, et al.  Deep high-resolution representation learning for visual recognition.  _IEEE transactions on pattern analysis and machine intelligence_ , 43(10):3349–3364, 2020.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * Xie et al. [2017] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He.  Aggregated residual transformations for deep neural networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1492–1500, 2017.
  * Chen et al. [2019a] Wuyang Chen, Xinyu Gong, Xianming Liu, Qian Zhang, Yuan Li, and Zhangyang Wang.  Fasterseg: Searching for faster real-time semantic segmentation.  _arXiv preprint arXiv:1912.10917_ , 2019a.
  * Li et al. [2020] Yanwei Li, Lin Song, Yukang Chen, Zeming Li, Xiangyu Zhang, Xingang Wang, and Jian Sun.  Learning dynamic routing for semantic segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8553–8562, 2020.
  * Liu et al. [2019] Chenxi Liu, Liang-Chieh Chen, Florian Schroff, Hartwig Adam, Wei Hua, Alan L Yuille, and Li Fei-Fei.  Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 82–92, 2019.
  * Bao et al. [2021] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei.  Beit: Bert pre-training of image transformers.  _arXiv preprint arXiv:2106.08254_ , 2021.
  * Chu et al. [2021] Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen.  Twins: Revisiting the design of spatial attention in vision transformers.  _Advances in Neural Information Processing Systems_ , 34:9355–9366, 2021.
  * Dosovitskiy et al. [2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Peng et al. [2022] Zhiliang Peng, Li Dong, Hangbo Bao, Qixiang Ye, and Furu Wei.  Beit v2: Masked image modeling with vector-quantized visual tokenizers.  _arXiv preprint arXiv:2208.06366_ , 2022.
  * Strudel et al. [2021] Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid.  Segmenter: Transformer for semantic segmentation.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 7262–7272, 2021.
  * Xie et al. [2021] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo.  Segformer: Simple and efficient design for semantic segmentation with transformers.  _Advances in Neural Information Processing Systems_ , 34:12077–12090, 2021.
  * Yu et al. [2022] Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan.  Metaformer is actually what you need for vision.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10819–10829, 2022.
  * He et al. [2019] Junjun He, Zhongying Deng, Lei Zhou, Yali Wang, and Yu Qiao.  Adaptive pyramid context network for semantic segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 7519–7528, 2019.
  * Huang et al. [2019a] Zilong Huang, Xinggang Wang, Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu.  Ccnet: Criss-cross attention for semantic segmentation.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 603–612, 2019a.
  * Yu et al. [2020] Changqian Yu, Jingbo Wang, Changxin Gao, Gang Yu, Chunhua Shen, and Nong Sang.  Context prior for scene segmentation.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 12416–12425, 2020.
  * Chen et al. [2017b] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille.  Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.  _IEEE transactions on pattern analysis and machine intelligence_ , 40(4):834–848, 2017b.
  * Chen et al. [2018] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam.  Encoder-decoder with atrous separable convolution for semantic image segmentation.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 801–818, 2018.
  * Yang et al. [2018] Maoke Yang, Kun Yu, Chi Zhang, Zhiwei Li, and Kuiyuan Yang.  Denseaspp for semantic segmentation in street scenes.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3684–3692, 2018.
  * Yu and Koltun [2015] Fisher Yu and Vladlen Koltun.  Multi-scale context aggregation by dilated convolutions.  _arXiv preprint arXiv:1511.07122_ , 2015.
  * Berman et al. [2018] Maxim Berman, Amal Rannen Triki, and Matthew B Blaschko.  The lovász-softmax loss: A tractable surrogate for the optimization of the intersection-over-union measure in neural networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 4413–4421, 2018.
  * Ke et al. [2018] Tsung-Wei Ke, Jyh-Jing Hwang, Ziwei Liu, and Stella X Yu.  Adaptive affinity fields for semantic segmentation.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 587–602, 2018.
  * Zhao et al. [2019] Shuai Zhao, Yang Wang, Zheng Yang, and Deng Cai.  Region mutual information loss for semantic segmentation.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Wang et al. [2021] Wenguan Wang, Tianfei Zhou, Fisher Yu, Jifeng Dai, Ender Konukoglu, and Luc Van Gool.  Exploring cross-image pixel contrast for semantic segmentation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 7303–7313, 2021.
  * Zhao et al. [2021] Xiangyun Zhao, Raviteja Vemulapalli, Philip Andrew Mansfield, Boqing Gong, Bradley Green, Lior Shapira, and Ying Wu.  Contrastive learning for label efficient semantic segmentation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 10623–10633, 2021.
  * Cheng et al. [2021] Bowen Cheng, Alex Schwing, and Alexander Kirillov.  Per-pixel classification is not all you need for semantic segmentation.  _Advances in Neural Information Processing Systems_ , 34:17864–17875, 2021.
  * Zhang et al. [2021] Wenwei Zhang, Jiangmiao Pang, Kai Chen, and Chen Change Loy.  K-net: Towards unified image segmentation.  _Advances in Neural Information Processing Systems_ , 34:10326–10338, 2021.
  * Bertasius et al. [2016] Gedas Bertasius, Jianbo Shi, and Lorenzo Torresani.  Semantic segmentation with boundary neural fields.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3602–3610, 2016.
  * Ding et al. [2019] Henghui Ding, Xudong Jiang, Ai Qun Liu, Nadia Magnenat Thalmann, and Gang Wang.  Boundary-aware feature propagation for scene segmentation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 6819–6829, 2019.
  * Ruan et al. [2019] Tao Ruan, Ting Liu, Zilong Huang, Yunchao Wei, Shikui Wei, and Yao Zhao.  Devil in the details: Towards accurate single and multiple human parsing.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 33, pages 4814–4821, 2019.
  * Wang et al. [2022b] Chi Wang, Yunke Zhang, Miaomiao Cui, Peiran Ren, Yin Yang, Xuansong Xie, Xian-Sheng Hua, Hujun Bao, and Weiwei Xu.  Active boundary loss for semantic segmentation.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36, pages 2397–2405, 2022b.
  * Yuan et al. [2020b] Yuhui Yuan, Jingyi Xie, Xilin Chen, and Jingdong Wang.  Segfix: Model-agnostic boundary refinement for segmentation.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XII 16_ , pages 489–506. Springer, 2020b.
  * Huang et al. [2019b] Lang Huang, Yuhui Yuan, Jianyuan Guo, Chao Zhang, Xilin Chen, and Jingdong Wang.  Interlaced sparse self-attention for semantic segmentation.  _arXiv preprint arXiv:1907.12273_ , 2019b.
  * Felzenszwalb et al. [2009] Pedro F Felzenszwalb, Ross B Girshick, David McAllester, and Deva Ramanan.  Object detection with discriminatively trained part-based models.  _IEEE transactions on pattern analysis and machine intelligence_ , 32(9):1627–1645, 2009.
  * Choi et al. [2011] Myung Jin Choi, Antonio Torralba, and Alan S Willsky.  A tree-based context model for object recognition.  _IEEE transactions on pattern analysis and machine intelligence_ , 34(2):240–252, 2011.
  * Galleguillos and Belongie [2010] Carolina Galleguillos and Serge Belongie.  Context based object categorization: A critical survey.  _Computer vision and image understanding_ , 114(6):712–722, 2010.
  * Mottaghi et al. [2014] Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille.  The role of context for object detection and semantic segmentation in the wild.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 891–898, 2014.
  * Hu et al. [2018] Han Hu, Jiayuan Gu, Zheng Zhang, Jifeng Dai, and Yichen Wei.  Relation networks for object detection.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3588–3597, 2018.
  * Xu et al. [2019] Hang Xu, Chenhan Jiang, Xiaodan Liang, and Zhenguo Li.  Spatial-aware graph relation network for large-scale object detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 9298–9307, 2019.
  * Kipf and Welling [2016] Thomas N Kipf and Max Welling.  Semi-supervised classification with graph convolutional networks.  _arXiv preprint arXiv:1609.02907_ , 2016.
  * Chen and Gupta [2017] Xinlei Chen and Abhinav Gupta.  Spatial memory for context reasoning in object detection.  In _Proceedings of the IEEE international conference on computer vision_ , pages 4086–4096, 2017.
  * Mezzadri [2006] Francesco Mezzadri.  How to generate random matrices from the classical compact groups.  _arXiv preprint math-ph/0609050_ , 2006.
  * LeCun et al. [1988] Yann LeCun, D Touresky, G Hinton, and T Sejnowski.  A theoretical framework for back-propagation.  In _Proceedings of the 1988 connectionist models summer school_ , volume 1, pages 21–28, 1988.
  * Rumelhart et al. [1986] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams.  Learning representations by back-propagating errors.  _nature_ , 323(6088):533–536, 1986.
  * Srivastava et al. [2014] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.  Dropout: a simple way to prevent neural networks from overfitting.  _The journal of machine learning research_ , 15(1):1929–1958, 2014.
  * Caesar et al. [2018] Holger Caesar, Jasper Uijlings, and Vittorio Ferrari.  Coco-stuff: Thing and stuff classes in context.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1209–1218, 2018.
  * Gong et al. [2017] Ke Gong, Xiaodan Liang, Dongyu Zhang, Xiaohui Shen, and Liang Lin.  Look into person: Self-supervised structure-sensitive learning and a new benchmark for human parsing.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 932–940, 2017.
  * Everingham et al. [2010] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes (voc) challenge.  _International journal of computer vision_ , 88(2):303–338, 2010.
  * Paszke et al. [2017] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer.  Automatic differentiation in pytorch.  2017\.
  * Jin [2023] Zhenchao Jin.  Sssegmenation: An open source supervised semantic segmentation toolbox based on pytorch.  _arXiv preprint arXiv:2305.17091_ , 2023.
  * Russakovsky et al. [2015] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.  Imagenet large scale visual recognition challenge.  _International journal of computer vision_ , 115(3):211–252, 2015.
  * Sakaridis et al. [2019] Christos Sakaridis, Dengxin Dai, and Luc Van Gool.  Guided curriculum model adaptation and uncertainty-aware evaluation for semantic nighttime image segmentation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 7374–7383, 2019.
  * Dai and Van Gool [2018] Dengxin Dai and Luc Van Gool.  Dark model adaptation: Semantic image segmentation from daytime to nighttime.  In _2018 21st International Conference on Intelligent Transportation Systems (ITSC)_ , pages 3819–3824. IEEE, 2018.
  * Gong et al. [2018] Ke Gong, Xiaodan Liang, Yicheng Li, Yimin Chen, Ming Yang, and Liang Lin.  Instance-level human parsing via part grouping network.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 770–785, 2018.
  * Li et al. [2019] Xia Li, Zhisheng Zhong, Jianlong Wu, Yibo Yang, Zhouchen Lin, and Hong Liu.  Expectation-maximization attention networks for semantic segmentation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 9167–9176, 2019.
  * Yuan et al. [2021] Yuhui Yuan, Lang Huang, Jianyuan Guo, Chao Zhang, Xilin Chen, and Jingdong Wang.  Ocnet: Object context for semantic segmentation.  _International Journal of Computer Vision_ , 129(8):2375–2398, 2021.
  * Hu et al. [2019] Xinxin Hu, Kailun Yang, Lei Fei, and Kaiwei Wang.  Acnet: Attention based network to exploit complementary features for rgbd semantic segmentation.  In _2019 IEEE International Conference on Image Processing (ICIP)_ , pages 1440–1444. IEEE, 2019.
  * Ren et al. [2015] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  _Advances in neural information processing systems_ , 28, 2015.
  * He et al. [2017] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2961–2969, 2017.
  * Lin et al. [2014] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_ , pages 740–755. Springer, 2014.
  * Chen et al. [2019b] Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, et al.  Mmdetection: Open mmlab detection toolbox and benchmark.  _arXiv preprint arXiv:1906.07155_ , 2019b.


# BERT for Long Documents: A Case Study of Automated ICD Coding

Arash Afkanpour &Shabir Adeel &Hansenclever Bassani11footnotemark: 1 Arkady
Epshteyn11footnotemark: 1 &Hongbo Fan11footnotemark: 1 &Isaac
Jones11footnotemark: 1 &Mahan Malihi11footnotemark: 1 Adrian
Nauth11footnotemark: 1 &Raj Sinha11footnotemark: 1 &Sanjana
Woonna11footnotemark: 1 &Shiva Zamani11footnotemark: 1 Elli Kanal &Mikhail
Fomitchev22footnotemark: 2 &Donny Cheung22footnotemark: 2 Google
<EMAIL_ADDRESS>Equal contributionTechnical leadership

###### Abstract

Transformer models have achieved great success across many NLP problems.
However, previous studies in automated ICD coding concluded that these models
fail to outperform some of the earlier solutions such as CNN-based models. In
this paper we challenge this conclusion. We present a simple and scalable
method to process long text with the existing transformer models such as BERT.
We show that this method significantly improves the previous results reported
for transformer models in ICD coding, and is able to outperform one of the
prominent CNN-based methods.

##  1 Introduction

The International Classification of Diseases (ICD) codes provide a standard
way of keeping track of diagnoses and procedures during a patient visit. These
codes are used worldwide for epidemiological studies, billing and
reimbursement, and research in health care. The codes are maintained by the
World Health Organization (WHO) and are revised and updated periodically. As
of 2022 the ICD codes are in the 11th revision.

Assigning ICD codes to a clinical note, such as a discharge summary, is done
by professional medical coders. Human coders require extensive training, and
the process of coding is often time-consuming, costly, and error-prone. Due to
these challenges there is an incentive to automate the coding process.
Therefore in recent years this problem has gained interest among machine
learning researchers in health care (See, Mullenbach et al. (2018); Li and Yu
(2020); Zhang et al. (2020) and references therein). On the surface, the
problem can be considered as a multi-label document classification problem.
However, there are aspects of the problem that make it particularly
challenging. The primary challenge is that there are tens of thousands of
classes. For instance, billable ICD-10-CM codes consist of approximately
73,000 codes. In addition, the distribution of the codes is not uniform. Many
of the codes are related to rare conditions and are mentioned infrequently in
text, which makes it difficult to train a reliable classifier for them.

Transformer-based language models developed based on self attention (Vaswani
et al., 2017) have become the state-of-the-art across many NLP problems by
outperforming previous solutions that were mostly based on recurrent neural
networks (RNN) and convolutional neural networks (CNN). So one would expect
that they perform well in ICD coding too. However, examining the literature of
ICD coding methods reveals that transformer-based solutions fail to outperform
CNN-based models. Many studies have applied the BERT language model (Devlin et
al., 2018) to this task, for example Pascual et al. (2021); Singh et al.
(2020); Biseda et al. (2020); Amin et al. (2019). More recently, Ji et al.
(2021) performed a comprehensive quantitative study to compare BERT and some
of its variants pre-trained on medical text against CNN-based models such as
Mullenbach et al. (2018) and Cao et al. (2020) to answer the question of
whether the magic of BERT (as observed across many NLP problems) also applies
to automated ICD coding. They concluded that BERT cannot outperform CNN-based
models in the full ICD code case.

Unlike RNN or CNN models, which in theory can process sequences of arbitrary
length, transformers’ computational complexity scales quadratically with
sequence length. This means that most of these models can handle limited size
sequences. For instance, BERT models usually are pre-trained and fine-tuned on
sequences with at most 512 tokens. Clinical notes normally contain long
snippets of text beyond the sequence limit of transformers. We hypothesize
that this constraint could explain the poor performance of transformers in
this task, and will present empirical evidence for that.

We emphasize that we do not claim to achieve state-of-the-art performance in
ICD coding, or that our design is the most efficient transformer architecture
for processing long text. For a review of efficient transformers see Tay et
al. (2020) and references therein. Our goal is to provide new empirical
evidence that shows even the standard transformer models can outperform some
of the previous prominent methods and are a viable solution for ICD coding.

##  2 Related work

Medori and Fairon (2010) applied a rule-based method to extract important
snippets of text and encode them with ICD codes. Perotte et al. (2014)
proposed SVM classification with bag-of-words features. They experimented with
both flat SVM (i.e. one classifier per code) and a hierarchical classifier.

With the success of deep learning in NLP tasks, many researchers focused on
using RNN and CNN models for ICD coding. CNN models provide a convenient way
to learn a contextual representation of text in NLP problems (Chen, 2015). For
example, Mullenbach et al. (2018) proposed the CAML model: a convolutional
layer on word2vec embedding vectors to learn a contextual representation for
each word. The word representations are combined into a class-specific
document representation using the attention mechanism. They also suggested a
method to leverage code descriptions via a regularization term. Li and Yu
(2020) proposed Multi-filter Residual CNN (MultiResCNN) that uses
convolutional layers with different kernel sizes to capture patterns with
different lengths. Additionally, they used residual blocks on top of the
convolutional layer. Similar to Mullenbach et al. (2018) they employed a per-
class attention mechanism to make the document representation attend to
different parts of the input for each code.

Recurrent neural networks (RNN) are also studied extensively for ICD coding.
Shi et al. (2017) applied LSTM at character and word level to encode both the
clinical note and the code description. Baumel et al. (2018) employs a two-
layer bidirectional Gated Recurrent Unit (GRU) model, where the first layer
encodes individual sentences, and the second layer encodes the document.

With the success of transformer architectures across many NLP tasks,
researchers focused their attention to designing such models for ICD coding.
BERT-XML (Zhang et al., 2020) with access to a large corpus of private data
managed to pre-train the model with sequence length of 1024. Most of the work
in this area, however, considered the standard BERT model and its variants
pre-trained on medical text to encode the document (Pascual et al., 2021;
Singh et al., 2020; Biseda et al., 2020; Amin et al., 2019). One observation
with these models was that they were unable to outperform CNN-based models. Ji
et al. (2021) performed a comprehensive study to answer a few research
questions on the suitability of BERT models for ICD coding. They studied and
compared different variants of BERT pre-training. They also proposed a
hierarchical attention method so that long clinical notes can be processed
with a BERT model with a limit of 512 tokens. Most importantly, they compared
different BERT variants against traditional CNN-based models, and through
extensive experiments showed that BERT-based models are not capable of
outperforming CNN-based models in ICD coding. In the next sections we show
that a simple method that enables processing of long text with transformers
will attain results that contradict the findings of Ji et al. (2021).

Figure 1: Model architecture proposed for handling long text inputs.

##  3 Method

In this section we explain our method for building a model to predict medical
codes. As illustrated in Figure 1, our model consists of an encoder that
calculates token-level representation of the input text. This can be done in
various ways, e.g. Mullenbach et al. (2018) used word2vec and a CNN layer to
calculate word-level representations. We choose the BERT language model for
this purpose. A class-specific representation of the document is then
calculated using class-specific attention vectors, similar to Mullenbach et
al. (2018). For $d$-dimensional token representations and $K$ classes, this
layer requires $d\times K$ parameters. Linear binary classifiers are built on
top of the document representation to produce the probability that the
document belongs to any of the $K$ classes. This layer requires $(d+1)\times
K$ parameters (one scalar for the offset).

Let $X=[x_{1},\ldots,x_{s}]$ denote the tokenized input sequence with $s$
tokens. Let $e_{1}(X),\ldots,e_{s}(X)$ denote the representation of tokens
$1,\ldots,s$ obtained from an encoder. That is,

$\displaystyle e_{i}(X)=\phi(x_{i}|X),\hskip 14.22636pti\in\\{1,\ldots,s\\},$

where $\phi$ is an encoder, such as BERT, that returns a context-dependent
representation for each token. For each class $c$, token-level representations
are combined into a single vector that represents the entire document using
the attention mechanism:

$\displaystyle z_{c}(X)=\sum_{i=1}^{s}\alpha_{c,i}(X)e_{i}(X),$

where

$\displaystyle\alpha_{c,i}(X)=\frac{\exp\left(\langle
e_{i}(X),q_{c}\rangle\right)}{\sum_{j=1}^{s}\exp\left(\langle
e_{j}(X),q_{c}\rangle\right)},$ (1) $\displaystyle i\in\\{1,\ldots,s\\},$

are the normalized attention coefficients and ${\langle}\cdot,\cdot{\rangle}$
denotes inner product, and $q_{c}$ is the $d$-dimensional attention vector for
class $c$. The predicted probability of the model for class $c$ is calculated
by

$\displaystyle
f_{c}(X)=\sigma\left({\langle}z_{c}(X),w_{c}{\rangle}+b_{c}\right),$

where $w_{c}$ is the weight vector for class $c$, $b_{c}$ is the scalar offset
for class $c$, and $\sigma$ is the sigmoid function.

###  3.1 Handling long text

Language models such as BERT can handle input text up to a certain length. For
example, BERT can take input of at most 512 tokens. While it is possible to
pre-train the model on longer sequences (mostly to learn useful positional
embedding vectors), memory requirement grows quadratically with input size. So
pre-training a BERT model on longer text is not scalable.

There are transformer-based models that can handle long sequences, such as
BigBird (Zaheer et al., 2020), ETC (Ainslie et al., 2020), Longformer (Beltagy
et al., 2020), and LongT5 (Guo et al., 2021). There are a few factors that
limit their usability in the medical coding task. For example, these models
are usually designed to train on TPU, so training on GPU is often a slow
process, if feasible, especially for longer sequences. Also, pre-trained
checkpoints of these models are limited, unlike the BERT models that have many
pre-trained variants including those pre-trained on medical text.

In this paper, we propose a simple idea, which enables us to use a vanilla
BERT model on long sequences. Inspired by the local attention feature of CNN
models, we propose to split the input text into (optionally overlapping)
segments of 512 tokens. These segments are passed sequentially to a BERT
model, and the token representations are concatenated to form
$[e_{1}(X),...,e_{512}(X),e_{513}(X),...,e_{1024}(X),...]$. One may argue that
a limitation of this approach is that the token representations are calculated
with a 512-token attention span. However, we have observed that in practice
this method performs well. In fact, we conjecture that in many cases short
snippets of text (as evidence) are sufficient for assigning the correct ICD
codes to the input document. Algorithm 1 shows the training procedure.

Algorithm 1 Training on a single example.

1:Input: tokenized input text of length $s$: $X=[x_{1},\ldots,x_{s}]$, sparse
binary label vector $Y=[y_{1},\ldots,y_{K}]$ for $K$ classes, where $y_{c}=1$
if the example belongs to class $c$, and $0$ otherwise.

2:Pad input text $X=[x_{1},\ldots,x_{s}]$ to length $\tau\times 512$ to obtain
$X^{\prime}=[x_{1},\ldots,x_{s},\ldots,x_{\tau\times 512}]$, where
$\tau=\lceil s/512\rceil$.

3:Split $X^{\prime}$ into segments of 512 tokens:
$S_{1}=[x_{1},\ldots,x_{512}],S_{2}=[x_{513},\ldots,x_{1024}],\ldots,S_{\tau}$.

4:Pass $S_{i}\text{'s},\hskip 2.84526pti\in\\{1,\ldots,\tau\\}$ sequentially
to the BERT module and obtain the corresponding token representations.

5:Concatenate token representations from all sequences to obtain
$[e_{1},\ldots,e_{s},\ldots,e_{\tau\times 512}]$.

6:Calculate class-specific document representations by
$z_{c}(X)=\sum_{i=1}^{s}\alpha_{c,i}(X)e_{i}(X)$, with $\alpha_{c,i}$ from Eq.
3.

7:Calculate model predictions for all classes:
$f_{c}(X)=\sigma\left({\langle}z_{c}(X),w_{c}{\rangle}+b_{c}\right),\hskip
2.84526ptc\in{1,\ldots,K}$.

8:Calculate and apply gradient updates for loss function
$\sum_{c=1}^{K}\ell(y_{c},f_{c}(X))$, where $\ell$ is binary cross-entropy.

##  4 Evaluation

We evaluate the accuracy of the proposed method with several sequence lengths
and compare it against the CAML method (Mullenbach et al., 2018), which is one
of the prominent CNN-based methods for ICD coding.

###  4.1 Data sets

For this task we chose the publicly-available MIMIC-III (Johnson et al., 2016)
and MIMIC-IV (Johnson et al., 2020) data sets. MIMIC-III is a large de-
identified data set of over 40,000 patients admitted to intensive care units
at the Beth Israel Deaconess Medical Center. The data set contains structured
and unstructured data, including lab measurements, vital signs, medications,
clinical notes, etc. Following previous studies, we focus on predicting ICD
codes for discharge summaries where each note corresponds to a hospital stay
event. MIMIC-IV is an update to MIMIC-III, which incorporates contemporary
data. It is sourced from two in-hospital database systems: a custom hospital
wide EHR and an ICU specific clinical information system.

Each discharge summary in MIMIC-III is manually coded by human coders with one
or more ICD-9 codes that specify diagnoses and procedures of that particular
stay. The data set contains 8,921 unique ICD-9 codes, including 6,918
diagnosis and 2,003 procedure codes. There are patients with multiple
admissions and therefore multiple discharge summaries. To be consistent with
the previous studies and to ensure that all of the notes of a patient are
assigned to one of train/validation/test sets we use the data split provided
by Mullenbach et al. (2018). This results in 47,724 discharge summaries for
training, 1,632 summaries and 3,372 summaries for validation and test sets
respectively.

The discharge summaries in MIMIC-IV are additionally labeled with ICD-10
codes. At the time of writing this paper the MIMIC-Note module, which contains
the discharge summaries, is not yet publicly available. In our experiments we
only consider the ICD-10 diagnosis set, which contains 72,748 codes in the
data set.

For tokenizing text we used the standard BERT vocabulary and tokenizer (Devlin
et al., 2018). Figure 2 shows the cumulative distribution function of the
number of tokens per note for MIMIC-III and MIMIC-IV.

Figure 2: Cumulative distribution function (CDF) of the number of tokens per
note for MIMIC-III (left) and MIMIC-IV (right) data sets.

###  4.2 Models

Our classification model uses a BERT language model with the method described
in Section 3. We dub this model _LongBERT_ below. The BERT checkpoint we use
in the experiments is a model with 2 transformer blocks and 256-dimensional
embedding vectors. The checkpoint can be downloaded from TensorFlow
Hub.111https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-2_H-256_A-4/2

The baseline model (_BERT-baseline_) was trained and evaluated on the first
512 tokens of input text. To measure the impact of sequence length we trained
and evaluated similar models on the first $s$ tokens of each note, with
$s\in\\{1024,2048,4096,8192\\}$. All BERT parameters and the additional
attention and classification parameters were fine-tuned during training. We
used the Adam optimizer (Kingma and Ba, 2014) with a learning rate of 2e-4.
The batch size was set to 4 in all experiments, except for the models trained
with the sequence length of 8192 which were trained with the batch size of 2
to avoid running out of memory. The models were trained for 1 million steps
(each step is one batch). No hyper-parameter tuning was performed except for
the number of training steps. The best model corresponds to the training step
that achieves the highest validation micro F1 score.

We compare these models against a CAML model trained on sequences of 2500
words following Mullenbach et al. (2018). The hyper-parameters were set
according to the optimal values obtained in Mullenbach et al. (2018). Training
was performed for 1 million steps, and the best model was selected according
to validation micro F1 score.

Following previous work, in the MIMIC-III experiments, training and evaluation
was performed on the full ICD-9 label set as well as the 50 most frequent
codes. In the MIMIC-IV experiment, we consider only the ICD-10 diagnosis
codes. Each ICD code has its own attention and classification weight vectors
in the models. Table 1 breaks down the number of parameters of the models in
the experiments.

| MIMIC-III full | MIMIC-III top 50 | MIMIC-IV diagnosis
---|---|---|---
Language model | 9,591,040 | 9,591,040 | 9,591,040
Attention layer | 2,283,776 | 12,800 | 18,623,488
Classification layer | 2,292,697 | 12,850 | 18,696,236
Total | 14,167,513 | 9,616,690 | 46,910,764
Table 1: Breakdown of the number of parameters of BERT-baseline and LongBERT
with 2 transformer blocks and 256-dimensional embedding vectors. MIMIC-III
full contains 8,921 classes, and MIMIC-IV diagnosis contains 72,748 classes.

###  4.3 Evaluation metrics

Our primary evaluation metric is micro-averaged F1 (micro F1 for short).
Micro-averaged values are calculated by treating each code as a (binary) label
for each note. That is, each (note, code) pair is counted as one instance for
calculating the metrics. Let,

micro precision $\displaystyle=$
$\displaystyle\frac{\sum_{x,c}TP(x,c)}{\sum_{x,c}TP(x,c)+FP(x,c)},$ micro
recall $\displaystyle=$
$\displaystyle\frac{\sum_{x,c}TP(x,c)}{\sum_{x,c}TP(x,c)+FN(x,c)},$

where $TP(x,c)=1$ if class $c$ is a true positive prediction for note $x$ and
0 otherwise. $FP(x,c)$ (false positive) and $FN(x,c)$ (false negative) are
defined analogously. Finally, micro F1 is the harmonic mean of micro precision
and micro recall:

micro F1 $\displaystyle=$ $\displaystyle 2\hskip 5.69054pt\frac{\text{micro
precision}\times\text{micro recall}}{\text{micro precision}+\text{micro
recall}}.$

The optimal threshold on model predictions, which is used to calculate
$TP/FP/FN$ counts, is obtained by a grid search to maximize the validation set
F1 score.

Additionally we report precision-recall AUC (PR-AUC), and ROC-AUC. In contrast
to F1 score, these metrics are independent of a specific operating point and
provide an aggregated view of model accuracy.

###  4.4 Results

Table 2 shows the results of the LongBERT and CAML models on the MIMIC-III
full-code test set. Table 3 shows accuracy metrics obtained on the MIMIC-IV
diagnosis code data set. Bold numbers represent the best value of each metric.
A clear trend observed in both data sets is that as the sequence length of
LongBERT increases, the accuracy of the model improves. These results
demonstrate that the capability to process long text is critical in achieving
high accuracy.

The LongBERT models with sequence lengths of 4096 and 8192 both outperform the
CAML model. This finding contradicts the previous finding of Ji et al. (2021).
While their hierarchical attention proposal and our method both handle long
text by breaking it into segments of 512 tokens, one key difference is that
they use the CLS token representation from each segment, whereas we use
individual token representations. The best MIMIC-III full-code performance
reported in Ji et al. (2021) was F1 = 0.47 with BioBERT full-text (Lee et al.,
2020) checkpoint and hierarchical attention, while our small vanilla BERT
model with sequence length of 8192 achieves F1 = 0.5680. These results show
that with a proper modeling approach transformer-based models are indeed
capable of outperforming CNN-based models in ICD coding.

| Seq. length | Micro F1 | Precision | Recall | PR-AUC | ROC-AUC
---|---|---|---|---|---|---
CAML | 2500 (words) | 0.5465 | 0.5973 | 0.5036 | 0.5361 | 0.9831
BioBERT full-text (Ji et al., 2021) | entire note | 0.470 | N/A | N/A | N/A | 0.974
BERT-baseline | 512 | 0.4149 | 0.4769 | 0.3672 | 0.3793 | 0.9745
LongBERT | 1024 | 0.4697 | 0.5421 | 0.4144 | 0.4309 | 0.9766
LongBERT | 2048 | 0.5036 | 0.5777 | 0.4463 | 0.4703 | 0.9794
LongBERT | 4096 | 0.5514 | 0.6038 | 0.5074 | 0.5305 | 0.9820
LongBERT | 8192 | 0.5680 | 0.6148 | 0.5278 | 0.5402 | 0.9827
Table 2: Accuracy metrics in the MIMIC-III full-code experiment. | Seq. length | Micro F1 | Precision | Recall | PR-AUC | ROC-AUC
---|---|---|---|---|---|---
CAML | 2500 (words) | 0.5439 | 0.5739 | 0.5169 | 0.5313 | 0.9889
BERT-baseline | 512 | 0.4010 | 0.4298 | 0.3757 | 0.3580 | 0.9883
LongBERT | 1024 | 0.4607 | 0.5094 | 0.4205 | 0.4254 | 0.9839
LongBERT | 2048 | 0.4852 | 0.5268 | 0.4497 | 0.4559 | 0.9852
LongBERT | 4096 | 0.5635 | 0.5925 | 0.5371 | 0.5450 | 0.9850
LongBERT | 8192 | 0.5703 | 0.6046 | 0.5397 | 0.5517 | 0.9871
Table 3: Accuracy metrics in the MIMIC-IV diagnosis experiment.

#### MIMIC-III top 50.

Following previous work, we also trained and evaluated the models on the
MIMIC-III 50 most frequent codes. Table 4 shows the results. Similar to the
full-code case we observe that processing longer segments results in higher
accuracy.

In this case, however, there is no clear winner between LongBERT and CAML.
While LongBERT achieves a higher micro F1 score, the CAML model has a higher
PR-AUC. We conjecture that the smaller performance difference between the two
models in this experiment compared to the full-code experiment is due to the
amount of information in the data sets. By removing many of the labels in the
top-50 experiment we essentially remove information. This information is more
helpful to larger models (i.e. transformers) than smaller models, such as
CAML. As a result, we observe a larger performance gap in the full-code
experiment between LongBERT and CAML.

We also note that the accuracy numbers of the CAML model in this experiment
are higher than those reported in Mullenbach et al. (2018). One difference
here is that we do not discard notes that aren’t assigned any of the top 50
codes as was done in the original paper. Such notes are used as negative
examples for the top 50 codes. Therefore our data set contains more negative
examples than the data set used in Mullenbach et al. (2018).

| Seq. length | Micro F1 | Precision | Recall | PR-AUC | ROC-AUC
---|---|---|---|---|---|---
CAML | 2500 (words) | 0.6390 | 0.6506 | 0.6278 | 0.6410 | 0.9102
BERT-baseline | 512 | 0.5027 | 0.5367 | 0.4727 | 0.5117 | 0.8360
LongBERT | 1024 | 0.5568 | 0.5923 | 0.5252 | 0.5406 | 0.8560
LongBERT | 2048 | 0.5908 | 0.5987 | 0.5832 | 0.5604 | 0.8834
LongBERT | 4096 | 0.6375 | 0.6157 | 0.6609 | 0.6229 | 0.9115
LongBERT | 8192 | 0.6522 | 0.6417 | 0.6629 | 0.6303 | 0.9181
Table 4: Accuracy metrics in the MIMIC-III top-50 experiment.

##  5 Discussion

Most of the existing BERT models pre-trained on generic or medical text can
take input segments of up to 512 tokens. Clinical notes, however, are much
longer than this limit. To deal with this limitation, much of the existing
works in automated ICD coding that use BERT limit the input to the model by
truncating the text or selecting specific spans of text. This results in loss
of information and poor performance.

In this paper we proposed a simple method to apply BERT models to sequences
longer than 512 tokens. Our method is simple and consists of two key
components: (i) apply BERT sequentially to (optionally overlapping) segments
of 512 tokens, and (ii) concatenate token-level representations from all
segments, and combine them using a class-specific attention layer.

We demonstrated that processing long text sequences minimizes information loss
and is critical for achieving high performance in automated ICD coding. We
also showed that contrary to previous findings, this method with even a small
vanilla BERT model outperforms CNN-based methods, and achieves competitive
performance.

Future steps include evaluating medical variants of BERT, and exploring other
transformer-based architectures that were designed to handle long sequences.

## Limitations

While our method enables the processing of text longer than 512 tokens, one of
the limitations of this approach is that context-dependent token
representations are still calculated using a window of 512 tokens. Despite
good performance of this method in practice, there could be cases where a
context window of longer than 512 must be used to make accurate predictions.

Furthermore, while our method reduces computational complexity from quadratic
(in sequence length) to linear, the memory requirement of the model could
still be prohibitive in certain cases. For instance, for sequence length of
8192, and a small BERT checkpoint with only two transformer blocks we had to
reduce batch size to 2 in order to train the models. Using a larger BERT
checkpoint for long sequences requires more memory and multiple GPUs, which
increases the cost of compute.

## References

  * Ainslie et al. (2020) Joshua Ainslie, Santiago Ontanon, Chris Alberti, Vaclav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, and Li Yang. 2020.  Etc: Encoding long and structured inputs in transformers.  _arXiv preprint arXiv:2004.08483_.
  * Amin et al. (2019) Saadullah Amin, Günter Neumann, Katherine Dunfield, Anna Vechkaeva, Kathryn Annette Chapman, and Morgan Kelly Wixted. 2019.  Mlt-dfki at clef ehealth 2019: Multi-label classification of icd-10 codes with bert.  In _CLEF (Working Notes)_ , pages 1–15.
  * Baumel et al. (2018) Tal Baumel, Jumana Nassour-Kassis, Raphael Cohen, Michael Elhadad, and Noémie Elhadad. 2018.  Multi-label classification of patient notes: case study on icd code assignment.  In _Workshops at the thirty-second AAAI conference on artificial intelligence_.
  * Beltagy et al. (2020) Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020.  Longformer: The long-document transformer.  _arXiv preprint arXiv:2004.05150_.
  * Biseda et al. (2020) Brent Biseda, Gaurav Desai, Haifeng Lin, and Anish Philip. 2020.  Prediction of icd codes with clinical bert embeddings and text augmentation with label balancing using mimic-iii.  _arXiv preprint arXiv:2008.10492_.
  * Cao et al. (2020) Pengfei Cao, Yubo Chen, Kang Liu, Jun Zhao, Shengping Liu, and Weifeng Chong. 2020\.  Hypercore: Hyperbolic and co-graph representation for automatic icd coding.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 3105–3114.
  * Chen (2015) Yahui Chen. 2015.  Convolutional neural network for sentence classification.  Master’s thesis, University of Waterloo.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Guo et al. (2021) Mandy Guo, Joshua Ainslie, David Uthus, Santiago Ontanon, Jianmo Ni, Yun-Hsuan Sung, and Yinfei Yang. 2021.  Longt5: Efficient text-to-text transformer for long sequences.  _arXiv preprint arXiv:2112.07916_.
  * Ji et al. (2021) Shaoxiong Ji, Matti Hölttä, and Pekka Marttinen. 2021.  Does the magic of bert apply to medical code assignment? a quantitative study.  _Computers in Biology and Medicine_ , 139:104998.
  * Johnson et al. (2020) Alistair Johnson, Lucas Bulgarelli, Tom Pollard, Steven Horng, Leo Anthony Celi, and Roger Mark. 2020.  Mimic-iv.  _version 0.4). PhysioNet. https://doi. org/10.13026/a3wn-hq05_.
  * Johnson et al. (2016) Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. 2016.  Mimic-iii, a freely accessible critical care database.  _Scientific data_ , 3(1):1–9.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_.
  * Lee et al. (2020) Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020.  Biobert: a pre-trained biomedical language representation model for biomedical text mining.  _Bioinformatics_ , 36(4):1234–1240.
  * Li and Yu (2020) Fei Li and Hong Yu. 2020.  Icd coding from clinical text using multi-filter residual convolutional neural network.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34(05), pages 8180–8187.
  * Medori and Fairon (2010) Julia Medori and Cédrick Fairon. 2010.  Machine learning and features selection for semi-automatic icd-9-cm encoding.  In _Proceedings of the NAACL HLT 2010 Second Louhi Workshop on Text and Data Mining of Health Documents_ , pages 84–89.
  * Mullenbach et al. (2018) James Mullenbach, Sarah Wiegreffe, Jon Duke, Jimeng Sun, and Jacob Eisenstein. 2018\.  Explainable prediction of medical codes from clinical text.  _arXiv preprint arXiv:1802.05695_.
  * Pascual et al. (2021) Damian Pascual, Sandro Luck, and Roger Wattenhofer. 2021.  Towards bert-based automatic icd coding: Limitations and opportunities.  _arXiv preprint arXiv:2104.06709_.
  * Perotte et al. (2014) Adler Perotte, Rimma Pivovarov, Karthik Natarajan, Nicole Weiskopf, Frank Wood, and Noémie Elhadad. 2014.  Diagnosis code assignment: models and evaluation metrics.  _Journal of the American Medical Informatics Association_ , 21(2):231–237.
  * Shi et al. (2017) Haoran Shi, Pengtao Xie, Zhiting Hu, Ming Zhang, and Eric P Xing. 2017.  Towards automated icd coding using deep learning.  _arXiv preprint arXiv:1711.04075_.
  * Singh et al. (2020) AK Singh, Mounika Guntu, Ananth Reddy Bhimireddy, Judy W Gichoya, and Saptarshi Purkayastha. 2020.  Multi-label natural language processing to identify diagnosis and procedure codes from mimic-iii inpatient notes.  _arXiv preprint arXiv:2003.07507_.
  * Tay et al. (2020) Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. 2020.  Efficient transformers: A survey.  _ACM Computing Surveys (CSUR)_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Zaheer et al. (2020) Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020.  Big bird: Transformers for longer sequences.  _Advances in Neural Information Processing Systems_ , 33:17283–17297.
  * Zhang et al. (2020) Zachariah Zhang, Jingshu Liu, and Narges Razavian. 2020.  Bert-xml: Large scale automated icd coding using bert pretraining.  _arXiv preprint arXiv:2006.03685_.