# W-Net: A Two-Stage Convolutional Network for Nucleus Detection in
Histopathology Image

Anyu Mao1, Jialun Wu1, Xinrui Bao2, Zeyu Gao1, Tieliang Gong1, and Chen Li1,∗
1School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an,
China
National Engineering Lab for Big Data Analytics, Xi’an Jiaotong University,
Xi’an, China  2School of Automation Science and Engineering, Xi’an Jiaotong
University, Xi’an, China
Email<EMAIL_ADDRESS>
###### Abstract

Pathological diagnosis is the gold standard for cancer diagnosis, but it is
labor-intensive, in which tasks such as cell detection, classification, and
counting are particularly prominent. A common solution for automating these
tasks is using nucleus segmentation technology. However, it is hard to train a
robust nucleus segmentation model, due to several challenging problems, i.e.,
the nucleus adhesion, stacking, and excessive fusion with the background.
Recently, some researchers proposed a series of automatic nucleus segmentation
methods based on point annotation, which can significant improve the model
performance. Nevertheless, the point annotation needs to be marked by
experienced pathologists. In order to take advantage of segmentation methods
based on point annotation, further alleviate the manual workload, and make
cancer diagnosis more efficient and accurate, it is necessary to develop an
automatic nucleus detection algorithm, which can automatically and efficiently
locate the position of the nucleus in the pathological image and extract
valuable information for pathologists. In this paper, we propose a W-shaped
network for automatic nucleus detection. Different from the traditional U-Net
based method, mapping the original pathology image to the target mask
directly, our proposed method split the detection task into two sub-tasks. The
first sub-task maps the original pathology image to the binary mask, then the
binary mask is mapped to the density mask in the second sub-task. After the
task is split, the task’s difficulty is significantly reduced, and the
network’s overall performance is improved. In order to evaluate the model
efficiency in system application, we integrate the automatic nucleus detection
model into the digital pathology image analysis system, Pathology Information
Management & Integration Platform (PIMIP). Our proposed network can automatic
identify the center of each nucleus. Combined with the NuClick, a semi-
supervised recognition model based on point annotation, we implement a fully
automatic nucleus annotation framework.

##  I Introduction

Body fluid examination, imaging examination, and endoscopy can show the
abnormalities inside the body, but whether the cancer is diagnosed must
undergo pathological biopsy. Pathological diagnosis can further determine the
cancer stage, and intuitively show the curative effect in the follow-up
treatment process.

Pathological diagnosis is labor-intensive in which tasks such as nucleus
counting and classification statistics are particularly prominent. These tasks
need to determine whether the nucleus exists and where it exists, so it is
necessary to do cell detection to locate the center of each nucleus, which is
the basis for further pathological diagnosis.

The resolution of a complete digital pathological image is extremely large. In
the image, the number of nuclei is enormous, but the shape of each nucleus is
tiny. If we only rely on pathologists’ manual annotation, it will be highly
repetitive to pathologists and take up time and energy for follow-up
observation and analysis of pathological tissues, and increase the pressure on
pathologists.

There is a noticeable shortage of professional pathologists worldwide, which
is not enough to support the pathological diagnosis needs of all humanity.
Although the number of pathologists is increasing year by year, it still can
not be proportional to the number of patients.

The automatic nucleus detection algorithm based on artificial intelligence can
rely on the machine to complete the locating of the center of each nucleus in
digital pathological images and further complete basic tasks such as counting
nuclei. So it can replace the highly repetitive manual work and efficiently
extract information valuable to doctors from digital pathological images.
Pathologists can save more time and energy to analyze more detailed
pathological information such as cell morphology.

In addition, recently, some researchers proposed a series of automatic nucleus
segmentation methods based on point annotation, which can significant improve
the model performance. But it requires more sophisticated nucleus detection,
that is, locating the center of each nucleus. Nevertheless, the point
annotation needs to be marked by experienced pathologists. In order to take
advantage of segmentation methods based on point annotation, further alleviate
the manual workload, and make cancer diagnosis more efficient and accurate, it
is necessary to develop an automatic nucleus detection algorithm, which can
automatically and efficiently locate the center of each nucleus in
pathological images for pathologists.

In pathology, there has been a method for coloring cells in tissues, that is
Hematoxylin-Eosin Staining (H&E). Current detection methods for nuclei in H&E
stained images are mostly based on low-level manual features, such as color,
edge, and texture. Traditional algorithms or machine learning methods based on
such features include local threshold [1] and morphological calculation [2] ,
region growing [3, 4], active contour model [5], level set [6], watershed
algorithm [7], k-means clustering [8], etc. However, for pathological images
with numerous nuclei, the time cost of using the above methods is too high.
For nuclei with irregular edges, methods based on edge features have
limitations, while for images with complex backgrounds, methods based on color
and texture have drawbacks.

With the rapid development of deep learning methods, a common solution for
locating nucleus is using nucleus segmentation technology. The nucleus
segmentation technology is challenging to train a robust model, and it is
difficult to solve the problems of nucleus adhesion, stacking, and excessive
fusion with the background. As for data preparation, the nucleus segmentation
technology requires accurate nuclear boundaries, which requires high
professionalism. However, the nucleus detection technology is easy to prepare
a large amount of data, and it is easy to distinguish stacked nuclei by
locating the center of each nucleus. Besides, automatic nucleus segmentation
methods based on point annotation need nucleus detection to provide the center
of each nucleus as necessary signals.

In order to solve the above problems, we have developed an automatic nucleus
detection model based on deep learning and integrated it into the digital
pathology analysis system, Pathology Information Management & Integration
Platform (PIMIP). Combined with the model named NuClick [9] which is an
automatic nucleus segmentation method based on point annotation, we build a
complete automatic nucleus annotation framework. It can assist pathologists in
their work, relieve their burden, and advance the digitization of pathology.

The main contributions of this paper are as follows:

  * •

Design of nucleus detection model: We have made adaptive processing of the
renal cell carcinoma data set. According to the characteristics of renal
cancer cell data, we improved the U-Net based model specifically the feature
extraction ability of the encoding part. We further split the detection task
into two sub-tasks. The first sub-task maps the original pathology image to
the binary mask, then the binary mask is mapped to the density mask in the
second sub-task, so we have proposed W-Net.

  * •

System integrated development: In order to verify the actual effect of our
model, we integrated the designed automatic nucleus detection model into the
digital pathology analysis system PIMIP to provide a specific nuclear center
for the automatic nucleus segmentation method based on point annotation and
realize a fully automatic nucleus annotation framework.

##  II Related work

In this chapter, from the perspective of network structure, we classify the
existing automatic nucleus detection methods and analyze the advantages and
disadvantages of various methods. Additionally, we introduce a pathological
image analysis system PIMIP and the NuClick, a semi-supervised nucleus
segmentation model based on point annotation, which is used in PIMIP.

###  II-A Automatic nucleus detection methods

There are many nucleus detection methods based on deep learning including
probability prediction models and image segmentation models. The above two
types of methods both use Convolutional Neural Network(CNN) to extract the
features of images, and then process the features to obtain final results. The
subsequent operations on the features distinguish two types of network
structures. One type is a network structure with only encoder, another is an
encoder-decoder mode network. In addition, these two structures have a clear
distinction between the form of the input and the efficiency of detection.

####  II-A1 Tiles analysis based on CNN downsampling

This type of models includes Structured Regression CNN (SR-CNN) [10],
Spatially Constrained CNN (SC-CNN) [11], Shape Prior CNN(SP-CNN) and Tunable
Shape Prior CNN (TSP-CNN) [12] , etc.

The structure of SR-CNN and SC-CNN is similar. They use CNN downsampling to
obtain features and then connect to fully connected layers. The input required
by the above two methods is a tile of a complete slide image which is cut by a
sliding window with fixed size. SP-CNN and TSP-CNN are detection methods based
on the prior shape of each nucleus, which introduce prior knowledge of the
shape of each nucleus to do the detection. Although the input data required by
these two methods in the detection part is a complete pathological image, the
correction processing part requires a sliding window to extract tiles from the
pathological image and compare it with prior knowledge.

These methods predict the nucleus by predicting the probability of pixels.
However, they adopt the form of a sliding window during data input, which is
extremely inefficient for large-resolution digital pathological images. Among
them, the setting of the sliding window’s size needs to consider factors such
as the size and number of nuclei, which requires professional opinions. The
data preparation for prior information of the nuclear shape is difficult. In
addition, the predicted value of any pixel will be affected by multiple
blocks, which will have a negative impact on accuracy.

####  II-A2 Full-image prediction method based on encoder-decoder model

This type of models includes Stacked Sparse Auto-encoder (SSAE) [13], Fully
Residual Convolutional Neural Network (FRCN) [14] and U-Net [15].

SSAE is composed of multiple layers of basic auto-encoders. The features are
normalized by softmax, and finally are restored to the original size through a
symmetrical decoder. In this method, whether the predicted value of the entire
block is close to 1 is used to determine whether the block is a block with or
without a nucleus. So it requires the input to be a picture extracted from a
large-resolution pathological image, and there is a strict requirement that
the block contains at most one nucleus. It makes the model training
inefficient.

FRCN uses residual blocks for downsampling, and uses short-hop connections to
avoid the problem of gradient disappearance, so that the model can learn
features better. FRCN and FRCN with structured regression [16] are both
probability prediction methods. The output is predicted values of each pixel
in the whole image. The higher the value, the closer to the center of each
nucleus. So its truth ground is a density mask and input can be whole
pathological images. However, only when the distinction between nuclei and the
background is obvious, the center of each nucleus can be highlighted. For
pathological images with more complicated backgrounds, the performance of this
model is poor.

In the medical image segmentation task, one of the best performing methods is
U-Net [17], which uses an encoder-decoder structure and a long-hop connection
between the encoder part and the decoder part, fusing low-level features with
high-level features. The input can be a complex pathological image, and the
output is a mask that distinguishes the target area from the background, which
can also meet the requirements of nucleus detection. U-Net can segment nuclei
from images with complex backgrounds, but the drawback is that the center of
each nucleus is not highlighted. Adhered or stacked nuclei may be mistakenly
detected as the same nucleus.

In summary, the goal of our work is to directly predict the center of each
nucleus from pathological images with complex backgrounds, so the encoder-
decoder structure can meet our needs. We hope to take advantage of FRCN and
U-Net, that is, nuclei can be detected from complex images, and the center of
each nucleus can be highlighted to avoid problems such as missed detection and
wrong detection.

###  II-B Pathology image analysis system

PIMIP is a digital pathology image analysis system of our team. The goal of
the system is to build a fully automatic digital pathology image annotation
framework, but the existing technology still has limitations.

The system relies on NuClick [18], a semi-supervised nucleus segmentation
model based on point annotations to draw nuclear masks. Previous methods
require a bounding box [19] or at least four annotations [20] as signals to
segment nuclei and draw the mask. NuClick only needs one point annotation that
is the center of each nucleus when segmenting the nucleus and drawing the
corresponding mask.

The point annotation needs to be manually annotated in the system, which is a
task with extremely high time cost and labor cost. In addition, there may be
problems such as excessive position deviation and manual omission. In order
for the framework of nucleus annotation to be more automatic, more efficient,
and more accurate, it is necessary to convert manual annotations into
automatic nucleus detection based on deep learning models.

The automatic nucleus detection model designed in this paper should be
integrated into PIMIP, which returns the coordinates of nuclear centers, and
provides it to NuClick model as guide signals to draw nuclear masks. Finally,
we can build a complete automatic nuclear annotation framework.

##  III Methods

We design the model based on the encoder-decoder structure. We improve the
U-Net based model specifically the feature extraction ability of the encoding
part from a multi-dimensional perspective. In order to reduce the difficulty
of the detection task and make the result more accurate, we split the task
into two sub-tasks by adding an intermediate prediction step. We propose
W-shaped network(W-Net) to improve the overall performance of the model.

Figure 1: W-Net. Our proposed method W-Net splits the detection task into two
sub-tasks. The first sub-task maps the original pathology image to the binary
mask, then the binary mask is mapped to the density mask in the second sub-
task. In two sub-tasks, we both use encoder-decoder structure. For the encoder
of the first sub-task, we use EfficientNet-B7 as its backbone. When extracting
the features of the binary mask, the background has been distinguished from
the nucleus, so a simpler structure of encoder can be used. We finally choose
EfficientNet-B0 for the second sub-task.

###  III-A Improve the encoder of U-Net

U-Net can be applied to nucleus detection. We only need to re-plan the
features we want the network to extract. In short, it is to change the form of
the label to meet the goals. However, the encoder used by the U-Net is simple.
Due to the limitation of the depth, the degree of extracted features is
limited, resulting in low recall rate. Many non-existent nuclei will be
detected. In addition, the simple encoder cannot eliminate the negative impact
caused by adhered and stacked nuclei.

Therefore, we improve the encoder by using a pre-trained network as its
backbone. At the same time, the decoder part is modified to restore the
features to images with original size based on the principle of symmetry. The
basic decoding block is a deconvolution layer connecting two convolution
layers and an activation layer, which is shown in Fig. 1. The number of
decoding blocks is adjusted according to the encoder to restore the size of
the images symmetrically. The deconvolution mode is bilinear interpolation.
Long-hop connections are still used to integrate low-level features and high-
level features. The following are two frameworks used as the encoder backbone.

####  III-A1 Deep Residual Network (ResNet)

The depth of the network is crucial to the performance of the model. When the
number of network layers is increased, the network can extract more complex
features and theoretically achieve better results. But the deep network will
have degradation problems. The residual block can effectively solve the above
problems. Therefore, the U-Net with ResNet [21] is better in performance of
detection. There are experiments using ResNet-34 as the U-Net encoder
backbone, while 50, 101, and 152 are not widely used, so we focus on these
four kinds of ResNet to find the most suitable number of layers.

####  III-A2 EfficientNet

ResNet improves accuracy by increasing the depth of the network, but it
affects the speed of model. Besides, only considering the depth of the network
is too simple, which will cause the improvement of network performance to fall
into a bottleneck. EfficientNet explores a more suitable combination of
network depth, width and image resolution, and has a good trade-off between
speed and accuracy [22]. Compared with ResNet, EfficientNet not only improves
the speed and reduces the amount of parameters to a certain extent, but also
maintains accuracy and even improves the performance of the network when the
amount of calculation is equivalent.

Therefore, we use EfficientNet as the encoder backbone to construct a U-Net
with EfficientNet. We try EfficientNet-B5, B6, and B7 which is different in
the zoom levels in multiple dimensions to find out the most suitable
structure.

###  III-B W-Net

U-Net with EfficientNet mentioned above greatly improves the ability of
feature extraction of the encoder, which reduces the problems of over
detection and error detection. The predicted results basically distinguish
each nucleus, and the nuclei with a lighter degree of adhesion can be
distinguished. But the stacked nuclei are not distinguished. Areas where
nuclear color is not highly distinguished from background’s color are even
more difficult to highlight the center of each nucleus.

Analyzing the characteristics of inputs, it can be found that the background
of some images is similar to the color of the nucleus. As shown in Fig.2 (a),
some of nuclei can almost blend with the background. Once the stacked nuclei
are fused with the background, it is difficult to determine where the highest
value of the nucleus center pixel starts. Therefore, it is very difficult to
determine the center coordinates in such case.

So the method of directly predicting centers of each nuclei from the
pathological image with complex backgrounds makes detection task difficult.
Therefore, we split the detection task into two sub-tasks. The first sub-task
maps the original pathology image to the binary mask that distinguishes the
background and nuclei, then the binary mask is mapped to the density mask in
the second sub-task that highlights the center of each nucleus. In this way,
when predicting the center of each nucleus, the stacked nuclei will not be
affected by excessive fusion with the background, and the task of highlighting
the center of each nucleus is greatly simplified.

Figure 2: The result of the data processing. (a) Original image. (b) Solid
dots. (c) Density mask.

In summary, our proposed method W-Net splits the detection task into two sub-
tasks. After the task is split, the task’s difficulty is significantly
reduced, and the network’s overall performance is improved. The W-Net
architecture is shown in Fig. 1.

In two sub-tasks, we both use encoder-decoder structure. When extracting
features from original images, the encoder with better ability of feature
extraction is used. We find that the U-Net with EfficientNet has better
performance. Therefore, for the encoder of the first sub-task, we use
EfficientNet as its backbone. When extracting the features of the binary mask,
the background has been distinguished from nuclei, so a simpler structure of
encoder can be used. Different choices and comparisons of encoders for the
second sub-task are introduced as follows.

####  III-B1 Basic encoder

U-Net cannot distinguish the adhered or stacked nuclei, but it can perform
better in binary masks that distinguish the background and nuclei, and
highlight the center of each nucleus.

####  III-B2 VGGNet

Since the basic encoder does not have pre-trained parameters, there is
randomness in the training process, and there are still limitations for
extracting the features. Among the pre-trained models with a small number of
layers, there is VGGNet [23] that is commonly used to extract image features.
VGGNet has a simple structure and uses a small convolution kernel instead of a
large convolution kernel. A small convolution kernel means more activation
functions, richer features, and stronger discrimination capabilities. The
small convolution kernel also reduces the parameters of the convolution layer.
Therefore, in the second sub-task, we use VGGNet to improve the performance of
network. VGGNet-13 and VGGNet-16 are tried as the backbone of the encoder for
the second sub-task.

####  III-B3 EfficientNet with different zoom levels

EfficientNet comprehensively considers the performance of the network from the
three dimensions of network depth, network width, and image resolution. U-Net
with EfficientNet performs well in detection. Binary masks eliminate the
negative impact of excessive fusion of the background and nuclei, which
greatly reduce the difficulty of predicting the center of each nucleus. It
will better utilize EfficientNet’s ability. We use EfficientNet-B0 as the
backbone of the encoder for the second sub-task.

##  IV Result

We compare the performance between U-Net, U-Net with ResNet, U-Net with
EfficientNet and W-Net. The data set used is annotated by our team.

###  IV-A Renal clear cell carcinoma data set

The data set used is named clear cell Renal Cell Carcinoma Nuclei Grading Data
set (ccRCC), which is composed of 1000 H&E stained tiles with a resolution of
512 pixels × 512 pixels. The labels in the ccRCC contain the coordinates of
centers of nuclei. There is a problem that if the label is only a single-point
pixel in the center of the nucleus, it will make the convolutional network
unable to extract features. Therefore, it is necessary to process the data.
The specific methods are as follows.

####  IV-A1 Solid dots

The center of each nucleus occupies one pixel, and it is marked as 1. The
proportion of single-point pixels is too small compared to the proportion of
background pixels marked as 0, which makes the encoder difficult to extract
features that the center of each nucleus is marked as 1. The output is close
to a whole black image and nuclei are almost undetectable.

Therefore, we expand the influence of the center of each nucleus and increase
the proportion of pixels marked as 1. We consider the shape of a dot as the
prior shape of the center of each nucleus, and describe the center of each
nucleus as a dot with a radius of 2 pixels, which is shown in Fig.2 (b).

TABLE I: The comparison of U-Net with different encoders Methods | P | R | F1
---|---|---|---
U-Net | 0.78 | 0.80 | 0.78
U-Res-Net | 0.81 | 0.84 | 0.82
U-E-Net | 0.82 | 0.87 | 0.84
TABLE II: The comparison of U-Net and W-Net  Methods | P | R | F1
---|---|---|---
U-Net | 0.78 | 0.80 | 0.78
W-UU-Net | 0.81 | 0.79 | 0.80
W-EU-Net | 0.82 | 0.84 | 0.83
W-EV-Net | 0.82 | 0.86 | 0.84
Proposed Method | 0.83 | 0.88 | 0.85

####  IV-A2 Density mask

Although the solid dots mask enlarges the influence of the center of each
nucleus, it also spreads the center of the nucleus, leading to the problem of
predicted center shift. Therefore, we need to highlight the center points. The
labels of breast cancer nucleus data set (BCData) [24] are generated based on
the method of the crowd counting model CSRNet [25]. It uses Gaussian kernel
convolution, so that the central pixel points are spread around, but maintain
highest. However, the pixel values of the image become extremely small,
resulting in the network not being able to extract features. Therefore,
considering the density of nuclei, the specific range of diffusion is
selected. According to the density mask generation method in FRCN with
structured regression, the center point keeps value 1, and it spreads into a
circle with a radius of d pixels, and the ratio of the decrease in pixel value
is related to distance to the center. Suppose the pixel value of a point is
$M_{ij}$, the distance that the point to the center is $D$, and $\alpha$ is a
given ratio. The formula is shown as (1), and the density mask generated
according to the formula (1) is shown as Fig. 2 (c).

$M_{ij}=\begin{cases}\frac{e^{\alpha\left(1-\frac{D\left(i,j\right)}{d}\right)-1}}{e^{\alpha}-1}&\text{
if }D\left(i,j\right)\leq d\\\ 0&\text{ otherwise}\end{cases}$ (1)

We divide the data set into three subsets, including 600 tiles in the training
set, 200 tiles in the validation set, and 200 tiles in the test set.

###  IV-B Experiment details

For the evaluation of the model, we choose F1 score. F1 score is the harmonic
average of precision (P) and recall (R). The maximum value is 1 and the
minimum value is 0. The higher the value is, the better the model is. The
formula is shown in (2).

$F_{1}=2\times\frac{P\times R}{P+R}$ (2)

True Positive (TP) indicates that the distance between the predicted center of
the nucleus and true center is less than 5 pixels. False Positive (FP)
indicates that the predicted point is not within the allowable range of the
true center, that is, the distance from the true center exceeds 5 pixels.
False Negative (FN) indicates the point that is not predicted among the truth
points. Therefore, the calculation formulas for P and R are shown in (3) and
(4).

$P=\frac{TP}{TP+FR}$ (3) $R=\frac{TP}{TP+FN}$ (4)

We used a GPU server to train the network model, which is configured as an RTX
2080Ti graphics card, with the storage up to 11000MiB. After training every
10% of the training samples, we verify the current network to adjust the
learning rate. The initial learning rate is set to 0.0001, and the minimum
learning rate is set to 0.00000001.

The loss function used for model training includes the cross-entropy loss
function for two classifications (BCE-Loss) and the average absolute error
loss function (L1-Loss). In the comparison experiment of U-Net with different
encoders, we mainly use L1-Loss. In the comparison experiment of W-Net, the
first sub-task is regarded as a binary classification problem with pixels
other than 0 or 1, so BCE-Loss is used. And the second sub-task is regarded as
the regression problem of the two-dimensional matrix to the density mask, so
the L1-Loss is used. In the W-Net training process, the loss calculated by the
two loss functions is quite different in magnitude. In order to prevent any
loss from being ignored, the ratio $\alpha$ of the two is adjusted. The loss
formula for W-Net is shown in (5).

$Loss_{W-Net}=Loss_{BCE}+\alpha\times Loss_{L1}$ (5)

###  IV-C Experiment result

We compare the performance between U-Net, U-Net with ResNet, U-Net with
EfficientNet and W-Net.

Figure 3: The effect of PIMIP’s nucleus annotation. (a) The result of
automatic nucleus detection using W-Net. (b) After transmitting the guiding
signal of the nucleus center gained by W-Net, the NuClick model is used to
complete the automatic nuclear mask drawing. We construct a complete automatic
nucleus annotation framework.

The comparison of U-Net with different encoders are shown in table 1.
Experiments are carried out on U-Net, U-Net with ResNet-34, 50, 101 and 152 as
encoders and U-Net with EfficientNet-b5, B6 and B7 as encoders. The
experimental data of U-Net is used as the baseline to show the effect of
improving encoders. It is found that when the encoder takes ResNet as the
backbone, the effect of ResNet-50 is the best. When EfficientNet is used as
the encoder backbone, EfficientNet-b7 is the best choice. U-Res-Net refers to
U-Net with ResNet-50 as its encoder backbone and U-E-Net refers to U-Net with
EfficientNet-B7 as its encoder backbone.

The comparison of U-Net and W-Net are shown in table 2. According to the above
group of comparative experiments, it is determined that the first sub-task
still uses EfficientNet-B7 as the encoder backbone. For the encoder backbone
of the second sub-task, we try the basic encoding blocks, VGGNet-13, 16 and
EfficientNet-B0 respectively. When trying VGGNet, it is found that VGGNet-16
works best. W-UU-Net means W-Net consisting of two U-Nets. W-EU-Net means that
EfficientNet-B7 is used as the first sub-task’s encoder backbone and the basic
encoder is used in the second sub-task. W-EV-Net means that EfficientNet-B7 is
used as the first sub-task’s encoder backbone and VGGNet-16 is used as the
second sub-task’s encoder backbone. The method we proposed, W-EE-Net, uses
EfficientNet-B7 as the first sub-task’s encoder backbone and EfficientNet-B0
as the second sub-task’s encoder backbone.

The effect of the basic encoder is not as good as using pre-trained models.
Due to the insufficient downsampling depth of the network, the network can not
extract better features, resulting in poor prediction results. When using
ResNet and EfficientNet with deeper network depth as encoder backbones, the
features extracted by the network are more conducive to the correct prediction
of the nuclear center. U-Net with EfficientNet performs better than W-Net with
ResNet in recall when the precision is improved. The reason is that ResNet
only considers the network depth to improve the network performance, while
EfficientNet considers network performance from the three dimensions of depth,
width, and image resolution.

The W-Net we proposed significantly surpasses U-Net in terms of precision and
recall rate, and compares with the best performing U-E-Net, the recall rate
has also been improved, making the overall performance of the model promoted.
The task is split, which reduces the difficulty of the task and gives better
play to the encoding ability. The U-Net needs to learn from pathological
images with a complex background to highlight the center of each nucleus. That
makes it too difficult to distinguish between the background and nuclei with a
high degree of fusion in a single encoding and decoding process. W-Net firstly
distinguishes the background and nuclei, solving the problem of fusion of
nuclei and the background, and then map the binary mask to the density mask to
highlight the center of each nucleus. After excluding the influence of graph
fusion, the difficulty of the task is reduced, so the performance of the
network is improved apparently.

###  IV-D System integration development

We integrated the designed W-Net into PIMIP. Firstly, the automatic nucleus
detection function is realized according to the MVC framework. A new button
for auto-detecting nuclei is added to the user interface, which triggers the
back-end model prediction function through the parameter transmission of the
controller. The prediction function stores the predicted coordinate points in
the server so that the controller can read and display it on the front-end
interface. The user interface also includes a manual correction function to
manually add or delete nuclei.

We connect the newly added automatic nucleus detection function with the
NuClick model. By transmitting the guiding signal of the nucleus center, the
NuClick model is used to complete the automatic nuclear mask drawing. We
construct a complete automatic nucleus annotation framework.

The result of PIMIP’s nucleus annotation is shown in Fig. 3. The performance
of the model in the system is good, and the system integration development is
successful.

##  V Conclusion and Discussion

###  V-A The Conclusion

In the process of pathological diagnosis, there is a large part of manual work
with a high degree of repetition, especially the work of annotating nuclei.
Moreover, the nucleus segmentation technology is hard to train a robust model,
and it is difficult to solve the problem of nucleus adhesion, stacking or
excessive fusion with the background.

In order to solve the above problems, we develop an automatic nucleus
detection algorithm based on deep learning, train a robust model, and
integrate it into PIMIP system to build a complete automatic nucleus
annotation framework to reduce the burden of pathologists.

We have made adaptive improvements to the network with encoder-decoder
structure, experimented with basic encoding blocks, U-Net with ResNet, and
innovatively tried the U-Net with EfficientNet. It is found that increasing
the depth of the network, and considering the depth, width and image
resolution comprehensively, is conducive to improving the network’s ability to
extract image features and improving the performance of the network in nucleus
detection.

By analyzing the prediction results of the U-Net, we finally split the
detection task into two sub-tasks and add an intermediate step to form a
learning path which maps the original image to the binary mask that
distinguishes the background and nuclei and then maps to the density mask that
highlights the centers of nuclei. So that W-Net is proposed, in which
EfficientNet-B7 is the backbone of the encoder for first sub-task, and
EfficientNet-B0 is the backbone of the encoder for second sub-task. The
experiments found that after the task was split, the difficulty of the task
was greatly reduced, and the overall performance of the model was improved.

We also have carried out system integration development. We integrated W-Net
into PIMIP system, and combined the detection function with the NuClick model
to realize full automatic nucleus annotation framework.

###  V-B The Discussion

In the process of improving the encoder of the U-Net, two types of backbones
have been tried. In the future, more innovative and better encoder backbones
should be tested. Improving the decoder is also important for promoting the
performance of the model.

Although W-Net shows better performance than the U-Net in predictions, the
speed is significantly reduced and the model occupies more storage space. The
problem of storage and speed also leads to great limitations in the choice of
the encoder structure. Therefore, the model will be pruned in the follow-up to
reduce unnecessary parameters and improve the speed of training and
prediction.

For the system development, the first problem is that the annotation interface
lacks more detailed guidance information. The manual correction function needs
to update in the real-time performance. Automatic nucleus detection models
have different features such as high speed or high accuracy. However, the
choice of models is only for code writers, and users lack independent choice.
Users should have the right to choose the appropriate model according to their
own needs.

All in all, in terms of the model design, we hope to increase the speed of
model training and prediction, reduce model parameters, and try more encoders
with different structures. After the model is further improved, a better
performance and more robust model is trained to be used for automatic nucleus
detection. The automatic nucleus detection function in the PIMIP system is
currently working well, but we hope to further improve the real-time
performance and user autonomy.

## Acknowledgment

This work has been supported by National Natural Science Foundation of China
(61772409); The consulting research project of the Chinese Academy of
Engineering (The Online and Offline Mixed Educational Service System for “The
Belt and Road” Training in MOOC China); Project of China Knowledge Centre for
Engineering Science and Technology; The innovation team from the Ministry of
Education (IRT_17R86); and the Innovative Research Group of the National
Natural Science Foundation of China (61721002). The results shown here are in
whole or part based upon data generated by the TCGA Research Network:
https://www.cancer.gov/tcga.

## References

  * [1] M. Oberlaender, V. J. Dercksen, R. Egger, M. Gensel, B. Sakmann, and H.-C. Hege, “Automated three-dimensional detection and counting of neuron somata,” _Journal of neuroscience methods_ , vol. 180, no. 1, pp. 147–160, 2009.
  * [2] H. Esmaeilsabzali, K. Sakaki, N. Dechev, R. D. Burke, and E. J. Park, “Machine vision-based localization of nucleic and cytoplasmic injection sites on low-contrast adherent cells,” _Medical & biological engineering & computing_, vol. 50, no. 1, pp. 11–21, 2012.
  * [3] A. N. Basavanhally, S. Ganesan, S. Agner, J. P. Monaco, M. D. Feldman, J. E. Tomaszewski, G. Bhanot, and A. Madabhushi, “Computerized image-based detection and grading of lymphocytic infiltration in her2+ breast cancer histopathology,” _IEEE Transactions on biomedical engineering_ , vol. 57, no. 3, pp. 642–653, 2009.
  * [4] S. Ali and A. Madabhushi, “An integrated region-, boundary-, shape-based active contour for multiple object overlap resolution in histological imagery,” _IEEE transactions on medical imaging_ , vol. 31, no. 7, pp. 1448–1460, 2012.
  * [5] B. Al-Diri, A. Hunter, and D. Steel, “An active contour model for segmenting and measuring retinal vessels,” _IEEE Transactions on Medical imaging_ , vol. 28, no. 9, pp. 1488–1497, 2009.
  * [6] J. A. Sethian, _Level set methods and fast marching methods: evolving interfaces in computational geometry, fluid mechanics, computer vision, and materials science_. Cambridge university press, 1999, vol. 3.
  * [7] A. Kucharski and A. Fabijańska, “Cnn-watershed: A watershed transform with predicted markers for corneal endothelium image segmentation,” _Biomedical Signal Processing and Control_ , vol. 68, p. 102805, 2021.
  * [8] Y. Zhao and X. Zhou, “K-means clustering algorithm and its improvement research,” in _Journal of Physics: Conference Series_ , vol. 1873, no. 1. IOP Publishing, 2021, p. 012074\.
  * [9] N. A. Koohbanani, M. Jahanifar, N. Z. Tajadin, and N. Rajpoot, “Nuclick: a deep learning framework for interactive segmentation of microscopic images,” _Medical Image Analysis_ , vol. 65, p. 101771, 2020.
  * [10] Y. Xie, F. Xing, X. Kong, H. Su, and L. Yang, “Beyond classification: Structured regression for robust cell detection using convolutional neural network,” in _Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015_ , N. Navab, J. Hornegger, W. M. Wells, and A. F. Frangi, Eds. Springer International Publishing, 2015.
  * [11] K. Sirinukunwattana, E. Shan, Y. W. Tsang, D. Snead, I. A. CrEe, and N. M. Rajpoot, “Locality sensitive deep learning for detection and classification of nuclei in routine colon cancer histology images,” _IEEE Transactions on Medical Imaging_ , vol. 35, no. 5, pp. 1196–1206, 2016.
  * [12] M. Tofighi, T. Guo, J. Vanamala, and V. Monga, “Prior information guided regularized deep learning for cell nucleus detection,” _IEEE Transactions on Medical Imaging_ , 2019.
  * [13] J. Xu, X. Lei, R. Hang, and J. Wu, “Stacked sparse autoencoder (ssae) for nuclei detection on breast cancer histopathology images.” in _IEEE International Symposium on Biomedical Imaging_ , 2014.
  * [14] Weidi, Xie, J., Alison, Noble, Andrew, and Zisserman, “Microscopy cell counting and detection with fully convolutional regression networks,” _Computer Methods in Biomechanics & Biomedical Engineering Imaging & Visualization_, 2018.
  * [15] T. Falk, D. Mai, R. Bensch, O. Cicek, A. Abdulkadir, Y. Marrakchi, A. Böhm, J. Deubner, Z. Jäckel, and K. Seiwald, “U-net: deep learning for cell counting, detection, and morphometry,” _Nature Methods_ , 2019.
  * [16] Y. Xie, F. Xing, X. Shi, X. Kong, H. Su, and L. Yang, “Efficient and robust cell detection: A structured regression approach,” _Medical Image Analysis_ , p. S1361841517301111, 2017.
  * [17] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” _Springer International Publishing_ , 2015\.
  * [18] N. A. Koohbanani, M. Jahanifar, N. Z. Tajadin, and N. Rajpoot, “Nuclick: A deep learning framework for interactive segmentation of microscopic images,” _Medical Image Analysis_ , p. 101771, 2020.
  * [19] G. Wang, W. Li, M. A. Zuluaga, R. Pratt, P. A. Patel, M. Aertsen, T. Doel, A. L. David, J. Deprest, and S. Ourselin, “Interactive medical image segmentation using deep learning with image-specific fine-tuning,” _IEEE Transactions on Medical Imaging_ , vol. PP, no. 99, 2017.
  * [20] D. P. Papadopoulos, J. Uijlings, F. Keller, and V. Ferrari, “Extreme clicking for efficient object annotation,” _IEEE Computer Society_ , 2017.
  * [21] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” _IEEE_ , 2016.
  * [22] M. Tan and Q. V. Le, “Efficientnet: Rethinking model scaling for convolutional neural networks,” _CoRR_ , vol. abs/1905.11946, 2019. [Online]. Available: http://arxiv.org/abs/1905.11946
  * [23] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _Computer Science_ , 2014.
  * [24] Z. Huang, Y. Ding, G. Song, L. Wang, and J. Chen, _BCData: A Large-Scale Dataset and Benchmark for Cell Detection and Counting_. Medical Image Computing and Computer Assisted Intervention – MICCAI 2020, 2020.
  * [25] Y. Li, X. Zhang, and D. Chen, “Csrnet: Dilated convolutional neural networks for understanding the highly congested scenes,” in _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2018.


# Targeting the Benchmark: On Methodology in Current
Natural Language Processing Research

David Schlangen
Computational Linguistics / Department of Linguistics
University of Potsdam, Germany
<EMAIL_ADDRESS>

###### Abstract

It has become a common pattern in our field: One group introduces a _language
task_ , exemplified by a _dataset_ , which they argue is _challenging_ enough
to serve as a _benchmark_. They also provide a baseline _model_ for it, which
then soon is _improved_ upon by other groups. Often, research efforts then
move on, and the pattern repeats itself. What is typically left implicit is
the argumentation for why this constitutes progress, and progress towards
what. In this paper, we try to step back for a moment from this pattern and
work out possible argumentations and their parts.

##  1 Introduction

The goal of any field of research is to make progress towards answering its
foundational questions. To do so, a _methodology_ is required that guides
attempts at providing or improving answer proposals. In natural language
processing, the object of study is human language, and any methodology for
doing research in this field will need to have some contact with examples of
this object. This contact has become more and more direct in the past decades,
with samples of language becoming more directly the material from which
proposals (in the form of statistical _models_) are derived. Recent years have
seen an increase in the collection of samples specifically for the purpose of
creating _benchmarks_ , against which progress in devising models can be
measured. It is this function of _benchmarking_ , and its role in a progress-
oriented methodology, that this paper aims to investigate.

Figure 1 illustrates the basic structure of a benchmarking methodology: A
_language task_ is devised that is a) restricted enough to be managable with
current methods, and b) deemed challenging for the _capabilities_ that it
involves.111This figure is from Schlangen (2019), of which this is a shorter
version developed in a slightly different direction. For this task, a
_dataset_ is collected, often via crowd sourcing, on which in turn models are
trained and compared, using evaluation metrics defined together with the task.
What can we learn by following such a methodology? Let’s look at the
components first and then at ways in which this methodology is, might, and
perhaps should be used.

Figure 1: Relations between Research Objects in a Benchmark-Driven Methodology

##  2 What is a Language Benchmark?

###  2.1 What is a Benchmark?

In computing, a benchmark is _“a problem that has been designed to evaluate
the performance of a system [which] is subjected to a known workload and the
performance of the system against this workload is measured. Typically the
purpose is to compare the measured performance with that of other systems that
have been subject to the same benchmark test.”_ Butterfield et al. (2016).

The use of this term in NLP is related: here, benchmark tasks are also
specifically designed for evaluation; however, an important difference is that
what is being evaluated is not a full system that has a separate main purpose,
but rather an _algorithm_ that is instantiated on the benchmark itself. We
will discuss the consequences of this below.

This kind of evaluation of learning algorithm has a long tradition in the
field of machine learning research.222For example, the UCI Machine Learning
Repository has been collecting and providing datasets for more than 20 years
now Dua and Graff (2019).  In this field, a new algorithm would normally be
tested on a large collection of datasets, possibly ranging from
classifications of flowers to classification of credit records, with no
assumption of any internal connection between the datasets. Again, NLP is
different here, as all datasets represent facets of the same underlying
phenomenon, language use.

We will argue that these two differences (life outside of benchmarking, and
internal connection between tasks) are important, but understudied. But first
we look at the notion of a _language task_ in some more detail.

###  2.2 What is a Language Task?

A _language task_ is a mapping between an _input space_ and an _output_ or
_action space_ , at least one of which contains natural language expressions.
The mapping has to conform to a _task description_ , which is typically given
only informally, making reference to theoretical or pre-theoretical constructs
external to the definition, such as “translation” or “is true of”. We call
this an _intensional description_. Typically, a task will also be specified
_extensionally_ through the provision of a _dataset_ of examples of the
mapping (that is, pairs of state and action). To collect such a dataset, the
task description (e.g., “classification of entailment relations between
sentence pairs”) must be operationalised into a collection instruction
(“please mark whether the a situation that is well described by sentence A
could normally also be described by sentence B”).

##  3 How Can It be Evaluated?

###  3.1 Relation Task / Dataset

Given a task and a dataset, the first question to ask is how well the latter
exemplifies the former. Investigating this is relatively straightforward.
First, the dataset should be _verified_ , which is to check whether the
provided input/output pairs can indeed be judged correct relative to the task
(in its intensional description). If the examples are collected specifically
for the purpose of exemplifying the task, this is the process of controlling
annotation, and standard methodologies exist Artstein and Poesio (2008);
Pustejovsky and Stubbs (2013). Care needs to be taken that the task is
actually well-defined enough to pose an unambiguous challenge to capable
language users.333Pavlick and Kwiatkowski (2019), for example, show that the
task of annotation textual entailments can lead to faultless annotator
disagreements.

_Validating_ a dataset is a less formalised process. It comprises arguing that
the dataset indeed exemplifies the task intension well. For example, pairs
only of images of giraffes and sentences describing them would arguably not
exemplify the general task of _image description_ very well (even if the
descriptions are accurate), while perhaps exemplifying the task of _giraffe
image description_.

Another way to evaluate a dataset is by trying to model it. If a model can
“solve” the dataset even when deprived of information that for theoretical or
pre-theoretical reasons is seen to be crucial, the dataset can be considered
an unsatisfactory exemplification of the task. E.g., in a _visual (polar)
question answering_ setting Antol et al. (2015), if in a dataset all and only
the expressions that mention giraffes are true, a model could seize on this
fact and perform well without needing the images, which would be evidence that
the dataset is deficient relative to the task description.444The task of
visual question answering provides an interesting example case of such a
development. After Antol et al. (2015) introduced the first large scale
dataset for this task, it quickly became clear that this dataset could be
handled competitively by models that were deprived of visual input (“language
bias”, as noted e.g. by Jabri et al., 2016). This problem was then addressed
by Goyal et al. (2017) with the construction of a less biased (and hence more
valid) corpus for the same task.

###  3.2 Relation Cognitive Capability / Task

While the dataset forms the visible surface of the task, it is the task itself
that needs to provide value. We can categorise tasks by how they are embedded
in further uses: a _product task_ task is one that can be argued to have
direct value to consumers (such as translation, or search); an _annotation
task_ is one where the task description is theoretically motivated and the
output a linguistically motivated object; a _benchmark task_ – which is the
type that concerns us here – finally is one which gets its value from how well
it tests a particular ability (and nothing else) and how well it discriminates
learners based on this ability.555Martinez-Plumed and Hernandez-Orallo (2018),
analysing AI benchmarks in general, distinguish between _difficulty_ (which
determines the ability level which must be reached to perform better than
chance on a task) and _discrimination_ (the slope of the graph ability level
vs. probability of correct response.

For a language benchmark task, the argument roughly goes as follows (even if
typically only made implicitly): To be good at task $T$, an agent must possess
a set $C_{T}$ of capabilities (of representational or computational nature).
If the $c\in C_{T}$ are capabilities that competent language users can be
shown or argued to possess and make use of in using language—let’s call the
set of these capabilities of a competent language user $C_{L}$, so that
$C_{T}\subseteq C_{L}$— then being able to model these capabilities (via
modelling the task) results in progress towards the ultimate goal, which is to
model competent language use. And hence, any task $T$ that comes with an
interesting set $C_{T}$ is a good task.666To give some examples of informal
versions of this argument, and chosing papers more or less randomly, here are
some quotes: From the paper that introduced the _visual question answering_
task Antol et al. (2015): “What makes for a compelling “AI-complete” task? We
believe that in order to spawn the next generation of AI algorithms, an ideal
task should (i) require multi-modal knowledge beyond a single sub-domain (such
as CV) and (ii) have a well-defined quantitative evaluation metric to track
progress. […] Open-ended questions require a potentially vast set of AI
capabilities to answer – fine-grained recognition […], object detection […],
activity recognition […], knowledge based reasoning […], and commonsense
reasoning […].” Williams et al. (2018), on computing entailments: “The task of
natural language inference (NLI) is well positioned to serve as a benchmark
task for research on NLU. […] In particular, a model must handle phenomena
like lexical entailment, quantification, coreference, tense, belief, modality,
and lexical and syntactic ambiguity.”

Under what conditions does this argument work? First of all, the assumed
connection to the set of capabilities must indeed be there. We have already
seen a way to challenge a claimed connection, through providing a model that
can “solve” a given task (via a dataset) while not having access to
information that, given our analysis of the task and interest in $C_{T}$,
should be involved in the capability.777Such an attack challenges the claim of
there being a _necessary_ connection between handling $T$ and possessing
capability $c$. It might still very well be that humans can only perform this
task if they possess capability $c$ (and all the knowledge involved in it),
because they wouldn’t be able to pick up the statistical correlations that
could be exploited.  (Although this challenge in the first instance only
targets the dataset and not the task itself.)

Secondly, following usual scientific methodology Popper (1934), we can rank
the value of an instantiation of this argument by how precisely the capability
is specified, from the trivially correct “task T involves the capability to do
task T” to a statement that could be wrong, e.g. “task T involves the
capability to compute the syntactic structure of a natural language sentence”.
Such a statement must make reference to theoretical constructs belonging to
the analysis of cognitive capabilities.

Furthermore, we can rank the motivation given for a task by how explicit it is
in delineating the set of capabilities it involves. For a given $c\in C_{T}$,
is “$c$ as required by $T$” fully _separable_ from any other tasks involving
$c$? Or is “$c$ as required by $T$” perhaps all that there is to know about
$c$, that is, is $c$ _exhaustively_ represented by $T$?

Finally, underlying the benchmarking methodology — where the benchmark is not
just a measuring tool, but also a modelling target — there has to to be the
assumption that some sort of transferable knowledge is generated by modelling
$T$, so that what the model has learned about (a sufficiently generally
specified) $c$ can be used in other tasks that involve $c$. (Let’s call this
_transferability_ ; which strictly speaking is a property of models, not of
tasks.) More on this below.

To sum up, a benchmark task gets its value from its connection to a particular
facet of language, a particular capability of language users; this in turn
seems to be difficult to specify without access to terms from theories of the
domain, which allow us to name these capabilities.888And one will indeed find
that papers introducing such tasks make mention of terms like _syntax_ ,
_semantics_ , _compositionality_ , _quantifiers_ , etc. ,999We can also note
that with this focus normally comes a certain top-down approach, where the
collected data is not investigated for how exactly the human participants
actually solved their task. (But see van Miltenburg (2019) for a detailed
study along those lines, for the task of image description.)

##  4 How are Language Benchmarks Used?

In the way that these tasks are set up, as single-step tasks that humans can
quickly do (“describe this image”, “is the elephant [in this image]
sleeping?”, “does sentence A follow from sentence B?”), it is tempting to see
a similarity to tasks used in intelligence testing (see e.g. Borsboom (2005)
for an introduction). There is a crucial difference, however: Where
intelligence testing works more in the way standard computing benchmarking
works (subjecting the _otherwise functioning_ learner to a standardised
workload), in NLP, benchmarks are both the testing instrument as well as the
training material.101010Not unlike a school that aims to improve its test
scores by preparing its students specifically for the tests; in that case,
however, this practice would be seen as undermining the value of the test. ,
111111For a recent paper also discussing the relation between AI benchmarking
and intelligence testing, see Chollet (2019).  The question then cannot be “to
what extent does system $\Sigma$ possess capability $c$”, it has to be “to
what extent can algorithm $A$ learn $c$ from dataset $D$?” — and what does
that tell us?

###  4.1 Single-Task Models

Let’s assume we have defined a task $T$ that we are sufficiently convinced is
well represented by dataset $D$. We have trained a model $M$ that performs
well on this dataset. What have we learned? We have learned that a learning
algorithm of the type of $M$ can model $D$. Further, we have learned that the
information to do task $T$ (as exemplified in $D$), is contained in $D$, and
$M$ can pick it up.

Under what conditions can we now say that we have modelled $T$, rather than
just $D$? If we have convinced ourselves that $D$ represents $T$ faithfully,
then we might be willing to make this leap, and with it, claim that we have
modelled $C_{T}$. But we can get further support by collecting more data
$D^{\prime}$ that also exemplifies $T$, but perhaps operationalises it
differently. The prediction should be at least that the learning algorithm can
also learn to model $D^{\prime}$; but more significantly, we’d also want the
model $M$ learned from $D$ to perform well on $D^{\prime}$. Similarly, if we
have another task $T^{\prime}$ of which we think that it involves similar
capabilities, we should expect it to be amenable to being modelled with a
learning algorithm of similar type to $M$.

What do we learn from a model $M^{\prime}$ (introducing architectural
innovation $\kappa$ over $M$) performing better on $T$ (via $D$)? We can take
this as indication that $\kappa$ may be what is responsible for increasing
performance, and hence what is leading to a more adequate model of $C_{T}$.

###  4.2 Multi-Task Models

With the advent of pre-training in NLP Peters et al. (2018); Devlin et al.
(2018), where a model is trained on (a typically large amount of) data under a
specific task-regime (typically language modelling, i.e. the task of
predicting the next word in a running text) and then becomes part of the model
for a target task, it has become common to test on a collection of tasks Wang
et al. (2019b, a). What do we learn from such a setup? In our Figure 1, if we
find a task on which we can pre-train a model $M_{P}$ that becomes a part of
models $M$ and $M^{\prime}$, and which makes them more powerful than models
that do not have access to the pre-trained model, then we can infer that
whatever $M_{P}$ models is a shared part of $M$ and $M^{\prime}$ as well (and
hence _involves_ the hypothesised joint capability $C^{\prime}$). This then
provides an instrument to study the tasks: if the pre-trained model works well
on some but not all, there must be something that those groups have in common.
To make this intelligible, however, recourse to theoretical terms must again
be taken. (E.g., assuming that these tasks involve the use of certain types of
representation, or certain actions over representations.)

##  5 But Are We Making Progress?

Within the logic of this methodology, we are clearly making enormous progress
at two links in the chain illustrated in Figure 1: For many of the established
tasks, models have been and continue to be proposed that perform better,
according to the metrics defined for the tasks. In addition, for many of the
tasks, better datasets have been collected, avoiding exploitable biases. Where
there is less activity is in systematically studying the implications of
success at one task for success at others. The presentation above was largely
idealised (or normative): In reality, there is very little explicitness about
the assumed connection between tasks and capabilities, and no theory of how
(or whether) language competence decomposes into capabilities that could be
learned separately and then be assembled into a whole, and there is very
little explicit knowledge about the vertical links in the Figure, from one
task / model to the next.

##  6 Conclusions

In this short paper, we have discussed the methodology of using _language
tasks_ to drive research on models of language competence. We have argued that
the success of this approach hinges on how well progress on one task can be
translated into progress on other tasks. While some steps have been taken in
this direction, current work still appears to mostly focus on isolated tasks
(or groups of tasks). Overcoming this, in our opinion, will require more
explicit considerations about how tasks and capabilities are connected, and
how the set of capabilities is structured. For this, a (re-)connection with
the fields that study the composition of language competence, linguistics and
cognitive and developemental psychology, seems to be advisable. Finally, the
stark difference between how humans can pick up new tasks, based on an
intensional description and a few examples, and how current models do this (by
needing massive amounts of data), and between how human language competence
develops (from simpler interactions to more complex ones) and how models
“develop” (from scratch on any task) has to be accounted for, pointing to a
future where the real benchmark might be the developmental trajectory, and the
tasks are only measures, not targets.

## References

  * Antol et al. (2015) Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015.  Vqa: Visual question answering.  In _International Conference on Computer Vision (ICCV)_.
  * Artstein and Poesio (2008) Ron Artstein and Massimo Poesio. 2008.  Inter-Coder Agreement for Computational Linguistics.  _Computational Linguistic_ , 34(4):555–596.
  * Borsboom (2005) Denny Borsboom. 2005.  _Measuring the Mind: Conceptual Issues in Contemporary Psychometrics_.  Cambridge University Press, Cambridge, UK.
  * Butterfield et al. (2016) Andrew Butterfield, Gerard Ekembe Ngondi, and Anne Kerr, editors. 2016.  _A Dictionary of Computer Science_ , 7th edition.  Oxford University Press, Oxford, UK.
  * Chollet (2019) François Chollet. 2019.  On the Measure of Intelligence.  _arXiv e-prints_ , page arXiv:1911.01547.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  BERT: pre-training of deep bidirectional transformers for language understanding.  _CoRR_ , abs/1810.04805.
  * Dua and Graff (2019) Dheeru Dua and Casey Graff. 2019.  UCI machine learning repository.
  * Goyal et al. (2017) Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017\.  Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering.  In _CVPR 2017_.
  * Jabri et al. (2016) Allan Jabri, Armand Joulin, and Laurens van der Maaten. 2016.  Revisiting Visual Question Answering Baselines.  In _European Conference on Computer Vision (ECCV)_.
  * Martinez-Plumed and Hernandez-Orallo (2018) Fernando Martinez-Plumed and José Hernandez-Orallo. 2018.  Dual indicators to analyse ai benchmarks: Difficulty, discrimination, ability and generality.  _IEEE Transactions on Games_ , pages 1–1.
  * van Miltenburg (2019) Emiel van Miltenburg. 2019.  _Pragmatic factors in (automatic) image description_.  Ph.D. thesis, Vrije Universiteit Amsterdam.
  * Pavlick and Kwiatkowski (2019) Ellie Pavlick and Tom Kwiatkowski. 2019.  Inherent Disagreements in Human Textual Inferences.  _Transactions of the Association for Computational Linguistics_ , 7:677–694.
  * Peters et al. (2018) Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018.  Deep contextualized word representations.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 2227–2237, New Orleans, Louisiana. Association for Computational Linguistics.
  * Popper (1934) Karl Popper. 1934.  _Logik der Forschung_.  Mohr Siebeck, Tübingen, Germany.
  * Pustejovsky and Stubbs (2013) James Pustejovsky and Amber Stubbs. 2013.  _Natural Language Annotation for Machine Learning_.  O’Reilly, Sebastopol, CA, USA.
  * Schlangen (2019) David Schlangen. 2019.  Language tasks and language games: On methodology in current natural language processing research.  _CoRR_ , abs/1908.10747.
  * Wang et al. (2019a) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019a.  SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems.  In _NeurIPS_ , July, pages 1–30.
  * Wang et al. (2019b) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019b.  GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding.  In _ICLR 2019_ , pages 1–20.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2018.  A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference.  In _Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)_.