# Design and Evaluation of Crowd-sourcing Platforms Based on Users’ Confidence
Judgments

Samin Nili Ahmadabadi  School of Electrical and Computer Engineering,
University of Tehran, College of Engineering, Tehran, Iran  Maryam Haghifam
Department of Computer Science, University of Toronto, Toronto, Canada  Vahid
Shah-Mansouri  School of Electrical and Computer Engineering, University of
Tehran, College of Engineering, Tehran, Iran<EMAIL_ADDRESS>Sara
Ershadmanesh  Department of Computational Neuroscience, MPI for Biological
Cybernetics, Tuebingen, Germany  School of Cognitive Sciences, Institute for
Research in Fundamental Sciences, Tehran, Iran

###### Abstract

Crowd-sourcing deals with solving problems by assigning them to a large number
of non-experts called crowd using their spare time. In these systems, the
final answer to the question is determined by summing up the votes obtained
from the community. The popularity of using these systems has increased by
facilitation of access to community members through mobile phones and the
Internet.
One of the issues raised in crowd-sourcing is how to choose people and how to
collect answers. Usually, the separation of users is done based on their
performance in a pre-test. Designing the pre-test for performance calculation
is challenging; The pre-test questions should be chosen in a way that they
test the characteristics in people related to the main questions.
One of the ways to increase the accuracy of crowd-sourcing systems is to pay
attention to people’s cognitive characteristics and decision-making model to
form a crowd and improve the estimation of the accuracy of their answers to
questions. People can estimate the correctness of their responses while making
a decision. The accuracy of this estimate is determined by a quantity called
metacognition ability. Metacoginition is referred to the case where the
confidence level is considered along with the answer to increase the accuracy
of the solution [1]. In this paper, by both mathematical and experimental
analysis, we would answer the following question: Is it possible to improve
the performance of the crowd-sourcing system by knowing the metacognition of
individuals and recording and using the users’ confidence in their answers?

###### keywords:

Crowd-sourcing, Meta-cognition, Majority Voting, Group Decision Making

##  1 Introduction

The goal of crowd-sourcing is to entrust answering questions, called tasks, to
non-experts and using their idle time at a low cost. In these systems, cost is
defined as the time consumed to decide and the payment to the users. Crowd-
sourcing tries to use the extra processing power of millions of human brains.
The experience with open-source software proved that a network of enthusiastic
volunteers and highly paid developers at Microsoft and other companies could
develop software. Wikipedia showed that this model could be used to create a
surprisingly distributed online encyclopedia [2].

Given that most crowd-sourcing systems are online, it usually dose not matter
where people are located. These systems help companies, industries, research,
etc., find the solution to their problems or the required data in a more
affordable and faster way. Also, crowd-sourcing systems can be an easy source
of income for people in their spare time. Furthermore, people with
disabilities can easily participate in these systems.

Many discussions in the field of crowd-sourcing systems deal with solutions to
avoid and minimize errors. In this regard, this research deals with the use of
the decision-making model of people in such systems. Knowing how people make
decisions makes it possible to predict the probability of the person answering
the question correctly. One unsolved question in this area is that is it
possible to improve the system’s performance by asking people’s confidence in
their answers and how this improvement can be done. In literature,
metacognition ability is defined based on users’ responses and confidence in
order to see how accurately they can assess their performance. Thus,
Metacognition ability is a predictor of how well users can respond and how
well they are reporting their confidence [3].

In typical crowd-sourcing systems, people are separated based on efficiency,
and the majority vote algorithm is used to aggregate votes. In order to
investigate the impact of metacognition and people’s decision-making model in
crowd-sourcing systems, in this paper, we present a new method for screening
people relying on metacognition and aggregation of votes with confidence and
measuring their performance against a conventional system. Metacognition
ability-based crowd-sourcing systems can be useful in two main cases:

  * •

Metacognition ability is the same among different tasks [4, 5]. Therefore,
this parameter can be measured once and recruit users for several tasks
according to their metacognition ability.

  * •

Sometimes, people choose an answer that are not completely sure about its
correctness. Asking how confident they are may help the system to change the
weight of their decision in the final aggregation.

In this research, our contributions are as follows:

  * •

We propose a novel crowd-sourcing system based on the users’ reported
confidence and their metacognition ability.

  * •

With the help of simulation, we check the superiority of our proposed system
for different communities. With the help of the results, we have identified
the characteristics of the communities in which our system works better.

  * •

By designing and implementing a real-world task, we prove our hypothesis that
our system outperforms the typical crowd-sourcing systems in certain
communities. We ask people to identify the gender of the author of several
tweets. This experiment’s results show that using the proposed crowd-sourcing
system reduces the error.

In Section 2, we introduce the components of crowd-sourcing systems and
examine the solutions for various problems in these systems. Then we will
introduce and review several practical and commercial examples of this field.
In the following, we will introduce the concept of metacognition ability by
examining and modeling the decision-making process of humans. Then, we will
describe the use of metacognition in crowd-sourcing systems. The proposed
model of the crowd-sourcing system, considering the first and second types of
decision-making, is introduced in Section 3. Through mathematical and
experimental analysis, we will examine our proposed system. According to our
findings, the crowd-sourcing system error can be decreased by asking people
how confident they are and khowing their metacognition ability. Furthermore,
in this manner, we can avoid repetitive pre-tests and recruit users based on
their metacognition ability which will be tested at the beginning of their
enrollment.

##  2 Background

###  2.1 Crowd-sourcing Systems

Crowd-sourcing is a method that relies on collective wisdom to answer
questions. Usually, this method is used for questions whose answer is
unavailable or there is not enough confidence in the available response. To
obtain the answer, the desired question is asked from several people, and
based on the crowd’s response, the final answer is extracted. There are
different types of crowd-sourcing systems, some of which are listed below:

  * •

In some crowd-sourcing systems, only one person provides a solution for each
problem; For example, in InnoCentive and kaggle, companies describe their
issues and questions. The people in this network offer a solution after
studying the problem. At the end, the crowd answer will be extracted by an
answer aggregation algorithm, which will be explained later. The users whose
answers are the same as the crowd answer will be rewarded.

  * •

Another group of crowd-sourcing systems is designed to use data sent from
mobile phones or computers [6]. In this type of system, there is no need for
people to answer questions. One example of this category is traffic estimation
by Google Map. This software receives the speed of cars from the sensors of
smartphones and using this information and data obtained in the previous days
and hours with the same method, it estimates the traffic and announces the
fastest route to destination. In this type of systems, people are not given
cash rewards, but they can use the data obtained by these softwares (for
example, routing according to traffic). Another example of this area is
finding a network map and classifying points in terms of antenna quality. For
this purpose, a software is installed on the smartphone, and this software
sends information about the network to a server in each region [7].

  * •

In another type of crowd-sourcing system, a problem arises that is difficult
to find an answer to or requires an expert. Since it is expensive to get help
from an expert, crowd-sourcing suggests that we leave this task to a large
number of non-experts and then use their answer set. In the rest of this
paper, a crowd-sourcing system is a system that uses the method of assigning a
task to a large number of people.

One of the most famous crowd-sourcing system is Amazon Mechanical Turk (AMT).
This site is a web marketplace that helps companies find people to solve
problems that computers can’t solve; For example, identifying and categorizing
photos and videos and annotating them, converting images into documents and
books [8], implementing the text of podcasts, etc. These tasks are called
HITs. Each HIT is designed in such a way that it is not necessary to spend a
lot of time doing them.

One of the uses of crowd-sourcing is categorizing and annotating images and
videos. Due to the large number of images available on the web, database,
etc., this work is not possible manually because it is time consuming. On the
other hand, for obtaining the best result, experts are needed and recruiting
them is usually expensive. Sometimes techniques can be used that automatically
annotate multimedia content using algorithms. The main disadvantage of these
methods is that they only support limited vocabulary for annotation, and in
complex cases, they lead to high errors [9]. For annotation complex cases, we
can use an expert which is expensive. Therefore, one of the standard solutions
is to use crowd-sourcing in this field. For example, [10] has been able to
label many images by designing a game. The paper predicts that if the game is
played continuously by 5,000 people, all the photos on Google (425,000,000
photos in 2005) will be tagged within 31 days. In this game, participants are
paid cash, and people play for fun only. Finally, a label that a certain
number of people agree on is selected for each photo. It is also possible to
classify images by quality using crowd-sourcing [11] or improve their quality
[12].

Another common application of these systems is to obtain a complete and
accurate map inside the buildings [13]. CrowdMap is a software designed for
finding indoor maps which can help a variaty of jobs such as interior
designers [14]. This software asks the user to film the space inside the
building. Then, a map inside the building is obtained using the people’s
geographic location and the video sent.

Other uses of crowd-sourcing include data collection for problem analysis. For
example, [15] shows different ads to people using crowd-sourcing software.
Then by analyzing their facial expressions and the questions , the system
tries to provide a way to determine the impact of ads on people.

Crowd-sourcing systems have made it possible for different people to
participate in business activities. Commercial companies can conduct market
research, development, and testing of new products, pricing, etc., through
these systems [16].

####  2.1.1 Steps of a crowd-sourcing system

In a crowd-sourcing system, first, a task is defined. Then the system
announces the task’s information to all the crowd-sourcing network’s people.
The task information is its topic and category, description, the time required
to do it, etc. Then the users who are enthusiastic about doing this task
announce their willingness. In some systems, users propose an amount they want
to be paid in exchange for performing the desired task. But sometimes, the
system calculates and announces the payment amount. Next, the system holds a
pre-test. Based on predefined criteria (such as performance in the pre-test
section), the system decides which users to recruit to contribute to the task.
After this step, the task will be fully announced to the selected users. After
completing the task, the system collects the answers, and the crowd’s response
is calculated using an answer aggregation method. Rewards will be paid to the
people whose answers are recorded in the assigned time and whose answers are
the same as the crowd’s answer.

User Recruitment Selecting users is done in two main ways: online and offline
[17]. In the offline method, the system waits to find the required number of
users and then announces the task to everyone. For this mode, it is assumed
that there are a number of simultaneous users in the system. One of the
crucial things in this selection is predicting the location of people so that
we can estimate how many people will be present in the situation where we want
the task to be done and adjust the filters for selecting people [18, 19].
Since offline recruiting is not easy, the online method is usually used. In
this method, any user can announce his willingness in a period of time, and
the system should decide whether to recruit the user or not.

Answer Aggregation Algorithms. In order to extract the answer from the crowd,
different algorithms can be used. The two main categories of them are
mentioned below:

  * •

Majority voting (MV): In this method, an answer is announced as the final
answer that the majority of the crowd has agreed on [20]. This method (as well
as weighted majority voting) is the most common in these systems. That is
because this method is easy to implement and the calculation overhead is much
less compared to the other methods.

  * •

Algorithms based on machine learning and pattern recognition: these algorithms
are one of the main answer aggregation algorithms in the field of annotation.
For example, [21] found the final answer through clustering. Also, in [22], it
detects spam text messages using SVM and Bayesian filters. Naive Bayes [23] or
deep Bayesian [24] can be used to find suitable photo labels.

Incentive mechanisms. Users’ rewards can be cash or non-cash. Some crowd-
sourcing systems are designed in such a way that participation in them is
enjoyable (such as systems developed in the form of a game) or ultimately
provide information and services to the users. But considering that doing the
task will be costly for the person (use of mobile phone charging, internet,
time, etc.), in some systems, a cash bonus is paid to the users.
There are different solutions for determining cash rewards. as an example,
paying the winner is one of the incentive methods. This method puts people
into a competitive phase and prevents them from cooperating with each other.
Although the non-cooperation of users causes different responses and comments
to be registered, it may reduce the number of users willing to do the task
because they are no different in this second and last place structure.
Therefore, except for people who are very interested in competition, other
people will not have the motivation to participate [25].
Another method is to pay a fixed amount to all the users [21, 26], which
reduces the number of calculations to find the payment amount and gives
everyone an incentive to participate in the task. Also, this method does not
incentivize people to cheat for more money.

Auction is also used in these systems. In this case, the user submits his bid.
Based on the bid price and budget, the system decides which users to use [17,
27].

Also, payment based on the required time and the difficulty of the question
can be utilized [28].

###  2.2 Meta-cognitive ability

Decision-making requires gathering, organizing, synthesizing, and evaluating
information. Due to each person’s difference in understanding and intuition of
the information needed to make a decision, people will not make the same
decision in the same situations. Decision-making in people can be divided into
two types: the first and the second type. The first type of decision-making
refers to the individual’s choice between the available options, while the
second type refers to the degree of confidence that the decision has been made
correctly [29]. The level of a person’s ability to correctly estimate his
level of confidence is called metacognition.

Metacognition refers to higher-order thinking that deals with the ability to
influence and monitor one’s cognitive processes, such as perception, memory,
and decision-making. Metacognition usually deals with determining whether a
person’s certainty of answering can be a predictor of the correctness of their
decision [3]. Metacognition is more simply defined as "thinking about
thinking." Metacognitive strategies are sequential processes that a person
uses to control cognitive activities and ensure the achievement of a cognitive
goal (for example, understanding a text). These processes help regulate and
monitor learning and include planning and monitoring cognitive activities as
well as examining the results of those activities. For example, after reading
the paragraph in the text, each person can ask themselves about the concepts
discussed in the paragraph. Their cognitive goal was to understand the text.
Self-questioning is a metacognitive monitoring strategy [30].

As mentioned, metacognition plays a vital role in learning. People are often
aware of their mistakes and report a level of confidence in their choices that
is related to their performance. These metacognitive assessments of decision
quality are essential for guiding behavior and subsequent decisions,
especially when external feedback is absent [31].

Suppose we consider a simple two-choice decision-making task. In that case, a
person with high metacognition is assigned to one who declares his correct
answers with high confidence and his incorrect answers with low confidence.

In the presented models for decision-making, it is assumed that a decision
variable is produced in the brain for each question. This variable is compared
to the individual’s decision threshold, and being greater or lower determines
which option they choose. The difference between the decision threshold and
decision variable will show one’s level of confidence [32]. If a person is
asked to declare their confidence in a leveled and discrete manner, they will
report this value according to the thresholds specified in their mind to
declare confidence levels. The parameters involved in the production of the
decision variable, the decision threshold, and the choice of confidence level
are different from person to person.

There are various methods to study the first and the second type decision-
makings. The most common method is using Signal Detection Theory (SDT) and
using Receiver Operating Characteristic (ROC) [1], which are explained below.

The answer to any question that a person submits, based on the correct answer
to the question, can have four states which are shown in Table 1.

Table 1: The possible modes of the first type of person’s answer to two-choice questions | Person’s response
---|---
$a_{1}$ | $a_{2}$
Correct answer | $a_{1}$ | Correct rejection | False alarm
$a_{2}$ | Miss | Hit

Hit rate (HR) and false alarm rate (FAR) are used to examine the individual’s
behavior to answer the questions. If we draw a graph whose horizontal axis is
FAR, its vertical axis is HR, and its points are the pairs (FAR, HR) and fit
it, the obtained curve will be the first type ROC. If the person responds
completely randomly, their ROC curve will be the line FAR=HR. Therefore, the
area above this line shows a person’s ability and expertise. The area under
the first type ROC diagram (AUROC1) is directly related to performance (the
rate of answering questions correctly).

To analyze Type II decision, assuming that there are only two levels of
confidence (high and low), we would have four states for Type II decision-
making, shown in Table 2.

Table 2: The possible modes of the second type of person’s answer to two-choice questions | Person’s confidence
---|---
High | Low
Answer | Correct | Hit 2 | Miss 2
Wrong | False alarm 2 | Correct rejection 2

Similarly, we use HR 2 and FAR 2 to draw the second type ROC. The area below
this diagram (AUROC2) represents a person’s metacognition level. This method
can be used for more confidence levels.

###  2.3 Metacognition in Crowd-sourcing Systems

If we ask people for their confidence when asking crowd-sourcing questions, we
can calculate their metacognition and performance. Their confidence can also
be used to aggregate votes. Existing examples of crowd-sourcing systems do not
typically use metacognition, and there are only a few examples in this field.
One of the examples close to these systems using metacognition is a crowd-
sourcing system using self-assessment introduced in [26].

It is stated in [26] that self-assessment can help attention and accuracy in
answering questions in users. In the model presented by this article, after
running a pre-test, it asks people what percentage of questions they have
answered correctly according to their opinion. In addition to high efficiency,
people will be recruited based on a relatively good performance estimation.

The Dunning-Kruger effect is a cognitive bias in which people assume their
competency is low when they have high performance and vice versa. This effect
and its influence on crowd-sourcing systems are studied in [26]. This paper
proposes a mathematical model for performance and confidence based on the DK
effect. By assuming MV, and WMV aggregation algorithms, they have studied the
impact of confidence reporting in crowd-sourcing systems. They have concluded
that WMV causes more errors than MV. According to our experiment, this finding
is invalid, and the proposed model in this paper cannot be used in all cases.

In the next section, we will introduce and analyze our proposed crowd-sourcing
system. In this system, we will answer whether asking about the confidence
level will improve the system’s performance. To check this point, we will
compare one of the common crowd-sourcing systems with our proposed system.

##  3 Decision-Making Model

As we mentioned in Section 2, there are two types of decisions: Type I and
Type II. A Type I decision is a choice between the possible options. Type II
decision represents the confidence level of the decision. In this section, we
are going to discuss the mathematical model for these types based on what [32]
has presented. For the sake of simplicity, we will model the decision when
humans face two-choice questions.

###  3.1 Type I Decision

When a question is presented, a decision variable is generated corresponding
to the evidence in favor of each option, $A=1$, and $A=2$. You will decide
among the options based on the value of this variable and your decision-making
threshold. A Gaussian model often models this random variable. The
distribution’s mean and variance vary from person to person depending on their
performance and bias. The distribution of the decision variable, $x$, is as
follows:

$x|A=i\sim\mathcal{N}(\mu_{i},\sigma_{i}),\qquad i\in\\{1,2\\}$ (1)

Imagine you are to answer a question whose actual answer is $A=1$. In your
mind, a random variable is generated that comes from a Gaussian model,
$\mathcal{N}(\mu_{1},\sigma_{1})$. If the generated variable is lower than
your Type I decision-making threshold, $c1$, you will choose $A=1$, otherwise
$A=2$.

Figure 1: Decision-making model. A normal random variable, $x$, is generated
for each question. Type I decision is made based on the value of $x$ and the
decision-making threshold, $c1$. The same happens with Type II decision,
confidence, due to $x$ and Type II decision-making threshold, $c2|A$.
The green area shows the probability of answering correctly if the answer is
option two and the confidence is reported as high. Also, the red area is equal
to the probability of being wrong, with the true answer being option 2,
reporting low confidence.

With the help of this model, the probability of answering correctly for users
can be calculated, also known as performance. The $u$th user’s performance,
shown by $Perf_{u}$, is as follows:

$\displaystyle Perf_{u}=$ $\displaystyle\Pr(A=1)\times\Pr(x\leq c1|A=1)$
$\displaystyle+\Pr(A=2)\times\Pr(x>c1|A=2).$ (2)

Due to (1), both $\Pr(x\leq c1|A=1)$ and $\Pr(x>c2|A=2)$ can be written as
$\operatorname{Q}(x)=\frac{1}{\sqrt{2\pi}}\int_{x}^{\infty}\exp(-\frac{u^{2}}{2})du$.
For defining this function, assume $Y$ is a Gaussian random variable with mean
$\mu$ and variance $\sigma^{2}$, then $V=\frac{Y-\mu}{\sigma}$ is standard
normal and

$\Pr(Y>y)=Pr(V>v)=\operatorname{Q}(v)=\frac{1}{\sqrt{2\pi}}\int_{v}^{\infty}\exp(-\frac{u^{2}}{2})du$

Therefor, The $Perf_{u}$ is calculated as

$\displaystyle Perf_{u}=$
$\displaystyle\Pr(A=1)\times[1-\operatorname{Q}(\frac{c1-\mu_{1}}{\sigma_{1}})]$
$\displaystyle+\Pr(A=2)\times\operatorname{Q}(\frac{c1-\mu_{2}}{\sigma_{2}}).$
(3)

###  3.2 Type II Decision

After choosing your answer to the question, you will decide how confident you
are about the correctness of your decision; This is called Type II decision.
This decision is similar to Type I, according to the decision variable, $x$,
and Type II decision-making threshold. By assuming $c_{max}$ is the maximum
level of confidence that can be reported, this threshold is $c2=j|A=i\quad
i\in\\{1,2\\},\leavevmode\nobreak\ j\in\\{1,...,c_{max}\\}$.

Assuming we have just two confidence levels (low and high), the levels are
divided by one threshold. In the previous example, if your choice was $A=1$
and $x$ is greater than $c2|A=1$, your confidence is low; otherwise high.

The meta-cognitive ability that was discussed in section 1 is how efficiently the user’s confidence ratings discriminate between their own correct and incorrect stimulus classifications [32]. Several methods measures meta-cognition ability, but in this study, we use the area under Type II ROC curve, AUROC2 [33]. The y-axis of this curve is Pr( confidence | correct) and the x-axis is $\Pr(\text{confidence}|\text{incorrect})$. Due to its definition, the greater AUROC2 means the user is more meta-cognitive.

All the parameters defined in this section differ among users and can vary
over time. They all can be obtained by fitting this model to their decision-
making behavior. In the next section, we will use this model to make decisions
in a crowd-sourcing system.

##  4 Method

In any crowd-sourcing system, after broadcasting the task information and
collecting volunteers’ requests, who desire to do the task, the system selects
the most proper subset of volunteers. This selection is either on the features
or skills each person claims or according to the pre-test results. The
selected volunteers, also known as users, are recruited and will participate
in the test phase. In this phase, the main questions are presented to the
users, and their answers are collected. The system extracts the crowd’s answer
based on an aggregation method and announces it as the correct answer. Users
receive incentives at the end of the task due to the system’s payment policy.
In this paper, we assume incentives for all the users are the same. In this
section, we first discuss about a conventional crowd-sourcing system, Response
Based Crowdsourcing System (ReBaCS), and then our proposed system,Confidence-
Based Crowd-Sourcing system (CoBaCS) will be introduced.

###  4.1 Response Based Crowd-Sourcing System (ReBaCS)

One of the popular and conventional types of crowd-sourcing systems are just
based on users’ responses, and the answer aggregation is done by the Majority
Voting (MV) algorithm. As mentioned earlier, each crowd-sourcing system has
two phases: the first, a pre-test phase that examines the volunteers’
capabilities, and the second, the test phase, that the main questions are
presented to the recruited users. In the pre-test phase, the users submit
their answers to the questions while the system knows the correct answers.
Users are selected due to their performance at the end of the pre-test phase;
by performance, we mean how correctly their answers were. To have proper user
selection, the pre-test questions should be in the same context as the main
crowd-sourcing task. The recruited users and volunteers’ performance reached a
minimum threshold in the test phase and must answer the main crowd-sourcing
questions. The crowd’s answer to each question is the one that more than half
of the crowd has consensus on. Users will be paid at the end of the task. Now
on, we will refer to these systems as Response Based Crowdsourcing System
(ReBaCS).

####  4.1.1 Error Calculation

For the sake of simplicity, we assume that all the questions in both the pre-
test and test phases are two-choice. The answer of the $n$th user to question
$q$ can be written as:

$r_{n,q}=\begin{cases}0&\text{if the $n$th user has chosen first option}\\\
1&\text{if the $n$th user has chosen second option}\end{cases}$ (4)

By assuming $N_{u}$ users have passed the pre-requisite test and are presented
in the system, the crowd’s answer for question $q$, $R_{\text{MV}_{q}}$, is
driven by MV as

$R_{\text{MV}_{q}}=\begin{cases}0,&\sum_{n=1}^{N}r_{n,q}<\frac{N_{u}}{2},\\\
1,&\text{o.w},\end{cases}$ (5)

If we denote the correct answer of the $q$th question by $A_{q}$, the system’s
performance is the proportion of the questions in which their actual response,
$A_{q}$, matches the crowd response, $R_{\text{MV}_{q}}$. If we assume $N_{Q}$
questions are available in the system, the system’s accuracy is defined as the
percentage of the correct answers as

$\displaystyle acc_{\text{ReBaCS}}$
$\displaystyle=\frac{\sum_{q=1}^{N_{Q}}\delta(R_{\text{MV}_{q}},A_{q})}{N_{Q}},$
(6) $\displaystyle\delta(x,y)$ $\displaystyle=\begin{cases}1,&\text{if
}x=y,\\\ 0,&\text{o.w.},\end{cases}$

By the above definition, the error of the system will be
$err_{\text{ReBaCS}}=1-acc_{\text{ReBaCS}}$. With the help of the total
probability theorem, the expectation of error can be rewritten as follows:

$\displaystyle\mathrm{E}_{N_{Q}}[err_{\text{ReBaCS}}]=$
$\displaystyle\Pr(A\neq R_{\text{MV}})$ (7) $\displaystyle=$
$\displaystyle\Pr(A=0)\times\Pr(R_{\text{MV}}=1|A=0)$
$\displaystyle+\Pr(A=1)\times\Pr(R_{\text{MV}}=0|A=1)$ $\displaystyle=$
$\displaystyle\Pr(A=0)\times\Pr\left(\sum_{n=1}^{N}r_{n}\geq\frac{N_{u}}{2}|A=0\right)$
$\displaystyle+\Pr(A=1)\times\Pr\left(\sum_{n=1}^{N}r_{n}<\frac{N_{u}}{2}|A=1\right).$

In these equations, $A$ shows the actual answer to a question, and
$R_{\text{MV}}$ refers to the crowds’ response based on the MV algorithm.
According to the decision-making model described in Section 3, each user
decides between the first and the second option in every question based on a
Bernoulli random variable as:

$\displaystyle\Pr(r_{n}=r|A=i)=$
$\displaystyle\begin{cases}\operatorname{Q}\left(\frac{c_{n}-\mu_{i,n}}{\sigma_{i,n}}\right)&r=1\\\
1-\operatorname{Q}\left(\frac{c_{n}-\mu_{i,n}}{\sigma_{i,n}}\right)&r=0\end{cases},\
\ i\in\\{0,1\\}$ (8)

where $\operatorname{Q}(x)$ is
$\frac{1}{\sqrt{2\pi}}\int_{x}^{\infty}\exp(-\frac{u^{2}}{2})du$, $c_{n}$ ,
$\mu_{i,n}$ and $\sigma_{i,n}$ are the decision making parameters for the
$n$th user, respectively. Each user’s decision-making characteristics are
encoded in how far the $\mu_{i,n}$s are from $c_{n}$ and the difference
between $\mu_{i,n}$s. Therefore, without loss of generality, we can assume
$c_{n}=0$ and $\sigma_{0,n}=\sigma_{1,n}=1$ for all users and let the value of
$\mu_{i,n}$ show their decision making policies. By these assumptions, the
equation in (8) becomes less complex:

$\displaystyle\Pr(r_{n}=r|A=i)=$
$\displaystyle\begin{cases}\operatorname{Q}(\mu_{i,n})&r=1\\\
1-\operatorname{Q}(\mu_{i,n})&r=0\end{cases}$ (9)

The summation of several independent Bernoulli random variables with the same
distribution parameters is a Binomial random variable. Therefore, by
considering that all $\mu_{i,n}$ are the same and equal to
$\mathrm{E}[\mu_{i,n}]=\mu_{i}$, the random variables defined as
$\gamma_{i}=\sum_{n=1}^{N_{u}}r_{n}|\big{(}A=i\big{)},\leavevmode\nobreak\
i\in\\{0,1\\}$ are binomial random variables. If we suppose $N_{Q}$ is large
enough, $\gamma_{i}$ follows a normal distribution as

$\displaystyle\gamma_{i}$ $\displaystyle\sim\mathcal{N}(m_{i,r},s_{i,r})$ (10)
$\displaystyle\qquad m_{i,r}=N_{u}\times\operatorname{Q}(-\mu_{i})$
$\displaystyle\qquad
s_{i,r}=\sqrt{N_{u}\times\operatorname{Q}(-\mu_{i})\times\big{(}1-\operatorname{Q}(-\mu_{i})\big{)}}$

With the equation in (10), $\mathrm{E}[E_{ReBaCS}]$ is calculated as:

$\displaystyle\mathrm{E}[err_{\text{ReBaCS}}]=$
$\displaystyle\Pr(A=0)\times\Pr\left(\frac{N_{u}}{2}\leq\gamma_{0}\leq
N_{u}\right)$
$\displaystyle+\Pr(A=1)\times\Pr\left(0\leq\gamma_{1}<\frac{N_{u}}{2}\right)$
$\displaystyle=$
$\displaystyle\Pr(A=0)\times\Big{(}\operatorname{Q}\left(\frac{\frac{N_{u}}{2}-m_{0,r}}{s_{0,r}}\right)-\operatorname{Q}\left(\frac{N_{u}-m_{0,r}}{s_{0,r}}\right)\Big{)}$
$\displaystyle+\Pr(A=1)\times\Big{(}\operatorname{Q}\left(\frac{-m_{1,r}}{s_{1,r}}\right)-\operatorname{Q}\left(\frac{\frac{N_{u}}{2}-m_{1,r}}{s_{1,r}}\right)\Big{)}$

In conclusion, by knowing the expectation of all users decision variables and
the number of users, expectation of ReBaCS error can be determined.

###  4.2 CoBaCS

As we discussed earlier, sometimes users are not 100% sure about the
correctness of their answers, so they choose the option they feel more
confident about. By knowing the meta-cognitive ability of each user, we can
estimate how accurately their confidence leads them. In other words, we can
answer this question: "Is the user’s confidence a proper estimator of his/her
accuracy?". If we select volunteers based on their meta-cognitive ability and,
in every question, ask them to submit their confidence as well as their
answers, we can improve system performance. So, in the Confidence-Based Crowd-
Sourcing system, CoBaCS, first we show the volunteers pre-test questions. As
we mentioned earlier, both their response and confidence are recorded. At the
end of this phase, users are selected due to their both performance and meta-
cognition ability. For measuring performance, the context and the complexity
of the pre-test and test phase questions must be the same. Still, according to
[4, 5] meta-cognitive ability is similar across domains. So, the developed
tasks in the literature can be applied, and there is no need to re-design
them.

In the test phase, the same as the pre-test, users report their confidence in
being correct in each question. After collecting users’ answers and
confidence, the system states the crowd answer by Weighted Majority Voting
(WMV) algorithm. This algorithm works similarly to MV with one difference;
answers are multiplied with their corresponding confidence. Using WMV, a user
with low confidence has a lower contribution to the crowd’s answer.

####  4.2.1 Error Calculation

In CoBaCS, we assume $r_{n,q}$ will be either 1 or -1 as

$r_{n,q}=\begin{cases}-1&\text{if the $n$th user has chosen first option}\\\
1&\text{if the $n$th user has chosen second option}\end{cases}$ (12)

Users are asked to report their confidence as discrete levels from 1 (i.e.,
lowest confidence) to $c_{max}$ (i.e., highest confidence). We denote the
reported confidence of user $n$ to the $q$th question by $c_{n,q}$. Then, we
have

$R_{\text{WMV}_{q}}=\begin{cases}-1&\text{if}\ \sum_{n=1}^{N}r_{n,q}\times
c_{n,q}<0\\\ 1&\text{o.w.}\end{cases}$ (13)

Similar to the error calculation of ReBaCS, CoBaCS performance is the fraction
of questions that the crowds’ consensus leads the system to the correct
answer. The expectation of error in this system is as follows:

$\displaystyle\mathrm{E}_{N_{Q}}[err_{\text{CoBaCS}}]=$
$\displaystyle\Pr(A=-1)\times\Pr(R_{\text{WMV}}=1|A=-1)$
$\displaystyle+\Pr(A=1)\times\Pr(R_{\text{WMV}}=-1|A=1)$ $\displaystyle=$
$\displaystyle\Pr(A=-1)\times\Pr\left(\sum_{n=1}^{N}r_{n}\times c_{n}\geq
0\left|A=-1\right.\right)$
$\displaystyle+\Pr(A=1)\times\Pr\left(\sum_{n=1}^{N}r_{n}\times
c_{n}<0\left|A=1\right.\right)$

Due to the independency of $r_{n}\times c_{n}$ across $n\in\\{1,...,N_{u}\\}$
and their limited variance, the sum of these variables can be approximated
with a normal random distribution as

$\displaystyle\lambda_{i}$ $\displaystyle=\sum_{n=1}^{N_{u}}\big{(}r_{n}\times
c_{n}\big{)}|\big{(}A=i\big{)},\qquad i\in\\{-1,1\\}$ (14)
$\displaystyle\qquad\lambda_{i}\sim\mathcal{N}(m_{i,c},s_{i,c})$
$\displaystyle\qquad m_{i,c}=N_{u}\times\mathrm{E}[r\times c|A=i]$
$\displaystyle\qquad s_{i,c}=\sqrt{N_{u}\times\mathrm{Var}[r\times c|A=i]}$

For determining the error of CoBaCS, the confidence level threshold of users
is needed as well as their first-order decision variables and the number of
users. By having these parameters, the expected error of CoBaCS can be written
as follows.

$\displaystyle\mathrm{E}[err_{\text{CoBaCS}}]=$
$\displaystyle\Pr(A=-1)\times\Pr(\lambda_{1}>=0)$ (15)
$\displaystyle+\Pr(A=1)\times\Pr(\lambda_{2}<0)$ $\displaystyle=$
$\displaystyle\Pr(A=-1)\times\operatorname{Q}(\frac{-m_{1,c}}{s_{1,c}})$
$\displaystyle+\Pr(A=1)\times(1-\operatorname{Q}(\frac{-m_{2,c}}{s_{2,c}}))$

###  4.3 Evaluations Environment

In this section, we evaluate the performance of these systems and compare them
in terms of error. To do so, we simulate the recruited population of these
cases. For computing error of ReBaCS, we consider that the recruited
population is made of users with various performances. We choose the
performance of each user randomly. With having users’ performances, the
parameters of Type I decision making described in Section 3 can be computed
($\mu_{i,n}\quad i\in\\{0,1\\},n\in\\{1,..,N\\}$). Then, the expected
parameters of decision-making will be obtained ($\mu_{i,r}\quad
i\in\\{0,1\\}$).

We assume each user has either low, medium or the high meta-cognitive ability
for meta-cognition ability without loss of generality. A user with low meta-
cognitive ability is one who rates his or her confidence low in the time of
being correct and high when the answer is wrong (AUROC2 $\approx 0$). The high
meta-cognitive user acts the contrary: low confidence in wrong answers and
high confidence in correct answers (AUROC2 $\approx 1$). The medium meta-
cognitive user is one who consistently rates confidence at the medium level
(AUROC2 $=0.5$). The users’ meta-cognition ability is chosen randomly.

We assumed that 200 questions and 100 users were participating in both ReBaCS
and CoBaCS systems. We have chosen these numbers to have enough questions to
calculate the system’s error correctly. Also, We aimed to have a reasonable
number of users remain in the system after filtering. Considering 100,000
samples of the population with different performances and meta-cognitive
ability, Figure 2 is obtained. As you can see in this figure, without having
any filters on performance and meta-cognitive ability, based on performance,
CoBaCS is in 52.38 % of viable populations better the ReBaCS. Although in some
populations, CoBaCS outperforms ReBaCS, in general, without any pre-testing
and filtering, there is no significant difference between ReBaCS and CoBaCS
systems.

Figure 2: Performance of ReBaCS and CoBaCS in various populations: without
filtering users at the beginning of the task, CoBaCS has no superiority over
ReBaCS.

In figure 3, we have demonstrated the effect of different filters in the
pretest phase on systems’ performance. Users are filtered according to their
performance in the horizontal direction and their meta-cognition ability in
the vertical direction. According to this figure, in the populations where the
experts (users with high performance) are present, it is wiser to use ReBaCS
if we have no hard meta-cognition filter. But, when the users are not experts
and have low performance, we can obtain better performance by filtering very
low metacognitive users using CoBaCS. In the first column of figure 3, there
are no criteria for users’ performance, and with existing users, even with
performance near 10%, the CoBaCS is much better when we filter very low
metacognitive users.

(a)

(b)

Figure 3: Difference between performance of ReBaCS and CoBaCS in different
populations: with no low meta-cognition ability users present in the system,
the CoBaCS will outperforme ReBaCS, even if very low performance users exist
in the population. But when experts are available, the ReBaCS has better
performance than CoBaCS. a) the histogram of the difference between
performance of ReBaCS and CoBaCS with applying various filters. b) the
percentage of the populations which CoBaCS outperformes ReBaCS.

In conclusion, according to the simulations’ results, when the CoBaCS is used,
and the population does not include experts, it is important to filter people
with low meta-cognition ability. It is noteworthy to mention that for
filtering users based on their performance, we should take pre-tests in the
context of the main task, which is challenging. On the other side, the
metacognition ability is not task dependent [4, 5]; we can compute users’
metacognition first, and then assign different taks to the ones with no low
metacognition ability. Also, in tasks with high difficulty, finding users who
are expert is hard and time-consuming. But, if they are available with hard
metacognition filters, CoBaCS has a lower error. Therefore, CoBaCS can
increase performance and reduce the cost in the time of difficulty. In Section
5, we will examine our proposed system with a real-world task.

##  5 Experimental Results and Analysis

###  5.1 Tasks and Procedures

We consider three tasks to study the role of meta-cognitive ability as a
filter and the effect of confidence reporting in crowd-sourcing systems. The
first task is a tweet task in which the participants are asked to guess the
gender of the Twitter account owner by seeing a few tweets of him/her. The
tweet task is designed to test the performance of a confidence-based crowd-
sourcing system, CoBaCS. Before the tweet task, we ran a pre-tweet task
designed to evaluate the gender bias in different topics to rank the questions
in the tweet task based on their difficulty. The other task is a word
recognition memory task which was used to measure the meta-cognitive ability
of subjects [34, 35]. The pre-tweet task was an online test using Google
Forms. The memory and tweet tasks were computerized and programmed in MATLAB
(Mathworks), using COGENT 2000 toolbox
111http://www.vislab.ucl.ac.uk/cogent.php.

Subjects: Eighty-six participants completed the pre-tweet task. Eighty-six
participants (42 women, 44 men) with a mean age of 24.24 (range = 17 to 37)
were instructed to do the other two tasks. In this study, there is no excluded
subject. We paid subjects a fixed amount of money (15 thousand Tomans), so the
incentive for all the subjects was the same. In addition, we ordered the
participants according to their performance in the Memory task plus twice
their performance in the tweet task. We paid a bonus (100 thousand Tomans) to
the top three participants who were higher in our ordering.

Memory task: The memory task was a Persian word recognition memory test based
on [34] and adapted from [35]. The study had two phases: encoding and recall.
During the encoding phase, 100 words from a set containing 200 words were
presented on the screen in a random order, separately and sequentially. In the
recall phase, immediately after the first phase, all 200 words were displayed
sequentially. Subjects decided whether each word was shown in the encoding
phase or not by pressing 1 (yes) or 2 (no) on the keypad using their left
hand. Next, they reported their confidence in being correct on a scale of 1
(low confidence) to 5 (high confidence) using a number pad with their right
hand. The procedure is shown in Figure 4. Individuals did not receive any
feedback on the correctness of their responses.

In the memory task, average memory performance of our subjects was 67.53%
(standard deviation 7.42%, range 53% -86.5% ), comparable to the results
reported in [34] (mean accuracy 71%, standard deviation 9%, range 57-91%) and
the results reported in [35] (mean accuracy 71%, standard deviation 1%, range
68 - 73 %).

The average meta-cognitive ability of subjects was 0.62 (standard deviation
0.06, range 0.508- 0.784). Reported results on meta-cognitive accuracy were
measured by area under ROC Type II curve, AUROC2 [33, 4, 32].

Figure 4: Memory Task. This task had two phases: 1) In the encoding phase,
subjects were asked to memorize half of the words which were chosen randomly
from a word set. 2) In the recall phase, after the presentation of each word
from the word set, the subjects reported whether the word was seen in the
encoding phase or not. After the response, they scaled their confidence about
the previous answer. The Persian words from left to write mean; the sun
rising, flight, fountain

Pre-tweet task. The goal of this task was to investigate how difficult it is
to guess the gender of a Twitter account based on the topics that have been
discussed in their tweets. We designed an online questionnaire using Google
Forms, including 17 titles. Sixty eight participants answered the questioner;
they were instructed to report _male_ or _female_ for each of the contents.
There was not any time limit to answer the questionnaire.

We define the gender of a tweet based on the gender that more than 60% of the
participants picked. A simple tweet is one in that the gender of the tweet is
the same as the actual gender of its Twitter account. By the results of this
task, we categorized selected tweets for the next task, the tweet task, into
four groups based on the actual gender of the writer of tweets (female or
male) and the difficulty of gender recognition according to the discussed
topic (simple or complicated).

Tweet task. We collected a set of thousands of Twitter accounts whose tweets
were in Persian. Then, one hundred accounts were selected, with more than one
hundred followers and one hundred followings, and also had tweeted more than
five hundred tweets. The last condition is to make sure that the accounts’
owners are familiar with twitting and that we have enough tweets from them to
choose for our task. We omitted the accounts that had a massive number of
tweets, but they have joined Twitter in recent months from our selections to
avoid bots. Half of the accounts’ owners were women, and the other half were
men. We chose three of the tweets of these one hundred accounts sudo-random;
thus, we had three hundred tweets in the Persian language. The selection of
accounts and tweets was that every four categories of Twitter accounts
(female/male and difficult/simple) occurred exactly 25 times (a quarter of the
total number of accounts). We call an account simple, and most of its selected
tweets are simple (based on the definition in the pre-tweet task).

At first, all the participants were asked to submit their gender and age.
Participants of the tweet task saw the three tweets of each account on the
same page on the monitor, and they decided whether the account belonged to a
male or female by pressing the key "1" or "2". Then, they reported their
confidence of correctness by pressing keys 1 (i.e., lowest confidence) to 5
(i.e., highest confidence) on the number pad. There was no time limit to
report the decision and confidence. At the end of the task, participants
reported their estimation of the number of correct answers; then, they saw
their actual performance.

In the tweet task, the average performance of our subjects was 61.79%
(standard deviation 4.56%, range 49%-72% ). The mean of meta-cognitive ability
was 0.58 (standard deviation 0.07, range 0.445- 0.777). In the literature,
There is no similar task to our tweet task, so a comparison of results is
impossible.

Figure 5: Tweet Task. This Task had 100 questions, and each question contained
three tweets, all from an account. The subjects were asked to guess the gender
of the owner of the account. Then, they rated their confidence in being
correct from 1 to 5.

###  5.2 Behavioral Results

Data Analysis: We would like to study the influence of the performance filter
and Metacognition filter on the error of two systems; ReBaCS and CoBaCS. Thus
we needed two scores for each participant regarding performance and
metacognitive ability. We saw that participants’ performance in the first
thirty trials of the tweet task was significantly correlated (Pearson
correlation $=0.59$, $p<0.05$) with their performance in the remaining
questions, the last seventy, of the tweet task. Therefore, we used the
performance in the first thirty questions (including 15 questions about men
tweet applicants and 15 questions about women tweet applicants) of the tweet
task as each participant’s performance score. For the score of metacognitive
ability, we used the AUROC2 for each participant in the memory task. It is
noticeable that the first thirty trials of the tweet task were not enough to
measure metacognitive ability in the tweet task to use it as the score of
metacognitive ability; thus, we measured this ability in the memory task.

Then, we randomly sampled 60% of participants, and we used four filters for
performance ability ($0.4,0.5,0.6,0.7$) and three filters for metacognitive
ability ($0.5,0.6,0.7$). Thus we applied twelve coupled filters on the 60
sampled participants. For example, for the coupled filter (performance 0.4,
metacognition 0.5), we excluded the participants whose performances were lower
than 0.4 and whose metacognitive abilities were lower than 0.5. To achieve the
performance of the ReBaCS system, we saw the performance of the remaining
people in the society after filtering in the last seventy questions in the
tweet task according to the majority voting; society’s answer to each question
was the answer of the majority. Similarly, to achieve the performance of the
CoBaCS system, we applied the WMV algorithm to measure the performance of the
filtered society. The difference in society performance between the two MW/WMV
algorithms and the 100% performance showed the error of two crowd-sourcing
systems, ReBaCS and CoBaCS. In this way, we estimated the error of two crowd-
sourcing systems with twelve filters for a randomly sampled society with 60
participants. We repeated the above process 100 times, randomly sampled a new
society with 60 percent, and measured the error of two systems after
implementing the twelve filters. The results of this analysis is shown in
Table 3 and Table 4. After obtaining the results, we ran a one-tailed T-test
to check if the ReBaCS error was significantly greater than the CoBaCS error.
The p-value for the first meta filter (meta>0.5) and performance greater than
40% was 0.03. For the same meta condition and performance greater than 70%,
the p-value was 0.01. For the rest, this value was below $2e-5$. Therefore,
for all conditions, the error of CoBaCs is significantly less than ReBaCS in
this experiment. To test our hypothesis with the filter effect, we used a one-
tailed T-test, and the result showed that regardless of the filter, the error
of ReBaCS is significantly greater than the CoBaCS (p-value = $2e-100$).

Table 3: RebaCS and CoBaCS Experiment Outcomex: Comparison of the Error of Systems. Filter Parameters | Min of Performance
---|---
40% | 50% | 60% | 70%
Min of Meta | 0.5 | ReBaCS | 38.5 $\pm$ 1.5 | 32.5 $\pm$ 1.6 | 32.2 $\pm$ 1.9 | 32.2 $\pm$ 2.6
CoBaCS | 32.0 $\pm$ 1.5 | 31.8 $\pm$ 1.4 | 31.5 $\pm$ 1.9 | 32.0 $\pm$ 2.8
0.6 | ReBaCS | 32.9 $\pm$ 2.1 | 33.6 $\pm$ 2.2 | 33.7 $\pm$ 2.2 | 34.9 $\pm$ 3.4
CoBaCS | 30.9 $\pm$ 2.1 | 31.9 $\pm$ 2.1 | 32.5 $\pm$ 2.3 | 33.9 $\pm$ 3.8
0.7 | ReBaCS | 37.1 $\pm$ 2.7 | 37.6 $\pm$ 2.8 | 41.5 $\pm$ 18.5 | -
CoBaCS | 32.5 $\pm$ 3.0 | 37.7 $\pm$ 3.5 | 37.0 $\pm$ 20.3 | -
Table 4: RebaCS and CoBaCS Experiment Outcomes: Mean of the number of subjects remaining after filtering Filter Parameters | Min of Performance
---|---
40% | 50% | 60% | 70%
Min of Meta | 0.5 | 52 | 49.6 | 36 | 8.4
0.6 | 30.9 | 29.8 | 22.3 | 5.4
0.7 | 4.4 | 4.2 | 1.7 | 0

Discussion. As you can see in tables 3 and 4, the mean of error of RebaCS is
greater than the other system in almost every case, Even in the first filter
(meta>0.5, performance>40%) which is equal to now having any filter (all the
population meet these criteria). It is noteworthy to mention that this
experiment tests our proposed system in one of the populations simulated in
the previous section. Based on our simulation and the results shown in 3, we
did not expect to obtain a better performance of CoBaCS in all the cases.
Still, we wanted to test if the performance of CobaCS significantly exceeds
ReBaCS.

In [36], it is mentioned that the subjects’ confidence is related to their
response time; greater response time leads to less confidence. If this is
true, we can measure the response time and not ask the users to rate their
confidence. During our task, we measured the users’ response time. According
to the gathered data, in the memory task, this hypothesis is true (correlation
coefficient is negative, p-value<0.05 for 71 users). On the other hand, in the
tweet task, the correlation coefficient for just 42 users out of 86 is
negative. This can be because the length of tweets is different, and each
person has a different reading speed. Therefore, the findings in [36] cannot
be applied to all the tasks.

As we mentioned in 2, the DK effect (Dunning-Kruger effect) refers to the
cognitive bias in which worthy person rates her or his performance low, and
the person with actual low performance reports that she or he had done the
task properly. Also, in [26] is mentioned that using weighted majority voting
in a population where the DK effect exists leads to more error compared with a
majority voting-based system. To test whether this finding can be applied to
our system, we asked users to rate their performance at the end of tweet
tasks. As Figure 6 shows, in our population, the DK effect exists, but the
CoBaCS performance was significantly better than the ReBaCS.

Figure 6: DK effect: users who guessed their performance was below 50%, their
actual performance was greater than 50%. Also, users who reported their
performance greater than 70% have actually performed with less than 70%
accuracy in most cases.

##  6 Conclusions

In this paper, we first studied crowd-sourcing systems. Then, we proposed our
method, CoBaCS, which asks the users for not only their responses but also
their confidence. Furthermore, this system did not include heavy mathematical
calculations and can be implicated and run easily in online and offline crowd-
sourcing systems. Based on the research done in the field of meta-cognition,
we measured the meta-cognition ability of the users and then used this
measurement to select proper users. According to our findings, both
mathematically and via a real-world experiment, our proposed system outperform
the most popular crowd-sourcing system (ReBaCS). As a result of our
simulation, when the task is hard, and there are not enough experts present in
the system, using CoBaCS is a wiser choice. Also, because the meta-cognition
ability is not task-dependent and can be measured once a user is asked a
variety type of questions, one can use CoBaCS and filter users with low meta-
cognition.

## References

  * [1] Maniscalco, B. & Lau, H.  Signal detection theory analysis of type 1 and type 2 data: meta-d, response-specific meta-d, and the unequal variance sdt model.  In _The cognitive neuroscience of metacognition_ , 25–66 (Springer, 2014).
  * [2] Howe, J.  The rise of crowdsourcing.  _Wired magazine_ 14, 1–4 (2006).
  * [3] Fleming, S. M.  Hmeta-d: hierarchical bayesian estimation of metacognitive efficiency from confidence ratings.  _Neuroscience of consciousness_ nix007 (2017).
  * [4] Veenman, M. V., Elshout, J. J. & Meijer, J.  The generality vs domain-specificity of metacognitive skills in novice learning across domains.  _Learning and instruction_ 7, 187–209 (1997).
  * [5] Mazancieux, A., Fleming, S. M., Souchay, C. & Moulin, C.  Retrospective confidence judgments across tasks: domain-general processes underlying metacognitive accuracy (2018).
  * [6] Sheng, X., Tang, J., Xiao, X. & Xue, G.  Sensing as a service: Challenges, solutions and future directions.  _IEEE Sensors journal_ 13, 3733–3741 (2013).
  * [7] Faggiani, A., Gregori, E., Lenzini, L., Luconi, V. & Vecchio, A.  Smartphone-based crowdsourcing for network monitoring: opportunities, challenges, and a case study.  _IEEE Communications Magazine_ 52, 106–113 (2014).
  * [8] Liu, Y. _et al._ A crowdsourcing based mobile image translation and knowledge sharing service.  In _9th International Conference on Mobile and Ubiquitous Multimedia, Limassol, Cyprus_ (2010).
  * [9] Ntalianis, K., Tsapatsoulis, N., Doulamis, A. & Matsatsinis, N.  Automatic annotation of image databases based on implicit crowdsourcing, visual concept modeling and evolution.  _Multimedia Tools and Applications_ 69, 397–421 (2014).
  * [10] Von Ahn, L. & Dabbish, L.  Labeling images with a computer game.  In _Proceedings of the SIGCHI conference on Human factors in computing systems, Gaithersburg, Maryland, USA_ (2004).
  * [11] Ribeiro, F., Florencio, D. & Nascimento, V.  Crowdsourcing subjective image quality evaluation.  In _2011 18th IEEE International Conference on Image Processing_ (Brussels, Belgium, 2011).
  * [12] Ghadiyaram, D. & Bovik, A. C.  Massive online crowdsourced study of subjective and objective picture quality.  _IEEE Transactions on Image Processing_ 25, 372–387 (2015).
  * [13] Shin, H., Chon, Y., Kim, Y. & Cha, H.  A participatory service platform for indoor location-based services.  _IEEE Pervasive Computing_ 14, 62–69 (2015).
  * [14] Chen, S., Li, M., Ren, K. & Qiao, C.  Crowd map: Accurate reconstruction of indoor floor plans from crowdsourced sensor-rich videos.  In _2015 IEEE 35th International conference on distributed computing systems_ (Columbus, OH, USA, 2015).
  * [15] McDuff, D., El Kaliouby, R. & Picard, R. W.  Crowdsourcing facial responses to online videos.  _IEEE Transactions on Affective Computing_ 3 (2012).
  * [16] Gatautis, R. & Vitkauskaite, E.  Crowdsourcing application in marketing activities.  _Procedia-Social and Behavioral Sciences_ 110, 1243–1250 (2014).
  * [17] Zhang, X. _et al._ Free market of crowdsourcing: Incentive mechanism design for mobile sensing.  _IEEE transactions on parallel and distributed systems_ 25, 3190–3200 (2014).
  * [18] Liu, W., Yang, Y., Wang, E., Han, Z. & Wang, X.  Prediction based user selection in time-sensitive mobile crowdsensing.  In _14th Annual IEEE International Conference on Sensing, Communication, and Networking (SECON), San Diego, USA_ (2017).
  * [19] Pu, L., Chen, X., Xu, J. & Fu, X.  Crowdlet: Optimal worker recruitment for self-organized mobile crowdsourcing.  In _IEEE INFOCOM 2016-The 35th Annual IEEE International Conference on Computer Communications_ (San Francisco, USA, 2016).
  * [20] Liu, X. _et al._ Cdas: a crowdsourcing data analytics system.  _Proceedings of the VLDB Endowment_ 5, 1040–1051 (2012).
  * [21] Nguyen, A. T., Lease, M. & Wallace, B. C.  Explainable modeling of annotations in crowdsourcing.  In _IUI_ (2019).
  * [22] Yadav, K., Kumaraguru, P., Goyal, A., Gupta, A. & Naik, V.  Smsassassin: crowdsourcing driven mobile-based system for sms spam filtering.  In _Proceedings of the 12th Workshop on Mobile Computing Systems and Applications_ (ACM, 2011).
  * [23] Kamar, E., Hacker, S. & Horvitz, E.  Combining human and machine intelligence in large-scale crowdsourcing.  In _Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems, Valencia, Spain_ , vol. 1 (2012).
  * [24] Yang, J., Drake, T., Damianou, A. & Maarek, Y.  Leveraging crowdsourcing data for deep active learning an application: Learning intents in alexa.  In _World Wide Web Conference, Lyon, France_ , 23–32 (2018).
  * [25] Abernethy, J. D. & Frongillo, R. M.  A collaborative mechanism for crowdsourcing prediction problems.  In _Advances in Neural Information Processing Systems_ (2011).
  * [26] Gadiraju, U., Fetahu, B., Kawase, R., Siehndel, P. & Dietze, S.  Using worker self-assessments for competence-based pre-selection in crowdsourcing microtasks.  _ACM Transactions on Computer-Human Interaction (TOCHI)_ 24, 30 (2017).
  * [27] Zhao, D., Li, X.-Y. & Ma, H.  How to crowdsource tasks truthfully without sacrificing utility: Online incentive mechanisms with budget constraint.  In _IEEE INFOCOM 2014-IEEE Conference on Computer Communications_ (Toronto, Canada, 2014).
  * [28] Bi, R., Zheng, X. & Tan, G.  Optimal assignment for deadline aware tasks in the crowdsourcing.  In _IEEE International Conferences on Big Data and Cloud Computing (BDCloud), Social Computing and Networking (SocialCom), Sustainable Computing and Communications (SustainCom)(BDCloud-SocialCom-SustainCom)_ (2016).
  * [29] Mazor, M., Charles, L., Mor, R. M. & Fleming, S.  Paradoxical evidence weighting in confidence judgments for detection and discrimination (2022).
  * [30] Livingston, J. A.  Metacognition: An overview. (2003).
  * [31] Fleming, S. M. & Daw, N. D.  Self-evaluation of decision-making: A general bayesian framework for metacognitive computation.  _Psychological review_ 124, 91 (2017).
  * [32] Maniscalco, B. & Lau, H.  A signal detection theoretic approach for estimating metacognitive sensitivity from confidence ratings.  _Consciousness and cognition_ 21, 422–430 (2012).
  * [33] Fleming, S. M. & Lau, H. C.  How to measure metacognition.  _Frontiers in human neuroscience_ 8, 443 (2014).
  * [34] Baird, B., Smallwood, J., Gorgolewski, K. J. & Margulies, D. S.  Medial and lateral networks in anterior prefrontal cortex support metacognitive ability for memory and perception.  _Journal of Neuroscience_ 33, 16657–16665 (2013).
  * [35] Sadeghi, S., Ekhtiari, H., Bahrami, B. & Ahmadabadi, M. N.  Metacognitive deficiency in a perceptual but not a memory task in methadone maintenance patients.  _Scientific reports_ 7, 1–8 (2017).
  * [36] Fiedler, K., Ackerman, R. & Scarampi, C.  Metacognition: Monitoring and controlling one’s own knowledge, reasoning and decisions.  _The psychology of human thought: An introduction_ 89–111 (2019).


11institutetext: Institute of Imaging & Computer Vision, RWTH Aachen
University, Germany
11email: {long.chen, martin.strauch<EMAIL_ADDRESS>
https://www.lfb.rwth-aachen.de/

# Semi-supervised Instance Segmentation with a Learned Shape Prior ††thanks:
This work was supported by the Deutsche Forschungsgemeinschaft (Research
Training Group 2416 MultiSenses-MultiScales).

Long Chen 11 0000-0002-5280-4727 Weiwen Zhang 11 Yuli Wu 11 Martin Strauch 11
0000-0001-6754-211X Dorit Merhof 11 0000-0002-1672-2185

###### Abstract

To date, most instance segmentation approaches are based on supervised
learning that requires a considerable amount of annotated object contours as
training ground truth. Here, we propose a framework that searches for the
target object based on a shape prior. The shape prior model is learned with a
variational autoencoder that requires only a very limited amount of training
data: In our experiments, a few dozens of object shape patches from the target
dataset, as well as purely synthetic shapes, were sufficient to achieve
results en par with supervised methods with full access to training data on
two out of three cell segmentation datasets. Our method with a synthetic shape
prior was superior to pre-trained supervised models with access to limited
domain-specific training data on all three datasets. Since the learning of
prior models requires shape patches, whether real or synthetic data, we call
this framework semi-supervised learning. The code is available to the
public111https://github.com/looooongChen/shape_prior_seg.

###### Keywords:

Semi-supervised Instance segmentation Shape prior Variational autoencoder Edge
loss

##  1 Introduction

Instance segmentation, where many instances of an object have to be segmented
in one image, is the basis of several practically relevant applications of
computer vision, such as cell tracking [1]. Many approaches [2, 3, 4] have
been proposed for instance segmentation, the majority of which are based on
supervised learning. The practical applicability of these methods is often
limited by the lack of a large training dataset with manually outlined
objects. Here, we introduce an instance segmentation approach that only relies
on a shape prior which can be learned from a considerably smaller number of
training samples or even synthetic data.

The shape is one of the most informative cues in object segmentation and
detection tasks. Anatomically constrained neural networks (ACNNs) [5] improve
segmentation results by including a shape prior for model regularization. For
segmentation refinement, a shape prior has been used by [6] as a separate
post-processing step. Segmentations generated by the shape prior model are
reconstructed to the original MRI images through several convolutional layers
in [7]. By minimizing the reconstruction error, the segmentation model can be
trained in an unsupervised fashion. All these works report promising results,
but are limited to cases where object position and extent are roughly the same
in all images, such as for the cardiac images in [5], the lung X-ray images in
[6] and the brain MRI scans in [7]. To our knowledge, this is the first work
considering instance segmentation based on a shape prior, i.e. we detect and
segment multiple, scattered object instances. Similar to [8], we use the
spatial transformer [9] to localize objects. The main advantage of using the
spatial transformer lies in its differentiability, making the whole framework
end-to-end trainable.

The main contributions of this work are: We propose (1) an semi-supervised
instance segmentation approach that seaches for target objects based a shape
prior, and (2) a novel loss computing the difference between two gradient
maps. This framework provides a way to achieve instance segmentation with a
small amount of manual annotations, or by utilizing unpaired annotations
(where the correspondence between annotations and images is unknown). We
compared our approach to the state-of-the-art supervised method, Mask R-CNN
[2], in different training scenarios. On three experimental datasets, our
approach is proved to be en par with a Mask R-CNN with full access to training
data, while it outperforms a pre-trained Mask R-CNN with limited access to
domain-specific training data.

##  2 Approach

As shown in Figure 1, our framework consists of three main parts: 1) the
localization network, 2) the spatial transformer [9], and 3) the patch
segmentation network. Based on the localization prediction, the spatial
transformer crops local patches and feeds them to the patch segmentation
network. The gradient maps of segmented patches are then stitched together.
The entire model is trained by minimizing the reconstruction error of the
gradient map.

During training, the model learns to predict the object position and to find
the correspondence between the image patch and the segmentation. The shape
prior model (gray part in Fig. 1; fixed during training) is guaranteed to
output a plausible shape, but the correspondence has to be learned by the
model itself.

Figure 1: Architecture of our framework: the localization network predicts the
object position and a presence score, based on which object patches are
cropped by a spatial transformer. A variational autoencoder with the decoder
part fixed (shape prior) is responsible for the patch segmentation. At last,
the gradient maps of segmented patches are stitched together. The model is
trained by minimizing the reconstruction loss of the gradient map with the KL-
divergence loss as regularization.

###  2.1 Localization network

The localization network consists of 8 convolutional layers and 4 max pooling
layers after every 2 convolutional layers. Given an image of size
$(H_{img},W_{img})$, the localization network will spatially divide the image
into an $(H_{img}/S_{cell},W_{img}/S_{cell})$ grid of cells, where $S_{cell}$
is the cell size and also the downsampling rate. Since 4 pooling layers with
stride 2 are used, we have $S_{cell}=16$.

Each cell is responsible to predict the presence of an object
$L_{presence}\in[0,1]$, its range described by the bounding box size
$(H_{obj},W_{obj})$ and the offset with respect to the cell center
$(O_{x},O_{y})$ (Figure 2(a)), with the implementation:

$\displaystyle L_{presence}=$ $\displaystyle\>sigmoid(f_{presence})$
$\displaystyle L_{scale}=$
$\displaystyle\>sigmoid(f_{scale})\cdot(S_{max}-S_{min})+S_{min}$
$\displaystyle L_{ratio}=$
$\displaystyle\>\exp(tanh(f_{ratio})\cdot\log(R_{max}))$
$\displaystyle(L_{x},L_{y})=$ $\displaystyle\>(0.5\cdot tanh(f_{x}),0.5\cdot
tanh(f_{y}))$

where $f_{[\cdot]}$ is the corresponding input feature map. $sigmoid(\cdot)$
and $tanh(\cdot)$ denote the sigmoid and tanh activation function. $S_{min}$,
$S_{max}$ and $R_{max}$ are hyperparameters, which are the minimal scale, the
maximal scale and the maximal aspect ratio, respectively. The position is
parameterized according to:

$\displaystyle(H_{obj},W_{obj})=$ $\displaystyle\>(L_{scale}\cdot
S_{cell}/\sqrt{L_{ratio}},\>L_{scale}\cdot S_{cell}\cdot\sqrt{L_{ratio}})$
$\displaystyle(O_{x},O_{y})=$ $\displaystyle\>(L_{x}\cdot
S_{cell},\>L_{y}\cdot S_{cell})$

It is worth mentioning that the maximal offset is $0.5\cdot S_{cell}$, which
means that an object will be detected by the cell in which its center lies.

Figure 2: (a) Demonstration of parameters of a bounding box. (b) Architecture
of the patch segmentation network, which is firstly trained with shape
patches. During the detector training, the decoder part is fixed and plays the
role of shape prior.

###  2.2 Patch crop and stitch

Given the location parameters obtained from the localization network, we use a
spatial transformer to crop local patches. The spatial transformer implements
the crop by sampling transformed grid points, which is differentiable,
enabling end-to-end training. The patch crop of the i-th cell can be described
by transform:

$\displaystyle T_{crop}^{i}$
$\displaystyle=\begin{bmatrix}W_{img}/W_{obj}^{i}&0&W_{img}\cdot(X_{cell}^{i}+O_{y}^{i})/W_{obj}^{i}\\\
0&H_{img}/H_{obj}^{i}&H_{img}\cdot(Y_{cell}^{i}+O_{x}^{i})/H_{obj}^{i}\\\
0&0&1\end{bmatrix}$

where $(X_{cell}^{i},Y_{cell}^{i})$ is the cell center.
$(O_{x}^{i},O_{y}^{i})$ and $(H_{obj}^{i},W_{obj}^{i})$ are the predicted
offset and size of the object. All cropped patches will be rescaled to size
$S_{patch}\times S_{patch}$ ($S_{patch}=32$ in this work) and segmented by the
patch segmentation network, as described in Section 2.3. After that, the
gradient map of segmented objects will be stitched together by adding up back
transformed patches through:

$\displaystyle T_{stitch}^{i}$
$\displaystyle=\begin{bmatrix}W_{obj}^{i}/S_{patch}&0&X_{cell}^{i}+O_{y}^{i}\\\
0&H_{obj}^{i}/S_{patch}&Y_{cell}^{i}+O_{x}^{i}\\\ 0&0&1\end{bmatrix}$

The gradient map is computed by applying the x- and y-directional Sobel filter
to the image and taking the square root of the summed square. The gradient map
is normalized to range 0 to 1. In this work, we use an input size of
$256\times 256$ for all experiments. Considering $S_{cell}=16$, 256 patches
are cropped in total.

###  2.3 Shape prior and patch segmentation network

Similar to [5, 6, 7], we employ a variational autoencoder (VAE) as our shape
model. As shown in Figure 2(b), the model is trained to reconstruct plausible
patch segmentation masks with the KL-divergence loss as
regularization.Compared to a standard autoencoder, a VAE learns a more
continuous latent space, which is expected to generate plausible new shapes
that do not appear in training data.

In this work, the VAE is trained with $32\times 32$ patches. The encoder and
decoder consist of 6 convolutional layers and 3 pooling/upsampling layers,
respectively. Based on our experiments, model training requires only a small
amount of data, especially when the shape variation is small. We train the
shape prior with either annotations from a single image or synthetic data
(Section 3).

After training, the decoder part will be used as the shape prior in the
detector (Figure 1). Its parameters will be fixed during the detector
training. The encoder will be reinitialized and trained together with the
localization network.

###  2.4 Training

The model is trained end-to-end by minimizing the gradient map reconstruction
error with the KL-divergence loss as regularization. In initial experiments,
we found the mean absolute/squared error (MAE/MSE) to be very unstable during
training: The shape prior model tends to generate distorted shapes or
degenerates into empty output. Thus, we propose the following novel loss:

$L_{edge}=1-\frac{\frac{1}{N}\sum_{i}min^{2}(G_{image}^{i},G_{reconstruction}^{i})}{\frac{1}{N}\sum_{i}G_{reconstruction}^{i}+\alpha}$
(1)

where $G_{image}$ and $G_{reconstruction}$ indicate the gradient map of the
image and the reconstructed gradient map. $N$ is the number of pixels. The
$min()$ operation are conducted pixelwise. The parameter $\alpha$ prevents the
model from pushing $G_{reconstruction}$ to zero and is set to 0.01
empirically.

Instead of optimizing the value of each pixel, as MSE and MAE, this loss
maximizes the proportion of the reconstructed gradient map under the image
gradient map. In addition, the square operator in the numerator is proved to
be crucial for stable training in our experiments. Our interpretation is that
the square operator modulates the back-propagated gradient with the
reconstructed gradient map, giving more emphasis to positions around the edge.

###  2.5 Pre- and post-processing

To reduce the influence of extreme values on the loss, we equalized the image
and the gradient map by clipping and streching. For all datasets, we truncated
the gradient map at 0.8 times the maximum and normalized the value to the
range 0 to 1. In addition, we also performed image equalization for the
Fluo-N2DH-SIM+ dataset due to the bright spots inside the cell (Figure 3). The
clip value was set to $1.2$ times the image mean.

As post-processing, we first filtered out predictions with $L_{presence}$
smaller than 0.1. Non-max suppression is then performed to eliminate duplicate
predictions: An instance mask is compared with another mask, when the
overlapping area is larger than $p_{non\\_max}=0.1$ with respect to its own
area. A mask is only retained if its score is the highest in all comparisons.

##  3 Experiments and results

###  3.1 Datasets and experiments

We evaluate our approach on three datasets: the BBBC006
dataset222https://data.broadinstitute.org/bbbc and two datasets Fluo-N2DH-SIM+
and PhC-C2DL-PSC from the cell tracking challenge [1]. In the following, we
use BBBC, FLUO and PHC as abbreviations. The BBBC dataset contains 768
microscopic images of human U2OS cells, while the FLUO (HL60 cells with
Hoechst staining) and PHC (pancreatic stem cells on a polystyrene substrate)
datasets are smaller with 215 and 202 annotated images.

For comparison, we also report the performance of the supervised method Mask
R-CNN. The following experiments are performed:

Ours-annotation: We first evaluate our approach with the shape prior learned
from manual annotations. We only took segmentation patches from one image.
Specifically, 67, 8 and 138 object patch masks were used for the BBBC, FLUO
and PHC shape model training. To model small shape changes and object
rotation, we performed rotation (in steps of 30 degrees) and elastic
deformation [11] to augment the training set. The scale range and maximal
aspect ratio was set to 2-3/3, 1-2/1.5 and 1-2/3, respectively.

Ours-synthetic: Since the objects are approximately circular, especially for
the BBBC and FLUO datasets, we could train the shape prior model with
synthetic data consisting of elastically deformed ellipses [11] with random
angle and major-minor axis ratio. The maximal major-minor axis ratio was 2,
1.5 and 3 for the BBBC, FLUO and PHC dataset, respectively.

MRCNN-scratch-one/full: We trained a Mask R-CNN from scratch using ResNet-50
backbone. The anchor box scale, aspect ratio and non-maximum suppression (NMS)
threshold were set to values equivalent to those used in our approach. Since
the Ours-annotation scenario can be considered as one image training, we also
trained a Mask R-CNN with one image for comparison.

MRCNN-finetune-one/full: Since the dataset in our experiments is small,
especially for FLUO and PHC, we pretrained the Mask R-CNN on the MS COCO
dataset333https://cocodataset.org/. Afterwards, we finetuned the model, with
only the head layers trainable, on the actual target dataset.

For the BBBC and PHC dataset, we cropped images to $256\times 256$ and
$128\times 128$ for training and test. All images were resized to $256\times
256$ for the network input. For the scenarios using one training image (Ours-
annotation, MRCNN-scratch-one, MRCNN-finetune-one), the images a01_s1,
02/t000, 02/t150 were used for BBBC, FLUO and PHC, respectively. MRCNN-
scratch-full and MRCNN-finetune-full used a01_s1-b24_s2, 02/t000-t149,
02/t150-t250 for training. Ours-synthetic requires no manual annotations. All
remaining images were kept for testing.

###  3.2 Results and discussion

We report the average precision444https://www.kaggle.com/c/data-science-
bowl-2018 (AP) over a range of IoU (intersection over union) thresholds from
0.3 to 0.9 as the evaluation score (Table 1). Our approach, including the
evaluation scenarios where the shape prior is learned from one image
annotation and synthetic data, outperforms the Mask R-CNN trained or finetuned
with one image, which shows the advantage of our approach in cases where few
or no annotations are available. Furthermore, our approach achieves comparable
results with the Mask R-CNN trained/finetuned with the full training set on
the BBBC and FLUO dataset, while the performance gap is apparent for the PHC
dataset.

While Mask R-CNN achieved the best mean AP (mAP) on the BBBC dataset, our
approach outperformed Mask R-CNN on the FLUO dataset by a relatively large
margin. The main reason is that the FLUO dataset is indeed a very small one
for Mask R-CNN training, even with finetuning. This again illustrates the
advantage of our method on small datasets.

Ground Truth

Ours-synthetic

Ours-annotation

MRCNN-finetune-one

MRCNN-finetune-full

Figure 3: Qualitative results: from top to bottom, the rows show the results
on the BBBC006, Fluo-N2DH-SIM+ and PhC-C2DL-PSC datasets, respectively.

On the PHC dataset, neither method performed particularly well. Both methods
tended to detect nearby objects as one if there was no clearly visible edge
between them. The average precision of our method in the low IoU range was
close to or better than that of Mask R-CNN. Figure 3 shows that our method
could detect most objects as well as the Mask R-CNN. However, our method has
been designed to heavily rely on the edge clue, so that the segmentation will
converge to strong edges. For the PHC dataset, the object boundaries do not
generally correspond to the strongest edges. This explains why objects were
undersegmented by our approach (Figure 3) and why the average precision
decreased rapidly with increasing IoU (Table 1).

The performance improvement through training the shape prior with manually
outlined shapes depends on the nature of the shape. On the FLUO dataset,
annotated data and synthetic data shape priors performed almost equally well,
while training with manual annotations was superior on the other two datasets,
even though only a few dozen shapes were used.

Table 1: Average precision ($AP$) over different $IoU$ for different datasets (the best two scores in bold). Experiments and abbreviations are introduced in Section 3.1. Dataset | $IoU$ | 0.3 | 0.4 | 0.5 | 0.6 | 0.7 | 0.8 | 0.9 | $\text{m}AP$
---|---|---|---|---|---|---|---|---|---
BBBC | Ours-annotation | .8345 | .8260 | .7977 | .7632 | .7083 | .6100 | .2660 | .6865
Ours-synthetic | .8171 | .8012 | .7641 | .7170 | .6525 | .5247 | .2042 | .6401
MRCNN-scratch-one | .6386 | .5934 | .5459 | .4769 | .3543 | .1759 | .0294 | .4020
MRCNN-scratch-full | .7901 | .7851 | .7708 | .7473 | .7128 | .6296 | .3374 | .6817
MRCNN-finetune-one | .7672 | .7524 | .7277 | .7020 | .6608 | .5492 | .1250 | .6121
MRCNN-finetune-full | .7997 | .7949 | .7851 | .7720 | .7521 | .6923 | .3485 | .7064
FLUO | Ours-annotation | .9605 | .9538 | .9312 | .8999 | .8228 | .6777 | .1332 | .7685
Ours-synthetic | .9600 | .9497 | .9336 | .8986 | .8324 | .6768 | .1378 | .7698
MRCNN-scratch-one | .0458 | .0324 | .0156 | .0018 | .0000 | .0000 | .0000 | .0014
MRCNN-scratch-full | .9333 | .9144 | .8703 | .7605 | .5765 | .2556 | .01073 | .6173
MRCNN-finetune-one | .8224 | .8133 | .7905 | .7389 | .5909 | .2404 | .0049 | .5716
MRCNN-finetune-full | .9361 | .9252 | .8955 | .8467 | .7265 | .4115 | .0197 | .6802
PHC | Ours-annotation | .6840 | .6034 | .4035 | .1468 | .0233 | .0028 | .0000 | .2662
Ours-synthetic | .6471 | .5611 | .3605 | .1326 | .0219 | .0027 | .0000 | .2466
MRCNN-scratch-one | .1124 | .0991 | .0847 | .0668 | .0353 | .0049 | .0000 | .0576
MRCNN-scratch-full | .6332 | .6001 | .5226 | .4467 | .2981 | .1079 | .0023 | .3730
MRCNN-finetune-one | .1647 | .1602 | .1460 | .1146 | .0633 | .0108 | .0000 | .0942
MRCNN-finetune-full | .6551 | .6380 | .5855 | .5014 | .3425 | .1144 | .0007 | .4053

##  4 Conclusion and outlook

We have proposed an instance segmentation framework which searches for target
objects in images based on a shape prior model. In practice, this allows
segmenting instances with a very limited amount of annotations, segmenting
synthesizable shapes without any annotation, as well as reusing object
annotations from other datasets.

The main limitation of our approach lies in the dependency on the edge cues.
Images should have a relatively clear background, which is, however, the case
for many biomedical datasets00footnotemark: 0. Future work will focus on
including area-based information, which will make our approach applicable to
further datasets, e.g. in cases where edges and object boundaries do not
always coincide.

## References

  * [1] Ulman, V., et al.: An Objective Comparison of Cell-tracking Algorithms. Nature Methods, 14, 1141-1152 (2017)
  * [2] He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask R-CNN. In: 2017 ICCV, 2980-2988
  * [3] Schmidt, U., Weigert, M., Broaddus, C., Myers, E.W.: Cell Detection with Star-Convex Polygons. In: 2018 MICCAI, 26–273
  * [4] Chen, L., Strauch, M., Merhof, D.: Instance Segmentation of Biomedical Images with an Object-Aware Embedding Learned with Local Constraints. In: 2019 MICCAI, 451-459
  * [5] Oktay, O., et al.: Anatomically Constrained Neural Networks (ACNNs): Application to Cardiac Image Enhancement and Segmentation. IEEE Transactions on Medical Imaging, 37(2), 384-395 (2018)
  * [6] Larrazabal, A. J., Martinez, C., Ferrante, E.: Anatomical Priors for Image Segmentation via Post-processing with Denoising Autoencoders. In: 2019 MICCAI, 585-593
  * [7] Dalca, A. V., Guttag, J., Sabuncu, M. R.: Anatomical Priors in Convolutional Networks for Unsupervised Biomedical Segmentation. In: 2018 CVPR, 9290-9299
  * [8] Crawford, E., Pineau, J.: Spatially Invariant Unsupervised Object Detection with Convolutional Neural Networks. In: 2019 AAAI, 3412-3420
  * [9] Jaderberg, M., Simonyan, K., Zisserman, A., Kavukcuoglu, K.: Spatial Transformer Networks. In: 2015 NIPS, 2017-2025
  * [10] Kingma, D. P., Welling, M.: Auto-Encoding Variational Bayes. In: 2014 ICLR
  * [11] Simard, P. Y., Steinkraus, D., Platt, J. C.: Best Practices for Convolutional Neural Networks Applied to Visual Document Analysis. In: Proceedings of the Seventh International Conference on Document Analysis and Recognition, pp. 958. IEEE (2003)


# Comment on ”Effects of shear methods on shear strengths and deformation
modes of two typical transition metal carbides and their unification”

Marcin Maździarz<EMAIL_ADDRESS>Institute of Fundamental Technological
Research Polish Academy of Sciences, Pawińskiego 5B, 02-106 Warsaw, Poland

###### Abstract

Recently, Chuanying Li, Tao Fu, Xule Li, Hao Hu, and Xianghe Peng in [Phys.
Rev. B 107, 224106] investigated the mechanical behavior of cubic HfC and TaC
under simple shear (SS) and pure shear (PS) using first-principles
calculations. Unfortunately, the paper contains some serious and fundamental
flaws in the field of continuum mechanics and nanomechanics. The results
presented appear to be qualitatively and quantitatively incorrect, they would
be correct if we were in the small/linear deformation/strain regime, which we
are not. A correct description therefore requires a finite/nonlinear
deformation/strain apparatus.

##  I Introduction

In the paper entitled ”Effects of shear methods on shear strengths and
deformation modes of two typical transition metal carbides and their
unification” by Chuanying Li et al. [1] cubic HfC and TaC crystals were
subjected to simple (SS) and pure shear (PS) by DFT simulations. The paper
mistakes concepts from the continuum mechanics, incorrectly enforces
deformations, and it appears that the computational results are qualitatively
and quantitatively incorrect.

  * •

The Authors write: ”During deformation, the lattice vectors need to be
changed, … Hence, the deformation can be imposed by transforming the i-1th
step lattice vector matrix Ri-1 to the deformed ith step lattice vector matrix
Ri as follows [36]:

$\textbf{R}^{i}=\textbf{R}^{i-1}\left[\textbf{I}+\left(\begin{array}[]{ccc}{0}&{0}&{0}\\\
{0}&{0}&{0}\\\ {0}&{\Delta{\varepsilon_{zy}}}&{0}\\\
\end{array}\right)\right]."$ (1)

This formula is incorrect.

In the non-linear mechanics of crystals there is well know hypothesis, called
the _Cauchy-Born rule_ [2], which assumes that under a homogeneous macroscopic
deformation, the primitive Bravais lattice vectors of a 3D crystal deform in
an affine manner via a 3$\times$3 matrix F:

$\textbf{a}_{i}=\textbf{F}\textbf{A}_{i},$ (2)

where ai stands for a spatial lattice vectors, Ai reference vectors and F the
deformation gradient [3].

If we even wanted to do it in steps, it is clear from the tensor calculus
point of view that Eq.1 should be as follows:

${{\bf R}^{def}}={\bf F}\cdot{{\bf R}^{ini}}.$ (3)

The vector, or matrix of vectors, is multiplied by the matrix on the left.

Moreover, the components of the deformation gradient F are not, in general,
components of the strain tensor.

  * •

The Authors write further: ”For PS, after each $\varepsilon_{zy}$, the atomic
coordinates and the other five independent strain components (except
$\varepsilon_{zy}$) are optimized simultaneously to reach a stress state with
$\sigma_{xx}$ = $\sigma_{yy}$ = $\sigma_{zz}$ = $\sigma_{xy}$ = $\sigma_{zx}$
= 0 [35,37,38]. For SS, after applying $\varepsilon_{zy}$, the atomic
coordinates are optimized but remain $\varepsilon_{xx}$ = $\varepsilon_{yy}$ =
$\varepsilon_{zz}$ = $\varepsilon_{xy}$ = $\varepsilon_{zx}$ = 0 [14,39]. The
schematics of PS and SS are illustrated in Figs. 1(a) and 1(b), respectively.”

The above excerpt from the Authors’ text contains several misrepresentations.
In general, the _Lagrangian finite strain tensor_ E, also called the _Green-
Lagrangian strain tensor_ or _Green – St-Venant strain tensor_ is defined as:

${\bf E}=\frac{1}{2}({\bf F}^{T}{\bf F}-{\bf I}).$ (4)

and its linear approximation the _infinitesimal strain tensor_ $\varepsilon$,
also called the _Cauchy’s strain tensor_ , _linear strain tensor_ , or _small
strain tensor_ takes the form:

${\bf\varepsilon}=\frac{1}{2}({\bf F}^{T}+{\bf F})-{\bf I}.$ (5)

The classical _finite simple shear deformation_ (SS) is an isochoric plane
deformation defined by the deformation gradient tensor F in the following form
[4] (and this implies the form of the tensor E and $\varepsilon$):

$\textbf{F}^{\text{SS}}=\left[\begin{array}[]{ccc}{1}&{0}&{0}\\\
{0}&{1}&{\gamma}\\\ {0}&{0}&{1}\\\
\end{array}\right]\Rightarrow\textbf{E}^{\text{SS}}=\left[\begin{array}[]{ccc}{0}&{0}&{0}\\\
{0}&{0}&{\frac{\gamma}{2}}\\\
{0}&{\frac{\gamma}{2}}&{\frac{{\gamma}^{2}}{2}}\\\
\end{array}\right]\Rightarrow\mathbf{\varepsilon}^{\text{SS}}=\left[\begin{array}[]{ccc}{0}&{0}&{0}\\\
{0}&{0}&{\frac{\gamma}{2}}\\\ {0}&{\frac{\gamma}{2}}&{0}\\\
\end{array}\right].$ (6)

It can be seen here that $\textbf{E}^{\text{SS}}_{\text{zz}}\neq 0$. This
would be the case if we were using the small strain tensor
$\mathbf{\varepsilon}^{\text{SS}}_{\text{zz}}$. The deformations in this work
are not small. It is important to remember what the consequences of overusing
linear theories can be. Because F is not objective and hence small strain
tensor $\varepsilon$ also and a rigid rotation can induce any stresses in a
deformable body and they should not, see [5] .

The _pure shear deformation_ (PS) [6] is a deformation in which the body is
elongated in one direction while being shortened perpendicularly and is
defined by:

$\textbf{F}^{\text{PS}}=\left[\begin{array}[]{ccc}{1}&{0}&{0}\\\
{0}&{\beta}&{0}\\\ {0}&{0}&{\frac{1}{\beta}}\\\
\end{array}\right]\Rightarrow\textbf{E}^{\text{PS}}=\left[\begin{array}[]{ccc}{0}&{0}&{0}\\\
{0}&{\frac{{{\beta}^{2}-1}}{2}}&{0}\\\ {0}&{0}&{\frac{{{\beta}^{-2}-1}}{2}}\\\
\end{array}\right]\Rightarrow\mathbf{\varepsilon}^{\text{PS}}=\left[\begin{array}[]{ccc}{0}&{0}&{0}\\\
{0}&{\beta-1}&{0}\\\ {0}&{0}&{{\beta}^{-1}-1}\\\ \end{array}\right].$ (7)

The details of the PS in the rotated coordinate frame along the shear plane
can be found in [4]. The Authors’ PS is not in fact a PS, but another
deformation.

The _pure shear stress_ (PSS) is such a deformation for which the Cauchy
stress tensor is a pure shear stress tensor of the form $\sigma$ =
_$\tau$_($e_{2}$ $\otimes$ $e_{3}$ \+ $e_{3}$ $\otimes$ $e_{2}$) with _$\tau$_
$\in$ $\mathbb{R}$ [4]. To enforce such a deformation F must take the form of
a simple shear composed with a triaxial stretch:

$\mathbf{\sigma}^{\text{PSS}}=\left[\begin{array}[]{ccc}{0}&{0}&{0}\\\
{0}&{0}&{\tau}\\\ {0}&{\tau}&{0}\\\
\end{array}\right]\implies\textbf{F}^{\text{PSS}}=\left[\begin{array}[]{ccc}{a}&{0}&{0}\\\
{0}&{b}&{c{\eta}}\\\ {0}&{0}&{c}\\\
\end{array}\right]\Rightarrow\textbf{E}^{\text{PSS}}=\left[\begin{array}[]{ccc}{\frac{a^{2}-1}{2}}&{0}&{0}\\\
{0}&{\frac{b^{2}-1}{2}}&{\frac{bc\eta}{2}}\\\
{0}&{\frac{bc\eta}{2}}&{\frac{{\eta}^{2}c^{2}+c^{2}-1}{2}}\\\
\end{array}\right].$ (8)

  * •

In ”FIG. 2. Mechanical response of samples sheared along [1$\bar{1}$0](110).
(a), (b) Variations of stress components of HfC and TaC with
$\varepsilon_{zy}$ under SS and PS, respectively. …” the Authors presented the
results of their DFT shear computations.

The HfC and TaC crystals analyzed here have cubic symmetry and there is no
reason for them to behave differently in shear than other cubic crystals, see
[7]. The shear component of the stress during SS deformation, by the way for
PSS also, should have a _sine_ -like character. In order to verify this, the
HfC[1$\bar{1}$0](110) crystal was subjected to SS and PSS by _ab-initio_
calculations based on density functional theory (DFT) [8, 9] within the
pseudopotential plane-wave approximation (PP-PW) performed by the ABINIT [10]
code. The projector augmented wave method (PAW) pseudopotentials used for the
PBE [11] generalized gradient approximation (GGA) exchange–correlation
functionals (XC) were obtained from the PseudoDojo project [12]. The valence
electron configurations and the _cut-off_ energy used for Hf and C atoms were
consistent with those utilized in the PAW pseudopotentials. The calculation
accuracy settings correspond to those used in [13].

a) | b)
---|---
|
Figure 1: HfC [1$\bar{1}$0](110): a) Simple Shear, b) Pure Shear Stress

It can be seen from Figure 1 that the results presented here are substantially
different from those presented by the Authors in their Figure 2 for HfC. For
SS, the $\sigma_{yz}$ shear component of the stress has a proper _sine_ -like
character and the $\sigma_{yy}$ component of the stress is about twice as
large. Zeroing the normal stresses in the PSS reduces the shear stress and
introduces its asymmetry. These results also differ substantially from those
presented by the Authors.

###### Acknowledgements.

This work was partially supported by the National Science Centre (NCN –
Poland) Research Projects: No. 2021/43/B/ST8/03207. Additional assistance was
granted through the computing cluster GRAFEN at Biocentrum Ochota, the
Interdisciplinary Centre for Mathematical and Computational Modelling of
Warsaw University (ICM UW) and Poznań Supercomputing and Networking Center
(PSNC).

## References

  * Li  _et al._ [2023] C. Li, T. Fu, X. Li, H. Hu, and X. Peng, Effects of shear methods on shear strengths and deformation modes of two typical transition metal carbides and their unification, Phys. Rev. B 107, 224106 (2023).
  * Born and K. [1954] M. Born and H. K.,  _Dynamical Theory of Crystal Lattices_ (Oxford University Press, Oxford, 1954).
  * Clayton [2011] J. Clayton,  _Nonlinear Mechanics of Crystals_, Solid Mechanics and Its Applications (Springer Netherlands, 2011).
  * Thiel  _et al._ [2019] C. Thiel, J. Voss, R. J. Martin, and P. Neff, Shear, pure and simple, International Journal of Non-Linear Mechanics 112, 57 (2019).
  * Maździarz [2019] M. Maździarz, A comment on the article “Ab initio calculations of pressure-dependence of high-order elastic constants using finite deformations approach” by I. Mosyagin, A.V. Lugovskoy, O.M. Krasilnikov, Yu.Kh. Vekilov, S.I. Simak and I.A. Abrikosov [Computer Physics Communications 220 (2017) 20–30], Computer Physics Communications 235, 293 (2019).
  * Ogden [2013] R. Ogden,  _Non-Linear Elastic Deformations_ , Dover Civil and Mechanical Engineering (Dover Publications, 2013).
  * Ogata  _et al._ [2008] S. Ogata, Y. Umeno, and M. Kohyama, First-principles approaches to intrinsic strength and deformation of materials: perfect crystals, nano-structures, surfaces and interfaces, Modelling and Simulation in Materials Science and Engineering 17, 013001 (2008).
  * Hohenberg and Kohn [1964] P. Hohenberg and W. Kohn, Inhomogeneous electron gas, Phys. Rev. 136, B864 (1964).
  * Kohn and Sham [1965] W. Kohn and L. J. Sham, Self-consistent equations including exchange and correlation effects, Phys. Rev. 140, A1133 (1965).
  * Gonze  _et al._ [2020] X. Gonze, B. Amadon, G. Antonius, F. Arnardi, L. Baguet, J.-M. Beuken, J. Bieder, F. Bottin, J. Bouchet, E. Bousquet, N. Brouwer, F. Bruneval, G. Brunin, T. Cavignac, J.-B. Charraud, W. Chen, M. Côté, S. Cottenier, J. Denier, G. Geneste, P. Ghosez, M. Giantomassi, Y. Gillet, O. Gingras, D. R. Hamann, G. Hautier, X. He, N. Helbig, N. Holzwarth, Y. Jia, F. Jollet, W. Lafargue-Dit-Hauret, K. Lejaeghere, M. A. Marques, A. Martin, C. Martins, H. P. Miranda, F. Naccarato, K. Persson, G. Petretto, V. Planes, Y. Pouillon, S. Prokhorenko, F. Ricci, G.-M. Rignanese, A. H. Romero, M. M. Schmitt, M. Torrent, M. J. van Setten, B. V. Troeye, M. J. Verstraete, G. Zérah, and J. W. Zwanziger, The ABINIT project: Impact, environment and recent developments, Computer Physics Communications 248, 107042 (2020).
  * Perdew  _et al._ [1996] J. P. Perdew, K. Burke, and M. Ernzerhof, Generalized Gradient Approximation Made Simple, Phys. Rev. Lett. 77, 3865 (1996).
  * Jollet  _et al._ [2014] F. Jollet, M. Torrent, and N. Holzwarth, Generation of Projector Augmented-Wave atomic data: A 71 element validated table in the XML format, Computer Physics Communications 185, 1246 (2014).
  * Maździarz  _et al._ [2022] M. Maździarz, R. Psiuk, A. Krawczyńska, M. Lewandowska, and T. Mościcki, Effect of zirconium doping on the mechanical properties of W1-xZrxB2 on the basis of first-principles calculations and magnetron sputtered films, Archives of Civil and Mechanical Engineering 22, 193 (2022).