# Mixed-Mode Chimera States in Pendula Networks

P. Ebrahimzadeh  Forschungszentrum Jülich GmbH, ZEA-2: Electronics Systems,
52428 Jülich, Germany<EMAIL_ADDRESS>M. Schiek
Forschungszentrum Jülich GmbH, ZEA-2: Electronics Systems, 52428 Jülich,
Germany<EMAIL_ADDRESS>Y. Maistrenko  Forschungszentrum Jülich GmbH,
ZEA-2: Electronics Systems, 52428 Jülich, Germany  Institute of Mathematics
and Technical Centre, NAS of Ukraine, Tereschenkivska Str. 3, 01030, Kyiv,
Ukraine<EMAIL_ADDRESS>
###### Abstract

We report the emergence of peculiar chimera states in networks of identical
pendula with global phase-lagged coupling. The states reported include both
rotating and quiescent modes, i.e. with non-zero and zero average frequencies.
This kind mixed-mode chimeras may be interpreted as images of bump states
known in neuroscience in the context of modelling the working memory. We
illustrate this striking phenomenon for a network of $N=100$ coupled pendula,
followed by a detailed description of the minimal non-trivial case of $N=3$.
Parameter regions for five characteristic types of the system behavior are
identified consisting: two mixed-mode chimeras with one and two rotating
pendula, classical weak chimera with all three pendula rotating, synchronous
rotation and quiescent state. The network dynamics is multistable: up to four
of the states can coexist in the system phase state as demonstrated through
the basins of attraction. The analysis suggests that the robust mixed-mode
chimera states can generically describe the complex dynamics of diverse
pendula-like systems widespread in nature.

††preprint: AIP/123-QED

> Chimera states generally refer to spatio-temporal patterns in networks of
> identical or close to identical oscillators, in which a group of oscillators
> is synchronized and the other group is asynchronous. For networks composed
> of Kuramoto oscillators with inertia, chimera states are manifested in the
> form of solitary states in which one or a few oscillators split off from the
> main synchronized cluster and start to rotate with a different average
> frequency. Chimeras of this kind include rotational modes and their
> frequencies are determined by the system parameters. In networks of
> excitable elements such as neurons, in contrary, chimeric spatiotemporal
> patterns typically arise in the form of bump states, where active spiking
> neurons (large amplitude) coexist with quiescent (subthreshold) ones. The
> bumps states are created due to the competition mechanism between attractive
> and repulsive couplings, which suppresses the quiescent group. Then, the
> pendulum network can be viewed as a model bringing together the properties
> of the Kuramoto oscillators with inertia and the excitable theta neuron
> model, for which we show the emergence of mixed-mode chimeras with non-zero
> and zero average frequencies of individual oscillators from different
> groups.

##  I  Introduction. Mixed-Mode Chimera States

Patterns of synchronization have been the subject of intensive study in
various fields, ranging from biology, social behavior and network science
among others [1, 2, 3]. To this end, "classical" chimera states refer to
spatio-temporal patterns emerging in networks of non-locally coupled
oscillators as coexistence of coherent and incoherent groups [4, 5, 6, 7, 8].
In recent works, the notion of chimera states has been generalized to the
property of frequency clustering, named weak chimera states [9]. Among weak
chimeras a distinctive role is played by so-called solitary states [10, 11,
12, 13, 14, 15, 16, 17, 18] in which one or a few (or even more) oscillators
split off from the main synchronized cluster and start rotating with a
different average frequency. Characteristic examples of this kind behavior are
supplied by the Kuramoto model with inertia [19, 20, 21, 22, 23, 24, 25, 26],
where solitary states arise at all types of the network coupling from global
to local [10]. An essential property of the solitary states is that the
desynchronized oscillators do not create localized groups in the space, in
contrast to the classical Kuramoto model without inertia [27, 28]. Instead,
the splitted oscillators appear to be distributed in the network space in
visually arbitrary manner subjected to the assigned initial conditions. This
fact causes a huge multiplicity of the coexisting stable solitary states of
different configurations, given not only by different number of splitted
elements but also their permutation in the space. The number of coexisting
solitary states grows exponentially and network shows spatial chaos in the
thermodynamic limit [29].

Figure 1:  Typical examples of mixed-mode chimera state with different number
of frequency clusters for model (1) of $N=100$ pendula. Left panel: velocity
time plots for $\mu=2$ and $\alpha=1.2$(a), $1.35$(b), $1.42$(c),$1.52$(d),
respectively, and with the same fixed initial conditions. Middle panel:
sustained frequency profiles of the mixed-mide chimeras (inter-cluster
dynamics). Right panel: average frequencies for the individual pendula of the
mixed-mode chimeras. Other parameters: $m=1.0,\varepsilon=0.1,r=1,\omega=0$.

In this work, we consider a network of globally coupled pendula

$m\ddot{\phi}_{i}+\varepsilon\dot{\phi}_{i}+r\sin\phi_{i}=w+\frac{\mu}{N}\sum_{j=1}^{N}\sin[\phi_{j}-\phi_{i}-\alpha],$
(1)

where $\phi_{i}$, $i=1,...,N$, are phase variables, $\mu$ is the coupling
strength, $\alpha$ is the phase-lag, $m$, $\varepsilon$ and $w$ are inertia,
damping, and natural frequency of each pendulum, respectively. Experimental
evidence of chimera states in the pendula network (1) was obtained for a ring
setup of coupled metronomes [30, 31, 32]. We find that, besides the amazing
chimera complexity [33] inherent to the Kuramoto model with inertia, model (1)
also gains new characteristic solutions caused by the presence of the
nonlinear gravitation terms $\sin\phi_{i}$. We name them mixed-mode chimera
states, in which a part of oscillators are in a quiescent mode (slightly
oscillating, however) and the others are rotating with a non-zero average
frequency. Note that, behavior of this type is widespread in neuroscience,
known as bump states[34, 35, 36, 37, 38] which combines the spatially
localized groups of persistent neuronal activity at the silent background of
not-firing subthreshold neurons. In particular, bump states are considered as
appropriate models for functioning of the working memory in the brain [39, 40,
41]. During the last two decades, bump states were intensively studied for
various excitable neuronal models, including their apparent relation to
chimera states [34, 35, 36, 37, 38].

The emergence of the mixed-mode chimera states in model (1) is manifested as
the mismatch in average frequency between the chimera clusters. One of the
clusters is similar to the quiescent background in bumps, i.e., with zero
average frequency, and the other one consist of rotating oscillators with non-
zero average frequency. An important characteristic is that to induce the
mixed-mode chimera of a given configuration, specially prepared initial
conditions are needed (similarly again to the bump states). In the dynamical
interpretation, this property follows from the complex structure of the basins
of attraction, in particular, due to fractal basin boundary.

The dynamics of a single pendulum in model (1) consists of a stable
equilibrium and stable limit cycle including their coexistence at some
parameter region [23]. For small values of $w$ (and $\varepsilon$), the fixed
point solution exists and is stable. Increasing $w$ results in the appearance
of a limit cycle which is born in a homoclinic bifurcation at the Tricomi
bifurcation curve [42]. With further increase of $w$ at line $w/r=1$, the
stable fixed point is eliminated in a saddle-node bifurcation, and the only
attractor is the limit cycle. In the bistability region between the Tricomi
curve and $w/r=1$, both fixed point and limit cycle can develope depending on
the initial conditions. We note that, at $r=0$ the model transforms into the
Kuramoto model with inertia, the stable equilibrium is eliminated and
dynamical regimes reduce to that of limit cycle with average frequency
$\langle\dot{\phi}\rangle=w/\varepsilon$. In our numerical simulation we fix
parameters $m=1$, $\varepsilon=0.1$, $r=1$, and setting the natural frequency
$w=0$. The complex chimera-like regimes in the model (1) arise due to the
influence of coupling, which include self-coupling playing the role of an
"external forcing".

##  II  Network $N=100$

Typical examples of mixed-mode chimeras in model (1) of $N=100$ pendula with
global coupling are illustrated in Fig 1. The figure reveals the appearance of
solutions with different number and size of the frequency clusters, where one
of the clusters is quiescent (only slightly oscillating) but the others are
rotationg with non-zero average frequency. The solutions are obtained by the
simulations starting from the same initial conditions for increasing values of
the phase-lag parameter $\alpha=1.2(a);1.35(b);1.42(c);1.52(d)$, where the
coupling strength is fixed to $\mu=2$. The left panel in the figure shows the
transient behavior of the solutions, middle panel emboldens the shape of the
sustained cluster oscillations (after the transient) within a narrow time
interval, and the average frequencies are depicted in the right panel. It can
be seen that the complexity of the mixed-mode chimeras grows with the increase
of $\alpha$. First, in Fig. 1(a), the frequency profile is rather simple,
resembling classical bump state: some pendula of the network rotate, the rest
are quiescent. The number of the rotating clusters grows with an increase of
$\alpha$: two such clusters can be seen in (b) and three in (c). With further
increase of $\alpha$ the dynamics inside the quiescent cluster becomes
irregular or even chaotic, showing a "fuzzy" structure of the individual
frequencies, see (d).

Each cluster in a mixed-mode chimera state is categorized by a common averaged
frequency of the oscillators $\bar{\omega}$, with $\bar{\omega}\neq 0$ for
rotating and $\bar{\omega}=0$ for oscillating cluster in the background. This
is in contrast to classical chimera states, where a synchronous rotating
cluster coexists with a group of incoherent oscillators with a bell shape
frequency profile. Furthermore, the mixed-mode chimeras are not chaotic
transients as the classical counterpart [43], i.e. do not collapse into a
synchronization or rotating wave states. In contrary, they are persistent
solutions similar to solitary states in the Kuramoto model with inertia [10].

Note that, frequency clustering in chimera states does not imply, in general,
phase clustering (see Ref. [44] for illustrative examples). In the model (1),
however, this is the case as soon as $\alpha<\pi/2$: after formation of a
mixed-mode chimera, not only frequencies but also phases coincide for the
oscillators within each cluster. The dynamics is then governed on an invariant
manifold of the reduced dimension

$\displaystyle
D(M)=\\{\phi_{1}=...=\phi_{n_{1}}=\Phi_{1}(t);...;\phi_{n_{M}}=...=\phi_{N}=\Phi_{M}(t);$
$\displaystyle\dot{\phi}_{1}=...=\dot{\phi}_{n_{1}}=\dot{\Phi}_{1}(t);...;\dot{\phi}_{n_{M}}=...=\dot{\phi}_{N}=\dot{\Phi}_{M}(t)\\}$

such that the in-manifold dynamics is given by the equations

$m\ddot{\Phi}_{i}+\varepsilon\dot{\Phi}_{i}+r\sin(\Phi_{i})=-n_{i}\tilde{w}+\mu\sum_{j\neq
i}^{M}n_{j}\sin[\Phi_{j}-\Phi_{i}-\alpha],$ (3)

where $\tilde{w}=\mu\sin(\alpha)$ is the modified eigenfrequency and $n_{i}$
is the ratio of pendula in the $i$’s cluster to the network size. For
different values of parameters $(\alpha,\mu)$ and the initial conditions, Eq.
(3) can develope in different dynamical regions (such as fixed point,
bistability, limit cycle, invariant torus or chaos).

An analytical approach with asymptotic description can be developed to
describe the cluster dynamics. One can represent $\Phi_{i}$’s as sum of a
monotonic term with respect to time and correction terms scaled by small
perturbation parameter $\lambda$

$\Phi_{i}=\bar{\omega}_{i}t+\sum_{l=1}^{\infty}\lambda^{l}\delta\Phi^{l}_{i}$
(4)

Inserting condition (4) into Eq. (3) and sorting by $\lambda$, one finds
(Appendix A) the average frequency of the rotating clusters

$\bar{\omega}_{i}=-\frac{n_{i}\tilde{w}}{\varepsilon}$ (5)

and the leading correction term

$\lambda\delta\Phi^{1}_{i}\sim\frac{r}{m\bar{\omega}_{i}^{2}}\sin\bar{\omega}_{i}t-\mu\sum_{j=1}^{M}\frac{n_{j}}{\Delta\bar{\omega}_{ji}^{2}}\sin[\Delta\bar{\omega}_{ji}t-\alpha],$
(6)

where $\Delta\bar{\omega}_{ji}=\bar{\omega}_{j}-\bar{\omega}_{i}$. The average
frequency of the clusters depend on the parameters $(\alpha,\mu)$ and the
number of pendulum in the cluster. The system dynamic from this point of view
for any $N$ could be a subject of future study. In the next section, we
consider the emergence and coexistence of mixed-mode chimeras for the minimal
but non-trivial network of $N=3$ coupled pendulua in model (1).

##  III Minimal Network $N=3$

The minimal network with chimeric behavior in model (1) consists of $N=3$
coupled pendula

$\displaystyle\ddot{\phi}_{1}+\varepsilon\dot{\phi}_{1}+\sin{\phi_{1}}$
$\displaystyle=$
$\displaystyle\frac{\mu}{3}[\sin(\phi_{2}-\phi_{1}-\alpha)+\sin(\phi_{3}-\phi_{1}-\alpha)-\sin{\alpha}],$
$\displaystyle\ddot{\phi}_{2}+\varepsilon\dot{\phi}_{2}+\sin{\phi_{2}}$
$\displaystyle=$
$\displaystyle\frac{\mu}{3}[\sin(\phi_{1}-\phi_{2}-\alpha)+\sin(\phi_{3}-\phi_{2}-\alpha)-\sin{\alpha}],$
$\displaystyle\ddot{\phi}_{3}+\varepsilon\dot{\phi}_{3}+\sin{\phi_{3}}$
$\displaystyle=$
$\displaystyle\frac{\mu}{3}[\sin(\phi_{1}-\phi_{3}-\alpha)+\sin(\phi_{2}-\phi_{3}-\alpha)-\sin{\alpha}],$

where we put $m=r=1$ and $w=0$ for simplicity. In this minimal system, we find
two types of mixed-mode chimera states with one and two rotating pendula,
respectively. We symbolically denote them by the average frequencies
$(\bar{\omega}_{1},\bar{\omega}_{2},\bar{\omega}_{3})$ of the pendula such
that $(0:1:1)$ corresponds to the mixed-mode chimera with two rotating pendula
and one oscillating, and $(0:0:1)$ denotes one rotating pendulum and two
oscillating. By $(1:2:2)$ we denote a "classical" chimera in which all pendula
are in a rotation mode such that two are synchronized and rotating faster than
the third one. In addition, there are two non-chimeric behaviors: synchronous
rotating state $(1:1:1)$ and trivial equilibrium $(0:0:0)$ given by in-phase
fixed point $\phi_{1}=\phi_{2}=\phi_{3}$.

Results of direct numerical simulation of the system (III) in the two-
parameter plane of the phase-lag $\alpha$ and coupling strength $\mu$ are
presented in Fig. 2. This figure reveals the appearance of regions of the
chimera states of the three types $(0:1:1)$, $(0:0:1)$, and $(1:2:2)$, all
arising at not-small values of $\alpha$ and $\mu$. Alternatively, if $\alpha$
and/or $\mu$ are close to zero, the network displays only the equilibrium
state $(0:0:0)$, which exists and is stable in the dotted region below the
saddle-node bifurcation curve $\mu\sin\alpha=1$.

Figure 2:  Intersecting regions of chimera states for system (III) of $N=3$
coupled pendula in the ($\alpha,\mu$) parameter plane. The chimeras reside on
wide areas of the parameter plane. Type-1 mixed-mode chimera (0:1:1) (light
blue region) extends for large values of $\mu$ for all $\alpha$, while the
type-2 mixed-mode chimera (0:0:1) only exists in the strip region (shown
dashed). Region for classical chimera (1:1:2) is shown in dark blue, it is
also extends for large $\mu$. Synchronous rotation state (light gray) emerges
in a homoclinic bifurcation at the Tricomi curve $T_{0}$; it exist and is
stable for all values of $(\alpha,\mu)$ above $T_{0}$. quiescent fix point
state (trivial equilibtium) exists for small $\alpha$ and $\mu$ until the
saddle-node bifurcation curve $\mu\sin{\alpha}=1$. Characteristic example of
the patterns are presented in Fig. 3 Figure 3:  Frequency time plots of
typical behaviors in system (III) of $N=3$ coupled pendula. Setting coupling
strength $\mu=2.4$, the following patterns are observed for different values
of $\alpha$: (a) $\alpha=0.8$: classical chimera state $(1:2:2)$ ; (b)
$\alpha=0.6$: type-1 mixed-mode chimera $(0:0:1)$ with two quiescent and one
rotating pendula; (c) $\alpha=0.4$: type-2 mixed-mode chimera $(0:1:1)$ with
one quiescent and two rotating pendula; and (d) $\alpha=0.2$: synchronous
rotation $(1:1:1)$. Other parameters as in Fig. 2. Figure 4:  Ring of unstable
asynchronous fixed points of system (III) of $N=3$ coupled pendula at
parameters $\alpha=1.2$ and $\mu=1.2$

Note that both, the mixed-mode chimera $(0:1:1)$ and the rotating chimera
$(1:2:2)$ reside in large parameter regions with the area of coexistence
(shown in blue and dark gray, respectively), and they extend up to the large
values of $\alpha$ and $\mu$. The second mixed-mode chimera $(0:0:1)$, in
contrary, exists in a smaller region (shown in dashed). Non-chimeric
synchronous rotating state $(1:1:1)$ also occupies a big parameter region. It
arises in a homoclinic bifurcation at the curve $T_{0}$ (equivalent to the
Tricomi curve) and is stable for all $\alpha,\mu$ above $T_{0}$.

To describe the mechanism of the bifurcations in system (III) of $N=3$
pendula, we start by identifying its fixed points in the attractive parameter
region $\alpha<\pi/2$. Note that system (III) represents a six-dimensional
system of differential equations, and due to the presence of the nonlinear
gravitation term $\sin\phi$, cannot be reduced to a 4-Dim system in phase
differences (contrary to the corresponding Kuramoto model with inertia)

System (III) has a $2D$ invariant synchronous manifold
$\mathfrak{M}=\\{\phi_{1}=\phi_{2}=\phi_{3}\\}$, its dynamics is given by a
single pendulum equation

$\ddot{\phi}_{i}+\varepsilon\dot{\phi}_{i}+\sin(\phi_{i})=-\mu\sin\alpha.$ (8)

Note that the forcing term in the right hand side of Eq. (8) arises due to the
phase-lagged character of the global coupling, including the self-coupling.
System (III) has two fixed points (equilibria) $O$ and $S$ inside the
synchronous manifold $\mathfrak{M}$. In the variables $\\{\phi;\dot{\phi}\\}$
they are written as: $O=\\{\phi^{*};0)\\}$ and $S=\\{\phi^{**};0)\\}$, where

$\displaystyle\phi^{*}$ $\displaystyle=$
$\displaystyle-\sin^{-1}[\mu\sin\alpha]$ (9) $\displaystyle\phi^{**}$
$\displaystyle=$ $\displaystyle\pi+\sin^{-1}[\mu\sin\alpha].$ (10)

Figure 5:  Basins of attraction of system (III) with $N=3$ for $\mu=1.2$ and
increasing values of $\alpha=0.98;1.2$; and $1.4$. Top and bottom panels are
different cross sections of the six dimensional phase space, fixing different
initial conditions for pendulum no.2 (shown in the figure), and the same
initial values for pendulum no.3 as $(\phi_{3},\dot{\phi}_{3})=(-0.85,0.21).$
Color gamma of five coexisting characteristic states are shown to the right.

Fixed points $O$ and $S$ exist in the ($\alpha,\mu$) parameter region below
the bifurcation curve $\mu=1/\sin\alpha$, at which they collide and disappear
in a saddle-node bifurcation. The in-manifold stability of the fixed points is
controlled by the characteristic equation for two Lyapunov exponents
$\lambda=\lambda_{1,2}$:

$\lambda^{2}+\varepsilon\lambda\pm\sqrt{1-\mu^{2}\sin^{2}}=0,$ (11)

where $(+)$ and $(-)$ stay for $O$ ans $S$, respectively. Then, $S$ is a
saddle and $O$ is a stable fixed point such that

$O:\begin{cases}\text{stable
focus},&0<\mu<\sqrt{1-\frac{\varepsilon^{4}}{16}}\frac{1}{\sin\alpha}\\\
\text{stable
node},&\sqrt{1-\frac{\varepsilon^{4}}{16}}\frac{1}{\sin\alpha}<\mu<\frac{1}{\sin\alpha}.\end{cases}$
(12)

Simple analytics confirm that fixed point $O$ is stable not only inside the
synchronous manifold $\mathfrak{M}$, but also in transverse directions.
Transverse stability of $O$ is obtained from the equation for transverse
Lyapunov exponents $\lambda=\lambda_{3-6}$:

$(\lambda^{2}+\varepsilon\lambda+\sqrt{1-\mu^{2}\sin^{2}}+\mu\cos\alpha)^{2}=0.$
(13)

It has two pairs of two-multiple roots, all having negative real parts (as
$\mu\cos\alpha>0$ at $\alpha<\pi/2$), which guaranties transverse stability of
$O$. Similar analytics allows to extend this property to larger number of
coupled pendula, revealing that $O$ is a stable equilibrium of model (1) at
any finite $N$. The stability follows from the characteristic equation for the
eigenvalues $\lambda=\lambda_{i},i=1,...,2N$ of $O$:

$(z+\sqrt{1-\mu^{2}\sin^{2}{\alpha}})(z+\sqrt{1-\mu^{2}\sin^{2}{\alpha}}+\mu\cos{\alpha})^{N-1}=0,$
(14)

where $z=\lambda(\lambda+\varepsilon)$. The first term in Eq. (14) controls
the in-manifold stability and the second one provides two $(N-1)$-multiple
transverse eigenvalues of $O$, all characterized by negative real parts at
$\alpha<\pi/2$.

Besides the two synchronous equilibria $O$ and $S$, system (III) has numerous
asynchronous fixed points in the region $\mu>1$ which, however, are unstable
at $\alpha<\pi/2$. A remarkable example is illustrated in Fig. 4: a continuum
of asynchronous fixed points are assembled in a one-dimensional ”ring” type
manifold. All points are unstable for $\alpha<\pi/2$, however, they can
stabilize in the repulsive region $\alpha>\pi/2$. Interestingly, a similar
situation was reported recently for the $N=4$ dimensional Kuramoto model with
inertia, where a continuum of so-called antipodal fixed points exists [33],
unstable at $\alpha<\pi/2$ but stabilizing at $\alpha>\pi/2$. We leave this
issue for future study.

A typical scenario of the multistable basin transition in system (III) is
illustrated in Fig. 5 for growing values of the parameter $\alpha$, with fixed
coupling strength $\mu=1.2$. Two different cross sections of the 6-dimensional
phase space are shown in upper and lower panels, respectively, by fixing the
initial conditions of pendula no.$2~{}\&~{}3$. The plots demonstrate a highly
developed multistability of the system (III). At $\alpha=0.98$ (left panel)
all five characteristic states coexist and occupy wide areas of the space.
Here, the parameter point $(\alpha,\mu)$ lies below the curve
$\mu\sin{\alpha}=1$. Beyond the curve (middle and right panels), synchronous
equilibrium $(0:0:0)$ disappears, and basins of only four states are
presented. Noticeably, at different cross sections (cr. upper and lower
panels) the basins of attraction occupy essentially different regions
although, in general, they are qualitatively similar. This scenario suggests
an essential dependence of the global system dynamics of model (1) on the
initial conditions. The shape of basins develope as $\alpha$ approaches
$\pi/2$, becoming even "puzzling" as $\alpha$ crosses over $\pi/2$ [33].

##  IV Conclusions

In conclusion, we have shown that mixed-mode chimera states naturally arise in
a small network of $N=3$ coupled pendula with global phase-lagged coupling,
and demonstrated the similar but much more developed behaviors of this kind in
large network of $N=100$ pendula. Properties of the mixed-mode chimeras
specify their analogy to the bump states from neuroscience, both arising as a
result of the coupling excitability. With a difference, however, that the
mixed-mode chimeras grow in networks with bistable individual dynamics above
the Tricomi bifurcation curve. The huge variability of coexisting chimera
states indicates, we suggest, an apparent transition to space chaos in model
(1) as $N$ tends to infinity. A detailed verification of this fact can be a
subject of future study. It would also be interesting to examine deeper the
qualitative connections between the pendula and the neuronal networks,
different at the first glance but generating similar patterns. Additionally,
the system dynamics in our case is not ruled to a large extend by self-
organizing processes (forming rather limited numbers of coherent structures),
but by the other ”space chaos” mechanism in which a combinatorially large
variety of the stable states can be created given an appropriate choice of the
initial conditions. We believe that the emergence and multiplicity of the
mixed-mode chimera states indicates a common, probably universal phenomenon in
networks of very different nature, due to the competition between inertia and
excitability.

###### Acknowledgements.

P. Ebrahimzadeh has been supported by the Helmholtz Association Initiative and
Networking Fund under project number SO-092 (Advanced Computing Architectures,
ACA) and HITEC program.

*

##  Appendix A Asymptotic solution of pendulum network

In order to construct an asymptotic description of model (1), we introduce a
re-scaled time $\tau=\frac{w}{\varepsilon}$ and
$\lambda\kappa=\frac{\varepsilon^{2}}{w^{2}}$ for some dummy variable $w$. The
equation of motion of the network then reads

$\displaystyle m\phi_{i}^{\prime\prime}+\lambda\kappa
w\phi_{i}^{\prime}+\lambda\kappa r\sin{\phi_{i}}$ $\displaystyle=$
$\displaystyle-\lambda\kappa n_{i}\tilde{w_{i}}$ $\displaystyle+$
$\displaystyle\lambda\kappa\mu\sum_{j}n_{j}\sin[\phi_{j}-\phi_{i}-\alpha],$

where the prime symbol denotes derivative with respect to re-scaled time
${}^{\prime}=\frac{\mathrm{d}}{\mathrm{d}\tau}$. One can decompose $\phi$ as a
monotonic and a perturbebd bounded function with small perturbation
$\phi_{i}\approx\phi_{0,i}+\lambda\delta\phi_{1,i}$. Inserting this condition
into Eq. A and sorting for perturbation parameter $\lambda$ one gets equations
of motion for the main and perturbed parts

$\displaystyle\phi^{\prime\prime}_{0,i}$ $\displaystyle=$ $\displaystyle 0,$
(16) $\displaystyle m\delta\phi^{\prime\prime}_{1,i}+\kappa
w\phi^{\prime}_{0,i}+\kappa r\sin{\phi_{0,i}}$ $\displaystyle=$
$\displaystyle-\kappa n_{i}\tilde{w_{i}}$ $\displaystyle+$
$\displaystyle\kappa\mu\sum_{j}n_{j}\sin[\phi_{0,j}-\phi_{0,i}-\alpha].$

Equation 16 reveals that the unperturbed part $\phi_{0}$ is indeed a monotonic
function in time and gives the average velocity of the pendulum as a constant.
The perturbed part $\delta\phi_{1}$, however, is a bounded function implying
Eq. (A.3) cannot have a non-zero torque. This can be prevented by equating
$w\phi_{0,i}^{\prime}=-n_{i}\tilde{w_{i}}$. Rewriting this equation in
original time scale, the dummy parameter $w$ drops and one gets the average
velocity of the pendulum

$\tilde{\omega}_{i}\equiv\dot{\phi}_{0,i}=-\frac{n_{i}\tilde{w_{i}}}{\varepsilon}.$
(18)

Inserting this condition into Eq. (A.3), one gets the equation of motion for
the perturbed part in original time scale

$\lambda\delta\ddot{\phi}_{1,i}=-r\sin(\tilde{\omega}_{i}t)+\mu\sum_{j}n_{j}\sin[\tilde{\omega}_{j}t-\tilde{\omega}_{i}t-\alpha],$
(19)

The right hand side is now only a function of time, and one finds the
perturbed function $\delta\phi$ by straightforward integration.

## References

  * Kuramoto [2003] Y. Kuramoto,  _Chemical oscillations, waves, and turbulence_ (Courier Corporation, 2003).
  * Pikovsky, Rosenblum, and Kurths [2001] A. Pikovsky, M. Rosenblum, and J. Kurths,  _Synchronization: A Universal Concept in Nonlinear Sciences_, Cambridge Nonlinear Science Series (Cambridge University Press, 2001).
  * Strogatz [2001] S. H. Strogatz, “Exploring complex networks,” nature 410, 268–276 (2001).
  * Panaggio and Abrams [2015] M. J. Panaggio and D. M. Abrams, “Chimera states: coexistence of coherence and incoherence in networks of coupled oscillators,” Nonlinearity 28, R67–R87 (2015).
  * Schöll [2016] E. Schöll, “Synchronization patterns and chimera states in complex networks: Interplay of topology and dynamics,” The European Physical Journal Special Topics 225, 891–919 (2016).
  * Omel’chenko and Knobloch [2019] E. Omel’chenko and E. Knobloch, “Chimerapedia: coherence–incoherence patterns in one, two and three dimensions,” New Journal of Physics 21, 093034 (2019).
  * Zakharova [2020] A. Zakharova,  _Chimera Patterns in Networks_ (Springer, 2020).
  * Parastesh  _et al._ [2021] F. Parastesh, S. Jafari, H. Azarnoush, Z. Shahriari, Z. Wang, S. Boccaletti, and M. Perc, “Chimeras,” Physics Reports 898, 1–114 (2021).
  * Ashwin and Burylko [2015] P. Ashwin and O. Burylko, “Weak chimeras in minimal networks of coupled phase oscillators,” Chaos: An Interdisciplinary Journal of Nonlinear Science 25, 013106 (2015).
  * Jaros  _et al._ [2018] P. Jaros, S. Brezetsky, R. Levchenko, D. Dudkowski, T. Kapitaniak, and Y. Maistrenko, “Solitary states for coupled oscillators with inertia,” Chaos: An Interdisciplinary Journal of Nonlinear Science 28, 011103 (2018).
  * Berner  _et al._ [2020] R. Berner, A. Polanska, E. Schöll, and S. Yanchuk, “Solitary states in adaptive nonlocal oscillator networks,” The European Physical Journal Special Topics 229, 2183–2203 (2020).
  * Jaros  _et al._ [2021] P. Jaros, R. Levchenko, T. Kapitaniak, and Y. Maistrenko, “Chimera states for directed networks,” Chaos: An Interdisciplinary Journal of Nonlinear Science 31, 103111 (2021).
  * Schülen  _et al._ [2021] L. Schülen, D. A. Janzen, E. S. Medeiros, and A. Zakharova, “Solitary states in multiplex neural networks: Onset and vulnerability,” Chaos, Solitons & Fractals 145, 110670 (2021).
  * Hellmann  _et al._ [2020] F. Hellmann, P. Schultz, P. Jaros, R. Levchenko, T. Kapitaniak, J. Kurths, and Y. Maistrenko, “Network-induced multistability through lossy coupling and exotic solitary states,” Nature communications 11, 1–9 (2020).
  * Franović  _et al._ [2022] I. Franović, S. Eydam, N. Semenova, and A. Zakharova, “Unbalanced clustering and solitary states in coupled excitable systems,” Chaos: An Interdisciplinary Journal of Nonlinear Science 32, 011104 (2022).
  * Maistrenko, Penkovsky, and Rosenblum [2014] Y. Maistrenko, B. Penkovsky, and M. Rosenblum, “Solitary state at the edge of synchrony in ensembles with attractive and repulsive interactions,” Physical Review E 89, 060901 (2014).
  * Maistrenko, Sudakov, and Maistrenko [2020] V. Maistrenko, O. Sudakov, and Y. Maistrenko, “Spiral wave chimeras for coupled oscillators with inertia,” The European Physical Journal Special Topics 229, 2327–2340 (2020).
  * Maistrenko, Sudakov, and Osiv [2020] V. Maistrenko, O. Sudakov, and O. Osiv, “Chimeras and solitary states in 3d oscillator networks with inertia,” Chaos: An Interdisciplinary Journal of Nonlinear Science 30, 063113 (2020).
  * Jaros, Maistrenko, and Kapitaniak [2015] P. Jaros, Y. Maistrenko, and T. Kapitaniak, “Chimera states on the route from coherence to rotating waves,” Physical Review E 91, 022907 (2015).
  * Ermentrout [1991] B. Ermentrout, “An adaptive model for synchrony in the firefly pteroptyx malaccae,” Journal of Mathematical Biology 29, 571–585 (1991).
  * Olmi  _et al._ [2015] S. Olmi, E. A. Martens, S. Thutupalli, and A. Torcini, “Intermittent chaotic chimeras for coupled rotators,” Physical Review E 92, 030901 (2015).
  * Olmi [2015] S. Olmi, “Chimera states in coupled kuramoto oscillators with inertia,” Chaos: An Interdisciplinary Journal of Nonlinear Science 25, 123125 (2015).
  * Belykh, Brister, and Belykh [2016] I. V. Belykh, B. N. Brister, and V. N. Belykh, “Bistability of patterns of synchrony in kuramoto oscillators with inertia,” Chaos: An Interdisciplinary Journal of Nonlinear Science 26, 094822 (2016).
  * Belykh, Jeter, and Belykh [2017] I. Belykh, R. Jeter, and V. Belykh, “Foot force models of crowd dynamics on a wobbly bridge,” Science advances 3, e1701512 (2017).
  * Brister, Belykh, and Belykh [2020] B. N. Brister, V. N. Belykh, and I. V. Belykh, “When three is a crowd: Chaos from clusters of kuramoto oscillators with inertia,” Physical Review E 101, 062206 (2020).
  * Kruk, Maistrenko, and Koeppl [2020] N. Kruk, Y. Maistrenko, and H. Koeppl, “Solitary states in the mean-field limit,” Chaos: An Interdisciplinary Journal of Nonlinear Science 30, 111104 (2020).
  * Kuramoto and Battogtokh [2002] Y. Kuramoto and D. Battogtokh, “Coexistence of coherence and incoherence in nonlocally coupled phase oscillators.” NONLINEAR PHENOMENA IN COMPLEX SYSTEMS 5, 380–385 (2002).
  * Abrams and Strogatz [2004] D. M. Abrams and S. H. Strogatz, “Chimera states for coupled oscillators,” Phys. Rev. Lett. 93, 174102 (2004).
  * Omelchenko  _et al._ [2011] I. Omelchenko, Y. Maistrenko, P. Hövel, and E. Schöll, “Loss of coherence in dynamical networks: spatial chaos and chimera states,” Physical review letters 106, 234102 (2011).
  * Kapitaniak  _et al._ [2014] T. Kapitaniak, P. Kuzma, J. Wojewoda, K. Czolczynski, and Y. Maistrenko, “Imperfect chimera states for coupled pendula,” Scientific reports 4, 1–4 (2014).
  * Wojewoda  _et al._ [2016] J. Wojewoda, K. Czolczynski, Y. Maistrenko, and T. Kapitaniak, “The smallest chimera state for coupled pendula,” Scientific reports 6, 1–5 (2016).
  * Ebrahimzadeh  _et al._ [2020] P. Ebrahimzadeh, M. Schiek, P. Jaros, T. Kapitaniak, S. van Waasen, and Y. Maistrenko, “Minimal chimera states in phase-lag coupled mechanical oscillators,” The European Physical Journal Special Topics 229, 2205–2214 (2020).
  * Brezetsky  _et al._ [2021] S. Brezetsky, P. Jaros, R. Levchenko, T. Kapitaniak, and Y. Maistrenko, “Chimera complexity,” Physical Review E 103, L050204 (2021).
  * Ermentrout [1998] B. Ermentrout, “Neural networks as spatio-temporal pattern-forming systems,” Reports on Progress in Physics 61, 353–430 (1998).
  * Laing and Chow [2001] C. R. Laing and C. C. Chow, “Stationary bumps in networks of spiking neurons,” Neural computation 13, 1473–1494 (2001).
  * Laing  _et al._ [2002] C. R. Laing, W. C. Troy, B. Gutkin, and G. B. Ermentrout, “Multiple bumps in a neuronal model of working memory,” SIAM Journal on Applied Mathematics 63, 62–97 (2002).
  * Owen, Laing, and Coombes [2007] M. R. Owen, C. R. Laing, and S. Coombes, “Bumps and rings in a two-dimensional neural field: splitting and rotational instabilities,” New Journal of Physics 9, 378 (2007).
  * Laing [2021] C. R. Laing, “Interpolating between bumps and chimeras,” Chaos: An Interdisciplinary Journal of Nonlinear Science 31, 113116 (2021).
  * Renart, Song, and Wang [2003] A. Renart, P. Song, and X.-J. Wang, “Robust spatial working memory through homeostatic synaptic scaling in heterogeneous cortical networks,” Neuron 38, 473–485 (2003).
  * Coombes [2005] S. Coombes, “Waves, bumps, and patterns in neural field theories,” Biological cybernetics 93, 91–108 (2005).
  * Miller, Lundqvist, and Bastos [2018] E. K. Miller, M. Lundqvist, and A. M. Bastos, “Working memory 2.0,” Neuron 100, 463–475 (2018).
  * Tricomi [1933] F. Tricomi, “Integrazione di un’equazione differenziale presentatasi in elettrotecnica,” Annali della Scuola Normale Superiore di Pisa-Classe di Scienze 2, 1–20 (1933).
  * Wolfrum and Omel’chenko [2011] M. Wolfrum and E. Omel’chenko, “Chimera states are chaotic transients,” Physical Review E 84, 015201 (2011).
  * Maistrenko  _et al._ [2017] Y. Maistrenko, S. Brezetsky, P. Jaros, R. Levchenko, and T. Kapitaniak, “Smallest chimera states,” Physical Review E 95, 010203 (2017).


# Distinguishing Question Subjectivity from Difficulty for Improved
Crowdsourcing

Yuan Jin, Ye Zhu  School of Information Technology
Deakin University, Australia yuan.jin<EMAIL_ADDRESS>and  Mark Carman,
Wray Buntine  Faculty of Information Technology
Monash University, Australia mark.carman<EMAIL_ADDRESS>
(2018)

###### Abstract.

The questions in a crowdsourcing task typically exhibit varying degrees of
difficulty and subjectivity. Their joint effects give rise to the variation in
responses to the same question by different crowd-workers. This variation is
low when the question is easy to answer and objective, and high when it is
difficult and subjective. Unfortunately, current quality control methods for
crowdsourcing consider only the question difficulty to account for the
variation. As a result, these methods cannot distinguish workers’ personal
preferences for different correct answers of a partially subjective question
from their ability/expertise to avoid objectively wrong answers for that
question. To address this issue, we present a probabilistic model which (i)
explicitly encodes question difficulty as a model parameter and (ii)
implicitly encodes question subjectivity via latent preference factors for
crowd-workers. We show that question subjectivity induces grouping of crowd-
workers, revealed through clustering of their latent preferences. Moreover, we
develop a quantitative measure of the subjectivity of a question. Experiments
show that our model (1) improves the performance of both quality control for
crowd-sourced answers and next answer prediction for crowd-workers, and (2)
can potentially provide coherent rankings of questions in terms of their
difficulty and subjectivity, so that task providers can refine their designs
of the crowdsourcing tasks, e.g. by removing highly subjective questions or
inappropriately difficult questions.

Crowdsourcing, Subjectivity, Difficulty, Statistical modelling

††copyright: rightsretained††conference: ACM conference; August 2018; London,
United Kingdom††journalyear: 2018††article: 4††price: 15.00††ccs: Information
systems Crowdsourcing††ccs: Information systems Answer ranking††ccs: Computing
methodologies Learning in probabilistic graphical models

##  1\. Introduction

Outsourcing tasks to a flexible online workforce (aka crowdsourcing) has
proven a successful paradigm for data collection in numerous fields due
primarily to its overall lower costs and shorter turnaround time as compared
to in-house expert-based data collection. The downside of online crowdsourcing
is that the quality of the answers collected from crowd-workers is usually not
guaranteed, even when multiple responses are collected and aggregated for each
question, and workers are trained and vetted using gold-standard questions. To
address this issue, many quality control methods for the crowdsourced answers
have been proposed (Whitehill et al., 2009a; Welinder et al., 2010). These
methods rely on the assumptions that most crowd-workers are reliable when
answering the questions and that a given worker is more likely to be reliable
should she agree with the majority of her co-workers on the majority of their
jointly answered questions. Thus, the methods have focused on modelling the
_accuracy/ability/expertise_ of individual workers, assuming this to be
correlated with the quality of the responses (Dawid and Skene, 1979; Raykar et
al., 2010). In recent years, it has become popular for quality control methods
to also model the influence that individual questions exert on the quality of
the responses (Whitehill et al., 2009b; Bachrach et al., 2012). Broadly
speaking, the following two key properties of questions have drawn the
modelling attention:

  * •

Difficulty. The modelling of question difficulty is founded on the assumption
that greater agreement on workers’ answers to a particular question indicates
less difficulty for them in determining the correct response. Quality control
methods often encode this assumption using a function in which worker
expertise counteracts question difficulty for predicting the probability of a
correct response. The probability is also known as the quality of the
response: the more difficult the question, the lower the quality of a
response, and vice versa. In addition, some methods (e.g. (Kamar et al.,
2015)) also consider the existence of _deceptive_ questions which are so
difficult that the assumption that the majority of worker responses are
correct no longer holds.

  * •

Subjectivity. In crowdsourcing, there are also many tasks that contain (purely
or partially) subjective questions (Nguyen et al., 2016). Intuitively, the
degree of subjectivity of a question (or equivalently, the data item described
by it) depends on the number of answer options that are correct. Being purely
subjective means all of the options are correct, while being partially
subjective means more than one but not all of them are correct. Unless it is
explicitly announced by the task provider that a question accepts all options
(e.g. movie rating by workers to build a movie recommender system (Lee et al.,
2013)), the number of correct answers to a question is unknown and assumed by
most quality control methods to be one, meaning the question is objective.
However, it is widely known that even expert assessors can disagree with each
other on the correct answer to a question in typical crowdsourcing tasks like
relevance judgement which is deemed to be “quite subjective" (or equivalently,
at least partially subjective) (Voorhees, 2000). In this case, the objectivity
assumption on the questions does not hold and most of the quality control
methods based on this assumption cannot distinguish the answering
accuracy/quality of workers from their preferences for the different answers
of questions.

(a) _Product Matching task:_ crowd-workers asked whether two product
descriptions referred to the same item or not.

(b) _Fashion Judgement task:_ crowd-workers asked whether a picture contains a
“fashion related item” or not.

Figure 1. Heatmaps showing inter-worker response similarity (% of response
agreement) for two different tasks: (a) a _relatively objective_ product
matching task and (b) a _more subjective_ fashion judging task, both involving
binary worker responses. Hierarchical clustering was performed to order
workers such that similar workers are close together. The three yellow blocks
in the figure for task (b) indicate three groups of response behaviour and
higher subjectivity for task (b).

For crowdsourcing tasks that contain questions whose subjectivity is either
unknown or known to be at least partial, novel quality control methods need to
be developed to capture any underlying answering/labelling pattern that
results from the subjectivity of the questions. One such pattern uncovered by
collaborative filtering (Koren et al., 2009) is that crowd-workers who share
similar preferences tend to respond similarly towards subjective questions
which share certain (latent) features.This pattern can also be observed in
crowdsourcing where groups emerge amongst crowd-workers in terms of the
answers they give to partially subjective questions. Figure 1 illustrates this
phenomenon by providing heat maps of pairwise worker similarity for two tasks:
(a) a relatively objective task and (b) a more subjective one. The objective
task required workers to judge whether a pair of products were the same based
on their names, descriptions and prices, while the subjective task asked
workers to judge whether an image contained “fashion related items”111The
datasets for the two tasks have been listed in section 5.. The similarity
between pairs of workers is calculated as the percentage of agreement across
the jointly answered questions222Pairs of crowd-workers not sharing any items
had their similarity to be $.00001$.  and hierarchical clustering has been
performed to group similar workers together. The three yellow boxes along the
diagonal for the more subjective task (b) indicates the three distinct groups
of worker response behaviour for this task, which was absent in the more
objective task (a). Since the workers were mostly reliable on both tasks, we
conjecture that the grouping of response behaviour for the workers in the
fashion judgement task reflects the underlying structures in their tendencies
for selecting the different correct answers of the same questions (due to
their subjectivity).

To enable the answer quality control for the above tasks and generally, any
crowdsourcing task that exhibits arbitrary degrees of question subjectivity
and difficulty, we are motivated to develop a statistical model encoding both
these properties. The resulting model is able to explain both the randomness
and the correlations in the answering behaviour of crowd-workers. More
specifically, when a task contains only purely subjective questions, groupings
of workers start to emerge due to the subjectivity of questions. A group
captures a particular correlation between the crowd-workers within it and the
latent correct answers for each of the questions. We model such a correlation
by factorising it into the latent preferences of the workers and the latent
features of the questions. The assumption is that the workers with similar
latent preferences tend to have similar perceptions of what constitutes a
correct answer for each of the questions. For instance, asking workers “which
colour for this shirt do you like?” is a purely subjective question for which
one group of workers who like blue colour in general will answer “blue”,
whereas another group who like green colour will choose “green”. There is no
reason to believe one of these two groups answer the question more correctly
than the other, and their distinct answering patterns tend to remain
consistent across similar questions asking about their colour favourites for
other items (e.g. trousers, hats).

If a question is _partially subjective_ , this means it possesses (i) a
certain degree of subjectivity which corresponds to either its tendency of
having two or more correct answers, and (ii) a certain level of difficulty.
This difficulty corrupts the crowd-workers’ perceptions as to (what tend to
be) the correct answers of the question determined by its subjectivity to
various extents depending on its level against the workers’ levels of
expertise. We model a greater extent of the corruption as a lower probability
that the worker’s answer is equal to the subjective (worker-specific) correct
answer to the question, thereby the lower quality of the worker’s answer. This
subjective correct answer characterizes the particular group to which the
worker belongs by sharing similar preferences with some other workers333We
refer the reader to the movie example in the previous paragraph..

In this paper we introduce a new quality-control framework for crowdsourcing
that _models both the subjective (i.e. worker-specific) truths regarding the
correct answers to individual questions and also the difficulty-dependent
probability that a worker’s answer to a question will equal her perceived
subjective truth_. We now summarise the contributions of the paper as follows:

  * •

A novel statistical model is proposed which encodes the question difficulty
explicitly and the question subjectivity implicitly via latent variables for
worker preferences and corresponding question features. The model accounts for
both the random and the systematic parts of the variance in crowdsourced
answers to refine the quality control over them.

  * •

A Monte Carlo simulation approach is provided for quantifying question
subjectivity as the expected number of subjective truths perceived by
different groupings of crowd-workers with respect to their preferences.

  * •

A meaningful ranking of questions in terms of either difficulty or
subjectivity is derived from the model parameter estimates. This can bring
practical benefits to crowdsourcing such as improving designs of tasks by
helping requesters to detect and remove highly subjective questions from the
tasks intended to be objective.

$r_{ij}$$l_{j}$$\boldsymbol{\theta}$$\boldsymbol{\gamma}$$e_{i}$$d_{j}$$\mu_{e},\sigma^{2}_{e}$$\mu_{d},\sigma^{2}_{d}$$\mathcal{IJ}$$\mathcal{I}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}$$\mathcal{J}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}$

(a) model for _fully-objective_ items

$r_{ij}$$\boldsymbol{u}_{ik}$$\boldsymbol{v}_{jk}$$\boldsymbol{\mu}_{u},\sigma^{2}_{u}$$\boldsymbol{\mu}_{v},\sigma^{2}_{v}$$\mathcal{IJ}$$\mathcal{IK}$$\mathcal{JK}$

(b) model for _fully-subjective_ items

$r_{ij}$$l_{ij}$$z_{ij}$$\boldsymbol{\phi}_{i}$$\boldsymbol{\alpha}$$e_{i}$$d_{j}$$u_{mk}$$v_{jk}$$\mu_{u},\sigma^{2}_{u}$$\mu_{v},\sigma^{2}_{v}$$\mu_{d},\sigma^{2}_{d}$$\mu_{e},\sigma^{2}_{e}$$\mathcal{MK}$$\mathcal{JK}$$\mathcal{IJ}$$\mathcal{I}$$\mathcal{J}$

(c) model for _partially-subjective_ items

Figure 2. (a) shows GLAD with a latent variable $l_{j}$ for each objective
truth, (b) shows a collaborative filtering model without objective truths, and
(c) is the proposed subjectivity-and-difficulty response (SDR) model for
partially-subjective questions that is able to distinguish question difficulty
from subjectivity.

##  2\. Related Work

###  2.1. Latent variable modelling in crowdsourcing

Most state-of-the-art answer/label quality control methods in crowdsourcing
have operated under the assumption that each question is purely objective.
These methods are primarily based on statistical modelling of the interactions
between crowd-workers and questions which determine either the marginal
probabilities of the workers’ answers equal to the corresponding correct
answers (Whitehill et al., 2009b; Bachrach et al., 2012; Welinder et al.,
2010) or the conditional probabilities of the answers given the correct
answers (Dawid and Skene, 1979; Venanzi et al., 2014; Kamar et al., 2015). In
comparison, the marginal probabilistic modelling is simpler than the
conditional modelling, and also better at mitigating answer sparsity problem
in crowdsourcing (Jung and Lease, 2013; Jung, 2014). The basic marginal
probabilistic modelling is GLAD (Whitehill et al., 2009b), which models the
correctness of each answer as a logistic function where the question
difficulty counteracts the expertise of the responding worker. Its graphical
representation is shown by Figure 2(a) with the following generative scheme
for a response $r_{ij}$ of worker $i$ given to question $j$:
$\boldsymbol{\theta}\sim Dir(\boldsymbol{\gamma});\;l_{j}\sim
Discrete(\boldsymbol{\theta});\text{ }\;r_{ij}\sim
Discrete(\boldsymbol{\pi}_{ij})$. This means a correct answer $l_{j}$ is drawn
for question $j$ from a discrete distribution parametrised by
$\boldsymbol{\theta}$, which was previously drawn from a Dirichlet
distribution parametrised by $\boldsymbol{\gamma}$. Then, a response $r_{ij}$
conditioned on $l_{j}$ is drawn from a discrete distribution with the $k$-th
component of its parameters $\boldsymbol{\pi}_{ij}$ calculated as follows:

(1)
$\displaystyle\;\;\pi_{ijk}=f(e_{i},d_{j})\;\;\text{if}\;\;k=l_{j}\;\;\text{else}\;\;\frac{1-f(e_{i},d_{j})}{K-1};\;\;f(e_{i},d_{j})=\frac{1}{1+e^{-(e_{i}/\exp(d_{j}))}}$

In this case, the function takes in the expertise factor $e_{i}$ of worker $i$
and the difficulty factor $d_{j}$ of question $j$. The output of the function
is the probability of the response $r_{ij}$ being correct. When
$e_{i}\rightarrow+\infty$ or $d_{j}\rightarrow 0$, this probability grows,
indicating a stronger positive correlation between $r_{ij}$ and $l_{j}$. When
$e_{i}\rightarrow 0$ or $d_{j}\rightarrow+\infty$ and the question has binary
options, the probability approaches 0.5, leading to no correlation between the
two, which suggests $r_{ij}$ is a random binary pick. When
$e_{i}\rightarrow-\infty$, the probability decreases to 0, indicating a
stronger negative correlation.

Although efficient in quality control of answers to objective questions,
current models based on marginal probabilistic modelling have hardly
considered modelling the subjectivity of questions, let alone inferring their
possible subjective truths. One of the only two papers that have made progress
in this regard is (Tian and Zhu, 2012). It assumes that a higher (lower) joint
degree of difficulty and subjectivity for an entire crowdsourcing task can
increase (decrease) the number of groups of answers given by the crowd-workers
to the questions. The expected size of each group becoming smaller (larger)
indicates overall weaker (stronger) correlations of answers given to the
questions. Despite attributing the variance of answers to both difficulty and
subjectivity, the paper makes no attempt to separate the two when it is
supposed to be only the difficulty accounting for the quality of answers.
Moreover, this work requires every question in a task to be answered by every
worker, which is unrealistic in practice. The other work (Nguyen et al., 2016)
has focused on modelling partially subjective questions with just ordinal
answers. It assumes each response to a question is generated by a Normal
distribution the mean and the variance of which are linearly regressed over
the observed features of the question. This means the model will poorly fit
any multi-modal distribution of answers to a question.

###  2.2. Latent variable modelling in collaborative filtering

In model-based collaborative-filtering (Koren et al., 2009), matrix
factorization is typically applied to predicting ordinal ratings provided by
users to items (e.g. movies, songs). Its categorical version, shown in Figure
2(b), is less commonly applied but is important for the construction of our
model for the quality control of crowdsourced categorical answers. It has a
generative process for the response $r_{ij}\sim
Discrete(\boldsymbol{\psi}_{ij}),\;\text{where}\;\boldsymbol{\psi}_{ij}=\\{\psi_{ijk}\\}_{k\in\mathcal{K}}$
and $\mathcal{K}$ is the set of answer options, with its $k$-th component
calcuated as:

(2)
$\displaystyle\psi_{ijk}=P(r_{ij}=k|\boldsymbol{U}_{i},\boldsymbol{V}_{j})=\exp(\boldsymbol{u}^{T}_{ik}\boldsymbol{v}_{jk})/\sum_{k^{\prime}\in\mathcal{K}}\exp(\boldsymbol{u}^{T}_{ik^{\prime}}\boldsymbol{v}_{jk^{\prime}})$

Here, $\boldsymbol{\psi}_{ij}$ is also called the soft-max function,
$\boldsymbol{u}_{ik}$ and $\boldsymbol{v}_{jk}$ are respectively the latent
preferences of worker $i$ and the latent features of item $j$ in relation to
the $k$-th answer option. The inner product term
$\boldsymbol{u}^{T}_{ik}\boldsymbol{v}_{jk}$ indicates how much tendency
worker $i$ responds to item $j$ with the $k$-th answer option.

##  3\. Proposed model

Our proposed model endeavours to combine the key characteristics of the latent
variable models specified in section 2.1 and section 2.2. We call it SDR model
(Subjectivity-and-Difficulty Response model), which comprises an _upstream
module_ which generates a subjective truth for a question based on the
worker’s perception of the correct answer, and a _downstream module_ which
imposes a difficulty-dependent corruption on the subjective truth for
generating the actual response from the worker to the question. More
specifically, in the upstream module, the latent subjective truth $l_{ij}$ of
question $j$ as perceived by crowd-worker $i$ is drawn from a soft-max
function specified by Eq. (2) except that the original $r_{ij}$ in the
equation is now replaced by $l_{ij}$. This function explains how the worker’s
latent preferences interact with the question’s latent features to generate
the subjective truth behind her response to the question. In the downstream,
conditioned on the latent subjective truth $l_{ij}$, the response $r_{ij}$
actually given by worker $i$ to question $j$ is determined by the logistic
function $f(e_{i},d_{j})$. It encodes how the worker expertise $e_{i}$
counteracts the question difficulty $d_{j}$ to corrupt the subjective truth
into the response, which will be defined later in this section. Essentially,
the above perception-corruption process is a generalisation of the corruption
process of the correct answer signals from objective questions modelled in
(Welinder et al., 2010) by additionally considering the question subjectivity.

Unfortunately the upstream+downstream model described above suffers from an
over-parameterisation issue whereby _both_ the upstream component (which
determines the worker-specific correct answer) and the downstream component
(which determines the noise resulting from worker inaccuracy) can
_independently and adequately_ explain the variance observed in worker
responses to the same question. In other words, the varied responses from
different workers to the same question could equally be due to different
perceptions on what constitutes the correct answer to the question or to
difficulty of the question causing low accuracy amongst the respondents. To
remedy this situation we explicitly enforce a group structure over workers in
order to limit the variation in the perceptions across workers. This is done
by changing the upstream module to have sparsity-inducing priors over the
latent preferences of crowd-workers. In this paper, we use the Latent
Dirichlet Allocation (LDA) (Blei et al., 2003) as such priors. The final
graphical representation of the SDR model is shown in Figure 2(c).The new
upstream module of our model assigns a probability vector
$\boldsymbol{\phi}_{i}$, which follows a Dirichlet with a concentration
parameter $\boldsymbol{\alpha}$, to each worker $i$. Each component
$\phi_{mi}$ of this probability vector reflects the worker’s tendency of
showing a particular preference $m$ among the set of preferences $\mathcal{M}$
she possesses when answering any question. Then, a preference assignment
$z_{ij}$ is drawn from $\boldsymbol{\phi}_{i}$ for determining the specific
preference worker $i$ will show for answering question $j$. As for preference
$m$, it has a weight $u_{mk}$ for each answer option $k$ to reflect how likely
each option is to be selected given the preference $m$ showed by any worker.
In this paper, we fix the dimension of $u_{mk}$ to be strictly 1. This weight
is multiplied with the latent feature $v_{jk}$ of question $j$ and the result
is input to a soft-max function for drawing the subjective truth $l_{ij}$
behind the response $r_{ij}$. The above generative process can be formulated
as: $\boldsymbol{\phi}_{i}\sim Dir(\boldsymbol{\alpha});\;\;z_{ij}\sim
Discrete(\boldsymbol{\phi}_{i});\;\;l_{ij}\sim
Discrete(\boldsymbol{\psi}_{z_{ij}})$ with the $k$-th component of the soft-
max function $\boldsymbol{\psi}_{z_{ij}}$ calculated as:

(3)
$\displaystyle\psi_{z_{ij}k}=\frac{\exp(u_{z_{ij}k}v_{jk})}{\sum\limits^{K}_{k^{\prime}=1}\exp(u_{z_{ij}k^{\prime}}v_{jk^{\prime}})}$

Embodying the sparsity-inducing effect of LDA, the preference probabilities
$\boldsymbol{\phi}_{i}$ are dedicated to revealing the underlying groups of
crowd-workers while the soft-max specified by Eq. (3) governs the positive
correlations between the latent correct answers to the same questions
perceived by the workers within the same group. When the number of preferences
in $\mathcal{M}$ = 1, the probability of the only preference $\phi_{i}$ is 1.
This has a two-fold meaning that each question has one correct answer and
every worker should perceive the correct answer of any question in the same
way. When the size of $\mathcal{M}$ is greater than 1, this indicates certain
numbers of underlying worker groups, which we can recover by applying K-means
clustering to the estimated preference probabilities
$\hat{\boldsymbol{\phi}}_{i}$ using the Elbow method to determine the right
number of the groups.

The downstream module corrupts the correlations between the subjective truth
$l_{ij}$ and the response $r_{ij}$. It draws $r_{ij}$ from a discrete
probability distribution $\boldsymbol{\pi}_{ij}$ specified in Eq. (1) except
the logistic function $f(e_{i},d_{j})$ has the following definition from
(Rasch, 1993):

(4) $\displaystyle f(e_{i},d_{j})=\frac{1}{1+e^{-(e_{i}-d_{j})}}$

The term $(e_{i}-d_{j})$ naturally explains the type of biases induced by
deceptive questions when the difficulty $d_{j}$ is much larger than the
expertise $e_{i}$, which is not captured in Eq. (1) as the term $\exp(d_{j})$
is never smaller than 0, meaning questions never bias workers to answer
incorrectly due to their difficulty. Moreover, when the estimated values for
this term are greater than zero for most responses, it means SDR deems them
more likely to be correct. With more of them deemed correct, the number of
inferred correct answers to any question tends to increase. As a result, the
size of latent preference set $\mathcal{M}$ should grow, from the perspective
of SDR, to fit the seemingly more diverse set of correlations between latent
correct answers across the questions. Thus, for our model to recover the right
number of latent preferences for crowd-workers from their responses, the
priors for $e_{i}$ and $d_{j}$ need to be set properly, which will be
elaborated more in section 5.1.

##  4\. Estimation

###  4.1. Model parameter estimation

We now provide equations used for parameter estimation, using the notation
$\psi_{z_{ij}k}$ from Eq. (3) and $f(e_{i},d_{j})=f_{ij}$ from Eq. (4) to
simplify the equations. The conditional probability for the preference
assignment $z_{ij}$ to worker $i$ when answering question $j$ given the model
parameters is:

(5) $\displaystyle
P(z_{ij}\\!=\\!m|e_{i},d_{j},\boldsymbol{u}_{m},\boldsymbol{v}_{j},\alpha)\propto\sum\limits_{k\in\mathcal{K}}\psi_{mk}f^{\delta_{ijk}}_{ij}\bigg{(}\frac{1-f_{ij}}{K-1}\bigg{)}^{1-\delta_{ijk}}\frac{N^{m}_{i\neg{j}}+\alpha_{m}}{\sum\limits_{m\in\mathcal{M}}N^{m^{\prime}}_{i\neg{j}}+\alpha_{m^{\prime}}}$

where $N^{m}_{i\neg{j}}$ denotes the number of questions excluding question
$j$ answered by worker $i$ given her preference $m$. The joint probability of
the other parameters given $z_{ij}$ and the hyper-parameters is:

(6)
$Q=p\Big{(}\\{e_{i}\\}_{i\in\mathcal{I}},\\{d_{j},\boldsymbol{v}_{j}\\}_{j\in\mathcal{J}},\\{\boldsymbol{u}_{m}\\}_{m\in\mathcal{M}}|\\{\boldsymbol{z}_{ij}\\}_{i\in\mathcal{I},j\in\mathcal{J}},\mu_{\\{e,d,u,v\\}},\sigma^{2}_{\\{e,d,u,v\\}}\Big{)}\\\
=-\sum\limits_{i\in\mathcal{I}}\sum\limits_{j\in\mathcal{J}}\log\Bigg{[}\sum\limits_{k\in\mathcal{K}}\Bigg{(}\psi_{z_{ij}k}f^{\delta_{ijk}}_{ij}\bigg{(}\frac{1-f_{ij}}{K-1}\bigg{)}^{1-\delta_{ijk}}\Bigg{)}\Bigg{]}-\sum\limits_{i\in\mathcal{I}}\log\big{(}p(e_{i}|\mu_{e},\sigma^{2}_{e})\big{)}-\\\
\sum\limits_{j\in\mathcal{J}}\log\big{(}p(d_{j}|\mu_{d},\sigma^{2}_{d})\big{)}-\sum\limits_{k\in\mathcal{K}}\sum\limits_{m\in\mathcal{M}}\log\big{(}p(u_{mk}|\mu_{u},\sigma^{2}_{u})\big{)}-\sum\limits_{j\in\mathcal{J}}\log\big{(}p(v_{jk}|\mu_{v},\sigma^{2}_{v})\big{)}$

The partial derivatives for $Q$ with respect to the other parameters:

(7) $\displaystyle\frac{\partial Q}{\partial
u_{mk}}=-\sum\limits_{i\in\mathcal{I}}\sum\limits_{j\in\mathcal{J}}\zeta_{ijm}v_{jk}\sum\limits_{k^{\prime}\in\mathcal{K}}f^{\delta_{ijk^{\prime}}}_{ij}\bigg{(}\frac{1-f_{ij}}{K-1}\bigg{)}^{1-\delta_{ijk^{\prime}}}\omega_{ijkk^{\prime}}+\frac{u_{mk}-\mu_{u}}{\sigma^{2}_{u}}$
(8) $\frac{\partial Q}{\partial
v_{jk}}=-\sum\limits_{i\in\mathcal{I}}u_{z_{ij}k}\sum_{k^{\prime}\in\mathcal{K}}f^{\delta_{ijk^{\prime}}}_{ij}\bigg{(}\frac{1-f_{ij}}{K-1}\bigg{)}^{1-\delta_{ijk^{\prime}}}\omega_{ijkk^{\prime}}+\frac{v_{jk}-\mu_{v}}{\sigma^{2}_{v}}$
(9) $\frac{\partial Q}{\partial
e_{i}}=-\sum\limits_{j\in\mathcal{J}}\sum_{k\in\mathcal{K}}\bigg{(}\frac{-1}{K-1}\bigg{)}^{1-\delta_{ijk}}f_{ij}(1-f_{ij})\psi_{z_{ij}k}+\frac{e_{i}-\mu_{e}}{\sigma^{2}_{e}}$
(10) $\frac{\partial Q}{\partial
d_{j}}=-\sum\limits_{i\in\mathcal{I}}\sum_{k\in\mathcal{K}}(-1)^{\delta_{ijk}}\bigg{(}\frac{1}{K-1}\bigg{)}^{1-\delta_{ijk}}f_{ij}(1-f_{ij})\psi_{z_{ij}k}+\frac{d_{j}-\mu_{d}}{\sigma^{2}_{d}}$

Here,
$\delta_{ijk}=\mathbbm{1}{\\{r_{ij}=k\\}},\zeta_{ijm}=\mathbbm{1}\\{z_{ij}=m\\}$
and
$\omega_{ijkk^{\prime}}=\psi_{z_{ij}k}\big{(}1-\psi_{z_{ij}k}\big{)}^{\mathbbm{1}\\{k=k^{\prime}\\}}\big{(}-\psi_{z_{ij}k^{\prime}}\big{)}^{\mathbbm{1}\\{k\neq
k^{\prime}\\}}$. The parameter estimation involves two alternating procedures:
sample $z_{ij}$ according to Eq. (5) and optimize $Q$ in Eq. (6) using LBFGS
based on Eq. (7), (8), (9) and (10).

###  4.2. True answer estimation

A single worker-specific correct answer $l_{ij}$ (as perceived by worker $i$)
fails to provide overall information about the correct answers to question
$j$. Thus, we should gather the $l_{ij}$ values from all workers who answer
each question. However, in practice, each question is assigned to only a
limited number (usually 3 or 5) of workers, making the estimate of the true
answer distribution poor. Our solution to improving this estimate is to first
find underlying clusters of workers (across all questions) by applying K-means
with the Elbow method based on 10-fold cross validation to the posterior means
$\hat{\boldsymbol{\Phi}}=\\{\hat{\boldsymbol{\phi}}_{i}|i\in\mathcal{I}\\}$ of
the latent preference probabilities of all the workers. With the centroid
$\hat{\boldsymbol{\phi}}_{c}$ of each resulting cluster $c$, we then calculate
the probability that the true answer $l_{cj}$ (as perceived by the workers in
cluster $c$) takes the value $k$ as follows:

(11)
$\displaystyle\begin{split}P(l_{cj}=k|c)=\sum_{m\in\mathcal{M}}P(l_{cj}=k|m)P(m|c)=\sum_{m\in\mathcal{M}}\Bigg{(}\frac{\exp(\hat{u}_{mk}\hat{v}_{jk})}{\sum_{k^{\prime}=1}^{K}\exp(\hat{u}_{mk^{\prime}}\hat{v}_{jk^{\prime}})}\hat{\phi}_{mc}\Bigg{)}\end{split}$

where $\hat{u}_{mk}$ and $\hat{v}_{jk}$ are the estimates of the weight
$u_{mk}$ for preference $m$ and the latent feature $v_{jk}$ of question $j$,
both specific to option $k$. The best guess regarding the correct answer
$l_{cj}$ according to the workers assigned to cluster $c$ is then:

(12) $\displaystyle\hat{l}_{cj}=\arg\max_{k\in\mathcal{K}}P(l_{cj}=k|c)$

Now we have a set of correct answer estimates
$\hat{\mathcal{L}}_{j}=\\{\hat{l}_{cj}|c\in\mathcal{C}\\}$ for question $j$
from all the worker clusters (with $\mathcal{C}$ being the set of the
clusters). For the task of true answer prediction, we can either arbitrarily
choose one from $\hat{\mathcal{L}}_{j}$ as the estimate of the correct answer
$l_{j}$ or choose by following certain strategies. Two simple strategies are
to choose $\hat{l}_{cj}$ from the cluster $c$ with the highest average
expertise over its workers, or from the cluster with the largest proportion of
workers assigned to it. The first strategy states that the correct answer
perceived by on average the most expert group of workers is the most
appropriate, while the second assumes it to be the one perceived by the
largest group of workers which represents the mainstream school-of-thought. In
this paper, we apply the second strategy because most crowdsourcing datasets
used in the experiments correspond to relatively simple tasks, where the
provided correct answers we believe are more likely to be mainstream opinions.
As for the first strategy, it might be more useful than the second for
revealing a minority group of expert workers who show distinct preferences on
partially or purely subjective questions from the majority of less expert
workers.

###  4.3. Subjectivity estimation

Despite not being directly estimated in the model, question subjectivity can
still be quantified and estimated after the model has been estimated. This is
achieved based on the reasonable assumption that the subjectivity of each
question is proportional to the number of correct answers it affords. Despite
not knowing the actual number of correct answers $|\mathcal{L}_{j}|$ to
question $j$, we can estimate the value by taking its expectation with respect
to the clusters of workers derived in section 4.2. More precisely, the
expected number of correct answers to question $j$ with respect to worker
clusters $\mathcal{C}$ is:
$\mathbb{E}_{\mathcal{C}}[|\mathcal{L}_{j}|]=\sum_{n=1}^{|\mathcal{K}|}nP_{\mathcal{C}}(|\mathcal{L}_{j}|=n)$.
In this equation, $n$ iterates over the possible number of correct answers
(from 1 to the size of $\mathcal{K}$). The probability
$P_{\mathcal{C}}(|\mathcal{L}_{j}|=n)$ denotes how likely it is that the
number of correct answers to question $j$ equals $n$, with respect to the
worker clusters $\mathcal{C}$. It is, however, difficult to calculate this
probability when $\mathcal{C}$ and $\mathcal{K}$ are large due to a
combinatorial explosion. Thus we apply Monte Carlo simulation to estimate (a
measure of) the subjectivity of question $j$ as
$\mathbb{E}_{\mathcal{C}}[|\mathcal{L}_{j}|]$ using Alogrithm 1.

Algorithm 1 Subjectivity estimation for question $j$

1:$\hat{v}_{j};\;\\{\hat{\boldsymbol{u}}_{m}\\}_{m\in\mathcal{M}};\;\;\hat{\boldsymbol{\Phi}}_{c}=\\{\hat{\boldsymbol{\phi}}_{c}\\}_{c\in\mathcal{C}};\;\;T=50,000$.

2:$\mathbb{E}_{\mathcal{C}}[|\mathcal{L}_{j}|]$.

3:$n_{j}\leftarrow 0$.  /* Initialise number of correct answers for question
$j$ to zero */

4:for $t=1...T$ do /* Sample over $T$ iterations.*/

5: $\hat{\mathcal{L}_{j}}\leftarrow\\{\\}$.  /* Initialise set of correct
answers to be sampled at iteration $t$. */

6: for $c=1...C$ do

7: $z_{cj}\sim Discrete(\hat{\boldsymbol{\phi}}_{c})$.  /* Sample group
preference assignment $z_{cj}$. */

8:
$\hat{l}_{cj}\sim\boldsymbol{\psi}_{z_{cj}},\;\text{where}\;\psi_{z_{cj}k}=\exp(\hat{u}_{z_{cj}k}\hat{v}_{jk})/\sum\limits_{k^{\prime}\in\mathcal{K}}\exp(\hat{u}_{z_{cj}k^{\prime}}\hat{v}_{jk^{\prime}})$.
/* Sample correct answer $\hat{l}_{cj}$ perceived by worker cluster $c$. */

9: $\hat{\mathcal{L}}_{j}\leftarrow\hat{\mathcal{L}}_{j}\cup\hat{l}_{cj}\;$
only if $\;\hat{l}_{cj}\not\in\hat{\mathcal{L}}_{j}$ /* Add sampled
$\hat{l}_{cj}$ to $\hat{\mathcal{L}_{j}}$ when it first appears. */

10: end for

11: $n_{j}\leftarrow n_{j}+|\hat{\mathcal{L}}_{j}|$. /* Increase $n_{j}$ by
number of distinct correct answers sampled at $t$. */

12:end for

13:$\mathbb{E}_{\mathcal{C}}[|\mathcal{L}_{j}|]\approx\frac{n_{j}}{T}$. /*
Divide $n_{j}$ by $T$ to estimate
$\mathbb{E}_{\mathcal{C}}[|\mathcal{L}_{j}|]$ as the question’s subjectivity.
*/

##  5\. Experiments

The evaluation of our proposed model consists of four parts. The first part is
its sensitivity to various degrees of subjectivity in different crowdsourcing
tasks. The second and the third parts are its performance of predicting
respectively the provided correct answers of questions and the answers to be
given by crowd-workers to unseen questions. The last part is its consistency
with human assessors in assessing the difficulty and the subjectivity of
questions. We have used 10 crowdsourcing datasets to evaluate the performance
of our model in the experiments corresponding to the four parts. Table 1
summarises these datasets as being either (primarily) objective or partially
subjective. Among them, the identification tasks of event time ordering, dog
and duck breeds, and same products concern objective factual knowledge, while
the judgement tasks of image beauty, document relevance 1&2444The questions of
relevance judgement task 2 come from the part of TREC 2011 crowdsourcing track
(Lease and Kazai, 2011) that does not contain the questions of relevance
judgement task 1. We collected crowdsourced judgements for the task 2 from
CrowdFlower., facial expression and adult content intrinsically contain
certain degrees of subjectivity.

###  5.1. SDR hyper-parameter setup

As discussed at the end of section 3, to find the right number of latent
preferences for crowd-workers, the hyper-parameters of the expertise $e_{i}$
and the difficulty $d_{j}$ in the SDR model need to be carefully set. This is
achieved through held-out validation which leverages noise within worker
responses for detecting signs that SDR may be overfitting the responses by
introducing more latent preferences than necessary. More specifically, we
construct a held-out validation dataset by randomly sampling a response from
each worker. Thus, the size of such a dataset equals the number of workers
participating in a task. Then, given a certain setting of the hyper-
parameters, we learn our model based on the remaining responses and use the
parameter estimates from the learned model to calculate the prediction
accuracy: $1\\!-\\!MAE$ (Mean Absolute Error) over the held-out dataset. We
repeat the model learning process with each hyper-parameter setting over the
same 100 random held-out validation data subsets. We then obtain the average
prediction accuracy for our model across these subsets for each hyper-
parameter setting. Finally, we choose the setting (including the number for
latent preferences) that yields the highest average prediction accuracy for
use in the experiments.

###  5.2. Sensitivity analysis

We first verify whether our model is sensitive to various degrees of
subjectivity in different crowdsourcing tasks. If a task is (almost entirely)
objective, the optimal size of latent preference set $\mathcal{M}$ should be
1, meaning that every crowd-worker now perceives the correct answers in the
same way. Consequently, the probabilities of latent preferences
$\boldsymbol{\phi}_{i}$ for worker $i$ collapse to $\phi_{i}=1$, and the set
of correct answers $\mathcal{L}_{j}$ for question $j$ collapses to a single
correct answer $l_{j}$. In this case, we conduct the held-out validation on
our model across the objective datasets each with the 100 randomly sampled
data subsets described in section 5.1. We expect that the average held-out
prediction accuracy for our model across these data subsets will decrease when
the number of latent preferences it has increases from 1 to 2, since in this
case the model starts to overfit by learning the noise in the training
responses to those objective tasks.

If a task is sufficiently subjective, our model should uncover the right
number of underlying groups of workers along with the right number of latent
preferences. We conduct the experiment in the same way as above to see the
difference in average prediction accuracy on held-out unseen responses with
the number of preferences increasing from 1 to 3 over the partially subjective
datasets. We expect the average prediction accuracy to be higher when the
number of preferences is greater than 1. Moreover, since Tian and Zhu (Tian
and Zhu, 2012) provided us with the number of worker clusters emerging
respectively from the five sub-tasks which constitute the image data in Table
1, we thus compare the corresponding numbers of clusters derived from our
model with theirs.

Table 1. The objective and the partially subjective datasets used in this paper.  Objective datasets | # Worker | # Item | # Response
---|---|---|---
Time (Snow et al., 2008) | 76 | 462 | 4,620
Dog (Zhou et al., 2012) | 109 | 807 | 8,070
Duck (Welinder et al., 2010) | 53 | 240 | 9,600
Product (Wang et al., 2012) | 176 | 8,315 | 24,945
Partially subjective datasets | # Worker | # Item | # Response
Image (Tian and Zhu, 2012) | 402 | 60 | 24,120
Rel1 (Buckley et al., [n. d.]) | 642 | 1,787 | 13,310
Rel2 (Lease and Kazai, 2011) | 83 | 585 | 1,755
Fashion (Loni et al., 2013) | 199 | 3,837 | 11,511
Face (Mozafari et al., 2012) | 27 | 584 | 5,242
Adult (adu, [n. d.]) | 269 | 333 | 3,324

###  5.3. Question correct answer prediction

To verify the ability of the SDR model to predict the question true answers,
we compare it with the following state-of-the-art quality control methods for
crowdsourcing. All of these methods assume that each question has a single
correct answer.

  * •

Majority Vote (MV): The predicted correct answer for each question is the one
chosen by the majority of the workers.

  * •

Multi-dimensional Wisdom of Crowds (MdWC) (Welinder et al., 2010): This model
endows both crowd-workers and questions with multi-dimensional latent factors,
and provides the workers with additional variables to account for their
answering biases.

  * •

Generative model of Labels, Abilities, & Difficulties (GLAD) (Whitehill et
al., 2009b): This model resembles MdWC except that its latent factors
(interpreted respectively as expertise and difficulty) are uni-dimensional,
and it does not have worker-specific bias variables.

  * •

Dawid-Skene (DS) (Dawid and Skene, 1979): Unlike GLAD and MdWc which model the
correctness probability of each worker’s response, this model focuses on the
(worker-specific) conditional probability of each response option given the
correct answer to each question.

  * •

Community Dawid-Skene (CDS) (Venanzi et al., 2014): This model extends DS by
clustering workers over some latent structures imposed on their conditional
response probability matrices (given all correct answer possibilities) to
alleviate the response sparsity problem.

The performance measure: _correct answer prediction accuracy_ , is calculated
as:
$\frac{1}{|\mathcal{J}|}\sum_{j\in\mathcal{J}}\mathbbm{1}\\{l_{j}=\hat{l}_{j}\\}$,
where $\hat{l}_{j}$ is inferred from the respective baselines. For our model,
it is:
$\frac{1}{|\mathcal{J}|}\sum_{j\in\mathcal{J}}\mathbbm{1}\\{l_{j}=\hat{l}_{cj}\\}$,
where $c=\arg\max_{c\in\mathcal{C}}N_{c}$ with $N_{c}$ the number of workers
assigned to cluster $c$ after Elbow K-means, and $\hat{l}_{cj}$ calculated by
Eq. (12). The hyper-parameters for each baseline except MV are optimised using
the held-out validation specified in section 5.1 on the exact same random
held-out validation subsets of each dataset in Table 1.

###  5.4. Worker answer prediction

Predicting the answers to be given by crowd-workers to unseen questions is
much more significant for (partially) subjective crowdsourcing tasks than it
is for the objective ones as the former type of tasks values more about the
different ways workers respond. For example, it is crucial to employ worker
answer prediction to test a recommender system built on crowdsourced ratings.
In this experiment, we evaluate the performance of all the models except MV on
predicting the next answer from each worker. We first sample one answer from
each worker to create a held-out test dataset, and then learn all the models
from the rest of the data with their hyper-parameters optimised as described
in section 5.1 using the exact same random validation data subsets. Finally,
we evaluate the prediction performance of the models on the held-out test data
using (1 - MAE). Due to the limitation of our computing power, in this
experiment, we reduce the number of held-out validation iterations for each
model to be 15 before a single iteration of held-out test is conducted. We
perform 15 such random tests before the average performance of each model is
elicited.

(a)

(b)

Figure 3. (a) shows the 3 worker clusters on identifying sky from images and
(b) shows the 4 worker clusters on judging beautiful images.

###  5.5. Subjectivity and difficulty coherence

In this experiment, we investigate whether the estimates of the difficulty and
the subjectivity of questions derived from the SDR model are consistent with
the judgements of five human assessors. We focused on the object
identification & image aesthetics task555Crowd-workers are asked whether an
image is beautiful or not. from (Tian and Zhu, 2012) as the total number of
its questions is 60, a manageable workload for the assessors to provide good-
quality judgements with sufficient levels of effort and concentration. The
assessors are either PhD or Master students, three of whom are avid
photographers with adequate knowledge about what constitutes beautiful images,
while the other two are novices who, during the group discussion, provided
suggestions as to how novices might react to different images. We ask them to
rank the images with respect to (i) difficulty and (ii) subjectivity. The
respective instructions we gave to them are:"rank all these images by how hard
they are for crowd-workers to judge correctly by avoiding possible incorrect
answers" and "rank them this time by how subjective they are for crowd-workers
to judge". The assessors first independently came up with their two rankings.
In the process, they could redo the two ranking tasks until they felt
confident to submit. The assessors then worked together to merge their
rankings into single rankings (for both difficulty and subjectivity) through
group discussion and majority vote. The resulting rankings were then compared
with the corresponding rankings based on the estimates from the learned SDR
model. In addition to ranking the images, the assessors were also asked to
categorise each image into one of the three levels of difficulty (namely
_easy_ , _medium_ , and _hard_), and into one of the three levels of
subjectivity (namely _objective_ , _partially subjective_ , and _purely
subjective_). We did this to see whether there existed any correlation between
the difficulty or subjectivity levels to which images were categorised, and
their corresponding estimates from the model.

##  6\. Results

(a)

(b)

(c)

(d)

(e)

(f)

Figure 4.  (a) and (d) show the correlation of the difficulty estimates and
that of the subjectivity estimates respectively with the corresponding
rankings judged by human assessors, while (b) and (e) show the correlation of
the difficulty estimates and that of the subjectivity estimates respectively
with the corresponding levels to which the images have been categorized by the
assessors. Finally, (c) shows the images as points with coordinates being the
difficulty and the subjectivity estimates, and has highlighted some images
with noteworthy coordinates, while (d) shows these images.

The results of the sensitivity analysis described in section 5.2 are shown in
Tables 2 and 3. We can see from Table 2 that the average prediction accuracy
of the SDR model with 1 latent preference is constantly higher than that of
the model with 2 preferences over all the objective datasets. According to
section 5.2, this result indicates there is just one underlying group of
workers for each of the tasks who perceive the questions’ correct answers in
the same way. It also shows that even though the expertise-difficulty
corruption introduced noises to the objective truths to form the actual
responses, our model was still able to recover the number of underlying group
of workers to be 1. From Table 3, our model with 2 preferences clearly
outperforms itself with 1 preference across all the partially subjective
tasks. This means multiple groups of workers have emerged due to the
sufficient subjectivity of these tasks. Moreover, the table shows that further
increasing the number of latent preferences to 3 no longer improves the
performance. This has most likely been caused by over-fitting, and also
suggests a two-dimensional latent space is accurate enough to explain the
worker clustering effects emerged from these tasks. To further prove our model
with 2 preferences can uncover the underlying groups of workers who have
perceived the partially subjective tasks differently, we show the density of
the workers’ latent preference probabilities $\hat{\boldsymbol{\phi}}_{i}$
estimated by our model from the image data (Tian and Zhu, 2012). Due to a
space limit, we only show two of them in Figure 3. According to (Tian and Zhu,
2012), the sub-task of judging whether images are beautiful is more subjective
than the sub-task of identifying skies in images. This is re-confirmed by our
model with its number of worker clusters for the former sub-task greater than
that for the latter shown by Figures 3(a) and 3(b).

Table 2.  Average accuracy of our model with 1 and 2 latent preferences on predicting the held-out validation response of each worker over 4 objective tasks.  Dataset |  The SDR model
---|---
m = 1 | m = 2
Time | 0.8967 | 0.8915
Dog | 0.6970 | 0.6625
Duck | 0.8427 | 0.8388
Product | 0.8396 | 0.8291
Table 3. Average accuracy of our model with 1, 2 and 3 latent preferences on predicting the held-out validation response of each worker over 10 partially subjective tasks the first 5 of which are sub-tasks of the Image task in (Tian and Zhu, 2012). Dataset |  The SDR model
---|---
m = 1 | m = 2 | m = 3
Beauty 1 | 0.6736 | 0.6944 | 0.6924
Beauty 2 | 0.6914 | 0.6998 | 0.6937
Sky | 0.8889 | 0.8962 | 0.8862
Building | 0.8997 | 0.9026 | 0.9007
Computer | 0.8098 | 0.8117 | 0.8074
Rel1 | 0.3956 | 0.3985 | 0.3983
Rel2 | 0.4426 | 0.4481 | 0.4481
Fashion | 0.7517 | 0.7589 | 0.7522
Face | 0.7181 | 0.7203 | 0.7123
Adult | 0.7469 | 0.7494 | 0.7446

The results of the question correct answer prediction described in section 5.3
are listed in Table 4. Across all the partially subjective datasets except the
image data, the SDR model, based on the largest-group strategy for choosing
the best worker clusters, is superior than the other 5 baselines666The
performance of all the models on the image data (in Table 1) has been too
close to bear any useful information as for which of them is better since the
number of the questions (i.e. 60) in the data is too small.. Especially, for
the tasks of relevance judgement 1&2 and fashion judgement, our model is able
to outperform the best baselines by 3%, 1.5% and 0.3% with almost 54, 9 and 13
more correctly predicted question answers respectively. Since our model is
reduced to being very similar to GLAD when dealing with the objective
datasets, it has achieved very similar results as GLAD did in correct answer
prediction over all the objective datasets except for the Duck data (Welinder
et al., 2010). In this task, our model is superior than GLAD (0.69 versus 0.62
from GLAD). This suggests that our model is at least as robust as GLAD when
predicting correct answers for objective tasks.

Table 4.  Accuracy of all the models on predicting the true answers of the four partially subjective datasets (the results for the Image task are not included as the number of items in this task is too small to show any significant difference in the performance of different models).  Dataset |  Question correct answer prediction
---|---
SDR | MV | GLAD | DS | CDS | MdWC
Rel1 | 0.4998 | 0.4522 | 0.4457 | 0.4309 | 0.4697 | 0.4674
Rel2 | 0.4752 | 0.4544 | 0.4567 | 0.4512 | 0.4604 | 0.4586
Fashion | 0.8733 | 0.8580 | 0.8689 | 0.8415 | 0.8463 | 0.8700
Face | 0.6423 | 0.6404 | 0.6130 | 0.5924 | 0.5986 | 0.6079
Adult | 0.7598 | 0.7568 | 0.7587 | 0.7534 | 0.7582 | 0.7556

The results of the worker answer prediction described in section 5.4 are shown
in Table 5. We can see that our model is not the best on 3 out of the 10
partially subjective datasets, topped by different baselines. Despite that,
our model has still performed adequately well (being second best on those
datasets). We conjecture that this is because all these 3 datasets are with
binary answer options which intrinsically constrain the answering behaviour of
crowd-workers. This results in overall weaker correlations both in the worker
responses and in the underlying correct answers across the questions. For the
other 7 datasets, 5 of them are with more than two answer options, thus
containing stronger answer correlations for our model to exploit to achieve
better performance. To examine whether the difference in the worker answer
prediction accuracy between any two algorithms is significant, we conducted
the Nemenyi post-hoc test (Demšar, 2006) based on Table 5. The result is shown
in Figure 5, according to which the performance difference between SDR and
either CDS, GLAD or DS is beyond the critical difference (CD), thereby being
statistically significant.

Table 5. Average Accuracy of all the models on predicting the unseen held-out test response of each worker across all the partially subjective datasets. Dataset |  Unseen worker answer prediction
---|---
SDR | GLAD | DS | CDS | MdWC
Beauty 1 | 0.6974 | 0.6884 | 0.6256 | 0.6927 | 0.6912
Beauty 2 | 0.7006 | 0.7011 | 0.6796 | 0.6842 | 0.6998
Sky | 0.9014 | 0.8772 | 0.8801 | 0.8862 | 0.8903
Building | 0.8987 | 0.8912 | 0.8956 | 0.9006 | 0.8976
Computer | 0.8284 | 0.8139 | 0.8115 | 0.8196 | 0.8336
Rel1 | 0.4067 | 0.4035 | 0.3654 | 0.3972 | 0.3987
Rel2 | 0.4386 | 0.4312 | 0.4257 | 0.4304 | 0.4340
Fashion | 0.7659 | 0.7593 | 0.6977 | 0.7621 | 0.7633
Face | 0.7224 | 0.7193 | 0.6625 | 0.7081 | 0.7148
Adult | 0.7386 | 0.7347 | 0.6767 | 0.7312 | 0.7354
Figure 5. Critical difference (CD) diagram of the Nemenyi post-hoc test
($\alpha=0.10$). The performance difference between two algorithms is
significant if the gap between their ranks is larger than CD. There is a
horizontal line connecting the two algorithms if the rank gap between them is
smaller than CD.

The results of the subjectivity and difficulty coherence evaluation have been
summarised in Figure 4 which consists of 6 sub-figures. Figures 4(a) and 4(d)
show overall there is a strong negative correlation between the model
estimates and the rankings judged by human assessors. More specifically, the
larger the estimate for either the difficulty or the subjectivity of an image,
the higher it tends to be ranked by human assessors. Moreover, Figures 4(b)
and 4(e) show that there exist clear positive correlations between the levels
of difficulty and subjectivity into which the images get categorised by the
human assessors, and the estimated values of these two properties inferred by
the SDR model.

To support our argument about the efficacy of the SDR model in revealing the
two key properties of images, we have selected four images highlighted in
different colours in Figure 4(c) with their image ids. We can see that image
34 is inferred by our model to be both easy and objective as both of its
estimates shown in Figure 4(c) are the smallest. This can be re-confirmed by
visual inspection of the image in Figure 4(f). It is very easy and clear to
see that there is no sky in the image 34. Image 29 has been identified by our
model to be hard with low subjectivity according to its estimates shown in
Figure 4(c). This is reasonable as the image indeed contains an
extraterrestrial sky which is hard for novice workers to realise, while expert
workers are able to realise and find the image objective. Images 2 and 23 both
belong to the image beauty judgement task from (Tian and Zhu, 2012) which
requires workers to select 6 most beautiful images from 12 images. Our model
has identified that image 2 is more subjective and harder to judge. This is
probably because image 2 delivers a view of scenery which is more likely to
resonate with workers while image 23 is merely showing an object. As a result,
workers tend to show more different feelings and opinions towards image 2. On
the other hand, image 23 does have better image quality and thus is easier for
workers to make their decisions on whether it is beautiful or not.

##  7\. Conclusions

In this paper, we have proposed the SDR (Subjectivity-and-Difficulty Response)
model, a novel quality-control framework for crowdsourcing that is able to
distinguish question subjectivity, which causes worker-specific truth for
individual questions, from question difficulty, which determines the
probability that a worker’s response to each question equals her perceived
subjective truth. Experiment results show that our model improves both the
correct answer prediction for questions and the held-out unseen response
prediction for crowd-workers compared to five baselines across numerous
partially subjective crowdsourcing datasets. Moreover, our model shows
robustness to both the objective and the partially subjective datasets by
discovering the right numbers of underlying worker groups for them. Finally,
our model is able to provide estimates of the difficulty and the subjectivity
of questions that are consistent with the judgements from human assessors.

## References

  * (1)
  * adu ([n. d.]) [n. d.].  Adult Datset.  https://github.com/ipeirotis/Get-Another-Label/tree/master/data. ([n. d.]).  Accessed: 2017-07-30.
  * Bachrach et al. (2012) Yoram Bachrach, Thore Graepel, Tom Minka, and John Guiver. 2012\.  How to grade a test without knowing the answers—a Bayesian graphical model for adaptive crowdsourcing and aptitude testing.  arXiv preprint arXiv:1206.6386 (2012).
  * Blei et al. (2003) David M Blei, Andrew Y Ng, and Michael I Jordan. 2003\.  Latent Dirichlet allocation.  Journal of machine Learning research 3, Jan (2003), 993–1022.
  * Buckley et al. ([n. d.]) Chris Buckley, Matthew Lease, and Mark D Smucker. [n. d.].  Overview of the trec 2010 relevance feedback track (notebook).
  * Dawid and Skene (1979) Alexander Philip Dawid and Allan M Skene. 1979.  Maximum likelihood estimation of observer error-rates using the EM algorithm.  Applied statistics (1979), 20–28.
  * Demšar (2006) Janez Demšar. 2006\.  Statistical comparisons of classifiers over multiple data sets.  Journal of Machine Learning Research 7 (2006), 1–30.
  * Jung (2014) Hyun Joon Jung. 2014\.  Quality assurance in crowdsourcing via matrix factorization based task routing. In Proceedings of the 23rd International Conference on World Wide Web. ACM, 3–8.
  * Jung and Lease (2013) Hyun Joon Jung and Matthew Lease. 2013.  Crowdsourced task routing via matrix factorization.  arXiv preprint arXiv:1310.5142 (2013).
  * Kamar et al. (2015) Ece Kamar, Ashish Kapoor, and Eric Horvitz. 2015.  Identifying and accounting for task-dependent bias in crowdsourcing. In Third AAAI Conference on Human Computation and Crowdsourcing.
  * Koren et al. (2009) Yehuda Koren, Robert Bell, and Chris Volinsky. 2009\.  Matrix factorization techniques for recommender systems.  Computer 42, 8 (2009).
  * Lease and Kazai (2011) Matthew Lease and Gabriella Kazai. 2011.  Overview of the trec 2011 crowdsourcing track. In Proceedings of the text retrieval conference (TREC).
  * Lee et al. (2013) Jongwuk Lee, Myungha Jang, Dongwon Lee, Won-Seok Hwang, Jiwon Hong, and Sang-Wook Kim. 2013\.  Alleviating the sparsity in collaborative filtering using crowdsourcing. In Workshop on Crowdsourcing and Human Computation for Recommender Systems (CrowdRec), Vol. 5.
  * Loni et al. (2013) Babak Loni, Maria Menendez, Mihai Georgescu, Luca Galli, Claudio Massari, Ismail Sengor Altingovde, Davide Martinenghi, Mark Melenhorst, Raynor Vliegendhart, and Martha Larson. 2013.  Fashion-focused Creative Commons Social Dataset. In Proceedings of the 4th ACM Multimedia Systems Conference (MMSys ’13). ACM, New York, NY, USA, 72–77.
  * Mozafari et al. (2012) Barzan Mozafari, Purnamrita Sarkar, Michael J. Franklin, Michael I. Jordan, and Samuel Madden. 2012\.  Active Learning for Crowd-Sourced Databases.  CoRR abs/1209.3686 (2012).
  * Nguyen et al. (2016) An Thanh Nguyen, Matthew Halpern, Byron C Wallace, and Matthew Lease. 2016.  Probabilistic modeling for crowdsourcing partially-subjective ratings.
  * Rasch (1993) Georg Rasch. 1993\.  Probabilistic models for some intelligence and attainment tests. ERIC.
  * Raykar et al. (2010) Vikas C Raykar, Shipeng Yu, Linda H Zhao, Gerardo Hermosillo Valadez, Charles Florin, Luca Bogoni, and Linda Moy. 2010.  Learning from crowds.  Journal of Machine Learning Research 11, Apr (2010), 1297–1322.
  * Snow et al. (2008) Rion Snow, Brendan O’Connor, Daniel Jurafsky, and Andrew Y Ng. 2008.  Cheap and fast—but is it good?: evaluating non-expert annotations for natural language tasks. In Proceedings of the conference on empirical methods in natural language processing. Association for Computational Linguistics, 254–263.
  * Tian and Zhu (2012) Yuandong Tian and Jun Zhu. 2012.  Learning from crowds in the presence of schools of thought. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 226–234.
  * Venanzi et al. (2014) Matteo Venanzi, John Guiver, Gabriella Kazai, Pushmeet Kohli, and Milad Shokouhi. 2014.  Community-based bayesian aggregation models for crowdsourcing. In Proceedings of the 23rd international conference on World wide web. ACM, 155–164.
  * Voorhees (2000) Ellen M Voorhees. 2000\.  Variations in relevance judgments and the measurement of retrieval effectiveness.  Information processing & management 36, 5 (2000), 697–716.
  * Wang et al. (2012) Jiannan Wang, Tim Kraska, Michael J Franklin, and Jianhua Feng. 2012\.  Crowder: Crowdsourcing entity resolution.  Proceedings of the VLDB Endowment 5, 11 (2012), 1483–1494.
  * Welinder et al. (2010) Peter Welinder, Steve Branson, Pietro Perona, and Serge J Belongie. 2010.  The multidimensional wisdom of crowds. In Advances in neural information processing systems. 2424–2432.
  * Whitehill et al. (2009a) Jacob Whitehill, Paul Ruvolo, Tingfan Wu, Jacob Bergsma, and Javier R. Movellan. 2009a.  Whose Vote Should Count More: Optimal Integration of Labels from Labelers of Unknown Expertise. In 23rd Annual Conference on Neural Information Processing Systems, NIPS’09. 2035–2043.
  * Whitehill et al. (2009b) Jacob Whitehill, Ting-fan Wu, Jacob Bergsma, Javier R Movellan, and Paul L Ruvolo. 2009b.  Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Advances in neural information processing systems. 2035–2043.
  * Zhou et al. (2012) Denny Zhou, Sumit Basu, Yi Mao, and John C. Platt. 2012\.  Learning from the Wisdom of Crowds by Minimax Entropy.  In Advances in Neural Information Processing Systems 25, F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger (Eds.). Curran Associates, Inc., 2195–2203.  http://papers.nips.cc/paper/4490-learning-from-the-wisdom-of-crowds-by-minimax-entropy.pdf