# A Study on the Effects of Diffusion of Information on Epidemic Spread

Semra Gündüç
Department of Computer Engineering
Faculty of Engineering, Ankara University
06345 Gölbası Ankara, Turkey

###### Abstract

In this work, the spread of a contagious disease on a society where the
individuals may take precautions is modeled. The primary assumption is that
the infected individuals transmit the infection to the susceptible members of
the community through direct contact interactions. In the meantime, the
susceptibles gather information from the adjacent sites which may lead to
taking precautions. The SIR model is used for the diffusion of infection while
the Bass equation models the information diffusion. The sociological
classification of the individuals indicates that a small percentage of the
population take action immediately after being informed, while the majority
expect to see some real advantage of taking action. The individuals are
assumed to take two different precautions. The precursory measures are getting
vaccinated or trying to avoid direct contact with the neighbors. A weighted
average of states of the neighbors leads to the choice of action.

The fully connected and Scale-free Networks are employed as the underlying
network of interactions. The comparison between the simple contagion diffusion
and the diffusion of infection in a responsive society showed that a very
limited precaution makes a considerable difference in the speed and the size
of the spread of illness. Particularly highly connected hubs nodes play an
essential role in the reduction of the spread of disease.

Keywords:  Social Networks, Epidemic, SIR model, Diffusion of Information,
Bass Model.

###### Contents

  1. 1 Introduction
  2. 2 Related Work
  3. 3 SIR Model with Element of Information Diffusion- An Agragate approach
  4. 4 Rules of Agent Based Model on Networks
  5. 5 Agent Based Model on Scale Free Networks
  6. 6 Discussions and Conclusions

##  1 Introduction

One of the most critical challenges of public health services is to prevent or
at least control epidemic outbreaks. The process requires the optimization of
the limited resources of time and supply. The resources optimization reaches
its peak when society consists of well-informed individuals. There are various
methods of informing the members of the society on the severity of the
epidemic and the methods of prevention. Planing the methodology of the spread
and increase of awareness together with an efficient vaccination program
requires preliminary modeling studies. Here in this work, a model of epidemic
spread is introduced to study the effect of the awareness on the prevention of
epidemic spread. The main contribution of the present work is the introduction
of a mathematical relation to describe the social behavior of the individual
during an epidemic spread. The social behavior studies of Roger [1] and the
mathematical model introduced by Bass [2] for the diffusion of innovation are
taken as the base of the adoption of information and awareness spread.

The present model aims to integrate the adoption of information with the
spread of contagious diseases using aggregate and agent-based models. The
spreading of infection is modeled by the well-known the susceptible-infected-
removed (recovered) (SIR) model [3] while the Bass model is employed for the
adoption of the information. The Bass model considers two categorically
different class of individuals for the simplicity of the parametrization Bass
[2]. The first group of individuals is named as innovators. The social
behavior studies of Roger [1] indicate that the innovators respond as soon as
they are informed. In the epidemic spreading case the innovators are the
individuals who adopt the idea of taking precaution as soon as they are
informed. The second group is the individuals respond after observing the
benefits of the new situation. The members of this group are called imitators.
The imitators wait to see the decisions of the neighbors to get vaccinated of
taking precaution. The majority of the individuals in a society are imitators.
The information on the spread of contagious disease is formulated by using the
Bass model dynamics while dynamics of the spread of the disease is governed by
the well known SIR model [3]. The SIR model in its original form is an
aggregate model. The members of the society are classified into three groups:
Susceptibles ($S$), infected ($I$) and recovered ($R$). Three coupled first
order differential equations describe the dynamics of the time variation of
the number of $S,\;I$ and $R$ type individuals [3]. SIR model cannot
accommodate intellectual individuals who can respond to incoming flux of
information. For this purpose, the Bass equation added as the fourth
differential equation; hence four coupled differential equations are used for
the realization of concurrent diffusion of knowledge and disease.

The agent-based model of the effects of awareness on the spread of contagious
disease is introduced. In the agent-based approach, the members of the society
(agents) live on the nodes of a network. The simples network is the fully
connected network in which every node is connected with every other node -
everyone knows everyone. Assuming all nodes are equal (uniformity assumption),
the fully connected agent-based models correspond to aggregate models. In
large societies, every individual can only interact with a limited number of
acquaintances. This social fact has opened possibilities of different network
topologies [12]. One of the most commonly observed real-world network topology
is the scale-free networks. In the scale-free networks, every node has a
finite but varying number of connections. A small number of nodes are
connected with a very high number of other nodes while the majority of the
nodes have only a limited number of connections. These highly connected nodes
play the hub role, connecting large parts of the network. Information
diffusion and spreading of the infection modeled on both fully connected and
scale-free networks. The fully connected systems give a similar result to the
aggregate model. The existence of hub nodes, in the scale-free network case
considerably change the situation. It is shown that the awareness level of the
individuals with a high number of neighbors is capable of stopping the disease
spread. Putting the Individuals in the center of decision-making mechanism,
enable the innovator group to defend themselves immediately but at the same
time, the imitators make their decision according to the states of their
neighbors. In both cases, the highly connected nodes naturally contribute to
the prevention of the infection spreading. Comparison of the same society with
different connection topologies the effectiveness of the natural prevention
strategies about the high is studied.

Although the vaccination is considered as the crucial protection strategy,
other prevention methods are usually preferred by the individuals for
uncomplicated virus infections. The individuals start to protect themselves
almost synchronously with the information spread about the disease. Hence the
information and disease spread are tightly related to the prevention of
epidemics, which is the primary motivation for the present work. The proposed
model is based on two inter-woven spreading mechanisms. First one is the
spread of disease and the second one is the spread of the information among
the members of the society. Individual awareness and reaction play an
essential role in combining these two effects. Although a wide range of
studies exists (Section 2), none of them has connected sociological behavior
classifications of individuals [1] with the prevention of disease spreading.
The protection effort is considered as the natural reaction of the members of
the society. This mechanism distinguishes the introduced model from the models
where the source of the prevention mechanism is independent of the
interactions among the members of the community.

The work is organized as follows: The following section is devoted to
outlining the related publications, the aggregate model of diffusion of
contagious disease is discussed in the third section. This section mainly
essential to set the rules of an agent-based model of the epidemic on networks
which is presented in the fourth section. Simulation results of the agent-
based model are given in the fifth section. The final section is reserved for
the discussions of the results and conclusions.

##  2 Related Work

Mathematical modeling of contagious disease spreading has many main lines of
research among which, i) Estimating the model parameters from the existing
data [6, 7], ii) resources opptimization [8, 9, 10], and iii) building models
of artificial societies through which the contagious diseases spread [11]
attracts considerable attention. The present work aims to address the question
of the relationship between social behavior and the spreading the infection
among the members of society.

The classical epidemic models, such as the SIR model assumes a homogeneous
society. The society consists of identical individuals who interact with each
other with the same probability. Societies are not uniform and homogeneous.
Every member of society has a characteristic social behavior. Moreover, the
interaction patterns among individuals are not uniform. Recently patterns of
non-uniformity of the social connections attracted attention. A better
understanding of the connectivity patterns of social networks leads to better
models of social phenomena. In particular, the epidemic models became more
accurate and realistic with a better understanding of the social networks [12]
in terms of complex network theory. Moreover, the complex-network analysis
methods enabled analyzing the dynamics of the transmission of the contiguous
disease more precisely [13, 14, 15, 16, 17]. The scale-free Networks which
cover the majority of social networks contain some nodes with very high
connectivity. The nodes with a large number of neighbors play the crucial role
[18] in spreading the infections. The infection paths are cut by immunizing
the highly connected nodes, which stops the spreading of the disease. The
scale-free networks are robust against random attacks [4]. Hence, random
vaccination cannot stop the spread of contagious disease [19]. Various groups
pursue the idea of immunizing individuals with a high number of neighbors.
Pastor and Vespignani proposed the Targeted immunization method which aims to
protect the society from epidemic behavior by vaccination of highly connected
nodes [19]. The high-risk immunization [20], based on the protection of high-
risk individuals to optimize the number of nodes to be feasibly immunized.
Also, Cohen et al. proposed another type of immunization strategy called
acquaintance immunization: A randomly chosen neighbor of a random node is
vaccinated [21].

All of the methods mentioned above require prior knowledge of the social
network topology. Moreover, outside intervention is necessary to organize the
vaccination of the high degree nodes. Despite all these developments, human
behavior element is only recently has attracted attention [22, 23]. Modern
societies are self-organizing systems. The actions of the individuals evolve
to collective phenomena. Unless it is a state of emergency, very little
outside control is necessary. The members of society take their actions and
react accordingly. Susceptibles may avoid contact with the infected, as a
precaution, by changing their connections [24]. The relation of the judgment
of an epidemic and the behavioral changes are studied in the context of H1N1
pandemic influenza appeared in Italy in 2009 [25]. It is shown that the
individual consciousness reduces the possibility of an uncontrolled spread of
a contegion [26, 27]. Ruan et all [28] proposed prevention strategies which
aim to increase the awareness of the individuals. Different behavioral groups
exhibit different reactions concerning the judgment of an epidemic [29, 30].
In opinion formation or consensus studies, the role of zealots is shown to be
very important. Similarly, the decision on taking necessary procedures against
a contagion vaccination is also an opinion formation process. The role and the
effect of the committed individuals on immunization has also been discussed
[31].

Next section is devoted to the introduction of the proposed model of SIR model
with the awareness spread.

##  3 SIR Model with Element of Information Diffusion- An Agragate approach

The SIR model is based on an artificial society of $N$ individuals. The
individuals are in three categories: susceptible ($S$), infective ($I$) and
removed ($R$) where, $N=S+I+R$. The SIR model assumes a uniform society in
which all individuals are in interaction with all others. Moreover, the
uniformity assumption extends also to the probabilities of the transmission
and recovery rates. Hence, the transmission and recovery rates of the disease
are the same for all members of the society. The SIR model has two free
parameters apart from the size of the society. The first parameter, $\beta$,
represents an average rate of encounters between the infected and susceptible
individuals. The second parameter, $\gamma$, represents the percentage of
recovery per unit time. Infected individuals recover from the illness and gain
immunity. The fractions of the susceptible, infected and removed individuals
can be written as, $s=S/N$, $i=I/N$, and $r=R/N$ respectively,

$\displaystyle\frac{ds}{dt}$ $\displaystyle=$ $\displaystyle-\beta\,i\,s$
$\displaystyle\frac{di}{dt}$ $\displaystyle=$
$\displaystyle\beta\,i\,s-\gamma\,i$ (1) $\displaystyle\frac{dr}{dt}$
$\displaystyle=$ $\displaystyle\gamma\,i$

where $s(t)$, $i(t)$, and $r(t)$ are the fractions of susceptible, infected
and removed individuals at a given time $t$.

Although the uniformity assumption is an oversimplification of the situation,
the model still successfully explains the spreading process of contagious
diseases. The first and most important prediction of the model is the
existence of a threshold value which is a function of transmission and
recovery rates. The model can also accommodate various assumptions for
immunization and vaccination.

In real societies, as soon as an infection starts to spread, information on
the spread rate and level of Infectiousness becomes available to the public
through two channels: Public broadcasting (official data) and word-of-mouth
(in technology-oriented societies, social media). One way of integration of
information diffusion and the SIR model requires reinterpretation of the Bass
model. The number of individuals who adopt information, at time $t$ is $V(t)$;
which leads to the percentage of vaccinated individuals, $v(t)=V(t)/N$. The
modified Bass equation becomes,

$\frac{dv(t)}{dt}=(p+q(v(t)+i(t))+r(t))(1-(v(t)+i(t)+r(t))$ (3)

where $v(t),i(t)$ and $r(t)$ are the percentages of vaccinated, infected and
recovered individuals. $s(t)=(1-(v(t)+i(t)+r(t))$ gives the percentage of
susceptibles who are open to infection. The parameters $p$ and $q$ are known
to be innovation and imitation parameters of the Bass model. Here, the
innovation parameter best understood as the probability of adoption the new
situation (get vaccinated) immediately after being informed. The imitations
parameter is the probability of adoption after having some experience through
the neighbors (Word-of-mouth). Imitators try to protect themselves after
seeing the benefit of vaccination through vaccinated, recovered or infected
neighbors.

Coupling SIR model equations with the Bass equation leads to modeling a
society with individuals who may take precursory measures under the spread of
contagious disease 3.

$\displaystyle\frac{ds}{dt}$ $\displaystyle=$
$\displaystyle-\beta\,i\,s-(p+q(v+i))s$ $\displaystyle\frac{di}{dt}$
$\displaystyle=$ $\displaystyle\beta\,i\,s-\gamma\,i$ (4)
$\displaystyle\frac{dr}{dt}$ $\displaystyle=$ $\displaystyle\gamma\,i$
$\displaystyle\frac{dv}{dt}$ $\displaystyle=$ $\displaystyle(p+q(v+i))s.$

In these four coupled first order differential equations, susceptible
individuals become infected with the rate $\beta$, infected individuals
recover with the rate of $\gamma$. At the same time some susceptible
$s=1-(v+i+r)$, take precursory measures and become immune. The parameter $p$
and $q$, determine the vaccination rates of the susceptibles who adopt the
idea of vaccination immediately after being informed and after observing the
benefit respectively. In this work, the probability of transmission
$\beta=1.0$ and the probability of recovery $\gamma=0.2$ are kept constant for
both aggregate and the agent-based models.

Figures 1(a) and 1(b) show the effect of opinion diffusion among the members
of society. The difference between the figures 1(a) and 1(b) is the number of
individuals who can avoid contamination.

For the information spread, only virtual interactions are sufficient. The
spread of contamination requires direct contact interaction. In this sense
even if the social network is the same since information and contamination
spread are different types of interactions with different rules, their spread
dynamics are different. The information of the spread of the infection reaches
the susceptible through the infected, recovered and vaccinated neighbors. Upon
this information the susceptible makes a decision: Take a precursory measure
or not. In the decision-making process, both infected and vaccinated
individuals contribute equally (Equation 3). Figure 1(a) show SIR equation
(Eqn. 3) solution with the above-given parameters. The figure shows the
expected susceptible-infective-recovered relation. The chosen parameters are
higher than the threshold value of the epidemic; hence all of the members of
the population becomes infected, and in time they recover. In $50$ discrete
time steps, the population consists only of the recovered individuals. Figure
1(b) shows the results of the extended SIR model (Eq. 3). The extension, Bass
equation as the fourth differential equation requires the innovation and
imitation parameters. From the marketing studies, it is shown that the
innovation parameter is much smaller than the imitation parameter. This
situation is intuitively understandable since the number of individuals who
take risks is less than the imitators, who follow the trend or observe the
benefits of the new situation, the innovation parameter is less than the
imitation parameter. The innovation and imitation parameter values, $p=0.05$
and $q=0.2$ are also used for both aggregate and agent-based models. In the
SIR model case, the total number of infected individuals summed up to $100\%$
of the population. The voluntary vaccination/protection program with the
above-given parameters changes the situation. The innovation and imitation
parameter values,$p=0.05$ and $q=0.2$, reduce the peak of the infected
individuals down to $20\%$ level. With such low infected ratio, the recovered
individuals remain around $40\%$, while $60\%$ of the population remains
unaffected by the infection.

(a) SIR Model

(b) SIR model with vaxination

Figure 1: Spread of epidemics in a uniform society.

Such a totalistic approach cannot produce the expected result of the
intelligent behavior. The characteristics of the real social networks
implemented in the above-described model. The following section has devoted
the application of the aggregate model on the different network topologies to
discuss the effects of the network topologies on the spread of contagious
disease under the intelligent selective vaccinations.

##  4 Rules of Agent Based Model on Networks

For agent-based simulations, the model society a collection of interacting
units (agents). The interactions rules and the network of connections of the
agents determine the dynamics of the evolving social phenomena. The network
contains $N$ nodes. Each node, $N_{i}$, carries a state variable, $S,\;I,\;R$
or $V$ for susceptible, infected, recovered and vaccinated states
respectively. At the initial stage all nodes have common infection,
$\beta_{i}=\beta$ and recovery $\gamma_{i}=\gamma$ probabilities. After the
beginning of the spread of infection, the collected information leads some
nodes to take precursory measures. The uniformity of the fully connected
networks did not allow individuals to make a choice. On the contrary with the
fully connected network case, here, agents can choose to reduce the risk of
being contaminated or choose to be vaccinated.

The states of the agents changed through mutual interactions. The number of
reciprocal interactions is the measure of the discrete time steps. The spread
of infection is followed in time steps until the stationary state is reached.

At one time step, the states of $N$ randomly selected nodes are sequentially
updated. Update of a node is the dynamical change through interaction with a
randomly chosen neighbor. Interaction of two connected sites sets their state
for the next time step. The rules of transition are as follows, a)If a
susceptible node interacts with an infected neighbor, they mutually
contaminate each other. b) If the chosen node is infected, may recover with
probability $\gamma$. c) If the node is susceptible and did not meet an
infected neighbor until the time of update, get vaccinated immediately with
the transition rate $p$ or gather information on the number of infected,
recovered and vaccinated neighbors. If the collected data triggers an action,
gets immunized with the probability $q_{1}$. The node changes its state from
$S$ to $V$, and this node gains immunity. Another precursor measure is to
reduce the probability of neighbor induced contamination. The node who take
precursory action reduce its transition probability with probability $q_{2}$,
avoids infection, remain susceptible.

Two separate sets of rules are necessary for the diffusion of contamination
and information gathering. The first set of rules are related to the contact
interaction of the neighboring agents. The proposed rules for transmission of
the contiguous disease on networks have the following steps:

  1. 1.

A randomly selected individual ($N_{i}$) interacts with a randomly selected
neighbor ($N_{j}$).

  2. 2.

If either one of them is infective, at time slice $t$ while its neighbor is
susceptible, the susceptible the partner becomes infected with a probability
$\beta$ at the time slice $t+1$

  3. 3.

If the selected individual ($N_{i}$) is already infected, recovers with
probability $\gamma$ at time slice $t+1$.

At each time step, all nodes mutually exchange information. The information
sent by every node contains the state of the node. Each node evaluates the
collected data. If the individual $N_{i}$ is susceptible at time step $t$
decides whether to take precursor or not. The rules of information gathering
and decision-making process is as follows:

If an individual living at vertex $i$ is susceptible, at time $t$,

  1. 1.

A node $N_{i}$ (innovator) gets vaccinated with probability $p$, regardless of
the state of the neighbors.

  2. 2.

If the individual is an imitator, take precursory measures with the
probability proportional with the states of the neighbours,

$P_{Neighbors}=\;\frac{\sum_{\alpha\in N\\\
N}k_{\alpha}\left(\delta_{N_{\alpha},I}+\delta_{N_{\alpha},V}+\delta_{N_{\alpha},R}\right)}{\sum_{\alpha\in
NN}k_{\alpha}}$

. If $P_{Neighbors}r$ the individual living at site $i$ becomes aware of the
danger and take action. Here, $r$ is a uniform random number, $0\leq r<1$. For
an imitator there are two possible choices:

    1. (a)

With probability $q_{1}$ get vaccinated and gain immunity,

    2. (b)

with probability $q_{2}$ take measures and reduce the probability of being
infected.

Any action other than vaccination does not give full immunity but reduce the
probability of infection $\beta_{i}\rightarrow\beta^{\prime}_{i}$. In this
study, $\beta^{\prime}_{i}$ is taken as $\beta_{i}/4$.

$\begin{array}[]{lll}{\rm if}&\;\;q_{1}>r_{1}&\;\;\;N_{i}=V\\\ {\rm
else\;if}&\;\;q_{2}>r_{2}&\;\;\;\beta_{i}=\beta_{i}/4\\\ {\rm else}&&{\rm
does\;noting.}\end{array}$

Here $r_{1}$ and $r_{2}$ are random numbers with uniform distribution, $q_{1}$
and $q_{2}$ are imitation parameters for vaccination and other protective
measures respectively. Initially, all vertices are assigned the same value of
infection probability value. The infection probability is reduced to one-
fourth for the individuals who choose the protection as a measure of
protection.

In the following subsection, the model will be tested using scale-free
networks will be tested to observe the effects of topology differences of the
social networks.

##  5 Agent Based Model on Scale Free Networks

One of the most prominent examples of real-life networks is the scale-free
networks. The Barabási-Albert [5] preferential-attachment algorithm is one of
the commonly used algorithms to create scale-free networks. In this work, the
Barabasi-Albert algorithm is employed to create scale-free networks. The
network starts with $m$ seed nodes which are mutually connected. Each new node
makes $m$ links to the existing nodes with probability,

$P=\frac{k_{i}}{\sum_{j}k_{j}}$

where $k_{i}$ is the degree of the $i^{\rm th}$ node. The sum of the degrees
of all existing nodes is the normalization term. The number of the initial
nodes plays an important role in the topology of the network. If the network
starts with only very few numbers of nodes, some of the nodes will be highly
connected. The highly connected nodes are called hub nodes since such nodes
transmit the effect to a very high number of nodes. This situation is typical
of the scale-free networks. As the number of initial nodes increases, the
number of hub nodes increase but their connectivity decrease. If the network
starts with a high number of nodes, none of the nodes will have a very high
degree, and the network resembles random networks.

The spread of contagious disease on scale-free networks are summarised in
Figure 2. The model society consists of $10000$ individuals. A scale-free
network provides the connections among the members of the community. Networks
with the number of seed nodes, $m=2,$ and $m=20$ are used to study the role of
highly connected nodes. Highly connected hub nodes are the characteristics of
the scale-free network with a small number of initial states. The increasing
number of seed nodes change the structure of the network. As the number of
initial nodes increases, the hub nodes lose their importance. With an
increasing number of seed nodes the scale-free networks gradually divert to
random networks.

Initially, all nodes are in the susceptible state, $S$. One of the randomly
selected nodes is assigned as infected $I$. Measurements are done over $100$
statistically independent initial configurations and new sets of connections.
Each configuration contained only one infected individual. The time evolution
of each new system is followed $50$ time steps.

Each system contains two elements: The network and the distribution of the
states of the nodes. Each node is assigned transmission, and recovery rates as
well as the innovation, imitation parameters. Initially, the parameter values
are uniformly distributed to all nodes. During the simulation, only the
infection transmission rate $\beta$ is allowed to change through interactions.
The same network is employed for both spreads of contamination and
information. Figures 2(a) and 2(c) show the spread of contamination in two,
same size, networks. The difference between these networks is in their
connectivity structure. The first network centered around $2$ seed nodes while
the second network started with $20$ seed nodes. Hence, the degree
distributions of highly connected nodes are different. Just a few highly
connected nodes exist in the $m=2$ case, while $m=20$ is more like a random
network. In both cases, the parameters related with the transmission of
illness ($\beta$ and $\gamma$), and parameters related with the adoption of
action ($p$,$q_{1}$ and $q_{2}$) parameters are the same for both networks.
Moreover, the fixed values of the probabilities, $\beta=1$ and $\gamma=0.2$
are the same as in the aggregate model case. The difference is due to the
change of the network topology. The number of infected individuals is higher
in the network with a large number of initial sites. Moreover, the peak value
of the number of infected is reached before the peak of the network originated
from only a small number of initial nodes. This is understandable since the
connectivity of the random networks spread of the infection equally well,
regardless of the choice of the position of the initial infected individual.
In the stong hub case, a limited number of initial nodes have very high
connectivity, but rest of the nodes are connected with only a small number of
neighbors. Figures 2(b) and 2(d) show the effects of the individual prevention
efforts for the networks initiated with $m=2$ and $m=20$ nodes .

The case of the societies with intelligent individuals who take precaution
when the disease information spread is presented by the figures 2(b) and 2(d).
The figures include the changes in the number of susceptibles $S$, infected,
$I$ recovered $R$, vaccinated $V$ and the total number of non-contaminated
individuals, $T$. Vaccinated or precautioned hub nodes reduce the probability
of being contaminated blocks the paths of contamination. In this sense, the
hub nodes play an important role in reducing the speed and the size of the
spread which also enables the other to take precaution. This effect is
presented by the figures, 2(a) and 2(b). Figure 2(a) show that at the peak,
the infected ratio reaches about $60\%$ of the population. The contamination
spreads to all members of the society at the end of $50$-time slices. Figure
2(a) show that, individual prevention efforts are very efficient in the case
of $m=2$. The peak value of the infected individuals hardly reaches one-fifth
of the population. After $50$ time steps, slightly less than half of the
population, $T$, remain unaffected by the disease.

(a) SIR model on scale-free network ( $m=2$)

(b) SIR with vaxination on scale-free network ($m=2$)

(c) SIR on scale-free network ($m=20$)

(d) SIR with vaxination on scale-free network ($m=20$)

Figure 2: Spread of contegious disiese in a society with scale-free network
topology.

Using the fixed infection transition and recovery rates, the spread of the
infection is faster on networks with a higher number of initial nodes ($m=20$)
than the small number of seed nodes ($m=2$). In the case of a high number of
seed nodes also the peak of the infected individuals reach the point which is
higher than the half of the population (Figure 2(c)). The majority of the
population has already been infected after $10$ time steps. Since the peak is
reached at about ten iterations, individuals have less time to take
precaution. The number of vaccinated remains less than the case of $m=2$.
Therefore the total number of unaffected individuals are also considerably
less than $m=2$ case.

(a) $m=2$ and $m=4$

(b) $m=20$ and $m=100$

Figure 3: The effects of information processing on the number of infected
individuals. a) $m=2$ and $m=4$, b) $m=20$ and $m=100$ .

Figure 3, discuss the role of connectivity in the spread of infection. Figure
3(a) and 3(b) show the spread of disease in scale-free networks with a low and
high number of hub nodes. Figure 3(a) exhibits the effect of the changes in
the number of highly connected hub nodes on the number of infected
individuals. The Figure 3(a) show the comparison between $m=2$ and $m=4$
networks. In the case where highly connected hub nodes exist ($m=2,4$) the
effects of the individual precaution attempts to reduce the initial high peak
value of the infected individuals. As the number of hub nodes increases, the
diffusion speed increases, consequently the prevention efforts are not
sufficient. The Figure 3(b) is devoted to the comparison of a high number of
hub nodes, $m=20$ and $m=100$, where it is seen that as the number of hub
nodes increase, a limiting link distribution is reached. At this point, the
diffusion speed of the infection reaches its peak.

##  6 Discussions and Conclusions

In this work, the aim is to model the spreading of disease on networks while
susceptibles exchange information on the state of the neighbors. The
accumulated knowledge is used to make decisions on the possible precautions.
The infection spreads among the members of the community through direct
contact interactions while the information exchange with the neighbor sites is
simultaneous and does not require direct contact. The susceptible members of
society may take measures to prevent infection. The proposed methods of
prevention are being vaccinated or trying to avoid direct contact with the
neighbors. The collected information on the states of the neighbors leads to
the choice of precursory action.

The fully connected and scale-free networks are used to test the proposed
model. On the fully connected system, the comparison between the SIR model and
the proposed model with awareness shows a considerable reduction in the spread
rate of infection. Even a limited number of precaution taken nodes are
sufficient for such significant success in fighting epidemics. In the scale-
free network case, the same observations valid and moreover it is seen that
the highly connected nodes are essential for the control of the spread of
disease. A precautioned hub node can block a high number of connections,
reducing the spread rate of infection.

All types of networks are under attack of different viruses, computer viruses,
and malice information. In real-life situations, nodes defend themselves from
the fatal attacks. Nodes continuously exchange information. The shared crucial
information help nodes to prepare themselves for possible malice situation.
This work aimed to model existing real-life protection mechanisms appearing in
networked systems. The role of connection topology is seen to play a crucial
role in prevention attempts. In general, contact networks are separate than
the communication networks the disease transmission is of concern. For
technological networks, both communication and interactions are part of the
same system. When the direct interactions and communications are on different
networks, multilayer networks must substitute traditional network structure.
The transmission of information and infectious disease on multilayer networks
will be the subject of future work.

## References

  * [1] E. Rogers, Diffusion of Innovations, 5th ed. (Free Press, New York, (2003).
  * [2] F. Bass, "A New Product Growth for Model Consumer Durables", Management Sci. 15, 215-227 (1969).
  * [3] W. O. Kermack, A. G. McKendrick, "A contribution to the mathematical theory of epidemics", Proc. R. Soc. A 115, 700-721 (1927).
  * [4] R. Albert, H. Jeong, A. L. Barabási, "Error and attack tolerance of complex networks", Nature 406, 378-382 (2000)
  * [5] A. L. Barabási and R. Albert "Emergence of scaling in random networks", Science 286, 509-512 (1999).
  * [6] S. Venkatramanan, B. Lewis, J. Chen, D. Higdon, A. Vullikanti, M. Marathe, "Using data-driven agent-based models for forecasting emerging infectious diseases" Epidemics 22 43–49 (2018)
  * [7] B. Pell , Y. Kuang , C. Viboud , G. Chowell, "Using phenomenological models for forecasting the 2015 Ebola challenge" Epidemics 22 62–70 (2018)
  * [8] L. Zhou,M. Fan, "Dynamics of an SIR epidemic model with limited medical resources revisited." Nonlinear Anal. Real World Appl. 13, 312–324 (2012)
  * [9] T. Yu, D. Cao ,S. Liu, "Epidemic model with group mixing: Stability and optimal control based on limited vaccination resources." Commun. Nonlinear Sci. Numer. Simul. , 61, 54–70 (2018).
  * [10] I. Abouelkheir, F. El Kihal , M. Rachik and I. Elmouki, "Time Needed to Control an Epidemic with Restricted Resources in SIR Model with Short-Term Controlled Population: A Fixed Point Method for a Free Isoperimetric Optimal Control Problem"
  * [11] Z. Wang, M. A. Andrews, Z. Wu, L. Wang, T. Chris, F. Bauch , "Coupled disease–behavior dynamics on complex networks: A review" Physics of Life Reviews 15 1–29 (2015)
  * [12] C. Castellano, S. Fortunato and V. Loreto "Statistical physics of social dynamics" Reviews of Modern Physics 81, 592-638 (2009)
  * [13] P. S. Romualdo A. Vespignani, "Epidemic Spreading in Scale-Free Networks". Physical review letters 4, 3200-3203, (2001).
  * [14] P. S. Romualdo and A. Vespignani, "Epidemic Dynamics and Endemic States in Complex Networks",Physical Review E 63, 066117, (2001).
  * [15] M.E.J.Newman, "Spread of epidemic disease on networks",Phys. Rev. E66 016128 (2002).
  * [16] Z. Wang, T. Chris Bauch, S. Bhattacharyya, A. d’Onofrio, P. Manfredi, M. Perc, N. Perra, M. Salathé, D. Zhao, "Statistical physics of vaccination", Phys. Rep. 664, 1-113, (2016).
  * [17] G. Chowell, L. Sattenspiel, S. Bansal, C. Viboud, "Mathematical models to characterize early epidemic growth: A review" Physics of Life Reviews 18 66–97 (2016)
  * [18] Z. Dezso and A.-L. Barabási, " Halting viruses in scale-free networks", Phys. Rev. E 65, 055103 (2002).
  * [19] P.S. Romualdo, A. Vespignani, "Immunization of complex networks". Phys.Rev. E 56, 036104-036107, (2002).
  * [20] F. Nian and X., J. Wang, "Efficient immunization strategies on complex networks" Theor. Biol., 264, 77-83, (2010).
  * [21] R. Cohen ,S. Havlin and D. Ben-Avraham, "Efficient immunization strategies for computer networks and populations", Phys. Rev. Lett., 91 247901 (2003).
  * [22] N. Ferguson, "Capturing human behaviour", Nature (London) 446, 733 (2007).
  * [23] S. Funk, M. Salathé, and V. A. A. Jansen, "Modelling the influence of human behaviour on the spread of infectious diseases: a review",J. R. Soc. Interface 7, 1247-1256 (2010).
  * [24] :T. Gross, C. J. Dommar D’Lima, and B. Blasius "Epidemic Dynamics on an Adaptive Network" Phys. Rev. Lett. 96, 208701 (2006)
  * [25] P. Poletti, M. Ajelli, and S. Merler,"The Effect of Risk Perception on the 2009 H1N1 Pandemic Influenza Dynamics", PLoS One 6, e16460 (2011).
  * [26] Q. Wu, X. Fu, M. Small, and X-J Xu, "The impact of awareness on epidemic spreading in networks" ,CHAOS 22, 013101 (2012)
  * [27] I. A. Baba, E. Hincal, "A model for influenza with vaccination and awareness Chaos." Solitons and Fractals 106 49–55 (2018)
  * [28] Z. Ruan , M. Tang and Z. Liu, "Epidemic spreading with information-driven vaccination", Phys. Rev. E, 86 036117 (2012).
  * [29] N. Perra, D. Balcan, B. Goncalves, A. Vespignani, "Towards a Characterization of Behavior-Disease Models", PLoS One 6, e23084 (2011).
  * [30] S. Xia , J. Liu ," A belief-based model for characterizing the spread of awareness and its impacts on individuals’ vaccination decisions." J. R. Soc. Interface 11: 20140013\. http://dx.doi.org/10.1098/rsif.2014.0013 (2014)
  * [31] X.T. Liu, Z.X. Wu, and L. Zhang "Impact of committed individuals on vaccination behavior" Phys. Rev. E 86, 051132 (2012)


# Ontology Reuse: the Real Test of Ontological Design

Piotr Sowiński  Katarzyna Wasielewska-Michniewska  Maria Ganzha  Marcin
Paprzycki  Costin Bădică  Warsaw University of Technology, Warsaw, Poland
Systems Research Institute, Polish Academy of Sciences, Warsaw, Poland
University of Craiova, Romania

###### Abstract

Reusing ontologies in practice is still very challenging, especially when
multiple ontologies are (jointly) involved. Moreover, despite recent advances,
the realization of systematic ontology quality assurance remains a difficult
problem. In this work, the quality of thirty biomedical ontologies, and the
Computer Science Ontology are investigated, from the perspective of a
practical use case. Special scrutiny is given to cross-ontology references,
which are vital for combining ontologies. Diverse methods to detect potential
issues are proposed, including natural language processing and network
analysis. Moreover, several suggestions for improving ontologies and their
quality assurance processes are presented. It is argued that while the
advancing automatic tools for ontology quality assurance are crucial for
ontology improvement, they will not solve the problem entirely. It is ontology
reuse that is the ultimate method for continuously verifying and improving
ontology quality, as well as for guiding its future development. Specifically,
multiple issues can be found and fixed primarily through practical and diverse
ontology reuse scenarios.

###### keywords:

Ontology evaluation Ontology engineering Semantic Web Ontology reuse Ontology
alignment

, , , ,

##  1 Introduction

Ontologies are invaluable in representing domain knowledge; possessing a
number of characteristics that make them a tempting choice when designing
intelligent information systems [1]. They are very expressive and flexible,
allowing one to describe a wide variety of subjects, in a precise manner.
Nowadays, with the introduction of the Linked Open Data principles [2],
accessing, reusing, and combining ontologies should have become easier than
ever. However, in practice, ontology reuse can still be very challenging [3].
This is especially true when multiple ontologies are involved, where ambiguous
or conflicting statements between ontologies often require human intervention
in order to be resolved [4].

Obviously, high quality of an ontology is vital to ensure its potential
reusability across a spectrum of applications. As ontologies model knowledge,
any issues in them immediately reflect on the quality of their application.
Many criteria for evaluating ontologies were proposed, and there are multiple
approaches to their evaluation [5, 6]. However, individual ontologies use very
different evaluation methods, or (in most cases) do not have any evaluation
plan in place [7]. Some approaches are hard to implement in practice – such as
the gold standard method, for which, often, there is no available gold
standard to be used. Obviously, expert-based evaluation is resource-intensive
and naturally limited in its scope. On the other hand, the automated, more
universal methods are usually too general to identify many issues in the
ontologies [8].

Overall, as of today, there do not seem to be any established, agreed upon,
and widely used standards and methodologies for ontology evaluation. Instead,
for years, there have been examples of evaluations that were _tailored to the
specific ontology_ [9, 10] or motivated by a _specific, practical use case_
[11, 8, 12]. Lack of a commonly accepted evaluation methodology is one of
problems negatively impacting potential reuse and adaptation of ontology-based
solutions, as it was investigated, for instance, for the enterprise domain
[13].

In earlier work [14] an unsupervised, ontology-based method for classifying
scientific publications has been described (see Section 2). During research,
when the classifier was to be adapted to different domains, numerous problems
with the used ontologies have been noticed. Each of those issues either (i)
increased the amount of work necessary to adapt the classifier to the domain,
or (ii) decreased the accuracy of predictions. These observations motivated
further research, which is reported in this contribution.

Specifically, the goal of this work is to demonstrate that the _varied and
demanding reusability scenarios_ provide the ultimate measure of the quality
of an ontological design. Here, the real-world-anchored use case motivates
exploring each issue and supports the conclusions drawn from the experiments.

In this context, in the following sections, ontologies from two different
fields of science are investigated. To provide additional context to the
study, Section 2 briefly describes the publication classification method that
serves as the driving use case. Section 3 explores thirty diverse biomedical
ontologies, related to the field of food safety. Section 4 examines a medium-
sized Computer Science Ontology.

It should be stressed that all code used in the reported work, as well as
additional results and materials, are available on GitHub, under an open-
source license 111https://github.com/Ostrzyciel/ontology-quality-2022.
Detailed instructions for reproducing the experiments are also provided there.

##  2 Use Case Description

The investigation presented in this contribution is driven by a specific use
case from our earlier work [14] – an unsupervised method for classifying
scientific publications, that aims to be domain-agnostic. Here, all domain
knowledge necessary to perform the classification is obtained from ontologies
relevant to the pertinent field of science.

The considered method uses the publication’s title, abstract, and (optionally)
unstructured keywords as an input. It then assigns to the article several
entities from the loaded ontologies that best describe its topic. The
classification pipeline consists of six steps. (1) Using a neural named entity
recognition method, possible mentions of entities of interest are identified
in the text. (2) The ontologies are searched for candidate entities that may
match the mention. This is done using a text index. (3) The neighborhoods
(most closely related entities) of the candidate entities are retrieved from
the ontologies. (4) Using text embeddings and string matching, the candidate
entities are compared to the mention in the text. Here, both the context of
the mention (its sentence) and the context of the entity (its neighborhood)
are taken into account. This two-way context sensitivity helps the method to
correctly classify ambiguous mentions in the text. (5) The relations between
the identified candidate entities are examined (entity-entity coherence).
Then, the least consistent (least strongly connected) candidates are removed.
(6) For the final result, the highest-scoring and most frequently occurring
candidate entities are selected. The result is further enhanced with their
parent entities. This allows for classification of publications with topics
that are only implied and never mentioned explicitly.

The method relies solely on the ontologies as its source of domain knowledge,
utilizing both textual labels (search index, text embeddings) and relations
between entities (neighborhood estimation, entity-entity coherence, results
enhancement). Unsurprisingly, the method’s performance was observed
empirically to be largely determined by the quality of the used ontologies.
Following the ontology evaluation criteria as described by Raad & Cruz [6],
accuracy, completeness, conciseness, consistency, and clarity of the
ontologies had a direct, significant impact on the classifier’s performance.
The observed issues are discussed and investigated in detail in the following
sections.

##  3 Evaluation of OBO Foundry Ontologies

Food safety was the first field, to which the publication classification
method was applied. As a source of domain knowledge thirty ontologies from the
Open Biological and Biomedical Ontology (OBO) Foundry [15] were used, along
with SNOMED CT [16] (full list of used ontologies is available on
GitHub222https://github.com/Ostrzyciel/ontology-
quality-2022/blob/main/obo/ONTOLOGIES.md). Here, we focus only on the OBO
ontologies, which range in size from 1 221 triples for BFO (Basic Formal
Ontology) to 16 105 832 triples for NCBITaxon. When combined, the thirty
ontologies contained 39 156 273 triples. These ontologies are highly
heterogenous, due to their different goals and being, typically, maintained by
independent groups. Most considered ontologies are _developed_ in the OBO
format333https://owlcollab.github.io/oboformat/doc/GO.format.obo-1_4.html,
which is then _translated_ into the standard Web Ontology Language (OWL), to
facilitate reuse.

The issue of reliably maintaining quality over independently developed
ontologies is an active area of research, with tools such as ROBOT [17] or the
OBO Dashboard [18], attempting to tackle it. OBO Dashboard, in particular,
performs a wide selection of tests on ontologies, essentially operationalizing
the review activities that would otherwise be performed by a human. However,
the set of tests is limited – for example the validity of cross-ontology
references is not checked. Most of the issues have to be fixed manually, which
proves difficult in practice, especially for the rarely-updated ontologies.
Nevertheless, it can be stipulated that OBO Foundry is making significant
efforts to assure the high quality of its ontologies. Keeping this in mind,
let us report issues that have been found.

###  3.1 Ease of Access

Before the ontologies could be used, they had to be first downloaded and
loaded into a triple store. All ontology files downloaded from the OBO Foundry
had the .owl extension, but their actual format varied. Most were distributed
in the RDF/XML format, which is in line with the official OBO Foundry
guidelines. However, the Common Anatomy Reference Ontology (CARO) used the OWL
functional syntax instead, and had to be converted into RDF before being
loaded. The Food-Biomarker Ontology (FOBI) was distributed in the OWL/XML
format, which could not be converted to RDF/XML, due to invalid element
errors. Thus, for FOBI, an earlier release was used that did not exhibit this
issue and could be successfully converted to RDF.

It is worth pointing out that although these issues may appear as minor
annoyances, they have significant implications. Inconsistency in file formats
makes automatic reuse of these ontologies much harder or even impossible, due
to the need for manual conversion of the files.

###  3.2 Rarely Used Properties

The publication classifier requires the user to assign weights to all object
properties that may be useful for discovering how different entities are
interrelated (steps 3 and 5 of the method). Similarly, weights are assigned to
text properties, which describe the entities (step 2). To ascertain these
weights, an obvious first step is to retrieve a list of all properties, sorted
by the number of times they were used. Although OBO Foundry ontologies strive
for reuse and have common upper ontologies defining relations, in the
resulting lists many rarely used properties were found, often seeming like
mistakes. Thus, those properties that had at most ten unique uses across all
ontologies have been manually inspected.

It was found that out of 140 rarely-used properties, 74 were in some way
erroneous, while being used across 278 unique triples. The most common type of
error (51 properties, 212 occurrences) were undefined properties in the
oboInOwl namespace [19]. These are, most likely, errors in translation from
the OBO format to OWL. Other issues included mistaken ontology prefixes,
typos, and invalid URIs. These issues were found in 24 out of thirty
ontologies.

###  3.3 Property Value Type Mismatch

Next, the classifier requires dividing properties into URI-valued and literal-
valued. This turned out to be a non-trivial task, as many properties were used
inconsistently – sometimes their objects were URIs, and sometimes they were
literals.

A list of all properties was created, with the number of times each was used
in triples in which the object is an URI, a blank node, or a literal. For
manual review, properties that exhibited conflicting usage patterns were
selected. Most did not have an rdfs:range property, which would define
formally their set of possible values. In fact, some properties were not
defined at all, in any ontology. In other cases, the range was specified very
broadly, allowing both URIs and literals. This presented a major challenge in
distinguishing between erroneous and valid annotations. In such cases it had
to be decided “intuitively” whether a given use is valid or not. The decision
was made on the basis of scarce documentation, or other occurrences of the
same property. After establishing the valid usage patterns, lists of erroneous
occurrences for each property were obtained, using SPARQL queries.

In the results, the oboInOwl:hasDbXref property was excluded, as it is
discussed separately in the following subsection. Overall, 12 296 erroneous
triples were identified in 20 ontologies. Often, in places where an URI-typed
value was expected, a string literal representing that URI was present. Thus,
the value itself was valid, but the datatype did not match. In other cases,
various ontologies had different conventions for applying the same property.
For some metadata properties, e.g., those for marking authorship, significant
inconsistencies in their use were observed. However, it was decided not to
mark them as errors, due to our lack of knowledge as to which convention
should be applied, in a given case. Nonetheless, the lack of a common
convention for representing metadata makes understanding and reusing
ontologies considerably harder.

###  3.4 Inconsistent Cross-ontology References

Food safety integrates a wide variety of topics, from microbiology, to
chemistry, and human diseases. Therefore, it is crucial to ensure that the
imported ontologies are well-connected. Such connections, in OBO Foundry, can
be either partial imports, (where an entity from one ontology is imported into
another), or cross-ontology references, made with various types of properties.
When examining inter-ontology references, it was observed that their usage
patterns varied and their targets were hard to resolve automatically. Thus,
all occurrences of cross-ontology references were retrieved and their targets
analyzed. For these references, the dominant property is oboInOwl:hasDbXref (3
809 415 uses), but some SKOS [20] properties are also used. There were 259
occurrences of skos:*Match properties, of which only three had valid URIs as
values. Others had literals of external identifiers, e.g., MESH:C536189.

The SKOS references were not included in further analysis. Instead, two
standard OBO properties from the oboInOwl namespace were analyzed: hasDbXref
and hasAlternativeId. Their values are supposed to be strings, having the form
of Namespace:Identifier, with the namespace (usually) corresponding to an OBO
Foundry ontology. All triples (3 908 752) containing these properties were
retrieved. Then, the obtained reference values were processed and classified
using a custom algorithm. The found namespace identifiers were cross-
referenced with the publicly available list of all OBO Foundry
ontologies444https://obofoundry.org/registry/ontologies.yml, to filter out
valid references. Out of all references, 52 122 (1.3%) were URIs, not
identifiers. Among them only 112 were valid URIs of entities in OBO Foundry.
Others pointed to very diverse resources, e.g., Wikipedia, GitHub, online
databases, scientific publications, structured vocabularies, and various other
websites.

Among non-URI references, 187 167 (4.8%) referred to OBO Foundry ontologies.
Others, usually, pointed to external biomedical databases and structured
vocabularies, such as MeSH or SNOMED CT. However, multiple prefixes were used
for some databases, (e.g., SCTID, SMID, SNOMEDCT_US, SNOMEDCT_US_2021_09_1 for
SNOMED CT), which complicates translating the identifiers into URIs. Using the
recently created Bioregistry [21], it was possible to resolve prefixes for
another 3 013 315 references (77.1%). Nonetheless, the identified databases
form an eclectic collection, including the Unified Medical Language System,
PubMed, English Wikipedia, ISBN, ORCID, and Google Patents. This shows that
the cross-reference properties are used for a wide variety of purposes.
Obviously, this complicates their potential reuse. Moreover, the references
did not always conform to the Namespace:Identifier format, with some using the
underscore (_) as a separator instead of a colon. For OBO identifiers, there
were 19 such cases, while for Bioregistry-resolved prefixes 26 572 were
identified. Finally, 214 ($<0.01$%) references pointed to empty RDF blank
nodes. These are, most likely, the results of errors during modeling, or
translation into OWL. All these findings are summarized in Table 1.

Table 1: Cross-ontology references in OBO Foundry Type | # references | % total
---|---|---
URI | 52 122 | 1.33%
| valid OBO URI | 112 |  $<0.01$%
| en.wikipedia.org | 17 365 | 0.44%
| orcid.org | 8 085 | 0.21%
| langual.org | 5 592 | 0.14%
| other | 20 968 | 0.54%
Textual | 3 856 416 | 98.66%
| recognized OBO identifier | 187 167 | 4.79%
| recognized other Bioregistry prefix | 3 013 315 | 77.09%
| unknown | 655 934 | 16.78%
Empty blank node | 214 |  $<0.01$%
Total | 3 908 752 | 100.00%

It is worth mentioning that the OBO ontology format has a number of header
tags (treat-xrefs-as-*) that allow one to more precisely describe how to treat
cross-references, on a per-ontology basis. This information, however, is lost
in the translation to OWL, and thus largely inaccessible.

The (in)consistency of OBO cross-ontology references has also been explored by
Laadhar et al. [10], who noted the overwhelming variety of uses for the cross-
reference properties, which hampers reuse. Their work gives valuable
suggestions of possible improvements, which augment the discussion presented
here.

Table 2: Summary of issues found in OBO Foundry ontologies

Ontology | Rare prop.1 | Prop. obj.2 | Xref: blank3 | Xref: URI4 | Xref: unk.5
---|---|---|---|---|---
AEO | 4 | 0 | 0 | 10 | 136
AGRO | 18 | 51 | 0 | 1 266 | 6 710
APOLLO-SV | 4 | 308 | 214 | 2 | 21
BFO | 0 | 0 | 0 | 0 | 0
BTO | 3 | 0 | 0 | 0 | 3 479
CARO | 1 | 6 | 0 | 380 | 1 800
CHEBI | 12 | 0 | 0 | 0 | 313 736
CL | 38 | 236 | 0 | 2 297 | 34 296
DOID | 2 | 2 | 0 | 1 | 12 824
DRON | 9 | 6 | 0 | 0 | 35 148
EHDAA2 | 3 | 0 | 0 | 5 | 67
ENVO | 3 | 1 612 | 0 | 3 299 | 1 649
FOBI | 5 | 0 | 0 | 0 | 0
FoodOn | 0 | 5 702 | 0 | 8 416 | 6 329
GAZ | 0 | 6 | 0 | 0 | 25 505
GO | 1 | 2 536 | 0 | 354 | 118 473
HP | 45 | 313 | 0 | 3 520 | 28 386
IAO | 0 | 22 | 0 | 0 | 0
MP | 47 | 388 | 0 | 15 253 | 37 229
NCBITaxon | 0 | 0 | 0 | 0 | 0
OBI | 0 | 1 295 | 0 | 0 | 0
PATO | 13 | 96 | 0 | 3 485 | 17 144
PCO | 3 | 19 | 0 | 9 | 41
PECO | 2 | 0 | 0 | 0 | 685
PO | 3 | 24 | 0 | 3 | 6 547
RO | 2 | 35 | 0 | 0 | 15
SYMP | 2 | 0 | 0 | 1 | 449
Uberon | 87 | 375 | 0 | 23 845 | 14 627
UO | 9 | 0 | 0 | 0 | 0
XCO | 3 | 0 | 0 | 0 | 494
All | 278 | 12 296 | 214 | 52 122 | 655 934

  * 1

Invalid occurrences of rarely-used properties.

  * 2

Property object type mismatch (URI instead of literal or vice versa).

  * 3

Cross-references pointing to blank nodes.

  * 4

Cross-references pointing to URIs instead of identifiers.

  * 5

Non-resolvable cross-reference identifiers.

###  3.5 Summary

In summary, 720 844 issues were found across thirty OBO Foundry ontologies
(treated jointly). These issues, divided into five categories, and broken down
by the source ontology, have been presented in Table 2. Note that due to the
use of partial imports (cf. MIREOT [22]), some ontologies overlap. Thus, the
actual number of issues in a given category may be lower than the sum of
issues encountered in individual ontologies. It is also worth noting that
_only two ontologies_ were found to be entirely problem-free, from the
perspective of the considered use case. These were BFO and NCBITaxon.

##  4 Evaluation of the Computer Science Ontology

The same classifier was adapted to the Computer Science Ontology (CSO) [23],
which models research topics in computer science. Its main areas of
application include publication classification and research trend detection.
It uses a small set of semantic properties, extending the SKOS data model
[20]. It also contains references to entities in external knowledge bases
(KBs) such as DBpedia [24], Wikidata [25], and YAGO [26], using the owl:sameAs
property. The most recent version of CSO (3.3) contains 14 290 topics and 163
470 triples, and was used in all experiments.

The ontology was generated automatically, using the Klink-2 algorithm [27].
Relationships in some research areas (Semantic Web and Software Architecture)
were manually reviewed by experts, but most of the burden of ontology quality
control and maintenance was left to the community. To facilitate this, the
authors of CSO built the CSO Portal555https://cso.kmi.open.ac.uk/, which
collects suggestions for improving the ontology.

In the following subsections, we examine various issues found in CSO. To
assure that the results properly identify the magnitude of individual issues,
each experiment was performed independently, starting from a fresh copy of the
ontology.

###  4.1 Synonym Description Structure

In CSO, alternative names of one concept are treated in such a way that each
synonym is its own entity of type cso:Topic. Clusters of equivalent topics are
connected using the cso:relatedEquivalent and cso:preferentialEquivalent
properties. There are 11 187 such clusters. This structure is in a stark
contrast to most other ontologies, which usually attach alternative names to a
single entity, using text properties (cf. Gene Ontology and other OBO
ontologies [28], Open Research Knowledge Graph [29], YAGO [26]).

When processing the ontology, it was found to be laborsome to work with this
structure. For example, when displaying the results of a query searching for
topics, one would have to filter out those that are not marked as
_preferential_. Similarly, adapting the publication classification method to
CSO would be impossible without writing additional logic to handle the
atypical structure. Therefore, CSO was transformed using SPARQL 1.1 updates
into an ontology with synonymous topics merged into a single entity. Here,
synonyms were attached using the skos:altLabel property. The transformation
changed the structure of the ontology, but it did not affect its information
content. The original 163 470 triples were transformed into 88 227 triples, a
46% decrease. An example of the applied transformation can be found in Figure
1.

Figure 1: An example of the changes made in CSO

###  4.2 Externally Inconsistent References

Next, CSO’s references to external KBs were examined. The goal was to be able
to use federation to provide additional information to the classifier.
However, multiple references were found to point to entities very distant from
computer science, such as musicians or rivers. Assuming that in the external
KB there should not be many connections between, e.g., popular music and
computer science, one could attempt to find the “outlier” entities that are
not well connected with the rest. These entities will, most likely, be invalid
references.

A list of all DBpedia entities referenced from CSO was obtained. Then, for
each entity, a list of links to other DBpedia entities was retrieved. This
data was used to construct an undirected wikilink graph that was analyzed
using the NetworkX library [30]. Three approaches to identifying outliers were
used in a sequence:

  * T1

Find all connected components (CCs) in the graph. Add the vertices from all
except the largest CC to the set of candidate outliers (result set).
Afterwards, remove these vertices from the graph, to leave only one CC.

  * T2

Identify and remove bridges in the graph, increasing the number of CCs [31].
Add the vertices from the newly created CCs to the result set, and then remove
them from the graph.

  * T3

Find “communities” in the graph, using the label propagation algorithm [32].
Add the vertices from these communities to the result set.

The outlier detection method identified 140 potentially erroneous alignments,
which were then manually evaluated by a reviewer, using a custom web
application. As the errors were usually easily discernible and there was
little room for ambiguity, only one reviewer was included in this experiment.
Here, each alignment has been marked as _valid_ , _invalid_ or having a
different issue, such as being out of scope of computer science entirely.
Table 3 presents the results of the review, broken down by the tactic that led
to marking the alignment as “suspect”. It can be observed that tactics T1 and
T2 were highly effective in discovering potentially erroneous alignments. The
false positive rate was relatively low, at only 25% of all suspected entities
not having any issues. Most occurrences in the _other issues_ category
referred to out-of-scope entities, which explains why their external
references were inconsistent with others.

Table 3: DBpedia alignments manual review results Verdict | # total |  # T1 |  # T2 |  # T3
---|---|---|---|---
invalid | 74 | 47 | 27 | 0
other issues | 31 | 14 | 15 | 2
valid | 35 | 15 | 13 | 7
Total | 140 | 76 | 55 | 9

###  4.3 Missing References to the Corresponding Knowledge Bases

CSO contains references to several external KBs, most of which have
corresponding entities for each other. For example: each DBpedia entity refers
to an English Wikipedia article, and each such article has a corresponding
entity in Wikidata. Thus, one may expect there to be a 1:1 mapping between
CSO’s references to DBpedia and Wikidata. To examine this, a series of SPARQL
queries was performed, looking for instances where this 1:1 mapping was not
the case.

The method did not identify any missing DBpedia references. However, it did
find 31 topics with missing references to 13 unique Wikidata items. Fixing the
issue is trivial, and a patch in the form of a Turtle
file666https://github.com/Ostrzyciel/ontology-
quality-2022/blob/main/cso/3_missing_refs/results/suggestion.ttl was prepared.

###  4.4 Logically Invalid References

The publication classifier uses relationships contained in the ontology to
determine the degree of similarity between entities. Each property type is
assigned a weight, which should be low for very close conceptual connections
(e.g., identity), and high for the more “remote” relationships. When assigning
the weights, one can assume that the owl:sameAs relation, according to its
definition, should represent identity (and thus, have zero weight). However,
CSO uses the owl:sameAs property for referencing external knowledge bases.
This property is defined as reflexive, symmetrical, and transitive [33], which
implies that if two different concepts were to be aligned to the same external
entity, they could be deduced to be identical, which is a logic violation.
Multiple examples of such problems have been noticed (see, Figure 2).

Figure 2: References to external KBs in CSO

Here, for example, _malicious software_ and _malware detection_ can be
reasoned to be identical, even though in the ontology they are (as expected)
completely separate concepts.

To find all such occurrences, the set of owl:sameAs relations in CSO was
expanded, using a reasoner that exploits the reflexive, symmetrical, and
transitive characteristics of this property. A SPARQL query was used to obtain
a list of CSO topic pairs, marked as identical (owl:sameAs), but not belonging
to the same cluster of synonyms. A total of 7 457 unique URI pairs was found
to be incorrectly marked as identical. This issue concerned 3 250 unique topic
URIs, out of 14 290 CSO topics in total.

For comparison, in the BioPortal, which hosts multiple biomedical ontologies,
the alignments that were constructed with the LOOM lexical matching algorithm
[34] are marked using the skos:closeMatch property, chosen due to its lack of
transitiveness, which helps avoid similar issues.

###  4.5 Intra-cluster Alignment Inconsistencies

CSO’s approach to synonym clusters requires caution to avoid inconsistencies
in references to external KBs. For example, having topics $A$ and $B$ from a
single synonym cluster, one would expect that if $A$ is aligned to an external
entity $X$, then also $B$ is aligned to $X$. Moreover, CSO topics from one
cluster should not be aligned to multiple different external entities from a
single external KB.

To validate these assumptions, SPARQL queries were performed to find DBpedia
alignments for each synonym cluster. The information was then processed with a
script to find clusters that violate either assumption. The method identified
130 synonym clusters that contained more than one unique DBpedia reference. Of
those, 124 clusters had two unique references, and six clusters had three.
Moreover, there were 962 entities with a DBpedia reference missing, spread
over 752 synonym clusters. Adding these missing references is easy and an
appropriate ontology patch was
prepared777https://github.com/Ostrzyciel/ontology-
quality-2022/blob/main/cso/5_intra_ref_consistency/results/suggestion.ttl. In
total, 882 out of 2 225 (39.6%) larger-than-one synonym clusters exhibited
some external reference consistency issues.

###  4.6 Term Conflation

When debugging the outputs of the publication classifier, several synonym
clusters were noticed to include conflated and erroneous terms. Finding more
such issues systematically would require the intervention of human experts. To
narrow down the set of “suspected” clusters, natural language processing
techniques can be used to determine the degree of similarity between terms.
Then, the least coherent synonyms could be inspected manually by human
reviewers.

Hence, using SPARQL queries, a list of all synonym clusters was obtained,
along with the corresponding topic labels. These labels were then encoded into
vectors using a sentence embedding model all-mpnet-
base-v2888https://huggingface.co/sentence-transformers/all-mpnet-base-v2,
based on the MPNet transformer [35]. Next, in each cluster, an all-pairs
similarity matrix was computed between the encoded labels (using cosine
similarity, however, any other similarity measure could have been used). For
each topic, the mean and the standard deviation of its similarity to other
topics were computed. Finally, clusters that contained at least three topics
and had low mean and standard deviation of similarity to other topics were
selected for review. This approach was designed to find clusters with topics
that are _systematically_ inconsistent, and thus most likely to be wrong.

For manual review, suspected clusters were evaluated by three experts.
Available ratings were: _definitely good_ , _probably good_ , _not sure_ ,
_probably wrong_ , and _definitely wrong_. The order of clusters was picked at
random, independently for each expert. The experts were allowed to use
external sources to aid their work, and were instructed to simply use their
best judgment when making the decision.

The intra-cluster similarity method identified 115 clusters of potentially
invalid synonyms. Using the simple majority vote rule, 84 of these clusters
were deemed _wrong_ by the reviewers. Moreover, 95 clusters were marked as
such by at least one reviewer, and all three reviewers agreed that 58 clusters
are wrong. To investigate inter-reviewer agreement, scores were converted to
an ordinal scale from -2 to 2, with _definitely wrong_ corresponding to -2,
and _definitely good_ corresponding to 2. Mean scores and standard deviations
for each reviewer are presented in Table 4. It can be observed that reviewers
had significantly different score distributions (Fleiss’ $\kappa=0.501$,
Krippendorff’s $\alpha=0.474$). This indicates moderate agreement, which is
desirable when seeking varied opinions on the synonymy of terms.

Table 4: Term conflation: reviewer’s scores distributions Reviewer | Mean score | St. dev.
---|---|---
Reviewer 1 | -1.16 | 1.60
Reviewer 2 | -0.85 | 1.67
Reviewer 3 | -0.13 | 1.79

It was found that, in CSO, frequently conflated were terms surrounding a
single subject. Here, as an example one can consider _classification system_ ,
_classification tasks_ , _classification results_. Other synonyms seemed of
little use, introducing only singular/plural variations, or different variants
of hyphen placement. Abbreviations were also occasionally mixed into the
labels – for example _neural network (nn)_ exists as a single term.

Additionally, the reviewers made plentiful remarks about scrutinized clusters.
In particular, many out-of-scope clusters were found, frequently from the
fields of pedagogy and genetics. This is consistent with the observations of
Han et al. [36], who point out that CSO fails to distinguish computer science
topics from those of related fields.

###  4.7 Summary

Overall, in above experiments, 3 137 synonym clusters and 4 287 topics were
found to be affected by at least one issue. For cases where the fix is
unambiguous, suggested patches in the form of Turtle files were prepared and
made available.

##  5 Discussion and Concluding Remarks

Over the course of a use case-driven investigation, 31 heterogeneous
ontologies were scrutinized with regard to their quality. Not only use case-
specific issues were found, but also those related to basic principles of
ontology design, clarity and consistency.

Overall, OBO Foundry ontologies present an extremely varied landscape of
sizes, quality assurance methods, and stakeholders involved. Thus, it is
interesting to observe that out of 30 studied ontologies, 28 were found to
involve at least one class of problems. Moreover, in majority of cases, found
problems concerned a very large number of triples (these were not “localized
problems” but clear cases of systemic ones). One method to tackle them would
be, for OBO Foundry, to further expand upon their set of guidelines, for
example by specifying, which properties to use for describing metadata.
Defining formally the ranges of properties could also prove valuable, as it
would enable automatic tools to check for potential problems. Cross-ontology
references also require improvements, to better conform with Linked Data
standards and have clearer meaning. Finally, the OBO to OWL translation should
be further improved, to better preserve semantic information.

In the case of CSO, it was found that some issues stem from the decisions made
when designing its overall structure, while others can be attributed to the
non-exhaustive quality assurance of the automatically generated ontology. It
can be hoped that the reported results and suggested fixes will be valuable to
the further development of CSO.

In the experiments, the usefulness of natural language processing methods and
network analysis for ontology quality assurance were clearly shown. Most
importantly, these techniques can greatly reduce the amount of human labor
required to perform the evaluation. Thus, ontological engineers should not be
afraid to broaden their toolboxes with less conventional techniques. This is
particularly the case since the joint use of approaches described above can
substantially improve quality of ontologies, while considerably reducing the
amount of work that has to be completed by human experts.

Although the automatic ontology inspection methods, such as the OBO Dashboard,
do help with the problem of ontology quality, they cannot solve it on their
own. There will always be limits to the scope of checks performed by
automated, general tools. Thus, it is vital to pay special attention to the
issues observed during reuse, and _continuously_ integrate resulting
observations to improve the ontologies. Reusability is, arguably, the
_essence_ of the Linked Open Data vision. Therefore, it should be the core
driving factor for ontology development. It was shown that using multiple
ontologies together is particularly hard, with numerous, still largely
unresolved, issues regarding cross-ontology references. Thus, this topic
deserves special attention in future research.

Overall, it was demonstrated how reusing ontologies, especially in demanding
applications, serves as the real test of their quality. The purpose of the
performed experiments is not to present them as general approaches to ontology
quality assurance. Rather, they exemplify the process of continuous ontology
reuse, evaluation, and improvement.

## Acknowledgments

We would like to thank Anastasiya Danilenka and Jan Sawicki for their
invaluable help in reviewing CSO’s entities.

## References

  * [1] Noy N, Mcguinness D.  Ontology Development 101: A Guide to Creating Your First Ontology.  Knowledge Systems Laboratory. 2001 01;32.
  * [2] Bizer C, Heath T, Berners-Lee T.  Linked data: The story so far.  In: Semantic services, interoperability and web applications: emerging concepts. IGI global; 2011. p. 205-27.
  * [3] Fernández-López M, Poveda-Villalón M, Suárez-Figueroa MC, Gómez-Pérez A.  Why are ontologies not reused across the same domain?  Journal of Web Semantics. 2019;57:100492.
  * [4] Cota G, et al.  The landscape of ontology reuse approaches.  Appl Practices Ontol Des, Extraction, Reason. 2020;49:21.
  * [5] Brank J, Grobelnik M, Mladenic D.  A survey of ontology evaluation techniques.  In: Proceedings of the conference on data mining and data warehouses (SiKDD 2005). Citeseer Ljubljana, Slovenia; 2005. p. 166-70.
  * [6] Raad J, Cruz C.  A survey on ontology evaluation methods.  In: Proceedings of the International Conference on Knowledge Engineering and Ontology Development, part of the 7th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management; 2015. p. 179-86.
  * [7] Amith M, He Z, Bian J, Lossio-Ventura JA, Tao C.  Assessing the practice of biomedical ontology evaluation: Gaps and opportunities.  Journal of biomedical informatics. 2018;80:1-13.
  * [8] Al-Sayed MM, Hassan HA, Omara FA.  Towards evaluation of cloud ontologies.  Journal of Parallel and Distributed Computing. 2019;126:82-106.
  * [9] Casellas N.  Ontology Evaluation through Usability Measures.  In: Meersman R, Herrero P, Dillon T, editors. On the Move to Meaningful Internet Systems: OTM 2009 Workshops. Berlin, Heidelberg: Springer Berlin Heidelberg; 2009. p. 594-603.
  * [10] Laadhar A, Abrahão E, Jonquet C.  Investigating one million XRefs in thirthy ontologies from the OBO world.  In: 11th International Conference on Biomedical Ontologies (ICBO); 2020\. p. G.1-12.
  * [11] Szmeja P, Wasielewska K, Ganzha M, Drozdowicz M, Paprzycki M, Fidanova S, et al.  Reengineering and extending the Agents in Grid Ontology.  In: International Conference on Large-Scale Scientific Computing. Springer; 2013. p. 565-73.
  * [12] Ganzha M, Paprzycki M, Pawlowski W, Szmeja P, Wasielewska K.  Towards Common Vocabulary for IoT Ecosystems - preliminary Considerations.  In: Nguyen NT, Tojo S, Nguyen LM, Trawinski B, editors. Intelligent Information and Database Systems - 9th Asian Conference, ACIIDS 2017, Kanazawa, Japan, April 3-5, 2017, Proceedings, Part I. vol. 10191 of Lecture Notes in Computer Science; 2017. p. 35-45.  Available from: https://doi.org/10.1007/978-3-319-54472-4_4.
  * [13] Wasielewska-Michniewska K, Ganzha M, Paprzycki M, Denisiuk A.  Application of ontologies in the enterprise – overview and critical analysis.  In: Proceedings of the Third International Conference on Information Management and Machine Intelligence: ICIMMI 2021, Algorithms for Intelligent Systems (AIS), TO APPEAR. Springer; 2022. .
  * [14] Sowiński P, Wasielewska-Michniewska K, Ganzha M, Paprzycki M.  Topical Classification of Food Safety Publications with a Knowledge Base.  arXiv preprint arXiv:220100374. 2022.
  * [15] Smith B, Ashburner M, Rosse C, Bard J, Bug W, Ceusters W, et al.  The OBO Foundry: coordinated evolution of ontologies to support biomedical data integration.  Nature biotechnology. 2007;25(11):1251-5.
  * [16] Donnelly K, et al.  SNOMED-CT: The advanced terminology and coding system for eHealth.  Studies in health technology and informatics. 2006;121:279.
  * [17] Jackson RC, Balhoff JP, Douglass E, Harris NL, Mungall CJ, Overton JA.  ROBOT: a tool for automating ontology workflows.  BMC bioinformatics. 2019;20(1):1-10.
  * [18] Jackson RC, Matentzoglu N, Overton JA, Vita R, Balhoff JP, Buttigieg PL, et al.  OBO Foundry in 2021: operationalizing open data principles to evaluate ontologies.  Database: The Journal of Biological Databases and Curation. 2021;2021.
  * [19] Moreira D, Mungall C, Shah N, Aitken S, Richter JD, Redmond T, et al.  The NCBO OBOF to OWL Mapping.  Nature Precedings. 2009:1-1.
  * [20] Miles A, Bechhofer S.  SKOS Simple Knowledge Organization System Reference.  W3C; 2009.
  * [21] Hoyt CT. An integrative registry of biological databases, ontologies, and nomenclatures; v0.0.6. Zenodo; 2020.  Available from: https://doi.org/10.5281/zenodo.4404608.
  * [22] Courtot M, Gibson F, Lister AL, Malone J, Schober D, Brinkman RR, et al.  MIREOT: The minimum information to reference an external ontology term.  Applied Ontology. 2011;6(1):23-33.
  * [23] Salatino AA, Thanapalasingam T, Mannocci A, Osborne F, Motta E.  The Computer Science Ontology: a large-scale taxonomy of research areas.  In: International Semantic Web Conference. Springer; 2018. p. 187-205.
  * [24] Auer S, Bizer C, Kobilarov G, Lehmann J, Cyganiak R, Ives Z.  DBpedia: A nucleus for a web of open data.  In: The semantic web. Springer; 2007. p. 722-35.
  * [25] Vrandečić D, Krötzsch M.  Wikidata: a free collaborative knowledgebase.  Communications of the ACM. 2014;57(10):78-85.
  * [26] Tanon TP, Weikum G, Suchanek F.  YAGO 4: A reason-able knowledge base.  In: European Semantic Web Conference. Springer; 2020. p. 583-96.
  * [27] Osborne F, Motta E.  Klink-2: integrating multiple web sources to generate semantic topic networks.  In: International Semantic Web Conference. Springer; 2015. p. 408-24.
  * [28] Ashburner M, Ball CA, Blake JA, Botstein D, Butler H, Cherry JM, et al.  Gene Ontology: tool for the unification of biology.  Nature genetics. 2000;25(1):25-9.
  * [29] Jaradeh MY, Oelen A, Farfar KE, Prinz M, D’Souza J, Kismihók G, et al.  Open Research Knowledge Graph: next generation infrastructure for semantic scholarly knowledge.  In: Proceedings of the 10th International Conference on Knowledge Capture; 2019. p. 243-6.
  * [30] Hagberg A, Swart P, S Chult D.  Exploring network structure, dynamics, and function using NetworkX.  Los Alamos National Lab.(LANL), Los Alamos, NM (United States); 2008.
  * [31] Bollobas B, Gehring FW, Halmos PR.  Modern Graph Theory. vol. 184 of Graduate Texts in Mathematics.  New York, NY: Springer; 2013.
  * [32] Cordasco G, Gargano L.  Community detection via semi-synchronous label propagation algorithms.  In: 2010 IEEE international workshop on: business applications of social network analysis (BASNA). IEEE; 2010. p. 1-8.
  * [33] Bechhofer S, Van Harmelen F, Hendler J, Horrocks I, McGuinness DL, Patel-Schneider PF, et al.  OWL web ontology language reference.  W3C recommendation. 2004;10(2):1-53.
  * [34] Ghazvinian A, Noy NF, Musen MA.  Creating mappings for ontologies in biomedicine: simple methods work.  In: AMIA Annual Symposium Proceedings. vol. 2009. American Medical Informatics Association; 2009. p. 198.
  * [35] Song K, Tan X, Qin T, Lu J, Liu TY.  MPNet: Masked and permuted pre-training for language understanding.  arXiv preprint arXiv:200409297. 2020.
  * [36] Han K, Yang P, Mishra S, Diesner J.  WikiCSSH: extracting computer science subject headings from Wikipedia.  In: ADBIS, TPDL and EDA 2020 Common Workshops and Doctoral Consortium. Springer; 2020. p. 207-18.