# Which topics are best represented by science maps? An analysis of clustering
effectiveness for citation and text similarity networks

Juan Pablo Bascur  0000-0002-4077-1024 Suzan Verberne  0000-0002-9609-9505
Nees Jan van Eck  0000-0001-8448-4521 Ludo Waltman  0000-0001-8249-1752

(June 2024)

###### Abstract

A science map of topics is a visualization that shows topics identified
algorithmically based on the bibliographic metadata of scientific
publications. In practice not all topics are well represented in a science
map. We analyzed how effectively different topics are represented in science
maps created by clustering biomedical publications. To achieve this, we
investigated which topic categories, obtained from MeSH terms, are better
represented in science maps based on citation or text similarity networks. To
evaluate the clustering effectiveness of topics, we determined the extent to
which documents belonging to the same topic are grouped together in the same
cluster. We found that the best and worst represented topic categories are the
same for citation and text similarity networks. The best represented topic
categories are diseases, psychology, anatomy, organisms and the techniques and
equipment used for diagnostics and therapy, while the worst represented topic
categories are natural science fields, geographical entities, information
sciences and health care and occupations. Furthermore, for the diseases and
organisms topic categories and for science maps with smaller clusters, we
found that topics tend to be better represented in citation similarity
networks than in text similarity networks.

##  1 Introduction

Science maps [4] are visualizations that provide an overview of the content of
collections of scientific publications. The goal of science mapping is to find
meaningful structures in the bibliographic metadata of publications (e.g, in
the references, the titles and abstracts, or the authors). These structures
can then be used for literature analysis or information retrieval [5, 9]. Some
of the uses of science maps are field delimitation [40], research policy [33],
and enhanced document browsing [3]. A well established practice to create
science maps is to cluster similar publications, and then to summarize the
content of the resulting clusters. Our focus in this paper is on science maps
created in this way.

When using science maps, it is important to be aware that scientific
publications usually have more than a single topic (e.g., a document about
lung cancer is about both lungs and cancer), but in a science map they
typically can be assigned to only one cluster, representing a single topic.
Losing information when creating science maps is unavoidable, but it does
raise the question of which of the topics addressed in a collection of
publications a clustering will be based on. This is not an idle question, as
there can be significant disagreement between expert-identified and cluster-
identified topics [16], indicating that expert-identified topics are poorly
represented by the clusters in a science map. More specifically, an expert
with an interest in a particular topic may find that publications related to
this topic are scattered over many different clusters, with most of the
publications in these clusters being unrelated to the expert’s topic of
interest. By providing a better understanding of the types of topics that are
well or less well represented in science maps, we hope our research will
contribute to a more effective use of these maps.

In this paper, we use the Medical Subject Headings (MeSH) terms to investigate
clustering for biomedical topics. Our focus is on clustering solutions based
on either citation or text similarity networks, which are the most common
document similarity metrics for creating science maps. We aim to find out
which MeSH terms are well represented by the clusters in a science map, a
phenomenon that we will refer to as clustering effectiveness. Our approach is
to group topics, represented by MeSH terms, into topic categories, represented
by branches of the MeSH tree, and to then evaluate clustering effectiveness at
the level of these topic categories.

Our research questions are as follows:

  * •

Which topic categories have the highest and lowest clustering effectiveness in
citation and text similarity networks?

  * •

Which topic categories have higher clustering effectiveness in citation
similarity networks than in text similarity networks, and vice versa?

In the remainder of this paper, we will discuss background literature,
describe our data, define our metrics, report our analyses and discuss our
results.

##  2 Background

This section has the following structure: In Subsection 2.1 we explain how
science maps are usually evaluated, in Subsection 2.2 we explore the criticism
of science maps that originates from one particular evaluation method, and in
Subsection 2.3 we explain the challenges of understanding the meaning of the
clusters in a science map.

###  2.1 Evaluation of science maps

The most common method to evaluate the quality of a science map is to ask
experts if the science map reflects their knowledge of the field of interest.
The utility of this evaluation method has recently been called into question
because it usually gives an inconclusive result: The experts tend to agree
with most of the science map but identify caveats about certain details [11].
Additionally, there are several issues intrinsic to the expert evaluation
method: The evaluation criteria may differ between experts; seeing the map may
affect the expert’s understanding of a field; the expert may be biased towards
the subfields of their interest; and the expert may have limited competence in
some subfields [11].

An alternative method to evaluate the quality of a science map is to consider
the intrinsic properties of the clustering process used to create science
maps. Commonly used intrinsic properties are desirable characteristics such as
homogeneous cluster sizes, few small clusters, stable clustering solutions
between different runs of the cluster algorithm, and a short computing time to
create the clusters [32]. An intrinsic properties evaluation method was
developed by Waltman et al. [36]. Their method assumes that there exists an
ideal map and then assesses how closely a clustering solution matches this
map. It evaluates the quality of a clustering solution based on one metric
using another unrelated baseline metric (e.g., a clustering solution based on
citation similarity can be evaluated using text similarity). Ahlgren et al.
[1], who created the clustering solutions that we use in our current work,
used this method with MeSH terms similarity as their baseline metric.

A third approach to evaluate the quality of a science map is to define a
ground truth made of documents that correspond to a given topic, and evaluate
the overlap between the clustering solution and the ground truth: either the
extent to which all documents of each field are contained in a single cluster
[16, 15], or the extent to which each cluster contains only documents of a
single field [27, 16, 17, 12]. Some studies obtained the ground truth from the
references of review articles [19, 30], but most studies obtained the ground
truth using expert knowledge. To our knowledge, MeSH terms have not been used
as ground truths, although Sjögårde, Ahlgren and Waltman [31] used MeSH terms
to label clusters in science maps. It is worth mentioning that our work has a
different goal than evaluating a science map based on a ground truth. Instead
of evaluating the quality of a science map based on a set of topics, we
evaluate which topics are most accurately represented in a science map.

###  2.2 Criticism of science maps based on ground truth evaluations

Evaluations that use expert knowledge ground truths have recently questioned
the quality of science maps by challenging their ability to identify fields of
science [12, 16, 15, 17]. For example, Held and Velden [17] found that science
maps provide clusters about organisms rather than clusters about the field of
invasive biology. One explanation for these negative results is that a
document can belong to several fields or topics but only to a single cluster
[16, 17] (although some maps allow documents to belong to multiple clusters
[38, 13]). Another explanation is that the choice of a clustering algorithm
can have a significant influence on the quality of a science map, and it is
impossible to know beforehand which clustering algorithm will give the best
result for a given map [14, 27].

Similar negative findings have also emerged in areas beyond science mapping.
For example, the field of complex systems has developed algorithms to clusters
the elements that share a given property (i.e., the cluster matches the ground
truth), but these algorithms fail in practical applications. On the other
hand, this field has succeeded in practical applications of algorithms that
infer the properties of an element based on the properties of the other
elements in a cluster (e.g., fraud in telecommunications networks, function in
biological networks) [10, 18, 25].

###  2.3 Meaning of the clusters

The negative findings discussed in the previous subsection suggest that
science maps, and clustering in general, offer poor representations of certain
ground truths. However, this does not mean that science maps are not useful.
As mentioned in Subsection 2.1, experts tend to agree that science maps
reflect their knowledge of a field. Also, in the field of complex systems,
Newman and Clauset [24] argued that, even if clusters do not reflect the
ground truth, they can still describe meaningful structures in the data. Our
work tries to find out what kinds of structures are described by the clusters
in a science map.

In this direction, Seitz et al. [29] found that the epistemic functions of
citations (i.e., what kind of knowledge is a citation contributing to in a
document) within a cluster are different from the epistemic functions of
citations between clusters. This suggests that clusters tend to represent
certain epistemic functions more than others. Also, the type of similarity
network might have an effect on the meaning of clusters. For example, Ding [8]
found significant differences between clusters emerging from co-authorship
networks of documents and clusters emerging from topic modeling of documents.
On the other hand, Velden et al. [35] found that there is a substantial
similarity between the topics found in science maps built from citation and
text similarity networks, although science maps built from citation networks
are better at distinguishing topics when words related to the topics have
multiple meanings.

##  3 Methods

This section has the following structure: In Subsection 3.1, we define how we
selected our data. In Subsection 3.2, we explain how we modified our data so
to better fit our experimental design. In Subsection 3.3, we explain how we
evaluate the clustering effectiveness of topic categories.

###  3.1 Data selection

#### Documents

The collection of documents that we use in our work comes from the work by
Ahlgren et al. [1]. This is a collection of 2,941,119 PubMed documents
published between 2013 and 2017.

#### Clustering solutions

The clustering solutions that we use are the ones generated by Ahlgren et al.
They created several clustering solutions for the above mentioned documents
using different similarity metrics and granularities. They used the Leiden
algorithm [34] for clustering, where the parameter Resolution controls the
granularity of the clustering solution (a higher Resolution value generates
smaller clusters). We select two similarity metrics, one for citation and one
for text, based on which pair of metrics produce similar cluster sizes at the
same Resolution. The citation metric is Extended direct citation, which is
calculated using direct citations between documents plus the citations to
documents outside the document collection [36]. The text metric is BM25 [26],
which uses the noun phrases in the titles and abstracts of the documents, and
weights them inversely to their frequency in the document collection [36]. For
each metric we selected the three clustering solutions that use the Resolution
values $2*10^{-6}$, $2*10^{-5}$ or $2*10^{-4}$, enabling us to evaluate
different cluster sizes. We selected these Resolution values because the first
and second value yield cluster sizes similar to those in the algorithmic
mapping of science [37] used in the CWTS Leiden Ranking [7], while the third
value enables us to evaluate clusters of smaller size.

#### Topics

Our topics are the MeSH terms, a controlled vocabulary thesaurus from the
National Library of Medicine (NLM) used for indexing PubMed. MeSH terms are
semi-automatically annotated to documents by the NLM [22]. We obtained the
MeSH terms annotated for each document in our document collection, plus the
metadata of the MeSH terms themselves, from the PubMed and MeSH databases
(version from 2023) available in the database system of the Centre for Science
and Technology Studies (CWTS) at Leiden University.

#### Topic categories

Our topic categories are the 16 nodes at the first level of the MeSH
hierarchical tree of topics [22], also known as the branches of the MeSH tree.
We use branches because they group the MeSH terms in epistemological
categories (e.g., organisms), which are the categories sometimes used for
topical analysis of clusters [16, 29]. A single MeSH term can have instances
in different branches of the MeSH tree. We will address this in Subsection
3.2.

###  3.2 Data prepossessing

#### Clustering solution cleaning

We cleaned the clustering solutions by removing the clusters with fewer than
10 documents because these clusters usually had documents that were
disconnected from the largest connected component of the similarity network.
Removing these clusters removed only a minor fraction of the total number of
documents. The statistics of each clustering solution after this process can
be seen in Table 1. In this table, the variable $S$ is the smallest set of
clusters that together cover at least half of the documents in the dataset.
This means that $S$ contains the biggest clusters in the clustering solution.
We report statistics for $S$ to provide some insight into the distribution of
cluster sizes.

Table 1: Statistics of the clustering solutions. $S$ is the smallest set of
clusters that together cover at least half of the documents in the dataset.

#### MeSH term expansion

We would like a MeSH term to be annotated on all documents related to the
topic of the MeSH term, but NLM typically only annotates up to 15 MeSH terms
per document, which means that more generic MeSH terms are not annotated. To
fix this, we expanded the number of MeSH terms annotated to a document by
annotating, for each NLM MeSH term, all MeSH terms that are upstream in the
MeSH tree, or in other words, all ancestors of the NLM MeSH term in the MeSH
tree.

For example, if a document has the NLM MeSH term Abdominal Pain, we also
annotated the upstream MeSH term Pain. While the former MeSH term belongs to
the branch Diseases [C], the latter one belongs not only to the branch
Diseases [C], but also to the branches Psychiatry and Psychology [F] and
Phenomena and Processes [G]. We annotated the MeSH term Pain paired with the
branch Diseases [C], and not with the other two branches. For simplicity, in
the rest of this paper we will refer to MeSH terms paired with a specific
branch simply as MeSH terms. Also, we will refer to the documents that have a
given MeSH term as the MeSH term documents and to the number of these
documents as the MeSH term size.

#### MeSH term removal

We removed some MeSH terms to improve the quality of our experiments. Our
first removal criterion is size. We removed MeSH terms with size greater than
300,000 (i.e., 10% of the document set) because these MeSH term documents can
saturate the clusters just by random chance, distorting our analysis. We also
removed the MeSH terms with size 500 or less, because we want the smallest
MeSH terms to be close but smaller than the median size of the clusters for
resolution $2*10^{-5}$.

Our second removal criterion is redundancy. Due to the MeSH term expansion
process, some MeSH terms had almost the same documents as their ancestor in
the MeSH tree, like Dogs and its ancestor Canidae. This redundancy could
distort our results. We therefore decided to remove the redundant MeSH terms
by grouping together MeSH terms that share many documents and retaining only
the smallest MeSH term from the group, which in our experience tends to be the
term that best represents the group. The extent to which MeSH terms share
documents was measured using Jaccard similarity, the grouping algorithm was
agglomerate hierarchical clustering with the Complete Linkage method [28], and
the criterion for forming MeSH term groups was for MeSH terms to have a
Jaccard similarity of at least 0.9. In cases where a group had more than one
smallest MeSH term, we selected the one at the lowest level in the MeSH tree
or the one with the largest number of instances in the MeSH tree.

#### Branch removal

To make our results more robust, we removed the branches with fewer than 100
MeSH terms. We ended up with the 14 branches shown in Table 2.

#### Size bins of MeSH terms

The size of a MeSH term can be expected to have an effect on its clustering
effectiveness. We therefore grouped the MeSH terms according to their size. We
refer to these groups as Size bins. To ensure the robustness of our results,
we only considered Size bins that had at least 10 MeSH terms per branch. This
resulted in five Size bins: 501-1,000, 1,001-2,000, 2,001-4,000, 4,001-8,000,
and 8,001-16,000. The number of MeSH terms per Size bin can be seen in Table
2.

Table 2: Number of MeSH terms per branch and Size bin.

###  3.3 Clustering effectiveness

#### Selection of clusters

To find out which MeSH terms are well represented by the clusters in a science
map, we introduce the notion of clustering effectiveness. Measuring the
clustering effectiveness of a MeSH term starts by selecting a subset of
clusters. Our cluster selection criterion is to select the clusters with the
largest number of MeSH term documents while making sure that the selected
clusters cover at least a given share of all MeSH term documents. We call this
share Coverage. We consider three Coverage values: 0.25, 0.50 and 0.75. Our
cluster selection criterion minimizes the number of selected clusters for a
given Coverage value. It is inspired by cluster quality metrics of Yuan, Zobel
and Ling [39]. We expect our cluster selection criterion to reflect the
clusters a user of a science map is likely to select while exploring the map.

#### Clustering effectiveness metrics

Once we have the selected clusters for a given MeSH term, we measure
clustering effectiveness using two metrics:

  * •

Purity: Purity represents the extent to which the selected clusters are
composed of MeSH term documents. It is the fraction of documents in the
selected clusters that are MeSH term documents. In mathematical terms, Purity
is defined as:

$Purity=\frac{\sum_{i=1}^{N}{|D_{i}\cap D_{M}|}}{\sum_{i=1}^{N}|D_{i}|}$ (1)

Here, $N$ denotes the number of selected clusters, $D_{i}$ denotes the
documents in selected cluster $i$ and $D_{M}$ denotes the MeSH term documents.
The higher Purity, the more effective the clustering. Purity is bounded
between zero and one.

####

  * •

Inverse count of clusters (ICC): ICC represents the extent to which the MeSH
term documents are contained in a small number of clusters. ICC is defined as
one divided by the number of selected clusters. In mathematical terms, ICC is
defined as:

$ICC=\frac{1}{N}$ (2)

The higher ICC, the more effective the clustering. Like Purity, ICC is bounded
between zero and one.

We use two metrics instead of one to control for MeSH term size and cluster
size: If there are few MeSH term documents, or if they are in big clusters,
then ICC will be high but Purity will be low, and vice versa.

The Purity and ICC of a MeSH term are calculated for a given Coverage value,
Resolution value and similarity network. We use C-Purity and C-ICC to refer to
Purity and ICC calculated for a citation network, and T-Purity and T-ICC to
refer to Purity and ICC calculated for a text network.

We also provide metrics for the difference in Purity and ICC between citation
and text networks for a given MeSH term. These metrics, referred to as rPurity
(Ratio Purity) and rICC (Ratio ICC), are calculated as the logarithm base 2 of
C-Purity or C-ICC divided by T-Purity or T-ICC. In mathematical terms, rPurity
and rICC are defined as:

$rPurity=\log_{2}\left(\frac{\text{{C-Purity}}}{\text{{T-Purity}}}\right)$ (3)
$rICC=\log_{2}\left(\frac{\text{{C-ICC}}}{\text{{T-ICC}}}\right)$ (4)

Positive values indicate that a citation network yields a higher clustering
effectiveness than a text network, and vice versa.

##  4 Results

###  4.1 Which topic categories have the highest and lowest clustering
effectiveness in citation and text similarity networks?

To answer our first research question, we consider the C-Purity and T-Purity
rankings of the 14 branches for each of the 45 combinations of parameter
values (i.e., three Resolution values combined with three Coverage values
combined with five Size bin values). Table 3 shows the number of times each
branch appears in each position in the C-Purity and T-Purity rankings. The
order of the branches in the table was determined manually so that the
branches that frequently occupy higher ranking positions are above of the ones
that occupy lower ranking positions. We found that the ICC rankings are
strongly correlated with the Purity rankings, so we do not show them.

From Table 3 we make the following observations:

  * •

Most of the branches occupy between one and four adjacent positions, which
shows that the position of the branches tends to be stable for different
parameter values.

  * •

For both C-Purity and T-Purity, the top five branches are almost always in
positions 1 to 7, and the bottom four branches are almost always in positions
8 to 14. We therefore consider the top five and bottom four branches as the
the ones with, respectively, the highest and lowest clustering effectiveness.

  * •

The top five and bottom four branches are the same for C-Purity and T-Purity,
showing that in this respect citation and text networks yield very similar
outcomes.

  * •

The top five branches are Diseases [C], Organisms [B], Anatomy [A],
Analytical, Diagnostic and Therapeutic Techniques, and Equipment [E] and
Psychiatry and Psychology [F].

  * •

The bottom four branches are Health Care [N], Disciplines and Occupations [H],
Information Science [L] and Geographicals [Z].

Figure 1 shows the distribution of the Purity and ICC values of each branch
for the 45 combinations of parameter values. The box plots for the different
branches heavily overlap with each other due to the effect of the parameter
values on Purity and ICC. From Figure 1 we observe that C-Purity, T-Purity,
C-ICC and T-ICC are substantially higher for the branch Diseases [C] than for
the other branches, while they are substantially lower for the branch
Geographicals [Z]. This also explains why in Table 3 these branches almost
always appear in position 1 and 14, respectively.

Table 3: Number of times each branch appears in each ranking position, using
either C-Purity (top) or T-Purity (bottom) as ranking criterion.

Figure 1: Box plots showing the distribution of C-Purity, C-ICC, T-Purity and
T-ICC for each branch. The median values of each box plot are reported along
the right Y axis. The branches are sorted as in Table 3.

###  4.2 Which topic categories have higher clustering effectiveness in
citation similarity networks than in text similarity networks, and vice versa?

To address our second research question, we first evaluate how the ratio
metrics rPurity and rICC correlate with the Size bin, Resolution and Coverage
parameters. The box plots in Figure 2 show the distribution of the rPurity and
rICC values for each value of the Size bin, Resolution and Coverage
parameters. Here we see that higher Resolution and Coverage are correlated
with higher rPurity and rICC. Also, higher Size bin is correlated with lower
rPurity and rICC, but this is a weak correlation.

Figure 2: Box plots showing the distribution of rPurity and rICC for each
value of Size bin, Resolution and Coverage.

The answer to our second research question depends on whether the rPurity and
rICC values of a branch are positive or negative. Positive values indicate
that the clustering effectiveness is higher in citation networks, while
negative values indicate that the clustering effectiveness is higher in text
networks. The box plots in Figure 3 show the distribution of the rPurity and
rICC values of each branch for the 45 combinations of parameter values. For
each branch, the rPurity and rICC distributions include both positive and
negative values. This reflects the dependence of the rPurity and rICC values
on the values of the Size bin, Resolution and Coverage parameters, as was
shown in Figure 2.

Because for each branch the rPurity and rICC distributions include both
positive and negative values, it is not possible to unequivocally conclude
that a branch has a higher clustering effectiveness in either citation
networks or text networks. Nevertheless, it is clear that the branches
Diseases [C] and Organisms [B] tend to have a higher clustering effectiveness
in citation networks than in text networks. rPurity and rICC are almost always
positive for these branches. In contrast, the branches Geographicals [Z],
Information Science [L], Named Groups [M], Analytical, Diagnostic and
Therapeutic Techniques, and Equipment [E] and Phenomena and Processes [G] tend
to have a higher clustering effectiveness in text networks than in citation
networks. However, the results for these branches are less stable, so we need
to be cautious in drawing strong conclusions.

Figure 3: Box plots showing the distribution of rPurity and rICC for each
branch.

##  5 Discussion

This section has the following structure: We discuss what we have learned for
our first research question in Subsection 5.1, for our second research
question in Subsection 5.2, and for the strengths and weaknesses of our work
in Subsection 5.3.

###  5.1 Which topic categories have the highest and lowest clustering
effectiveness in citation and text similarity networks?

Our results show that the MeSH branches with the highest and lowest clustering
effectiveness are the same for citation and text similarity networks. Despite
the different purposes of writing and citing [21], the way scientists write
and the way they cite yield similar rankings of MeSH branches in terms of
clustering effectiveness. It would be interesting to see if the top and bottom
branches are also the same in other similarity networks, like co-tweeting [6],
co-authorship [23], and patent co-citation [20].

The branch Disciplines and Occupations [H], which contains the MeSH terms for
natural science fields, is among the branches with the lowest clustering
effectiveness. This shows that how scientists cite each other is only weakly
related to how they define scientific fields. This might suggest the need for
alternative approaches to defining scientific fields, for instance based on
science map clusters.

Held and Velden [17] reported that a given science map was poor at showing the
field of invasive biology, and instead placed documents related to the field
in clusters about species. Our results are in line with this, because invasive
biology belongs to Disciplines and Occupations [H], one of the bottom four
branches in our results, while species belongs to Organisms [B], one of the
top five branches.

###  5.2 Which topic categories have higher clustering effectiveness in
citation similarity networks than in text similarity networks, and vice versa?

Our results show that which networks yield a higher clustering effectiveness
depends strongly on the Resolution and Coverage values, with higher Resolution
and higher Coverage increasing the clustering effectiveness for citation
networks relative to text networks. Importantly, this does not mean that
higher Resolution and higher Coverage increase the clustering effectiveness
for citation networks in an absolute sense. It means that higher Resolution
and higher Coverage increase the ratio between the clustering effectiveness
for citation networks and the clustering effectiveness for text networks.

Ahlgren et al. [1] developed a method to measure the accuracy of the clusters
in a science map. Using their data and visualization method, we found that the
accuracy of citation networks relative to text networks increases as the
Resolution value increases. This is in line with our results. Unfortunately,
we do not know the mechanism behind this dependency. Our findings for
Resolution could be useful for users of science maps: It tells them that, if
they have two science maps, one based on citations and another based on text,
then decreasing the size of the clusters will make the citation one more
effective relative to the text one, and vice versa.

In the context of field delimitation tasks, where a user of a science map
identifies the clusters that contain the documents of a field, Coverage is
analog to the completeness of the field delimitation. Our findings for
Coverage suggest that citation networks are better for exhaustive field
delimitation, while text networks are better for less exhaustive field
delimitation.

Our results also indicate that, omitting the effect of Resolution and
Coverage, the branches Diseases [C] and Organisms [B] tend to have higher
clustering effectiveness in citation networks than in text networks. To
exemplify what this means for users, we consider the use case of Held and
Velden [17] discussed above: They would like to have a clustering of the field
of invasive biology, but in their science map invasive biology documents are
spread over clusters about organisms. If instead of a citation network a text
network is used, the organisms will probably be clustered less effectively,
which may give the opportunity for invasive biology documents to form their
own clusters instead of being part of clusters about organisms.

###  5.3 Strengths and weaknesses

We see the use of MeSH terms as an important strength of our work. An
alternative approach could be to ask experts to assign documents to topics,
but this cannot be done at the scale at which MeSH terms provide document-
topic links. Also, MeSH terms link documents to topics at a scale that no
other classification scheme, like the Mathematics Subject Classification, the
ACM Computing Classification System, or the Physics Subject Headings, is able
to provide.

We also improved the utility of the MeSH terms by using Coverage, MeSH term
expansion, MeSH term removal and MeSH branches in our experimental design.
Coverage diminished the effect of mislabeled documents (e.g., the document
with DOI 10.1007/s12603-020-1457-6 is incorrectly labeled with the MeSH term
Alcohol Drinking) by ignoring a certain share of the documents with a
particular MeSH term. MeSH term expansion allowed us to have a collection of
documents for each MeSH term that represent the topic of the MeSH term more
accurately. MeSH term removal allowed us to ensure that our results are not
affected by redundant MeSH terms. Using the MeSH branches as topic categories
allowed us to use a curated scheme of topic categories. However, some topic
categories may be absent from the MeSH tree (e.g., topics linking diseases
with their medicines) and some lower levels of the MeSH tree may be more
informative as topic categories (e.g., the children of the branch Disciplines
and Occupations [H] are Natural Science Disciplines and Health Occupations,
which may be more informative as topic categories than the branch itself).

Another strength of our work is that we evaluated clustering effectiveness per
MeSH term, while other studies, like Waltman et al. [36], evaluated a
clustering solution as a whole. Our method is also insensitive to the effect
of size differences between MeSH terms and clusters (e.g., if clusters are
much bigger than MeSH terms, it is impossible to have maximum Purity, and if
they are much smaller, it is impossible to have maximum ICC) because our focus
is on comparing the clustering effectiveness of different topic categories
instead of achieving optimal clustering effectiveness.

A weakness of our work is that we used only one clustering algorithm, the
Leiden algorithm, an algorithm that is commonly used by the science mapping
community. Other studies used multiple algorithms: Held, Laudel and Gläser
[16, 15] analyzed clusters created by the Leiden algorithm and the Infomap
algorithm. Held [14] assessed the suitability of the Leiden, Louvian, OSLM and
Infomap algorithms for creating clusters. Beyond science maps, Rossetti,
Pappalardo and Rinzivillo [27] showed that different clustering algorithms
(Louvain, Infohiermap, cFinder, Demon, iLCD and Ego-Network) have differential
performance for different types of networks (DBLP co-authorship network,
Amazon co-purchase network, YouTube users network, and LiveJournal users
network).

Another weakness of our work is that we used only one citation similarity
metric (extended direct citation) and only one text similarity metric (BM25).
Future work should ideally evaluate multiple citation and text similarity
metrics, because different citation metrics and different text metrics may
yield different results.

##  6 Conclusion

In this paper we explored science maps of mostly biomedical topics, analyzing
the clustering effectiveness for different topic categories. We hope our work
will contribute to a more effective use of science maps. We addressed the
following research questions:

#### Which topic categories have the highest and lowest clustering
effectiveness in citation and text similarity networks?

We found that the answer is the same for citation and text similarity
networks. Paraphrasing the topic category names, the topic categories with the
highest clustering effectiveness are diseases, psychology, anatomy, organisms
and the techniques and equipment used for diagnostics and therapy, while the
topic categories with the lowest clustering effectiveness are natural science
fields, geographical entities, information sciences and health care and
occupations. Also, the diseases category has a substantially higher clustering
effectiveness than all other categories, while the geographical entities
category has a substantially lower clustering effectiveness.

#### Which topic categories have higher clustering effectiveness in citation
similarity networks than in text similarity networks, and vice versa?

We found that there are two factors that can make any topic category have
higher clustering effectiveness in either network. The first factor is the
size of the clusters generated by the clustering process (i.e., the Resolution
parameter). The smaller the size, the higher the clustering effectiveness in
citation networks relative to text networks. The second factor, specific to
our experimental setting, is the percentage of all topic documents that must
be covered by the selected clusters (i.e., the Coverage parameter). The higher
this percentage, the higher the clustering effectiveness in citation networks
relative to text networks. Regardless of these two factors, we found that the
diseases and organisms topic categories tend to have higher clustering
effectiveness in citation networks than in text networks.

####

Our work has shown that there is a strong tendency for clusters in science
maps to represent some topics better than others. Further research could
explore how to control which topics are clustered better, so that users of
science maps can adjust the maps to their needs.

##  7 Data availability

The code used to run the experiments and the data needed to replicate the
results are available in Zenodo [2].

## References

  * [1] P. Ahlgren, Y. Chen, C. Colliander, and N. J. van Eck, Enhancing direct citations: A comparison of relatedness measures for community detection in a large set of pubmed publications, _Quantitative Science Studies_ 1 no. 2 (2020), 714–729. https://doi.org/10.1162/qss_a_00027.
  * [2] J. P. Bascur, Which topics are best represented by science maps? An analysis of clustering effectiveness for citation and text similarity networks (data), June 2024. https://doi.org/10.5281/zenodo.11181030.
  * [3] J. P. Bascur, S. Verberne, N. J. van Eck, and L. Waltman, Academic information retrieval using citation clusters: in-depth evaluation based on systematic reviews, _Scientometrics_ 128 no. 5 (2023), 2895–2921.
  * [4] C. Chen, Science mapping: a systematic review of the literature, _Journal of Data and Information Science_ 2 no. 2 (2017), 1–40. https://doi.org/10.1515/jdis-2017-0006.
  * [5] M. J. Cobo, A. G. López-Herrera, E. Herrera-Viedma, and F. Herrera, Science mapping software tools: Review, analysis, and cooperative study among tools, _Journal of the American Society for Information Science and Technology_ 62 no. 7 (2011), 1382–1402. https://doi.org/10.1002/asi.21525.
  * [6] R. Costas, S. de Rijcke, and N. Marres, “heterogeneous couplings”: Operationalizing network perspectives to study science-society interactions through social media metrics, _Journal of the Association for Information Science and Technology_ 72 no. 5 (2021), 595–610. https://doi.org/10.1002/asi.24427.
  * [7] CWTS, Leiden ranking fields, 2023, [Online; accessed 20-March-2023]. Available at https://www.leidenranking.com/information/fields.
  * [8] Y. Ding, Community detection: Topological vs. topical, _Journal of Informetrics_ 5 no. 4 (2011), 498–514. https://doi.org/10.1016/j.joi.2011.02.006.
  * [9] N. J. van Eck, _Methodological advances in bibliometric mapping of science_ , no. EPS-2011-247-LIS, 2011.
  * [10] S. Fortunato, Community detection in graphs, _Physics Reports_ 486 no. 3-5 (2010), 75–174. https://doi.org/10.1016/j.physrep.2009.11.002.
  * [11] J. Gläser, Opening the black box of expert validation of bibliometric maps, in _Lockdown Bibliometrics: Papers not submitted to the STI Conference 2020 in Aarhus_ , 2020, pp. 27–36.
  * [12] R. Haunschild, H. Schier, W. Marx, and L. Bornmann, Algorithmically generated subject categories based on citation relations: An empirical micro study using papers on overall water splitting, _Journal of Informetrics_ 12 no. 2 (2018), 436–447. https://doi.org/10.1016/j.joi.2018.03.004.
  * [13] F. Havemann, J. Gläser, and M. Heinz, Memetic search for overlapping topics based on a local evaluation of link communities, _Scientometrics_ 111 (2017), 1089–1118. https://doi.org/10.1007/s11192-017-2302-5.
  * [14] M. Held, Know thy tools! Limits of popular algorithms used for topic reconstruction, _Quantitative Science Studies_ 3 no. 4 (2022), 1054–1078. https://doi.org/10.1162/qss_a_00217.
  * [15] M. Held, G. Laudel, and J. Gläser, Topic reconstruction from networks of papers may not be possible if only one algorithm is applied to only one data model, in _Lockdown Bibliometrics: Papers not submitted to the STI Conference 2020 in Aarhus_ , 2020, p. 18.
  * [16] M. Held, G. Laudel, and J. Gläser, Challenges to the validity of topic reconstruction, _Scientometrics_ 126 (2021), 4511–4536. https://doi.org/10.1007/s11192-021-03920-3.
  * [17] M. Held and T. Velden, How to interpret algorithmically constructed topical structures of scientific fields? A case study of citation-based mappings of the research specialty of invasion biology, _Quantitative Science Studies_ 3 no. 3 (2022), 651–671. https://doi.org/10.1162/qss_a_00194.
  * [18] D. Hric, R. K. Darst, and S. Fortunato, Community detection in networks: Structural communities versus ground truth, _Physical Review E_ 90 no. 6 (2014), 062805. https://doi.org/10.1103/PhysRevE.90.062805.
  * [19] R. Klavans and K. W. Boyack, Which type of citation analysis generates the most accurate taxonomy of scientific and technical knowledge?, _Journal of the Association for Information Science and Technology_ 68 no. 4 (2017), 984–998. https://doi.org/10.1002/asi.23734.
  * [20] K.-K. Lai and S.-J. Wu, Using the patent co-citation approach to establish a new patent classification system, _Information Processing & Management_ 41 no. 2 (2005), 313–330. https://doi.org/10.1016/j.ipm.2003.11.004.
  * [21] L. Leydesdorff, Theories of citation?, _Scientometrics_ 43 (1998), 5–25. https://doi.org/10.1007/BF02458391.
  * [22] National Institutes of Health, Medical subject headings, Available at https://www.nlm.nih.gov/mesh/meshhome.html, Accessed: 2023-09-07.
  * [23] M. E. Newman, Coauthorship networks and patterns of scientific collaboration, _Proceedings of the National Academy of Sciences_ 101 no. suppl_1 (2004), 5200–5205. https://doi.org/10.1073/pnas.0307545100.
  * [24] M. E. Newman and A. Clauset, Structure and inference in annotated networks, _Nature Communications_ 7 no. 1 (2016), 11863. https://doi.org/10.1038/ncomms11863.
  * [25] L. Peel, D. B. Larremore, and A. Clauset, The ground truth about metadata and community detection in networks, _Science Advances_ 3 no. 5 (2017), e1602548. https://doi.org/10.1126/sciadv.1602548.
  * [26] S. Robertson and H. Zaragoza, The Probabilistic Relevance Framework: BM25 and Beyond, _Foundations and Trends in Information Retrieval_ 3 no. 4 (2009), 333–389. https://doi.org/10.1561/1500000019.
  * [27] G. Rossetti, L. Pappalardo, and S. Rinzivillo, A novel approach to evaluate community detection algorithms on ground truth, in _Complex Networks VII: Proceedings of the 7th Workshop on Complex Networks CompleNet 2016_ , Springer, 2016, pp. 133–144. https://doi.org/10.1007/978-3-319-30569-1_10.
  * [28] SAS Institute Inc., Clustering methods, Available at https://support.sas.com/documentation/cdl/en/statug/63033/HTML/default/viewer.htm#statug_cluster_sect012.htm, 2009, Accessed: 2023-09-07.
  * [29] C. Seitz, M. Schmidt, N. Schwichtenberg, and T. Velden, A case study of the epistemic function of citations—implications for citation-based science mapping, in _Proceedings of the 18th International Conference of the International Society for Scientometrics and Informetrics (ISSI)_ , 2021.
  * [30] P. Sjögårde and P. Ahlgren, Granularity of algorithmically constructed publication-level classifications of research publications: Identification of topics, _Journal of Informetrics_ 12 no. 1 (2018), 133–152. https://doi.org/10.1016/j.joi.2017.12.006.
  * [31] P. Sjögårde, P. Ahlgren, and L. Waltman, Algorithmic labeling in hierarchical classifications of publications: Evaluation of bibliographic fields and term weighting approaches, _Journal of the Association for Information Science and Technology_ 72 no. 7 (2021), 853–869. https://doi.org/10.1002/asi.24452.
  * [32] L. Šubelj, N. J. Van Eck, and L. Waltman, Clustering scientific publications based on citation relations: A systematic comparison of different methods, _PLOS One_ 11 no. 4 (2016), e0154404. https://doi.org/10.1371/journal.pone.0154404.
  * [33] R. Sullivan, S. Eckhouse, and G. Lewison, Using bibliometrics to inform cancer research policy and spending, _Monitoring financial flows for health research_ (2007), 67–78.
  * [34] V. A. Traag, L. Waltman, and N. J. Van Eck, From Louvain to Leiden: Guaranteeing well-connected communities, _Scientific Reports_ 9 no. 1 (2019), 5233. https://doi.org/10.1038/s41598-019-41695-z.
  * [35] T. Velden, K. W. Boyack, J. Gläser, R. Koopman, A. Scharnhorst, and S. Wang, Comparison of topic extraction approaches and their results, _Scientometrics_ 111 no. 2 (2017), 1169–1221. https://doi.org/10.1007/s11192-017-2306-1.
  * [36] L. Waltman, K. W. Boyack, G. Colavizza, and N. J. van Eck, A principled methodology for comparing relatedness measures for clustering publications, _Quantitative Science Studies_ 1 no. 2 (2020), 691–713. https://doi.org/10.1162/qss_a_00035.
  * [37] L. Waltman and N. J. Van Eck, A new methodology for constructing a publication-level classification system of science, _Journal of the American Society for Information Science and Technology_ 63 no. 12 (2012), 2378–2392. https://doi.org/10.1002/asi.22748.
  * [38] S. Xu, J. Liu, D. Zhai, X. An, Z. Wang, and H. Pang, Overlapping thematic structures extraction with mixed-membership stochastic blockmodel, _Scientometrics_ 117 no. 1 (2018), 61–84. https://doi.org/10.1007/s11192-018-2841-4.
  * [39] M. Yuan, J. Zobel, and P. Lin, Measurement of clustering effectiveness for document collections, _Information Retrieval Journal_ 25 no. 3 (2022), 239–268. https://doi.org/10.1007/s10791-021-09401-8.
  * [40] M. Zitt, Meso-level retrieval: IR-bibliometrics interplay and hybrid citation-words methods in scientific fields delineation, _Scientometrics_ 102 no. 3 (2015), 2223–2245. https://doi.org/10.1007/s11192-014-1482-5.


* [19] K. Hasselmann, Stochastic climate models part i. theory, tellus 28 (6) (1976) 473–485.
  * [20] C. Frankignoul, K. Hasselmann, Stochastic climate models, part ii application to sea-surface temperature anomalies and thermocline variability, Tellus 29 (4) (1977) 289–305.
  * [21] E. N. Lorenz, Deterministic nonperiodic flow, Journal of atmospheric sciences 20 (2) (1963) 130–141.
  * [22] B. B. Mandelbrot, B. B. Mandelbrot, The fractal geometry of nature, Vol. 1, WH freeman New York, 1982.
  * [23] B. A.L., A. R., Emergence of scaling in random networks., Science 286 (2009) 509–512.
  * [24] D. J. Watts, S. H. Strogatz, Collective dynamics of ‘small-world’networks, nature 393 (6684) (1998) 440–442.
  * [25] E. Bonabeau, Agent-based modeling: Methods and techniques for simulating human systems, Proceedings of the national academy of sciences 99 (suppl_3) (2002) 7280–7287.
  * [26] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al., Training language models to follow instructions with human feedback, arXiv preprint arXiv:2203.02155 (2022).
  * [27] C. E. Shannon, A mathematical theory of communication, The Bell system technical journal 27 (3) (1948) 379–423.
  * [28] C. Manning, H. Schutze, Foundations of statistical natural language processing, MIT press, 1999.
  * [29] D. E. Rumelhart, G. E. Hinton, R. J. Williams, Learning representations by back-propagating errors, nature 323 (6088) (1986) 533–536.
  * [30] G. Cybenko, Approximation by superpositions of a sigmoidal function, Mathematics of control, signals and systems 2 (4) (1989) 303–314.
  * [31] Y. Bengio, R. Ducharme, P. Vincent, A neural probabilistic language model, Advances in neural information processing systems 13 (2000).
  * [32] T. Mikolov, K. Chen, G. Corrado, J. Dean, Efficient estimation of word representations in vector space, arXiv preprint arXiv:1301.3781 (2013).
  * [33] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, I. Polosukhin, Attention is all you need, Advances in neural information processing systems 30 (2017).
  * [34] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, O. Klimov, Proximal policy optimization algorithms, arXiv preprint arXiv:1707.06347 (2017).
  * [35] R. S. Sutton, A. G. Barto, Reinforcement learning: An introduction, MIT press, 2018\.
  * [36] G. Parisi, Complex systems: a physicist’s viewpoint, Physica A: Statistical Mechanics and its Applications 263 (1) (1999) 557–564, proceedings of the 20th IUPAP International Conference on Statistical Physics.
  * [37] N. Goldenfeld, L. P. Kadanoff, Simple lessons from complexity, Science 284 (5411) (1999) 87–89.
  * [38] P. W. Anderson, More is different, Science 177 (4047) (1972) 393–396.
  * [39] J. H. Miller, S. Page, Complex adaptive systems, Princeton university press, 2009\.
  * [40] J. Kwapień, S. Drożdż, Physical approach to complex systems, Physics Reports 515 (3-4) (2012) 115–226.
  * [41] P. Charbonneau, Natural complexity: a modeling handbook, Vol. 5, Princeton University Press, 2017.
  * [42] S. Thurner, R. Hanel, P. Klimek, Introduction to the theory of complex systems, Oxford University Press, 2018.
  * [43] R. Albert, A.-L. Barabási, Statistical mechanics of complex networks, Reviews of modern physics 74 (1) (2002) 47.
  * [44] M. E. Newman, A.-L. E. Barabási, D. J. Watts, The structure and dynamics of networks., Princeton university press, 2006.
  * [45] R. Cohen, S. Havlin, Complex networks: structure, robustness and function, Cambridge university press, 2010.
  * [46] E. Estrada, The structure of complex networks: theory and applications, Oxford University Press, 2012.
  * [47] M. Posfai, A.-L. Barabasi, Network Science, Citeseer, 2016.
  * [48] S. Wolfram, Statistical mechanics of cellular automata, Reviews of modern physics 55 (3) (1983) 601.
  * [49] A. Ilachinski, Cellular automata: a discrete universe, World Scientific Publishing Company, 2001.
  * [50] J. L. Schiff, Cellular automata: a discrete view of the world, John Wiley & Sons, 2011.
  * [51] D. Helbing, Agent-based modeling, Springer, 2012.
  * [52] S. De Marchi, S. E. Page, Agent-based models, Annual Review of political science 17 (2014) 1–20.
  * [53] E. N. Lorenz, K. Haman, The essence of chaos, Pure and Applied Geophysics 147 (3) (1996) 598–599.
  * [54] S. H. Strogatz, Nonlinear dynamics and chaos with student solutions manual: With applications to physics, biology, chemistry, and engineering, CRC press, 2018\.
  * [55] P. Bak, C. Tang, K. Wiesenfeld, Self-organized criticality, Physical review A 38 (1) (1988) 364.
  * [56] H. J. Jensen, Self-organized criticality: emergent complex behavior in physical and biological systems, Vol. 10, Cambridge university press, 1998.
  * [57] D. L. Turcotte, Self-organized criticality, Reports on progress in physics 62 (10) (1999) 1377.
  * [58] K. Christensen, N. R. Moloney, Complexity and criticality, Vol. 1, World Scientific Publishing Company, 2005.
  * [59] D. Marković, C. Gros, Power laws and self-organized criticality in theory and nature, Physics Reports 536 (2) (2014) 41–74.
  * [60] S. Kullback, R. A. Leibler, On information and sufficiency, The annals of mathematical statistics 22 (1) (1951) 79–86.
  * [61] J. Rissanen, Information and complexity in statistical modeling, Vol. 152, Springer, 2007.
  * [62] C. Beck, Generalised information and entropy measures in physics, Contemporary Physics 50 (4) (2009) 495–510.
  * [63] M. I. Jordan, T. M. Mitchell, Machine learning: Trends, perspectives, and prospects, Science 349 (6245) (2015) 255–260.
  * [64] S. TC, Z. L, Machine learning in complex networks, Springer, 2016.
  * [65] Y. Tang, J. Kurths, W. Lin, E. Ott, L. Kocarev, Introduction to focus issue: When machine learning meets complex systems: Networks, chaos, and nonlinear dynamics, Chaos: An Interdisciplinary Journal of Nonlinear Science 30 (6) (2020) 063151.
  * [66] H. Stanley, Introduction to Phase Transitions and Critical Phenomena, International series of monographs on physics, Oxford University Press, 1987.
  * [67] S. N. Dorogovtsev, A. V. Goltsev, J. F. F. Mendes, Critical phenomena in complex networks, Rev. Mod. Phys. 80 (2008) 1275–1335.  doi:10.1103/RevModPhys.80.1275.
  * [68] D. Stauffer, Scaling theory of percolation clusters, Physics reports 54 (1) (1979) 1–74.
  * [69] J. W. Essam, Percolation theory, Reports on progress in physics 43 (7) (1980) 833\.
  * [70] B. Bollobás, O. Riordan, Percolation, Cambridge University Press, 2006.
  * [71] D. Stauffer, A. Aharony, Introduction to percolation theory, CRC press, 2018.
  * [72] D. Easley, J. Kleinberg, Networks, crowds, and markets: Reasoning about a highly connected world, Cambridge university press, 2010.
  * [73] S. F. Railsback, V. Grimm, Agent-based and individual-based modeling: a practical introduction, Princeton university press, 2019.
  * [74] D. J. MacKay, D. J. Mac Kay, Information theory, inference and learning algorithms, Cambridge university press, 2003.
  * [75] K. P. Murphy, Probabilistic machine learning: an introduction, MIT press, 2022.
  * [76] M. Pascual, J. A. Dunne, R. A. Dunne (Eds.), Ecological Networks: Linking Structure to Dynamics in Food Webs, Oxford University Press, Oxford, UK, 2006\.
  * [77] M. Schroeder, Fractals, chaos, power laws: Minutes from an infinite paradise, Courier Corporation, 2009.
  * [78] B. Mandelbrot, How long is the coast of britain? statistical self-similarity and fractional dimension, science 156 (3775) (1967) 636–638.
  * [79] J. D. Farmer, E. Ott, J. A. Yorke, The dimension of chaotic attractors, Physica D: Nonlinear Phenomena 7 (1-3) (1983) 153–180.
  * [80] J. Feder, Fractals, Springer Science & Business Media, 1988.
  * [81] K. Falconer, Fractal geometry: mathematical foundations and applications, John Wiley & Sons, 2004.
  * [82] F. Hausdorff, Dimension und äußeres maß, Mathematische Annalen 79 (1-2) (1918) 157–179.
  * [83] A. Besicovitch, On sets of fractional dimensions iii: Sets of points of nondifferentiability of absolutely continuous functions and of divergence of fejér sums, Math. Annalen 110 (1934) 331–335.
  * [84] A. S. Besicovitch, On the sum of digits of real numbers represented in the dyadic system. on sets of fractional dimensions ii, Mathematische Annalen 110 (1) (1935) 321–330.
  * [85] D. Stauffer, S. de Oliveira, P. de Oliveira, J. de Sá Martins, Biology, Sociology, Geology by Computational Physicists, ISSN, Elsevier Science, 2006.
  * [86] C. Domb, J. L. Lebowitz (Eds.), PHASE TRANSITIONS AND CRITICAL PHENOMENA. VOL. 11, 1987.
  * [87] M. Gitterman, Phase Transitions: Modern Applications, World Scientific, 2014.
  * [88] J. Yeomans, Statistical Mechanics of Phase Transitions, Clarendon Press, 1992.
  * [89] G. Jaeger, The ehrenfest classification of phase transitions: Introduction and evolution, Archive for History of Exact Sciences 53 (1) (1998) 51–81.  doi:10.1007/s004070050021.
  * [90] J. Cardy, Scaling and Renormalization in Statistical Physics, Cambridge Lecture Notes in Physics, Cambridge University Press, 1996.  doi:10.1017/CBO9781316036440.
  * [91] R. Solé, Phase transitions, Vol. 3, Princeton University Press, 2011.
  * [92] M. Gitterman, Phase transitions: Modern applications, World Scientific Publishing Company, 2013.
  * [93] D. Ruelle, F. Takens, On the nature of turbulence, Les rencontres physiciens-mathématiciens de Strasbourg-RCP25 12 (1971) 1–44.
  * [94] J. C. Sprott, Strange attractors: Creating patterns in chaos, Vol. 9, M & T Books New York, 1993.
  * [95] C. Grebogi, E. Ott, J. A. Yorke, Chaos, strange attractors, and fractal basin boundaries in nonlinear dynamics, Science 238 (4827) (1987) 632–638.
  * [96] P. Holmes, Poincaré, celestial mechanics, dynamical-systems theory and “chaos”, Physics Reports 193 (3) (1990) 137–163.
  * [97] H. Froehling, J. Crutchfield, D. Farmer, N. Packard, R. Shaw, On determining the dimension of chaotic flows, Physica D: Nonlinear Phenomena 3 (3) (1981) 605–617.
  * [98] H. Hentschel, I. Procaccia, The infinite number of generalized dimensions of fractals and strange attractors, Physica D: Nonlinear Phenomena 8 (3) (1983) 435–444.
  * [99] D. G. Tarboton, R. L. Bras, I. Rodriguez-Iturbe, The fractal nature of river networks, Water Resources Research 24 (8) (1988) 1317–1322.
  * [100] A. Rinaldo, I. Rodriguez-Iturbe, R. Rigon, E. Ijjasz-Vasquez, R. L. Bras, Self-organized fractal river networks, Phys. Rev. Lett. 70 (1993) 822–825.
  * [101] C. Song, S. Havlin, H. A. Makse, Self-similarity of complex networks, Nature 433 (7024) (2005) 392–395.
  * [102] T. Wen, K. H. Cheong, The fractal dimension of complex networks: A review, Information Fusion 73 (2021) 87–102.
  * [103] L. Gou, B. Wei, R. Sadiq, Y. Sadiq, Y. Deng, Topological vulnerability evaluation model based on fractal dimension of complex networks, Plos one 11 (1) (2016) e0146896.
  * [104] Y. Wu, Z. Chen, K. Yao, X. Zhao, Y. Chen, On the correlation between fractal dimension and robustness of complex networks, Fractals 27 (04) (2019) 1950067\.
  * [105] Z. Li, G. Chen, Global synchronization and asymptotic stability of complex dynamical networks, IEEE Transactions on Circuits and Systems II: Express Briefs 53 (1) (2006) 28–33.  doi:10.1109/TCSII.2005.854315.
  * [106] C. W. Wu, Synchronization in complex networks of nonlinear dynamical systems, World scientific, 2007.
  * [107] A. Arenas, A. Díaz-Guilera, J. Kurths, Y. Moreno, C. Zhou, Synchronization in complex networks, Physics reports 469 (3) (2008) 93–153.
  * [108] M. Chen, Chaos synchronization in complex networks, IEEE Transactions on Circuits and Systems I: Regular Papers 55 (5) (2008) 1335–1346.
  * [109] A. Barrat, M. Barthelemy, A. Vespignani, Dynamical processes on complex networks, Cambridge university press, 2008.
  * [110] J. Zhang, M. Small, Complex network from pseudoperiodic time series: Topology versus dynamics, Physical review letters 96 (23) (2006) 238701.
  * [111] E. P. Borges, D. O. Cajueiro, R. F. S. Andrade, Mapping dynamical systems onto complex networks, The European Physical Journal B 58 (2007) 469–474.
  * [112] L. d. F. Costa, F. A. Rodrigues, G. Travieso, P. R. Villas Boas, Characterization of complex networks: A survey of measurements, Advances in physics 56 (1) (2007) 167–242.
  * [113] H. Kantz, T. Schreiber, Nonlinear time series analysis, Vol. 7, Cambridge university press, 2004.
  * [114] P. J. Brockwell, R. A. Davis, Time series: theory and methods, Springer science & business media, 2009.
  * [115] J. D. Hamilton, Time series analysis, Princeton university press, 2020.
  * [116] L. Lacasa, B. Luque, F. Ballesteros, J. Luque, J. C. Nuno, From time series to complex networks: The visibility graph, Proceedings of the National Academy of Sciences 105 (13) (2008) 4972–4975.
  * [117] R. V. Donner, Y. Zou, J. F. Donges, N. Marwan, J. Kurths, Recurrence networks—a novel paradigm for nonlinear time series analysis, New Journal of Physics 12 (3) (2010) 033025.
  * [118] N. Masuda, R. Lambiotte, A guide to temporal networks, World Scientific, 2016.
  * [119] C. M. Bishop, N. M. Nasrabadi, Pattern recognition and machine learning, Springer, 2006.
  * [120] Y. S. Abu-Mostafa, M. Magdon-Ismail, H.-T. Lin, Learning from data, Vol. 4, AMLBook New York, 2012.
  * [121] I. Goodfellow, Y. Bengio, A. Courville, Deep learning, MIT press, 2016.
  * [122] J. Pathak, B. Hunt, M. Girvan, Z. Lu, E. Ott, Model-free prediction of large spatiotemporally chaotic systems from data: A reservoir computing approach, Phys. Rev. Lett. 120 (2018) 024102.  doi:10.1103/PhysRevLett.120.024102.
URL https://link.aps.org/doi/10.1103/PhysRevLett.120.024102

  * [123] B. Penkovsky, X. Porte, M. Jacquot, L. Larger, D. Brunner, Coupled nonlinear delay systems as deep convolutional neural networks, Phys. Rev. Lett. 123 (2019) 054101.  doi:10.1103/PhysRevLett.123.054101.
URL https://link.aps.org/doi/10.1103/PhysRevLett.123.054101

  * [124] P. Amil, M. C. Soriano, C. Masoller, Machine learning algorithms for predicting the amplitude of chaotic laser pulses, Chaos: An Interdisciplinary Journal of Nonlinear Science 29 (11) (2019) 113111.
  * [125] K. Doya, Bifurcations of recurrent neural networks in gradient descent learning, IEEE Transactions on neural networks 1 (75) (1993) 218.
  * [126] Y. Bengio, P. Frasconi, P. Simard, The problem of learning long-term dependencies in recurrent networks, in: IEEE International Conference on Neural Networks, 1993, pp. 1183–1188 vol.3.  doi:10.1109/ICNN.1993.298725.
  * [127] R. Pascanu, T. Mikolov, Y. Bengio, On the difficulty of training recurrent neural networks, in: S. Dasgupta, D. McAllester (Eds.), Proceedings of the 30th International Conference on Machine Learning, Vol. 28 of Proceedings of Machine Learning Research, PMLR, Atlanta, Georgia, USA, 2013, pp. 1310–1318.
  * [128] S. Fortunato, Community detection in graphs, Physics Reports 486 (3) (2010) 75–174.
  * [129] D. O. Cajueiro, Optimal navigation in complex networks, Physical Review E 79 (4) (2009) 046103.
  * [130] D. O. Cajueiro, Optimal navigation for characterizing the role of the nodes in complex networks, Physica A: Statistical Mechanics and its Applications 389 (9) (2010) 1945–1954.
  * [131] S. Kong, L. He, G. Zhang, L. Tao, Z. Zhang, Identifying multiple influential nodes for complex networks based on multi-agent deep reinforcement learning, in: S. Khanna, J. Cao, Q. Bai, G. Xu (Eds.), PRICAI 2022: Trends in Artificial Intelligence, Springer Nature Switzerland, Cham, 2022, pp. 120–133.
  * [132] A. Asgharian Rezaei, J. Munoz, M. Jalili, H. Khayyam, A machine learning-based approach for vital node identification in complex networks, Expert Systems with Applications 214 (2023) 119086.
  * [133] L. Lü, T. Zhou, Link prediction in complex networks: A survey, Physica A: Statistical Mechanics and its Applications 390 (6) (2011) 1150–1170.
  * [134] A. Grover, J. Leskovec, Node2vec: Scalable feature learning for networks, in: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, Association for Computing Machinery, 2016, p. 855–864.
  * [135] H. Xia, H. Wang, Z. Xuan, Opinion dynamics: A multidisciplinary review and perspective on future research, International Journal of Knowledge and Systems Science (IJKSS) 2 (4) (2011) 72–91.
  * [136] P. J. Brancazio, Physics of basketball, American Journal of Physics 49 (4) (1981) 356–365.
  * [137] R. Cross, The bounce of a ball, American Journal of Physics 67 (3) (1999) 222–227.
  * [138] K. Nagel, M. Paczuski, Emergent traffic jams, Physical Review E 51 (4) (1995) 2909\.
  * [139] D. Helbing, P. Molnár, I. J. Farkas, K. Bolay, Self-organizing pedestrian movement, Environment and Planning B: Planning and Design 28 (3) (2001) 361–383.
  * [140] M. Treiber, A. Kesting, Traffic Flow Dynamics: Data, Models and Simulation, Springer, 2012.
  * [141] G. Caldarelli, A. Chessa, Data Science and Complex Networks: Real Case Studies with Python, Oxford University Press, Oxford, UK, 2016.
  * [142] F. Vega-Redondo, Complex social networks, 1st Edition, Cambridge University Press, Cambridge, UK, 2007.
  * [143] V. Latora, M. Marchiori, Is the boston subway a small-world network?, Physica A: Statistical Mechanics and its Applications 314 (1-4) (2002) 109–113.
  * [144] Y. Kuramoto, T. Yamada, Pattern formation in oscillatory chemical reactions, Progress of Theoretical Physics 56 (3) (1976) 724–740.
  * [145] M. C. Cross, P. C. Hohenberg, Pattern formation outside of equilibrium, Reviews of Modern Physics 65 (3) (1993) 851.
  * [146] E. Bonabeau, G. Théraulaz, Swarm smarts, Scientific American 282 (3) (2000) 72–79.
  * [147] G. Spezzano, Swarm Robotics, Springer, 2012.
  * [148] C. R. Harris, K. J. Millman, S. J. van der Walt, R. Gommers, P. Virtanen, D. Cournapeau, E. Wieser, J. Taylor, S. Berg, N. J. Smith, R. Kern, M. Picus, S. Hoyer, M. H. van Kerkwijk, M. Brett, A. Haldane, J. F. del Río, M. Wiebe, P. Peterson, P. Gérard-Marchant, K. Sheppard, T. Reddy, W. Weckesser, H. Abbasi, C. Gohlke, T. E. Oliphant, Array programming with NumPy, Nature 585 (7825) (2020) 357–362.
  * [149] P. Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski, P. Peterson, W. Weckesser, J. Bright, S. J. van der Walt, M. Brett, J. Wilson, K. J. Millman, N. Mayorov, A. R. J. Nelson, E. Jones, R. Kern, E. Larson, C. J. Carey, İ. Polat, Y. Feng, E. W. Moore, J. VanderPlas, D. Laxalde, J. Perktold, R. Cimrman, I. Henriksen, E. A. Quintero, C. R. Harris, A. M. Archibald, A. H. Ribeiro, F. Pedregosa, P. van Mulbregt, SciPy 1.0 Contributors, SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python, Nature Methods 17 (2020) 261–272.  doi:10.1038/s41592-019-0686-2.
  * [150] A. Hagberg, P. Swart, D. S Chult, Exploring network structure, dynamics, and function using networkx, Tech. rep., Los Alamos National Lab.(LANL), Los Alamos, NM (United States) (2008).
  * [151] J. D. Hunter, Matplotlib: A 2d graphics environment, Computing in Science & Engineering 9 (3) (2007) 90–95.  doi:10.1109/MCSE.2007.55.
  * [152] Wes McKinney, Data Structures for Statistical Computing in Python, in: Stéfan van der Walt, Jarrod Millman (Eds.), Proceedings of the 9th Python in Science Conference, 2010, pp. 56 – 61\.  doi:10.25080/Majora-92bf1922-00a.
  * [153] A. Meurer, C. P. Smith, M. Paprocki, O. Čertík, S. B. Kirpichev, M. Rocklin, A. Kumar, S. Ivanov, J. K. Moore, S. Singh, T. Rathnayake, S. Vig, B. E. Granger, R. P. Muller, F. Bonazzi, H. Gupta, S. Vats, F. Johansson, F. Pedregosa, M. J. Curry, A. R. Terrel, v. Roučka, A. Saboo, I. Fernando, S. Kulal, R. Cimrman, A. Scopatz, Sympy: symbolic computing in python, PeerJ Computer Science 3 (2017) e103.
  * [154] H. Sayama, PyCX: a Python-based simulation code repository for complex systems education, Complex Adaptive Systems Modeling 1 (1) (2013) 1–10.
  * [155] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, E. Duchesnay, Scikit-learn: Machine learning in Python, Journal of Machine Learning Research 12 (2011) 2825–2830.
  * [156] J. Kazil, D. Masad, A. Crooks, Utilizing python for agent-based modeling: The mesa framework, in: R. Thomson, H. Bisgin, C. Dancy, A. Hyder, M. Hussain (Eds.), Social, Cultural, and Behavioral Modeling, Springer International Publishing, Cham, 2020, pp. 308–317.
  * [157] H. Stanley, V. Afanasyev, L. Amaral, S. Buldyrev, A. Goldberger, S. Havlin, H. Leschhorn, P. Maass, R. Mantegna, C.-K. Peng, P. Prince, M. Salinger, M. Stanley, G. Viswanathan, Anomalous fluctuations in the dynamics of complex systems: from dna and physiology to econophysics, Physica A: Statistical Mechanics and its Applications 224 (1) (1996) 302–321, dynamics of Complex Systems.
  * [158] R. N. Mantegna, H. E. Stanley, Introduction to econophysics: correlations and complexity in finance, Cambridge University Press, 1999.
  * [159] M. Gligor, M. Ignat, Econophysics: a new field for statistical physics?, Interdisciplinary Science Reviews 26 (3) (2001) 183–190.
  * [160] C. Schinckus, Introduction to econophysics: towards a new step in the evolution of physical sciences, Contemporary Physics 54 (1) (2013) 17–32.
  * [161] F. Slanina, Essentials of Econophysics Modelling, Oxford University Press, 2014\.
  * [162] W. Newell, J. Wentworth, D. Sebberson, A theory of interdisciplinary studies, Issues in integrative studies 19 (2001) 1–25.
  * [163] H. H, M. A (Eds.), Interdisciplinary approaches to nonlinear complex systems, Springer, 2012.
  * [164] R. Kutner, M. Ausloos, D. Grech, T. Di Matteo, C. Schinckus, H. E. Stanley, Econophysics and sociophysics: Their milestones & challenges (2019).
  * [165] R. Mantegna, Hierarchical structure in financial markets., European Physical Journal B 11 (1999) 193–197.
  * [166] M. Boss, H. Elsinger, M. Summer, S. T. 4, Network topology of the interbank market, Quantitative Finance 4 (6) (2004) 677–684.
  * [167] B. M. Tabak, M. Takami, J. M. Rocha, D. O. Cajueiro, S. R. Souza, Directed clustering coefficient as a measure of systemic risk in complex banking networks, Physica A: Statistical Mechanics and its Applications 394 (2014) 211–216.
  * [168] N. Basalto, R. Bellotti, F. De Carlo, P. Facchi, E. Pantaleo, S. Pascazio, Hausdorff clustering of financial time series, Physica A: Statistical Mechanics and its Applications 379 (2) (2007) 635–644.
  * [169] D. O. Cajueiro, S. B. Bastos, C. C. Pereira, R. F. Andrade, A model of indirect contagion based on a news similarity network, Journal of Complex Networks 9 (5) (2021) cnab035.
  * [170] D. D. Gatti, G. Fagiolo, M. Gallegati, M. Richiardi, A. Russo, Agent-Based Models in Economics: A Toolkit, Cambridge University Press, Cambridge, 2018.
  * [171] M. Steinbacher, M. Raddant, F. Karimi, E. Camacho Cuena, S. Alfarano, G. Iori, T. Lux, Advances in the agent-based modeling of economic and social behavior, SN Business & Economics 1 (99) (2021) 1–24.
  * [172] D. A. HSIEH, Chaos and nonlinear dynamics: Application to financial markets, The Journal of Finance 46 (5) (1991) 1839–1877.
  * [173] D. Sornette, Why Stock Markets Crash, Princeton University Press, Princeton, NJ, 2017.
  * [174] C. A. Hidalgo, R. Hausmann, The building blocks of economic complexity, Proceedings of the National Academy of Sciences 106 (26) (2009) 10570–10575.
  * [175] C. A. Hidalgo, Economic complexity theory and applications, Nature Reviews Physics 3 (1) (2021) 92–113.
  * [176] R. L. Costa, G. Vasconcelos, Long-range correlations and nonstationarity in the brazilian stock market, Physica A: Statistical Mechanics and its Applications 329 (1-2) (2003) 231–248.
  * [177] D. O. Cajueiro, B. M. Tabak, The hurst exponent over time: testing the assertion that emerging markets are becoming more efficient, Physica A: statistical mechanics and its applications 336 (3-4) (2004) 521–537.
  * [178] R. Cont, Long range dependence in financial markets, in: Fractals in engineering: New trends in theory and applications, Springer, 2005, pp. 159–179.
  * [179] R. Morales, T. Di Matteo, R. Gramatica, T. Aste, Dynamical generalized hurst exponent as a tool to monitor unstable periods in financial time series, Physica A: statistical mechanics and its applications 391 (11) (2012) 3180–3189.
  * [180] S. Salcedo-Sanz, D. Casillas-Pérez, J. Del Ser, C. Casanova-Mateo, L. Cuadra, M. Piles, G. Camps-Valls, Persistence in complex systems, Physics Reports 957 (2022) 1–73.
  * [181] H. E. Hurst, Long-term storage capacity of reservoirs, Transactions of the American society of civil engineers 116 (1) (1951) 770–799.
  * [182] J. Moreira, J. K. L. da Silva, S. O. Kamphorst, On the fractal dimension of self-affine profiles, Journal of Physics A: Mathematical and General 27 (24) (1994) 8079.
  * [183] C.-K. Peng, S. V. Buldyrev, S. Havlin, M. Simons, H. E. Stanley, A. L. Goldberger, Mosaic organization of dna nucleotides, Physical review e 49 (2) (1994) 1685.
  * [184] P. Windrum, G. Fagiolo, A. Moneta, Empirical validation of agent-based models: Alternatives and prospects, Journal of Artificial Societies and Social Simulation 10 (2) (2007) 8.
  * [185] R. Franke, F. Westerhoff, Structural stochastic volatility in asset pricing dynamics: Estimation and model contest, Journal of Economic Dynamics and Control 36 (8) (2012) 1193–1211.
  * [186] A. F. Peralta, J. Kertész, G. Iñiguez, Opinion dynamics in social networks: From models to data, arXiv preprint arXiv:2201.01322 (2022).
  * [187] R. V. Barroso, J. I. A. V. Lima, A. H. Lucchetti, D. O. Cajueiro, Interbank network and regulation policies: an analysis through agent-based simulations with adaptive learning, Journal of Network Theory in Finance 2 (2016) 53–86.
  * [188] L. F. Adão, D. Silveira, R. A. Ely, D. O. Cajueiro, The impacts of interest rates on banks’ loan portfolio risk-taking, Journal of Economic Dynamics and Control 144 (2022) 104521.
  * [189] J. A. Scheinkman, M. Woodford, Self-organized criticality and economic fluctuations, The American Economic Review 84 (2) (1994) 417–421.
  * [190] X. Gabaix, The granular origins of aggregate fluctuations, Econometrica 79 (3) (2011) 733–772.
  * [191] X. Gabaix, J.-M. Lasry, P.-L. Lions, B. Moll, The dynamics of inequality, Econometrica 84 (6) (2016) 2071–2111.
  * [192] D. F. Batten, Complex landscapes of spatial interaction, The Annals of Regional Science 35 (2001) 81–111.
  * [193] M. Batty, The size, scale, and shape of cities, Science 319 (5864) (2008) 769–771.
  * [194] G. De Roo, E. A. Silva, A planner’s encounter with complexity, Ashgate Publishing, Ltd., 2010.
  * [195] M. Barthelemy, The statistical physics of cities, Nature Reviews Physics 1 (6) (2019) 406–415.
  * [196] P. A. Longley, M. Batty, Spatial analysis: modelling in a GIS environment, John Wiley & Sons, 1997.
  * [197] T. C. Bailey, A. C. Gatrell, Interactive spatial data analysis, Longman Scientific & Technical Essex, 1995.
  * [198] R. White, G. Engelen, I. Uljee, Modeling cities and regions as complex systems: From theory to planning applications, MIT Press, 2015.
  * [199] D. Rind, Complexity and climate, Science 284 (5411) (1999) 105–107.
  * [200] T. W. Crawford, J. P. Messina, S. M. Manson, D. O’Sullivan, Complexity science, complex systems, and land-use research, Environment and Planning B: Planning and Design 32 (6) (2005) 792–798.
  * [201] R. R. Rindfuss, B. Entwisle, S. J. Walsh, L. An, N. Badenoch, D. G. Brown, P. Deadman, T. P. Evans, J. Fox, J. Geoghegan, M. Gutmann, M. Kelly, M. Linderman, J. Liu, G. P. Malanson, C. F. Mena, J. P. Messina, E. F. Moran, D. C. Parker, W. Parton, P. Prasartkul, D. T. Robinson, Y. Sawangdee, L. K. Vanwey, P. H. Verburg, Land use change: complexity and comparisons, Journal of Land Use Science 3 (1) (2008) 1–10.
  * [202] K. Rus, V. Kilar, D. Koren, Resilience assessment of complex urban systems to natural disasters: A new literature review, International Journal of Disaster Risk Reduction 31 (2018) 311–330.
  * [203] P.-A. Balland, D. Rigby, The geography of complex knowledge, Economic Geography 93 (1) (2017) 1–23.
  * [204] D. Hartmann, F. L. Pinheiro, Economic complexity and inequality at the national and regional level, arXiv preprint arXiv:2206.00818 (2022).
  * [205] M. Batty, Urban modelling, Cambridge University Press Cambridge, 1976.
  * [206] H. A. Makse, J. S. Andrade, M. Batty, S. Havlin, H. E. Stanley, et al., Modeling urban growth patterns with correlated percolation, Physical Review E 58 (6) (1998) 7054.
  * [207] M. Batty, Y. Xie, Z. Sun, Modeling urban dynamics through gis-based cellular automata, Computers, Environment and Urban Systems 23 (3) (1999) 205–233.
  * [208] E. A. Oliveira, J. S. Andrade Jr, H. A. Makse, Large cities are less green, Scientific reports 4 (1) (2014) 4235.
  * [209] Z. Dobesova, Experiment in finding look-alike european cities using urban atlas data, ISPRS International Journal of Geo-Information 9 (6) (2020) 406.
  * [210] A. Lavallin, J. A. Downs, Machine learning in geography–past, present, and future, Geography Compass 15 (5) (2021) e12563.
  * [211] D. O’Sullivan, S. M. Manson, Do physicists have geography envy? and what can geographers learn from it?, Annals of the Association of American Geographers 105 (4) (2015) 704–722.
  * [212] U. Bilal, C. P. de Castro, T. Alfaro, T. Barrientos-Gutierrez, M. L. Barreto, C. M. Leveau, K. Martinez-Folgar, J. J. Miranda, F. Montes, P. Mullachery, et al., Scaling of mortality in 742 metropolitan areas of the americas, Science advances 7 (50) (2021) eabl6325.
  * [213] C. P. de Castro, G. F. Dos Santos, A. D. de Freitas, M. I. Dos Santos, R. F. S. Andrade, M. L. Barreto, Socio-economic urban scaling properties: Influence of regional geographic heterogeneities in brazil, PLoS one 15 (12) (2020) e0242778.
  * [214] D. Pumain, Geography, physics and synergetics (1992).
  * [215] D. Nüst, E. Pebesma, Practical reproducibility in geography and geosciences, Annals of the American Association of Geographers 111 (5) (2021) 1300–1310.
  * [216] P. Kedron, A. E. Frazier, A. B. Trgovac, T. Nelson, A. S. Fotheringham, Reproducibility and replicability in geographical analysis, Geographical Analysis 53 (1) (2021) 135–147.
  * [217] G. PARISI, Physics complexity and biology, Advances in Complex Systems 10 (supp02) (2007) 223–232.
  * [218] A. Ma’ayan, Complex systems biology, Journal of the Royal Society Interface 14 (134) (2017) 20170391.
  * [219] E. Schrödinger, R. Penrose, What is Life?: With Mind and Matter and Autobiographical Sketches, Canto, Cambridge University Press, 1992.  doi:10.1017/CBO9781139644129.
  * [220] S. Kauffman, Answering schrödinger’s “what is life?”, Entropy 22 (8) (2020).
  * [221] Z. N. Oltvai, A.-L. Barabási, Life’s complexity pyramid, Science 298 (5594) (2002) 763–764.
  * [222] A. F. Siegenfeld, Y. Bar-Yam, An introduction to complex systems science and its applications, Complexity 2020 (2020) 6105872.
  * [223] P. Medawar, An Unsolved Problem of Biology: An Inaugural Lecture Delivered at University College, London, 6 December, 1951, Inaugural lecture, H.K. Lewis and Company, 1952.
  * [224] T. J. P. Penna, A bit-string model for biological aging, Journal of Statistical Physics 78 (5) (1995) 1629–1633.
  * [225] S. M. De Oliveira, J. S. Martins, K. Luz-Burgoa, A. Ticona, T. Penna, The penna model for biological aging and speciation, Computing in Science & Engineering 6 (3) (2004) 74–81.
  * [226] E. Ravasz, A. L. Somera, D. A. Mongru, Z. N. Oltvai, A.-L. Barabási, Hierarchical organization of modularity in metabolic networks, Science 297 (5586) (2002) 1551–1555.
  * [227] A.-L. Barabási, Z. N. Oltvai, Network biology: understanding the cell’s functional organization, Nature Reviews Genetics 5 (2) (2004) 101–113.
  * [228] F. R. Pinu, D. J. Beale, A. M. Paten, K. Kouremenos, S. Swarup, H. J. Schirra, D. Wishart, Systems biology and multi-omics integration: viewpoints from the metabolomics research community, Metabolites 9 (4) (2019) 76.
  * [229] G. de Anda-Jáuregui, E. Hernández-Lemus, Computational oncology in the multi-omics era: state of the art, Frontiers in oncology 10 (2020) 423.
  * [230] D. G. Spiller, C. D. Wood, D. A. Rand, M. R. White, Measurement of single-cell dynamics, Nature 465 (7299) (2010) 736–745.
  * [231] D. M. Camacho, K. M. Collins, R. K. Powers, J. C. Costello, J. J. Collins, Next-generation machine learning for biological networks, Cell 173 (7) (2018) 1581–1592.
  * [232] C. Xu, S. A. Jackson, Machine learning and complex biological data (2019).
  * [233] A.-L. Barabási, Network medicine—from obesity to the “diseasome” (2007).
  * [234] A.-L. Barabási, N. Gulbahce, J. Loscalzo, Network medicine: a network-based approach to human disease, Nature reviews genetics 12 (1) (2011) 56–68.
  * [235] M. Heinemann, S. Panke, Synthetic biology—putting engineering into biology, Bioinformatics 22 (22) (2006) 2790–2799.
  * [236] J. Pleiss, The promise of synthetic biology, Applied microbiology and biotechnology 73 (2006) 735–739.
  * [237] P. Jeffries, N. Johnson, P. Ming Hui, Financial Market Complexity: What Physics Can Tell Us About Market Behaviour, Oxford University Press, 2003.
  * [238] D. Challet, M. Marsili, Y.-C. Zhang, Minority games: interacting agents in financial markets, OUP Oxford, 2004.
  * [239] R. Kitamura, S. Nakayama, Can travel time information influence network flow? implications of the minority game, Transportation research record 2010 (1) (2007) 12–18.
  * [240] P. Bouman, L. Kroon, P. Vervest, G. Maróti, Capacity, information and minority games in public transport, Transportation Research Part C: Emerging Technologies 70 (2016) 157–170.
  * [241] B. A. Mello, D. O. Cajueiro, Minority games, diversity, cooperativity and the concept of intelligence, Physica A: Statistical Mechanics and its Applications 387 (2-3) (2008) 557–566.
  * [242] B. C. Lustosa, D. O. Cajueiro, Constrained information minority game: How was the night at el farol?, Physica A: Statistical Mechanics and its Applications 389 (6) (2010) 1230–1238.
  * [243] N. Capodieci, G. A. Pagani, G. Cabri, M. Aiello, An adaptive agent-based system for deregulated smart grids, Service Oriented Computing and Applications 10 (2016) 185–205.
  * [244] D. Challet, Y.-C. Zhang, Emergence of cooperation and organization in an evolutionary game, Physica A: Statistical Mechanics and its Applications 246 (3) (1997) 407–418.
  * [245] R. Savit, R. Manuca, R. Riolo, Adaptive competition, market efficiency, and phase transitions, Phys. Rev. Lett. 82 (1999) 2203–2206.
  * [246] W. B. Arthur, Inductive reasoning and bounded rationality, The American economic review 84 (2) (1994) 406–411.
  * [247] A. C. Coolen, The mathematical theory of minority games: statistical mechanics of interacting agents, OUP Oxford, 2005.
  * [248] M. Ortisi, Limiting Behavior of Interacting Particle Systems: with application to Minority Game, LAP LAMBERT Academic Publishing, 2011.
  * [249] I. Lorscheid, B.-O. Heine, M. Meyer, Opening the ‘black box’of simulations: increased transparency and effective communication through the systematic design of experiments, Computational and Mathematical Organization Theory 18 (2012) 22–62.
  * [250] F. M. Klingert, M. Meyer, Effectively combining experimental economics and multi-agent simulation: suggestions for a procedural integration with an example from prediction markets research, Computational and Mathematical Organization Theory 18 (2012) 63–90.
  * [251] G. Bianconi, A. Arenas, J. Biamonte, L. D. Carr, B. Kahng, J. Kertesz, J. Kurths, L. Lu, C. Masoller, A. E. Motter, M. Perc, F. Radicchi, R. Ramaswamy, F. A. Rodrigues, M. Sales-Pardo, M. S. Miguel, S. Thurner, T. Yasseri, Complex systems in the spotlight: next steps after the 2021 nobel prize in physics, Journal of Physics: Complexity 4 (1) (2023) 010201.
  * [252] A. Cook, H. A. Blom, F. Lillo, R. N. Mantegna, S. Miccichè, D. Rivas, R. Vázquez, M. Zanin, Applying complexity science to air traffic management, Journal of Air Transport Management 42 (2015) 149–158.
  * [253] A. Cook, D. Rivas (Eds.), Complexity Science in Air Traffic Management, Routledge, 2016.
  * [254] P. Nowack, J. Runge, V. Eyring, et al., Causal networks for climate model evaluation and constrained projections, Nature Communications 11 (1) (2020) 1415\.  doi:10.1038/s41467-020-14973-w.
  * [255] S. Galea, M. Riddle, G. A. Kaplan, Causal thinking and complex system approaches in epidemiology, International Journal of Epidemiology 39 (1) (2009) 97–106.
  * [256] R. Pastor-Satorras, C. Castellano, P. Van Mieghem, A. Vespignani, Epidemic processes in complex networks, Rev. Mod. Phys. 87 (2015) 925–979.
  * [257] S. Mei, X. Zhang, M. Cao, Power grid complexity, Springer Science & Business Media, 2011.
  * [258] Y. Yang, T. Nishikawa, A. E. Motter, Small vulnerable sets determine large network cascades in power grids, Science 358 (6365) (2017) eaan3184.
  * [259] M. Anand, A. Gonzalez, F. Guichard, J. Kolasa, L. Parrott, Ecological systems as complex systems: Challenges for an emerging science, Diversity 2 (3) (2010) 395–410.
  * [260] P. G. Lind, L. R. Da Silva, J. S. Andrade Jr, H. J. Herrmann, Spreading gossip in social networks, Physical Review E 76 (3) (2007) 036117.
  * [261] J.-P. Onnela, J. Saramäki, J. Hyvönen, G. Szabó, D. Lazer, K. Kaski, J. Kertész, A.-L. Barabási, Structure and tie strengths in mobile communication networks, Proceedings of the national academy of sciences 104 (18) (2007) 7332–7336.
  * [262] B. Mello, L. Batistuta, R. Boueri, D. Cajueiro, Measuring the flow of information among cities using the diffusion power, Physics Letters A 374 (2) (2009) 126–130.
  * [263] A. Vespignani, Modelling dynamical processes in complex socio-technical systems, Nature physics 8 (1) (2012) 32–39.
  * [264] Z.-K. Zhang, C. Liu, X.-X. Zhan, X. Lu, C.-X. Zhang, Y.-C. Zhang, Dynamics of information diffusion and its applications on complex networks, Physics Reports 651 (2016) 1–34.
  * [265] B. Lengyel, E. Bokányi, R. Di Clemente, J. Kertész, M. C. González, The role of geography in the complex diffusion of innovations, Scientific reports 10 (1) (2020) 15065.
  * [266] J. H. Fowler, Connecting the congress: A study of cosponsorship networks, Political analysis 14 (4) (2006) 456–487.
  * [267] J. H. Fowler, Legislative cosponsorship networks in the us house and senate, Social networks 28 (4) (2006) 454–465.
  * [268] G. Koger, S. Masket, H. Noel, Partisan webs: Information exchange and party networks, British Journal of Political Science 39 (3) (2009) 633–653.
  * [269] J. H. Fowler, M. T. Heaney, D. W. Nickerson, J. F. Padgett, B. Sinclair, Causality in political networks, American politics research 39 (2) (2011) 437–480.
  * [270] P. F. Nery, B. Mueller, Co-sponsorship networks in the brazilian congress: An exploratory analysis of caucus influence, Estudos Econômicos (São Paulo) 52 (2022) 83–111.
  * [271] S. Aral, L. Muchnik, A. Sundararajan, Distinguishing influence-based contagion from homophily-driven diffusion in dynamic networks, Proceedings of the National Academy of Sciences 106 (51) (2009) 21544–21549.
  * [272] S. Aral, D. Walker, Identifying influential and susceptible members of social networks, Science 337 (6092) (2012) 337–341.
  * [273] F. E. Lennon, G. C. Cianci, N. A. Cipriani, T. A. Hensing, H. J. Zhang, C.-T. Chen, S. D. Murgu, E. E. Vokes, M. W. Vannier, R. Salgia, Lung cancer—a fractal viewpoint, Nature reviews Clinical oncology 12 (11) (2015) 664–675.
  * [274] A. N. Anderson, A. B. McBratney, J. W. Crawford, Applications of fractals to soil studies, Advances in Agronomy 63 (1997) 1–76.
  * [275] F. Almeida, C. Calistru, The main challenges and issues of big data management, International Journal of Research Studies in Computing 2 (1) (2013) 11–20.
  * [276] C. Philip Chen, C.-Y. Zhang, Data-intensive applications, challenges, techniques and technologies: A survey on big data, Information Sciences 275 (2014) 314–347.
  * [277] S. Yin, O. Kaynak, Big data for modern industry: challenges and trends [point of view], Proceedings of the IEEE 103 (2) (2015) 143–146.
  * [278] C. Allen, W. Wallach, I. Smit, Why machine ethics?, IEEE Intelligent Systems 21 (4) (2006) 12–17.
  * [279] J. Moor, The nature, importance, and difficulty of machine ethics, IEEE Intelligent Systems 21 (4) (2006) 18–21.  doi:10.1109/MIS.2006.80.
  * [280] J. Kleinberg, J. Ludwig, S. Mullainathan, Z. Obermeyer, Prediction policy problems, American Economic Review 105 (5) (2015) 491–495.