# A Large-Scale Characterization of
How Readers Browse Wikipedia

Tiziano Piccardi  EPFL<EMAIL_ADDRESS>,  Martin Gerlach  Wikimedia
Foundation<EMAIL_ADDRESS>,  Akhil Arora  EPFL<EMAIL_ADDRESS>and
Robert West  EPFL<EMAIL_ADDRESS>
(2018)

###### Abstract.

Despite the importance and pervasiveness of Wikipedia as one of the largest
platforms for open knowledge, surprisingly little is known about how people
navigate its content when seeking information. To bridge this gap, we present
the first systematic large-scale analysis of how readers browse Wikipedia.
Using billions of page requests from Wikipedia’s server logs, we measure how
readers reach articles, how they transition between articles, and how these
patterns combine into more complex navigation paths. We find that navigation
behavior is characterized by highly diverse structures. Although most
navigation paths are shallow, comprising a single pageload, there is much
variety, and the depth and shape of paths vary systematically with topic,
device type, and time of day. We show that Wikipedia navigation paths commonly
mesh with external pages as part of a larger online ecosystem, and we describe
how naturally occurring navigation paths are distinct from targeted navigation
in lab-based settings. Our results further suggest that navigation is
abandoned when readers reach low-quality pages. These findings not only help
in identifying potential improvements to reader experience on Wikipedia, but
also in better understanding of how people seek knowledge on the Web.

††copyright: acmcopyright††journalyear: 2018††doi:
10.1145/1122445.1122456††conference: ; ;

##  1\. Introduction

Homo sapiens is a born knowledge seeker. Many of the most consequential
evolutionary, cultural, and technological advances in humans have related to
enhancing their ability to find, ingest, process, and transfer knowledge, from
the evolution of bipedalism, large brains, and language to the invention of
writing, telecommunication, and the Web. Given the central importance of
knowledge seeking to human nature—captured by the very name Homo sapiens and
epitomized by the view of humans as informavores (Machlup, 1983)—,
understanding how humans seek information is of key significance across
disciplines, in the basic as well as the applied sciences. In the basic
sciences, biologists, psychologists, anthropologists, among others, stand to
gain fundamental insights into how humans function, whereas in the applied
sciences, such insights can enable the design of more effective tools and
information environments, such that humans can more readily find relevant
knowledge in an ever-surging flood of information.

Closely observing humans as they seek knowledge is, however, challenging,
since it requires measuring predominantly mental behaviors at a great level of
detail. As a consequence, although much work has been dedicated to shedding
light on human knowledge seeking behavior (see Sec. 2), it has faced important
limitations: Surveys (Younger, 2010) and thinking-out-loud studies (Muramatsu
and Pratt, 2001) are prone to cognitive biases, as humans generally perform
poorly at introspection (Nisbett and Wilson, 1977). Lab-based experiments
(Lydon-Staley et al., 2021) typically involve small samples consisting of
biased populations (e.g., university students) and are thus frequently not
representative and might lack statistical power. Studies based on surrogate
tasks (e.g., navigation games (West et al., 2009)), although measuring
navigation-related skills, do not capture real-world, self-motivated knowledge
seeking and may thus lack external validity (Olteanu et al., 2019). Finally,
studies based on aggregated versions of real-world knowledge seeking traces
(specifying page-to-page transition counts instead of full traces (Dimitrov et
al., 2017; Gildersleve and Yasseri, 2018)), although capturing local, page-
level choices accurately, may lack relevant trace-level information (e.g.,
relating the start of a trace to its end).

We overcome these limitations in the context of encyclopedic knowledge
seeking—an important special case of human knowledge seeking—by leveraging a
large-scale dataset of digital traces compiled from one month’s worth of
English Wikipedia’s complete server logs, which offer unprecedented
opportunities for observing humans interacting with knowledge in great detail.
Wikipedia is the largest encyclopedia ever built, is freely accessible across
the globe—attracting billions of monthly views from an audience whose
representativeness far surpasses that of lab-based studies—, fulfills a wide
range of information needs (Singer et al., 2017; Lemmerich et al., 2019), and
records all pageloads in its server logs. The logs are thus uniquely suited
for providing a geographically and temporally complete mirror of real-world,
self-motivated encyclopedic knowledge seeking.

In contrast to prior work, which has leveraged Wikipedia’s server logs to shed
light on specific aspects of reader behavior (including reasons for visiting
Wikipedia (Singer et al., 2017; Lemmerich et al., 2019), engagement with
citations and external links (Piccardi et al., 2020, 2021; Maggio et al.,
2020), studying variation in dwell time (TeBlunthuis et al., 2019), and
measuring geo-localized collective behavior (Tizzoni et al., 2020)), the
present paper is the first to employ the logs in a principled, broad analysis
with the goal of systematically elucidating the nature and structure of
encyclopedic knowledge seeking pathways. By analyzing billions of navigation
traces extracted from the logs (Sec. 3), we span three levels of aggregation
in our research questions:

  1. (1)

Unigram level: How do readers reach Wikipedia articles? (Sec. 4)

  2. (2)

Bigram and trigram level: How do readers transition from one article to the
next? (Sec. 5)

  3. (3)

Session level: What is the structure of entire reading sessions? (Sec. 6)

We find that Wikipedia navigation traces expose a wide variety of structures;
although shallow sessions consisting of single pageloads dominate, we observe
a long tail of long, complex traces, whose depth and shape vary systematically
with topic, device type, and time of day. We highlight that Wikipedia
navigation does not happen in isolation, but is embedded in sessions where
users transition fluidly to and from the external Web. This aspect, as well as
other differences that emerge, distinguishes real-world, in-the-wild Wikipedia
usage from the targeted navigation behavior captured by lab-based studies.
Finally, we find strong evidence that users stop navigating when reaching low-
quality articles.

These results have important implications for Wikipedia and beyond.
Understanding how readers explore content on Wikipedia is critical for framing
its role in fulfilling information needs and for making design decisions
regarding its structure, format, accessibility, and supportive tools such as
recommender systems. Going beyond Wikipedia, these findings deepen our
understanding of how humans navigate information when seeking knowledge.

##  2\. Related work

Navigation on the Web. Characterizing the user navigation on the Web is a
challenging task because of the limited availability of data. Previous work
focused on modeling navigation patterns based on server logs of large websites
or by using modified browser versions. A common finding is that people
frequently revisit the same content multiple times (Tauscher and Greenberg,
1997; Anderson et al., 2014). This repeat consumption behavior, which is
abandoned when the person gets bored of the content (Benson et al., 2016),
makes human mobility on the Web predictable (Kulshrestha et al., 2020).
Although researchers found that Web users are not strictly Markovian (the page
visited next does not depend exclusively on the current state) (Chierichetti
et al., 2012), many prediction models approximate the navigation of users on a
network with Markov chains (Pirolli and Pitkow, 1999; Mabroukeh and Ezeife,
2009; Deshpande and Karypis, 2004) and hybrid models (Narvekar and Banu, 2015;
Khalil et al., 2009; Jindal et al., 2020; Awad and Khan, 2007).

In terms of applications, navigation traces have proven useful as a tool to
improve the website navigability by identifying missing links (Paranjape et
al., 2016; Lamprecht et al., 2016; West et al., 2015) and other usability
issues that normally require the work of domain experts (Geng and Tian, 2015).
Similarly, navigation logs can be used to compute semantic relatedness of
pages by studying what content is typically accessed together (Singer et al.,
2013; Dallmann et al., 2016).

Reader behavior on Wikipedia. Researchers have studied how readers behave when
reading Wikipedia. Recent work focuses on the interaction with external links
(Piccardi et al., 2021) and references (Piccardi et al., 2020; Maggio et al.,
2020), and on the reading time of articles (TeBlunthuis et al., 2019).
Researchers have concluded that Wikipedia users have reading patterns that
fall in different categories, such as exploration, focus, trending, and
passing (Lehmann et al., 2014), and that readers prefer links that lead to the
periphery of the network, about semantically similar content and located at
the top of the article (Lamprecht et al., 2017; Dimitrov et al., 2017). Other
studies have investigated the inter-event time in the navigation logs of
Wikipedia and found strong regularities in the temporal rhythms, which suggest
a good rule-of-thumb for segmenting sessions after inactivity periods of one
hour (Halfaker et al., 2015).

These studies are complemented by investigations of the motivations for
visiting Wikipedia (Lemmerich et al., 2019; Singer et al., 2017), which
describe a variety of factors such as current events, media coverage of a
topic, personal curiosity, work or school assignments, or boredom.

Closest in spirit to the present work, multiple approaches have been used to
study human navigation on Wikipedia. The public clickstream (Wulczyn and
Taraborelli, 2015) contains transition counts for pairs of articles. Although
the clickstream constitutes an aggregated and filtered version of the server
logs, it has been used to study how different topics relay more traffic than
others (Dimitrov et al., 2018; Gildersleve and Yasseri, 2018), and how
readers’ navigation paths tend to start general and become incrementally more
focused at every step (Rodi et al., 2017).

Other approaches to understanding readers’ navigation have identified
different types of curiosity during Wikipedia exploration by relying on data
shared by volunteers (Lydon-Staley et al., 2021), while yet others have
characterized human navigation as manifested in digital traces obtained via
Wikipedia navigation games such as Wikispeedia (West et al., 2009), where
players start from a random article and are tasked to reach a target page in
as few clicks as possible by following links only. These trajectories, denoted
as targeted navigation here, show how efficient people are at finding short
paths (West and Leskovec, 2012b; Helic, 2012; West and Leskovec, 2012a). In
contrast to natural navigation, targeted navigation posits an unambiguous
definition of success (i.e., reaching the target article), which allows
researchers to study how people drift away from the best path and when they
abandon the search (Koopmann et al., 2019; Scaria et al., 2014).

##  3\. Materials and methods

The data sources exploited in this study include user traces mined from
Wikipedia’s server logs and features extracted from articles.

###  3.1. Pageloads

To study how readers navigate Wikipedia, we analyze the server logs of the
English edition collected for four weeks between 1 and 28 March 2021. This
data contains an entry for each time a Wikipedia page is loaded. It is
automatically collected for analytic purposes on Wikimedia’s infrastructure
and deleted after 90 days.

We limit our analysis to the pageload requests for articles (MediaWiki
namespace 0), filtering out requests from bots. To protect readers’ privacy,
we remove sensitive information in several steps: discarding pageloads from
readers who edited or were logged in during the time of data collection;
discarding all requests from countries with at least one day with fewer than
300 pageloads; generating (pseudo) user identifiers by hashing IPs and user
agent strings, as done in previous work (Paranjape et al., 2016); and dropping
IP, user agent, and fine-grained geo information. In total, these
anonymization steps lead to the removal of around 3% of the data. In addition,
we perform the following filtering steps. First, we drop pageloads of the
Main_Page article, as it does not represent any specific entity. These
requests may come from users who set Wikipedia as the browser’s default page.
Second, we remove traffic from massively shared IPs, which would make it hard
to study individual activities, by dropping all user identifiers with more
than 2,800 pageloads, or on average 100 per day, removing 28k (0.0019%) user
identifiers. The final dataset contains 6.52B pageloads associated with 1.47B
user identifiers.

###  3.2. Article features

To characterize the content viewed by the readers, we collect a set of article
features. To ensure alignment between the server logs and the articles’
content, we compute the features for the revisions of the public snapshot
released at the end of March 2021. We obtain article features such as the
number of outgoing links, the PageRank, and article quality score (Halfaker,
2017), and topic. For the latter, we use two approaches: (1) ORES (Halfaker
and Geiger, 2019) probabilities for 64 manually curated topics, used for
assigning topical labels to articles; (2) WikiPDA (Piccardi and West, 2021)
topic vectors, used for placing articles in a 300-dimensional topic space.

##  4\. Results: unigram level

We use the term “$n$-gram” to designate a sequence of $n$ subsequent pageloads
from the same user. We start our analysis with unigrams ($n=1$) and enumerate
how readers can reach Wikipedia articles. We classify Web traffic according to
HTTP referrers and quantify the frequency of each type (Table 1). In total, 4B
(61.5%) pageloads have external or empty referrers and are thus entry points
to Wikipedia.

Table 1.  Statistics of referrers of single pageloads.  Origin | Desktop | Mobile | Total
---|---|---|---
Search engines | 45.97% | 48.77% | 45.72%
Wikipedia |  |  |
Articles | 35.64% | 35.75% | 35.72%
Main page | 1.65% | 0.70% | 1.06%
Lang. switching | 1.62% | 0.50% | 0.92%
Categories | 0.59% | 0.25% | 0.39%
Search page | 0.38% | 0.22% | 0.29%
Special pages | 0.07% | 0.01% | 0.03%
Portals | 0.03% | 0.01% | 0.02%
Others | 0.01% | 0.07% | 0.03%
Unspecified origin | 12.64% | 13.03% | 12.88%
External websites | 1.36% | 0.70% | 0.95%
Others | 4.36% | 1.73% | 2.73%

Search engines. The most common way to reach the content of Wikipedia is
through external search engines, at 3.1B pageloads (45.9% of all recorded
traffic, or 77.4% of external traffic). This volume reflects the significant
value offered by Wikipedia in fulfilling the information needs of search
engine users (Vincent and Hecht, 2021; Andreescu et al., 2021).

Wikipedia. Clicks from other articles account for 35.7% of all traffic.
Interestingly, as observed in previous work (Mitrevski et al., 2020), 6.6% of
these pageloads happen through links that do not exist in the link network
itself, but likely through other interactions such as Wikipedia’s search drop
down menu. Content can also be reached from other pages on the Wikipedia
platform: (1) the main page, (2) category pages, (3) Wikipedia’s internal
search, (4) portals, (5) other Wikipedia pages, including talk pages or pages
in other languages (language switching).

Unspecified origin. In 12.9% of all traffic, we observe an empty referrer
field. Multiple reasons can produce a request without an explicit origin,
including direct access via the browser history, redirects from apps,
bookmarks, search toolbars, or when the link source has explicitly turned on
the noreferrer property.

External websites. In total, 0.95% of the requests originated from external
websites that are not search engines nor Wikipedia domains. Among those, the
most common sources are Facebook (15.6%), Reddit (9.6%), YouTube (8.0%), and
Twitter (4.3%).

Others. Other external visits come from Android Web views and custom embedded
visualizations, with the most common being the Telegram and Reddit sync apps,
and Facebook on Android devices.

(a) Distrib. of inter-event times

(b) Prob. of internal transition

(c) External transitions with link

(d) Distance between pageloads

Figure 1.  Statistics of bigrams as a function of the inter-event time between
two pageloads. Dashed curves represent the distributions with AA patterns
included.

##  5\. Results: bigram and trigram level

Next, we move from unigrams to bigrams to understand how readers transition
between Wikipedia articles. We study events aggregated by user identifier and
sorted by time to investigate the properties of consecutive pageloads and
their inter-event time. Here it is important to note that the Wikipedia server
instructs the browser to disable the cache, such that the server logs contain
essentially all pageloads events, including cases when the readers reloaded an
article, e.g., by using the back button.

Bigrams. The logs contain 3.95B bigrams (i.e., two subsequent pageloads by the
same user with less than one hour in between (Halfaker et al., 2015)). The
emerging patterns, described next, are summarized in Table 2.

Table 2.  Frequencies of bigram and trigram patterns.  Device | AB | AA | ABC | ABA | ABB | AAB | AAA
---|---|---|---|---|---|---|---
Desktop | 0.900 | 0.099 | 0.749 | 0.121 | 0.047 | 0.049 | 0.031
Mobile | 0.880 | 0.119 | 0.719 | 0.143 | 0.055 | 0.053 | 0.027
Total | 0.888 | 0.111 | 0.732 | 0.134 | 0.052 | 0.052 | 0.029

The most frequent bigram pattern (“AB” in Table 2) corresponds to transitions
between two different articles. It can happen both through internal and
external navigation (cf. Fig. S1). This pattern represents around 89% of all
bigrams. The other possible bigram pattern (“AA” in Table 2), corresponds to
the consecutive reload of the same article. Representing 11% of all bigrams,
it is rather common (84% share the same referrer). This pattern appears at
least once in 37% of the navigation histories of readers with at least two
pageloads in the month of data collection. This pattern in the log can be
generated by different client behaviors (cf. Fig. S1), including repeated
consumption as described in previous work (Tauscher and Greenberg, 1997;
Benson et al., 2016), user activities involving external navigation, or
artificial reloads by the browser when a tab unloaded from memory is restored.

Trigrams. Finally, we also briefly consider the 2.98B trigrams present in the
logs. The most common trigram pattern (73%, “ABC” in Table 2) represents
transitions between three different articles. A variety of behaviors can
generate this pattern, including sequential clicks or multitab behavior (cf.
Fig. S1). The second most common trigram pattern (13%, “ABA” in Table 2) can
be generated by intentionally revisiting the same page or by clicking the back
button (cf. Fig. S1). In 89% of ABA instances, the first and last event also
share the same referrer. The remaining trigram patterns (ABB, AAB, AAA) are
combinations of the bigrams described above.

Dynamics of transitions. In order to understand the dynamics of these
transitions, we investigate the inter-event time between the two pageloads in
each bigram. The interval between two consecutive pageloads peaks at very
short times, with a median of 74 seconds (63 and 93 seconds for mobile and
desktop devices, respectively). However, as Fig. 1 shows, the distribution is
long-tailed, with 22% of pairs separated by more than one hour.

Investigating the referrer of the second page of the bigrams reveals that
readers frequently do not use internal links to transition between two
articles, but external pages by leaving and re-entering Wikipedia. These
external transitions are not rare: in 35.2% (or 40.1% when including AA
patterns) of the bigrams with less than one hour between the two events, the
second page was reached through external navigation. This observation is
corroborated by Fig. 1, which shows that for pairs with an inter-event time
greater than 3 minutes and 48 seconds, transitions via internal links are even
less common than transitions via external navigation. External transitions
tend to be semantically coherent: considering all 1.4B AB-type bigrams where
the second page is reached via search, in 15% of the cases, the first page
explicitly contained the link. This proportion increases to 30% [60–70%] when
considering pairs with an inter-event time of less than one hour [less than 10
seconds] (Fig. 1). The topical coherence of these transitions is also visible
by observing the jump size in topic space. Fig. 1 plots the average topical
distance (measured by the cosine of WikiPDA vectors, cf. Sec. 3.2) as a
function of inter-event time, showing that external navigation recorded within
a few minutes from the previous pageload shows topical distance comparable to
internal navigation.

##  6\. Results: session level

Using our insights about navigation at the unigram, bigram, and trigram
levels, we can now characterize entire navigation sessions. We start by
introducing two different approaches to conceptualize navigation sessions
(Sec. 6.1) and discuss how each captures different aspects of readers’
navigation. We then describe the properties of reader navigation by focusing
on three aspects of the resulting sessions: contextual features defining when
and how sessions start (Sec. 6.2), structural features of sessions (Sec. 6.3),
and finally, the evolution of various article properties over the course of
navigation sessions (Sec. 6.4).

###  6.1. Conceptualizing reader sessions

We introduce two notions of a user session, each capturing different aspects
of navigation pathways (details below): (1) navigation trees connect pageloads
hierarchically based on referrer information, whereas (2) reading sequences
order pageloads linearly based on temporal information. From the original
6.52B pageloads, we obtain 3.7B navigation trees and 2.51B reading sequences.

Navigation trees (Paranjape et al., 2016) describe how readers traverse
Wikipedia by following internal links. We generate a tree by connecting
pageloads via the referrer contained in HTTP headers. Pages reached through
internal transitions are added as children of the most recent load of the
referrer, while pageloads with external or Main_Page referrers generate a new
tree. If a page is loaded multiple times from the same referrer, the parent
node retains only the first instance as a child. This method has the advantage
of representing coherent sessions created through clicks on internal links and
of capturing multitab behavior. The downside is the difficulty to capture
content consumption over time for subsequent pages not reached through
internal clicks, even if close in time (a common pattern, cf. Sec. 5).

Reading sequences describe how readers consume content in temporal order. They
are defined as linear sequences of all pageloads by the same user ordered by
time. Sequences are split if the inter-event time between two consecutive
pageloads separated by external navigation exceeds a threshold value of one
hour, following recommendations from previous studies (Halfaker et al., 2015)
and common practice (Singer et al., 2017; Lemmerich et al., 2019). Within such
sessions, we keep only the first pageload of each article, in order to only
capture the first exposure of the respective content. This method generates
topically less coherent sessions, capturing the temporal and linear sequence
of pageloads of a reader within a defined period of time, both via internal
and external transitions (e.g., multiple external searches). This method has
the disadvantage of being a simplification of how readers explore the link
network, and a fixed threshold of one hour may not be ideal in every context.

(a) Number of trees

(b) Features from regression

Figure 2.  Statistics about time of day of sessions

###  6.2. Session context: time and device

We study the context of a session by focusing on the time of the first
pageload and the device used to access Wikipedia. This section focuses on
navigation trees, but reading sequences give qualitatively similar results
(cf. Fig. S5, Fig. S3).

Time. To remove confounding via different timezones, we use the geolocation
information to normalize the time of all pageloads to local time. The
distribution of session starting times follows a regular circadian rhythm
(Fig. 2 and Fig. S4). Both access methods (desktop and mobile) show a similar
pattern during the day, with a substantial increase of mobile sessions in the
evening. Wikipedia has fewer sessions during weekends, but with similar
temporal distributions as working days. The desktop distribution shows dents
at 12:00 and 18:00, mirroring work rhythms with a lunch break around noon and
the end of work in the evening (and possibly commuting).

In order to understand which features are associated with requests at
different times of the day, we fitted a logistic regression to predict if a
pageload was observed during the day or evening/night. We represent each
pageload by its topic probabilities (obtained from ORES, cf. Sec. 3.2) and the
type of device (desktop or mobile). Binarizing the target variable by
representing daytime (9:00–18:00) as the positive class, we obtain an AUC of
0.586. Inspecting feature importance (Fig. 2) shows that desktop devices and
articles associated with STEM and education are associated with sessions
starting during the day, whereas topics about entertainment are predictors of
sessions starting during the evening or night.

Device. Fig. 2 indicates that people prefer different devices at different
times of day. Next, we study whether specific topics are associated with
device types by representing each pageload with the vector of topic
probabilities (obtained from ORES) and a feature indicating if the page was
loaded during the daytime. We again fit a logistic regression to predict the
device used, with an AUC of 0.639. Inspecting feature importance shows that
people tend to access STEM and business content from desktop devices, and
biographies, entertainment, and medicine from mobile (Fig. S3).

(a) Session length histogram

(b) Session length CCDF

(c) Average length by time

(d) Regression coefficients

Figure 3.  Session-length statistics.

###  6.3. Structure of sessions

Session length. We measure session length as the number of pageloads in the
navigation tree or the reading sequence, respectively. Most sessions consist
of a single pageload (Fig. 3), but the length distribution also exposes a long
tail (Fig. 3). Therefore, we summarize session lengths via the geometric mean
(arithmetic mean in parentheses). By construction, reading sequences tend to
be longer because, unlike navigation trees, they merge both external and
internal transitions.

In the case of reading sequences, the average session length shows differences
with respect to the access method, with an average length of 1.41 (1.99) for
mobile, and 1.54 (2.40) for desktop. This difference is less pronounced for
navigation trees, where mobile sessions contain on average 1.23 (1.5)
articles, vs. 1.24 (1.5) for desktop. The average session length varies during
the day, with readers engaging in longer sessions during the evening and
night, for both navigation trees and reading sequences (Fig. 3 and Fig. S4).

To understand what properties are associated with short sessions consisting of
a single pageload, we fitted a logistic regression to predict if the reader
will continue after loading the first page in a navigation tree (results are
qualitatively identical for reading sequences; Fig. S5), representing each
first pageload with its topic probabilities (obtained from ORES), device type,
and time of day, and obtaining a model with an AUC of 0.606. Inspecting the
coefficients of the regression (Fig. 3), we find that longer [shorter]
sessions are associated with topical content around entertainment [STEM and
medicine]. This observation is corroborated by the substantial difference in
average navigation tree size across topics (Table S1).

Shape of navigation trees. In order to better understand how readers navigate
the link network, we analyze the shape of navigation trees (in contrast, the
shape of reading sequences is, by construction, always a linear chain). The
three most common patterns (Fig. 4, left) are described as follows, in order
of decreasing frequency: (1) a linear chain of pageloads; (2) fanning out from
one page to several different pages, e.g., by opening multiple tabs or rolling
back and selecting a different path; (3) a combination of the two (one-step
chain followed by fanning out). These three patterns remain the most frequent
for all tree sizes (Fig. 4, right).

We further characterize the different strategies associated with navigation
trees in terms of tree depth (i.e., average length of paths from the root to
the leaves) and breadth (i.e., average out-degree of non-leaves in the tree)
for trees starting with different topics. Noting that the two metrics are
almost perfectly anti-correlated and that the relative ordering of topics is
stable across all tree sizes (Fig. S6), we define an aggregate tree-breadth
ranking for each topic based on the average rank across tree sizes (Table S2).
This shows that entertainment topics are associated with wider trees with
higher branching, and STEM topics are characterized by deeper trees with a
more chain-like structure.

Figure 4.  Shape of navigation trees. Frequency of patterns for trees size
$N\leq 4$ (left panel). Dominance of top three patterns (see main text) for
larger trees (right panel).

(a) From the first article

(b) From the prev. article

(c) Quality

(d) Out-degree

(e) Pagerank

Figure 5.  Within-session evolution of 5 article properties. Each curve
represents sessions of different lengths.

###  6.4. Within-session article-property evolution

To shed light on navigation dynamics, we track the evolution of different
article properties within sessions. Our evolution analysis revolves around
three domains: topic space (distance from the first and previous articles),
quality, and network centrality (out-degree and PageRank). Here, reading
sequences are represented as defined above, whereas a navigation tree is
represented by the linear path from the root to the temporally last leaf, from
where the reader ceased to click further via internal links.

It is important to note that these two approaches can produce different
sequences of pageloads: e.g., a pageload in position 1 of a navigation tree
could be in position 4 of a reading sequence (as in Fig. S2). Also, the last
pageload of each sequence can have different interpretations: for navigation
trees, the reader stopped link-based navigation on that page, whereas for
reading sequences, the reader left Wikipedia for at least one hour.

In order to better interpret our observations, we compare them with three null
models corresponding to different random walkers. We randomly sample 120M
paths from the navigation trees, and run (from the tree’s starting article)
(1) an unbiased random walker that selects the next steps with uniform
probability from the available links and generates a sequence of the same
length as the original path; (2) a extrinsic-stop biased random walker that
selects the next step based on the pairwise transition probabilities obtained
from the public clickstream and generates a sequence of the same length as the
original path; (3) an intrinsic-stop biased random walker that selects the
next step—or stops—based on the pairwise transition probabilities from the
public clickstream (Rodi et al., 2017). We consider sessions up to length 15,
stratifying by session length.

Topic space. We measure the topical distance between articles via the KL
divergence of their respective WikiPDA topic distribution vectors (Sec. 3.2).
For robustness, we tried different topic models (WikiPDA and ORES) and
different distance metrics (KL divergence, Euclidean, cosine, and
Wasserstein), obtaining qualitatively similar results. First, we study how
readers diffuse in topic space starting from the first article, which plays a
special role, as it represents the entry point to Wikipedia. On average,
readers diffuse in topic space, moving further from the first article with
every step (Fig. 5). Reading sequences and navigation trees exhibit the same
trend, with a shift due to the tendency of reading sequences to ignore
external navigation. All the random walkers show similar increasing
trajectories (Fig. S8), diffusing faster than natural navigation when the
random walker is unbiased, or biased but extrinsically stopped.

Second, we measure the semantic step size in topic space by tracking how the
topical distance to the previous article evolves. Both navigation trees and
reading sequences exhibit a U-shape, suggesting that readers tend to first
reduce their semantic step size, before diverging and finally abandoning (Fig.
5). The discrepancy between navigation trees and reading sequences is
consistent with the previous observation on diffusion from the first article.
Interestingly, this U-shape is similar to the trajectories generated by the
intrinsic-stop biased random walker (Fig. S8), as also reported in previous
work (Rodi et al., 2017). In contrast, the other two random walk models show
that by selecting a random link or stopping at predefined lengths, the average
distance from the previous article tends to stabilize to an equilibrium value.

Quality. The evolution of article quality shows a sharp drop at the beginning,
for both reading sequences and navigation trees (Fig. 5). This behavior can be
interpreted as a form of regression to the mean, since many sessions start
from popular pages with high quality, which thus contribute more to the
distribution. By moving one step in the link network, readers naturally reach
a page that is, on average, of lower quality. The intuition is confirmed by
the behavior of the unbiased random walker, which shows the same drop with the
first step (Fig. S8).

In contrast to reading sequences, navigation trees show a sharp drop in
quality with the _last_ pageload. This indicates that readers have a higher
chance to stop Wikipedia-internal navigation when reaching a low-quality page,
and as a result, continue navigating in a different branch of the tree or via
an external transition.

Compared to the random walkers (Fig. S8), readers tend to navigate across
pages with less variance in quality. The random walkers’ traces support the
hypothesis that there are articles with a higher chance of terminating the
navigation: while the unbiased and extrinsic-stop biased walkers show no
termination pattern, the intrinsic-stop biased walker shows a final drop as in
human navigation. The organic stopping of this random walker, mirroring
readers’ behavior more closely, increases the chances to abandon the
navigation on pages of low quality that, according to the clickstream data,
relay less traffic.

Network centrality. Finally, we are interested in how reader sessions evolve
in the network with respect to different centrality measures. We start with
out-degree (the number of outgoing links in article bodies). Similar to
article quality, the out-degree shows a sharp drop with the first step (Fig.
5) for navigation trees and reading sequences, likely caused by the presence
of many sessions starting from pages with a high out-degree. We also find a
sharp drop for the last pageload in the sequence of the navigation trees,
suggesting that readers have a higher chance of stopping Wikipedia-internal
navigation upon reaching a page with low out-degree.

In the case of the random walkers, we draw similar conclusions as for article
quality. Whereas unbiased random walks and extrinsic-stop biased random walks
show a decrease and stabilization of out-degree, the intrinsic-stop random
walker, as humans, terminates on pages of lower degree (Fig. S8). Compared to
random walkers, human navigation is more stable: after the initial drop, they
have a higher chance to stay on pages with around 150 links.

Finally, we characterize how the PageRank of visited articles changes during
sessions. We observe that the PageRank mirrors the evolution of quality and
out-degree with regard to the initial drop (Fig. 5). Readers tend to enter
more frequently on popular pages with high centrality and naturally move to a
less central node in one step. Also for this case, a drop is visible in the
last step of the navigation trees, indicating that, when the readers reach an
article leading to the network periphery, they have higher chances to stop the
Wikipedia-internal navigation. The random walkers (Fig. S8) show that unbiased
walks naturally converge in a few steps to the most central pages with very
high PageRank. The extrinsic-stop biased walker, on the contrary, after an
initial drop, tends to move to central nodes at a much lower speed. Finally,
the intrinsic-stop biased walker, again, shows a final drop from a stable
value before abandoning the navigation, similar to human readers.

Aggregation by page. The quantities in Fig. 5 correspond to a micro-average
over all sessions, where the average behavior can be dominated by sessions
starting from the most popular pages. Therefore, we also calculate a macro-
average by aggregating on a starting-page level to make each first article
contribute equally. The diffusion in topic space is qualitatively similar in
both aggregation methods (Fig. S7 and Fig. S7). In contrast, for quality, out-
degree, and PageRank, the overall trend is inverted, i.e., instead of a sharp
drop, we observe a sharp increase in these metrics after the first step (Fig.
S7, Fig. S7, and Fig. S7). This discrepancy could be caused by the presence of
many low-quality (Piccardi et al., 2018) and low-degree articles, such that
readers at the first step tend to move to better articles in search of
information. Interestingly, the drop towards the last pageload in a session
appears across both aggregation methods.

##  7\. Discussion

We have provided a systematic characterization of the navigation pathways of
readers in Wikipedia through a large-scale study of the site’s server logs.
Our results provide a rigorous framework to capture and describe information
seeking on one of the largest platforms for open knowledge.

###  7.1. Summary of findings

Starting from the raw logs, we developed a systematic pre-processing pipeline
which allowed us to quantify how readers reach, and transition between, pages.
First, the most common way to reach a page is through an external search
engine, followed in frequency by internal navigation from other Wikipedia
articles; other sources, such as external websites (mostly social media sites)
and other Wikipedia content (such as categories or special pages), are much
less frequent, but still substantial in absolute numbers. Second, readers
frequently transition between pages via external search engines instead of
using a direct Wikipedia link. These external transitions are characterized by
larger topic jumps and larger inter-event times between pageloads; they must,
however, still be considered semantically meaningful, for, in many cases, a
link for internal navigation would be available. Third, by analyzing
sequential patterns, we find that consecutive reloads and revisits to a
previous article are rather common (10% or more each).

We continued by characterizing how readers combine the above patterns into
extended navigation sequences. First, we introduced two approaches to capture
paths of readers: navigation trees based only on internal navigation, and
reading sequences based on the time-ordered pageloads including internal and
external transitions. Second, we described how sessions are affected by their
context in terms of device type and time of day. We find that topics related
to STEM [entertainment] are more associated with working [evening and night]
hours. Third, we measured the size and structure of sessions. While most
sessions consist of a single pageload (68–78% depending on the aggregation
method), the size distribution shows a long tail with tens of millions of
sessions consisting of 10 or more pageloads. The topic not only affects the
size but also the shape of trees: while sessions starting from articles on
entertainment generally consist of more pageloads, such trees are also broader
(higher branching factor) than sessions starting, e.g., from STEM topics,
which are smaller and deeper. Fourth, we investigated the within-session
evolution of article properties. In topic space, longer sessions diffuse far
from the origin with a characteristic U-shape pattern suggesting that readers
reduce their steps first, before diverging and finally abandoning the session.
The first and last pageload of a session show special behavior regarding the
evolution of article quality and network properties. Popular pages are
naturally more common as first articles, thus engendering a form of regression
to the mean with the second step. An inverted effect appears when sessions are
aggregated at the page level, so every starting article is represented
equally. The articles at the end of the navigation are typically lower-quality
pages, suggesting that readers stop following the internal navigation when
they reach these pages acting as network sinks.

###  7.2. Implications

Complexity of data. Our results show that the dataset of navigation paths of
readers in Wikipedia extracted from server logs constitute a non-trivial
dataset requiring extreme care in order to avoid drawing spurious conclusions.
First, in contrast to existing pre-processing pipelines for sequence analysis
(e.g., tokenization steps in NLP), we still lack an understanding of best
practices for navigation paths, and as a result had to develop a set of
domain-specific heuristics. Second, operationalizing navigation paths makes
strong assumptions: while navigation trees from pure internal navigation are
more topically coherent with more complex structure, reading sequences from
temporally ordering all of the user’s pageloads are less coherent but provide
a linear sequence that is not broken by external searching (which is common).
The latter typically introduces an additional cutoff for sessions if
consecutive pageloads are separated by more than one hour (Halfaker et al.,
2015); however, our analysis suggests other potential data-informed choices
such as the time separation of internal and external transitions at
approximately 4 minutes. Naturally, the suitable choice depends on the
question of interest. Third, our analysis shows that the data can exhibit
Simpson’s paradoxes; e.g., the inversion of the within-session evolution of
page properties such as PageRank depends on the aggregation level. Fourth, the
prevalence of trivial patterns (e.g., reload or revisit) points to potential
caveats when applying prediction models to session-based recommendation (Wang
et al., 2021).

Diversity. There is extraordinary diversity in the ways readers browse
Wikipedia, modulated by topic, device, time of day, etc. This reflects the
diversity found in previous studies on the different motivations and
information needs of readers across the globe (Singer et al., 2017; Lemmerich
et al., 2019; Johnson et al., 2020). This heterogeneity indicates caution
against simplistic models aiming to capture a single average behavior.

Online ecosystem. The usage of Wikipedia is embedded in a larger online
ecosystem. Multiple studies have shown the importance of Wikipedia to search
engines (McMahon et al., 2017; Vincent and Hecht, 2021), as a gateway to the
Web (Piccardi et al., 2020, 2021), and as a main educational resource for
online learning more generally (Kross et al., 2021). Our results show that
this interplay between external and internal (with respect to Wikipedia) also
plays a crucial part on an intra-session level when navigating for knowledge.

Navigation in the wild. The navigation of readers on Wikipedia differs from
targeted navigation in lab-based settings (West and Leskovec, 2012b; Helic,
2012; West and Leskovec, 2012a). We do not observe typical strategies
characterized by, e.g., navigation via hubs (an initial increase followed by a
drop in out-degree) or gradually decreasing steps in semantic space towards
the target. Instead, we find a range of other patterns, such as a U-shape for
the step-size in semantic space and an immediate sharp drop followed by
largely constant centrality measures (out-degree, PageRank). This highlights
conceptual limitations of targeted-navigation experiments to generalize their
findings to how humans seek knowledge more generally.

Furthermore, our results provide a more nuanced picture on the conclusions
derived from publicly available data, most notably the Wikipedia clickstream
(Wulczyn and Taraborelli, 2015), which provides aggregate data on the number
of times a link was clicked. For example, we can observe that the overall
tendency to navigate towards peripheral nodes (Dimitrov et al., 2017) is
mainly driven by the first step after reaching Wikipedia, with subsequent
steps showing much smaller differences in centrality measures (with the
exception of the last step, see below). One possible interpretation is a
regression-to-the-mean effect as popular pages (the starting points of
navigation) are generally skewed towards higher centrality and quality.

Content and navigation. Our results contribute to describing the relation
between contents and navigation, expanding the prior understanding of how
readership and popularity are influenced by visual position (Dimitrov et al.,
2017) or quality (Zhu et al., 2020). Our results go beyond the population
level, suggesting that low-quality pages lead readers to stop navigating along
a specific branch in the navigation tree (and continuing along a different
branch or stopping altogether). This is specifically important in the context
of knowledge gaps in Wikipedia (Redi et al., 2020), in order to address the
uneven representation of, e.g., articles on women, where a better
understanding of the interaction between content, readers, and editors (Shaw
and Hargittai, 2018; Foundation, 2019) is crucial to allow for more informed
decision-making in designing interventions.

###  7.3. Limitations and future work

Limitations. In terms of limitations, we capture navigation paths only via
events in the server logs. Moving forward, how people engage with content
could be more accurately observed via client-side instrumentation. The
aggregation based on IP and user agent information also has limitations; e.g.,
we had to discard the sessions of large organizations with shared IP
addresses. Finally, we only focused on a single language, English. While this
already revealed a rich spectrum of phenomena, additional variation can be
expected from a comparison across languages (Lemmerich et al., 2019).

Future work. To overcome these limitations, future work should capture the
variation in navigation across Wikipedia’s over 300 languages. Moreover, in
order to better serve the different information needs of readers, a better
understanding is needed about how patterns in navigation correspond to
underlying motivations (Singer et al., 2017) or other traits such as curiosity
(Lydon-Staley et al., 2021). Finally, in order to capture knowledge seeking
more generally, researchers should capture navigation beyond individual
platforms to take into account the interdependence of Wikipedia with the rest
of the Web.

###  7.4. Conclusion

Seventy-seven years ago, in 1945, Vannevar Bush sketched his vision of an
information management device—the “memex”—that would allow users to not only
retrieve documents quickly, but to also easily interlink documents (Bush et
al., 1945). With the advent of the Web, the hyperlink structure envisioned by
Bush has since become a reality—but Bush’s vision went further: he saw the
trails taken by users as first-class citizens of the hypertext environment, as
important as the text content itself: “Wholly new forms of encyclopedias will
appear, ready made with a mesh of associative trails running through them,
ready to be dropped into the memex and there amplified” (Bush et al., 1945).
In this regard, our technological reality has not caught up with Bush’s vision
yet, and the present work should be seen as a small step toward achieving it:
we have started by describing the “associative trails running through”
Wikipedia, and we hope that its future versions will build on these insights
to incorporate tools and features that will allow readers to continually
benefit from each other’s encyclopedic trail blazing.

## References

  * (1)
  * Anderson et al. (2014) Ashton Anderson, Ravi Kumar, Andrew Tomkins, and Sergei Vassilvitskii. 2014.  The Dynamics of Repeat Consumption. In _Proc. International World Wide Web Conference (WWW)_.
  * Andreescu et al. (2021) Dan Andreescu, Kinneret Gordon, Isaac Johnson, and Nicholas Perry. 2021.  Searching for Wikipedia.  https://techblog.wikimedia.org/2021/06/07/searching-for-wikipedia/.  accessed: 13 October 2021.
  * Awad and Khan (2007) Mamoun A Awad and Latifur R Khan. 2007.  Web navigation prediction using multiple evidence combination and domain knowledge.  _IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans_ 37, 6 (2007), 1054–1062.
  * Benson et al. (2016) Austin R. Benson, Ravi Kumar, and Andrew Tomkins. 2016\.  Modeling User Consumption Sequences. In _Proc. International World Wide WebConference (WWW)_.
  * Bush et al. (1945) Vannevar Bush et al. 1945\.  As we may think.  _The atlantic monthly_ 176, 1 (1945), 101–108.
  * Chierichetti et al. (2012) Flavio Chierichetti, Ravi Kumar, Prabhakar Raghavan, and Tamas Sarlos. 2012.  Are Web Users Really Markovian?. In _Proc. International World Wide WebConference (WWW)._
  * Dallmann et al. (2016) Alexander Dallmann, Thomas Niebler, Florian Lemmerich, and Andreas Hotho. 2016.  Extracting semantics from random walks on wikipedia: Comparing learning and counting methods. In _Proc. Conference on Web and Social Media (ICWSM)_.
  * Deshpande and Karypis (2004) Mukund Deshpande and George Karypis. 2004.  Selective markov models for predicting web page accesses.  _ACM transactions on internet technology (TOIT)_ 4, 2 (2004), 163–184.
  * Dimitrov et al. (2018) Dimitar Dimitrov, Florian Lemmerich, Fabian Flöck, and Markus Strohmaier. 2018.  Query for Architecture, Click through Military: Comparing the Roles of Search and Navigation on Wikipedia. In _Proc. Conference on Web Science (WebSci)_.
  * Dimitrov et al. (2017) Dimitar Dimitrov, Philipp Singer, Florian Lemmerich, and Markus Strohmaier. 2017.  What Makes a Link Successful on Wikipedia?. In _Proc. International World Wide Web Conference (WWW)_.
  * Foundation (2019) Wikimedia Foundation. 2019\.  Medium-term plan 2019: The model for engagement.  https://meta.wikimedia.org/wiki/Wikimedia_Foundation_Medium-term_plan_2019#The_model_for_engagement.  accessed: 13 October 2021.
  * Geng and Tian (2015) Ruili Geng and Jeff Tian. 2015.  Improving Web Navigation Usability by Comparing Actual and Anticipated Usage.  _IEEE Transactions on Human-Machine Systems_ 45, 1 (2015), 84–94.
  * Gildersleve and Yasseri (2018) Patrick Gildersleve and Taha Yasseri. 2018.  Inspiration, Captivation, and Misdirection: Emergent Properties in Networks of Online Navigation.  _Complex Networks IX_ (2018), 271–282.
  * Halfaker (2017) Aaron Halfaker. 2017\.  Interpolating quality dynamics in Wikipedia and demonstrating the Keilana effect. In _Proc. International Symposium on Open Collaboration (OpenSym)_.
  * Halfaker and Geiger (2019) Aaron Halfaker and R.Stuart Geiger. 2019.  ORES: Lowering Barriers with Participatory Machine Learning in Wikipedia.  _Proc. Human-Computer Interaction (HCI)_.
  * Halfaker et al. (2015) Aaron Halfaker, Os Keyes, Daniel Kluver, Jacob Thebault-Spieker, Tien Nguyen, Kenneth Shores, Anuradha Uduwage, and Morten Warncke-Wang. 2015.  User Session Identification Based on Strong Regularities in Inter-Activity Time. In _Proc. International World Wide Web Conference (WWW)_.
  * Helic (2012) Denis Helic. 2012\.  Analyzing user click paths in a Wikipedia navigation game. In _Proc. International Convention MIPRO_.
  * Jindal et al. (2020) Honey Jindal, Neetu Sardana, and Raghav Mehta. 2020\.  Efficient web navigation prediction using hybrid models based on multiple evidence combinations.  _International Journal of Computers and Applications_ 42, 7 (2020), 715–728.
  * Johnson et al. (2020) Isaac Johnson, Florian Lemmerich, Diego Sáez-Trumper, Robert West, Markus Strohmaier, and Leila Zia. 2020.  Global gender differences in Wikipedia readership. In _Proc. Conference on Web and Social Media (ICWSM)_.
  * Khalil et al. (2009) Faten Khalil, Jiuyong Li, and Hua Wang. 2009.  An integrated model for next page access prediction.  _International Journal of Knowledge and Web Intelligence_ 1, 1-2 (2009), 48–80.
  * Koopmann et al. (2019) Tobias Koopmann, Alexander Dallmann, Lena Hettinger, Thomas Niebler, and Andreas Hotho. 2019\.  On the Right Track! Analysing and Predicting Navigation Success in Wikipedia. In _Proc. Conference on Hypertext and Social Media (HT)_.
  * Kross et al. (2021) Sean Kross, Eszter Hargittai, and Elissa M Redmiles. 2021\.  Characterizing the Online Learning Landscape: What and How People Learn Online.  _ACM Hum.-Comput. Interact._ 5, CSCW1 (Feb. 2021), 19.
  * Kulshrestha et al. (2020) Juhi Kulshrestha, Marcos Oliveira, Orkut Karacalik, Denis Bonnay, and Claudia Wagner. 2020.  Web Routineness and Limits of Predictability: Investigating Demographic and Behavioral Differences Using Web Tracking Data.  _arXiv preprint arXiv:2012.15112_ (2020).
  * Lamprecht et al. (2016) Daniel Lamprecht, Dimitar Dimitrov, Denis Helic, and Markus Strohmaier. 2016.  Evaluating and Improving Navigability of Wikipedia: A Comparative Study of Eight Language Editions. In _Proc. International Symposium on Open Collaboration (OpenSym)_.
  * Lamprecht et al. (2017) Daniel Lamprecht, Kristina Lerman, Denis Helic, and Markus Strohmaier. 2017.  How the structure of Wikipedia articles influences user navigation.  _New Review of Hypermedia and Multimedia_ 23, 1 (2017), 29–50.
  * Lehmann et al. (2014) Janette Lehmann, Claudia Müller-Birn, David Laniado, Mounia Lalmas, and Andreas Kaltenbrunner. 2014\.  Reader Preferences and Behavior on Wikipedia. In _Proc. Conference on Hypertext and Social Media (HT)_.
  * Lemmerich et al. (2019) Florian Lemmerich, Diego Sáez-Trumper, Robert West, and Leila Zia. 2019.  Why the world reads Wikipedia: Beyond English speakers. In _Proc. International Conference on Web Search and Data Mining (WSDM)_.
  * Lydon-Staley et al. (2021) David M Lydon-Staley, Dale Zhou, Ann Sizemore Blevins, Perry Zurn, and Danielle S Bassett. 2021.  Hunters, busybodies and the knowledge network building associated with deprivation curiosity.  _Nature Human Behaviour_ 5, 3 (2021), 327–336.
  * Mabroukeh and Ezeife (2009) Nizar R Mabroukeh and Christie I Ezeife. 2009.  Semantic-rich markov models for web prefetching. In _Proc. International Conference on Data Mining Workshops (ICDMW)_. IEEE, 465–470.
  * Machlup (1983) Fritz Machlup. 1983\.  The study of information: Interdisciplinary messages.  (1983).
  * Maggio et al. (2020) Lauren A Maggio, Ryan M Steinberg, Tiziano Piccardi, and John M Willinsky. 2020.  Meta-Research: Reader engagement with medical content on Wikipedia.  _Elife_ 9 (2020), e52426.
  * McMahon et al. (2017) Connor McMahon, Isaac Johnson, and Brent Hecht. 2017\.  The Substantial Interdependence of Wikipedia and Google: A Case Study on the Relationship Between Peer Production Communities and Information Technologies.  _Proc. Conference on Web and Social Media (ICWSM)_.
  * Mitrevski et al. (2020) Blagoj Mitrevski, Tiziano Piccardi, and Robert West. 2020\.  WikiHist. html: English Wikipedia’s Full Revision History in HTML Format. In _Proc. Conference on Web and Social Media (ICWSM)_.
  * Muramatsu and Pratt (2001) Jack Muramatsu and Wanda Pratt. 2001.  Transparent Queries: investigation users’ mental models of search engines. In _Conference on Research & Development in Information Retrieval (SIGIR)_.
  * Narvekar and Banu (2015) Meera Narvekar and Shaikh Sakina Banu. 2015.  Predicting user’s web navigation behavior using hybrid approach.  _Procedia Computer Science_ 45 (2015), 3–12.
  * Nisbett and Wilson (1977) Richard E Nisbett and Timothy D Wilson. 1977.  Telling more than we can know: Verbal reports on mental processes.  _Psychological review_ 84, 3 (1977), 231.
  * Olteanu et al. (2019) Alexandra Olteanu, Carlos Castillo, Fernando Diaz, and Emre Kıcıman. 2019.  Social data: Biases, methodological pitfalls, and ethical boundaries.  _Frontiers in Big Data_ 2 (2019), 13.
  * Paranjape et al. (2016) Ashwin Paranjape, Robert West, Leila Zia, and Jure Leskovec. 2016\.  Improving Website Hyperlink Structure Using Server Logs. In _Proc. International Conference on Web Search and Data Mining (WSDM)_.
  * Piccardi et al. (2018) Tiziano Piccardi, Michele Catasta, Leila Zia, and Robert West. 2018\.  Structuring Wikipedia articles with section recommendations. In _Conference on Research & Development in Information Retrieval (SIGIR)_.
  * Piccardi et al. (2020) Tiziano Piccardi, Miriam Redi, Giovanni Colavizza, and Robert West. 2020.  Quantifying engagement with citations on Wikipedia. In _Proc. International World Wide Web Conference (WWW)_.
  * Piccardi et al. (2021) Tiziano Piccardi, Miriam Redi, Giovanni Colavizza, and Robert West. 2021.  On the Value of Wikipedia as a Gateway to the Web. In _Proc. International World Wide Web Conference (WWW)_.
  * Piccardi and West (2021) Tiziano Piccardi and Robert West. 2021.  Crosslingual Topic Modeling with WikiPDA.  _Proc. International World Wide Web Conference (WWW)_.
  * Pirolli and Pitkow (1999) Peter L T Pirolli and James E Pitkow. 1999.  Distributions of surfers’ paths through the World Wide Web: Empirical characterizations.  _World Wide Web_ 2, 1 (1999), 29–45.
  * Redi et al. (2020) Miriam Redi, Martin Gerlach, Isaac Johnson, Jonathan Morgan, and Leila Zia. 2020.  A Taxonomy of Knowledge Gaps for Wikimedia Projects (Second Draft).  (Aug. 2020).  arXiv:2008.12314 [cs.CY]
  * Rodi et al. (2017) Giovanna Chiara Rodi, Vittorio Loreto, and Francesca Tria. 2017\.  Search strategies of Wikipedia readers.  _PLOS ONE_ 12, 2 (02 2017), 1–15.
  * Scaria et al. (2014) Aju Thalappillil Scaria, Rose Marie Philip, Robert West, and Jure Leskovec. 2014.  The Last Click: Why Users Give up Information Network Navigation. In _Proc. International Conference on Web Search and Data Mining (WSDM)_.
  * Shaw and Hargittai (2018) Aaron Shaw and Eszter Hargittai. 2018.  The Pipeline of Online Participation Inequalities: The Case of Wikipedia Editing.  _The Journal of communication_ 68, 1 (Feb. 2018), 143–168.
  * Singer et al. (2017) Philipp Singer, Florian Lemmerich, Robert West, Leila Zia, Ellery Wulczyn, Markus Strohmaier, and Jure Leskovec. 2017.  Why we read Wikipedia. In _Proc. International World Wide Web Conference (WWW)_.
  * Singer et al. (2013) Philipp Singer, Thomas Niebler, Markus Strohmaier, and Andreas Hotho. 2013.  Computing Semantic Relatedness from Human Navigational Paths: A Case Study on Wikipedia.  _Int. J. Semant. Web Inf. Syst._ 9, 4 (Oct. 2013), 41–70.
  * Tauscher and Greenberg (1997) Linda Tauscher and Saul Greenberg. 1997.  Revisitation Patterns in World Wide Web Navigation. In _Proc. Conference on Human Factors in Computing Systems (CHI)_.
  * TeBlunthuis et al. (2019) Nathan TeBlunthuis, Tilman Bayer, and Olga Vasileva. 2019\.  Dwelling on Wikipedia: Investigating Time Spent by Global Encyclopedia Readers. In _Proc. International Symposium on Open Collaboration (OpenSym)_.
  * Tizzoni et al. (2020) Michele Tizzoni, André Panisson, Daniela Paolotti, and Ciro Cattuto. 2020.  The impact of news exposure on collective attention in the United States during the 2016 Zika epidemic.  _PLoS computational biology_ 16, 3 (March 2020), e1007633.
  * Vincent and Hecht (2021) Nicholas Vincent and Brent Hecht. 2021.  A Deeper Investigation of the Importance of Wikipedia Links to Search Engine Results.  _Proc. ACM Hum.-Comput. Interact._ 5, CSCW1 (April 2021), 1–15.
  * Wang et al. (2021) Shoujin Wang, Longbing Cao, Yan Wang, Quan Z Sheng, Mehmet A Orgun, and Defu Lian. 2021\.  A Survey on Session-based Recommender Systems.  _ACM Comput. Surv._ 54, 7 (July 2021), 1–38.
  * West and Leskovec (2012a) Robert West and Jure Leskovec. 2012a.  Automatic Versus Human Navigation in Information Networks.  _Proc. Conference on Web and Social Media (ICWSM)_ (2012).
  * West and Leskovec (2012b) Robert West and Jure Leskovec. 2012b.  Human Wayfinding in Information Networks. In _Proc. International World Wide Web Conference (WWW)_.
  * West et al. (2015) Robert West, Ashwin Paranjape, and Jure Leskovec. 2015\.  Mining Missing Hyperlinks from Human Navigation Traces: A Case Study of Wikipedia.  _Proc. International World Wide Web Conference (WWW)_.
  * West et al. (2009) Robert West, Joelle Pineau, and Doina Precup. 2009\.  Wikispeedia: An Online Game for Inferring Semantic Distances between Concepts. In _Proc. International Joint Conference on Artificial Intelligence (IJCAI)_.
  * Wulczyn and Taraborelli (2015) Ellery Wulczyn and Dario Taraborelli. 2015.  Wikipedia clickstream.  https://meta.wikimedia.org/wiki/Research:Wikipedia_clickstream.
  * Younger (2010) Paula Younger. 2010\.  Internet-based information-seeking behaviour amongst doctors and nurses: a short review of the literature.  _Health Information & Libraries Journal_ 27, 1 (2010), 2–10.
  * Zhu et al. (2020) Kai Zhu, Dylan Walker, and Lev Muchnik. 2020.  Content Growth and Attention Contagion in Information Networks: Addressing Information Poverty on Wikipedia.  _Information Systems Research_ 31, 2 (June 2020), 491–509.

## Supplementary Material

Figure S1. Examples of patterns in the logs and the multitude of client-side
behaviors that can leave these digital traces. Black arrows represent click,
red arrow are back button, yellow are multitab clicks Figure S2. This set of
log events yields three navigation trees, represented by arrows and composed
of ACE, DG, and F. The reading sequences method creates two sessions
represented as gray boxes: ABCDE and FG. Square boxes are clicks from external
origins

(a) Navigation trees

(b) Reading sequences

Figure S3. Feature contributions to a logistic model predicting if the the
session is started from a mobile or desktop device.

(a) Number of reading sequences

(b) Session length by time

Figure S4. Total count (Fig. S4), and average length (Fig. S4) of the reading
sequences started at different time of the day.

(a) Multiple-pageloads

(b) Daytime

Figure S5. Feature contributions to the logistic model predicting if the
reading sequence is composed by more than one pageload (Fig. S5), and if the
reading sequence started during daytime (Fig. S5). Table S1. Top and bottom 10
topics with respect to (geometric) average tree size (geographical topics
excluded). Tree size
---
Top 10 (larger trees) | Bottom 10 (smaller trees)
1.377 | Films | 1.152 | Earth and environment
1.373 | Entertainment | 1.148 | Food and drink
1.340 | Television | 1.145 | Biology
1.327 | Military and warfare | 1.138 | Technology
1.324 | Music | 1.128 | Physics
1.295 | Comics and Anime | 1.122 | Software
1.284 | History | 1.114 | Medicine & Health
1.272 | Biography | 1.112 | Computing
1.269 | Sports | 1.104 | Mathematics
1.264 | Transportation | 1.100 | Chemistry
Table S2. Rank with respect to average degree of navigation trees, by topic (geographical topics excluded). A separate rank was computed per tree size (3–15), and arithmetic means over tree sizes are reported, alongside standard deviations. Top 10 (wider trees) | Bottom 10 (deeper trees)
---|---
Rank (mean) | Std | Starting Topic | Rank (mean) | Std | Starting Topic
1.00 | 0.00 | Films | 27.42 | 2.72 | Linguistics
2.50 | 0.87 | Television | 29.42 | 0.95 | Earth and environment
3.58 | 0.76 | Entertainment | 29.50 | 1.19 | Space
4.50 | 1.85 | Comics and Anime | 30.08 | 2.78 | History
4.67 | 1.31 | Education | 31.92 | 1.11 | Computing
6.58 | 1.98 | Video games | 32.92 | 1.55 | Software
7.92 | 2.43 | Literature | 34.67 | 1.75 | Chemistry
8.50 | 2.36 | Fashion | 34.75 | 1.30 | Physics
8.83 | 1.07 | Performing arts | 35.50 | 1.26 | Mathematics
10.42 | 2.29 | Internet culture | 35.67 | 1.65 | Libraries & Information

Figure S6. Relation between the average depth and average degree for
navigation trees of different sizes.

(a) From the first article

(b) From the prev. article

(c) Quality

(d) Out-degree

(e) Pagerank

Figure S7.  Comparison of the evolution of five different properties when
aggregating navigation trees by sequence and by page. Gray trajectories added
for readability. Each curve represents sessions of different lengths.

(a) From the first article

(b) From the prev. article

(c) Quality

(d) Out-degree

(e) Pagerank

Figure S8. Properties evolution of the trajectories generated by the three
random walk models compared with the natural navigation based on navigation
trees. Each curve represents sessions of different lengths.