# The Stories We Tell About Data:
Media Types for Data-Driven Storytelling

Zhenpeng Zhao and Niklas Elmqvist111Corresponding author<EMAIL_ADDRESS>
University of Maryland, College Park

(January 2022)

## Abstract

The emerging practice of data-driven storytelling is framing data using
familiar narrative mechanisms such as slideshows, videos, and comics to make
even highly complex phenomena understandable. However, current data stories
are still not utilizing the full potential of the storytelling domain. One
reason for this is that current data-driven storytelling practice does not
leverage the full repertoire of media that can be used for storytelling, such
as the spoken word, e-learning, and video games. In this paper, we propose a
taxonomy focused specifically on media types for the purpose of widening the
purview of data-driven storytelling simply by putting more tools into the
hands of designers. Using our taxonomy as a generative tool, we also propose
three novel storytelling mechanisms, including for live-streaming, gesture-
driven oral presentations, and textual reports that dynamically incorporate
visual representations.

Keywords: Data-driven storytelling; narrative visualization; taxonomy; design
guidelines.

(a) Annotated chart created in ChartAccent.

(b) Data-driven story of a session in the video game Civilization 3 (2001).

(c) Will Ferrell annotating everyday activities in Stranger Than Fiction
(2006).

Figure 1: Many possible media types are available today for data-driven
storytelling.

##  1 Introduction

From the hunter returning from his latest foray to tell tall tales of stalking
his prey, to the shaman spinning a yarn about the origins of the gods, the
stars, and the moon, storytelling is the oldest form of human communication,
record-keeping, and entertainment. [35, 47, 55] Stories—sequences of events
involving characters and places—are particularly well suited for this purpose
because their chronological structure enables recall, entices listeners, and
facilitates understanding. [20, 35] For this reason, narration and
storytelling retain important roles even in today’s information society, where
these properties are particularly important in helping people get to grips
with an increasingly complex world. This has recently given rise to data-
driven storytelling where narrative techniques are utilized for telling
stories about data [49], often using visual media. [17, 50]

Examples of such data-driven storytelling abound, and include data comics, [5,
62] infographics, [24] data videos, [1, 2] data-driven slideshows, [29] and
even story sketching. [34] The common denominator is that they are based on
story arcs evolving over time to build an argument, explain a phenomenon, or
report a finding. They diverge by the form of media, or channels of
communication, they use: sequential art, animated graphics, presentation
slides, and handwriting, respectively.

However, while visual representations are particularly useful for
storytelling, there are many additional types of media—digital and
otherwise—that can be co-opted for telling stories about data. An existing
framework proposed by Segel and Heer [49] outline seven genres of narrative
visualization, but conflates the storytelling mechanism with the media being
used. Furthermore, Segel and Heer also note that their sample is limited (58
items) and does not cover the full scope of possible media that can be used
for storytelling, [49] such as video games, infotainment, and other e-learning
tools. Meanwhile, much research in data-driven storytelling has used these
seven genres as a starting point, which suggests that the community may be
limiting itself by needlessly adhering to a framework that was intended to be
generative rather than prescriptive.

What about using the spoken word for data-driven storytelling, i.e.,
supporting speakers talking to an audience? What about the written word, i.e.,
data-driven prose, such as for inclusion in a textual report? And what about
other forms of human communication, such as theatre, poetry, and even dance?
Much innovation remains in data-driven storytelling, but this requires going
beyond existing frameworks. Hence this work.

We present a new taxonomy focused on media for data-driven storytelling with
the purpose of opening the field to a wider set of future possibilities. Our
work started with collecting evidence of data-driven storytelling using novel
and diverse media, from the spoken word to interpretative dance and
choreography. We then use this wealth of data to derive a taxonomy and
classify all of these examples into a coherent framework. We use the framework
to postulate some potential future media types for data-driven storytelling.
Finally, by generalizing across storytelling practice for different media, we
derive design guidelines for data-driven storytelling. We conclude the paper
with a summary and our plans for future work.

##  2 Data-Driven Storytelling

Storytelling is the conveyance of a sequence of events, often involving
characters and places—stories—using speech, sound, and visuals, [20] and has a
history spanning thousands of years. [47, 55] A visual narrative is a story
told primarily using visual media, such as illustrations, photographs,
animations, video, and—now—visualization. [17, 50] In particular,
visualization has a specific proclivity for communication by virtue of its
graphical form, yielding the notion of communication-minded visualization [57]
to support collaborative analysis. Combining the idea of communication-driven
visualization with storytelling yields the notion of data-driven storytelling:
narrative techniques for telling stories about data. [49]

Figure 2: Seven genres of narrative visualization by Segel and Heer. [49]

###  2.1 Existing Media Types

Our focus in this paper is on communication mechanisms—or media—used for
conveying data-driven stories. In this paper, we define “media” as the channel
or the tools used to store and deliver information. While data-driven
storytelling is a nascent research topic in visual communication and
visualization, there has so far been no focus directed to the specific media
used. Instead, existing efforts tend to revolve around the seven genres of
narrative visualization proposed by Segel and Heer [49] (Figure 2):

  * •

Magazine Style: A data-driven image integrated in a page of text, where the
text refers to and explains the image.

  * •

Annotated Chart: Chart adorned with descriptive text and labels for the
purpose of explaining its contents.

  * •

Partitioned Poster: A poster or dashboard consisting of multiple images, each
with separate data.

  * •

Flow Chart: Visually ordered sequence of images and annotations designed to
tell a story.

  * •

Comic Strip: Sequence of frames containing images and text organized in a
comic-style strip layout.

  * •

Slide Show: Deck of slides combining images and text to sequentially tell a
story.

  * •

Film/Video/Animation: Motion graphics that incorporate data-driven imagery and
visualizations, often animated.

However, as readily admitted by Segel and Heer, their findings are limited to
a sample of 58 examples. They also do not claim that their genres are
exhaustive, noting for example that their work did not include video games or
e-learning tools. Furthermore, the above seven genres conflate the media used
for storytelling with the format, method, and components employed. The seven
genres have proven to be extremely powerful for categorizing research in this
field; they have even played a prescriptive role, with Graph Comics [4] and
Data Comics [62] arising as examples of the Comic Strip genre, and Data Videos
[1, 2] drawing inspiration from the Film/Video/Animation genre. However, there
certainly is room for expanding the framework further.

###  2.2 Visual Communication

Visual forms of communication, including icons, illustrations, schematics,
photographs, and full-motion video, have long been considered to be one of the
most compelling and approachable storytelling formats. [11, 50] Presenting
insights from data to the masses requires taking the visualization literacy
[9] of everyday users into account. Thus, the notion of “casual visualization”
[42] is important.

Complex skills are today often learned through step-by-step instructions on
the internet. [22] Several projects have endeavored to increase the richness
of such video tutorials: ToolClips [22] provide contextual access to video
assistance, Pause-and-Play [41] links tutorials to the real application, MixT
[12] automatically generates video from demonstrations, and a recent approach
makes software tutorial videos interactive. [38]

Beyond traditional providers (HBO, MSNBC, ABC, etc) as well as the new breed
(Amazon Prime, Hulu, and Netflix), open video sharing sites such as YouTube
have democratized access to visual storytelling. With such openness comes
entirely new storytelling formats that were previously infeasible, such as
vlogs (video blogs), [21] reaction videos, [3] and video commentaries,
including the ever-popular Let’s Play videos. [58] An additional recent
development in internet video is the increased focus on live, so-called
streaming, content, where the media is received and presented to the consumer
at the same time it is delivered by the provider. This development was
particularly driven by live gameplay content, where prominent gamers—called
streamers—broadcast their game screen on services such as Twitch.tv. While
streaming is also used for heavily produced events such as eSports
tournaments, where professional gamers compete over thousands of dollars in
prize money in games such as Overwatch, League of Legends, and Dota 2, the
vast majority of live streams on Twitch and elsewhere are created using
specialized broadcasting software such as OBS, XSplit, GameShow, etc.

Amini et al. [1] recently identified data videos as motion graphics combining
both sound and visuals to tell a data story. Pointing to prominent examples
from the New York Times and the Guardian, their work encourages professional
storytellers to use visuals to craft their narratives. Their follow-up Data
Clips [2] work is an authoring tool for creating data-driven clips
incorporating visualizations that can be assembled into longer data videos.

###  2.3 Visualization for Communication

Data visualization is the use of interactive graphical representations of data
to aid cognition. [10] There are arguably two main uses of data visualization
[57]:

  * •

Exploration: Visualization for exploratory data analysis [54] to gain insights
and generate hypotheses; and

  * •

Explanation: Visualization for informing external stakeholders about data
using visual means.

Data-driven storytelling is a natural extension of visualization for
explanation (the latter). The production, presentation, and dissemination of
results is a grand challenge of visualization and visual analytics. [53]
Gershon and Page first proposed using storytelling for visualization [18], and
their work has since been followed up by workshops, [14, 15] surveys, [25, 49]
and even commercial tools. [29] Viégas and Wattenberg remark upon the
proclivity of visualization for communication by virtue of its graphical form,
and encourage focusing on so-called communication-minded visualization [57]
where communication enables collaborative analysis.

To reach its full potential, communication capabilities should be integrated
into the visualization tools themselves [53]; for example, Tableau
incorporates story points, [29] and most commercial tools support exporting
interactive dashboards to the web.

##  3 Evidence of Data-Driven Storytelling Media

To illustrate the prevalence and variety of different data-driven narratives
in the world, we here enumerate and discuss a set of representative and
innovative such examples. The purpose is to provide a basis for a taxonomy
that can be used to classify the storytelling media used for the data-driven
narrative. For each example, we use an informal classification scheme to
describe the media in more detail. This scheme will then feed into our
taxonomy in the following section.

We use the following criteria for our selection:

  * •

Has a story format, i.e., a progression (or arc) over the course of the
artifact;

  * •

Leverages data to drive its narrative;

  * •

Uses an approach that is novel over Segel and Heer’s seven genres [49]; and

  * •

Inhabits a unique position w.r.t other examples.

Figure 3: Interactive signal processing of the LIGO-Virgo collaboratory data
for detecting gravitational waves in a Jupyter notebook [27].

Figure 4: Interactive article on Gaussian processes [23] from Distill, an
online machine learning journal. Manipulating widgets will modify text and
figures accordingly.

Figure 5: An explorable report for multiverse analysis, as proposed by
Dragicevic et al. [16] Clicking underlined parameters in the text toggles
between different animations and changes the visual representations.

Figure 6: Jeffrey Heer’s interactive article (written in Idyll [13])
explaining the Barnes-Hut approximation for graph layout. As is typical for
this type of artifact, the article contains multiple widgets that allows the
reader interact with the displayed model.

###  3.1 Exhibit 1: Interactive Articles

Online journalism is changing to the point where so called interactive
articles are becoming a new, popular, and dominant form of visual
communication. [13] These data-driven narratives—or explorable explanations
[56]—combine traditional journalistic storytelling with interactive components
that allow a potentially large audience to engage with them. Major newspapers
such as the New York Times, The Guardian, and the Washington Post have long
made such interactive articles a popular part of their online presence; in
fact, the New York Times graphics department was long a key driver for the D3
[7] toolkit (while Michael Bostock was a member until 2015).

While already tremendously successful in practice—ten of the New York Times’
forty most popular articles in 2014 were interactive articles from its Upshot
department [13]—the concept of interactive articles is also beginning to take
hold in the academic realm. Already in 2011, publishing giant Elsevier
launched the Executable Paper Grand
Challenge222https://www.journals.elsevier.com/international-journal-of-human-
computer-studies/news/introducing-executable-papers to find ways to improve
reproducibility of research by including executable code inside an interactive
article. The winner of this challenge was the Collage authoring environment,
[39] and the Elsevier journal Computers & Graphics even published a special
issue testing the environment, [51] but unfortunately, the movement seems to
have died down within Elsevier.

Instead of starting from the article and going to interactivity, what about
starting from code and going towards an article? As embodiments of Knuth’s
literate programming [28] paradigm, where traditional source code is embedded
in descriptive natural language, computational notebooks [19, 40, 27, 43, 52,
59]—which combine executable code, its output, and descriptive text and other
media in a single document—is based on this premise. As a result,
computational notebooks have become a lingua franca for presenting findings,
supporting material, and breakthroughs in many STEM disciplines [45]; for
example, when gravitational waves were first observed by the LIGO-Virgo
collaboration in 2015, the announcement in February 2016 was accompanied by a
Jupyter notebook complete with all the collected data.333https://www.gw-
openscience.org/ However, computational notebooks are generally rather
technical and expose interactive mostly through source code rather than
widgets and controls.

Distill,444https://distill.pub/ on the other hand, is an online academic
journal for machine learning research that is based on interactive, explorable
articles. Figure 4 shows an example of a Distill article on understanding
Gaussian processes where smooth controls allows the reader to change
parameters, causing the visual representations to update accordingly.
Similarly, the Dragicevic et al. [16] propose explorable reports that are
designed specifically for multiverse analysis, where clickable controls in a
paper allows for varying the reporting for many different statistical analyses
in order to show the fragility or robustness of the findings. Figure 5 shows
an example of an example “mini-paper” implemented using this method; clicking
on the blue underlined text will cycle between matching results and images.

Most recently and inspired by these efforts, Conlen and Heer [13] proposed
Idyll, a domain-specific language for authoring interactive articles that is
specifically targeted at journalists and designers. Using a rich standard
library of components, Idyll allows even non-technical users to quickly create
compelling and highly interactive content. Figure 6 shows an example
interactive article created in Idyll on the Barnes-Hut approximation for graph
layouts.555https://jheer.github.io/barnes-hut/ In fact, Matthew Conlen, the
creator of Idyll, has also founded Parametric
Press666https://parametric.press/, an online magazine for interactive articles
built in Idyll.

Informal classification: Key in all of the interactive articles reviewed above
is that they are interactive; they allow—even invite—interaction by the
reader. In some cases, this is done through low-level and technical means,
such as changing source code in a Jupyter notebook, but in more “polished”
cases intended for a more general audience, such as Distill and Idyll, the
interaction is performed through standard widgets.

The other significant characteristic of interactive articles is inherent in
the actual medium itself: they are based on a typical document format where
the main context is textual in nature. As with previous examples of new media
being introduced into society, it is clear that data-driven stories will
mostly augment rather than replace existing media types. In other words, a
fruitful approach to invent future storytelling media for data is to study
existing ones.

Figure 7: Using SketchStory [34] to give an interactive and visual
presentation on data.

Figure 8: Explaining computer science concepts using ballet.

Figure 9: Visualizing movement data in dance.draw [32].

Figure 10: Physical dance demonstration of a bubble sort algorithm [63].

###  3.2 Exhibit 2: Data-Driven Performance

Some storytelling takes place in person and is enacted by one or several
performers. SketchStory by Lee et al. [34] (Figure 7) is close to traditional
data-driven storytelling, and supports a person giving a presentation using an
interactive whiteboard that responds to sketch input to generate interactive
graphics. However, SketchStory is curiously alone in this area—few
visualization or data science tools exist to support in-person presentations.
This was also the motivation for our GestureStory concept described later in
this paper.

However, there are additional ways that people have used live performance to
communicate data. Figure 10 shows an image from a performance where dancers
demonstrate how the bubble sort algorithm works. [63] This algorithm pushes
the largest element to the right and forms a ordered sequence of numbers. For
clarity, each dance is labeled in the this picture. In reality, the dancers
wear uniforms with numbers on them, where each dancer stands for a different
element in the array. The storyline is the movement of all the array elements.
The data is the ordered sequence.

A similar data-driven dance performance was created by the Dance.Draw project
[32, 33] (Figure 9), where the movements of dancers in a physical space was
conveyed using visual representations. This mechanism could also be used as a
vehicle for conveying data-driven stories. Finally, the Data-Driven Dance
project777http://www.datadrivendance.org/ (Figure 8) has three performances
designed to convey abstract data and blur the boundary between science and
art: [arra]stre (data-driven dance for computer science theory), [data]storm
(ocean storms, networks, and weather), and [pain]byte (biomedical engineering,
chronic pain, and dance).

Informal classification: In-person performances typically take place in an
auditorium or studio, which supports a large audience. Disregarding video
recordings of the performance (which would be another form of media), this
does require the audience to be physically co-located with the performance,
and to consume it in synchronously, in real-time. This means that the
performance is not stored; it is emphemeral. The components of the performance
are human bodies in motion over the duration of the performance, and can also
include text, sound, light, and visuals (typically projected in the
workspace).

Figure 11: Graphics displayed on the pitch for a football game.

Figure 12: Augmented reality forecast on NBC News.

Figure 13: Augmented reality forecast on CBS News.

Figure 14: Immersive augmented reality hurricane experience on the Weather
Channel.

###  3.3 Exhibit 3: Storytelling in Composite/Augmented Reality

Augmented reality [48] is making significant inroads in society as technology
becomes availability and new ideas to leverage the technology are introduced.
Aspects of technology has in fact already been used in a limited capacity in
broadcasting for a long time, particularly in sports, where it can be most
accurately named “composite” graphics. Figure 11 shows a typical example of
informative graphics inlaid on an American football field on a live broadcast.

The practice has also recently been employed in weather reporting. In fact,
weather reporting has a long history of using image composition to combine the
reporter’s image with a virtual background through chromakeying. Augmented
reality allows for more advanced effects. Figure 12 shows a studio
incorporating both virtual (the logo and temperature gauge) and real component
(the weatherman) in the same shot. In Figure 13, this is taken to the next
level by creating a 3D depiction of a hurricane in front of the reporter.
Finally, Figure 14 shows the popular and much-discussed “immersive hurricane”
footage from the Weather Channel, where the impact of Hurricane Florence on
North Carolina in September 2018 was shown in a live broadcast.

Informal classification: Clearly, augmented reality has much potential for
storytelling, but its parameters depend on how it is used. When experienced by
a person using an AR headset, the audience is a single person, whereas the
broadcasting examples discussed here are clearly intended for a broad
audience. Similarly, the interaction depends on whether it is experienced in
person—in which case the person can navigate freely in the real world as well
as interact with virtual objects—or viewed as composite graphics on a screen,
where the interaction is limited to playback.

###  3.4 Exhibit 4: Data-Driven Storytelling in Video Games

While Segel and Heer [49] explicitly note that they chose not to include video
games in their survey, games have long been instances where visualization is
often integrated. [8] As it turns out, they have also been used for data-
driven storytelling. Figure 1b depicts an image from the replay session that
is shown at the end of a completed game session, i.e., after one of the
players (human or computer) has achieved one of the victory conditions:
conquering all opponents, winning the space race, taking over most of the
land, or scoring a diplomatic or cultural victory. The replay shows a history
of how each civilization was founded, expanded across the world, and was
eventually defeated. While the interaction is limited, playback controls
allows the user to go back and forward in the history. Similar session
playback tools—often called theater mode—can be found in Halo 3 and Call of
Duty: Black Ops.

Informal classification: The audience for most theater modes is individual
players who wants to study their own and other players’ performance. However,
many theater modes typically also allow the player to cut and paste clips
together, eventually producing a resulting video to share with others. The
resulting video will have the same features as a data video (see above). A
playback session typically uses a map view, so the bandwidth requirement is
lower than full-motion video, thus reducing the cognitive load. One feature of
most theater modes is that they make it easy to navigate in 2D or 3D in the
scene, thus changing the viewpoint. While the action itself cannot be changed
(since it represents events that already happened), this interaction is
powerful in that it can, for example, allow a player to put themselves in the
shoes of another player to see what an encounter looked like from their
viewpoint.

Table 1: Classification of our representative sample of different media used
for data-driven storytelling. Examples in italics are categories.

##  4 Taxonomy of Data-driven Storytelling Media

In this paper, we propose to identify, study, and classify media that have
traditionally been used for storytelling. The purpose of this activity is to
expand existing genres of narrative visualization to encompass the entire
scope of storytelling in society. This, in turn, would generate a wealth of
new research ideas for how to best use such media for data-driven
storytelling. Based on our survey of the existing evidence of a wide variety
of media for data-driven storytelling, we found the following dimensions
useful in classifying them:

  * •

Audience Cardinality (A): Who is the intended recipient for the story?

  * •

Space and Time (S/T): What is the temporal and spatial delivery mechanism for
the story?

  * •

Media Components (VC): What are the visual and sound building blocks employed?

  * •

Data Components (DC): How is the data conveyed to the viewer?

  * •

Viewing Sequence (SQ) How is the media viewed by the viewer?

The method we used to derive these dimensions included reviewing a large
sample of examples of data-driven storytelling, and then narrowing our
selection down a set of representative examples. These examples have been
classified into the taxonomy and can be viewed in Table 1.

###  4.1 Audience Cardinality (A)

In our notion of the audience of the data-driven story, we also include the
storytellers: whether it is one or several people who are creating or viewing
the narrative, respectively. A group performance, such as the bubble sort
dance in the example above, would be an example of many storytellers (the
dancers) conveying data to many recipients (the audience in the dance studio).

Audience has the below values:

  * •

One-to-one (1:1): one storyteller and one recipient, such as in a private
conversation.

  * •

One-to-many (1:N): one storyteller and many recipients, such as a speaker
giving a talk to a group.

  * •

Many-to-one (N:1): many storytellers and one recipient, such as an entire
staff briefing a commander.

  * •

Many-to-many (N:N): many storytellers and many recipients, such as a dance
troupe giving a performance to an audience.

###  4.2 Space and Time (S/T)

We borrow the notion of space and time from CSCW, [6] where the space-time
matrix has long been used to characterize forms of groupware based on the
spatial and temporal relations of the human users. It is a useful property
because both space and time have a significant impact on the delivery and
storage mechanism for the data-driven story.

Space and time has two values, one for each dimension:

  * •

Space: relative physical locations of storyteller and recipient.

    * –

Co-located (coloc): the storyteller and the recipient are in the same physical
space.

    * –

Distributed (distr): the storyteller and the recipient are not in the same
physical space.

  * •

Time: temporal locations of storyteller and recipient.

    * –

Synchronous (sync): the storyteller is delivering the story to the recipient
in real time.

    * –

Distributed (distr): the storyteller is delivering the story in a form that
the will be consumed by the recipient at a later time.

###  4.3 Media Components (MC)

The nature of the media captures the composition of the media being used for
data-driven storytelling. Since we are typically talking about composite media
types, the Media Components is a set variable that can include one or several
of the below:

  * •

Audio (aud): audio, such as speech recordings, ambient noise, or sampled sound
effects.

  * •

Music (mus): ordered sound forming a musical piece.

  * •

Photographs (pho): pixmap images.

  * •

Live video (vid): animated pixmap images.

  * •

Static graphics (gra): non-dynamic vector graphics.

  * •

Animated graphics (ani): dynamic vector graphics.

  * •

Text (txt): textual representations.

###  4.4 Data Components (DC)

The core purpose of a data-driven storytelling artifact is to convey data from
the storyteller to the viewer. The form that this takes is captured in the
Data Components (DC) dimension. It is a set variable that will take or several
of the below values:

  * •

Table (tab): data table representation;

  * •

Map (map): spatial maps;

  * •

Statistical graphics (stat): classic statistical graphics such as barcharts,
scatterplots, and linecharts;

  * •

Discrete event visualization (event): event timelines;

  * •

Continuous time visualization (time): time-series data;

  * •

Graph visualization (graph): network visualizations, such as node-link or
adjacency matrix representations; and

  * •

Text visualization (txtvis): textual visualizations, such as wordclouds or
visual concordances.

###  4.5 Viewing Sequence (SQ)

The level of interactivity associated with a storytelling artifact governs its
level of engagement, cognitive load, and adaptiveness. This variable takes one
of the following values:

  * •

No interaction (noint): the artifact cannot be interacted with. It will play
out from start to finish.

  * •

Playback control (playback): the viewer can stop and rewind or at least
restart the narrative.

  * •

Navigation (nav): the artifact allows the user to zoom and pan around in the
representation.

  * •

Basic control (basic): the artifact allows the user to click and focus on part
of the representation to highlight or trigger other effects.

  * •

Full control (control): the artifact yields full filtering, linking, and
transformation control to the user.

##  5 Applications

Given our taxonomy, we here explore a few designs for data-driven storytelling
in detail.

###  5.1 Sequential Art for Data: Data Comics

Data comics is an approach to show how sequential art [36]—also known as
comics—can be used as a novel method for storytelling. Several researchers
have studied this phenomenon, including Zhao et al., [60, 61, 62] Bach et al.,
[4, 5] and Kim et al. [26] We here discuss a practical system for building
data comics.

This data comics authoring system allows the users to build narratives using
comic layouts of panels that contain both images, text, figures, and live
visualizations. Comic features, such as motion lines, captions, thought and
speech bubbles, build on the universal language of comics. Our corresponding
implementation provides an authoring system for creating a data comic by
clipping images, data, and visualizations using a web browser and combining
them into the comic layout.

Figure 15: Data Comic on the European debt crisis.

Figure 15 is an example of a data comic produced using our prototype
implementation by clipping sources and images from the web. More specifically,
the designer starts by collecting the data visualizations and other material.
The designer then crops subgraphs and splits them across panels to allow the
reader to focus on one part at a time (rather than seeing the entire thing at
once).

In terms of our taxonomy, Data Comics is designed for both entertainment,
information, and education. The author cardinality (A) is one-to-many, since
that a comics is typically designed to be viewed on a computer or personal
device, and thus it has a distributed and asynchronous space and time (S/T)
value. Comics are generally not interactive, but they do support navigation.
The media components include static graphics, and text, whereas the data
components include virtually any static visualization element. Comics are
permanent in that they can be stored and transmitted easily.

###  5.2 Live-Streaming Data Analysis: DataTV

DataTV is a prototype system for authoring live-streaming data videos using a
single, integrated desktop interface. The approach is based on the notion that
live-streamed video has not yet been explored for data-driven storytelling.
The prototype (Figure 16) supports three separate modes for (1) production,
(2) recording, and (3) editing, in a highly streamlined and optimized workflow
that allows a single content creator to control the entire process, even
during live streaming. The tool incorporates multimedia sources such as live
webcams, live audio recordings, web browsers, image viewers, and full-motion
video. In particular, it supports live recording of any selected window on the
user’s desktop, such as those containing an interactive visualization, such as
a web browser or dedicated application window. Furthermore, the tool
incorporates advanced video functionality, such as chroma-keying (making parts
of a stream transparent, such as for blue or green screens), picture-in-
picture, hand-drawn annotations (for highlighting important parts of a
stream), viewport control (zooming and panning), and advanced source
composition operations (transitions, stretching, and fitting).

Figure 16: Webcam and Keshif visualization being recorded for the Nobel Prize
Winner data video. A data video can be made with user as the presenter.

In terms of our taxonomy, Data TV is designed for both entertainment,
information, and education. The author cardinality (A) is one-to-many, since
that a live streaming is typically designed to be viewed on a computer or
personal device, and thus it has a distributed and synchronous space and time
(S/T) value. Live streaming generally does not support interaction (unless the
streamer provides a chat channel for viewers to give feedback), so the viewing
sequence is playback only. The media components include audio, live video,
static graphics, and text, whereas the data components include virtually any
static visualization element. Live streaming is generated once, and the
interaction cannot be replicated.

###  5.3 Supporting Oral Tradition: GestureStory

Inspired by our taxonomy and its gaps, we propose a third application for
data-driven storytelling: body language and gestures. Body language has long
been a part of storytelling, but it is only recently where modern technology
has enabled people to use their gestures and entire bodies to control digital
devices. What if we could use data-driven storytelling to support a speaker
talking to an audience, simply by using the speaker’s body language and
gestures as input to control the audio-visual material (slides and data
visualizations) used to support the presentation? Inspired by Lee et al.’s
work on SketchStory, [34] where pen input drives the narrative, we call this
idea GestureStory.

A GestureStory tool would enable the presenter or even the audience to control
the visual components of a presentation consisting of data visualizations
using their gestures and voices. The application would presumably use a depth
camera and microphone to recognize body movements and spoken words. These can
be used to control interactive operations such as panning, zooming, moving,
annotating, and etc. The visual components can be made to follow the gesture
controls while textual annotations can be added by transcribing the
presenter’s speech.

In terms of our taxonomy, a gesture story tool is designed for entertainment,
information, and education. The author cardinality is one-to-many, since a
presentation is typically designed to be viewed in the same location as the
presenter while being presented, and thus it has a distributed and synchronous
space and time (S/T) value. The benefit of a live presentation is that the
audience can interact with the presenter by asking questions, so this media
supports at least basic interaction control. The media components include
audio, static graphics, animation, and text, whereas the data components
include table most of the static visualization element. Gesture stories are
generated with the author’s improvisations, and thus can not be replicated
easily. We classify it as part of our “data-driven performance” category.

##  6 Implications for Design

We have successfully labeled some thirty representative examples using our
taxonomy (see Table 1). The examples cover movies, documentaries, web articles
with data visualization, infographics, comics, social media, visualization
tools, games, dance, and sketching tools, which are a large part of the major
categories for storytelling. This comprehensive classification provides good
evidence that our taxonomy is sufficient and complete.

Our taxonomy is an extension that builds on the foundation that Segel and Heer
[49] laid in 2010. While this foundation has proven instrumental in the
guiding the development of data-driven storytelling, their model is limited in
scope and conflates the delivery mechanism with the media used for the
message. We believe our taxonomy provides a more comprehensive view of media
for data-driven storytelling while still building on their foundational work.
Using our taxonomy, designers will be able to widen the horizon of data-driven
storytelling. By providing a taxonomy with detailed dimensions, we explored
the possible values for each dimension. By expanding the list of dimensions
and dimensions values, we can also keep tracking of the emerging media types.
In particular, the terms and dimensions in our taxonomy provides a
standardized vocabulary to use when discussing data-driven storytelling. This
enables researchers and practitioners alike to classify their own work so that
existing and new media can be systematically organized with a common ground.

However, the true value of a taxonomy such as ours is in generating new ideas
by identifying gaps in the literature. By grouping and labeling the dimensions
of existing media, our taxonomy can help researchers identify new areas to
explore in the future. For example, this new design space can be generated by
exploring previously untested combinations of dimensions. We have done so in
the previous section: the Data Comics authoring system as well as the Data TV
and GestureStory applications are all based on our taxonomy.

A common theme for many of the novel storytelling methods such as Distill,
Idyll, and even some data comics is interactivity. This clearly goes in hand
with the old adage attributed to Confucius that ”I hear and I forget/I see and
I remember/I do and I understand.” In other words, learning is best scaffolded
by interactivity that allows the learner to change parameters and study its
effects. An concrete example is Omar Shehata’s Parametric Press article
“Unraveling the JPEG”888https://parametric.press/issue-01/unraveling-the-
jpeg/, which provides many interactive tools for exploring the surprising
depths of the JPEG image format. It is clear that the future of data-driven
storytelling will include interactive media.

A particular such interactive medium is video games. We have already surveyed
some examples of data-driven storytelling in games (or created through the use
of games in what is known as Machinima), and we think that there is
significant potential for this in the future. In particular, games for
education and/or games in augmented reality may be particularly fertile
grounds for adoption data-driven storytelling.

Let us close on Marshall McLuhan’s famous note that “the medium is the
message.” [37] It is clear that the medium used to convey a story will also
affect the message and content of that story. This is not only true because
the medium we choose affects the audience that will be able to consume it, but
also that the unique aspects of the medium gives specific capabilities to the
storyteller. We look forward to seeing how the data-driven storytelling
community will explore such novel media in the future.

##  7 Conclusion and Future Work

We have proposed a taxonomy of media for data-driven storytelling for the
purpose of widening the discourse on which storytelling techniques can be used
for telling stories about data. Our work began with a survey of the wide range
of evidence of such novel data-driven stories, resulting in us identifying
several representative dimensions to categorize these media types. We then
defined our taxonomy and used it to classify a large collection of novel data
stories. Based on these ideas, we propose several novel applications for data-
driven storytelling, and then close the paper with some take-aways for
designers looking to expand their repertoire in this domain.

In the future, we anticipate continuing our investigations into novel methods
for data-driven storytelling. We believe that this taxonomy will be helpful in
guiding our efforts in this endeavor.

## Acknowledgments

This work was partially supported by the U.S. National Science Foundation
award IIS-1539534. Any opinions, findings, and conclusions or recommendations
expressed in this material are those of the authors and do not necessarily
reflect the views of the funding agency.

## References

  * Amini et al. [2015] Fereshteh Amini, Nathalie Henry Riche, Bongshin Lee, Christophe Hurter, and Pourang Irani.  Understanding data videos: Looking at narrative visualization through the cinematography lens.  In _Proceedings of the ACM Conference on Human Factors in Computing Systems_ , pages 1459–1468, New York, NY, USA, 2015. ACM.  doi: 10.1145/2702123.2702431.
  * Amini et al. [2017] Fereshteh Amini, Nathalie Henry Riche, Bongshin Lee, Andres Monroy-Hernández, and Pourang Irani.  Authoring data-driven videos with dataclips.  _IEEE Transactions on Visualization and Computer Graphics_ , 23(1):501–510, 2017.  doi: 10.1109/TVCG.2016.2598647.
  * Anderson [2011] Sam Anderson.  Watching people watching people watching.  New York Times Magazine, 2011.
  * Bach et al. [2016] Benjamin Bach, Natalie Kerracher, Kyle Wm. Hall, Sheelagh Carpendale, Jessie Kennedy, and Nathalie Henry Riche.  Telling stories about dynamic networks with Graph Comics.  In _Proceedings of the ACM Conference on Human Factors in Computing Systems_ , pages 3670–3682, New York, NY, USA, 2016. ACM.  doi: 10.1145/2858036.2858387.
  * Bach et al. [2017] Benjamin Bach, Nathalie Henry Riche, Sheelagh Carpendale, and Hanspeter Pfister.  The emerging genre of data comics.  _IEEE Computer Graphics & Applications_, 38(3):6–13, 2017.  doi: 10.1109/MCG.2017.33.
  * Baecker [1993] Ronald M. Baecker.  _Readings in Groupware and Computer-Supported Cooperative Work_.  Morgan Kaufmann Publishers, San Francisco, CA, USA, 1993.
  * Bostock et al. [2011] Michael Bostock, Vadim Ogievetsky, and Jeffrey Heer.  D3: Data-driven documents.  _IEEE Transactions on Visualization and Computer Graphics_ , 17(12):2301–2309, 2011.
  * Bowman et al. [2012] Brian Bowman, Niklas Elmqvist, and T. J. Jankun-Kelly.  Toward visualization for games: Theory, design space, and patterns.  _IEEE Transactions on Visualization and Computer Graphics_ , 18(11):1956–1968, 2012.  doi: 10.1109/TVCG.2012.77.
  * Boy et al. [2014] Jeremy Boy, Ronald A. Rensink, Enrico Bertini, and Jean-Daniel Fekete.  A principled way of assessing visualization literacy.  _IEEE Transactions on Visualization and Computer Graphics_ , 20(12):1963–1972, 2014.
  * Card et al. [1999] Stuart K. Card, Jock D. Mackinlay, and Ben Shneiderman.  _Readings in Information Visualization: Using Vision to Think_.  Morgan Kaufmann, San Francisco, CA, USA, 1999.
  * Chevalier et al. [2016] Fanny Chevalier, Nathalie Henry Riche, Catherine Plaisant, Amira Chalbi, and Christophe Hurter.  Animations 25 years later: New roles and opportunities.  In _Proceedings of the ACM Conference on Advanced Visual Interfaces_ , pages 280–287, New York, NY, USA, 2016. ACM.  doi: 10.1145/2909132.2909255.
  * Chi et al. [2012] Pei-Yu Chi, Sally Ahn, Amanda Ren, Mira Dontcheva, Wilmot Li, and Björn Hartmann.  MixT: automatic generation of step-by-step mixed media tutorials.  In _Proceedings of the ACM Symposium on User Interface Software and Technology_ , pages 93–102, New York, NY, USA, 2012. ACM.  doi: 10.1145/2380116.2380130.
  * Conlen and Heer [2018] Matthew Conlen and Jeffrey Heer.  Idyll: A markup language for authoring and publishing interactive articles on the web.  In _Proceedings of the ACM Symposium on User Interface Software and Technology_ , pages 977–989, New York, NY, USA, 2018. ACM.  doi: 10.1145/3242587.3242600.
  * Diakopoulos et al. [2011] Nick Diakopoulos, Joan DiMicco, Jessica Hullman, Karrie Karahalios, and Adam Perer.  Telling stories with data: The next chapter—a visweek 2011 workshop, 2011.
  * DiMicco et al. [2010] Joan DiMicco, Matt McKeon, and Karrie Karahalios.  Telling stories with data—a visweek 2010 workshop, 2010.
  * Dragicevic et al. [2019] Pierre Dragicevic, Yvonne Jansen, Abhraneel Sarma, Matthew Kay, and Fanny Chevalier.  Increasing the transparency of research papers with explorable multiverse analyses.  In _Proceedings of the ACM Conference on Human Factors in Computing Systems_ , pages 65:1–65:15, New York, NY, USA, 2019. ACM.  doi: 10.1145/3290605.3300295.
  * Eisner [2008] Will Eisner.  _Graphic Storytelling and Visual Narrative_.  W. W. Norton & Company, New York, NY, USA, 2008.
  * Gershon and Page [2001] Nahum D. Gershon and Ward Page.  What storytelling can do for information visualization.  _Communications of the ACM_ , 44(8):31–37, 2001\.
  * Google [2019] Google.  Colaboratory, 2019.  URL https://colab.research.google.com/.
  * Gottschall [2012] Jonathan Gottschall.  _The Storytelling Animal: How Stories Make Us Human_.  Mariner Books, New York, NY, USA, 2012.
  * Griffith and Papacharissi [2010] Maggie Griffith and Zizi Papacharissi.  Looking for you: An analysis of video blogs.  _First Monday_ , 15(1), January 2010.
  * Grossman and Fitzmaurice [2010] Tovi Grossman and George W. Fitzmaurice.  ToolClips: an investigation of contextual video assistance for functionality understanding.  In _Proceedings of the ACM Conference on Human Factors in Computing Systems_ , pages 1515–1524, New York, NY, USA, 2010. ACM.  doi: 10.1145/1753326.1753552.
  * Görtler et al. [2019] Jochen Görtler, Rebecca Kehlbeck, and Oliver Deussen.  A visual exploration of gaussian processes.  _Distill_ , 2019.  doi: 10.23915/distill.00017.  https://distill.pub/2019/visual-exploration-gaussian-processes.
  * Harris [1999] Robert L. Harris.  _Information Graphics: A Comprehensive Illustrated Reference_.  Oxford University Press, Oxford, United Kingdom, 1999.
  * Hullman and Diakopoulos [2011] Jessica Hullman and Nicholas Diakopoulos.  Visualization rhetoric: Framing effects in narrative visualization.  _IEEE Transactions on Visualization and Computer Graphics_ , 17(12):2231–2240, 2011.
  * Kim et al. [2019] Nam Wook Kim, Nathalie Henry Riche, Benjamin Bach, Guanpeng Xu, Matthew Brehmer, Ken Hinckley, Michel Pahud, Haijun Xia, Michael J. McGuffin, and Hanspeter Pfister.  DataToon: Drawing dynamic network comics with pen + touch interaction.  In Stephen A. Brewster, Geraldine Fitzpatrick, Anna L. Cox, and Vassilis Kostakos, editors, _Proceedings of the ACM Conference on Human Factors in Computing Systems_ , pages 105:1–105:12, New York, NY, USA, 2019\. ACM.  doi: 10.1145/3290605.3300335.
  * Kluyver et al. [2016] Thomas Kluyver, Benjamin Ragan-Kelley, Fernando Pérez, Brian E. Granger, Matthias Bussonnier, Jonathan Frederic, Kyle Kelley, Jessica B. Hamrick, Jason Grout, Sylvain Corlay, Paul Ivanov, Damián Avila, Safia Abdalla, and Carol Willing.  Jupyter notebooks – a publishing format for reproducible computational workflows.  In _Positioning and Power in Academic Publishing: Players, Agents and Agendas_ , pages 87–90, Amsterdam, Netherlands, 2016. IOS Press.  doi: 10.3233/978-1-61499-649-1-87.
  * Knuth [1984] Donald E. Knuth.  Literate programming.  _The Computer Journal_ , 27(2):97–111, 1984.  doi: 10.1093/comjnl/27.2.97.
  * Kosara [2013] Robert Kosara.  Story points in Tableau Software.  Keynote at Tableau Customer Conference, September 2013.
  * Kosara and Mackinlay [2013] Robert Kosara and Jock D. Mackinlay.  Storytelling: The next step for visualization.  _IEEE Computer_ , 46(5):44–50, 2013.
  * Kwon et al. [2014] Bum Chul Kwon, Florian Stoffel, Dominik Jäckle, Bongshin Lee, and Daniel Keim.  VisJockey: Enriching data stories through orchestrated interactive visualization.  In _Proceedings of the Computation+Journalism Symposium_ , pages 1–5, New York, NY, USA, 2014. Brown Institute for Media Innovation.
  * Latulipe and Huskey [2008] Celine Latulipe and Sybil Huskey.  Dance.draw: exquisite interaction.  In _Proceedings of the British HCI Group Annual Conference on HCI_ , pages 47–51, 2008.  URL https://dl.acm.org/citation.cfm?id=1531839.
  * Latulipe et al. [2011] Celine Latulipe, David Wilson, Sybil Huskey, Berto Gonzalez, and Melissa Word.  Temporal integration of interactive technology in dance: Creative process impacts.  In _Proceedings of ACM Creativity & Cognition_, pages 107–116, New York, NY, USA, 2011. ACM.  doi: 10.1145/2069618.2069639.
  * Lee et al. [2013] Bongshin Lee, Rubaiat Habib Kazi, and Greg Smith.  SketchStory: Telling more engaging stories with data through freeform sketching.  _IEEE Transactions on Visualization and Computer Graphics_ , 19(12):2416–2425, 2013.  doi: 10.1109/TVCG.2013.191.
  * Leitch [1986] Thomas M. Leitch.  _What Stories Are: Narrative Theory and Interpretation_.  Pennsylvania State University Press, University Park, PA, 1986.
  * McCloud [1994] Scott McCloud.  _Understanding Comics: The Invisible Art_.  William Morrow Paperbacks, 1994.
  * McLuhan [1964] Marshall McLuhan.  _Understanding Media: The Extensions of Man_.  McGraw-Hill Education, 1964.
  * Nguyen and Liu [2015] Cuong Nguyen and Feng Liu.  Making software tutorial video responsive.  In _Proceedings of the ACM Conference on Human Factors in Computing Systems_ , pages 1565–1568, New York, NY, USA, 2015. ACM.  doi: 10.1145/2702123.2702209.
  * Nowakowski et al. [2011] Piotr Nowakowski, Eryk Ciepiela, Daniel Harezlak, Joanna Kocot, Marek Kasztelnik, Tomasz Bartynski, Jan Meizner, Grzegorz Dyk, and Maciej Malawski.  The collage authoring environment.  In _Proceedings of the International Conference on Computational Science_ , pages 608–617, 2011.  doi: 10.1016/j.procs.2011.04.064.
  * ObservableHQ [2018] ObservableHQ.  Observable, 2018.  URL https://observablehq.com.  Accessed September 2019.
  * Pongnumkul et al. [2011] Suporn Pongnumkul, Mira Dontcheva, Wilmot Li, Jue Wang, Lubomir D. Bourdev, Shai Avidan, and Michael F. Cohen.  Pause-and-play: automatically linking screencast video tutorials with applications.  In _Proceedings of the ACM Symposium on User Interface Software and Technology_ , pages 135–144, New York, NY, USA, 2011. ACM.  doi: 10.1145/2047196.2047213.
  * Pousman et al. [2007] Zachary Pousman, John T. Stasko, and Michael Mateas.  Casual information visualization: Depictions of data in everyday life.  _IEEE Transactions on Visualization and Computer Graphics_ , 13(6):1145–1152, 2007.
  * Rädle et al. [2017] Roman Rädle, Midas Nouwens, Kristian Antonsen, James R. Eagan, and Clemens N. Klokmose.  Codestrates: Literate computing with webstrates.  In _Proceedings of the ACM Symposium on User Interface Software and Technology_ , pages 715–725, New York, NY, USA, 2017. ACM.  doi: 10.1145/3126594.3126642.
  * Ren et al. [2017] Donghao Ren, Matthew Brehmer, Bongshin Lee, Tobias Höllerer, and Eun Kyoung Choe.  ChartAccent: Annotation for data-driven storytelling.  In _Proceedings of the IEEE Pacific Symposium on Visualization_ , pages 230–239, Piscataway, NJ, USA, 2017. IEEE.  doi: 10.1109/PACIFICVIS.2017.8031599.
  * Rule et al. [2019] Adam Rule, Amanda Birmingham, Cristal Zuniga, Ilkay Altintas, Shih-Cheng Huang, Rob Knight, Niema Moshiri, Mai H. Nguyen, Sara Brin Rosenthal, Fernando Pérez, and Peter W. Rose.  Ten simple rules for writing and sharing computational analyses in Jupyter Notebooks.  _PLOS Computational Biology_ , 15(7):1–8, 07 2019\.  doi: 10.1371/journal.pcbi.1007007.
  * Satyanarayan and Heer [2014] Arvind Satyanarayan and Jeffrey Heer.  Authoring narrative visualizations with Ellipsis.  _Computer Graphics Forum_ , 33(3):361–370, 2014\.  doi: 10.1111/cgf.12392.
  * Schank and Abelson [1995] Roger C. Schank and Robert P. Abelson.  Knowledge and memory: The real story.  In Jr. Robert S. Wyer, editor, _Advances in Social Cognition_ , volume 8, pages 1–85, Hillsdale, NJ, USA, 1995. Lawrence Erlbaum Associates.
  * Schmalstieg and Höllerer [2016] Dieter Schmalstieg and Tobias Höllerer.  _Augmented Reality: Principles and Practice_.  Addison-Wesley, Boston, MA, USA, 2016.
  * Segel and Heer [2010] Edward Segel and Jeffrey Heer.  Narrative visualization: Telling stories with data.  _IEEE Transactions on Visualization and Computer Graphics_ , 16(6):1139–1148, 2010.  doi: 10.1109/TVCG.2010.179.
  * Sless [1981] David Sless.  _Learning and Visual Communication_.  Wiley, New York, NY, USA, 1981.
  * Spagnuolo and Veltkamp [2013] Michela Spagnuolo and Remco C. Veltkamp.  Special issue on executable papers for 3d object retrieval.  _Computers & Graphics_, 37(5):A7–A8, 2013\.  doi: 10.1016/j.cag.2013.04.006.
  * Tabard et al. [2008] Aurélien Tabard, Wendy E. Mackay, and Evelyn Eastmond.  From individual to collaborative: the evolution of Prism, a hybrid laboratory notebook.  In _Proceedings of the ACM Conference on Computer Supported Cooperative Work_ , pages 569–578, New York, NY, USA, 2008. ACM.  doi: 10.1145/1460563.1460653.
  * Thomas and Cook [2005] James J. Thomas and Kristin A. Cook.  _Illuminating the Path: The Research and Development Agenda for Visual Analytics_.  IEEE Computer Society, Los Alamitos, CA, USA, 2005.
  * Tukey [1977] John W. Tukey.  _Exploratory Data Analysis_.  Addison-Wesley, Reading, MA, USA, 1977.
  * Vansina [1985] Jan Vansina.  _Oral Tradition as History_.  University of Wisconsin Press, Madison, WI, USA, 1985.
  * Victor [2011] Bret Victor.  Explorable explorations.  http://worrydream.com/ExplorableExplanations/, 2011.
  * Viégas and Wattenberg [2006] Fernanda Viégas and Martin Wattenberg.  Communication-minded visualization: A call to action.  _IBM Systems Journal_ , 45(4):801–812, 2006.  doi: 10.1147/sj.454.0801.
  * Wadeson [2013] Danny Wadeson.  GamerTube: PewDiePie and the YouTube Commentary Revolution.  Polygon, September 6 2013.
  * Wickham and Grolemund [2017] Hadley Wickham and Garrett Grolemund.  _R for Data Science: Import, Tidy, Transform, Visualize, and Model Data_.  O’Reilly Media, Sebastopol, CA, USA, 2017.
  * Zhao et al. [2015a] Zhenpeng Zhao, William Benjamin, Niklas Elmqvist, and Karthik Ramani.  Sketcholution: Interaction histories for sketching.  _International Journal of Human-Computer Studies_ , 82:11–20, 2015a.  doi: 10.1016/j.ijhcs.2015.04.003.
  * Zhao et al. [2015b] Zhenpeng Zhao, Rachael Marr, and Niklas Elmqvist.  Data comics: Sequential art for data-driven storytelling.  Technical Report 15, Human-Computer Interaction Laboratory, University of Maryland, College Park, 2015b.
  * Zhao et al. [2019] Zhenpeng Zhao, Rachael Marr, Jason Shaffer, and Niklas Elmqvist.  Understanding partitioning and sequence in data-driven storytelling.  In _Proceedings of the iConference_ , volume 11420 of _Lecture Notes in Computer Science_ , pages 327–338, Cham, Switzerland, 2019\. Springer.  doi: 10.1007/978-3-030-15742-5“˙32.
  * Zoltán and László [2017] Kátai Zoltán and Tóth László.  Sorting algorithms shown by dance, 2017.  http://www.pdviz.com/different-sorting-algorithm-demonstrated-with.


# On the symmetry of the finitistic dimension

Henning Krause  Fakultät für Mathematik
Universität Bielefeld
D-33501 Bielefeld
Germany<EMAIL_ADDRESS>Dedicated to the memory of Helmut
Lenzing.

(Date: 11 November, 2022)

###### Abstract.

For any ring we propose the construction of a cover which increases the
finitistic dimension on one side and decreases the finitistic dimension to
zero on the opposite side. This complements recent work of Cummings.

###### Key words and phrases:

Finitistic dimension

###### 2020 Mathematics Subject Classification:

16E10

The finitistic dimension is a homological invariant of a ring which is
conjectured to be finite when the ring is a finite dimensional algebra over a
field [2]. In recent work [3] Cummings introduces for any finite dimensional
algebra a related algebra; its purpose is to increase the finitistic dimension
on one side and to decrease the finitistic dimension to zero on the opposite
side. In this note we propose the construction of such an _asymmetric cover_
for any ring and we establish the same properties. This specialises to
Cummings’ construction for finite dimensional algebras over an algebraically
closed field and yields examples of rings such that the finitistic dimension
is infinite while the finitistic dimension of the opposite ring is zero. We
need to distinguish between the small and the big finitistic dimension but our
results cover both cases.

* * *

Let $A$ be an associative ring. We consider the category of (right)
$A$-modules and identify left $A$-modules with modules over the opposite ring
$A^{\mathrm{op}}$. For a module $M$ we write $\operatorname{rad}M$ for its
_radical_ and $\operatorname{soc}M$ for its _socle_. We set
$\operatorname{top}M=M/\operatorname{rad}M$. The functor

$(-)^{*}:=\operatorname{Hom}_{A}(-,A)$

yields a duality between right and left $A$-modules. We consider the _trivial
extension_

$T(A):=A\ltimes A^{\natural}$

which is given by the bimodule $A^{\natural}:={{}_{A}A_{A}}$. This ring is by
definition the abelian group $T(A)=A\oplus A^{\natural}$ with multiplication
given by the formula

$(x,y)\cdot(x^{\prime},y^{\prime})=(xx^{\prime},xy^{\prime}+yx^{\prime}).$

Note that $A^{\natural}$ is a two-sided ideal with
$T(A)/A^{\natural}\xrightarrow{\raisebox{-1.72218pt}[0.0pt][0.0pt]{$\scriptstyle{\sim}$}}A$.

######  Lemma 1.

Let $A$ be a semisimple ring. Then

$\operatorname{rad}T(A)=A^{\natural}=\operatorname{soc}T(A)\qquad\text{and}\qquad\operatorname{top}T(A)\cong\operatorname{soc}T(A).$

###### Proof.

The first assertion is clear. Left multiplication with $(0,1)$ gives a map
$T(A)\to T(A)$ which induces an isomorphism
$\operatorname{top}T(A)\xrightarrow{\raisebox{-1.72218pt}[0.0pt][0.0pt]{$\scriptstyle{\sim}$}}\operatorname{soc}T(A)$
∎

For a ring $A$ we denote by $\Sigma(A)$ the set of isomorphism classes of
simple $A$-modules. Set

$\bar{S}:=\coprod_{S\in\Sigma(A)}S\qquad\text{and}\qquad\bar{A}:=\prod_{S\in\Sigma(A)}T(\operatorname{End}_{A}(S)).$

We view $\bar{S}$ as $\bar{A}$-$A$-bimodule, with left action via
$T(\operatorname{End}_{A}(S))\twoheadrightarrow\operatorname{End}_{A}(S)$ for
each $S$ in $\Sigma(A)$, and consider the triangular matrix ring

$\tilde{A}:=\begin{bmatrix}A&0\\\ \bar{S}&\bar{A}\end{bmatrix}.$

The idempotents $e=\left[\begin{smallmatrix}1&0\\\
0&0\end{smallmatrix}\right]$ and $f=\left[\begin{smallmatrix}0&0\\\
0&1\end{smallmatrix}\right]$ provide an $\tilde{A}$-module decomposition
$\tilde{A}=P\oplus Q$, where

$P:=e\tilde{A}\cong A\qquad\text{and}\qquad
Q:=f\tilde{A}\cong\bar{S}\oplus\bar{A}.$

We call $\tilde{A}$ a _cover_ of $A$ because the idempotent $e\in\tilde{A}$
yields an isomorphism

$\operatorname{End}_{\tilde{A}}(P)\cong e\tilde{A}e\cong A.$

The following lemma expresses the distinct property of the cover $\tilde{A}$,
namely that every simple $\tilde{A}$-module embeds into $\tilde{A}$.

######  Lemma 2.

We have $S^{*}\neq 0$ for every simple $\tilde{A}$-module $S$.

###### Proof.

We claim that each simple $\tilde{A}$-module arises as the image of a morphism
$\tilde{A}=P\oplus Q\to Q$, using that

$\operatorname{Hom}_{\tilde{A}}(P,Q)\cong
e\tilde{A}f=\bar{S}\qquad\text{and}\qquad\operatorname{Hom}_{\tilde{A}}(Q,Q)\cong
f\tilde{A}f=\bar{A}.$

With Lemma 1 we compute $\operatorname{soc}Q$ and obtain a decomposition into
simples:

$\operatorname{soc}Q=\bar{S}\oplus\operatorname{soc}\bar{A}\cong\coprod_{S\in\Sigma(A)}\big{(}S\oplus\operatorname{End}_{A}(S)^{\natural}\big{)}.$

A simple $\tilde{A}$-module $T$ comes either with a nonzero map $P\to T$ or a
nonzero map $Q\to T$. In the first case $T$ identifies with a simple
$A$-module via the inclusion $A\to\tilde{A}$ given by
$x\mapsto\left[\begin{smallmatrix}x&0\\\ 0&1\end{smallmatrix}\right]$, and
therefore with a summand of $\bar{S}\subseteq\operatorname{soc}Q$. In the
second case $T$ identifies with a simple $\bar{A}$-module via the inclusion
$\bar{A}\to\tilde{A}$ given by $x\mapsto\left[\begin{smallmatrix}1&0\\\
0&x\end{smallmatrix}\right]$, and therefore with a summand of
$\operatorname{soc}\bar{A}\subseteq\operatorname{soc}Q$. In any case one
obtains a monomomorphism $T\to Q\hookrightarrow\tilde{A}$. ∎

Let $\mathcal{P}(A)$ denote the class of $A$-modules $M$ that admit a finite
resolution

$0\longrightarrow P_{n}\longrightarrow\cdots\longrightarrow
P_{1}\longrightarrow P_{0}\longrightarrow M\longrightarrow 0$

with all $P_{i}$ finitely generated projective. We denote by

$\operatorname{fin{.}dim}A:=\sup\\{\operatorname{proj{.}dim}M\mid
M\in\mathcal{P}(A)\\}$

the _small finitistic dimension_ of $A$; this is a slight variation of the
usual definition which seems natural as the modules in $\mathcal{P}(A)$ are
precisley the ones which become compact (or perfect) when viewed as an object
in the derived category of $A$.

The following lemma is [5, Lemma 7.2.8] and its proof is sketched for the
convenience of the reader; cf. the discussion in [2, §5].

######  Lemma 3.

For a ring $A$ we have $\operatorname{fin{.}dim}A=0$ if and only if $M^{*}\neq
0$ for every finitely presented $A^{\mathrm{op}}$-module $M$.

###### Proof.

Let $\operatorname{proj}A$ denote the category of finitely generated
projective $A$-modules. The condition $\operatorname{fin{.}dim}A=0$ means that
every monomorphism in $\operatorname{proj}A$ splits. The duality

$(-)^{*}\colon(\operatorname{proj}A)^{\mathrm{op}}\xrightarrow{\
\raisebox{-1.72218pt}[0.0pt][0.0pt]{$\scriptstyle{\sim}$}\
}\operatorname{proj}(A^{\mathrm{op}})$

translates this into the condition on finitely presented
$A^{\mathrm{op}}$-modules. ∎

######  Theorem 4.

For a ring $A$ we have

$\operatorname{fin{.}dim}\tilde{A}\geq\operatorname{fin{.}dim}A\qquad\text{and}\qquad\operatorname{fin{.}dim}\tilde{A}^{\mathrm{op}}=0.$

###### Proof.

The idempotent $e\in\tilde{A}$ with $e\tilde{A}e\cong A$ gives rise to a fully
faithful functor

$-\otimes_{A}e\tilde{A}\colon\operatorname{Mod}A\longrightarrow\operatorname{Mod}\tilde{A}$

which is exact and maps projectives to projectives; also it preserves finite
generation. This yields the first assertion. The second assertion follows from
Lemma 3 because we have $M^{*}\neq 0$ for every finitely presented
$\tilde{A}$-module $M$ by Lemma 2. ∎

* * *

There is a somewhat more natural construction of a cover when the ring $A$ is
semilocal. Recall that $A$ is _semilocal_ if the ring $A/\operatorname{rad}A$
is semisimple. In this case we have an idempotent $\varepsilon\in
A/\operatorname{rad}A$ and a Morita equivalence

$A/\operatorname{rad}A\;\sim\;\varepsilon(A/\operatorname{rad}A)\varepsilon\;\cong\;\prod_{S\in\Sigma(A)}\operatorname{End}_{A}(S).$

We set

$\widetilde{A}:=\begin{bmatrix}A&0\\\
A/\operatorname{rad}A&T(A/\operatorname{rad}A)\end{bmatrix}$

and this is closely related to the cover $\tilde{A}$ via the idempotent
$\tilde{\varepsilon}=\left[\begin{smallmatrix}1&0\\\
0&(\varepsilon,0)\end{smallmatrix}\right]$.

######  Lemma 5.

For a semilocal ring $A$ we have a Morita equivalence

$\widetilde{A}\;\sim\;\tilde{\varepsilon}\widetilde{A}\tilde{\varepsilon}\;\cong\;\tilde{A}.$

###### Proof.

We use some general facts. Let $\Lambda=\left[\begin{smallmatrix}A&0\\\
M&B\end{smallmatrix}\right]$ be a triangular matrix ring and $e\in B$ an
idempotent such that $B$ and $eBe$ are Morita equivalent. Set
$\bar{e}=\left[\begin{smallmatrix}1&0\\\ 0&e\end{smallmatrix}\right]$. Then
$\Lambda$ and $\bar{e}\Lambda\bar{e}=\left[\begin{smallmatrix}A&0\\\
eM&eBe\end{smallmatrix}\right]$ are Morita equivalent. Also, the trivial
extensions $T(B)$ and $T(eBe)=(e,0)T(B)(e,0)$ are Morita equivalent. ∎

We may identify $\widetilde{A}$ with $\tilde{A}$, as we are mostly interested
in homological properties. In fact, $\widetilde{A}\cong\tilde{A}$ when $A$ is
semiperfect and basic. Note that the definition of $\widetilde{A}$ does not
depend on any choices. In particular, we have an identity
$\widetilde{A^{\mathrm{op}}}=\widetilde{A}^{\mathrm{op}}$.

Next we discuss some ring theoretic properties which are preserved under the
passage from $A$ to its cover $\tilde{A}$. Recall that a ring is _semiprimary_
if it is semilocal and its radical is a nilpotent ideal.

######  Remark 6.

If $A$ is semilocal then $\tilde{A}$ is semilocal. This follows from the
isomorphism

$\operatorname{top}\tilde{A}=\operatorname{top}P\oplus\operatorname{top}Q\xrightarrow{\raisebox{-1.72218pt}[0.0pt][0.0pt]{$\scriptstyle{\sim}$}}A/\operatorname{rad}A\oplus(A/\operatorname{rad}A)^{\natural}=\operatorname{soc}Q$

which is induced by the morphism $\tilde{A}=P\oplus Q\to
Q\hookrightarrow\tilde{A}$ given by left multiplication with
$\left[\begin{smallmatrix}0&0\\\ 1&(0,1)\end{smallmatrix}\right]$. Moreover,
in this case the inclusion $A\hookrightarrow\tilde{A}$ yields the identity
$(\operatorname{rad}A)^{n}=(\operatorname{rad}\tilde{A})^{n}$ for all $n>1$.

######  Remark 7.

If the ring $A$ is left or right perfect then the same holds for $\tilde{A}$.
This follows from Remark 6, since $A$ is right perfect if and only if $A$ is
semilocal and $\operatorname{rad}A$ is right T-nilpotent.

There is an analogue of Theorem 4 for the _big finitistic dimension_

$\operatorname{Fin{.}dim}A:=\sup\\{\operatorname{proj{.}dim}M\mid
M\in\operatorname{Mod}A,\,\operatorname{proj{.}dim}M<\infty\\}.$

We use the following fact which is a slight variation of [2, Theorem 6.3].

######  Proposition 8.

For a ring $A$ we have $\operatorname{Fin{.}dim}A=0$ if and only if $A$ is
right perfect and $\operatorname{fin{.}dim}A=0$.

###### Proof.

Suppose that $A$ is right perfect and $\operatorname{fin{.}dim}A=0$. We need
to show that every monomorphism $\phi\colon M\to N$ between projective
$A$-modules splits. This holds when $M$ and $N$ are finitely generated since
$\operatorname{fin{.}dim}A=0$. Because $A$ is right perfect, any projective
$A$-module decomposes into a direct sum of finitely generated modules and can
therefore be written as a filtered colimit of finitely generated direct
summands. Choose such a presentation $M=\operatorname*{colim}_{i}M_{i}$. Then
$\phi$ is a filtered colimit of split monomorphisms $\phi_{i}\colon M_{i}\to
N$, and therefore
$\operatorname*{colim}_{i}\operatorname{Coker}\phi_{i}\cong\operatorname{Coker}\phi$
is a filtered colimit of projectives. Thus $\operatorname{Coker}\phi$ is
projective and $\phi$ splits. For the other implication we refer to [2]. ∎

######  Theorem 9.

For a ring $A$ we have

$\operatorname{Fin{.}dim}\tilde{A}\geq\operatorname{Fin{.}dim}A.$

Moreover, $\operatorname{Fin{.}dim}\tilde{A}^{\mathrm{op}}=0$ if and only if
$A$ is left perfect.

###### Proof.

The first assertion is easily checked as in the proof of Theorem 4. If
$\operatorname{Fin{.}dim}\tilde{A}^{\mathrm{op}}=0$, then $\tilde{A}$ is left
perfect by Proposition 8, and this implies that $A$ is left perfect. For the
converse suppose that $A$ is left perfect. Then $\tilde{A}$ is left perfect by
Remark 7. Thus $\operatorname{Fin{.}dim}\tilde{A}^{\mathrm{op}}=0$ by Theorem
4 and Proposition 8. ∎

* * *

The preceding results demonstrate a failure of symmetry for the notion of
‘finite finitistic dimension’, as pointed out in the recent work of Cummings
[3]. In particular we have the following examples.

Recall that a noetherian ring is _regular_ if all its finitely generated
modules have finite projective dimension.

######  Corollary 10.

Let $A$ be a commutative noetherian ring that is regular of infinite Krull
dimension. Then

$\operatorname{fin{.}dim}\tilde{A}=\infty\qquad\text{and}\qquad\operatorname{fin{.}dim}\tilde{A}^{\mathrm{op}}=0.$

###### Proof.

The finitistic dimension $\operatorname{fin{.}dim}A$ is infinite by Theorem
1.6 and Corollary 1.7 in [1]. Thus the assertion follows from Theorem 4. ∎

Specific examples of regular rings of infinite Krull dimension have been
constructed by Nagata; cf. [5, Example 7.2.20].

We continue with an example due to Kirkman and Kuzmanovich [4]. Let $k$ be a
field and consider the quotient $\Lambda=kQ/I$ of the path algebra $kQ$ given
by the quiver

${Q:}$${\circ}$${\circ\quad(i\in\mathbb{N})}$$\scriptstyle{a_{i}}$$\scriptstyle{b_{i}}$

(with $k$-basis given by the paths in $Q$ and multiplication induced by the
composition of paths, where for any pair of paths $\alpha,\beta$ we write
$\beta\alpha$ for the composite when the terminal vertex of $\alpha$ equals
the initial vertex of $\beta$) modulo the ideal $I$ that is generated by the
elements

$b_{t}a_{s}b_{r}\quad(r,s,t\in\mathbb{N})\qquad
b_{t}a_{s}-a_{t}b_{t}\quad(t>s)\qquad a_{r}b_{r}\quad(r\in\mathbb{N}).$

Note that $\Lambda$ is a semiprimary ring with
$(\operatorname{rad}\Lambda)^{4}=0$.

######  Corollary 11.

The ring $\tilde{\Lambda}$ is semiprimary satisfying

$\operatorname{Fin{.}dim}\tilde{\Lambda}=\operatorname{fin{.}dim}\tilde{\Lambda}=\infty\qquad\text{and}\qquad\operatorname{Fin{.}dim}\tilde{\Lambda}^{\mathrm{op}}=\operatorname{fin{.}dim}\tilde{\Lambda}^{\mathrm{op}}=0.$

###### Proof.

From Remark 6 it follows that the ring $\tilde{\Lambda}$ is semilocal with
$(\operatorname{rad}\tilde{\Lambda})^{4}=0$. Thus $\tilde{\Lambda}$ is
semiprimary. In [4] it is shown that $\operatorname{fin{.}dim}\Lambda=\infty$.
Then the assertion follows from Theorem 9, using that $\Lambda$ is left
perfect. ∎

## References

  * [1] M. Auslander and D. A. Buchsbaum, Homological dimension in noetherian rings. II, Trans. Amer. Math. Soc. 88 (1958), 194–206.
  * [2] H. Bass, Finitistic dimension and a homological generalization of semi-primary rings, Trans. Amer. Math. Soc. 95 (1960), 466–488.
  * [3] C. Cummings, _Left-right symmetry of finite finitistic dimension_ , arXiv:2211.04394v1 [math.RT].
  * [4] E. Kirkman and J. Kuzmanovich, Algebras with large homological dimensions, Proc. Amer. Math. Soc. 109 (1990), no. 4, 903–906.
  * [5] H. Krause, Homological theory of representations, Cambridge Studies in Advanced Mathematics, 195, Cambridge University Press, Cambridge, 2022.


11institutetext:  1\. INRIA Paris, ARAMIS project-team, 75013 Paris, France
2\. Sorbonne Universités, UPMC Univ Paris 06, Inserm, CNRS, Institut du
cerveau et la moelle (ICM), AP-HP - Hôpital Pitié-Salpêtrière, 75013 Paris,
France
3\. Sorbonne Universités, UPMC Univ Paris 06, Inserm, CNRS, LIB, AP-HP, 75013
Paris, France
4\. INSEAD, Bd de Constance, 77305 Fontainebleau, France

# Yet Another ADNI Machine Learning Paper? Paving The Way Towards Fully-
reproducible Research on Classification of Alzheimer’s Disease

Jorge Samper-González1,2 Ninon Burgos1,2 Sabrina Fontanella1,2 Hugo Bertin3
Marie-Odile Habert3 Stanley Durrleman1,2 Theodoros Evgeniou4 Olivier
Colliot2,1 (🖂)  the Alzheimer’s Disease Neuroimaging Initiative

###### Abstract

In recent years, the number of papers on Alzheimer’s disease classification
has increased dramatically, generating interesting methodological ideas on the
use machine learning and feature extraction methods. However, practical impact
is much more limited and, eventually, one could not tell which of these
approaches are the most efficient. While over 90% of these works make use of
ADNI an objective comparison between approaches is impossible due to
variations in the subjects included, image pre-processing, performance metrics
and cross-validation procedures. In this paper, we propose a framework for
reproducible classification experiments using multimodal MRI and PET data from
ADNI. The core components are: 1) code to automatically convert the full ADNI
database into BIDS format; 2) a modular architecture based on Nipype in order
to easily plug-in different classification and feature extraction tools; 3)
feature extraction pipelines for MRI and PET data; 4) baseline classification
approaches for unimodal and multimodal features. This provides a flexible
framework for benchmarking different feature extraction and classification
tools in a reproducible manner. Data management tools for obtaining the lists
of subjects in AD, MCI converter, MCI non-converters, CN classes are also
provided. We demonstrate its use on all (1519) baseline T1 MR images and all
(1102) baseline FDG PET images from ADNI 1, GO and 2 with SPM-based feature
extraction pipelines and three different classification techniques (linear
SVM, anatomically regularized SVM and multiple kernel learning SVM). The
highest accuracies achieved were: 91% for AD vs CN, 83% for MCIc vs CN, 75%
for MCIc vs MCInc, 94% for AD-A$\displaystyle\beta$\+ vs
CN-A$\displaystyle\beta$\- and 72% for MCIc-A$\displaystyle\beta$\+ vs
MCInc-A$\displaystyle\beta$+. The code is publicly available111Code at
https://gitlab.icm-institute.org/aramislab/AD-ML \- Depends on the Clinica
software platform, publicly available at http://www.clinica.run .

##  1 Introduction

Alzheimer’s disease (AD) accounts for over 20 million cases worldwide.
Biological and brain imaging markers of AD contain information about
complementary processes of the disease progression: in anatomical MRI the
atrophy due to gray matter loss, hypometabolism in fluorodeoxyglucose (FDG)
PET, the accumulation of amyloid-beta protein in the brain tissue in amyloid
PET imaging, as well as concentrations of tau and amyloid-beta proteins in
cerebrospinal fluid through lumbar puncture. A major interest is then to
analyze those markers to identify AD at an early stage.

To this end, a large number of machine learning approaches have been proposed
(see [16, 4, 14] for a review). Such approaches differ by: i) brain image
processing pipelines, 2) feature extraction and selection; 3) machine learning
algorithms (classification and/or regression methods). Furthermore, while
initial efforts had made use of a single imaging modality (usually anatomical
MRI) [15, 19, 12, 10, 18], a large number of works have proposed to combine
multiple modalities (MRI and PET neuroimaging, fluid biomarkers) [2, 7, 9, 1,
17, 6]. In particular, the combination of anatomical MRI and FDG PET is the
most common.

Validation and comparison of such approaches require a large number of
patients with multimodal data and followed over time. The vast majority (over
90%) of published works use the publicly available Alzheimer’s Disease
Neuroimaging Initiative (ADNI). However, objective comparison between their
results is impossible because they use: i) different subsets of patients (with
unclear specification of selection criteria); ii) different preprocessing
pipelines (and thus it is not clear if the superior performance comes for the
classification or the preprocessing); iii) different evaluation metrics; iv)
different cross-validation procedures. At the end, it is very difficult to
conclude which methods perform the best and even if a given modality provides
useful additional information. As a result, the practical impact of these
works has remained very limited.

Comparison papers [12, 11] and challenges [3, 5] have been an important step
towards objective evaluation of machine learning methods, by allowing to
benchmark different approaches on the same dataset and with the same
preprocessing. Nevertheless, such studies provide a ”static” assessment of
approaches. Evaluation datasets are used in their current state at the time of
the study, whereas new patients are continuously included in studies such as
ADNI. Similarly, they are limited to the classification and preprocessing
methods that were used at the time of the study. It is thus difficult to
complement them with new approaches.

In this paper, we propose a framework for reproducible evaluation of machine
learning algorithms in AD and demonstrate its use on multimodal classification
of PET and MRI data of the ADNI database. Specifically, our contributions are
three-fold: i) a framework for management of ADNI data and their continuous
update with new subjects; ii) a modular set of preprocessing pipelines,
feature extraction and classification methods, that provide a baseline for
benchmarking of different components; iii) a large-scale evaluation of the
added value of the combination of anatomical MRI and FDG PET on 967 patients.
By providing a fully automatic conversion into BIDS format, we offer a huge
saving of time to users, compared to simply making public the list of used
subjects. This is particularly true for complex multimodal datasets like ADNI
(with plenty of incomplete data, multiple instances of a given modality and
complex metadata). Such a standardized data curation also allows future
inclusion of other datasets in a transparent way, perfect for external
validation (e.g. the Australian Imaging Biomarker and Lifestyle study, AIBL).

Currently, we include relatively standard T1 and (FDG and AV45) PET pipelines
and voxel-based SVM classification. The idea is not to impose their use, but
to provide a set of baseline tools against which new methods can be easily
compared. Researchers working on novel methods can then easily replace a given
part of the pipeline (e.g. feature extraction, classification) with their own
approach, and evaluate the added value of this specific new component over the
baseline approach provided.

##  2 Material and methods

In order to serve methodological developments and to provide a common
framework for evaluation and comparison of methods, we developed a unified set
of tools for data management, image preprocessing, feature extraction and
classification. Data management tools allow to easily update the dataset as
new subjects become available in ADNI. Processing and classification tools
have been designed in a modular way using Nipype library to allow the
development and testing of other methods as replacement for a given step.
Then, the impact of each method in the results can be objectively measured. A
simple command line interface is provided and the code can also be used as a
Python library.

###  2.1 Dataset

ADNI, together with follow ups ADNI GO and ADNI2, is a publicly available
database. Overall, it comprises over 300 patients with AD, over 850 patients
with mild cognitive impairment (MCI) and over 350 control subjects. An
important difficulty with ADNI lies in the organization of the public
database. Imaging data, in the state it is downloaded, lacks of a clear
structure, and there are multiple image acquisitions for a given visit of a
subject. The complementary image information is contained in numerous csv
files, making the exploration of the database and subject selection very
complicated.

###  2.2 A standardized data structure

To organize the data, we selected the Brain Imaging Data Structure (BIDS)[8],
a community standard to store multiple neuroimaging modalities. Being based on
a file hierarchy, rather than a database management system, BIDS can be easily
deployed in any environment. For ADNI, among the multiple scans for a visit,
we selected a single scan by imaging modality for each subject. In the case of
T1 scans, gradwarp and B1-inhomogeneity corrected images were selected when
available, otherwise the original image was selected. When repeated MRI were
available for a single session, the higher quality scan (as identified in ADNI
csv files) was chosen. 1.5T images were preferred for ADNI-1 since they are
available for a larger number of patients. For PET scans, the co-registered
averaged across time frames images were selected.

Very importantly, we provide the code that performs these selections in a
fully automated way. The code receives as input a folder containing the
download of all ADNI data (in their raw format) and creates a BIDS organized
version (T1 MRI, FDG and AV-45 PET). This allows direct reproducibility by
other groups without having to redistribute ADNI data, which is not allowed.
We also provide tools for subject selection according to desired imaging
modalities, time of follow up and diagnose, which make possible to use the
same groups with the largest possible number of subjects across studies.
Finally, we propose a BIDS-inspired standardized structure for all outputs of
the experiments.

###  2.3 Preprocessing and feature extraction pipelines

Two pipelines for processing anatomical T1 MRI and PET images were developed.
Pipelines have a modular structure based on Nipype allowing to easily connect
and/or replace components. For anatomical T1 MRI, SPM12 was used to perform
tissue segmentation (GM, WM, CSF) based on the Unified Segmentation procedure.
Next, a DARTEL template is created for all the subjects and a registration to
MNI space (DARTEL to MNI) is carried on each of them. The result is that all
the images are in a common space, providing a voxel-wise correspondence across
subjects. For PET images, we perform a registration of PET data onto the
corresponding T1 image in native space. An optional partial volume correction
(PVC) step is included using different tissue maps from the T1 in native
space. Then the registration into MNI space using the same transformation as
for the corresponding T1 and the generation of a parametric image by
normalization to a reference region (the reference region is eroded pons for
FDG PET, eroded pons and cerebellum combined for amyloid PET) and masking of
non-brain regions are performed.

###  2.4 Classification methods

The obtained images for all the subjects and for each modality lie in a common
space so they can be analyzed voxel-wise. The currently implemented
classifiers use voxel-as-features (i.e. GM maps for T1 MRI and parametric FDG
PET images). Other types of features such as regional features could be easily
implemented in the future. We implemented different classifiers for unimodal
(T1 MRI or FDG PET) and multimodal classification.

####  2.4.1 Unimodal classification.

Linear SVM. The first method included to classify single modality images is a
linear SVM. For the specified modality, a linear kernel is calculated using
the inner product for each pair of images (using all the voxels) for the
provided subjects. This kernel is used as input for the generic SVM (makes use
of scikit-learn library222http://scikit-learn.org). Given its simplicity it is
useful as a baseline to compare the performance of methods.

##### Spatially and anatomically regularized SVM.

The L2 regularization of the standard SVM does not take into account the
spatial and anatomical structure of brain images. As a result, the solutions
are not easily interpretable (the obtained hyperplanes are highly scattered).
To overcome this issue, we implement a classifier that combines a spatial
regularization and an atlas based anatomical regularization by using the
Fisher metric as proposed by [13]. The spatial proximity takes into account
the distance between voxels while anatomical proximity is defined as if two
voxels belong to the same atlas defined region or tissue (in our case GM, WM
and CSF probability maps). This approach was initially proposed by [13] in the
case of T1 MRI. Here, we extend it to FDG PET images. The implementation of
this approach provides a baseline classifier with interpretable maps that can
be used for comparison of other methods.

####  2.4.2 Multimodal classification.

For multimodal classification, we included a simple implementation of Multiple
Kernel Learning (MKL) for the case of two modalities. A kernel is obtained
from the linear combination of two kernels obtained for each imaging modality
separately, and then provided as input to the SVM: $\displaystyle
K_{\text{T1-FDG}}=\alpha K_{\text{T1}}+(1-\alpha)K_{\text{FDG}},\quad
0\leq\alpha\leq 1$

Parameter $\displaystyle\alpha$ that maximizes the balanced accuracy is
determined by cross validation. This parameter weights the initial kernels
into the combined kernel and therefore represents the contribution of each
imaging modality into the classification result.

###  2.5 Validation

####  2.5.1 Cross-validation.

To assess classification performance without bias, it is important to
carefully perform cross-validation (CV). In particular, CV must not only
concern the training of the classifier, but also the optimization of
hyperparameters. While the former is usually dealt with correctly, the latter
has not always been appropriately performed in the literature. Here, we
implemented two nested CV procedures for hyperparameter optimization and
classifier training. By default, a 10-fold CV is implemented in our framework,
and an input parameter lets the user select the number of desired folds.

####  2.5.2 Metrics.

As output of the classification, we report AUC, accuracy, balanced accuracy,
sensitivity, specificity and in addition the predicted class for each subject
so that the user could calculate any other metric with this information. Also,
an image with the SVM weights for each feature is provided. Such weights live
in the same space as the MRI and PET images and can thus be used to visualize
the brain regions that contributed to the classification.

##  3 Results

We applied the preprocessing pipelines to all (1519) baseline T1 MR images and
all (1102) baseline FDG PET images from ADNI 1, GO and 2. Subjects were
grouped as AD(239), MCI converter(164), MCI non converter(309) or CN(255),
according to diagnosis determined for 36 months of follow up (967 subjects in
total). Another grouping was done also taking into account amyloid imaging as
AD-A$\displaystyle\beta$+(125), MCIc-A$\displaystyle\beta$+(81),
MCIc-A$\displaystyle\beta$-(5), MCInc-A$\displaystyle\beta$+(105),
MCInc-A$\displaystyle\beta$-(131) and CN-A$\displaystyle\beta$-(111).

Classification for AD vs CN, MCIc vs CN and MCIc vs MCInc,
AD-A$\displaystyle\beta$\+ vs CN-A$\displaystyle\beta$\- and
MCIc-A$\displaystyle\beta$\+ vs MCInc-A$\displaystyle\beta$\+ classification
tasks was performed using voxel-based linear SVM and spatially and
anatomically regularized SVM for each modality separately (T1 and FDG PET).
Also the multiple-kernel SVM was used on the combination of linear and
regularized kernels for T1 and FDG PET obtained in the previous step.
Classification results were averaged over 10 runs. Results can be observed in
Table 1.

Overall, the use of FDG PET provided better results than T1 in general
(balanced accuracy of 91% for AD vs CN, 83% for MCIc vs CN, 75% for MCIc vs
MCInc, 94% for AD-A$\displaystyle\beta$\+ vs CN-A$\displaystyle\beta$\- and
72% for MCIc-A$\displaystyle\beta$\+ vs MCInc-A$\displaystyle\beta$+), but it
significantly outperforms it in the case of conversion prediction (MCIc vs
MCInc and MCIc-A$\displaystyle\beta$\+ vs MCInc-A$\displaystyle\beta$\+
tasks).

The combination of different modality kernels confirms this result, providing
as best linear combination the case when only FDG PET data was selected,
giving no weight to GM kernel.

The use of amyloid imaging to refine groups improved the classification
results only for FDG PET in the case of AD-A$\displaystyle\beta$\+ vs
CN-A$\displaystyle\beta$\- and not in every task as expected.

Figure 1: Normalized $\displaystyle w^{opt}$ coefficients $\displaystyle\geq$
0 for AD vs CN: (a) T1 linear SVM, (b) T1 regularized SVM, (c) FDG PET linear
SVM, (d) FDG PET regularized SVM

##  4 Conclusions

We proposed a framework for reproducible machine learning experiments on
automatic classification using the ADNI database. It features a standardized
data management and a modular architecture for preprocessing, feature
extraction and classification. Of note, the dataset can be continuously
updated as more data become available. We provide processing pipelines for T1
MRI and PET images as well as classification tools for unimodal and multimodal
data.

We applied this framework to classify T1 MRI and FDG PET data. We demonstrate
that FDG PET imaging provides superior accuracy over T1 MRI when predicting
Alzheimer’s disease conversion in MCI patients. Furthermore, combination of
both modalities did not result in superior accuracy over FDG PET alone.
However, more sophisticated combinations may lead to improvements. The use of
amyloid imaging to refine groups of subjects didn’t improve results in
general, but it might be due to the significantly lower subject counts. The
classification accuracy for AD vs CN and MCIc vs CN are in line with the state
of the art. Nevertheless, there are some papers that report higher accuracy
for prediction of conversion. Our lower accuracy might be due to the fact that
we used all available patients, avoiding cherry-picking effects.

The code associated to this work will be made available at the time of the
conference. We believe that it will be a useful tool for the community to
progress towards more reproducible results in this field.

Image type |  Classifier |  Task | AUC |  Bal | Sens | Spec
---|---|---|---|---|---|---
acc
T1 | Linear SVM | AD vs CN | 94.2% | 88.8% | 92.8% | 84.8%
MCIc vs CN | 85.5% | 80.5% | 66.8% | 89.3%
MCIc vs MCInc | 73.8% | 66.5% | 64.9% | 69.3%
AD-A$\displaystyle\beta$\+ vs CN-A$\displaystyle\beta$- | 93.6% | 87.9% | 90.5% | 85.5%
MCIc-A$\displaystyle\beta$\+ vs MCInc-A$\displaystyle\beta$+ | 73.6% | 65.8% | 69.8% | 60.3%
FDG PET | Linear SVM | AD vs CN | 96.7% | 91.1% | 95% | 87.2%
MCIc vs CN | 89.1% | 83.5% | 74.7% | 89.2%
MCIc vs MCInc | 80.5% | 75.2% | 78.5% | 69.4%
AD-A$\displaystyle\beta$\+ vs CN-A$\displaystyle\beta$- | 98.6% | 94.4% | 95.6% | 93.4%
MCIc-A$\displaystyle\beta$\+ vs MCInc-A$\displaystyle\beta$+ | 80.8% | 72.8% | 77% | 67%
Table 1: Classification results (mean of 10 runs)

## References

  * [1] B. Jie et al.  Manifold regularized multitask feature learning for multimodality disease classification.  Human Brain Mapping, 36(2):489–507, February 2015.
  * [2] D. Zhang et al.  Multimodal classification of Alzheimer’s disease and mild cognitive impairment.  NeuroImage, 55(3):856–867, April 2011.
  * [3] E. Bron et al.  Standardized evaluation of algorithms for computer-aided diagnosis of dementia based on structural MRI: the CADDementia challenge.  NeuroImage, 111:562–579, May 2015.
  * [4] F. Falahati et al.  Multivariate data analysis and machine learning in Alzheimer’s disease with a focus on structural magnetic resonance imaging.  Journal of Alzheimer’s disease: JAD, 41(3):685–708, 2014.
  * [5] G. Allen et al.  Crowdsourced estimation of cognitive decline and resilience in Alzheimer’s disease.  Alzheimer’s & Dementia, 12(6):645–653, June 2016.
  * [6] HJ. Yun et al.  Multimodal Discrimination of Alzheimer’s Disease Based on Regional Cortical Atrophy and Hypometabolism.  PloS One, 10(6):e0129250, 2015.
  * [7] J. Young et al.  Accurate multimodal probabilistic prediction of conversion to Alzheimer’s disease in patients with mild cognitive impairment.  NeuroImage. Clinical, 2:735–745, 2013.
  * [8] K. Gorgolewski et al.  The brain imaging data structure, a format for organizing and describing outputs of neuroimaging experiments.  Scientific Data, 3:160044, 2016.
  * [9] K. Gray et al.  Random forest-based similarity measures for multi-modal classification of Alzheimer’s disease.  NeuroImage, 65:167–175, January 2013.
  * [10] M. Liu et al.  Ensemble sparse classification of Alzheimer’s disease.  NeuroImage, 60(2):1106–1116, April 2012.
  * [11] M. Sabuncu et al.  Clinical Prediction from Structural Brain MRI Scans: A Large-Scale Empirical Study.  Neuroinformatics, July 2014.
  * [12] R. Cuingnet et al.  Automatic classification of patients with Alzheimer’s disease from structural MRI: a comparison of ten methods using the ADNI database.  NeuroImage, 56(2):766–781, May 2011.
  * [13] R. Cuingnet et al.  Spatial and anatomical regularization of svm: a general framework for neuroimaging data.  IEEE transactions on pattern analysis and machine intelligence, 35(3):682–696, 2013.
  * [14] S. Haller et al.  Principles of classification analyses in mild cognitive impairment (MCI) and Alzheimer disease.  Journal of Alzheimer’s disease: JAD, 26 Suppl 3:389–394, 2011.
  * [15] S. Kloppel et al.  Automatic classification of MR scans in Alzheimer’s disease.  Brain: A Journal of Neurology, 131(Pt 3):681–689, March 2008.
  * [16] S. Rathore et al.  A review on neuroimaging-based classification studies and associated feature extraction methods for alzheimer’s disease and its prodromal stages.  NeuroImage, 2017.
  * [17] S. Teipel et al.  The relative importance of imaging markers for the prediction of Alzheimer’s disease dementia in mild cognitive impairment - Beyond classical regression.  NeuroImage. Clinical, 8:583–593, 2015.
  * [18] T. Tong et al.  Multiple instance learning for classification of dementia in brain MRI.  Medical Image Analysis, 18(5):808–818, July 2014.
  * [19] Y. Fan et al.  Spatial patterns of brain atrophy in MCI patients, identified via high-dimensional pattern classification, predict subsequent cognitive decline.  NeuroImage, 39(4):1731–1743, February 2008.

## Supplementary material

Image type |  Classifier |  Task | AUC |  Bal | Sens | Spec
---|---|---|---|---|---|---
acc
T1 | Linear SVM | AD vs CN | 94.2% | 88.8% | 92.8% | 84.8%
MCIc vs CN | 85.5% | 80.5% | 66.8% | 89.3%
MCIc vs MCInc | 73.8% | 66.5% | 64.9% | 69.3%
AD-A$\displaystyle\beta$\+ vs CN-A$\displaystyle\beta$- | 93.6% | 87.9% | 90.5% | 85.5%
MCIc-A$\displaystyle\beta$\+ vs MCInc-A$\displaystyle\beta$+ | 73.6% | 65.8% | 69.8% | 60.3%
Regul SVM | AD vs CN | 94.9% | 90% | 92.5% | 87.3%
MCIc vs CN | 86.5% | 81.2% | 74.7% | 85.4%
MCIc vs MCInc | 74.8% | 67.3% | 66.8% | 68.8%
AD-A$\displaystyle\beta$\+ vs CN-A$\displaystyle\beta$- | 94.9% | 88.1% | 92.5% | 84.2%
MCIc-A$\displaystyle\beta$\+ vs MCInc-A$\displaystyle\beta$+ | 73.5% | 65.4% | 56.8% | 76.2%
FDG PET | Linear SVM | AD vs CN | 96.7% | 91.1% | 95% | 87.2%
MCIc vs CN | 89.1% | 83.5% | 74.7% | 89.2%
MCIc vs MCInc | 80.5% | 75.2% | 78.5% | 69.4%
AD-A$\displaystyle\beta$\+ vs CN-A$\displaystyle\beta$- | 98.6% | 94.4% | 95.6% | 93.4%
MCIc-A$\displaystyle\beta$\+ vs MCInc-A$\displaystyle\beta$+ | 80.8% | 72.8% | 77% | 67%
Regul SVM | AD vs CN | 96.7% | 91.1% | 94.8% | 86.9%
MCIc vs CN | 88.8% | 83.0% | 76.5% | 87.1%
MCIc vs MCInc | 80.4% | 73.7% | 74.5% | 71.9%
AD-A$\displaystyle\beta$\+ vs CN-A$\displaystyle\beta$- | 98.7% | 95.4% | 96.9% | 93.8%
MCIc-A$\displaystyle\beta$\+ vs MCInc-A$\displaystyle\beta$+ | 78.0% | 71.3% | 73.5% | 67.8%
Table 2: Classification results for T1 and FDG PET modalities (mean of 10
runs)

Image type |  Classifier |  Task |  Bal | Sens | Spec
---|---|---|---|---|---
acc
T1+FDG PET | Linear SVM | AD vs CN | 90% | 93% | 87%
MCIc vs CN | 81% | 88% | 73%
MCIc vs MCInc | 73% | 79% | 68%
AD-A$\displaystyle\beta$\+ vs CN-A$\displaystyle\beta$- | 95% | 95% | 94%
MCIc-A$\displaystyle\beta$\+ vs MCInc-A$\displaystyle\beta$+ | 73% | 77% | 69%
Regul SVM | AD vs CN | 92% | 95% | 89%
MCIc vs CN | 82% | 87% | 77%
MCIc vs MCInc | 72% | 75% | 69%
AD-A$\displaystyle\beta$\+ vs CN-A$\displaystyle\beta$- | 96% | 97% | 95%
MCIc-A$\displaystyle\beta$\+ vs MCInc-A$\displaystyle\beta$+ | 72% | 77% | 67%
Table 3: Best classification results for MKL combining T1 and FDG PET