# T3: Domain-Agnostic Neural Time-series Narration

Mandar Sharma  Computer Science
Virginia Tech
<EMAIL_ADDRESS>John S. Brownstein  Boston Children’s Hospital
Harvard Medical School
<EMAIL_ADDRESS>Naren Ramakrishnan  Computer Science
Virginia Tech
<EMAIL_ADDRESS>
###### Abstract

The task of generating rich and fluent narratives that aptly describe the
characteristics, trends, and anomalies of time-series data is invaluable to
the sciences (geology, meteorology, epidemiology) or finance (trades, stocks,
or sales and inventory). The efforts for time-series narration hitherto are
domain-specific and use predefined templates that offer consistency but lead
to mechanical narratives. We present T3 (Time-series-To-Text), a domain-
agnostic neural framework for time-series narration, that couples the
representation of essential time-series elements in the form of a dense
knowledge graph and the translation of said knowledge graph into rich and
fluent narratives through the transfer learning capabilities of PLMs (Pre-
trained Language Models). T3’s design primarily addresses the challenge that
lies in building a neural framework in the complete paucity of annotated
training data for time-series. The design incorporates knowledge graphs as an
intermediary for the representation of essential time-series elements which
can be linearized for textual translation. To the best of our knowledge, T3 is
the first investigation of the use of neural strategies for time-series
narration. Through extensive evaluations, we show that T3 can improve the
lexical diversity of the generated narratives by up to 65.38% while still
maintaining grammatical integrity. The practicality and deployability of T3 is
further validated through an expert review ($n=21$) where 76.2% of
participating experts wary of auto-generated narratives favored T3 as a
deployable system for time-series narration due to its richer narratives. Our
code-base, models, and datasets, with detailed instructions for
reproducibility is $publicly$ hosted111https://github.com/Mandar-Sharma/TCube.

###### Index Terms:

time-series analysis, time-series-to-text, data-to-text, pre-trained language
models, natural language generation

##  I Introduction

Real-world data is often temporal in nature. From the global outbreaks of
infectious diseases to the prices of stocks, all chronologically recorded data
takes the form of a time-series. Thus, its mining and analysis has been of
significant interest to the scientific community [1]. Time-series narration
aims to portray the discerning characteristics of a time-series obtained from
such analysis through a textual narrative. The efficacy of narratives as an
aid to data comprehension has been validated through studies in digital
libraries [2] as well as causal networks [3]. Petre, in his advocacy for the
importance of textual representations of data [4], humorously notes, A picture
is worth a thousand words - isn’t it? And hence graphical representation is by
its nature universally superior to text - isn’t it? Why then isn’t the
anecdote itself expressed graphically?.

Figure 1: Sample narratives generated by T3 for the United Kingdom COVID19,
Kansas CO pollution, and United States merchandise exports datasets.

Time-series narration falls under the umbrella of data-to-text, a sub-field of
NLG (Natural Language Generation) that aims to produce meaningful and coherent
textual descriptions of non-linguistic data [5, 6]. Although data-to-text has
garnered significant interest over the years, recent efforts for textual
description of data have been focused on either tabular data [7, 8, 9] or
graph data [10, 11]. The attention that these data types have garnered
simultaneously highlight the two key challenges for time-series data. The
first being in the design and training of such a system in the paucity of gold
datasets and the second in its evaluation standards.

  * •

End-to-end models for data-to-text generation showcase learning a direct
input-output mapping from data to text [12, 13] through the use of annotated
datasets such as WikiBio [12] and E2E [14] for tabular data and WebNLG and
DART [15, 16] for RDF (Resource Description Framework) triples [17]. In both
tabular data and RDF triples, the information to be presented in the narrative
is present in the data itself and is copied to the output token - making end-
to-end learning possible. In contrast, time-series requires further processing
for the discovery of underlying patterns to be narrated. Thus, due to the
inherent numerical and continuous nature of time-series, one needs to consider
time-series as a whole rather than a sum of its individual constituents. Thus,
one would have to either follow the traditional modular pipeline architecture
[5] where non-linguistic data is transformed into text through several
intermediate steps, or formulate a novel approach suited to time-series data
altogether.

  * •

The gold narratives in the aforementioned datasets offers a common ground for
automated evaluation of competing frameworks on the basis of word-based
metrics such as BLEU [18] and its variants [19, 20, 21]. Thus, there are
domain-familiar metrics present to showcase how one framework can perform
better than another. For time-series data, without human annotations
corresponding to the data, automated evaluation through said word-based
metrics is not possible.

As will be discussed in the related works section, there have been several
previous efforts for time-series narration. Although these pioneering efforts
have laid significant groundwork for this field, the recent work in time-
series narration falls short in two crucial areas: First, they are domain-
specific, modeled specifically for use in fields such as meteorology,
intensive care, health monitoring and so on. Second, the proposed systems have
not actualized the recent advances in language processing, rather, relying on
the traditional pipeline architecture. Graefe et. al. [22] note news consumers
get more pleasure out of reading human-written as opposed to computer-written
content. Thus, these template-based narratives can be met with a dismissive
response by its users due to its seemingly mechanical nature - we further
elaborate on this in our expert review section.

To address these challenges, we present T3 : Timeseries-To-Text, which stands
out from previous forays in this task through a) its domain-agnostic nature
and b) its coupling of dense knowledge graph based representation of essential
time-series elements and the translation of said knowledge graph into rich and
fluent narratives through the transfer-learning capabilities of large PLMs
(Pre-trained Language Models) fine-tuned to this specific task - tackling the
paucity of annotated data. Figure 1 highlights the diversity in the narratives
generated by T3 along with the automatic extrapolations and abbreviations
deduced by the language models. The terms ‘United Kingdom’, ‘United States’,
and ‘Carbon Monoxide’ are automatically abbreviated to ‘UK’, ‘US’, and ‘CO’
respectively. Similarly, the system extrapolates information such as adding
‘as a measurement of air quality’ when mentioning carbon monoxide values,
adding ‘the state of’ to Kansas, and introducing the term ‘trade volume’ when
describing export values. Our contributions are summarized as follows:

  * •

To the best of our knowledge, T3 is the first foray into neural time-series
narration. Our rigorous evaluations across multi-domain datasets showcases
that T3 consistently produces 65.38% more diverse narratives with the same
grammatical integrity as the existing baselines.

  * •

Through an expert review $(n=21)$, we validate the performance, practicality,
and linguistic superiority of $T^{3}$. 76.2% of participating experts who were
wary of auto-generated narratives favored T3 as a deployable system as
compared to existing baselines.

  * •

We benchmark the performance of several time-series segmentation and regime-
shift detection algorithms as well as prominent PLMs for outlining the best
approach to a domain-agnostic time-series narration framework.

  * •

Our code-base, pre-trained models, the datasets used, along with a detailed
notebook guide for reproducibility are made public1.

##  II Narratives: Good, Bad, and Boring

Textual narratives are swiftly becoming important components of visualization
systems, either as a way to generate data insights to accompany visualizations
[23] or to structure visualizations for better communication [24]. Research
into what makes an effective narrative is still in its infancy and is
necessarily tied to the underlying analytical task and domain. For temporal
data, we identify the following crucial facets:

Level of detail: Should the narrative capture an executive summary or provide
in-depth access to the underlying data?

Language diversity: Greater diversity in language prevents monotony but could
detract from conveying key messages and conclusions. Lower diversity, on the
other hand, supports comparison of different narratives, but leads to glossing
over by analysts - defeating the very purpose of these narratives.

Verbalizing numbers: The verbalization of quantitative or probabilistic data
(using Kent’s words of estimative probability [25] or the NIC/ Mercyhurst
standardization) and trends is considered important in specific domains (such
as intelligence analysis [26]), however, other applications argue for direct
access to the original numeric information.

Human performance aspects: Understanding the characteristics of narratives
that lead to improved human performance is an ongoing research problem [27].
Narratives provide increased comprehension, interest, and engagement and are
known to contribute distinct cognitive pathways of comprehension with
increased recall, ease of comprehension, and shorter reading times [28].
Conversely, the challenge of the written word implies slowness and error-prone
behavior due to short-term memory limits.

In essence, successful narrative research requires a standardization of both
the generation and evaluation space, and an understanding of how a narrative
fits into the larger comprehension process of the analyst. As an example, a
bad narrative for a fictional monthly sales-volume dataset, in the form of The
sales numbers for January 2019 were 1500 while the sales numbers for February
2019 were 2000. Similarly, the sales numbers for …, falls to meet all the
above criterion: it is lexically repetitive, portrays no information about the
data that would have been difficult to discern visually, and presents the
numbers as-is with no verbalization.

##  III Related Work

While some of the earliest work on time-series narration can be traced back to
1994 with the Forecast Generator (FOG) [29], a framework for generating
bilingual (English/French) textual summaries of weather forecasts, in the
recent decades, Ehud Reiter’s research group has laid significant groundwork
for this domain. Their SUMTIME-MOUSAM project [30] generates short textual
summaries of weather forecasts and SUMTIME-TURBINE [31] generates the same for
sensor readings from a gas turbine. The design of these SUMTIME systems
highlights the importance of domain expertise in relaying the information
embedded in a raw time-series in a manner relevant to the end user. Following
this, their SUMTIME project was extended to SUMTIME-NEONATE [32], which
generates textual summaries of time-series data intended to aid medical
professionals in monitoring infants in neonatal intensive care units. In 2003
[33], the authors highlight the use of Gricean maxims of cooperative
communication [34] for the selection of the most crucial information to be
relayed to the end user. The authors further investigate the impact of word
choice in textual summarization by avoiding words specific to one idiolect and
words whose meanings varied in different idiolects [35].

Kacprzyk et. al. [36] propose the use of Zadeh’s calculus of linguistically
quantified propositions with varying t-norms to summarize time-series
segmented with Piece-wise Linear Approximations. Castillo-Ortega et. al. [37],
propose linguistic summarization of time-series based on the hierarchical
structure of time. The multiple candidate summaries are evaluated with a
multi-objective evolutionary algorithm. In the physiological domain, Banaee
et. al. [38] propose a system to summarize the data streams from health
monitoring systems in a clinician and patient centric manner. Dubey et. al.
[39] propose the use of Case-based Reasoning from records of previous
summaries to summarize weather reports.

Thus, there has been significant investigation into this domain. However, the
research emphasis has heavily been in the identification of the information to
relay to the end user rather than relaying the information in a manner
engaging to the end user - having the narratives themselves be rich and
fluent. The textual output of the above mentioned systems follow the
traditional modular pipeline architecture of Reiter and Dale [5]. Commercial
services such as The Automatic
Statistician222https://automaticstatistician.com/ and Narrative
Science333https://narrativescience.com/ offer data summarization through
visualization and narratives. Although their technology and code is
proprietary, a perusal through offered
samples444https://automaticstatistician.com/examples/ for time-series
summarization hints towards templated generation where variables from analysis
are plugged into preset templates.

##  IV Preliminaries

In this section we outline some necessary background in time-series
segmentation, detecting shifting regimes, and PLMs, as a foundation for T3’s
architecture.

###  IV-A Segmentation

Given a time-series $T$ of length $n$, a segmentation of $T$ contains a set of
distinct temporal cut-points $S=\\{c_{1},c_{2},..,c_{k}\\}$ corresponding to
$k$ straight lines where $k<<n$ [40]. The segmentation approach can be limited
by the number of segments $k$ produced, or by a predefined threshold for
segment-wise or cumulative error. As time-series of varying types and lengths
need to be approximated with varying number of segments, we evaluate the
following candidate segmentation algorithms based on a preset error threshold
to promote domain-agnosticism.

Sliding Windows: The data points from a time-series are added to a sliding
window until the maximum approximation error is met and a segment is formed.
This process repeats with the window starting from the next data point.

Bottom-Up: The algorithm starts with the finest approximation such that a
time-series of length $n$ is approximated by $\frac{n}{2}$ segments. The
algorithm iteratively merges the lowest cost adjacent segments until the
stopping criteria is met.

SWAB: An acronym for the integration of Sliding Windows and Bottom-Up, SWAB
[41] first defines an initial buffer $w$ on which Bottom-Up is performed. The
first segment from $w$ is reported and the corresponding data points are
removed from it. Remaining points from the series are read into $w$ till the
linear fit on it reaches an error threshold. This process is repeated until
the buffer $w$ reaches the end of the time-series.

###  IV-B Regime-shifts

Regime shift or switching refers to changes in the state or structure of a
time-series. For domain-agnosticism, we require the shift-detection algorithms
to be unsupervised, universal approximators, and input length invariant. Thus,
based on these criterion, we evaluate the following candidates:

Rrepresentation Learning: Franceschi et. al.’s [42] unsupervised
representation learning algorithm, hereby noted as RL, learns representations
of time-series elements using an encoder architecture based on causal dilated
convolutions with a triplet loss arrangement that employs time-based negative
sampling.

Matrix Profile: The Matrix Profile [43, 44] is a multi-purpose annotation
(profile) of a time-series $T$ where the $i^{th}$ location on the profile
records the distance of the sub-sequence in $T$ at the $i^{th}$ location to
its nearest neighbor.

Figure 2: The two stage T3 framework: In Stage I, the system extracts trends,
regimes, and peaks from the input time-series which is formulated into a
knowledge graph. In Stage II, a PLM fine-tuned for graph-to-text generation
generates the narrative from the input graph.

###  IV-C Pre-trained Language Models

Transfer learning in language processing has been democratized and made
universal with the advent of PLMs [45] which share the multi-headed attention
core architecture of transformers [46]. Transfer learning, in the context of
PLMs, is essentially the adaptation of these massive language models to
downstream tasks such as data-to-text, question answering, summarization and
much more via a fine-tuning process on task-specific data. Through the effors
of Thomas Wolf et. al. [47], second-generation seq2seq PLMs such as Google’s
T5 [48] and Facebook’s BART [49] and auto-regressive PLMs such as Open-AI’s
GPT-2 [50] and many more have been made accessible to the larger community.

The motivation behind using PLMs for this task not only stems from the fact
that they lead the benchmark for a multitude of downstream language processing
tasks [51] but also due to the evidence that PLMs, due to their apparent
acquisition of worldly knowledge [52], in some cases refuse to generate false
outputs even when the input to the system is corrupted [11]. As Open AI’s
GPT-3 [53] has not been released for public access at the time of publication
of this paper, we have not been able to incorporate it into our experiments.

###  IV-D Decoding Strategies

The PLMs we intend to investigate—viz. Open-AI’s GPT-2, Facebook’s BART, and
Google’s T5—though differing in their architectures and training strategies,
share an auto-regressive decoder. Auto-regressive language generation is based
on the assumption that the probability distribution of a sequence of words can
be decomposed into the product of conditional next word distributions. If
$W_{0}$ be the initial context word sequence and $T$ be the length of the
sequence to be generated, then the probability distribution can be defined as:

$P(w_{1:T}|W_{0})=\prod_{t=1}^{T}P(w_{t}|w_{1:t-1},W_{0})$ (1)

Basic Sampling: This strategy is based on randomly picking a word $w_{t}$
based on its conditional probability distribution $w_{t}\sim P(w|w_{1:t-1})$.
Thus, the next word in the sequence is chosen based on its conditional
probability of occurrence.

Top-K Sampling: In top-K sampling [54], the $K$ words most likely to occur
next in the sequence are chosen and the probability mass is redistributed
among these $K$ words. This leads to a more human-like text generation.

Top-p Sampling: Top-p sampling [55], also known as nucleus sampling, addresses
a core issue in top-K sampling. Since top-K re-distributes the probability
mass among the top $K$ chosen words, it has the potential to break down in
particularly sharp or flat distributions. If a distribution is sharp, the
limit on the selection of just $K$ words can lead to insensible text
generation. On the other hand, for flat distributions, the limit prevents the
generation from being diverse. Thus, instead of limiting the sampling space to
$K$ words, top-p samples from the smallest possible set of words whose
cumulative probability exceeds a predefined probability $p$.

##  V T3 Framework

###  V-A The Architecture

The two-stage design of T3, as illustrated in figure 2, is motivated by the
need to produce rich and fluent narratives of time-series data with the least-
possible human intervention. Subsections VI-A and VI-B highlight thorough
experimentation that motivate the specific choices for the segmentation and
regime-shift detection algorithms for T3 while subsections VI-C and VI-D
highlight the same for our choice of PLMs.

Stage I: The time-series is first log-transformed to approximately conform the
data to normality before information extraction. This log-transformed series
is segmented into $k$ linear segments where the individual slopes of these $k$
segments indicates the trends followed by the data in their respective
intervals. Simultaneously, sequential data-points with similar properties are
clustered together based on their learned representations. These clusters
represent changing regimes in the dataset. The above time-series
characteristics are encoded into a RDF-based knowledge graph. Figure 3
illustrates a sample knowledge graph (curtailed) as extracted from T3’s first
stage for the United States COVID19 time-series.

Stage II: Anterior to T3’s execution, the PLMs are fine-tuned with both WebNLG
and DART datasets for graph-to-text translation. The knowledge graph from
Stage I is thus translated into a rich and descriptive narrative by these PLMs
using sampling techniques for strategic language generation.

Figure 3: Sample T3 knowledge graph (curtailed) for U.S. COVID19 Cases.

###  V-B Datasets

Time-series: To promote domain-agnosticism, the datasets used for evaluating
T3 are drawn from five different fields -
COVID19555https://ourworldindata.org/, Direction of Trade
Statistics666https://data.imf.org/, Carbon Monoxide
Pollution777https://data.world/data-society/, World Population4, and Climate
Change4. Based on the amount and consistency of the data, we consider the same
ten countries (United States, India, Brazil, Russia, United Kingdom, France,
Spain, Italy, Turkey, and Germany) across these datasets. The CO (Carbon
Monoxide) units, however, are extracted for the U.S. states with EPA state
codes 1 through 10. Table I provides a brief statistical summary of these
datasets.

Fine-tuning: RDF-based datasets WebNLG v3.0 and DART v1.1 are used for fine-
tuning the PLMs in T3. Table I briefly summarizes the statistics of these
datasets where $N_{x}$ represents the number of samples for
$x\in\\{train,dev,test\\}$ and $V$, $WSR$, and $SSR$ represent the vocabulary
size, words per SR (Surface Realization), and sentence per SR respectively.

###  V-C Fine-tuning, Training, and Decoding Specifications

Tokens $<$X$>$ where $X\in\\{H,R,T\\}$ are appended to the start of the Head
(subject), Relationship (predicate), and Tail (object) entities of each RDF
triple. The Adam optimizer [56] with a linearly decreasing learning rate is
used to fine-tune the PLMs with learning rates initially set to 3e-5 for T5
and BART and 5e-4 for GPT-2. For uniformity, the maximum token lengths for all
PLMs are set to their default maximum (512) with a batch size of 4. For
strategic decoding, based on the average length ($\sim$100 words) and the
average number of unique words ($\sim$50) present in the generated narratives
we set $k$ as 50. Similarly, based on popular practice, we set $p$ as 92%.

TABLE I: Datasets statistics.

| $N$ | $\mu$ | $\sigma$
---|---|---|---
COVID19 Cases | 351 | 1.75e4 | 1.95e4
DOTS Exports (MM) | 254 | 1.35e4 | 6.27e3
U.S. CO Pollution (Units) | 4722 | 0.39 | 0.22
World Population | 22 | 8.02e7 | 4.82e7
Global Temperature (°C) | 3166 | 8.15 | 6.91

| $N_{train}$ | $N_{dev}$ | $N_{test}$ | $V$ | $WSR$ | $SSR$
---|---|---|---|---|---|---
WebNLG | 35426 | 4464 | 5150 | 8000 | 22.5 | 1.4
DART | 62659 | 6980 | 12551 | 33200 | 21.6 | 1.5

##  VI Experiments

Through our experimentation, we seek to address three core questions regarding
the design and need for a domain-agnostic time-series narration framework:

  * •

For the design of a domain-agnostic narration framework, how do we choose
among the prominent time-series analysis tools at our disposal? (Sections A
and B)

  * •

How do state-of-the-art language models stack up against each other for the
task of translating knowledge graphs to natural language? (Section C)

  * •

Does T3 deliver richer and more diverse narratives as compared to traditional
approaches? Does T3 hallucinate? Would domain experts find this favorable
and/or practical? (Sections D and VII)

TABLE II: Comparison of time-series segmentation algorithms based on total $SSE$ of residuals and $r^{2}$ fit across datasets. | COVID19 Cases | DOTS Exports | U.S. CO Pollution | World Population | Global Temperature
---|---|---|---|---|---
| $SSE$ | $r^{2}$ | $SSE$ | $r^{2}$ | $SSE$ | $r^{2}$ | $SSE$ | $r^{2}$ | $SSE$ | $r^{2}$
Sliding Window | 300.61 | 0.12 | 6.91 | 0.08 | 73.91 | 0.15 | 0.92 | 0.61 | 838.35 | 0.16
Bottom-Up | 27.07 | 0.13 | 4.98 | 0.07 | 67.53 | 0.16 | 0.75 | 0.64 | 67.36 | 0.16
SWAB | 27.16 | 0.14 | 4.90 | 0.08 | 67.11 | 0.16 | 0.11 | 0.37 | 65.21 | 0.17

###  VI-A Trend Detection

In order to evaluate our candidate segmentation algorithms, we must first
determine the right value of allowable maximum linear-fit error appropriate
for our datasets. The evaluation of the total SSE (Sum Squared of Errors) of
residuals vs $k$ (the numbers of segments produced), as a function of the
maxmimum linear-fit error, hints at 2.75 as a potential error sweet spot. The
figure below presents this analysis for the U.S. COVID19 dataset - the left
marker indicates the trade-off point between the total SSE and $k$ while the
right marker indicates the point where both total SSE and $k$ stabilize.

Table II outlines the performance of the selected segmentation algorithms
across our datasets with the maximum linear-fit threshold set to 2.75. We
observe that SWAB consistently performs the best in terms of both the $r^{2}$
goodness-of-fit and $SSE$, making it the segmentation algorithm of choice for
T3.

Out of the $k$ segments produced for each time-series, if the slope of
$k_{i-1}^{th}$ segment follows that of the $k_{i}^{th}$ segment, we rearrange
them as a single segment for continuity. This is illustrated in the figure
above for the U.S. COVID19 time-series where the original $k$ segments are
consolidated based on their slopes to 6 long segments ($k>6$) that indicate
the core trends followed by the time-series over significant time-spans.

###  VI-B Regime Shift Detection

For the evaluation of our candidate regime-shift detection algorithms, we
force these algorithms to produce a known number of regime shifts validated
through visual interpretation of the data - regime shifts in COVID19 cases
should correspond to waves of outbreak, as illustrated in the figure below,
whereas those in DOTS Exports should correlate to inflation or deflation in
the economy.

Table III outlines the performance of Matrix Profile and RL across our
datasets based on the standard deviations ($\sigma$) of the formed regimes.
Our evaluations lead us to conclude that the performance of Matrix Profile and
RL are on-par and vary based on the individual dataset. In our implementation,
an RL instance trained on the COVID19 dataset showcases high cross-domain
transferability when applied to other series in our catalog. The Matrix
Profile, however, requires a window-size definition prior to its execution
which varies based on the input time-series. The tendency of RL to favor
automation makes it the regime-shift detection algorithm of choice for T3.

TABLE III: Comparison of regime shift detection algorithms based on $\sigma$. | Matrix Profile ($\sigma$) | RL ($\sigma$)
---|---|---
COVID19 Cases | 7.29 | 9.43
DOTS Exports | 8.68 | 8.98
U.S. CO Pollution | 2.35 | 2.41
World Population | - | 15.26
Global Temperature | 2.20 | 2.19

###  VI-C Graph-to-text Translation

TABLE IV: Evaluation of PLMs on graph-to-text translation on the WebNLG dataset, DART dataset, and their combination. Dataset | WebNLG | DART | WebNLG + DART
---|---|---|---
Model | BLEU | ROUGE | METEOR | chrF++ | BLEU | ROUGE | METEOR | chrF++ | BLEU | ROUGE | METEOR | chrF++
GPT-2 | 14.2 | 4.28 | 20.22 | 37.13 | 15.56 | 5.23 | 21.79 | 37.68 | 18.65 | 7.54 | 23.61 | 39.22
BART Base | 32.13 | 51.81 | 33.49 | 59.08 | 33.77 | 54.27 | 35.86 | 61.15 | 37.89 | 58.22 | 37.80 | 64.52
BART Large | 32.04 | 51.10 | 34.68 | 59.98 | 34.75 | 55.32 | 36.47 | 61.75 | 38.36 | 58.18 | 38.15 | 64.82
T5 Small | 33.94 | 56.46 | 35.4 | 61.56 | 34.52 | 55.96 | 36.33 | 61.74 | 38.52 | 59.05 | 38.21 | 65.06
T5 Base | 36.75 | 57.76 | 37.25 | 64.17 | 36.40 | 57.00 | 37.44 | 63.23 | 39.88 | 59.71 | 38.91 | 65.95
TABLE V: Comparison of the performance of T3 with that of Templated Generation based on language evaluation metrics. | COVID19 Cases | DOTS Exports | U.S. CO Pollution | World Population | Global Temperature
---|---|---|---|---|---
| RE | TTR | G | RE | TTR | G | RE | TTR | G | RE | TTR | G | RE | TTR | G
Templated Generation | 17.79 | 0.26 | 1 | 54.73 | 0.45 | 1 | 64.34 | 0.22 | 1 | 66.28 | 0.46 | 1 | 55.24 | 0.37 | 1
T3 with $T5$ | 64.48 | 0.31 | 0.99 | 67.54 | 0.47 | 1 | 69.22 | 0.28 | 0.99 | 69.56 | 0.49 | 1 | 67.45 | 0.39 | 0.99
$T5_{top-K}$ | 65.67 | 0.38 | 0.98 | 67.57 | 0.51 | 0.98 | 64.43 | 0.33 | 0.99 | 74.19 | 0.56 | 1 | 66.06 | 0.46 | 1
$T5_{top-p}$ | 68.02 | 0.37 | 0.99 | 66.27 | 0.48 | 1 | 65.15 | 0.32 | 1 | 71.82 | 0.54 | 1 | 66.98 | 0.45 | 1
$BART$ | 70.71 | 0.42 | 0.94 | 67.30 | 0.46 | 0.97 | 68.16 | 0.33 | 0.99 | 75.04 | 0.55 | 0.99 | 63.95 | 0.42 | 0.92
$BART_{top-K}$ | 69.60 | 0.43 | 0.94 | 69.47 | 0.47 | 0.96 | 72.10 | 0.32 | 0.99 | 76.81 | 0.56 | 0.99 | 64.58 | 0.40 | 0.94
$BART_{top-p}$ | 67.53 | 0.40 | 0.94 | 68.36 | 0.47 | 0.97 | 67.35 | 0.32 | 0.99 | 76.58 | 0.57 | 0.99 | 65.46 | 0.41 | 0.93

The task of translating a graph to text is predominantly a Machine Translation
task. Thus, the PLM architecture of preference are seq2seq models such as
Google’s T5 and Facebook’s BART. However, for completeness we also include an
auto-regressive model - OpenAI’s GPT-2 in our evaluation. The performance of
these models are bench-marked across three dataset configurations: WebNLG,
DART, and a combination of both. Table IV shows our evaluation results for
these PLMs on automated word-based metrics. From this table, there are three
key takeaways: For every model, the performance improves with the third
dataset configuration (both the WebNLG and DART datasets). The $T5_{Base}$
model significantly outperforms the competitors while $GPT2$ falls short
across all benchmarks. Finally, although $T5_{Small}$ outperforms
$BART_{Large}$, their performance is almost competitive. From these
observations, $T5_{Base}$ and $BART_{Large}$, with the third dataset
configuration, are T3’s preferred language models.

###  VI-D T3 Evaluation

To evaluate the performance of $T^{3}$, we measure its performance with
respect to our baseline - the templated generation framework. The templated
generation takes in the data from Stage I of T3 , however, instead of passing
it to Stage II, it feeds it to a template designed for the desired domain. The
narratives generated by these systems are evaluated based on three core
dimensions of linguistic quality:

  * •

The Flesch’s RE (Reading Ease) score [57] measures the readability of a text
based on the average length of its sentences and the average number of
syllables of its words888https://pypi.org/project/textstat/. Ranging from 0 to
100, increasing scores represent increasing levels of readability.

  * •

The TTR (Type Token Ratio)999https://pypi.org/project/lexical-diversity/ is a
measure of text diversity where the tokens refers to the total number of words
in a given text while types refers to the number of non-repeating unique
words. Simply calculated as $TTR=\frac{Types}{Tokens}$, the closer the TTR is
to 1, the more lexical variety there is in a given text.

  * •

The $G$ (grammar score)101010https://pypi.org/project/language-tool-python/,
represents the grammatical integrity of the text. Similar to TTR, the closer
$G$ is to 1, the better the grammar of the text.
$G=1-\frac{\textrm{Number of grammatical errors in a sentence}}{\textrm{Number
of words in a sentence}}$

For each of our five datasets described in section 5-B, the RE score, TTR, and
Grammar score (G) are averaged-out for the aforementioned ten
countries/states. The performance of T3 is evaluated with three decoding
strategies: $T^{3}$ with $PLM_{top-K}$ represents the use of top-K sampling
scheme, $T^{3}$ with $PLM_{top-p}$ represents the use of top-p sampling
scheme, and $T^{3}$ with simply $PLM$ refers to the default sampling scheme
where words are sampled from the base conditional probability distribution
without the use of top-K or top-p strategies. Table V illustrates the
comparative performance of T3 with templated generation. From this, we make
four key observations:

  1. 1.

T3 significantly outperforms templated generation in lexical diversity. The
highest increase in lexical diversity was observed in the COVID19 dataset
where T3 increases the TTR by 65.38% while the lowest observed increase was in
the DOTS Exports dataset where T3 increases the TTR by 13.33%.

  2. 2.

T3 remains closely competitive with templated generation in maintaining
grammatical integrity. As templated generation uses pre-defined sentence
planning, the grammar is expected to be perfect ($TTR=1$). While T3 achieves
perfect grammatical integrity in the DOTS Exports, U.S. CO Pollution, and
World Population datasets, the highest observed loss in grammatical integrity
was 7.9% in the Global Temperature dataset.

  3. 3.

T3 consistently outperforms templated generation in terms of readability,
although not significantly. We attribute this to the distinct sentences formed
when each element of the knowledge graph is translated to text.

  4. 4.

In terms of PLM selection, we observe that T5 tends to lean more towards
grammatical integrity while BART tends to produce more linguistically diverse
text. Similar observations are made for the sampling strategies: top-p
sampling leads to more grammatical consistent texts while top-K sampling
promotes linguistic diversity.

##  VII Expert Review

We conduct an expert review $(n=21)$ [58] to validate the practicality of
$T^{3}$. The review simultaneously acts as a human evaluation of $T^{3}$’s
narratives as well. 85.7% of the recruited experts had expertise in data
science, 76.2% in data visualization, and 66.7% in NLP. When asked to rate
their trust in machine-generated narratives on a 1 to 5 Likert scale, the
response from the experts resembled a right-skewed bell-curve where 42.9% of
the experts had chosen a rating of 3 (neither complete trust or distrust in
machine-generated narratives). In agreement with [22], 61.9% of the recruited
experts acknowledged being dismissive of machine-generated narratives, while
the remaining claimed equal treatment of both machine and human generated
narratives.

Figure 4: Histogram of Likert Ratings based on Narrative Type.

The experts, each, were presented with 2 time-series datasets, where each
time-series was accompanied with 4 narratives - a baseline templated
narrative, 2 narratives randomly sampled from $T^{3}$, and finally, a sub-par
T3 narrative (generated by repeatedly sampling from T3 until a a sub-par
narrative was generated). For each of these narratives, the experts were asked
to rate its coherence, linguistic diversity, grammatical integrity, and data
fidelity (does the model tend to hallucinate?) on a 1 to 5 Likert scale.
Figure 4 presents an overview of the findings: $T^{3}$ and templated
generation were rated comparably in terms of coherence, grammatical integrity,
and data fidelity. However, $T^{3}$ was rated considerably higher in terms of
linguistic diversity - in alignment with our experimental findings. In their
concluding remarks, 76.2% of the experts chose $T^{3}$ over templated
narratives for deployable systems. For the remaining 23.8% of the experts that
chose templated narratives, their sentiment resonates with the need for
mission-critical data fidelity.

##  VIII Conclusion and Future Work

We have presented $T^{3}$, a domain-agnostic neural framework for time-series
narration. Through our experiments, we outline a strategy forward for
universal time-series narration. There are numerous avenues to pursue to
augment the space of time-series narration. From the analysis of time-series
data to the realization of natural language summaries, work in each of these
space will bring us closer to better data-to-text systems. With a dataset of
time-series and narrative pairs, a promising direction for future exploration
lies in learning direct mappings from numbers to text, extending beyond just
time-series.

##  IX Acknowledgements

This work was partially supported by DARPA (Defense Advanced Research Projects
Agency) under contract number FA8650-17-C-7720. The views, opinions and/or
findings expressed in this publication are solely those of the author(s).

## References

  * [1] T. chung Fu, “A review on time series data mining,” _Engineering Applications of Artificial Intelligence_ , vol. 24, no. 1, pp. 164 – 181, 2011\.
  * [2] S. Latif and F. Beck, “Vis author profiles: Interactive descriptions of publication records combining text and visualization,” _IEEE transactions on visualization and computer graphics_ , vol. 25, no. 1, pp. 152–161, 2018.
  * [3] A. C. et. al., “Once upon a time in visualization: Understanding the use of textual narratives for causality,” _IEEE Transactions on Visualization and Computer Graphics_ , pp. 1–1, 2020.
  * [4] M. Petre, “Why looking isn’t always seeing: readership skills and graphical programming,” _Communications of the ACM_ , vol. 38, no. 6, pp. 33–44, 1995\.
  * [5] E. Reiter and R. Dale, _Building Natural Language Generation Systems_ , ser. Studies in Natural Language Processing. Cambridge University Press, 2000.
  * [6] A. Gatt and E. Krahmer, “Survey of the state of the art in natural language generation: Core tasks, applications and evaluation,” _J. Artif. Int. Res._ , vol. 61, no. 1, p. 65–170, Jan. 2018.
  * [7] T. Liu, K. Wang, L. Sha, B. Chang, and Z. Sui, “Table-to-text generation by structure-aware seq2seq learning,” in _Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence_. AAAI Press, 2018, pp. 4881–4888.
  * [8] R. Puduppully, L. Dong, and M. Lapata, “Data-to-text generation with content selection and planning,” in _The Thirty-Third AAAI Conference on Artificial Intelligenc_. AAAI Press, 2019, pp. 6908–6915.
  * [9] C. Rebuffel, L. Soulier, G. Scoutheeten, and P. Gallinari, “A hierarchical model for data-to-text generation,” in _Advances in Information Retrieval_. Cham: Springer International Publishing, 2020, pp. 65–80.
  * [10] Y. Z. et. al., “Triple-to-text: Converting RDF triples into high-quality natural languages via optimizing an inverse KL divergence,” in _Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval_. ACM, 2019, pp. 455–464.
  * [11] L. F. R. Ribeiro, M. Schmitt, H. Schutze, and I. Gurevych, “Investigating pretrained language models for graph-to-text generation,” _ArXiv_ , vol. abs/2007.08426, 2020.
  * [12] R. Lebret, D. Grangier, and M. Auli, “Neural text generation from structured data with application to the biography domain,” in _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_. Austin, Texas: Association for Computational Linguistics, Nov. 2016, pp. 1203–1213.
  * [13] S. Gehrmann, F. Dai, H. Elder, and A. Rush, “End-to-end content and plan selection for data-to-text generation,” in _Proceedings of the 11th International Conference on Natural Language Generation_. Tilburg University, The Netherlands: Association for Computational Linguistics, Nov. 2018, pp. 46–56.
  * [14] J. Novikova, O. Dušek, and V. Rieser, “The E2E dataset: New challenges for end-to-end generation,” in _Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue_. Association for Computational Linguistics, 2017, pp. 201–206.
  * [15] C. Gardent, A. Shimorina, S. Narayan, and L. Perez-Beltrachini, “Creating training corpora for NLG micro-planners,” in _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Vancouver, Canada: Association for Computational Linguistics, Jul. 2017, pp. 179–188.
  * [16] D. R. et. al., “Dart: Open-domain structured data record to text generation,” _ArXiv_ , vol. abs/2007.02871, 2020.
  * [17] G. Klyne and J. Carroll, “Resource description framework (rdf): Concepts and abstract syntax,” 2003.
  * [18] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_. Philadelphia, Pennsylvania, USA: Association for Computational Linguistics, Jul. 2002, pp. 311–318.
  * [19] C.-Y. Lin, “ROUGE: A package for automatic evaluation of summaries,” in _Text Summarization Branches Out_. Barcelona, Spain: Association for Computational Linguistics, Jul. 2004, pp. 74–81.
  * [20] A. Lavie and A. Agarwal, “METEOR: An automatic metric for MT evaluation with high levels of correlation with human judgments,” in _Proceedings of the Second Workshop on Statistical Machine Translation_. Prague, Czech Republic: Association for Computational Linguistics, Jun. 2007, pp. 228–231.
  * [21] M. Popović, “chrF: character n-gram F-score for automatic MT evaluation,” in _Proceedings of the Tenth Workshop on Statistical Machine Translation_. Lisbon, Portugal: Association for Computational Linguistics, Sep. 2015, pp. 392–395.
  * [22] A. Graefe, M. Haim, B. Haarmann, and H.-B. Brosius, “Readers’ perception of computer-generated news: Credibility, expertise, and readability,” _Journalism_ , vol. 19, no. 5, pp. 595–610, 2018.
  * [23] A. Srinivasan, S. M. Drucker, A. Endert, and J. Stasko, “Augmenting visualizations with interactive data facts to facilitate interpretation and communication,” 2018.
  * [24] A. Srinivasan, H. Park, A. Endert, and R. C. Basole, “Graphiti: Interactive specification of attribute-based edges for network modeling and visualization,” vol. 24, no. 1, pp. 226–235, 2017.
  * [25] S. Kent, _Words of Estimative Probability_ , 1964.
  * [26] R. J. Heuer Jr, “Analysis of competing hypotheses,” _Psychology of Intelligence Analysis_ , pp. 95–110, 1999.
  * [27] R. Metoyer, Q. Zhi, B. Janczuk, and W. Scheirer, “Coupling story to visualization: Using textual analysis as a bridge between data and interpretation,” in _Proceedings of the ACM Conference on Intelligent User Interfaces_ , 2018, pp. 503–507.
  * [28] M. F. Dahlstrom, “Using narratives and storytelling to communicate science with nonexpert audiences,” vol. 111, no. Supplement 4. National Acad Sciences, 2014, pp. 13 614–13 620.
  * [29] E. Goldberg, N. Driedger, and R. Kittredge, “Using natural-language processing to produce weather forecasts,” _IEEE Expert_ , vol. 9, pp. 45–53, 1994.
  * [30] S. G. Sripada, E. Reiter, J. Hunter, J. Yu, and I. P. Davy, “Modelling the task of summarising time series data using ka techniques,” in _Applications and Innovations in Intelligent Systems IX_. Springer, 2002, pp. 183–196.
  * [31] J. Yu, E. Reiter, J. Hunter, and S. Sripada, “Sumtime-turbine: a knowledge-based system to communicate gas turbine time-series data,” in _International Conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems_. Springer, 2003, pp. 379–384.
  * [32] S. G. Sripada, E. Reiter, J. Hunter, and J. Yu, “Summarizing neonatal time series data,” in _10th Conference of the European Chapter of the Association for Computational Linguistics_. Budapest, Hungary: Association for Computational Linguistics, Apr. 2003.
  * [33] S. Sripada, E. Reiter, J. Hunter, and J. Yu, “Generating english summaries of time series data using the gricean maxims,” in _KDD ’03_ , 2003.
  * [34] H. Grice, “Logic and conversation,” _Syntax and Semantics_ , vol. 3, pp. 41–58, 1975.
  * [35] E. Reiter, S. Sripada, J. Hunter, J. Yu, and I. Davy, “Choosing words in computer-generated weather forecasts,” _Artificial Intelligence_ , vol. 167, no. 1-2, pp. 137–169, 2005.
  * [36] J. Kacprzyk, A. Wilbik, and S. Zadrożny, “Linguistic summarization of time series using a fuzzy quantifier driven aggregation,” _Fuzzy Sets and Systems_ , vol. 159, no. 12, pp. 1485–1499, 2008.
  * [37] R. Castillo Ortega, N. Marín, D. Sánchez, and A. G. Tettamanzi, “Linguistic summarization of time series data using genetic algorithms,” in _EUSFLAT_ , vol. 1, no. 1. Atlantis Press, 2011, pp. 416–423.
  * [38] H. Banaee, M. U. Ahmed, and A. Loutfi, “A framework for automatic text generation of trends in physiological time series data,” in _2013 IEEE International Conference on Systems, Man, and Cybernetics_. IEEE, 2013, pp. 3876–3881.
  * [39] N. Dubey, S. Chakraborti, and D. Khemani, “Textual summarization of time series using case-based reasoning: a case study,” in _Workshop on Reasoning about Time in CBR-RATIC_ , 2018, pp. 164–174.
  * [40] N. Muralidhar et. al., “Cut-n-reveal: Time series segmentations with explanations,” _ACM Transactions on Intelligent Systems and Technology (TIST)_ , vol. 11, no. 5, pp. 1–26, 2020.
  * [41] E. J. Keogh, S. Chu, D. Hart, and M. Pazzani, “Segmenting time series: A survey and novel approach,” 2002.
  * [42] J. Franceschi, A. Dieuleveut, and M. Jaggi, “Unsupervised scalable representation learning for multivariate time series,” in _Advances in Neural Information Processing Systems_ , 2019, pp. 4652–4663.
  * [43] C.-C. e. a. Yeh, “Matrix profile i: all pairs similarity joins for time series: a unifying view that includes motifs, discords and shapelets,” in _2016 IEEE 16th international conference on data mining (ICDM)_. Ieee, 2016, pp. 1317–1322.
  * [44] A. V. Benschoten, A. Ouyang, F. Bischoff, and T. Marrs, “Mpa: a novel cross-language api for time series analysis,” _Journal of Open Source Software_ , vol. 5, no. 49, p. 2179, 2020.
  * [45] X. Qiu, T. Sun, Y. Xu, Y. Shao, N. Dai, and X. Huang, “Pre-trained models for natural language processing: A survey,” 2020.
  * [46] A. V. et. al., “Attention is all you need,” in _Advances in Neural Information Processing Systems_ , I. Guyon, U. von Luxburg, S. Bengio, H. M. Wallach, R. Fergus, S. V. N. Vishwanathan, and R. Garnett, Eds., 2017, pp. 5998–6008.
  * [47] T. e. a. Wolf, “Transformers: State-of-the-art natural language processing,” in _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing_ , 2020, pp. 38–45.
  * [48] C. R. et. al., “Exploring the limits of transfer learning with a unified text-to-text transformer,” _Journal of Machine Learning Research_ , vol. 21, no. 140, pp. 1–67, 2020.
  * [49] M. e. a. Lewis, “BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension,” in _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , Jul. 2020, pp. 7871–7880.
  * [50] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language models are unsupervised multitask learners,” 2019.
  * [51] M. Kale and A. Rastogi, “Text-to-text pre-training for data-to-text tasks,” in _Proceedings of the 13th International Conference on Natural Language Generation_. Dublin, Ireland: Association for Computational Linguistics, Dec. 2020, pp. 97–102.
  * [52] C. Wang, X. Liu, and D. Song, “Language models are open knowledge graphs,” _ArXiv_ , vol. abs/2010.11967, 2020.
  * [53] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell _et al._ , “Language models are few-shot learners,” _arXiv preprint arXiv:2005.14165_ , 2020.
  * [54] A. Fan, M. Lewis, and Y. Dauphin, “Hierarchical neural story generation,” in _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics_. Melbourne, Australia: Association for Computational Linguistics, Jul. 2018, pp. 889–898.
  * [55] A. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi, “The curious case of neural text degeneration,” in _8th International Conference on Learning Representations, Addis Ababa, Ethiopia_ , 2020.
  * [56] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_ , Y. Bengio and Y. LeCun, Eds., 2015.
  * [57] R. Flesch, “A new readability yardstick.” _The Journal of applied psychology_ , vol. 32 3, pp. 221–33, 1948.
  * [58] M. Tory and T. Moller, “Evaluating visualizations: do expert reviews work?” _IEEE computer graphics and applications_ , vol. 25, no. 5, pp. 8–11, 2005\.