# HINTs: Sensemaking on large collections of documents with Hypergraph
visualization and INTelligent agents

Sam Yu-Te Lee, Kwan-Liu Ma Sam Yu-Te Lee and Kwan-Liu Ma are with the
Department of Computer Science, University of California at Davis, One Shields
Ave., Davis, California 95616\. E-mail: ytlee<EMAIL_ADDRESS>
###### Abstract

Sensemaking on a large collection of documents (corpus) is a challenging task
often found in fields such as market research, legal studies, intelligence
analysis, political science, computational linguistics, etc. Previous works
approach this problem either from a topic- or entity-based perspective, but
they lack interpretability and trust due to poor model alignment. In this
paper, we present HINTs, a visual analytics approach that combines topic- and
entity-based techniques seamlessly and integrates Large Language Models (LLMs)
as both a general NLP task solver and an intelligent agent. By leveraging the
extraction capability of LLMs in the data preparation stage, we model the
corpus as a hypergraph that matches the user’s mental model when making sense
of the corpus. The constructed hypergraph is hierarchically organized with an
agglomerative clustering algorithm by combining semantic and connectivity
similarity. The system further integrates an LLM-based intelligent chatbot
agent in the interface to facilitate sensemaking. To demonstrate the
generalizability and effectiveness of the HINTs system, we present two case
studies on different domains and a comparative user study. We report our
insights on the behavior patterns and challenges when intelligent agents are
used to facilitate sensemaking. We find that while intelligent agents can
address many challenges in sensemaking, the visual hints that visualizations
provide are necessary to address the new problems brought by intelligent
agents. We discuss limitations and future work for combining interactive
visualization and LLMs more profoundly to better support corpus analysis.

###### Index Terms:

Text visualization, sensemaking, hypergraph, hierarchical clusters, corpus
analysis, large language models

##  I Introduction

Text data is ubiquitous. From news articles and social media posts to
scientific publications, the tremendous amount of text data that is produced
poses not only opportunities but also a great challenge to anyone who needs to
analyze them. Visual Analytics (VA) mitigates this challenge by combining
mathematical models and visualizations to automate the sensemaking process and
reduce the cognitive load. Model Alignment [18], the alignment of analysis
tasks, visual encodings, and model choices, greatly affects users’
interpretation and trust in visual analytic systems. In modern text analysis,
the models that affect visual analytics system design is often replaced with
the Natural Language Processing (NLP) tasks that Deep NLP models (e.g., BERT
[21]) can perform, such as named entity recognition, topic modeling, or
sentiment analysis. However, these NLP tasks often align poorly with users’
analysis tasks. For example, topic models are commonly used to model the
topical structure of text documents. Most topic models characterize topic as a
probabilistic distribution spanning a given vocabulary [53]. This
transformation from a topic, an apprehensible concept, to a probabilistic
distribution, a mathematical concept that models can operate on, hinders
proper model alignment. The misalignment between NLP tasks and user tasks
limits the usage of visual analytics systems for users who are not familiar
with the underlying models.

Recent advances in large language models (LLMs) present a promising solution
to this problem. LLMs have proven their performance as intelligent agents in
open-context question-answering (prompting) due to their strong capability to
understand user intent [11]. Visualization researchers have adopted them to
assist data transformation [55] or directly generate visualization [36], in
which LLMs are used as “Agents” that delegate some tasks. However, applying
LLMs in the data-preparation stage of the visualization pipeline is under-
explored. Topics [4], sentiments [9], concepts and entities [43, 14] are
common analysis targets in text analysis that require a dedicated data
preparation stage. Researchers have been using NLP models to extract them, but
by using LLMs as a general-purpose NLP task solver, visualization creators can
now extract more than just topics, sentiments, or entities. With prompting,
visualization creators can extract precisely what they want to visualize from
the documents. It also enables them to provide domain-specific contexts for
the extraction, easily achieving human-level extraction results automatically.
In the previous example, instead of relying on abstruse and unfathomable
probabilistic models, visualization creators can process the text data and
summarize the topics of the documents using a prompt. A visualization creator
can ask in a prompt: “What are the topics of these articles?”, and the model
would give a human-like response, such as “The articles are about Covid-19”.
If there is a pre-defined list of topics in mind, it can be added directly in
the prompt and the model can assign topics accordingly.

Such advance in data processing capabilities opens many doors for
visualization creators. In this work, we present HINTs (Hypergraph
visualization and INTelligent agents), a VA system that combines LLM’s ability
both as “Agents” and as general NLP task solvers to help users make sense of a
large corpus. The HINTs system models a corpus as a hypergraph, where the
nodes are documents and salient keywords. Once the hypergraph is constructed,
it is then hierarchically clustered and visualized with enhanced space-filling
curve layouts [37]. We showcase how LLMs are used in the data preparation
stage to take into account the user analysis tasks and visual encodings. The
system supports interactive exploration and reorganization of the documents,
after which users can use an intelligent chatbot agent to conduct further
analysis of the documents. To the best of our knowledge, no published visual
analytics system has adopted LLMs for Model Alignment. Using the system, we
demonstrate two different ways that integrate LLMs into a VA system and
advocate for their incorporation into other parts of the visualization
pipeline.

The contributions of our work are as follows:

  * •

We introduce an LLM-based data preparation pipeline that is capable of
extracting topics and salient keywords from a given corpus for better model
alignment.

  * •

We extend space-filling curve layouts to visualize hierarchical clusters in
large hypergraphs interactively.

  * •

We develop a novel VA system, HINTs, that facilitates users to make sense of a
corpus. From the user study of HINTs, we report our observations on user
behavior patterns and insights on the benefit of visualization.

##  II Related Works

Our work builds upon existing visual analytic systems for document
intelligence and advances in the application of LLMs. We first give a
breakdown of the data preparation approaches in existing visual analytic
works, then switch focus to the visual designs in those works. Finally, we
introduce existing advances in LLMs in different application scenarios, with a
focus on using them as a general NLP task solver.

###  II-A Data Preparation for Large Collections of Text

####  II-A1 Topic-based approaches

Topic-based approaches employ variations of topic models to organize the
documents. Each topic is often presented as a “bag of words”, which can be in
the form of a sequence of words [58, 15, 22, 26, 3, 32] or word clouds [41,
15]. The modeling result is used as an overview of the corpus for subsequent
analysis tasks. Despite their popularity, using topic modeling results as an
overview, as well as its “bag-of-word” representation, is reported by Lee et
al. [33] to be problematic, especially for non-expert users. Chuang et al.
[18] concluded that these problems arise from a misalignment between the
analysis task, visual encoding, and model. Sensemaking is challenging without
a basic understanding of the model because the “bag-of-words” representation
diverges from the user’s mental model of a topic. This misalignment limits the
usage of topic models for non-expert users and makes the system prone to
producing false insights.

####  II-A2 Entity-based approaches

A line of work that makes successful model alignments is the entity-based
approach. ‘Entities’ usually include named entities (people, organizations,
locations), or meaningful concepts known to an existing knowledge base. The
earliest of such approaches is Jigsaw [49], where entities are linked if they
appear in the same document. FacetAtlas [14] generalizes the idea of entity to
‘facets’ which can be entities or any keywords or user’s interest.
ConceptVector [43] uses ‘concept’ to represent a similar idea. Generally,
entity-based approaches exhibit better model alignments than topic-based
approaches [18], but the polysemy of natural language makes them prone to
produce false positives [43]. In our work, we use keywords to represent
entities or concepts that appear in the documents. Previous works use
computational metrics such as TF-IDF or saliency [17] to extract keywords.
However, computational metrics often cannot encode high-level semantics very
well, causing them to sometimes unalign with human judgments. Instead, we rely
on LLMs’ ability to understand natural language to identify significant
keywords from unstructured texts. We ensure that the keywords are salient by
exploiting LLM’s ability to understand semantic contexts.

####  II-A3 Embedding-based approaches

Finally, an important line of work organizes documents by directly modeling
their semantic similarity [50]. Documents are first projected into a high-
dimensional vector space where similarity can be measured, and then a
dimensionality reduction technique (e.g., t-SNE) is used to project the
documents back into a two-dimensional space for visualization. Earlier works
construct a sparse vector using term-frequency based scores such as TF-IDF or
BM25 [16, 48]. More recently, the success of pre-trained language models like
BERT [21] popularizes the idea of embedding documents in a dense vector space
[39, 51, 45]. The embedding can then be used for document retrieval [30, 29]
or visualization. Embedding-based approaches also exhibit proper model
alignment, as the semantic distance in vector space fits the user’s mental
model in the analysis task (i.e., finding similar documents). However, the
result often lacks explainability and prevents users from trusting the result.
Recently, Raval et al.[46] proposed the use of LLMs to provide explainability
to embeddings-mappings visualizations. We adopt a similar approach in our
system to provide interpretability to the clustering result.

###  II-B Visual Interfaces for Corpus Analysis

In this section, we review visual interfaces designed for the exploration or
analysis of a corpus. Even though there are some overlaps with the previous
section, here we focus on their visual representations.

Since the most prominent technique for organizing a corpus is topic modeling,
many of the visual representations are centered around the topic modeling
results, most of them involving topic co-occurrences or hierarchies. Serendip
[3] uses a reorderable matrix of documents and topics for the exploration of
topic occurrences and their importance. HierarchicalTopics [22] uses a tree
visualization where each node is a topic to visualize the topic hierarchies.
Due to the topic model’s “bag-of-word” representation, many previous works use
word clouds in their visualization. SolarMap and FacetAtals [13, 14] use a
radial word cloud for word appearances in topics. VisTopic [58] combines a
sunburst diagram and a modified tag cloud to visualize the topic hierarchy and
word occurrences at the same time. An exception is PhraseMap [52], where a map
metaphor is used to visualize the extracted technical phrases. Similar to our
approach, they use a Gosper curve to create the layout. Different from topic-
and entity-based approaches, embedding-based approaches typically employ
dimensionality reduction techniques and use a scatter plot as their primary
visualization [43, 12, 38, 16].

One key distinction of the visualization between topic-based and embedding-
based approaches is the unit of abstractions. In topic-based approaches, the
unit of abstractions is “bag-of-words” (topics) and their hierarchies. In
embedding-based approaches, the unit of abstraction is the documents. We note
that for a better model alignment in the context of corpus sensemaking and
analysis, the unit of abstraction must be the documents since the unit of
analysis is the document. Further discussion on this matter is presented in
Section III.

###  II-C LLMs as General NLP Task Solvers

LLMs have been proven to excel at many NLP tasks, such as sentiment analysis
or summarization, and even more complicated tasks that in the past could only
be done by humans. Gilardi et al. [24] reported that ChatGPT can generate
annotations that outperform crowd-workers on commercial platforms by about 25
percent accuracy on average, and cost over thirty times less financially.
Beyond simple annotations, Wu et al. [56] shows that LLMs can replace multi-
stage crowdsourcing pipelines with careful prompt designs. The NLP tasks
closest to our work involve Information Extraction (IE), a line of NLP tasks
that aims to identify structured information of interest from unstructured
text. Some of its subtasks include Named Entity Recognition (NER), Relation
Extraction (RE) and Event Extraction (EE) [40, 57]. Li et al. [34]
comprehensively evaluated the capabilities of ChatGPT for common IE tasks.
They found that the model excels under the Open-IE setting, where the model
directly extracts information from documents but performs poorly under the
Standard-IE setting, where the model chooses a correct label from candidate
labels. Similarly, Zhang et al. [59] reported that ChatGPT performs poorly on
extractive summarization, where sentences in the text are treated as
“candidates” and the model is instructed to select sentences to form
summarization. A common reason for the poor performance in these tasks is that
they are essentially supervised learning tasks, tasks that the model is not
trained to perform. On the other hand, some works have shown the limitations
of LLMs such as “hallucination” [8] and “faithfulness” [60] issue reported by
existing works. To make the best use of ChatGPT for NLP tasks, we are informed
by these early experiments and carefully design our extraction tasks
accordingly.

Figure 1: Data processing pipeline of HINTs. Starting from a corpus of
unstructured texts, each document goes through the data preparation stage to
extract the salient keywords. Then the documents and keywords are both
embedded into a vector space. The model stage constructs a document hypergraph
and a keyword hypergraph, which are then clustered separately by combining
connectivity similarity and semantic similarity. The clustered hypergraphs are
hosted on the server and visualized in the user interface. Users can expand,
filter, or search the hypergraphs to explore the corpus and select documents
to be analyzed with a chatbot.

##  III Design Rationale

The HINTs system is designed for experts and analysts to make sense of a
corpus. Our design rationale is based on the design guidelines proposed by
Chuang et al. [18]. We reuse their definitions of Model Alignment, Progressive
Disclosure, and Unit of Analysis when describing our design rationale. We
first identify common analysis tasks from previous works. Then, we derive our
design considerations (DC) from the analysis tasks. We take the DCs into
account during the data preparation and visualization design process in
Section IV.

###  III-A Analysis Tasks

We derive the analysis tasks from topic- and entity-based approaches. Topic-
based approaches aim to support document understanding by visualizing the
topic structure of the documents. Entity-based approaches support
investigative analysis by visualizing entity occurrences and relations. We aim
to support both tasks simultaneously as they are fundamental to subsequent
tasks and intertwined in a real-world scenario.

We first generalize entities to Keywords, which are the core entities or
concepts that appear in the documents. For example, in a news article, the
keywords can be named entities in the title or protagonists of the news event.
In a research article, the keywords can be the proposed models or employed
techniques. This definition of keywords is generalizable enough for us to take
user intent into account during the extraction. Building around topics and
keywords, our system designs visualizations and interactions to support the
sensemaking of a corpus. Specifically, we support the following four analysis
tasks:

T1: Understanding topic structures. Topic-based approaches have shown that the
topic structures of a corpus are essential to the sensemaking of the corpus.
Each topic, consisting of a subset of documents, represents a significant
semantic facet occurring in the corpus. By breaking down a corpus into topics,
users break down the cognitively demanding sensemaking process into feasible
subtasks in which they incrementally understand a single topic.

T2: Understanding keyword occurrences. Keywords also play an essential role in
the sensemaking process, as shown by the entity-based approaches, so we treat
them with equal importance to the topics. A subtle difference between keywords
and topics is that keywords are carried by the documents, while topics carry
the documents. Thus from the user’s mental model, the keyword’s occurrences
are of interest, and the understanding of such occurrences contributes to the
understanding of a document, a topic, and the whole corpus.

T3: Navigating to documents of interest. A corpus, by its definition, contains
too many documents for users to quickly find their target of interest. Through
the understanding of topic structures and keyword occurrences, the target of
interest is found and a subsequent step is to navigate to the target, either
to verify or deepen existing understanding or resolve confusion. Although
navigation does not directly contribute to sensemaking, it is an essential
step to making progress.

T4: Understanding the content of a document and the meaning of a keyword. The
most fine-grained level of sensemaking is to understand the content of a
document and the meaning of a keyword. This can occur in cases where users
need to get a clearer understanding of a topic or meet an unknown keyword that
hinders a clear understanding. Previous works often ignore such a task or
assume that the users already have enough knowledge to understand a topic or a
keyword, which is questionable. We include this task to complete the
sensemaking process and fill the gaps in existing research.

###  III-B Design Considerations

To support the aforementioned analysis tasks, we derive the following design
considerations (DCs):

  * •

DC1: Overview of topic structures and keyword occurrences (T1, T2). Given a
corpus, the topic structures and keyword occurrences can be complex and cover
a wide range of documents and keywords. The overview seeks to cover all the
documents and keywords while hiding the details. This sets the ground for the
user to discover their targets of interest.

  * •

DC2: Progressive Disclosure (T1, T2, T3). To facilitate incremental
sensemaking, it is important to support users to drill down from a high-level
overview to intermediate abstractions. This includes disclosure of a specific
topic’s sub-structure, the containing documents, and the occurrences of
keywords. It not only supports a better understanding of the topics and
keywords but also directs users to find the target of interest.

  * •

DC3: Model Alignment (T1, T2). Having good interpretability for the presented
topics and keywords is essential to support sensemaking, which is enabled by
carefully considering Model Alignment. Our choice of models should align well
with the user’s mental model when conducting the analysis tasks for better
interpretability. This entails that our model should directly operate on the
units of analysis (i.e., documents and keywords)

  * •

DC4: Detailed analysis of the target of interest (T4). The investigation of
topic structures and keyword occurrences often leads to a target of interest,
which is a subset of documents. After such investigation, previous works
usually only provide the user with a list of these documents. This is perhaps
due to the lack of a unified way to analyze the target of interest under
different contexts. The advance of LLMs presents a promising solution to this
problem by transforming any analysis task into a question-answering task,
enabling us to consider T4 in our system design.

##  IV Methodology

As shown in Figure 1, starting from a corpus of unstructured texts, we first
use LLMs to extract the main keywords from each document. The keywords are
disambiguated and linked to a knowledge base if available. We create and store
document embeddings and keyword embeddings for further usage.

All LLM-based preprocessing used OpenAI’s “gpt-3.5-turbo-16k-0613” model. We
provide all our prompts in supplemental material. In the next stage, we
construct a document hypergraph and a keyword hypergraph, which are then
clustered separately by combining connectivity similarity and semantic
similarity. The clustering result is hosted on the server and visualized in an
interactive user interface. During the development of HINTs, we tested the
pipeline on two datasets from different domains: a news article dataset
(AllTheNews) [1] and a visualization publication dataset (VisPub) [28]. In all
the prompt-based extraction modules, we used two different versions of prompts
to take the user’s analysis tasks into account and guarantee optimal
extraction results for the users. Below, we describe each component in detail.

###  IV-A Data Preparation

The data preparation stage consists of five steps: summarization, keyword
extraction, keyword disambiguation, document and keyword embedding, and topic
label generation. We use LLMs to perform all five steps as they outperform
traditional approaches in accuracy and flexibility. Additionally, we discuss
how we design these tasks while taking the users’ analysis tasks (Subsection
III-A) into consideration.

####  IV-A1 Summarization

During sensemaking, users need a quick understanding of the content of the
selected documents (T1). Summarization helps condense the content into a
concise form, making it easier for users to make sense of. Zhang et al[59]
reported that human users found the summaries generated by ChatGPT to be more
interpretable and trustworthy than traditional approaches, making LLMs a
better choice for our purpose. We employed an instruction-based zero-shot
prompt, where the model is instructed to act as a text summarizer. The
summarized documents are also used in subsequent data preparation pipelines
for other prompts to better process and extract the most important
information.

####  IV-A2 Keyword Extraction

We define keywords as phrases that are significantly discussed in the
documents. We highlight the importance of “significantly discussed” here, as
they serve as a unit of analysis in the user’s tasks (T2). The keywords are
extracted with prompts, building upon the summaries generated in the previous
step. We use slightly adjusted versions of the description of a “keyword” in
the prompts according to the dataset domain, making them even more tightly
connected with the user’s analysis tasks. For news articles, keywords are
named entities or protagonists in the news event. For research papers,
keywords are models, techniques, or algorithms proposed or used by the paper.

In addition, we break down the process of keyword extraction into two steps to
avoid the problem of extrinsic hallucination as discussed by Bang et al. [8].
Below, we take keyword extraction in a news article dataset as an example.
First, a sentence describing the most important event is generated from the
summary. The event should be the primary focus of the news article, such as a
geopolitical event like the G20 summit. A few-shot prompt was adopted to give
the LLM examples of what kind of events would be described as the main event.
Next, all the main keywords involved in that event are extracted with another
prompt, injecting the main event extracted in the previous step: “A major
event is reported by a news article: {event_description}, what are the main
keywords that are majorly involved in the event?”. This provides the model
with more context and yields better accuracy and lower chances of generating a
hallucinated output.

####  IV-A3 Keyword Disambiguation

Once the main keywords are extracted, a disambiguation step is done to create
connections between documents (T2). In early experiments, we found that LLMs
do not excel at disambiguating keywords. We thus prioritize using an entity
linking model [47] to disambiguate keywords. Existing models adopt a
supervised learning approach which requires training data, so they are usually
limited to a specific domain. For datasets where an existing entity-linking
model [6] is available, we use it to disambiguate keywords. Otherwise, we
employ an embedding-based matching algorithm. The algorithm is
straightforward: For every extracted keyword, a first prompt generates an
explanation of the keyword in a few sentences. Then the explanations are
embedded to create a pair-wise similarity matrix. The pairs of keywords with a
similarity score above a threshold are considered to be the same keyword. Then
a second prompt generates a unified title for the matched keywords to be
displayed in the user interface.

####  IV-A4 Document and Keyword Embedding

Embeddings are dense vectors that can be used to measure similarities between
objects, which are often created on words, sentences, or documents in text
analysis. We adopted a similar approach here for both documents and keywords.
For documents, we embed the summaries generated in Subsubsection IV-A1. For
keywords that can be linked to an external knowledge base, we embed the
description of the keyword in the knowledge base. Otherwise, we embed the
explanation of the keyword generated with a simple prompt: “What is
{keyword}?”. While previous works embed the keywords directly, we found that
embedding the explanations produces more semantically precise results. We used
OpenAI’s “text-embedding-ada-002” model for all embeddings, allowing us to
measure the semantic similarity between documents, keywords, and user queries
in the same vector space. The document and keyword embeddings are used
subsequently to generate semantic clusters and visualized to support T1 and T2
as described in Subsubsection IV-B2.

####  IV-A5 Topic Label Generation

Later in the preprocessing pipeline, an agglomerative clustering algorithm is
used to cluster the documents and keywords (Subsubsection IV-B2). The
algorithm outputs a hierarchy, where leaf nodes are documents or keywords and
internal nodes are clusters. However, the clusters are not readily
interpretable for the users, hindering the support of T1. To address this, we
assign human-readable labels to each cluster with prompts. Our approach is
similar to a recent work by Raval et al. [46], where LLMs are used to generate
explanations for user-selected points. However, our task is more complicated
because the prompt needs to (1) consider the hierarchical structure, and (2)
be able to process large clusters containing thousands of documents without
breaking the token limit.

Our topic label generation follows a bottom-up approach: (1) At the bottom
level (one level above the leaf nodes), assign topic labels to the clusters
using the document summaries. The bottom-level clusters usually contain only a
few documents, so the token limit is unlikely to be exceeded. We insert the
summaries generated in Subsubsection IV-A1 into the prompt, then instruct the
LLM to generate a label composed of a single noun phrase for the given
documents. (2) At any level above, the cluster size increases quickly and the
token limit is likely to be exceeded. We thus assign labels to the clusters
using the labels of their children and randomly sampled document summaries.
This guarantees that the token limit will not be exceeded. In early
experiments. we found that the children’s labels alone were not enough to
generate a meaningful label for the parent cluster. To improve, we insert the
document summaries randomly sampled from the cluster. We enforce that each
child has at least one article being sampled, and distribute the remaining
token space proportionally to the size of each child.

Figure 2: Illustration of the dual of a hypergraph. Left: a hypergraph with 4
nodes $v_{1},v_{2},v_{3},v_{4}$ and 3 hyperedges
$e_{1}=(v_{1},v_{2},v_{3}),e_{2}=(v_{2},v_{3}),e_{3}=(v_{3},v_{4})$. Right:
the dual of the hypergraph. Nodes in the dual hypergraph are now
$e_{1},e_{2},e_{3}$ and hyperedges are
$v_{1}=(e_{1}),v_{2}=(e_{1},e_{2}),v_{3}=(e_{1},e_{3}),v_{4}=(e_{2},e_{3})$.

###  IV-B Models

####  IV-B1 Hypergraph Construction

A hypergraph is a generalization of a graph in which an edge can connect any
number of nodes [23]. A hyperedge thus represents a multi-way relationship
between nodes. In our work, we model two types of hypergraphs: document
hypergraph and keyword hypergraph. In a document hypergraph, nodes are
documents and hyperedges are keywords, and symmetrically in a keyword
hypergraph, nodes are keywords and hyperedges are documents. By modeling the
corpus as hypergraphs, the system supports users to conduct both topic- and
entity-based analysis simultaneously under a unified framework (DC1). Below,
we explain the relation between these two types of hypergraphs and the
construction process.

A hyperedge can be used to represent two types of multi-way relationships: In
a document hypergraph, a hyperedge (connecting document nodes) can be
constructed between documents that mention the same keyword. In this case, the
hyperedge represents the co-mention of a keyword. In a keyword hypergraph, a
hyperedge (connecting keyword nodes) can be constructed between keywords if
they are mentioned together in the same document. In this case, the hyperedge
represents a co-occurrence relationship between keywords. In other words, the
two hypergraphs are built from the same corpus, using the same relationship
information, but from different perspectives. This allows us to utilize the
dual of a hypergraph to simplify the construction process. The dual of a
hypergraph is simply another hypergraph, where the nodes and hyperedges are
interchanged, as shown in Figure 2. Using the duality feature, we first model
the documents as nodes and keywords as hyperedges to construct the document
hypergraph $H_{D}$. The keyword hypergraph $H_{C}$ can then be easily
constructed by taking the dual of $H_{D}$. This construction process also
allows us to map user interactions on documents and keywords to operations on
a single hypergraph (DC1). Even better, the duality feature between documents
and keywords matches the user’s mental model (DC3), making the visualization
and interaction intuitive.

####  IV-B2 Hierarchical Clustering

Once the two hypergraphs are constructed, we separately hierarchically cluster
them using an agglomerative clustering algorithm. Clusters in the document
hypergraph represent topics that are discussed in the dataset. Clusters in the
keyword hypergraph represent keywords (entities or concepts) that are similar
to each other. For better interpretability of the clustering result, we
further assign labels to each cluster, as explained in Subsubsection IV-A5.
Below, we describe our motivation for using an agglomerative clustering
algorithm and its process.

Common clustering algorithms for graphs consider only graph connectivity.
However, for the best interpretability of the clustering result, the node
embeddings must be also used in the clustering process to model semantic
similarity. We thus focus on discussing attributed node clustering algorithms.
Although there are existing approaches that can cluster attributed nodes on
graphs such as EVA [19] and iLouvain [20], they are not designed for
hypergraphs. Although Kamiński et al. [10] have shown that it is feasible to
generalize the modularity metric for graphs to hypergraphs (thus generalizing
modularity-based clustering algorithms), we found in early experiments that it
is not scalable and hard to incorporate node attributes. Finally, we decided
to follow the approach proposed by Kumar et al. [31]; i.e., transform the
hypergraph into a graph and apply graph clustering algorithms. We decide to
apply an agglomerative clustering algorithm [50] to support progressive
disclosure of the cluster structures (DC2).

In agglomerative clustering, the key is to define node similarity and cluster
similarity. We can easily incorporate node attributes into the clustering
process by defining the similarity between nodes and clusters as a weighted
sum of attribute similarity $S_{s}$ and connectivity similarity $S_{c}$. Since
we’re dealing with texts, we refer to the attribute similarity between nodes
as semantic similarity. The semantic similarity $S_{s}(i,j)$ is the cosine
similarity of the embeddings (dense vectors) of the two nodes, denoted as
$vec_{i}$ and $vec_{j}$. The connectivity similarity $S_{c}$ is the weighted
Topological Overlap (wTO) [25], which is a weighted generalization of the
Overlap Coefficient [54], as shown in Equation 1.

$\displaystyle\mathbf{S_{s}}(i,j)=\frac{vec_{i}\cdot
vec_{j}}{||vec_{i}||\cdot||vec_{j}||}$ (1a)
$\displaystyle\mathbf{S_{c}}(i,j)=\frac{\sum_{u=1}^{N}{w_{i,u}w_{u_{j}}}+w_{i,j}}{\min(k_{i},k_{j})+1-|w_{i,j}|}$
(1b)

where $k_{i}=\sum_{j=1}^{N}|w_{i,j}|$ is the total weight of the edges
connected to node $i$. Finally, a weighting factor $\alpha$ is used to balance
the two similarities, as shown in Equation 2.

$\mathbf{S}=\alpha\mathbf{S_{s}}+(1-\alpha)\mathbf{S_{c}}$ (2)

For the similarity between clusters, we used centroid similarity, i.e., the
similarity between two clusters is the similarity between the centroids of the
two clusters. The algorithm takes a hypergraph $H=(V,E)$ and the embeddings of
each node as input, and outputs a sequence of partitions $P=P_{1},P_{2},\dots
P_{k}$. Each partition corresponds to a level in the hierarchy. Pseudo-codes
of the algorithm can be found in supplemental materials.

Figure 3: The HINTs system. (a) The peripheral area of Cluster View shows the
mentioned keywords of highlighted documents using Gilbert curves. (b) The
center area of Cluster View shows the topic structure of the corpus using
Gosper curves. (c) The Document View shows a list of selected documents. (d)
The Chatbot View provides a chatbot interface to answer user questions with
the option to insert selected documents in the prompt.

##  V Visualization of HINTs

The visualization of HINTs extends the existing Space-Filling Curve (SFC)
layout for large graphs proposed by Muelder et al. [37]. Using the duality of
hypergraphs, our visualization layout treats documents and keywords as both
nodes and edges and uses two different curves to lay out the two hypergraphs.
We describe our motivation for using such a layout method and our techniques
to improve the readability.

###  V-A SFC for HyperGraph

In the design guidelines derived by Abdelaal et al. [2] in a recent network
visualization evaluation study, node-link-based approaches are recommended
when: (1) tasks involve the identification of network clusters, and (2) the
network is sparse. Condition (1) is fulfilled as explained in Section III, and
(2) is guaranteed by the keyword extraction process described in Subsubsection
IV-A2. Therefore, we decided to use node-link-based approaches for our system.

In all the variations of node-link-based approaches for hypergraph
visualization [23], we find the extra-node representation introduced by
Ouvrard et al. [42] most flexible and intuitive. An extra-node representation
improves the existing clique expansion of hypergraphs by adding extra nodes to
represent hyperedges, effectively transforming the hypergraph visualization
problem into a bipartite graph visualization problem. After that, any node-
link-based graph visualization method can be applied. In our system, we use
the SFC layout method to lay out the extra-node representation of the
hypergraph. The SFC layout method uses pre-computed clustering to order nodes
in a sequence and then applies a space-filling curve on the node sequence to
map it to a two-dimensional screen space [37]. SFC layout is known for its
efficiency and aesthetics in visualizing large graphs [35]. Moreover, SFC
layouts support progressive disclosure (DC2) organically, as the layout is
generated based on the clustering result (DC3). The preprocessing and modeling
stage (Section IV) generates the document hypergraph $H_{D}$ and the keyword
hypergraph $H_{C}$, each with its hierarchical clusters and visualized as two
separate SFCs, as shown in Figure 3.

Specifically, we divide the layout space into two parts: the peripheral and
the center area. For the peripheral area, we concatenate four Gilbert curves
[61]. A Gilbert curve is a generalized version of the Hilbert curve, which
instead of filling a squared area, can traverse any rectangular region in a
way similar to the Hilbert Curve. In Figure 5-a, the first Gilbert curve
starts from the lower left (dark blue) and ends at the lower right (dark red).
Through rotation and flipping, the start and end curve points for neighboring
Gilbert curves are concatenated smoothly. Using concatenated Gilbert curves
enables us to fill a ring-like space with the efficiency and aesthetics of SFC
layouts.

For the center area, we found in early prototyping that using the same curve
as the peripheral region is confusing for the user, as the peripheral and
center areas would become visually indistinguishable. We decided to use a
simple Gosper curve (Figure 5-c) to lay out the nodes for better aesthetics.
The resulting visualization looks similar to GosperMap [5], but we did not
employ the advanced techniques proposed in GosperMap. The interactions to
support the exploration and reorganization of the dataset are the main focus
of the system, which are also not limited to any specific curve. After the
curves are generated, we can apply the curves on the node sequences to
generate the two-dimensional layout. We put the document hypergraph in the
center area because the documents are the main analysis targets for the user.
Consequently, the keyword hypergraph is put in the peripheral area.

Figure 4: Spacing strategy of the SFC layouts. The space of each cluster
$S_{i}$ is proportional to the cluster size. When $C_{i}$ is expanded, its
sub-clusters redistribute $S_{i}$ to limit the changes locally.

###  V-B Improving the readability

####  V-B1 Automatic Cluster Expansion

In most cases, the default clustering result is not optimal for the user’s
targeted analysis tasks. We identify two common problems in early prototyping:
(1) clusters may be too big, weakening the semantic meaning that the clusters
can convey. (2) clusters may have only one sub-cluster, which makes the parent
cluster redundant. Both problems can be mitigated by automatically expanding a
cluster, i.e.breaking the cluster into sub-clusters Figure 6-d. We employ
rule-based detection to identify clusters that need to be expanded. For a
cluster $C$, we expand it if the following conditions are met: (1) $C$ has
only one sub-cluster $C_{s}$; (2) $C$ has more than $n=kN$ nodes, where
$k\in[0,1]$ and $N$ is the size of the hypergraph. Through trial and error, we
find that $k=0.3$ gives the most balanced results.

####  V-B2 Spacing Strategy

Spacing between each node is critical for the readability and aesthetics of
SFC layouts. To highlight different clusters, we employ our spacing strategy
on clusters instead of nodes. Given a space-filling curve of a specific order,
we first calculate the length of the curve $L$, the total amount of space
available for the nodes, and $L-N$, the amount of space to be redistributed,
where $N$ is the size of the hypergraph. Our goal is to distribute the space
between clusters to ensure the best readability. In early prototyping, we
found that distributing the space proportional to the cluster size gives the
best readability as well as stability. Specifically, we define the space of a
cluster as the blank space it has behind it on the curve, which is calculated
by $(L-N)\frac{N_{c}}{N}$, where $N_{c}$ is the size of the cluster (Figure
4). Another consideration when designing our spacing strategy is stability. We
want to ensure that the layout change is minimized when clusters are expanded.
Naively, when a cluster is expanded, the whole layout needs to be recalculated
because now the cluster sequence is changed. To avoid such recalculation, we
design our spacing strategy in this way so that the sub-clusters can simply
take over the space of their parent cluster, as shown in Figure 4. Since the
total volume of the sub-clusters is the size of the parent cluster and the
space is proportional, the sub-clusters can take over exactly the space of the
parent cluster without any overflow or underflow. This ensures a local change
in the layout when a cluster is expanded.

####  V-B3 Concave Hull Approximation

After applying the SFC layout, we use a concave hull algorithm [44] to
generate an approximation polygon for each cluster. The polygons are used to
generate borders and calculate label positions for the clusters. The concave
hull is generated based on a cluster of points: in our case, the nodes in a
cluster. This means that the algorithm can be applied to any curve we used for
the SFC layout. This is desirable because we are using two different curves in
our system, and the center area curve choice is flexible. Using the same
algorithm guarantees a unified aesthetic across curves.

The concave hull algorithm is designed for approximating points, but we need
to approximate circles. Naively we can use the center of the circles as the
points for the algorithm, but this would result in a polygon that is too small
and crosses the circles on the boundary. To address this issue we use a simple
trick: we add extra points to the cluster by extrapolating the original
points. For example, a Gilbert curve moves perpendicularly, so the resulting
polygon would have perpendicular corners. We can therefore add eight extra
points around each original point so that the extrapolation forms a three-by-
three grid. On the boundary of the cluster, these extra points prevent the
concave hull from passing across the original points. Since the concave hull
algorithm has an $O(n\log n)$ time complexity, the performance overhead
introduced by the extra points is negligible.

(a) Four concatenated Gilbert curves. Color transitions from dark blue to dark
red indicate the traverse direction.

(b) An example cluster in Gilbert curve filling layout

(c) An order-4 Gosper curve

(d) An example cluster in Gosper curve filling layout

Figure 5:  Illustrations of the space-filling curves and example clusters
formed by the curves.

####  V-B4 Borders

The borders are generated by applying a smoothing algorithm on the polygons.
For Gosper curves, we use the polygon as control points to generate a cubic
basis spline as the border. For Gilbert curves, we use a similar approach but
with a cubic Bezier curve. More specifically, for each pair of consecutive
points, we use a smoothing factor to interpolate the control point. This
results in a sketchy style at the border corners. Two examples are given in
Figure 5.

####  V-B5 Label Position

Labeling the clusters is essential for users to explore the dataset. We use
the topic assignment described in Subsubsection IV-A5 to label the clusters.
For clusters, we use the centroid of the approximated polygon to position the
label. When a cluster is expanded, the sub-clusters within need to be clearly
labeled as well. Using the centroid of the sub-cluster as the label position
is not a good choice because the label would cause a serious cluttering issue.
Therefore, for a sub-cluster, we first calculate its centroid $C_{s}ub$. Then
we extend the line from the parent cluster centroid $C_{p}$ to $C_{s}ub$ and
calculate the intersection point of the extended line and the parent border.
The intersection point is used to position the sub-cluster label, resulting in
a radial layout, as shown in Figure 6-d. The generalizability of the concave
hull algorithm makes our labeling position calculation applicable to any curve
used for the SFC layout.

##  VI HINTs System Design

The HINTs system consists of three main views: Cluster View, Document View,
and Analysis View. Below, we describe the views, visualizations, and
interactions, and how they assist the exploration and reorganization of a
corpus.

###  VI-A Cluster View

Cluster View is the main view of the HINTs system (Figure 3-a and -b). It
visualizes the corpus as two hypergraphs using the SFC layout. Using the
Cluster View, users can get an overview of the topic structure and keyword
occurrences simultaneously (DC1). Interactions such as expansion, filtering,
and navigation are provided to support DC2. Below, we discuss user
interactions and the coordination with other views.

####  VI-A1 Hover and click

By default, the cluster labels are hidden to reduce clutter. Hovering over the
clusters triggers a highlight effect and shows the cluster label. Users can
select a cluster by clicking on the cluster label. This triggers the Document
View to show articles in the cluster and the mentioned keywords (T3).
Additionally, clicking on the cluster temporarily expands the cluster to
expose its sub-structure (T1), as shown in Figure 6-b. The sub-structure is
colored in different colors while maintaining the original cluster’s shape.
The labels of each sub-cluster are also shown radially. When the user selects
any cluster, the mentioned keywords are highlighted (T2). Under such cases,
hovering over the keyword clusters will show not only the cluster label but
also the highlighted keywords in a list, as shown in Figure 3-a. Users can
also select a keyword to filter the articles in the Document View (T3).

####  VI-A2 Expansion

Users can break a cluster into smaller clusters so that a deeper level of
topic structures can be explored (T1). This operation is necessary since the
agglomerative clustering result is not always semantically optimal, and some
clusters may be too vague for the user. Upon expansion, we use an animation to
show how the parent cluster is broken down into smaller pieces to maintain the
visual memory. The sub-clusters redistribute the spacing of their parent
cluster proportionally to their size, so that the expansion only affects the
layout locally.

####  VI-A3 Filtering

Once the users have found the target of interest, they can click the filter
button to remove irrelevant documents and keywords from the view (T3). The
filtering functionality effectively creates a sub-hypergraph based on node
selection, supporting any interactions that are available in the original
hypergraph. Note that we only support filtering on document node selection and
keyword nodes are filtered accordingly. Although filtering on keyword nodes is
technically possible, we do not find the operation intuitive. We decided to
remove this feature to keep the exploration straightforward.

####  VI-A4 Searching

HINTs supports embedding-search to retrieve documents relevant to user queries
(T3). The search functionality is implemented by first embedding the user
query into the same vector space as the documents, then ranking the documents
based on their relevancy to the user query using cosine similarity, thus
supporting users to query in natural language. The server returns the
documents sorted by relevancy score to the front end, and the user can control
the number of documents to be highlighted by a relevancy threshold. The
highlighted documents and mentioned keywords are visually distinguished in the
Cluster View (Figure 6-b).

###  VI-B Document View

The Document View displays a list of selected articles (Figure 3-c), either
from the same cluster or returned by the search functionality (T3). Each
document is an interactive card with the title, summary, ID, and relevancy
score. The main keywords of the document are highlighted in yellow in the
summary to assist quick browsing. If the user is under search mode, documents
are sorted according to relevancy, and those above the relevancy threshold are
highlighted. Users can click any document card to add or remove it as “query
documents”, which are used in the Analysis View.

###  VI-C Analysis View

The Analysis View (Figure 3-d) is an intelligent chatbot agent that assists
users in analyzing the selected documents (T4). We use the OpenAI
“gpt-3.5-turbo-16k-0613” to generate responses. We designed the interface by
enhancing a typical chatbot interface like ChatGPT with Retreival-Augmented-
Generation (RAG), i.e., users can insert selected documents into the prompt by
selecting document cards or adding a list of documents directly by clicking on
the title of the Document View. The chatbot then gives answers to user
questions based on the selected documents. They can also select if they want
to insert the summary or the full content of the document, in case of
information loss during the summarization. The capability of LLMs to answer
open-domain questions turns out to be more useful than we initially expected
for the sensemaking process, as our users exhibit diverse usage patterns
during the user study. More details are discussed in Section VIII.

##  VII Usage Scenario

We illustrate the use of HINTs through two usage scenarios. We chose two
different domains to demonstrate the flexibility of HINTs. In the first usage
scenario, we highlight the visualizations for interactions to support the
exploration of a news article dataset. In the second usage scenario, we focus
on the usage of the Analysis View, an intelligent chatbot agent to support the
analysis of selected documents. Together, we demonstrate the functionalities
that HINTs support to facilitate the sensemaking of a large corpus.

###  VII-A Scenario 1: Collecting materials for a news story

In this scenario, we use the AllTheNews [1] dataset to demonstrate the
effectiveness of HINTs in assisting the exploration of the topics and keywords
in news articles. We randomly sampled 8192 news articles published in 2016
from the dataset for this case study.

Alice is a journalist currently working on a story about the US presidential
election. She wants to collect articles about the 2016 election as materials
for her story. She can only vaguely remember certain events back when the
election was happening, so she wants to freshen up her memory and find real
evidence to reference. Alice begins by opening the HINTs system displaying the
AllTheNews dataset and searches “US presidential election”. The Cluster View
highlights relevant articles in each of the six topics with clear labels and
colors (Figure 6-a), and she can easily identify topics that are not related
to the election. such as “Ride-hailing Technology” (Figure 6-a, C-2). Before
she filters the whole corpus with current search results, she wants to make
sure that the relevancy threshold is reasonable so that she does not miss too
many relevant articles. The topic “Security Threats in the Middle East”
(Figure 6-a, C-3) catches her attention because foreign policy towards the
Middle East has always been a hot topic in the election, but the visualization
suggests otherwise. She clicks on the topic label to highlight the topic and
keywords. She hovers over each of the highlighted keyword clusters, but she
does not find any keywords related to the election. Upon quickly scanning
through the titles of the highlighted articles, she confirms that the topic is
not related to the election and raises the threshold until the cluster is
filtered out.

Figure 6: User interactions in usage scenario (a) The search result of “US
presidential election”. Relevant articles are highlighted. (b) The user
expands the topic “Controversies and Challenges”, showing the sub-topics
within and mentioned keywords in the peripheral area. (c) After clicking
“Filter Search”, irrelevant articles are removed, and the layout is
regenerated. (d) The user expands a sub-topic “Concerns and Controversies in
the United States”.

After that, she finds the remaining topics too vague to make sense of. She
first drills down the topic “Controversies and Challenges” (Figure 6-a, C-1)
by expanding it (Figure 6-b). She finds that “Criminal Justice” and “Political
Protests against the Trump election” (Figure 6-b, C-1-1 and C-1-2) are clearly
two related sub-topics. Upon reading the summaries of the articles, she
confirms the accuracy of the sub-topic labels and marks them down.

After the inspection, she is confident with the current relevancy threshold.
She clicks the “Filter Search” button to reorganize the corpus (Figure 6-c).
Once again, the topic “Concerns and Controversies in the United States”
(Figure 6-c, C-4) seems to contain a lot of related articles, but she could
not make sense of it solely from the topic label. To dive deeper, she splits
the cluster into separated sub-clusters (Figure 6-d). The sub-cluster “The
2016 US Presidential Election and Controversies” (Figure 6-d, C-4-1) catches
her attention. Upon inspecting the keywords, she becomes curious about how
“Vladimir Putin” is related to the election, so she clicks the keyword label
to inspect the articles. She also finds several interesting topics, such as
“Presidential Eligibility Confusion” (Figure 6-d, C-4-2). By browsing the
highlighted keywords and reading the titles of the articles, she understands
that it was about a controversy between Donald Trump and Ted Cruz during the
election. The wide range of topics brings her memory back, and she continues
to click on every topic she deems relevant to the election to collect
materials for her story.

This case study demonstrates that HINTs provides a highly interpretable
visualization for the topic structure and keyword connections in a corpus.
Alice can quickly understand the topics through the labels, and she can always
drill down to a deeper level for more specific topics if the labels are too
vague. The keyword connections and document summaries provide her with enough
information to decide the relevance of a topic to her story. Next, we
demonstrate how the Analysis View can assist users in analyzing the selected
documents.

Figure 7:  The VisPub corpus visualization was filtered by the keyword “text
analysis” and reorganized by the user in usage scenario 2. After expansion and
filtering, the remaining topics are at a reasonable level of granularity. The
user is currently selecting the orange cluster “Visualization of Web Data,
Textual Data, and Social Media Information” to prompt the Analysis View for
more detail.

###  VII-B Scenario 2: Literature Review

In this usage scenario, we use the visualization publication dataset (VisPub)
[28] to demonstrate how HINTs can assist users in conducting literature
research.

Bob is a Ph.D. student who is currently working on literature research in text
analysis in visualization for his research project. Bob starts by loading the
VisPub dataset into HINTs. Scanning through the topic structure, he can easily
understand topics like “Visualization of 3D Models, Volumes and Data”,
“Visualization of Medical Imaging and Blood Vessel Structures”, and
“Visualizations of Hierarchical Structures”. Since he is only interested in
text visualization, he eliminates those clusters solely based on the topic
labels. After a quick scan through all the topics presented, he proceeds to
search for “text analysis” and applies the filter. The search result shows
that there are three related topics: “Visualization of Complex Data and
Information Spaces”, “Visualization of Web Data, Textual Data, and Social
Media Information”, and “Visualization of Graphs and Networks”. From there, he
starts to scan through each topic to decide if it should be included in his
literature research. The mentioned keywords (author keywords) are particularly
helpful in this process. They include data types, model names, and
visualization techniques, providing rich information about the topic. He can
already get a gist of the research problems in text visualization, such as
“Reasoning”, “Topic Models” or “Temporal Patterns”. He finds those topics
interesting, but not all of them are directly related to his research. By
clicking on the keywords and roughly reading the abstracts, he can quickly
decide whether to include it in his literature research. He expands and
eliminates topics until he is satisfied with the reorganization result (Figure
7). After he feels confident about the overall picture, he starts to
investigate the reorganized corpus in detail.

First, he finds a rather general cluster “Visualization of Web Data, Textual
Data …” in the result, talking about the design space of text visualization.
He selects the cluster, inserts the summaries to the prompt template, and
prompts in the Analysis View: “Given the selected documents, what should I
consider when designing text visualization?”. The chatbot responds with 10
bullet points, such as “Amount of text”, “Placement of text” and
“Customizability” while referencing the articles. The natural language
explanation with references helps him quickly make sense of the topic. He then
proceeds to ask questions in more detail while inserting the full content,
such as “What is considered a balanced ratio of text and charts?”. In the
response, a key point that the chatbot emphasizes is that users prefer charts
with more textual annotations, as long as the placement of text is well-
designed.

With the insertion, the chatbot provides accurate answers tailored to the
inserted article, rather than replying generically. Bob continues this process
by selecting other topics he is unfamiliar with and asking the chatbot
questions. By the end, he is able to develop a clear understanding of the
topics in text visualization and how researchers address them. We present the
complete chat history in supplemental materials.

##  VIII User Study

The design goal of HINTs is to facilitate the sensemaking of a large corpus.
Our approach combines the usage of LLMs as both a general NLP task solver and
an intelligent chatbot agent, an abstraction of clustered hypergraph, and an
interactive visual interface. To further evaluate the HINTs system as well as
better understand user behavioral patterns and challenges during the
sensemaking of a large corpus, we conducted a comparative user study. Below,
we first introduce our study design, and then report the results and discuss
our findings.

###  VIII-A Task Design

Although HINTs use LLMs as both a general NLP task solver and an intelligent
chatbot agent, the key goal of our work is to advocate for the integration of
LLMs in the data preparation pipeline under the principle of Model Alignment.
We thus designed a comparative, task-driven, between-subject study to
highlight the benefits that visualizations designed under proper Model
Alignment can bring.

During the user study, the participants are instructed to conduct a literature
review on a subfield of visualization, namely explainable AI (XAI) or high-
dimensional data visualization, by exploring the VisPub [28] dataset. All our
participants are graduate students in the visualization field. The
participants are instructed to choose a subfield that they are less familiar
with to reduce the effects of prior knowledge. As a result of the literature
review, the participants need to create an outline of the important topics in
the selected subfield and provide a short introduction to each topic. The task
is designed to mimic the scenario of preparing materials for a course
presentation, where students are instructed to first conduct a small-scale
literature review of a subfield and introduce it to other students, in which
the sensemaking of the subfield is a critical step.

We divide our participants into two groups: HINTs group, which uses the HINTs
system to make sense of the subfield; and Baseline group, which uses a basic
RAG-enhanced chatbot interface that mimics the ChatGPT interface.

###  VIII-B Procedure

The study begins by collecting demographic data of the participants, including
age, gender, and years of experience with visualization research. Then, we
introduce the VisPub dataset, the literature review task, and the expected
outcome to the participants. After that, we introduce the interface to use,
including the visual encodings, interactions, and intended usage of each
component. Due to the complexity of HINTs, this process took longer for the
HINTs group than the Baseline group. The participants are encouraged to play
around and get familiar with the system.

After that, the literature review task begins, with a maximum time of one
hour. The participants are instructed to put the literature review outline in
a Google doc. During the task, the participants are asked to follow the think-
aloud protocol, so that their user experiences can be captured in audio. We
also record their screen for further analysis. After the task, the
participants filled out a NASA Task Load Index (TLX) [27] to provide usability
feedback. Then, we interview the participants about their satisfaction with
their drafted outline, their user experience, and any relevant feedback.

In addition to the self-reported Likert scales and interviews, to evaluate the
literature review outlines, we invite experts in XAI and high-dimensional data
visualization to score the outlines in terms of breadth and accuracy on a
scale of 1–7 (higher is better). We report these results together in Section
IX.

###  VIII-C Setup and Participants

The study was conducted fully in-person using the same setup: a 2020 Macbook
Mini attached to a 27-inch display. Both user interfaces were accessed through
the Firefox browser. The Google doc for drafting the literature review outline
is in a Chrome browser for easy switching. All participants used the Macbook
Mini’s trackpad and keyboard for interaction. Before the literature review
task began, we provided enough time for participants to get familiar with the
trackpad and keyboard.

A total of 12 (6 for each group) participants were recruited. Due to the task
design, all participants are graduate students in the visualization field,
with at least one year of research experience and at most 6 years of
experience, including 6 females and 6 males and ages ranging from 21 to 29. To
rule out the effect of prior knowledge during the literature review task, we
asked the participants to choose a subfield and self-report their familiarity
with the subfield on a scale of 1 to 7. All participants reported their
familiarity to be less than 3.

##  IX Result and Discussion

Below, we describe the usability evaluation results, as well as our findings
on users’ behavioral patterns, issues with our current approach, and potential
future solutions.

###  IX-A Usability

The self-reported results of the NASA TLX Likert scale are shown in Figure 8.
The x-axis is a 21-point scale divided into three sections: Low (0–7), Medium
(7–14), and High (14–21). Besides the seven scales in NASA TLX, we also added
the expert review scores: Breadth and Accuracy. We compare the two groups by
aligning them vertically. Overlapping scores are adjusted by 0.2 for visual
clarity.

####  IX-A1 Comparable Usability to Baseline

Overall, the results show that the HINTs is a highly usable and user-friendly
system, as we find no significant difference between the self-reported Mental
Demand, Effort, and Frustration. Achieving comparable usability results as the
Baseline group is significant as the Baseline’s chatbot interface is very
simple and requires minimal effort to learn and use. Although the HINTs Group
participants reported higher scores for Physical Demand, this is expected as
the only interaction for the Baseline group participants is to type, while the
HINTs system incorporates more sophisticated interactions. Being more
physically demanding while keeping the experience effortless and enjoyable
further proves the success of the interface design. We also got no negative
feedback from HINTs group participants about the visualization or
interactions. We attribute these usability successes to a proper Model
Alignment.

####  IX-A2 Over-confidence of Baseline Group

Another interesting observation from the usability results is the comparison
between self-reported Performance and expert-reviewed Breadth and Accuracy.
Though we observe no significant difference from the expert reviews, the self-
reported Performance of the Baseline group is higher than that of the HINTs
group. This over-confidence can be explained by some observations we made
during the user study. First, the Baseline group had no access to external
tools to validate the response of the chatbot, so participants mostly relied
on prior knowledge, or even “trick questions” for verification. Whenever the
model gave responses that fit the participants’ prior knowledge or passed
these trick questions, participants gave more trust in the model and were more
confident in their literature review. On the other hand, for the HINTs group
participants, the visualizations provided visual hints on the articles that
they should investigate by clustering or highlighting. However, due to the
time limit, many of them were not fully investigated. As a result,
participants were less confident in the completeness of their outlines. This
can be confirmed by the observation that the HINTs group participants felt
more time pressure than the Baseline Group as suggested by the difference in
Temporal Demand. In addition, when the participants were asked in the
interview “If given more time, what would you do to perfect the literature
review outline?”, the Baseline Group participants mostly wanted to do further
verification, while the HINTs group participants considered further
investigation on certain sub-topics or techniques, suggesting awareness of the
incompleteness of the outline.

Figure 8: Result of the self-reported NASA TLX Likert scales and expert
reviewed Accuracy and Breadth for HINTs group participants (purple) and
Baseline group participants (blue). The x-axis is a 21-point Likert scale,
divided into three sections: Low (0–7), Medium (7–14), and High (14–21). The
scores show no significant difference in Accuracy, Breadth, Frustration,
Effort, and Mental Demand. HINTs group participants reported higher Physical
Demand and Temporal Demand, while Baseline group participants reported higher
Performance, reflecting a higher confidence in their generated outline.

###  IX-B Commonalities In Comparative Behavior Patterns

Beyond the evaluation of the system, we report insights on the comparative
behavior patterns to contribute to a better understanding of the sensemaking
challenges with the help of LLMs. Overall, both groups of participants exhibit
many commonalities in their behaviors when a powerful chatbot is at hand to
help the sensemaking process, but with interactive visualizations, the
participants were able to avoid certain pitfalls and made more consistent
progress. We report these observations in detail in the following sections.

####  IX-B1 Documents and Keywords Are Equally Vital

As implied by previous topic- and entity-based works, both documents and
keywords are equally vital for users to make sense of a corpus. Baseline group
participants, although did not have direct access to the dataset, frequently
asked for example articles for a topic or a technique. The HINTs group
participants used the cluster structures to find similarities and differences
between different topics and then decided what hierarchy the literature review
outline should have. For the keywords, both groups of participants used the
chatbot heavily to generate explanations for unfamiliar keywords to help them
make sense of a topic or compare different topics. Participants in HINTs group
additionally used the keywords to navigate desired documents. This confirms
that the interplay of documents and keywords plays a central role in corpus
sensemaking.

####  IX-B2 Generative Texts Relieves Writing Burden

Another commonality of both groups of participants is that the chatbot is
heavily used to generate summaries of topics or explanations of keywords, and
then the generated texts are directly used (or after a slight edition) in the
literature review outline. In real-world sensemaking scenarios, note-taking is
an essential step that helps track the sensemaking progress, while previous
works usually excluded it from the sensemaking process. Our observations made
in the user study show that when making sense of a document or a corpus, even
though reading and writing are both cognitively demanding, reading is a step
that can not be avoided while writing can not. Since writing (or note-taking)
usually happens after the completion of a sensemaking, users can use their
gained understanding to quickly verify the generated texts. Powerful text
generation tools such as LLMs can thus not only significantly lower the burden
of writing, but also generate writings beyond human-level quality. This
suggests new opportunities for the integration of LLMs in sensemaking.

###  IX-C Improvements Brought By Visualizations

Besides commonalities in the behavior patterns, we also observed three
improvements in user experience which we attribute to the visual cues provided
by the visualizations. Overall, visualizations provide constant visual hints
for the participants to anchor on during the sensemaking. Below, we describe
each in detail.

####  IX-C1 Cold Start

Since the literature review task requires participants to investigate a
subfield (XAI or high dimensional data visualization) in the visualization
research, all participants started by narrowing down the dataset to find the
directly related subset. For the Baseline group participants, this is done by
prompting “Give me a summary of (subfield)”. However, after this narrowing
down, the Baseline group participants had difficulty finding the next action
to take. The HINTs group participants also started by narrowing down the
dataset using the search bar, but the subsequent actions for the HINTs
participants were clear: to explore the highlighted clusters, as indicated by
the visualization. This implies that the visualization can effectively provide
guidance for users to direct their sensemaking.

####  IX-C2 Prompting Intention Nuances

Even though we used the same underlying LLM model for the Baseline and HINTs
interface chatbots, participants exhibit different usages of such chatbots.
Baseline group participants frequently asked for clarification of a confusing
statement or explanation of a keyword. The conversation quickly became chaotic
if the model failed to provide satisfactory or consistent explanations,
especially if the model mistook a “clarification” question from the
participant as a “skepticism” question and overturned their previous
conclusions completely. The HINTs group participants resolved such confusion
by asking simple explanation questions to the chatbot and made sense of the
responses by combining the summaries and keywords in the Document View, rarely
asking for further clarification. This implies that when a challenging
cognitive process emerges (e.g., understanding an unknown keyword), people
have less trust in the automatic responses of the intelligent agent and prefer
to resolve the difficulty with the most direct manipulation of resources at
hand; i.e., the documents and keywords. This emphasizes the necessity of
interactive visualizations in sensemaking.

####  IX-C3 Losing Track of Progress

A common pitfall for the Baseline Group participants is losing track of their
progress. The participants might ask for a list of subtopics to explore but
unconsciously fixate on only one without realizing that there were five more
subtopics. We attribute this phenomenon to two reasons. First, sensemaking is
not linear, but hierarchical. It is our intuition to take a depth-first-search
approach to systematically organize the chaotic unknowns. However, the user
interface of the Baseline group presents a linear conversation between the
participant and the chatbot with only texts. The visual representation quickly
diverges from the participants’ mental model as they go deeper into the
sensemaking hierarchy. On the other hand, the graphical representation of the
HINTs system is consistent with the participants’ mental model, helping them
keep track of their progress by reminding them of their current status and
expected subsequent actions. Second, even though LLMs are powerful chatbots,
many of their unresolved issues distract users from their sensemaking of the
visualization subfield. For example, LLMs give answers that are too general in
most cases. Participants have to provide extra instructions to get a more
satisfactory response. Another example is the inconsistent responses. Even
though the full chat history is fed to the model, the chatbot still sometimes
gives contradictory conclusions or easily overturns previous conclusions upon
user skepticism. To resolve such confusion, participants can only ask more
questions for clarification. Such distraction rarely happened for the HINTs
group participants, as the participants mainly used the chatbot for simple
explanations or summaries, as mentioned earlier.

##  X Limitations and Future Work

The case study and user study demonstrate that HINTs effectively supports the
sensemaking of a corpus. We attribute this outcome to a good model alignment
and a combined usage of LLM as a general NLP task solver and intelligent
agent. Still, there are some limitations to the current approach. First,
designing prompts for domain-specific use cases is a time-consuming trial-and-
error process. There is currently a lack of effective evaluation methods for
these customized extraction tasks, so the evaluation of the effectiveness of
the extraction pipeline relies on human judgment. Addressing this challenge is
a promising direction for future work. We envision a combination of LLMs,
computational evaluation metrics, and visualizations to assist prompt
evaluation.

Another limitation is the amount of user interactions needed to converge on a
satisfying corpus organization. Although previous study [7] suggests that
users gain more trust and confidence with more interactions with the system,
it would be better if the system could automatically generate a satisfying
organization based on user feedback. Our current automatic cluster expansion
does not consider user intent. We believe LLM’s ability to understand user
intent can be utilized to address this limitation.

Our user study participants call for deeper integration among the
visualization, the documents, and the intelligent agent (chatbot). Our current
implementation only supports users to inject documents as context in the
prompt, but no hints are provided to show which documents are indeed used by
the agent. It would be more useful if the system could show the referenced
documents in the agent’s response and link them back to the visualizations. In
addition, comparative sensemaking of different clusters is also a wanted
feature from the participants. Although users of HINTs could inject documents
from two different clusters and ask the agent for comparisons, it is not
intuitive and the comparison made by the agent is not satisfactory.
Participants would like to have more control over what to compare and a
graphical representation of the comparison result. We leave the comparative
sensemaking of a large corpus as future work.

##  XI Conclusion

In this paper, we introduce HINTs, a visual analytics system that assists
users in making sense of a corpus through exploration, reorganization, and
analysis. Previous works have shown that topic- and entity-based analysis are
essential to sensemaking on a corpus, but failed to align the NLP tasks and
user analysis tasks for proper Model Alignment. We fill in this gap by
combining interactive visualization of a corpus through hypergraph and space-
filling curves and state-of-the-art large language model as both a general NLP
task solver and an intelligent agent. Our evaluation proves the effectiveness
of our system and the benefit of using LLMs in the data preparation stage of
visualization creation to guarantee proper Model Alignment. We advocate for
further integration of LLMs and other stages of the visualization pipeline.
Our observations and lessons learned from the user study inform future works
on the behavior patterns and challenges that users exhibit when collaborating
with an intelligent chatbot agent for a sensemaking task, and the benefits of
incorporating interactive visualizations. We especially emphasize the need for
a proper evaluation of the generated textual responses in both the data
preparation stage and the chatbot. We envision a combination of computational
metrics and interactive visualizations for such evaluation.

## References

  * [1] All The News.  https://components.one/datasets/all-the-news-2-news-articles-dataset.
  * [2] M. Abdelaal, N. D. Schiele, K. Angerbauer, K. Kurzhals, M. Sedlmair, and D. Weiskopf.  Comparative evaluation of bipartite, node-link, and matrix-based network representations.  IEEE Transactions on Visualization and Computer Graphics, 29(1):896–906, 2022.
  * [3] E. Alexander, J. Kohlmann, R. Valenza, M. Witmore, and M. Gleicher.  Serendip: Topic model-driven visual exploration of text corpora.  In 2014 IEEE Conference on Visual Analytics Science and Technology (VAST), pp. 173–182, 2014. doi: 10 . 1109/VAST . 2014 . 7042493
  * [4] D. Atzberger, T. Cech, W. Scheibel, M. Trapp, R. Richter, J. Döllner, and T. Schreck.  Large-scale evaluation of topic models and dimensionality reduction methods for 2d text spatialization.  arXiv preprint arXiv:2307.11770, 2023.
  * [5] D. Auber, C. Huet, A. Lambert, B. Renoust, A. Sallaberry, and A. Saulnier.  Gospermap: Using a gosper curve for laying out hierarchical data.  IEEE transactions on visualization and computer graphics, 19(11):1820–1832, 2013.
  * [6] T. Ayoola, S. Tyagi, J. Fisher, C. Christodoulopoulos, and A. Pierleoni.  ReFinED: An efficient zero-shot-capable approach to end-to-end entity linking.  In NAACL, 2022.
  * [7] T. A. Bach, A. Khan, H. Hallock, G. Beltrão, and S. Sousa.  A systematic literature review of user trust in ai-enabled systems: An hci perspective.  International Journal of Human–Computer Interaction, pp. 1–16, 2022.
  * [8] Y. Bang, S. Cahyawijaya, N. Lee, W. Dai, D. Su, B. Wilie, H. Lovenia, Z. Ji, T. Yu, W. Chung, et al.  A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity.  arXiv preprint arXiv:2302.04023, 2023.
  * [9] Z. J. Beasley, A. Friedman, and P. Rosen.  Through the looking glass: insights into visualization pedagogy through sentiment analysis of peer review text.  IEEE Computer Graphics and Applications, 41(6):59–70, 2021.
  * [10] P. P. . F. T. Bogumił Kamiński.  Community detection algorithm using hypergraph modularity.  In Complex Networks & Their Applications IX: Volume 1, Proceedings of the Ninth International Conference on Complex Networks and Their Applications COMPLEX NETWORKS 2020, pp. 152–163. Springer, 2021.
  * [11] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al.  Language models are few-shot learners.  Advances in neural information processing systems, 33:1877–1901, 2020.
  * [12] P. Caillou, J. Renault, J.-D. Fekete, A.-C. Letournel, and M. Sebag.  Cartolabe: A web-based scalable visualization of large document collections.  IEEE Computer Graphics and Applications, 41(2):76–88, 2020.
  * [13] N. Cao, D. Gotz, J. Sun, Y.-R. Lin, and H. Qu.  Solarmap: Multifaceted visual analytics for topic exploration.  In 2011 IEEE 11th International Conference on Data Mining, pp. 101–110. IEEE, 2011.
  * [14] N. Cao, J. Sun, Y.-R. Lin, D. Gotz, S. Liu, and H. Qu.  Facetatlas: Multifaceted visualization for rich text corpora.  IEEE transactions on visualization and computer graphics, 16(6):1172–1181, 2010.
  * [15] I. Cho, W. Dou, D. X. Wang, E. Sauda, and W. Ribarsky.  Vairoma: A visual analytics system for making sense of places, times, and events in roman history.  IEEE Transactions on Visualization and Computer Graphics, 22(1):210–219, 2016. doi: 10 . 1109/TVCG . 2015 . 2467971
  * [16] J. Choo, C. Lee, C. K. Reddy, and H. Park.  Utopian: User-driven topic modeling based on interactive nonnegative matrix factorization.  IEEE Transactions on Visualization and Computer Graphics, 19(12):1992–2001, 2013. doi: 10 . 1109/TVCG . 2013 . 212
  * [17] J. Chuang, C. D. Manning, and J. Heer.  Termite: Visualization techniques for assessing textual topic models.  In Proceedings of the international working conference on advanced visual interfaces, pp. 74–77, 2012.
  * [18] J. Chuang, D. Ramage, C. Manning, and J. Heer.  Interpretation and trust: Designing model-driven visualizations for text analysis.  In Proceedings of the SIGCHI conference on human factors in computing systems, pp. 443–452, 2012.
  * [19] S. Citraro and G. Rossetti.  Eva: Attribute-aware network segmentation.  In Complex Networks and Their Applications VIII: Volume 1 Proceedings of the Eighth International Conference on Complex Networks and Their Applications COMPLEX NETWORKS 2019 8, pp. 141–151. Springer, 2020.
  * [20] D. Combe, C. Largeron, M. Géry, and E. Egyed-Zsigmond.  I-louvain: An attributed graph clustering method.  In Advances in Intelligent Data Analysis XIV: 14th International Symposium, IDA 2015, Saint Etienne. France, October 22-24, 2015. Proceedings 14, pp. 181–192. Springer, 2015.
  * [21] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [22] W. Dou, L. Yu, X. Wang, Z. Ma, and W. Ribarsky.  Hierarchicaltopics: Visually exploring large text collections using topic hierarchies.  IEEE Transactions on Visualization and Computer Graphics, 19(12):2002–2011, 2013.
  * [23] M. T. Fischer, A. Frings, D. A. Keim, and D. Seebacher.  Towards a survey on static and dynamic hypergraph visualizations.  In 2021 IEEE visualization conference (VIS), pp. 81–85. IEEE, 2021.
  * [24] F. Gilardi, M. Alizadeh, and M. Kubli.  Chatgpt outperforms crowd-workers for text-annotation tasks.  arXiv preprint arXiv:2303.15056, 2023.
  * [25] D. M. Gysi, A. Voigt, T. d. M. Fragoso, E. Almaas, and K. Nowick.  wto: an r package for computing weighted topological overlap and a consensus network with integrated visualization tool.  BMC bioinformatics, 19(1):1–16, 2018.
  * [26] D. Han, G. Parsad, H. Kim, J. Shim, O.-S. Kwon, K. A. Son, J. Lee, I. Cho, and S. Ko.  Hisva: A visual analytics system for studying history.  IEEE Transactions on Visualization and Computer Graphics, 28(12):4344–4359, 2022. doi: 10 . 1109/TVCG . 2021 . 3086414
  * [27] S. G. Hart and L. E. Staveland.  Development of nasa-tlx (task load index): Results of empirical and theoretical research.  In Advances in psychology, vol. 52, pp. 139–183. Elsevier, 1988.
  * [28] P. Isenberg, F. Heimerl, S. Koch, T. Isenberg, P. Xu, C. Stolper, M. Sedlmair, J. Chen, T. Möller, and J. Stasko.  vispubdata.org: A metadata collection about IEEE visualization (VIS) publications.  IEEE Transactions on Visualization and Computer Graphics, 23(9):2199–2206, Sept. 2017. doi: 10 . 1109/TVCG . 2016 . 2615308
  * [29] G. Izacard, M. Caron, L. Hosseini, S. Riedel, P. Bojanowski, A. Joulin, and E. Grave.  Unsupervised dense information retrieval with contrastive learning.  Transactions on Machine Learning Research, 2022.
  * [30] V. Karpukhin, B. Oguz, S. Min, P. Lewis, L. Wu, S. Edunov, D. Chen, and W.-t. Yih.  Dense passage retrieval for open-domain question answering.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6769–6781. Association for Computational Linguistics, Online, Nov. 2020. doi: 10 . 18653/v1/2020 . emnlp-main . 550
  * [31] T. Kumar, S. Vaidyanathan, H. Ananthapadmanabhan, S. Parthasarathy, and B. Ravindran.  A new measure of modularity in hypergraphs: Theoretical insights and implications for effective clustering.  In Complex Networks and Their Applications VIII: Volume 1 Proceedings of the Eighth International Conference on Complex Networks and Their Applications COMPLEX NETWORKS 2019 8, pp. 286–297. Springer, 2020.
  * [32] H. Lee, J. Kihm, J. Choo, J. Stasko, and H. Park.  ivisclustering: An interactive visual document clustering via topic modeling.  In Computer graphics forum, vol. 31, pp. 1155–1164. Wiley Online Library, 2012.
  * [33] T. Y. Lee, A. Smith, K. Seppi, N. Elmqvist, J. Boyd-Graber, and L. Findlater.  The human touch: How non-expert users perceive, interpret, and fix topic models.  International Journal of Human-Computer Studies, 105:28–42, 2017.
  * [34] B. Li, G. Fang, Y. Yang, Q. Wang, W. Ye, W. Zhao, and S. Zhang.  Evaluating chatgpt’s information extraction capabilities: An assessment of performance, explainability, calibration, and faithfulness.  arXiv preprint arXiv:2304.11633, 2023.
  * [35] K.-L. Ma and C. W. Muelder.  Large-scale graph visualization and analytics.  Computer, 46(7):39–46, 2013. doi: 10 . 1109/MC . 2013 . 242
  * [36] P. Maddigan and T. Susnjak.  Chat2vis: Generating data visualisations via natural language using chatgpt, codex and gpt-3 large language models.  IEEE Access, 2023.
  * [37] C. Muelder and K.-L. Ma.  Rapid graph layout using space filling curves.  IEEE Transactions on Visualization and Computer Graphics, 14(6):1301–1308, 2008.
  * [38] A. Narechania, A. Karduni, R. Wesslen, and E. Wall.  Vitality: Promoting serendipitous discovery of academic literature with transformers & visual analytics.  IEEE Transactions on Visualization and Computer Graphics, 28(1):486–496, 2021.
  * [39] A. Narechania, A. Karduni, R. Wesslen, and E. Wall.  vitality: Promoting serendipitous discovery of academic literature.  2022\.
  * [40] Z. Nasar, S. W. Jaffry, and M. K. Malik.  Named entity recognition and relation extraction: State-of-the-art.  ACM Computing Surveys (CSUR), 54(1):1–39, 2021.
  * [41] D. Oelke, H. Strobelt, C. Rohrdantz, I. Gurevych, and O. Deussen.  Comparative exploration of document collections: a visual analytics approach.  In Computer Graphics Forum, vol. 33, pp. 201–210. Wiley Online Library, 2014.
  * [42] X. Ouvrard, J. L. Goff, and S. Marchand-Maillet.  Networks of collaborations: Hypergraph modeling and visualisation.  CoRR, abs/1707.00115, 2017.
  * [43] D. Park, S. Kim, J. Lee, J. Choo, N. Diakopoulos, and N. Elmqvist.  Conceptvector: Text visual analytics via interactive lexicon building using word embedding.  IEEE Transactions on Visualization and Computer Graphics, 24(1):361–370, 2018. doi: 10 . 1109/TVCG . 2017 . 2744478
  * [44] J.-S. Park and S.-J. Oh.  A new concave hull algorithm and concaveness measure for n-dimensional datasets.  Journal of Information science and engineering, 28(3):587–600, 2012.
  * [45] R. Qiu, Y. Tu, Y.-S. Wang, P.-Y. Yen, and H.-W. Shen.  Docflow: A visual analytics system for question-based document retrieval and categorization.  IEEE Transactions on Visualization and Computer Graphics, 2022.
  * [46] S. Raval, C. Wang, F. Viégas, and M. Wattenberg.  Explain and trust: An interactive machine learning framework for exploring text embeddings.  IEEE Transactions on Visualization and Computer Graphics, 2023.
  * [47] W. Shen, J. Wang, and J. Han.  Entity linking with a knowledge base: Issues, techniques, and solutions.  IEEE Transactions on Knowledge and Data Engineering, 27(2):443–460, 2014.
  * [48] E. Sherkat, S. Nourashrafeddin, E. E. Milios, and R. Minghim.  Interactive document clustering revisited: A visual analytics approach.  In 23rd International Conference on Intelligent User Interfaces, pp. 281–292, 2018.
  * [49] J. Stasko, C. Gorg, Z. Liu, and K. Singhal.  Jigsaw: Supporting investigative analysis through interactive visualization.  In 2007 IEEE Symposium on Visual Analytics Science and Technology, pp. 131–138, 2007. doi: 10 . 1109/VAST . 2007 . 4389006
  * [50] M. Steinbach, G. Karypis, and V. Kumar.  A comparison of document clustering techniques.  2000\.
  * [51] Y. Tu, O. Li, J. Wang, H.-W. Shen, P. Powałko, I. Tomescu-Dubrow, K. M. Slomczynski, S. Blanas, and J. C. Jenkins.  Sdrquerier: A visual querying framework for cross-national survey data recycling.  IEEE Transactions on Visualization and Computer Graphics, 2023.
  * [52] Y. Tu, R. Qiu, Y.-S. Wang, P.-Y. Yen, and H.-W. Shen.  Phrasemap: Attention-based keyphrases recommendation for information seeking.  IEEE Transactions on Visualization and Computer Graphics, 2022.
  * [53] I. Vayansky and S. A. Kumar.  A review of topic modeling methods.  Information Systems, 94:101582, 2020.
  * [54] M. Vijaymeena and K. Kavitha.  A survey on similarity measures in text mining.  Machine Learning and Applications: An International Journal, 3(2):19–28, 2016.
  * [55] C. Wang, J. Thompson, and B. Lee.  Data formulator: Ai-powered concept-driven visualization authoring.  IEEE Transactions on Visualization and Computer Graphics, 2023.
  * [56] T. Wu, H. Zhu, M. Albayrak, A. Axon, A. Bertsch, W. Deng, Z. Ding, B. Guo, S. Gururaja, T.-S. Kuo, et al.  Llms as workers in human-computational algorithms? replicating crowdsourcing pipelines with llms.  arXiv preprint arXiv:2307.10168, 2023.
  * [57] W. Xiang and B. Wang.  A survey of event extraction from text.  IEEE Access, 7:173111–173137, 2019.
  * [58] Y. Yang, Q. Yao, and H. Qu.  Vistopic: A visual analytics system for making sense of large document collections using hierarchical topic modeling.  Visual Informatics, 1(1):40–47, 2017.
  * [59] H. Zhang, X. Liu, and J. Zhang.  Extractive summarization via chatgpt for faithful summary generation.  arXiv preprint arXiv:2304.04193, 2023.
  * [60] W. Zhou, S. Zhang, H. Poon, and M. Chen.  Context-faithful prompting for large language models, 2023.
  * [61] J. Červený.  https://github.com/jakubcerveny/gilbert/commits/master generalized hilbert (”gilbert”) space-filling curve for rectangular domains of arbitrary (non-power of two) sizes., 2019.

|  Kwan-Liu Ma is a distinguished professor of computer science at the
University of California, Davis, where he leads VIDI Research Group. Professor
Ma received his PhD degree in computer science from the University of Utah in
1993. His research interests include visualization, computer graphics, human
computer interaction, and machine learning. For his significant research
accomplishments, Professor Ma has received many recognitions, among others the
NSF PECASE award in 2000, IEEE Fellow 2012, the IEEE VGTC Visualization
Technical Achievement Award in 2013. the IEEE Visualization Academy in 2019,
and ACM Fellow in 2024. He has served as papers co-chair for SciVis, InfoVis,
EuroVis, PacificVis, and Graph Drawing, and on the editorial board of IEEE
TVCG (2007-2011) and IEEE CG&A (2007-2019). Professor Ma presently serves on
the editorial boards of the Journal of Visual Informatics, and the Journal of
Computational Visual Media, and ACM Transactions on Interactive Intelligent
Systems. Contact him via email<EMAIL_ADDRESS>and research and other
interests.
---|---
|  Sam Yu-Te Lee received a BS degree in computer science from Nanjing
University in 2020. He is currently a PhD student in computer science at the
University of California, Davis. His research interests include text data
visualization and analytics.
---|---

Task |  Dataset |  Prompt
---|---|---
Summarization |  All The News |  { Role: ‘system’ Content: ‘You are a summarization system that summarizes events happened between the main keywords of a news article. The user will provide you with a news article to summarize. Try to summarize the article with no more than three sentences. Reply starts with ‘The article discussed … }
Keyword Extraction |  All The News |  Sub-task 1: Extract event { Role: ‘system’ Content: ‘You are a state-of-the-art event extraction system. Your task is to extract only the most important event from news articles. Strictly extract only one event. This event should be the most important event in the article. Also extract the trigger that indicates the occurrence of the event. The events should be human-readable. Reply in this format: [event – trigger] ’ }, { Role: ‘user’, Content: ‘This is the news article:{sentence}’ } Sub-task2: Extract keywords { Role: ‘system’ Content: ‘You are a named entity recognition model. You will be given an article and the event recognized in that article by the user. The format of the input defining event in article will be: [event - category]; Extract the main characters involved in that event. The number of main characters should be 2 or less strictly. Ignore any other characters other than the two main characters. Be as concise as possible. Reply with the following format: [character 1],[character 2] …’ }, { Role: ‘system’, Name: ‘example_user’ Content: ‘Article: The article discussed the extensive history of doping in Russia, dating back to the 1983 Soviet Union’s detailed instructions to inject top athletes with anabolic steroids in order to ensure dominance at the Los Angeles Olympics. Event: [Russia’s doping scandal - sports scandal] }, { Role: ‘system’, Name: ‘example_system’, Content: [Russia], [Dr.Sergei] }, { Role: ‘user’, Content: ‘Article: {sentence} Event: {event}’ },

Task |  Dataset |  Prompt
---|---|---
Keyword Disambiguation |  VisPub |  { Role: ‘system’ Content: ‘User will provide a list of keywords from research paper abstracts.’ The input will be in the format: keyword 1, keyword 2, … Which phrase would best describe the list of keywords? The phrase should be very specific and similar to the keywords and less than 5 words. If the words are too similar to each other, simply assign one of the words as the topic. Strictly assign one topic to a list of keywords. Reply in the format: [research topic]; }, { Role: ‘system’, Name: ‘example_user’, Content: ‘[storytelling, data storytelling]’ }, { Role: ‘system’, Name: ‘example_system’, Content: ‘[data storytelling]’ }, { Role: ‘user’, Content: ‘[keywords]’ }

Task |  Dataset |  Prompt
---|---|---
Topic Label Generation for documents |  All The News |  Bottom Level: { Role: ‘system’ Content: ‘You are a news article summarization system. The user will provide you with a set of summarized news articles, your job is to further summarize them into one noun phrase. Use words that are already in the articles, and try to use as few words as possible. }, { Role: ‘system’ Name: ‘example_user’ Content: ‘Article 1: The article discussed an engineering company that creates mobile research robots for the military, which released a new video showcasing the ‘next generation’ of its humanoid Atlas robot. The footage showed the robot escaping and being tormented with a hockey stick, but it quickly recovered. In December 2013, Google bought Boston, the company behind the robots, … Article 2: The article discussed the rise of robots and artificial intelligence (AI) in various industries, including Amazon’s commercial delivery to a customer in Cambridge, England and the testing of driverless vehicles and drones. It also mentioned the potential job losses due to automation, but argued that the panic is exaggerated and that new jobs will be created. Article 3: {summary of article 3 …} Article 4: {summary of article 4 …}’ }, { Role: ‘system’, Name: ‘example_system’ Content: ‘Robotic Advancements and Concerns’ }, { Role: ‘user’, Content: ‘Article 1: {article summary 1}, Article 2: {article summary 2} …’ } { Role: ‘system’ Content: ‘You are a news article categorization system. The user will provide you with a list of sub-topics of news articles and a few examples from the sub-topics. Your job is to further categorize the sub-topics into a single noun phrase that best summarizes all the sub-topics. Try to reuse the words in the examples. } Intermediate Level: { Role: ‘system’ Name: ‘example_user’ Content: ‘Sub-Topics: Increasing Gun Violence in Chicago, Crime Rates and Policing Tactics, Misconceptions about Crime in the United States, Global Events Article 1: The article discussed the major events of 2016, including the Orlando nightclub shooting, Bastille Day attack in Nice, French priest assassination, and cafe attack in Dhaka. … Article 2: The article discussed how serious crimes in the city increased last year while the number of arrests decreased … }, { Role: ‘system’ Name: ‘example_system’ Content: ‘Crimes in the United States’ },

|  VisPub |  Bottom Level: { Role: ‘system’ Content: ‘You are a visualization research paper summarization system. The user will provide you with a set of abstracts of visualization research papers. They are manually categorized by another person, so they are discussing the same topic. Your job is to find out what that topic is. Reply with less than five words. ’ }, { Role: ‘system’ Name: ‘example_user’ Content: ‘Abstract 1: Many datasets such as scientific literature collections contain multiple heterogeneous facets which derive implicit relations, as well as explicit relational references between data items … Abstract 2: We present PivotPaths, an interactive visualization for exploring faceted information resources … }, { Role: ‘system’ Name: ‘example_system’ Content:‘Faceted browsing visualization’ }, { Role: ‘user’ Content: ‘Abstract 1: {abstract 1}, Abstract 2: {abstract 2} …’ } Intermediate Level: { Role: ‘system’ Content: ‘You are a visualization research paper summarization system. You generate topics for a set of visualization research papers. The user will provide you with a list of sub-topics and a few example abstracts from the sub-topics. Your job is to further categorize the sub-topics into a single noun phrase that best summarizes all the sub-topics. Try to reuse the words in the sub-topics. Reply with a single noun phrase without any line breaks. Be concise. }, { Role: ‘system’ Name: ‘example_user’ Content: ‘Sub-Topics: Underwater 3D scene reconstruction from acoustic imag- ing sonar data, Visualization of Sound Propagation in Room Acoustics, Under- water seabed visualization; Abstract 1: The development of a high speed multi-frequency continuous scan sonar at Sonar Research Development Ltd has resulted in the acquisition of extremely accurate, high resolution bathymetric data. … Abstract 2: …’ }, { Role: ‘system’ Name: ‘example_system’ Content: ‘Visualization of Underwater Acoustic Scenes and Sound Propagation’ }, { Role: ‘user’ Content: ‘Sub-Topics: {sub topics}, Abstract 1: {abstract 1}, Abstract 2: {abstract 2}, …’ }
---|---|---

Task |  Dataset |  Prompt
---|---|---
Topic Label Generation for keywords |  All The News |  { Role: ‘system’ Content: ‘You are an entity summarization system. The user will provide you with a list of entities, they can be people, places, or things. The user wants to get a gist of what entities are in the list. First, split the entities into different categories. Then, assign each category a human-readable name. If entities in a category are all related to a specific entity, use that entity as the category. Limit the number of categories to be less than 5 by keeping only the important categories. Reply with the following format: Category 1, Category 2, Category 3, … Do not reply more than 5 categories.’ }, { Role: ‘system’, Name: ‘example_user_1’ Content: ‘Entities: Refugee, Refugee camp, Internally displaced person, Asylum seeker’ }, { Role: ‘system’, Name: ‘example_system_1’, Content: ‘Refugee’ }, { Role: ‘system’, Name: ‘example_user_2’ Content: ‘Entities: JetBlues, American Airlines, Delta Airlines, Southwest Airlines, United Airlines, Alaksa Airlines’ }, { Role: ‘system’, Name: ‘example_system_2’, Content: ‘US airline companies’ }, { Role: ‘user’, Content: ‘Entities: {Entity List}’ },
|  VisPub |  { Role: ‘system’ Content: ‘You are a visualization research paper keyword summarization system. The user will provide you with a list of keywords, they are terminologies in visualization research papers. The user wants to get a gist of what keywords are in the list, but the list is too long. Pick out only a few keywords that best represents the list. Avoid picking out overlapping keywords. Limit the number of picked keywords to be less than 5 by keeping only the important ones. Reply with the following format: Keyword_1, Keyword_2, Keyword_3, … }, { Role:  ‘user’, Content: ‘Keywords: {keyword list}’ }

##  Appendix A Chat Responses for usage scenario 2

Q:  Given the selected documents, what should I consider when designing text
visualization?

A:  When designing text visualization, here are some key considerations based
on the selected articles:
When designing text visualization, you should consider the following aspects
based on the selected articles:
1\. Amount of text: Determine the appropriate amount of text to include in the
visualization. The articles suggest that adding text annotations to
visualizations can enhance understanding and should not be penalized.
Participants preferred charts with more textual annotations over charts with
fewer annotations or text alone.
2\. Semantic content: Consider the type of information conveyed by the text.
Text that describes statistical or relational components of a chart leads to
more takeaways referring to statistics or relational comparisons.
3\. Placement of text: Text should be placed strategically within the
visualization. Different kinds of information are best placed in the title or
closer to the data, depending on the semantic level.
4\. Integration of text and charts: Aim for a tight coupling between text and
charts to support the synthesis of information. Investigate ways to establish
interactive references between text and charts, allowing users to construct
connections effortlessly.
5\. Real-time analysis: If dealing with real-time data, consider the
scalability and adaptiveness of the visualization. Develop interactive systems
that enable visual analysis of streaming social media data or dynamically
cluster posts related to relevant topics.
6\. Support for specific tasks: Tailor the visualization to specific tasks
such as clinical chart review, argument structure analysis, or anomaly
detection in social media. Incorporate insights from domain experts and
conduct iterative design processes to ensure usability.
7\. Enhancement of comprehension: Explore techniques that facilitate enhanced
comprehension of complex documents by coupling text content with data
visualizations. Generate on-demand visualizations based on the reader’s focus
within the document and evaluate their effectiveness against traditional
reading experiences.
8\. Customizability and integration with existing workflows: Provide tools
that are customizable to the needs of users, such as healthcare providers and
physicians, while preserving and extensively linking to the original text.
Avoid obscuring or substituting the original clinical text that contains
qualitative and rhetorical flavors.
9\. Facilitating analysis of complex textual differences: Develop visual
analytics environments that support the alignment of complex textual
differences, prevailing in medieval text traditions or other literature
domains. Provide interactive visualizations for investigating repetition,
variance, and other textual hierarchies.
10\. Evaluation and validation: Perform user studies, expert reviews, and
evaluations to validate the effectiveness and utility of the visualization
techniques. Collect feedback from users and iterate designs based on their
needs and preferences. Consider these guidelines when designing text
visualizations to ensure effective communication and comprehension of
information.

Q:  Given the selected documents, what is considered a balanced ratio of text
and charts?

A:  Based on the selected article, it is suggested that a balanced ratio of
text and charts can vary depending on the context and purpose of the
visualization. The article mentions that heavily annotated charts were not
penalized and participants preferred charts with more textual annotations.
However, it also emphasizes the importance of considering the qualitative
properties and placement of the text within the chart. Overall, it is
recommended to use a combination of text and charts that provides additional
context, guidance, and relevant information to the readers without
overwhelming or distracting them from the visualizations.

##  Appendix B Agglomerative Algorithm

Algorithm 1 Agglomerative Clustering

Input: $H=(V,E)$, $Vec=\\{vec_{i}|\,i\in V\\}$
Output: Sequence of partitions $\mathbf{P}=P_{1},P_{2},\dots P_{k}$

1:while $|V|>1$ do

2: $P_{k}=init\\_partition(V)$ $\triangleright$ Initialize each node as a
cluster

3: $\mathbf{S_{s}}=cosine\\_similarity(V\times V,Vec)$

4: $\mathbf{S_{c}}=wTO(V\times V,E)$

5: for $i\in V$ do

6: $j=most\\_similar\\_node(i,S_{s},S_{c})$

7: $P_{k}=merge\\_clusters(i,j)$ $\triangleright$ Merge the two clusters

8: end for

9: $H^{\prime}=(V^{\prime},E^{\prime})=construct\\_hypergraph(P_{k})$
$\triangleright$ clusters are the new nodes

10: $Vec^{\prime}=centroid\\_similarity(V^{\prime})$

11: $V=V^{\prime},E=E^{\prime},Vec=Vec^{\prime}$ $\triangleright$ Update for
next iteration

12:end while