# Graph Neural Operators for Classification of Spatial Transcriptomics Data

Junaid Ahmed
Johns Hopkins University
BlueLightAI Inc.
E-mail<EMAIL_ADDRESS>Alhassan S. Yasin
Johns Hopkins University
Yasin Research Group
E-mail<EMAIL_ADDRESS>
(December 2022)

### Abstract

The inception of spatial transcriptomics has allowed improved comprehension of
tissue architectures and the disentanglement of complex underlying biological,
physiological, and pathological processes through their positional contexts.
Recently, these contexts, and by extension the field, have seen much promise
and elucidation with the application of graph learning approaches. In
particular, neural operators have risen in regards to learning the mapping
between infinite-dimensional function spaces. With basic to deep neural
network architectures being data-driven, i.e. dependent on quality data for
prediction, neural operators provide robustness by offering generalization
among different resolutions despite low quality data. Graph neural operators
are a variant that utilize graph networks to learn this mapping between
function spaces. The aim of this research is to identify robust machine
learning architectures that integrate spatial information to predict tissue
types. Under this notion, we propose a study incorporating various graph
neural network approaches to validate the efficacy of applying neural
operators towards prediction of brain regions in mouse brain tissue samples as
a proof of concept towards our purpose. We were able to achieve an F1 score of
nearly 72% for the graph neural operator approach which outperformed all
baseline and other graph network approaches.

##  1 Introduction

A neural network takes in a single input and returns a single output. However,
this is not sufficient in capturing systematic relationships by way of the
spatial domain. A neural operator [1] takes in a function and outputs a
function by way of partial differential equations. These are effective in
modeling complex systematic relationships and may be specifically applicable
towards interpolation and segmentation tasks. Neural operators seem promising
towards spatial transcriptomics not just in determining the items present in
the cell, but actual spatial positions of those items as well.

To obtain a fundamental understanding of neural operators, we consider that
the spatial transcriptomics data can be viewed as a continuous function as
opposed to a static set of at-time collections. Furthermore, we view the
samples of data as being discretized from the continuous function that is
defined on the spatial domain $\mathcal{D}$ of each tissue slide. Neural
operators are essentially built on the premise of partial differential
equations, and therefore are designed to work with discretized functions.

The key part of a neural operator is defining a kernel layer that works on a
function with learnable parameters. Controlling the properties of the kernel
function ultimately controls the layer. There is quite a bit of research to
determine optimal kernel functions, for example through the implementation of
neural networks and transformers. Additionally, there is some effort being
dedicated towards determining a universal approximation kernel function. For
the aim of this research, we have identified two kernel functions: a simple
kernel involving the Euclidean spatial coordinates, and a graph network based
kernel layer.

The equation below describes the simple kernel for the spatial neural
operator.

$u(x)=\sigma\Big{(}W_{v_{t}}(x)\sum_{y\in N}\kappa_{\phi}(x,y)v_{t}(y)\Big{)}$

For this operator, $\kappa(x,y)$ represents the neural network which accepts
the Euclidean coordinates as inputs and outputs an NxN matrix. $W$ is a linear
transformation that acts on $v$. $\sigma$ was a constant value for
manipulating the kernel that was not parameterized.

The equation below illustrates the general idea of a graph neural operator, or
GraphPDE [2].

$v_{t+1}(x)=\sigma\Big{(}W_{v_{t}}(x)+\frac{1}{|N(x)|}\sum_{y\in
N}\kappa_{\phi}(\epsilon(x,y))v_{t}(y)\Big{)}$

For graphs constructed on the spatial domain $\mathcal{D}$, the latter portion
of this equation is the same as the message passing aggregation of graph
neural networks in accordance with the edge attributes. So simply, the graph
kernel is a an aggregation of messages. Again here $W$ is a linear
transformation that acts on $v$ and $\sigma$ is an activation function. The
additional caveats we consider are $v_{t}$ as the node features, $e(x,y)$ as
the edge features, and $N(x)$ as the neighborhood of x according to the graph.
$\kappa_{\phi}(e(x,y))$ is indicative of the neural network which takes graph
edge features as inputs (as opposed to Euclidean coordinates of the spatial
neural operator) and outputs an NxN matrix.

##  2 Materials and Methods

To validate that our materials were prudent for analysis, we established the
baseline models we intended to use for evaluating our filtered data. The
baseline models included a Logistic Regression Classifier, a neural network, a
random forest, and an XGBoost classifier [Table 6.1]. These were not
considered experimental models because, for one, they have been pre-
established in industry as best suited techniques in the scope of
classification tasks, and second, our aim is to validate graph network
approaches as viable state of the art classification methods.

###  2.1 Dataset

The goal is to select an optimal spatial transcriptomics dataset. We plan to
pursue a supervised classification task to validate the efficacy of using
state of the art graph network approaches towards tissue type prediction. For
a dataset to be considered towards our intended research, one requirement is
for the dataset to have plentiful collected spots across a large feature set
of genes. Second, it must have been gathered across a plethora of tissue
samples through robust profiling techniques. It is generally difficult to find
datasets in this domain which satisfy all of these requirements. Datasets may
not be well suited for supervised classification by either the lack of a
distinctive target feature(s), or the more prevalent case is that there aren’t
enough tissue samples to decipher a decent generalization.

We used a central repository of curated spatial transcriptomics datasets [6]
in order to find suitable datasets as per our requirements. The mouse brain
atlas dataset [3] we selected for our research contains about 34,000 spots and
23,000 genes with 75 samples. This satisfies our requirement of usable
datasets by way of number of tissue sections and having been profiled by
Illumina spatial transcriptomics profiling techniques. Captured within are the
gene expression signatures which define the spatial organization of
molecularly discrete subregions. Figure 1 illustrates how the dataset is a
“molecular atlas” [3], curated to firstly define the identity of brain
regions, as well as establish a molecular code for the mapping and targeting
of discrete neuroanatomical domains.

Figure 1: UMAP visualization constructed from the subsampled and normalized
gene set ($n\approx 232$) across all spots where colors are indicative of the
associated brain region. The plot shows strong imbalance in the dataset where
a significant number of spots are localized in the Cerebrem as opposed to the
other regions. This will be strongly considered towards balanced data
preparation for modeling.

Normalization is a key component to preparing this data before analysis. We
required a strategy to distill the large feature set to a trainable set.
Additionally, we discovered that introducing sparsity, as per their raw counts
set, was not helpful in differentiating variable genes. Fortunately, the
authors had provided a pre-normalized set that facilitated our discoveries.

###  2.2 Exploratory Data Analysis

Following the supplemental information provided by the authors of our intended
dataset, we discovered that the authors had already gone through a
sophisticated filtration step [3] which allowed them to determine around 230
of the top genes that would provide the most interesting variation in the
dataset.

Additionally, we discovered that the target feature consisted of 15 classes,
yet most of the data was categorized under non-highly differentiable labels,
and despite an abundance of classes, there was high class imbalance. The
authors were using an ontology provided by the Allen Brain Atlas for adult
mouse brains [5]. After referencing the original ontology, we discovered three
high level classes that would optimally separate the data among the 15
classes: the brain stem, the cerebellum, and the cerebrum.

###  2.3 Preprocessing

####  2.3.1 Validation set split

As the data we were analysing was sampled based on their associative tissue
slides, we couldn’t simply use a randomized sampling approach to split the
dataset into its respective modeling sets. We enacted a strategy to use a
fraction of the total samples to be held out for validation. With 75 samples
in total, we intended to use 7 samples as part of this holdout set. However,
there was a clear class imbalance for the target classes. We plotted the
samples against their total inherent class representation and annotated those
which had representation above a specified threshold of at least 10 classes.
Finally, we randomly selected 7 of the highly representative samples to be
held out for validation.

####  2.3.2 Filtration and Binning

To round out the preprocessing, we filtered our feature set to only those
genes which the authors for the source dataset identified. Furthermore, we
categorized the 15 classes of the target feature into the three isolated
classes we discovered through the ABA ontology [5].

####  2.3.3 Reformatting Spots to Graphs

As the final step in our preprocessing, we used Pytorch Geometric to load the
dataset as graphs rather than singular data points. We applied a Radius Graph
transformation in order to capture and represent points in a grid like manner.
The radius parameter was a tuned until we found an optimal arrangement of
points within each graph.

##  3 Results

For our experimental models, we trained a Graph Convolution Network and three
neural operator models 6.1. All models were trained with a balanced class
weighting. Both accuracy and F1 metrics were considered for evaluation,
however the F1 metric was chosen as the primary evaluation metric as it is the
preferred metric in evaluating highly imbalanced sets. Table 1 reflects the
evaluation metrics for each model that we trained against this dataset.

Table 1: Classification Accuracies and F1 scores for the Baseline and GNN
models (n=10). Classification Performance
---
Baseline Models |  Accuracy |  F1-Score |  # of Params
LR |  $48.96\pm 0.00\%$ |  $52.50\pm 0.00\%$ |  3
FCN |  $59.26\pm 0.00\%$ |  $58.20\pm 0.00\%$ |  5
RF |  $67.90\pm 0.27\%$ |  $62.30\pm 0.12\%$ |  4
XGB |  $68.45\pm 0.00\%$ |  $61.79\pm 0.00\%$ |  0
GNN Models |  |  |
SpatialKernel |  $64.08\pm 6.61\%$ |  $54.16\pm 2.40\%$ |  2
GraphSAGE |  $43.61\pm 1.56\%$ |  $44.09\pm 2.64\%$ |  3
GCN |  $52.18\pm 3.35\%$ |  $55.91\pm 2.11\%$ |  2
GAT |  $50.31\pm 3.09\%$ |  $53.51\pm 2.25\%$ |  4
GIN |  $53.99\pm 2.27\%$ |  $54.00\pm 1.74\%$ |  3
SpatialGCN |  $62.61\pm 5.72\%$ |  $51.99\pm 2.54$% |  2
GraphPDE |  $67.63\pm 1.35$% |  $71.06\pm 0.59$% |  5

###  3.1 Graph Convolutional Network

The GCN [Figure 2] has a very low performance when factoring in balanced class
weights. It predicts all 3 classes for the most part.

Figure 2: Predicted vs True class representation of the GCN model.

###  3.2 Neural Operators

For the operator networks, we chose to test both simple and convoluted
variations to validate the robustness of each. For our simple approach, we
built a kernel network termed SpatialKernel which incorporated a Gaussian norm
and linear layers. We further tested the simple approach by replacing the
linear layers with GCN convolution layers for the SpatialGCN model. Finally,
for the convoluted approach, we implemented the GraphPDE approach [2] with a
graph network as the kernel layer.

####  3.2.1 SpatialKernel

The SpatialKernel model [Figure 3] uses three linear layers, wherein a
Gaussian kernel is computed on the positional features and multiplied to the X
tensor before each linear layer. This model performs worse than the GCN,
however it seems to be doing a better job in generalizing across the classes.

Figure 3: Predicted vs True class representation of the SpatialKernel model.

####  3.2.2 SpatialGCN

The SpatialGCN model [Figure 4] combines both the SpatialKernel and GCN such
that each layer is a GCNConv layer, and a gaussian kernel is computed before
each convolutional layer. This model shows better performance than the
previous two models, however it is not predicting for the third class at all.

Figure 4: Predicted vs True class representation of the SpatialGCN model.

####  3.2.3 GraphPDE

The GraphPDE network [Figure 5] required some additional customization in the
data loading procedures by incorporation of edge attributes. Following this
procedural addition, we trained the foundation GraphPDE model against the
dataset using only six hidden layers. This model takes in more parameters
compared to the other experimental models. This model seems to be the most
reliable both in terms of performance and generalization across classes. It
has the highest reported accuracy and F1-score than the other graph network
methods. It is slightly better than the XGBoost model (the baseline model to
beat) in accuracy and far better than it or any other in F1-score performance.

Figure 5: Predicted vs True class representation of the GraphPDE model.

###  3.3 Other Methods

Other models for which we considered training against the dataset were
supplementary. They were merely used to demonstrate the robustness of the
GraphPDE model against high caliber graph neural network models. Discussion on
their performance can be found in the Supplementary Information section.

##  4 Conclusion

In this paper, we demonstrated the feasibility of using graph learning
techniques to classify mouse brain regions with a particular highlight into
the appication of neural operators. The GraphPDE model proved to be robust
both in performance and generaralization across classes. It outperformed all
the baselines and similar caliber graph networks with an F1-score of 72%. This
research presents graph neural operators as valid performers in supervised
classification tasks on spatial transcriptomics data.

##  5 Discussion and Future Work

The study could be improved in multiple facets. First, it was a fast tracked
research endeavor considering a limited timeline. Many of the GNN models of
interest employed some semblance of early fusion tactics incorporating
multimodal data. With additional time permitting, we would consider additional
data fusion techniques to compare against neural operators.

Second, the dataset at the point of discovery was one of few nearly perfect
curated sets. This dataset has sufficient samples, spots, and genes, however
it lacks in generalization. The data comes from one mouse brain. Replicating
results gathered to other mouse brains, should they be gathered, may affect
the final performance of each model. In a more general sense, we may consider
applying this technique to datasets that are not of mouse brains, such as the
mouse spinal cord dataset [4], and even cross-species analysis in human
anatomy datasets.

Third, the implemented GraphPDE model was shallow with a depth of only six
layers. With the benefit of additional time, we would investigate the space of
optimal hyperparameters across all models, and further determine whether a
deeper GraphPDE network could provide stronger performance. Furthermore, on
the note of hyperparameters, there were many models for which we didn’t
parameterize certain values. For example, we could have considered
parameterizing the sigma constant in the SpatialKernel model to allow
learnability for the most optimal value.

Another method we may consider in the future is late fusion techniques. The
methods we utilized reflected early fusion techniques where the data is
treated in a multimodal fashion through a single model pass. For future works,
we may try to train separate classifiers: one for the feature information and
the other either for the positional embeddings or an image classifier for
microscope images of the respective samples. Next, we would consider feeding
outputs from both of these into a single aggregator model to derive a final
output.

Finally, we employed one technique out of many possible neural operator
approaches in this research. Other neural operator techniques incorporate
state of the art architectures, for example, transformers as a kernel
approximation as opposed to the graph network used in the GraphPDE. We may
consider using this and other high caliber neural operator approaches for
further research.

## References

  * [1] Kovachki, N., Li, Z., Liu, B., Azizzadenesheli, K., Bhattacharya, K., Stuart, A., and Anandkumar, A. Neural operator: Learning maps between function spaces.  arxiv (2021).
  * [2] Li, Z., Kovachki, N., Azizzadenesheli, K., Liu, B., Bhattacharya, K., Stuart, A., and Anandkumar, A. Neural operator: Graph kernel network for partial differential equations.  arxiv (2020).
  * [3] Ortiz, C., Navarro, J., Jurek, A., Märtin, A., Lundeberg, J., and Meletis, K. Molecular atlas of the adult mouse brain.  Science Advances 6, 26 (2020).
  * [4] Russ, D. E., Cross, R. B. P., Li, L., Koch, S. C., Matson, K. J. E., Yadav, A., Alkaslasi, M. R., Lee, D. I., Le Pichon, C. E., Menon, V., and Levine, A. J. A harmonized atlas of mouse spinal cord cell types and their spatial organization.  Nature Communications 12, 1 (2021).
  * [5] Whetzel, P., Noy, N., Shah, N., Alexander, P., Nyulas, C., Tudorache, T., and Musen, M. Bioportal: enhanced functionality via new web services from the national center for biomedical ontology to access and use ontologies in software applications.  Nucleic Acids (2011).
  * [6] Xu, Z., Wang, W., Yang, T., Chen, J., Huang, Y., Gould, J., Du, W., Yang, F., Li, L., Lai, T., Hua, C., Hu, S., Cai, J., Li, H., You, L., Zeng, W., Wang, B., Liu, L., Xu, X., Chen, F., and Wei, X. Stomicsdb: a database of spatial transcriptomic data.  bioRxiv (2022).

##  6 Appendix

###  6.1 Abbreviations

ABA | Allen Brain Atlas
---|---
FCN | Fully Connected Network
GAT | Graph Attention Network
GIN | Graph Isomorphism Network
GraphPDE | Graph Neural Operator
GNN | Graph Neural Network
LR | Logistic Regressor
RF | Random Forest
XGB | XGBoost model

###  6.2 Supplementary Information

####  6.2.1 Graph Attention Network

The GAT [Figure 6] is not comparable to the top performers in terms of
accuracy, nor is it predictive towards all three classes. Furthermore it takes
longer to train, so it is not a worthy tradeoff.

Figure 6: Predicted vs True class representation of the GAT model.

####  6.2.2 GraphSAGE

The GraphSAGE [Figure 7] network performs worse than the GAT model and
slightly better than the simple SpatialKernel model. It is not highly
predictive towards all classes.

Figure 7: Predicted vs True class representation of the GraphSAGE model.

####  6.2.3 Graph Isomorphism Network

This network [Figure 8] performs better than the GAT and the GraphSAGE models,
which isn’t very telling among the breadth of GNN model performers, yet is
comparable to the suite of actual graph baselines. It is more accurate than
the GCN and the loss is minimal compared to other models. It lacks in terms of
representation of all three classes, and both with low accuracy and F1-score,
it is not a worthy comparator against the GraphPDE.

Figure 8: Predicted vs True class representation of the GIN model.


# Uncovering Latent Themes of Messaging on Social Media by Integrating LLMs: A
Case Study on Climate Campaigns

Tunazzina Islam, Dan Goldwasser

###### Abstract

This paper introduces a novel approach to uncovering and analyzing themes in
social media messaging. Recognizing the limitations of traditional topic-level
analysis, which tends to capture only the overarching patterns, this study
emphasizes the need for a finer-grained, theme-focused exploration.
Conventional methods of theme discovery, involving manual processes and a
human-in-the-loop approach, are valuable but face challenges in scalability,
consistency, and resource intensity in terms of time and cost. To address
these challenges, we propose a machine-in-the-loop approach that leverages the
advanced capabilities of Large Language Models (LLMs). This approach allows
for a deeper investigation into the thematic aspects of social media
discourse, enabling us to uncover a diverse array of themes, each with unique
characteristics and relevance, thereby offering a comprehensive understanding
of the nuances present within broader topics. Furthermore, this method
efficiently maps the text and the newly discovered themes, enhancing our
understanding of the thematic nuances in social media messaging. We employ
climate campaigns as a case study and demonstrate that our methodology yields
more accurate and interpretable results compared to traditional topic models.
Our results not only demonstrate the effectiveness of our approach in
uncovering latent themes but also illuminate how these themes are tailored for
demographic targeting in social media contexts. Additionally, our work sheds
light on the dynamic nature of social media, revealing the shifts in the
thematic focus of messaging in response to real-world events.

## Introduction

The social media landscape is highly diverse and dynamic, serving as a
battleground for various interest groups, including politicians, advertisers,
and other stakeholders. These entities strategically use these platforms for
microtargeting (Hersh 2015; Barbu 2014), a process that utilizes data-driven
techniques to target potential users and advance their interests.
Microtargeting capitalizes on the rich user data collected by social networks,
making the understanding of this targeted messaging a significant challenge.

In recent years, social media has emerged as a fertile ground for diverse
discussions and expressions, encompassing a broad spectrum of topics and
themes. In this context, a ‘topic’ within social media content refers to a
broad category under which various finer-grained ‘themes’ can be identified.
The topic serves as an umbrella term that encompasses a range of more specific
and nuanced themes. For example, under the ‘Climate Change’ topic, a theme
might be ‘Renewable Energy Solutions’. Identifying themes within a topic
involves delving into these specific aspects, revealing the detailed and
granular nuances within the broader category. Given the dynamic nature of
social media, accurately discerning and understanding the diverse themes
within a broader topic is a complex and continuously evolving task.

Traditionally, thematic analysis (TA) has been the method of choice for this
task, serving as a qualitative research technique focused on pattern
identification, where themes emerging from the data guide further analysis
(Roberts, Dowell, and Nie 2019; Vaismoradi et al. 2016; Vaismoradi, Turunen,
and Bondas 2013; Braun and Clarke 2012, 2006; Tuckett 2005). However, with the
explosion of textual data in the digital age, there’s a growing inclination
towards computational methods for text analysis (Sivarajah et al. 2017;
Jagadish et al. 2014; Fan, Han, and Liu 2014).

Among these, supervised text analysis methods like Support Vector Machines
(SVMs) (Joachims 1998), Naive Bayes classifiers (McCallum, Nigam et al. 1998),
and Neural Networks (LeCun, Bengio, and Hinton 2015) have been effective in
theme categorization. These methods use pre-labeled datasets to learn and
identify patterns, aiding in the categorization of new data. However, they
present challenges, notably the need for extensive labeled datasets, which can
be resource-intensive to create. Moreover, their performance heavily relies on
the quality and representativeness of the training data. If the training data
is biased or lacks diversity, the model’s ability to generalize and accurately
identify themes in varied or new contexts is significantly compromised.

On the other hand, unsupervised text analysis methods, such as Latent
Dirichlet Allocation (LDA) (Blei, Ng, and Jordan 2003) and non-negative matrix
factorization (NMF) (Lee and Seung 1999), have gained popularity in theme
discovery. These methods independently identify themes from data without the
need for pre-labeled datasets. While useful, they also have limitations. Topic
modeling techniques like LDA can produce topics that are challenging to
interpret or lack coherence, complicating the extraction of meaningful themes
(Röder, Both, and Hinneburg 2015; Lau, Newman, and Baldwin 2014; Mimno et al.
2011; Chang et al. 2009). Recent advancements have seen the integration of
human judgment through a human-in-the-loop approach, which, while enhancing
accuracy, suffers from issues of scalability and high resource demand (Pacheco
et al. 2022a, b; Lund et al. 2017; Hu et al. 2014). Recognizing this, our work
explores a new approach, integrating the strengths of LLMs (Brown et al. 2020)
with a streamlined algorithmic process to efficiently discover and
characterize themes in social media messaging. This exploration leads us to
the following research questions (RQ), which are crucial for assessing the
potential of LLMs to enhance our analysis by providing deeper and more nuanced
insights:

  1. 1.

RQ1: Can LLMs determine if two given texts, without prior knowledge of
existing codes, are discussing the same topic?

  2. 2.

RQ2: If provided with a definition of a theme, can LLMs successfully
categorize other texts under that specific theme?

To enhance theme discovery, we propose an iterative three-step machine-in-the-
loop framework. We define large textual collections as repositories of textual
instances (e.g., ads, tweets, posts, documents) where each instance is
potentially associated with a set of annotated themes (seed set from Islam,
Zhang, and Goldwasser (2023) that is used in this paper showed in Table 1). A
high-level overview of our method is shown in Figure 1.

Figure 1: Framework overview.

Firstly, we initiate with candidate generation, a phase that includes
clustering of instances using a clustering algorithm, ensuring consistency and
coherence of the clusters using LLMs, and summarizing each cluster using LLMs.
This phase also incorporates LLMs for redundancy checks and merging
overlapping clusters, thus refining the cluster pool.

Secondly, we acknowledge the potential need for human validation. This
optional phase allows for a critical assessment of the newly formed clusters,
questioning their merging, description, and overall coherence. This step
serves as a bridge, combining the computational efficiency of algorithms with
the nuanced understanding of human judgment.

Finally, the assignment phase utilizes the power of LLMs in a few-shot
prompting manner. This innovative approach determines if a new post/text
aligns with the summarized clusters, thus assigning it to a relevant cluster.
To showcase our approach, we use climate campaigns (dataset released by Islam,
Zhang, and Goldwasser (2023)) on social media as a case study. The dataset
focuses on climate related English ads based on the United States from January
$2021$ \- January $2022$ (Islam, Zhang, and Goldwasser 2023). Qualitative
researchers use different theories to ground theme discovery, each associated
with a different set of concepts. In this case, the theme discovery process is
grounded using energy industry and climate change related stances (Islam,
Zhang, and Goldwasser 2023). Our contributions are summarized as follows:

  1. 1.

We suggest a LLMs-in-the-loop approach to uncover latent themes of social
media messaging.

  2. 2.

We conduct quantitative and qualitative analysis on real-world dataset to
demonstrate the effectiveness of our proposed method. We show that our model
outperforms the baselines.

  3. 3.

Based on the newly discovered themes, we analyze demographic targeting on
social media messaging.

  4. 4.

We demonstrate how the thematic emphasis of advertisements evolves in response
to real-world events.

Pro-energy |  Economy_pro, ClimateSolution, Pragmatism, Patriotism, AgainstClimatePolicy.
---|---
Clean-energy |  Economy_clean, FutureGeneration, Environmental, HumanHealth, Animals, SupportClimatePolicy, AltEnergy, PoliticalAffiliation.

Table 1: Pre-existing themes from Islam, Zhang, and Goldwasser (2023).

## Related Work

Several works have done for understanding messaging on social media (Islam,
Roy, and Goldwasser 2023; Islam, Zhang, and Goldwasser 2023; Islam and
Goldwasser 2022; Capozzi et al. 2021; Silva and Benevenuto 2021; Silva et al.
2020; Serrano et al. 2020; Mejova and Kalimeri 2020; Capozzi et al. 2020;
Andreou et al. 2019). Most of the approaches to understanding messaging have
relied on broader categories. However, in this paper, we focus on gaining a
deeper understanding of messaging.

Previous works (Islam, Roy, and Goldwasser 2023; Islam, Zhang, and Goldwasser
2023; Islam and Goldwasser 2022; Capozzi et al. 2021) used a predefined set of
labels, themes, and arguments to analyze messaging. These were fixed and
established based on existing topics or theoretical frameworks, such as Moral
Foundations Theory (MFT) (Haidt and Graham 2007). However, previous works
often fall short of capturing the nuances of messaging choices. For instance,
within the Economy topic, it may neglect to recognize conflicting arguments,
such as Argument 1: “Increasing the minimum wage boosts economic growth,”
versus Argument 2: “Increasing the minimum wage harms small businesses and
reduces employment opportunities.” Recently, Pacheco et al. (2022a, b)
suggested a human-in-the-loop approach to uncover new themes. In contrast to
their work, we use LLMs inference.

Researchers in the fields of qualitative research (QR) and Natural Language
Processing (NLP) are currently investigating how LLMs can be utilized for TA.
Dai, Xiong, and Ku (2023) proposed LLM-in-the-loop approach for TA. Xiao et
al. (2023) combined GPT-3 with expert-drafted codebooks for supporting
qualitative analysis. Gao et al. (2023) suggested a collaborative interface
for qualitative analysis in their work, utilizing LLMs for generating initial
codes and aiding in decision-making processes. In this paper, we develop a new
set of themes (or codes) specifically tailored for analyzing messaging in a
particular domain, which helps speed up the development of domain-specific
labels.

## Framework

Our methodology is meticulously designed to discover themes in social media
messaging through a robust three-step process. This approach not only
capitalizes on advanced computational techniques but also integrates human
insight, ensuring a comprehensive and accurate analysis. Importantly, our
framework is flexible regarding the choice of representation. Both the
embedding technique used for generating representations and the scoring
function can be readily substituted with alternatives.

### Theme and Instance Representation

We represent instances and explanatory phrases under each theme using the
Sentence BERT (Reimers and Gurevych 2019) embedding. To evaluate the degree of
relevance between an instance and a specific theme, we adopt a straightforward
distance-based method. This involves calculating the cosine distance between
the instance and every explanatory phrase and example related to the themes,
selecting the smallest distance score as the representative value. Our
framework is agnostic of the representation used. The underlying embedding
objective, as well as the scoring function, can easily be replaced.

#### Grounding

Before assigning instances to themes, every instance is grouped into unnamed
clusters, effectively leaving them unassigned. Upon the first invocation of
this process, instances are assigned to the theme that is nearest to them, but
only if the newly calculated distance is shorter than, or at most equal to,
the distance from their prior assignment. Previous assignments can correspond
either to different themes or to the unnamed space. Note that this way, some
instances can remain unassigned based on a threshold.

Figure 2: Example of incoherent cluster.

### Candidate Generation

The initial phase of our methodology, candidate generation, begins with the
clustering of unassigned instances. We employ K-means clustering (Jin and Han
2010), a method chosen for its effectiveness in handling large datasets, which
consist of varied and complex social media messaging. The K-means algorithm
segments these instances into distinct clusters based on their thematic and
linguistic similarities.

Figure 3: Example of merged cluster.

#### Coherency check

To ensure the consistency and coherence of these clusters, we perform a
cluster coherence check. We create subsets of instances from each cluster,
with each subset containing $4$-$5$ instances. We then prompt LLMs in a zero-
shot setting, asking if these instances belong together. Based on the
responses provided by the LLMs, we identify incoherent clusters as noise and
exclude them. It is important to note that we discard only the clusters, not
the instances associated with them. These instances are still utilized for
cluster membership assignments in the subsequent stage of our framework. Fig.
2 shows an example of an incoherent cluster from the dataset. These are the
top $5$ ads that are closest to the centroid of a cluster. $1^{st}$ and
$2^{nd}$ ads are about supporting motor oil, $3^{rd}$ one is against the oil
and gas industry, $4^{th}$ and $5^{th}$ ads are about supporting biofuel. LLMs
categorize this cluster as an incoherent cluster (answering RQ1).

#### Cluster Summarization

To make the clusters interpretable, we prompt LLMs in a zero-shot setting to
generate a multi-document summary based on top-$K$ instances of the clusters
($k=5$).

#### Redundancy check

We embed the summaries using SBERT (Reimers and Gurevych 2019) and compute the
cosine similarity score between the embedding summaries of cluster pairs.
Based on a threshold value ($\geq 0.6$), we merge two clusters. Fig. 3 shows
an example of a merged cluster from the data. These cluster summaries are
related to “initiatives to plant new trees and protect existing forests to
save planet” and based on the threshold, we merge them.

Figure 4: Prompt template for coherency check (shown as zero-shot) Figure 5:
Prompt template for generating summary (shown as zero-shot)) Figure 6: Prompt
template for LLMs Mapper (shown as four-shot)

### Human Validation

The second phase introduces an optional but crucial step – human validation.
Two NLP and Computational Social Science (CSS) researchers did one $2$ hours
session to assess the clusters formed by the algorithms. Their expertise is
particularly invaluable in evaluating the nuances of language and context that
automated systems might overlook. They scrutinize the merging decisions, the
descriptiveness of the clusters, and overall coherence, bridging the gap
between computational efficiency and the nuanced understanding of human
judgment.

For the cluster coherency check, the annotators are asked to verify if the top
$5$ ads from a cluster are coherent or not. They are asked to provide a score
of $1$ if the ads are coherent and a score of $0$ if they are not.

For the cluster summary check, the annotators are asked to verify if the
generated summary is correct or not. They are asked to provide a score of $1$
if the summaries are correct and a score of $0$ if they are not.

### Map Instances to Themes

In the final phase, we provide summaries of all clusters (including pre-
existing + newly discovered) and prompt LLMs in a four-shot setting to assess
whether a new instance aligns with the summarized clusters. To alleviate the
issues of being hallucinated, we force LLMs to answer the questions by
selecting from the given summarized clusters. The LLMs, equipped with the
context from the initial phases, effectively categorize new texts into the
relevant clusters (answering RQ2). This phase is crucial for maintaining the
dynamism of the clusters, allowing the system to adapt to new data and
emerging themes.

Method | Num. | Num. Covered Ads
---|---|---
| Themes | thr $<0.6$ | thr $<0.5$ | thr $<0.4$ | thr $<0.3$
Pre-existing | 13 | 14652 | 9725 | 3731 | 558
+After Iter1 | 20 | 18702 | 14583 | 8646 | 2944
+After Iter2 | 25 | 18988 | 15052 | 9079 | 3180

Table 2: Coverage. THR: Threshold, NUM: Number.

(a) Assignment with LLMs mapper.

(b) Assignment with SBERT.

Figure 7: Confusion Matrix after $2^{nd}$ round of iteration + assignment.

(a) Baseline: 10 LDA Topics.

(b) Baseline: 30 LDA Topics.

(c) Ours: After $2^{nd}$ round of iteration.

Figure 8: Correlations between themes and stances.

## Experiments

We use $21372$ ads and $13$ themes (Table 1) related to climate campaigns
released by Islam, Zhang, and Goldwasser (2023). At first, we assign pre-
existing themes to the ads using SBERT. Pre-existing themes can cover a small
portion of ads ($1^{st}$ row of Table 2). The threshold is determined by the
distance from the centroid to an ad, with a shorter distance being preferable.
For a threshold of $<0.5$, there are $11647$ ads that remained uncovered,
which we consider as a noisy set. We then apply k-means clustering (with
$k=29$) to this noisy set. Following the candidate generation phase, we
identify $7$ new themes in the $1^{st}$ iteration. We use GPT-3.5 for cluster
coherency check, generating cluster summaries, and redundancy check. In
$2^{nd}$ iteration, we repeat this process using $6789$ ads remaining from
iteration 1 (again applying a threshold of $<0.5$), resulting in the discovery
of $5$ additional themes. The complete set of final themes is presented in
Table 3. Summaries of $25$ themes (clusters) are provided in Appendix A.

Iter1 |  BidenGasPriceIncrease, AgainstCorporateInterests, GasTax, Deforestation, Carbon, CustomerBasedAltEnergy, FoodSecurity
---|---
Iter2 |  EnergyAffordabilityandSustainabilityLegislation, EcofriendlyConsumerChoices, PlasticWasteandEnvironmentalImpact, PromoteSustainableTransportation, WaterManagementandSustainability

Table 3: Resulting Themes. Iter: Iteration.

Method | Acc. | Macro avg. F1
---|---|---
SBERT Assignment | 84.05 | 79.32
LLMs Mapper (Ours) | 88.15 | 89.24

Table 4: Mapping Quality w.r.t Human Judgements

### Prompt Templates

This section shows the prompt templates used to check cluster coherency (Fig.
4), generate cluster summary (Fig. 5), and map ads to themes (Fig. 6).

Age Group | State |  Entity |  Claim |  Theme
---|---|---|---|---
13-24 | CA |  Young People, Climate Change Denier, Climate Movement. |  Importance of action against climate change to clean air & water and to reduce carbon emissions. |  Environmental
TX |  Natural Gas Systems, Coal Energy Systems, Nuclear Energy Systems, Wind Turbines, Solar Panels. |  Failures in natural gas, coal, and nuclear systems caused almost double outages in Texas compared to frozen wind turbines and solar panels. |  AltEnergy
25-54 | CA |  Carbon Emissions, Sacramento Municipal Utility District. |  Eliminating all carbon emissions from the power supply by 2030. |  Carbon
TX |  Climate Change, Renewable Energy Plan. |  Climate change impacts are irreversible by 2030, urging an immediate switch to renewable energy. |  AltEnergy
55+ | CA |  Western Burrowing Owls, Aramis Industrial Solar Power Plant. |  Aramis solar plant’s construction will harm wildlife and habitats, especially affecting Western burrowing owls. |  Animals
TX |  U.S. Senators or Congressman, Citizens’ Climate Lobby. |  Urgent need for action on climate change. |  SupClimPolicy
Table 5: Most mentioned entities, claim, and theme of targeted ads for young,
working-age, older population of CA and TX. SupClimPolicy:
SupportClimatePolicy.

### Results and Analysis

Our experiments show that our framework can be used to uncover a set of themes
that cover a large portion of the messaging regarding climate campaigns (Table
2). With a threshold $<0.4$, pre-existing themes can cover only $17.5\%$ ads.
After $1^{st}$ iteration using our machine-in-loop approach, we cover $40.5\%$
and after $2^{nd}$ iteration, we cover $42.5\%$ ads (Table 2).

Gender |  Entity |  Claim |  Theme
---|---|---|---
Male |  Alex Askew, Karen Greenhalgh |  Alex Askew’s record of raising taxes contrasted with Karen Greenhalgh’s promise to cut taxes and lower living costs. |  GasTax
Female |  Parents, Children/Kids |  Urgent need for action against climate change to protect the future of children and the planet. |  FutureGen
Table 6: Mentioned entities, claim, and theme of targeted ads for only male and only female population. FutureGen: FutureGeneration. State | Stance |  Entity |  Claim |  Theme
---|---|---|---|---
VT | Clean-energy |  Community |  Climate change impacts health, family, and community. |  HumanHealth
Pro-energy |  Climate Action Council |  Climate Action Council plans to increase fuel and carbon taxes. |  GasTax
ND | Pro-energy |  Community |  Oil and gas supports jobs and community growth. |  Economy_pro
Biden |  Biden’s energy policies increase energy costs. |  BidenGasPrInc
Clean-energy |  Water system |  Solution of water scarcity problem. |  WaterMng&Sustain
Table 7: Entity, claim, and theme of targeted ads for blue and red states. The
entity ‘Community’ is targeted differently by advertisers. While targeting the
blue state Vermont, the emphasis is on the impacts of climate change on
community health. In contrast, when targeting the red state North Dakota, the
focus shifts to how oil and gas supports community growth. BidenGasPrInc:
BidenGasPriceIncrease, WaterMng&Sustain: WaterManagementandSustainability.

### Mapping Quality

To evaluate the mapping quality of our model, we select ads that fall under
threshold $<0.4$ and then randomly sample $1072$ ads. We ask the annotators to
provide a score of $1$ if the ad aligns with the theme, $0$ for no alignment.
There is no clear definition of alignment, and we let the annotators make this
judgment. To evaluate mapping quality, we compare the performance of LLMs
mapper with SBERT assignment (Baseline) of ads to themes. Results are outlined
in Table 4. The resulting mapping from ads to the theme is fairly accurate
with respect to human judgments (Fig. 7(a)) with the accuracy of $88.15\%$ and
macro avg. F1 score of $89.24\%$. Fig. 7 shows the confusion matrix after
$2^{nd}$ round of iteration and assignment with LLMs mapper (7(a)) and SBERT
mapper (7(b)).

### Qualitative Analysis

Given that we can characterize themes as the reasons cited by people to
support climate debate, we consider assignments to be better if they are more
cohesive (e.g., if they are more strongly correlated with climate-related
stance). To evaluate this, we perform a correlation test between the
identified themes and the stance expressed in the ads (i.e., pro-energy or
clean-energy). We calculate the Pearson correlation matrices and present them
in Fig. 8. We compare our discovered themes with a set of topics extracted
using LDA. We observe that our themes (Fig. 8(c)) have stronger correlations
with stances than the derived LDA topics (Fig. 8(a),8(b)).

### Hyperparameters

To obtain topics from LDA, we use Gensim (Rehurek and Sojka 2011)
implementation. We follow the preprocessing steps shown in Hoyle et al. (2021)
and estimate the number of topics in a data-driven manner by maximizing. We do
a grid search over a set of {10, 20, 30} for the LDA baseline.

To determine the value of $k$ in k-Means, we follow both The Elbow Method and
The Silhouette Method. Our optimal $k$ value is $29$.

For SBERT embedding, we use the sentence transformer model ‘all-mpnet-base-v2’
with default parameters.

(a) Exxon Mobil.

(b) American Petroleum Institute.

(c) Consumer Energy Alliance.

(d) Apex clean energy.

(e) American Wind Energy Association.

(f) Climate Power Education Fund.

Figure 9: Pro-energy and clean-energy Themes differ based on regions. Funding
entities are categorized based on Corporation, Industry Association, and
Advocacy Group.

(a) Event: Gas price increase on 12 May, 2021.

(b) Event: Hurricane Ida hits on 29 August, 2021.

Figure 10: Advertisers’ use of the top-$5$ themes before and after the events.
(a) Themes ‘GasTax’ and ‘BidenGasPriceIncrease’ emerged mainly after the
occurrence of the event Gas Price Increase. (b) Theme ‘SupportClimatePolicy’
emerged after the event Hurricane Ida hits.

## Demographic Targeting

Analyzing demographic targeting in social media messaging is crucial for
understanding the varying impacts of targeted content on different population
segments. This analysis is vital for comprehensively understanding how
messaging is tailored and disseminated to influence specific demographic
groups.

### How Themes Differ based on Age Group

In the dataset (released by Islam, Zhang, and Goldwasser (2023)), there are
$7$ age groups. In this paper, we categorize these into three distinct age
groups: young people (ages $13$-$24$), working-age people (ages $25$-$54$),
and the older population (age $55+$). We analyze the targeted ads for these
$3$ age categories, focusing on the states of California (CA) and Texas (TX).
The reason for choosing these two states is that TX is known as the energy
capital of the world***https://www.eia.gov/todayinenergy/detail.php?id=49356.
Recently, CA has emerged as one of the most vocal states in the fight against
climate change†††https://www.pewtrusts.org/en/research-and-
analysis/blogs/stateline/2022/10/06/california-takes-leading-edge-on-climate-
laws-others-could-follow. To identify the top-k most frequently mentioned
entities in the texts, we prompt LLMs in a zero-shot setting. Similarly to
extract the most mentioned claim, zero-shot prompting of LLMs has been used.

Table 5 summarizes the results. Ads targeted for young population of CA focus
on ‘Environmental’ theme having claim ‘Importance of action against climate
change to clean air and water’. For the working-age demographic in CA, the ads
focus on the ‘Carbon’ theme, making claims like ‘The Sacramento Municipal
Utility District (SMUD) aims to eliminate all carbon emissions from their
power supply by 2030, contributing to the #CleanPowerCity movement’. In TX,
both young and working-age groups are the focus of ads under the ‘AltEnergy’
theme, each with specific claims. Interestingly, the older population in CA is
targeted by ads with claims opposing the construction of solar plants, such as
The construction of solar plants will harm wildlife and habitats’, aligning
with the ‘Animals’ theme. This approach is interesting, as it positions
advertisers against solar plant construction while still engaging with the
broader context of clean-energy theme.

### How Themes Differ based on Gender

To analyze how messaging is tailored for a specific gender, we extract ads
targeted exclusively at males and females, focusing on the themes that are
uncommon between these categories. We find that ads having ‘GasTax’ theme
focus only male population. On the other hand, ‘FutureGeneration’ themed ads
only target female population with mentioned entities ‘Parents’,
‘Children/Kids’ and the claim emphasizing the ‘Urgency of taking action
against climate change to protect children’s future’ (Table 6).

### How Themes Differ State-wise

To explore the state-wise differences in emerging themes and claims, we use
historical voting data‡‡‡https://www.visualcapitalist.com/u-s-presidential-
voting-history-by-state/ to categorize states as red or blue. We focus on
Vermont (VT), a blue state, and North Dakota (ND), a red state. A notable
finding is the different emphasis placed on the entity ‘Community’ by VT and
ND. In VT, the focus is on the claim Climate change impacts health, family,
and community’ (under the theme ‘HumanHealth’), while in ND, the emphasis is
on Oil and gas support jobs and community growth’ (falling under the theme
‘Economy_pro’). These contrasting perspectives are detailed in Table 7.
Besides, ads having pro-energy stance focus on ‘GasTax’ theme while targeting
blue state. Conversely, ads having clean-energy stance focus on
‘WaterManagementandSustainability’ theme while targeting red state (Table 7).

Pro-energy |  Corporation: Exxon Mobil, Industry Association: American Petroleum Institute, Advocacy Group: Consumer Energy Alliance.
---|---
Clean-energy |  Corporation: Apex clean energy, Industry Association: American Wind Energy Association, Advocacy Group: Climate Power Education Fund.

Table 8: List of funding entities.

### How Themes Differ based on Geographic Regions

To grasp the ways in which fossil fuel industries and their supporting groups
shape public opinion, Islam, Zhang, and Goldwasser (2023) categorize pro-
energy funding entities into three distinct types: Corporation, Industry
Association, and Advocacy Group. Similarly, we categorize the clean-energy
funding entities into those categories. In this paper, we choose only one
funding entity from each category resulting $6$ funding entities, including
$3$ pro-energy and $3$ clean-energy funding entities. Table 8 shows the list
of funding entities included in our analysis.

We divide the USA into $5$ regions, encompassing all $50$ states: West,
Southwest, Midwest, Southeast, and Northeast to analyze the geographic
targeting of the sponsored ads (Fig. 9). Exxon mobile targets Northeast region
sponsoring ads focused on ‘energy affordability and reliability’ (‘Pragmatism’
theme). For the rest of the regions, it sponsor ads focusing on ‘how energy
can play in supporting jobs, wages and community growth’ (‘Economy_pro’
theme). On the other hand, the pro-energy advocacy group Consumer Energy
Alliance focuses on the ‘Economy_pro’ theme when targeting the Midwest, but
shifts to the ‘ClimateSolution’ theme for the Southeast region (Fig. 9(c)).

From Fig. 9(d), we notice that Apex clean energy targets the West region
focusing on ‘EcofriendlyConsumerChoices’ and
‘PromoteSustainableTransportation’ themes, while for other regions (except the
Southwest) it emphasizes on ‘CustomerBasedAltEnergy’ theme. However, American
Wind Energy Association uses ‘PoliticalAffiliation’ theme to target the
Southwest and Midwest regions (Fig. 9(e)).

## Theme Shifts Triggered by Key Events

To investigate how theme differs based on real-world event, we pick 2 defining
events and investigate the changes of the themes. The events are as follow:

  * •

Gas Price Increase: On May $12$, $2021$, US national average retail gas prices
rose above $\$3$ a gallon for the first time since
$2014$§§§https://www.bloomberg.com/news/articles/2021-05-12/u-s-gasoline-
prices-rise-above-3-gallon-first-time-since-2014, after a cyber attack shut
operations at Colonial
Pipeline¶¶¶https://en.wikipedia.org/wiki/Colonial˙Pipeline˙ransomware˙attack.

  * •

Hurricane Ida: On August $29$, $2021$, Hurricane Ida made landfall near Port
Fourchon, Louisiana, as a Category 4 hurricane causing life-threatening storm
surges, wind damage, heavy rainfall, and power outages that affected
approximately one million homes and businesses along the US Gulf
Coast∥∥∥https://en.wikipedia.org/wiki/Hurricane˙Ida.

The usage of themes in response to these events are shown in Fig. 10. Fig.
10(a) illustrates a comparative bar chart of the sponsored ad themes (top-$5$)
used five days before and after a significant event, labeled as the Gas Price
Increase. We observe that the frequency of themes ‘AltEnergy’,
‘AgainstCorporateInterests’, and ‘ClimateSolution’ increased following the
event. Additionally, the themes ‘GasTax’ and ‘BidenGasPriceIncrease’ emerged
predominantly in the aftermath (Fig. 10(a)).

Fig. 10(b) shows the shift in the thematic focus of sponsored advertisements
over a period of six months before and after Hurricane Ida hits in the USA.
The emerging themes after the hurricane, such as ‘SupportClimatePolicy’,
reflect a growing public and corporate response to environmental issues
highlighted by climate change. There is a noticeable presence of themes
related to alternative energy (Fig. 10(b)).
Overall, Fig. 10 shows a shift in the thematic focus of ads in response to the
real-world event.

## Conclusion

We present a machine-in-the-loop framework for discovering new themes from
social media messaging in a particular domain, which can help speed up the
development of domain-specific labels. We show that themes generated from our
approach can cover a larger portion of ads, and those themes are coherent
through quantitative measures and human evaluation. Additionally, we
demonstrate that our resulting set of themes is cleaner and more explainable
than themes obtained with topic modeling approaches.

Furthermore, our analysis demonstrates how these themes are specifically
tailored for demographic targeting on social media and how the thematic focus
of messaging shifts in response to real-world events.

## Limitations

Though we show the effectiveness of our framework focusing on climate
campaigns, it is a domain-agnostic framework. However, the main idea of
characterizing themes using LLMs and associating the ads to their closest
themes is applicable to other cases with no changes.

Our approach relies on GPT-3.5 to generate a multi-document summary, check
cluster coherency, and map ads to themes. We chose GPT-3.5 instead of the
open-source counterparts due to computational resource constraints.

LLMs are pre-trained on a huge amount of human-generated text. As a result,
they may inherently contain many human biases (Blodgett et al. 2020; Brown et
al. 2020). We did not consider any bias in our task.

## Ethics Statement

To the best of our knowledge, we did not violate any ethical code while
conducting the research work described in this paper. We report the technical
details needed for reproducing the results.

In this paper, we did not propose any new dataset rather, we only experiment
with an existing dataset which are adequately cited.

The author’s personal views are not represented in any qualitative result we
report, as it is solely an outcome derived from a machine learning model.

## Broader Impact

Our approach marks a significant advancement in the thematic analysis of
social media messaging, which might provide valuable insights into the
strategic use of messaging for targeted communication and campaign efficacy.
We leave this exploration for future work.

## References

  * Andreou et al. (2019) Andreou, A.; et al. 2019.  Measuring the Facebook advertising ecosystem.  In _NDSS_.
  * Barbu (2014) Barbu, O. 2014.  Advertising, microtargeting and social media.  _Procedia-Social and Behavioral Sciences_ , 163: 44–49.
  * Blei, Ng, and Jordan (2003) Blei, D. M.; Ng, A. Y.; and Jordan, M. I. 2003.  Latent dirichlet allocation.  _the Journal of machine Learning research_ , 3: 993–1022.
  * Blodgett et al. (2020) Blodgett, S. L.; Barocas, S.; Daumé III, H.; and Wallach, H. 2020.  Language (Technology) is Power: A Critical Survey of “Bias” in NLP.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 5454–5476.
  * Braun and Clarke (2006) Braun, V.; and Clarke, V. 2006.  Using thematic analysis in psychology.  _Qualitative research in psychology_ , 3(2): 77–101.
  * Braun and Clarke (2012) Braun, V.; and Clarke, V. 2012.  _Thematic analysis._ American Psychological Association.
  * Brown et al. (2020) Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33: 1877–1901.
  * Capozzi et al. (2021) Capozzi, A.; De Francisci Morales, G.; Mejova, Y.; Monti, C.; Panisson, A.; and Paolotti, D. 2021.  Clandestino or Rifugiato? Anti-immigration Facebook Ad Targeting in Italy.  In _CHI_.
  * Capozzi et al. (2020) Capozzi, A.; Morales, G. D. F.; Mejova, Y.; Monti, C.; Panisson, A.; and Paolotti, D. 2020.  Facebook Ads: Politics of Migration in Italy.  In _ICSI_.
  * Chang et al. (2009) Chang, J.; Gerrish, S.; Wang, C.; Boyd-Graber, J.; and Blei, D. 2009.  Reading tea leaves: How humans interpret topic models.  _Advances in neural information processing systems_ , 22.
  * Dai, Xiong, and Ku (2023) Dai, S.-C.; Xiong, A.; and Ku, L.-W. 2023.  LLM-in-the-loop: Leveraging Large Language Model for Thematic Analysis.  _arXiv preprint arXiv:2310.15100_.
  * Fan, Han, and Liu (2014) Fan, J.; Han, F.; and Liu, H. 2014.  Challenges of big data analysis.  _National science review_ , 1(2): 293–314.
  * Gao et al. (2023) Gao, J.; Guo, Y.; Lim, G.; Zhan, T.; Zhang, Z.; Li, T. J.-J.; and Perrault, S. T. 2023.  CollabCoder: A GPT-Powered Workflow for Collaborative Qualitative Analysis.  _arXiv preprint arXiv:2304.07366_.
  * Haidt and Graham (2007) Haidt, J.; and Graham, J. 2007.  When morality opposes justice: Conservatives have moral intuitions that liberals may not recognize.  _Social Justice Research_ , 20(1): 98–116.
  * Hersh (2015) Hersh, E. D. 2015.  _Hacking the electorate: How campaigns perceive voters_.  Cambridge University Press.
  * Hoyle et al. (2021) Hoyle, A.; Goel, P.; Hian-Cheong, A.; Peskov, D.; Boyd-Graber, J.; and Resnik, P. 2021.  Is automated topic model evaluation broken? the incoherence of coherence.  _Advances in neural information processing systems_ , 34: 2018–2033.
  * Hu et al. (2014) Hu, Y.; Boyd-Graber, J.; Satinoff, B.; and Smith, A. 2014.  Interactive topic modeling.  _Machine learning_ , 95: 423–469.
  * Islam and Goldwasser (2022) Islam, T.; and Goldwasser, D. 2022.  Understanding COVID-19 Vaccine Campaign on Facebook using Minimal Supervision.  In _2022 IEEE International Conference on Big Data (Big Data)_ , 585–595.
  * Islam, Roy, and Goldwasser (2023) Islam, T.; Roy, S.; and Goldwasser, D. 2023.  Weakly Supervised Learning for Analyzing Political Campaigns on Facebook.  In _Proceedings of the International AAAI Conference on Web and Social Media_ , volume 17, 411–422.
  * Islam, Zhang, and Goldwasser (2023) Islam, T.; Zhang, R.; and Goldwasser, D. 2023.  Analysis of Climate Campaigns on Social Media Using Bayesian Model Averaging.  In _Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society_ , AIES ’23, 15–25. New York, NY, USA: Association for Computing Machinery.  ISBN 9798400702310.
  * Jagadish et al. (2014) Jagadish, H. V.; Gehrke, J.; Labrinidis, A.; Papakonstantinou, Y.; Patel, J. M.; Ramakrishnan, R.; and Shahabi, C. 2014.  Big data and its technical challenges.  _Communications of the ACM_ , 57(7): 86–94.
  * Jin and Han (2010) Jin, X.; and Han, J. 2010.  _K-Means Clustering_ , 563–564.  Boston, MA: Springer US.  ISBN 978-0-387-30164-8.
  * Joachims (1998) Joachims, T. 1998.  Text categorization with support vector machines: Learning with many relevant features.  In _European conference on machine learning_ , 137–142. Springer.
  * Lau, Newman, and Baldwin (2014) Lau, J. H.; Newman, D.; and Baldwin, T. 2014.  Machine reading tea leaves: Automatically evaluating topic coherence and topic model quality.  In _Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics_ , 530–539.
  * LeCun, Bengio, and Hinton (2015) LeCun, Y.; Bengio, Y.; and Hinton, G. 2015.  Deep learning.  _nature_ , 521(7553): 436–444.
  * Lee and Seung (1999) Lee, D. D.; and Seung, H. S. 1999.  Learning the parts of objects by non-negative matrix factorization.  _Nature_ , 401(6755): 788–791.
  * Lund et al. (2017) Lund, J.; Cook, C.; Seppi, K.; and Boyd-Graber, J. 2017.  Tandem anchoring: A multiword anchor approach for interactive topic modeling.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 896–905.
  * McCallum, Nigam et al. (1998) McCallum, A.; Nigam, K.; et al. 1998.  A comparison of event models for naive bayes text classification.  In _AAAI-98 workshop on learning for text categorization_ , volume 752, 41–48. Madison, WI.
  * Mejova and Kalimeri (2020) Mejova, Y.; and Kalimeri, K. 2020.  COVID-19 on Facebook ads: competing agendas around a public health crisis.  In _Proceedings of the 3rd ACM SIGCAS Conference on Computing and Sustainable Societies_ , 22–31.
  * Mimno et al. (2011) Mimno, D.; Wallach, H.; Talley, E.; Leenders, M.; and McCallum, A. 2011.  Optimizing semantic coherence in topic models.  In _Proceedings of the 2011 conference on empirical methods in natural language processing_ , 262–272.
  * Pacheco et al. (2022a) Pacheco, M. L.; Islam, T.; Mahajan, M.; Shor, A.; Yin, M.; Ungar, L.; and Goldwasser, D. 2022a.  A Holistic Framework for Analyzing the COVID-19 Vaccine Debate.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , 5821–5839.
  * Pacheco et al. (2022b) Pacheco, M. L.; Islam, T.; Ungar, L.; Yin, M.; and Goldwasser, D. 2022b.  Interactively uncovering latent arguments in social media platforms: A case study on the covid-19 vaccine debate.  In _Proceedings of the Fourth Workshop on Data Science with Human-in-the-Loop (Language Advances)_ , 94–111.
  * Rehurek and Sojka (2011) Rehurek, R.; and Sojka, P. 2011.  Gensim–python framework for vector space modelling.  _NLP Centre, Faculty of Informatics, Masaryk University, Brno, Czech Republic_ , 3(2): 2.
  * Reimers and Gurevych (2019) Reimers, N.; and Gurevych, I. 2019.  Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , 3982–3992. Hong Kong, China: Association for Computational Linguistics.
  * Roberts, Dowell, and Nie (2019) Roberts, K.; Dowell, A.; and Nie, J.-B. 2019.  Attempting rigour and replicability in thematic analysis of qualitative research data; a case study of codebook development.  _BMC medical research methodology_ , 19: 1–8.
  * Röder, Both, and Hinneburg (2015) Röder, M.; Both, A.; and Hinneburg, A. 2015.  Exploring the space of topic coherence measures.  In _Proceedings of the eighth ACM international conference on Web search and data mining_ , 399–408. ACM.
  * Serrano et al. (2020) Serrano, J. C. M.; et al. 2020.  The Political Dashboard: A Tool for Online Political Transparency.  In _ICWSM_.
  * Silva and Benevenuto (2021) Silva, M.; and Benevenuto, F. 2021.  COVID-19 ads as political weapon.  In _Proceedings of the 36th Annual ACM Symposium on Applied Computing_ , 1705–1710.
  * Silva et al. (2020) Silva, M.; et al. 2020.  Facebook Ads Monitor: An Independent Auditing System for Political Ads on Facebook.  In _WWW_.
  * Sivarajah et al. (2017) Sivarajah, U.; Kamal, M. M.; Irani, Z.; and Weerakkody, V. 2017.  Critical analysis of Big Data challenges and analytical methods.  _Journal of business research_ , 70: 263–286.
  * Tuckett (2005) Tuckett, A. G. 2005.  Applying thematic analysis theory to practice: A researcher’s experience.  _Contemporary nurse_ , 19(1-2): 75–87.
  * Vaismoradi et al. (2016) Vaismoradi, M.; Jones, J.; Turunen, H.; and Snelgrove, S. 2016.  Theme development in qualitative content analysis and thematic analysis.
  * Vaismoradi, Turunen, and Bondas (2013) Vaismoradi, M.; Turunen, H.; and Bondas, T. 2013.  Content analysis and thematic analysis: Implications for conducting a qualitative descriptive study.  _Nursing & health sciences_, 15(3): 398–405.
  * Xiao et al. (2023) Xiao, Z.; Yuan, X.; Liao, Q. V.; Abdelghani, R.; and Oudeyer, P.-Y. 2023.  Supporting Qualitative Analysis with Large Language Models: Combining Codebook with GPT-3 for Deductive Coding.  In _Companion Proceedings of the 28th International Conference on Intelligent User Interfaces_ , 75–78.

##  Appendix A Appendix

### Cluster Summaries

Summaries of final $25$ themes (clusters) are shown below:

1\. Summary_Economy_pro:  The provided texts collectively emphasize the
importance of the oil and natural gas industry in supporting local jobs,
economic growth, and community development. They express concerns about
proposed bans on natural gas and oil, warning of potential job losses and
negative economic impacts. The texts call for support of the oil and gas
industry, highlighting its contributions to local economies and communities
while advocating for a sustainable future.

2\. Summary_ClimateSolution:  The provided texts collectively advocate for the
role of natural gas in advancing a cleaner and more sustainable energy future.
They highlight natural gas as an essential tool for reducing carbon emissions,
providing affordable and reliable energy, and complementing renewable energy
sources like wind and solar. The texts emphasize the importance of using
natural gas to address the dual challenge of meeting energy needs and
protecting the environment while working towards ambitious carbon reduction
goals, such as reaching net-zero emissions by 2045. Texts focus on
greenwashing aspect and the promotion of a controversial energy source under
the guise of environmental benefit. The text covers the complex and sometimes
contradictory narratives around certain fossil fuels in the context of climate
and energy policy.

3\. Summary_Pragmatism: The provided texts collectively stress the vital role
of reliable and affordable natural gas and oil in enabling modern life. They
highlight that in 2020, natural gas and oil accounted for nearly $70\%$ of
America’s energy supply, and projections suggest they will continue to meet
approximately $70\%$ of the nation’s energy needs by 2050, as indicated by the
U.S. Energy Information Administration. The texts call attention to the
significance of American oil and gas production in maintaining affordable
energy and invite individuals to share their priorities in this regard.

4\. Summary_Patriotism: The provided texts collectively emphasize the
importance of U.S. energy leadership and innovation, highlighting the
country’s position as a top global energy producer while actively working to
reduce greenhouse gas emissions. They express concerns about potential
cutbacks in local oil and gas production, citing potential consequences such
as increased reliance on foreign oil imports and higher gas prices for
Californians. The texts argue against shutting off American oil and natural
gas production, pointing out that such actions could strengthen other nations
while negatively impacting U.S. jobs, economic security, and national
security. Additionally, they underscore the broader impacts of U.S. oil and
natural gas production on daily life, including its role in fertilizer
production and its significance in ensuring food supply and availability. The
texts call for action to protect America’s energy future and support natural
gas and consumer choice through petitions, and they highlight the U.S.’s
achievement as the world’s No. 1 exporter of liquefied natural gas,
particularly in response to energy demands in Europe.

5\. Summary_AgainstClimatePolicy: The provided texts collectively convey a
sense of urgency and concern regarding the oil and gas industry, emphasizing
the need for support and resistance against what they perceive as threats to
the industry’s survival. They criticize progressive and Democratic energy
policies, particularly the Green New Deal, as potentially devastating to
American workers and the economy. The texts also criticize the Biden
administration’s energy policies, linking them to high gas prices and a
reliance on emergency reserves. They call for action to prioritize and protect
the American energy industry, with an emphasis on opposing what they view as
radical policies and mandates. Additionally, the texts highlight the role of
the oil and gas industry in addressing climate challenges through innovation
and expertise while expressing fears of job loss and economic consequences if
certain policies are implemented.

6\. Summary_Economy_clean: The provided texts collectively advocate for a
clean energy future in the United States and emphasize the potential for job
creation and economic benefits associated with clean energy initiatives. They
highlight the growth of clean energy sectors and argue that investments in
renewable energy are generating good-paying jobs across various occupations
and regions. The texts contrast clean energy with fossil fuels, suggesting
that clean energy is not only environmentally responsible but also
economically advantageous. They call on Congress to support clean energy
policies and investments as a means to strengthen the American economy, create
jobs, lower energy costs, and mitigate climate change, ultimately promoting a
clean energy-focused approach to economic growth.

7\. Summary_HumanHealth: The provided texts collectively emphasize the
significant and urgent health impacts of climate change, highlighting the
range of dangers it poses, including cardiovascular and respiratory issues,
food security, the spread of diseases, and more. They stress the need for
awareness and action to address these health challenges linked to climate
change, with mentions of governmental initiatives and organizations dedicated
to addressing climate-related health concerns. The texts underscore that
climate change is already harming people and communities, affecting vulnerable
populations, and call for support and action to mitigate these adverse health
effects.

8\. Summary_FutureGeneration: The provided texts collectively emphasize the
grave threat of climate change to children’s health and well-being. They cite
alarming statistics, including data from the World Health Organization, that
underscore the vulnerability of young children to climate-related health
issues. The texts stress the urgent need to protect children from the impacts
of climate change and highlight the various health risks they face, including
premature births, birth defects, and increased emergency room visits. They
call for action to address climate change to safeguard the future of children
and grandchildren, emphasizing that inaction comes at a high cost to
children’s health and well-being. The texts also acknowledge the anxiety and
concerns that climate change is causing among young people and suggest ways to
support and address their anxieties.

9\. Summary_Environmental: The provided texts collectively address the urgency
and breadth of the climate crisis, emphasizing its far-reaching impacts on
various aspects of society and the environment. They highlight alarming trends
such as the rapid warming of oceans, the increasing frequency of extreme
weather events, and the dire consequences of climate change on ecosystems,
economies, and public health. The texts call for collective action and
emphasize the role of individuals in combating climate change, urging the
sharing of ideas and solutions. They also stress the need to address the root
causes, including greenhouse gas emissions from fossil fuels, and promote
carbon capture as a key strategy for decarbonizing the global energy system.
Overall, the texts underscore the severity of the climate emergency and the
importance of taking immediate action to mitigate its effects and protect
vulnerable communities and ecosystems.

10\. Summary_Animals: The provided texts collectively highlight the severe
impact of climate change on wildlife, fisheries, birds and ecosystems. They
mention rising temperatures, increased droughts, fires, intense storms, and
floods that threaten various species and their habitats. Studies reveal that
climate change could lead to the extinction of one-third of species in the
next 50 years, with particular vulnerabilities in South America. The texts
emphasize the urgent need for action to protect wildlife from the consequences
of climate change, emphasizing the role of renewable energy development that
must consider the needs of wildlife. They call for an understanding of the
interconnected crises of climate change and biodiversity loss, urging
collective efforts to address both challenges.

11\. Summary_AltEnergy:  The provided texts collectively advocate for the
adoption of renewable energy sources as a cleaner, cheaper, and more
sustainable alternative to fossil fuels. They emphasize the benefits of
transitioning to renewables, including reducing carbon footprints, enhancing
grid reliability, and mitigating the impact of rising gas prices. The texts
highlight the rapid growth of renewable energy in the United States, with
clean power sources like wind, solar, and hydrogen accounting for a
significant share of electricity generation. They stress the urgency of making
this transition to combat climate change, citing the irreversible impacts
looming by 2030. Additionally, they mention various projects and initiatives
aimed at promoting clean energy and energy independence, such as repurposing
coal plants into renewable energy hubs and proposed solar farms. Finally, they
encourage energy-conscious practices like using electricity during off-peak
hours to harness clean energy effectively.

12\. Summary_SupportClimatePolicy:  The provided texts collectively advocate
for the passage of the Build Back Better Act, emphasizing its significance in
addressing climate change and promoting a clean energy future. They highlight
the key provisions of the bill, including cutting pollution by $50\%$ by 2030,
creating clean energy jobs, and lowering utility bills. The texts stress the
urgency of taking action to combat climate change and emphasize that it’s a
historic opportunity to invest in meaningful climate action and transition to
a carbon-zero grid. Additionally, some texts criticize the fossil fuel
industry for its role in climate change and call for holding polluters
accountable. Overall, the focus is on the importance of the Build Back Better
Act in advancing climate goals and environmental justice.

13\. Summary_PoliticalAffiliation: The provided texts collectively highlight
concerns about the influence of the oil and gas industry on climate change
policies. They emphasize that despite claims of caring about climate change,
these industries spend significant amounts on lobbying to block regulations
and protect their subsidies. The texts mention the substantial sums spent on
climate lobbying by major oil and gas companies and argue that these actions
hinder meaningful climate action. They also discuss political contributions to
lawmakers and emphasize the need to address climate change without being
influenced by industry lobbying. Additionally, some texts reference specific
instances where lobbyists have allegedly undermined climate legislation or
influenced infrastructure bills to prioritize industry interests over
environmental concerns. Overall, the focus is on revealing the industry’s
efforts to shape climate policies to their advantage.

14\. Summary_BidenGasPriceIncrease:  The provided texts collectively express
strong disapproval and criticism of President Joe Biden’s administration,
focusing on various issues such as rising gas prices, inflation, the southern
border crisis, and energy policies. Some texts call for Joe Biden’s
resignation and invite readers to participate in approval polls, while others
highlight specific concerns, like the increase in gas prices and the
cancellation of certain pipelines, portraying these actions as detrimental to
the American economy. Overall, these texts present a negative view of Joe
Biden’s leadership and policy decisions.

15\. Summary_AgainstCorporateInterests: The provided texts largely revolve
around progressive candidates running for Congress and their calls for
grassroots support. These candidates emphasize their commitment to economic
justice, racial equity, climate action, and progressive policies such as a
Green New Deal, Medicare for All, and universal housing. They criticize their
opponents for receiving significant contributions from corporations and
lobbyists and highlight the need for people-powered campaigns. Some texts also
mention the importance of supporting incumbent progressive Congress members,
such as the Squad, to pass transformative legislation like the Build Back
Better Agenda. Overall, these texts aim to rally support for progressive
candidates and causes while advocating for a shift away from corporate
influence in politics.

16\. Summary_GasTax: The provided texts revolve around opposing proposed gas
tax increases in various states. They emphasize concerns about the financial
burden on families, rising gas prices, and the potential for gasoline
shortages. The texts encourage people to sign petitions or participate in
rallies to voice their opposition to these tax hikes. Additionally, some texts
highlight the current tax rates and the impact of previous tax increases,
framing the issue as a matter of transparency and affordability for consumers.
Overall, the focus is on mobilizing public sentiment against gas tax increases
in different states.

17\. Summary_Deforestation: The texts primarily focus on environmental
conservation and efforts to combat climate change. They highlight the
importance of preserving forests like the Tongass National Forest in Alaska,
emphasizing their role in storing carbon, providing habitat, and mitigating
climate change effects. They urge readers to take action, such as signing
petitions or participating in tree-planting initiatives. Some texts discuss
the potential benefits of ”super trees” that are more effective at carbon
storage and stress the importance of preserving mature trees in the face of
climate change. Overall, the texts call for collective action to protect and
restore natural environments and address climate-related challenges by
planting more trees and preserving forests.

18\. Summary_Carbon: These texts collectively emphasize the importance of
reducing carbon emissions and mitigating one’s carbon footprint. They
introduce various strategies and initiatives for individuals and communities
to lower their carbon impact, such as carbon pricing, using energy-efficient
lights, enrolling in carbon reduction programs, calculating personal carbon
footprints, offsetting tailpipe emissions, and supporting sustainability
efforts in industries like cement and concrete. The overarching theme is the
need for environmental awareness and action to combat climate change and
promote a more sustainable future.

19\. Summary_CustomerBasedAltEnergy: These texts collectively highlight
various aspects of renewable and sustainable energy sources. They address
common myths and misconceptions about solar energy, emphasizing its
versatility and suitability for different types of buildings and applications.
They promote community solar programs in Illinois, offer information and
resources for learning about solar power, and invite individuals to explore
solar technology and installations. Additionally, the texts touch upon other
forms of renewable energy, such as wind and nuclear power, showcasing their
potential to provide reliable, clean energy and create jobs in various states.
The overarching theme is the promotion and education about renewable energy
sources and their benefits.

20\. Summary_FoodSecurity: The provided texts collectively address the
complexities of food security, emphasizing the global nature of the challenge
and the need for cross-sector collaboration to meet the increasing food demand
caused by a growing global population and climate change. They also shed light
on issues of food justice and sovereignty, particularly in marginalized
communities, where access to healthy foods is limited. Additionally, the texts
discuss the concept of food elitism, questioning marketing practices and their
impact on food equity, arguing for the importance of affordable and
sustainable food systems. Furthermore, the issue of food deserts is explored,
where residents face challenges related to the availability, accessibility,
and affordability of nutritious food, with a recognition of the intricate
factors contributing to food insecurity. These texts collectively stress the
urgency of addressing food-related challenges from multiple perspectives,
encompassing policy, social justice, and sustainable agriculture.

21\. Summary_EnergyAffordabilityandSustainabilityLegislation: These texts all
convey similar messages, focusing on various representatives, including Rep.
John Autry, Rep. Kelly Alexander, Rep. Kyle Hall, Rep. John Sauls, and Rep.
John A. Torbett, who are collectively working to reduce power bills for
families and businesses in North Carolina. They sponsor legislation aimed at
exploring changes that could potentially result in annual energy bill
reductions exceeding $\$600$ million. The texts encourage readers to learn
more about the Alliance for Energy Security’s collaboration with policymakers
in an effort to make energy in North Carolina more affordable, environmentally
friendly, and dependable.

22\. Summary_EcofriendlyConsumerChoices: The broader theme of these texts is
environmental sustainability and eco-conscious consumer choices. They focus on
promoting a laundry detergent product that aligns with sustainability goals by
offering a range of environmentally friendly features, such as being cruelty-
free, using minimal plastic packaging, and supporting carbon-neutral shipping.
The theme encourages individuals to make environmentally responsible choices
in their daily lives, particularly in the context of laundry and reducing
plastic waste.

23\. Summary_PlasticWasteandEnvironmentalImpact: The overall summary of these
texts is a collective call to address the urgent issue of plastic waste and
pollution in the environment. They highlight the alarming amount of plastic
entering our oceans, stress the need for collective action involving both
consumers and brands to combat this crisis, introduce innovative solutions
like pyrolysis to reduce plastic waste and carbon footprints, and emphasize
the importance of better handling and regulation within the petrochemical
industry to prevent plastic pollution. The texts underscore the broader theme
of environmental responsibility and the imperative to find sustainable
alternatives to single-use plastics and improve the entire plastic production
and waste management lifecycle.

24\. Summary_PromoteSustainableTransportation: These texts collectively
emphasize the importance of electric vehicles (EVs) in combating air pollution
and improving air quality. They highlight the role of clean car standards and
the accessibility of EVs, particularly in Los Angeles, where local rebate
programs make them a cost-effective choice. Shifting from gas-powered vehicles
to electric options is promoted for its potential to reduce emissions and
decrease exposure to pollution, with a focus on the environmental and health
benefits of adopting EVs and the need to take steps to clean the air in
Colorado and beyond.

25\. Summary_WaterManagementandSustainability: These texts collectively
emphasize the discussion of water treatment and billing issues in various
cities. The texts touch upon topics such as the cost of building new water
treatment plants versus refurbishing existing ones, concerns about water
quality and the presence of toxic chemicals in tap water, allegations of
potential corruption or kickbacks in decision-making processes, and the
financial aspects of water billing, including rates, fees, and expenditures
related to water infrastructure. Additionally, some texts emphasize the
importance of proper maintenance and infrastructure investment to ensure clean
and safe drinking water for residents.