# Clusters of science and health related Twitter users become more isolated
during the COVID-19 pandemic

Francesco Durazzi
Department of Astronomy and Physics (DIFA)
University of Bologna
Bologna 40127, Italy
<EMAIL_ADDRESS>
Martin Müller 11footnotemark: 1
Digital Epidemiology Lab
Ecole polytechnique fédérale de Lausanne (EPFL)
1202 Geneva, Switzerland
<EMAIL_ADDRESS>
Marcel Salathé
Digital Epidemiology Lab
Ecole polytechnique fédérale de Lausanne (EPFL)
1202 Geneva, Switzerland
<EMAIL_ADDRESS>
Daniel Remondini
Department of Astronomy and Physics (DIFA)
University of Bologna
Bologna 40127, Italy
<EMAIL_ADDRESS>
F.D. and M.M. contributed equally to this work.To whom correspondence may be
addressed. Email<EMAIL_ADDRESS>
###### Abstract

COVID-19 represents the most severe global crisis to date whose public
conversation can be studied in real time. To do so, we use a data set of over
350 million tweets and retweets posted by over 26 million English speaking
Twitter users from January 13 to June 7, 2020. We characterize the retweet
network to identify spontaneous clustering of users and the evolution of their
interaction over time in relation to the pandemic’s emergence. We identify
several stable clusters (super-communities), and are able to link them to
international groups mainly involved in science and health topics, national
elites, and political actors. The science- and health-related super-community
received disproportionate attention early on during the pandemic, and was
leading the discussion at the time. However, as the pandemic unfolded, the
attention shifted towards both national elites and political actors,
paralleled by the introduction of country-specific containment measures and
the growing politicization of the debate. Scientific super-community remained
present in the discussion, but experienced less reach and became more isolated
within the network. Overall, the emerging network communities are
characterized by an increased self-amplification and polarization. This makes
it generally harder for information from international health organizations or
scientific authorities to directly reach a broad audience through Twitter for
prolonged time. These results may have implications for information
dissemination along the unfolding of long-term events like epidemic diseases
on a world-wide scale.

_K_ eywords COVID-19 $\cdot$ scientific experts $\cdot$ infodemic $\cdot$
complex networks $\cdot$ Twitter

##  1 Introduction

Twitter has been widely used as a tool for emergency response in previous
crises and disasters and a large body of work has focused on optimizing
communication in order to adjust or nudge human behaviour under such
conditions [1, 2, 3, 4]. Research finds that during a time of crisis, when
information is scarce and heavily sought after, social media enables the
critical flow of information due to its collaborative nature [5, 6]. However,
the underlying network and community structure is likely to have a significant
influence over which information users are exposed to [7, 8, 9]. In
particular, retweet interactions have been shown to reflect real-life
community structure, such as political parties [10], and to characterize the
information sharing dynamics of different clusters of users [11]. Likewise,
communities of users sharing misleading news tend to be more connected and
clustered [12]. Also tweet replies and quotes allow the reconstruction of a
network structure, but the meaning and stance of these links is more ambiguous
as compared to retweets, which typically imply agreement with or validation of
the original message [13].

The structure of such information networks has been assessed in various
disease outbreak scenarios, most prominently in the context of the Zika virus
outbreak 2015–2016 in the US [14]. Among other findings the work suggests that
key international experts acted as “boundary spanners”, who were able to
spread information quickly through the network. Even though it has been
observed that false information travels faster in social networks [15, 16,
17], the work finds that Twitter users have made efforts to diffuse reliable
information by such key experts. Related work demonstrates how the association
between the geographical location of users dictates topics of conversations,
thereby matching the temporal and geographical spread of the Zika outbreak
[18].

Previously listed work allows for anecdotal evidence of crisis-induced
behaviour on social media, but compared to the COVID-19 crisis the events were
1) shorter in time scale 2) more localized and 3) orders of magnitude smaller
in terms of analyzed data. As the virus spread across the world within only a
few weeks, Twitter became the primary news source among expert groups and
medical personnel. Reliable sources, such as peer-reviewed literature or other
officially vetted channels, were simply too slow to be useful during this time
[19]. However, this initially positive role of Twitter and other social media
has now been overshadowed by the spread of numerous conspiracy theories and
other low-quality mis- and disinformation about the pandemic, peaking in what
the WHO now considers an “infodemic” [20, 21]. Preliminary work focused on
this issue and used social network analysis in order to determine drivers of
the conspiracy theory which claims a link between 5G and COVID-19 [22].
Although misinformation is a major concern, more recent work suggests that
experts and authorities are being heard and have received disproportionate
attention [23], but that key specialists may experience low reachability [24].
Moreover, other work finds that key medical professionals and scientific
experts may experience lower “sustained amplification”, meaning that the
attention given to this group has not been constant, and overall lower,
compared to media outlets or key political figures [25]. Our work attempts to
clarify the somewhat ambiguous premises about the role of experts during the
pandemic, focusing on the patterns of retweet interactions between users.
Previous works on the topic classified Twitter users involved in the COVID-19
discussion with Natural Language Processing methods [23, 24] or user
information extracted from external databases [25], which allowed the
labelling of only a fraction of the studied population. Our approach, instead,
gathered all the users into network communities, which were a-posteriori
associated to specific categories. In this way, with the help of a
comprehensive dataset of more than 350 million tweets, we identify the key
communities of English speaking Twitter users involved in the COVID-19 debate,
starting from early January to June 2020. We provide a detailed analysis of
the evolution of this massive communication network, in particular regarding
the interaction dynamics within and between the communities, showing how the
attention toward these super-communities follows different trajectories over
time.

##  2 Results

###  2.1 Network community identification

In the giant component of the directed network (see Methods), the out-degree
distribution (i.e. number of retweets received by each user) follows a very
skewed distribution, that can be fitted by a power law with exponential cut-
off (see Supplementary Fig. S1). Users with an out-degree higher than 1500
represent the 0.1% of the users in the network, but their tweets have been
retweeted >200M times (77.0% of all retweets). The community detection
algorithm reveals thousands of communities, spanning from millions down to
duplets, with size decreasing very sharply (see Supplementary Fig. S2). By
combining multiple repetitions of the clustering algorithm (see Methods), we
identify 15 communities (labeled with letters from A to O, in decreasing order
of size) with more than $10^{5}$ users, encompassing 97.9% of all users in the
giant component.

Figure 1:  Retweet network of a randomly sampled connected component of 1M
users, colored by community. Node size is proportional to node out-degree. In
the table, the column “S” designates the color of the super-communities used
throughout this work, “C” lists the community color in the network layout and
“Name” the respective community name. The “Dominant category” column specifies
the most abundant user category in the community (excluding “Other”). “Size”
denotes the ratio of users in the community with respect to the total number
of users in the network.

Figure 1 shows the network of 1M users (corresponding to 4.3% of all nodes)
randomly sampled but forming a connected component. Even if the layout
algorithm has no information about the communities we identified, it recovers
a similar stratification of the nodes. We observe that communities A and B
fill a great portion of the network, without being as densely connected as the
other communities. Some communities (C, H, F, L) are more peripheral and
exhibit a neat modular structure, which implies a high degree of internal
connectivity.

###  2.2 Characterization of communities

In order to characterize the 15 largest communities, we 1) infer the presumed
location of the users and 2) use the results of a Machine Learning classifier,
which was trained to predict the user category based on a user’s description
(see Methods). Self-reported location information was available for 54% of
nodes in the network. Since the diversity in nationality of the communities is
of interest, we calculate the entropy of the distribution of countries for
each community (similarly to the alpha diversity index in ecological
communities): a high entropy indicates an almost uniform distribution of users
across all countries, whereas a low entropy implies an uneven distribution,
possibly skewed toward a specific country. Some communities are strongly
associated with user location, in particular communities D (UK), E
(Philippines and Southeast Asia), F (India), G (African countries), J
(Canada), L (Pakistan) and M (Australia). The US represents more than a half
of the users from communities C, H, I, and K, while communities A, B, N and O
are more heterogeneous in terms of location, containing a large fraction of
tweets located outside of the US.

In some communities, the most retweeted users have a strong association to
political and cultural topics. In order to better understand these mixed
communities, we investigate the predicted user categories for each community
(see Methods, Table S1 and Supplementary Fig. S3). Out of the 13 categories,
the category “Other” is the majority category for all considered communities,
being assigned to 78% of all users and having received 22% of all retweets.
Science, being the second largest category (at 4.7% of all users), was
represented at 9% or higher in communities B, D, G, J and M. Other
communities, such as H or C, have a higher fraction of users who identify with
a political movement.

In this study, we are particularly interested in the specific user categories
“Science”, “Healthcare”, “Media”, “Politics & Government”, “Public Services”,
and “Political Supporter”. The percentage of users in these six categories and
the entropy measure for internationality were used for hierarchical clustering
at the community level (see Figure 2 and Material and methods section). With
the help of the resulting dendrogram, we recognize four super-communities,
which we name based on the categories over-represented in each super-community
(Z-score > 0):

  * •

International sci-health: communities exhibiting an increased number of users
of category “Science” or “Healthcare” and a high international diversity.

  * •

National elite: communities exhibiting an increased number of users belonging
to official categories (i.e. “Healthcare”, “Media”, “Politics & Government”,
and “Public Services”) and a low international diversity.

  * •

Political: communities exhibiting an increased number of users associated with
political activism only (category “Political Supporter”) or involved in
politics (“Politics & Government”).

  * •

Other: communities which are not linked to any of the categories of interest,
because no category has a Z-score > 0 except “Internationality”. This super-
community includes artists, entrepreneurs, and non-governmental activists.

The naming of super-communities is based on the over-represented categories
listed above, and further validated by the manual inspection of the top users
in the respective communities. In particular, only two communities (B and G)
are assigned to “International sci-health”, showing a clearly distinguishable
pattern. These communities consist of a large fraction of users who are
presumably working in scientific or health-related fields or who frequently
retweet top medical and scientific expert. Community B’s top users include
well-known news agencies, as well as the WHO, making this community led by
official media and scientific information spreaders. Community G is similar,
but it has more users from African countries, in particular Nigeria and South
Africa. National elites, i.e. communities I, J, D and M, are communities with
a high proportion of official categories not only Science and Healthcare
related, but linked to specific countries. Among the political super-
community, communities F and L have the highest proportions of users and
institutions involved in politics, while communities C, H and K are driven by
US-specific political debates. Upon visual inspection of a sample of accounts,
it emerges that community C and K are more often associated with the
Democratic party, and H with users from the Republican party. All other
communities, including community A (comprising 32% of all users), show
characteristics which were not deemed relevant for this analysis.

Figure 2:  Clustering of communities into super-communities. The heatmap shows
the Z-scores (i.e. standardized values) for seven chosen features. The four
super-communities denote the emerging clusters.

We quantify the attention received by the communities through the number of
retweets their users received (Figure 3). This measure can be decomposed into
an external and an internal component, depending on whether the retweets are
received from outside or inside the community, as further investigated in the
next section. Community C and H, assigned to the political super-community,
are the most retweeted communities, though not the largest. The attention
received by these two communities is mainly internal (more than $80\%$).
Community B, assigned to the sci-health super-community, is the second largest
one in terms of number of users but received low overall attention ($5^{th}$
most retweeted in total). Nevertheless, it has a high level of reach, ranking
$2^{nd}$ with respect to the external attention component.

Figure 3:  Internal and external components of the attention towards each
community. Each bar represents the number of total retweets received by users
in each community, divided into retweets from the same (blue, with the
percentage represented on top of each bar) and from other (orange)
communities.

###  2.3 Network dynamics

In order to trace the fluctuation of the retweet dynamics between users
throughout the pandemic, we reconstructed several networks by splitting the
Twitter data into non-overlapping windows of 1 week, with the purpose of
having a reasonable numbers of time windows with enough statistics within the
observation period. We collapsed the 1-week retweet networks into networks
with four nodes, aggregating the links between users previously assigned to
the four super-communities (see Figure 4a) and thereby compute the so-called
mixing matrix [26]. In these networks, we draw a link from super-community i
to j, with a weight $w_{ij}$ equal to the number of times super-community j
retweeted i in a given week.

We assign each node a size attribute $N_{i}$, representing the number of users
of super-community $i$ retweeting or being retweeted in the given week. Figure
4b shows the temporal evolution of this attribute. Two principal observations
can be made: first, the value of $N$ varies over time in correspondence with
the phases of the epidemic. We identify an initial peak in total number of
users in the beginning of February (peak $a$), and a second one at the end of
March (peak $b$). These two peaks presumably correspond to the first diffusion
of news about COVID-19 in China and, later, to worldwide diffusion of the
pandemic (see Supplementary Fig. S4). In the time between peak $a$ and $b$ the
number of users in the COVID-19 debate has doubled, followed by a slow decay
between April and June. Second, most of the change in the number of users
stems from the “Other” super-community. This implies that the three super-
communities of interest remained present throughout the entire observation
period at a relatively static level.

In Figure 4c, we show the average attention per user of super-community $i$,
defined as:

$A^{i}_{u}=\frac{\sum_{j}w_{ij}}{N_{i}}$ (1)

i.e. the weighted out-degree of super-community $i$, normalized by its size.
The international sci-health super-community faces an increase in average
attention per user in January and stabilizes at a higher level compared to
other super-communities until the beginning of March. After a narrow peak in
March, the political super-community plateaus in April at roughly three times
the attention level of national elites and international sci-health.

We then split the total attention, i.e. the sum of all weighted edges $W$, for
each super-community $i$ into an internal and external component for every
weekly network:

$a_{i}^{ext}=\frac{\sum_{j\neq i}w_{ij}}{W}$ (2)
$a_{i}^{int}=\frac{w_{ii}}{W}$ (3) $a_{i}^{ext}+a_{i}^{int}=1$ (4)

The external component $a_{i}^{ext}$ represents the attention given to super-
community $i$ from the other super-communities, while the internal component
$a_{i}^{int}$ quantifies self-amplification. Figure 4d shows that the external
attention component is decreasing overall, indicating a decrease in attention
between super-communities. This is particularly true for the international
sci-health super-community, which received broad attention in the very
beginning of the pandemic, peaking again in mid-February, and then decaying in
a monotonic way until the end of our sampling, while the pandemic was
spreading to all the countries worldwide.

Figure 4:  Evolution of weekly aggregated networks by super-community, with
dotted lines corresponding to the statistics across all users. (a) Diagram
representing the networks collapsed to the super-community level. Edge
direction represents the flow of information via retweets, i.e. from retweeted
to retweeting super-community. (b) Size of the super-communities in terms of
number of users. (c) Average attention per user. (d) External component of the
attention toward super-communities. (e) Internal component of the attention
toward super-communities. Indicated as $a$ and $b$ are the first and second
peak in terms of network size, as shown in Figure 4b.

Figure 4e shows that the internal attention component is increasing overall,
highlighting an increased self-amplification within the network. We observe
that this increase is mostly driven by the political super-community and to a
lesser degree by the national elites, suggesting also an increase of political
polarization within the retweets network, at the poles of which the
antagonistic political communities lie. The international sci-health super-
community, on the other hand, decreased internal sharing of content after
March. Overall, we note that the dynamics partially mirror Figure 4c, since
the internal attention component makes up most of the total attention given to
the super-communities.

Considering the average number of tweets per user within each super-community
(defined as "activity", see Supplementary Figure S5), we observe that while
the activity increased over time for the political and national elites super-
communities, there was a reduction for the international sci-health super-
community.

To further characterize the evolution of the attention patterns, we performed
an analysis of the network community structure dividing the observation period
into monthly windows. Moreover, since it is known from literature that
critical events may affect the sentiment of the discussion [27] and therefore
impact the community structure [28], we quantified on a weekly basis the
variation of sentiment ("subjectivity" and "polarity") in tweets’ content over
time by using a deep learning and a a lexicon-based algorithm (see Material
and methods section). These analyses (detailed in the Supplementary Text 1.3)
showed a stable network community structure, and no significant trend in
sentiment was observed over time. Regarding this last statement, we remark
that the null result may be due to sentiment analysis algorithms not
specifically tuned to COVID-related topics. This limitation, in particular,
has already been observed for lexicon-based algorithms [29].

###  2.4 Sustained attention towards top users

So far, in our analysis the observed dynamics of super-communities is a result
of the average behavior across all the users in the Twittersphere, while in
reality most of the dynamics are driven by a relatively small set of users who
receive disproportionate attention. Further, we have focused on the number of
retweets (node out-degree) as a canonical measure of attention, but the
retweet count of a single viral tweet might exaggerate the user’s real impact
in the overall debate.

In this section, we address these caveats. Here, we only consider the top 1000
users for each super-community in terms of retweets received, i.e. 4000 users
receiving 55.0% of all retweets. Furthermore, we adopt an alternative measure
to characterize the attention given to users, namely a retweet $h$-index, as
previously introduced by [30] and used by [25] in the context of COVID-19
posts on Twitter. Originally proposed in the context of academic citations
[31], the $h$-index in this case reflects that the user has received at least
$h$ retweets on $h$ of their original tweets.

Figure 5a and 5b compare the rank in terms of retweets received $r_{rt}$ and
$h$-index $r_{h}$ both on the user and the super-community level,
respectively. A user placed in the top-left in the figure plane suggests that
few of their tweets received punctual attention at high virality, whereas the
bottom-right suggests sustained or long-lasting attention at low virality.

Figure 5:  Static and temporal communication patterns of the top 1000 users of
each super-community. (a) Comparison between the rank in terms of $h$-index
and retweets by user, as well as respective marginal distributions. (b) The
average rank by super-community with bootstrapped 95% confidence intervals.
(c) Weekly $h$-index rank computed within a rolling time window of 1 month.
(d) Weekly rank in retweets computed within a rolling time window of 1 month.
(e) Vector plot of retweet and $h$-index ranks by super-community. Each arrow
denotes the change in ranks within one week. The first and last week are
marked with a square and circle, respectively.

Generally, users belonging to the political super-community are ranked highest
both in terms of retweets and $h$-index, receiving most of the attention both
on a punctual and an extended time scale. National elites and international
sci-health super-communities behave very similarly: they rank medium to high
in terms of $h$-index, but low in the number of retweets received (low
virality). Lastly, the “Other” category ranks generally lowest in $h$-index
and intermediate in retweets, thus is characterized by a higher virality in
terms of attention.

In order to understand the temporal dimension of these results, we formulate
the previous metrics in a time-dependent fashion on the same set of top users.
We consider all tweets and retweets posted within a rolling time window of 1
month width and a 1 week step. We then compute the rank by $h$-index (Figure
5c) and retweets (Figure 5d) averaged by super-community. Additionally, we
show the resulting data as a vector plot (Figure 5e). We find that
international sci-health users scored very highly in both metrics initially
and then experienced a drop in ranks. Similarly, national elites faced an
initial decline in both metrics but then increased in ranks above the level of
international sci-health after April. The temporal view of this data therefore
adds nuance to the picture obtained in Figure 5b.

We conclude that although national elites and the international sci-health
super-communities share a considerable overlap in the static view (cf. Figure
5b), they reveal distinct temporal dynamics (cf. Figure 5c–e). This result
reflects how the international sci-health super-community faced a decline in
attention, while the discussion has been moving onto more local grounds, with
the political and national debate gaining momentum.

##  3 Discussion

In this work we reconstruct and describe the complex network structure of the
user retweets in relation to the COVID-19 pandemic, and study the evolution of
the interactions between and the attention towards its communities. We also
make use of the network structure to characterize the role of international
communities involved in science and health related topics in the English-
speaking Twittersphere.
Using a community detection algorithm, we identify 15 user communities,
exhibiting a stable structure throughout the observation period. While
previous studies rely on user descriptions to cluster them (providing a proper
labelling of only a fraction of the users), we firstly identify communities
based on the whole retweet link structure, then we find the prevalent user
categories within each community, thus allowing the classification of all
users in the available dataset. We are able to group these communities into
four main super-communities related to the prevalent user categories and the
degree of internationality (namely international sci-health, national elite,
political, and other) and assess their interaction patterns over time,
focusing on the flow of intra- and inter- community retweet volume.

In the Twitter landscape of COVID-19 we identify a single major group with
interests in scientific and health topics and with a highly international
distribution of users, and multiple country-specific communities which appear
to engage more in the respective national debates. Additionally, we find
several large country-specific communities which are mostly characterized by
political activism, thus highlighting the substantial politicization in the
discussion surrounding the pandemic.

Our results emphasize the role of the international sci-health super-community
in the beginning of the, at the time, largely unknown pandemic. This super-
community received disproportionate attention and had broad reach across many
cultural and political communities, as reflected by the high total volume of
retweets received, in particular from users that we characterized as belonging
to non-scientific communities. As demonstrated by the high internal attention
component, the users of this super-community also shared content frequently
among themselves, possibly in an attempt to rapidly share scientific insights
about the novel virus.

In a second critical moment in March, the number of COVID-19 cases exploded in
almost all parts of the world, leading to massive media attention, which is
well reflected in the increase in community sizes, as well as in the total
number of tweets. It is noteworthy that this added attention has not been
allocated to international sci-health experts anymore but rather to local
political leaders, reflecting $1)$ the loss of global influence of the former
in the evolution of the debate, and $2)$ the fact that users were redirecting
their attention towards their local authorities and referents. This might be
due to the fact that COVID-19 pandemic was no more an external phenomenon
arousing a general interest, but it was starting to directly impact people’s
lives. In conjunction to this shift in attention, the international sci-health
super-community decreased its activity (original tweets per user), while for
national elites and political super-communities it increased.

As the pandemic unfolded in April, and while many English speaking countries
underwent a strict lockdown, we observe a growing politicization of the
debate, reflected by the fact that content by the political communities is now
shared most. Meanwhile, the analysis reveals the picture of a less retweeted
international sci-health super-community. Furthermore, compared to January and
February, and in contrast to all other groups, this super-community also
reduced its activity and the interactions among themselves, even though their
size remained constant.

The analysis of the sustained amplification patterns (measured by the retweet
h-index) of highly retweeted users extends our previous analysis performed at
a super-community level: compared to political leaders, top users in the
international sci-health super-community received intermediate levels of
sustained attention but their content lacked virality. Additionally, the
results show the importance of the national elites in influencing the
discussion after the end of March: national elites’ top users show a positive
trend in both punctual and sustained attention received, to the point of
surpassing the international sci-health super-community.

Our work, characterizing the evolution over time of these observables,
provides an explanation for the discrepancies found in recent literature on
the role of scientific information spreaders in the first months of the
COVID-19 pandemic. By characterizing the retweet network structure dynamics,
in parallel to the worldwide spreading of the pandemic, we are able to confirm
that the scientific sci-health super-community was mostly heard early on in
the pandemic, as found in [23] which utilized the same algorithm for user
classification. This also fits in well with previous literature, which
confirms that Twitter users amplify information from trusted sources in crisis
situations [32, 33, 34]. Further, we find that the attention towards sci-
health users has declined over time, whereas other authors [24] indicated a
low reach of doctors and experts, without acknowledging the leading role in
the first months of international communities interested in science and
healthcare. For the same reason, we can only partially confirm previous work
[25] that suggested low sustained attention to top medical experts.
We do not know the exact overlap between our sci-health super-community and
the expert groups defined in the previous papers, thus it is hard to make a
direct comparison. Nevertheless our study identified the communities gathered
around the top sci-health users and added a temporal dimension to the
analysis, showing that the sci-health super-community ranked highest both in
sustained and punctual attention in the very beginning of the pandemic and
only later became increasingly isolated in terms of attention. Our results
underline the different role of sci-health over time, seen as possible
boundary spanners only during the early phase of the pandemic. We observed
that the sentiment of tweet content remained quite stable over time. This
might suggest that the cause of this drop in attention is not due to a
different emotional response or a change in perception of the messages. It’s
possible that the algorithms we used for sentiment analysis, which were not
specifically trained on COVID related text, may have not been sensitive
enough. Overall, the international sci-health super-community lost some weight
in the discussion when the epidemic issue became more impactful on a local
scale and politics came into play, attracting more attention, and possibly
because it reduced the tweeting activity compared to the first months. Anyway,
the emerging political and national authorities did not have the same degree
of inter-community reach, making the discussion more compartmentalized.

##  4 Conclusion

Our work leads to two main conclusions: 1) under the unique circumstance of an
emerging virus causing a global pandemic, the Twitter platform allowed
thousands of international users with experience and interest in sci-helath
topics (including leading experts such as the WHO) to quickly and efficiently
exchange information, while also being amplified by non-scientific
communities. This effect was localized only in the early phase of the
pandemic, before its spread worldwide, since 2) as the pandemic developed,
Twitter users directed more of their attention towards the national debates,
overall leading to less interacting communities with respect to the initial
phase.

Our study is based on a comparatively large dataset encompassing a total of
354M tweets by 26M users, which can be considered as comprehensive (see
Methods). However, a limitation of this work is that a substantial part of the
network consists of the super-community labelled as “Other”, encompassing
around 50% of all users. It is difficult to make general statements about the
true nature of this group, as it includes a very diverse set of users, with
most of them reporting unspecific profile information. Our work is based on
the self-reported status of users, thus future research could be required to
properly understand their true nature.

As a final consideration, we can hypothesize that, as long as the COVID
pandemic was a novelty, Twitter users from all the super-communities tended to
give more attention to the messages from the sci-health super-community
because it was initially considered as the most authoritative reference in
this topic. Our network analysis highlights that in this first phase the
amount of internal debate within political and national elite communities was
reduced, possibly due to the fact that the pandemic was not impacting all
local levels directly. After a few months from its emergence, the epidemic
started to create social and health problems in many countries, prompting
Twitter users to turn their attention to politics and national issues. It also
is noteworthy that the international sci-health super-community itself reduced
the level of self-amplification and its tweeting activity. We verified that
this attention shift can not be clearly explained by a change in the sentiment
of the tweet content, since the algorithms we used provided a constant output
over the observed time period.

Our results suggest that an exclusively scientific discussion risks losing
audience when the health crisis starts to heavily impact society and to feed
country-specific debates, thus scientists and health institutions should
maintain a regular tweeting activity and reshape their valuable content in
order to involve themselves usefully in local discussions, targeting and
merging with the stable national Twitter communities.

##  5 Materials and methods

###  5.1 Data collection

Twitter data was collected through the Twitter API, specifically through the
filter streaming endpoint, using the Crowdbreaks platform [35]. The data used
in this work consists of a total of $353,993,900$ tweets (thereof
$267,026,740$ retweets) posted by $26,262,332$ users in a 146 day observation
period, i.e. from January 13 to June 7, 2020. These tweets have been
identified by Twitter to be in English language and match one or more of the
keywords “wuhan”, “ncov”, “coronavirus”, “covid” and “sars-cov-2”. Note that
keywords have changed over time, as the new names for virus were introduced
(for details refer to Supplementary Text 1.1 and Supplementary Table S2). The
data is complete with respect to these keywords, except during a period
between mid-March to mid-April when volume exceeded the 1% threshold imposed
by Twitter and was subsampled by an (unknown) degree.

###  5.2 User categorization

In order to be able to interpret the identified network communities, accounts
were categorized by occupational role and account type. This categorization
was conducted using a Machine Learning classifier which was trained to
determine the category of an account solely based on the user description
(user bio). The category assigned to each user is based on self-reported
information, though it reports a self-declared expertise or interest in a
particular category more than a verified expertise in it. The classifiers were
first published in the context of related work on the attention given to
experts during COVID-19 [23]. In this work, we use the published English
language BERT model (bert-english-pt) in order to determine the category of
each account in our dataset. In the aforementioned study [23], the categories
have been determined in an iterative coding process to best categorize users
into 13 categories, which are: Adult content, Arts & Entertainment, Business,
Healthcare, Media, Non-governmental organization (NGO), Political Supporter,
Government & Politics, Public Services, Religion, Science, Sports, and Other.
For further details on the coding process or the training of the machine
learning classifiers, please refer to the referenced study [23].

###  5.3 Geo-localization

Tweet objects contain both structured and unstructured forms of geographical
information. In this work, we employed a procedure to geo-localize tweets on
the country level using the Python library local-geocode
(https://github.com/mar-muel/local-geocode), please refer to Supplementary
Text 1.2 for detailed explanations). A user’s country location was determined
from the majority of the user’s geo-localized tweets. Geolocation could be
inferred for 75% of tweets belonging to 54% of the considered users.

###  5.4 Network analysis

We study the full directed retweet network, consisting of all retweets
collected during the entire 147 day observation period (267M retweets). The
nodes of the network represent users who have at least once retweeted or have
been retweeted by another user. An edge was established from user A to user B
if B retweeted A at least once during the whole period of data collection.
Therefore, the edge direction indicates the flow of information: the higher
the out-degree, the more the user has been retweeted. We assigned a weight to
this edge equal to the number of times user B has retweeted user A. The
reconstructed (weighted and directed) network has 22.9M nodes and 177M unique
edges. In order to study the communities in this network, we consider only the
largest connected component of the network, consisting of 22.5M nodes and 176M
unique edges. The discarded components consisted of small star-like
communities (size range 1-280, median:2), making up 1.57% of the nodes and
0.13% of the edges in the original network. Since Twitter, as many other
social networks, is characterized by a high degree of homophily, defined as
the tendency of similar individuals to form ties with each other [36, 37]. We
made use of Louvain algorithm for community detection [38], which maximizes
the network modularity. We symmetrized the network since we were not
interested in the direction of retweet flux, but rather we constructed
communities on the total volume of tweet exchange between users that defines
the tightness of their bond. We used the Python package Networkit [39] for its
efficiency in handling large networks, considering the default for the
resolution parameter $\gamma=1$. Due to the stochasticity of the clustering
algorithm, we ran 50 trials with different random initialization and assigned
each node to the community it was most frequently associated with. The
identification of largest communities was found to be stable both among
repeated runs of the algorithm (Supplementary Fig. S6). In the main text, we
show the results for the network obtained from the whole time-series of
retweets. In Supplementary Text 1.3 we show that networks reconstructed from
monthly time-windows are very stable over-time both at community and super-
community level. Thus, the results of the community detection can be
considered as fairly robust. The coordinates of the network layout in Figure 1
were processed by Gephi software [40], using the ForceAtlas2 algorithm [41]
with gravity set to 0.05 with the “stronger gravity” option enabled. The out-
degree distribution of the network (Supplementary Fig. S1) was analyzed
through the package powerlaw [42].

###  5.5 Sentiment analysis

We performed sentiment analysis through the Python’s library TextBlob, which
operates a lexicon based analysis of the text to compute the subjectivity and
the polarity of a message [43]. This algorithm assign to each original tweet
1) a polarity score, corresponding to a sentiment which spans from negative
(-1) to positive (+1), and 2) a subjectivity score, where a value of 1
generally refer to personal opinion, emotion or judgment whereas a value of 0
refers to objective and factual information. We then classified each retweet
in the dataset as Negative, Neutral or Positive depending on whether the
shared tweet had polarity less than -1/3, between -1/3 and +1/3 or higher than
1/3 respectively. For comparison, we also applied a roBERTa deep learning
model for sentiment classification into negative, neutral and positive
sentiment [44].

###  5.6 Super-communities definition

We mapped each community detected on the retweet network to a vector
concatenating the relative abundance of each user category within the
community and the internationality measure. We standardized the features
(Z-scores) and built an agglomerative hierarchical dendrogram with the Ward’s
method for variance minimization. We identified the knee point of the average
inter-clusters distance at 3 clusters, through the maximum of the 2nd
derivative (see Supplementary Fig. S7). The emerging clusters were
[E,N,A,O,B,G], [I,J,D,M], [F,L,H,C,K]. We decided to split the first cluster
into [E,N,A,O] and [B,G] because of the overabundance of "Science" users in
the latter 2 communities (Z-score > 0), which is a relevant category for the
topic of interest. Furthermore, [E, N, A, O] are not over-represented by any
of the categories of interest, but rather populated by users associated to
Sports, Adult content, Business, Arts & Entertainment (see Supplementary Fig.
S3).

#### Data availability.

The datasets and the code generated and/or analysed during the current stury
are available through the public GitHub repository
https://github.com/FraDurazzi/twitter-network-covid19. The full Twitter
dataset used in this work is available on Zenodo [45]:
https://doi.org/10.5281/zenodo.4267033. More information is available upon
reasonable request from the authors.

#### Author contributions.

M.M. collected the data. F.D. and M.M. analyzed the data. F.D., M.M., D.R. and
M.S. designed the study and wrote the paper.

#### Competing interests.

The authors declare no competing interests.

#### Acknowledgments.

We thank Marion Koopmans for her valuable comments and ideas that helped to
design the study. This project was funded through the European Union’s Horizon
2020 research and innovation programme under grant agreement No. 874735
“Versatile emerging infectious disease observatory - forecasting, nowcasting
and tracking in a changing world (VEO)”.

## References

  * [1] Rui Chen, Raj Sharman, H. Raghav Rao, and Shambhu J. Upadhyaya.  Coordination in emergency response management.  Communications of the ACM, 51(5):66–73, 2008.
  * [2] Jessica Li and H.R. Rao.  Twitter as a Rapid Response News Service: An Exploration in the Context of the 2008 China Earthquake.  The Electronic Journal of Information Systems in Developing Countries, 42(1):1–22, 2010.
  * [3] María Martínez-Rojas, María del Carmen Pardo-Ferreira, and Juan Carlos Rubio-Romero.  Twitter as a tool for the management and analysis of emergency situations: A systematic literature review.  International Journal of Information Management, 43:196–208, 2018\.
  * [4] Marcel Salathé, Clark C. Freifeld, Sumiko R. Mekaru, Anna F. Tomasulo, and John S. Brownstein.  Influenza A (H7N9) and the Importance of Digital Epidemiology.  New England Journal of Medicine, 369(5):401–404, 2013.
  * [5] Melissa W. Graham, Elizabeth J. Avery, and Sejin Park.  The role of social media in local government crisis communications.  Public Relations Review, 41(3):386–394, 2015.
  * [6] Amanda Lee Hughes and Leysia Palen.  Twitter adoption and use in mass convergence and emergency events.  International Journal of Emergency Management, 6(3/4), 2009.
  * [7] Kazutoshi Sasahara, Wen Chen, Hao Peng, Giovanni Luca Ciampaglia, Alessandro Flammini, and Filippo Menczer.  Social influence and unfollowing accelerate the emergence of echo chambers.  Journal of Computational Social Science, 2020.
  * [8] M Conover, J Ratkiewicz, and M Francisco.  Political polarization on twitter.  In Icwsm, 2011.
  * [9] M. E.J. Newman.  Modularity and community structure in networks.  In Proceedings of the National Academy of Sciences of the United States of America, 2006.
  * [10] Darko Cherepnalkoski and Igor Mozetič.  Retweet networks of the European Parliament: evaluation of the community structure.  Applied Network Science, 1, 2016.
  * [11] Alexandre Bovet and Hernán A. Makse.  Influence of fake news in Twitter during the 2016 US presidential election.  Nature Communications, 10, 2019.
  * [12] Francesco Pierri, Carlo Piccardi, and Stefano Ceri.  Topology comparison of Twitter diffusion networks effectively reveals misleading information.  Scientific Reports, 10, 2020.
  * [13] Danah Boyd, Scott Golder, and Gilad Lotan.  Tweet, tweet, retweet: Conversational aspects of retweeting on twitter.  In 2010 43rd Hawaii International Conference on System Sciences, pages 1–10, 2010.
  * [14] Loni Hagen, Thomas Keller, Stephen Neely, Nic DePaula, and Claudia Robert-Cooperman.  Crisis Communications in the Age of Social Media: A Network Analysis of Zika-Related Tweets.  Social Science Computer Review, 36(5):523–541, 2018.
  * [15] Marcelo Mendoza, Barbara Poblete, and Carlos Castillo.  Twitter under crisis: Can we trust what we RT?  In SOMA 2010 - Proceedings of the 1st Workshop on Social Media Analytics, 2010.
  * [16] Michela Del Vicario, Alessandro Bessi, Fabiana Zollo, Fabio Petroni, Antonio Scala, Guido Caldarelli, H. Eugene Stanley, and Walter Quattrociocchi.  The spreading of misinformation online.  Proceedings of the National Academy of Sciences of the United States of America, 2016.
  * [17] Soroush Vosoughi, Deb Roy, and Sinan Aral.  The spread of true and false news online.  Science, 359(6380):1146–1151, 2018.
  * [18] Anthony Stefanidis, Emily Vraga, Georgios Lamprianidis, Jacek Radzikowski, Paul L Delamater, Kathryn H Jacobsen, Dieter Pfoser, Arie Croitoru, and Andrew Crooks.  Zika in Twitter: Temporal Variations of Locations, Actors, and Concepts.  JMIR Public Health and Surveillance, 3(2), 2017.
  * [19] Hans Rosenberg, Shahbaz Syed, and Salim Rezaie.  The Twitter pandemic: The critical role of Twitter in the dissemination of medical information and misinformation during the COVID-19 pandemic.  Canadian Journal of Emergency Medicine, 22(4):418–421, jul 2020\.
  * [20] WHO Director General Tedros Adhanom Ghebreyesus at the Munich Security Conference on February 15, 2020, 2020.
  * [21] Matteo Cinelli, Walter Quattrociocchi, Alessandro Galeazzi, Carlo Michele Valensise, Emanuele Brugnoli, Ana Lucia Schmidt, Paola Zola, Fabiana Zollo, and Antonio Scala.  The COVID-19 social media infodemic.  Scientific Reports, 10, mar 2020.
  * [22] Wasim Ahmed, Josep Vidal-Alaball, Joseph Downing, and Francesc López Seguí.  COVID-19 and the 5G conspiracy theory: Social network analysis of twitter data.  Journal of Medical Internet Research, 22(5), 2020.
  * [23] Kristina Gligoric, Manoel Horta Ribeiro, Martin Müller, Olesia Altunina, Maxime Peyrard, Marcel Salathé, Giovanni Colavizza, and Robert West.  Experts and authorities receive disproportionate attention on Twitter during the COVID-19 crisis.  Preprint at https://arxiv.org/abs/2008.08364, 2020.
  * [24] A. Mourad, A. Srour, H. Harmanani, C. Jenainati, and M. Arafeh.  Critical Impact of Social Networks Infodemic on Defeating Coronavirus COVID-19 Pandemic: Twitter-Based Study and Research Directions.  IEEE Transactions on Network and Service Management, 2020.
  * [25] Ryan J Gallagher, Larissa Doroshenko, Sarah Shugars, David Lazer, and Brooke Foucault Welles.  Sustained Online Amplification of COVID-19 Elites in the United States.  Preprint at https://arxiv.org/abs/2009.07255, 2020.
  * [26] Mark Newman.  Networks: An introduction.  OUP Oxford, Oxford, 2010.
  * [27] David Garcia and Bernard Rimé.  Collective emotions and social resilience in the digital traces after a terrorist attack.  Psychological Science, 30(4):617–628, apr 2019.
  * [28] M. Mitrović, G. Paltoglou, and B. Tadić.  Networks and emotion-driven user communities at popular blogs.  The European Physical Journal B 2010 77:4, 77(4):597–609, sep 2010\.
  * [29] David Zimbra, Ahmed Abbasi, Daniel Zeng, and Hsinchun Chen.  The state-of-the-art in twitter sentiment analysis: A review and benchmark evaluation.  ACM Transactions on Management Information Systems, 9(2):5, apr 2018\.
  * [30] Miha Grčar, Darko Cherepnalkoski, Igor Mozetič, and Petra Kralj Novak.  Stance and influence of twitter users regarding the brexit referendum.  Computational Social Networks, 4(6), 2017.
  * [31] J. E. Hirsch.  An index to quantify an individual’s scientific research output.  Proceedings of the National Academy of Sciences of the United States of America, 102(46):16569–16572, 2005.
  * [32] Christian Reuter and Marc André Kaufhold.  Fifteen years of social media in emergencies: A retrospective review and future directions for crisis Informatics.  Journal of Contingencies and Crisis Management, 26(1):41–57, 2018\.
  * [33] Moritz Wagner, Vasileios Lampos, Ingemar J. Cox, and Richard Pebody.  The added value of online user-generated content in traditional methods for influenza surveillance.  Scientific Reports, 8, 2018.
  * [34] Soo Yong Shin, Dong Woo Seo, Jisun An, Haewoon Kwak, Sung Han Kim, Jin Gwack, and Min Woo Jo.  High correlation of Middle East respiratory syndrome spread with Google search and Twitter trends in Korea.  Scientific Reports, 6, 2016.
  * [35] Martin M. Müller and Marcel Salathé.  Crowdbreaks: Tracking health trends using public social media data and crowdsourcing.  Frontiers in Public Health, 7, 2019.
  * [36] Elanor Colleoni, Alessandro Rozza, and Adam Arvidsson.  Echo Chamber or Public Sphere? Predicting Political Orientation and Measuring Political Homophily in Twitter Using Big Data.  Journal of Communication, 64(2):317–332, apr 2014.
  * [37] Hywel T.P. Williams, James R. McMurray, Tim Kurz, and F. Hugo Lambert.  Network analysis reveals open forums and echo chambers in social media discussions of climate change.  Global Environmental Change, 32:126–138, may 2015.
  * [38] Vincent D. Blondel, Jean Loup Guillaume, Renaud Lambiotte, and Etienne Lefebvre.  Fast unfolding of communities in large networks.  Journal of Statistical Mechanics: Theory and Experiment, 10:1–12, 2008.
  * [39] Christian L. Staudt and Henning Meyerhenke.  Engineering Parallel Algorithms for Community Detection in Massive Networks.  IEEE Transactions on Parallel and Distributed Systems, 27(1):171 – 184, 2016.
  * [40] Mathieu Bastian, Sebastien Heymann, and Mathieu Jacomy.  Gephi: An Open Source Software for Exploring and Manipulating Networks.  In Third International AAAI Conference on Weblogs and Social Media, 2009.
  * [41] Mathieu Jacomy, Tommaso Venturini, Sebastien Heymann, and Mathieu Bastian.  ForceAtlas2, a continuous graph layout algorithm for handy network visualization designed for the Gephi software.  PLoS ONE, 9(6), 2014.
  * [42] Jeff Alstott, Ed Bullmore, and Dietmar Plenz.  powerlaw: A python package for analysis of heavy-tailed distributions.  PLoS ONE, 9, 2014.
  * [43] Steven Loria.  textblob documentation.  Release 0.15, 2, 2018.
  * [44] Francesco Barbieri, Jose Camacho-Collados, Luis Espinosa Anke, and Leonardo Neves.  TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification.  In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1644–1650, Online, November 2020. Association for Computational Linguistics.
  * [45] Martin Müller, Francesco Durazzi, Daniel Remondini, and Marcel Salathé.  COVID-19 Twitter data, keyword stream 2020-01-13 to 2020-06-06.  _Zenodo_ https://doi.org/10.5281/zenodo.4267033, 2020.


††thanks: These two authors contributed equally to this work.††thanks: These
two authors contributed equally to this work.

# CTGNN: Crystal Transformer Graph Neural Network for Crystal Material
Property Prediction

Zijian Du  School of Information Science and Technology, Fudan University,
Shanghai 200433, China.  Department of Physics, Fudan University, Shanghai
200433, China.  Luozhijie Jin  School of Information Science and Technology,
Fudan University, Shanghai 200433, China.  Le Shu  School of Information
Science and Technology, Fudan University, Shanghai 200433, China.  Yan Cen
<EMAIL_ADDRESS>Department of Physics, Fudan University, Shanghai 200433,
China.  Yuanfeng Xu  School of Science, Shandong Jianzhu University, Jinan
250101, Shandong, China  Yongfeng Mei  Department of Materials, Fudan
University, Shanghai 200433, China.  Hao Zhang<EMAIL_ADDRESS>School of
Information Science and Technology, Fudan University, Shanghai 200433, China.
Department of Optical Science and Engineering and Key Laboratory of Micro and
Nano Photonic Structures (Ministry of Education), Fudan University, Shanghai
200433, China.  State Key Laboratory of Photovoltaic Science and Technology,
Fudan University, Shanghai 200433, China

###### Abstract

The combination of deep learning algorithm and materials science has made
significant progress in predicting novel materials and understanding various
behaviours of materials. Here, we introduced a new model called as the Crystal
Transformer Graph Neural Network (CTGNN), which combines the advantages of
Transformer model and graph neural networks to address the complexity of
structure-properties relation of material data. Compared to the state-of-the-
art models, CTGNN incorporates the graph network structure for capturing local
atomic interactions and the dual-Transformer structures to model intra-crystal
and inter-atomic relationships comprehensively. The benchmark carried on by
the proposed CTGNN indicates that CTGNN significantly outperforms existing
models like CGCNN and MEGNET in the prediction of formation energy and bandgap
properties. Our work highlights the potential of CTGNN to enhance the
performance of properties prediction and accelerates the discovery of new
materials, particularly for perovskite materials.

##  I Introduction

Deep learning (DL) and machine learning (ML) has brought significant impacts
to a variety of scientific fields such as biologyzhou2020graph ,
chemistryjumper2021highly ,physicskirkpatrick2021pushing  and
mathematicsdavies2021advancing . While in materials science, the use of deep
learning has led to important progress in material properties
predictionxie2018crystal , materials generationxie2021crystal  ,
etcPredictingBandGaps ; Butler2018 ; lecun2015deep ; wu2020comprehensive .
Several DL models have been developed to capture material modality and predict
their properties, such as Crystal Graph Convolutional Neural Network
(CGCNN)xie2018crystal , MatErials Graph Network (MEGNET)chen2019graph ,
Atomistic Line Graph Neural Network (ALIGNN)choudhary2021atomistic , improved
Crystal Graph Convolutional Neural Networks (iCGCNN)park2020developing ,
OrbNetqiao2020orbnet , and similar variantsgasteiger2020directional ;
gasteiger2020fast ; shui2020heterogeneous ; schutt2017quantum ;
anderson2019cormorant ; zhang2020molecular ; schütt2018schnetpack;
jha2018elemnet ; westermayr2020combining ; wen2021bondnet ;
isayev2017universal . They have achieved great success in applications, such
as learning properties from multi-fidelity datachen2021learning , discovering
stable lead-free hybrid organic–inorganic perovskiteslu2018accelerated ,
mapping the crystal-structure phasechen2021automating , and designing material
microstructureslee2021fast .

Despite the graph-based DL model, the Transformervaswani2017attention  model
provides a new way to capture the material information, and some models based
on Transformers to predict material properties have been developed, such as
MatFormeryan2022periodic , Graphformermin2022transformer  , etc. These models
integrate the Transformer as the core network, utilizing the connections
within graphs as the queries, keys, and values (QKV) in the attention
mechanisms, distinguishing them from traditional graph neural networks.
Therefore, they lose the conventional graph structuremin2022transformer . Some
other models based on Transformer models use the structures of graph neural
networks such as ADA-GNNHuang2024ada-gnn: , TG-GNNPPR:PPR527806 ,
GATGNNLouis2020GlobalAB , etc. But these models further introduce high
complexity on the basis of Transformer architecture, which are not conducive
to model training. To address the aforementioned limitations, in this work,
the Crystal Transformer Graph Neural Network (CTGNN) is proposed, which
combines the Transformer structures’ message capturing capabilities and
traditional inductive bias of GNNs.

Generally, the GNN-based models extract structural data such as bond length,
angles, and neighbour atoms, which are important information to predict the
materials properties. In contrast to traditional GNNs which only capture bond
length, our proposed CTGNN employs an angular encoder kernel to encode angle
features, and the dual-Transformer structures are built, which include one
Transformer architecture focusing on intra-crystal interactions to model the
immediate chemical environment of atoms, and another to analyze inter-atomic
relationships within an atom’s neighborhood facilitates a thorough
understanding of material behaviors on both local and broader scales. In this
work, we conducted a series of ablation experiments to verify the importance
of our angular encoding and Transformer architecture in improving model
accuracy. We also tested the performance of CTGNN on some widely-used
materials database, achieving better results than other models we used for
comparison.

##  II Model

###  II.1 Transformer Model

The Transformer modelvaswani2017attention , a key component in the CTGNN
architecture, is based on the multihead-self-attention mechanisms. The multi-
head design allows the model to efficiently process sequential data in
parallel, enhancing its abilities, while the self-attention method allows it
to learn relationships between sequences, which is the core of the algorithm.

The self-attention mechanism can be described by

$\text{Attention}(Q,K,V)=\text{softmax}\left(\frac{QK^{T}}{\sqrt{d_{k}}}\right)V$
(1)

where $Q$, $K$, and $V$ represent queries, keys, and values, while $d_{k}$ is
the dimension of the key vectors, serving as a reweighting factor that
stablizes training process. The multi-head-self-attention allows the model to
simultaneously process information from different groups,

$\text{MultiHead}(Q,K,V)=\text{Concat}(\text{head}_{1},\ldots,\text{head}_{h})W^{O}$
(2)

where $\text{head}_{i}=\text{Attention}(QW_{i}^{Q},KW_{i}^{K},VW_{i}^{V})$,
and $W^{O}$ is a parameter matrix.

Then the feed-forward Networks, consisting of two linear transformations with
a ReLU activation in between, are given by

$\text{FFN}(x)=\max(0,xW_{1}+b_{1})W_{2}+b_{2}$ (3)

where $W_{1}$ and $W_{2}$ are weight matrices, and $b_{1}$ and $b_{2}$ are
bias vectors. These networks are applied independently to each position in the
sequence.

###  II.2 CTGNN Model

Figure 1: Our CTGNN framework. Transformer Layer denotes as Transformer
encoder for atom features and neighbor features. After the Transformer Layers,
CGCNN convolution layers are used. And finally pooling and predicting layers
are used to get the prediction. Topological features include angular and
distances (RBF) information.

The framework of the proposed CTGNN model is shown in Fig1. In this framework,
each atom is denoted as a node while each atom connection as an edge. The node
$i$ in the graph $G$ is characterized by a vector $v_{i}$, while the
connection between two nodes $i$,$j$ is denoted as an edge vector
$u(i,j)_{k}$. The node and edge features are updated by Transformer-based
graph convolution calculations.

First, the node features are updated by the intra-crystal Transformer layer
as,

$v_{i}^{(t)}=\text{MultiHead}(Q,K,V)+v_{i}^{(t-1)}$ (4)
$Q=K=V=\text{Linear}(v_{i}^{(t-1)})$ (5)

where $\text{MultiHead}(Q,K,V)$ is the multi-head self-attention mechanism
defined as:

$\text{Attention}(Q,K,V)=\text{softmax}\left(\frac{QK^{T}}{\sqrt{d_{k}}}\right)V$
(6)
$\text{MultiHead}(Q,K,V)=\text{Concat}(\text{head}_{1},\ldots,\text{head}_{h})W^{O}$
(7) $\text{head}_{i}=\text{Attention}(QW_{i}^{Q},KW_{i}^{K},VW_{i}^{V})$ (8)

After the multi-head attention, the feed-forward network (FFN) is applied,

$v_{i}^{(t)}=\text{FFN}(v_{i}^{(t)})$ (9)
$\text{FFN}(x)=\max(0,xW_{1}+b_{1})W_{2}+b_{2}$ (10)

Similarly, the edge features are updated by the inter-atomic Transformer
layer,

$u(i,j)_{k}^{(t)}=\text{MultiHead}(Q,K,V)+u(i,j)_{k}^{(t-1)}$ (11)
$Q=K=V=\text{Linear}(u(i,j)_{k}^{(t-1)})$ (12)

where the same multi-head self-attention and FFN mechanisms are applied to the
edge features,

$u(i,j)_{k}^{(t)}=\text{FFN}(u(i,j)_{k}^{(t)})$ (13)

After updating the node and edge features with the intra-crystal and inter-
atomic dual-Transformer layers, the model concatenates the features by,

$z_{i,j,k}^{(t)}=v_{i}^{(t)}\oplus v_{j}^{(t)}\oplus u(i,j)_{k}$ (14)

where $\oplus$ means concatenation and $z_{i,j,k}^{(t)}$ represents the
combined feature of atoms and edges.

Then the atom feature vectors are updated through a non-linear graph
convolution function,

$v_{i}^{(t+1)}=v_{i}^{(t)}+\sum_{j,k}\sigma(z_{i,j,k}^{(t)}W_{f}^{(t)}+b_{f}^{(t)})\odot
g(z_{i,j,k}^{(t)}W_{s}^{(t)}+b_{s}^{(t)})$ (15)

where $g$ denotes the activation function, $\sigma$ is the sigmoid function
served as a gate, $\odot$ denotes element-wise multiplication, $W_{f}^{(t)}$
and $W_{s}^{(t)}$ are the convolution weight matrices, and $b_{f}^{(t)}$ and
$b_{s}^{(t)}$ are the bias vectors.

After $R$ convolutional iterations, the network updates the feature vector
$v_{i}^{(R)}$ for each atom. A pooling layer then aggregates these vectors
into a global feature vector $v_{c}$ for the entire crystal by,

$v_{c}=\text{Pooling}(\\{v_{i}^{(R)}|\ i\in\text{atoms}\\})$ (16)

The pooled vector $v_{c}$ is a vector containing all the information of the
subgraphs of the crystal. It is permutation invariant, so it can capture the
information ignoring the noise and rotate translation. It is then passed
through fully-connected layers to predict the target property $\hat{y}$, with
the training process minimizing the cost function $J(y,\hat{y})$, representing
the difference between the predicted property $\hat{y}$ and the DFT-calculated
property $y$.

###  II.3 Crystal Angular Encoder

To enrich the information of edges, we go beyond merely distance information
by adding angular information into the edges, which allows for a more detailed
depiction of the spatial relationships between atoms. For a given atom $i$ and
its neighbor $j$, we calculated the related angles $\theta_{x}$, $\theta_{y}$,
and $\theta_{z}$ between the edge vector $\mathbf{r}_{ij}$ and the axes
vectors $\mathbf{x}$, $\mathbf{y}$, and $\mathbf{z}$ by,

$\theta_{x}=\cos^{-1}\left(\frac{\mathbf{r}_{ij}\cdot\mathbf{x}}{\|\mathbf{r}_{ij}\|\|\mathbf{x}\|}\right),$
(17)

$\theta_{y}=\cos^{-1}\left(\frac{\mathbf{r}_{ij}\cdot\mathbf{y}}{\|\mathbf{r}_{ij}\|\|\mathbf{y}\|}\right),$
(18)

$\theta_{z}=\cos^{-1}\left(\frac{\mathbf{r}_{ij}\cdot\mathbf{z}}{\|\mathbf{r}_{ij}\|\|\mathbf{z}\|}\right).$
(19)

These angles are then encoded into a feature vector using an angular encoder,
which discretizes each angle into one of several predefined bins, written as,

$\text{Encoded}_{\theta_{x}}[k]=\begin{cases}1,&\text{if }\ \
k\cdot\Delta\theta\leq\theta_{x}<(k+1)\cdot\Delta\theta\\\
0,&\text{otherwise}\end{cases},$ (20)

where $\displaystyle\Delta\theta=\frac{2\pi}{\text{bins}}$ and $k$ ranges from
0 to $\text{bins}-1$. Analogous encoding processes apply to $\theta_{y}$ and
$\theta_{z}$. The edge feature vector which combining RBF (Radial Basis
Function) and angular features, is used as the edge feature which is further
processed in the model.

##  III benchmark

In order to evaluate the performance, we used JARVIS-DFTchoudhary2014jarvis ,
dated 2021.8.18 as the training database. The dataset comprises 25,922
materials with bandgap, formation energy, and etc. For training, validation
and testing splits, JARVIS-DFTchoudhary2014jarvis  database and its properties
are split into 60% training, 20% validation, and 20% testing sets. To further
evaluate the performance, we merge two distinct datasets of perovskite
materials kim2017hybrid ; nakajima2017discovery  to create a more diverse and
representative dataset which contains 3489 perovskite structures with
formation energy and bandgap, key properties for perovskites. For comparison,
the state-of-the-art GNN models of CGCNN and MEGNET are also used to model the
formation energies and bandgaps of the materials involved in the teo materials
database, and the resuts are listed in Table1

Table 1: benchmark on jarvist dataset and perovskite dataset. $E_{f}$ and $E_{g}$ denotes formation energy and bandgap respectively. target |  model |  MAE |  R2
---|---|---|---
Pero($E_{f}$) (eV/atom) |  CGCNN |  0.027 |  0.988
MEGNet |  0.032 |  0.982
CTGNN |  0.013 |  0.996
Pero($E_{g}$) (eV) |  CGCNN |  0.285 |  0.855
MEGNet |  0.296 |  0.845
CTGNN |  0.156 |  0.960
Jarvist($E_{g}$) (eV) |  CGCNN |  0.531 |  0.914
MEGNet |  0.493 |  0.908
CTGNN |  0.469 |  0.910

As listed in Table 1, our proposed CTGNN model demonstrates superior
performance on the perovskites dataset on formation energy and bandgap
prediction. The plots of the target and prediction distribution are also shown
in Fig 2. Specifically, CTGNN achieves the lowest MAE on the formation energy
prediction on the Pero dataset, compared to CGCNN and MEGNet models. With the
MAE of 0.013 eV/atom, CTGNN is 51.85% and 59.38% better than CGCNN and MEGNet,
whose MAE is 0.027 and 0.032 eV/atom respectively. The R2 is also the highest,
with 0.8% and 1.4% improvements. When it comes to the bandgap prediction on
the Pero dataset, CTGNN also surpasses CGCNN and MEGNet models with a lower
MAE and higher R2. To be more detailed, the MAE is 0.156 eV, which is 45.26%
and 47.30% better. The Jarvist dataset exhibits the same trend, with the CTGNN
model enjoying the lowest MAE of 0.469 eV, which is 11.67% and 4.87% better, a
significant improvement. These results underline the effectiveness of CTGNN in
handling complex material datasets, especially in the perovskites datasets
over traditional methods like CGCNN and MEGNet, highlighting its potential in
the perovskites era.

Figure 2: Plots of predicted formation energy and bandgap versus target for
CGCNN, MEGNet, and CTGNN models, respectively. the upper and right part are
the target and prediction data distribution.

##  IV Ablation Study

To understand the contribution of different components in the CTGNN model, we
conducted an ablation study on the Pero dataset for bandgap prediction. The
study involved removing key components from the model and evaluating the
resulting performance. The results are summarized in Table 2.

Table 2: Ablation study on Pero dataset for bandgap prediction. Model |  MAE (eV) |  R2
---|---|---
CTGNN (full model) |  0.156 |  0.960
Without Angular Encoding |  0.188 |  0.946
Without inter-atomic Transformer |  0.190 |  0.945
Without Transformer |  0.285 |  0.855

As shown in Table 2, removing the angular encoding and the neighbor
Transformer from the CTGNN model leads to a decrease in performance.
Specifically, without the angular encoding, the MAE increases to 0.188 eV and
the R2 decreases to 0.946. Similarly, without the inter-atomic Transformer,
the MAE increases to 0.190 eV and the R2 decreases to 0.945. When the dual-
Transformer is totally removed The performance drop to 0.285 from 0.190.

##  V Conclusion

CTGNN model represents a significant progress in the field of material
computing, particularly in perovskite materials. By innovatively combining the
advantages of Transformer model and graph neural networks, CTGNN can capture
both the local and global interactions in materials efficiently. The addition
of angular kernels allows for a more comprehensive representation of atomic
structures, surpassing the traditional models which only focus on distances.
Our results demonstrate that CTGNN outperforms existing models in predicting
key material properties such as formation energy and bandgap, which is
confirmed by the benchmark tests on multiple datasets. CTGNN not only enhance
the ability to predict material properties with greater accuracy, but also
provide a solid foundation for discovering new materials.

## Acknowledgements

This work is supported by the National Key R&D Program of China
(2023YFA1608501), and Natural Science Foundation of Shandong Province under
grants no. ZR2021MA041. Mr. L. Jin and Z. Du also want to acknowledge the
support of FDUROP (Fudan’s Undergraduate Research Opportunities Program)
(24052, 23908).

### Competing Interests statement

The authors declare no competing interests.

### Author Contributions

## References

  * (1) Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun.  Graph neural networks: A review of methods and applications.  AI open, 1:57–81, 2020.
  * (2) John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al.  Highly accurate protein structure prediction with alphafold.  Nature, 596(7873):583–589, 2021.
  * (3) James Kirkpatrick, Brendan McMorrow, David HP Turban, Alexander L Gaunt, James S Spencer, Alexander GDG Matthews, Annette Obika, Louis Thiry, Meire Fortunato, David Pfau, et al.  Pushing the frontiers of density functionals by solving the fractional electron problem.  Science, 374(6573):1385–1389, 2021.
  * (4) Alex Davies, Petar Veličković, Lars Buesing, Sam Blackwell, Daniel Zheng, Nenad Tomašev, Richard Tanburn, Peter Battaglia, Charles Blundell, András Juhász, et al.  Advancing mathematics by guiding human intuition with ai.  Nature, 600(7887):70–74, 2021.
  * (5) Tian Xie and Jeffrey C Grossman.  Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties.  Physical review letters, 120(14):145301, 2018.
  * (6) Tian Xie, Xiang Fu, Octavian-Eugen Ganea, Regina Barzilay, and Tommi Jaakkola.  Crystal diffusion variational autoencoder for periodic material generation.  arXiv preprint arXiv:2110.06197, 2021.
  * (7) Ya Zhuo, Aria Mansouri Tehrani, and Jakoah Brgoch.  Predicting the band gaps of inorganic solids by machine learning.  The Journal of Physical Chemistry Letters, 9(7):1668–1673, 2018\.  PMID: 29532658.
  * (8) Keith T. Butler, Daniel W. Davies, Hugh Cartwright, Olexandr Isayev, and Aron Walsh.  Machine learning for molecular and materials science.  Nature, 559(7715):547–555, 2018.  Published 2018/07/01.
  * (9) Yann LeCun, Yoshua Bengio, and Geoffrey Hinton.  Deep learning.  nature, 521(7553):436–444, 2015.
  * (10) Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip.  A comprehensive survey on graph neural networks.  IEEE transactions on neural networks and learning systems, 32(1):4–24, 2020.
  * (11) Chi Chen, Weike Ye, Yunxing Zuo, Chen Zheng, and Shyue Ping Ong.  Graph networks as a universal machine learning framework for molecules and crystals.  Chemistry of Materials, 31(9):3564–3572, 2019.
  * (12) Kamal Choudhary and Brian DeCost.  Atomistic line graph neural network for improved materials property predictions.  npj Computational Materials, 7(1):185, 2021.
  * (13) Cheol Woo Park and Chris Wolverton.  Developing an improved crystal graph convolutional neural network framework for accelerated materials discovery.  Physical Review Materials, 4(6):063801, 2020.
  * (14) Zhuoran Qiao, Matthew Welborn, Animashree Anandkumar, Frederick R Manby, and Thomas F Miller.  Orbnet: Deep learning for quantum chemistry using symmetry-adapted atomic-orbital features.  The Journal of chemical physics, 153(12), 2020.
  * (15) Johannes Gasteiger, Janek Groß, and Stephan Günnemann.  Directional message passing for molecular graphs.  arXiv preprint arXiv:2003.03123, 2020.
  * (16) Johannes Gasteiger, Shankari Giri, Johannes T Margraf, and Stephan Günnemann.  Fast and uncertainty-aware directional message passing for non-equilibrium molecules.  arXiv preprint arXiv:2011.14115, 2020.
  * (17) Zeren Shui and George Karypis.  Heterogeneous molecular graph neural networks for predicting molecule properties.  In 2020 IEEE International Conference on Data Mining (ICDM), pages 492–500. IEEE, 2020.
  * (18) Kristof T Schütt, Farhad Arbabzadah, Stefan Chmiela, Klaus R Müller, and Alexandre Tkatchenko.  Quantum-chemical insights from deep tensor neural networks.  Nature communications, 8(1):13890, 2017.
  * (19) Brandon Anderson, Truong Son Hy, and Risi Kondor.  Cormorant: Covariant molecular neural networks.  Advances in neural information processing systems, 32, 2019.
  * (20) Shuo Zhang, Yang Liu, and Lei Xie.  Molecular mechanics-driven graph neural network with multiplex graph for molecular structures.  arXiv preprint arXiv:2011.07457, 2020.
  * (21) KT Schuett, Pan Kessel, Michael Gastegger, KA Nicoli, Alexandre Tkatchenko, and K-R Muller.  Schnetpack: A deep learning toolbox for atomistic systems.  Journal of chemical theory and computation, 15(1):448–455, 2018\.
  * (22) Dipendra Jha, Logan Ward, Arindam Paul, Wei-keng Liao, Alok Choudhary, Chris Wolverton, and Ankit Agrawal.  Elemnet: Deep learning the chemistry of materials from only elemental composition.  Scientific reports, 8(1):17593, 2018.
  * (23) Julia Westermayr, Michael Gastegger, and Philipp Marquetand.  Combining schnet and sharc: The schnarc machine learning approach for excited-state dynamics.  The journal of physical chemistry letters, 11(10):3828–3834, 2020\.
  * (24) Mingjian Wen, Samuel M Blau, Evan Walter Clark Spotte-Smith, Shyam Dwaraknath, and Kristin A Persson.  Bondnet: a graph neural network for the prediction of bond dissociation energies for charged molecules.  Chemical science, 12(5):1858–1868, 2021.
  * (25) Olexandr Isayev, Corey Oses, Cormac Toher, Eric Gossett, Stefano Curtarolo, and Alexander Tropsha.  Universal fragment descriptors for predicting properties of inorganic crystals.  Nature communications, 8(1):15679, 2017.
  * (26) Chi Chen, Yunxing Zuo, Weike Ye, Xiangguo Li, and Shyue Ping Ong.  Learning properties of ordered and disordered materials from multi-fidelity data.  Nature Computational Science, 1(1):46–53, 2021.
  * (27) Shuaihua Lu, Qionghua Zhou, Yixin Ouyang, Yilv Guo, Qiang Li, and Jinlan Wang.  Accelerated discovery of stable lead-free hybrid organic-inorganic perovskites via machine learning.  Nature communications, 9(1):3405, 2018.
  * (28) Di Chen, Yiwei Bai, Sebastian Ament, Wenting Zhao, Dan Guevarra, Lan Zhou, Bart Selman, R Bruce van Dover, John M Gregoire, and Carla P Gomes.  Automating crystal-structure phase mapping by combining deep learning with constraint reasoning.  Nature Machine Intelligence, 3(9):812–822, 2021.
  * (29) Xian Yeow Lee, Joshua R Waite, Chih-Hsuan Yang, Balaji Sesha Sarath Pokuri, Ameya Joshi, Aditya Balu, Chinmay Hegde, Baskar Ganapathysubramanian, and Soumik Sarkar.  Fast inverse design of microstructures via generative invariance networks.  Nature Computational Science, 1(3):229–238, 2021.
  * (30) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * (31) Keqiang Yan, Yi Liu, Yuchao Lin, and Shuiwang Ji.  Periodic graph transformers for crystal material property prediction.  Advances in Neural Information Processing Systems, 35:15066–15080, 2022.
  * (32) Erxue Min, Runfa Chen, Yatao Bian, Tingyang Xu, Kangfei Zhao, Wenbing Huang, Peilin Zhao, Junzhou Huang, Sophia Ananiadou, and Yu Rong.  Transformer for graphs: An overview from architecture perspective.  arXiv preprint arXiv:2202.08455, 2022.
  * (33) Jiao Huang, Qianli Xing, and Jinglong Ji.  Ada-gnn: Atom-distance-angle graph neural network for crystal material property prediction.  https://synthical.com/article/2e48f156-8b9d-4f18-a52a-3e730cdd1ede, 0 2024\.
  * (34) Taohong Zhang, Xuxu Guo, Hen Chen, Suli Fan, Qianqian Li, Saian Chen, Xueqiang Guo, and Han Zheng.  Tg-gnn: Transformer based geometric enhancement graph neural network for molecular property prediction, 2022.
  * (35) Steph-Yves M. Louis, Yong Zhao, Alireza Nasiri, Xiran Wong, Yuqi Song, Fei Liu, and Jianjun Hu.  Global attention based graph convolutional neural networks for improved materials property prediction.  ArXiv, abs/2003.13379, 2020.
  * (36) Kamal Choudhary.  Jarvis-dft, 2014.
  * (37) Chiho Kim, Tran Doan Huan, Sridevi Krishnan, and Rampi Ramprasad.  A hybrid organic-inorganic perovskite dataset.  Scientific Data, 4(1):1–11, 2017.
  * (38) Takahito Nakajima and Keisuke Sawada.  Discovery of pb-free perovskite solar cells via high-throughput simulation on the k computer.  The journal of physical chemistry letters, 8(19):4826–4831, 2017\.