# Open-vocabulary Auditory Neural Decoding Using fMRI-prompted LLM

Xiaoyu Chen  0000-0001-8945-3150 Laboratory of Brain Atlas and Brain-Inspired
Intelligence, State Key Laboratory of Multimodal Artificial Intelligence
Systems, CASIASchool of Artificial Intelligence, University of Chinese Academy
of ScienceBeijingChina<EMAIL_ADDRESS>,  Changde Du
0000-0002-0084-433X Laboratory of Brain Atlas and Brain-Inspired Intelligence,
State Key Laboratory of Multimodal Artificial Intelligence Systems,
CASIABeijingChina<EMAIL_ADDRESS>,  Liu Che  Laboratory of Brain Atlas
and Brain-Inspired Intelligence, State Key Laboratory of Multimodal Artificial
Intelligence Systems, CASIAUniversity of Chinese Academy of
ScienceBeijingChina<EMAIL_ADDRESS>,  Yizhe Wang  CASIABeijingChina
<EMAIL_ADDRESS>and  Huiguang He  0000-0002-0684-1711 Laboratory of Brain
Atlas and Brain-Inspired Intelligence, State Key Laboratory of Multimodal
Artificial Intelligence Systems, CASIASchool of Artificial Intelligence,
University of Chinese Academy of ScienceBeijingChina<EMAIL_ADDRESS>
###### Abstract.

Decoding language information from brain signals represents a vital research
area within brain-computer interfaces, particularly in the context of
deciphering the semantic information from the fMRI signal. However, many
existing efforts concentrate on decoding small vocabulary sets, leaving space
for the exploration of open vocabulary continuous text decoding. In this
paper, we introduce a novel method, the Brain Prompt GPT (BP-GPT). By using
the brain representation that is extracted from the fMRI as a prompt, our
method can utilize GPT-2 to decode fMRI signals into stimulus text. Further,
we introduce a text-to-text baseline and align the fMRI prompt to the text
prompt. By introducing the text-to-text baseline, our BP-GPT can extract a
more robust brain prompt and promote the decoding of pre-trained LLM. We
evaluate our BP-GPT on the open-source auditory semantic decoding dataset and
achieve a significant improvement up to $4.61\%$ on METEOR and $2.43\%$ on
BERTScore across all the subjects compared to the state-of-the-art method. The
experimental results demonstrate that using brain representation as a prompt
to further drive LLM for auditory neural decoding is feasible and effective.

Do, Not, Us, This, Code, Put, the, Correct, Terms, for, Your, Paper

††ccs: Do Not Use This Code Generate the Correct Terms for Your Paper††ccs: Do
Not Use This Code Generate the Correct Terms for Your Paper††ccs: Do Not Use
This Code Generate the Correct Terms for Your Paper††ccs: Do Not Use This Code
Generate the Correct Terms for Your Paper

##  1\. Introduction

“The limits of my language mean the limits of my world” - Ludwig Wittgenstein.
Wittgenstein’s statement refers to the standpoint that a person’s entire
understanding of the world is reflected in the things they can describe in
language. So it is important for human-centric artificial intelligence, for
example, the brain-computer interface, to understand the language information
from the human brain.

Figure 1. We focus on decoding semantic information from fMRI in the auditory
neural decoding scenario and use fMRI signals as prompts to guide a pre-
trained GPT-2 to achieve decoding.

In recent years, due to the rapid development of deep learning and its
widespread application in brain-computer interfaces, especially in neural
encoding and decoding (Scotti et al., 2024; Lin et al., 2022; Du et al., 2023;
Li et al., 2022; Wang et al., 2022), significant progress has been made in the
research field of extracting language information from brain signals. For
example, reconstructing audio of auditory stimuli from brain signals (Yang et
al., 2015; Pasley et al., 2012; Santoro et al., 2017; Défossez et al., 2023),
or reconstructing corresponding text (Affolter et al., 2020; Défossez et al.,
2023; Pereira et al., 2018; Xi et al., 2023; Tang et al., 2023). In this
article, we focus on reconstructing language information by decoding the
original text (see Figure 1 for an illustration). Specifically, we focus on
text decoding in auditory neural decoding scenarios. For the types of brain-
computer interfaces, we focus on decoding text from functional magnetic
resonance imaging (fMRI), a widely used non-invasive brain-computer interface
in both research and practical applications.

Figure 2. The training stages of our method. The upper part: we use the BERT
and GPT-2 for the encoder and decoder of our text-to-text baseline. In this
baseline, the BERT representation will be mapped into a text prompt which is
used for reconstructing the original text using GPT-2. The lower part: we use
a transformer fMRI encoder to extract the fMRI prompt and add a contrastive
loss to align the fMRI prompt to the text prompt. Then, the GPT-2 will decode
the text according to the fMRI prompt.

To accomplish this goal, two primary challenges must be addressed. Firstly,
the temporal resolution of fMRI signals presents a significant obstacle.
Despite its commendable spatial resolution and non-invasive characteristics,
fMRI signals exhibit significantly low temporal resolution. For instance, in
the context of commonly spoken English, where the average speaking speed
exceeds 2 words per second (Tang et al., 2023), the BOLD response to neural
activity rises and falls over approximately 10 seconds which is extremely
slower than the speech stimuli (Logothetis, 2003). Secondly, another essential
challenge is the significant difference between fMRI modality and text
modality. In fact, in auditory information decoding scenarios, the text does
not present as the stimulus signal received by the subject. Rather, it
contains the semantics of the stimulus signal. Therefore, the quality of modal
alignment significantly influences the effectiveness of the decoding model.

The low temporal resolution in fMRI necessitates our model to decode multiple
words from a single fMRI signal during the decoding process. To tackle this
ill-posed inverse problem, we propose a solution by incorporating a language
prior through a pre-trained Large Language Model (LLM) - Generative Pre-
trained Transformer 2 (GPT-2) (Radford et al., 2019). As illustrated in the
bottom part of Figure 2, we establish a mapping from fMRI to prompts and train
GPT-2 to employ autoregressive methods, generating corresponding text based on
the provided fMRI prompts. By applying the cross-entropy loss to the output
logit of GPT-2, the fMRI encoder can learn the suitable prompts for the target
text.

To mitigate the substantial modal disparity between the fMRI signal and the
text data, we introduced another text-to-text baseline during the training
stage. As depicted in the upper section of Figure 2, we utilized Bidirectional
Encoder Representations from Transformers (BERT) (Devlin et al., 2018) as the
text encoder to extract the text prompt. Subsequently, we reconstructed the
text using GPT-2 in a manner similar to the brain-to-text. Since the input and
output of the text-to-text baseline are identical, eliminating modal
differences, we designate the text prompt as the optimal prompt for the
ground-truth text. Consequently, we incorporated a contrastive loss to align
the fMRI prompt with the text prompt, enhancing decoding performance.

In summary, our main contributions are as follows:

  * •

We propose Brain Prompt-GPT (BP-GPT), which is a novel structure that can use
the fMRI prompt to decode the text of speech stimuli in an end-to-end
structure.

  * •

We introduced a language prior by a pre-trained LLM (GPT-2) to compensate for
the low temporal resolution of fMRI; through contrastive learning, we
encourage fMRI prompts to align with text prompts, thereby reducing the impact
of modal differences on decoding performance.

  * •

We evaluated our BP-GPT model on an open-source auditory semantic decoding
dataset and achieved a significant improvement of up to $4.61\%$ on METEOR and
$2.43\%$ on BERTScore across all subjects compared to the state-of-the-art
method. These results demonstrate the feasibility and advantages of our
approach.

##  2\. Related Work

###  2.1. Large language models

Since the introduction of the transformer architecture (Vaswani et al., 2017),
numerous transformer-based Large Language Models (LLMs) have emerged. While
all these models utilize stacked attention layers to construct their networks,
each exhibits unique characteristics. Broadly speaking, existing LLMs can be
categorized into three groups: encoder-only, decoder-only, and encoder-
decoder.

The encoder-only LLMs are mostly used to extract features of input text and
are used for different downstream tasks. The most representative one is the
BERT (Devlin et al., 2018) model. BERT adopts a bidirectional architecture and
trains using a masked language model and next-sense prediction task, allowing
it to extract representations of text based on the previous and following
text. A popular optimized version of BERT is RoBERTa (Liu et al., 2019), which
builds upon BERT’s architecture and pre-training objectives, refining the
training process to achieve improved performance. It removes the next sentence
prediction objective and trains on more data and for longer epochs, leading to
better representations.

Decoder-only LLMs stem from the decoder component of the transformer
architecture. Their key feature lies in masking future positions to ensure
that predictions of the current token are based solely on preceding tokens.
The most prominent example of decoder-only LLMs is the GPT (Generative Pre-
trained Transformer) series (Radford et al., 2018, 2019; Brown et al., 2020;
Achiam et al., 2023), which introduces a large-scale autoregressive language
model based on the Transformer architecture. These models have demonstrated
the effectiveness of unsupervised pre-training followed by fine-tuning across
various natural language processing tasks. Beyond their technical aspects,
they have significantly advanced natural language understanding and generation
tasks, including translation, question answering, code generation, and even
creative writing. Notably, with the emergence of ChatGPT, the GPT series has
become the most popular choice among LLMs.

In addition to the aforementioned architectures, there are models that utilize
the encoder-decoder architecture to leverage the strengths of both components.
Examples include BART (Lewis et al., 2019) and T5 (Raffel et al., 2020). While
both models are based on the encoder-decoder architecture, each adopts a
unique approach and possesses distinct capabilities for natural language
processing and generation tasks. T5 emphasizes a text-to-text framework for
unified processing of various tasks, whereas BART focuses on bidirectional and
auto-regressive training for sequence-to-sequence tasks such as summarization
and translation.

###  2.2. Decoding the Brain Signals into Text

In the early stages of text decoding, model outputs often comprised a single
vocabulary and only supported a limited vocabulary set. For instance, in the
Brain2word (Affolter et al., 2020), researchers employed a classification
model to decode participants’ brain signals when they read words. In the study
by Defossez et al. (Défossez et al., 2023), contrastive learning was applied
to decode words or phrases from auditory brain signals. Additionally, Pereira
et al. (Pereira et al., 2018) and Sun et al. (Sun et al., 2019) explored the
decoding of sentences from brain signals in their respective works.

In recent years, due to the success of LLMs, some work has begun to attempt to
use LLMs to decode complete texts from various types of brain data. For
example, Wang et al. (Wang and Ji, 2022) proposed using BART (Lewis et al.,
2019) to decode text from the EEG and eye tracking signals of the subjects
during reading, achieving the decoding of open vocabularies. In DeWave (Duan
et al., 2023), Duan et al. proposed using a quantized variant encoder and BART
to improve the decoding of EEG2text, achieving decoding without external event
markers like handling or eye tracking. Unlike decoding using EEG, in UniCoRN
(Xi et al., 2023), Xi et al. also used BART to decode fMRI signals into text.
Tang et al (Tang et al., 2023). used linear regression and GPT, based on the
neural encoding architecture, to complete text decoding using similarity
measurement. Our work, along with UniCoRN and Tang et al.’s most relevant, is
based on fMRI decoding. Different from UniCoRN which treats fMRI as a foreign
language and uses machine translation structure for text decoding, we adopted
a prompt approach. Compared to Tang et al., which adopts a neural encoding
architecture, our method adopts a more direct neural decoding architecture,
which is an end-to-end approach.

Figure 3. An illustration of the inference stage is provided here. During this
stage, the fMRI prompt is considered as the preceding text for the target text
generation. Subsequently, GPT-2 generates the text in an autoregressive
manner, relying on both the fMRI prompt and the generated text. For deciding
the length of decoding text, we compared two strategies in this work. The
first one is to use the word rate model to predict the length of text; The
second one is to use special tokens and fine-tune the GPT-2, the decoding
process will end when GPT-2 generates enough special tokens ($ in our
implementation).

##  3\. Method

###  3.1. A Text to Text Baseline

The fundamental concept of our method draws inspiration from the encoding and
decoding processes applied to text and its corresponding modalities. In the
field of Natural Language Processing (NLP) or Computer Vision (CV), various
tasks demand the encoding and decoding of text, such as text-to-image
generation (Ramesh et al., 2021) image captioning (Zhou et al., 2020; Mokady
et al., 2021). In this part, we align both the encoding input and decoding
target with the text corresponding to auditory stimuli. Due to the remarkable
progress made by LLMs recently (Devlin et al., 2018; Radford et al., 2019;
Brown et al., 2020) and its continued evolution at an astonishing speed
(Achiam et al., 2023), we have established the encoder and decoder of our
text-to-text model entirely based on the pre-trained LLMs, minimizing the
volume of parameters that need to be trained. This design facilitates the
seamless integration of new LLMs, enabling easy upgrades for our method.

We chose BERT as the text encoder and GPT-2 as the text decoder in this work
and drew inspiration from the recent image caption works (Zhou et al., 2020;
Mokady et al., 2021) which use encoded representation as the prompt of target
text in the decoding. Specifically, we first encode the text into the
representation space using the last hidden state of BERT. Then a mapping
network is applied to map the BERT representation into the prompt:

(1) $\mathit{P}_{i}^{T}=\mathbf{M}_{\theta}(\mathbf{BERT}(x_{i})),$

where the $\mathbf{M}(\cdot)$ denote the mapping network, and
$\mathit{P}_{i}^{T}=(p^{T}_{1},\ldots,p^{T}_{k})$ is the extracted prompt
sequence.

To reconstruct the text, we feed the prompt into GPT-2 to generate the
original text. The text was generated using the next-token-prediction paradigm
and the prompt is considered as the preceding text of the target text. In the
training, the parameters of BERT are fixed. For the GPT-2, fine-tuning is not
a mandatory option here due to the presence of the mapping network. So only
the parameters of the mapping network have to be optimized. We use a cross-
entropy loss between the output of GPT-2 and the tokens of the target text for
the optimization of the mapping network:

(2) $\displaystyle\mathcal{L}_{text}$ $\displaystyle=-\sum_{i=1}^{N}\log
p_{\theta}(\mathit{W}|\mathit{P}_{i}^{T})$ (3)
$\displaystyle=-\sum_{i=1}^{N}\sum_{j=1}^{\mathcal{L}}\log
p_{\theta}(w_{j}|p^{T}_{1},\ldots,p^{T}_{k},w_{1},\ldots,w_{j-1}),$

where the $\mathit{W}=(w_{1},\ldots,w_{j})$ is the tokens of the text, and $k$
is the length of prompt.

###  3.2. fMRI to Text Decoding

There are two main challenges in decoding text from the fMRI signals: (1) the
low temporal resolution of fMRI and (2) the significant modal difference
between fMRI and text. In this part, we will introduce our method which could
address the two challenges above. Our method consists of two essential
components. The first is the fMRI-prompted text decoding method, which
introduces the prompt paradigm into fMRI decoding and addresses the problem of
low temporal resolution of fMRI. The second is aligning the fMRI prompt to the
text prompt and therefore reduce the impact of modal differences and address
the last challenge.

####  3.2.1. fMRI-prompted text decoding.

Due to the low temporal resolution of fMRI, we need to decode multiple words
from each fMRI sample point. We adopted a similar structure as the text-to-
text baseline to solve this problem. Specifically, we use a fMRI encoder model
to encode the fMRI into the representation space and use this representation
as the prompt of the text generation process of GPT-2:

(4) $\mathit{P}_{i}^{B}=\mathbf{E}_{\eta}(x_{i}^{B}),$

where the $\mathbf{E}_{\eta}$ is the fMRI encoder, and $x_{i}^{B}$ denote the
fMRI. $\mathit{P}_{i}^{B}=(p^{B}_{1},\cdots,p^{B}_{k})$ denote the fMRI prompt
which is extracted by the fMRI encoder.

Since we did not introduce a pre-trained fMRI encoder here, the fMRI-to-text
part did not require a mapping network. Furthermore, the fMRI encoder can be
trained using the cross-entropy loss on the output of the GPT-2. Here, fine-
tuning the GPT-2 is not a mandatory option either. When we choose not to fine-
tune the GPT-2, the loss is completely dependent on the parameters of the fMRI
encoder $\eta$ and has the following form:

(5) $\displaystyle\mathcal{L}_{brain}$ $\displaystyle=-\sum_{i=1}^{N}\log
p_{\eta}(\mathit{W}|\mathit{P}_{i}^{B})$ (6)
$\displaystyle=-\sum_{i=1}^{N}\sum_{j=1}^{\mathcal{L}}\log
p_{\eta}(w_{j}|p^{B}_{1},\ldots,p^{B}_{k},w_{1},\ldots,w_{j-1}).$

However, fine-tuning the GPT-2 may bring performance improvements in
downstream tasks. Meanwhile, in the text decoding task of this work, the fine-
tuning of the GPT-2 can bring more specific benefits. We will introduce this
specific benefits in Section 3.4 and compare these two options in the later
experimental section.

####  3.2.2. Align with the Optimal Prompt

Given the considerable significant modal differences between fMRI and text,
extracting effective fMRI prompts through the fMRI encoder poses challenges.
Conversely, the text baseline inherently circumvents this modal difference. So
we argue that the prompt derived from the text baseline can be deemed as the
optimal prompt for the text to be generated. To utilize the optimal prompt, we
employ contrastive learning to align the fMRI prompt with the text prompt.

Specifically, we set the fMRI prompt and text prompt of the same text as the
positive pair and calculate the similarity between the positive pair using the
following formula:

(7)
$\mathit{S}_{p}=\exp(cos(\mathit{P}^{i}_{B}\cdot\mathit{P}^{i}_{T})/\tau),$

where the $\tau$ refers to the temperature hyperparameter.

For the negative pair, we use the fMRI prompt and text prompt from different
texts and the fMRI prompt of different texts. The similarity between the
negative pairs can be formulated as:

(8)
$\begin{split}\mathit{S}_{n}=&\exp(cos(\mathit{P}^{i}_{B}\cdot\mathit{P}^{j}_{B})/\tau)+\exp(cos(\mathit{P}^{i}_{B}\cdot\mathit{P}^{j}_{T})/\tau),i\neq
j.\end{split}$

Based on the definition above, the contrastive loss has the following form:

(9) $L_{\mathcal{C}}=-\mathbb{E}\left[\log\frac{S_{p}}{S_{n}}\right].$

###  3.3. Training

We will divide the training process into two stages. In the initial stage, the
text-to-text baseline is trained for text encoding and decoding. This process
enables the model to learn to extract the optimal prompt for the target text,
which is then utilized as the target of contrastive learning in the subsequent
stage. Then, we train our decoding model in the second stage using the
following formula:

(10) $\displaystyle L=L_{brain}+\alpha L_{C},$

where the $\alpha$ is a hyperparameter for the contrastive loss.

###  3.4. Inference

As is shown in Figure 3, in the inference stage, we extract the fMRI prompt
first and generate the text using GPT-2 depending on the fMRI prompt. The text
will be generated word by word in a next-token-prediction paradigm.

As our focus is on text decoding within an auditory neural decoding scenario,
the auditory stimuli received by the subjects exclusively consist of words
without punctuation. This will cause trouble for the model during the
inference phase, as we cannot determine the end of generation through the stop
token (usually, periods are used). Although punctuation can be manually added
during text annotation for audio stimuli, this introduces two concerns.
Firstly, the speaker might have delivered a spontaneous speech without
adhering to the ground-truth speech draft. Secondly, since the decoding often
uses the fMRI signals within a fixed-length window, and the end of this window
does not always match the end of the sentence, even if we add the punctuation
manually, they are very likely unable to indicate the end of the generation.

For the reasons outlined above, we chose not to include punctuation in the
model training process. To find the end of the generation in the inference
stage, we offer two strategies here, an illustration of these two strategies
is shown in Figure 3.

In the first one, we adopt an approach from recent work (Tang et al., 2023),
utilizing a word rate model to predict the number of words perceived by
participants. The text generation process will be stopped when the length of
the generated text meets the word count predicted by the word rate model.

The second strategy is to use the special token to segment text based on the
repetition time (TR) of fMRI. In this work, we add $ to the gound-truth text
during the training, and stop the generation when we meet enough $ in the
inference text. We also add an equal mark to point out the beginning of the
gourd-truth text.

###  3.5. Mapping Network and fMRI Encoder Model

The core of our model lies in mapping the original fMRI signal or
representation of the text encoder into the prompt. Given the complexities in
these transformations, especially for the fMRI-to-text part, we employed the
transformer structures (Vaswani et al., 2017) to accomplish the mapping of
original features to the prompt. The transformer’s inherent capability for
global attention enables the extraction of semantic relationships from the
input fMRI sequence, yielding a fMRI prompt better suited for the target text.

| BLEU-1$\uparrow$ | METEOR$\uparrow$ | BERTScore$\uparrow$
---|---|---|---
UTS01 | UTS02 | UTS03 | UTS01 | UTS02 | UTS03 | UTS01 | UTS02 | UTS03
T2T+WR | 0.1968 | 0.2085 | 0.1862 | 0.1414 | 0.1466 | 0.1266 | 0.8192 | 0.8205 | 0.8163
T2T+WR+fine-tune | 0.2296 | 0.2421 | 0.241 | 0.1692 | 0.1699 | 0.176 | 0.8242 | 0.8278 | 0.8259
T2T+Spe | 0.2189 | 0.2044 | 0.2187 | 0.2032 | 0.204 | 0.2082 | 0.8325 | 0.8328 | 0.8343
T2T+Spe+fine-tune | 0.2621 | 0.2613 | 0.2554 | 0.2627 | 0.2597 | 0.2549 | 0.8432 | 0.8451 | 0.8417

Table 1. Text-to-text performance. ’WR’ refers to use word rate model in the
inference, and ’Spe’ refers to the special tokens. In the results without
annotation ’fine-tune’, the parameter of GPT-2 is fixed.

(a) UTS01

(b) UTS02

(c) UTS03

Figure 4. The cortical flatmaps for the auditory cortex (in red color) of the
different subjects we used.

##  4\. Experiment

###  4.1. Dataset

We evaluate our method on an fMRI dataset obtained during a passive natural
language listening task (LeBel et al., 2023) along with its extended data
(Tang et al., 2023). This dataset comprises fMRI data from 8 subjects recorded
while they passively listened to naturally spoken English stories. The stories
were sourced from The Month and New York Times Modern Love podcasts.
Specifically, the first 3 subjects (UTS01 to UTS03) in the dataset had access
to stories from both The Month and New York Times Modern Love, expanding the
total number of stories to 84 for these subjects. For the remaining 5 subjects
(UTS04 to UTS08), all 27 stories are from The Month. To maintain consistency
with existing works (LeBel et al., 2023; Tang et al., 2023), we selected the
story ”Where There’s Smoke”, shared by all subjects, as the test set.

In the vanilla section of the dataset, the stories were divided into 5
sections, each containing 5 stories and an additional test story (”Where
There’s Smoke”). This configuration resulted in a total duration of
stimulating audio exceeding 6 hours for all subjects. In the extended section
of the dataset, the first three subjects underwent an additional 10 sessions,
increasing the overall dataset duration to 81 hours across all subjects. All
stories feature a single speaker narrating an autobiographical tale without a
prepared script. The texts were manually transcribed by one listener and
automatically aligned to the audio using the Penn Phonetics Lab Forced Aligner
(P2FA) (Yuan et al., 2008). Consequently, the transcriptions lack punctuation
marks, and some words may be repeated as the speaker is thinking and
organizing their language. Additionally, certain sounds, such as ”cough,”
”laugh,” ”lip smack,” ”misc noise,” and ”silence,” were annotated.

###  4.2. Implementing Details

In our work, we choose the first three subjects that have the extended data
for all of our experiments. For the region of interest (ROI), we use the voxel
in the auditory cortex for our experiment (see Figure 4 for the cortical flat
maps). The temperature of contrastive loss is set to $\tau=0.1$, and the
weight of contrastive loss is set to $\alpha=1$. We split the fMRI sequence
and corresponding text into multiple 20-second windows, with zero overlaps.
For the prompt length, we use $k=30$.

For the mapping network, the BERT representation will be first mapped to a
512-dimensional vector before passing forward to the transformer. We use an
8-layer transformer here, with 8 attention heads in each layer. The fMRI
encoder has the same architecture as the mapping network, except for a linear
layer for the input.

All the codes are implemented using PyTorch (Paszke et al., 2019), and
training on an Nvidia A-100 GPU with AdamW optimizer (Loshchilov and Hutter,
2017). The batch size is set to 32.

###  4.3. Baseline and Evaluation Metrics

We compare our method to Tang et al. (Tang et al., 2023), which is the state-
of-the-art method in the dataset we used. In their method, they use a neuron
encoding structure, which uses the GPT to generate proposal words and encode
the proposal words to fMRI to find the most matching word. For a fair
comparison, we use the same story (”Where There’s Smoke”) in the dataset as
the test set and divide the entire story into 20-second windows, calculate
evaluation metrics within each window, and take the average of all windows as
the metric score for the entire test set.

Similar to Tang et al. (Tang et al., 2023), we use identical language
similarity metrics to evaluate our method in several aspects. BLEU (Papineni
et al., 2002) indicates the number of individual translated segments that
appear in the ground-truth text. METEOR (Denkowski and Lavie, 2014) computes
the harmonic mean of unigram precision and recall. And BERTScore (Zhang et
al., 2019) computes a similarity score for each token in the candidate
sentence with each token in the reference sentence using contextual
embeddings.

| BLEU-1$\uparrow$ | METEOR$\uparrow$ | BERTScore$\uparrow$
---|---|---|---
UTS01 | UTS02 | UTS03 | UTS01 | UTS02 | UTS03 | UTS01 | UTS02 | UTS03
Tang et al. | 0.2331 | 0.2426 | 0.2470 | 0.1621 | 0.1677 | 0.1703 | 0.8077 | 0.8104 | 0.8116
BP-GPT | 0.2159 | 0.2111 | 0.2113 | 0.2082 | 0.1976 | 0.2034 | 0.832 | 0.8322 | 0.8332

Table 2. Compare our method with the existing work. The BP-GPT refers to using
the special tokens for the inference and fine-tuning the GPT-2.

###  4.4. Evaluation the Text-to-text Baseline

Since we treat the text prompt as the optimal prompt for decoding in our
method, we first evaluate the performance of the text-to-text baseline in our
work. We consider 4 settings in this part. For the first two settings, we use
the word rate model in the inference and fine-tune or fix the parameters of
GPT-2 in the training. For the last two settings, we use the special tokens
and also compare two options for fine-tuning or fixing the parameters of
GPT-2.

The experiment results are listed in Table 1. From the experimental results,
we can find that our text-to-text baseline can effectively encode and decode
text under various experimental settings. This result supports our further
application of this prompt paradigm in fMRI-to-text decoding. Also, we find
that fine-tuning the GPT-2 in the training can always bring improvements in
performance, regardless of the inference strategy we used. For the inference
strategies, rather than using a word rate model to infer the text length, we
find that adding special tokens in the ground-truth text in the training stage
can effectively improve the decoding performance. Moreover, we find that fine-
tuning the GPT-2 for the special tokens can bring a significant improvement in
the results. Specifically, this setting can bring up to $5.69\%$ on BLEU-1,
$5.95\%$ on METEOR, and $1.23\%$ on BERTScore among all the subjects. Based on
these experiment results, we choose the setting that uses special tokens and
fine-tune the GPT-2 as the text-to-text baseline for all the following
experiments which use the special tokens as inference strategies. Also, the
text-to-text baseline for the word rate model strategies in the following
experiments is also fine-tuned.

###  4.5. Evaluation of fMRI to Text Decoding

In this part, we evaluate the decoding performance of our model. To align with
existing work (Tang et al., 2023), we report the performance of the first
three subjects: UTS01, UTS02, and UTS03, who have experienced all 84 stories
in the experiment. For the setting of our method, we add special tokens in the
ground-truth text for the training. The GPT-2 is fine-tuned in the training,
and the inference is stopped when the GPT-2 generates enough $. We refer to
this setting as the BP-GPT in all the experiments.

As is shown in Table 2, our method achieves comparable or even better
performance. Specifically, on the METEOR, our method can achieve an
improvement ranging from $2.99\%$ to $4.61\%$ on all the subjects. For the
BERTScore, we also achieve an improvement ranging from $1.87\%$ to $2.43\%$ on
all the subjects.

###  4.6. Ablation Study

In this part, we make ablation studies that evaluate the contributions of
contrastive learning and different inference strategies for the performance.
We report these experiment results in Table 3 and Table 4. For the inference
strategies, we mark ’WR’ for the results that use the word rate model to
distinguish it from the results that use the special tokens in the inference.

| BLEU-1$\uparrow$ | METEOR$\uparrow$ | BERTScore$\uparrow$
---|---|---|---
UTS01 | UTS02 | UTS03 | UTS01 | UTS02 | UTS03 | UTS01 | UTS02 | UTS03
BP-GPT-wo-contras (WR) | 0.1855 | 0.1851 | 0.1979 | 0.1372 | 0.1358 | 0.1442 | 0.8134 | 0.8131 | 0.8172
BP-GPT (WR) | 0.2052 | 0.1944 | 0.2017 | 0.1451 | 0.1476 | 0.1497 | 0.8192 | 0.8164 | 0.8198
BP-GPT-wo-contras | 0.2027 | 0.2041 | 0.2043 | 0.1943 | 0.1958 | 0.1946 | 0.829 | 0.8278 | 0.8281
BP-GPT | 0.2159 | 0.2111 | 0.2113 | 0.2082 | 0.1976 | 0.2034 | 0.832 | 0.8322 | 0.8332

Table 3. Abalation study: contrastive learning. Experiments that use the word
rate model in the inference are marked with ’WR’.

| BLEU-1$\uparrow$ | METEOR$\uparrow$ | BERTScore$\uparrow$
---|---|---|---
UTS01 | UTS02 | UTS03 | UTS01 | UTS02 | UTS03 | UTS01 | UTS02 | UTS03
BP-GPT-wo-fine-tune (WR) | 0.198 | 0.1936 | 0.1997 | 0.1343 | 0.1409 | 0.139 | 0.8157 | 0.8145 | 0.8129
BP-GPT (WR) | 0.2052 | 0.1944 | 0.2017 | 0.1451 | 0.1476 | 0.1497 | 0.8192 | 0.8164 | 0.8198
BP-GPT-wo-fine-tune | 0.2083 | 0.1949 | 0.2065 | 0.206 | 0.1924 | 0.2022 | 0.8318 | 0.8295 | 0.8298
BP-GPT | 0.2159 | 0.2111 | 0.2113 | 0.2082 | 0.1976 | 0.2034 | 0.832 | 0.8322 | 0.8332

Table 4. Abalation study: inference strategy. Experiments that use the word
rate model in the inference are marked with ’WR’.

(a) BLEU-1 and METEOR

(b) BERTScore

Figure 5. The performance under different prompt length.

####  4.6.1. Contrastive Learning.

To demonstrate the effectiveness of contrastive learning, we compare the
performance of our method with or without contrastive learning. We include the
experiment results on both inference strategies since the challenge of the
significant modal difference between the fMRI and text exists both in these
settings. We want to explore through these experiments whether aligning the
fMRI prompt with the text prompt can bring performance improvements in various
inference strategies. Here, different inference strategies will lead to
different text-to-text baselines. As the target of the conservative learning,
the text prompt is extracted using the text-to-text baseline that has the same
inference strategy as the fMRI-to-text model. We would like to refer the
Section 4.4 for more details.

We report the experiment results in Table 3. By comparing Table 3 with Table
1, we can find that there is a big gap between the performance of fMRI-to-text
decoding and the performance of the text-to-text decoding, indicating the
modal differences between text and fMRI impact the decoding performance
seriously. Also, through the results, we find that aligning the fMRI prompt
with the text prompt always brings performance improvements, no matter what
inference strategies have been chosen. Specifically, when using a word rate
model at the inference stage, aligning the fMRI prompt to the text prompt can
bring an improvement up to $1.97\%$ on BLEU-1, $1.18\%$ on METEOR, and
$0.58\%$ on BERTScore. While choosing the special tokens for inference,
contrastive learning can bring an improvement up to $1.32\%$ on BLEU-1,
$1.39\%$ on METEOR, and $0.44\%$ on BERTScore. This result proves that our
approach of aligning fMRI prompts with text prompts is feasible and effective.

####  4.6.2. Inference Strategy.

Due to the characteristics of decoding tasks in auditory decoding scenarios,
the choice inference strategy has become particularly important. We further
take the ablation study on it. Specifically, we compared four experimental
settings, including the performance of two inference schemes with a fine-tuned
and not fine-tuned GPT-2. The result is reported in Table 4. Also, same as in
Section 4.6.1, the aligning target is the corresponding text prompt under the
same inference strategy.

As is shown in Table 4, we find that using special tokens to indicate the end
of decoding can always bring a performance improvement. Also, fine-tuning the
GPT-2 can bring more improvement with the special tokens. We believe that
fine-tuning the GPT-2 in the training can make the parameters of both the fMRI
encoder and GPT-2 adapt to the special token. However, if fine-tuning is not
performed, only the fMRI encoder will learn to adjust the fMRI prompt to
enable GPT-2 to output $ at the end of each text fragment that corresponds to
a fMRI TR, bringing a negative influence on the performance.

###  4.7. Prompt Length

In our method, fMRI drives GPT-2 to generate decoding targets through the
prompt. Therefore, the length of the prompt is crucial for the decoding
performance. In this part, we investigated the relationship between prompt
length and decoding performance and reported the result in Figure 5. We use
our BP-GPT setting, that is use special tokens in inference with a fine-tuned
GPT-2 model.

As is shown in the figure, the decoding performance increases with the length
of the prompt. Although a longer prompt length can bring better results, it
also incurs greater hardware costs, whether it is for the mapping network or
the fMRI encoder. Due to GPU memory limitations, the maximum prompt limit
during our experiment was 30. However, based on the experimental results, we
expect a better performance with a longer prompt length.

###  4.8. Conclution and Future Works

In this work, we propose a decoding method capable of extracting text from
fMRI signals within the auditory neural decoding scenario. The central concept
of our method involves employing an fMRI-prompted Large Language Model (LLM)
for decoding. Specifically, we utilize an fMRI encoder to extract fMRI
representations, which serve as prompts for the pre-trained GPT-2 model.
Through the application of cross-entropy loss, our fMRI encoder learns the
appropriate prompt for GPT-2 to generate the target text. In order to reduce
modal differences between the text and fMRI, we introduce a contrastive loss
to align the fMRI prompt with the text prompt. Experimental results
demonstrate the effectiveness and advantage of our approach.

In summary, our prompt-based LLM decoding method offers ease of implementation
and extends to various text-based neural decoding tasks. Furthermore, with the
ongoing advancement of LLMs, our method remains readily compatible with
updated and superior LLMs, facilitating performance improvements effortlessly.
Moving forward, our focus will be on applying our prompted LLM decoding
paradigm to a broader range of neural decoding fields and integrating it with
additional LLMs.

## References

  * (1)
  * Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023\.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_ (2023).
  * Affolter et al. (2020) Nicolas Affolter, Beni Egressy, Damian Pascual, and Roger Wattenhofer. 2020.  Brain2word: decoding brain activity for language generation.  _arXiv preprint arXiv:2009.04765_ (2020).
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020\.  Language models are few-shot learners.  _Advances in neural information processing systems_ 33 (2020), 1877–1901.
  * Défossez et al. (2023) Alexandre Défossez, Charlotte Caucheteux, Jérémy Rapin, Ori Kabeli, and Jean-Rémi King. 2023.  Decoding speech perception from non-invasive brain recordings.  _Nature Machine Intelligence_ 5, 10 (2023), 1097–1107.
  * Denkowski and Lavie (2014) Michael Denkowski and Alon Lavie. 2014.  Meteor universal: Language specific translation evaluation for any target language. In _Proceedings of the ninth workshop on statistical machine translation_. 376–380.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ (2018).
  * Du et al. (2023) Changde Du, Kaicheng Fu, Jinpeng Li, and Huiguang He. 2023\.  Decoding visual neural representations by multimodal learning of brain-visual-linguistic features.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ (2023).
  * Duan et al. (2023) Yiqun Duan, Jinzhao Zhou, Zhen Wang, Yu-Kai Wang, and Chin-Teng Lin. 2023.  Dewave: Discrete eeg waves encoding for brain dynamics to text translation.  _arXiv preprint arXiv:2309.14030_ (2023).
  * LeBel et al. (2023) Amanda LeBel, Lauren Wagner, Shailee Jain, Aneesh Adhikari-Desai, Bhavin Gupta, Allyson Morgenthal, Jerry Tang, Lixiang Xu, and Alexander G Huth. 2023.  A natural language fMRI dataset for voxelwise encoding models.  _Scientific Data_ 10, 1 (2023), 555.
  * Lewis et al. (2019) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019\.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  _arXiv preprint arXiv:1910.13461_ (2019).
  * Li et al. (2022) Rui Li, Yiting Wang, Wei-Long Zheng, and Bao-Liang Lu. 2022\.  A multi-view spectral-spatial-temporal masked autoencoder for decoding emotions with self-supervised learning. In _Proceedings of the 30th ACM International Conference on Multimedia_. 6–14.
  * Lin et al. (2022) Sikun Lin, Thomas Sprague, and Ambuj K Singh. 2022\.  Mind reader: Reconstructing complex images from brain activities.  _Advances in Neural Information Processing Systems_ 35 (2022), 29624–29636.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ (2019).
  * Logothetis (2003) Nikos K Logothetis. 2003\.  The underpinnings of the BOLD functional magnetic resonance imaging signal.  _Journal of Neuroscience_ 23, 10 (2003), 3963–3971.
  * Loshchilov and Hutter (2017) Ilya Loshchilov and Frank Hutter. 2017.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_ (2017).
  * Mokady et al. (2021) Ron Mokady, Amir Hertz, and Amit H Bermano. 2021.  Clipcap: Clip prefix for image captioning.  _arXiv preprint arXiv:2111.09734_ (2021).
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002\.  Bleu: a method for automatic evaluation of machine translation. In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_. 311–318.
  * Pasley et al. (2012) Brian N Pasley, Stephen V David, Nima Mesgarani, Adeen Flinker, Shihab A Shamma, Nathan E Crone, Robert T Knight, and Edward F Chang. 2012\.  Reconstructing speech from human auditory cortex.  _PLoS biology_ 10, 1 (2012), e1001251.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019\.  Pytorch: An imperative style, high-performance deep learning library.  _Advances in neural information processing systems_ 32 (2019).
  * Pereira et al. (2018) Francisco Pereira, Bin Lou, Brianna Pritchett, Samuel Ritter, Samuel J Gershman, Nancy Kanwisher, Matthew Botvinick, and Evelina Fedorenko. 2018.  Toward a universal decoder of linguistic meaning from brain activation.  _Nature communications_ 9, 1 (2018), 963.
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018\.  Improving language understanding by generative pre-training.  (2018).
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019\.  Language models are unsupervised multitask learners.  _OpenAI blog_ 1, 8 (2019), 9.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of machine learning research_ 21, 140 (2020), 1–67.
  * Ramesh et al. (2021) Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. 2021\.  Zero-shot text-to-image generation. In _International conference on machine learning_. Pmlr, 8821–8831.
  * Santoro et al. (2017) Roberta Santoro, Michelle Moerel, Federico De Martino, Giancarlo Valente, Kamil Ugurbil, Essa Yacoub, and Elia Formisano. 2017.  Reconstructing the spectrotemporal modulations of real-life sounds from fMRI response patterns.  _Proceedings of the National Academy of Sciences_ 114, 18 (2017), 4799–4804.
  * Scotti et al. (2024) Paul Scotti, Atmadeep Banerjee, Jimmie Goode, Stepan Shabalin, Alex Nguyen, Aidan Dempster, Nathalie Verlinde, Elad Yundler, David Weisberg, Kenneth Norman, et al. 2024\.  Reconstructing the mind’s eye: fMRI-to-image with contrastive learning and diffusion priors.  _Advances in Neural Information Processing Systems_ 36 (2024).
  * Sun et al. (2019) Jingyuan Sun, Shaonan Wang, Jiajun Zhang, and Chengqing Zong. 2019.  Towards sentence-level brain decoding with distributed representations. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 33. 7047–7054.
  * Tang et al. (2023) Jerry Tang, Amanda LeBel, Shailee Jain, and Alexander G Huth. 2023\.  Semantic reconstruction of continuous language from non-invasive brain recordings.  _Nature Neuroscience_ (2023), 1–9.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ 30 (2017).
  * Wang et al. (2022) Yixin Wang, Shuang Qiu, Dan Li, Changde Du, Bao-Liang Lu, and Huiguang He. 2022\.  Multi-modal domain adaptation variational autoencoder for EEG-based emotion recognition.  _IEEE/CAA Journal of Automatica Sinica_ 9, 9 (2022), 1612–1626.
  * Wang and Ji (2022) Zhenhailong Wang and Heng Ji. 2022.  Open vocabulary electroencephalography-to-text decoding and zero-shot sentiment classification. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 36. 5350–5358.
  * Xi et al. (2023) Nuwa Xi, Sendong Zhao, Haochun Wang, Chi Liu, Bing Qin, and Ting Liu. 2023\.  UniCoRN: Unified Cognitive Signal ReconstructioN bridging cognitive signals and human language.  _arXiv preprint arXiv:2307.05355_ (2023).
  * Yang et al. (2015) Minda Yang, Sameer A Sheth, Catherine A Schevon, Guy M Mckhann Ii, and Nima Mesgarani. 2015\.  Speech reconstruction from human auditory cortex with deep neural networks. In _Sixteenth Annual Conference of the International Speech Communication Association_.
  * Yuan et al. (2008) Jiahong Yuan, Mark Liberman, et al. 2008\.  Speaker identification on the SCOTUS corpus.  _Journal of the Acoustical Society of America_ 123, 5 (2008), 3878\.
  * Zhang et al. (2019) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019.  Bertscore: Evaluating text generation with bert.  _arXiv preprint arXiv:1904.09675_ (2019).
  * Zhou et al. (2020) Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, and Jianfeng Gao. 2020\.  Unified vision-language pre-training for image captioning and vqa. In _Proceedings of the AAAI conference on artificial intelligence_ , Vol. 34. 13041–13049.


# Percolation transition of k-frequent destinations network for urban mobility

Weiyu Zhang1,† Furong Jia1,† Jianying Wang2 Yu Liu1 Gezhi Xiu1 1Institute of
Remote Sensing and Geographical Information Systems, Peking University,
Yiheyuan Road 5, Haidian District, Beijing, 100871, China,
2Institute of Space and Earth Information Science, The Chinese University of
Hong Kong

###### Abstract

Urban spatial interactions are a complex aggregation of routine visits and
random explorations by individuals. The inherent uncertainty of these random
visitations poses significant challenges to understanding urban structures and
socioeconomic developments. To capture the core dynamics of urban interaction
networks, we analyze the percolation structure of the $k$-most frequented
destinations of intracity place-to-place flows from mobile phone data of eight
major U.S. cities at a Census Block Group (CBG) level. Our study reveals a
consistent percolation transition at $k^{*}=130$, a critical threshold for the
number of frequently visited destinations necessary to maintain a cohesive
urban network. This percolation threshold proves remarkably consistent across
diverse urban configurations, sizes, and geographical settings over a 48-month
study period, and can largely be interpreted as the joint effect of the
emergence of hubness and the level of mixing of residents. Furthermore, we
examine the socioeconomic profiles of residents from different origin areas
categorized by the fulfillment level of $k^{*}=130$ principal destinations,
revealing a pronounced distinction in the origins’ socioeconomic advantages.
These insights offer a nuanced understanding of how urban spaces are
interconnected and the determinants of travel behavior. Our findings
contribute to a deeper comprehension of the structural dynamics that govern
urban spatial interactions.

Intracity spatial interactions illustrate the interconnectedness of small
urban areas, creating a dynamic network driven by human movement across
various functional zones [1, 2, 3]. The complexity of this network stems from
its diverse connections, each marked by different travel frequencies between
points [4, 5]. Unlike traditional weighted networks, which often display
consistent patterns, this spatial interaction network encompasses a wide array
of destinations and inherits diverse travel behaviors, including commuting,
leisure, and transfer travels [6, 7, 8]. To fully understand this network, it
is crucial to focus on the irregular and unstable interactions, which
traditional models often overlook [9, 10], aspects that traditional models
fail to represent adequately. Identifying the network’s fundamental structure
of stability, or ‘backbone’, is essential for a comprehensive analysis of
urban spatial dynamics and for understanding human mobility patterns.

To effectively map the backbone that captures the core dynamics of place-to-
place interactions, including the nuanced activities of local residents, we
employ the concept of the percolation process from physics [11, 12, 10]. By
considering the city as a complex network of location-based interactions [13,
14], an intriguing pattern emerges: focusing solely on the most frequented
destinations within each area causes the once cohesive city-wide network to
fragment into smaller, isolated clusters. This fragmentation, similar to a
percolation transition, marks a critical point where the network shifts from
being globally interconnected to predominantly disconnected [15, 16].

This observation leads us to a natural delineation of the backbone for the
intracity place-to-place interaction network. From a global perspective, the
essential requirement for this network is the maintenance of city-wide
connectedness [17]; a scenario where infrequent but crucial destinations
become inaccessible is untenable for the network to function effectively as a
backbone [10]. Locally, it is vital to include a predetermined number of
primary destinations for each site to accurately capture and represent the
stable, recurrent patterns of individual visitations.

The network defined by the most frequently visited destinations, situated at
the threshold of connectivity criticality, emerges as a robust framework for
delineating the backbone of intracity place-to-place interactions. This
approach skillfully balances the dual goals of reflecting the pervasive
mobility patterns within the city and preserving the integral connectedness of
its diverse locales.

Empirically, we utilize monthly aggregated mobile phone data from eight major
U.S. cities spanning four years to analyze this phenomenon [18]. Our findings
highlight a remarkable consistency in the percolation transition points across
different urban layouts and over time. Further, an in-depth analysis of the
subgraphs formed at the threshold of the 130 most frequented destinations
reveals a consistent power-law degree distribution across these cities.

Despite the importance of the most frequented destinations network underscored
by the criticality of connectivity, a key question remains: to what extent or
frequency should a destination be included as an edge in the network’s
backbone? Our analysis shows that an area’s most-frequent destinations
significantly correlate with a wide range of socioeconomic properties. This
not only highlights the practical relevance of our approach but also enhances
its explanatory power in understanding urban mobility and planning. The
selected frequented destinations provide crucial insights into the
socioeconomic dynamics, emphasizing the significance of this focused analysis
in urban studies.

## Results

### A model to extract the urban mobility backbone through percolation

To understand the backbone of urban movements, it is essential to explore the
reasons behind intracity travel, which often includes routine habits such as
commuting (exploitation) and spontaneous trips to less frequented destinations
for dining or entertainment (exploration) [19, 20, 21]. To track these
patterns, we use a matrix $W$ that represents a month’s worth of visits among
$N_{c}$ defined small areas in a city, such as census tracts in the US. Each
element $(i,j)$ in this matrix records the total visits from area $i$ to area
$j$, capturing both regular and occasional visits. Conceptually, $W$ combines
a stable interaction matrix $\langle W\rangle$ and a random exploration matrix
$R$, the latter containing independent random elements to represent
exploratory movements.

To isolate recurrent visitation patterns, we simplify $W$ by considering only
the top $k$ destinations from each area, thus forming a reduced matrix
$\langle W\rangle$. This strategy assumes that each location depends on $k$
other locations for daily needs like commuting, shopping, and leisure, with
$k$ selected to ensure the city remains interconnected without excessive
overlap or redundancy.

We introduce a critical parameter $k^{*}$, the smallest number of destinations
needed to maintain city-wide connectivity. This parameter helps identify a
core network with far fewer connections than the full matrix $W$, effectively
representing regular movement patterns. Visualizations of the most-frequent-
destination subnetwork for cities like Los Angeles, New York City, Chicago,
and Dallas (Fig. 1 E-H) contrast with the complete visitation network of these
cities (Fig. 1 A-D), showcasing a clear backbone connecting various subcenters
rather than a diffuse, heatmap-like spatial distribution. This difference
validates the most-frequent-destination subnetwork as a structural backbone of
urban spatial interactions.

Further, we test whether this network of the top $k$ most visited destinations
effectively captures the city’s key spatial interactions, both structurally
and socioeconomically. By analyzing data from eight US cities over four years,
we find a consistently optimal number of destinations, $k^{*}$, around 130 for
each city. This network, focusing on the top 130 destinations, provides the
best insight into the socioeconomic dynamics, reinforcing the relevance and
robustness of our methodological approach.

### Consistency of connectivity level across cities and over time

To ascertain the significance of focusing on major outflows in urban mobility,
we analyze the critical connectivity defined as the minimum number of
frequently visited destinations in each area required to maintain network
connectedness, denoted as $G_{\text{city}}(t)$. Our findings indicate that as
the number of considered outflows increases, the network’s reach expands,
extending connectivity from urban cores to suburban areas. The critical
threshold, $k^{*}$, is identified at the point where even the least connected
Census Block Groups (CBGs) are integrated into the network, ensuring city-wide
connectivity. Fig. 1I illustrates the consistency of the critical percolation
transition point $k^{*}$ across various cities, as identified by their
statistical distribution of $k^{*}$ over 48 months from 2018 to 2022.
Remarkably, $k^{*}$ consistently hovers around 130, a constancy that holds
true regardless of city size, urban layout, or stage of development. This
uniformity highlights the global connectedness and local necessity for
specific visitation patterns, offering valuable insights into urban structures
as networks of small areas interconnected by their 130 most frequent outflows.

Moreover, the criticality of connectivity at $k^{*}=130$ coincides with a
significant transition in the level of visitation correlations. Fig. 1J
depicts the relationship between the average node-wise clustering coefficient
(CC) of the largest component of a city’s $k$-most-frequent-destination
subnetwork and the number of destinations $k$. The clustering coefficient,
defined as the number of triangles in the network divided by the number of
total triads, initially decreases as $k$ increases from zero to around 100,
indicating that more areas are included in the giant component via a few
critical links, without forming dense interactions. However, as $k$ approaches
130, the CC begins to increase, suggesting that triangular links are added to
the destination sets, forming complex interactions characteristic of a motif-
driven structure. This shift indicates that the interactions around the 130th
destinations mark a turning point from a predominantly hierarchical
interaction network to a more complex, motif-driven structure, where each
motif involves visitation among more than two nodes, adding depth and
stability to the network.

Figure 1: The universal structural conciseness of 130-most-frequent
destination network. A-D are the original flow pattern extracted from mobile
phone data, while the backboned patterns shown in E-H are the spatial
organization of the most-frequent-destination network. I is the consistent
critical percolation transition point across various U.S. cities, as
determined by the number of most frequented destinations required to maintain
city-wide connectivity. The data, collected over a 48-month period,
demonstrates the remarkable stability of the k=130 threshold.J is the
clustering coefficient variations of $G_{k}$. K is the increase hubness index
that grows from $G_{\infty}$ to $G_{k^{*}}$.

The observed consistency in the critical number of frequently visited
destinations raises an interesting question: What commonalities exist among
these principal destinations? Typically, the most frequented destinations
include key urban fixtures such as shopping centers, grocery stores, and
workplaces. These destinations are likely reflective of the origins’
preferential choices, suggesting that they are not only centers of daily
necessity but also pivotal to the overall urban connectivity.

When analyzing the structure of the subgraph $G(k^{*})$, which represents the
network at the critical threshold, it exhibits a more heavy-tailed
distribution compared to the original visitation network. This implies that
while principal destinations serve routine needs, they also accommodate a
diverse range of visitation patterns, highlighting the variety in urban
mobility and the roles these destinations play in daily urban life. Moreover,
the proportion of visits to these principal destinations provides insights
into their accessibility, making the frequency and distribution of visits to
these locations an indirect measure of urban accessibility.

In subsequent subsections, we delve deeper into assessing the efficacy and
characteristics of the critical network $G(k^{*})$. This analysis will extend
our understanding of urban mobility by emphasizing the criticality of specific
destinations in sustaining city-wide connectivity and catering to the diverse
needs of urban residents. Through examining $G(k^{*})$, we aim to elucidate
the qualitative aspects of urban networks, focusing on the distribution
patterns of principal destinations and their implications for accessibility
within the urban fabric. This comprehensive examination helps in understanding
how urban structures facilitate or hinder daily activities and mobility.

### Inflow degree distribution at criticality

The analysis of the urban connectivity framework reveals a distinct pattern: a
geographical unit is deemed an integral part of the urban fabric once its
primary 130 destinations have been identified as significant urban elements
(as shown in Fig. 1E-H). Considering that the connectivity threshold of 130 is
substantially lower than the total number of Census Block Groups (CBGs)
present in any urban area ($\sim 1000$), it implies that the most frequented
130 locations predominantly encompass major urban hubs. We validate this
observation by assessing the hubness of nodes, specifically the “hubness”
within the network $G_{k}$ in Fig. 1K, following the methodology outlined in
[22]. In this context, “top hubness” refers to the mean hubness index of the
top 10 nodes, which possess the highest hubness values within the network
$G_{k}$. As the network $G_{\infty}$ is refined towards the connectivity
threshold $G_{130}$ by eliminating less frequent connections, the top hubness
for each city generally remains consistent. However, there is a notable surge
in top hubness at the exact threshold of 130. This suggests that the
importance of certain locations is contingent upon their inclusion among the
most frequently visited destinations.

To further test this hypothesis, we investigated the inflow degree
distribution within the connectivity frameworks of various cities, as well as
within networks at different connectivity levels, specifically at $k^{*}$,
$2k^{*}$, and $k^{*}/2$. Here, $k^{*}$ denotes the minimum number of
destinations needed to ensure that the city’s network, $G_{\text{city}}(t)$,
remains fully connected. The choice of these levels is strategic, aiming to
highlight the network’s behavior around the critical threshold. Networks that
include at least $k^{*}$ most visited destinations may encompass additional,
potentially more randomly chosen locations. This inclusion could expand the
network’s structure, potentially deviating from a preferential attachment
model and consequently impacting the power law degree distribution
characteristic of these networks.

Fig. 2 illustrates the Complementary Cumulative Distribution Function
($P_{c}$) for inflow degree distributions, comparing networks centered around
the $k^{*}$, $2k^{*}$, and $k^{*}/2$ most visited destinations for each origin
within eight cities. In all cases, the inflow degree distributions adhere to a
power law across all variations of $G_{city}^{k^{*}}(t)$,
$G_{city}^{k^{*}/2}(t)$, and $G_{city}^{2k^{*}}(t)$. This uniformity
underscores a consistent pattern of preferential visits to critical locations.
However, as we expand the network to include destinations beyond the critical
$k^{*}$, up to $2k^{*}$, there’s a noticeable increase in the proportion of
nodes with lower inflow degrees, signaling a shift toward more distributed
visitation patterns across less frequently chosen destinations.

Figure 2: Inflow Degree Distribution at Critical Connectivity Thresholds. This
figure presents the Complementary Cumulative Distribution Function (CCDF) for
the in-degree distributions within the urban mobility network at three
distinct levels of connectivity: k* (the percolation threshold), 2k* (double
the threshold), and k*/2 (half the threshold). The visualization highlights
the transition in visitation patterns from preferential to more distributed as
the network’s connectivity increases beyond the critical threshold. The data
underscores the nuanced dynamics of urban mobility and the influence of the
$k^{*}=130$ threshold on shaping visitation frequencies to key destinations.

Therefore, the distinct emphasis on the most frequented destinations
diminishes as the network reaches and surpasses the critical connectivity
threshold, $k^{*}$. This shift from a predominantly preferential selection
mechanism towards a more randomized visitation pattern at and beyond $k^{*}$
signifies that the critical point also marks a transition in urban mobility
behaviors.

### Socioeconomic composition of frequent destinations

The subnetwork identified at the percolation transition point, while offering
a condensed structural overview, also raises questions about its functional
capacity to elucidate levels of regional urbanization. The selection of 130
destinations, although extensive, aligns with the reality that individuals
engage in a diverse array of activities beyond mere routine tasks like work
and groceries. This includes essential services such as gas stations and mixed
visitations that vary by residents’ proximity to city centers. For example,
residents in areas distant from city cores may need to travel further for
work, thereby frequenting life-supplying points of interest near their
workplaces. Conversely, those residing near central workplaces might frequent
nearby locations for basic needs, with additional exploratory visits largely
driven by leisure. This diversity in urban dynamics necessitates an
examination of whether the principal destination flows within a small area
account for a significant portion of total outflows and how these flows
correlate with the area’s broad socioeconomic characteristics.

To effectively quantify the impact of the 130 most frequented destinations
within local areas, we introduce the Proportion of Principal Destinations
(PPD) metric. This metric measures the proportion of flows directed towards
the 130 most frequent destinations from each small area. A high PPD suggests
that outflow visitations are concentrated, indicating that residents’ daily
needs are met without requiring visits to many locations. Moreover, a high PPD
might also suggests a lower level of residential diversity, as it indicates a
more homogeneous pattern of destination use within the area.

Fig. 3 shows the spatial distribution of PPD values in New York MSAs. Notably,
the lowest PPD values often identify public facilities with little
participation of residential activity such as transportation hubs including
the JFK Airport and major parks like Central Park and Van Cortlandt Park. On
the other hand, the indication of low PPD values in residential areas are
found in the areas with poor security status such as Brownsville (with high
crime rates and gang presence) in Brooklyn and Mott Haven in the Bronx (known
as the a high-density and mainly low-income neighborhood). Another
representative area with a low PPD value is the Starrett City in the southwest
of Brooklyn known as the largest government-subsidized residential complex in
the United States. Its residents live averaged below 50% of the poverty line.
With no obvious hubness in its geographical locations, the low PPD areas
mentioned above find the concerning places of socioeconomic and security
status from an accessibility prospective. Similar conclusions are found in
other cities (see SI).

Figure 3: Spatial Distribution of the Proportion of Principal Destinations
(PPD) in New York Metropolitan Area. This map delineates the geographical
variation of PPD values, which represent the share of mobility flows directed
towards the top 130 most frequented destinations within each Census Block
Group (CBG). The visualization reveals a stark contrast between areas with
high PPD values, indicating concentrated mobility patterns and better
accessibility, and areas with low PPD values, which may suggest challenges in
accessibility and socioeconomic disparities.

The above indication of principal destination coverage on socioeconomic status
can also be quantitatively verified by a correlation analysis with a set of
variables collected from [23] and American Community Survey (ACS) data [24].
Here we compare the population-weighted correlation Generally we find the
neighborhoods with better socioeconomic status have more concentrated travel
patterns: Median Household Income correlates with PPD by 0.25, and the
Proportion of Population Without Health Insurance is negatively correlated
with PPD by -0.16, which also suggests a potential exposure risk of low-income
families to access the medical establishments. This coincides with the
regional cases discussed above that rich communities tend to have less
principal destinations to visit to accommodate their daily needs. On the other
hand, education level is another principal factor that explains PPD:
Highschool Graduation Rates (-0.13), College Enrollment Rate (-0.13) are both
negatively correlated with PPD, while Population with a Master’s Degree’s
correlation with PPD is 0.20. This shows that an educated populations over
college tend to have a wider range of destination, while among the highschool
graduates, those with higher education level tend to be more occupied with
their professions so that to visit only fewer places. The PPD’s correlation
with residential percentage is only -0.06.

We further use the socioeconomic association of the PPD to validate the
significance of percolation transition point by extending the regression of
the PPD of different numbers of principal destinations from 130 to others. In
Fig. 4, we trace the $p$-value of the regressions of PPD from different
numbers of principal destinations as an indicator of the significance of the
connectivity level’s transition point. Most of the socioeconomic factors shows
a pronounced shift in significance around these connectivity transition points
130. Notably, factors like Median Household Income, Population With A
Bachelor’s Degree, and Building Density display heightened significance as the
number of frequent destinations approaches the connectivity transition point,
suggesting these factors may play a critical role in influencing the overall
connectivity of the urban landscape in New York. However for subnetworks of
more than 130 frequent destinations, a discernible drop in significance for
many factors, indicating that beyond a certain level of connectivity (or
number of neighbors), additional increases in k-values have diminishing
effects on the relevance of these socioeconomic factors. This pattern suggests
that there is an optimal level of connectivity, represented by a specific
k-value, that maximizes the influence of these factors on the urban network’s
structure. Beyond this point, the network might be oversaturated, leading to
diminishing returns in the context of these socioeconomic indicators. These
show that the connectivity transition points coincide with the optimal level
of urban connectivity that should be targeted to maximize the impact of
various socioeconomic factors in shaping the urban fabric of New York.

Figure 4: Correlation Analysis of Socioeconomic Factors with the Proportion of
Principal Destinations (PPD). B illustrates the relationship between various
socioeconomic indicators and the PPD metric, which measures the concentration
of mobility flows to the top 130 destinations. The data points in A showcase a
significant shift in the correlation coefficients around the critical
connectivity threshold of k=130, indicating the socioeconomic variables’
varying influence on urban mobility patterns at different levels of
connectivity. The analysis provides valuable insights into how different
neighborhoods’ socio-economic profiles impact the structure and efficiency of
urban mobility networks.

## Discussion

In this article, we study intracity human mobility through a percolation model
to extract the core network of urban flows and examine the socioeconomic
implications of these mobility patterns. Through the lens of the Proportion of
Principal Destinations (PPD), we have delved into the spatial and
socioeconomic fabric of urban environments, revealing nuanced insights into
how people move within cities and the underlying factors influencing these
movements.

Our findings confirm the criticality of the 130 most frequented destinations
in shaping the urban mobility backbone. The consistency of this number across
diverse urban settings highlights a universal aspect of urban mobility that
transcends the unique geographical and socio-economic characteristics of
individual cities. This revelation not only enriches our understanding of
urban dynamics but also offers a robust framework for urban planning and
policy interventions. The correlations between PPD values and socio-economic
indicators unveil the complex interplay between urban mobility and the socio-
economic landscape. High PPD values in affluent areas contrast sharply with
the lower PPDs observed in underprivileged neighborhoods, underscoring the
disparities in accessibility and convenience across different urban areas.

Acknowledgement: This research was funded by the National Natural Science
Foundation of China (41830645 and 41971331). G.X. was funded by the China
Scholarship Council.

## References

  * Pflieger and Rozenblat [2010] G. Pflieger and C. Rozenblat, Introduction. urban networks and network theory: the city as the connector of multiple networks (2010).
  * Bettencourt [2021] L. M. Bettencourt, Complex networks and fundamental urban processes, in  _Handbook of cities and networks_ (Edward Elgar Publishing, 2021) pp. 41–61.
  * Sevtsuk and Mekonnen [2012] A. Sevtsuk and M. Mekonnen, Urban network analysis, Revue internationale de géomatique–n 287, 305 (2012).
  * Barrat  _et al._ [2004] A. Barrat, M. Barthelemy, R. Pastor-Satorras, and A. Vespignani, The architecture of complex weighted networks, Proceedings of the national academy of sciences 101, 3747 (2004).
  * Hashemi and Karimi [2016] M. Hashemi and H. A. Karimi, A weight-based map-matching algorithm for vehicle navigation in complex urban networks, Journal of Intelligent Transportation Systems 20, 573 (2016).
  * Schlosser  _et al._ [2020] F. Schlosser, B. F. Maier, O. Jack, D. Hinrichs, A. Zachariae, and D. Brockmann, Covid-19 lockdown induces disease-mitigating structural changes in mobility networks, Proceedings of the National Academy of Sciences 117, 32883 (2020).
  * Louail  _et al._ [2015] T. Louail, M. Lenormand, M. Picornell, O. Garcia Cantu, R. Herranz, E. Frias-Martinez, J. J. Ramasco, and M. Barthelemy, Uncovering the spatial structure of mobility networks, Nature communications 6, 6007 (2015).
  * Rajput and Mostafavi [2023] A. A. Rajput and A. Mostafavi, Latent sub-structural resilience mechanisms in temporal human mobility networks during urban flooding, Scientific Reports 13, 10953 (2023).
  * Serrano  _et al._ [2009] M. Á. Serrano, M. Boguná, and A. Vespignani, Extracting the multiscale backbone of complex weighted networks, Proceedings of the national academy of sciences 106, 6483 (2009).
  * Li  _et al._ [2015] D. Li, B. Fu, Y. Wang, G. Lu, Y. Berezin, H. E. Stanley, and S. Havlin, Percolation transition in dynamical traffic network with evolving critical bottlenecks, Proceedings of the National Academy of Sciences 112, 669 (2015).
  * Shante and Kirkpatrick [1971] V. K. Shante and S. Kirkpatrick, An introduction to percolation theory, Advances in Physics 20, 325 (1971).
  * Li  _et al._ [2021] M. Li, R.-R. Liu, L. Lü, M.-B. Hu, S. Xu, and Y.-C. Zhang, Percolation on complex networks: Theory and application, Physics Reports 907, 1 (2021).
  * Helbing [2001] D. Helbing, Traffic and related self-driven many-particle systems, Reviews of modern physics 73, 1067 (2001).
  * Chowdhury  _et al._ [2000] D. Chowdhury, L. Santen, and A. Schadschneider, Statistical physics of vehicular traffic and some related systems, Physics Reports 329, 199 (2000).
  * Wang  _et al._ [2023] R. Wang, Q. Wang, and N. Li, Percolation transitions in urban mobility networks in america’s 50 largest cities, Sustainable Cities and Society 91, 104435 (2023).
  * He  _et al._ [2021] H. He, H. Deng, Q. Wang, and J. Gao, Percolation of temporal hierarchical mobility networks during covid-19, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 380, 20210116 (2021).
  * Sim  _et al._ [2015] A. Sim, S. N. Yaliraki, M. Barahona, and M. P. H. Stumpf, Great cities look small, Journal of The Royal Society Interface 12, 20150315 (2015).
  * Kang  _et al._ [2020] Y. Kang, S. Gao, Y. Liang, M. Li, J. Rao, and J. Kruse, Multiscale dynamic human mobility flow dataset in the us during the covid-19 epidemic, Scientific data 7, 390 (2020).
  * Liu  _et al._ [2012] Y. Liu, C. Kang, S. Gao, Y. Xiao, and Y. Tian, Understanding intra-urban trip patterns from taxi trajectory data, Journal of geographical systems 14, 463 (2012).
  * Song  _et al._ [2010] C. Song, Z. Qu, N. Blumm, and A.-L. Barabási, Limits of predictability in human mobility, Science 327, 1018 (2010).
  * Pappalardo  _et al._ [2015] L. Pappalardo, F. Simini, S. Rinzivillo, D. Pedreschi, F. Giannotti, and A.-L. Barabási, Returners and explorers dichotomy in human mobility, Nature communications 6, 8166 (2015).
  * Kleinberg [1999] J. M. Kleinberg, Authoritative sources in a hyperlinked environment, Journal of the ACM (JACM) 46, 604 (1999).
  * Xi  _et al._ [2023] Y. Xi, Y. Liu, T. Li, J. Ding, Y. Zhang, S. Tarkoma, Y. Li, and P. Hui, A satellite imagery dataset for long-term sustainable development in united states cities, Scientific Data 10, 866 (2023).
  * Bureau [2022] U. S. C. Bureau, American community survey data, https://www.census.gov/programs-surveys/acs (2022).