# Pretrained Mobility Transformer: A Foundation Model for Human Mobility

Xinhua Wu Haoyu He Yanchao Wang Qi Wang
Department of Civil and Environmental Engineering
Northeastern University
<EMAIL_ADDRESS>
###### Abstract

Ubiquitous mobile devices are generating vast amounts of location-based
service data that reveal how individuals navigate and utilize urban spaces in
detail. In this study, we utilize these extensive, unlabeled sequences of user
trajectories to develop a foundation model for understanding urban space and
human mobility. We introduce the Pretrained Mobility Transformer (PMT), which
leverages the transformer architecture to process user trajectories in an
autoregressive manner, converting geographical areas into tokens and embedding
spatial and temporal information within these representations. Experiments
conducted in three U.S. metropolitan areas over a two-month period demonstrate
PMT’s ability to capture underlying geographic and socio-demographic
characteristics of regions. The proposed PMT excels across various downstream
tasks, including next-location prediction, trajectory imputation, and
trajectory generation. These results support PMT’s capability and
effectiveness in decoding complex patterns of human mobility, offering new
insights into urban spatial functionality and individual mobility preferences.

##  1 Introduction

Today, ubiquitous mobile devices are generating vast quantities of data,
timestamped with both time and location, from large populations. This
location-based service (LBS) data captures human mobility behaviors with
unprecedented precision and sampling rates [1]. Users’ movement patterns
across various urban areas provide comprehensive insight into individual
behaviors related to work, leisure, entertainment, shopping, and more [2, 3].
The richness and detail of this data open up new avenues for understanding
complex human behaviors in urban contexts.

Despite the potential embedded in these vast resources, they are often
underutilized due to the challenges of handling and interpreting high-volume,
unstructured datasets. Particularly, the unlabeled sequences of user
trajectories have not been fully explored and understood. The lack of research
in these sequences means we missed the opportunity to gain a more in-depth
understanding and knowledge of user movement, such as inter-area
accessibility, area spatial functionality, and user mobility preferences,
which can be extracted in an autoregressive manner and applied to downstream
tasks.

To bridge both methodological and knowledge gaps, we propose Pretrained
Mobility Transformer (PMT) in this work. PMT offers a straightforward and
viable framework for training a human mobility foundation model using LBS
data. Specifically, each geographic area is abstracted as a specific token,
with each token corresponding to a trainable spatial embedding. In this
approach, user trajectories can be represented as a sequence composed of
tokens, akin to tokens in large language models forming paragraphs. We trained
the model using two types of autoregressive methods: predicting the next
location and filling in masked locations. Considering the periodic nature of
human mobility behaviors, we also designed temporal encoding to incorporate
time information.

We conducted experiments within three metropolitan statistical areas (MSAs) in
the United States, specifically Boston-Cambridge-Newton, MA-NH, Los Angeles-
Long Beach-Anaheim, CA, and New York-Newark-Jersey City, NY-NJ-PA. The data
spanned a two-month period, with the selected users representing 1%-2% of the
total population. PMT consistently demonstrated exceptional performance across
the different MSAs, accurately capturing local mobility patterns. Remarkably,
despite the absence of explicit latitude and longitude for geographic areas,
our proposed PMT still constructed relative geographical relationships between
different regions, as represented by the similarity of spatial embeddings.
More interestingly, we found that the spatial embeddings of various regions
also captured, to some extent, the socio-economic attributes of those areas,
such as educational and income levels.

We highlight the contributions of this work as follows: (1) To the best of our
knowledge, this is the first mobility foundation model trained extensively
using large-scale LBS data. The proposed PMT, based on the transformer
architecture, is a straightforward, effective, and scalable model for decoding
complex patterns of human mobility. (2) The spatial embeddings in PMT
effectively capture the geographical and socio-demographic characteristics of
regions, providing a new perspective for understanding urban spaces with a
data-driven approach. (3) The proposed PMT achieved superior performance
consistently across multiple downstream tasks, including next-location
prediction, trajectory imputation, and trajectory generation. PMT with a
larger parameter size showed significant advantages in its performance.

##  2 Related Work

Human Mobility Modelling. Statistical physicists have established theoretical
models to uncover universal patterns in human mobility. CTRW [4] describes
human traveling behavior on various spatiotemporal scales by a two-parameter
continuous-time random walk model, which ignores the considerable temporal and
spatial regularity in individual movements [2, 5]. EPR [6, 7] views the time
duration between an individual’s trips as a distribution and frames the
process of choosing a travel destination as a probabilistic choice between
exploring a new location and returning to a previously visited location. STS-
EPR [8] further considers the impact of social factors during exploration and
return behaviors. While these models offer valuable perspectives on human
movement, their constrained parameters and predefined structures limit their
capability in capturing individual’s complex and heterogeneous mobility
patterns.

Mobility-related Tasks. The development of deep learning and the availability
of big mobility data have spurred a series of studies on tasks related to
mobility. At the individual level, these tasks include next-location
prediction [9, 10, 11, 12], trajectory imputation [13, 14, 15] and trajectory
generation [16, 17, 18]. At the collective level, tasks include crowd flow
prediction [19, 20, 21] and flow generation [22, 23]. Notably, these proposed
methodologies are tailored for specific types of tasks and leverage a diverse
array of datasets, such as public bicycle data [24], taxi data [25], mobile
phone record data [26, 27], social media data [28], and points of interest
(POI) check-in data [29]. Some of these datasets, such as those from taxis and
social media, represent small, biased samplings of the population.
Additionally, given the variability in data availability and quality across
different regions, reproducing these models in other areas poses significant
challenges. In contrast to these efforts, our work aims to develop a
foundation model for understanding human mobility behavior, capable of
directly solving or being fine-tuned to address various mobility-related
tasks. To ensure the generalization of our approach, we exclusively utilize
massive human mobility data generated by mobile devices for autoregressive
training. The dataset has an original sampling rate of 8%-10% of the
population.

Geographic Embedding. The study of identifying the distinct functions (spatial
semantics) of urban areas predominantly employed feature-based methodologies.
This involves developing features for different regions based on information
such as POI distribution and origin-destination flow [30, 31]. These features,
depending on their interpretability and dimensions, can be further used in
clustering algorithms to categorize functional areas [32]. Recently, with the
rise of representation learning [33], more studies captured the latent
functions of areas with higher embedding dimensions using deep learning. In
terms of the objects of embedding, past research has focused on embedding
cellular towers [34], road segments [35, 36], POIs [37, 38, 39], geographic
unit [40] and user trajectories [35, 41]. From the perspective of embedding
methods, [36, 37, 38, 39] utilized the skip-gram model proposed by word2vec
[42], [40] applied graph neural networks with the idea of node2vec [43], and
[34, 35, 41] incorporated encoder-decoder RNN structures. Our work
distinguishes itself in two respects. First, in terms of the embedding
objects, we directly embed urban areas (polygons) rather than just parts of
the city like POIs, aiming for a more comprehensive understanding of human
mobility behavior. This is similar to [40], but notably we use detailed
individual mobility records rather than aggregated origin-destination flow for
training. Second, methodologically, we employ large pretrained transformer
models to extract embeddings of geographic areas, which is motivated by the
breakthroughs of pretraining models in NLP, such as BERT [44] and GPT [45].

##  3 Preliminary

Location-based Service Data. The widespread use of mobile devices with GPS has
led to the creation of large amounts of data from location-based services.
Each entry in LBS data contains the geographical coordinates, specifically the
latitude and longitude of the user, as well as the timestamp in which the data
was recorded. A snapshot of synthetic LBS data is provided in Appendix A.
Despite the high volumes and spatiotemporal scales, LBS data faces some
limitations, including spatial inaccuracies and inconsistent sampling time and
rates. To address these issues, spatial and temporal aggregation is typically
performed, as shown in Figure 1. Each user is assigned to a spatial region
based on their locations within a specific time period. Due to the
incompleteness of LBS data, the user location could be unknown within some
time windows, which is represented by the dark boxes in Figure 1.

User Trajectory Sequence. In this study, we model a user’s movement using a
sequence of location identifiers over consecutive time intervals, represented
as: $X=(x_{1},x_{2},x_{3},\ldots,x_{t},\ldots,x_{T})$. Here, $T$ denotes the
total number of time windows, and each $x_{t}$ is an identifier for a U.S.
Census Block Group (CBG) where the user is located at time $t$. CBGs are
geographical units used in the U.S. Census, typically containing between 600
and 3,000 residents. We use CBGs to track locations because they offer a good
balance between detail and privacy; they are small enough to provide
significant location data yet large enough to prevent precise tracking of
individual users (e.g., the exact locations of homes and workplaces). A
placeholder token is used if the location data is missing. Each time window is
set to 30 minutes. The time window duration aligns with the frequency of
location data collection, which averages 50 to 100 location records per user
per day. Choosing a 30-minute window helps to avoid overly sparse sequences.
In cases where a user’s data shows movement across different CBGs within the
same time window, we only record the last CBG in which the user was located.
Temporal occupancy is defined as the proportion of elements in the sequence
that have specific geographic locations. For example, the sequence shown in
Figure 1 has a temporal occupancy of $\nicefrac{{5}}{{8}}=62.5\%$ .

Figure 1: An illustrative user trajectory sequence with $T=8$ time steps.
Location information is available for five time steps, denoted by light-
colored boxes. The remaining three time steps, which lack user location data,
are represented by dark-colored boxes.

##  4 Methods

We introduce PMT and its implementation in this section. PMT’s model
architecture is a multi-layer transformer based on the original implementation
of [46] without the encoder-decoder structure. The overall framework of per-
training is shown in Figure 2. The trajectory sequences outlined in Section 3
serve as inputs for autoregressive training and the embedded sequence is
processed by multiple layers of transformer blocks to generate the sequences
required for a specific pretraining task.

In this work, we employ three models with parameter sizes of 1.6M, 21M, and
550M, respectively. These models are characterized by different dimensions:
embedding size ($D)$, hidden size ($H)$, the number of transformer layers
($L)$, and the number of self-attention heads ($A$). The specifications for
these models are as follows: PMT-1.6M ($D=128,H=256,L=4,A=4$), PMT-21M
($D=512,H=1024,L=8,A=8$), and PMT-550M ($D=1024,H=2048,L=16,A=16$).

Figure 2: Overall pretraining procedures for PMT. In the input layer, light-
colored, dark-colored, and gray-shadowed boxes distinctly represent elements
in the trajectory sequence that possess location information, are devoid of
location information, and have obscured location information, respectively.
Apart from the causal masking, the same architectures are used in both
pretraining tasks.

###  4.1 Spatiotemporal Embedding

Compared to pretraining models for NLP [45] and CV [47], the interaction
between geographical space and time in trajectory sequences is more crucial
and complex. For example, the Central Business District (CBD) during the day
and residential areas at night are hotspots for human activities.
Additionally, there is a distinct periodicity (e.g., 24 hours, 7 days, etc.)
in individual travel behaviors, reflecting consistent patterns over time. To
better capture the spatiotemporal relationships, we divided the sequence
embedding into spatial embedding and temporal encoding, with their sum
constituting the final embedding. Spatial embedding maps each predefined
spatial area (i.e., CBG) to a fixed-length vector, similar to how tokens are
mapped to vectors in large language models. The parameters for spatial
embedding are learnable and initialized randomly without any prior knowledge.

In terms of temporal encoding, we employed a predefined encoding scheme
instead of using learnable parameters. In this work, temporal encoding
includes absolute time encoding and periodic encoding. Absolute time encoding,
identical to positional encoding in [46], is designed to furnish comprehensive
temporal information within trajectory sequences. Meanwhile, periodic encoding
injects crucial periodical information that reflects human mobility patterns,
specifically daily encoding and weekly encoding. Periodic encoding facilitates
more efficient model pretraining and enhances the model’s ability to
generalize across different temporal contexts in downstream tasks.

Let $X$ be a trajectory sequence of length $T$, and $D$ the desired embedding
dimension. Temporal Encoding, $TE\in\mathbb{R}^{T\times D}$, is the
concatenation of absolute time encoding, $\text{AE}\in\mathbb{R}^{T\times
D/2}$, daily encoding, $\text{DE}\in\mathbb{R}^{T\times D/4}$, and weekly
encoding, $\text{WE}\in\mathbb{R}^{T\times D/4}$. AE, DE and WE are defined as
Equation 1. A visualization of the implemented temporal encoding is presented
in Appendix B.

$\displaystyle\text{AE}_{(t,j)}$
$\displaystyle=\begin{cases}\sin\left(\frac{t}{10000^{4i/D}}\right)&\text{if
}j=2i\\\ \cos\left(\frac{t}{10000^{4i/D}}\right)&\text{if }j=2i+1\end{cases}$
(1) $\displaystyle\text{DE}_{(t,j)}$
$\displaystyle=\begin{cases}\sin\left(\frac{\theta_{d}(t)}{10000^{8i/D}}\right)&\text{if
}j=2i\\\ \cos\left(\frac{\theta_{d}(t)}{10000^{8i/D}}\right)&\text{if
}j=2i+1\end{cases}$ $\displaystyle\text{WE}_{(t,j)}$
$\displaystyle=\begin{cases}\sin\left(\frac{\theta_{w}(t)}{10000^{8i/D}}\right)&\text{if
}j=2i\\\ \cos\left(\frac{\theta_{w}(t)}{10000^{8i/D}}\right)&\text{if
}j=2i+1\end{cases}$

where $t$ is the index of the time step and $j$ is the dimension.
$\theta_{d}(t)$ and $\theta_{w}(t)$ are the angular coordinates for daily and
weekly cycles, respectively, both mapped to the interval $[0,2\pi]$. Given the
30-minute time window, $\theta_{d}(t+48i)=\theta_{d}(t)$ and
$\theta_{w}(t+7\times 48i)=\theta_{w}(t)$ for all integer $i$.

###  4.2 Pretraining Tasks

Two different pretraining tasks, next-location prediction and mask imputation,
are designed for the training of PMT. These pretraining tasks are analogous to
the pretraining methods of GPT [48] and BERT [44], respectively.

Next-location Prediction Task. As shown on the left side of Figure 2, with the
user trajectories as input, the model is to predict the next location given
the previous trajectory. A causal masking strategy is applied to prevent the
leakage of future information. As mentioned above, the dark-colored boxes in
Figure 2 indicate time steps where location data is missing in the user’s
trajectory. We hypothesize that these instances of missing data might subtly
reflect the user’s movement patterns. For instance, when a user reaches their
destination, they are likely to turn off the navigation app, ceasing to upload
their location. Consequently, sequence positions lacking geographical
information are processed as a special token rather than omitted or masked.
They share a learnable spatial embedding and are utilized in subsequent self-
attention transformer layers; however, due to the absence of ground truth,
these positions are not included in the loss calculation.

Mask Imputation Task. In this task, we randomly mask positions within a
trajectory sequence, denoted by the gray-shadowed boxes on the right side of
Figure 2, and task the model with predicting these obscured geographic
locations. Similarly, masks where the ground truth is missing are not used to
calculate the loss. Causal masks were not used, allowing the model to complete
the sequence from all unmasked positions. We set the masking probability at
50% to create a challenging scenario that simple interpolation would not be
able to accomplish. However, this high level of masking, combined with pre-
existing missingness in geographical information within the sequence,
considerably destabilized the training process, especially for the PMT-550M.
To address this issue, we initialized the model with available parameters from
the Next-location Prediction Task. Here, a reasonable suspicion is whether
this could lead to leakage of future trajectory. Nonetheless, our subsequent
experiments (see Appendix E) showed consistent imputation performance across
both training and testing datasets, indicating that the model is not affected
by any significant data leakage during pretraining.

##  5 Experiments and Results

###  5.1 Experiment Setting

In this study, mobile location data from the three metropolitan statistical
areas (MSAs) introduced above were used for experiments. Hereafter, they are
referred to as BOS, LA, and NYC, respectively. Data in these three MSAs are
described in Table 1.

The data spans from January 1, 2020, to February 28, 2020, making the length
of each trajectory sequence 2,832. We selected sequences with a temporal
occupancy (see Section 3) greater than 50%. The sequences were allocated into
two groups, with two-thirds designated for pretraining and the remaining one-
third for testing the model’s performance on downstream tasks. This allocation
was to guarantee that there was no overlap of individuals between the
pretraining and the testing. More details regarding training are presented in
Appendix C.

Table 1: Data description for three MSAs. Temporal occupancy in the table refers to the average temporal occupancy across all trajectory sequences, while the sampling rate indicates the ratio of the number of filtered trajectory sequences to the total population. MSA | Population | CBG | Sequence | Average temporal occupancy | Sampling rate
---|---|---|---|---|---
BOS | 4.9M | 3.5k | 59k | 75.4% | 1.20%
LA | 12.9M | 8.2k | 249k | 77.2% | 1.93%
NYC | 23.6M | 14.4k | 465k | 76.5% | 1.97%

###  5.2 What does the Spatial Embedding Say?

In this subsection, we explore the meaning of spatial embeddings in PMT that
represent geographic areas (i.e., CBGs). Despite PMT models being trained
solely on unlabeled trajectory sequences, we discovered that these spatial
embeddings effectively capture the geographical and deeper socio-demographic
characteristics of the CBGs.

Spatial Embedding Similarity and Geographic Distance. We calculated the cosine
similarity (normalized to 0-1) among all CBGs within the same MSA and the
distances (Euclidean distance) between the centroids of all CBGs. Table 2
shows their negative correlation, indicating that CBGs with higher similarity
are also geographically closer to each other. PMT-550M and PMM-21M models
exhibit a weaker negative correlation, potentially attributable to two
factors. First, these models utilize deeper transformer architectures, which
may produce more abstract representations in their embedding layers. Second,
these models likely construct spatial embeddings that go beyond geographic
proximity. For example, factors such as transportation facilities and public
transport can also impact the accessibility of CBGs.

Table 2: The correlation between spatial embedding similarity and Euclidean distance. Model | Pearson correlation coefficient | Spearman’s correlation coefficient
---|---|---
BOS |  LA |  NYC |  BOS |  LA |  NYC
PMT-1.6M |  -0.29 |  -0.18 |  -0.34 |  -0.34 |  -0.19 |  -0.38
PMT-21M |  -0.11 |  -0.17 |  -0.15 |  -0.15 |  -0.21 |  -0.19
PMT-550M |  -0.06 |  -0.12 |  -0.07 |  -0.09 |  -0.14 |  -0.10

Spatial Embedding Similarity and Racially Dominant CBGs. Using the data from
the American Community Survey, we define CBGs where over 50% of the population
is black as black-dominant CBGs, and those where over 50% of the population is
White as white-dominant CBGs. The black-black comparison calculates the
similarity between all pairs of CBGs within black CBGs, while the black-white
comparison calculates the similarity of all CBG pairs composed of one black
CBG and one white CBG. Figure 3 shows that the distribution of black-black
similarity is significantly shifted to the right compared to the black-white
similarity across three MSAs, suggesting a stronger cohesion within black-
dominant CBGs. This could also indicate that minority populations are more
likely to participate in activities within their own racial communities,
beyond simply living in close proximity.

Figure 3: Spatial embedding similarity distribution of black-black comparison
and black-white comparison (PMT-1.6M).

Spatial Embedding Similarity and Socio-demographics. Individuals from
different socio-economic backgrounds often display distinct behavioral
patterns. We theorize that spatial embeddings in PMT, generated through
autoregressive training using extensive user trajectory data, might contain
socio-demographic information specific to certain areas. To validate this
hypothesis, we employed CBGs’ spatial embedding as inputs to predict their
socio-demographic characteristics. To control for randomness, we conducted one
hundred trials, randomly selecting half of the CBGs for training and the other
half for validation in each trial. A simple neural network with two hidden
layers was trained over five epochs per trial. The results, as shown in Table
3, indicate that this model can predict the population, education levels, and
income with moderate accuracy, suggesting that PMT embeddings indeed
incorporate socio-demographic information, indicating that PMT may implicitly
model urban characteristics and functionalities during its pre-training.

Table 3: $R^{2}$ of demographic prediction using spatial embedding (PMT-1.6M). | BOS | LA | NYC
---|---|---|---
Resident population | $0.296\pm 0.016$ | $0.199\pm 0.010$ | $0.185\pm 0.009$
Bachelor’s degree proportion | $0.475\pm 0.014$ | $0.558\pm 0.011$ | $0.483\pm 0.010$
Median household income | $0.321\pm 0.012$ | $0.380\pm 0.011$ | $0.409\pm 0.009$

###  5.3 Downstream Task Evaluations

We evaluated the performance of the PMT on several downstream tasks, including
next-location prediction, trajectory imputation, and trajectory generation. We
found that the proposed PMT models (PMT-21M and PMT-550M) significantly
outperformed baselines (see Appendix D) across various tasks. Even the
PMT-1.6M, with significantly fewer parameters than the baselines, exhibited
superior performance in some tasks. Additionally, there seems to be a scaling
law; increasing the model parameters benefits PMT in encoding spatiotemporal
information and human mobility, thereby enhancing performance across various
downstream tasks.

Next Location Prediction. This task, mirroring our first pretraining one, is
to predict the next location of a user, given their previous trajectory. The
task is critical for accurate next-location prediction is crucial for
personalized recommendation services, such as POI recommendations. The
historical trajectory length ranges from 30 minutes to two months to simulate
cold-start and warm-start scenarios in next-location prediction.

Table 4 shows the prediction results for both daytime and the entire day
separately. Given the highly predictable nature of human behavior during the
night, daytime predictions exhibit lower accuracy compared to predictions for
the entire day. Notably, PMT-550M and PMT-21M consistently outperform
baselines across all MSAs. While PMT-1.6M performs better than the baselines
in BOS, its performance is inferior to the baselines in LA and NYC. This could
be attributed to the higher number of CBGs in these two MSAs, where PMT-1.6M
might not adequately capture the rich and complex patterns with its small
parameter size.

Table 4: Performance Metrics for Next-Location Prediction. This table presents the cross entropy loss (without label smoothing) for predictions made during daytime hours, defined as 7:30 AM to 7:30 PM local time. The best-performing result is highlighted in bold, while the second-best is underlined for easy comparison. | Model | Daytime | All Day
---|---|---|---
| Loss | Acc@1 | Acc@3 | Acc@10 | Loss | Acc@1 | Acc@3 | Acc@10
BOS | FBM | / | 57.72% | 82.96% | 90.87% | / | 70.87% | 88.13% | 93.63%
LSTM | 1.31 | 79.07% | 85.93% | 90.23% | 0.82 | 86.49% | 91.08% | 93.80%
DeepMove | 1.11 | 80.13% | 87.44% | 91.96% | 0.71 | 87.15% | 92.14% | 94.97%
PMT-1.6M | 1.15 | 80.45% | 88.14% | 92.40% | 0.78 | 87.37% | 92.58% | 95.24%
PMT-21M | 1.05 | 81.71% | 89.01% | 93.23% | 0.70 | 88.23% | 93.16% | 95.79%
PMT-550M | 1.02 | 81.81% | 89.23% | 93.58% | 0.67 | 88.28% | 93.29% | 96.01%
LA | FBM | / | 57.19% | 81.04% | 88.44% | / | 69.36% | 86.40% | 91.64%
LSTM | 1.38 | 78.93% | 85.21% | 88.91% | 0.92 | 85.98% | 90.38% | 92.78%
DeepMove | 1.22 | 79.73% | 86.19% | 90.18% | 0.81 | 86.54% | 91.11% | 93.67%
PMT-1.6M | 1.37 | 79.49% | 86.23% | 90.05% | 0.98 | 86.19% | 90.96% | 93.47%
PMT-21M | 1.13 | 81.70% | 87.92% | 91.72% | 0.78 | 87.84% | 92.17% | 94.60%
PMT-550M | 1.11 | 81.64% | 88.10% | 92.00% | 0.76 | 87.76% | 92.29% | 94.79%
NYC | FBM | / | 56.86% | 82.05% | 89.51% | / | 69.30% | 86.76% | 92.19%
LSTM | 1.83 | 76.85% | 81.64% | 85.17% | 1.24 | 84.36% | 87.68% | 90.07%
DeepMove | 1.18 | 79.75% | 86.54% | 90.78% | 0.80 | 86.34% | 91.17% | 93.93%
PMT-1.6M | 1.48 | 78.13% | 85.37% | 90.00% | 1.15 | 84.35% | 89.77% | 93.11%
PMT-21M | 1.08 | 81.98% | 88.39% | 92.42% | 0.76 | 87.80% | 92.32% | 94.45%
PMT-550M | 1.04 | 82.02% | 88.71% | 92.80% | 0.72 | 87.84% | 92.55% | 95.22%

Trajectory Imputation. As users have control over uploading their location
information, trajectory sequences derived from LBS data frequently exhibit low
temporal occupancy. This constraint hampers a comprehensive understanding of
individual mobility patterns. The aim of trajectory imputation is to
reconstruct user locations for all time intervals based on existing incomplete
sequences. We specifically assessed the model’s ability to fill in missing
data across various levels of data sparsity. We randomly removed 50% to 90% of
the location information from the trajectory sequences over a two-month
period. This simulates different degrees of sparsity observed in location-
based service data.

Figure 4 shows the model’s imputation accuracy across different degrees of
data sparsity. PMT-550M and PMT-21M consistently outperform other models
across a range of sparsity levels, whereas PMT-1.6M, despite initially lagging
behind baselines at lower missingness rates, surpasses them as sparsity
increases. This suggests PMT’s advantages in capturing the spatiotemporal
regularities of human mobility from sparse trajectories. PMT-550M and PMT-21M
exhibit comparable accuracy, but PMT-550M shows better performance in terms of
cross-entropy, as detailed in Appendix E.

Figure 4: Accuracy of imputation across varying sparsity levels. The removal
ratio represents the proportion of locations randomly omitted from the
trajectory sequences. Considering the pre-existing gaps in the sequences, the
actual missingness rate is higher than indicated by the removal ratio alone.
At a 90% removal ratio, the average temporal occupancy of sequences is
approximately 7.5%.

Trajectory Generation. Differing from next-location prediction, trajectory
generation aims to produce extended sequences of human mobility behaviors over
a longer duration. This can inform potential future traffic flows and offer
reasonable synthetic trajectories for mobility simulation. The long-term
generation is achieved through multi-step predictions by the model. Maximum
probability-based sampling was employed for both baselines and PMT.

Table 5 presents the results of 8-hour (9 am-5 pm) and 24-hour (9 am-9 am)
trajectory generation. We utilized a modified n-gram precision [49] (see
Appendix D) at the individual level to assess the similarity between generated
trajectories and actual trajectories, where a higher n-gram precision
indicates better representation of user stay and movement behaviors across
different areas. Additionally, we aggregated the generated and actual
trajectories, computing discrepancies in the number of individuals locating in
different CBGs. A smaller aggregated level error indicates a better
representation of human macroscopic distribution across the city. An
interesting observation is that, despite PMT-1.6M performing well at the
individual level, it fails to accurately replicate the aggregated level
population distribution. However, with an increase in model parameters,
PMT-550M and PMT-21M significantly outperform the baselines both at the
individual and aggregated levels. This suggests that larger PMT models can
more effectively capture the temporal characteristics of human mobility.

Table 5: Performance of generation. A modified n-gram precision score is used for individual-level evaluation and MAPE is calculated only for the most populous 25% of CBGs at each time interval. The best result is bolded, and the second best is underlined in the table. | Model | Individual level | Aggregated level
---|---|---|---
| 8 hour | 24 hour | 8 hour | 24 hour
| 2-gram | 3-gram | 2-gram | 3-gram | RMSE | MAPE | RMSE | MAPE
BOS | LSTM | 0.532 | 0.481 | 0.448 | 0.415 | 2.72 | 29.87% | 4.18 | 45.78%
DeepMove | 0.538 | 0.479 | 0.622 | 0.582 | 2.27 | 27.39% | 3.21 | 30.93%
PMT-1.6M | 0.634 | 0.570 | 0.765 | 0.720 | 2.91 | 26.80% | 5.11 | 20.66%
PMT-21M | 0.642 | 0.578 | 0.766 | 0.721 | 2.21 | 24.52% | 2.36 | 18.40%
PMT-550M | 0.646 | 0.581 | 0.763 | 0.718 | 2.10 | 25.29% | 2.10 | 19.73%
LA | LSTM | 0.528 | 0.478 | 0.479 | 0.442 | 4.52 | 25.07% | 7.72 | 38.62%
DeepMove | 0.545 | 0.494 | 0.622 | 0.582 | 3.89 | 23.30% | 5.25 | 26.28%
PMT-1.6M | 0.636 | 0.576 | 0.754 | 0.707 | 4.36 | 22.43% | 8.02 | 20.66%
PMT-21M | 0.657 | 0.594 | 0.764 | 0.715 | 3.37 | 20.71% | 3.40 | 15.61%
PMT-550M | 0.651 | 0.589 | 0.764 | 0.715 | 3.19 | 20.85% | 2.81 | 15.41%
NYC | LSTM | 0.489 | 0.442 | 0.466 | 0.431 | 5.98 | 28.89% | 7.96 | 42.01%
DeepMove | 0.558 | 0.507 | 0.583 | 0.545 | 3.89 | 21.63% | 6.79 | 29.70%
PMT-1.6M | 0.616 | 0.549 | 0.736 | 0.690 | 6.66 | 24.11% | 13.95 | 19.56%
PMT-21M | 0.659 | 0.586 | 0.772 | 0.724 | 3.36 | 20.26% | 3.27 | 14.75%
PMT-550M | 0.653 | 0.581 | 0.768 | 0.721 | 3.31 | 20.16% | 3.23 | 14.82%

##  6 Discussion

This study introduces the PMT, a model designed to decode intricate patterns
of human mobility using extensive LBS data. Through experiments conducted in
three U.S. metropolitan areas, we found that PMT is able to not only predict
and complete trajectory sequences accurately but also capture geographic and
socio-demographic attributes. PMT excels across various downstream tasks,
underscoring its effectiveness in modeling human mobility and urban spaces.
This work represents a significant advancement in leveraging data-driven
approaches to unravel the complexities of urban spatial functionality and
individual mobility preferences, paving the way for further insights and
explorations in urban planning, transportation, mobility, and related domains.

Despite the contribution, two concerns remain regarding the human mobility
foundation model. The first one is the issue of sample imbalance. Although the
sampling rate of LBS data is relatively high, it originates from mobile
devices. Elderly people and the extremely impoverished who do not use smart
devices may be overlooked in the data. Policies informed by the model could be
unfair to these populations. The second one is the concern about user privacy.
Even though we control the trajectory resolution at the CBG level, we are
concerned that by fine-tuning with known personal information of a few users,
PMT could accurately infer private details such as gender, age, race, and
income based on users’ long-term trajectories.

## References

  * [1] Hugo Barbosa, Marc Barthelemy, Gourab Ghoshal, Charlotte R James, Maxime Lenormand, Thomas Louail, Ronaldo Menezes, José J Ramasco, Filippo Simini, and Marcello Tomasini.  Human mobility: Models and applications.  Physics Reports, 734:1–74, 2018.
  * [2] Marta C Gonzalez, Cesar A Hidalgo, and Albert-Laszlo Barabasi.  Understanding individual human mobility patterns.  nature, 453(7196):779–782, 2008.
  * [3] Qi Wang, Nolan Edward Phillips, Mario L Small, and Robert J Sampson.  Urban mobility and neighborhood isolation in america’s 50 largest cities.  Proceedings of the National Academy of Sciences, 115(30):7735–7740, 2018.
  * [4] Dirk Brockmann, Lars Hufnagel, and Theo Geisel.  The scaling laws of human travel.  Nature, 439(7075):462–465, 2006.
  * [5] Chaoming Song, Tal Koren, Pu Wang, and Albert-László Barabási.  Modelling the scaling properties of human mobility.  Nature physics, 6(10):818–823, 2010.
  * [6] Luca Pappalardo, Filippo Simini, Salvatore Rinzivillo, Dino Pedreschi, Fosca Giannotti, and Albert-László Barabási.  Returners and explorers dichotomy in human mobility.  Nature communications, 6(1):8166, 2015.
  * [7] Luca Pappalardo, Salvatore Rinzivillo, and Filippo Simini.  Human mobility modelling: exploration and preferential return meet the gravity model.  Procedia Computer Science, 83:934–939, 2016.
  * [8] Giuliano Cornacchia, Giulio Rossetti, and Luca Pappalardo.  Modelling human mobility considering spatial, temporal and social dimensions.  arXiv preprint arXiv:2007.02371, 2020.
  * [9] Jie Feng, Yong Li, Chao Zhang, Funing Sun, Fanchao Meng, Ang Guo, and Depeng Jin.  Deepmove: Predicting human mobility with attentional recurrent networks.  In Proceedings of the 2018 world wide web conference, pages 1459–1468, 2018.
  * [10] Zain Ul Abideen, Heli Sun, Zhou Yang, Rana Zeeshan Ahmad, Adnan Iftekhar, and Amir Ali.  Deep wide spatial-temporal based transformer networks modeling for the next destination according to the taxi driver behavior prediction.  Applied Sciences, 11(1):17, 2020.
  * [11] Dejiang Kong and Fei Wu.  Hst-lstm: A hierarchical spatial-temporal long-short term memory network for location prediction.  In IJCAI, volume 18, pages 2341–2347, 2018.
  * [12] Weizhen Dang, Haibo Wang, Shirui Pan, Pei Zhang, Chuan Zhou, Xin Chen, and Jilong Wang.  Predicting human mobility via graph convolutional dual-attentive networks.  In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining, pages 192–200, 2022.
  * [13] Guangshuo Chen, Aline Carneiro Viana, Marco Fiore, and Carlos Sarraute.  Complete trajectory reconstruction from sparse mobile phone data.  EPJ Data Science, 8(1):1–24, 2019.
  * [14] Gang Liu and Jukka-Pekka Onnela.  Bidirectional imputation of spatial gps trajectories with missingness using sparse online gaussian process.  Journal of the American Medical Informatics Association, 28(8):1777–1784, 2021.
  * [15] Keivin Isufaj, Mohamed Mokhtar Elshrif, Sofiane Abbar, and Mohamed Mokbel.  Gti: A scalable graph-based trajectory imputation.  In Proceedings of the 31st ACM International Conference on Advances in Geographic Information Systems, pages 1–10, 2023.
  * [16] Xi Liu, Hanzhou Chen, and Clio Andris.  trajgans: Using generative adversarial networks for geo-privacy protection of trajectory data (vision paper).  In Location privacy and security workshop, pages 1–7, 2018.
  * [17] Xingrui Wang, Xinyu Liu, Ziteng Lu, and Hanfang Yang.  Large scale gps trajectory generation using map based on two stage gan.  Journal of Data Science, 19(1):126–141, 2021.
  * [18] Haoyu He, Xinhua Wu, and Qi Wang.  Forecasting urban mobility using sparse data: A gradient boosted fusion tree approach.  In Proceedings of the 1st International Workshop on the Human Mobility Prediction Challenge, pages 41–46, 2023.
  * [19] Renhe Jiang, Zekun Cai, Zhaonan Wang, Chuang Yang, Zipei Fan, Quanjun Chen, Kota Tsubouchi, Xuan Song, and Ryosuke Shibasaki.  Deepcrowd: A deep model for large-scale citywide crowd density and flow prediction.  IEEE Transactions on Knowledge and Data Engineering, 35(1):276–290, 2021.
  * [20] Senzhang Wang, Jiannong Cao, Hao Chen, Hao Peng, and Zhiqiu Huang.  Seqst-gan: Seq2seq generative adversarial nets for multi-step urban crowd flow prediction.  ACM Transactions on Spatial Algorithms and Systems (TSAS), 6(4):1–24, 2020.
  * [21] Huaxiu Yao, Xianfeng Tang, Hua Wei, Guanjie Zheng, and Zhenhui Li.  Revisiting spatial-temporal similarity: A deep learning framework for traffic prediction.  In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 5668–5675, 2019.
  * [22] Xin Yao, Yong Gao, Di Zhu, Ed Manley, Jiaoe Wang, and Yu Liu.  Spatial origin-destination flow imputation using graph convolutional networks.  IEEE Transactions on Intelligent Transportation Systems, 22(12):7474–7484, 2020.
  * [23] Filippo Simini, Gianni Barlacchi, Massimiliano Luca, and Luca Pappalardo.  Deep gravity: enhancing mobility flows generation with deep neural networks and geographic information.  arXiv preprint arXiv:2012.00489, 2020.
  * [24] Bowen Du, Hao Peng, Senzhang Wang, Md Zakirul Alam Bhuiyan, Lihong Wang, Qiran Gong, Lin Liu, and Jing Li.  Deep irregular convolutional residual lstm for urban traffic passenger flows prediction.  IEEE Transactions on Intelligent Transportation Systems, 21(3):972–985, 2019.
  * [25] Junkai Sun, Junbo Zhang, Qiaofei Li, Xiuwen Yi, Yuxuan Liang, and Yu Zheng.  Predicting citywide crowd flows in irregular regions using multi-view graph convolutional networks.  IEEE Transactions on Knowledge and Data Engineering, 34(5):2348–2359, 2020.
  * [26] Hugo Barbosa, Fernando B de Lima-Neto, Alexandre Evsukoff, and Ronaldo Menezes.  The effect of recency to human mobility.  EPJ Data Science, 4:1–14, 2015.
  * [27] Haoyu He, Hengfang Deng, Qi Wang, and Jianxi Gao.  Percolation of temporal hierarchical mobility networks during covid-19.  Philosophical Transactions of the Royal Society A, 380(2214):20210116, 2022.
  * [28] Yilan Cui, Xing Xie, and Yi Liu.  Social media and mobility landscape: Uncovering spatial patterns of urban human mobility with multi source data.  Frontiers of Environmental Science & Engineering, 12:1–14, 2018.
  * [29] Qiang Gao, Fan Zhou, Goce Trajcevski, Kunpeng Zhang, Ting Zhong, and Fengli Zhang.  Predicting human mobility via variational attention.  In The world wide web conference, pages 2750–2756, 2019.
  * [30] Song Gao, Krzysztof Janowicz, and Helen Couclelis.  Extracting urban functional regions from points of interest and human activities on location-based social networks.  Transactions in GIS, 21(3):446–467, 2017.
  * [31] Yandong Wang, Yanyan Gu, Mingxuan Dou, and Mengling Qiao.  Using spatial semantics and interactions to identify urban functional regions.  ISPRS International Journal of Geo-Information, 7(4):130, 2018.
  * [32] Jing Yuan, Yu Zheng, and Xing Xie.  Discovering regions of different functions in a city using human mobility and pois.  In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 186–194, 2012.
  * [33] Yoshua Bengio, Aaron Courville, and Pascal Vincent.  Representation learning: A review and new perspectives.  IEEE transactions on pattern analysis and machine intelligence, 35(8):1798–1828, 2013.
  * [34] Yiwei Song, Dongzhe Jiang, Yunhuai Liu, Zhou Qin, Chang Tan, and Desheng Zhang.  Hermas: a human mobility embedding framework with large-scale cellular signaling data.  Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 5(3):1–21, 2021.
  * [35] Tao-Yang Fu and Wang-Chien Lee.  Trembr: Exploring road networks for trajectory representation learning.  ACM Transactions on Intelligent Systems and Technology (TIST), 11(1):1–25, 2020.
  * [36] Kang Liu, Song Gao, Peiyuan Qiu, Xiliang Liu, Bo Yan, and Feng Lu.  Road2vec: Measuring traffic interactions in urban road system from massive travel routes.  ISPRS International Journal of Geo-Information, 6(11):321, 2017.
  * [37] Yao Yao, Xia Li, Xiaoping Liu, Penghua Liu, Zhaotang Liang, Jinbao Zhang, and Ke Mai.  Sensing spatial distribution of urban land use by integrating points-of-interest and google word2vec model.  International Journal of Geographical Information Science, 31(4):825–848, 2017.
  * [38] Bo Yan, Krzysztof Janowicz, Gengchen Mai, and Song Gao.  From itdl to place2vec: Reasoning about place type similarity and relatedness by learning embeddings from augmented spatial contexts.  In Proceedings of the 25th ACM SIGSPATIAL international conference on advances in geographic information systems, pages 1–10, 2017.
  * [39] Wei Zhai, Xueyin Bai, Yu Shi, Yu Han, Zhong-Ren Peng, and Chaolin Gu.  Beyond word2vec: An approach for urban functional region extraction and identification by combining place2vec and pois.  Computers, environment and urban systems, 74:1–12, 2019.
  * [40] Zhicheng Liu, Fabio Miranda, Weiting Xiong, Junyan Yang, Qiao Wang, and Claudio Silva.  Learning geo-contextual embeddings for commuting flow prediction.  In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 808–816, 2020.
  * [41] Qiang Gao, Fan Zhou, Kunpeng Zhang, Goce Trajcevski, Xucheng Luo, and Fengli Zhang.  Identifying human mobility via trajectory embeddings.  In IJCAI, volume 17, pages 1689–1695, 2017.
  * [42] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean.  Efficient estimation of word representations in vector space.  arXiv preprint arXiv:1301.3781, 2013.
  * [43] Aditya Grover and Jure Leskovec.  node2vec: Scalable feature learning for networks.  In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining, pages 855–864, 2016.
  * [44] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [45] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  Advances in neural information processing systems, 33:1877–1901, 2020.
  * [46] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [47] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al.  Segment anything.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015–4026, 2023.
  * [48] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al.  Improving language understanding by generative pre-training.  2018\.
  * [49] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.  Bleu: a method for automatic evaluation of machine translation.  In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318, 2002.
  * [50] Rafael Müller, Simon Kornblith, and Geoffrey E Hinton.  When does label smoothing help?  Advances in neural information processing systems, 32, 2019.
  * [51] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.  Dropout: a simple way to prevent neural networks from overfitting.  The journal of machine learning research, 15(1):1929–1958, 2014.
  * [52] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [53] Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  Neural computation, 9(8):1735–1780, 1997.

##  Appendix A Location-based service data

Figure 5: A synthetic example of LBS data. It is important to note that, in
order to protect user privacy, the latitude and longitude of a user’s primary
addresses (such as home or workplace) will be shifted to the centroid of the
corresponding CBG.

##  Appendix B Temporal Encoding

Figure 6: A temporal encoding of 256 dimensions. The leftmost 128 dimensions
represent absolute time encoding, dimensions from 129 to 192 in the middle
represent day encoding, and the rightmost 64 dimensions represent week
encoding.

##  Appendix C Training Details

We trained our models on one machine with 4 NVIDIA A100 @80G GPUs. In our
PMT-550M architecture, each training step required approximately 3.5 seconds
for a batch size of 16 and 6.0 seconds for a batch size of 32. To mitigate
overfitting, we employed label smoothing [50] and dropout [51] regularization
techniques. The Adam optimizer [52] was used with the following parameters:
$\beta_{1}=0.9$, $\beta_{2}=0.99$, and $\epsilon=10^{-9}$. A dynamic learning
rate similar to [46] is used over the training, according to the following
formula, with the maximum upper bound of the learning rate set to 1e-4:

$\text{lr}=\frac{1}{\sqrt{d_{\text{Embedding}}}}\times\min\left(\frac{1}{\sqrt{\text{step}}},\frac{\text{step}\times\text{warmup\\_steps}^{-1.5}}{\sqrt{\text{warmup\\_steps}}}\right)\times
c$

where $d_{\text{Embedding}}$ is the embedding dimension of the model,
warmup_steps is the number of warm-up steps and $c$ is an additional
multiplier coefficient.

The hyperparameters during the training process are shown in Table 6. We
acknowledge that due to limited computational resources, the hyperparameters
are not optimized, especially for the largest PMT-550M model, where
significant convergence had not been observed. There is still room for
improvement in model performance on the same dataset and architecture.

Table 6: Hyperparameters for training. The format "*/*" denotes the hyperparameters for the next location prediction task and mask imputation task, respectively. | Model | Label smoothing | Dropout | Warmup steps | $c$ | Batch Size | Epoch
---|---|---|---|---|---|---|---
BOS | PMT-1.6M | 0.1 | 0.1 | 1,000/1,000 | 1/1 | 32/16 | 20/20
PMT-21M | 0.1 | 0.1 | 1,000/1,000 | 1/1 | 32/16 | 40/20
PMT-550M | 0.1 | 0.3 | 5,000/5,000 | 1/1 | 32/16 | 40/20
LA | PMT-1.6M | 0.1 | 0.1 | 1,000/1,000 | 1/1 | 32/16 | 20/10
PMT-21M | 0.1 | 0.1 | 1,000/1,000 | 1/1 | 32/16 | 20/10
PMT-550M | 0.1 | 0.3 | 20,000/2,000 | 1/0.05 | 32/16 | 20/10
NYC | PMT-1.6M | 0.1 | 0.1 | 1,000/1,000 | 1/0.01 | 32/16 | 20/10
PMT-21M | 0.1 | 0.1 | 1,000/1,000 | 1/1 | 32/16 | 20/10
PMT-550M | 0.1 | 0.3 | 40,000/2,000 | 1/0.05 | 32/16 | 15/4

##  Appendix D Evaluation

###  D.1 Baselines

In this section, we describe the baseline models used for various tasks. We
detail their hyperparameters in Table 7.

Frequency-based Model (FBM). This method follows a simple logic, where users
are most likely to appear where they most frequently visit. In its
implementation, we calculate the locations where users most frequently appear
during daytime and nighttime, and provide the top k predictions based on
frequency. The frequency of users appearing at different locations is computed
based on data from the entire study period.

Long Short-Term Memory (LSTM). LSTM [53] is a typical recurrent neural network
for sequence prediction problems. The unidirectional nature of LSTM models
limits the utilization of future information. Therefore, for imputation tasks,
we used a bidirectional LSTM (BiLSTM) for comparison.

DeepMove. DeepMove [9] is a model designed for mobility prediction. The model
leverages historical and recent user mobility patterns using the combination
of gated recurrent units (GRUs) and an attention module. To extend the
capability of this model in imputation tasks, we employed bidirectional GRU
layers. This modified model is referred as BiDeepMove.

Table 7: Hyperparameters of baseline models. Model | Embedding Size | Hidden Size | RNN Layers | Total Parameters
---|---|---|---|---
Location | Time | User
LSTM | 128 | - | - | 256 | 2 | 7M
BiLSTM | 128 | - | - | 256 | 2 | 12M
DeepMove | 512 | 512 | 64 | 1024 | 3 | 60M
BiDeepMove | 512 | 512 | 64 | 1024 | 3 | 130M

###  D.2 Metrics for generation evaluation

Modified n-gram Precision. Given the generated trajectory sequence
$\hat{X}=\\{\hat{x}_{1},\hat{x}_{2},\ldots,\hat{x}_{T}\\}$ and the reference
sequence $X=\\{x_{1},x_{2},\ldots,x_{T}\\}$. The modified n-gram precision
function $p_{n}(\hat{X};X)$ is defined as:

$p_{n}(\hat{X};X)=\frac{\sum_{s\in
G_{n}(X)}\min(C(s,\hat{X}),C(s,X))}{\sum_{s\in G_{n}(X)}C(s,X)}$

Where $G_{n}(X)$ is the set of n-grams of length $n$ in sequence $X$. Any
n-grams containing unknown locations will not be included, e.g., for
$X=\\{a,b,[\text{nan}],b,c,c,c\\}$, $G_{2}(X)=\\{ab,bc,cc,cc\\}$,
$G_{3}(X)=\\{bcc,ccc\\}$. $C(s,X)$ is count of n-gram $s$ in $G_{n}(X)$.

Aggregated Level Metrics. Given the generated trajectory sequence
$\hat{X}=\\{\hat{x}_{1},\hat{x}_{2},\ldots,\hat{x}_{T}\\}$ and the
corresponding reference sequence $X=\\{x_{1},x_{2},\ldots,x_{T}\\}$, we only
consider $T^{\prime}$ for aggregation, where $X_{t}\neq[\text{nan}]$ for all
$t\in T^{\prime}$. This ensures that the sum of aggregations is identical
between the generated and reference trajectories.

While root mean square error (RMSE) is computed for all areas, mean absolute
percentage error (MAPE) is calculated for the top 25% of CBGs with the highest
population. This strategy mitigates potential distortions in MAPE caused by
extremely low ground truth values.

##  Appendix E Supplementary Results

Table 8: Performance of the imputation task. The format "*/*" denotes the cross entropy (no label smoothing) for the training set (included in pretraining) and the testing set (excluded in pretraining), respectively. | Model | Removal rate
---|---|---
| 50% | 60% | 70% | 80% | 90%
BOS | PMT-1.6M | 0.72/0.75 | 0.76/0.79 | 0.82/0.85 | 0.91/0.95 | 1.12/1.16
PMT-21M | 0.62/0.64 | 0.66/0.68 | 0.72/0.75 | 0.81/0.84 | 1.00/1.03
PMT-550M |  0.57/0.59 |  0.61/0.64 |  0.67/0.70 |  0.77/0.80 |  0.93/0.96
LA | PMT-1.6M | 1.15/1.20 | 1.20/1.25 | 1.27/1.32 | 1.40/1.44 | 1.62/1.67
PMT-21M | 0.72/0.73 | 0.77/0.78 | 0.84/0.86 | 0.96/0.97 | 1.17/1.18
PMT-550M |  0.70/0.71 |  0.75/0.76 |  0.82/0.84 |  0.92/0.94 |  1.10/1.13
NYC | PMT-1.6M | 1.70/1.73 | 1.73/1.76 | 1.78/1.81 | 1.86/1.89 | 2.02/2.06
PMT-21M | 0.75/0.76 | 0.80/0.81 | 0.87/0.88 | 0.99/1.00 | 1.20/1.22
PMT-550M |  0.66/0.67 |  0.71/0.72 |  0.77/0.79 |  0.87/0.89 |  1.05/1.07