# Towards a Knowledge guided Multimodal Foundation Model for Spatio-Temporal
Remote Sensing Applications

Praveen Ravirathinam  University of Minnesota, Twin CitiesMinneapolisUnited
States<EMAIL_ADDRESS>,  Ankush Khandelwal  University of Minnesota, Twin
CitiesMinneapolisUnited States<EMAIL_ADDRESS>,  Rahul Ghosh  University of
Minnesota, Twin CitiesMinneapolisUnited States<EMAIL_ADDRESS>and  Vipin
Kumar  University of Minnesota, Twin CitiesMinneapolisUnited States
<EMAIL_ADDRESS>
(2018)

###### Abstract.

In recent years, there is increased interest in foundation models for
geoscience due to vast amount of earth observing satellite imagery. Existing
remote sensing foundation models make use of the various sources of spectral
imagery to create large models pretrained on masked reconstruction task. The
embeddings from these foundation models are then used for various downstream
remote sensing applications. In this paper we propose a foundational modeling
framework for remote sensing geoscience applications, that goes beyond these
traditional single modality masked autoencoder family of foundation models.
This framework leverages the knowledge guided principles that the spectral
imagery captures the impact of the physical drivers on the environmental
system, and that the relationship between them is governed by the
characteristics of the system. Specifically, our method, called MultiModal
Variable Step Forecasting (MM-VSF), uses mutlimodal data (spectral imagery and
weather) as its input and a variable step forecasting task as its pretraining
objective. In our evaluation we show forecasting of satellite imagery using
weather can be used as an effective pretraining task for foundation models. We
further show the effectiveness of the embeddings from MM-VSF on the downstream
task of pixel wise crop mapping, when compared with a model trained in the
traditional setting of single modality input and masked reconstruction based
pretraining.

Foundation model, Spatiotemporal modelling, Remote Sensing, Knowledge guided

††copyright: acmlicensed††journalyear: 2018††doi: XXXXXXX.XXXXXXX

##  1\. Introduction

Figure 1. Abstract Representation of an Environmental System

Increased availability and ease of access to large scale satellite data has
motivated the development of deep learning models that use this data to
perform tasks such as land cover mapping(Ghosh et al., 2021; Ravirathinam et
al., 2024; Kussul et al., 2017), wildfire mapping (Nayak et al., 2018; Seydi
et al., 2022; Zhao et al., 2018), crop yield prediction (Kuwata and Shibasaki,
2015; You et al., 2017), flood forecasting(Bentivoglio et al., 2022) etc. In
more recent times, methods have been developed to use large amounts of data in
a self supervised fashion to pretrain their model weights using a pretraining
task, such as masked reconstrcution (He et al., 2022), which would then be
finetuned for downstream tasks. Such models, called foundation models, have
been shown to generalise their embeddings over various tasks, called
downstream tasks, in the image (Yuan et al., 2021; Singh et al., 2022) and
text domains(Touvron et al., 2023) (e.g. large language model such as GPT
(Achiam et al., 2023)). Motivated by the success of such models, there is a
huge interest in building foundation models for geoscience
applications(Jakubik et al., 2023; Cha et al., 2023; Khanna et al., 2023; Mall
et al., 2023; Deng et al., 2024; Hong et al., 2024; Guo et al., 2024; Mai et
al., 2022; Bastani et al., 2023). Most geoscience foundation models can be
placed into one of two groups based on the data they use : (1) weather-climate
(Nguyen et al., 2023; Pathak et al., 2022) that are typically used for weather
forecasting or climate modeling and (2) spectral data from remote sensing
satellites (Cong et al., 2022; Jakubik et al., 2023) that are largely used for
identifying land-use land-cover change dynamics. The focus of this paper is on
the land-use land cover applications that are typically handled by remote
sensing foundation models. Such remote sensing foundation models are typically
built using the vast amounts of spectral data from various satellites
available using a pretraining task such as masked reconstruction (Cong et al.,
2022; Gao et al., 2022; Cha et al., 2023; Mendieta et al., 2023; Liu et al.,
2024; Sun et al., 2022). However, these methods do not account for
interrelationships that exist between different components of the
environmental systems.

If one is to envision the environment as a system, it can be thought of as
various physical drivers such as weather, moisture, etc. act upon a region
(e.g. a farm, catchment) that has physical properties such as land cover type,
soil characteristics , etc. to result in a response such as crop growth,
streamflow, emissions which can be observed by sensors and satellites such as
Sentinel, Landsat etc. This broad picture is captured in Figure 1 and is
called the forward process. However, if one is interested in properties such
as land cover type, the inverse process can be used to infer these values
given the physical drivers, sensor observations, and possibly the system
response(Ravirathinam et al., 2024). To the best of our knowledge, most
existing remote sensing foundation models currently use only the spectral
imagery in their pretraining as well as downstream tasks (Jakubik et al.,
2023; Cong et al., 2022), thus leaving out the key information that the
spectral imagery captures the impact of the physical drivers on the
environmental system, and the relationship between them (and other observables
such as crop yield or stream flow) is governed by the characteristics of the
system (e.g., land cover type, crop management). In this work, we show how to
realise this vision in both the pretraining and downstream task. Our proposed
approach differs from existing work along several dimensions as outlined
below.

Remote sensing based foundation models are usually built using one modality of
data, which is typically spectral imagery (Wang et al., 2022; Cha et al.,
2023; Sun et al., 2022). However, from Figure 1, we can see that various
components interact with each other leading to interesting phenonmenons. For
example, weather acts on a crop field which leads to crop growth, which is
seen by satellite imagery. Such examples highlight the need to capture
interactions between physical components and drivers in the foundation model,
which are not included in the pretraining tasks of existing geoscience models.
In our approach, we include both spectral imagery and weather in our
pretraining task to ensure that the interaction between these components is
captured in the embeddings of our model. In particular, we show how to include
weather in the pretraining task of forecasting and how its inclusion benefits
the downstream task of pixel wise crop mapping.

Majority of remote sensing based foundation model methods use masked
reconstruction as their pretraining task (Jakubik et al., 2023; Tseng et al.,
2023). However, by leveraging the scientific knowledge about the
interrelationships between different components of the environmental system,
we can go beyond this pretraining task. In particular, using the forward
process, we can see forecasting of satellite imagery as a potential
pretraining task. Without the use of weather, forecasting of satellite imagery
would not be very effective, but with its inclusion it may be possible the
model would be able to use this weather data to effectively forecast an image,
and in the process create much more meaningful embeddings as compared to
reconstruction. In this work, we present a methodology for the pretraining
task of satellite image forecasting with weather and show how its improves the
embeddings and in turn the performance on downstream tasks.

Another major feature of remote sensing satellite data is it’s spatio temporal
nature. Most of the remote-sensing downstream tasks (crop classification,
land-use land cover change detection, crop yield prediction) require such a
spatiotemporal input, however their required time-scales may vary. For example
crop mapping requires a much longer time scale than land-use land cover (LULC)
change detection. Most foundation models built for geoscience, either do not
account for the temporal nature of satellite data or are limited to processing
the temporal data of a fixed length (Jakubik et al., 2023). For a unified
foundation model it is important to not only include mechanisms that take
advantage of the temporal nature of remote sensing data in their approach, but
also have temporal flexibility in the length of its input, especially for
generalizing across various downstream tasks.

In this paper, we present a novel spatio temporal multimodal foundation model
that uses knowledge guided concepts to strengthen its embeddings. We call our
approach MultiModal Variable step Forecasting (MM-VSF), that uses both
satellite and weather data (two entirely different modalities) to perform the
pretraining task of forecasting (variable forecasting) and show how the
embeddings created by our approach are richer when compared to those created
using the task of reconstruction. We also present a temporally flexible
architecture that efficiently makes use of the temporal nature of both
satellite and weather data to give results that go beyond simple temporal
autocorrelation of satellite data. We show how the embeddings created by our
pretraining method generalise better for the downstream task of crop mapping
when compared to other pretraining methods.

##  2\. Method

Like most existing foundational models for geoscience, our architecture also
follows a heavy encoder and lightweight decoder format. Keeping our decoder
lightweight forces the embedding created by the encoder to be rich enough to
solve the pretraining task and in turn would be rich for various downstream
tasks. Unlike previous methodologies we incorporate multiple modalities into
our pretraining, namely spectral imagery and weather.

###  2.1. Pretraining Task

The most common pretraining task in geoscience has been reconstruction of
spectral imagery. To enrich the embedding created, typically varying amounts
of the input spectral image are masked, making the reconstruction task of the
entire image harder leading to better embeddings. However, simple
reconstruction embeddings capture just that particular image and might not be
suited for downstream tasks that rely on multi temporal contexts such as crop
mapping or land cover land use change. To solve this, previous works included
multiple timestamps in their input, however most of these methods stacked
these images together (Jakubik et al., 2023), thus removing the temporal
aspect. However, some methods added a timestamp positional embedding so that
the model has a sense of time (Khanna et al., 2023). This led to moderate
success in handling downstream tasks that require multi temporal contexts.

Another common pretraining task is forecasting of imagery. Typically, this
pretraining task has been used in weather related foundation models and not in
spectral imagery based foundation models. Foundation models created using
masked forecasting has shown great success in weather related downstream tasks
(Nguyen et al., 2023; Pathak et al., 2022). To enrich the embeddings created,
these works add variable future time forecasting, i.e vary the amount of time
into the future the model needs to forecast to. This pretraining task using
just spectral imagery would not make much sense, as it would be very
challenging to predict how an image would look in the future based on just one
image and a future time. At the minimum we would need more information such as
geographic location, time of year, country etc.

In our work, we combine these pretraining tasks in a unique multimodal
fashion. From Figure 1, we have seen that various components of the
environment interact with each other. Additionally, it has been shown in the
past that inclusion of weather has led to improved results in downstream
geoscience tasks (Ravirathinam et al., 2024). Thus we propose to include
weather in the pretraining task. One simple strategy is to add weather as an
additional modality to reconstruct in the pretraining task. However, this
method does not infuse weather dynamics into the embeddings. That is why we
propose using multimodal forecasting as a pretraining task. Our idea is to use
the current satellite imagery and provide weather data till a particular day
on which we would ask the model to predict the satellite image at that point
in the future. Asking the model to predict a satellite image in the future
without any weather information would be sub-optimal, but providing weather
can help the model narrow down the search space in figuring out how the land
cover would change. For example, if there was a lot of rain, the lakes would
be fuller, if there was a lot of sunlight then growth of crop/other vegetation
would accelerate. As a result, pretraining in this future image prediction
scheme would infuse land growth/change dynamics into the embeddings that the
model creates. Our hypothesis is that this extra knowledge infusion would help
greatly in downstream tasks that rely on land growth and change dynamics such
as crop prediction, land cover land use change, etc.

To summarise, our proposed pretraining task is to predict a spectral image in
the future (response) using a series of spectral images in the past (context)
along with the weather till the future date (query) we want to predict. We
call this pretraining task as Variable step Forecasting ($VSF$), where the
model is expected to forecast k steps into the future. During pre-training, k
is chosen randomly for different instances. During inference, k is specified
by the user. We also mask out patches in the spectral imagery to make the
prediction harder and lead to better embeddings. We will describe our masking
strategy for pretraining in a future section.

###  2.2. Dataset

Our dataset consists of Sentinel imagery for spectral imagery and ERA5 land
analysis data for our weather component. We chose these two sources due to
their temporal resolutions and more importantly their availability globally.
Due to these factors, we randomly sampled around 10000 locations across land
areas globally. Each location is of size 128x128 Sentinel pixels and for
spectral imagery we collected all images for that region in that year. Due to
missing data and improper coverage of some regions, the number of Sentinel
samples from each region would vary. For example, regions in well covered
regions such as US, might have upto 70 image instances in a year for that
region, whereas regions like India might not be as well covered and would have
40 image instances. For each instance, we collect six bands namely (B2, B3,
B4, B8, B9, B12), bands which have shown to be the most useful in land cover
related tasks as well as have been used in other works. For each image
instance, we also collect the day of the year it came from, thus forming a
series with a length the same as the number of image instances for that
location with each element having a value between 1 and 365. We get our
weather data from ERA5 Land Daily Aggregates. This globally available data
source consists of various bands that are useful for land cover related tasks.
To keep it simple and data efficient we chose 5 bands from this dataset,
namely (temperature 2m min, temperature 2m max, total precipitation sum, $u$
component of wind 10m, $v$ component of wind 10m). The frequency of this data
is daily and it is available at a resolution of 11k meters, a very coarse
resolution compared to the Sentinel pixels. Given the coarse resolution, for
most patches in our analysis we get only one value band set per timestamp,
thus making our weather data of length 365 with 5 values per timestamp.

To summarise, for each location our data comprises of 3 main components:

  * •

Spectral Imagery Series: A series of Sentinel2 Imagery each of 6 bands and of
shape 128x128. Length of this series depends on coverage of the location.

  * •

Weather Data Series: A series of ERA5 Land data of 5 bands and of shape 1x1,
with a series length of 365 (one per day)

  * •

Day of Year Series: A series of the day of the year number for each spectral
image in the series. The length of this series is same as the spectral imagery
series.

###  2.3. Architecture

Figure 2. MultiModal Variable step Forecasting (MM-VSF) Architecture diagram

Figure 2 depicts $MM-VSF$’s architecture diagram. Since our task is
forecasting, we need an architecture that is able to capture various
modalities of the data at hand. This includes spatial and temporal modalities
of satellite data, temporal modality of weather data as well how these
modalities interact with each other.

For extracting the spatial features from the spectral imagery we use a vision
transformer (ViT). ViT have been shown to be very useful in extracting
important spatial features (He et al., 2022), even in the geoscience context
(Jakubik et al., 2023) and in the presence of high masking. ViT encoder takes
a single spectral image and converts into a patch grid of embedding size of
your choice. Now given that our input is an image series, we propose using a
shared ViT to extract spatial embedding from spectral imagery of each input
timestamp. This would lead to a robust encoder, as it has to learn how to
embed images from all timestamps in a fashion that is suitable for
forecasting. For each timestamp image, we get embeddings on the unmasked
patches of our defined dimension. We also make use of patch positional
embedding in this stage. In summary, from this stage we are left with a series
of spectral image embeddings on unmasked patches for each timestamp.

As mentioned before, due to the coarse resolution of the weather data, for
each image location, we only have one weather data value per time-stamp.
Therefore, we need only a temporal embedding method and dont need any spatial
component. For encoding our weather data we use a bidirectional LSTM. We
initially tried using a temporal transformer instead of an LSTM, but this led
to poor reconstruction and in turn poor embeddings. Even when trying to solely
reconstruct masked weather, LSTM based approaches proved to have higher
accuracy in reconstruction over the transformer based approaches. We use the
LSTM based approach to get a weather embedding for each timestamp, and then
subsample from this series the weather embeddings corresponding to the input
image timestamps, in a fashion similar to WSTATT(Ravirathinam et al., 2024).
We call this extraction and matching of embeddings temporal embedding
matching.

From the input to the model, we have a series of the day of the year values
for the spectral images in the input as well as the final forecast image day
of year. It has been shown in other work that inclusion of the day of year
information in the embedding has led to improvement of results by giving the
embedding a context of the absolute time. So to incorporate this in our
architecture, we use a shared linear layer with tanh activation that takes the
day of the year and gives an embedding of our specified dimension. So we will
have a series of embeddings of our specified dimension where each embedding
corresponds to the day of the year. We will use these embeddings later on to
provide certain components with information about day of year. Orthogonally,
using the raw day of year values, we can create another series that
corresponds the number of days in between the images, i.e the delta in
timestamps. Similar to how we created an embedding for day of year values, we
can also create a series for the delta timestamps series. Such embeddings for
delta days have been shown to be useful in forecasting tasks (Nguyen et al.,
2023). As a result, here as well we apply a linear layer followed by tanh on
the delta timestamp values to create a delta timestamps embedding series. This
embedding series will be used in the forecasting components of our
architecture to provide information about how far to forecast.

From the previous steps, we now have a two time series of embeddings, one with
each element of the series corresponding to the spatial embeddings of a
particular timestamp and the other series corresponding to the temporally
matched weather embeddings. We also have the series of embeddings for the day
of year timestamps. Since all these series are of the same length, we add them
all along the temporal dimension to create a multimodal embedding series. This
multimodal embedding series, is still however, temporally unbound, i.e there
is no temporal relation between the embeddings. So to extract the temporal
embeddings, we use a BERT style transformer, with forward only attention. We
run the temporal series analysis on embedding patches that come from the same
spatial location in the overall image. Due to our uniform spatiotemporal
masking, the number of such embedding patches across all locations should be
the same, which is dependant on the amount of masking one does. For example,
if there are 6 images in the series and the masking is 50%, then in the
temporal transformer would have 3 inputs in its series. Now across spatial
patches, the timestamps from which these embeddings come will vary. This
feature allows our temporal transformer to learn robust embeddings using the
multimodal information from the combination of the spectral image embedding
information, weather embedding information and the day of the year embedding
information. Another important aspect here is that we make sure that there
only forward attention present in the temporal transformer, i.e the temporal
embeddings created are not bidirectional in nature. This feature makes sure
there is no information leakage from future timestamp information to previous
timestamps embeddings. Therefore following this stage, we are left with a
series of embeddings
$Emb_{STW}=[Emb_{:t_{1}}^{t_{1}},\dots,Emb_{:t_{C}}^{t_{C}}]$. Any downstream
task will use this embedding series for finetuning. Depending on the task, one
can choose to all use embeddings or choose to use only the final embedding.
Another thing to remember is that while finetuning, typically no masking is
present, so there will be more features per timestamp embedding.

Now our pretraining task is forecasting, so let us now delve into the decoder
part of the foundation model. The most obvious way to do forecasting is to
take the final embedding of the series $Emb_{STW}$ and use it to forecast an
image in the future. However, we can go a step further, as since our
embeddings are constructed in a forward only method, we can use each of the
embeddings to forecast an image in its respective future (i.e in a day of the
year past the day of the year of the corresponding embedding from the input
series). So apart from choosing a day of the year in the future for the final
timestamp to forecast, we can also choose intermediate days for the rest of
the embeddings in the series to forecast. There are many ways to choose these
intermediate days, but the most straightforward and computationally least
expensive are the day of year days of the input series but one timestamp
shifted, i.e the embedding that corresponds to timestamp t1 of the input
series would be used to forecast the image at day of year timestamp t2 from
the input series, embedding of timestamp t2 would be used to forecast image at
timestamp t3 and so on. Such multiple forecasting allows for the decoder to be
more robust, as similar to encoder, the decoder also has shared weights across
timestamps. So we can get a series of future day of year numbers for our
forecast and also calculate the delta days series as well. We can use the same
tanh and Linear layers as before to get the forecast day of year embedding
series and the forecast delta days embedding series. We can also use the
forecast day of year numbers to get the weather embeddings for those
corresponding days, similar to the temporal embedding matching process in the
encoder. The logic here is that, with the current image embedding and weather
information till the day of the year to forecast to, the model can predict how
the image would look in the future. Expecting a future image to look accurate
with no information apart from day of year would be very feasible, but with
weather information until that day, it is possible to estimate how it might
look. For example, if there was a lot of rain in that period, then water
bodies might have grown, or if there was a lot of sunlight, crop growth would
have accelerated etc. All this is made possible by the inclusion of weather,
which ties back to our original hypothesis that inclusion of weather would
lead to stronger embeddings.

Following these steps we have four embedding series, which we add together, to
form the basis embedding series on which forecasting is to be done. Now on
these embeddings, we apply a series of linear layers and activations to act as
the forecaster. The point of these layers is to morph the embedding from the
initial timestamp space to the future timestamp space using the information
from the image embedding and weather data before passing the embedding to the
decoder. We found that including these layers led to great benefits in terms
of performance on pretraining task, as these layers help in giving more depth
to the embeddings.

Following this stage, we repopulate these embeddings in their respective
unmasked positions in the timestamps and zero out the masked patches to pass
to the decoder, as done by most methods in transformer based autoencoder
methods(Jakubik et al., 2023; He et al., 2022). We use a light-weight decoder
(similar to the one used in ViT) to ensure that the main focus of the model is
to create good embeddings with the encoder to capture the best information so
that even a lightweight decoder can perform the pretraining task required.
Also, in any downstream task, we would need the embeddings from the encoder
and the decoder would not be used. Similar, to the encoder, we have only one
shared vision transformer in the decoder that performs the operations on each
timestamp using the same weights. Following the vision transformer, we have a
linear layer to bring the values back to the spectral image space and reshape
the output to match the required size of the spatiotemporal stack.

###  2.4. Masking

Figure 3. Example of 50 percent spatiotemporally uniform masking on a 4x4
image timeseries of 4 timestamps

Since we are dealing with a spatiotemporal based input, we adopt to using a
spatiotemporally uniform masking method. Previous methods mainly adopt to
random masking of patches and do not account for the temporal aspect. Since
our proposed architecture has both spatial and temporal components, we needed
a masking method that is fair both spatially and temporally. In our masking
strategy, there are an equal number of masked patches per timestamp as well as
an equal number of masked patches at each spatial patch location along the
temporal axis. Figure 3 shows an example of such masking for a image series of
4 4x4 grids with 50% masking. From the Figure, we can see that in each
timestamp image there are 8 patches masked and if one focuses on a particular
patch location along the temporal dimension one can notice that 2 patches are
available. This ensures that all temporal patch series that can be created
from unmasked patches at each patch location would be of same length, which
eases the implementation of our temporal component. Also since, each timestamp
individually has the same number of unmasked patches, the shared vision
transformer will also have same number of outputs per timestamp.

###  2.5. Implementation details

For the pretraining phase, we choose an input series length 6 image, and
selected a random image after the final image as the final image to forecast.
We also use 50% spatiotemporally masking and a patch size of 8 for the vision
transformer. Model was trained till convergence on 4 A100 Nvidia GPUs using
Adam Optimizer and Mean Squared Error loss.

##  3\. Experimental Evaluation

Figure 4. Forecast based Pretraining Task comparison. 50% Masking is not
shown. Row 1 depicts a crop field with, Green arrows depict regions of growth,
and Red arrows depict regions of harvest. Note how $MM-VSF$ captures both
these phenomena better than $SM-VSF$. Row 2 depicts an example where $MM-VSF$
is able to add snowfall accurately compared to $SM-VSF$. Row 3 depicts an
example where $MM-VSF$ does not include any land cover due to terrain but $SM-
VSF$ adds false greenness

In this section we evaluate various aspects of our proposed foundational model
framework. First, we evaluate the performance on the forecasting-based
pretraining task and highlight some examples to show that our method’s
embeddings are capturing aspects beyond just encoding the image. We further
evaluate the performance of our embeddings when finetuned to the crop mapping
downstream task. We compare our results with embeddings created using the
reconstruction pretraining task.

###  3.1. Baselines

Table 1 depicts the choices for input series to architectures and choices for
pretraining tasks for models. Recall that our model’s ($\boldsymbol{MM-VSF}$)
pretraining task is k steps variable forecasting ($\boldsymbol{VSF}$) in the
future, i.e P2 from the table, and our input series is multimodal
($\boldsymbol{MM}$), both satellite data and weather data, i.e I2 from the
table. However, from the table we can see that there are three other
variations, based on input and pretraining task choices, as shown below.

  * •

$\boldsymbol{SM-MAE}:$ Single Modality Masked AutoEncoder ($SM-MAE$) with the
Input series as satellite data and the pretraining task of Masked
Reconstruction, i.e (I1, P1)

  * •

$\boldsymbol{MM-MAE}:$ MultiModal Masked AutoEncoder ($MM-MAE$) with the Input
series as satellite data and weather data and the pretraining task of Masked
Reconstruction, i.e (I2, P1)

  * •

$\boldsymbol{SM-VSF}:$ Single Modality Variable step Forecasting ($SM-VSF$)
with the Input series as Satellite data and the pretraining task of Variable
Step Forecasting, i.e (I1, P2)

Note that the above combinations can be implemented by only changing the
inputs and the loss functions, without significant architecture changes. Thus
these variations, which we call baselines, can also be considered as ablations
of different components of our model. Also note that $SM-MAE$ is closest to
the existing remote sensing based foundation models (e.g.,(Jakubik et al.,
2023; Cong et al., 2022; Sun et al., 2022)), as they also use only satellite
data and pretrain using MAE.

Table 1. Choices of input and pretraining task for the model. | Input Choices
---|---
I1 | Single Modality (Satellite data)
I2 | Multimodal (Satellite + Weather data)
| Pretraining task Choices
P1 | Masked Reconstruction (MAE)
P2 | Variable Step Forecasting (VSF)

###  3.2. Pretraining Task: Forecasting

Here we evaluate the relative utility of using both weather and spectral data
(MM) vs spectral data (SM) only on the forecasting-based pretraining task.
With the help of a few examples, we illustrate that the embedding produced by
$MM-VSF$ are more powerful than those $SM-VSF$, as $MM-VSF$ is able to capture
the dynamic relationship between weather and the changes in the physical
environment on the land captured by spectral data. Figure 4 shows a comparison
of the images from these models,i.e $SM-VSF$ and $MM-VSF$, on 3 independent
examples. Each row correspond to a sample, with the first 6 images
corresponding to the satellite component of the input series to the model, the
weather component is not shown in the image but is passed along with the
satellite component(as shown in Figure 2). Now, the output would also be a
series of 6 images, with the last image corresponding to a future day as
specified in the user input (this is shown above the image in the groundtruth
column). Note that Figure 4 only shows the final forecast image for each
method, as this is where one would expect the most impact from using weather.
Both schemes are able to construct earlier images quite well, although MM-VSF
can improve over SM-VSF in most cases. Row 1 depicts an example of a crop
field, with the final forecast image being 120 days following the 6th image in
the input series. We can see that from the groundtruth image, that harvest has
occurred in the circular fields and growth has happened in the top left corner
field. Comparing the forecasted images from $SM-VSF$ and $MM-VSF$, we can see
that $MM-VSF$ is able to capture both the harvest and the growth of the crops.
We can also see that $SM-VSF$ is not able to capture these changes. This shows
that the inclusion of weather allows the embeddings created by MM-VSF to
capture the land cover dynamics that are driven by weather. Such dynamics
cannot be captured by SM-VSF, as it only has access to the past spectral
imageries, and thus it can only capture temporal autocorrelation amongst the
spectral images. Row 2 shows another example of a crop field later in the
year, with the final forecast image being 90 days after the final image in the
input series. We can see from the groundtruth image 90 days later that snow is
present in the field, which is captured by $MM-VSF$ but not $SM-VSF$. This
illustrates the ability of MM-VSF to capture the relationship between
precipitation and temperature (i.e., precipitation during cold winter days can
fall as snow). SM-VSF forecast shows a faded green field, which shows that it
is not able to capture this relationship. One can also notice that evergreen
regions within the forecasted image of $MM-VSF$ also have less snowfall when
compared to the fields, showing that terrain information is also being
captured. This is further reflected in Row 3, where a mountainous region is
depicted and the final forecast image is 255 days following the final input
image. We can see that even after 255 days there is not much change in the
region, and this is correctly captured by our method ($MM-VSF$). $SM-VSF$
seems to add some greenness which is not present. These examples illustrate
that inclusion of weather not only helps in predicting land cover change but
also helps in identifying land cover terrain, and the model realises that some
terrains no matter the weather do not change very much. Without weather, there
is no way for the model to learn this which is reflected in the forecast image
of $SM-VSF$. We further observe that all images from the forecast of $SM-VSF$
appear blocky, inspite of long periods of training. This can be explained as
the model unable to forecast accurate images without weather information.

###  3.3. Downstream Task: Crop Mapping

Here we evaluate the performance of our proposed method ($MM-VSF$) in
producing embeddings by fine-tuning to the crop mapping downstream task. We
compare our results with embeddings created using SM-SAE, which is the closest
to the existing remote sensing foundation models.

####  3.3.1. Dataset and Region of Analysis

Figure 5. Geogrpahic location of the T11SKA Sentinel Tile and its
corresponding CDL labels. Each color in the CDL image corresponds to a land
cover class. Table 2. Comparison on downstream task of crop mapping across the
pretraining tasks. Finetuning is done only using 2018 data

| 2018 | 2019
---|---|---
Crop class | $SM-MAE$ | $MM-VSF$ | $SM-MAE$ | $MM-VSF$
Corn | 0.7268 | 0.8409 | 0.4135 | 0.5708
Cotton | 0.9663 | 0.9679 | 0.8346 | 0.9125
Winter_Wheat | 0.7655 | 0.7928 | 0.1156 | 0.1777
Tomatoes | 0.8719 | 0.888 | 0.768 | 0.7341
Grapes | 0.8608 | 0.8638 | 0.7398 | 0.7543
Almonds | 0.8073 | 0.789 | 0.3851 | 0.299
Walnut | 0.8234 | 0.822 | 0.0238 | 0.5384
Pistachio | 0.7667 | 0.7673 | 0.507 | 0.7003
Alfalfa | 0.7207 | 0.7637 | 0.6892 | 0.7057
Grass | 0.8564 | 0.8611 | 0.776 | 0.8445
Urban | 0.5761 | 0.5552 | 0.6111 | 0.6191
Average | 0.7947 | 0.8102 | 0.5331 | 0.6233

Our data for finetuning also comes from Sentinel2 and ERA5 land data. The
region of analysis for testing our downstream task is the T11SKA tile in the
California Central Valley, a region that is rich in crop cover and has been
used for crop type mapping in various other works[]. Like other works, we get
our land cover labels for this region from the Cropland Data Layer, an
annually released land cover map for the entire continuous US by the USDA. A
diagrammatic representation of the CDL labels and the geographic location of
the T11SKA tile can be seen in Figure 5. As can be seen there are many crop
classes present, making it a challenging region to map. Similar to WSTATT
(Ravirathinam et al., 2024), we adopt a grid based training method, by
splitting the entire region into train, validation and test grids. We also
follow their preprocessing steps including combination and erosion.

####  3.3.2. Crop mapping Architecture and Implementation details

Figure 6. Layout of architecture for downstream task of crop mapping

Since crop mapping is a pixel wise output, we need an architecture that gives
us a pixel wise output, in particular, we would need a decoder that takes the
embedding series given by the foundation model encoder and constructs a pixel
wise classification map using this series in a semantic segmentation fashion.
Figure 6 shows a schematic of the downstream task architecture for crop
mapping. The embedding series mentioned would correspond to the series
$Emb_{STW}$, i.e the output series of the encoder. To map this embedding
series to a pixel wise crop map, we follow an attention based approach,
similar to WSTATT(Ravirathinam et al., 2024). This strategy assigns a
weightage to each timestamp embeddings and does an aggregated sum to form a
multitemporal attention-based embedding. This multitemporal embedding is then
acted by a series of upscaling and convolution layers with activation, then
passed through an output Linear layer to form a pixel wise map.

For our input series, we chose to have 10 spectral images, biweekly from May
to Sept, and while finetuning for crop mapping there is no masking of spectral
imagery done. Note that the number of timestamps passed in the downstream task
is different from the number passed during pretraining. This highlights the
temporal flexibility of our approach. We evaluate our proposed method MM-VSF
with SM-MAE, which is the common setting in existing remote sensing based
foundation models (e.g. (Jakubik et al., 2023; Cong et al., 2022)). Both $MM-
VSF$ (ours) and $SM-MAE$ (widely adopted setting) return embedding series and
the decoder layers including the attention mechanism are finetuned, i.e. the
encoder weights to obtain the embedding series are fixed from the respective
pretraining task. Both finetuned models are trained with the same data from
only the year of 2018 with their respective best hyperparameter setting.

####  3.3.3. Performance on downstream task

Table 2 compares the performance of $MM-VSF$ and $SM-MAE$ when finetuned on
our crop mapping downstream task. We can see that when tested on test regions
in the year of training, i.e 2018, the $MM-VSF$ finetuned model moderately
beats the $SM-MAE$ finetuned model performing better in almost all classes,
however, when testing in test regions on the next year, i.e 2019, we can see a
huge improvement from our approach over $SM-MAE$. This shows the impact and
generalizability of our approach over the standard approach. The fact that our
embeddings when finetuned are good enough to generalise across years shows
that important imformation has been captured in the embedding that goes beyond
just the information contained in spectral imagery.

Note that with the minimal effort of choosing a time-frame for input series
and adding a task-specific decoder, we adapted our $MM-VSF$ framework for the
Crop-mapping downstream task. This provides empirical evidence of our
framework’s flexibility to various remote-sensing based spatiotemporal tasks.

##  4\. Conclusion

In this paper we proposed a novel multimodal spatiotmeporal foundation model,
$MM-VSF$, that uses multimodal input of satellite and weather data and a
knowledge guided pretraining task of variable step forecasting. This leads to
superior embeddings when compared with embeddings achieved by models using
single modality input and trained with standard pretraining task of
reconstruction. Our pretraining task evaluation of $MM-VSF$ forecasting
capability showed that our foundation model is able to learn aspects that go
beyond temporal autocorrelation. We showed that $MM-VSF$ can be finetuned for
a crop mapping model that is generalisable across years. Our model is
temporally flexible and can adapt to geoscience downstream tasks that include
spatiotemporal remote-sensing data. Our study is the first step towards
incorporating knowledge guided principles in pretraining tasks and adopting
multimodal approaches to improve embeddings.

###### Acknowledgements.

We acknowledge NSF Award 2313174 and NSF/NIFA NAIRI Award no. 2023-67021-39829
for funding and Minnesota Supercomputing Institute (MSI) for providing
computing resources.

## References

  * (1)
  * Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023\.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_ (2023).
  * Bastani et al. (2023) Favyen Bastani, Piper Wolters, Ritwik Gupta, Joe Ferdinando, and Aniruddha Kembhavi. 2023.  Satlaspretrain: A large-scale dataset for remote sensing image understanding. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 16772–16782.
  * Bentivoglio et al. (2022) Roberto Bentivoglio, Elvin Isufi, Sebastian Nicolaas Jonkman, and Riccardo Taormina. 2022.  Deep learning methods for flood mapping: a review of existing applications and future research directions.  _Hydrology and Earth System Sciences Discussions_ 2022 (2022), 1–50.
  * Cha et al. (2023) Keumgang Cha, Junghoon Seo, and Taekyung Lee. 2023.  A billion-scale foundation model for remote sensing images.  _arXiv preprint arXiv:2304.05215_ (2023).
  * Cong et al. (2022) Yezhen Cong, Samar Khanna, Chenlin Meng, Patrick Liu, Erik Rozi, Yutong He, Marshall Burke, David Lobell, and Stefano Ermon. 2022.  Satmae: Pre-training transformers for temporal and multi-spectral satellite imagery.  _Advances in Neural Information Processing Systems_ 35 (2022), 197–211.
  * Deng et al. (2024) Cheng Deng, Tianhang Zhang, Zhongmou He, Qiyuan Chen, Yuanyuan Shi, Yi Xu, Luoyi Fu, Weinan Zhang, Xinbing Wang, Chenghu Zhou, et al. 2024\.  K2: A foundation language model for geoscience knowledge understanding and utilization. In _Proceedings of the 17th ACM International Conference on Web Search and Data Mining_. 161–170.
  * Gao et al. (2022) Zhihan Gao, Xingjian Shi, Hao Wang, Yi Zhu, Yuyang Bernie Wang, Mu Li, and Dit-Yan Yeung. 2022.  Earthformer: Exploring space-time transformers for earth system forecasting.  _Advances in Neural Information Processing Systems_ 35 (2022), 25390–25403.
  * Ghosh et al. (2021) Rahul Ghosh, Praveen Ravirathinam, Xiaowei Jia, Chenxi Lin, Zhenong Jin, and Vipin Kumar. 2021.  Attention-augmented spatio-temporal segmentation for land cover mapping. In _2021 IEEE International Conference on Big Data (Big Data)_. IEEE, 1399–1408.
  * Guo et al. (2024) Xin Guo, Jiangwei Lao, Bo Dang, Yingying Zhang, Lei Yu, Lixiang Ru, Liheng Zhong, Ziyuan Huang, Kang Wu, Dingxiang Hu, et al. 2024\.  Skysense: A multi-modal remote sensing foundation model towards universal interpretation for earth observation imagery. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 27672–27683.
  * He et al. (2022) Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. 2022.  Masked autoencoders are scalable vision learners. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 16000–16009.
  * Hong et al. (2024) Danfeng Hong, Bing Zhang, Xuyang Li, Yuxuan Li, Chenyu Li, Jing Yao, Naoto Yokoya, Hao Li, Pedram Ghamisi, Xiuping Jia, et al. 2024\.  SpectralGPT: Spectral remote sensing foundation model.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ (2024).
  * Jakubik et al. (2023) Johannes Jakubik, Sujit Roy, C. E. Phillips, Paolo Fraccaro, Denys Godwin, Bianca Zadrozny, Daniela Szwarcman, Carlos Gomes, Gabby Nyirjesy, Blair Edwards, Daiki Kimura, Naomi Simumba, Linsong Chu, S. Karthik Mukkavilli, Devyani Lambhate, Kamal Das, Ranjini Bangalore, Dario Oliveira, Michal Muszynski, Kumar Ankur, Muthukumaran Ramasubramanian, Iksha Gurung, Sam Khallaghi, Hanxi, Li, Michael Cecil, Maryam Ahmadi, Fatemeh Kordi, Hamed Alemohammad, Manil Maskey, Raghu Ganti, Kommy Weldemariam, and Rahul Ramachandran. 2023.  Foundation Models for Generalist Geospatial Artificial Intelligence.  arXiv:2310.18660 [cs.CV] https://arxiv.org/abs/2310.18660
  * Khanna et al. (2023) Samar Khanna, Patrick Liu, Linqi Zhou, Chenlin Meng, Robin Rombach, Marshall Burke, David B Lobell, and Stefano Ermon. 2023.  Diffusionsat: A generative foundation model for satellite imagery. In _The Twelfth International Conference on Learning Representations_.
  * Kussul et al. (2017) Nataliia Kussul, Mykola Lavreniuk, Sergii Skakun, and Andrii Shelestov. 2017.  Deep learning classification of land cover and crop types using remote sensing data.  _IEEE Geoscience and Remote Sensing Letters_ 14, 5 (2017), 778–782.
  * Kuwata and Shibasaki (2015) Kentaro Kuwata and Ryosuke Shibasaki. 2015.  Estimating crop yields with deep learning and remotely sensed data. In _2015 IEEE international geoscience and remote sensing symposium (IGARSS)_. IEEE, 858–861.
  * Liu et al. (2024) Fan Liu, Delong Chen, Zhangqingyun Guan, Xiaocong Zhou, Jiale Zhu, Qiaolin Ye, Liyong Fu, and Jun Zhou. 2024.  Remoteclip: A vision language foundation model for remote sensing.  _IEEE Transactions on Geoscience and Remote Sensing_ (2024).
  * Mai et al. (2022) Gengchen Mai, Chris Cundy, Kristy Choi, Yingjie Hu, Ni Lao, and Stefano Ermon. 2022.  Towards a foundation model for geospatial artificial intelligence (vision paper). In _Proceedings of the 30th International Conference on Advances in Geographic Information Systems_. 1–4.
  * Mall et al. (2023) Utkarsh Mall, Cheng Perng Phoo, Meilin Kelsey Liu, Carl Vondrick, Bharath Hariharan, and Kavita Bala. 2023.  Remote sensing vision-language foundation models without annotations via ground remote alignment.  _arXiv preprint arXiv:2312.06960_ (2023).
  * Mendieta et al. (2023) Matías Mendieta, Boran Han, Xingjian Shi, Yi Zhu, and Chen Chen. 2023.  Towards geospatial foundation models via continual pretraining. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 16806–16816.
  * Nayak et al. (2018) Guruprasad Nayak, Varun Mithal, Xiaowei Jia, and Vipin Kumar. 2018.  Classifying multivariate time series by learning sequence-level discriminative patterns. In _Proceedings of the 2018 SIAM International Conference on Data Mining_. SIAM, 252–260.
  * Nguyen et al. (2023) Tung Nguyen, Johannes Brandstetter, Ashish Kapoor, Jayesh K Gupta, and Aditya Grover. 2023.  ClimaX: A foundation model for weather and climate.  _arXiv preprint arXiv:2301.10343_ (2023).
  * Pathak et al. (2022) Jaideep Pathak, Shashank Subramanian, Peter Harrington, Sanjeev Raja, Ashesh Chattopadhyay, Morteza Mardani, Thorsten Kurth, David Hall, Zongyi Li, Kamyar Azizzadenesheli, et al. 2022\.  Fourcastnet: A global data-driven high-resolution weather model using adaptive fourier neural operators.  _arXiv preprint arXiv:2202.11214_ (2022).
  * Ravirathinam et al. (2024) Praveen Ravirathinam, Rahul Ghosh, Ankush Khandelwal, Xiaowei Jia, David Mulla, and Vipin Kumar. 2024.  Combining Satellite and Weather Data for Crop Type Mapping: An Inverse Modelling Approach. In _Proceedings of the 2024 SIAM International Conference on Data Mining (SDM)_. SIAM, 445–453.
  * Seydi et al. (2022) Seyd Teymoor Seydi, Mahdi Hasanlou, and Jocelyn Chanussot. 2022.  Burnt-Net: Wildfire burned area mapping with single post-fire Sentinel-2 data and deep learning morphological neural network.  _Ecological Indicators_ 140 (2022), 108999.
  * Singh et al. (2022) Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. 2022.  Flava: A foundational language and vision alignment model. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 15638–15650.
  * Sun et al. (2022) Xian Sun, Peijin Wang, Wanxuan Lu, Zicong Zhu, Xiaonan Lu, Qibin He, Junxi Li, Xuee Rong, Zhujun Yang, Hao Chang, et al. 2022\.  RingMo: A remote sensing foundation model with masked image modeling.  _IEEE Transactions on Geoscience and Remote Sensing_ 61 (2022), 1–22.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023\.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ (2023).
  * Tseng et al. (2023) Gabriel Tseng, Ruben Cartuyvels, Ivan Zvonkov, Mirali Purohit, David Rolnick, and Hannah Kerner. 2023.  Lightweight, pre-trained transformers for remote sensing timeseries.  _arXiv preprint arXiv:2304.14065_ (2023).
  * Wang et al. (2022) Di Wang, Qiming Zhang, Yufei Xu, Jing Zhang, Bo Du, Dacheng Tao, and Liangpei Zhang. 2022.  Advancing plain vision transformer toward remote sensing foundation model.  _IEEE Transactions on Geoscience and Remote Sensing_ 61 (2022), 1–15.
  * You et al. (2017) Jiaxuan You, Xiaocheng Li, Melvin Low, David Lobell, and Stefano Ermon. 2017.  Deep gaussian process for crop yield prediction based on remote sensing data. In _Proceedings of the AAAI conference on artificial intelligence_ , Vol. 31.
  * Yuan et al. (2021) Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. 2021\.  Florence: A new foundation model for computer vision.  _arXiv preprint arXiv:2111.11432_ (2021).
  * Zhao et al. (2018) Yi Zhao, Jiale Ma, Xiaohui Li, and Jie Zhang. 2018.  Saliency detection and deep learning-based wildfire identification in UAV imagery.  _Sensors_ 18, 3 (2018), 712.