11institutetext: Center for Research in Computer Vision, University of Central
Florida, USA
11email<EMAIL_ADDRESS>
# GAMa: Cross-view Video Geo-localization

Shruti Vyas  Chen Chen  Mubarak Shah

###### Abstract

The existing work in cross-view geo-localization is based on images where a
ground panorama is matched to an aerial image. In this work, we focus on
ground videos instead of images which provides additional contextual cues
which are important for this task. There are no existing datasets for this
problem, therefore we propose GAMa dataset, a large-scale dataset with ground
videos and corresponding aerial images. We also propose a novel approach to
solve this problem. At clip-level, a short video clip is matched with
corresponding aerial image and is later used to get video-level geo-
localization of a long video. Moreover, we propose a hierarchical approach to
further improve the clip-level geo-localization. It is a challenging dataset,
unaligned and limited field of view, and our proposed method achieves a Top-1
recall rate of 19.4% and 45.1% @1.0mile. Code and dataset are available at
following link.

##  1 Introduction

Video geo-localization is a challenging problem with many applications such as
navigation, autonomous driving, and robotics [11, 24, 5]. The problem to
estimate geo-location of the source of a ground video is also faced by first
respondents now and then. To solve this problem, there are two main
approaches; same-view geo-localization [19, 17, 3] and cross-view geo-
localization [13, 35, 16]. In same-view geo-localization, the query ground
image is matched with a street view image from the reference set or gallery.
Research in videos is limited to same-view geo-localization where a frame-by-
frame approach is followed. This approach relies on availability of ground
view images for all the locations which may not be possible always.

In such scenarios, cross-view geo-localization is more useful where the query
image is matched with the corresponding aerial or satellite image. However,
cross-view geo-localization is a much difficult problem since there is a large
domain shift between ground and aerial view. The limited field-of-view in the
ground view makes this even harder and it is sometimes difficult even for
humans to identify the correct location of a given image or video.

Figure 1: Left: Sample ground video and aerial image pair from the proposed
GAMa dataset. The stars represent location of video frames every sec. and mark
the trajectory of the video; Right: An example of larger aerial region for a
video where stars represent the GPS points labeled every second. The GPS
points are marked as $X_{1}$, $X_{2}$, .. $X_{n}$. Range $\mu$ is derived from
these GPS points and we also show a histogram of the number of samples at a
given value of $\mu$, in training set. The red dotted lines show the lower and
and upper threshold (i.e. 0.001 and 0.004) for selecting the videos

The existing works in cross-view geo-localization follow an image based
approach where a ground image is matched with an aerial image [16, 29, 26]. In
such an approach, the contextual information available with the video is lost.
We focus on the geo-localization of ground videos instead of images to utilize
the context, i.e. how the view in one frame is related/located w.r.t. another
frame. To the best of our knowledge there are no existing datasets which are
publicly available and can be used for this problem. Thus, we have collected a
new dataset, named as GAMa (Ground-video to Aerial-image Matching), which
contains ground videos with GPS labels and corresponding aerial images. It
consists of $\sim$ 1.9M aerial images and 51K ground videos where each video
is around 40 seconds long. An example video with representative frames and
corresponding aerial image is shown in Figure 1. We will make this dataset
publicly available for further research.

We propose GAMa-Net as a benchmark method to solve this problem at clip level
where we match every 0.5 second (short clip) from a long video with the
corresponding aerial view. A frame-by-frame approach can also be used for
video geo-localization, where a 2D convolutional network is used to get the
spatial features frame-by-frame. However, following a frame-by-frame approach
ignores the rich contextual information available in a video. Considering
this, we propose a 3D-convolution based approach for learning the ground level
features from a video. The proposed approach is trained using an image-video
contrastive loss which tries to match the ground view with the aerial view.
This matching provides a clip-level geo-localization.

Next, we propose a hierarchical approach which helps in improving the clip-
level geo-localization performance while providing a video-level geo-
localization with the help of clip-level predictions. It takes the set of
aerial images corresponding to the clips of a long video, and matches them
against a larger geographical area. Therefore, it makes use of the contextual
information available with the sequence of clips corresponding to a longer
video.

We evaluate the proposed approach on GAMa dataset and demonstrate its
effectiveness for clip/video geo-localization. We provide an analysis and
propose a set of baselines to benchmark the dataset. We make the following
contributions:

  * •

A novel problem formulation i.e. cross-view video geo-localization and a
large-scale dataset, GAMa, with ground videos and corresponding aerial images.
This is the first video dataset for this problem to the best of our knowledge.

  * •

We propose GAMa-Net, which performs cross-view video geo-localization at clip-
level by matching a ground video with aerial images using an image-video
contrastive loss.

  * •

We also propose a novel hierarchical approach which provides video-level geo-
localization and utilizes aerial images at different scales to improve the
clip-level geo-localization.

##  2 Related works

Traditional features of classical computer vision were earlier used for image
matching in geo-localization [20, 33]. As deep learning has proven successful
in feature learning most of the recent studies have adopted CNN based approach
for learning discriminative features for image matching [25]. The problem of
image or video geo-localization is solved using either the same view, which is
mostly the ground view, or cross-view. Same-view Geo-localization makes use of
the large collections of geo-tagged images available online [15, 34, 19, 17,
3, 2]. The problem is approached with the assumption that there is a reference
dataset consisting of geo-tagged ground images and there is an image
corresponding to each query image. The problem is then solved as image
retrieval of the matching reference to determine the location of the query.
There is some research in video geo-localization as well which is solved at
frame level which is followed by trajectory smoothing [6, 17, 3]. However, a
more complete coverage by overhead reference data such as satellite/aerial
imagery has spurred a growing interest in cross-view geo-localization.

Cross-view Geo-localization. Most of the recent work adopt CNN based
approaches. Several studies have explored CNN architectures for matching
ground-level query images to overhead satellite images [28, 26, 36, 12, 8,
29]. Triplet loss is mostly used optimization function in these studies [8,
16]. Certain studies however report better results with contrastive loss[15].
Field-of-view (FOV) also plays an important role in deciding the recall rate
and ground panorama is highly accurate as compared to limited FOV [8, 35].
Similarly, videos also contain more visual information as seen through the
trajectory of the camera and can be expected to provide more accurate geo-
localization as compared to images or frames with similar FOV.

Current works on cross-view geolocalization follow image based approach since
the existing datasets only contain image pairs for ground and aerial view [25,
16, 9, 31, 18]. However, some papers do report testing their model on videos
using a frame-by-frame approach [9]. Most popular datasets, Cross-View USA
(CVUSA) dataset [30], CVACT [13], and Vo et al. [28] contain ground panorama
aligned with corresponding aerial image. Recent publications have shown very
high recall rate on these datasets while using panoramas however these values
are quite low when using limited FOV and unaligned images, i.e. top@1 recall
of upto 14% [22, 31]. VIGOR [35] dataset also contains panorama however being
unaligned it is more realistic. All these datasets use ground panorama which
is not realistic from video geo-localization, as videos have limited FOV,
neither do they have time series data required for such training. It is
possible to get unaligned images and limited FOV from these datasets however,
there is no existing dataset with ground videos and aerial image pairs to
solve cross-view geolocalization in videos and the proposed dataset addresses
this gap.

Cross-view is also used for fine geo-localization of UAVs or robots. Camera
feed (also frame-by-frame) and a known small region/map, of about a mile is
used to find a more exact location in the given map [7]. Sometimes a prior is
given to estimate the vehicle pose [10]. However, our focus is on coarse geo-
localization where the gallery spans over multiple cities.

A frame-by-frame geolocalization of videos is also possible with the proposed
dataset where 2D convolutional networks are used to extract the spatial
features. However, while following a frame-by-frame approach the contextual
information as available from a video is ignored. Fusion of features obtained
from 2D-CNN is also possible however it is more memory intensive as compared
to a 3D-CNN network. Considering these limitations and challenges, we propose
a videos based cross-view geo-localization.

Table 1: Statistics of the proposed GAMa dataset. Note: Large aerial images are at 1792X1792 resolution whereas small aerial images or tiles are at 256x256 resolution Parameter | Train | Test | Train(day) | Test(day)
---|---|---|---|---
Videos | 45029 | 6506 | 21144 | 3103
Large aerial images | 45029 | 6506 | 21144 | 3103
Clips | 1.68M | 243k | 790k | 116k
CN small aerial images | 1.68M | 243k | 790k | 116k
UCN small aerial images | 2.21M | 319k | 1.04M | 152k

##  3 GAMa dataset

The proposed GAMa (Ground-video to Aerial-image Matching) dataset comprises of
select videos from BDD100k [32] and aerial images from apple maps.

Ground video selection: Most of the videos in BDD100k dataset are 40 sec long
and usually have GPS label every second. The selection of videos from dataset
was based on the range of latitude and longitude for a given video where we
use a range parameter $\mu$. We label the GPS points at $n^{th}$ second as
$X_{n}$ ($lat_{n}$, $long_{n}$), where the corresponding latitude is $lat_{n}$
and longitude is $long_{n}$ (Figure 1R, Aerial image). Thus, for the whole
video we have GPS points as $X_{1}$($lat_{1}$, $long_{1}$), $X_{2}$($lat_{2}$,
$long_{2}$),.., $X_{n}$($lat_{n}$, $long_{n}$). If max latitude = $lat_{k}$
and min latitude = $lat_{l}$, then Latitude range = $lat_{k}$ – $lat_{l}$.
Also, if max longitude = $long_{p}$ and min longitude = $long_{q}$, then
Longitude range = $long_{p}$ – $long_{q}$. Range, $\mu$ = max(Latitude range,
Longitude range) In order to eliminate stationary and very fast videos, we
select videos with $\mu$ from 0.001 to 0.004. Figure 1R. shows the
distribution of training videos with range, $\mu$. The distribution was
similar for training and test sets. This selection left us with 46596 training
and 6728 testing videos which were further screened based on the availability
of GPS labels.

Aerial images: For the selected videos, aerial images are downloaded as tiles
from Apple maps at 19x zoom [1]. The dataset comprises of one large aerial
image (1792x1792) corresponding to each video of around 40 sec. and 49
uncentered small aerial images (256x256) for these large aerial regions. Table
1 summarizes the dataset statistics. Since, most of the videos have a GPS
label every second we divide the videos into smaller clips of 1 sec. each and
for each clip we have an aerial tile.

In Figure 1, we see an example of a large aerial region, along with the video
frames at each second.

Figure 2: Left(L): Sample aerial images from GAMa dataset. a) A large aerial
image matched to a video and division into small uncentered(UCN) aerial
images, b) Small UCN aerial image (Zoomed-in) and possible GPS labels for a
matching clip, and c) small centered (CN) aerial image as centered around the
GPS label; Right(R): An outline of the proposed approach. From a given ground
video, clips of 0.5 sec are input to GAMa-Net, one clip at a time is matched
to an aerial image. The sequence of aerial images thus obtained from a video
is input to the Screening network to retrieve the corresponding large aerial
region for video-level geolocalization. Top predictions of larger aerial
regions provide the updated gallery for GAMa-Net

Aerial image centering: In Figure 2(L)a, we show an example of a large aerial
image. We have a centered (CN) and an uncentered (UCN) set of small aerial
images, Table 1. UCN aerial images are obtained by dividing the large aerial
image into smaller tiles. The GPS label in these UCN smaller aerial images can
be anywhere besides the center, Figure 2(L)b. There are three labels in the
figure and for each of these GPS points the same tile will be considered as
the ground truth. For making a centered (CN) set, as shown in Figure 2(L)c we
take a crop centered around the corresponding GPS point however it has lead to
overlap among the aerial images since in some videos we have a distribution
where GPS points are nearby. In the dataset, we still have a one on one
correspondence among aerial images and short ground clips. The overlap among
the aerial images however increases the difficulty level for top-1 retrieval
thus we evaluate with distance threshold.

The dataset covers multiple cities, and the ground videos are distributed all
over the US and also from middle eastern region. However, most of the videos
are from four US cities; New York, Berkeley, San Francsico, and Bay area. They
show different weather conditions, including sunny, overcast, and rainy, as
well as different times of the day including day and night.

There is occlusion in videos and shadows of the skyscrapers in aerial images.
Limited FOV in videos and all stated characteristics bring it closer to a
realistic scenario however also makes it a difficult dataset for
geolocalization. In Table 2, we show a comparison with existing datasets for
cross-view geolocalization.

Table 2: Comparison of GAMa dataset with existing cross-view geo-localization datasets. Please, note that the previous datasets do not contain ground videos | Vo[28] | CVACT[13] | CVUSA[30] | GAMa (proposed)
---|---|---|---|---
Ground videos | no | no | no | 51535
Panorama |  450,000 | 128,334 | 44,416 | no
Aerial images |  450,000 | 128,334 | 44,416 | 1.92M
Aerial img resolution | - | 1200x1200 | 750x750 | 256x256 & 1792x1792
Multiple cities | yes | no | yes | yes

##  4 Method

An overview of the proposed approach which works on clip and video level is
show in Figure 2R. Ground-video to Aerial-image Matching Network (GAMa-Net)
learns features for ground view clips and aerial images; and bring the
matching pair closer in the feature space by applying a contrastive loss. This
provides a clip-level geolocalization for a long video. In addition, we
propose an hierarchical approach, where we do video-level geolocalization and
use it to update the gallery of aerial images by selecting top matched large
aerial regions (Figure 2R). The reduced gallery helps improve the clip-level
geolocalization by screening out some of the visually similar however
incorrect aerial images.

###  4.1 GAMa-Net: Clip-level Geo-localization

The proposed network takes as input a short clip from a ground video and
matches it with corresponding aerial image. An overview of the proposed
network is shown in Figure 3.

Visual encoders In GAMa-Net, we have a video encoder i.e. Ground Video Encoder
(GVE) to get features from ground video frames and an image encoder for aerial
image features i.e. Aerial Image Encoder (AIE). GVE uses 3D-ResNet18 as
backbone and a two layer transformer encoder. Given a ground video C, GVE
provides features for a 8 frame clip $C_{i}$ at the $i^{th}$ second of the
video. There is a skip-rate of one frame so we are covering around half a
second in the clip, $C_{i}$. AIE on the other hand uses 2D-ResNet18 as the
backbone. It takes the corresponding aerial image, $A_{i}$ as the input and
learns the features to match with that of the ground video.

Figure 3: Network diagram for GAMa-Net proposed for clip-level
geolocalization. We use 3D-CNN as our base network for learning features from
a ground clip, around 0.5sec. Similarly, for aerial image features, we use a
2D CNN backbone. Since only some part of aerial images are covered by the
video feed, using a transformer encoder improves the learning. Number of
frames, k=8 with skip rate=1

Transformer encoding: In the ground video all the visual features are not of
equal importance for matching with the aerial view. This is true for aerial
images as well and few features are more important when matching with ground
videos. For example, the top of a building as seen from aerial images is not
visible from a ground video and hence cannot be used to match the pair. To
address this, we experimented with multi-headed self attention. We input the
features obtained from convolutional networks to a transformer encoder
framework which comprises of 4 heads and 2 layers, and uses positional
encoding. A small neural network i.e. projection head is used to map the
representations to the space where contrastive loss is applied. We use a MLP
with one hidden layer to obtain the ground video and aerial image feature
vectors ${e^{c}}_{i}$ and ${e^{a}}_{i}$, respectively.

Image-video contrastive loss: We utilize contrastive loss formulation, base on
NT-Xent [4, 14, 23], to train our network. For a given ground video the
corresponding aerial image is considered a positive sample and rest of the
samples are considered as negatives. This is a image-video contrastive loss
applied on features from two different visual modalities i.e. ground videos
and aerial images. In the loss formulation, the focus is on reducing the
distance between the positive pair. We have a total of $2(N)$ data points in
any mini-batch with $N$ examples. The image-video contrastive loss for a pair
of positive examples is defined as,

$\displaystyle
l_{c,a}=-\log\frac{\exp(\mathrm{sim}({e^{c}}_{i},{e^{a}}_{i})/\tau)}{\sum^{N}_{k=1}\mathbbm{1}_{[k\neq
i]}\exp(\mathrm{sim}({e^{c}}_{i},{e^{c}}_{k})/\tau)+\sum^{N}_{k=1}\mathbbm{1}_{[k\neq
i]}\exp(\mathrm{sim}({e^{c}}_{i},{e^{a}}_{k})/\tau)},$ (1)

where ${e^{c}}_{i}$ and ${e^{a}}_{i}$ is a positive pair, ${e^{c}}_{i}$ and
(${e^{c}}_{k}$ or ${e^{a}}_{k}$) are negative pairs, $\mathbbm{1}_{[k\neq
i]}\in\\{0,1\\}$ is an indicator function with value 1 if $k\neq i$, $\tau$ is
a temperature parameter, $\mathrm{sim}$ is the cosine similarity between a
pair of features. The final loss is computed for all the positive pairs, both
$(e^{c},e^{a})$ and $(e^{a},e^{c})$.

###  4.2 Hierarchical approach

In this approach, we introduce video-level geolocalization which also helps in
reducing the search space for GAMa-Net (Figure 2R). The clips of a given video
are temporally related and provide contextual information to help improve the
geolocalization. Similarly, while looking at this problem from aerial image
view point, the sequence of aerial images corresponding to clips from a given
video are also related geographically and some of the correct prediction at
clip-level can be used to update the gallery. Using a smaller gallery also
reduces the possibility of error in feature matching.

Approach: We have four steps in this approach. In Step-1, we use GAMa-Net
which takes one clip (0.5 sec) at a time and matches with an aerial image.
Using multiple clips of a video, we get a sequence of aerial images for the
whole video, i.e. around 40 small aerial images. In Step-2, we use these
predictions of aerial images and match them to the corresponding larger aerial
region. We use a screening network to match the features however the features
are from the same view i.e aerial view. In Step-3, we use the predictions to
reduce the gallery by only keeping top ranked large aerial regions
corresponding to a video. These large aerial regions define our new gallery
for a given video. In Step-4, we use GAMa-Net i.e. the same network as in
Step-1, however geo-localize using the updated gallery.

Visually correct predictions from Step-1 which may not be the ground truth are
used to reduce the gallery using this approach. This approach helps improve
the clip-level geolocalization since the reduction of the gallery is based on
the fact that all the clips of a given video are geolocated nearby. Thus, the
aerial images predicted by GAMa-Net can be used to find that large aerial
region where all these clips have been captured. In this case, the probability
of finding all those visually correct aerial images in the same region is
higher when we are searching at the correct geolocation. Hence, it is likely
to match with correct large aerial region provided that meaningful and enough
information is available.

Figure 4: Left(L): Network diagram for the Screening network used for video-
level geo-localization in our proposed Hierarchical approach. Right(R):
Predictions at video level, where a single frame of video is shown along with
the correct matched large aerial region. More examples are in supplementary

Screening Network: Video-level geo-localization
We use the screening network to match a sequence of smaller aerial images with
the corresponding larger aerial region. The network is similar to GAMa-Net,
however the sequence of aerial images is input to Small Aerial Encoder (SAE)
with 2D-ResNet18 backbone and the sequence of features are later averaged to
obtain a 512D feature vector (Figure 4L). We also experimented with a
3D-ResNet18 backbone which is discussed later. The feature vector thus
obtained is matched with the feature vector of corresponding large aerial
image from Large Aerial Encoder (LAE). We apply contrastive loss similar to
Equation 1. The predictions from this network are used to update the gallery
for GAMa-Net.

##  5 Experiments and Results

Implementation and training details: We implement our GAMa-Net and screening
network using PyTorch [14] and train using Adam optimizer with a learning rate
(lr) of 8e-5. We use a lr scheduler with lr decay rate of 0.1. The screening
network is trained in two step; first with the actual ground truth sequences,
and then finetuned with predictions from GAMa-Net. The finetuning step allows
the network to adapt to noisy aerial sequences which will be used during
inference. We use pre-trained weights from Kinetics-400 for 3D-ResNet18 and
ImageNet weights for 2D-ResNet18. The ground videos in the proposed dataset
are from different times of the day, for faster training we have used only day
videos in our experiments unless stated otherwise.

Evaluation: We use top-k recall for clip-level and video-level matching at
different values of k. Given a video query, its closest k reference neighbors
in the feature space are retrieved as predictions. Similar to image based
geolocalization methods we use recall rates at top-1, top-5, top-10, and top
1%. More details can be found in [28, 8, 21]. We have UCN and CN sets of
aerial images corresponding to each clip. In UCN set there is one-to-one
correspondence between the clip and aerial image. The GPS point can be
anywhere within the aerial image however in the CN set there is an overlap
among the aerial images. To keep the evaluation similar to UCN set it is
considered a correct match if the predicted GPS is within the range of 0.05
mile of the correct location. We also report top-1@t rate where t is a
distance threshold to be used for correct prediction.

Baselines: We propose several baselines for comparison. For ground image based
baselines we use the center frame of the clip as an input. Image-CBn, our
proposed baseline uses 2D-CNN ResNet18 model to encode the ground video frame
with similar contrastive loss formulation as GAMa-Net. We use two different
loss functions for video based baselines (Triplet-Bn and CBn); margin triplet
loss [27] and contrastive loss [4]. In these baselines, we utilize 2D-CNN
ResNet18 for aerial images and 3D-CNN ResNet18 for ground videos.

Table 3: Comparison with proposed baselines, where CBn-Contrastive Baseline & Bn-Baseline. Recall rates in parentheses() are with UCN aerial images  Model | Video | R@1 | R@5 | R@10 | R@1%
---|---|---|---|---|---
Image-CBn | x | 9.5(1.5) | 18.8(5.5) | 24.6(9.0) | 87.7(50.7)
Shi et al.[22] | x | 9.6 | 18.1 | 26.6 | 71.9
L2LTR [31] | x | 11.7 | 20.8 | 28.2 | 87.1
Triplet-Bn | ✓ | <1(<1) | <1(1.2) | 1.1(2.4) | 33.1(33.8)
CBn | ✓ | 11.5(3.2) | 21.7(10.9) | 27.8(16.7) | 89.2(65.0)
GAMa-Net | ✓ | 15.2 | 27.2 | 33.8 | 91.9
GAMa-Net (Hierarchical) | ✓ | 18.3 | 27.6 | 32.7 | -

###  5.1 Results

The evaluations of GAMa-Net and a comparison with baselines is shown in Table
3. We observe a Top-1 recall rate of 18.3% and 15.2%, using GAMa-Net with and
without the hierarchical approach, respectively. On UCN set, we observe poor
performance as compared to CN set which was expected.

Comparison: In the first row, we show results with our image-based baselines
which uses a single frame from the ground video. We also compare with other
image based methods using a single frame as input and observe that the
proposed approach outperforms all these baselines. While training with a
single image is faster we do not have the temporal information which in this
case can be perceived as relative positioning or contextual information. As
the camera moves along a path or trajectory we can see the buildings or
objects pass-by giving an idea of their respective location. The information
of distance/relative-positioning as seen by a 3D-CNN is the additional
information when we train with videos. As shown in the Table 3, using a video
provides better results as compared to images.

Few studies have reported better performance with contrastive loss [15],
however triplet loss is also frequently used for geolocalization [35, 17]. We
also observe better performance with contrastive loss as compared to the
triplet loss (Table 3). In GAMa-Net, we have features from two different
visual modalities i.e. one from ground videos and other from aerial images,
which is different from the traditional contrastive loss. Similarly, training
with triplet loss also use image-video features. These differences are likely
responsible for poor training with triplet loss.

Figure 5: Three sample predictions by different models. Top-row shows frames
of query clips. Second row is for combined model, and third row is GAMa-Net.
Bottom row shows predictions by GAMa-Net with Hierarchical approach ( gallery
reduced to 1% of larger aerial regions). Correct predictions have a green
outline. Please note that in case of close GPS labels there are multiple
correct aerial images

Qualitative analysis: Figure 5 show sample Top-5 predictions with different
models where the leftmost is Top-1 and the rightmost is 5th. The combined
model makes visually meaningful predictions. In the leftmost example, the
camera is passing under a fly-over and the predictions show a similar
locations. The middle example is of a road without any crossings or red lights
in sight, and right-most example is of a city street with crossing markings on
road. The predictions by combined model match these specifications. However,
in these samples, the ground truth is in top-1% but not in top-5 images. The
predictions by GAMa-Net, with multi-headed self-attention improves the network
performance and correct prediction moves up in the top-5 (More results in
supplementary).

####  5.1.1 Hierarchical approach

Table 4: Screening network matches large aerial regions using a sequence of matched aerial images. In this Table, we compare 3D-CNN and 2D-CNN backbone for aerial image sequence of 8 or 32 images Model | R@1 | R@10 | R@1% | R@10%
---|---|---|---|---
3D-CNN (seq of 8 aerial) | 7.5 | 24.9 | 39.8 | 67.9
3D-CNN (seq of 32 aerial) | 3.8 | 19.8 | 36.1 | 74.9
2D-CNN (seq of 8 aerial) | 12.2 | 35.3 | 49.3 | 77.2
2D-CNN (seq of 32 aerial) | 8.2 | 20.9 | 29.3 | 83.8

The predictions from the screening network are used for video-level geo-
localization and to reduce the gallery for GAMa-Net. We experimented with both
2D-CNN and 3D-CNN backbone for aerial image sequence. As shown in Table 4,
better results were achieved with 2D-CNN backbone likely due to better spatial
features. A comparison of number of input aerial images show that 2D-CNN with
8 frames retains more ground truths at Top-1 (12.2%), Top-10 (35.3%) and
Top-1% (49.3%). We do not have 32 samples in some of the samples which likely
effects the performance. However, when we evaluated GAMa-Net after gallery
reduction, Screening network i.e. 2D-CNN network with 32 aerial images
provided the best results. Figure 4R shows qualitative results for video-level
geo-localizaion. It shows one frame of the video input to GAMa-Net, the
predicted seq of aerial images thus obtained was used to identify the larger
aerial regions. The bottom row shows the correct matched large aerial regions
for video-level geo-localization. More results are in supplementary.

Table 5: Results with GAMa-Net, without and with hierarchical approach Gallery size | R@1 | R@5 | R@10 | R@1%
---|---|---|---|---
Full | 15.2 | 27.2 | 33.8 | 91.9
| With Hierarchical Approach
Top-10 | 16.2 | 22.8 | 25.5 | -
Top-1% | 18.3 | 27.6 | 32.7 | -
Top-10% | 16.6 | 28.2 | 34.6 | 76.6

Gallery sizes: In Table 5, we discuss the results with GAMa-Net using various
gallery sizes, i.e. Top-10, Top-1%, and Top-10% of large aerial regions,
identified using screening network. We see better Top-1 results with Top-1%
gallery as compared to Top-10 and Top-10%. There is a trade-off between
reduced search space which improves matching and retaining the ground truth.
As observed from Table 4, in Top-10 we have ground truth for only 20.9% of
videos which increases to 83.8% at 10% gallery. However, this percentage will
be different if we consider clips since we get different number of clips from
each video, as per GPS labels. We evaluated GAMa-Net, which is clip level,
using a reduced gallery. Even when ground truth was not available for many
samples we see a Top-10 recall rate of 25.5%. When gallery is reduced to
Top-10 and Top-1% of larger aerial regions we do not have enough clips to make
upto 1% of the total gallery. In Figure 5, we show three examples of aerial
image matching and it is evident that predictions by GAMa-Net are improved
with hierarchical approach.

Table 6: Results with our best model and ablations for comparison. Here we show Top-1@threshold for comparison, with threshold values of 0.1, 0.2, 0.5 and 1.0 mile. CBn-UCN and CBn-CN, represents contrastive baselines with uncentered and centered aerial images, respectively. Hierarchical approach uses one percent of the gallery Model | Recall @
---|---
| Top-1 | Top-5 | Top-10 | Top-1% |<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>
CBn-UCN | 3.2 | 10.9 | 16.7 | 65.0 | 3.6 | 5.9 | 11.3 | 18.6
CBn-CN | 11.5 | 21.7 | 27.8 | 89.2 | 15.7 | 19.0 | 25.1 | 32.1
Combined | 11.6 | 23.4 | 30.4 | 92.4 | 15.6 | 19.0 | 25.0 | 32.7
Video-Tx | 14.1 | 25.4 | 31.8 | 90.8 | 18.7 | 22.1 | 28.1 | 35.7
GAMa-Net | 15.2 | 27.2 | 33.8 | 91.9 | 19.6 | 23.0 | 28.7 | 36.1
Dual-Tx | 14.6 | 26.1 | 32.7 | 91.8 | 19.0 | 22.2 | 28.2 | 35.5
Hierarchical | 18.3 | 27.6 | 32.7 | - | 23.5 | 27.8 | 34.9 | 43.6

###  5.2 Ablations

Combined model: In Table 6, we observe better performance with combined model
as compared to baselines, and a Top-1% recall of 92.4%. As expected, the
performance improves as we increase the distance threshold. In the combined
model, we have various augmentations which includes spatial and temporal
centering, and random crop. Since images are not aligned in GAMa we
stochastic-ally rotate the aerial view (0, 90, 180, 270 deg) for view-point
invariance. All these augmentations have been reported to help improve geo-
localization. We also include hard negatives for better training however
transformer encoder is not a part of combined model.

Transformer encoder: Similar to an aerial image, not all visual features in
the ground video have the same importance for cross-view geolocalization.
Thus, we implemented transformer encoder on ground video(Video-Tx) and aerial
images(Aerial-Tx), individually. In both cases, we observe an increase in
recall @Top-1. The performance is better with GAMa-Net(i.e. Aerial-Tx) and we
observe 15.2% Top-1 recall which is higher than 14.1% with Video-Tx (Table 6).
Observing an improvement in both cases we used transformer encoder on both
sides(Dual-Tx) however it did not perform better than the GAMa-Net.

Hierarchical approach: We use hierarchical approach to improve the performance
of GAMa-Net by reducing the gallery. Top-1@threshold (Table 6) shows that
using the hierarchical approach makes the matching more effective by
predicting the aerial images closer to the correct GPS location or ground
truth. The difference in Top-1 recall, with and without hierarchical approach,
is even higher at higher thresholds i.e 7.5% at 1.0 mile vs 3.9% at 0.1 mile.
An increase in recall to 43.6% at 1.0 mile threshold shows that the ground
truth is not very far from the Top-1.

##  6 Discussion

Videos and contextual information: In videos, we have more information which
can be considered as contextual information from geo-localization point of
view. It enables the network to locate a given frame or view with respect to
the other frames in the video. One frame may contain features to complement
another and together they are likely to provide more or complete information
required for geo-locating. In Table 3, we have compared video based method
with frame based baselines. Better recall rate with videos show that the
network is able to utilize the additional information available with videos.

Centered vs Uncentered: In an UNC aerial image the corresponding GPS point can
be anywhere in the tile. In cases where the GPS point is near the boundary,
the visual information from the video is less likely to match the
corresponding tile. As expected, after centering (CBn-CN) we observed an
improvement in the recall rate as compared with the uncentered set (i.e. CBn-
UCN).

Table 7: Training on full dataset and evaluation on day videos Model | Recall @
---|---
| Top-1 | Top-1% |<EMAIL_ADDRESS>|<EMAIL_ADDRESS>
Combined | 14.7 | 94.8 | 19.2 | 36.7
GAMa-Net | 17.5 | 94.7 | 22.3 | 39.3
Hierarchical | 19.4 | - | 25.0 | 45.1

Full Dataset: GAMa dataset is a large dataset which has its pros and cons.
With large amount of data networks are better trained however this also
increases the training time and memory requirement. Here we discuss results
with models trained on full dataset (Table 7) i.e. both day and night videos.
There is an improvement of 1-4% in recall at all k. Thus, using the
hierarchical approach our best R@Top-1 and R<EMAIL_ADDRESS>is 19.4% and 45.1%,
respectively.

Comparison: Our results are comparable to existing image based methods. One
recent study reports<EMAIL_ADDRESS>on CVUSA when images are unaligned and have
around 70% field of view (FOV) [31]. Most of the cross-view geolocalization
datasets such as CVUSA report high R@k while using ground panorama and aligned
aerial images which is unrealistic since a normal camera lens has a FOV of
around 72% and it is not possible to get aligned images always. In GAMa
dataset, the aerial images and ground videos are unaligned, and ground videos
have a limited FOV which makes it a more realistic and difficult dataset.

Challenges: In GAMa dataset among the two sets of small aerial images, i.e. CN
and UCN, the UCN set is more realistic however difficult for geolocalization.
Unaligned aerial images increase this difficulty however, the orientation
information can be extracted from the GPS information and is likely to improve
the performance [13]. Also, the ground videos have varying lengths and in some
of the cases GPS label is not available every second. Thus, the video length
available for geolocalization is less than 40 sec. Additionally, there is
occlusion because of the car hood and other objects. Such cases are more
likely to appear in fail cases.

Limitations: The proposed hierarchical approach performs better with longer
videos (8 sec. or more), however it can be used with shorter clips as well.
The aim behind using a hierarchical approach is to filter out confusing
samples to improve the retrieval rate. However, this sometimes leads to
filtering of the ground truth from the gallery. The model is also likely to
fail with indoor videos since it will not be possible to match the features
with an aerial image. However, this limitation is common to all cross-view
geo-localization methods.

##  7 Conclusions

In this work, we focus on the problem of video geolocalization via cross-view
matching. We propose a new dataset for this problem which has more than 51K
ground videos and 1.9 million satellite images. The dataset spans multiple
cities and is a more realistic dataset, unaligned and limited FOV. We believe
that this dataset will be useful for future research in cross-view video geo-
localization. Our proposed GAMa-Net effectively makes use of the rich
contextual information available with video. In addition, we propose a
hierarchical approach which also utilize the contextual information to further
improve the geo-localization.

## References

  * [1] Satellite images. https://www.apple.com/maps/, accessed: 2021-01
  * [2] Arandjelovic, R., Gronat, P., Torii, A., Pajdla, T., Sivic, J.: Netvlad: Cnn architecture for weakly supervised place recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 5297–5307 (2016)
  * [3] Chaabane, M., Gueguen, L., Trabelsi, A., Beveridge, R., O’Hara, S.: End-to-end learning improves static object geo-localization from video. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 2063–2072 (2021)
  * [4] Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: International conference on machine learning. pp. 1597–1607. PMLR (2020)
  * [5] Grigorescu, S., Trasnea, B., Cocias, T., Macesanu, G.: A survey of deep learning techniques for autonomous driving. Journal of Field Robotics 37(3), 362–386 (2020)
  * [6] Hakeem, A., Vezzani, R., Shah, M., Cucchiara, R.: Estimating geospatial trajectory of a moving camera. In: 18th International Conference on Pattern Recognition (ICPR’06). vol. 2, pp. 82–87. IEEE (2006)
  * [7] Hosseinpoor, H., Samadzadegan, F., Dadras Javan, F.: Pricise target geolocation and tracking based on uav video imagery. International Archives of the Photogrammetry, Remote Sensing & Spatial Information Sciences 41 (2016)
  * [8] Hu, S., Feng, M., Nguyen, R.M., Lee, G.H.: Cvm-net: Cross-view matching network for image-based ground-to-aerial geo-localization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 7258–7267 (2018)
  * [9] Hu, S., Lee, G.H.: Image-based geo-localization using satellite imagery. International Journal of Computer Vision 128(5), 1205–1219 (2020)
  * [10] Kim, D.K., Walter, M.R.: Satellite image-based localization via learned embeddings. In: 2017 IEEE International Conference on Robotics and Automation (ICRA). pp. 2073–2080. IEEE (2017)
  * [11] Li, A., Hu, H., Mirowski, P., Farajtabar, M.: Cross-view policy learning for street navigation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 8100–8109 (2019)
  * [12] Lin, T.Y., Cui, Y., Belongie, S., Hays, J.: Learning deep representations for ground-to-aerial geolocalization. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 5007–5015 (2015)
  * [13] Liu, L., Li, H.: Lending orientation to neural networks for cross-view geo-localization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5624–5633 (2019)
  * [14] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., Chintala, S.: Pytorch: An imperative style, high-performance deep learning library. In: Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., Garnett, R. (eds.) Advances in Neural Information Processing Systems 32, pp. 8024–8035. Curran Associates, Inc. (2019), http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf
  * [15] Radenović, F., Tolias, G., Chum, O.: Fine-tuning cnn image retrieval with no human annotation. IEEE transactions on pattern analysis and machine intelligence 41(7), 1655–1668 (2018)
  * [16] Regmi, K., Borji, A.: Cross-view image synthesis using geometry-guided conditional gans. Computer Vision and Image Understanding 187, 102788 (2019)
  * [17] Regmi, K., Shah, M.: Video geo-localization employing geo-temporal feature learning and gps trajectory smoothing. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 12126–12135 (2021)
  * [18] Rodrigues, R., Tani, M.: Are these from the same place? seeing the unseen in cross-view image geo-localization. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 3753–3761 (2021)
  * [19] Sarlin, P.E., DeTone, D., Malisiewicz, T., Rabinovich, A.: Superglue: Learning feature matching with graph neural networks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 4938–4947 (2020)
  * [20] Senlet, T., Elgammal, A.: Satellite image based precise robot localization on sidewalks. In: 2012 IEEE International Conference on Robotics and Automation. pp. 2647–2653. IEEE (2012)
  * [21] Shi, Y., Liu, L., Yu, X., Li, H.: Spatial-aware feature aggregation for image based cross-view geo-localization. Advances in Neural Information Processing Systems 32, 10090–10100 (2019)
  * [22] Shi, Y., Yu, X., Campbell, D., Li, H.: Where am i looking at? joint location and orientation estimation by cross-view matching. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4064–4072 (2020)
  * [23] Sohn, K.: Improved deep metric learning with multi-class n-pair loss objective. Advances in neural information processing systems 29 (2016)
  * [24] Tian, X., Shao, J., Ouyang, D., Shen, H.T.: Uav-satellite view synthesis for cross-view geo-localization. IEEE Transactions on Circuits and Systems for Video Technology (2021)
  * [25] Tian, Y., Chen, C., Shah, M.: Cross-view image matching for geo-localization in urban environments. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 3608–3616 (2017)
  * [26] Toker, A., Zhou, Q., Maximov, M., Leal-Taixé, L.: Coming down to earth: Satellite-to-street view synthesis for geo-localization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6488–6497 (2021)
  * [27] Vassileios Balntas, Edgar Riba, D.P., Mikolajczyk, K.: Learning local feature descriptors with triplets and shallow convolutional neural networks. In: Richard C. Wilson, E.R.H., Smith, W.A.P. (eds.) Proceedings of the British Machine Vision Conference (BMVC). pp. 119.1–119.11. BMVA Press (September 2016). https://doi.org/10.5244/C.30.119, https://dx.doi.org/10.5244/C.30.119
  * [28] Vo, N.N., Hays, J.: Localizing and orienting street views using overhead imagery. In: European conference on computer vision. pp. 494–509. Springer (2016)
  * [29] Wang, T., Zheng, Z., Yan, C., Zhang, J., Sun, Y., Zhenga, B., Yang, Y.: Each part matters: Local patterns facilitate cross-view geo-localization. IEEE Transactions on Circuits and Systems for Video Technology (2021)
  * [30] Workman, S., Souvenir, R., Jacobs, N.: Wide-area image geolocalization with aerial reference imagery. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 3961–3969 (2015)
  * [31] Yang, H., Lu, X., Zhu, Y.: Cross-view geo-localization with layer-to-layer transformer. Advances in Neural Information Processing Systems 34 (2021)
  * [32] Yu, F., Chen, H., Wang, X., Xian, W., Chen, Y., Liu, F., Madhavan, V., Darrell, T.: Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 2636–2645 (2020)
  * [33] Zamir, A.R., Shah, M.: Accurate image localization based on google maps street view. In: European Conference on Computer Vision. pp. 255–268. Springer (2010)
  * [34] Zemene, E., Tesfaye, Y.T., Idrees, H., Prati, A., Pelillo, M., Shah, M.: Large-scale image geo-localization using dominant sets. IEEE transactions on pattern analysis and machine intelligence 41(1), 148–161 (2018)
  * [35] Zhu, S., Yang, T., Chen, C.: Vigor: Cross-view image geo-localization beyond one-to-one retrieval. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3640–3649 (2021)
  * [36] Zhu, Y., Sun, B., Lu, X., Jia, S.: Geographic semantic network for cross-view image geo-localization. IEEE Transactions on Geoscience and Remote Sensing (2021)

##  Appendix 0.A Overview

In this supplementary, we have included additional qualitative results. In
Section 0.B, we show and discuss some qualitative results for video-level
geolocalization. In Section 0.C, we have included additional qualitative
results for clip-level geolocalization and ablations.

Figure 6: An outline of the proposed approach. Given a ground video, in Step-1
0.5 sec clips from the video are input to GAMa-Net. It takes one clip at a
time and matches it with an aerial image. In Step-2, the sequence of aerial
images obtained from GAMa-Net is input to the Screening network to retrieve
the corresponding larger aerial region. This is our video-level geo-
localization. In Step-3, top predictions of these larger aerial regions
provides the updated gallery for a video. Step-4, is prediction by GAMa-Net
while using the updated gallery

An outline of the proposed approach in Figure 6 shows how video level geo-
localization is used to improve the clip-level results. The GAMa-Net outputs
aerial image predictions at clip-level using the full gallery i.e. $G_{f}$. A
video comprises of a number of clips (upto 40 clips per video), thus a
sequence of aerial images is obtained from each query video, this sequence is
then input to the screening network. The screening network uses this sequence
of small aerial images to predict a large aerial region corresponding to the
query video. Then we select Top-1% large aerial images to update the gallery
(the updated gallery $G_{up}$) for GAMa-Net and reduce the search space. We
are hopeful that with further research this approach has the potential to be
generalized to even larger scale. As shown in main paper we observe an
improvement in Top-1 recall using this approach. The screening of the
locations at large region level removed the confusing samples.

##  Appendix 0.B Qualitative Results: Video-level Geo-localization

In Figure 7, we show some of the predictions of larger aerial regions as
ranked by the screening network. For each sample, the first row shows frames
of the video spanning over 35 sec. of time duration. In the subsequent row, we
have the ground truth larger aerial region which is followed by Top-5
predictions by the screening network. The examples shown here are mostly
correct predictions where Top-1 is same as the ground truth.

In the first sample (Figure 7), we see that the video frames are partially
outdoors and partially indoors. Indoor setting appears to be a parking lot or
an underpass. The screening network is able to correctly localize the larger
aerial region while using the information available from all the clips. The
prediction when used to reduce the gallery of GAMa-Net is likely to enable a
better prediction by screening-out the far away regions. Frame-by-frame it
would have been difficult to localize the indoor frames. However, using the
hierarchical approach the network is able to use the context from outdoor
frames. We also observed this from the predictions, with GAMa-Net only 12
clips out of the 38 clips had ground truth prediction in top-10 and after
reducing the gallery using the prediction from screening network this number
increased to 28 clips. In the second sample (Figure 7), we can see that all
Top-5 predictions are visually similar. These predictions appears to be from
the same region and most are from around a mile radius of the ground truth
larger aerial region.

In the third sample, because of a car, there is occlusion in part of the
video. GAMa-Net correctly localizes the initial clips however fails in the
clips with occlusion. After screening the gallery using the correct larger
aerial prediction, most of these clips were correctly geolocalized by GAMa-Net
in Top-1, Top-5 or Top-10. Similarly, in fourth and fifth sample the occlusion
is in all the frames either because of the car hood or rear view mirror. In
the fourth sample, similar improvement in clip-level geolocalization was
observed with GAMa-Net because of correct screening of the larger aerial
region at the video-level. We see correct video-level Top-1 with fifth sample,
however the improvement in the clip-level predictions by GAMa-Net was less and
only 3-4 more clips had predictions added to top-10 as compared to retrieval
from the full gallery. The second top prediction of this sample is visually
similar to the ground truth however a closer look shows that it is a different
image. In the last or sixth sample we can see that multiple correct
predictions because of the visual similarity are in Top-5.

Figure 7: Here, we show results for video-level geo-localization using the
Screening network. In each sample, we show seven frames of the query video,
followed by the ground truth aerial image and top-5 predictions of larger
aerial regions.

With all these video samples, because of some of the correct clip-level
predictions by GAMa-Net, the screening network is able to localize at video-
level and identify the correct larger aerial region. However, the last sample
had correct clip-level prediction for a single clip out of 31. It is likely
that the visual similarity of the incorrect aerial image predictions helped
with correct video-level geolocalization at Top-1 using screening network.

##  Appendix 0.C Additional Qualitative Results: Clip-level Geo-localization

In Figure 8 and Figure 9, we show additional results using the proposed GAMa-
Net with Hierarchical approach, where we use the video-level predictions to
improve clip-level geo-localization. Figure 8 shows examples of correct Top-1
predictions. We can see that most of the Top-5 predictions are nearby the
ground truth. Because of the nearby GPS labels in video clips we have
overlapping in aerial images and all these images appear in the Top-5
predictions along with the ground truth. Figure 9 shows examples of fail
cases, we can see that most of the fails are due to shadows or occlusion or
poor quality of aerial images. The model makes meaningful predictions however
fails in difficult or confusing samples e.g. in the last sample traffic lights
are visible in the video and predictions are with zebra crossings however does
not retrieve the correct aerial image in Top-5.

Figure 8: Here we show sample success cases with GAMa-Net using Hierarchical
approach where Top-1 prediction is correct. In each sample, four frames of the
query clips are followed by ground truth aerial image and Top-5 predictions
Figure 9: Here we show sample fail cases with GAMa-Net using Hierarchical
approach where Top-1 prediction is correct. In each sample, four frames of the
query clips are followed by ground truth aerial image and Top-5 predictions
Figure 10: Here we show success cases using GAMa-Net without Hierarchical
approach where top-1 prediction is correct. In each sample, four frames of the
query clips are followed by the ground truth aerial image and Top-5
predictions Figure 11: Here we show fail cases using GAMa-Net without
Hierarchical approach. In each sample, four frames of the query clips are
followed by the ground truth aerial image and top-5 predictions

In Figure 12 and Figure 13, we show some additional results with the combined
model which does clip-level geolocalization by retrieving a matching aerial
image. The network however is an ablation of the proposed GAMa-Net and does
not have a transformer encoder.

Figure 12: Here we show success cases for combined model where Top-1
prediction is correct. In each sample, four frames of the query clips are
followed by ground truth aerial image and top-5 predictions Figure 13: Here we
show fail cases for combined model. In each sample, four frames of the query
clips are followed by ground truth aerial image and Top-5 predictions


11institutetext: Department of Informatics, Systems and Communication,
University of Milano-Bicocca, Viale Sarca 336, 20126 Milano, Italy
https://mmsp.unimib.it/
11email<EMAIL_ADDRESS>22institutetext:
NeuroMI, Milan Center for Neuroscience,
Piazza dell’Ateneo Nuovo 1, 20126 Milano, Italy

# SER_AMPEL: a multi-source dataset for speech emotion recognition of Italian
older adults ††thanks: This research is partially supported by the FONDAZIONE
CARIPLO “AMPEL: Artificial intelligence facing Multidimensional Poverty in
ELderly” (Ref. 2020-0232) and by the co-funding European Union – Next
Generation EU, in the context of The National Recovery and Resilience Plan,
Investment Partenariato Esteso PE8 ”Conseguenze e sfide dell’invecchiamento”,
Project Age-It (Ageing Well in an Ageing Society)

Alessandra Grossi 1122 0000-0003-1308-8497 Francesca Gasparini 1122
0000-0002-6279-6660

###### Abstract

In this paper, SER_AMPEL, a multi-source dataset for speech emotion
recognition (SER) is presented. The peculiarity of the dataset is that it is
collected with the aim of providing a reference for speech emotion recognition
in case of Italian older adults. The dataset is collected following different
protocols, in particular considering acted conversations, extracted from
movies and TV series, and recording natural conversations where the emotions
are elicited by proper questions. The evidence of the need for such a dataset
emerges from the analysis of the state of the art. Preliminary considerations
on the critical issues of SER are reported analyzing the classification
results on a subset of the proposed dataset.

###### Keywords:

Speech emotion recognition older adults Italian language multi-source dataset
cross-language SER cross-corpus SER

##  1 Introduction

In recent decades, the elderly population has undergone continuous growth with
a consequent increase in attention to well-being and active ageing. In this
context, loneliness and social isolation of the elderly are factors to
consider because they have a close relationship with depression, dementia and
other serious medical conditions [33]. The use of technology can reduce
loneliness and social isolation, mitigating their negative outcomes on mental
and physical health. In particular, there is a grown interest in studying
social robots to increase psychological well being [15] [8]. As well as in the
case of human-human interactions [23], emotions play a relevant role in
creating an effective interaction between older adults and social robots.
During social interactions, human beings convey their emotions not only by
words but also by bodily, vocal or facial expressions [23].

In this work we focus on Speech Emotion Recognition (SER) [40] [1] to
recognize emotions in natural conversations, to provide a more natural and
social communication in human-robot interactions [18] [21].

Several researches have been developed in the field of SER during the last
three decades [13], starting from uni-modal approaches, that consider only
linguistic or acoustic information, to multi-modal approaches that rely on
both of them [2]. Several traditional approaches that adopt handcrafted
features, both in the temporal and/or frequency domains as well as deep
learning ones, have been adopted and are summarized in review manuscripts such
as the ones of Akçay et al [1] and Wani et al [39].

Despite the numerous works in the state of the art that face the SER task,
there are several critical issues that make it difficult to recognize emotions
especially in natural conversations. In the review paper by Fahad et al. [13]
some of these challenges and the approaches developed to address them are
summarized. In particular, SER models struggle to recognize emotions when
considering people of different languages or ages. Moreover, most of the
datasets available in the literature are composed of acted utterances [30],
while only few of them consider natural conversations [36] [14] [27].
Within this context, we address the problem of SER in the case of an elderly
population that speaks Italian. To this end, the first urgency is the
collection of a proper dataset of Italian conversations that considers
different ages, in particular older people.

This paper is thus structured as follows. In section 2, the datasets available
in the literature are introduced, analyzing their limits. In section 3, our
multi-source SER_AMPEL dataset, collected to solve some of the critical issues
found in the available datasets, is presented. Models previously trained on
acted datasets are then applied on a portion of the SER_AMPEL data, and the
obtained classification results are reported in section 4. From this analysis,
interesting considerations can be drawn and future lines of research can be
identified that are discussed in section 5.

##  2 SER datasets

Several datasets have been considered in the literature for SER purpose.
Following the suggestions proposed by Koolagudi et al. [22], they can be
classified into three groups, depending on how the emotions are elicited.

  * •

Acted datasets: these datasets collect utterances from actors/actresses that
try to simulate emotions; the utterances can be short phrases (few seconds),
usually repeated with different emotions. The utterances can also be phrases
without meaning. Most of the datasets available belong to this category
[5][7][6][25]

  * •

Evoked or Elicited datasets: conversations are recorded in situations properly
created in order to evoke certain emotions [26][14].

  * •

Spontaneous or Natural datasets: conversations among different persons are
recorded in real-world environments, such as call-centers, or public places
[13][3][32]. These datasets are more realistic and authentic, but collecting
balanced instances of each emotion is really difficult.

We have analyzed 46 datasets described in the literature and reported our
analysis in a table available at the following link:
https://mmsp.unimib.it/download-1/. In this table, among the various
information, we have reported the name and reference of the dataset, if it is
freely available, number of subjects, gender and age, language, type of
collection process (with respect to the three categories introduced), and in
particular if there are older adults among the subjects.

From this analysis it emerges that: i) the languages of the collected datasets
are mainly English and Chinese, as depicted in the pie chart of Figure 1,
right; ii) multi-language datasets have been proposed [28] [19] as language
could have an influence in how emotions are expressed [24]; iii) most of the
available dataset are acted; iv) few works face the problem of SER in case of
elderly, or varying the age [4] [20] [38] [34], and old subjects are rarely
present in available datasets [32] [7] [29] [14], as also depicted in the pie
chart of Figure 1, left; v) the number of available datasets is 12 out of 46.

Figure 1: Pie charts depicting the SER datasets that include older subjects
with respect to all the others (left) and depicting the relative proportions
among datasets of different languages, right.

##  3 SER_AMPEL: a multi-source dataset

The SER_AMPEL is a multi-source dataset that has the intent of providing a
reference in case of SER for Italian older adults. It is composed of different
subsets, acquired following different protocols.

The three subsets of the SER_AMPEL dataset are:

  * •

NOLD evoked subset: this is a dataset of natural conversations among Auser
111Auser is an Italian association that promotes active ageing and that
supports older persons, www.auser.it volunteers and older adults, where the
emotions are elicited by proper questions and/or music. The list of all the 43
questions is reported in appendix 0.A. Note that for each conversations only
few questions, between 2 and 15 depending on the protocol, were used. Instead,
the audio files adopted were chosen by the subject himself/herself.

  * •

NYNG evoked subset: this is a dataset of natural conversations among young
adults where the emotions are elicited in the same way of NOLD.

  * •

AOLD acted subset: portion of conversations of Italian older dubbers segmented
from Series and Movies.

The SER_AMPEL dataset is not yet completed, as the acquisition of some subsets
is still ongoing, as well as the segmentation, labelling and transcription
steps.
In particular, the required steps are:

  1. 1.

segment all the recorded conversations into phrases of about 15 seconds;

  2. 2.

label all the segments with respect to two different emotional models: the
continuous Valence-Arousal-Dominance model, introduced by Mehrabian and
Russell [31] and the categorical one defined by Ekman [12];

  3. 3.

transcribe the speech into text, automatically and/or manually.

In Table 1 a synthetic description of the subsets of data already collected is
reported and described, considering the classification introduced in section
2, the number of subjects and gender, and type of transcription.

Table 1: Main characteristics of each subset of SER_AMPEL dataset. Subset |  No. of
Speakers |  No. of
Female |  Age (in years):
$\mu\pm\sigma$ |  No of
Recordings |  Average duration |  Transcriptions
---|---|---|---|---|---|---
NOLD | 81 | 31 | 73.03 ± 5.4 | 591 | 40 sec | manual
NYNG | 10 | 6 | 31.09 ± 11,38 | 150 | 1 min | manual
AOLD | 10 | 3 | 71.4 ± 5.3 | 120 | 15 sec | automatic

##  4 Method and Results

A general SER model based on XGBoost [9], that considers only the acoustic
part of the speech, was originally trained on a dataset of English utterances,
acted by young and adult actors and actresses [7]. The classifier has been
defined in order to recognize the three sentiment classes: positive, neutral
and negative. Starting from this model, two domain adaptation strategies
(Kullback-Leiber Important Estimation Procedure (KLIEP) [37] and Transfer
AdaBoost (TrAdaBoost) [11]) have been proposed to adapt the model to older
people and to Italian language respectively [16]. The dataset adopted to train
the general model and the two adapted ones are CREMA-D [7] and EMOVO [10]. To
better understand what follows, a brief description of these two datasets is
necessary.

CREMA-D is an English acted dataset, that consists of 7442 audio and video
recordings of professional actors, playing 12 utterances each one expressed in
six emotional states (happy, sad, anger, fear, disgust and neutral) at
different intensity levels. The utterances last about 5 seconds. All the acted
sentences have a neutral semantic content. 91 actors and actresses have been
involved (43 female), among them 6 are older persons ($>=$ 60 years) and 85
adults (between 20 and 59 years). Data collected from adults are used to train
the general model, while data coming from the older persons (hereafter called
OLD) are used to adapt it.

EMOVO is an Italian acted dataset of 588 audio recordings. Six young Italian
actors (3 males and 3 females) with a mean age of 27.1 years, acted 14
utterances, simulating 7 emotional states: neutral, disgust, fear, anger, joy,
surprise, and sadness. This dataset (hereafter called ITA) is used to adapt
the general model for Italian speakers.

The best results obtained in the previous paper are reported in Table 2. The
models, without domain adaptation (DA) and with DA, trained and tested on the
same English dataset (CREMA-D) perform better than the models where training
and test sets are of different languages. The domain adaptation strategy
applied to adapt the general model (trained on English utterances) to the
Italian dataset (ITA test set), increases the performance with respect to the
general model without DA. While, in the case of the OLD test set, it seems
that there are no differences in the performance of the general model and the
DA one.

Table 2: Comparison of the models using, as test set, the older persons of the CREMA-D dataset (OLD), and the EMOVO dataset (ITA) respectively. The results are reported considering the general model without domain adaptation (No DA) and the best DA strategy (TrAdaBoost). Three evaluation metrics are considered: Macro F1-score, Accuracy and single class F1-score. Classifier | DA Strategy |  Test Set |  Macro F1-score |  Negative F1-score |  Neutral F1-score |  Positive F1-score | Accuracy
---|---|---|---|---|---|---|---
XGBoost | No DA | OLD | 62% | 0,79 | 0,55 | 0,51 | 70%
ITA | 35% | 0,71 | 0,28 | 0,07 | 57%
TrAdaBoost | OLD | 62% | 0,79 | 0,56 | 0,52 | 70%
| ITA | 44% | 0,68 | 0,25 | 0,39 | 56%

In Table 3, the results obtained applying these models to AOLD subset are
reported. AOLD is adopted as test set as it is already labelled and
transcribed, while the rest of the SER_AMPEL dataset still requires either
labelling and/or transcription.

In order to use the pre-trained models on this new dataset, the instances of
AOLD have been labelled as positive, neutral and negative. The values of
valence have been used to define these classes, considering the high valence
scores (5-4) as positive sentiment, low valence scores (1-2) as negative
sentiment, and the single central valence score (3) as neutral sentiment. Due
to the fact that the dataset collection is still on going, the cardinality of
the three groups is unbalanced, with 89 instances labeled as negative, 27 as
positive and only 3 neutral. The performance while using the general model
trained on English adults, is aligned with the results achieved with the same
model on the ITA dataset. These low performances can be attributed both to the
different language between training and test sets, but also to the presence of
different utterances. Note that, the higher result obtained in Table 2 first
row, is referred to a model where training and test set come from the same
dataset, where the utterances pronounced by adults (training) and older people
(test) are the same. Moreover the DA strategies, seem to be not effective,
both in the case of trying to adapt to Italian language and to age. A
difference between CREMA-D and EMOVO datasets and the AOLD one is related to
the phrases pronounced. In fact the AOLD dataset is composed of realistic
conversations of about 15 seconds, while the other two datasets contain only
short utterances, of about 5 seconds, that in some cases correspond to no-
sense phrases.

Table 3: Performance comparison using the young and adults data from CREMA-D as training set and AOLD data as test set. Two different datasets were adopted as target set for the domain adaptation: Italian acted utterances from EMOVO (rows 2-3) and older adults data from CREMA-D. The analysis are performed considering three different Domain Adaptation (DA) strategies (no Domain Adaptation, KLIEP DA and TrAdaBoost DA). Three evaluation metrics are considered: macro F1-score, accuracy, and single class F1-score. |  Macro F1-score |  Negative F1-score |  Neutral F1-score |  Positive F1-score |  Accuracy
---|---|---|---|---|---
No Domain Adaptation | 32% | 0,74 | 0,00 | 0,22 | 60%
KLIEP (DA on Ita) | 36% | 0,85 | 0,00 | 0,35 | 75%
TrAdaBoost (DA on Ita) | 30% | 0,42 | 0,14 | 0,35 | 37%
KLIEP (DA on Eld) | 36% | 0,67 | 0,00 | 0,40 | 56%
TrAdaBoost (DA on Eld) | 31% | 0,76 | 0,00 | 0,17 | 62%

##  5 Conclusions

The development of SER algorithms that adapt their behaviour to different
conditions and scenarios is a fundamental step for the definition of systems
able to interact in a natural and intuitive way with fragile people such as
older adults. These technologies could be employed, for instance, in call
centers or mobile communications for the definition of services that adapt
their behaviour according to the interlocutor’s emotions and mood [17].
Furthermore, similar systems could be embedded into nursing-care or assistive
robots that continuously interact with older adults or patients to help them
in their daily lives. From the subject’s voice, the robot could be able to
infer the emotional state of the speaker and adapt its behaviour in order to
define a more efficient and trustworthy interaction between robot and people
[35]. In this paper the SER task is specifically considered with respect to
Italian older people. From the results here obtained, it emerges the need of a
specific dataset, uniform both for what concerns the language and the age of
the subjects. To guarantee the generalizability of the SER models that can be
developed, multi sources for data collection should be considered. The
SER_AMPEL dataset, here introduced and still ongoing, is conceived with these
aims. Such a dataset could ensure a large collection of speech data useful for
the definition of Italian SER algorithms that take into account several
factors, for example, the subject’s age, the mode of acquisition, or how the
emotions are elicited. Furthermore, the dataset could allow the analysis of
issues that may affect speech data collected in natural environments, such as
the presence of different kinds of noise as well as the occurrence of
utterances or words in dialect.

## References

  * [1] Akçay, M.B., Oğuz, K.: Speech emotion recognition: Emotional models, databases, features, preprocessing methods, supporting modalities, and classifiers. Speech Communication 116, 56–76 (2020)
  * [2] Atmaja, B.T., Sasou, A., Akagi, M.: Survey on bimodal speech emotion recognition from acoustic and linguistic information fusion. Speech Communication (2022)
  * [3] Batliner, A., Steidl, S., Nöth, E.: Releasing a thoroughly annotated and processed spontaneous emotional database: the fau aibo emotion corpus (2008)
  * [4] Boateng, G., Kowatsch, T.: Speech emotion recognition among elderly individuals using multimodal fusion and transfer learning. In: Companion Publication of the 2020 International Conference on Multimodal Interaction. pp. 12–16 (2020)
  * [5] Burkhardt, F., Paeschke, A., Rolfes, M., Sendlmeier, W.F., Weiss, B., et al.: A database of german emotional speech. In: Interspeech. vol. 5, pp. 1517–1520 (2005)
  * [6] Busso, C., Bulut, M., Lee, C.C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J.N., Lee, S., Narayanan, S.S.: Iemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation 42, 335–359 (2008)
  * [7] Cao, H., Cooper, D.G., Keutmann, M.K., Gur, R.C., Nenkova, A., Verma, R.: Crema-d: Crowd-sourced emotional multimodal actors dataset. IEEE transactions on affective computing 5(4), 377–390 (2014)
  * [8] Chen, S.C., Jones, C., Moyle, W.: Social robots for depression in older adults: a systematic review. Journal of Nursing Scholarship 50(6), 612–622 (2018)
  * [9] Chen, T., Guestrin, C.: Xgboost: A scalable tree boosting system. In: Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining. pp. 785–794 (2016)
  * [10] Costantini, G., Iaderola, I., Paoloni, A., Todisco, M.: Emovo corpus: an italian emotional speech database. In: International Conference on Language Resources and Evaluation (LREC 2014). pp. 3501–3504. European Language Resources Association (ELRA) (2014)
  * [11] Dai, W., Yang, Q., Xue, G.R., Yu, Y.: Boosting for transfer learning. vol. 227, pp. 193–200 (01 2007). https://doi.org/10.1145/1273496.1273521
  * [12] Ekman, P.: Are there basic emotions? (1992)
  * [13] Fahad, M.S., Ranjan, A., Yadav, J., Deepak, A.: A survey of speech emotion recognition in natural environment. Digital Signal Processing 110, 102951 (2021)
  * [14] Fan, W., Xu, X., Xing, X., Chen, W., Huang, D.: Lssed: a large-scale dataset and benchmark for speech emotion recognition. In: ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 641–645. IEEE (2021)
  * [15] Fraune, M.R., Komatsu, T., Preusse, H.R., Langlois, D.K., Au, R.H., Ling, K., Suda, S., Nakamura, K., Tsui, K.M.: Socially facilitative robots for older adults to alleviate social isolation: A participatory design workshop approach in the us and japan. Frontiers in Psychology p. 6256 (2022)
  * [16] Gasparini, F., Grossi, A.: Sentiment recognition of italian elderly through domain adaptation on cross-corpus speech dataset. In: CEUR Workshop Proceeding. vol. 3367, pp. 12–28 (2022)
  * [17] Gupta, P., Rajput, N.: Two-stream emotion recognition for call center monitoring. In: Eighth Annual Conference of the International Speech Communication Association. Citeseer (2007)
  * [18] Hegel, F., Spexard, T., Wrede, B., Horstmann, G., Vogt, T.: Playing a different imitation game: Interaction with an empathic android robot. In: 2006 6th IEEE-RAS International Conference on Humanoid Robots. pp. 56–61. IEEE (2006)
  * [19] Hozjan, V., Kacic, Z., Moreno, A., Bonafonte, A., Nogueiras, A.: Interface databases: Design and collection of a multilingual emotional speech database. In: LREC (2002)
  * [20] Jian, Q., Xiang, M., Huang, W.: A speech emotion recognition method for the elderly based on feature fusion and attention mechanism. In: Third International Conference on Electronics and Communication; Network and Computer Technology (ECNCT 2021). vol. 12167, pp. 398–403. SPIE (2022)
  * [21] Jones, C., Deeming, A.: Affective human-robotic interaction. In: Affect and emotion in human-computer interaction, pp. 175–185. Springer (2008)
  * [22] Koolagudi, S.G., Rao, K.S.: Emotion recognition from speech: a review. International journal of speech technology 15(2), 99–117 (2012)
  * [23] Lange, J., Heerdink, M.W., Van Kleef, G.A.: Reading emotions, reading people: Emotion perception and inferences drawn from perceived emotions. Current Opinion in Psychology 43, 85–90 (2022)
  * [24] Latif, S., Qayyum, A., Usman, M., Qadir, J.: Cross lingual speech emotion recognition: Urdu vs. western languages. In: 2018 International Conference on Frontiers of Information Technology (FIT). pp. 88–93. IEEE (2018)
  * [25] Livingstone, S.R., Russo, F.A.: The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english. PloS one 13(5), e0196391 (2018)
  * [26] Martin, O., Kotsia, I., Macq, B., Pitas, I.: The enterface’05 audio-visual emotion database. In: 22nd international conference on data engineering workshops (ICDEW’06). pp. 8–8. IEEE (2006)
  * [27] Morrison, D., Wang, R., De Silva, L.C.: Ensemble methods for spoken emotion recognition in call-centres. Speech communication 49(2), 98–112 (2007)
  * [28] Parada-Cabaleiro, E., Costantini, G., Batliner, A., Baird, A., Schuller, B.: Categorical vs dimensional perception of italian emotional speech (2018)
  * [29] Pichora-Fuller, M.K., Dupuis, K.: Toronto emotional speech set (TESS) (2020). https://doi.org/10.5683/SP2/E8H2MF, https://doi.org/10.5683/SP2/E8H2MF
  * [30] Ringeval, F., Sonderegger, A., Sauer, J., Lalanne, D.: Introducing the recola multimodal corpus of remote collaborative and affective interactions. In: 2013 10th IEEE international conference and workshops on automatic face and gesture recognition (FG). pp. 1–8. IEEE (2013)
  * [31] Russell, J.A., Mehrabian, A.: Evidence for a three-factor theory of emotions. Journal of research in Personality 11(3), 273–294 (1977)
  * [32] Schuller, B.W., Batliner, A., Bergler, C., Messner, E.M., Hamilton, A., Amiriparian, S., Baird, A., Rizos, G., Schmitt, M., Stappen, L., et al.: The interspeech 2020 computational paralinguistics challenge: Elderly emotion, breathing & masks (2020)
  * [33] National Academies of Sciences, E., Medicine, et al.: Social isolation and loneliness in older adults: Opportunities for the health care system. National Academies Press (2020)
  * [34] Soğancıoğlu, G., Verkholyak, O., Kaya, H., Fedotov, D., Cadée, T., Salah, A.A., Karpov, A.: Is everything fine, grandma? acoustic and linguistic modeling for robust elderly speech emotion recognition. arXiv preprint arXiv:2009.03432 (2020)
  * [35] Spezialetti, M., Placidi, G., Rossi, S.: Emotion recognition for human-robot interaction: Recent advances and future perspectives. Frontiers in Robotics and AI p. 145 (2020)
  * [36] Steidl, S.: Automatic classification of emotion related user states in spontaneous children’s speech. Logos-Verlag Berlin, Germany (2009)
  * [37] Sugiyama, M., Nakajima, S., Kashima, H., Buenau, P., Kawanabe, M.: Direct importance estimation with model selection and its application to covariate shift adaptation. Advances in neural information processing systems 20 (2007)
  * [38] Verma, D., Mukhopadhyay, D.: Age driven automatic speech emotion recognition system. In: 2016 International Conference on Computing, Communication and Automation (ICCCA). pp. 1005–1010. IEEE (2016)
  * [39] Wani, T.M., Gunawan, T.S., Qadri, S.A.A., Kartiwi, M., Ambikairajah, E.: A comprehensive review of speech emotion recognition systems. IEEE Access 9, 47795–47814 (2021)
  * [40] Wu, X., Zhang, Q.: Design of aging smart home products based on radial basis function speech emotion recognition. Frontiers in Psychology 13, 882709–882709 (2022)

##  Appendix 0.A Appendix A

Table 4: Questions selected to evoke Positive Sentiment ID |  Questions
---|---
Q_01 | Tell us about a fun adventure you had with your friends
Q_02 | What is your favourite time of day? Please, describe it
Q_03 | Are there any hobbies or activities you began during the lockdown? If so, please describe them
Q_04 | Where would you like to go on vacation?
Q_05 | What is your favorite season and why?
Q_06 | Tell us about the happiest moment of your life
Q_07 | Do you have grandchildren? Please, tell us about them
Q_08 | Describe a happy moment spent with your parents
Q_09 | Are you married? If so, how did you meet each other?
Q_10 | Have you ever traveled? If so, tell us about your most beautiful trip
Q_11 | Describe the happiest moment you experienced during the lockdown
Q_12 |  Were you able to keep in touch with friends and relatives during the lockdown? If so, please describe how
Q_13 | Has the lockdown period allowed you to rediscover activities or passions that you had set aside?
Q_14 | What did you enjoy most during the lockdown period?
Table 5: Questions selected to evoke Neutral Sentiment ID |  Questions
---|---
Q_15 | Please, describe your ordinary day / Which activities do you daily perform?
Q_16 | Describe in detail the path to go shopping
Q_17 | Where do you usually read the news?
Q_18 | Please describe a room of your home in detail
Q_19 | What did you have for lunch/dinner?
Q_20 | Age, gender, place of birth, educational background and work history
Q_21 | Please, describe the place where you live
Q_22 | Please, describe the type of dwelling where you live
Q_23 | Please, describe what a car looks like
Q_24 | Please, describe what a chair looks like
Table 6: Questions selected to evoke Negative Sentiment ID |  Questions
---|---
Q_25 | Tell us about a news story you recently heard that saddened you?
Q_26 | Is there anything in your neighborhood you want to change?
Q_27 | What did you miss most during the lockdown period?
Q_28 | What do you think about the recent price increase?
Q_29 | What is the most dramatic moment in someone’s life?
Q_30 | What activities or hobbies have you been unable to carry out during the lockdown?
Q_31 | Have you ever suffered the loss of a loved one?
Q_32 | Have you ever been let down by someone you cared about?
Q_33 |  Has the economic situation ever affected your happiness during any moment of your life?
Q_34 | Do you remember a moment of your childhood that saddened you deeply?
Q_35 | Have you ever suffered a disappointment of love?
Q_36 | What saddened you most during lockdown?
Q_37 | What saddened you most during the pandemic from a global point of view?
Table 7: Questions not related to a specific sentiment ID |  Questions
---|---
Q_38 | Do you think the experience of the pandemic could have affected you in any way?
Q_39 | What is an object you particularly care about and what does it mean for you?
Q_40 |  What is your perception of the area or the city where you live? How do they feel about your neighbourhood? Is there anything you like or dislike? How do you move within your neighbourhood, and from your home to the rest of the city?
Q_41 |  What is your vision of the future city? How do you imagine the neighbourhood where you live, and the city in general, when your grandchildren will be older adults?
Q_42 |  Think about of a song you like, would you describe an episode of your life related to that song?
Q_43 | What do you think about the experiment you have just performed?