# Combining social media and survey data to nowcast migrant stocks in the
United States

Monica Alexander111University of Toronto<EMAIL_ADDRESS>
Kivan Polimis222University of Washington
Emilio Zagheni333Max Planck Institute for Demographic Research

###### Abstract

Measuring and forecasting migration patterns, and how they change over time,
has important implications for understanding broader population trends, for
designing policy effectively and for allocating resources. However, data on
migration and mobility are often lacking, and those that do exist are not
available in a timely manner. Social media data offer new opportunities to
provide more up-to-date demographic estimates and to complement more
traditional data sources. Facebook, for example, can be thought of as a large
digital census that is regularly updated. However, its users are not
representative of the underlying population. This paper proposes a statistical
framework to combine social media data with traditional survey data to produce
timely ‘nowcasts’ of migrant stocks by state in the United States. The model
incorporates bias adjustment of the Facebook data, and a pooled principal
component time series approach, to account for correlations across age, time
and space. We illustrate the results for migrants from Mexico, India and
Germany, and show that the model outperforms alternatives that rely solely on
either social media or survey data.

##  1 Introduction

Accurate, reliable and timely estimates of migration indicators, such as flows
and stocks, are crucial for understanding population dynamics and demographic
change, for designing effective economic, social and health policies, and for
supporting migrants and their families. However, data on migration from
traditional sources, such as censuses, surveys or administrative registers,
are often insufficient. Even when these sources exist, the data available may
lack the granularity of information required to understand migration trends,
or are not released in a manner that is timely enough to monitor changes in
trends. As migration flows can change substantially over a short period of
time — such as in response to a natural disaster or war and conflict — relying
on out-dated data is often not sufficient.

Timely and reliable information about migration stocks is important not only
to understand migration patterns. It is key also to monitor fertility,
population health and mortality. Even when accurate data on births and deaths
exist, often demographers are faced with large uncertainty in population
counts, which, in various disaggregations, form the denominators for standard
demographic rates. Most of the uncertainty is driven by lack of appropriate
information on how migration stocks change over time and space.

As a consequence of data availability issues, we need to consider how non-
traditional data can be leveraged to complement existing sources in order to
improve estimates and predictions of migration indicators over time. Previous
work has explored the use of data such as call detail records (Blumenstock,
2012; Pestre et al., 2020), air traffic data (Gabrielli et al., 2019), tax
file records (Engels and Healy, 1981) and other sources like billing addresses
or school enrollment (Foulkes and Newbold, 2008) to estimate migration.
Additionally, an increasingly large body of work has investigated the use of
social media data, from websites such as Twitter (Zagheni et al., 2014),
Facebook (Zagheni et al., 2017) and LinkedIn (State et al., 2014). Provided
that the data can be obtained in a reliable, timely, and ethical way,
information about the users of social media websites is potentially an
incredibly rich demographic data source. Data on these populations are
essentially collected in real time, and while individual-level information is
usually restricted, many of the social media websites provide a certain amount
of aggregate-level information through their advertising platforms (Cesare et
al., 2018). In particular, Facebook’s Advertising platform allows information
to be extracted on the relative size of groups by key demographics such as
age, sex, location of residence and country of origin, thereby potentially
acting as a measure of the relative size of migrant groups in a particular
country.

While these data have clear potential for use in demographic research, with
respect to timeliness and the size of the sample being considered, there are
some notable issues that need to be overcome. In particular, for any given
population subgroup of interest, the corresponding users of Facebook or any
other social media platform are unlikely to be a representative sample. An
additional challenge is to use these data in a way that meaningfully combines
new ‘signal’ or information about migration trends with existing knowledge on
probable migration trends from historical data sources.

This paper proposes a statistical framework to combine social media data from
Facebook, with traditional survey data from the ACS, in order to produce
timely ‘nowcasts’ of migrant stocks by state in the United States. The
framework consists of a Bayesian hierarchical model which incorporates bias
adjustment of the Facebook data, a demographic time series approach to account
for historical past trends, and a geographic pooling component which allows
information about the age structure of migrants to be shared across space. The
model also accounts for the different types of uncertainty that are likely to
be present in Facebook and traditional survey data. The resulting model
produces estimates and short-term projections of migrant stocks by US state of
destination and country of origin, and is shown to out-perform valid modeling
alternatives.

The remainder of the paper is structured as follows. First, we briefly discuss
previous demographic research which incorporates social media. Then we outline
the data sources used, and in particular how the Facebook data were collected.
Section 4 discusses the model set-up, assumptions and computation. We then
present results for Mexican, Indian and German migrants by US state, and
validate model performance against reasonable alternatives. Finally, the
strengths and limitations of the model are discussed, together with avenues
for future research.

##  2 Background

The lack of good-quality data on migration is a global problem, with data
sparsity issues prevalent in both developed and developing countries (Landau
and Achiume, 2017). This has prompted scholars to investigate the use of other
types of data to monitor migration trends. In particular, with the rise of
social media use around the world, new data that have potential for
demographic research have emerged.

Scholars began using social media and web data to estimate and track
demographic indicators over time in the early 2010s. The earliest papers
illustrated how geo-located data from email services and web-based
applications such as Twitter, Google Latitude, Foursquare or Yahoo! can be
used (Ferrari et al., 2011; Noulas et al., 2011; Zagheni and Weber, 2012).
Initial research focused on evaluating spatial mobility of populations at a
city or regional level. For example, Ferrari et al. (2011) used Twitter data
to study patterns of urban movement in New York. In the first effort to tackle
global trends, Zagheni and Weber (2012) linked the geographic locations of IP
addresses of Yahoo! emails to the user’s self-reported demographic data to
estimate age- and sex-specific migration flows in a large number of countries
around the world.

Recent efforts have focused on using data from social media and networking
websites such as Twitter, LinkedIn and, more recently, Facebook and Instagram.
These websites provide public access to an Application Programming Interface
(API), which makes it possible to send requests and receive responses from
these websites for data like tweet hashtag counts, the number of jobs in a
certain industry, or number of cell phone users in a particular area.
Researchers have utilized these APIs to extract publicly available demographic
and location data for use in social research, in particular to study outcomes
such as migration (Yildiz et al., 2017; Zagheni et al., 2017), fertility
(Rampazzo et al., 2018), gender equality (Fatehkia et al., 2018; Garcia et
al., 2018), and health (Araujo et al., 2017a). For instance, Garcia et al.
(2018) used Facebook data to create an index of the internet gender divide in
217 countries, showing that this indicator encapsulated gender equality
indices in education, health and economic opportunity. Yildiz et al. (2017)
used a combination of geo-located tweets and image recognition software to
obtain estimates of internal migration in England.

In work relevant to this paper, Zagheni et al. (2017) presented a proof of
concept for estimating migration stocks in the United State by age, sex and
state, using Facebook’s Advertising Platform. More recently, Alexander et al.
(2019) used the same type of data to track changes in migrants over time, in
the context of estimating out-migration from Puerto Rico following Hurricane
Maria in September 2017.

The main gap in the literature is related to the lack of a suitable
statistical model for combining ‘traditional’ data sources on migrants — in
the form of censuses, nationally representative surveys, or other vital
statistics — with migration information from social media data. The goal of
this paper is thus to develop a probabilistic framework that allows
representative and historical time series to be combined, in a sound
statistical framework, with non-representative – but timely – sources from
social media.

##  3 Data

###  3.1 Facebook Advertising data

Facebook for Business has developed a targeted advertising platform, called
Ads Manager that provides a graphical user interface to allow advertisers to
micro-target specific audiences. Demographic characteristics that can be
targeted include information directly reported by Facebook users, such as age
or sex, and information indirectly inferred from using Facebook’s platform or
affiliated websites, such as location and behavioral interests. Before
launching an advertisement, an advertiser can select a variety of
characteristics (e.g., Australians living in California, who are female, and
aged 30-35) and get an estimate of the ‘potential reach’ (monthly active
users) to this subgroup. These estimates can be obtained, in a programmatic
way, for a variety of different expatriate (‘expat’) groups by age, sex, and
education.

We use the estimates of potential reach by expat group, age and sex to track
sizes of migration stocks over time. These estimates can be obtained before
the launch of an advertisement, and as such are obtained free of charge. We
use the Ads Manager backend application, Facebook’s Marketing API, to extract
estimates of potential reach over time programmatically with the Python module
pySocialWatcher (Araujo et al., 2017b). With pySocialWatcher, we collected
data across 11 age groups (10 UN age groups from 15-19 to 60-65; an 11th group
for the entire available Facebook population of 13-65 was also used), three
gender groups (female, male, and total population) and multiple education
categories. Data was collected using Amazon Web Services (AWS) EC2 Instance
servers.

As part of a broader project on using social media in demographic research, we
started data collection in January 2017. For each wave of data collection we
obtained state-level estimates of all Facebook users (by age, sex, and gender)
as well as state-level estimates of 90 expat groups. We have been collecting a
new wave of data every 2-3 months. 444The waves used in this paper are: Wave
1: January 2017; Wave 2: April 2017; Wave 3: June 2017; Wave 4: October 2017;
Wave 5: January 2018; Wave 6: March 2018

###  3.2 American Community Survey

The American Community Survey (ACS) is an annual survey of the U.S. Census
Bureau, designed to supplement the decennial census. Based on the long-form
version of the census, the ACS collects information on topics including
population, housing, employment and education from a nationally representative
sample. Data on migrant stocks can be readily obtained from the ACS. In
particular, in every year of the ACS, the survey has contained a question
asking the birthplace of the person; if it is inside the United States, the
state is recorded, and if it is outside the United States, the country is
recorded. This birthplace variable is recorded as a three digit code to
indicate the US state or country of birth. In addition to the birthplace
variable, the ACS has information on current state of residence. Thus, we can
tabulate the number of migrants from a particular country living in a
particular state by looking at the combination of these two variables. From a
modeling perspective, we are interested in the proportion of migrants from a
particular origin of the total population by five-year age groups
($15-19,20-24,\dots,50-54$) in each state.

We calculated the migrant stock proportions using the 1-year ACS for each year
between 2001 and 2017 using micro-data available through the Integrated Public
Use Microdata (IPUMS) US project (Ruggles et al., 2000). Standard errors
around the calculated proportions based on sampling variation were calculated
based on ACS accuracy guidelines (US Census Bureau, 2020) and using the Delta
method.

##  4 Model

We are considering two data sources of migration trends in the US: data from
Facebook’s Advertising Platform, and the ACS. The overall goal of the modeling
strategy is to combine information from both these sources to produce
estimates of current and future migrant stocks. To do this, the model should
have three main characteristics. Firstly, we want to adjust for biases in
Facebook data to effectively use up-to-date information on migration patterns
from this source. Secondly, we want to be able to incorporate longer time
series of information from the ACS. Finally, the data should be combined in a
probabilistic way, in order to objectively weigh information from both
sources. We propose a Bayesian hierarchical model which achieves these goals.
In this section we describe the model in detail.

For a particular migrant group, define $\mu_{xts}$ to be the (‘true’)
proportion of migrants of the total populations in age group $x$ at time $t$
and in state $s$. This quantity $\mu_{xts}$ is the main parameter of interest
to be estimated. We have observations of this proportion, which will be
denoted $p_{xts}$. The observed proportions are either from Facebook
($p_{xts}^{FB}$) or from the ACS ($p^{ACS}_{xts}$). The $p_{xts}$s are
observed, and it is assumed that these are somehow related to the underlying
true proportions, $\mu_{xts}$, with some associated error. We use the term
‘true’ in a statistical sense, referring to a latent variable of interest.

###  4.1 Facebook bias adjustment

The first goal is to adjust the Facebook data to account for the non-
representativeness of the Facebook user population. Previous research from
Zagheni et al. (2017) showed that, while the bias in the Facebook migrant data
is substantial, it is also relatively systematic by age and migrant group and
can be modelled.

Following their approach, we introduce a regression model which relates the
proportions of migrants in Facbook, $p_{xts}^{FB}$, to the proportions in the
ACS in a similar time period, $p_{xts}^{ACS}$, plus a series of age and state
variables. In particular, for a particular migrant group, express
$p_{xts}^{ACS}$, on a log scale, as

$\log p_{xts}^{ACS}=\alpha_{0}+\alpha_{1}\log p_{xts}^{FB}+\mathbf{\beta
X}+\varepsilon_{FB}$ (1)

where $\mathbf{X}$ is a covariate matrix containing an indicator variable for
each age group ($15-19,20-24,\dots,50-54$) and each of the 50 states plus
Washington D.C. This means that we estimate a fixed effect for each age group
and state. In addition, we assume that the error is i.i.d and that

$\varepsilon_{FB}\sim N(0,\sigma^{2}_{FB}).$

Estimates of the coefficients $\alpha_{0}$, $\alpha_{1}$ and the vector of
$\beta$’s are obtained using the first wave of the Facebook data and the 2016
ACS data. Once obtained, these coefficient estimates are then used to adjust
subsequent waves of Facebook data, i.e. we calculate

$\log p_{xts}^{*}=\hat{\alpha}_{0}+\hat{\alpha}_{1}\log
p_{xts}^{FB}+\mathbf{\hat{\beta}X}$

where $\log p_{xts}^{*}$ is a ‘bias-adjusted’ version of the Facebook data.
This is taken to be our ‘best guess’ of what the migrant stocks in group $xts$
are, based on the Facebook data alone. Note that an estimate of
$\sigma^{2}_{FB}$ is also obtained, that is, the variance of the error terms,
which becomes important in the final model (see section 4.3 below).

###  4.2 Time series modeling of ACS using principal components

In addition to using data from Facebook, we also want to incorporate the
relatively long historical time series of information on migrant stocks
obtained from the ACS. A reasonable short term forecast based on ACS should
model historical trends and project them forward.

There are many different time series models that could be used in this
context. Perhaps the simplest approach would be to project forward a moving
average of the time series for each age group and state combination.
Alternatively, we could use a classical Box-Jenkins approach and model the
time series of migrant stocks in each age group and state separately using an
appropriately specified ARIMA model. However, these methods would not place
any constraints on the age structure of migration. Given this demographic
context, we expect that the age distribution of migration displays strong
patterns and changes in a relatively regular way over time. This is because of
regularities in the age at migration as well as historical trends which
include different waves of migrants, who also age over time. As such, we chose
to incorporate this prior knowledge into our model through a principal
components approach.

Principal component-based models have a long history in demographic modeling,
with the most well known example being the Lee-Carter mortality model (Lee and
Carter, 1992). The idea is that a set of age-specific demographic rates
observed over time can be expressed as a combination of a series of
characteristic patterns (or principal components). The Lee-Carter approach
uses the mean age-specific mortality schedule and first principal component,
which is interpreted as age-specific contributions to mortality change over
time. This model can easily be extended to include higher-order principal
components, which various researchers have done.

Apart from the Lee-Carter model and variants (e.g. Li et al. (2004), Lee
(2000), Renshaw and Haberman (2006)), principal component models have been
recently used to estimate and forecast mortality (e.g. Alexander et al.
(2017)), fertility (e.g. Schmertmann et al. (2014)) and overall population
(Wiśniowski et al. (2015)). Here, we extend this idea to parsimoniously
estimate and project migration stocks by age and state.

####  4.2.1 Model overview

Age-specific migration schedules are decomposed into independent age and time
components. The time component is then projected forward as a time series,
taking auto-correlated error into account. We propose a log-linear model for
$p_{xts}$:

$\log p_{xts}^{ACS}=\beta_{ts,1}Z_{x,1}+\beta_{ts,2}Z_{x,2}+\varepsilon_{xts}$
(2)

where $Z_{x,1}$ and $Z_{x,2}$ are the first and second ‘principal components’,
$\beta_{ts,1}$ and $\beta_{ts,2}$ are state and time-specific coefficients, to
be estimated, and $\varepsilon_{xts}$ is an error term. The principal
components are obtained via Singular Value Decomposition (SVD), as outlined in
the next section. To obtain estimates of $\beta_{ts,1}$ and $\beta_{ts,2}$, we
impose some smoothing over time and pooling of information across space, as
outlined in section 4.2.3. Finally, as discussed in Section 4.2.4 we place a
time series model on the error term, $\varepsilon_{xts}$, to account for auto-
correlation.

####  4.2.2 Obtaining the principal components

The principal component terms $Z_{x,1}$ and $Z_{x,2}$ aim to capture the main
sources of systematic variation in migration patterns across age. They are
obtained by first creating a matrix of (logged) historical age-specific
migration schedules based on ACS data from 2001 to 2016. Singular Value
Decomposition (SVD) is then performed on this matrix to obtain principal
components of the age-specific migration. In particular, let $\bf{X}$ be a
$N\times G$ matrix of log-migration stock rates, where $N$ is the number of
state-years and $G$ is the number of age-groups. In this case, we had $N=51$
states + DC $\times 16$ years $=816$ observations of $G=9$ age-groups
($15-19,20-24,\dots,50-54$). The SVD of $\bf X$ is

$\bf X=\bf{UDV^{\prime}},$ (3)

where $\bf U$ is a $N\times N$ matrix, $\bf D$ is a $N\times G$ matrix and
$\bf V$ is a $G\times G$ matrix. The first two columns of $\bf V$ (the first
two right-singular values of $\bf X$) are $Z_{x,1}$ and $Z_{x,2}$.

For example, Fig. 1 shows the resulting $Z_{.,1}$ and $Z_{.,2}$ for the
Mexican migrant group in the US. These were obtained via the following steps:

  1. 1.

Calculate $p_{xts}^{ACS}$, i.e. the proportion of migrants in age group $x$,
year $t$ and state $s$ for each age group, year and state in ACS 2001-2016.

  2. 2.

Create $\bf{X}$ where each element is $\log p_{xts}^{ACS}$, every row is a
state-year and every column is an age group.

  3. 3.

Perform SVD on $\bf X$ and extract the first two columns555We used the ‘svd’
function in R. of $\bf V$.

Figure 1: Principal Components for Mexico

The principal components shown in Fig. 1 can be interpreted as a baseline
migration age schedule ($Z_{.,1}$) and age-specific contributions to change
over time ($Z_{.,2}$). In the model, the coefficient on $Z_{.,1}$
($\beta_{ts,1}$) moves the overall level of Mexican migrants up or down,
depending on the year and state. The coefficient on $Z_{.,2}$ allows the age
distribution to shift to older or younger ages. For $Z_{.,2}$, the sign
changes from negative to positive at age 35. This means that the larger and
more positive the value of $\beta_{ts,2}$, the older the migrant age
distribution.666Note that the interpretation of $Z_{.,1}$ and $Z_{.,2}$ is
similar to the interpretation of the $a_{x}$ and $b_{x}$ terms in the usual
Lee-Carter model.

####  4.2.3 Sharing information across time and space

The model specified in Eqn. 2 requires the estimation of two coefficients,
$\beta_{ts,1}$ and $\beta_{ts,2}$ for each time $t$ and state $s$. One option
would be to estimate each of these coefficients separately for every year and
state. However, we would like to incorporate the knowledge that trends in
migration over time are likely to exhibit relatively regular patterns. In
addition, for the coefficient on the second principal component — which allows
for the age distribution of migrants to shift to the left or right, we would
like to share information about the patterns in migration across geographic
space.

The coefficient on the first principal component, $\beta_{ts,1}$, is modeled
as a random walk, i.e.

$\beta_{ts,1}\sim N(\beta_{t-1s,1},\sigma_{\beta_{1}}^{2})$

This allows for information about the level of migration within each state to
be smoothed over time. The random walk structure allows for the estimate in
the current time period, $\beta_{ts,1}$, to be partially informed by the
previous period.

For the coefficient on the second principal component, we place the following
hierarchical structure on the $\beta$’s:

$\beta_{ts,2}\sim N(\Phi_{t},\sigma_{\beta}^{2})$ $\Phi_{t}\sim
N(\Phi_{t-1},\sigma_{\Phi}^{2})$

where $p=1,2$. The $\Phi_{t}$ term represents essentially a national mean; as
such the $\beta_{ts,2}$’s are a draw from a national distribution with some
mean and variance. In this way, information about how the age distribution is
ageing over time is shared across states. The more information about migration
there is available for a particular state (i.e., the larger the migrant
population), the less the estimate of $\beta_{ts,2}$ is influenced by the
overall mean. Conversely, states with smaller migrant populations where the
trends over time are less clear from the data are partially informed by
patterns in larger states.

Note that the geographical hierarchical structure is not present on the first
coefficient, as this represents an overall level of migration. Pooling
information across space about the level of migration would artificially
increase migrant proportions in smaller states.

####  4.2.4 Auto-correlated error

The final piece of the time series model is the error term
$\varepsilon_{xts}$. This term is included in the model to allow for extra
variation in migration age schedules that is not otherwise picked up by the
principal components. We expect the extra variation to be autocorrelated, and
as such we model the error term as an AR(1) process:

$\varepsilon_{xts}\sim
N(\rho_{xs}\varepsilon_{xt-1s},\sigma_{\varepsilon}^{2})$

where $\rho_{xs}\in[0,1]$.

####  4.2.5 Projection

The model described above is fit to ACS data from 2001-2016. However,
estimates in more recent years can easily be obtained by projecting the time
series aspects of this model forward. In particular, for time $t+1$:

  * •

Obtain an estimate for $\beta_{t+1s,1}$ from $\beta_{t+1s,1}\sim
N(\beta_{ts,1},\sigma^{2}_{\beta})$.

  * •

Obtain an estimate for $\beta_{t+1s,2}$ from $\beta_{t+1s}\sim
N(\Phi_{t+1},\sigma^{2}_{\Phi})$ and $\Phi_{t+1}\sim
N(\Phi_{t},\sigma_{\Phi}^{2})$.

  * •

Obtain an estimate for $\varepsilon_{xt+1s}$ from $\varepsilon_{xt+1s}\sim
N(\rho_{xs}\varepsilon_{xts},\sigma_{\varepsilon}^{2}).$

  * •

Calculate $\log p_{xt+1s}^{ACS}$ based on Eqn. 2.

###  4.3 Bringing it all together

Sections 4.1 and 4.2 described two ways to obtain current ‘nowcasts’ of
migrant stocks. One option would be to take the most recent data obtained from
Facebook, adjust using the bias-adjustment model, and take the resulting
estimate as our nowcast. Another option would be to project forward the ACS
model to the time period of interest. Ideally, we would like to incorporate
both sources into our final estimate. Perhaps an option would be to just take
an average of the two resulting estimates. However, we would like to weigh the
estimates from both sources more objectively, taking different sorts of
uncertainty into consideration.

Our solution is to combine both models into one framework, and use the results
from both methods as data points for our ‘best estimate’ nowcast. This is
illustrated in Fig. 2. Facebook inputs are calibrated with the ACS via the
adjustment model. ACS data are used to obtained principal components based on
past migration data. The modeling structure allows for information exchange
over time and across geographic space. The key piece of the combined model,
which has yet to be explained, is the data model (or likelihood), which allows
data from the different sources to have different associated error.

Figure 2: Modeling framework

####  4.3.1 Data model

As outlined above, we observe migrant proportions $p_{xts}$ from either
Facebook or the ACS. The data model assumes

$\log p_{xts}\sim N(\log\mu_{xts},\sigma_{p}^{2})$

i.e. the log of the observed proportion is assumed to have mean
$\log\mu_{xts}$ and variance $\sigma_{p}^{2}$, where $\sigma_{p}^{2}$ depends
on the data source:

$\sigma_{p}^{2}=\begin{cases}\sigma_{s}^{2},&\text{if ACS}\\\
\sigma_{s}^{2}+\sigma_{FB}^{2}+\sigma_{ns}^{2},&\text{if
Facebook}\end{cases}\\\ $

Here, $\sigma^{2}_{s}$ refers to sampling error, and is assumed to be present
in both ACS and Facebook data. For the ACS data, sampling errors are
calculated based on guidelines from the US Census Bureau (2020). For Facebook
data, the sampling error is calculated assuming the binomial approximation to
the Normal distribution and calculating

$\sigma^{2}_{s}=\frac{p_{xts}\cdot(1-p_{xts})}{N_{xts}^{FB}}$

where $N_{xts}^{FB}$ is the total size of the Facebook population in subgroup
$x,t,s$.

For the Facebook data there are two additional error terms. $\sigma_{FB}^{2}$
refers to the error associated with our bias-adjustment model (Eqn. 1) and is
estimated within this model. This captures the fact that our adjustment model
is imperfect and that extra variation remains. Additionally, we allow for a
non-sampling error with $\sigma_{ns}^{2}$, which aims at capturing additional
uncertainty like variation in the way potential reach is estimated across
waves.

For a given population size, the sampling error is going to be of similar size
for ACS and Facebook data. As such, the error term associated with the
Facebook data, which is the sum of three terms, will always be bigger than for
ACS. In practice, this means that estimates from the model will follow (i.e.
give more weight to) the ACS data.

####  4.3.2 Summary of full model

The full model is summarized below. Equation 4 is the data model. Equations
5-9 relate to the ACS time series model. Equation 12 is related to the
Facebook regression model. Equations 10 and 11 allow the observation of the
proportion of interest to come from a different source (Facebook or ACS),
which has a different associated variance. Note that $\mu_{xts}$ is estimated
on a yearly basis, but it is assumed that $j$ waves of Facebook data are
collected within any one year.

$\displaystyle\log p_{xts}$ $\displaystyle\sim$ $\displaystyle
N(\log\mu_{xts},\sigma^{2})$ (4) $\displaystyle\log\mu_{xts}$ $\displaystyle=$
$\displaystyle\beta_{ts,1}Z_{x,1}+\beta_{ts,2}Z_{x,2}+\varepsilon_{xts}$ (5)
$\displaystyle\beta_{ts,1}$ $\displaystyle\sim$ $\displaystyle
N(\beta_{t-1s,1},\sigma_{\beta_{1}}^{2})$ (6) $\displaystyle\beta_{ts,2}$
$\displaystyle\sim$ $\displaystyle N(\Phi_{t,2},\sigma_{\beta}^{2})$ (7)
$\displaystyle\Phi_{t}$ $\displaystyle\sim$ $\displaystyle
N(\Phi_{t-1},\sigma_{\Phi}^{2})$ (8) $\displaystyle\varepsilon_{xts}$
$\displaystyle\sim$ $\displaystyle
N(\rho_{xs}\varepsilon_{xt-1s},\sigma_{\varepsilon}^{2})$ (9) $\displaystyle
p_{xts}$ $\displaystyle=$ $\displaystyle\begin{cases}p_{xts}^{ACS},&\text{if
}2001\leq t\leq 2016\\\ p_{xtsj}^{*},&\text{if }t\geq 2017\end{cases}$ (10)
$\displaystyle\sigma^{2}$ $\displaystyle=$
$\displaystyle\begin{cases}\sigma_{s}^{2},&\text{if ACS}\\\
\sigma_{s}^{2}+\sigma_{FB}^{2}+\sigma_{ns}^{2},&\text{if Facebook}\\\
\end{cases}$ (11) $\displaystyle p_{xtsj}^{*}$ $\displaystyle\sim$
$\displaystyle N({\alpha_{0}}+{\alpha_{1}}\cdot p_{xtsj}^{\text{
Facebook}}+X{\Gamma},\sigma^{2}_{FB})$ (12)

####  4.3.3 Priors

Weakly-informative priors were placed on the coefficients in the Facebook
bias-adjustment model, as well as the principal component coefficients in the
initial periods:

$\displaystyle\alpha_{0}$ $\displaystyle\sim$ $\displaystyle N(0,100)$
$\displaystyle\alpha_{1}$ $\displaystyle\sim$ $\displaystyle N(0,100)$
$\displaystyle\Gamma_{0}$ $\displaystyle\sim$ $\displaystyle N(0,100)$
$\displaystyle\beta_{1,s,1}$ $\displaystyle\sim$ $\displaystyle N(0,100)$
$\displaystyle\Phi_{1}$ $\displaystyle\sim$ $\displaystyle N(0,100)$

In addition, we put weakly-informative half-Normal priors on the two standard
deviation terms to be estimated:

$\displaystyle\sigma_{FB}$ $\displaystyle\sim$ $\displaystyle N_{+}(0,1)$
$\displaystyle\sigma_{ns}$ $\displaystyle\sim$ $\displaystyle N_{+}(0,1).$

####  4.3.4 Computation

The model was fitted in a Bayesian framework using the statistical software R.
Samples were taken from the posterior distributions of the parameters via a
Markov Chain Monte Carlo (MCMC) algorithm. This was performed using JAGS
software (Plummer et al., 2003). Standard diagnostic checks using trace plots
and the Gelman and Rubin diagnostic were used to check convergence (Gelman et
al., 2013).

Best estimates of all parameters of interest were taken to be the median of
the relevant posterior samples. The 95% Bayesian credible intervals were
calculated by finding the 2.5% and 97.5% quantiles of the posterior samples.

All code and data are available on GitHub: https://github.com/MJAlexander/fb-
migration-bayes

##  5 Results

We illustrate the model on male migrants from three different countries:
Mexico, India, and Germany. These three migrant groups represent three
different scenarios of levels and trends over time, as illustrated by the
trends in the ACS data shown in Figures 3(a) and 3(b).

Firstly, Mexican migrants make up a relatively large share of the overall
population, but the proportion has been generally declining since around 2007.
The age distribution at the national level peaks in the 40-44 year old age
group. Secondly, Indian migrants make up a moderate proportion of the total
population, but this share is increasing over time. The age distribution peaks
at younger ages (30-34), compared to Mexicans. Finally, German migrants make
up a low and declining share of the population. In contrast to the other
migrant groups, the age distribution of German migrants at the national level
is relatively flat, increasing slightly across age.

(a) By proportion of the total population

(b) By proportion of each age group, 2016

Figure 3: German, Indian and Mexican migrants

###  5.1 Bias adjustment of Facebook data

We firstly illustrate the results of the bias-adjustment step of the Facebook
data. Figure 4 shows, for each US state and five-year age group where data are
available, the proportion of migrants in each age group for the ACS data in
2016 (black dots), the un-adjusted Facebook data (blue dots), and the
estimated bias-adjusted Facebook data (red line and associated shaded area)
for Mexican migrants. Similar plots for migrants from India and Germany are
shown in Appendix A. The interpretation is that if the bias-adjustment step is
working reasonably well, the red line would be close to the black dots. In
general, this appears to be the case.

For all three migrant groups, the raw Facebook data are generally lower than
the ACS data, but the bias-adjustment model adjusts these values upwards. In
general, across the three migrant groups and across states, the shape of the
age distributions in the Facebook and ACS data are similar, with more
substantial under-representation in Facebook in the older age groups. These
systematic differences mean that the model works well to adjust the raw
Facebook data based on age and state effects.

Figure 4: Bias adjustment of Facebook data for Mexican migrants

###  5.2 Nowcasts by age group and state

Now we move on to short term projections by age and state. Figure 5 shows the
estimated age distribution in 2008 (red) and projected distribution in 2018
(blue) for Mexican migrants. Similar plots for India and Germany can be found
in Appendix LABEL:appendix_age_facets.

For Mexico (Figure 5), the relatively high proportions in the border states
and on the West coast are apparent, with the highest proportions in
California, Texas, Nevada and Arizona. Additionally, the age distribution of
Mexican migrants is generally aging over time (shifting to the right), which
is consistent with relatively constant stocks.

Figure 5: Estimated and projected age distributions of Mexican migrants by
state, 2008 and 2018

####  5.2.1 Projected time series

Figures 6 to 8 zoom in on two states for each migrant group and show how the
Facebook data are used to project forward the time series to the most recent
two years (2017 and 2018). The full estimated and projected time series from
2001 to 2018 is shown. In the figures, each facet is a five-year age group.
The red dots and associated shaded area represent the ACS data and sampling
standard errors; these data are broadly available from 2001 to 2016, although
some observations are missing (if sample sizes in the ACS were too small to
capture information about migrants in that particular state and age group).
The blue dots represent the (adjusted) Facebook observations, which are
available in years 2017 and 2018. The black line and associated shaded area is
the model estimate and 95% uncertainty intervals.

Mexican males in California (Figure 6(a)) represent by far the highest
proportions of any of the migrant origin/ state combinations considered. The
proportion is as high as 0.25 in some age groups, for example 25-29 year olds
in 2001 and 40-44 year olds in 2018. As a consequence, the sampling error
around the ACS data for this migrant group is relatively small and the model
estimates closely follow these data. For the most recent two years, where only
Facebook data are available, note that the model estimates do not follow the
data as closely and the uncertainty around the model estimates increases. This
reflects the fact that there are more sources of error associated with the
Facebook data.

In Georgia (Figure 6(b)) the levels of Mexican migrants are around half as
high as in California. Due to smaller sample sizes in Georgia, the standard
errors around the ACS data are much larger, and as such the model estimates do
not follow the data as closely. However, the trends for Mexican migrants in
California, and Georgia are broadly the same: decreases in the younger age
groups, and increases in the older age groups, representing an aging stock of
migrants.

For Indian males in California and Georgia (Figure 7), the proportions are
much lower than for the Mexican migrant population, peaking at around 3-4% of
the population in the 30-39 year old age groups. The proportions are
increasing over time, however, particularly in the 25-44 age bracket. Finally,
for German male migrants in California and Georgia (Figure 8), we see low and
constant migrant proportions. The uncertainty around the ACS data is already
relatively high, and so there is not so much of an increase in uncertainty in
the final two years.

(a) California

(b) Georgia

Figure 6: Mexican male migrants by age group, California and Georgia,
2001-2018

(a) California

(b) Georgia

Figure 7: Indian male migrants by age group, California and Georgia, 2001-2018

(a) California

(b) Georgia

Figure 8: German male migrants by age group, California and Georgia, 2001-2018

##  6 Validation

We evaluated the performance of the Bayesian model compared to other
reasonable forecasting alternatives. To do this, we ran the model on data from
2001 to 2016, and forecast migration stocks in 2017. We then compared these
forecasts to the actual ACS data in 2017. We compared the accuracy of the
Bayesian model forecast to forecasts produced by three other models:

  1. 1.

Three-year moving average of the ACS data. This is one of the simplest options
available and does not require the Facebook data or any statistical modeling.

  2. 2.

Facebook data only: Estimates are based just on the available Facebook data in
2017, after it has been adjusted for biases.

  3. 3.

ACS time series model: Here, we ran the Bayesian hierarchical time series
model described in section 4 above, but just using data from the ACS (no
Facebook).

In order to assess model performance, we compare the root mean squared error
(RMSE):

$RMSE=\sqrt{\frac{\sum_{n}\left(\hat{p}_{g,2017}-p^{ACS}_{g2017}\right)^{2}}{N}}$
(13)

where $\hat{p}_{g,2017}$ is the estimated proportion of migrants from a
particular group $g$, $p^{ACS}_{g2017}$ is the equivalent proportion from the
ACS and $N$ is the size of the group. Here, the $g$ can refer to any
combination of age group, state and migrant origin.

Table 1 shows the overall RMSE for the four models for Mexican, Indian and
German migrants. The main result is that in each of the three migrant groups,
the Bayesian model presented (which combines the ACS and bias-adjusted
Facebook data and thus is referred to as the ’combined model’), produces the
lowest RMSE and thus the most accurate forecasts. The overall results also
illustrate that the Bayesian hierarchical time series model produces
substantially more accurate forecasts compared to a simple moving average or
the bias-adjusted Facebook data alone, producing RMSEs that are up to an order
of magnitude smaller. This gain in accuracy is much larger than the gain
moving from ACS-only to the combined model, although there is still a gain in
each case.

Model | Mexico | India | Germany
---|---|---|---
Moving average | 0.01280 | 0.0480 | 0.0129
Facebook | 0.01210 | 0.00584 | 0.0142
ACS | 0.00995 | 0.00453 | 0.00263
Combined | 0.00970 | 0.00356 | 0.00261
Table 1: Overall RMSE by model and migrant origin

Figure 9 illustrates the RMSE by age group and model type for each of the
three migrant groups. Similar plots by state can be found in Appendix C. For
Mexico, there is generally an incremental decline in the RMSE moving from the
Facebook-only model, to moving average, to ACS, to the combined model. The
RMSE is highest in the 30-34 year old age group, which is also where the
proportion of migrants is highest (see Figure 6). For India (Figure 9b)), the
RMSE is particularly high from the Facebook-only model in the 25-34 age
bracket. As a consequence, the combined model RMSEs in those age groups are
higher in the combined model versus the ACS model. For Germany, the gain in
accuracy in moving to the hierarchical time series set-up is much larger. This
is most likely related to the fact that the proportions of German migrants are
in general a lot lower than for Mexico or India, and so there are noticeable
gains in pooling information across state, age and time.

(a) Mexico

(b) India

(c) Germany

Figure 9: RMSE by age group and model

To summarize, the validation exercise comparing the one-year-out predictions
from a range of models to the migrant proportions reported in the ACS
illustrate both (i) the strength of the proposed Bayesian hierarchical time
series model as a general framework, and (ii) the additional information
obtained from including up-to-date Facebook data compared to just historical
ACS data alone.

##  7 Discussion

As the size and frequency of migration movements continues to increase
worldwide, new sources of data are being considered in order to better
understand both historical and future migration trends. There is a growing
body of work considering the feasibility of using social media data to achieve
such goals, from platforms such as Facebook, Twitter and LinkedIn. While the
granularity of social media use varies widely (from individual geo-tagged
tweets, to aggregated advertising demographic data, as is in this paper), the
common challenges of using such data remain: firstly, to adequately adjust for
known biases in the social media data, primarily as a consequence of the non-
representativeness of the population of social media users; and secondly, to
meaningfully combine information from social media data with information from
more traditional data sources, such as surveys or censuses.

In this paper we presented a statistical framework to achieve these goals in
the context of producing short-term projections of migrant stocks in the
United States. The model includes a bias-adjustment process of the Facebook
data, and a ‘principal components time series’ model, which allows for the
projection of trends in stocks into the future, considering both Facebook and
ACS data. The model allows for different types of uncertainty around the
different data sources, and shares information on migration trends over time
and pools across geographic space. Illustrative results were presented for
three separate migrant groups: Mexicans, Indians and Germans. The results of
the validation exercise, comparing projections with 2017 ACS data, suggest
that the proposed model improves prediction of short-term trends when compared
to viable alternatives.

The validation exercise illustrated the substantial gain in accuracy achieved
when moving to the Bayesian hierarchical time series model, regardless of
whether or not the Facebook data were included. While the benefits of
including the Facebook data in this particular case were relatively marginal,
more generally the Facebook data has the advantage of being up-to-date and
essentially available in real time. Thus, in a situation of a ‘shock’, such as
a natural disaster or other event, the collection of Facebook data allows for
a more immediate estimate of the effects of that shock on migration. The
combination of these data with past trends allows for the identification of
surprising increases or decreases, that are out of the expected bounds based
on historical patterns.

There are several limitations of the proposed model, which naturally lead into
avenues for future work. Firstly, the bias-adjustment model assumes that the
systematic bias in the Facebook data (by age and state) is constant over time.
In reality, it is reasonable to believe that the biases in the Facebook data
are changing over time, as the composition of the underlying Facebook
population changes. The relationship between the age/location composition of
the Facebook and the actual population (as measured by the ACS) could be
investigated in future work. Secondly, the bias-adjustment model also assumes
that the non-sampling error is constant over Facebook’s ‘waves’ of data
collection — that is, sources of error that include changes in how the
population of reach is calculated, or other computational reasons, are assumed
to be constant. In practice, and in other work using these data (Alexander et
al. 2019), we have observed that this is probably not the case, and needs to
be further investigated to better understand non-migration-related
fluctuations over time. While we only consider two data sources in this paper,
the general statistical framework could easily be extended to include
information from other sources. Future work could also include taking
advantage of the rich demographic and socioeconomic data available through the
Facebook Advertising Platform, including information on education and
occupation.

While this work focused on a model for the estimation of migrant stocks, the
philosophy of combining social media data with more traditional data sources
in one statistical framework – allowing for different sources of uncertainty –
can be readily extended to model other demographic indicators. Indeed, the
underlying time series model was itself an extension of principal component
techniques that have been previously been used in demography to study
mortality and fertility. We show the strength of combining more traditional
demographic modeling techniques and data with newer sources of data to gain
insights into underlying population processes.

##  Appendix A Bias adjustment plots for migrants from India and Germany

Figure 10: Bias adjustment of Facebook data for Indian migrants Figure 11:
Bias adjustment of Facebook data for German migrants

##  Appendix B Age distributions in 2008 and 2018 for Indian and Germany
migrants by state

For India (Figure 12) we see that the proportions across age groups have
generally increased over the decade, with relatively high proportions
concentrated in the northeast region of the country. Unlike Mexico, the age
distribution is fairly constant, with the highest proportions generally being
in the 30-34 age group. The German male migrant populations (Figure 13) by
state show relatively flat, low and unchanging levels over the decade
2008–2018.

Figure 12: Estimated and projected age distributions of Indian migrants by
state, 2008 and 2018 Figure 13: Estimated and projected age distributions of
German migrants by state, 2008 and 2018

##  Appendix C Validation results by state

The figures below show the RMSE for each model, by state of destination in the
US, for Mexican, Indian and German migrants. The figures show that, in
general, the combined model out-performs the other three alternative models.
For Mexico (Figure 14), this is particularly the case for the states with
relatively high proportions of Mexican migrants. Results are more variable for
Indian migrants (Figure 15), with the combined model performing relatively
poorly in the West coast states but well in the mid-West states. For German
migrants (Figure 16), the geographic pooling appears to vastly improve the
projection accuracy.

Figure 14: RSME by state and model type for Mexican migrants in 2018 Figure
15: RSME by state and model type for Indian migrants in 2018 Figure 16: RSME
by state and model type for German migrants in 2018

## References

  * Alexander et al. (2017) Alexander, M., E. Zagheni, and M. Barbieri (2017).  A flexible bayesian model for estimating subnational mortality.  Demography 54(6), 2025–2041.
  * Alexander et al. (2019) Alexander, M., E. Zagheni, and K. Polimis (2019).  The impact of hurricane maria on out-migration from puerto rico: Evidence from facebook data.  Population and Development Review 45(3), 617–630.
  * Araujo et al. (2017a) Araujo, M., Y. Mejova, I. Weber, and F. Benevenuto (2017a).  Using facebook ads audiences for global lifestyle disease surveillance: Promises and limitations.  In Proceedings of the 2017 ACM on Web Science Conference, pp. 253–257.
  * Araujo et al. (2017b) Araujo, M., Y. Mejova, I. Weber, and F. Benevenuto (2017b).  Using facebook ads audiences for global lifestyle disease surveillance: Promises and limitations.  WebSci ’17, New York, NY, USA. ACM.
  * Blumenstock (2012) Blumenstock, J. E. (2012).  Inferring patterns of internal migration from mobile phone call records: evidence from rwanda.  Information Technology for Development 18(2), 107–125.
  * Cesare et al. (2018) Cesare, N., H. Lee, T. McCormick, E. Spiro, and E. Zagheni (2018).  Promises and pitfalls of using digital traces for demographic research.  Demography 55(5), 1979–1999.
  * Engels and Healy (1981) Engels, R. A. and M. K. Healy (1981).  Measuring interstate migration flows: an origin—destination network based on internal revenue service records.  Environment and Planning A 13(11), 1345–1360.
  * Fatehkia et al. (2018) Fatehkia, M., R. Kashyap, and I. Weber (2018).  Using facebook ad data to track the global digital gender gap.  World Development 107, 189–209.
  * Ferrari et al. (2011) Ferrari, L., A. Rosi, M. Mamei, and F. Zambonelli (2011).  Extracting urban patterns from location-based social networks.  In Proceedings of the 3rd ACM SIGSPATIAL International Workshop on Location-Based Social Networks, pp. 9–16.
  * Foulkes and Newbold (2008) Foulkes, M. and K. B. Newbold (2008).  Using alternative data sources to study rural migration: examples from illinois.  Population, Space and Place 14(3), 177–188.
  * Gabrielli et al. (2019) Gabrielli, L., E. Deutschmann, F. Natale, E. Recchi, and M. Vespe (2019).  Dissecting global air traffic data to discern different types and trends of transnational human mobility.  EPJ Data Science 8(1), 1–24.
  * Garcia et al. (2018) Garcia, D., Y. M. Kassa, A. Cuevas, M. Cebrian, E. Moro, I. Rahwan, and R. Cuevas (2018).  Analyzing gender inequality through large-scale facebook advertising data.  Proceedings of the National Academy of Sciences 115(27), 6958–6963.
  * Gelman et al. (2013) Gelman, A., J. B. Carlin, H. S. Stern, D. B. Dunson, A. Vehtari, and D. B. Rubin (2013).  Bayesian data analysis.  Chapman and Hall/CRC.
  * Landau and Achiume (2017) Landau, L. B. and E. T. Achiume (2017).  International migration report 2015: Highlights.  DEVELOPMENT AND CHANGE 48(5), 1182–1195.
  * Lee (2000) Lee, R. (2000).  The lee-carter method for forecasting mortality, with various extensions and applications.  North American actuarial journal 4(1), 80–91.
  * Lee and Carter (1992) Lee, R. D. and L. R. Carter (1992).  Modeling and forecasting us mortality.  Journal of the American statistical association 87(419), 659–671.
  * Li et al. (2004) Li, N., R. Lee, and S. Tuljapurkar (2004).  Using the lee–carter method to forecast mortality for populations with limited data.  International Statistical Review 72(1), 19–36.
  * Noulas et al. (2011) Noulas, A., S. Scellato, C. Mascolo, and M. Pontil (2011).  An empirical study of geographic user activity patterns in foursquare.  In Fifth international AAAI conference on weblogs and social media.
  * Pestre et al. (2020) Pestre, G., E. Letouzé, and E. Zagheni (2020).  The abcde of big data: assessing biases in call-detail records for development estimates.  The World Bank Economic Review 34(Supplement_1), S89–S97.
  * Plummer et al. (2003) Plummer, M. et al. (2003).  Jags: A program for analysis of bayesian graphical models using gibbs sampling.  In Proceedings of the 3rd international workshop on distributed statistical computing, Volume 124, pp. 1–10. Vienna, Austria.
  * Rampazzo et al. (2018) Rampazzo, F., E. Zagheni, I. Weber, M. R. Testa, and F. Billari (2018).  Mater certa est, pater numquam: What can facebook advertising data tell us about male fertility rates?  In Twelfth International AAAI Conference on Web and Social Media.
  * Renshaw and Haberman (2006) Renshaw, A. E. and S. Haberman (2006).  A cohort-based extension to the lee–carter model for mortality reduction factors.  Insurance: Mathematics and economics 38(3), 556–570.
  * Ruggles et al. (2000) Ruggles, S., C. A. Fitch, P. K. Hall, and M. Sobek (2000).  Ipums-usa: Integrated public use microdata series for the united states.  Handbook of international historical microdata for population research. Minneapolis: Minnesota Population Center, 259–284.
  * Schmertmann et al. (2014) Schmertmann, C., E. Zagheni, J. R. Goldstein, and M. Myrskylä (2014).  Bayesian forecasting of cohort fertility.  Journal of the American Statistical Association 109(506), 500–513.
  * State et al. (2014) State, B., M. Rodriguez, D. Helbing, and E. Zagheni (2014).  Migration of professionals to the us.  In International Conference on Social Informatics, pp. 531–543. Springer.
  * US Census Bureau (2020) US Census Bureau (2020).  PUMS accuracy. https://www2.census.gov/programs-surveys/acs/tech_docs/pums/accuracy/.
  * Wiśniowski et al. (2015) Wiśniowski, A., P. W. Smith, J. Bijak, J. Raymer, and J. J. Forster (2015).  Bayesian population forecasting: extending the lee-carter method.  Demography 52(3), 1035–1059.
  * Yildiz et al. (2017) Yildiz, D., J. Munson, A. Vitali, R. Tinati, and J. A. Holland (2017).  Using twitter data for demographic research.  Demographic Research 37, 1477–1514.
  * Zagheni et al. (2014) Zagheni, E., V. R. K. Garimella, I. Weber, and B. State (2014).  Inferring international and internal migration patterns from twitter data.  In Proceedings of the 23rd International Conference on World Wide Web, pp. 439–444.
  * Zagheni and Weber (2012) Zagheni, E. and I. Weber (2012).  You are where you e-mail: using e-mail data to estimate international migration rates.  In Proceedings of the 4th annual ACM web science conference, pp. 348–351.
  * Zagheni et al. (2017) Zagheni, E., I. Weber, and K. Gummadi (2017).  Leveraging facebook’s advertising platform to monitor stocks of migrants.  Population and Development Review 43(4), 721–734.


# D2SLAM: Semantic visual SLAM based on the influence of Depth for Dynamic
environments

1st Ayman Beghdadi  Computer Engineering Department
IBISC Lab, Paris-Saclay University
Evry, France
<EMAIL_ADDRESS>2nd Malik Mallem  Computer Engineering
Department
IBISC Lab, Paris-Saclay University
Evry, France
<EMAIL_ADDRESS>3rd Lotfi Beji  Mechanical Engineering Department
IBISC Lab, Paris-Saclay University
Evry, France
<EMAIL_ADDRESS>
###### Abstract

Taking into account the dynamics of the scene is the most effective solution
to obtain an accurate perception of unknown environments within the framework
of a real autonomous robotic application. Many works have attempted to address
the non-rigid scene assumption by taking advantage of deep learning
advancements. Most new methods combine geometric and semantic approaches to
determine dynamic elements that lack generalization and scene awareness. We
propose a novel approach that overcomes the limitations of these methods by
using scene depth information that refines the accuracy of estimates from
geometric and semantic modules. In addition, the depth information is used to
determine an area of influence of dynamic objects through our Objects
Interaction module that estimates the state of both non-matched keypoints and
out of segmented region keypoints. The obtained results demonstrate the
efficacy of the proposed method in providing accurate localization and mapping
in dynamic environments.

###### Index Terms:

Visual SLAM, Dynamic environments, Robust, Semantic segmentation, Depth
influence

##  I Introduction

Visual Simultaneous Localization And Mapping (vSLAM) has been widely
investigated over the past decade for deploying many applications in robotics,
Augmented Reality (AR), Virtual Reality (VR), Micro Aerial Vehicle (MAV), or
Unmanned Autonomous Vehicles (UAV) [1, 2, 3]. This technology is based on
extracting and interpreting visual information to locate the robot in an
unknown environment [4]. Generally, the proposed methods rely on the rigid
scene assumption to design high-performance vSLAM frameworks. However, their
deployment in real applications has highlighted the impact of scene dynamics
on their performance. Indeed, the vSLAM process attempts to locate the camera
based on its environment using visual landmarks tracked during the camera’s
movement. Thus, moving visual landmarks would imply an error in location
estimation and mapping, affecting thus its performance. To solve this problem,
many methods [5, 6, 8, 7] deal with the dynamics of the scene through two main
approaches used jointly or separately. These approaches classify keypoints
into static and dynamic states in order to reject those in a dynamic state
from the SLAM process. First, the semantic segmentation approach determines
the dynamic objects according to their nature and the type of environment,
where only humans are dynamic in indoor environments. Many indoor methods [6]
perform semantic segmentation into person class in order to reject keypoints
of likely dynamic objects. This approach is quite efficient but lacks
flexibility due to the limited zone of the segmented region and the
uncertainty around the mask edges. Subsequently, the geometric approach [9]
was introduced to consider the whole image and deal with the other moving
objects. It estimates the reprojection error of matched keypoints of two
successive frames using the epipolar geometry constraint. A reprojection error
above a certain threshold implies that the considered keypoint is dynamic and
therefore should be rejected. Unfortunately, this approach has a limited
impact since it only determines the state of the matched keypoints. To
overcome these limitations we propose a novel approach based on the ORB-SLAM3
framework [10] by taking into account the influence of scene depth on the
geometric and semantic segmentation reasoning. It is worth noticing that the
geometric approach is based on 2D reasoning without considering the 3D
information from the depth. This is one of its limitations. Indeed, the
displacement of a distant point induces a smaller reprojection error than a
closer moving point with the same motion amplitude. Therefore, we introduce an
adaptive threshold related to the point depth to account for the relative
reprojection error. Each reprojection error of a matched keypoint lower than
this adaptive threshold is then fed into a normal probability distribution
model to estimate the corresponding motion probability. In contrast, keypoints
with an error higher than the threshold are considered dynamic and rejected.
For semantic reasoning, some methods [7, 11] handle the state uncertainty of
the points around the inner edges of the mask by using a probabilistic
function based either on the minimum distance from these points to the edges
or on the camera motion. This reasoning leads to ambiguity and imprecision in
the way objects at different distances are taken into account in the
estimation of the keypoints states. Indeed, a probabilistic function without
depth consideration induces a constant size of the uncertainty zone, which
would inevitably produce inconsistent results in the estimation of keypoints
states. To solve this inconsistency, we introduce an adaptive impact factor in
the proposed keypoints state probability. This allows to adjust the size of
the area according to the depth of the segmented objects.

Figure 1: Architecture of our proposed method

Furthermore, to overcome the raised limitations of the existing methods, we
propose a novel approach that combines information from geometric and semantic
approaches with depth information to determine the state of some keypoints
that are neither matched nor from a segmented area. Our method is based on the
assumption that in indoor environments dynamics result only from humans
activity. Thereby, the probability that static objects become dynamic
increases if they are within the zone of human influence, i.e. at a depth and
an image position close to human. The neighborhood of the non-matched
keypoints in this zone of influence is analyzed to determine if there are
dynamic points resulting from the geometric approach, which would have the
proximity of position and depth with this one. The spatial information about
the elements within this neighborhood informs us about the probability of the
state of this unpaired point. By doing so we could overcome the limitations of
the geometric approach. The main contributions of our work are summarized as
follows:

  * •

An adaptive method to estimate the state of matched keypoints using geometric
approach through an adaptive depth-related threshold.

  * •

A refinement of the state estimation of keypoints near the inner and outer
edges of segmented objects using an adaptive depth-related impact factor in a
probabilistic function.

  * •

A novel approach for keypoints state estimation based on the area of influence
of dynamic objects with both non-matched keypoints and keypoints out of
segmented areas.

The remainder of the paper is organized as follows. Section II summarizes the
relevant related works. Section III is devoted to the description of our
method. The obtained results are presented are discussed in section IV.
Finally, concluding remarks and perspectives are provided in section V.

##  II Related work

In this section, we present related works on vSLAM methods that attempt to
deal with dynamic environments. These methods consider the non-rigid scene
assumption and can be grouped into two categories. Most of them are based on
the ORB-SLAM2 or ORB-SLAM 3 frameworks and include deep learning networks. The
first category corresponds to dynamic vSLAM methods based on semantic
information that combine geometric and semantic segmentation approaches. Among
these methods, DynaSLAM [5] associates multi-view geometry with semantic
information from the MASK R-CNN model to detect moving object to be rejected.
DS-SLAM [8] extracts semantic and motion information through a segmentation
model SegNet and a moving consistency check. These complementary tasks remove
outliers and then improve the estimated pose. Dynamic-SLAM [6] enriches the
SSD object detection method to a semantic level to determine the shape of
dynamic objects. This novel approach is completed by a selective tracking
algorithm applied to keypoints of dynamic objects that increases pose
estimation robustness. DP SLAM [7] is based on the estimation of the moving
probability propagation of the dynamic keypoints combining the epipolar
geometry constraints and semantic segmentation into a Bayesian filter. This
method processes the state probability ambiguity of the keypoints at the
boundary of masks. DGS-SLAM [12] uses a multinomial residual network to detect
dynamic objects combining the motion information from consecutive frames and
potential motion information from the semantic segmentation. This approach is
enhanced by a robust camera pose tracking strategy that uses a keypoints
classification. These modules are then employed in a segmentation module which
extracts potential moving points thanks to a semantic frame selection
strategy. The second category consists of methods based on optical flow
networks to estimate the scene dynamics without any prior semantic
consideration. FlowFusion [13] performs simultaneously the camera pose
estimation and the static background reconstruction. This method deals with
dynamic environments by using an optical flow network that generates an
indirect semantic segmentation of dynamic objects that makes the tracking more
robust. RDMO-SLAM [11] is a multimodal dynamic vSLAM that combines geometric
and semantic segmentation information approaches. The limitations of the
semantic segmentation module due to its slow process are overcome by
predicting missing semantic labels through a dense optical flow thread.
Finally, the semantic information about the scene dynamic is improved by
applying a moving probability estimation based on camera rotation. All these
methods can only estimate the state of points contained in a segmented area or
matched points. This limits their applicability in other zones of the scene.

##  III Method

Our vSLAM system is illustrated in fig. 1. RGB images undergo geometric and
semantic segmentation modules. This first process allows to determine the
keypoints state. The geometric module calculates the reprojection error of
matched keypoints using the epipolar geometric constraint. This reprojection
error is then compared to an adaptive threshold to decide on whether to reject
a given keypoint or to fed it into the probability function of keypoint state.
The semantic segmentation module is based on the YOLACT++ [14] method, which
generates masks of people in the scene. Segmented areas are considered dynamic
due to the nature of the segmented object category. In addition, the accuracy
of the keypoint state is modeled by a probabilistic function based on the
distance of the keypoints from the edges of the mask. These two modules and
the depth images are connected to their respective adaptive modules, which
will consider the influence of depth on these tasks. Once the keypoints are
sorted, the remaining keypoints considered static are refined through the
object interaction module. This module estimates the state of the keypoints in
the zone of influence of people using depth information associated with the
masks and the dynamic keypoints of the geometric module. (see fig.4).

###  III-A Semantic segmentation approach

The state of each keypoint of the semantic segmentation module is obtained by
calculating their probability of displacement with respect to their distance
from the the mask edges. First, this minimal distance $\Delta d_{m}$ between a
keypoint $p_{i}$ and a point $m_{i}^{n}$ on the edge of the mask $n$ is
defined as:

$\Delta d_{m}=\min\|p_{i}-m_{i}^{n}\|$ (1)

This distance is inserted into a binomial logistic regression model that will
estimate the probability of the keypoints state according to the mask $n$ as
follows:

$P(S_{p_{i}^{n}})=\frac{1}{exp(-\beta(z_{p_{i}})\cdot\Delta d_{m})+1}$ (2)

Figure 2: Areas of segmentation uncertainty

Where $\beta$$(z_{p_{i}})$ is the adaptive impact factor proportional to the
depth $z_{p_{i}}$ of the keypoint. The adaptive impact factor
$\beta(z_{pi})\in[0.05;0.25]$ will assign the size of the uncertainty area in
and out of the mask as described in fig.3. Thus, the greater the distance, the
more accurate the estimation of the state of this point, respectively static
outside the mask and dynamic inside. We calculate the adaptive impact factor
so as to obtain the limit of the uncertainty zone of a size corresponding to a
state probability lower than $75\%$. In summary, the probability range of a
keypoint to be dynamic can be defined as follows:

$P(S_{p_{i}^{n}})=\begin{cases}\left]0.25;0.5\right[,&\text{if $p_{i}\in
Uncertaintyareaout$}.\\\ \left]0.5;0.75\right[,&\text{if $p_{i}\in
Uncertaintyareain$}.\\\ \left]0.75;1.0\right[,&\text{if $p_{i}\in
Certaintyareain$}.\end{cases}$ (3)

Probabilities greater than 75% are considered reliable estimates of the state
as shown in fig.3.

###  III-B Geometric approach

The geometric module is based on the geometric epipolar constraint that links
matched keypoints of two distinct images.

Figure 3: Illustration of the reprojection error

Note the keypoint $q_{i}^{t-1}$ of a frame $t-1$ and its corresponding
keypoint $p_{i}^{t}$ of the next frame $t$, $F$ denoting the fundamental
matrix that describes the camera motion between these two frames and the
epipolar line $l_{q_{i}}$ being the reprojection of the keypoint $q_{i}$ in
the frame $t$ defined as follows:

$l_{p_{i}}=F\cdot(q_{i})^{T}=\left[\begin{array}[]{l}X_{q_{i}}\\\ Y_{q_{i}}\\\
Z_{q_{i}}\end{array}\right]$ (4)

The reprojection error $\Delta\varepsilon(p_{i})$ describes the distance
between the epipolar line $l_{q_{i}}$ of the keypoint $q_{i}$ and its matched
keypoint $p_{i}$ in the current frame. This error is denoted by the following
equation:

$\Delta\varepsilon(p_{i})=\frac{|p_{i}F(q_{i})^{T}|}{\sqrt{\|X_{q_{i}}\|^{2}+\|Y_{q_{i}}\|^{2}}}$
(5)

Keypoint $p_{i}$ is considered dynamic if its reprojection error
$\Delta\varepsilon(p_{i})$ is greater than an adaptive threshold
$\alpha(z_{p_{i}})$ related to the keypoint depth $Z_{p_{i}}$. This adaptive
threshold based on the assumption that, the farther a point is, the more its
reprojection error represents a large movement due to projective geometry
constraints. Thus, this threshold $\alpha(z_{pi})\in[0.5;0.9]$ and is
inversely proportional to the depth. If this error is smaller than the
threshold, then it feds the following normal probability density function to
estimate the state probability of the keypoint:

$P(g_{p_{i}})=\frac{1}{\sqrt{2\pi}\sigma}\exp\left(-\frac{(\Delta\varepsilon(p_{i}))^{2}}{2\sigma^{2}}\right)$
(6)

Where $\sigma$ is set to 1 and represents the standard deviation of this
distribution. The moving probability obtained is then normalized by the
adaptive threshold.

###  III-C Moving probability update

The moving probability $P(p_{i})$ represents the state probability of the
keypoints where $P(p_{i})\in State$ with $State=\\{static(s),dynamics(d)\\}.$
The moving probability $P(p_{i})$ for a matched and segmented keypoint $p_{i}$
combines the geometric probability model $P(g_{p_{i}})$ and the semantic
segmentation probability model $P(S_{p_{i}})$ as follows:

$P(p_{i})=\omega P(g_{p_{i}})+(1-\omega)P(S_{p_{i}})$ (7)

Where $\omega$ is a weight that describes the relevance of the probabilistic
model for different keypoint situations. In uncertain areas, $\omega$ is set
to 0.5, while in the reliable area, it is set to 0.1. In case of non-matched
keypoint include in the mask area, the moving probability $P(p_{i})$
corresponds to the semantic segmentation probability $P(S_{p_{i}})$.
Conversely, if it is a matched keypoint out of the mask area, then the moving
probability $P(p_{i})$ corresponds to the geometric probability
$P(g_{p_{i}})$. We update the moving probability function using Bayesian
filter as expressed:

$bel(p_{i})=\eta P(\Omega_{i}|p_{i})\int P(p_{i}|q_{i})bel(q_{i})dq_{i}$ (8)

Where $\eta$ is an impact factor to normalize probabilities. Initials prior
probability $p_{0}$ and observation likelihood $P(\Omega_{i}|p_{i})$ are set
to 0.5. Keypoints with a moving probability greater than 0.5 are considered
dynamic points and therefore rejected.

###  III-D Objects interaction module

The Objects Interaction Module (OIM) assesses possible interactions between
humans and inert objects. It is based on the assumption that, the more a point
considered static has an image position and a depth close to a human, the more
likely it is to interact with it. Thus, we define an interaction zone related
to the human depth (see the colored areas in fig. 4 that indicate the
probability of objects interaction) in which we estimate the correlation
between the position and depth distances of the point with the nearest human.
Suppose a considered static keypoint denoted $p_{i}$ is in this area and owns
a position and depth distance lesser than an adaptive threshold. The dynamics
keypoints $p_{id}$ from the geometric module in its neighborhood are studied
in that case. These dynamics keypoints are retained when they satisfy the
following conditions:

$\begin{cases}||p_{i}-p_{id}||<\delta(z_{p_{i}}),&\text{$\delta(z_{p_{i}})\in[11;48]$}\\\
\Delta(i)=|p_{i}(z)-p_{id}(z)|<\rho,&\rho=0.7\end{cases}$ (9)

Where $\delta(z_{p_{i}})=48-4*z_{p_{i}}$ is an adaptive position distance
threshold in pixels, and $\rho$ is a depth distance threshold in meters. We
calculate the center of gravity $G(p_{i})$ of these dynamic points weighting
it by its depth distance as follows:

Figure 4: Probability of objects interaction in areas

Interaction points are dynamic points obtain through the OIM.

$G(p_{i})=\frac{\sum_{i=1}^{k}p_{id}\cdot\frac{\rho-\Delta(i)}{\rho}}{\sum_{i=1}^{k}\frac{\rho-\Delta(i)}{\rho}}$
(10)

The distance $\Delta(G_{pi})$ of this center from the point $p_{i}$ provides
information on the state of $p_{i}$. Indeed, this distance informs us about
the relevance of the link characterizing these dynamic keypoints and the point
$p_{i}$. A small distance implies for the static point either very close
proximity to these dynamic points or that they encompass it. Thus, the OIM
changes the keypoint state to dynamic when $\Delta(G_{pi})$ is smaller than a
threshold $\gamma(z_{pi})\in[10;28]$ inversely proportional to the depth.

##  IV Experiments and results

We have evaluated our vSLAM method in the fre3 sequence of the public dataset
TUM RGB-D dedicated to assessing vSLAM methods in dynamic environments. This
dedicated sequence provides RGB and depth images, and ground truth trajectory.
It is broken down into two types of sequences, namely sitting (s) and walking
(w), which correspond to scenarios with low and high levels of dynamics. All
experiments were achieved on an Intel XENON CPU and Nvidia GPU RTX2080 SUPER.
To perform the segmentation, We used the official YOLACT++ model with the
backbone Resnet50-FPN pre-trained on the COCO dataset with an image size of
550*550. Our method has been compared to other state-of-the-art dynamic vSLAM
methods according to the Absolute Trajectory Error (ATE).

TABLE I: Evaluation of the objects interaction module (RMSE) Sequences | D2SLAM (w/o-OIU) | D2SLAM (w/-OIU)
---|---|---
| ATE | T. RPE | R. RPE | ATE | T. RPE | R. RPE
fr3/w/xyz | 0.0162 | 0.0236 | 0.648 | 0.0163 | 0.0237 | 0.642
fr3/w/half | 0.0291 | 0.0413 | 0.903 | 0.0290 | 0.0409 | 0.923
fr3/w/static | 0.0160 | 0.0282 | 0.565 | 0.0101 | 0.0151 | 0.358
fr3/w/rpy | 0.0326 | 0.0466 | 1.047 | 0.0309 | 0.0444 | 1.003
fr3/s/xyz | 0.0141 | 0.0202 | 0.633 | 0.0138 | 0.0199 | 0.626
fr3/s/half | 0.0204 | 0.0293 | 0.842 | 0.0199 | 0.0287 | 0.826
fr3/s/static | 0.0069 | 0.0101 | 0.332 | 0.0070 | 0.0102 | 0.328
fr3/s/rpy | 0.0231 | 0.0335 | 0.799 | 0.0224 | 0.0322 | 0.801

First, we highlighted the contribution of the Objects Interaction Module (OIM)
to our system by assessing the performance of our method with (w/-OIM) and
without (w/o-OIM) this module (see tab.I). We observed a better improvement
for high dynamic scene scenarios, especially for the ATE score. Table II shows
our benchmarking, where we highlighted in bold the best results of each
assessment for better readability.

TABLE II: Evaluation of the Absolute Trajectory Error in meters (m) Sequences | RMSE
---|---
| ORB-SLAM3 | DynaSLAM | DS-SLAM | D2SLAM
fr3/w/xyz | 0.725 | 0.0164 | 0.0247 | 0.0163
fr3/w/half | 0.399 | 0.0296 | 0.0303 | 0.0290
fr3/w/static | 0.358 | 0.0068 | 0.0081 | 0.0101
fr3/w/rpy | 0.807 | 0.0354 | 0.444 | 0.0309
fr3/s/xyz | 0.0092 | 0.0127 | / | 0.0138
fr3/s/half | 0.0221 | 0.0186 | / | 0.0199
fr3/s/static | 0.0096 | 0.0108 | 0.0065 | 0.0070
fr3/s/rpy | 0.0222 | / | / | 0.0224

As shown in Table II, the results obtained show that the performance of our
method is globally better than that of the techniques considered. Indeed, we
record relatively significant gains in terms of ATE for some sequences and
slightly below the state of the art for others. This clearly demonstrates the
interest of taking into account the parameters related to the proximity of
humans and the depth in the estimation of the trajectory. The gain in accuracy
in the estimation of the trajectory is even more significant in the case where
the camera has high amplitudes of movement (sequences xyz, halfsphere and
rpy).

##  V Conclusion

Through this study we have shown the interest of taking into account the
proximity of the points of interest by integrating the depth and distance
information with respect to the dynamic persons to improve the estimation of
ATE. This allowed us to take into account the interaction that exists between
humans and inert objects by combining the information of the geometric module
and the proximity information. The results obtained in terms of accuracy in
the estimation of the trajectory are convincing and show the interest and the
relevance of our approach. In this study we have limited ourselves to the
analysis of the trajectory estimation. In a future work we will exploit this
approach in the study of other aspects related to pose estimation.

## References

  * [1] Mur-Artal, R., Tardós, J. D. (2017). Orb-slam2: An open-source slam system for monocular, stereo, and rgb-d cameras. IEEE transactions on robotics, 33(5), 1255-1262.
  * [2] Forster, C., Pizzoli, M., Scaramuzza, D. (2014, May). SVO: Fast semi-direct monocular visual odometry. In 2014 IEEE international conference on robotics and automation (ICRA) (pp. 15-22). IEEE.
  * [3] Qin, T., Li, P., Shen, S. (2018). Vins-mono: A robust and versatile monocular visual-inertial state estimator. IEEE Transactions on Robotics, 34(4), 1004-1020.
  * [4] Beghdadi, A., Mallem, M. (2022). A comprehensive overview of dynamic visual SLAM and deep learning: concepts, methods and challenges. Machine Vision and Applications, 33(4), 1-28.
  * [5] Bescos, B., Fácil, J. M., Civera, J., Neira, J. (2018). DynaSLAM: Tracking, mapping, and inpainting in dynamic scenes. IEEE Robotics and Automation Letters, 3(4), 4076-4083.
  * [6] Xiao, L., Wang, J., Qiu, X., Rong, Z., Zou, X. (2019). Dynamic-SLAM: Semantic monocular visual localization and mapping based on deep learning in dynamic environment. Robotics and Autonomous Systems, 117, 1-16.
  * [7] Li, A., Wang, J., Xu, M., Chen, Z. (2021). DP-SLAM: A visual SLAM with moving probability towards dynamic environments. Information Sciences, 556, 128-142.
  * [8] Yu, C., Liu, Z., Liu, X. J., Xie, F., Yang, Y., Wei, Q., Fei, Q. (2018, October). DS-SLAM: A semantic visual SLAM towards dynamic environments. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 1168-1174). IEEE.
  * [9] Hartley, R., Zisserman, A. (2003). Multiple view geometry in computer vision. Cambridge university press.
  * [10] Campos, C., Elvira, R., Rodríguez, J. J. G., Montiel, J. M., Tardós, J. D. (2021). Orb-slam3: An accurate open-source library for visual, visual–inertial, and multimap slam. IEEE Transactions on Robotics, 37(6), 1874-1890.
  * [11] Liu, Y., Miura, J. (2021). RDMO-SLAM: Real-time visual SLAM for dynamic environments using semantic label prediction with optical flow. IEEE Access, 9, 106981-106997.
  * [12] Yan, L., Hu, X., Zhao, L., Chen, Y., Wei, P., Xie, H. (2022). DGS-SLAM: A Fast and Robust RGBD SLAM in Dynamic Environments Combined by Geometric and Semantic Information. Remote Sensing, 14(3), 795.
  * [13] Zhang, T., Zhang, H., Li, Y., Nakamura, Y., Zhang, L. (2020, May). Flowfusion: Dynamic dense rgb-d slam based on optical flow. In 2020 IEEE International Conference on Robotics and Automation (ICRA) (pp. 7322-7328). IEEE.
  * [14] Bolya, D., Zhou, C., Xiao, F., Lee, Y. J. (2020). Yolact++: Better real-time instance segmentation. IEEE transactions on pattern analysis and machine intelligence.