# Stock Embeddings: Learning Distributed Representations for Financial Assets
††thanks: This publication has emanated from research conducted with the
financial support of Science Foundation Ireland under Grant number
18/CRT/6183. For the purpose of Open Access, the author has applied a CC BY
public copyright licence to any Author Accepted Manuscript version arising
from this submission.

Rian Dolphin  School of Computer Science
University College Dublin
Dublin, Ireland
<EMAIL_ADDRESS>Barry Smyth  School of Computer Science
University College Dublin
Dublin, Ireland
<EMAIL_ADDRESS>Ruihai Dong  School of Computer Science
University College Dublin
Dublin, Ireland
<EMAIL_ADDRESS>
###### Abstract

Identifying meaningful relationships between the price movements of financial
assets is a challenging but important problem in a variety of financial
applications. However with recent research, particularly those using machine
learning and deep learning techniques, focused mostly on price forecasting,
the literature investigating the modelling of asset correlations has lagged
somewhat. To address this, inspired by recent successes in natural language
processing, we propose a neural model for training stock embeddings, which
harnesses the dynamics of historical returns data in order to learn the
nuanced relationships that exist between financial assets. We describe our
approach in detail and discuss a number of ways that it can be used in the
financial domain. Furthermore, we present the evaluation results to
demonstrate the utility of this approach, compared to several important
benchmarks, in two real-world financial analytics tasks.

###### Index Terms:

Latent Representation, Embedding, Similarity, Stock Market, Distributional
Semantics

##  I Introduction

The stock market provides a challenging but appealing target for predictive
analysis [1, 2, 3], and has attracted the attention of machine learning (ML)
research for some time. While state-of-the art ML techniques, such as modern
deep learning-based methods, have demonstrated excellent performance on
financial forecasting tasks [4], less attention has been paid to a range of
other important tasks. One such area is portfolio optimisation,
diversification and hedging. This relies heavily on being able to understand
the relationships and similarities between financial assets; to identify
groups of assets that will perform well collectively, for example by
delivering a stable and predictable increase in value over time with reduced
risk of capital loss. It is this problem of measuring relationships and
similarities between financial assets that we seek to address in this paper.

Allocating capital within an investment portfolio is a two-step process [5],
_forecasting_ and _allocation_ : (1) an investor must form some beliefs about
the future return and risk potential of each asset _individually_ ; and (2)
given their beliefs, the investor must then decide how to allocate capital
between these assets taking into account the relationships and correlations
that may exist between pairs of assets and their returns. Most relevant recent
research has focused on the forecasting step, usually on an individual asset
basis, by proposing models to predict asset returns and volatility [4], with
less attention paid to optimising the allocation of assets (_portfolio
design_); indeed modern portfolio theory largely remains rooted in [5], which
saw Harry Markowitz receive the Nobel Prize in Economics in 1990.

In this paper, inspired by language models that have proven to be very
successful in the NLP domain [6, 7, 8], we propose a method for learning stock
embeddings which are capable of capturing the underlying relationships between
stocks by studying distributional similarities in their historical returns. In
particular, we propose a novel algorithm and training methodology for the
learning of distributed representations of financial assets, based purely on
historical returns data, which can be used to identify financial assets that
are likely to behave similarly and differently, both important elements of
portfolio design. After summarising related work in the next section, we go on
to describe this approach and then present a number of example case-studies to
showcase how the learned representations might be useful in financial
applications. Finally, before concluding, we present the results of an initial
evaluation to demonstrate the effectiveness of the learned representations.

##  II Related Work

The rapid advancement of ML in recent years has fueled interest in applying
modern techniques within the financial domain, with applications ranging from
tackling financial cybercrime [9] to stock market forecasting [10]. Within the
subset of applications focusing on the stock market, the literature
overwhelmingly focuses on return or volatility forecasting for individual
assets [11, 10]. Although these forecasting models have achieved improvements,
there are many other problems within the stock market which have not received
the same level of attention. One such area is the modelling of asset
correlations and relationships.

Modelling relationships and correlations between asset returns is vital for a
number of financial tasks such as portfolio optimisation, hedging and sector
classification [12]. Markowitz used covariance of returns to inform a risk
reduction strategy in his pioneering paper on modern portfolio theory [5] and,
as a result, correlation has become the conventional measure of similarity
between the returns time series of two financial assets. A number of other
techniques for measuring asset similarity have been proposed in recent years
including a geometric measure of similarity [13] and an adjusted correlation-
based approach [14]. As previously mentioned, ML based solutions to this task
are lacking, with many relying on textual data as discussed below.

Though computing pairwise correlation or covariance estimates between assets
is an actionable solution, we hypothesise that the idea of representing
financial assets using dense vector embeddings has the potential to outperform
traditional correlation based approaches. The learning of distributed
representations has been extensively studied in recent years with arguably the
most well-known application being word embeddings in language modeling [15].
Motivated by algorithms such as Word2Vec [6], neural embedding methods have
become more prevalent across multiple domains such as recommendation systems
[16] and medicine [17].

The application of embeddings in the financial domain have, for the most part,
been limited to the NLP idea of word embeddings. For example, [10, 18, 19]
tackle stock returns forecasting using event embeddings obtained from
financial news. [20] train company embeddings by applying BERT to the textual
data from annual reports and [21] use word embeddings to select similar
stocks. Despite applying the idea of embeddings within the financial domain,
the aforementioned literature still relies on the aggregation of pretrained
word embeddings from language models, rather than on using a novel technique
to learn embeddings from non-textual financial data such as historical
returns.

In contrast, the method proposed in this paper does not draw on pretrained
word embeddings or any form of textual data. Instead, we propose a novel
methodology for generating stock embeddings using only historical returns
data.

##  III Architecture & Approach

This paper proposes a method for training distributed representations of
stocks using a probabilistic neural framework. Inspired by the distributional
semantics area of natural language processing, we propose a model architecture
and training procedure that uses the idea of _context stocks_ to learn
embeddings of target stocks. In linguistics, the distributional hypothesis,
which underpins a number of popular language models [6], captures the idea
that a word is characterized by the company it keeps [8], i.e., words that
occur in the same contexts tend to have similar meanings.

In the financial domain, research has shown a similar hypothesis to be true:
companies with similar characteristics, such as operating in the same business
sectors, tend to exhibit similar stock price fluctuations [22]. As a result,
by engineering the selection of context stocks to reflect this hypothesis, and
adding noise reduction strategies, our proposed framework generates embeddings
that can capture relationships between financial assets purely based on
historical pricing data.

###  III-A Generating Training Data

Figure 1: Generating training data, i.e., target:context stock sets

Consider a universe of stocks $U=\\{a_{1},...,a_{|U|}\\}$ and for each stock
$a_{i}$ we have a vector
$\mathbf{p}_{a_{i}}=\\{p_{0}^{a_{i}},...,p_{T}^{a_{i}}\\}$ containing its
prices at discrete time intervals (daily or weekly for example). From the
pricing data we then compute a returns vector
$\mathbf{r}_{a_{i}}=\\{r_{1}^{a_{i}},...,r_{T}^{a_{i}}\\}$ using Equation 1.

$r_{t}^{a_{i}}=\frac{p_{t}^{a_{i}}-p_{t-1}^{a_{i}}}{p_{t-1}^{a_{i}}}$ (1)

From these returns vectors we can generate _target:context sets_. For a
context size $C$, the context stocks for target asset $a_{i}$ at time $t$ are
the $C$ stocks which have the closest return at that point in time. The
closest return is defined by the lowest absolute value difference in return
for candidate stock $a_{j}$, formulated as $|r_{t}^{a_{i}}-r_{t}^{a_{j}}|$. An
example of this process is outlined in Figure 1. Here we have AAPL as the
context stock and $t$ is January 3rd 2000\. We compute the absolute value
difference between the return of AAPL at that point in time with the return of
each other stock at the same point in time. Then, we choose the $C$ stocks
with the lowest values as the context stocks, excluding AAPL itself. In this
case, IBM and MSFT have very small differences with AAPL and so are likely to
be chosen as context stocks. In total, we generate a target:context set for
every stock at each point in time, which results in a total of $|U|\times T$
sets for training.

An example of a target:context set for $C=3$ might be
$\mathcal{S}(a_{0},t)=[a_{0}:a_{270},a_{359},a_{410}]$, which corresponds to
_[AAPL : IBM, MSFT, ORCL]_ , since we store the index value of the stock
rather than strings (270 is the index corresponding to IBM in our universe of
stocks, for example). This tells us that, at a certain point in time $t$, the
three stocks with the closest returns to Apple Inc. were IBM, Microsoft and
Oracle.

###  III-B Base Model Architecture

Figure 2: Model Architecture

The proposed model architecture is illustrated in Figure 2. The model design
is such that _the stock embeddings are the model parameters_. As such, each
row in the weight matrix $\mathbf{W}\in\mathbb{R}^{|U|\times N}$ is a stock
embedding, which are randomly initialised.

The first step is to compute the hidden layer, which is an element-wise
average of the context stock embeddings. To be more precise, the input to the
model is a one-hot encoded version of the context set, and so, consists of $C$
one-hot vectors
$\\{\mathbf{x}_{j_{1}},\mathbf{x}_{j_{2}},...,\mathbf{x}_{j_{C}}\\}$, one for
each context stock. These vectors are used to extract the embeddings
corresponding to the $C$ context stocks. For example, computing
$\mathbf{W}^{T}\cdot\mathbf{x}_{j_{1}}$ will extract a single row from
$\mathbf{W}$ — the embedding corresponding to the first context stock
$a_{j_{1}}$. In its most basic form, the hidden layer, $\bm{h}$, is an
element-wise average of the extracted embeddings, and is formulated in
Equation 2.

$\bm{h}=\frac{1}{C}\mathbf{W}^{T}(\mathbf{x}_{j_{1}}+\mathbf{x}_{j_{2}}+...+\mathbf{x}_{j_{C}})$
(2)

Thus the hidden layer, $\bm{h}$, is an $N$-dimensional vector and can be
thought of as an aggregate embedding representation of the context stocks. In
Equation 2, each embedding receives an equal weighting of $C^{-1}$; however,
in Section III-C we describe a more complex weighting approach intended to
reduce noise.

The next step is to estimate the probability of the target stock _given_
$\bm{h}$ by applying Equation 3.

$\mathbb{P}(\text{Target }|\text{ Context})=softmax(\mathbf{W}\bm{h})$ (3)

Ensured by using the softmax activation, the output is a posterior probability
distribution expressing the probability of each stock in the universe being
the target stock, given the context stocks observed. Since the dot product
represents a measure of similarity between vectors, the model assigns higher
probability to stocks whose embeddings are similar to hidden layer embedding
$\bm{h}$. In this way, when back-propagation is applied, stocks which commonly
co-occur in target:context sets will end up closer in the embedding space. As
a result, the embeddings will learn nuanced relationships that are present in
the historical returns data. We note that the ground truth here,
$\mathbf{y}_{i}$ in Figure 2, is a one-hot vector of length $|U|$ indicating
the true target stock.

###  III-C Noise Reduction Strategies

Financial returns data are notoriously noisy [23], and so, in addition to the
base model architecture, we propose two amelioration strategies to improve
performance. Firstly, a weighting strategy based on overall distributional co-
occurrence is introduced. With this, the hidden layer $\bm{h}$ is computed via
a weighted average, and is implemented by scaling each $\mathbf{x}_{j}$ using
a weight, which is proportional to the rate at which the given context stock
$a_{j}$ appears in the context of the target stock $a_{i}$ over the whole
training dataset. This is outlined in Equation 4, where ${\mathds{1}}$ denotes
the indicator function.

$w_{i,j,t}\propto\beta_{i,j}=\frac{1}{T}\sum_{t=1}^{T}{\mathds{1}}\Big{(}a_{j}\in\mathcal{S}(a_{i},t)\Big{)}$
(4)

The constant of proportionality here, $k_{i,t}$, is computed such that the
weightings over all context stocks sum to one.

$w_{i,j,t}=k_{i,t}\cdot\beta_{i,j}\hskip 14.22636pt:\hskip
14.22636ptk_{i,t}=\frac{1}{\sum_{j:j\in\mathcal{S}(a_{i},t)}\beta_{i,j}}$ (5)

Secondly, the distribution of returns over a short time period, such as daily,
contains a large proportion of values close to 0, which indicates little
movement in stock price. In an effort to isolate meaningful cases and reduce
noise, a context set $\mathcal{S}(a_{i},t)$ was deleted from the training data
if the target stock return, $r_{t}^{a_{i}}$, was within the interquartile
range (IQR) of returns on that day. As a result, only sets where the target
stock had a movement significantly different from the market average on a
given day were included in training.

##  IV Case Studies

In this section, for illustrative purposes, we describe two ways in which the
distributed representations that can be learned may be used in financial
settings to: (1) to identify nearest neighbour stocks; and (2) to complete
useful stock analogies similar to those used to in the original Word2Vec work
[6].

###  IV-A Daily Returns Dataset

For these case studies, and the evaluation that follows in Section V, we use a
publicly available dataset of daily pricing data111All data and code is
available at https://github.com/rian-dolphin/stock-embeddings for 611 US
stocks during the 2000-2018 period. In addition to daily returns, each stock
is also associated with a _sector_ and _industry_ classification label. The
former corresponds to the business sector the company operates in and there
are eleven sectors in total, including Finance, Health Care and Technology,
for example. The industry label represents a finer-grained classification; a
stock in the Technology sector may have Computer Software as its industry
label, for example, to contrast it with another Technology stock in the
Electronic Components industry.

###  IV-B Identifying Nearest Neighbour Assets

TABLE I: Examples of Top-3 Nearest Neighbours for Given Query Stocks | Query Stock
---
Sector - Industry
3 Nearest Neighbours - Sector - Industry | Similarity
| JP Morgan Chase
---
Finance
Major Bank
| Bank of America Corp - Finance - Major Bank
---
State Street Corp - Finance - Major Bank
Wells Fargo & Company - Finance - Major Bank
| 0.88
---
0.82
0.81
| Analog Devices
---
Technology
Semiconductors
| Maxim Integrated - Technology - Semiconductors
---
Texas Instruments - Technology - Semiconductors
Xilinx, Inc. - Technology - Semiconductors
| 0.93
---
0.91
0.90
| Chevron Corporation
---
Energy
Integrated Oil & Gas
| Exxon Mobil - Energy - Integrated Oil & Gas
---
BP P.L.C. - Energy - Integrated Oil & Gas
Occidental Petroleum - Energy - Oil & Gas Production
| 0.89
---
0.82
0.78

Ideally, we should expect the distributed representations of related stocks to
to be closer, by some suitable similarity metric, than dissimilar stocks, and
the ability to identify similar (and dissimilar) stocks is an important tool
for portfolio design. Here we use cosine similarity as our similarity metric
in order to provide examples of the _k_ nearest neighbours for a sample of
example stocks; similar results can be obtained when using alternative metrics
such as Euclidean distance.

Table I shows the top-3 ($k=3$) nearest neighbours for JP Morgan Chase, Analog
Devices, and Exxon Mobil Corp, three well-known companies in very different
sectors. In each case the nearest neighbours pass the sanity test in that they
belong to similar sectors and industries. For example, the three nearest
neighbours of JP Morgan, a major bank, are also all major banks. Remember,
that no sectoral or industry information has been used in determining these
nearest neighbours, because only daily returns information has been used to
generate the distributed representations used for similarity assessment.

Nearest neighbour stocks, can be used in a variety of ways by investors. By
focusing on the $k$ nearest neighbour stocks we can develop a basic stock
recommendation system which, when given a target stock – a novel stock for the
investor or one already in their portfolio – can generate a ranked list of
similar stocks based on their historical returns data. Conversely, the ability
to identify maximally dissimilar stock is an important way to improve
portfolio diversity in order to provide an investor with an ability to guard
against volatility and _hedge_ against sudden sectoral shocks.

###  IV-C Analogical Inference in Financial Domains

In the NLP domain, one of the most appealing and useful features of word
embeddings has been their ability to capture relationships that go deeper that
simple syntactic regularities. Word embeddings have provided access to a form
of analogical inference using simple vector operations over the embeddings
[24]. The classic example is the analogy Man is to King as Woman is to ? which
can be solved algebraically as vector(King) - vector(Man) + vector(Woman) to
produce a vector that is closest to the vector representation of the word
Queen [6].

Similar algebraic operations can be used in the financial domain, albeit in a
manner that is not quite as intuitive as the word analogies that we are all so
familiar with from everday life. For example, in Table II we solve for similar
analogies to identify, for example, Hewlett Packard is the stock in the
Technology sector that is most similar to American Express in the Finance
sector, or that Johnson & Johnson in Healthcase is most similar to Exxon Mobil
in Energy. Such analogies can be very useful in an investment context when
investors wish to extend their portfolios into different sectors in which they
have little or no investment experience and/or to improve their portfolio
diversity and limit overall risk and volatility.

TABLE II: Analogical Examples Analogy | Similarity
---|---
American Express is to Finance as Hewlett Packard is to Technology | 0.8
Exxon Mobil is to Energy as Johnson & Johnson is to Health Care | 0.81
State Street is to Finance as Xerox is to Technology | 0.8

##  V Evaluation

In this section we provide the results of an initial evaluation of the
proposed stock embeddings approach. We describe two experiments to evaluate
different aspects of the learned distributed representations. In the first, we
evaluate the ability of the distributed representations to be used for sector
classification. In the second, we describe and evaluate an approach to
generate hedged portfolios that benefit from lower volatility characteristics
than more conventional hedging strategies, which rely on simple price
correlations. In both experiments we use the daily returns dataset described
previously and generate our embeddings with a context size of 3 and an
embedding dimension of 20. Both noise reduction techniques (distributional co-
occurrence and significant movement) were also used in training.

###  V-A Sector Classification

The complex web of hidden factors and unpredictable events that influence
stock prices make investing an undeniably risky endeavour. In particular,
investing in individual stocks exposes the investor to both broader market
risk (systematic) and asset-specific risk (idiosyncratic). Exchange traded
funds (ETFs) have become a popular alternative to stock picking since the
_diversification_ they offer greatly reduces idiosyncratic risk and their cost
is relatively low.

ETFs are securities traded on public exchanges that offer partial ownership in
a portfolio containing a large number of stocks. Often these portfolios are
constructed in such a way that they aim to track a particular geographical
region or specific market sector so that an investor can obtain highly
diversified exposure to any market segment that they wish. ETFs have become so
popular that, by the end of 2016, their market share exceeded 10% of the total
market capitalisation traded on US exchanges, representing more than 30% of
overall trading volume [25].

However, companies offering ETF products need to decide on the constituents of
the underlying portfolios. Sometimes the portfolios are constructed to track
certain indices, such as the S&P500. However the selection criterion can be
highly subjective, particularly for market sector ETFs. For example, a company
like Amazon is classified as consumer discretionary by the Global Industry
Classification Standard (GICS). Should it be included in a consumer
discretionary ETF with the likes of Ford Motor and McDonald’s? Or would it be
better suited to a technology or consumer staples ETF? Indeed, a case can be
made for Amazon to be considered as all of the above.

The ability to segment stocks into market sectors is also important for many
other types of financial and economic analysis — measuring economic activity,
identifying peers and competitors, quantifying market share and bench-marking
company performance — none of which would be possible without industry
classifications [12]. The merit of particular assets are primarily determined
relative to their peers within the same asset class and business sector. Thus,
a well-defined sector classification system makes it easier for analysts to
compare companies’ relative valuations and to build sector-specific return and
risk estimates [26].

With these use cases in mind, we will show how the stock embeddings can be
used to successfully classify stocks into business sectors and pick up on
inconsistencies in existing classification schemes. First, we explore a
visualisation of the latent space spanned by the learned embeddings, examining
clustering and relationships between stocks. Then we consider some interesting
cases which seem, on the surface, not to behave how we might expect. Finally,
we train a classification model based on the embeddings which assigns sector
labels to companies.

####  V-A1 Clustering Stocks using Embeddings

Visualising latent embeddings in a lower dimensional space can often be useful
to identify relationships and clustering behaviour. Figure 3 shows a graphical
representation of the embeddings for stocks in four of the largest business
sectors: Energy, Finance, Public Utilities and Technology. Each node
represents a stock, colored by sector, and an edge indicates that the two
nodes it connects have greater than 0.7 cosine similarity between their
embeddings. The plot is generated using a force-directed graph drawing
algorithm in Gephi[27].

The clustering of stocks into business sectors is clearly evident in the plot.
We can also see that nearly all of the edges in the graph are between nodes
from the same sector. From this we conclude that the proposed model
architecture and training procedure result in embeddings that have
successfully picked up on relationships between stocks.

Figure 3: Graphical Visualisaion of Stock Embeddings Coloured by Business
Sector. Edges Formed by an Embedding Similarity Threshold

When we consider that the training data used is purely derived from historical
returns data, this is a very positive result because it suggests that it is
possible to reconstruct important sectoral information from the embeddings,
and indeed are likley to be able to do so in a way that is more nuanced than
might be possible using simple sectoral labels.

####  V-A2 High Similarity Mismatches

TABLE III: Examples of high similarity mismatches — stocks with very high similarity that have different sector labels | Stock A
---
Sector - Industry
| Stock B
---
Sector - Industry
Similarity
| Lennar Corporation
---
Basic Industries - Homebuilding
| KB Home
---
Capital Goods - Homebuilding
0.98
| Bristow Group Inc.
---
Transportation - Transportation Services
| Unit Corporation
---
Energy - Oil & Gas Production
0.92
| Cirrus Logic, Inc.
---
Technology - Semiconductors
| Xcerra Corporation
---
Capital Goods - Electrical Products
0.93

Though nearly every edge in the Figure 3 occurs between nodes from the same
sector, there are some cases where this is not true. In other words, there are
some cases where two stocks have a very high embedding similarity but their
sector labels don’t match. Does this highlight a flaw in the embeddings, where
stocks achieve very high embedding similarity when they should not? To answer
this we provide some examples of these _high similarity mismatches_. We
consider pairs of stocks that have a cosine similarity between embeddings
greater than 0.9 and are members of different sector classes. A number of
examples are shown in Table III.

The first example in Table III is Lennar Corporation. Lennar is a home
construction and real estate company which has been recorded as a member of
the Basic Industries sector and the Homebuilding industry (remember here that
industry is a finer-grained classification of the sector). Lennar Corporation
actually has four stocks that have greater than 0.9 similarity but are _not_
in the Basic Industries sector, we can see one of these in Table III. KB Home
is classified in the Capital Goods sector but we note that it is also in the
Homebuilding industry. In fact, these companies both operate a very similar
business model focused around home construction. This is the same for the
other three stocks which have very high similarity with Lennar; all are in the
Homebuilding industry but were subjectively classified into a different sector
to Lennar.

Immediately we can see that the human approach of assigning sector
classifications can be flawed and inconsistent. Why is a company in the same
industry but not the same sector when industry is just a finer grained
description of sector? As previously mentioned, sector classification is a
vital task in the financial domain for many reasons. We propose that the
embeddings could be a useful and objective tool to complement the existing
methods by which companies are grouped.

Another example is the case of Cirrus Logic and Xcerra Corporation. Though
these companies are segmented into a different sector and industry, both of
their business models revolve around dealing with semiconductors, Xcerra
develops equipment for testing semiconductors and Cirrus is a semiconductor
supplier. Similarly, Unit Corporations engage in oil and gas production while
Bristow Group offers offshore oil and gas transportation.

These two examples are not labelling inconsistencies in the same way as the
Homebuilding example, but rather, they indicate that the embeddings pick up on
interesting relationships which are not captured in the sector classification
schemes. This particular property of the embeddings, to pick up on nuanced
relationships between stocks, will be explored further in Section V-B.

####  V-A3 Sectoral Classification

Using the embeddings generated by the proposed model architecture we can use a
classification model to segment companies into business sectors in an
objective manner, based purely on relationships in historical returns. To do
this we train a classification model with embeddings as input and sector label
as the output.

There are a number of considerations here which will undoubtedly limit the
accuracy of the classification model. Firstly, the training data is derived
solely from historical returns data, which are influenced by a complex network
of unpredictable factors. Secondly, as we saw in Section V-A2 when looking at
high similarity mismatches, there are a number of stocks which are either
labelled inconsistently or subjectively, and could be classified into a number
of different sectors. As a result, sector classification in the financial
domain is a very challenging problem and even more so when using only returns
data.

To train the classification model we used $k$-fold cross validation with
$k=5$. Within the data there is a large class imbalance between sectors which
can introduce algorithmic bias and impact negatively on results [28]. And so,
to remedy this we applied Synthetic Minority Oversampling Technique (SMOTE) to
the training data. As a classification model we use a support vector
classifier.

TABLE IV: Results from the sector classification model which takes embeddings as inputs Model | Precision | Recall | F1 | Accuracy
---|---|---|---|---
Embedding | 0.57 | 0.54 | 0.55 | 54%
Embedding + IQR | 0.59 | 0.56 | 0.56 | 56%
Embedding + Weight | 0.59 | 0.57 | 0.57 | 57%
Embedding + Weight + IQR | 0.62 | 0.60 | 0.60 | 60%

Table IV shows the performance in the sector classification task. The highest
accuracy of 60% is achieved by the Embedding model with both noise reduction
techniques, thus indicating their utility. This is an impressive result
considering there are 11 sector classes222The sector classes are: Basic
Industries, Capital Goods, Consumer Durables, Consumer Non-Durables, Consumer
Services, Energy, Finance, Health Care, Public Utilities, Technology,
Transportation as well as the aforementioned limitations. A more in depth
analysis shows a variation in accuracy across sectors, with the more populated
sectors being very accurately classified (F1 $>0.9$ in some cases) while other
minority sectors have quite low accuracy. With this in mind, we believe the
accuracy could be improved with a larger dataset.

An objective method of sector classification as described in this work has the
potential to be very useful in practice to pick up on and prevent inconsistent
company segmentation, which is a well-documented issue [29].

###  V-B Hedging/Diversification Example

As mentioned previously, much of the existing literature applying
computational methods to the stock market has focused on returns forecasting.
However, maximising returns is not the primary consideration for all
investors; in fact, in some cases portfolio protection is a higher priority.
For example, the primary focus of a portfolio manager for a defined benefit
pension fund is having enough money to cover the agreed benefits. As a result,
maximising returns in this context is a secondary consideration to successful
risk management. In order to achieve protection, investors and portfolio
managers implement the ideas of diversification and hedging, and measure their
effectiveness in terms of volatility. Thus it is not only important to be able
to identify a set of similar stocks, but also a set of dissimilar stocks,
which can be expected to behave in opposition to the similar stocks [14]. This
is because, in a trading context, traders will often need to offset or hedge
their positions in target stocks by also trading in maximally dissimilar
stocks; the idea being that under-performance in a selected stock can be
offset by gains in a dissimilar stock, thereby allowing a trader to limit
their overall risk.

The usual practice is to implement hedging by using negatively-correlated
assets leading to hedging strategies that rely on different types of
correlation metrics. Here we consider an alternative approach by using the
embeddings generated to identify maximally dissimilar stocks and inform a
hedging strategy. For evaluation purposes we consider the case where an
investor holds an open position in a given stock (the query stock) and wants
to use another single stock (the hedge stock) to hedge and reduce their risk,
measured as volatility, as much as possible.

for  _Query_stock in stocks_ do

temp_min $\leftarrow\infty$

Hedge_stock $\leftarrow$ None

for  _temp_stock in stocks_ do

if  _similarity(Query_stock, temp_stock) $<$ temp_min_ then

temp_min $\leftarrow$ similarity(Query_stock, temp_stock)

Hedge_stock $\leftarrow$ temp_stock

end if

end for

Simulate portfolio performance with equal weight between Query_stock and
Hedge_stock and return the volatility.

end for

Algorithm 1 Algorithm for finding hedged portfolios and simulating realised
volatility

To implement this, we create a hedged two-asset portfolio for each stock in
the dataset, as outlined in Algorithm 1. This results in a sample size of 611
portfolios, each of which contains two assets, the query stock and the hedge
stock. The hedge stock is chosen as the lowest similarity counterpart to the
query stock, where similarity is either cosine similarity between embeddings
or one of the baselines which are listed later. We can then simulate the
performance of the portfolios using each similarity metric in turn, recording
the realised volatilities for comparison.

Figure 4: Distribution of Volatility for Hedged Portfolios TABLE V: Portfolio hedging experiment results along with Tukey HSD test indicating significantly lower volatility than Pearson baseline at $\alpha=0.01$. Method | Avg Volatility | Significant
---|---|---
Pearson | 23.8% | -
Spearman | 24.0% | ✗
Geometric | 23.9% | ✗
Embedding | 22.9% | ✓
Embedding + IQR | 21.3% | ✓
Embedding + Weight | 22.8% | ✓
Embedding + Weight + IQR | 21.9% | ✓

Of course, a train-test split is vital here to ensure that the experiment
mimics a real-world out-of-sample trading application as much as possible.
Given the time sensitive nature of financial data the split used a date
cutoff, with the first 70% of data (approx 2000 - 2013) being used to train
the embeddings and compute the other baseline similarity metrics. The latter
30% of data (approx 2013 - 2018) is then used to simulate the portfolios and
compute realised volatility.

In modern portfolio theory [5], similarity is defined in terms of Pearson
correlation and it is still the go-to method in both industry and academia. We
propose that cosine similarity between the proposed embeddings may be a better
choice. In the evaluation we include Pearson correlation as a baseline along
with Spearman rank-order correlation coefficient and a recently proposed
geometric shape similarity [13].

Figure 4 shows the distribution of volatility over the 611 portfolios for the
Pearson baseline, the proposed embeddings with no noise reduction and the best
performing variation of the proposed architecture333All variations and
baselines not included in Figure 4 for visual clarity. See Table V for more
detail.. From Figure 4 we can see that the embedding approaches result in a
greater concentration of lower volatility portfolios than the Pearson
baseline, which has a fatter long-tail indicating a greater tendency towards
high volatility portfolios.

Table V displays the average volatility results. To ensure the robustness of
results, we reran the experiment 100 times where, instead of choosing the
single most dissimilar stock as the hedge stock, we randomly chose one of the
25 most dissimilar for each target stock on each iteration. Overall, the
proposed embedding approach with IQR noise reduction results in portfolios
with the lowest average volatility at 21.3%. Post-hoc Tukey HSD tests indicate
that the volatility in all of the embedding based methods is statistically
significantly lower than the Pearson baseline at $\alpha=0.01$; none of the
other approaches generate any statistically significant differences in mean
volatility over the Pearson approach.

Thus, we have shown how the proposed embeddings might be used to inform a
hedging strategy that is superior to a number of baselines, at least within
the simplified setting used for this experiment. Obviously real-world settings
are based on more complex portfolios with many different stocks that need to
be collectively hedged, and it is a matter for future work to further evaluate
our embeddings approach in these more realistic settings. That being said, the
approach used here still serves as an important indication of success: had the
embeddings approach not be able to demonstrate improved volatility in these
simple two-stock portfolios then it would cast doubts on its likely future
success in more complex portfolios. Similarly, there is more work to be
carried out when it comes to understanding the dynamics of the distributed
representations and how they are learned: how does changing the context size
or embedding dimension impact these findings, for example.

##  VI Conclusion and Future Work

This work has focused on the problem of measuring relationships between the
returns of financial assets. We proposed an approach to learning distributed
representations for financial assets, based on daily returns data.We
demonstrated the utility of these representations using a number of example
case-studies and by evaluating their efficacy in two important sub-tasks,
sector classification and portfolio hedging. The results speak to the
potential benefits of our approach and provide a useful starting point for
further exploration and development.

In the future we plan to further evaluate the approach using more complex
portfolio management settings and additional datasets. For example, the
proposed technique is applicable to any group of financial assets whose
pricing information is available, and so, we can expand the universe to
include multiple asset classes in addition to equities, such as bonds, options
and other derivatives. Additionally, we plan to carefully explore the
parameter space of the model used to learn the embeddings, along with
different approaches to generate context stocks. For example, if Apple’s stock
price rises on a certain day, the stock prices of its suppliers will often be
affected in the days that follow. Can such _lagged embeddings_ be learned? Are
they useful? And if so, how can they be used in practice?

## References

  * [1] L. Bachelier, “Théorie de la spéculation,” in _Annales scientifiques de l’École normale supérieure_ , vol. 17, 1900, pp. 21–86.
  * [2] B. G. Malkiel and E. F. Fama, “Efficient capital markets: A review of theory and empirical work,” _The journal of Finance_ , vol. 25, no. 2, pp. 383–417, 1970.
  * [3] E. F. Fama, “The behavior of stock-market prices,” _The journal of Business_ , vol. 38, no. 1, pp. 34–105, 1965.
  * [4] O. B. Sezer, M. U. Gudelek, and A. M. Ozbayoglu, “Financial time series forecasting with deep learning: A systematic literature review: 2005–2019,” _Applied Soft Computing_ , vol. 90, p. 106181, 2020.
  * [5] H. Markowitz, “Portfolio selection,” 1952.
  * [6] T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient estimation of word representations in vector space,” _arXiv preprint arXiv:1301.3781_ , 2013\.
  * [7] J. Pennington, R. Socher, and C. D. Manning, “Glove: Global vectors for word representation,” in _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_ , 2014, pp. 1532–1543.
  * [8] J. R. Firth, “A synopsis of linguistic theory, 1930-1955,” _Studies in linguistic analysis_ , 1957.
  * [9] J. Nicholls, A. Kuppa, and N.-A. Le-Khac, “Financial cybercrime: A comprehensive survey of deep learning approaches to tackle the evolving financial crime landscape,” _IEEE Access_ , 2021.
  * [10] X. Ding, Y. Zhang, T. Liu, and J. Duan, “Deep learning for event-driven stock prediction,” in _Twenty-fourth international joint conference on artificial intelligence_ , 2015.
  * [11] L. Yang, T. L. J. Ng, B. Smyth, and R. Dong, “Html: Hierarchical transformer-based multi-task learning for volatility prediction,” in _Proceedings of The Web Conference 2020_ , 2020, p. 441–451.
  * [12] R. L. Phillips and R. Ormsby, “Industry classification schemes: An analysis and review,” _Journal of Business & Finance Librarianship_, vol. 21, no. 1, pp. 1–25, 2016.
  * [13] S.-H. Chun and Y.-W. Ko, “Geometric case based reasoning for stock market prediction,” _Sustainability_ , vol. 12, no. 17, p. 7124, 2020.
  * [14] R. Dolphin, B. Smyth, Y. Xu, and R. Dong, “Measuring financial time series similarity with a view to identifying profitable stock market opportunities,” in _International Conference on Case-Based Reasoning_. Springer, 2021, pp. 64–78.
  * [15] X. Wang, Y. Wang, B. Weng, and A. Vinel, “Stock2vec: A hybrid deep learning framework for stock market prediction with representation learning and temporal convolutional network,” _arXiv preprint arXiv:2010.01197_ , 2020\.
  * [16] O. Barkan and N. Koenigstein, “Item2vec: neural item embedding for collaborative filtering,” in _2016 IEEE 26th International Workshop on Machine Learning for Signal Processing (MLSP)_. IEEE, 2016, pp. 1–6.
  * [17] E. Choi, M. T. Bahadori, E. Searles, C. Coffey, M. Thompson, J. Bost, J. Tejedor-Sojo, and J. Sun, “Multi-layer representation learning for medical concepts,” in _proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining_ , 2016, pp. 1495–1504.
  * [18] X. Ding, Y. Zhang, T. Liu, and J. Duan, “Knowledge-driven event embedding for stock prediction,” in _Proceedings of coling 2016, the 26th international conference on computational linguistics: Technical papers_ , 2016, pp. 2133–2142.
  * [19] D. Cheng, F. Yang, X. Wang, Y. Zhang, and L. Zhang, “Knowledge graph-based event embedding framework for financial quantitative investments,” in _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_ , 2020, pp. 2221–2230.
  * [20] T. Ito, J. Camacho Collados, H. Sakaji, and S. Schockaert, “Learning company embeddings from annual reports for fine-grained industry characterization,” 2020\.
  * [21] M. Hirano, H. Sakaji, S. Kimura, K. Izumi, H. Matsushima, S. Nagao, and A. Kato, “Selection of related stocks using financial text mining,” in _2018 IEEE International Conference on Data Mining Workshops (ICDMW)_. IEEE, 2018, pp. 191–198.
  * [22] P. Gopikrishnan, B. Rosenow, V. Plerou, and H. E. Stanley, “Identifying business sectors from stock price fluctuations,” _arXiv preprint cond-mat/0011145_ , 2000.
  * [23] J. B. De Long, A. Shleifer, L. H. Summers, and R. J. Waldmann, “Noise trader risk in financial markets,” _Journal of political Economy_ , vol. 98, no. 4, pp. 703–738, 1990.
  * [24] K. Ethayarajh, D. Duvenaud, and G. Hirst, “Towards understanding linear word analogies,” _arXiv preprint arXiv:1810.04882_ , 2018.
  * [25] I. Ben-David, F. Franzoni, and R. Moussawi, “Exchange traded funds (etfs),” National Bureau of Economic Research, Tech. Rep., 2016.
  * [26] S. G. M. I. . MSCI, “Global industry classification standard (gics) methodology,” 2020.
  * [27] M. Bastian, S. Heymann, and M. Jacomy, “Gephi: An open source software for exploring and manipulating networks,” 2009. [Online]. Available: http://www.aaai.org/ocs/index.php/ICWSM/09/paper/view/154
  * [28] W. Blanzeisky and P. Cunningham, “Algorithmic factors influencing bias in machine learning,” _arXiv preprint arXiv:2104.14014_ , 2021.
  * [29] L. K. Chan, J. Lakonishok, and B. Swaminathan, “Industry classifications and return comovement,” _Financial Analysts Journal_ , vol. 63, no. 6, pp. 56–70, 2007.


# Self-Supervised Object Goal Navigation with In-Situ Finetuning

So Yeon Min1,2, Yao-Hung Hubert Tsai1, Wei Ding1,
Ali Farhadi1, Ruslan Salakhutdinov2, Yonatan Bisk2, Jian Zhang1 The authors
are with 1 Apple and 2 Carnegie Mellon University, USA. Correspondence:
<EMAIL_ADDRESS>
###### Abstract

A household robot should be able to navigate to target objects without
requiring users to first annotate everything in their home. Most current
approaches to object navigation do not test on real robots and rely solely on
reconstructed scans of houses and their expensively labeled semantic 3D
meshes. In this work, our goal is to build an agent that builds self-
supervised models of the world via exploration, the same as a child might -
thus we (1) eschew the expense of labeled 3D mesh and (2) enable self-
supervised in-situ finetuning in the real world. We identify a strong source
of self-supervision (Location Consistency - LocCon) that can train all
components of an ObjectNav agent, using unannotated simulated houses. Our key
insight is that embodied agents can leverage location consistency as a self-
supervision signal – collecting images from different views/angles and
applying contrastive learning. We show that our agent can perform
competitively in the real world and simulation. Our results also indicate that
supervised training with 3D mesh annotations causes models to learn simulation
artifacts, which are not transferrable to the real world. In contrast, our
LocCon shows the most robust transfer in the real world among the set of
models we compare to, and that the real-world performance of all models can be
further improved with self-supervised LocCon in-situ training.

##  I Introduction

In practical deployment scenarios where a robot is delivered to a customer’s
house, it should quickly generalize and adapt to new objects and layouts. For
the task of object goal navigation (ObjectNav), this means that the robot
should adapt both 1) its visual perception (which pixel is which object?) to
the objects and 2) the navigation policy (where to look next?) to the house
layout in the deployment setting. Because this should happen automatically,
without the user annotating data, we propose in-situ learning (i.e. learning
in the deployed real-world setting) of ObjectNav agents. Our proposed in-situ
adaptation is not applicable to existing approaches [1, 2] as they depend on
expensive labeled semantic 3D meshes of scanned and reconstructed houses to
train their agents and perception stacks.

Figure 1: Real World ObjectNav and in-Situ Training (a) Picture of our robot
in one of the airbnbs rented for ObjectNav experiments. (b) Example ObjectNav
(real-world) failure mode and its remedy with in-situ self-supervised
(location consistency) training. The agent starts at the living room and
wrongly detects a small patch of a black object as “TV.” The segmentation
model avoids this distractor after in-situ location consistency training.

For this purpose, we introduce a novel and strong source of self-supervision
(Location Consistency - LocCon), and propose a method to train all components
of an ObjectNav agent without labeled 3D meshes. At a high level, our approach
is composed of two stages: Stage I (visual perception) utilizes the embodiment
of an agent to train a semantic segmentation model, and then Stage II (Nav
Policy) uses the trained semantic segmentation model to self-label the 3D
mesh, which then can be used to train an ObjectNav policy. In particular, in
Stage I, we use location consistency as a self-supervision signal: an agent
collects images from different views/angles for a given location. We scale the
location-consistency collection to 100 unlabeled houses and apply contrastive
training to fine-tune the backbone of a pretrained semantic segmentation model
[3] (LocCon training). In Stage II, we train a navigation policy inspired by
PONI [2], which learns potential functions over map frontiers (analytic
functions of unexplored area and geodesic distances to goal locations), with
self-labeled semantic maps.

Our analysis finds that (1) our fully self-supervised ObjectNav agent can
perform competitively in the real world, (2) in-situ training with LocCon
further improves visual perception performance at deployment, and (3) despite
the presence of noisy artifacts in simulation, our self-supervised training
can learn useful semantics for sim2real transfer. We present a comprehensive
set of experiments in real houses and simulation (Gibson ObjectNav datasets
[1, 4, 5]).

First, we perform a sim2real experiment that shows our agent trained in
simulation can be directly run in the real world for object goal navigation,
demonstrating the robustness of our approach. We find that the largest error
mode of real-world ObjectNav is visual perception errors, which often
nullifies the need to adopt a better ObjectNav policy. This analysis, together
with evidence from previous work [1] that the Nav Policy, captures abstract
knowledge of house layouts which transfers to the real world, shows that in-
situ training is most needed for updating visual perception. Second, we
perform self-supervised location-consistency training directly in the real
houses – a step towards agents learning on their own. In particular, with
location consistency data collected from a real robot from 4 AirBnB houses, we
show that segmentation models can be improved with our training scheme.
Finally, within simulation, we compare our self-supervision pipeline against
training with annotations (FullSup agent). We find that FullSup’s segmentation
model learns the visual noise from simulation - which is irrelevant and
harmful for real-world transfer. This causes FullSup to perform worse and
makes it a harmful initialization in the real world.

Our first-of-a-kind full stack system for self-supervised ObjectNav enables us
to investigate where simulation is most helpful or harmful in training real
world ObjectNav agents. The primary approach in the literature has been to
focus on collecting scans of houses for reconstruction and training in
simulation; however, our results indicate that simulated data often introduces
both reconstruction and annotation noise, thereby inhibiting real world
transfer. Where supervised learning incentives models to overfit to such
artifacts, we find that our self-supervised training is robust to
reconstruction errors and help real world transfer. This disparity begs the
question - What data collection and annotation efforts are most effective in
the real world and simulation? We argue, that while simulation holds promise,
in the absence of high-quality annotations or renderings, researchers should
adopt our self-supervised approach which can be applied directly in the real
world for in-situ training with raw, unannotated, images that can be easily
collected.

Figure 2: Fine-tuning visual perception using self-supervision: First, we
produce a 3D voxel grid of the scene from depth sensors of a robot. We
randomly select a location and put 50cm3 cube, and make the robot view it from
8 different angles and 40 different poses. If the ray reflects back from this
cube to the robot’s camera, we save the corresponding egocentric RGB frame.
These collected images are labeled with “location consistency” pseudo-labels
in a self-supervised manner. We also show examples of collected “location
consistent” images. The ResNet backbone of pre-trained semantic segmentation
models can be fine-tuned with our data using contrastive loss.

##  II Related Work

We enumerate related work for visual perception learning and policy learning.
We will transition the discussions from supervised learning, which requires a
lot of annotated labels, to self-supervised learning, which avoids labels.

Visual Representation Learning. In ObjectNav, ground truth 3D semantic labels
are often given in the dataset and simulators, such as Matterport3D [6],
Replica [7], Habitat [8], Gibson [4], iGibson [9], and Habitat-Matterport3D
[10]. Existing methods often leverage the ground truth semantic 3D labels as
strong supervision signals for ObjectNav [1, 2]. Nonetheless, because of
lacking real-world semantic labels, it is unclear if the methods will
generalize well from the simulator to the real world.

00footnotetext: While [1], a closely related work of [11], shows real world
transfer of their semantic mapper and semantic nav policy, [11] itself does
not show real world transfer results of its self-supervised components.

To avoid the need for ground truth 3D semantic labels, previous work has
proposed workarounds: Mimic behaviors from a large collection of human
demonstrations [12], gather large-scale image collections from 3D mesh and
apply SSL techniques for static images [13], or leverage the embodiment of a
moving agent for self-supervised learning by taking the action of the agent
and the interaction with the environment into account when building their
representations [14, 11, 15].111Most self-supervised learning approaches
leverage only static images without considering the embodiment of an agent.
SimCLR[16], MoCo [17], DINO [18], and InfoMax[19] build consistent
representations for augmented variants of an same image and maximize their
difference to other images.  For example, in SEAL [11] an agent roams in an
environment, records predictions from an initial semantic segmentation model,
propagates the predictions from high-confidence to low-confidence regions, and
finally uses the predictions to fine-tune the semantic segmentation model. The
resulting model is then used for navigation. In principle, SEAL and LocCon
could be used together; however, LocCon is more general as, unlike SEAL, it
does not assume a pre-defined category of objects for training. In CRL [14],
the agent learns a policy to search for images that are deemed helpful for
contrastive learning. In both SEAL and CRL, self-supervisedly collecting data
requires a new training/calibration of policies - which can be a hindrance to
real-world transfer; our method, on the other hand, uses algorithmic data
collection upon a 3D map, which is simpler and does not require learning. The
simplicity and independence of our method allows for easy transfer of our
self-supervised learning scheme in the real world (§V-B).

| SSL Visual | SSL Nav | Real Robot | Real
---|---|---|---|---
Method | Perception | Policy | Transfer | World SSL
SEAL [11] | ✓ | ✗ | ✗∗ | ✗
OVRL [13] | ✓ | ✗ | ✗ | ✗
ZER [5] | ✓ | ✓ | ✗ | ✗
CoW, ZSON [20, 21] | ✓ | ✓ | ✗ | ✗
Ours | ✓ | ✓ | ✓ | ✓
TABLE I: Comparison of self-supervised ObjectNav methods. Our work (1) employs
end-to-end self-supervised training both for the visual perception and
navigation policy, and (2) demonstrates the sim2real transfer of both
components to the real world.33footnotemark: 3

Policy Learning. In ObjectNav, policies fall into two categories: end-to-end
and modular approaches. End-to-end methods [22, 23, 24, 25, 26, 27] directly
predict low-level actions (e.g. “move forward”, “turn left”) from input RGB-D
images and the camera pose. Modular methods [1, 2, 21, 13, 28] consist of a
pipeline of components - semantic mapping (persistent semantic and spatial
memory), a high-level semantic policy (decides the general direction to move
towards the predicted goal location), and low-level navigation modules (a low-
level planner for navigating to the high-level goal chosen by the semantic
policy). Learning is typically constrained to 1) the semantic (instance)
segmentation model inside the semantic mapper and 2) the high-level semantic
policy; our method is modular.

Existing methods for semantic policy training often use the object goal’s
ground truth location, either to define reward functions for reinforcement
learning [1, 13] or potential functions for supervised learning [2] \- these
are expensive requirements that make these approaches difficult to train in
the real world. To avoid the need for ground truth object goal locations, ZER,
ZSON, and CoW [5, 21, 20] perform image goal navigation (navigating to a
target image rather than a category). These images are converted to goal
embeddings via CLIP [29] or ResNet[30]-based encoders, allowing for object
categories to be mapped to scenes in service of performing ObjectNav. In lieu
of web-scale pretraining, given an object goal our method learns to find the
object in our self-supervised semantic 3D map without the use of any ground
truth location information.

While not related to self-supervision, [26] uses data at scale to improve
performances in ObjectNav. More specifically, it trains EmbCLIP [27] on 10K
diverse houses procedurally generated. While we use more homogenous scenes
from Gibson [4] and HM3D [31], the implication of our work well agrees with
[26] in that scaling brings gains. On the other hand, [32], a concurrent work,
presents large-scale comparisons of ObjectNav methods when transferred to the
real world. However, the focus of [32] and our work are different in that we
focus on the need of/method for self-supervised real-world in-situ training.

Figure 3: ObjectNav policy training and pipeline. (a) Self-supervised
ObjectNav policy training: The agent constructs self-labeled semantic maps of
the environment using the segmentation model trained in Stage I. Then, we
create a partial map out of full maps, and compute potential functions upon
the partial map from the unexplored areas and locations of goal objects.
Finally, we apply PONI policy training with the self-labeled potential
functions. (b) ObjectNav agent pipeline: Following previous work [1, 2, 33],
we equip our ObjectNav agent with the semantic mapper and the Object Nav
policy. The learned components are colored in green.

##  III Task Explanation

An ObjectNav [34, 35] agent aims to navigate to an instance of the given
object category such as “table” or “sofa” as efficiently as possible, with its
perception of the world and semantic priors. The agent is placed at a random
location in the environment and receives the goal object category (e.g.
“table”). With each action ($a_{t}$) the agent takes, it records visual
observations ($V_{t}$) and pose information ($x_{t}$).

In this work, we perform ObjectNav both on real robots and within simulation.
This leads to several key changes in the action space, and sensor noise in
both visual observations and pose information. In both conditions, visual
observations will consist of first-person RGB-D images.

#### Simulation

Within simulation, we have an idealized action space $A$ that contains four
actions: forward (0.25 meters), left (30°), right (30°), and stop. The agent
takes the stop action when it believes it has reached the goal. The episode is
considered successful, if the agent’s final location is within a distance $1m$
of the goal. Additionally, we set a maximum number of timesteps (500).

#### Robot Platform

For the real-world Object Nav, we rented three AirBnB’s in North America,
using the same set of object categories used in simulation. Using the fast-
marching method, the robot plans collision-free paths connecting its position
to the desired goal location; then, using the ROS move_base package, it
regulates velocities and yaw rates to follow the planned paths. Since 0.25
meters and 30 degrees are too small for velocity/yaw rate control, we directly
use $(x,y,o)$ (where $x,y$ are the “forward” and “left” positions from the
starting pose in meters and $o$ is the orientation in radian) as the action
space. After each time step, we pass the next desired global pose to
move_base. To account for velocity control and the body radius, the task is
considered successful if the robot stops within 1.5 meters of the desired
goal. Since the action space is no longer discrete, we set a maximum duration
of time (7 minutes) instead of timesteps. Furthermore, to account for
localization errors, we allow up to two trials per task. Additional robot
hardware specifications are in §V-A.

##  IV Methods

Our agent is composed of a semantic mapper and a navigation policy (Fig. 3
(b)), following previous work [1, 2]. Mapping converts egocentric RGB-D and
robot pose information to a 5cm3 voxel representation of the scene, with
labels for occupancy, explored areas, and semantic categories for each voxel
[1]. Core to mapping, is learning semantic (instance) segmentation, which
enables object category labeling. The navigation policy (Fig. 3 (a)) is
responsible for spatial understanding and deciding where to look/move next.

###  IV-A Location Consistency for Visual Perception

The self-supervised finetuning of the semantic segmentation model is composed
of two steps (Fig. 2). First, we algorithmically program a robot to navigate
around objects to link images with location consistency. Second, we train the
model in an off-line manner, using a contrastive loss.

#### Location consistency data collection

In a given scene, we initiate an agent and create a 3D occupancy map ($O$) of
the environment with frontier-based exploration [36]; the robot continues to
move to unexplored areas of the map until there is no unexplored area left.
Once complete, we use $O$ to sample the (next) location that the agent will
observe from multiple views (Fig. 2). First, we sample a point $(x,y)$ among
the occupied regions ($O$ summed across height). Second, from $O[x,y]$, we
sample $z$ (height) among occupied voxels. Third, we place a 50 cm3 cube $C$
centered around $(x,y,z)$ and have the agent move around this cube at
increment angles of 45o (i.e. 0, 45, 90, …) and distances of $0.3,0.6,0.9,1.2$
meters from $(x,y,z)$. “Intermediate stop” poses (where the agent stops
temporarily to stare at $C$) are placed in navigable regions of $O$. When the
agent pauses at an “intermediate stop”, it ray casts and if it hits $C$ the
current egocentric RGB view $R$ is recorded as an image of $C$. When the agent
has visited all “intermediate stops” for the current $C$, it saves all the
views as a “location-consistent” datum. The agent then repeats the entire
process by sampling and collecting data from a new location $(x,y)$ from $O$
until 70$\%$ of the map is covered or the agent has moved 7,000 steps.

We conduct the same data collection process in multiple real houses to obtain
a collection of images. We only consider images from the same cube $C$ as
location-consistent. In the physical environment, there are additional
complications as agents cannot achieve perfect angles and distances, but the
procedure still succeeds and is not adversely affected by these minor
perceptual changes. We enumerate factors that could influence the data
collection result, such as camera extrinsics, architectural choices, and robot
morphology; we hope future work explores these questions.

#### Contrastive training (LocCon)

As with previous work, we initialize our backbone with a pre-trained semantic
segmentation model, but training then commences using the aforementioned
location consistency data and contrastive training. The location serves as the
label tieing together images from disparate views (Fig. 2).

We denote a model under training as $M$ (backbone and segmentation head), and
its backbone as $M_{b}$. We denote ${I}$ as an image, $\\{{I}^{k}_{{\rm
pos},i}\\}_{i=1}^{n}$ as $n$ images collected from the same $k$-th cube
$C_{k}$ or augmented variants of these images, $\\{{I}^{k}_{{\rm
neg},j}\\}_{j=1}^{m}$ as $m$ images not collected from $C_{k}$, ${\rm
sim}(u,v)=M_{b}(u)^{\top}M_{b}(v)/\|M_{b}(u)\|\|M_{b}(v)\|$ as the cosine
similarity between two embedding vectors $M_{b}(u)$ and $M_{b}(v)$, and $\tau$
as a temperature hyper-parameter. We consider the popular contrastive loss as
in prior work [17, 16]:

$\ell_{B}=\sum_{k=1}^{K}\sum_{i=1}^{n}\,-{\rm log}\,\frac{e^{{\rm
sim}({I},{I}_{{\rm pos},i})/\tau}}{e^{{\rm sim}({I},{I}_{{\rm
pos},i})/\tau}+\sum_{j=1}^{m}e^{{\rm sim}({I},{I}_{{\rm neg},j})/\tau}}.$ (1)

To synchronize the segmentation head with the pretrained backbone, we
experimented with the addition of a regularizer, but we found no effect on
downstream performance. For each minibatch, the loss is therefore simply
$\ell_{B}$. We select $64$ different locations in an environment, and for each
location we sample four images from different views and apply image
augmentations to obtain two augmented variants of each of the four images.
This construction, with a mini-batch of 512 images, results in each image
having seven positively-paired images ($n=7$) and $504$ negatively-paired
image ($m=504$). We are interested in the semantic segmentation of 15 semantic
categories Table VII), following previous work [1, 2]. Since the pre-trained
model comes with a segmentation head for more than 15 classes, we treat the
rest as “background.” Adam optimizer with weight decay $1e-4$ and the learning
rate $\in\\{1e-6,1e-5,1e-4\\}$ that outputs the best validation accuracy
($1e-5$) was used.

###  IV-B A navigation policy from self-labeled scenes

The ObjectNav policy determines the next intermediate stop to find the goal,
based on the partial observations of the agent (Fig. 3 (a)). Unlike in prior
work [2], we demonstrate how training can be done without a labeled navigation
mesh.

We follow a process similar to the pre-mapping of §IV-A, but now also record
semantic information. We equip an agent with the semantic segmentation model
trained in §IV-A, and enforce frontier-based exploration; this creates a
semantic map of the scenes, self-labeled by the visual perception model. Then,
we learn the “potential functions” over the frontiers of partial maps; a high
potential function at a point in the map indicates that the point is worth
visiting. The total potential function is a sum of the “area potential
function” (high intensity means that the point is where the agent should
“explore” to gain a more complete map) and the “object potential function”
(high intensity means that the point is likely to be close to the object being
looked for). The area and object potentials can be directly calculated from
the geodesic distances obtained from the “full map”, and the policy network
learns the potentials given a “partial map,” similar to one that the agent
will create during a task.

Figure 4: Example ObjectNav task in a real house. The goal object is toilet.
Visual perception, semantic mapping, and the ObjecNav policy were all accurate
enough to lead to a task success.

##  V Real-World Experiments

In §V-A, we first show that our agent, trained with self-supervision only, can
perform ObjectNav competitively in the real world and in simulation; in §V-B,
we show that self-supervised training of visual perception results in good
real world transfer, and real-world in-situ training can further remedy
ObjectNav error modes.

The structure of our ObjectNav agent is shown in Fig. 3 (b). The agent relies
on our self-supervised segmentation model (Self-Seg) and ObjectNav Policy
(Self-Pol). Self-Seg is trained by taking an “off-the-shelf” model (aka
deeplabv2[3] pre-trained with COCO-things 164k[37]) and applying LocCon
(§IV-A) with data sampled from 100 houses, randomly selected from a pool of 25
houses from the training split of Gibson-mini and 720 houses from HM3D [31].
We again highlight that no labeled 3d mesh was used in this training, since
they are expensive and mostly unavailable for HM3D. Self-Pol is trained using
the method of §IV-B, from semantic maps created and self-labeled (with Self-
Seg) from frontier-based exploration in the 25 Gibson houses.

We define notations used in the remaining sections. We denote the “off-the-
shelf” segmentation model (aka deeplabv2 [3] pre-trained with COCO-things 164k
[37]) as O-t-S. We replace Self-Seg and Self-Pol with their fully supervised
(using all the available simulator annotations) counterparts; we call this
agent “FullSup agent” (and its components Full-Seg and Full-Pol) and our self-
supervised agent “SelfSup agent.” Full-Seg, initialized with O-t-S., is
trained with fully supervised training from 75K randomly sampled, labeled
images from the 25 Gibson houses, and Full-Pol is trained with PONI using the
ground-truth labeled 3d mesh of the 25 houses. We follow the same protocols of
[3] and [2] (e.g. the pool of hyperparameters to optimize within, output
dimensions) to train Full-Seg and Full-Pol, respectively.

###  V-A Real World ObjectNav

To show the transfer of our ObjectNav agent (§IV-B, §VI-A) to real houses, we
built a robot from a Jackal base that has similar specifications to the
simulation settings (§III). Perception is performed with an Intel Realsense
D435 RGB-D camera mounted at 0.88 meters from the floor and all computation
was run locally in real-time on an Nvidia Orin (Fig. 4 (a)).

Category | Success
---|---
Total |  066.6%
Chair |  033.3%
Couch | 100.0%
Potted Plant |  066.6%
Bed | 100.0%
Toilet | 100.0%
TV |  033.3%

TABLE II: ObjectNav Results Success rate of 18 tasks in 3 AirBnBs.

For localization, we pre-map the environment with the ROS RTAB-map package
when first entering a new house. When executing a policy, the robot receives
its pose from the ROS SLAM package and uses the existing map only for its
occupancy map instead of that from the semantic mapper; all other outputs of
the semantic mapper, including mapping of explored areas and semantic
categories, are as in simulation.

Experiments were performed in three rented AirBnBs where we set up three tasks
for each of the 6 goal objects. Because a Jackal base is larger than the agent
radius in simulation, we start the agent in areas that are wider than 1.5
meters, which is usually in the living room or the kitchen. An example
experiment is shown in Figure 4. Across 18 tasks, we obtain success rate of
$66.6\%$. We show two example runs in the supplementary video. Out of 6
failures, 4 were due to misrecognition of goal objects (e.g. recognizing a
“white couch” as “bed”), and 1 was due to depth/ segmentation misalignment, 1
was due to localization failure.

###  V-B Self-Supervised In-Situ Training of Visual Perception

In §V-A, we saw that a major error mode in real world ObjectNav is visual
perception; the robot stops at wrong goal objects or goal objects are seen but
misrecognized. Since we find that visual perception generalization failures
somewhat nullify efficient exploration/ nav policy, we focus on in-situ
learning for visual perception. We show that LocCon training of semantic
segmentation can remedy these issues.

To replicate applying LocCon in a the real world, we collected location
consistency images from 4 airbnbs, of which three are disjoint from the ones
in §V-A. We replicate the sampling procedure of §IV-A as follows; we use a
pre-map of the environment to sample an $(x,y)$ and then pick $z$ as in §IV-A.
We place a tripod with Intel Realsense D435 RGBD camera mounted at 0.88 meters
at 5 to 10 locations around the picked $(x,y,z)$ among navigable points from
which $(x,y,z)$ is visible, mimicking the “intermediate stops” of §IV-A. We
collected a total of 479 images and 70 $(x,y,z)$’s. For validation, we
annotate five images (and exclude them from the training set) from the AirBnB
that was used in §V-A, so that all goal object classes appear at least once.
We show examples of train/ val images in the supplementary video.

In Table III, we show the Mean IOU of O-t-S., Self-Seg (Self.), and Full-Seg
(Full.), when evaluated and in-situ trained on real world data with LocCon.
First, we find that Full. gives the worst and Self. gives the best
initialization in the real world. This suggests that our LocCon training makes
simulation training useful for Sim2Real transfer, despite the data challenges
shown in §VI. Second, we find that our LocCon training gives performance boost
for all models.

| O-t-S. |  | Self. |  | Full.
---|---|---|---|---|---
Initial | 61.1 |  | 61.9 |  | 58.8
with In-Situ | 62.2 |  | 62.8 |  | 61.7
Performance $\Delta$ | 1.1 $\uparrow$ |  | 0.9 $\uparrow$ |  | 2.9$\uparrow$

TABLE III: Semantic segmentation gains from real-world in-situ LocCon
finetuning.

While we do not include plots due to space limmitations, we found that as we
increase the number of training images for LocCon training, the IOU increases,
in both simulation and the real world.

We retrofit Self-Seg trained with In-Situ (Table III) to real world ObjectNav
(§V-A) and examine if this adapted model can change any failures to non-
failures (Fig. §1). More specifically, we replicate the same trajectory the
agent took in each of the failed tasks, but use Self-Seg trained with In-Situ
(Table III) in the semantic mapper. We observe that this turns one out of six
failed tasks as a non-failure (Fig. 1).

##  VI Simulation Experiments

In §VI-A, we present ObjectNav results in simulation. In §VI-B, we analyze
navigation error modes, with special attention to those caused by simulation
artifacts (Fig.5). In §VI-C, we provide an in-depth analysis of self-
supervision vs. full supervision (with annotations) for visual perception, and
conclude that the latter tends to learn artifacts from bad rendering and
annotations. In §VI-D, we analyze the performance and behavior of a self-
supervised nav policy.

###  VI-A ObjectNav in simulation

While our focus is on the deployable real-world system, we also evaluate our
agent in simulation (two validation sets of Gibson ObjectNav) for fair
comparison to both existing supervised models [1] (Gibson-1) and fully self-
supervised approaches [5] (Gibson-2). Table IV shows ObjectNav results of our
approach, compared with other fully supervised and partially (visual
perception is self-supervised while policy was trained with annotated
mesh)/fully self-supervised methods. Succ. denotes task success rate across
all validation tasks, SPL the Succ. weighted by path length, and DTS the
average distance to the 1 meter (stop threshold) radius of the true goal. Our
method outperforms fully self-supervised methods on Gibson-2 and performs as
competitively as methods with more supervision on Gibson-1.

From Table IV, we do observe that directly training with simulation
annotations gives advantages; however, we find that this in-domain advantage
is due to adapting to the artifacts and noise of simulation, not learning the
indoor semantics that is transferrable to the real world (Fig. 5). In §VI-B,
§VI-C, §VI-D, we provide in-depth analyses of the challenges of supervised
training with simulation artifacts.

|  Gibson - 1 [34] |  Gibson - 2[5]
---|---|---
Method |  Succ. $\uparrow$ |  SPL $\uparrow$ |  DTS $\downarrow$ |  Succ. $\uparrow$ |  SPL $\uparrow$
Fully Supervised w. Annotations
DD-PPO [38] | 15.0 | 10.7 | 3.24 | N.A. | N.A.
SemExp [1] |  65.5∗ |  36.5∗ |  1.45∗ |  42.8∗ |  20.8∗
PONI [2] |  66.2∗ |  36.8∗ |  1.41∗ |  34.9∗ |  20.3∗
Partially Self-Supervised
SemExp + SEAL [11] | 62.7 | 33.1 | N.A. | N.A. | N.A.
Fully Self-Supervised
ZER [5] | N.A | N.A | N.A | 11.3 | N.A
ZSON [21] | N.A | N.A | N.A | 31.3 | 12.0
Ours | 60.0 | 31.2 | 1.89 | 36.0 | 19.2
TABLE IV: ObjectNav validation results in Simulation (Gibson-1 and -2). We
compare fully supervised, partially self-supervised, and fully self-supervised
methods with our approach. We perform competitively with SOTA fully supervised
methods on Gibson-1 and significantly outperforms end-to-end self supervised
methods on Gibson-2. Cells with N.A. are due to gaps in the literature; ZER
and ZSON only report results on Gibson-2, while the other methods only do so
on Gibson-1. $*$ stands for our own implementation of these methods, since the
original code of [2] that reports results of SemExp and PONI on Gibson-1 was
not fully available.

###  VI-B Error Modes

Table V shows the breakdown of the error modes of FullSup agent (denoted
Full.) and SelfSup agent (denoted Self.). We find that simulation
reconstruction and annotation can be very noisy from certain views and can
produce RGB that are highly unlikely in the real world (Fig. 5),

Error Modes | Self. | Full.
---|---|---
Misrecognition | 13.2% | 16.0%
Bad Rendering∗ | 8.0% | 1.2%
The Rest
Policy Error | 3.6% | 2.4%
Mapping Error | 6.0% | 7.2%
Mislabeled Goal∗ | 5.6% | 5.2%
Depth Render Error | 1.6% | 0.8%
$\Rightarrow$ Total | 16.8% | 15.6%

TABLE V: Error modes of Gibson-1 ObjectNav. ∗: Error modes unique to
simulation.

and first enumerate the errors that are unique to simulation, highlighted with
$*$ in Table V. These errors are Bad Rendering (Fig. 5(a); deformed rendering
causes Self. to incorrectly detect goal objects and incorrectly terminate Nav
task) and Mislabeled Goal (Fig. 5(b); the goal objects are incorrectly
labeled). We find that Self. is more susceptible to these errors, with
especially Bad Rendering happening $\sim$7 times more in Self. than Full.
While this gap essentially explains the performance gap between SelfSup agent
and fully supervised methods (Table IV), we note that these errors are caused
by simulation artifacts. This agrees with our result in §V-B, that adapting to
these errors harm real-world transfer.

Among errors not caused by reconstruction/ annotation challenges, the most
common error mode for both agents is misrecognition \- goal objects not
recognized even though an instance of it was seen or misrecognized (e.g.
“couch” as “bed”); surprisingly, Self. is more robust to this error. The other
errors consist of policy error (bad exploration led to not finding a goal
object), mapping/ planning error (agent goes out of the map and gets lost),
and depth render error (agent stops outside the 1m radius due to depth error).

Figure 5: Data challenges: bad rendering and annotations.

Metric | Test with Sim Noise |  | Test w.o. Sim Noise
---|---|---|---
O-t-S. | Self. | Full. |  | O-t-S. | Self. | Full.
Mean IOU | 38.3 | 44.1 | 52.7 |  | 44.2 | 53.1 | 53.1
Freq. IOU | 92.2 | 93.0 | 94.0 |  | 94.9 | 95.6 | 95.6
Pixel Acc. | 95.1 | 95.9 | 96.5 |  | 96.8 | 97.4 | 97.4

TABLE VI: Semantic segmentation results on simulated data. Results are on the
6 goal objects (IOUs of all 15 objects are in Table VII); Self-Seg and Full-
Seg are denoted Self. and Full.

###  VI-C Full. vs Self. Semantic Segmentation

We further analyze the behavior of Self-Seg (denoted Self.) and Full-Seg
(denoted Full.), to better understand the results of Table V. We present
results on two test sets, which each consist of 500 images from 5 validation
houses of Gibson. We realize that simply randomly teleporting agents and
collecting images (Test with Sim Noise) results in many images with bad
rendering (that will not appear in the real world) and bad annotations (Fig.
5). Thus, we further evaluate on a test set of 500 images with obvious
reconstruction/ annotation errors filtered by the authors (Test without Sim
Noise).

Table VI shows that Self-Seg (Self.) yields performance on par with Full-Seg
(Full.) across all metrics, when evaluated without simulation noise. However,
on Test with Sim Noise, the performance of Self. sharply drops while that of
Full. nearly stays the same. This leads to the conclusion that supervised
training with reconstruction noise leads to learning simulation noise and
artifacts; in Fig 5, we observe that Full. learns to both ignore bad
renderings and adapt to the bad annotations of simulation. This explains the
$\sim$7x gap in the Bad Rendering error in Table V and the result that Full.
harms real-world transfer, presented in §V-B.

Table VII shows the class IOU of the three models across object categories.
Interesting cases are bolded, but we draw the reader’s attention in particular
to cases where full supervision has hurt performance. This appears to happen
with small objects (e.g. Cup and Book); we find these objects are often
mislabeled. Because Self. does not use labels, we are unaffected by this noise
– a good sign for more realistic scenarios. However, at present, small objects
pose a major challenge for all models regardless of their supervision.

Category | O-t-S. | Self | Full
---|---|---|---
Chair | 41.9 | 40.3 | 42.6
Couch | 57.7 | 60.7 | 62.4
Potted Plant | 25.0 | 29.6 | 30.1
Bed | 49.0 | 64.8 | 75.8
Toilet | 5.53 | 37.1 | 48.7
TV | 32.7 | 41.5 | 14.5
Dining Table | 11.0 | 11.9 | 19.6
Oven | 17.6 | 35.1 | 54.4

Category | O-t-S. | Self | Full
---|---|---|---
Sink | 17.6 | 25.7 | 34.8
Refrigerator | 25.3 | 36.7 | 45.2
Vase | 0.88 | 1.79 | 4.93
Bottle | 0.00 | 0.22 | 0.00
Clock | 0.00 | 0.00 | 0.00
Cup | 28.4 | 33.2 | 0.00
Book | 11.2 | 5.33 | 1.18
Total Mean | 27.4 | 32.5 | 33.2

TABLE VII: Semantic segmentation by object category. We provide a category
level IOU breakdown of results in Table VI. Self. shows a significant boost
over O-t-S. for Toilet, Bed, and Oven; Full. can hurt performance sometimes
(e.g. Book, Cup, TV).

###  VI-D Full. vs Self. Nav Policy

First, we investigate if the gap of policy error in Table V is meaningful. In
Table VIII, we compare ObjectNav performance on Gibson - 1 with a cross
product of two segmentation models and three ObjectNav policy models; the
former are Self-Seg and Full-Seg, and the latter are Self-Pol, Full-Pol, and
SemExp [1] trained with ground truth locations from labeled 3d meshes.

| Segmentation Models
---|---
Nav Policy | Self-Seg | Full-Seg
Self-Pol | 60.0/ 31.2 | 64.4/ 33.2
Full-Pol444The lower performance of Full-Sup PONI may be because of misalignment between the pre-trained model provided by [2] and our reimplementation of the navigation components which we could not reproduce with the provided code. We also compare against Full-Sup SemExp. | 56.3/ 28.9 | 58.8/ 28.8
SemExp | 59.0/ 32.3 | 63.4/ 32.2

TABLE VIII:  ObjectNav “Success Rate/ SPL” on Gibson-1 for segmentation models
with different navigation policy models.

Table VIII shows that Self-Pol performs as well as Full-Pol across
segmentation models, implying that existing expensive methods that train the
Nav policy with self-supervision, such as using ground truth locations of
objects, collecting demonstrations, or using ImageNav (in which the goal
location comes for free as mentioned in [21]) are not needed.

Furthermore, this result implies that policy training is fairly agnostic to
noise; in Table V, we see that Self-Seg introduces more noise to the self-
labeled semantic map (from Bad Rendering) than the mislabeling in the mesh
(from Mislabeled Goal). Still, Self-Pol, while trained with more noisy data,
performs on par with (even slightly better than) its counterparts trained with
mesh annotations. We hypothesize that this is because the Nav Policy is
trained to capture abstract knowledge on house layouts (as mentioned in [1,
2]) and thus is fairly robust to both noises in Fig. 5.

##  VII Conclusion

We present a self-supervised method to train all components of an ObjectNav
agent that works in the real world. We find that our method can provide robust
transfer to the real world, while supervised training with simulation
annotations can actually harm transfer performances. Most importantly, we show
that our method can work for in-situ training to further improve ObjectNav
performance, which encourages future research efforts on in-situ training. Our
aim is to further encourage the development of physical robot systems that
adapt without requiring human intervention.

## References

  * [1] Devendra Singh Chaplot, Dhiraj Prakashchand Gandhi, Abhinav Gupta, and Russ R Salakhutdinov.  Object goal navigation using goal-oriented semantic exploration.  Advances in Neural Information Processing Systems, 33, 2020.
  * [2] Santhosh K. Ramakrishnan, Devendra Singh Chaplot, Ziad Al-Halah, Jitendra Malik, and Kristen Grauman.  Poni: Potential functions for objectgoal navigation with interaction-free learning.  In Computer Vision and Pattern Recognition (CVPR), 2022 IEEE Conference on. IEEE, 2022.
  * [3] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille.  Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.  IEEE transactions on pattern analysis and machine intelligence, 40(4):834–848, 2017.
  * [4] Fei Xia, Amir R. Zamir, Zhiyang He, Alexander Sax, Jitendra Malik, and Silvio Savarese.  Gibson env: Real-world perception for embodied agents.  In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9068–9079, 2018.
  * [5] Ziad Al-Halah, Santhosh Kumar Ramakrishnan, and Kristen Grauman.  Zero experience required: Plug & play modular transfer learning for semantic visual navigation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17031–17041, 2022.
  * [6] Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang.  Matterport3d: Learning from rgb-d data in indoor environments.  arXiv preprint arXiv:1709.06158, 2017.
  * [7] Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, et al.  The replica dataset: A digital replica of indoor spaces.  arXiv preprint arXiv:1906.05797, 2019.
  * [8] Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al.  Habitat: A platform for embodied ai research.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9339–9347, 2019.
  * [9] Chengshu Li, Fei Xia, Roberto Martín-Martín, Michael Lingelbach, Sanjana Srivastava, Bokui Shen, Kent Vainio, Cem Gokmen, Gokul Dharan, Tanish Jain, et al.  Igibson 2.0: Object-centric simulation for robot learning of everyday household tasks.  arXiv preprint arXiv:2108.03272, 2021.
  * [10] Santhosh Kumar Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alexander Clegg, John M Turner, Eric Undersander, Wojciech Galuba, Andrew Westbury, Angel X Chang, Manolis Savva, Yili Zhao, and Dhruv Batra.  Habitat-matterport 3d dataset (HM3d): 1000 large-scale 3d environments for embodied AI.  In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
  * [11] Devendra Singh Chaplot, Murtaza Dalal, Saurabh Gupta, Jitendra Malik, and Russ R Salakhutdinov.  Seal: Self-supervised embodied active learning using exploration and 3d consistency.  Advances in Neural Information Processing Systems, 34:13086–13098, 2021.
  * [12] Ram Ramrakhya, Eric Undersander, Dhruv Batra, and Abhishek Das.  Habitat-web: Learning embodied object-search strategies from human demonstrations at scale.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5173–5183, 2022.
  * [13] Karmesh Yadav, Ram Ramrakhya, Arjun Majumdar, Vincent-Pierre Berges, Sachit Kuhar, Dhruv Batra, Alexei Baevski, and Oleksandr Maksymets.  Offline visual representation learning for embodied navigation.  arXiv preprint arXiv:2204.13226, 2022.
  * [14] Yilun Du, Chuang Gan, and Phillip Isola.  Curious representation learning for embodied intelligence, 2021.
  * [15] Jianwei Yang, Zhile Ren, Mingze Xu, Xinlei Chen, David Crandall, Devi Parikh, and Dhruv Batra.  Embodied visual recognition.  arXiv preprint arXiv:1904.04404, 2019.
  * [16] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In International conference on machine learning, pages 1597–1607. PMLR, 2020.
  * [17] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729–9738, 2020.
  * [18] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9650–9660, 2021.
  * [19] Aaron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  arXiv preprint arXiv:1807.03748, 2018.
  * [20] Samir Yitzhak Gadre, Mitchell Wortsman, Gabriel Ilharco, Ludwig Schmidt, and Shuran Song.  Clip on wheels: Zero-shot object navigation as object localization and exploration.  arXiv preprint arXiv:2203.10421, 2022.
  * [21] Arjun Majumdar, Gunjan Aggarwal, Bhavika Devnani, Judy Hoffman, and Dhruv Batra.  Zson: Zero-shot object-goal navigation using multimodal goal embeddings.  arXiv preprint arXiv:2206.12403, 2022.
  * [22] Oleksandr Maksymets, Vincent Cartillier, Aaron Gokaslan, Erik Wijmans, Wojciech Galuba, Stefan Lee, and Dhruv Batra.  Thda: Treasure hunt data augmentation for semantic navigation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15374–15383, 2021.
  * [23] Arsalan Mousavian, Alexander Toshev, Marek Fišer, Jana Košecká, Ayzaan Wahid, and James Davidson.  Visual representations for semantic target driven navigation.  In 2019 International Conference on Robotics and Automation (ICRA), pages 8846–8852. IEEE, 2019.
  * [24] Wei Yang, Xiaolong Wang, Ali Farhadi, Abhinav Gupta, and Roozbeh Mottaghi.  Visual semantic navigation using scene priors.  arXiv preprint arXiv:1810.06543, 2018.
  * [25] Raphael Druon, Yusuke Yoshiyasu, Asako Kanezaki, and Alassane Watt.  Visual object search by learning spatial context.  IEEE Robotics and Automation Letters, 5(2):1279–1286, 2020.
  * [26] Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Jordi Salvador, Kiana Ehsani, Winson Han, Eric Kolve, Ali Farhadi, Aniruddha Kembhavi, and Roozbeh Mottaghi.  Procthor: Large-scale embodied ai using procedural generation, 2022.
  * [27] Apoorv Khandelwal, Luca Weihs, Roozbeh Mottaghi, and Aniruddha Kembhavi.  Simple but effective: Clip embeddings for embodied ai.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14829–14838, 2022.
  * [28] Saurabh Gupta, James Davidson, Sergey Levine, Rahul Sukthankar, and Jitendra Malik.  Cognitive mapping and planning for visual navigation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2616–2625, 2017.
  * [29] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International Conference on Machine Learning, pages 8748–8763. PMLR, 2021.
  * [30] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition, 2015.
  * [31] Santhosh K Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alex Clegg, John Turner, Eric Undersander, Wojciech Galuba, Andrew Westbury, Angel X Chang, et al.  Habitat-matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai.  arXiv preprint arXiv:2109.08238, 2021.
  * [32] Theophile Gervet, Soumith Chintala, Dhruv Batra, Jitendra Malik, and Devendra Singh Chaplot.  Navigating to objects in the real world.  arXiv preprint arXiv:2212.00922, 2022.
  * [33] So Yeon Min, Devendra Singh Chaplot, Pradeep Ravikumar, Yonatan Bisk, and Ruslan Salakhutdinov.  Film: Following instructions in language with modular methods.  arXiv preprint arXiv:2110.07342, 2021.
  * [34] Peter Anderson, Angel Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, et al.  On evaluation of embodied navigation agents.  arXiv preprint arXiv:1807.06757, 2018.
  * [35] Manolis Savva, Angel X Chang, Alexey Dosovitskiy, Thomas Funkhouser, and Vladlen Koltun.  Minos: Multimodal indoor simulator for navigation in complex environments.  arXiv preprint arXiv:1712.03931, 2017.
  * [36] Brian Yamauchi.  A frontier-based approach for autonomous exploration.  In Proceedings 1997 IEEE International Symposium on Computational Intelligence in Robotics and Automation CIRA’97.’Towards New Computational Principles for Robotics and Automation’, pages 146–151. IEEE, 1997\.
  * [37] Holger Caesar, Jasper Uijlings, and Vittorio Ferrari.  Coco-stuff: Thing and stuff classes in context.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1209–1218, 2018.
  * [38] Erik Wijmans, Abhishek Kadian, Ari Morcos, Stefan Lee, Irfan Essa, Devi Parikh, Manolis Savva, and Dhruv Batra.  Dd-ppo: Learning near-perfect pointgoal navigators from 2.5 billion frames.  arXiv preprint arXiv:1911.00357, 2019.