# Statistical relational learning and neuro-symbolic AI: what does first-order
logic offer?

Vaishak Belle The material below is adapts discussions from a recent book by
the author [7], but has been extended to incorporate details and discussions
about neuro-symbolic AI.University of Edinburgh, UK<EMAIL_ADDRESS>
###### Abstract

In this paper, our aim is to briefly survey and articulate the logical and
philosophical foundations of using (first-order) logic to represent
(probabilistic) knowledge in a non-technical fashion. Our motivation is three
fold. First, for machine learning researchers unaware of why the research
community cares about relational representations, this article can serve as a
gentle introduction. Second, for logical experts who are newcomers to the
learning area, such an article can help in navigating the differences between
finite vs infinite, and subjective probabilities vs random-world semantics.
Finally, for researchers from statistical relational learning and neuro-
symbolic AI, who are usually embedded in finite worlds with subjective
probabilities, appreciating what infinite domains and random-world semantics
brings to the table is of utmost theoretical import.

##  1 Introduction

Statistical relational learning and neuro-symbolic AI are having a growing
impact on recent discussions regarding the connection between (deep) learning
and logical syntax [24, 32, 17, 6]. These fields are motivated by the use of
relations and logical reasoning, specifically entailment and satisfaction, in
learning paradigms. They recognize that the world is inherently relational;
objects have properties and attributes, employees have employers that are part
of organisations, individuals are part of social networks. More elaborately,
in biological networks, genes are the fundamental units of heredity that
determine the traits of an organism. The process by which genes are expressed
in proteins involves several steps, including transcription and translation,
that are critical for the proper functioning of cells and organisms. If we
need to reason about biological data, then algorithms that harness these
relationships and processes are fundamental.

Logical reasoning is crucial to deduce properties about these relationships.
For example, if we know that a dog is a mammal and a cat is a mammal, we
should be able to deduce that they both give birth to live young, except
unless those mammals that are monotremes. On the other hand, if we know that
insects cannot be mammals, then we can also deduce that a dog cannot be an
insect.

In the insular world of (deep) machine learning, where we assume that objects
are atomic and independent, the incorporation of logical artifacts provides a
much richer understanding of the world and its objects, leading to a more
nuanced and accurate interpretation of the data.

A second important area where the impact of symbols extends to is the notion
of explainability in artificial intelligence. While traditionally this area
has been about interpreting decision boundaries, it now includes attempts to
use logical and programmatic objects to characterize the prediction model.
See, for example, recent celebrated successes on deep concept induction [28,
18]. There is the more complex area of constructing mental models of the users
and providing explanations relative to that putative user [25]. This area is
also governed by symbolic and formal principles. Symbolic principles such as
sufficiency and necessity are also being used to explore causal dimensions for
explanations [15].

Despite this success and the many syntactic flavors explored in the research
community, the underlying logical language is essentially propositional.
Almost always, the outputs of the learning model (such as a neural network)
are either interpreted probabilistically over a finite set of propositional
variables, or as the numeric truth value accorded to a finite set of
propositions (in fuzzy logical approaches). Even in a (probabilistic) logic
programming setting such as ProbLog, cycle breaking strategies are considered
to reduce query-driven reasoning to probabilistic labels over finitely many
propositions via weighted model counting [39, 19].

In this paper, _our aim is to briefly survey and articulate the logical and
philosophical foundations of using (first-order) logic to represent
(probabilistic) knowledge in a non-technical fashion_. Our motivation is three
fold. First, for machine learning researchers unaware of why the research
community cares about relational representations, this article can serve as a
gentle introduction. Second, for logical experts who are newcomers to the
learning area, such an article can help in navigating the differences between
finite vs infinite, and subjective probabilities vs random-world semantics.
Finally, for researchers from statistical relational learning (StarAI) and
neuro-symbolic AI (NeSyAI), who are usually embedded in finite worlds with
subjective probabilities, appreciating what infinite domains and random-world
semantics brings to the table is of utmost theoretical import: in fact, one
may even argue it is needed to make the area well-defined! StarAI and NeSyAI
wishes to generalize to unforeseen examples, and deal with statistical
properties, and yet their prove properties for finite models and degrees of
belief.

Of course, the use of a relational language (with a finite domain
notwithstanding), that only serves as syntactic sugar for the specification
for probabilistic models, is not without its advantages. It can greatly simply
the specification of relations and hierarchies over random variables, and can
also speed up probabilistic inference in some cases by appealing to first-
order resolution [40]. _But by incorporating probabilistic reasoning into
first-order logic, we might be able to better deal with infinite domains and
incomplete knowledge in a more principled way, and allowing for the
specification of random properties (and not simply subjective probabilities,
as assumed by current mainstream languages) in an open-world setting._
However, there will still challenges to be addressed, such as the
computational complexity of reasoning with large knowledge bases and the need
for effective methods of learning from data.

##  2 Why representation matters

It is worth reflecting on the nature of knowledge representation and how it
affects modeling and learning. Regardless of whether we use a logical
representation (such as a logic program), a probabilistic one (such as a
Bayesian network), or a connectionist one (such as a neural network), our goal
is to represent information and capture the knowledge of a robot or vision
system. We want to understand how this representation produces interesting
conclusions that determine what the robot knows and understands about the
world.

While acquisition is undoubtedly important, for now, we are focusing on the
language used for representing information. It is crucial to equip this
language with tools for modeling and reasoning about the world. In a statement
remarkably similar in spirit, Pearl writes [37]:

_This is why you will find me emphasizing and reemphasizing notation,
language, vocabulary and grammar. For example, I obsess over whether we can
express a certain claim in a given language and whether one claim follows rom
others. My emphasis on language also comes from a deep conviction that
language shapes our thoughts. You cannot answer a question that you cannot
ask, and cannot ask a question that you have no words for._

Which language should we choose, and what should we represent? Turing machines
are foundational structures for investigating the computable universe. They
offer minimal, yet powerful read and write operations. However, being a
computational abstraction means we have no means to concretely talk about what
the robot knows about the environment it operates in. Likewise, binary system
code runs robotic software, but is too coarse and opaque to explain a
purposeful agent. Vectorized representations and non-linear decision
boundaries as obtained by sigmoid neural units over many interconnected layers
are also too coarse to explain the functioning of a deep network. Although
these models seem capable of capturing a potentially infinite hypothesis space
and high-dimensional data, including high-resolution images such as
mammograms, an operational description is too opaque for humans to decipher.

Therefore, we require a modeling framework that is understandable to domain
experts and allows for the updating and provision of new information, while
also being suitable for computational processes. In particular, by a
representation, we mean a mapping from one entity to another. The former is
concrete and computationally processable. For example, “rain” might stand for
the assertion that it is raining today. Reasoning is the computational
processing of representations to produce new knowledge. We focus on formal
(symbolic) logic, where all information is stored explicitly or implicitly
using symbols.

The distinction between explicit and implicit knowledge is important: explicit
knowledge is often the information that is directly modelled or provided by
the domain expert. This could come in the form of rules, databases, knowledge
bases, graphs, or any other structured data. Implicit knowledge is what is
obtained from the explicit knowledge through one or more reasoning steps.

Reasoning can be understood in two ways. First, it can define the mathematical
framework for obtaining explicit knowledge from implicit knowledge. In logic,
we often provide a semantics and/or proof theory that determines when
sentences are “entailed” from a knowledge base (a set of sentences). Second,
reasoning can provide an implementation strategy for computing entailments: an
algorithm that takes a knowledge base and a query as input and outputs whether
the query is entailed. Standard platforms for designing such algorithms
include theorem provers, model checkers, and satisfiability solvers.

##  3 Logic and probability: how did we get here?

Leibniz perhaps first suggested that thinking could be understood as symbol
manipulation, but it was only with the results of Boole and Frege that an
algebraic view of logic became possible. Symbolic logic was primarily used for
studying the foundations of mathematics until the 19th century. David
Hilbert’s program, Kurt Gödel’s seminal incompleteness theorem, and other such
inquiries studied the arithmetic of natural numbers and what logically follows
from that arithmetic. John McCarthy, among others, began to see first-order
logic as a formalism for capturing commonsense and language only since the
1960s [33, 34, 36]. He suggested that the system’s knowledge could be
represented in a formal language, and a general-purpose algorithm could then
determine the necessary actions to solve the problem at hand. A major
advantage of this approach is that the representation can be scrutinized and
understood by external observers, allowing them to improve the system’s
behavior by making statements to it.

Numerous such languages emerged in the years to follow, but first-order logic
remained at the forefront as a general and powerful option. Unfortunately, one
of the main arguments against a logical approach is the pervasive uncertainty
present in almost every domain of interest. This uncertainty can manifest in
various forms, such as measurement errors (e.g. readings from a thermometer),
the absence of categorical assertions (e.g. smoking may be a factor for
cancer, but cancer is not an absolute consequence for smokers), and the
presence of numerous “latent” factors. These factors include causes that the
modeler may not have taken into account, which question the legitimacy of the
model.

Modeling uncertain worlds required a rigorous formulation, which has been
achieved through probabilistic models, of course. Probabilistic models,
obtained either by explicit specification or implicitly induced by modern
machine learning methods such as deep learning, have supercharged the
application of statistical methods in language understanding, vision, and data
analysis more generally.

Despite the success of probabilistic models, we observe that they are
essentially propositional, yet are deployed in an inherently relational world.
That is, they easily make sense of “flat” data, where atomic events are
treated as independent and identically distributed random variables. This
realization has led to the design of probabilistic concept languages,
culminating in the area of statistical relational learning, neuro-symbolic AI,
and other hybrid languages.

Note that there is a considerable history on the unification of logic and
probability, including early work by Carnap and Gaifman [13, 20]. Interested
readers can refer to [6] for a discussion on the historical development.

##  4 First-order logic

First-order logic is, as discussed above, a widely studied formalism. To
contrast it with popular proposals in statistical learning, let us start with
an example often used in that community, which involves smoking within social
networks. We will then discuss variations allowed by propositional and first-
order logic, while also briefly demonstrating their features. As will become
clear, virtually every formalism from that community does not support
arbitrary first-order logic. There is one good reason for this: first-order
logic is only semi-decidable, and computing entailments is not tractable [11].
(Although first-order logic with reals as its domain is decidable, it is best
used to reason about arithmetic properties, as seen in satisfiability modulo
solvers.)

Let us imagine two individuals $a$ (for Alice) and $b$ (for Bob) such that $a$
is a smoker and the two are friends. Using $S(x)$ as a predicate to mean $x$
is a smoker and $F(x,y)$ to mean $x$ is a friend of $y$, we have: $S(a)\land
F(a,b)$. There are two propositional assertions here, connected using a
conjunction, to say both are true in the knowledge base (KB).

Suppose we found that smokers influence those in their social networks to also
smoke, in which case, we would wish to add $S(b)$ to the KB. However, rather
than working this out by hand, we might wish to express this observation using
an implication rule: $S(a)\land F(a,b)\supset S(b)$. Such rules involve only
constants, and so, with quantifiers, we will be able to say that the smokers
rule applies to every individual: $\forall x,y.~{}S(x)\land F(x,y)\supset
S(y)$. The quantification is over a “domain of discourse”, which defines the
constants to which the expression following the quantification applies.

Naturally, the knowledge base (KB) contains “facts”, i.e., ground atoms, such
as $S(a)$ and $F(a,b)$, in addition to quantified formulas. These facts
constitute explicit knowledge. By applying these facts to the rules, we can
derive $S(b)$, which is implicit in the sense of the terminology used earlier.
In this way, the truth value for every instance of $S(x)$ can be determined
from explicit or implicit knowledge, similar to a database. This denotes the
case of “complete knowledge.”

Incompleteness is usually expressed using disjunctions and existential
quantifiers. For example, assuming a third constant $c$ for Carol: $(S(a)\lor
S(c))\land\neg(S(a)\land S(c))$ says that either Alice or Carol is a smoker
but not both.

The use of an existential statement can declare the existence of an individual
with a certain property, but not specify who that individual might be. For
example, the following sentence states that there is a friend of Alice who is
not Carol: $(\exists x.~{}F(x,b))\land\neg F(c,b)$. Suppose we further
stipulate that no one can be their own friend: $\forall x.~{}\neg F(x,x)$. Of
course, if our domain were finite and consisted only of the individuals
discussed so far - $\\{a,b,c\\}$ \- then $F(a,b)$ would follow from the above
two sentences. However, if the domain is infinite, then infinitely many
candidates are possible friends of Bob, but we are not certain of any one of
them.

###  4.1 Infinite domains

In addition to incomplete information about domains, other types of incomplete
knowledge can also be expressed with infinite domains. For instance, when
quantifiers range over an infinite set, the sentence $\forall x.~{}x\neq
a\supset S(x)$ states that there are infinitely many individuals who are
smokers, except for Alice who might or might not be a smoker. Therefore, it
models unknown atoms. To represent unknown values, we can use the following
sentence: $\forall x(S(x)\supset eyecolor(x)\neq green)$, which declares that
among smokers in our population, of whom there may be infinitely many, none
has green eyes.

Although there are a wide range of formalisms in statistical relational
learning, constants are typically fixed in advance and are finite. Functions
are rarely allowed, and often the interpretation of predicates is fixed as
well: that is, the set of ground atoms is fixed, as well as their truth
values. When these formalisms allow atoms to be decorated with probabilities,
there is some flexibility in terms of the interpretation of predicates, in the
sense that a positive instance is more likely than a negative instance, or
vice versa.

As noted, the full first-order language, especially over a possibly infinite
domain, can allow for unknown sets of atoms, their possible values, and
varying sets of objects (e.g., when the set of humans is left open). This
makes standard first-order logic a powerful and general-purpose language for
representing knowledge, and we believe it can be a promising candidate for the
next generation of probabilistic relational models.

In fact, keeping the domain undefined is often referred to as “open-world”
modelling. Note that closeness is not difficult to implement in open-world
models, should we so wish. For example, the sentence below says that Alice and
Bob are both smokers, and everyone else is not: $\forall x[((x=a\lor
x=b)\supset S(x))\land((x\neq a\land x\neq b)\supset\neg S(x))].$

###  4.2 Probabilities on formulas

The standard way to represent incomplete knowledge in symbolic logic is
through the use of disjunctions and existential quantifiers. A logical
disjunction expresses qualitative uncertainty, but does not commit to which of
the disjuncts is more likely than the other.

For example, suppose we toss a coin and find that it lands on heads nine out
of ten times. We might believe it is a loaded coin and ascribe a probability
of 0.9 to observing heads on the next toss. Suppose we read three plausible
reports, two of which suggest that Alice is not a smoker, but the third
asserts that she is. We might ascribe a probability of 2/3 to $S(a)$. However,
given that a report about Alice being a smoker might have been more recent or
that Alice accidentally admitted she occasionally smokes, we might treat it as
a true but lesser-known fact and ignore the other reports. Whichever the case
may be, we need a language to reason about the _degrees of belief_ – i.e.,
subjective probabilities – of the agent.

One of the simplest and most widely studied approaches to dealing with degrees
of belief in logic involves the concept of _possible worlds_ , where each
world represents a distinct interpretation of the underlying knowledge base.
We ascribe a probability (or more generally, a weight) to each world. To
determine the probability of a statement $S(a)$, we sum the probabilities of
all worlds where $S(a)$ is true. We may also use weights and consider the
ratio of that sum to a _normalization factor_ , which is simply the sum of the
probabilities of all worlds.

###  4.3 Probabilities on atoms

Suppose $S(a)$ is believed with a probability of 1. What is the probability
that Bob is also a smoker? Given that $S(a)$ is entailed by KB, meaning it
holds in every possible model of the KB, we cannot infer anything about $S(b)$
without further information. The probability of $S(b)$ depends on the possible
worlds. If $S(b)$ is also entailed by the KB, it would be accorded a
probability of 1 as well. However, if some models of the KB did not satisfy
$S(b)$, then the probability of $\neg S(b)$ can be obtained by summing the
probabilities of all such worlds, say this is $r$. The probability of $S(b)$
would then be $(1-r)$. If the worlds all have equal weights, and $S(b)$ is
false in $r$ out of $n$ worlds, then the probability of $S(b)$ would be
$(n-r)/n$. From a logical viewpoint, the extension of the predicate $S$ is
different in every world, and it so happens that $a$ is in every such
extension while $b$ is only in some.

Suppose we were able to assign a probability of 4/5 to Alice and Bob being
smokers based on the deterministic rule mentioned above. This rule can be
represented in the KB as $4/5\colon~{}(S(a)\land F(a,b))\supset S(b)$,
indicating that in 4 out of 5 worlds, Alice influences her friend to smoke. If
all worlds had equal weights, this would mean that Alice influences her friend
to smoke in 4/5 of those worlds.

To further understand the rule, let’s consider two scenarios with different
probabilities for Alice being a smoker. Assume $\\{S(a),F(a,b)\\}$ is believed
with probability 1. The noisy rule states that $S(b)$ is true in four-fifths
of the worlds, giving us a probability of 4/5 for $S(b)$.

Now, let’s assume $F(a,b)$ is believed with probability 1, but $S(a)$ only
with a probability of 2/3. Assuming equal weights to worlds for ease of
explanation, $S(a)$ holds in two-thirds of the worlds, and in four-fifths of
those worlds, the noisy rule requires that $S(b)$ holds. Therefore, the
probability of $S(b)$ is $2/3\times 4/5=8/15.$

Let’s consider a simple example where a unique distribution would not arise.
Suppose $\forall x,y(F(x,y))$ is believed with probability 1. This means that
every individual is a friend of everybody else (including themselves). Now
suppose $S(a)\lor S(c)$ is believed with probability 1. In this case, no
specification is provided for the individual probabilities of $S(a)$ and
$S(c)$. It is not acceptable to assign a probability of 0 to both $S(a)$ and
$S(c)$. However, provided that this requirement is satisfied, there are many
possibilities. Most statistical relational languages will either disallow such
specifications or make assumptions about how the distribution should be
defined for such disjunctions. Notable exceptions allowing for imprecise
probabilities do exist, but they are often not in a first-order formalism. See
[4, 9] for examples that involve first-order logic and multiple distributions.

Suppose we are interested in the probability of $S(b)$ given the noisy smokers
rule above and a similar instance resulting from $S(c)$:
$4/5\colon~{}(S(c)\land F(c,b))\supset S(b)$. It is not hard to see that the
probability of $S(b)$ should be 4/5. This is because every world should
satisfy $S(a)$ or $S(c)$ or both. Due to the universal rule about everyone
being friends, $F(a,c)$ and $F(b,c)$ both hold. Consequently, in four-fifths
of the worlds (assuming equal weights for simplicity), the smokers rule would
imply that $S(b)$ must hold. Therefore, its probability is also 4/5.

Suppose we instead said $S(a)\lor S(c)$ with a probability of 1/5. In one-
fifth of the possible worlds, $S(a)$ or $S(c)$ or both would be true. In four-
fifths of those worlds, the smokers rule would mean that $S(b)$ must be true
as well. Therefore, the probability of $S(b)$ would be $4/5\times 1/5=4/25$
assuming equal weights.

###  4.4 Probabilities on quantified formulas

The examples from the previous section could be expressed more succinctly with
the following noisy quantified rule: $4/5\colon~{}\forall x,y(S(x)\land
F(x,y)\supset S(y)).$ This rule states that in four-fifths of the worlds
(assuming equal weights to worlds), friends of smokers are smokers too.

Suppose we know for certain that Alice and Bob are friends, and we ascribe a
probability of 2/3 to the atom $S(a)$. Then, in two-thirds of the worlds,
$S(a)$ holds. In four-fifths of these worlds, the universal rule, together
with the certain knowledge that $F(a,b)$, means that $S(b)$ holds. The
probability of $S(b)$ is then $2/3\times 4/5=8/15.$

However, one might find it striking how the reading of this rule contrasts
with the statistical observation that motivates it. Presumably, we would like
to say that in 80% of the population, friends of smokers are also smokers.
This is statistical information, of course, but the way the rule is written
requires that the rule must hold in four-fifths of the worlds. In other words,
if in every world there is some smoker whose friend is not a smoker, then the
rule clearly holds in no world, and so its probability should be 0.

This is an early observation in the field of probabilistic logics [1, 2, 3,
23, halpern90an]. The possible-worlds model with probabilities over worlds is
used to define degrees of belief, but it is not appropriate for statistical
information. To represent statistical information, a different formal setup is
required. This setup, called the _random-worlds_ semantics, allows
probabilities to be ascribed to domain elements. With this approach, we can
reason about the probability of a randomly chosen person being a smoker, or
the probability of two randomly chosen individuals being friends, and so on.
This would be useful in cases where the smokers universal is known to not be
true in the real world.

In most statistical relational and neuro-symbolic languages, the intended
statistical interpretation is often ignored, resulting in noisy quantified
rules. These rules are commonly interpreted using the possible-worlds model.

Likewise, probabilities on existentially quantified formulas are to be
understood the same way. For example: $1/5\colon~{}\exists x(S(x))$ says that
in one-fifth of the worlds (assuming equal weights on worlds), there is
someone who is a smoker. It is _not_ saying that one out of five individuals
is a smoker or that there is a 20% chance that a randomly chosen individual is
a smoker.

Both of these topics require a random-worlds semantics, and integrating that
with a possible-worlds semantics could be an exciting area of research for
mainstream statistical relational and neuro-symbolic AI.

###  4.5 What lies beneath

ProbLog, probabilistic databases, Markov logic networks, Bayesian networks,
factor graphs [38], and their neuro-symbolic extensions can be understood
either in terms of a computational framework called _weighted model counting_
[chavira08on, 39], or via a fuzzy semantics that accords non-binary truth
values to atoms. Let us discuss them in turn.

Given a logical language with n propositions, and therefore, $2^{n}$ possible
truth assignments (or worlds) ${\mathcal{M}}=\\{M\\_1,\ldots,M\\_{2^{n}}\\}$,
consider a weight function $\mu$ mapping literals to non-negative reals. By
extension, let the weight of a world $\mu(M)$ be defined as the product of the
weights of the literals that hold at $M$. Then the weighted model count of a
propositional formula $\phi$ is defined as:

$\displaystyle{\it
WMC}(\phi)\doteq{\sum\\_{\\{M\in{\mathcal{M}},~{}M\models\phi\\}}\mu(M)}\bigg{/}{\sum\\_{\\{M\in{\mathcal{M}}\\}}\mu(M)}.$

A Bayesian network can be encoded as a propositional knowledge base, with the
conditional dependences expressed as implications [chavira08on, 39]. In Markov
logic networks, propositions appearing in a clause correspond to the dependent
random variables in an undirected graphical model. The probability function of
the nodes can be further expressed in terms of a weight function of the above
sort. In this case, the probability of some query $q$ (a propositional
formula) is obtained using:

${{\it WMC}({\it KB}\land q)}\bigg{/}{{\it WMC}({\it KB})}.$

In other words, relative to the models of KB, we compute the weights of those
worlds additionally satisfying $q$.

There are a few notable exceptions to picture in the statistical relational
learning community; see [5] for more discussions. For example, in BLOG [35],
function symbols are allowed in the language over an infinite domain, thus
supporting open-world modeling. However, for computational reasons, various
restrictions are placed both for the language and its implementation. Well-
defined programs, for example, restrict the use of logical connectives. When
computing probabilities, it avoids instantiating infinitely many terms by
sampling values for function symbols. When it comes to termination, moreover,
a number of structural conditions need to hold, including one that the
quantifiers over formulas can only range over finite sets.

In a second approach to the semantics of neural learning in logic, binary
truth is relaxed. In particular, because the outputs of neural networks are
real-valued and need to be interpreted logically, there has been a recent
revival of fuzzy logic in neuro-symbolic AI [10, 21, 29]. Fuzzy logic allows
for degrees of truth, unlike classical logic which assumes a proposition is
either true or false. In fuzzy logic, logical connectives are interpreted as
maximum and product operators, where the maximum operator corresponds to
disjunction and the product operator corresponds to conjunction.

When the inputs to these operators are either 0 or 1, it can be seen that
maximum and product collapses to the classcal interpretation. However, when
the inputs are reals between 0 and 1, formulas are evaluated to values that do
not correspond to being strictly true or false. This makes the semantics of
complex formulas hard to interpret. Although there is some research on first-
order fuzzy logic over non-finite domains, much of the work on neuro-symbolic
AI is limited to finite domains [21].

A common argument in favor of fuzzy logic is that it may be more feasible
computationally [41, 42], although this comes at the cost of a hard-to-
interpret semantics. However, recent discussions have raised the point that
neural networks are trained for conditional probabilities, making it
reasonable to interpret the output in a probabilistic manner [41, 42]. As a
result, it may be more appropriate to consider using probabilistic (logical)
languages. Striking a balance between ease of computation and interpretability
of semantics is crucial, and perhaps the final decision may depend on the
specific application.

##  5 Outlook and conclusions

In the above sections, we discussed the importance of reasoning using the
friends-smokers example. We also explored the advantages of first-order logic
in terms of the expressiveness it offers for open-world scenarios. However, we
did not delve into the strategies that allow us to reason in a decidable way
with fragments of first-order logic. Fortunately, there are numerous methods
available to tackle this issue, and we shall mention them in brief.

As is well known, areas such as descriptive complexity and finite model theory
demonstrate that logic could serve as an abstraction for understanding
computation vis-à-vis representation. It is also well-known that there is a
trade-off between the expressiveness of the language and the tractability of
the reasoning task.

On the one hand it is widely acknowledged that relations and more generally,
the expressiveness of first-order logic is extremely useful for capturing
language and concepts required for common-sense reasoning. This explains the
use of first-order logic in database theory, statistical relational learning,
automated planning, knowledge representation (e.g., description logics, and
reasoning about actions) and verification. But at the same time the entailment
problem is only semi-decidable.

Since Goedel’s celebrated results on the intractability of logic, there has
been a wide range of approaches to deal with this trade-off especially in the
knowledge representation, symbolic computation and the statistical relational
learning communities. They can be broadly classified into three categories.
The easiest approach is to simply restrict the language to propositional
logic. Popular frameworks in statistical relational learning such as Markov
logic networks and relational Bayesian networks as well as relational
probabilistic planning languages take this approach [16, 26, 31, 43].

A second approach is to limit the expressiveness of the language in terms of
the size of the predicates (but not necessarily in terms of the size of the
domain). Description logics are popular concept representation languages in
knowledge representation that only allow predicates with utmost two arguments.
When the domain is further assumed to be finite, certain other computational
tasks beyond entailment turn out to be tractable [12, 27].

A third approach, stemming from the knowledge representation community but
less popular in other communities, is to limit reasoning. Here, the entailment
relation itself is weakened by way of introducing non-traditional semantics
[30]. However, because the semantical apparatus is often complicated, they
have understandably not been pursued by or used in other communities.

Somewhere in between the three approaches, there is also the well-known Horn
clause restriction, seen most prominently in logic programming and its
derivatives. It potentially allows for an infinite Herbrand universe, but the
grounding of the program is only performed against a ground query, which
permits a propositional encoding sometimes [19]. This enables the use of logic
programming in a wide variety of applications, including statistical
relational learning by decorating clauses further with probabilities. But most
commong flavors are limited in only allowing one positive literal in a clause,
and is non-monotonic.

A fourth approach is to exploit symmetries in first-order representations. For
example, the area of lifted reasoning avoids grounding a probabilistic
relational model by appealing to first-order resolution [40]. In somewhat a
similar spirit, a recent result shows that there is an intuitive fragment of
first-order disjunctive knowledge, for which reasoning is decidable and can be
reduced to propositional satisfiability [8]. Knowledge bases in this fragment
essentially amounts to universally quantified first-order clauses, but without
arity restrictions and without restrictions on the appearance of negation.
Queries, however, are expected to be ground formulas. They achieve this result
is by showing how the entailment over infinitely many infinite-sized
structures can be reduced to a search over finitely many finite-size
structures. The crux of the argument lies in showing that names not mentioned
in the knowledge base and the query behave identically (in a suitable formal
sense).

Overall, if we are extend the statistical relational and neuro-symbolic
formalisms to full first-order expressiveness, some of these strategies need
to be combined and extended carefully, and this is an exciting area for future
research.

## References

  * [1] M. Abadi and J. Halpern.  Decidability and expressiveness for first-order logics of probability.  Information and Computation, 112(1):1 – 36, 1994.
  * [2] F. Bacchus, A. Grove, J. Y. Halpern, and D. Koller.  Forming beliefs about a changing world.  In Proc. AAAI, pages 222–229, 1994.
  * [3] F. Bacchus, A. J. Grove, J. Y. Halpern, and D. Koller.  From statistical knowledge bases to degrees of belief.  Artificial intelligence, 87(1-2):75–143, 1996.
  * [4] F. Bacchus, J. Y. Halpern, and H. J. Levesque.  Reasoning about noisy sensors and effectors in the situation calculus.  Artificial Intelligence, 111(1–2):171 – 208, 1999.
  * [5] V. Belle.  Open-universe weighted model counting.  In AAAI, 2017.
  * [6] V. Belle.  Logic meets learning: From aristotle to neural networks.  In Neuro-Symbolic Artificial Intelligence: The State of the Art, pages 78–102. IOS Press, 2021.
  * [7] V. Belle.  Toward robots that reason: Logic, probability & causal laws.  2023\.
  * [8] V. Belle and B. Juba.  Implicitly learning to reason in first-order logic.  Advances in Neural Information Processing Systems, 32, 2019.
  * [9] V. Belle and G. Lakemeyer.  Reasoning about probabilities in unbounded first-order dynamical domains.  In IJCAI, 2017.
  * [10] T. R. Besold, A. d. Garcez, S. Bader, H. Bowman, P. Domingos, P. Hitzler, K.-U. Kühnberger, L. C. Lamb, D. Lowd, P. M. V. Lima, et al.  Neural-symbolic learning and reasoning: A survey and interpretation.  arXiv preprint arXiv:1711.03902, 2017.
  * [11] E. Boerger, E. Grädel, and Y. Gurevich.  The classical decision problem.  Springer Verlag, 1997.
  * [12] D. Calvanese, G. De Giacomo, D. Lembo, M. Lenzerini, and R. Rosati.  Data complexity of query answering in description logics.  Artificial Intelligence, 195:335–360, 2013.
  * [13] R. Carnap.  Logical foundations of probability.  Routledge and Kegan Paul London, 1951.
  * [14] M. Chavira and A. Darwiche.  On probabilistic inference by weighted model counting.  Artif. Intell., 172(6-7):772–799, 2008.
  * [15] H. Chockler, D. Kroening, and Y. Sun.  Explanations for occluded images.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1234–1243, 2021.
  * [16] J. Corander, T. Janhunen, J. Rintanen, H. J. Nyman, and J. Pensar.  Learning chordal markov networks by constraint satisfaction.  In Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States., pages 1349–1357, 2013.
  * [17] A. d’Avila Garcez and L. C. Lamb.  Neurosymbolic ai: The 3rd wave.  arXiv e-prints, pages arXiv–2012, 2020.
  * [18] K. Ellis, A. Solar-Lezama, and J. Tenenbaum.  Sampling for bayesian program learning.  In NIPS, pages 1289–1297, 2016.
  * [19] D. Fierens, G. V. den Broeck, I. Thon, B. Gutmann, and L. D. Raedt.  Inference in probabilistic logic programs using weighted CNF’s.  In UAI, pages 211–220, 2011.
  * [20] H. Gaifman.  Concerning measures in first order calculi.  Israel Journal of Mathematics, 2(1):1–18, 1964.
  * [21] A. S. d. Garcez, K. Broda, D. M. Gabbay, et al.  Neural-symbolic learning systems: foundations and applications.  Springer Science & Business Media, 2002.
  * [22] J. Halpern.  An analysis of first-order logics of probability.  Artificial Intelligence, 46(3):311–350, 1990.
  * [23] J. Y. Halpern.  Reasoning about Uncertainty.  MIT Press, 2003.
  * [24] P. Hitzler.  Neuro-symbolic artificial intelligence: The state of the art.  2022\.
  * [25] S. Kambhampati.  Challenges of human-aware ai systems.  AI Magazine, 41(3), 2020.
  * [26] S. Kok and P. Domingos.  Learning markov logic networks using structural motifs.  In ICML, pages 551–558, 2010.
  * [27] D. Koller, A. Levy, and A. Pfeffer.  P-classic: a tractable probablistic description logic.  In Proc. AAAI / IAAI, pages 390–397, 1997.
  * [28] B. M. Lake, R. Salakhutdinov, and J. B. Tenenbaum.  Human-level concept learning through probabilistic program induction.  Science, 350(6266):1332–1338, 2015.
  * [29] L. C. Lamb, A. Garcez, M. Gori, M. Prates, P. Avelar, and M. Vardi.  Graph neural networks meet neural-symbolic computing: A survey and perspective.  arXiv preprint arXiv:2003.00330, 2020.
  * [30] Y. Liu and H. J. Levesque.  Tractable reasoning in first-order knowledge bases with disjunctive information.  In Proc. AAAI, pages 639–644, 2005.
  * [31] D. Lowd and A. Rooshenas.  Learning markov networks with arithmetic circuits.  In AISTATS, 2013.
  * [32] R. Manhaeve, S. Dumancic, A. Kimmig, T. Demeester, and L. De Raedt.  Deepproblog: Neural probabilistic logic programming.  Advances in Neural Information Processing Systems, 31, 2018.
  * [33] J. McCarthy.  Programs with common sense.  In Semantic Information Processing, pages 403–418. MIT Press, 1968\.
  * [34] J. McCarthy.  Situations, Actions and Causal Laws.  Technical report, Stanford University, 1963. Also in M. Minsky (ed.), Semantic Information Processing, MIT Press, Cambridge, MA, 1968.
  * [35] B. Milch, B. Marthi, S. J. Russell, D. Sontag, D. L. Ong, and A. Kolobov.  BLOG: Probabilistic models with unknown objects.  In Proc. IJCAI, pages 1352–1359, 2005.
  * [36] L. Morgenstern and S. A. McIlraith.  John McCarthy’s legacy.  Artificial Intelligence, 175(1):1 – 24, 2011.
  * [37] J. Pearl and D. Mackenzie.  The book of why.
  * [38] L. D. Raedt, K. Kersting, S. Natarajan, and D. Poole.  Statistical relational artificial intelligence: Logic, probability, and computation.  Synthesis Lectures on Artificial Intelligence and Machine Learning, 10(2):1–189, 2016.
  * [39] T. Sang, P. Beame, and H. A. Kautz.  Performing bayesian inference by weighted model counting.  In AAAI, pages 475–482, 2005.
  * [40] G. Van den Broeck.  Lifted Inference and Learning in Statistical Relational Models.  PhD thesis, KU Leuven, 2013.
  * [41] E. van Krieken, E. Acar, and F. van Harmelen.  Analyzing differentiable fuzzy implications.  arXiv preprint arXiv:2006.03472, 2020.
  * [42] E. van Krieken, E. Acar, and F. van Harmelen.  Analyzing differentiable fuzzy logic operators.  Artificial Intelligence, 302:103602, 2022.
  * [43] H. Younes and M. Littman.  PPDDL 1. 0: An extension to pddl for expressing planning domains with probabilistic effects.  Technical report, Carnegie Mellon University, 2004.


# SQUIRL: Robust and Efficient Learning from Video Demonstration of Long-
Horizon Robotic Manipulation Tasks

Bohan Wu, Feng Xu, Zhanpeng He, Abhi Gupta, and Peter K. Allen This work is
supported by NSF Grant CMMI-1734557. Authors are with Columbia University
Robotics Group, New York, NY, 10027, USA

###### Abstract

Recent advances in deep reinforcement learning (RL) have demonstrated its
potential to learn complex robotic manipulation tasks. However, RL still
requires the robot to collect a large amount of real-world experience. To
address this problem, recent works have proposed learning from expert
demonstrations (LfD), particularly via inverse reinforcement learning (IRL),
given its ability to achieve robust performance with only a small number of
expert demonstrations. Nevertheless, deploying IRL on real robots is still
challenging due to the large number of robot experiences it requires. This
paper aims to address this scalability challenge with a robust, sample-
efficient, and general meta-IRL algorithm, SQUIRL, that performs a new but
related long-horizon task robustly given only a single video demonstration.
First, this algorithm bootstraps the learning of a task encoder and a task-
conditioned policy using behavioral cloning (BC). It then collects real-robot
experiences and bypasses reward learning by directly recovering a Q-function
from the combined robot and expert trajectories. Next, this algorithm uses the
Q-function to re-evaluate all cumulative experiences collected by the robot to
improve the policy quickly. In the end, the policy performs more robustly
(90%+ success) than BC on new tasks while requiring no trial-and-errors at
test time. Finally, our real-robot and simulated experiments demonstrate our
algorithm’s generality across different state spaces, action spaces, and
vision-based manipulation tasks, e.g., pick-pour-place and pick-carry-drop.

##  I Introduction

We aspire robots to become generalists who acquire new complex skills robustly
and quickly. The robotic system, whether planned or learned, needs to leverage
its existing knowledge to solve a new but related task in an efficient yet
high-performance manner. Thanks to recent advances in machine learning and
sim-to-real transfer mechanisms, short-horizon robotic manipulation such as
grasping has improved in performance. However, many real-world robotic
manipulation tasks are long-horizon, diverse, and abundant in volume. In the
absence of a scalable and systematic way to construct simulation environments
for a large number of tasks, the robot needs to learn a new task directly in
the physical world from only a handful of trials, due to the high cost of
collecting real-robot trial-and-errors and experiences.

Figure 1: Learning from a single video demonstration of a long-horizon
manipulation task via Soft Q-functioned Meta-IRL (SQUIRL). In the pick-pour-
place example above, the robot needs to approach, pick-up and carry the grey
bottle, pour the iron pebble inside the bottle into a specific container, and
finally place the bottle back on the table. During training, the robot is
given a single video demonstration for each of the 117 training tasks. After
learning from these 117 videos, the robot also practices 90 trial-and-errors
in total on these tasks. From such combined expert and robot trajectories, the
robot learns the general skills of pouring robustly. At test time, given a
single video demonstration of pouring into a new, unseen red container at a
new position, the robot successfully replicates this new task without the need
for any trial-and-errors.

We observe that real-world robotic skill acquisition can become more sample-
efficient in several important ways. First, we notice that humans learn tasks
quickly by watching others perform similar tasks. Among many forms of task
representations such as rewards, goal images, and language instructions, human
demonstrations guide exploration effectively and can lead to significant
sample efficiency gains. Furthermore, learning from video demonstrations
sidesteps hand-designing a proper reward function for every new task. In the
case of a vision-based task, video demonstrations also conveniently provide
the same pixel state space for the robot.

In learning from demonstrations (LfD), the robot should be sample-efficient in
two dimensions – it should use as few expert demonstrations (“demonstrations”
hereafter) as possible and take as few trial-and-errors (practices) as
possible on its own to learn a robust policy. Among LfD methods, behavioral
cloning (“BC” hereafter) is sample-efficient but susceptible to compounding
errors. Here, compounding errors refer to the problem in which every time a
behavioral-cloned robot makes a small error, it makes a larger error down the
road as it drifts away from the expert state distribution. In contrast, IRL
alleviates compounding errors by allowing the robot to try the tasks out in
the real world and measure its behavior against the expert. However, due to
the need to learn a reward function, IRL can require many trial-and-errors in
the real world, while BC does not require such robot experiences. We posit
that leveraging off-policy experiences of trial-and-errors is essential to
making IRL sample-efficient enough for real robots. Here, “off-policy
experiences” refer to the cumulative experiences that the robot has collected
thus far during training. In contrast, “on-policy experiences” are the most
recent experiences that the robot has collected using its current policy.
Humans leverage lifelong, cumulative experiences to learn quickly at present.
We envision robots to acquire new skills more quickly by learning from off-
policy (i.e., cumulative) experiences.

Finally, many real-world tasks are related and share structures and knowledge
that can be exploited to solve a new but similar task later. For example,
humans can quickly learn to pick and place a new object after learning to pick
and place many known objects. Meta-learning, explicitly utilizing this
property, aims to learn a new but related task quickly if it has already
learned several similar tasks in the past.

Figure 2: Fig.2: Pick-Pour-Place Robot Setup at Test Time. Given an RGB image
from the top-down (black) or 45°camera (also black), the UR5-Seed robot is
tasked to approach and pick-up the grey cylindrical bottle, pour the iron
pebble already inside the bottle into a specific container on the table and
finally place the bottle back on the table.

With these motivations, we introduce SQUIRL, a meta-IRL algorithm that learns
long-horizon tasks quickly and robustly by learning from 1) video
demonstrations, 2) off-policy robot experiences, and 3) a set of related
tasks. Fig.1 explains this algorithm using the example of a set of long-
horizon pick-pour-place tasks, using the UR5-Seed111Site:
www.seedrobotics.com/rh8d-dexterous-hand.html robot setup shown in Fig.2. In
this task, we have the containers (green, yellow, orange, and red), a
cylindrical bottle (grey), and an iron pebble inside the bottle. The robot
needs to first approach and pick-up the grey bottle, pour the iron pebble
inside the bottle into a specific container on the table, and then finally
place the bottle back on the table, as shown in each row of images in Fig.1.
At the beginning of each task, the bottle is not yet in hand, but the iron
pebble is already in the bottle. At training time, the robot is given a single
video demonstration for each of the 117 pick-pour-place tasks, as shown in the
first two rows of images in Fig.1. Every new combination of container
positions represents a different pick-pour-place task. Furthermore, the robot
only needs to pour into one of the containers in a single task. Therefore,
pouring into different containers also represents different tasks. After
learning from these 117 demonstrations, the robot also practices 90 trial-and-
errors on these tasks in total. From such a combination of expert and robot
trajectories, the robot learns the general skills of pick-pour-place robustly.
In all 117 training tasks, only two of the four containers appear on the
table: the green and yellow containers, as shown in the first two rows of
images in Fig.1. The orange and red containers are excluded during training
and only appear at test time, as shown in the last row of images in Fig.1. We
do so to evaluate our algorithm’s generalizability to unseen containers at
test time. As shown in the last row of images in Fig.1, the robot successfully
pours into a new container (red) at test time, at a new position never seen
before during training, without the need for any trials or practices.

To achieve such fast generalization to new tasks, our algorithm learns a task
encoder network and a task-conditioned policy. The task encoder generates a
32-dimensional task embedding vector that encodes task-specific information.
The policy network then learns to generalize to new tasks by accepting this
task embedding vector as input, thus becoming “task-conditioned”. During
training, our algorithm first bootstraps learning by training both the task
encoder and the policy jointly via the BC loss. The robot then collects 10
trials across 10 tasks using the warmed-up policy and the task encoder. Next,
using the combined experiences of the expert and the robot, our algorithm
bypasses reward learning by directly learning a task-conditioned Q-function.
Using this Q-function, our algorithm then reuses and re-evaluates all
cumulative experiences of the robot to improve the policy quickly. This cycle
repeats until the $90^{th}$ trial. Finally, at test time, the task encoder
generates a new task embedding from a single video demonstration of a new
task. This embedding is then inputted into the task-conditioned policy to
solve the new task without any trial-and-errors and yet in a high-performance
manner. In summary, our contributions are:

  1. 1.

A robust meta-IRL algorithm that outperforms ($90\%$\+ success) its behavioral
cloning counterpart in real-robot and simulated vision-based long-horizon
manipulation;

  2. 2.

A novel Q-functioned IRL formulation that circumvents reward learning and
improves IRL sample efficiency;

  3. 3.

An efficient method that leverages off-policy robot experiences for training
and requires no trials at test time;

  4. 4.

A general approach that tackles various long-horizon robotic manipulation
tasks and works with both vision and non-vision observations and different
action spaces.

##  II Related Work

###  II-A Inverse Reinforcement Learning (IRL) and Meta-IRL

Inverse reinforcement learning (IRL) models another agent’s (typically the
expert’s) reward function, given its policy or observed behavior. Previous
works have approached the IRL problem with maximum margin methods [1][2] and
maximum entropy methods [3][4][5]. In particular, maximum entropy methods
recover a distribution of trajectories that have maximum entropy among all
distributions and match the demonstrated policy’s behaviors. While these
methods have shown promising results in continuous control problems, they
suffer from low sample efficiency due to the need for evaluating the robot’s
policy, which can be alleviated by meta-learning (i.e., meta-IRL). SMILe [6]
and PEMIRL [7] are two meta-IRL algorithms based on AIRL [8] that leverage a
distribution of tasks to learn a continuous task-embedding space to encode
task information and achieve fast adaptation to a new but similar task. Our
work differs from [6][7] in four crucial ways. First, our meta-IRL algorithm
works with real robots and image observations. Second, instead of a reward
function, we directly model a Q-function that the policy can optimize, in
order to increase IRL sample efficiency. Third, we train the task encoder with
the behavioral cloning (BC) gradient as opposed to the IRL gradient for
stabler and more efficient learning. Lastly, we bootstrap policy and task
encoder learning using BC before training via meta-IRL.

###  II-B Real-robot Learning from Demonstrations (LfD)

Our work is related to real-robot LfD [9], such as [10][11][12]. In
particular, [13] developed IRL on real robots without learning from raw
pixels. Other works (e.g., [14][15][16][17]) used BC for real-robot LfD.
Another work [18] developed goal-conditioned BC on a simulated robot to learn
long-horizon tasks by playing with objects in the scene. While enjoying
efficient learning by casting imitation learning into a supervised learning
problem, BC suffers from the covariate shift between the train and test data.
In comparison, IRL achieves robust performance by modeling the state-action
joint distribution instead of the conditional action distribution in BC [19].
Different from previous works, our meta-IRL algorithm works on real-robot
vision-based tasks, and its Q-functioned IRL policy gradient can be directly
combined with the BC gradient signal to approach both the sample efficiency of
BC and the robustness of IRL.

###  II-C One-shot Meta-imitation Learning on Real Robots

Our algorithm attempts to enable robots to quickly and robustly imitate a
single unseen video demonstration by learning from a distribution of tasks
with shared structure, i.e., one-shot robot meta-imitation learning. For
example, [20] combines gradient-based meta-learning and BC on a real robot to
learn quickly from video demonstrations. [21] then extends [20] to enable
robots to learn from human-arm demonstrations directly. [22] then improves
[21] to meta-imitation-learn multi-stage real-robot visuomotor tasks in a
hierarchical manner. However, constrained by the covariate shift problem of
BC, these works show limited task performance (e.g., around $50\%$ success
rate for the training tasks). In contrast, our algorithm learns a vision-based
manipulation task robustly ($90\%+$ success rates) and efficiently (117
videos, 90 trials) by utilizing the generalization ability of task embeddings
[23] and a novel Q-functioned IRL formulation.

##  III Preliminaries

###  III-A Off-policy Reinforcement Learning via Soft Actor-Critic

Standard RL models a task $\mathcal{M}$ as an MDP defined by a state space
$\mathcal{S}$, an initial state distribution $\rho_{0}\in\Pi(\mathcal{S})$, an
action space $\mathcal{A}$, a reward function
$\mathcal{R}:\mathcal{S}\times\mathcal{A}\to\mathbb{R}$, a dynamics model
$\mathcal{T}:\mathcal{S}\times\mathcal{A}\to\Pi(\mathcal{S})$, a discount
factor $\gamma\in[0,1)$, and a horizon $H$. Here, $\Pi(\cdot)$ defines a
probability distribution over a set. The robot acts according to stochastic
policy $\pi:\mathcal{S}\to\Pi(\mathcal{A})$, which specifies action
probabilities for each $s$. Each policy $\pi$ has a corresponding
$Q^{\pi}:\mathcal{S}\times\mathcal{A}\to\mathbb{R}$ function that defines the
expected discounted cumulative reward for taking an action $a$ from $s$ and
following $\pi$ onward.

Off-policy RL, particularly Soft Actor-Critic (SAC) [24], reuses historical
experiences to improve learning sample efficiency by recovering a “soft”
Q-function estimator $Q_{\theta}$. A policy can then be learned by minimizing
the KL divergence between the policy distribution and the exponential-Q
distribution: $\pi^{*}=\operatorname*{arg\,min}_{\pi\in\Pi}D_{KL}(\pi(a\mid
s)\;\|\;\frac{\exp(Q^{\pi_{old}}_{\theta}(s,a))}{Z(s)})$

###  III-B Timestep-centric IRL as Adversarial Imitation Learning

The purpose of IRL is to learn the energy function $f_{\theta}$ implicit in
the provided expert demonstrations and use $f_{\theta}$ to learn a policy that
robustly matches the expert performance. In particular, timestep-centric IRL
aims to recover an energy function $f_{\theta}(s,a)$ to rationalize and match
the demonstrated expert’s action conditional distribution: $p_{\theta}(a\mid
s)=\frac{\exp(f_{\theta}(s,a))}{Z_{\theta}}\propto\exp(f_{\theta}(s,a))=\overline{p_{\theta}}(a\mid
s)$, where $Z_{\theta}$ is the partition function, an integral over all
possible actions given state $s$. In other words, IRL minimizes the KL
divergence between the actual and predicted expert conditional action
distributions: $\pi_{E}(a\mid s)$ and $p_{\theta}(a\mid s)$.

Adversarial IRL [8][25] provides a sampling-based approximation to MatEntIRL
[4] in an adversarial manner. Specifically, AIRL [8] learns a generative
policy $\pi_{\psi}$ and a binary discriminator $D_{\theta}$ derived from
energy function $f_{\theta}$:

$\displaystyle D_{\theta}(s,a)=P((s,a)\text{ is generated by expert})$
$\displaystyle=\frac{\overline{p_{\theta}}(a\mid
s)}{\overline{p_{\theta}}(a\mid s)+\pi_{\psi}(a\mid
s)}=\frac{\exp(f_{\theta}(s,a))}{\exp(f_{\theta}(s,a))+\pi_{\psi}(a\mid s)}$
(1)

and $\theta$ is trained to distinguish state-action pairs sampled from the
expert vs. the policy, using binary cross entropy loss:

$\displaystyle\mathcal{L}^{IRL}=-\mathbb{E}$
${}_{(s,a)\sim\pi_{\psi},\pi_{E}}[y(s,a)\log(D_{\theta}(s,a))$
$\displaystyle+(1-y(s,a))\log(1-D_{\theta}(s,a))]$ (2)

where $y(s,a)=\mathds{1}\\{(s,a)\text{ is generated by expert }\pi_{E}\\}$.

Meanwhile, the policy is trained to maximize the MaxEntIRL Objective [4], or
equivalently, to match the expert’s state-action joint distribution via
reverse KL divergence [19].

###  III-C One-shot Meta-imitation Learning from A Single Video

In one-shot meta-imitation learning, the robot is trained to solve a large
number of tasks drawn from a task distribution $p(\mathcal{M})$. The total
number of tasks in this task distribution can be finite or infinite. Each
imitation task $\mathcal{M}_{train}^{i}$ consists of a single video
demonstration $\mathcal{D}^{i}_{\pi_{E}}$. During training, the robot can also
generate limited practice trajectories (e.g., 90). For example, in the Pick-
Pour-Place experiment in Fig.1, the robot receives a single video
demonstration for each of the 117 tasks. Each task is characterized by a
different combination of container positions, or pouring into the green vs.
the yellow container. At test time, the robot receives a single video of a new
task $\mathcal{M}_{test}^{i}$ drawn from $p(\mathcal{M})$. For example, a new
Pick-Pour-Place test task can be a new combination of container positions or
pouring into a new container (e.g., the red or orange container). The robot
then needs to solve this task the first time without trial-and-error.

###  III-D Embedding-based Meta-learning

Embedding-based meta-learning [7][23] learns a task-specific embedding vector
$z$ that contains task-level abstraction to adapt to a new but related task
quickly. This method aims to learn a task-conditioned policy $\pi(a|s,z)$ that
maximizes task-conditioned expected returns:
$\max_{\pi}\mathbb{E}_{(s_{t},a_{t})\sim\pi,\rho_{0}}[\sum_{t=1}^{T}r(s_{t},a_{t}|c)+\alpha\mathcal{H}(\pi(a_{t}|s_{t},c))]$,
by learning an embedding space $Z$ that maximizes the mutual information
between $z$ and task context $c$. The goal is to make this learned embedding
space generalizable to new tasks so that at test time, the policy can quickly
adapt to unseen tasks with no or few practices. A key advantage of embedding-
based meta-learning is the ability to learn from off-policy experiences.
However, current methods are mostly if not only demonstrated in non-vision
tasks in simulation.

##  IV Mathematical Formulation for SQUIRL

###  IV-A SQUIRL: Timestep-centric IRL as Soft Q-Learning

Previous works in timestep-centric IRL such as [6][7][8] have interpreted the
energy function $f_{\theta}$ in Eq.III-B as a reward function $r_{\theta}$ and
later recover a Q or advantage function based on reward $r_{\theta}$ for
policy improvement. To improve IRL sample efficiency, we propose to bypass
this reward learning and directly interpret $f_{\theta}(s,a)$ as the soft
Q-function [24] $Q^{\pi_{mix}}_{\theta}(s,a)$. This soft Q-function models the
expert’s behavior as maximizing both the Q-value and its entropy (i.e.,
randomness) simultaneously. It also encourages the robot to explore the real
world to imitate the expert more robustly. Under this formulation,
approximating the expert’s conditional action distribution is equivalent to
recovering a soft Q-function under which the expert is soft Q-optimal:

$\displaystyle\operatorname*{arg\,min}_{\theta}D_{KL}(\pi_{E}(a\mid
s)\;\|\;p_{\theta}(a\mid s))$ $\displaystyle=$
$\displaystyle\operatorname*{arg\,max}_{\theta}\mathbb{E}_{a\sim\pi_{E}(a\mid
s)}[Q^{\pi_{mix}}_{\theta}(s,a)]-\log Z_{\theta}$ (3)

Eq.3 rationalizes the expert behavior intuitively because the expert should be
optimal with respect to the cumulative reward [3], not the immediate reward.
Here, $Q^{\pi_{mix}}_{\theta}$ is under a mixture policy $\pi_{mix}$ between
the robot and expert’s policies.

###  IV-B SQUIRL as Expert Imitation and Adversarial Learning

Under SQUIRL, the policy learning objective (Eq.4) is also equivalent
(derivations on website) to matching: 1) the exponential-Q distribution of the
discriminator $\theta$ (Eq.5), 2) the generator’s objective in Generative
Adversarial Networks (GANs) [26] (Eq.6), and 3) the joint state-action
distribution of expert [19] (Eq.7):
$\pi^{*}=\operatorname*{arg\,min}_{\pi\in\Pi}\mathcal{L}^{RL}(\pi)$, where

$\displaystyle\mathcal{L}^{RL}(\pi)=D_{KL}(\pi_{\psi}(a\mid
s)\;\|\;\frac{\exp{Q^{\pi_{mix}}_{\theta}(s,a)}}{Z(s)})$ (4)
$\displaystyle=D_{KL}(\pi_{\psi}(a\mid s)\;\|\;p_{\theta}(a\mid s))$ (5)
$\displaystyle=\mathbb{E}_{(s,a)\sim\pi_{mix}}[\log(1-D_{\theta}(s,a))-\log(D_{\theta}(s,a))]$
(6) $\displaystyle=D_{KL}(\rho_{\pi_{\psi}}(s,a)\;\|\;\rho_{\pi_{E}}(s,a))$
(7)

Meanwhile, the discriminator $\theta$ is matching its Q-function to the log-
distribution of the expert’s conditional action distribution (Section III-B).
Therefore, when this Q-function is optimal:
$Q^{\pi_{mix}}_{\theta}=Q^{\pi_{mix}}_{\theta^{*}}$, the robot’s policy
objective (Eq.4) is also matching the expert’s conditional action
distribution:

$\psi^{*}=\operatorname*{arg\,min}_{\psi}E_{\rho_{\pi_{mix}}(s)}[D_{KL}(\pi_{\psi}(a\mid
s)\;\|\;\pi_{E}(a\mid s))]$ (8)

###  IV-C Comparison to the Behavioral Cloning (BC) Objective

While BC attempts to learn a policy that also matches the expert’s conditional
action distribution [19], the fundamental difference is that the KL-divergence
in BC’s case is computed under the expert’s narrow state distribution
$\rho_{\pi_{E}}(s)$:
$\psi_{BC}^{*}=\operatorname*{arg\,min}_{\psi}E_{\rho_{\pi_{E}}(s)}[D_{KL}(\pi_{E}(a\mid
s)\;\|\;\pi_{\psi}(a\mid s))]$. In contrast, ours (Eq.8) is computed under
$\rho_{\pi_{mix}}(s)$: the state distribution of the combined cumulative
experience of the robot and the expert, which is a much wider distribution
than the expert distribution. We hypothesize that this, along with matching
the joint state-action distribution of the expert (Eq.7), makes our algorithm
less susceptible to compounding errors than BC, as experimentally tested in
Section VI.

Figure 3: SQUIRL: Soft Q-functioned Meta-IRL. To begin, our algorithm
bootstraps learning for the policy (orange) and the task encoder (yellow) via
behavioral cloning (the left third of Fig.3). Next, our algorithm uses the
warmed-up policy and task encoder to generate 10 trials in the physical world
(not in simulation). Using the combined expert and robot trajectories, our
algorithm learns a task-conditioned soft Q-function (green) that rationalizes
the expert’s behaviors as maximizing both cumulative reward and entropy (i.e.,
randomness). Using this Q-function, our algorithm then quickly improves the
policy using all cumulative robot and expert timesteps. This cycle repeats
until convergence, totaling 90 trials (the middle third of Fig.3). Finally, at
test time (the right third Fig.3), our algorithm generates a new embedding $z$
for the new task, and inputs this embedding into the task-conditioned policy
to solve the new task without any practices.

##  V SQUIRL: Soft Q-functioned Meta-IRL

Shown in Fig.3, our algorithm learns three neural networks jointly – a task
encoder (yellow), a task-conditioned policy (orange), and a task-conditioned
soft Q-function (green):

  1. 1.

$\Psi_{\phi}(c)$: a task encoder that encodes a sampled batch of $C=64$ expert
state-action pairs $c=\\{s^{i}_{1:C},a^{i}_{1:C}\\}$ from a task $i$ into a
single 32-dim embedding vector $z^{i}\in\mathbb{R}^{32}$ (by computing the
mean vector across 64 embeddings) that enables generalization to new tasks.
This batch of expert state-action pairs is randomly sampled and thus does not
encode time information. Both the policy and the Q-function accept this
embedding vector as input.

  2. 2.

$\pi_{\psi}(s,z^{i})$: a task-conditioned policy the robot uses to perform a
task $i$ given state $s$ and the task embedding vector
$z^{i}\in\mathbb{R}^{32}$ outputted by the task encoder $\Psi_{\phi}(c)$.

  3. 3.

$Q_{\theta}(s,a,z^{i})$: a task-conditioned soft Q-function used to train the
policy $\pi_{\psi}(s,z^{i})$ to more robustly mimic the expert’s behavior for
the robotic manipulation task $i$.

To begin, the robot is given an expert trajectory of state-action pairs
$\mathcal{D}_{\pi_{E}}$ for each of the 117 training tasks. The robot first
uses these expert trajectories to bootstrap training for both its policy
$\pi_{\psi}$, and the task encoder $\Psi_{\phi}$ via behavioral cloning
(Eq.9). This way, the robot can distinguish the train tasks better and learn
more quickly in the real world. Next, the robot generates 10 trials (state-
action pairs) $\overline{\mathcal{D}}_{\pi_{\psi}}$ in the physical world (not
simulation) using its warmed-up policy and task encoder. Then, the robot uses
both the expert’s and its state-action pairs to train a discriminator
$\theta$. This discriminator classifies which state-action pairs come from the
expert $\pi_{E}$ vs. the robot $\pi_{\psi}$. At first, the robot is
distinctively worse than the expert at performing the tasks. This makes it
easy for the discriminator to classify. By doing so, the discriminator learns
a Q-function $Q^{\pi_{mix}}_{\theta}$ using Eq.3.

Using the learned Q-function $Q^{\pi_{mix}}_{\theta}$, the robot trains its
policy $\pi_{\psi}$ via Eq.4. Meanwhile, the robot also has the option to
continue updating its task-conditioned policy and task encoder via behavioral
cloning (Eq.9). Since training the policy via Eq.4 is equivalent to indirectly
imitating the expert (Eq.7 and 8), as derived in Section IV-B, the
trajectories generated by the policy gradually become more similar to the
expert. This makes the state-action pairs more difficult for the discriminator
to classify. This difficulty, in turn, forces the discriminator to learn a
more precise Q-function, which then encourages the policy to mimic the expert
even more closely. This cycle repeats until convergence (90 trials in total),
at which point: 1) the policy matches the expert performance, 2) the task
encoder learns to generalize to new tasks, and 3) the discriminator continues
to struggle to distinguish state-action pairs correctly despite having learned
an accurate Q-function.

###  V-A Rationale for Bypassing Reward Learning via SQUIRL

SQUIRL learns a Q-function without rewards because 1) the policy is ultimately
trained by the Q-function, not rewards, thus bypassing reward learning
improves IRL sample efficiency, and 2) circumventing reward learning avoids
off-policy Q-learning from a constantly changing reward function and makes
training easier and more stable empirically.

###  V-B Architectures for Policy, Task Encoder, and Q-function

For all non-vision tasks, we parameterize $\pi_{\psi},\Psi_{\phi},Q_{\theta}$
with five fully-connected (FC) layers. For vision tasks, we use a 5-layer CNN
followed by a spatial-softmax activation layer for the RGB image. This
activation vector is then concatenated with the non-vision input vector and
together passed through five FC layers. Our algorithm is general and works
with many other network architectures, state, and action spaces.

###  V-C Incorporating BC to Bootstrap and Accelerate Learning

Since our algorithm’s IRL objective (Eq.8) is compatible with BC, as explained
in Section IV-C, our algorithm can jointly be trained with BC to stabilize and
accelerate learning without conflicting gradient issues (line 16 in Algorithm
1):

$\mathcal{L}^{BC}=\mathbb{E}_{(s,a)\sim\pi_{E}}[\left\lVert\pi_{\psi}(s,\Psi_{\phi}(c))-a\right\rVert^{2}]$
(9)

This, combined with the off-policy nature of our algorithm, also allows the
robot to bootstrap learning by first “pre-training” via BC (Eq.9) using the
expert demonstrations, before improving performance further via meta-IRL
training.

Algorithm 1 SQUIRL: Soft Q-functioned Meta-IRL (Train)

Input: One expert video demonstration trajectory of state-action pairs
$\mathcal{D}^{i}_{\pi_{E}}=\\{s^{i}_{1:H},a^{i}_{1:H}\\}$ for each of the $n$
training tasks $i=1:n$, where $H$ is the horizon of the task (e.g.,
$n=117,H=100$)

1: Initialize soft Q-function $Q_{\theta}$, policy $\pi_{\psi}$, task encoder
$\Psi_{\phi}$, and an empty buffer of off-policy robot trajectories
$\mathcal{D}^{i}_{\pi_{\psi}}\leftarrow\\{\\}$ for each training task $i=1:n$

2: Warm-up policy and task encoder via $\mathcal{L}^{BC}$ (Eq.9)

3: while not converged do

4: Sample a batch of $m$ task indices $\\{i^{1:m}\\}$ from all training tasks
$i=1:n$, (e.g., $m=10$)

5: for $i=i^{1:m}$ do

6: Infer task embedding
$z^{i}=\mathbb{R}^{\mathcal{Z}}\leftarrow\Psi_{\phi}(c)$, where
$c=\\{s^{i}_{1:C},a^{i}_{1:C}\\}\sim\mathcal{D}^{i}_{\pi_{E}}$ (e.g.,
$\mathcal{Z}=32,C=64$)

7: Generate a robot trajectory of state-action pairs
$\overline{\mathcal{D}}^{i}_{\pi_{\psi}}=\\{s^{i}_{1:H},a^{i}_{1:H}\\}$ from
task $i$ using $\pi_{\psi},z^{i}$

8:
$\mathcal{D}^{i}_{\pi_{\psi}}\leftarrow\mathcal{D}^{i}_{\pi_{\psi}}\cup\overline{\mathcal{D}}^{i}_{\pi_{\psi}}$

9: end for

10: for $j=1:J$ (e.g., $J=400$) do

11: Sample another batch of $m$ task indices $\\{i^{1:m}\\}$

12: $\theta\leftarrow\theta-\nabla_{\theta}\mathcal{L}^{IRL}$ (Eq.III-B) using
a combined batch of $\mathcal{B}=128$ robot and expert timesteps:
$\overline{\mathcal{D}}^{i}_{\pi_{\psi}}\cup\overline{\mathcal{D}}^{i}_{\pi_{E}}$
and $z^{i}$, where
$\overline{\mathcal{D}}^{i}_{\pi_{\psi}}\sim\mathcal{D}^{i}_{\pi_{\psi}}$,
$\overline{\mathcal{D}}^{i}_{\pi_{E}}\sim\mathcal{D}^{i}_{\pi_{E}}$,
$i=\\{i^{1:m}\\}$

13: end for

14: for $k=1:K$ (e.g., $K=2000$) do

15: Sample another batch of $m$ task indices $\\{i^{1:m}\\}$

16: if necessary then
$\\{\psi,\phi\\}\leftarrow\\{\psi,\phi\\}-\nabla_{\psi,\phi}\mathcal{L}^{BC}$
(Eq.9) using a batch of $\mathcal{B}$ expert timesteps
$\overline{\mathcal{D}}^{i}_{\pi_{E}}\sim\mathcal{D}^{i}_{\pi_{E}},z^{i}$,
$i=\\{i^{1:m}\\}$ end if

17: $\psi\leftarrow\psi-\nabla_{\psi}\mathcal{L}^{RL}$ (Eq.4) using a combined
batch of $\mathcal{B}$ robot and expert timesteps:
$\overline{\mathcal{D}}^{i}_{\pi_{\psi}}\cup\overline{\mathcal{D}}^{i}_{\pi_{E}}$
and $z^{i}$, where
$\overline{\mathcal{D}}^{i}_{\pi_{\psi}}\sim\mathcal{D}^{i}_{\pi_{\psi}}$,
$\overline{\mathcal{D}}^{i}_{\pi_{E}}\sim\mathcal{D}^{i}_{\pi_{E}}$,
$i=\\{i^{1:m}\\}$

18: end for

19: end while

20: return soft Q-function $Q_{\theta}$, policy $\pi_{\psi}$, task encoder
$\Psi_{\phi}$

Algorithm 2 SQUIRL: Soft Q-functioned Meta-IRL (Test)

Input: $\pi_{\psi}$, $\Psi_{\phi}$, $Q_{\theta}$, and a single expert video
demonstration of state-action pairs
$\mathcal{D}^{i}_{\pi_{E}}=\\{s^{i}_{1:H}$, $a^{i}_{1:H}\\}$ from a new task
$i$ unseen during training

1: Infer task embedding vector
$z^{i}=\mathbb{R}^{\mathcal{Z}}\leftarrow\Psi_{\phi}(c)$, where
$c=\\{s^{i}_{1:C},a^{i}_{1:C}\\}\sim\mathcal{D}^{i}_{\pi_{E}}$ (e.g.,
$\mathcal{Z}=32,C=64$)

2: Rollout robot trajectory in the real world using $\pi_{\psi}$, $z^{i}$

Approach Box

Lower to Box

Grasp Box

Pick up Box

Carry Box

Drop Box

Figure 4: Pick-Carry-Drop Experiment. The robot needs to approach, lower to,
grasp, pick-up, carry, and drop the box to solve the task.

###  V-D Using Expert Demonstration as Both the Input Task Context Variables
and Training Signal for the Task Encoder

Learning robust task embeddings enables robots to generalize to new tasks
quickly [23]. To this end, our algorithm proposes to use 64 expert timesteps
as the input task context variable $c$ into the task encoder, as opposed to 64
robot timesteps. This is because context variables should explore the task and
environment sufficiently well to expose the key information of the task, and
expert demonstration timesteps are an ideal candidate compared to the
timesteps from the robot’s suboptimal policy. As a result, the context
variable $c$ input into the task encoder only includes the states and actions
of the expert, but not the rewards or the next states.

In addition, we choose the BC loss $\mathcal{L}^{BC}$ in Eq.9 as the training
loss for learning the task encoder $\Psi_{\phi}$. This BC loss is stable since
the expert timesteps are fixed. In contrast, the IRL loss $\mathcal{L}^{IRL}$
(Eq.III-B) and the policy loss $\mathcal{L}^{RL}$ (Eq.4) are less stable
because the training data distribution for both losses are non-stationary.
This design choice also allows us to learn a robust task embeddings first via
BC pre-training before performing meta-IRL training via SQUIRL. We empirically
observe that such pre-training can improve the training stability and the
sample efficiency of SQUIRL, but the final policy performance is similar with
or without BC pre-training. In summary, our algorithm is detailed in Algorithm
1 (train) and Algorithm 2 (test), with hyperparameters detailed
here222Hyperparameters in Algorithm 1 and 2. Policy gradient batch size
$\mathcal{B}$: 1024 (non-vision), 128 (vision); task embedding batch size $C$:
64; all learning rates: $3e^{-4}$; starting SAC alpha: $1e^{-5}$; SAC target
entropy: $-300$; IRL updates per epoch $J$: $400$; policy updates per epoch
$K$: $2000$; task embedding size $\mathcal{Z}$: 32; meta-batch size $m$: 10;
discount rate $\gamma$: 0.99.

##  VI Experiments and Results Analysis

We evaluate the generality and robustness of our algorithm across long-horizon
vision and non-vision tasks with continuous state and action spaces in both
simulation (Pick-Carry-Drop, a horizon of 1024 timesteps, 30 train tasks) and
real-world (Pick-Pour-Place, a horizon of 100 timesteps, 117 train tasks).
There is only a single expert demonstration for each of the train or test
tasks. We compare with the PEARL-BC baseline, which is the behavioral cloning
version of PEARL [23]. Evaluation: We evaluate real-robot and simulation
experiments on 50 and 500 trials respectively across 50 seen and unseen tasks.
We report mean and standard deviation (“stdev” hereafter). The performance
difference between different experiments is statistically significant if the
difference in mean is at least either standard deviation away. Experimental
video is at http://crlab.cs.columbia.edu/squirl.

Approach Bottle

Grasp Bottle

Carry Bottle

Pour Orange Cup

Carry Bottle

Place Bottle

Figure 5: Pick-Pour-Place at Test Time. To solve this task, the robot needs to
first approach, grasp and carry the grey bottle, pour the iron pebble inside
the bottle into a specific container, and carry and place the bottle back on
the table. At the beginning of each task, the bottle is not in hand, and but
the iron pebble is already in the bottle. Top row: top-down camera images.
Bottom row: 45°camera images.

###  VI-A Simulation Experiment: Pick-Carry-Drop

Description. We modify the planar Stacker task [27] to create “Pick-Carry-
Drop”. Shown in Fig.4, a robot is tasked to approach, pick, carry, and drop
the black box into the stack marked in green. The task is successful if the
box is dropped into the stack within 1024 timesteps, and failed otherwise.

State Space. We evaluate our algorithm on both the vision and the non-vision
version of the task, to demonstrate that SQUIRL is general across different
state space modalities. The state space for the vision version includes 1) the
joint angles and velocities for its 5-DOFs, 2) a one-hot vector indicating the
current stage of the task, and 3) an RGB image shown in Fig.4. The non-vision
version’s state space replaces the RGB image with the position of the black
box.

Action Space. The robot controls its 5-DOF joint torques.

Task Definition. There are a total of 30 training tasks in this experiment,
each corresponding to a different drop location:
$x\in\\{-0.15,-0.14,\ldots,0.14\\}$. During test time, we randomly sample a
new, real-valued drop location from the maximum valid range:
$x\in[-0.25,0.25]$. The green drop location is invisible in both the vision
and the non-vision version of the task. Therefore, the robot needs to infer
the green drop location (i.e., task information) solely from the provided
expert video demonstration. On the other hand, the starting pose of the robot
and the location of the black box are all initialized randomly at the
beginning of each task.

Robot Trials. The robot uses 150 training trials in total.

Expert Demonstration. We trained an expert policy from scratch via RL to
provide expert demonstrations. The reward function used to train the expert
policy comprises of six stages, each stage with a reward of 10. Designing this
reward function has taken significant human effort, which exhibits the value
of directly learning from video demonstrations.

TABLE I: Pick-Carry-Drop Results (% Drop Success$\pm$Stdev) Tasks | Seen | Unseen | Seen | Unseen
---|---|---|---|---
| Vision | Non-Vision
SQUIRL (BC + IRL) | 95.8$\pm$1.7 | 95.0$\pm$1.5 | 97.3$\pm$3.0 | 96.9$\pm$2.0
Baseline (PEARL-BC) | 77.8$\pm$1.6 | 76.5$\pm$0.7 | 90.8$\pm$2.5 | 89.5$\pm$1.6
Ablation: No BC Joint Training or BC Pre-training
SQUIRL (IRL Only) | 93.8$\pm$1.8 | 93.2$\pm$1.6 | 94.7$\pm$1.7 | 93.9$\pm$1.4

Simulation Results and Analysis. As shown in Table I, our algorithm, “SQUIRL
(BC + IRL)”, pre-trains via BC and then trains the policy using both the BC
loss (Eq.9) and the IRL policy gradient loss (Eq.4). It statistically
significantly outperforms the PEARL-BC baseline in both the vision
(95.8%$\pm$1.7 vs. 77.8%$\pm$1.6) and non-vision (97.3%$\pm$3.0 vs.
90.8%$\pm$2.5) version of the task for seen tasks. For unseen tasks, we
observed similar outperformance (95.0%$\pm$1.5 vs. 76.5%$\pm$0.7 in the vision
case and 96.9%$\pm$2.0 vs. 89.5%$\pm$1.6 in the non-vision case).
Qualitatively, in the PEARL-BC’s case, the robot sometimes misses the drop
location as it attempts to drop the box or fails to pick up the box when the
box gets stuck by the walls of the stack (kindly see website). The performance
drop of the baseline from the non-vision version (90.8%$\pm$2.5 and
89.5%$\pm$1.6 for seen and unseen tasks) to the vision version (77.8%$\pm$1.6
and 76.5%$\pm$0.7 for seen and unseen tasks) is mainly because vision-based
manipulation tends to suffer from larger compounding errors. Nevertheless, as
evident in the statistical similarities between seen and unseen tasks for
SQUIRL (95.8%$\pm$1.7 vs. 95.0%$\pm$1.5 for vision) and PEARL-BC
(77.8%$\pm$1.6 vs. 76.5%$\pm$0.7 for vision), both algorithms can generalize
to unseen tasks, due to the generalizability of task embeddings.

Ablation: IRL Gradient Only. To compare the performance contribution of
SQUIRL’s meta-IRL core training procedure directly against PEARL-BC, we
created “SQUIRL (IRL only)”, which trains the policy using only the policy
gradient loss in Eq.4 (no BC joint training or pre-training). This ablated
version still outperforms the PEARL-BC baseline (93.8%$\pm$1.8 vs.
77.8%$\pm$1.6 for seen vision tasks, 93.2%$\pm$1.6 vs. 76.5%$\pm$0.7 for
unseen vision tasks). Nevertheless, by combining BC and IRL gradients, “SQUIRL
(BC + IRL)” improves performance slightly further (95.8%$\pm$1.7 and
95.0%$\pm$1.5). Intuitively, while BC only matches the expert’s conditional
action distribution under the expert’s  state distribution, BC’s supervised
learning signal is stabler than IRL. Joint training with BC and IRL gradients
can be interpreted as combining the stability of BC and the robustness of
Q-functioned IRL, by matching the conditional action distribution of the
expert under the broader state distribution of the expert-robot mixture
experience (Eq.8), in addition to matching the expert’s joint state-action
distribution (Eq.7).

###  VI-B Real-Robot Experiment: Pick-Pour-Place

Description. We evaluated our algorithm on the UR5-Seed robot (Fig.2) to
perform a set of long-horizon pick-pour-place tasks. As shown in Fig.2, in
each task, there is a grey cylindrical bottle, an iron pebble that is already
in the bottle, and more than one container on the table. The robot is tasked
to approach and pick-up the grey bottle, pour the iron pebble into a specific
container, and place the bottle back on the table. The task is a success only
if the pebble is poured into the correct container and the bottle is placed
upright on the table within $H=100$ timesteps, and a failure otherwise.

State Space. The state space contains a top-down or 45°camera’s RGB image
(Fig.5), and 2 binary indicators for whether the robot has poured or closed
the hand, respectively.

Action Space. The action space includes the Cartesian unit directional vector
for the end-effector movement. During each timestep, the robot can adjust the
end-effector by 2cm along any 3D direction. The action space also includes a
binary indicator to control the arm vs. the hand and a trinary indicator to
close, open, or rotate the hand for pouring.

Orthogonality to State and Action Representions. While Pick-Pour-Place can be
tackled by first localizing the correct container via object detection
(alternative state space) and then executing motion-planning trajectories to
pour (alternative action space), our algorithm is general across and
orthogonal to alternative state and action spaces.

Task Definition. As shown in each row of images in Fig.1, each task is defined
by the positions and colors of the containers, and by the correct container to
pour into. There are always only the green and yellow containers in the 117
train tasks. 25 of the 50 test tasks have the green and yellow containers at
new positions. The remaining 25 test tasks add the red and the orange unseen
containers, or either. Since there is always more than one container in the
RGB image, the robot will not know which container to pour into without the
expert demonstration. Therefore, the robot needs to depend solely on the task
encoder’s ability to extract the correct task information from the expert
demonstration.

Robot Trials. The robot collects 90 training trials in total.

Expert Demonstration. We collect demonstrations via teleoperation using a
Flock of Birds sensor333Flock of Birds is a 6D pose tracker from Ascension
Technologies Corp.. Using the human wrist pose detected by the sensor in real-
time, we move, open, close, or rotate the robot hand for pouring. We collected
$117$ video demonstrations across 117 tasks for training. It takes 1-2 minutes
to collect one demonstration.

TABLE II: Pick-Pour-Place Results (% Pour Success$\pm$Stdev) Tasks | RGB Image | Seen | Unseen
---|---|---|---
SQUIRL (BC + IRL) | Top-Down (90°) | 92.0$\pm$4.5 | 90.0$\pm$7.1
Baseline (PEARL-BC) | 70.0$\pm$7.1 | 68.0$\pm$11.0
Baseline (Standard-BC) | 60.0$\pm$10.0 | 56.0$\pm$11.4
SQUIRL (BC + IRL) |  $45\degree$ (Ablation) | 90.0$\pm$7.1 | 88.0$\pm$8.4

Real-robot Results and Analysis. As shown in Table II, our algorithm
outperforms the PEARL-BC baseline statistically significantly in both seen
tasks (92.0%$\pm$4.5 vs. 70.0%$\pm$7.1) and unseen tasks (90.0%$\pm$7.1 vs.
68.0%$\pm$11.0). This observed outperformance mainly originates from our soft
Q-functioned IRL formulation, which forces the robot to imitate the expert
under a much wider state distribution provided by the expert-robot mixture
trajectories, instead of the narrow state distribution of the expert
demonstrations. This helps reduce compounding errors during task execution.
The low performance of the PEARL-BC baseline is mainly due to additional
compounding errors induced by real-world sensory noises such as unstable
lighting conditions and small perturbation to camera positions. Qualitatively,
the PEARL-BC baseline sometimes pours into the wrong container, misses the
target container by a few centimeters, or moves past the target container
while failing to pour in time (kindly see website for examples). Nevertheless,
from the statistical similarity between seen and unseen tasks for both our
algorithm (92.0%$\pm$4.5 vs. 90.0%$\pm$7.1) and PEARL-BC (70.0%$\pm$7.1 vs.
68.0%$\pm$11.0), we see that the learned task encoder is still effectively
generalizing to a new, related task.

Comparison to the “Standard-BC” Baseline. We also compared to “Standard-BC”
(60.0%$\pm$10.0 and 56.0%$\pm$11.4 for seen and unseen tasks), which performs
no meta-learning and learns every train or test task independently from
scratch via BC. As a result, the neural network overfits to the single
demonstration and fails to generalize to real-world sensory (camera) noises at
test time. Note that Standard-BC’s unseen-task performance is slightly lower
than seen tasks since the unseen tasks are more challenging with at most 4
containers on the table, compared to only 2 containers in seen tasks.

Ablation: Non-top-down Camera. We also tested our algorithm with a $45\degree$
RGB image (90.0%$\pm$7.1 and 88.0%$\pm$8.4 for seen and unseen tasks) against
a top-down RGB image (92.0%$\pm$4.5 and 90.0%$\pm$7.1 for seen and unseen
tasks). The statistical similarity between the two shows that SQUIRL is
general and can accept a non-top-down RGB input image.

##  VII Conclusion

We introduced SQUIRL, a robust, efficient, and general Soft Q-functioned meta-
IRL algorithm, towards enabling robots to learn from limited expert (one per
task) and robot (90 in total) trajectories. This algorithm is statistically
significantly more robust than behavioral cloning and requires no trial-and-
errors at test time. Finally, this general algorithm has been tested to work
with various long-horizon manipulation tasks, and across vision and non-vision
state and action spaces. In the future, we will extend this algorithm to learn
from direct human-arm demonstrations instead of teleoperation. This will lower
the cost of collecting real-world expert demonstrations further. We also aim
to incorporate hierarchical learning into SQUIRL to solve much longer horizon
manipulation tasks by reusing low-level subpolicies.

## References

  * [1] P. Abbeel and A. Ng, “Apprenticeship learning via inverse reinforcement learning,” _International Conference on Machine Learning_ , 2004.
  * [2] N. Ratliff, B. Andrew, and M. Zinkevich, “Maximum margin planning,” _International Conference on Machine Learning (ICML)_ , 2006.
  * [3] B. Ziebart, “Modeling purposeful adaptive behavior with the principle of maximum causal entropy,” _PhD Thesis_ , 2010.
  * [4] B. D. Ziebart, A. Maas, J. A. Bagnell, and A. K. Dey, “Maximum entropy inverse reinforcement learning,” in _Proc. AAAI_ , 2008.
  * [5] A. Boularias, J. Kober, and J. Peters, “Relative entropy inverse reinforcement learning,” in _Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics_ , ser. Proceedings of Machine Learning Research, vol. 15. PMLR, 11–13 Apr 2011.
  * [6] S. K. Seyed Ghasemipour, S. S. Gu, and R. Zemel, “Smile: Scalable meta inverse reinforcement learning through context-conditional policies,” in _Advances in Neural Information Processing Systems_ , 2019.
  * [7] L. Yu, T. Yu, C. Finn, and S. Ermon, “Meta-inverse reinforcement learning with probabilistic context variables,” in _NeurIPS_ , 2019.
  * [8] J. Fu, K. Luo, and S. Levine, “Learning robust rewards with adverserial inverse reinforcement learning,” in _ICLR_ , 2018.
  * [9] B. D. Argall, S. Chernova, M. Veloso, and B. Browning, “A survey of robot learning from demonstration,” _Robotics and Autonomous Systems_ , vol. 57, no. 5, pp. 469–483, 2009.
  * [10] D. Xu, S. Nair, Y. Zhu, J. Gao, A. Garg, L. Fei-Fei, and S. Savarese, “Neural task programming: Learning to generalize across hierarchical tasks,” in _International Conference on Robotics and Automation_ , 2018.
  * [11] D.-A. Huang, S. Nair, D. Xu, Y. Zhu, A. Garg, L. Fei-Fei, S. Savarese, and J. C. Niebles, “Neural task graphs: Generalizing to unseen tasks from a single video demonstration,” in _CVPR_ , 2019.
  * [12] D.-A. Huang, Y.-W. Chao, C. Paxton, X. Deng, L. Fei-Fei, J. C. Niebles, A. Garg, and D. Fox, “Motion reasoning for goal-based imitation learning,” in _ICRA_ , 2020.
  * [13] C. Finn, S. Levine, and P. Abbeel, “Guided cost learning: Deep inverse optimal control via policy optimization,” in _ICML_ , 2016.
  * [14] T. Zhang, Z. McCarthy, O. Jow, D. Lee, X. Chen, K. Goldberg, and P. Abbeel, “Deep imitation learning for complex manipulation tasks from virtual reality teleoperation,” in _ICRA_ , 2018.
  * [15] J. Kober and J. Peters, “Imitation and reinforcement learning - practical algorithms for motor primitive learning in robotics,” _IEEE Robotics and Automation Magazine_ , vol. 17, no. 2, pp. 55–62, 2010.
  * [16] P. Pastor, H. Hoffmann, T. Asfour, and S. Schaal, “Learning and generalization of motor skills by learning from demonstration,” in _International Conference on Robotics and Automation (ICRA)_ , 2009.
  * [17] P. Sermanet, C. Lynch, Y. Chebotar, J. Hsu, E. Jang, S. Schaal, S. Levine, and G. Brain, “Time-contrastive networks: Self-supervised learning from video,” in _ICRA_ , 2018.
  * [18] C. Lynch, M. Khansari, T. Xiao, V. Kumar, J. Tompson, S. Levine, and P. Sermanet, “Learning latent plans from play,” in _CoRL_ , 2019.
  * [19] S. K. S. Ghasemipour, R. Zemel, and S. Gu, “A divergence minimization perspective on imitation learning methods,” in _CoRL_ , 2019.
  * [20] C. Finn, T. Yu, T. Zhang, P. Abbeel, and S. Levine, “One-shot visual imitation learning via meta-learning,” in _CoRL_ , 2017.
  * [21] T. Yu, C. Finn, A. Xie, S. Dasari, T. Zhang, P. Abbeel, and S. Levine, “One-shot imitation from observing humans via domain-adaptive meta-learning,” in _Robotics: Science and Systems (RSS)_ , 2018.
  * [22] T. Yu, P. Abbeel, S. Levine, and C. Finn, “One-shot hierarchical imitation learning of compound visuomotor tasks,” in _IROS_ , 2019.
  * [23] K. Rakelly, A. Zhou, D. Quillen, C. Finn, and S. Levine, “Efficient off-policy meta-reinforcement learning via probabilistic context variables,” _International Conference on Machine Learning (ICML)_ , 2019.
  * [24] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,” _International Conference on Machine Learning (ICML)_ , 2018.
  * [25] J. Ho and S. Ermon, “Generative adversarial imitation learning,” in _Advances in neural information processing systems_ , 2016.
  * [26] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in _Advances in neural information processing systems_ , 2014.
  * [27] Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li, D. de Las Casas, D. Budden, A. Abdolmaleki, J. Merel, A. Lefrancq, T. Lillicrap, and M. Riedmiller, “DeepMind control suite,” Tech. Rep., Jan. 2018.