# A Study of Causal Confusion in Preference-Based Reward Learning

Jeremy Tien1, Jerry Zhi-Yang He1, Zackory Erickson2, Anca D. Dragan1, and
Daniel Brown1 1InterACT Lab, UC Berkeley. 2 Carnegie Mellon University
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Learning robot policies via preference-based reward learning is an
increasingly popular method for customizing robot behavior. However, in recent
years, there has been a growing body of anecdotal evidence that learning
reward functions from preferences is prone to spurious correlations and reward
gaming or hacking behaviors. While there is much anecdotal, empirical, and
theoretical analysis of causal confusion and reward gaming behaviors both in
reinforcement learning and imitation learning approaches that directly map
from states to actions, we provide the first systematic study of causal
confusion in the context of learning reward functions from preferences. To
facilitate this study, we identify a set of three preference learning
benchmark domains where we observe causal confusion when learning from offline
datasets of pairwise trajectory preferences: a simple reacher domain, an
assistive feeding domain, and an itch-scratching domain. To gain insight into
this observed causal confusion, we present a sensitivity analysis that
explores the effect of different factors—including the type of training data,
reward model capacity, and feature dimensionality—on the robustness of rewards
learned from preferences. We find evidence that learning rewards from pairwise
trajectory preferences is highly sensitive and non-robust to spurious features
and increasing model capacity, but not as sensitive to the type of training
data. Videos, code, and supplemental results are available at
https://sites.google.com/view/causal-reward-confusion.

##  I Introduction

Preference-based reward learning [1, 2] is a well-studied technique for
learning from human input that takes the form of pairwise preferences or
rankings. Although preference learning holds the potential of allowing robots
to learn specifications for tasks without requiring a human to write down an
explicit reward function and to adapt robot behavior to individual preferences
and needs, much prior work focuses on learning reward functions that are
linear combinations of well-specified, hand-crafted features [1, 3, 4]. These
methods work well, but they put a large burden on the feature designer to make
sure that the reward features are well specified and span the space of desired
reward functions.

Recent work has explored learning reward functions from preferences using deep
learning [2, 5, 6, 7]. However, there is growing anecdotal evidence that these
methods are prone to learning reward functions that overfit to spurious
correlations in the data, especially when learning reward functions from
limited numbers of preferences queries [2, 6, 8]. The effects of reward
misspecification have recently been studied in the context of reinforcement
learning agents that optimize a proxy reward function [9], and causal
confusion has been studied in imitation learning methods that directly mimic
expert demonstrations via supervised learning [10, 11, 12], yet there has not
been a systematic study of causal confusion in reward function learning from
preferences.

As an example of the type of causal confusion we study in this paper, consider
an assistive feeding task—the assistive robot’s goal is to feed food particles
into the patient’s mouth safely without spilling. Note that successful robot
executions will make contact with the patient’s mouth and apply small amounts
of force, whereas failed executions may not make any contact with the patient.
In this setting, we find that a preference-based learning approach will often
miss the correct causal relationship between “contact” and “feeding”, leading
the robot to optimize a policy that seeks any kind of contact with the
patient, including contact with the patient’s torso or head (see website for
videos of such behavior).

In this paper, we seek to systematically study causal confusion in preference-
based reward learning for robotics tasks. We first demonstrate across three
different robot learning benchmark tasks that, even with large numbers of
pairwise preferences over trajectories, the learned reward functions do not
lead to desirable behavior, indicating causal confusion over the true reward
function. To better understand this phenomenon, we focus our study on three
factors that may affect causal confusion: (1) the type of features used to
learn the reward function, (2) the capacity of the reward function model, and
(3) the preference training data generation mechanism. We find evidence that
increasing the dimensionality of the reward function feature space exacerbates
causal confusion. We also find evidence that increasing model capacity also
can lead to causal confusion, especially when learning rewards over larger
feature spaces. Finally, we compare several methods for collecting pairwise
preferences. We find that human-demonstrations that seek to be pedagogic lead
to better sample efficiency, but that all the training data collection methods
we study are prone to causal confusion.

In summary, our paper provides the following contributions: (1) We identify a
set of 3 robot preference learning environments that exhibit causal confusion
when learning reward functions from preferences. This combination of domains
and preference training data forms the first concrete set of benchmarks for
studying causal confusion when learning reward functions. (2) We perform the
first systematic study of causal confusion in preference-based reward learning
by varying the feature-space dimensionality, the capacity of the reward
function model, and the preference data used to train the reward function
model. (3) We identify spurious features and model capacity as the main
sources of causal confusion when learning reward functions from pairwise
preferences.

##  II RELATED WORK

Reward Hacking and gaming behaviors are known to commonly occur in
reinforcement learning [13, 14] and reward learning [2, 6, 15]. However, these
behaviors are often only mentioned anecdotally [14]. Recently, Pan et al. [9]
proposed to systematically analyze reward misspecification in RL by creating a
set of domains where the agent optimizes a proxy reward function and studying
when this leads to incorrect behavior. By contrast, we study cases where the
reward function must be learned from human preferences.

Preference Learning is a popular method for training AI systems when a reward
function is unavailable [1, 2, 3, 6, 4, 5, 16]. Preferences are often easier
to provide than demonstrations or raw numerical values [17, 18, 1] since they
do not require expert proficiency or fine-grained feedback. However,
optimizing a reward function that has been trained using preferences can
sometimes lead to unintended behaviors. Evidence of this has been documented
for Atari games [2, 6, 16], simple pointmass navigation tasks [8], and
language model fine-tuning [19]. Recent work has attempted to detect learned
reward functions that overfit to spurious correlations [20]; however, there
has only been limited study of causal confusion when learning reward functions
for robotics tasks [21]. In this paper, we provide the first concrete and
systematic study of causal confusion across three robot control benchmarks
when learning and optimizing reward functions from preference labels over
trajectories.

Causal Confusion in Imitation Learning has previously been studied in the
context of behavioral cloning [22, 23], a supervised learning approach to
imitation learning, where a policy is trained to directly map from states to
actions using demonstrated state-action pairs. Prior work has shown that
behavioral cloning approaches can suffer causal confusion due to “causal
misidentification,” where giving imitation learning policies more information
leads to worse performance [10] due to temporally correlated noise in expert
actions [12]. While we do not consider demonstrator noise, we do find strong
evidence of causal misidentification when learning reward functions from
preferences—access to a richer set of features increases the chance of causal
confusion over the true reward function, leading to poor policy performance.
Zhang et al. [11] investigates causal confusion in imitation learning by using
causal diagrams for simplified imitation learning tasks with discrete actions
and observing performance when features available to the demonstrator are
different from those of the imitator. We also study what happens when the
observation space of the imitator changes; however, we focus on imitation via
preference learning, consider continuous-action robotics tasks, and evaluate
causal confusion when using non-linear reward function approximation.

##  III Reward Learning from Preferences

We are interested in studying causal confusion that occurs when misaligned
reward functions are learned from pairwise preferences over trajectories. We
consider environments that are modeled as finite horizon MDPs [24], with state
space $\mathcal{S}$, action space $\mathcal{A}$, horizon $T$, and reward
function $r:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$. Note that we
assume that the reward function is unobserved and must be learned from
preferences over trajectories. We assume that preferences over trajectories
are based on the cumulative reward associated with each trajectory. Thus,
using the popular Bradley-Terry model of preferences [25], the probability a
trajectory, $\tau_{B}$, is preferred over another trajectory, $\tau_{A}$, is
given by

$P(\tau_{A}\prec\tau_{B})=\frac{\exp(r(\tau_{B}))}{\exp(r(\tau_{A}))+\exp(r(\tau_{B})},$
(1)

where $r(\tau)=\sum_{(s,a)\in\tau}r(s,a)$ and where we define a trajectory,
$\tau$, by a sequence of state-action pairs:
$\tau=(s_{0},a_{0},\ldots,s_{T},a_{T})$..

To learn a reward function from preferences, we assume access to a set of
pair-wise preference labels, $\mathcal{P}$, over trajectories,
$\tau_{1},\ldots,\tau_{N}$, where $(i,j)\in\mathcal{P}$ implies that
$\tau_{i}\prec\tau_{j}$. We then optimize a reward function
$r_{\theta}:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$, parameterized
by $\theta$ that maximizes the likelihood of the pairwise preference data [2]:

$\mathcal{L}(\theta)=\prod_{(i,j)\in\mathcal{P}}\frac{\exp(r_{\theta}(\tau_{j}))}{\exp(r_{\theta}(\tau_{i}))+\exp(r_{\theta}(\tau_{j}))}.$
(2)

This likelihood function is differentiable, allowing us to leverage linear and
non-linear function approximation to learn the reward function from trajectory
preferences. We explore learning both linear and neural network reward
function models and provide specific training details in the relevant sections
below.

We note that, for the sake of systematically studying causal confusion, our
focus in this paper is on learning reward functions from offline sets of
trajectory preferences. We do this for three reasons. First, using offline
trajectory preference data facilitates reproducibility by allowing us to
create benchmarks (with fixed training datasets) for studying causal confusion
in reward learning, similar to existing standard supervised learning
benchmarks. Second, learning from offline data is known to often lead to
overfitting in batch or offline reinforcement learning [26], and prior work
has shown anecdotal evidence that learning a reward function from a fixed set
of preferences can lead to causal confusion [2, 6, 8]—hence, learning from
offline data naturally lends itself to a study of causal confusion. Third,
learning reward functions from offline data, as described above, removes the
need for running reinforcement learning [2] or trajectory optimization [3] in
the inner loop, as is required for many active preference learning methods.
This enables us to learn a reward function by optimizing the likelihood in
Equation 2 (which amounts to optimizing a standard cross-entropy loss with the
predicted cumulative rewards as logits) and then learn a corresponding policy
using any off-the-shelf reinforcement learning algorithm to optimize
$r_{\theta}$. This sequential approach—learning a reward function followed by
learning a policy—significantly improves the scalability and simplicity of
experiments, facilitating the rapid testing and evaluation needed here for a
systematic study of causal confusion.

In practice, we use the Adam optimizer in PyTorch to learn the reward
function, $r_{\theta}$, and then use PPO [27] or SAC [28] for policy
optimization given $r_{\theta}$.

##  IV Environments for Preference Learning

One of our contributions is to identify a set of robot learning benchmarks
that exhibit causal confusion when learning reward functions from preferences.
As part of this contribution, we have open-sourced our code and training
datasets of trajectory preferences for reproducibility and to facilitate
future research on training reward functions that are robust to causal
confusion. This combination of domains and training data form the first
concrete set of benchmarks for studying causal confusion when learning reward
functions for robotic tasks. Links to download and install source code for the
domains and links to download the preference training data used in our
experiments are available at https://sites.google.com/view/causal-reward-
confusion.

We briefly describe the three domains that we consider:

(a) Reacher-v2

(b) FeedingSawyer-v1

(c) ScratchItchJaco-v1

Figure 1: Environments. We propose and study three benchmark environments in
which performing reward function learning from pairwise trajectory preferences
leads to causal confusion over the true reward function.

  1. 1.

Reacher (MuJoCo) environment from OpenAI gym [29] as shown in Figure 1(a). The
goal of this task is to move an end effector attached to a two-link robot arm
to a desired goal location on a 2D plane.

  2. 2.

Feeding (Assistive Gym), a robotic assistive feeding task that is part of
Assistive Gym [30], as shown in Figure 1(b). The goal of this task is to feed
the human using a spoon carrying pieces of food.

  3. 3.

Itch Scratching (Assistive Gym), as shown in Figure 1(c). The goal of this
task is to repeatedly scratch a desired itch location on the human’s arm using
the robot’s end effector.

We first note that each of these domains comes with a predefined “true” reward
function, $r$ that was hand designed by the original developers of each
environment. This enables us to create synthetic demonstrations and preference
labels. Note, that while we use the ground-truth reward function for obtaining
preference labels ($r(\tau_{1})<r(\tau_{2})\implies\tau_{1}\prec\tau_{2}$), we
assume no access to this reward function during policy learning, but rather
seek to learn a policy that obeys a users preferences by first learning a
model $r_{\theta}$ of the true reward function from preference labels
$\mathcal{P}$ and then using the learned reward function to optimize a policy.
We then evaluate the policy using the ground-truth reward function $r$.

We also note that all of the benchmark tasks are variations on moving an end
effector toward a target—feeding requires additional stability to prevent
spillage, and itch scratching requires a repetitive scratching motion at the
target. In contrast to prior work on deep preference-based reward learning
that has shown success on repetitive tasks such as Atari games and MuJoCo
locomotion [2, 5], we found that the above three tasks are surprisingly
difficult to learn via offline preference data. We leave the identification
and creation of additional benchmark domains for future work.

##  V Evidence of Causal Confusion

To demonstrate that each of these domains has problems with causal confusion,
we first used pairwise preferences over trajectories to train a neural network
reward function approximator to map from the original state observations for
each domain to a real valued scalar reward signal which we then optimize via
RL.

###  V-A Synthetic Preference Generation

To enhance scalability and reproducibility, we automatically generate a large
amount of synthetic trajectory preferences. This was done using a pretrained
RL policy for each of these domains that was trained using the ground-truth
reward function provided with each of these environments. We then generate a
large number of diverse trajectories by adding $\epsilon$-greedy noise during
policy rollouts, where $\epsilon$ is the probability that the policy takes an
action uniformly at random from its action space. Thus, $\epsilon=0$
corresponds to the fully trained RL policy and $\epsilon=1$ corresponds to a
uniformly random policy. As noted by Brown et al. [7], adding this type of
disturbance will result in monotonically decreasing performance in
expectation.

To generate pairwise preferences over trajectories, we randomly sample pairs
of trajectories with replacement (while ensuring that the two trajectories
within a pair are nonidentical) from a set of 120 total trajectories generated
with 20 $\epsilon$-greedy rollouts for $\epsilon\in\\{0,0.2,0.4,0.6,0.8,1\\}$.
From these pairwise preferences, we randomly set aside 10% as a validation
set, and use the rest to train. We then use the ground-truth reward functions
provided by each environment to provide ground-truth preference labels. Using
the dataset of preferences, we train a neural network reward function
approximator with two hidden layers (128 units and 64 units, respectively) and
Leaky ReLU activations, after which we perform 1,000,000 timesteps of
reinforcement learning with PPO [27] (for Feeding and Itch Scratching) and SAC
[28] (for Reacher) using the learned reward function in place of the ground-
truth reward function. We optimize the reward function approximator using
stochastic gradient descent with weight decay of $0.01$ and early-stopping on
the validation loss (with a patience of 10 epochs).

###  V-B Results

In Table I, we show both the pairwise classification accuracy on the train and
validation sets as well as the subsequent performance of the resulting learned
policies after training on 2,000 pairwise preference labels. Surprisingly, we
find that despite having high validation and test accuracy on distinguishing
between better or worse trajectories, the learned reward functions do not lead
to the correct behavior in the learned policy (as indicated by the low
cumulative reward and task success values), even when given large amounts of
pairwise preferences. In addition to this, we observe that the learned reward
function consistently assigns higher rewards to the policy trained from
preference learning rather than the policy trained on the ground truth reward;
these results provide strong empirical evidence of causal confusion. Figure 2
displays qualitative examples of some of the learned policies. See the
supplementary website for videos of these and other examples.

TABLE I: Empirical Evidence of Causal Confusion. We compare policies optimized
with a reward learned from preferences (PREF) against policies optimized with
the ground truth reward (GT). PREF is given 2000 pairwise preferences over a
wide range of trajectories. Both PREF and GT are optimized with 1000000 RL
iterations. Despite high pairwise classification accuracy, the final policy
performance achieved by PREF under the ground truth reward is extremely low
compared with that of GT. However, the reward learned from preferences
consistently prefers the PREF policy over the GT policy. Results are averaged
over three random seeds. Reacher does not have a predefined task success, but
Feeding and Itch Scratching do.
| Pref. Learning Acc. | RL Policy Performance
---|---|---
Domain (Method) | Train | Validation | Learned Reward | True Reward | Task Success
Reacher (pref) | 0.889 | 0.882 | 13.593 | -17.972 | ——
Reacher (gt) | —— | —— | -4.143 | -6.701 | ——
Feeding (pref) | 0.811 | 0.812 | 28.952 | -139.482 | 0.003
Feeding (gt) | —— | —— | 14.217 | 97.295 | 0.940
Itch Scratching (pref) | 0.895 | 0.897 | 39.070 | -41.940 | 0.000
Itch Scratching (gt) | —— | —— | 15.258 | 269.231 | 0.820

(a) Feeding

(b) Itch Scratching

Figure 2: Undesirable behavior resulting from causal confusion. Figure 2(a):
The robot learns to spill the food particles in front of the human. Figure
2(b): The robot learns to hover its end-effector around without scratching the
itch target.

##  VI Factors that May Lead to Causal Confusion

Our results in the previous section demonstrate strong evidence of causal
confusion during reward learning. In particular, we found that preference-
based reward learning fails on the Reacher, Feeding, and Itching tasks, even
when given large numbers of pairwise preferences—for comparison, prior work
has shown successful reward learning for Atari games and MuJoCo locomotion
tasks with no more than 24 ranked trajectories (276 pairwise preferences) [5].

In the remainder of this paper, we systematically vary different aspects of
preference-based reward learning to gain insight into the following questions:
(1) How does the choice of observation feature space affect causal confusion?
(2) How does the model capacity of the reward-learning network affect causal
confusion? and (3) How does the data used for preference learning affect
causal confusion?

(a) Reacher, Cumulative Reward

(b) Feeding, Cumulative Reward

(c) Feeding, Task Success

(d) Itch Scratching, Cumulative Reward

(e) Itch Scratching, Task Success

Figure 3: Sensitivity of preference learning to raw features in the reward
network’s observation space. Raw features in the Feeding task consist of spoon
and head position, robot joint angles, and amount of force applied on the
human, for a total raw observation dimensionality of 25. The raw features in
the Reacher environment are end effector position, arm angles, target
coordinates, and angular velocities, for a total raw observation
dimensionality of 11. The Itch Scratching environment’s raw features consist
of tool and target position, robot and human joint angles, and the force
applied by the tool, for a total dimensionality of 30. For each environment,
raw features are progressively concatenated onto the entire set of privileged
features.

###  VI-A The Choice of Observation Feature Space

In this section we seek to understand how the choice of observation feature
space affects reward learning. We first use hand-crafted features for reward
learning and then study reward learning when these hand-crafted features are
combined with raw observation features. Surprisingly, the concatenation of the
two—despite having strictly more information—leads to worse reward learning
results than using hand-crafted features alone. This type of causal
misidentification mirrors previous results shown in the context of behavioral
cloning [10].

Hand-crafted Privileged Features: Based on the results in Section V, we find
that preference-learning does not work well using the original, high-
dimensional observation spaces of our three benchmark domains. To better study
the effect of different features on reward learning and causal confusion, we
use a set of hand-designed ‘privileged’ features, as defined in Table II, that
directly correspond to the ground truth reward. We then train a linear reward
model using these features. When training, we empirically found that the model
tends to learn a large bias, and that removing the bias helps incentivize RL
to learn more desirable behavior. Thus, we remove biases from the final layers
of subsequent models.

TABLE II: Privileged features for each environment.
Environment |  Privileged Features
---|---
Reacher |  Distance to target
Feeding |  Distance to mouth
|  Num. food particles in mouth
|  Num. food particles on floor
ScratchItch |  Distance to itch target
|  Force applied at target

We train the reward-learning model on 2000 pairwise preferences of randomly-
selected whole trajectories from the $\epsilon$-greedy rollouts that are at
least 60 ‘ranks apart’ when ranked using the ground truth reward (defined as
$\Delta_{pair}=60$). We employ early-stopping with a patience of 10 epochs and
maximum of 100 epochs, and apply $l_{1}$-regularization with $\lambda=0.1$.
PPO is then used to learn a policy from the learned reward for Feeding and
Itch Scratching, whereas SAC is used for Reacher.

As expected, given features directly corresponding to the ground truth reward,
preference learning is generally able to successfully learn weights over these
features leading RL to optimize for a good policy. We found that Feeding
achieves a mean cumulative reward of 132.18 and task success rate of 99.7%,
and Itch Scratching achieves a mean cumulative reward of 205.12 and task
success rate of 90.7%. Reacher achieves a relatively lower cumulative reward
of $-18.53$ ($-3.75$ being considered good), likely due to the fact that the
privileged features fail to account for action size, which the ground truth
reward penalizes.

Adding Spurious Features: The above results demonstrate that preference-based
reward learning works if we use a small set of hand-crafted features and a
linear reward function—indicating the occurrence of causal confusion. We now
study the effect of concatenating these privileged features with the features
from the raw observation space and performing reward learning on this
augmented feature-space. We add raw features based on the default order that
they are given in the environment, although methods that select features that
are important for reward learning could be explored.

Figure 3 displays a sensitivity analysis on the number of raw features in this
augmented feature space of both raw and privileged observations.
Interestingly, we find that the ability of the agent to successfully complete
its task drops off as more raw features are introduced, suggesting that
certain raw features likely exhibit spurious correlations with the true
privileged features. Note that the reward network still has access to the
privileged features as we progressively augment the feature space with raw
features; however, it appears that simply ignoring the concatenated raw
features when learning the reward is difficult, even in a linear network,
leading to causal confusion and poor policy performance.

We next inspected the learned linear reward models to understand which
features contributed the most to the learned reward functions. Interestingly,
in both the Feeding and Itch Scratching environments, the feature that
corresponds to the total amount of force applied by the spoon or scratching
tool is consistently given a large weighting factor in the learned reward
function when compared to other raw features. Specifically, the amount of
force applied by the spoon on the human is given a large weight in the Feeding
task, and the total amount of force applied by the scratching tool is given a
large weight in the Itch Scratching task. This phenomenon is likely a result
of the fact that all successful Feeding and Itch Scratching demonstrations
applied some force on the human (touching the mouth or the arm), whereas the
majority of failing demos simply dropped the food or swung the end effector
without applying any force on the human. This proves to be a somewhat
problematic proxy; as seen in Figure 4, rather than actually feed or scratch,
the robot learns the undesirable behavior of applying a large amount of force
to the human’s head to maximize the spoon force on human (Figure 4(a)) or
applying a large amount of force on its own base to maximize total tool force
(Figure 4(b)).

(a) Feeding

(b) Itch Scratching

Figure 4: Visualizing the spurious force feature. Figure 4(a): In the Feeding
task, the linear reward model learns a high weight on the amount of force
applied on the human. This leads to the robot learning to apply force to
various parts of the human (torso, head, etc.) rather than properly feeding.
Figure 4(b): In the Itch Scratching task, the linear reward model learns a
high weight on the total amount of force applied by the scratching tool. This
results in the robot preferring positions where large amounts of force are
registered by the scratching tool—here, the scratching tool is pressed against
the base of the Jaco robot.

Discussion: Our findings in this section demonstrate the need for caution when
performing reward learning. Using raw observational feature spaces originally
designed for RL tasks may not be sufficient for reward learning. In the three
tasks we considered, the raw observation spaces for RL introduce spurious
correlations that lead to causal confusion about the true reward function,
which results in poor performance when using RL to optimize the learned reward
function. Using hand-selected features results in good performance, but
designing the right set of features is difficult. On top of that, we find
evidence that increasing the number of features available to the reward
network can exacerbate causal confusion, likely because it increases the
probability that the learned reward network focuses on spurious correlations.

###  VI-B The Effect of Reward Function Model Capacity

We next study the effect of the model capacity for the reward function
approximator, $r_{\theta}$. Analyzing the effect of model capacity on causal
confusion is inspired by recent results in the reinforcement learning setting,
where Pan et al. [9] find that increased model capacity for the policy
network, often increases the likelihood of reward hacking behavior. By
contrast, we study the model capacity of the reward-learning network, rather
than the model capacity of the RL policy, and seek to understand how the size
of the reward function model affects performance.

Experimental Setup: Similar to the above feature-space sensitivity analysis,
we learn a reward function from 2000 ground-truth pairwise preference labels
over randomly selected trajectory pairs with a $\Delta_{pair}$ of 60. The
model is optimized using stochastic gradient descent with weight decay
($\lambda=0.01$) and regularized with $l_{1}$-regularization
($\lambda_{1}=0.01$). We use early-stopping with a patience of 10 epochs and
an upper limit of 100 epochs.

In order to measure the effect of reward network model capacity, we train the
following models with increasing order of complexity: linear model, one-layer
fully-connected neural network with a hidden dimension of 64, 128; two-layer
fully-connected networks with hidden dimensions of (64, 64), (128, 64), (128,
128), (256, 128) and (256, 256). For each model architecture, we train the
reward network on pairwise preferences, run the RL optimization for 3 seeds,
and evaluate the cumulative ground truth reward and task success on 100
rollouts from a validation seed.

(a) Feeding, Cumulative Reward

(b) Feeding, Task Success

Figure 5: Performance of preference learning as a function of reward network
capacity. Privileged Only denotes training the reward network on just the
privileged features defined in Table II, and Augmented denotes training the
reward network on the privileged features augmented with 10 raw features from
the Feeding environment.

Discussion: We find that, at least for augmented features, as the model
capacity of the reward-learning network increases, the mean task success rate
and cumulative ground truth reward the RL agent receives roughly decreases,
all else being equal. Figure 5 shows agent performance in each environment as
a function of reward-learning model capacity, trained with observations
consisting of either purely privileged features or a mixture of raw and
privileged features.

As expected, the models trained on just the privileged features perform better
than those trained on the augmented features, but, notably, the larger models
generally perform worse than the smaller models, especially in the augmented
feature space, where raw features are present. These findings go against the
common belief of increasing model size to enable better generalization that is
widely held in other machine learning domains [31, 32, 33]. Conversely, our
results suggest that in the robotic domains we consider, the increased
expressivity of the reward function does not enable better interpretation of
the raw features; rather, the increased expressivity appears to increase the
potential for inferring the wrong reward function, which the RL agent is then
able to game or hack to a greater degree. This poses serious implications for
the future; as models continue to grow in size and gain access to greater
amounts of compute, the potential for reward hacking when learning reward
functions is likely to also increase.

###  VI-C The Effect of Data Collection

Further, we explore the process through which we generate training data and
the differences in agent performance that can result. Specifically, we study
(1) how to generate trajectories for demonstrations, and (2) how to select or
sample pairwise preferences from these generated trajectories.

Trajectory Generation: To study the effect of training data on causal
confusion for preference-based reward learning, we examine two trajectory
generation methods:

  1. 1.

RL+noise. We first train an RL policy on a given ground truth reward, then
generates demonstrations by adding $\epsilon$-greedy noise to rollouts from
the trained policy. Demonstrations are labeled using the ground truth reward.

  2. 2.

T-REX. Brown et al. [5] suggests periodically checkpointing a policy trained
on the ground-truth reward can be used to generate demonstrations with varying
levels of performance. We also explore demonstrations provided via human-
teleoperation that purposefully provide a range of demonstration qualities.

Pairwise Preference Selection: Given a set of ranked trajectories, we study
two ways of creating pairwise preferences:

  1. 1.

Random $\Delta_{pair}$-sampling. We randomly sample pairs of trajectories with
replacement and discard pairs that are within $\Delta_{pair}$ ranks of each
other. We do this until we have the desired number of pairwise preferences. We
hypothesize that enforcing a $\Delta_{pair}$ difference within pairs ensures
that the differences between the two trajectories are more salient and thus
easier for the reward network to learn. Without this (say, if we compare two
similar trajectories), the reward network may try to disambiguate them using
some spurious or unimportant feature(s).

  2. 2.

Systematic all-pairs selection. We select $M$ evenly-spaced demonstrations
from the set of provided demonstrations and generate all possible $M\choose 2$
pairwise preferences. This method was originally proposed by Brown et al. [5]
and requires no additional hyperparameters; however, as noted above, we
hypothesize that using all pairs may lead to more causal confusion over the
true reward function.

(a) Synthetic T-REX vs. RL+noise

(b) Teleoperated T-REX vs. RL+noise

(c) Sensitivity of demos to raw features

Figure 6: Analysis of trajectory generation and pairwise preference selection
methods. Figure 6(a): In the Reacher Environment, we use random
$\Delta_{pair}$-sampling with $\Delta_{pair}=60$ to sample pairwise
preferences from synthetic T-REX and RL+noise. Figure 6(b): In the Feeding
environment, we use systematic all-pairs selection to generate preferences
from 20 human-teleoperated T-REX and RL+noise trajectories (the 120 demos case
uses $\Delta_{pair}$-sampling with $\Delta_{pair}=60$). Figure 6(c): We use 10
human-teleoperated demonstrations on (1) the raw features, (2) all the raw and
privileged features, and (3) 10 raw and all the privileged features.

Experimental Setup: In each of the four configurations (2 generation methods
$\times$ 2 preference selection methods), we train a linear model, using an
augmented feature space consisting of half of the raw observation features (10
in Feeding, 5 in Reacher) and all of the privileged features (3 in Feeding, 1
in Reacher) from each environment. We train the reward learning network for
100 epochs with early stopping (patience of 10), and add weight decay with
$\lambda=0.01$ and $l_{1}$-regularization with $\lambda_{1}=0.01$. In the
RL+noise trajectory generation method, we generate 20 rollouts at each level
of $\epsilon=[0.0,0.2,0.4,0.6,0.8,1.0]$. For the human-teleoperated T-REX
trajectories, we had the human demonstrator perform 8 demonstrations that
‘successfully complete the task’, 7 demonstrations that ‘fail at the task’,
and 5 demonstrations that ‘half-succeed, half-fail’ (eg., demonstration would
spill some food, then feed the rest in the case of the Feeding task), for a
total of 20 teleoperated demonstrations. To generate the T-REX trajectories
taken from a checkpointed policy, we take 20 rollouts at each of the following
checkpoints (expressed as a proportion of the total number of training
iterations): $[0.01,0.05,0.1,0.2,0.8,1.0]$. We use a fixed $\Delta_{pair}=60$
for our random $\Delta_{pair}$-sampling configurations.

Discussion: Figure 6 displays our analysis of the aforementioned trajectory
generation and preference selection methods when used in conjunction with one
another. Firstly, we observe in Figure 6(a) that performing preference
learning on using pairwise preferences over synthetic demonstrations generated
by adding $\epsilon$-greedy noise to a pretrained policy (RL+noise) performs
on par with, if not better than, the approach suggested by Brown et al. [5]
(T-REX) which takes rollouts from different partially trained policy
checkpoints and then providing pairwise preferences over these rollouts. The
implication is that we do not find any evidence that the causal confusion for
reward inference is due to the trajectory generation being too simplistic or
biased. Future work should examine whether active methods that obtain pairwise
preferences by synthesizing informative trajectory pairs from scratch [3] lead
to better performance.

Next, we note that human-teleoperated demonstrations that attempt to give a
diverse set of good, mediocre, and poor demonstrations enable high preference
learning performance in fewer demonstrations, as shown in Figure 6(b).
However, when given more demonstrations and sampled with $\Delta_{pair}=60$,
using synthetically-generated RL+noise trajectories approaches the performance
of using human-teleoperated demonstrations. This reinforces the previous
finding—at least for large data regimes, the trajectory generation method is
not the culprit of the poor reward inference performance demonstrated in the
main results.

Lastly, we observe that human demonstrations, though preferred over synthetic
demonstrations, can also lead to causal confusion over the true reward
function. In Figure 6(c), the mean cumulative reward increases with the
addition of privileged features and the removal of raw features from the input
observation of the reward network.

##  VII CONCLUSION

We present new benchmarks for studying causal confusion when learning rewards
from pairwise preferences over trajectories. We also provide the first
systematic study of factors that may lead to causal confusion for reward
learning. In particular, we find that spurious features and increased model
capacity lead to causal confusion over the true reward function, even when
learning from thousands of pairwise preferences. We hope that our empirical
study and open-source benchmarks and datasets will inspire and facilitate
future work on learning reward functions that are robust to causal confusion,
even when using high-capacity reward function approximators to learn complex
robotic tasks from high-dimensional observation spaces.

## ACKNOWLEDGMENTS

We thank the members of the InterACT lab for helpful discussion and advice.
This work was supported in part by the NSF NRI, ONR YIP, and NSF CAREER
awards.

## References

  * [1] C. Wirth, R. Akrour, G. Neumann, J. Fürnkranz, _et al._ , “A survey of preference-based reinforcement learning methods,” _Journal of Machine Learning Research_ , vol. 18, no. 136, pp. 1–46, 2017.
  * [2] P. F. Christiano, J. Leike, T. B. Brown, M. Martic, S. Legg, and D. Amodei, “Deep reinforcement learning from human preferences,” in _NIPS_ , 2017.
  * [3] D. Sadigh, A. D. Dragan, S. Sastry, and S. A. Seshia, “Active preference-based learning of reward functions.” in _Robotics: Science and Systems_ , 2017\.
  * [4] E. Biyik and D. Sadigh, “Batch active preference-based learning of reward functions,” in _Conference on robot learning_. PMLR, 2018, pp. 519–528.
  * [5] D. Brown, W. Goo, P. Nagarajan, and S. Niekum, “Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations,” in _International Conference on Machine Learning_. PMLR, 2019, pp. 783–792.
  * [6] B. Ibarz, J. Leike, T. Pohlen, G. Irving, S. Legg, and D. Amodei, “Reward learning from human preferences and demonstrations in atari,” _arXiv preprint arXiv:1811.06521_ , 2018.
  * [7] D. S. Brown, W. Goo, and S. Niekum, “Better-than-demonstrator imitation learning via automatically-ranked demonstrations,” in _Conference on robot learning_. PMLR, 2020, pp. 330–359.
  * [8] Z. Javed, D. S. Brown, S. Sharma, J. Zhu, A. Balakrishna, M. Petrik, A. Dragan, and K. Goldberg, “Policy gradient bayesian robust optimization for imitation learning,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 4785–4796.
  * [9] A. Pan, K. Bhatia, and J. Steinhardt, “The effects of reward misspecification: Mapping and mitigating misaligned models,” _arXiv preprint arXiv:2201.03544_ , 2022.
  * [10] P. De Haan, D. Jayaraman, and S. Levine, “Causal confusion in imitation learning,” _Advances in Neural Information Processing Systems_ , vol. 32, 2019.
  * [11] J. Zhang, D. Kumor, and E. Bareinboim, “Causal imitation learning with unobserved confounders,” _Advances in neural information processing systems_ , vol. 33, pp. 12 263–12 274, 2020.
  * [12] G. Swamy, S. Choudhury, J. A. Bagnell, and Z. S. Wu, “Causal imitation learning under temporally correlated noise,” _arXiv preprint arXiv:2202.01312_ , 2022.
  * [13] A. Y. Ng, D. Harada, and S. Russell, “Policy invariance under reward transformations: Theory and application to reward shaping,” in _Icml_ , vol. 99, 1999, pp. 278–287.
  * [14] V. Krakovna, J. Uesato, V. Mikulik, M. Rahtz, T. Everitt, R. Kumar, Z. Kenton, J. Leike, and S. Legg, “Specification gaming: the flip side of ai ingenuity,” _DeepMind Blog_ , 2020.
  * [15] J. Z.-Y. He and A. D. Dragan, “Assisted robust reward design,” _arXiv preprint arXiv:2111.09884_ , 2021.
  * [16] D. Brown, R. Coleman, R. Srinivasan, and S. Niekum, “Safe imitation learning via fast bayesian reward inference from preferences,” in _International Conference on Machine Learning_. PMLR, 2020\.
  * [17] S. Sidney, “Nonparametric statistics for the behavioral sciences,” _The Journal of Nervous and Mental Disease_ , vol. 125, no. 3, p. 497, 1957.
  * [18] M. G. Kendall, “Rank correlation methods.” 1948.
  * [19] N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano, “Learning to summarize with human feedback,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 3008–3021, 2020.
  * [20] D. Brown, S. Niekum, and M. Petrik, “Bayesian robust optimization for imitation learning,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 2479–2491, 2020.
  * [21] A. R. Dario Amodei, Paul Christiano, “Learning from human preferences,” _OpenAI Blog_ , 2017. [Online]. Available: https://openai.com/blog/deep-reinforcement-learning-from-human-preferences/
  * [22] D. A. Pomerleau, “Alvinn: An autonomous land vehicle in a neural network,” _Advances in neural information processing systems_ , vol. 1, 1988.
  * [23] F. Torabi, G. Warnell, and P. Stone, “Behavioral cloning from observation,” _arXiv preprint arXiv:1805.01954_ , 2018.
  * [24] M. L. Puterman, _Markov decision processes: discrete stochastic dynamic programming_. John Wiley & Sons, 2014\.
  * [25] R. A. Bradley and M. E. Terry, “Rank analysis of incomplete block designs: I. the method of paired comparisons,” _Biometrika_ , vol. 39, no. 3/4, pp. 324–345, 1952.
  * [26] S. Levine, A. Kumar, G. Tucker, and J. Fu, “Offline reinforcement learning: Tutorial, review, and perspectives on open problems,” _arXiv preprint arXiv:2005.01643_ , 2020.
  * [27] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” _arXiv preprint arXiv:1707.06347_ , 2017\.
  * [28] T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel, _et al._ , “Soft actor-critic algorithms and applications,” _arXiv preprint arXiv:1812.05905_ , 2018.
  * [29] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba, “Openai gym,” _arXiv preprint arXiv:1606.01540_ , 2016.
  * [30] Z. Erickson, V. Gangaram, A. Kapusta, C. K. Liu, and C. C. Kemp, “Assistive gym: A physics simulation framework for assistive robotics,” in _2020 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2020, pp. 10 169–10 176.
  * [31] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [32] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [33] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, _et al._ , “Language models are few-shot learners,” _Advances in neural information processing systems_ , vol. 33, pp. 1877–1901, 2020.