# Probe-Based Interventions for Modifying Agent Behavior

Mycal Tucker1 William Kuhl1 Khizer Shahid1 Seth Karten2 Katia Sycara2 Julie
Shah1 1Massachusetts Institute of Technology
2Carnegie Mellon University
{mycal<EMAIL_ADDRESS>{bkuhl<EMAIL_ADDRESS>{skarten,
<EMAIL_ADDRESS>
###### Abstract

Neural nets are powerful function approximators, but the behavior of a given
neural net, once trained, cannot be easily modified. We wish, however, for
people to be able to influence neural agents’ actions despite the agents never
training with humans, which we formalize as a human-assisted decision-making
problem. Inspired by prior art initially developed for model explainability,
we develop a method for updating representations in pre-trained neural nets
according to externally-specified properties. In experiments, we show how our
method may be used to improve human-agent team performance for a variety of
neural networks from image classifiers to agents in multi-agent reinforcement
learning settings.

##  1 Introduction

We envision a world in which autonomous agents learn patterns from data on
their own but, when given human guidance, they respond appropriately. Such
agents would enable high individual and team performance, as well as bridge
the gap between data-driven and symbolic reasoning.

Neural nets are a natural architectural choice for such agents. As universal
function approximators, with enough data, they can learn correlations to map
from inputs to desired outputs. As a result, they have been applied in a wide
range of contexts, including image recognition, self-driving cars, and
communication domains PyTorch (2022); Singh et al. (2018). Any single neural
net is trained to map from its inputs (e.g., sensor data from a self-driving
car) to outputs (e.g., throttle control).

While neural networks generally are quite flexible, any given neural net, once
trained, acts as a black box that maps from inputs to outputs. For example, a
neural net on a self-driving car that normally accepts images cannot accept
verbal commands at test time. However, we wish for agents to act more like
humans, who can change their behaviors based on new sources of information.
The challenge is to do so with pre-trained neural net agents that have never
been trained to accept the new form of input.

Figure 1: If a neural agent (blue, left) fails to see a nearby car (red, top),
it may act unsafely. We enable humans to inject information into
representations, modifying behavior without fully dictating it.

In this work, the key insight is that, if a neural net already models some
concept in its representation space (e.g., “there is another car nearby”), the
problem of “injecting” information becomes one of updating that symbolic
representation. While neural representation spaces may be difficult for humans
to understand, one may train separate neural net “probes” to map from
representations to symbolic properties Alain and Bengio (2016); Tucker et al.
(2021). Lastly, one may use these probes to update representations according
to a desired property. For example, as depicted in Figure 1, one may train a
probe to predict, from a representation, if there is another car nearby, and
then use that probe to update the representation to reflect the fact that
there actually is a car that the neural net failed to notice.

In this paper, we make three contributions. First, we define the problem of
“human-assisted decision-making” to formalize an intuitive desire of
leveraging both human and agent observations in decision-making settings.
Next, we describe how techniques that had earlier been developed for neural
net explanations may be re-purposed to solve our problem. Third, we propose an
improvement over these techniques that mitigates a limitation of prior art,
based on a theory of redundant encodings. In experiments, we show how our
method can be used to improve the performance of an image classifier (without
retraining the classifier), prevent crashes in a simulated road intersection
environment in which cars fail to see each other, or enabled flawed agents in
a collaborative grid-world environment to find each other.

##  2 Related Work

###  2.1 Interpretability and Explainability

Interpretability and explainability techniques focus on developing models that
are easily understandable, or techniques to better understand pre-trained
models Molnar (2019).

On the one hand, through special architectural or training choices, one may
train agents with interpretable representation or decision-making systems. For
example, in models trained with concept-whitening, it is easy to identify
which neuron activations correspond to which concepts Chen et al. (2020). More
generally, techniques that shape representation spaces according to human
design principles (e.g., prototype-based classification, disentanglement,
etc.) can be used to support human understanding. In some domains, like style-
transfer, interepretable representations support mixing aspects of different
representations to produce desired outputs John et al. (2019).

On the other hand, other research attempts to create human-understandable
explanations of pre-trained neural networks with minimal assumptions about how
those networks have been trained. For example, post-hoc explanation techniques
like LIME produce linear approximations of neural network decision boundaries
Ribeiro et al. (2016). Other methods train simple classifiers to expose what
information is present in neural representations, but not how such information
is used Alain and Bengio (2016).

These techniques are useful for understanding neural networks, but differ from
our work in two ways. First, unlike interpretability research, we do not
assume that we can train the neural net model. Second, we fundamentally seek
to modify neural net behavior as opposed to only understanding it.

###  2.2 Causal Analysis of Neural Networks

Our work most directly builds upon work previously developed for causal
analysis of neural models. In causal analysis literature, researchers seek to
identify how models use particular properties in their decision-making. For
example, an image classifier may often correctly classify images of dogs, but
causal analysis may reveal that the model changes its predictions if a single
input pixel changes. This is an important causal property of the model.

Researchers have applied causal analysis techniques both on input-space
features (e.g., if this pixel changes, how does the model prediction change)
and latent features Zeiler and Fergus (2014). We focus here on latent feature
analysis. One technique, quantitative testing of concept activation vectors
(TCAV), measures how a neural network’s predictions change when a
representation is updated according to a conceptual property Kim et al.
(2018). For example, the authors find a direction in the latent space that is
correlated with images of striped objects, and they find that if they move
representations along that direction, they increase the probability of the
model predicting zebras. In domains in which humans know what the “right”
causal path for a particular output is, this provides a useful quality check.

More recently, researchers similarly used causal analysis techniques to study
if large language models use syntactic properties in predictions. While
similarly motivated to TCAV - they wish to understand what latent properties
models use in making predictions - they adopted more general frameworks.
Applying multiple linear classifiers appeared to induce better effects than
the single classifier used by TCAV Elazar et al. (2020). Furthermore, Tucker
et al. Tucker et al. (2021) developed a method to use non-linear classifiers
to extract and modify syntactic properties. Lastly, Giulianelli et al. (2018)
showed how one could update representations in LSTM-based language models and
improve subject-verb agreement in sentences.

We focus on modifying agent behavior rather than just understanding how they
act. We implement the technique proposed by Tucker at al. Tucker et al.
(2021), which encompasses TCAV as a special case, and use them as baselines
for generating updated embeddings in non-language settings. Lastly, we show
how a new design further improves upon these baselines, enabling better
control of agent behavior.

##  3 Human-Assisted Decision Making Problem Formulation

We define a new problem, “human-assisted decision-making,” based on the
Decentralized Partially-Observable Markov Decision Process framework in which
different agents only observe subsets of the true state Bernstein et al.
(2013). At training time, $N$ artificial agents train to maximize some shared
reward by learning policies ($\pi_{1}$ to $\pi_{N}$) that map from
observations of the true state, $S$, to actions. Reward is a function of the
joint state and actions taken by all agents:
$R(S,\pi_{1}(o_{1}),\pi_{2}(o_{2}),...)$.

At test time, we define a human participant with an associated observation
function, $O_{h}:S\rightarrow o_{h}$. Human observations, an individual
agent’s policy, and the agent’s observations, may be combined into a “hybrid”
policy, $h:o_{h},o_{i},\pi_{i}\rightarrow a_{i}$. That is, the hybrid policy
uses the human’s observation, the agent’s observation, and the agent’s policy
to produce a new action. This parallels human supervisory control of automated
systems frameworks in which a human monitors an agent’s output and decides
when to intervene, but the roles of the human and agent are reversed.

The human-assisted decision-making problem is therefore one of generating a
“good” hybrid policy, $h$, that reconciles human and agent observations within
an agent policy. The aim is to generate a hybrid policy that maximizes
$R(S,h_{1}(o_{h},o_{1},\pi_{1}),\pi_{2}(o_{2}),...,\pi_{N}(o_{N}))$. This
paper frames the human-assisted decision-making problem in reinforcement
learning contexts, but the framework may be adapted to other settings.

##  4 Technical Approach

We proposed probe-based interventions as an approach to the human-assisted
decision-making problem. Our method relies upon using human observations to
update the internal representations of neural agent policies.

###  4.1 Probe Design and Training

For simplicity, we describe how to use probe-based interventions in feed-
forward image classifiers, but our experiments illustrate the value of this
approach for a variety of domains and neural architectures.

We characterize a feedforward neural network model, $M$, as composed of $N$
layers, which map from input image $x\in R^{X}$ to an output $y\in R^{Y}$. We
denote the representation generated after layer $k$ as $z_{k}\in R^{Z}$. $M$
may be thought of as two neural networks: $z_{k}=M_{k-}(x)$ and
$y=M_{k+}(z_{k})$ where the subscripts for $M$ denote the indices of layers
before or after layer $k$.

A probe is defined as a neural network mapping from representations generated
at layer $k$ to predictions of a property, $s\in R^{S}$. For example, a probe
may predict, given a representation generated by an image classifier, if the
image was originally of an animal or a vehicle. The probe may be a linear
classifier, but it may equally well be a non-linear, multi-layer neural
network.

The probe may be trained using a dataset of $(z_{k},s)$ pairs. In TCAV,
although not explicitly framed as training a neural net, a linear probe is
trained with embeddings generated by supplying a dataset of input images
describing some concepts, and those images are encoded to generate the
corresponding $z_{k}$ Kim et al. (2018). In other works studying language
models, non-linear probes are trained to map from representations of sentences
in language models to aspects of the syntactic structure of the sentence
Tucker et al. (2021). Fundamentally, these probes may be trained just like any
other neural network to map from representations to predictions.

###  4.2 Counterfactual Generation

Next, we show how to use such probes to update representations. Adopting the
method of Tucker et al. (2021), “counterfactual” representations,
$z^{\prime}$, are created by updating an original encoding, $z$, along the
gradient of the loss function, $L$, computed on the output of the probe, $p$,
and a desired property, $s^{\prime}$. That is, starting at $z^{\prime}=z$, we
incrementally update as follows, with step size $\delta$:

$z^{\prime}=z^{\prime}+\delta\nabla_{z^{\prime}}L(p(z^{\prime}),s^{\prime})$
(1)

One may apply such updates until a stopping criterion, such as number of steps
or minimum probe loss, is reached. To then evaluate the effect of using
counterfactual embeddings, one may pass $z^{\prime}$ through the remainder of
the model to generate a counterfactual outputs:
$y^{\prime}=M_{k+}(z^{\prime})$. If the original output, $y$, and $y^{\prime}$
systematically differ, one may say that $M$ uses a representation that encodes
information about $s$.

While this method is borrowed from Tucker et al. Tucker et al. (2021), other
techniques similarly use gradient updates, although only for linear
classifiers. TCAV measures, in the limit of small gradient steps, the change
in model prediction with respect to the change in embedding. Giulianelli
Giulianelli et al. (2018) perform a single gradient update of fixed magnitude.

Figure 2: A graphical depiction of generating counterfactual embeddings. The
first $k$ layers of an agent generate an embedding, $z$ (depicted here in 2D
for simplicity). A counterfactual embedding, $z^{\prime}$, is generated by
moving in the embedding space along the gradient of the probe loss until the
probe predicts “Car.” Here, the probe is depicted as a linear classifier, but
it may be a multi-layer neural network. Lastly, $z^{\prime}$ is passed through
the rest of the agent to produce an action or decision.

In this work, we use Tucker et al.’s Tucker et al. (2021) incremental,
gradient-based method for updating representations. We note that, because this
method depends upon gradients calculated through trained probes, different
probe designs may produce different types of updates.

Fundamentally, this method for counterfactual generation is our proposed
approach to the human-assisted decision-making problem. We use probes to
update the original representations generated by the agent to reflect new
information supplied by the human, and we pass the resulting representation
through the remainder of the agent’s policy.

###  4.3 Dropout Probes

While the existing methods develop a gradient-based mechanism for creating
counterfactual representations according to a trained probe, they could be
susceptible to probe overfitting, which can produce undesirable counterfactual
updates.

In particular, consider an image classifier that encodes a single property -
is the input image an animal or a vehicle - in a redundant manner. For
example, if the representation vector is 32-dimensional but the model encodes
the animal vs. vehicle property identically in two neurons, then a probe could
arbitrarily learn to use either neuron in making its prediction.
Unfortunately, even if the model also used representations of animal vs.
vehicle information, if the model used the other, redundant neuron, creating
counterfacutal embeddings according to the probe would not affect model
behavior. Prior art has hinted at this possibility: TCAV, for example,
instantiates many linear classifiers to see if, on average, they produce an
effect, and Elazar et al. Elazar et al. (2020) use an iterative approach to
create multiple classifiers. This can be thought of as a mechanism to ensure
greater alignment between probe and model.

Instead of using a suite of classifiers, we instead propose the simple
modification of adding a dropout layer before probes. During training, the
dropout layer randomly masks a subset of inputs with some probability $p$.
When generating counterfactual embeddings, dropout may be disabled.

With dropout, a probe must learn to use all subsets of representations that
are informative of the predicted property, $s$. Thus, we hypothesized that
$\nabla_{z^{\prime}}L(p(z^{\prime}),s^{\prime})$ (the gradient along which one
updates $z^{\prime}$), is less likely to be orthogonal to the model’s decision
boundary’s gradient, and therefore that creating counterfactuals using dropout
probes would change model behavior more than when using standard probes.

##  5 Experiments

We conducted three suites of experiments to evaluate: 1) the ability of probe-
based interventions to affect neural net behavior and 2) the advantage of
using dropout in probes. First, we demonstrated how to boost an image
classifier’s performance be injecting symbolic information about high-level
categories in the data. Second, in a traffic-junction environment, we showed
how probe-based interventions supplied by humans could prevent collisions even
if cars didn’t directly observe each other. Lastly, in a multi-agent
reinforcement learning environment, in which the agents needed to communicate
to succeed at the task, we showed how our interventions enabled us to inject
information into agents without needing to understand all aspects of the
learned policies.

Experiments followed a standard procedure: 1) an agent was trained on the
primary task (e.g., image classification) and then frozen, 2) a probe was
trained on an auxiliary task to map from representations to a desired
property, 3) at test time, we used the trained agents and probes to update
embeddings and modify behavior. Updated embeddings were generated using an SGD
optimizer with learning rate 0.1 and momentum 0.9 for a fixed number of
timesteps (specified in experiments) or until the probe loss was below 0.001.

In each experiment setup, we performed 5 trials by training new agents from
scratch, and we studied the effect of dropout rate by training probes with
different dropout rates, from 0 to 0.9 at increments of 0.1, for the same
model. Anonymized code for some experiments is available online; complete code
will be edited for clarity and released upon
acceptance.111https://bit.ly/31xqbbf

###  5.1 Image Classifier Boosting

We showed that we could improve an image classifier’s performance by injecting
hierarchical information into representations, and that using dropout was an
important part of boosting performance.

First, following a standard online tutorial for an image classifier, we
trained a 6-layer convolutional neural net (CNN) on the CIFAR10 dataset,
achieving roughly 50% percent accuracy Krizhevsky et al. ; PyTorch (2022).
Although this does not match state of the art classification performance on
CIFAR, it follows standard practices and allows us to demonstrate how our
method allowed use to improve model performance.

Probes were trained on a binary classification task: given representations
taken from the third layer of the image classifier, the probe was trained to
predict if the image was one of an animal or a vehicle. At test time, we
sought to improve the main image classifier’s accuracy by injecting the
“animal vs. vehicle” information into the representations. We compared the
effect of using probes with different numbers of layers and different dropout
rates. Our results, generated across 5 trials for different classifiers and
probes and evaluated over 1000 test images, are plotted in Figures 3 and 4.

[trim=0.12cm 0.0cm 0.0cm 0.0cm, clip=true]figures/tikz/cifar_numsteps

Figure 3: Updating representations in the CIFAR10 classifier based on
animal/vehicle class, using 1-layer probes, improved model accuracy.
Interventions produced the greatest benefits when using high-dropout probes
($x$ axis) for many gradient steps (different lines).

[trim=0.12cm 0.0cm 0.0cm 0.0cm, clip=true]figures/tikz/cifar_probedepth

Figure 4: Using multi-layer dropout probes for interventions in the CIFAR10
classifier produced greater benefits than linear probes. As before,
introducing dropout also boosted performance.

Several important trends are visible in Figure 3. First, interventions using
any probe design improved classification accuracy from the normal performance
of around 51% to 53%. We emphasize that this improvement occurred despite no
classifier retraining. Second, updating representations for more gradient
steps induced greater benefits. This is visible via the different lines in the
plot: results when updating for 600 steps achieved greater accuracy than when
updating for 400, which outperformed only 200 steps. Lastly, adding dropout to
the probes consistently improved the effect of interventions. Updating
embeddings using probes with no dropout, as introduced in prior art (Kim et
al. (2018); Tucker et al. (2021)), corresponded to the leftmost data in the
plot, with dropout 0. The benefit of using dropout was most obvious when using
600 steps: model performance was further increased from 53% accuracy (leftmost
points) to 54%.

Lastly, we investigated the effects of using dropout and multi-layer probes
for interventions and plotted the results in Figure 4. In these experiments,
all interventions were performed used 600 gradient update steps, but with
probes using 1, 2, or 3 layers. The multi-layer probes used ReLU activations
for the hidden layers, allowing them to capture non-linearities in the
representation space. As before, dropout boosted performance. Furthermore, the
two-layer probe dramatically outperformed the linear probe. Interestingly, the
three-layer probe did not show as large a benefit. Given these results, we
hypothesize that using non-linear probes is beneficial (confirming findings by
Tucker et al. Tucker et al. (2021)), but that there may be a limit to the
benefits of larger probe models. The TCAV baseline, which used a linear
approach, corresponded to the bottom, 1-layer method.

Overall, these experiments confirmed two important trends in a classification
domain: 1) using probes to inject information into representations
consistently boosted performance and 2) non-linear dropout probes performed
better than linear probes with no dropout.

###  5.2 Avoiding Collisions in Traffic Junction

Based on the introduction’s self-driving car example, we next demonstrated how
probes could be used to prevent collisions in a simulated driving scenario.
Neural agents were trained in a 2D grid setting (borrowed from prior art) to
drive through an intersection without colliding, with up to 5 cars driving in
the grid at the same time Singh et al. (2018). Each agent was a feed-forward
three-layer neural network mapping from inputs (current location and occupancy
of the 8 neighboring grid cells) to a binary action (go/stop). Agents received
positive reward for driving through the intersection safely and 0 reward for
collisions. Over 3000 training episodes using REINFORCE, agents converged to
policies in which they collided less than 1% of the time Williams (1992).

We trained 3-layer probes to predict, given the representations output by
agents’ second layers, whether another car was also adjacent to the
intersection. Based on the benefits of non-linear probes found in the prior
section, we experimented with 2- and 3-layer probes and found no substantial
difference, so we opted for the theoretically more powerful 3-layer probes. At
test time, we changed the environment to demonstrate the benefits of probe-
based interventions. We artificially corrupted one car’s observations by
programatically altering all observations to mask any nearby cars there. For
simplicity, we tested with only two cars at a time: one corrupted and one
normal.

In this scenario, only 86% of trials completed without a collision - where a
collision was defined as the two cars entering the intersection
simultaneously. We then re-evaluated agent success rates when we used probes
to update the representation of the corrupted car to reflect the true
occupancy of the roads. We performed interventions for 50 gradient updates at
each timestep in the episode, evaluated over 100 episodes.

For all dropout rates from 0.0 to 0.8, we found no significant difference in
the effect of interventions. Across the 5 trained teams, the median
performance when using interventions always exceeded 99% accuracy - matching
the non-corrupted environment performance. The only exception was probes with
dropout of 0.9, which achieved a success rate of 94%. This reinforces the
pattern observed in the image classification experiments: some dropout may
help, but if the dropout rate is too high, the probes fail to learn the right
pattern, so interventions are not useful.

###  5.3 Parent-Child

In our final suite of experiments, we used a “Parent-Child” domain - a
collaborative version of predator-prey from prior literature - to study the
effect of probe-based interventions when humans were unaware of all
information needed to produce the right action Singh et al. (2018).

In this two-agent setting, a stationary child and a movable parent were
spawned in random locations in a $5\times 5$ grid. As depicted in Figure 5,
the parent observed its location and the surround cell locations at all
timesteps; the child only observed its location at timestep 0. At the start of
each episode, each grid location was populated by an obstacle with 10%
probability. The agents received shared reward of 1 if the parent ended the
episode (after 20 timesteps) at the child’s location and otherwise got reward
0. If the parent collided with an obstacle, it could not move for the rest of
the episode.

Agents were instantiated using the LSTM-based IC3Net architecture, allowing
them to communicate with each other agent at all timesteps, and trained with
REINFORCE Singh et al. (2018); Williams (1992). The parent could move up,
down, left, right, or stay still; the child could not move. Thus, the optimal
policy was for the child to communicate its location to the parent and for the
parent to navigate to the child, avoiding obstacles along the way.

We trained 5 parent-child teams to convergence using different random seeds to
convergence at 100% success rates. For a given team, for both the parent and
child, we trained $h$ and $c$ probes that accepted the LSTM’s hidden or cell
states, respectively. The parent probes were trained to predict the obstacle
occupancy grid of the cells surrounding the parent; the child probes were
trained to predict the child’s location in that 2D world. To gain further
statistical power, for each parent-child team, we trained probes from scratch
5 times with different random seeds. Overall, given the 5 pairs of agents, 2
types of probes per agent, 5 random seeds for probes, and 10 dropout rate
levels, this corresponded to 500 trained probes.

As in the traffic junction experiment, we created “corrupted” versions of
these environments to see if probes could be used to overcome perceptual
failures of the agents. For consistency, all corrupted versions started with
the same initial conditions: the child in the upper right and the parent in
the lower left. In corrupting the child’s perception, we replaced the
observation of its true location with data from the upper left. In corrupting
the parent’s perception, we masked all observations of obstacles. We then
tested if, using our probes, we could remedy child corrupted data alone,
parent corrupted data alone, or even both simultaneously. We used both the $h$
and $c$ probes to update both representations when performing interventions.

[trim=0.12cm 0.0cm 0.0cm 0.0cm, clip=true]figures/tikz/lost_child

[trim=0.12cm 0.0cm 0.0cm 0.0cm, clip=true]figures/tikz/lost_parent

Figure 5: In the parent-child environment, the blindfolded parent (P) did not
observe nearby obstacles, and the lost child (C) observed its location as
$(0,0)$ even when located at $(0,4)$. In “lost-child” settings (middle),
interventions caused the parent to find the child with nearly 80% accuracy,
whereas without interventions the success rate was 0%. In the “blindfolded-
parent” setting, interventions similarly boosted performance. Especially with
the blindfolded-parent, probes with dropout between 0.5 and 0.8 often
outperformed probes without dropout.

Our results in the lost-child settings are reported in Figures 5. In this
experiment, wherein the child was actually at $(4,0)$ but observed its
location as $(0,0)$, the baseline performance without interventions was 0%:
that is, the parent never found the child. Conversely, using interventions, we
increased the success rate to up to 80%, as shown in Figure 5. On the $x$
axis, we varied the dropout rates of the probes, on the $y$ axis we measured
the team success rates. The different curves correspond to intervening on the
child’s representations for the first timestep, first and second timestep,
etc.. For all probe types, intervening for more timesteps always helped:
however, intervening for only the first 2 timesteps with probes with dropout
rate 0.9 was still enough to have the parent reach the child nearly 80% of the
time.

Performance as a function of dropout was more subtle, though: small but
positive dropout rates seemed to worsen behavior when only performing
interventions for one or two timesteps, but as the dropout rate increased
above 0.5, interventions became more effective than when not using dropout.
This non-linear behavior is surprising and warrants further investigation in
future work.

Similarly, we measured success rate as a function of probe dropout rate in the
blindfolded-parent environment. Without interventions, because the parent
could not observe obstacles, roughly 50% of episodes resulted in collisions
and therefore failures. Conversely, when we intervened to update the parent
representations, especially when using dropout probes, we decreased the
likelihood of collisions and increased the success rate. In addition to
varying our analysis by dropout rate, we also investigated the effect of more
gradient steps when updating representations and plotted the results as
different curves. For all dropout rates, more gradient steps produced better
effects, although the effect was greatest for dropout rates between 0.1 and
0.8.

Lastly, in experiments conducted in which both the parent and child
observations were corrupted, we demonstrated that probe-based interventions on
both agents mitigated the observation failures, but that higher-dropout probes
were most effective. Success rate without interventions was 0%, but reached
72% for probes with dropout 0.7. Full results are omitted for brevity but are
available in Appendix B.

##  6 Discussion

Jointly, the results from the prior section demonstrate important benefits of
test-time interventions on representations. First, we showed how an imperfect
image classifier could be improved by injecting high-level information into
representations. This indicates a possible strategy for companies seeking to
improve AI performance without retraining a model from scratch. Second, in the
traffic junction experiment, we demonstrated that, in simple cases where the
injected information and expected behavior had a clear relationship, we could
induce the right behavior. Although simplified, this demonstrates how humans
might be able to affect the behavior of AI systems that they cannot directly
control.

Lastly, in the parent-child environment, we showed more subtle capabilities.
Despite not understanding the emergent communication protocol, humans were
able to interact with agents by only updating the relevant parts of
representations. This therefore illustrates the value of our method in cases
in which humans may want to guide agent behavior without dictating all aspects
of it. For example, in the blidfolded-parent scenario, the human could
indicate the presence of obstacles that should be avoided without having to
know what the goal location was (and therefore whether the parent should avoid
an obstacle by going above or below it).

##  7 Conclusion

In this work, we proposed probe-based interventions as a mechanism for
inserting information into neural model representations - a potential approach
to the human-assisted decision-making problem. We found that prior methods,
initially developed for model explanability work could be adapted to this
problem, and we showed how a new, dropout-based intervention method often
outperformed other techniques. We tested our method in a variety of contexts,
from image classification to emergent communication domains, demonstrating the
wide applicability of our approach. While we have taken an initial step
towards supporting human interventions, future work could develop more
sophisticated intervention methods or evaluate the tradeoffs between minimal
and sufficient changes to representations. Lastly, we are particularly
interested in continuing work to train models that better respond to
interventions.

## References

  * Alain and Bengio [2016] Guillaume Alain and Yoshua Bengio.  Understanding intermediate layers using linear classifier probes.  arXiv preprint arXiv:1610.01644, 2016.
  * Belghazi et al. [2018] Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm.  Mutual information neural estimation.  In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 531–540. PMLR, 10–15 Jul 2018\.
  * Bernstein et al. [2013] Daniel S Bernstein, Shlomo Zilberstein, and Neil Immerman.  The complexity of decentralized control of markov decision processes, 2013\.
  * Chen et al. [2020] Zhi Chen, Yijie Bei, and Cynthia Rudin.  Concept whitening for interpretable image recognition.  Nature Machine Intelligence, 2(12):772–782, 2020.
  * Elazar et al. [2020] Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg.  Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals.  arXiv e-prints, page arXiv:2006.00995, June 2020.
  * Giulianelli et al. [2018] Mario Giulianelli, Jack Harding, Florian Mohnert, Dieuwke Hupkes, and Willem Zuidema.  Under the hood: Using diagnostic classifiers to investigate and improve how language models track agreement information.  In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 240–248, Brussels, Belgium, November 2018. Association for Computational Linguistics.
  * John et al. [2019] Vineet John, Lili Mou, Hareesh Bahuleyan, and Olga Vechtomova.  Disentangled representation learning for non-parallel text style transfer.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 424–434, Florence, Italy, July 2019. Association for Computational Linguistics.
  * Kim et al. [2018] Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al.  Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav).  In International conference on machine learning, pages 2668–2677. PMLR, 2018.
  * [9] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton.  Cifar-10 (canadian institute for advanced research).
  * Molnar [2019] Christoph Molnar.  Interpretable Machine Learning.  2019\.
  * PyTorch [2022] PyTorch.  Training a classifier, 2022.
  * Ribeiro et al. [2016] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin.  Model-agnostic interpretability of machine learning.  arXiv preprint arXiv:1606.05386, 2016.
  * Singh et al. [2018] Amanpreet Singh, Tushar Jain, and Sainbayar Sukhbaatar.  Learning when to communicate at scale in multiagent cooperative and competitive tasks.  arXiv preprint arXiv:1812.09755, 2018.
  * Tucker et al. [2021] Mycal Tucker, Peng Qian, and Roger Levy.  What if this modified that? syntactic interventions with counterfactual embeddings.  In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 862–875, Online, August 2021. Association for Computational Linguistics.
  * Williams [1992] Ronald J Williams.  Simple statistical gradient-following algorithms for connectionist reinforcement learning.  Machine learning, 8(3):229–256, 1992.
  * Zeiler and Fergus [2014] Matthew D Zeiler and Rob Fergus.  Visualizing and understanding convolutional networks.  In European conference on computer vision, pages 818–833. Springer, 2014.

##  Appendix A Redundancy Analysis

The main paper motivates dropout probes from a theory of redundant
representations; here, we use a technique from prior art to demonstrate that
such redundancy existed in one of our experiments.

We define a measure of property-specific representational redundancy as
follows:

$R(S,Z,Z_{1},Z_{2})=I(S,Z_{1})+I(S,Z_{2})-I(S,Z)$ (2)

That is, redundancy, $R$, in nats, is equal to the difference of two mutual
information terms. The first, $I(S,Z_{1})+I(S,Z_{2})$, equals the sum of the
mutual information between $Z_{1}$ and $S$ and $Z_{2}$ and $S$ where we let
$S$ be a random variable describing a property of interest, and $Z_{1}$ and
$Z_{2}$ are mutually-exclusive and collectively exhaustive subsets of a $Z$.
The second term is merely the mutual information between a representation,
$Z$, and $S$.

We applied this metric to study parents in the parent-child domain. $Z$ was
defined as the 128-dimensional hidden state; $Z_{1}$ and $Z_{2}$ were the
first and second halves of $Z$, respectively. Lastly, $S$ was the
8-dimensional binary vector representing the presence of obstacles in the
cells surrounding the parent. In other words, we sought to measure how much
“the information about nearby obstacles that is in the whole representation is
copied in the first and second halves.”

Analytically calculating the terms for computing $R$ is challenging, but a
technique dubbed MINE may be used to quickly approximate pairwise mutual
information Belghazi et al. [2018]. In essence, the technique trains a neural
net to maximize a lower bound on the mutual information between pairs of
variables.

We therefore employed MINE to calculate the 3 terms necessary for measuring
the redundancy of obstacle information in parents’ hidden states. Averaged
over the 5 trained teams, $I(S,Z_{1})\approx I(S,Z_{2})=0.7$ and $I(S,Z)=0.8$.
(All standard deviations were less than 0.1.) Therefore, as measured in nats,
$R(S,Z,Z_{1},Z_{2})=0.7+0.7-0.8=0.6$. That is, there are 0.6 nats of common
information about the presence of obstacles in the first and second halves of
$Z$. This proves that the information is encoded redundantly, which in turn
motivates dropout probes for interventions.

We repeated such experiments for the latent representations used in the
traffic experiment (for $S$ denoting if another car was near the intersection)
and for the hidden and cell state activations for both parent and child in the
parent-child settings (for $S$ denoting nearby obstacles for parents and the
child’s location for children). In all cases, we found positive redundancy,
indicating that this phenomenon is widespread.

Here, we have only shown that redundancy is positive given our choice of
$Z_{1}$ and $Z_{2}$ as the first and second halves of a representation. This
suffices for a proof by existence, but other work defining different subsets
of representations could better characterize redundancy.

##  Appendix B Parent-Child Full Results

In the main paper, we omitted the full results of experiments in which both
the parent and child observations were corrupted. Here, we include the full
results in Figure 6.

[trim=0.12cm 0.0cm 0.0cm 0.0cm, clip=true]figures/tikz/both

Figure 6: When both the child and parent observations were corrupted, probe-
based inerventions improved over baseline performance, particularly when using
higher-dropout probes ($x$ axis) for more gradient steps (different curves).

The baseline success rate, without interventions, was zero. However, with
interventions, the parent found the child up to 72% of the time. This
indicates that the high-dropout probes were able to update both the child’s
representation of its location and the parent’s representation of nearby
obstacles. This hints at a sort of compositionality of interventions.

##  Appendix C Implementation Details

The CIFAR10 classifier was trained per the hyperparameters specified in the
online tutorial Krizhevsky et al. ; PyTorch [2022].

The traffic agents were trained using the IC3Net architecture for feedfoward
agents Singh et al. [2018]. Each agent comprised 4 dense layers with hidden
dimension 128; weights were shared among agents. Agents were trained with an
RMSProp optimizer with learning rate 0.001, $\alpha=0.97$, and
$\epsilon=10^{-6}$, the default values provided with IC3Net, for 3000
episodes.

The parent-child agents were trained using the IC3Net architecture for LSTM-
based agents Singh et al. [2018]. Agents shared weights and were parametrized
with 2 dense layers encoding observations with hidden dimension 128 and an
LSTM cell with 128-dimensional $c$ and $h$ cells. Agents were trained for 1000
episodes, with the same optimizer as the one for the traffic experiments.