To illustrate the basic features of this formulation we construct a model1 of a simplified Trust Task based on the multi-round Investor-Trustee game (King-Casas et al., 2008). I (*self*) am to play consecutive rounds with the Trustee (*other*). At each round *t* I earn a wage *wself* , usually set at 20 units of play money. I can then invest one of a discrete set of fractions *f self* , *low*,..., *f self* , *high* of my wage in a joint venture with the *other*. The investment is multiplied by a gain *g*, representing the surplus value created by the joint venture (usually *g* = 3). The *other* then returns a fraction of the *invested* amount. The round ends with the following returns:

$$\begin{array}{l}r\_t^{\text{self}} = \boldsymbol{\nu}^{\text{self}} - \boldsymbol{\nu}^{\text{self}}f^{\text{self}} + \boldsymbol{\nu}^{\text{self}}f^{\text{self}}f^{\text{other}}\\r\_t^{\text{other}} = \boldsymbol{\nu}^{\text{self}}f^{\text{self}}\boldsymbol{g} - \boldsymbol{\nu}^{\text{self}}f^{\text{self}}f^{\text{other}}\end{array} \tag{6}$$

1Here we present the model step-by-step; see also the Discussion section regarding the rationale behind specific modeling choices.

Our Trust-Task is simpler than the standard Investor-Trustee game, with respect to the levels of investment and repayment available to the players. We make available only two levels, thus rendering a matrix representation of the exchange more straightforward and allowing experimenters to enforce (psychologically) interesting choices. The available response fractions *f* correspond only to *Cooperation* (action 1) or *Defection* (action 2). A matrix of monetary returns for *self* and *other* that can be used for this simplified task is shown in **Table 2**.

The task is a multi-round game—partners have to make decisions, taking into account long-term consequences of their choices. This is a difficult problem—and we will see that appropriate use of interpersonal representations may be used as a shortcut.

#### *Interpersonal representations and prosocial utilities*

We now consider the issue of how preferences are constituted in the generative model. To construct our minimal model, we assume the following:

The observable component of world states is disclosed by action (*uo*, *us* ) and the hidden component (*es* ,*eo*) concerns the traits to be inferred. The fact that a "good" person is more likely to cooperate—and to *attract cooperation*—highlights the fact that esteem can augment the utility of cooperation. An analogous reasoning applies to defection.

Preferential biases induced by esteem can be specified in terms of an augmented return that includes the payoff and esteem. Following the format of **Table 2** we write:

**Table 2 | Trust Task monetary returns matrix with only two choices for each partner.**

*These returns are defined by payoffs rs 11* > *rs 12* ≥ *rs 22* > *rs 21 for the self (in the Investor role) and ro 21* > *ro 11* ≥ *ro 22* > *ro 12 for the other (in the Trustee role). The task is constructed as a sequential game, with my self playing first, and is typically asymmetric. In the example in brackets I have a "wage" of 20 units. I can choose to invest f self*,*low* = *20% or f self*, *high* = *80%; the other can choose to return 40 or 140% of my investment. All amounts have been rounded.*

$$\begin{aligned} r^s(\mu^o = 1, \mu^s = 1, e^s, e^o) &= \mathfrak{k}\_r^s r^s\_{22} + e^s + e^o \\ r^o(\mu^o = 1, \mu^s = 1, e^s, e^o) &= \mathfrak{k}\_r^o r^s\_{22} + e^s + e^o \end{aligned} \tag{7}$$

**Table 3** gives the augmented returns for each combination of outcomes.

With this setup observable outcomes can take just 5 values: A "starting state" and four outcomes: *o*2 = {*us* = 1, *uo* = 1}, *o*3 = {*us* = 2, *uo* = 1} and so on, for all combinations of cooperation and defection. For each round, each player has to model the transition probabilities *P*(*sT*|*st*, *u*˜). If *ro*(*uo t* , *us t*,*es* ,*eo*) denotes the augmented return for the *other*, *self* can use a softmax function to calculate the probabilities of actions taken by the *other* (following Equation 1):

$$P(\mu\_t^o | \mu\_t^s, e^s, e^o) = \frac{\exp(r^o)}{\sum\_c \exp(r^o)}\tag{8}$$

However, this requires that *self* knows the beliefs of *other* about hidden esteems (*es* ,*eo*). We will assume that *self* uses beliefs about their esteem to model the beliefs of the *other*. We will see later that this is not an unreasonable assumption. Furthermore, we assumed that players can resolve just two levels of esteem *eo* = *p* (for prosocial) or *eo* = *n* (for non-social or antisocial). To further simplify things, we assume that the self esteem is neutral, *eo* = 0. Prior beliefs about choices will then be influenced by "who I would like you to be" and "what I would like (us) to get." These simplifications create a discrete hidden state space with 10 states. These correspond to the five observable states, for each of the two levels of the other's esteem *eo* ∈ {*p*, *n*}. The action chosen by *self* were sampled from posterior beliefs over choices based on the prior beliefs over policies of Equation 2. These prior beliefs depended on the utilities in **Table 3**.

#### **IMPLEMENTATION OF THE TRUST GAME IN ITERATED PLAY**

We implemented the multi-round version of the Trust game by using the posterior beliefs about the partner, at the end of each round, as the priors for the next round.

The software routines were written using the SPM academic freeware platform in matlab (MATLAB, 2012). The SPM platform, including the DEM toolbox used here, is available

*The entries of Table 1 are weighted by a sensitivity parameter and then augmented by an interpersonal component to form socialized returns. The interpersonal component consists of the esteem for each partner plus the esteem for the other partner (weighted equally in this example). Positive esteems enhance cooperative utility whereas negative esteem increases the utility of defecting. We have assumed here that* β*s r* = β*o r* = β*r .*

under GPL (GNU General Public License, version 3, 2007). It can be accessed via www.fil.ion.ucl.ac.uk/spm/software/spm12. Additional scripts are available from the corresponding author on demand, also under GPL.

## **RESULTS**

In order to perform simulations we used the monetary values in **Table 1**. To calculate the numerical values corresponding to **Table 2**, we chose a value for β*r* such that the resulting probabilities according to Equation 8 would be very distributed. Furthermore, for the purposes of this demonstration, we chose the *other* to be antisocial; i.e., have a negative esteem, and naïve; i.e., only influenced by immediate outcomes (as per Equation 8). The preferences (priors) that these choices translate into for the *other* are shown in **Figure 2A**. The *other* would prefer the self to cooperate and the *other* herself to defect (cd in **Figure 2A**). Their second best preference would be mutual cooperation (cc), which still has a substantial monetary outcome. The *other* is indifferent about the remaining two options, in which the *self* defects (dc, dd). In **Figure 2**, we have included the starting state (start) as a potential outcome—as is required by the model specification in the code we used. We set the starting state probability to zero, as it never actually materializes as an outcome and agents do not need to consider a preference for it.

The situation is a little more complicated, and more interesting, with respect to the goals of the *self* that this scheme gives rise to. These are shown in **Figure 2B**. Whereas our antisocial, naïve

**FIGURE 2 | Pattern of social utilities ln** *P(sT***|***m)* **= σ***(rs (sT ),* **β***)***. (A)** Preferences of the *other*. This simple *other* only considers observable states of each round—the starting state (start), and each of the four *self*-action—*other*-action combinations shown in **Table 3**. The "start" state is only indicated for completeness: agents correctly never consider it as an outcome. **(B)** Preferences (goals) of the *self*. Preferences over all 10 hidden states are shown; See text for detailed description.

*other* did not consider separate states for prosocial vs. antisocial self, we endowed the *self* with preferences depending on the type of the *other* and hence we consider the full 10-state outcome space for each round of the exchange.

**Figure 2B** shows that the preference of the *self* for mutual cooperation is more pronounced if the *other* is prosocial. As one might expect, given an antisocial *other* the second-best preference for *self* is for the *other* to cooperate while *self* defects. More interestingly, given a prosocial *other* the second-best preference for the *self* is to cooperate, while the prosocial *other* defects. Heuristically, *self* is forgiving toward prosocial but not antisocial *others*.

## **A SINGLE-ROUND**

The basic behavior of *self* when choosing a policy through free energy minimization is shown in **Figure 3**. Initially, *self* believes that the *other* is equally likely to be *p* or *n*. In other words, at the beginning of a series of exchanges, we assume people are agnostic as to the character or esteem of their opponent. Notice that although there are 10 hidden states, there are only five observable states—because the esteem (of the other) is hidden and has to be inferred.

At the first time step *self* just observes the starting state and believes the *other* is equally likely to be *prosocial* or *antisocial*, corresponding to hidden states 1 or 6. Still, under the influence of their utilitarian priors *self* assigns a higher probability to the cooperative policy (control state 1). With the parameters used in this example, this is a modest preference: as it happens, the

**FIGURE 3 | Inferences made by** *self* **during a single round, where** *self* **initially believes that the** *other* **is just as likely to be prosocial as antisocial.** The numbering of states from 1 to 10 corresponds to the 10 states in **Figure 2B. (A)** This shows that the observable state changed from state 1, the starting state, to 5, corresponding to mutual defection during this example round. **(B)** Initially the belief of *self* was equally shared between playing a prosocial partner or an antisocial partner (state 1 or 6). At the end of the round, belief was shared between mutual defection with a prosocial (s5) or antisocial (s10) partner, but no longer equally so. Defection made the *self* infer that the *other* was more likely to be antisocial: *P*(s10) > *P*(s5). The column "Full priors" corresponds to **Figure 2B**. **(C)** Control state 1 (cooperation) is slightly favored despite agnosticism, at this stage, as to the type of the *other*. As it happened however the *self* still chose to defect, as choice is probabilistic **(D).** The underlying true states: in this example the *other* is antisocial.

choice selected was to defect—to which the *other* responded by also defecting. *Self* therefore observes outcome state 5. Finally, on the basis of this outcome, *self* infers that they are more likely to be playing an *antisocial other*, which is the case. Clearly, in a single round, *self* cannot make use of this inference. However, if we now replace the prior beliefs about the *other* with the posterior beliefs and play a further round, we can emulate Bayesian updating of beliefs about the *other.* We now turn to the simulation of iterated play using this method of updating beliefs.

## **ITERATED PLAY**

During iterated play, beliefs about the *other* evolve. This has a knock-on effect on the goals or priors for each round—that produce a progressive change in preferred policies as one learns about the *other* and adjusts one's behavior accordingly. The result of a multi-round game is shown in **Figure 4** and reveals several interesting features:

The agent infers fairly quickly that the *other* is antisocial and reduces cooperative play. In this example, they still engage a considerable amount of cooperative play – outcome state 4 in **Figure 4C** is self-cooperate, other-defect *o*4 = {*us* = 1, *uo* = 2}. These outcomes reflect the preference of *self*, not a lack of confidence or expected precision. The evolution of expected precision is interesting. Precision reflects whether the available policies can

**FIGURE 4 | (A)** A sequence of 32 rounds of the simplified Trust task. Over the course of approximately 10 rounds, *self* becomes confident that *other* is antisocial. **(B)** This increasing belief results in a declining belief in (preference for) cooperating. **(C)** In this example the actions chosen are quite variable and: **(D)** expected precision changes relatively slowly. The variability of responses is due to the relatively weak preferences over different outcomes used here; this is to illustrate how one quantity (e.g., expected precision) changes with respect to another (e.g., players' choices) over a single round or over a sequence of rounds.

fulfill the goals or utilitarian priors. Initially, there was prior belief that fully cooperative play might be achieved, given the *other* might be prosocial*.* When it looked as if this was the case (outcome state 2 in **Figure 4C**), precision jumped optimistically (**4D**). However, overall, there is a slower increase in expected precision, as the agent realizes the true nature of the opponent (i.e., that the *other* is antisocial). This illustrative example highlights the important interplay between prior beliefs about outcomes, inference on hidden states or characteristics of opponents and, crucially, confidence in the ensuing beliefs.

## **DISCUSSION**

In this paper, we applied active inference to interpersonal decision making. Using a simple example, we captured key aspects of single and repeated exchanges. This example belongs to the large family of partially observable Markov decision problems (POMDP) but its solution is distinguished by explicit consideration of the agent's goals as *prior* distributions over outcomes. Because behavior depends upon beliefs, this necessarily entails beliefs that have precision. In other words, it is not sufficient simply to consider the goals of interpersonal exchange, one also has two consider the confidence that those goals can be attained. We have focused on optimizing this precision of beliefs about different policies—as opposed to sensitivity to different outcomes. In what follows, we consider the difference between sensitivity and precision. We then consider the nature of interpersonal inference and how it shapes decision-making. Finally, we discuss further developments along these lines.

## **SENSITIVITY OVER OUTCOMES vs. PRECISION OVER POLICY CHOICE**

One of the key consequences of our formulation is the separation of choice behavior into two components. The first is inherent in the prior distribution itself, which reflects goals that are not directly represented in the exchange—as might be codified by various matching rules or exploratory drives. The second is optimized by the agent during the exchange itself in order to maximize utility or returns, in light of what is realistic. As described in Friston et al. (2013), this decomposition can be seen clearly by expressing the negative divergence—that constitutes prior beliefs—in terms of entropy (promoting exploration of allowable states) and expected utility:

$$\begin{aligned} -D\_{KL}[P(s\_T|s\_t, \tilde{u})||P(s\_T|m)] &= H[P(s\_T|s\_t, \tilde{u})] \\ &+ E\_{P(s\_T|s\_t, \tilde{u})} \{ \ln P(s\_T|m) \} \end{aligned} (9)$$

Therefore minimizing the difference between attainable and desired outcomes can always be expressed in terms of maximizing expected utility, under the constraint that the entropy or dispersion of the final outcomes is as high as possible.

This separation of choice behavior—into (context-sensitive) beliefs about policies vs. (context invariant) beliefs about which outcomes are desirable—is reflected by an introduction of precision γ to complement the softmax sensitivity β. Both parameters play the role of precision or sensitivity (inverse temperature). β determines how sensitive prior beliefs are to rewards or the relative utility of different outcomes. However, this does not specify the confidence or precision that these outcomes can be attained. This is where the precision parameter γ comes in—it encodes the confidence that desired outcomes can be reached, based on current beliefs about the world and allowable policies. For example, one can be very uncertain about the contingencies that intervene between the current state and final outcome, even if one is confident that a particular outcome has much greater utility than another.

Crucially, the precision of the probability distribution over alternative policies can itself be inferred in a Bayes-optimal sense. This represents a departure from classical formulations. It arises because we are formulating policy selection in terms of inference. Choices are based upon beliefs (or inference) and beliefs in turn are held with greater or lesser confidence. The Bayes-optimal selection of precision over policies is a key thing that the current formulation brings to the table, above and beyond classical formulations.

## **INTERPERSONAL REPRESENTATIONS AS MOTIVATING BELIEFS**

Our modeling demonstrates that the formulation of interpersonal representations in terms of plausible and desirable outcomes accommodates a number of psychological findings and points to interesting theoretical and empirical questions.

First, our model replicates basic features of other successful models of interactive games. The 'esteem' traits in our model parallel the role of fairness-related coefficients in other models (Xiang et al., 2012). Second, our model infers the type of the partner (e.g., **Figure 4A**) and adjusts its policy so that it is not exploited (**Figure 4B**). Thirdly, posterior beliefs are based upon a generative model that entails beliefs about beliefs (utility functions) of others. This endows the generative model with an elemental theory of mind. Furthermore, Bayesian inference about esteem, and therefore intentions, constitute an elementary form of *mentalizing* (Allen et al., 2008).

In our case the fact that interpersonal representations contribute to the agent's beliefs about the desirability of outcomes *biases inference about states perceived* and actions selected. The perceptual update in **Figure 1** contains a contribution from precision. This is a remarkable effect of approximate Bayesian inference. In our example (**Figure 4B**) the result is that the agent is biased toward co-operativity, despite believing that the *other* is as likely to be uncooperative as not (**Figure 4A**). This is an interpersonal analog of optimism bias, or 'giving the benefit of the doubt'. There is experimental evidence in the Trust task that beliefs about prosocial traits in the *other* result in preference structures akin to the prosocial side of **Figure 2A**. When Investors are made to believe that the Trustee is of 'moral character' they entrust larger amounts (in our terms, cooperate in a sustained manner) even if the experimenter manipulates Trustee behavior so that the Investor does not make more money as a result (Delgado et al., 2005).

Our treatment suggests that interpersonal representations can help predict (and seek out) the outcomes of interactions. The idea that a self-esteem aspect of self-representation helps predict social outcomes is a central empirical finding of research by Leary and co-workers (Leary et al., 1995). Aspects of other-representation that help predict active social outcomes can be found in negative ideas about others, that healthy people harbor in certain contexts. As mentioned, exaggerated suspicion about others can serve to manage false-negative errors in the detection of social difficulties (Kramer, 1994). Computationally, more sophisticated agents can predict interactions better. Under certain constraints, however, interpersonal beliefs in the form of prosocial biases help achieve behavior that emulates such sophisticated thinking, a key theoretical finding of Yoshida and co-workers (Yoshida et al., 2008).

Interpersonal inference suggests that the use of selfrepresentations to predict outcomes requires an *assessment of context.* In our Trust task, my partner and I can just consider one round in the future, provided we have inferred our types appropriately and, implicitly, the effective nature of the exchange (cooperative or competitive, etc.). Our simulation contains an interesting example of what happens if the wrong representations are assumed. The game is cooperative but, in our example, the other is antisocial (and unsophisticated). The *other*'s preference, stemming from their negative "niceness" (esteem), is to defect while the *self* cooperates, followed by mutual cooperation. Note that this preference structure is the only element in our naïve *other*'s cognitive machinery. When the *self* infers this preference structure they switch to a more uncooperative policy, thus undermining the *other*'s goals. Had the *other* been "nice" enough, or had they believed the *self* to be "nice" enough, the *self* would have inferred this and the *other*'s predictions, or goals, would be fulfilled.

We see that goals are not prescribed by immediate reward but by more generic beliefs. Clearly, there are an enormous number of forms for these beliefs that we could consider that help predict and realize different outcomes in different contexts. In the present context, one might consider the long-term payoffs that accrue from a collaborative policy for the agent or for everybody. Crucially, collaboration entails a consilience in terms of prosocial preferences or utility. The key thing about prosocial utility is that it can be symmetrical with respect to me and my opponent. For example, I may altruistically value the total reward accrued by myself and my opponent *if* I think they are prosocial, but only my own rewards if they are antisocial. In our simple illustration, and with the right choice of parameters, this would result in a very similar pattern of exchange to that seen in **Figure 3**. Alternatively, through some aversion to inequality, *self* might prefer equitable outcomes (irrespective of who gets most).

In our simulations the effect of esteem operates like a *social Pavlovian bias*, biasing beliefs irrespective of their consequences. A Pavlovian bias enhances certain actions in certain contexts. For example, it enhances passivity in a context of threat or vigorous approach in a context of opportunity, irrespective of instrumental outcomes. Our social Pavlovian bias promotes certain actions in the context of certain personal esteems irrespective of instrumental outcomes. Here, we chose a scheme of social Pavlovian biases that makes direct links between contemporary research into these fundamental biases (Guitart-Masip et al., 2012) and the large body of clinical- and social- psychological work on affectively charged representations of people. This work spans Aristotelian ethics, forensic psychotherapy (Gilligan, 2000) through to attribution theory (Thewissen et al., 2011).

We placed emphasis on prior beliefs as they may absorb various beliefs about long-term outcomes. These utilitarian beliefs entail the agents' cognitive-affective horizon, beyond which the agent has no knowledge and no control. This contrasts with the dynamics of the exchange, wherein the agent has both beliefs about states and beliefs about control. We envisage that the present approach will help disentangle these two components in the setting of interpersonal dynamics.

Although our ultimate aim is to study how self-representation is inferred under active inference, in this introductory study we have kept self-representation constant. Although we hope to examine this in future work here we note that a Bayesian framework naturally predicts that ordinary self-representation should be less responsive to evidence than the representation of others. Setting aside beliefs about changeability of the self, as well as the real possibility that aspects of self-representation may be learnt "once and for all" during childhood, inference about self-representations must take place on the basis of a much greater evidence base than inference about strangers. Therefore each new piece of evidence is expected to have less impact on self-representation than other-representation.

## **MODELING CHOICES, LIMITATIONS AND OUTLOOK**

## *What does my partner think of me?*

It may appear that we made a gross simplification in modeling the *self* using their own representation to estimate how the *other* sees the self. A more general formulation might be more conventional, where the beliefs of the *self* (self-representation and reputation with respect to others) are separate. Yet this is not a modeling choice made to make the model simpler. For example, clinical psychology indicates that beliefs about the *self* are highly correlated with beliefs about how others see the *self*. Moreover, patients with unwarranted beliefs about themselves and others that look "psychologically defensive" show no greater social desirability than healthy controls (Moutoussis et al., 2013). We suggest that the *self* uses beliefs about their esteem to model the beliefs of the *other*, a generalization of the "sociometer" theory with a view to testing the limits of this assumption's predictive power.

## *Depth-of-thought*

Our model uses a very simple *other*, who makes no inferences about itself. Clearly, this is not a realistic simulation of *other*. Furthermore, our model *self* does did not explicitly calculate distant outcomes before applying the prior "horizon." The latter is partly justified as most people look to the future to quite a limited extent. In the Trust Task, only about a quarter of Investors show up to two levels of recursive interpersonal thought (Xiang et al., 2012). Having said this, further work needs to consider agents that explicitly simulate outcomes for a small number of steps into the future and apply inference and preferences to patterns of such outcomes.

## *Normative self-representations*

We envisage that self representations would enter into the assessment of proximal gains in the light of long-term outcomes; for example, "What sort of person am I, if I treat the other player like this?"; "If that's the sort of person I am, how am I likely to be treated in the future?" This extension of the simple model above will be crucial if the *other* makes inferences about the *self*. Our long-term aim, test the hypothesis that the normative role of self-representation is to predict the likely outcomes of social interactions, is likely to require such complex thinking. We envisage that beliefs about the opponent can, through conditional dependencies among Bayesian estimates about me and my opponent, affect beliefs about me. This may be crucial for understanding psychopathology in interpersonal exchange.

## *Model parameterization*

We discussed above that interpersonal, affectively charged representations may be parameterized in a number of related ways. We chose a very simple parameterization for the purposes of demonstration. In the light of a wider literature, the validity of different models for interpersonal representation and the relationships between them remain to be clarified. One important contribution of formal models, of the sort we have introduced here, is that they can provide quantitative predictions of choice behavior. In principle, this means that one can use observed choices to estimate the parameters of a given model and—more importantly use Bayesian model comparison to adjudicate between different forms or hypothetical schemes.

## **SUMMARY**

In conclusion, we have sketched an elementary model of *self* and *other* representation during interpersonal exchange; within which these representations have important functional roles. We have seen that it is fairly straightforward to place optimal decision schemes in an active inference framework. This involves replacing optimal policies, defined by utility functions, with prior beliefs about outcomes. The advantage of doing this is that one can formulate action and perception as jointly minimizing the same objective function, which provides an upper bound on surprise or (negative log Bayesian) model evidence. This enables optimal control to be cast as a pure inference problem, with a clear distinction between action and inference about (partially) observed outcomes. Using a simple example, we have demonstrated how desirable goals can embody and express prosocial preferences as well as beliefs about the type of an opponent. Specifically, we have shown how these beliefs can be updated during iterated play and how they can guide interpersonal choices. Although rudimentary, these simulations illustrate a formal basis for interpersonal inference.

## **AUTHOR CONTRIBUTIONS**

Michael Moutoussis formulated the core hypotheses regarding interpersonal representation from both a clinical-psychological and a Bayesian perspective; formulated the test task and its parameterization; programmed the structure of the task; ran the simulations; and drafted much of the manuscript. Raymond J. Dolan provided supervision to Michael Moutoussis; provided management for the whole project; provided psychiatric expertise; and edited the manuscript. Nelson J. Trujillo-Barreto and Wael El-Deredy provided help with variational Bayesian modeling; and reviewed the manuscript. Karl J. Friston placed the hypotheses regarding interpersonal Bayesian inference within the framework of Active Inference; contributed heavily to the mathematical formulation; wrote and provided key code within the SPM environment for solving discrete partially observable Markov problems through the message-passing free-energy-minimization algorithm; edited the manuscript; and provided supervision to Michael Moutoussis.

## **ACKNOWLEDGMENTS**

This work was funded by the Wellcome Trust ['Neuroscience in Psychiatry' strategic award, 095844/Z/11/Z]. This work was supported by the Wellcome Trust Ray Dolan holds a Senior Investigator Award [098362/Z/12/Z]. The Wellcome Trust Centre for Neuroimaging is supported by core funding from the Wellcome Trust [091593/Z/10/Z]. Michael Moutoussis also receives support by the Biomedical Research Council.

## **REFERENCES**

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 29 August 2013; accepted: 03 March 2014; published online: 25 March 2014. Citation: Moutoussis M, Trujillo-Barreto NJ, El-Deredy W, Dolan RJ and Friston KJ (2014) A formal model of interpersonal inference. Front. Hum. Neurosci. 8:160. doi: 10.3389/fnhum.2014.00160*

*This article was submitted to the journal Frontiers in Human Neuroscience.*

*Copyright © 2014 Moutoussis, Trujillo-Barreto, El-Deredy, Dolan and Friston. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## What makes us conscious of our own agency? And why the conscious versus unconscious representation distinction matters

## **Glenn Carruthers \***

ARC Centre of Excellence in Cognition and Its Disorders, Macquarie University, Sydney, NSW, Australia

#### **Edited by:**

Nicole David, University Medical Center Hamburg-Eppendorf, Germany

#### **Reviewed by:**

Cristina Becchio, Università degli Studi di Torino, Italy Lauren Swiney, Queen's University Belfast, UK

#### **\*Correspondence:**

Glenn Carruthers, ARC Centre of Excellence in Cognition and Its Disorders, Macquarie University, Level 3 Australian Hearing Hub, 16 University Ave., Sydney, NSW 2109, Australia e-mail: Glenn.rj.carruthers@ gmail.com

Existing accounts of the sense of agency tend to focus on the proximal causal history of the feeling. That is, they explain the sense of agency by describing the cognitive mechanism that causes the sense of agency to be elicited. However, it is possible to elicit an unconscious representation of one's own agency that plays a different role in a cognitive system. I use the "occasionality problem" to suggest that taking this distinction seriously has potential theoretical pay-offs for this reason. We are faced, then, with a need to distinguish instances of the representation of one's own agency in which the subject is aware of their sense of own agency from instances in which they are not. This corresponds to a specific instance of what Dennett calls the "Hard Question": once the representation is elicited, then what happens? In other words, how is a representation of one's own agency used in a cognitive system when the subject is aware of it? How is this different from when the representation of own agency remains unconscious? This phrasing suggests a Functionalist answer to the Hard Question. I consider two single function hypotheses. First, perhaps the representation of own agency enters into the mechanisms of attention. This seems unlikely as, in general, attention is insufficient for awareness. Second, perhaps, a subject is aware of their sense of agency when it is available for verbal report. However, this seems inconsistent with evidence of a sense of agency in the great apes. Although these two single function views seem like dead ends, multifunction hypotheses such as the global workspace theory remain live options which we should consider. I close by considering a non-functionalist answer to the Hard Question: perhaps it is not a difference in the use to which the representation is put, but a difference in the nature of the representation itself. When it comes to the sense of agency, the Hard Question remains, but there are alternatives open to us.

**Keywords: consciousness, self-consciousness, sense of agency (SoA), hard question, functionalism, vehicle theory**

## **INTRODUCTION**

In this paper I argue that we, as a community investigating the sense of agency, are not doing enough to answer what Dennett has called the "Hard Question" of consciousness. Our existing models do a very good job of explaining when a representation of own agency is elicited. I illustrate this with two historically important accounts: the comparator model of Frith et al. and Wegner et al. inference to apparent mental state causation. Following Revonsuo, I consider these to be proximal etiological explanations. Although powerful so far as they go, these accounts, on their own, do not provide us with the explanatory resources to distinguish conscious and unconscious representations of one's own agency. This is not a problem we can ignore. I use the "occasionality problem" to suggest that there are potential theoretical benefits to taking this distinction more seriously as conscious and unconscious representations of own agency play very different roles in cognition. I conclude by considering how we might approach the Hard Question for the sense of agency. I consider two Functionalist approaches (i) that a representation of own agency is conscious if it is taken as the object of attention; and (ii) that a representation of own agency is conscious if it is available for verbal report. Although such approaches offer clear research agendas, both of these specific approaches seem non-starters on empirical grounds. That said multifunction hypotheses such as the global workspace theory remain viable Functionalist positions. Finally I consider a Vehicle theory approach to the Hard Question. Such an approach also offers some clear research questions, but currently no clear answers. As of now, the Hard Question remains under-considered for the sense of agency even though there exist a variety of questions we can ask to make progress on it if we take either a Functionalist or a Vehicle approach. These are questions we would all do well to consider.

## **STANDARD ACCOUNTS OF THE SENSE OF AGENCY**

Standard explanations of the sense of agency are of a particular type. Revonsuo (2006, pp. 20–22) calls this type of explanation "proximal etiological explanation". Such explanations have two defining characteristics. First, they enumerate *causes* of the sense of agency. Second, the explanations are *cognitive* explanations. The specific causes posited are mental representations and computations. To understand these accounts as explanations of the sense of agency then, is to understand them as a description of what aspects of the mind, i.e., mental representations and computations, cause a subject to experience their own agency. The sense of agency itself is taken to be just another representation in this causal chain.

These traits are shared by prominent accounts of the sense of agency. Consider first the comparator model. This model gets its name from the use of three hypothetical comparisons performed by the motor control system. Each of these comparisons performs specific functions for motor control and motor learning (Wolpert and Ghahramani, 2000). One of these comparisons also elicits the sense of agency. This is the comparison that will concern us here (for the full account and its broader applicability see Frith et al., 2000a). On this model it is hypothesized that performing an action requires the formation of a goal state or motor intention (Pacherie, 2008), which represents where the body needs to move to in order to perform the action. From this, the motor control system formulates a motor command, which specifies how to move the body from where it is to where it needs to be in order to attain the goal. Two copies of the motor command are formed; one is sent to the periphery and elicits the requisite contractions of the effector muscles to perform the movement needed to attempt the action. This movement, of course, affects the sensory organs, allowing the motor control system to represent the movement after it occurs. The second copy of the motor command, sometimes called the "efference copy" or "corollary discharge", is used by the motor system to form a prediction of what sensory feedback will be received due to the action. This predicted feedback can be used to represent the action as it occurs (Frith et al., 2000a,b; Blakemore et al., 2002).

Now we get to the sense of agency. It is hypothesized that the collection of representations and computations introduced above cause the sense of agency to be elicited. Specifically when feedback from the senses to the motor control system (actual sensory feedback), matches an internally generated prediction of what this feedback will be (predicted sensory feedback), a sense of agency is elicited (Frith et al., 2000a, p. 1784). These two representations matching in this context means that they represent the same action. The comparator model has been considered a promising explanation of the sense of agency and is able to explain some important discoveries (for recent reviews of what and how see Carruthers, 2012).

Wegner et al. have suggested that the sense of agency is elicited by a rather different kind of computation. On their model, the sense of agency is elicited when one infers that one or other of one's mental states caused the action of one's body (Wegner and Wheatley, 1999, p. 480; Wegner, 2003, p. 67). If correct, we can provide a proximal etiological explanation of the sense of agency by explaining how this inference is made. To make the inference the subject needs to represent their mental states *qua* potential causes of action, represent which body in the world is their own (i.e., a sense of embodiment) and represent the action which is occurring or has occurred. Next they must represent that one or other mental state causes the action of their body. This is the role of the inference to apparent mental state causation. According to Wegner et al. this inference is made when three facts about the relationship between the mental state and action are recognized. First, the mental state must be consistent with the action in that it specifies the action that actually occurs. Second, the mental state must seem to occur at an appropriate time before the action occurs, for example a memory of an action won't be inferred as a cause of that action. Third the thought must appear to be the only possible cause of the action, i.e., if something else, another person or gust of wind, say, could have caused the action then the inference will not be made, or at least not made with a high degree of certainty. Wegner et al. call these the principles of "consistency", "priority" and "exclusivity" respectively (Wegner and Wheatley, 1999, pp. 482–487; Wegner, 2003, p. 67). Like the comparator model, this account has been considered a promising approach to the sense of agency and it can explain some important discoveries (see Wegner, 2002 for reviews; Carruthers, 2010).

Numerous authors have followed the general approach of these classic hypotheses. Recently, several authors have proposed that the sense of agency is elicited by a process that integrates the output of several such computations (Synofzik et al., 2008, 2009; Moore and Fletcher, 2012; Carruthers, in press). This work is characterized by considerable progress in investigations into the computations that elicit the sense of agency. However, there is a limitation to this approach. Knowing how the representation of agency is elicited doesn't distinguish between cases where it is elicited but remains unconscious and cases where it is elicited and the subject is aware of it. Unless we are to view the sense of agency as unique amongst all mental representations in that it can only *ever* be conscious, we must allow for the possibility of a representation of own agency to be elicited and remain unconscious. In the next section I consider the occasionality problem which has been presented as an objection to the comparator model, but which is generalizable to other accounts. I use this as an example to show that taking seriously the distinction between unconscious representations and a conscious sense of agency can have theoretical pay-offs in this area as each of these play different roles in the broader cognitive system. In particular I suggest that if we take this distinction seriously then the occasionality problem doesn't arise.

## **THE OCCASIONALITY PROBLEM AND UNCONSCIOUS REPRESENTATIONS OF ONE'S OWN AGENCY**

It would, I take it, be bizarre if there were no unconscious representations of own agency. But, is there any theoretical benefit for sense of agency research as it it currently done to considering this fact explicitly? In this section I argue that there is. That by considering the different roles conscious and unconscious representations of own agency play, in particular that only the absence conscious representations can be noticed by the subject, we can avoid the "occasionality problem". In the next section I consider some ways in which we might attempt to explain the difference between conscious and unconscious representations of own agency. The occasionality problem, it should be noted, was originally formulated as an objection to the comparator model, but it applies equally to Wegner et al. account described above. To see the problem we first need to take a step back and consider clinical phenomena that the above models need to explain.

One of the central explananda for the accounts introduced above is thought to be delusions of alien control. This delusion, commonly seen as a symptom of schizophrenia, is a patient's belief that not they, but rather some other agent, control the patient's actions. This is expressed in reports such as:

I felt like an automaton, guided by a female spirit who had entered me during it [an arm movement].

I thought you [the experimenter] were varying the movements with your thoughts.

I could feel God guiding me [during an arm movement] (Spence, 2001, p. 165).

There is a growing consensus that explanations focusing on the sense of agency alone cannot explain every feature of this delusion (Synofzik et al., 2008; Carruthers, 2009). In particular, such accounts do not have the resources to explain why patients attribute the action to another specific agent. What such accounts can explain is why patients fail to attribute their actions to themselves. According to the comparator model, in healthy subjects the comparison between actual and predicted sensory feedback causes a sense of agency to be elicited for actions the subject performs. However, it is hypothesized that this computation goes wrong for the patient suffering delusions of alien control. They do not represent a match between predicted and actual sensory feedback when they should and so no sense of agency is elicited. Without this sense, the patient has no experiential basis for a self-attribution of action- they do not feel as though they perform the action- and so actions are not selfattributed. For those interested in the details of why this occurs, there is some experimental evidence that these patients have an underlying deficit in forming or using predicted sensory feedback (Frith and Done, 1989; Blakemore et al., 2000; Carruthers, 2012).

As with the comparator model, Wegner et al. inference to apparent mental state causation is unlikely to explain every feature of this delusion. Like the comparator model it may offer an account of how the sense of agency is lost. According to this view the sense of agency would be lost when one of the principles of priority, exclusivity or consistency is not met. I have argued elsewhere (forthcoming) that on this model it is reasonable to hypothesize that it is the principle of priority which is violated, as there is some evidence that patients suffering from delusions of alien control display abnormalities in the representation of the timing of their actions (Voss et al., 2010).

Now we are in a position to examine the occasionality problem (de Vignemont and Fourneret, 2004; Proust, 2006, p. 89; Synofzik et al., 2008). This problem starts from the observation that those suffering from delusions of alien control only attribute *some* of their actions to other agents. None of the models above appear to have, on their own, the resources to explain this observation. At the core of this objection is an accusation of a false prediction. A model like the comparator model predicts patients lack a sense of agency for their actions because they cannot represent a match at the comparison between actual and predicted sensory feedback. This does not offer us principled grounds for distinguishing those actions that the patient self-attributes and those that they attribute to others. If the comparison fails then the model should predict that patients lack a sense of agency for all of their actions. This is not the case, so the comparator model appears to be incorrect. This problem arises again when we consider Wegner et al. account. Hypothesizing that these patients fail to represent their own agency for their actions because they misrepresent the timing of their actions (thus violating the principle of priority) again fails to explain why only some actions are misattributed. In essence these accounts suggest that such patients always lack a representation of their own agency, but it seems that this lack only matters to the subject some of the time.

de Vignemont and Fourneret, (2004, p. 9) have suggested that the system which elicits the sense of agency, whether it be the comparator model or something else, fails only occasionally and in a context specific way. If it is true that the comparator or inference to apparent mental state causation face intermittent failures, then the occasionality problem disappears. However, the questions of how and why the mechanism occasionally fails have not been answered and nothing about the actions themselves or features such as their personal significance affect whether or not they are self-attributed (Proust, 2006, p. 89). More importantly there is no evidence independent of reports of the delusion that the comparator model or the representation of the timing of actions fails only occasionally for such patients. Until such evidence is forthcoming it is difficult for this solution to shake the appearance of being *ad hoc* and it is worth considering other accounts. More so, as I will suggest below, if we consider the different roles conscious and unconscious representations of own agency play in cognition, which we should do anyway, then there is no need to add additional assumptions of this type.

An argument from the occasionality problem against the hypotheses described, like that sketched above, assumes that the result of the process leading to a representation of own agency is a conscious sense of agency. If we drop this assumption the problem needn't arise. To see why this assumption is being made let us consider the relationship between experiences and delusions. So, what is the evidence that patients suffering delusions of alien control lack a sense of agency? One might be tempted to think that they *say so*. But, this isn't typically the case. Rather a deficit in a conscious sense of agency is inferred from the fact that patients attribute their own actions to another agent. This inference is justified by some standard assumptions in the study of delusions. The state of the art in delusions research is strongly influenced by Maher (1988, 1974) hypothesis that delusions are attempts to explain anomalous experience. Now there may be controversy regarding whether this explanatory attempt involves normal or deficient reasoning (Davies et al., 2002; Gerrans, 2002), but both sides agree that the delusion arises from an attempt to make sense of an anomalous experience. This supposition is not universally accepted, of course (Campbell, 2001; Bayne and Pacherie, 2004), but what matters here is that this assumption is needed if we are to justify inferring that patients lack a sense of agency from their acts of other attribution. We can justify this inference if the lack of a sense of agency is the anomalous experience which the delusion of alien control is an attempt to make sense of. So first, why should the absence of a sense of agency be an anomaly that needs to be explained? Well, it would be, if a conscious sense of agency typically accompanied one's actions. If this is the case, we would expect that its absence would be noticed and felt to be in need of explanation. After all, if one feels one's body move, but one does not seem to be the agent behind the movement, then one would naturally search for a reason that one moved.

We see this assumption that there is a conscious sense of agency accompanying all actions at play in the argument from the occasionality problem. The general failure of a process like the comparator should mean that the sense of agency that is usually present is not. This is an anomaly to be explained by the patient. The patient should show delusions of alien control for all of their actions, but they do not, therefore the comparator model (or which ever process we are considering) is false, quod erat demonstrandum (QED).

To avoid this conclusion, all we need do is drop the assumption that a conscious sense of agency always accompanies our actions. Instead, we need only hypothesize that a representation of own agency which may or may not be conscious accompanies our actions. In other words, the output of processes like the comparator or inference to apparent mental state causation is a representation of the subject's agency which is sometimes conscious and sometimes not. An absence of a sense of agency is thus not always an anomaly which the patient need explain. An absence of representation is not a representation of absence, as the saying goes, and it is particularly not a representation of absence *to the subject*. It is the subject noticing (i.e., representing to themselves) that the sense of agency is absent which is hypothesized to lead to delusions of alien control, not it's mere absence. This noticing of the absence will occur when the sense of agency is expected and so we might say the absence of a sense of agency is only an anomaly when the subject expects to experience it.

A possible objection to this line of response is to assert, based on introspection, that a conscious sense of agency accompanies all of our actions in the normal case. As such, it is always expected and any absence is an anomaly to be explained. However, introspection gives us poor grounds to assume that there is a ubiquitous sense of agency. What would lead one to assume that there is a conscious sense of agency accompanying every action? We can see where this assumption comes from, and how poorly grounded it is, by an analogy with visual consciousness. A favorite example purporting to show that we are not conscious of as much as we think we are comes from Dennett (1991). This example is so easy to replicate that given minimal resources you can do it yourself right now. All you need is a well shuffled deck of playing cards.  It is important that you continue to stare at this point throughout the entire demonstration. Without looking randomly select a card and hold it out to one side at arm's length. Gradually move it toward the center of your vision. At what point can you see the color and number on the card? The typical finding is that it is only about 2 or 3◦1 from the point one is looking at that these features become visible (Dennett, 1991, p. 54). The reason for this is to do with the nature of photoreceptors outside of the fovea on the retina and need not concern us here. What I wish to draw attention to, however, is that on first experiencing this demonstration most people seem surprised (Dennett, 1991, p. 68). Pre-theoretically, we expect to be able to discriminate objects easily when they are presented in our peripheral vision. Dennett suggests, and I agree, that this expectation is based on a folk-theoretical belief that vision presents us with a relatively uniformly clear and colored world in which objects are easily distinguished. But, as this simple demonstration shows, as do other more rigorous experiments, e.g., Brooks et al. (1980), this is at best only true of the foveated world, and even then with some exceptions (Caplovitz et al., 2008).2

Why do we believe this is true of our peripheral vision? We can speculate on many possible reasons for this. One reason might be that things we use as public representations of what we see, e.g., photographs or videos, are somewhat like this. There may be a misbegotten analogy between visual depictions and visual experience. Another more universal proposal comes from Schwitzgebel (2008: p. 255) as well as Dennett (1991: p. 68) who suggests that objects in our peripheral vision appear distinct and colored because they are when we look at them. Whenever one looks to see what object is in one's periphery one finds it clear, distinct and colored. As such we tend to assume that we always experience those objects as such. This claim provides us with a useful analogy for understanding why accounts like the comparator model and Wegner et al. inference to apparent mental state causation needn't suppose that a conscious sense of agency accompanies every action.

If a model like one of those above is right, then it would be true that our actions are normally accompanied by a representation of our own agency. However, the subject need not be aware of their own agency. The representation could be unconscious but, because the representation is formed with every action, it is there whenever we go "looking" for it, or more generally when it is expected to occur to the subject, i.e., consciously. Just as objects in our periphery always appear clear, distinct and colored when we go looking for them, our representation of agency is always experienced when we go looking for it, thus meeting our expectations. Just as this may lead us to believe that objects in our periphery always appear clear, distinct and colored, this may lead us to believe we always experience a sense of agency accompanying our actions rather than merely representing it.

1As a rough guide 1 degree is approximately the angle subtended by a point either side of your thumb nail held at arms length.

2Now of course in the normal case our eyes saccade constantly allowing us to build a much more detailed visual representation than is possible from staring at a fixation point making the area of clear vision significantly larger than the 2 to 3◦ observable in a fixation task. This doesn't affect the central Dennettian claim that periphery is not clear and colored in the way that we would typically assume. Nor do we typically reflect on how much moving our eyes is necessary for seeing the way we do.

Accepting this conclusion then, the comparator model or the inference to apparent mental state causation need not suppose that representing one's own agency is always a conscious sense of agency. Still, one may wonder, how exactly does this affect the occasionality problem? After all, it would still seem to be the case that these models predict that the unconscious representation of agency would be missing for every action performed by the patient suffering delusions of alien control, so should the model still predict that the patient would show the delusion for every action?

The answer to this is no. However, to see why, we need to return to the purported role of consciousness in the formation of delusions such as delusions of alien control. Recall Maher's proposal that delusions are attempts to make sense of anomalous experiences. In the case we are interested in here, the delusion of alien control arises because the patient attempts to make sense of the absence of a sense of agency. They expect a sense of agency, but it is not there when they "look", giving rise to an anomalous experience that must be explained. On this view then, an absent sense of agency is only anomalous when it is expected. A subject would not notice the absence of an unconscious representation. It is only when the representation would otherwise become conscious that its absence would be noticeable. Again the absence of the representation is not the same as the subject representing to themselves that something is absent. The upshot of this is that if we hypothesize that the comparator or inference produces an unconscious representation of agency, which only becomes conscious when it is needed by the subject (say in self-recognition or introspecting to see what experiences one has), we find that the occasionality problem is no problem at all.

It is not so much that the problem is solved as it doesn't arise in the first place, all because conscious and unconscious representations of own agency play different roles in cognition. Only conscious representations can be expected by the subject, and only their absence can be noticed by the subject. The normal case is that actions are not accompanied by a conscious sense of agency (only an unconscious representation) and so a lack of this feeling is typically not an anomaly that the patient suffering delusions of control needs to explain. It is only when they would "look for" (however this analogy is to be cashed out mechanistically—see below) this representation that it is expected and so its absence is an anomaly that needs to be explained.

This consideration of the occasionality problem shows us that there are theoretical benefits to taking seriously the distinction between conscious and unconscious representations of own agency. By doing so and considering the different roles conscious and unconscious representations play in cognition we see that the occasionality problem doesn't arise. As such we don't need to add assumptions to our models, such as assuming that they only fail some of the time, which lack supporting evidence. However, we do have a new set of issues to consider. What then is the analogy of "looking for" the representation of agency that produces the expectation of the sense of agency needed to explain delusions of alien control? This question is no less than what distinguishes an unconscious representation of agency from a conscious sense of agency, and this is what Dennett has called the "Hard Question" of consciousness.

## **THE HARD QUESTION**

The approach to the sense of agency used by traditional accounts such as the comparator model and the inference to mental state causation are only designed to answer one question about the sense of agency: how is a representation of one's own agency elicited? This is a vitally important question in the study of the sense of agency, but to think it is the *only* question is to treat awareness as the end of the line of a computation, the dreaded Cartesian Theatre, and to deny the possibility of an unconscious representation of one's own agency. In addition to this question, we also need to ask of accounts of agency what Dennett calls the "Hard Question" [not to be confused with any purported "Hard Problems" (Chalmers, 2003) 3 ]: after the representation of own agency is elicited by one or other of these mechanisms, well, then what happens (Dennett, 1991, p. 255)? What is the difference between a representation of my own agency of which I become aware and one that languishes forever in the apparent irrelevance of unconsciousness?

The analogy employed above of "looking" for the sense of agency suggests one possible answer. Perhaps an unconscious representation of agency becomes a conscious representation when the subject's attention is directed to it? In the following section I consider and discuss this possibility. Having found this wanting, I consider a further possibility, that the answer to the Hard Question is that the representation enters into the mechanisms required for verbal report. I argue that this answer is also unsatisfactory, as it is inconsistent with behavioral evidence of a sense of agency in non-verbal animals. These first two options are Functionalist theories. They propose that consciousness is playing a certain role in cognition. Although these two specific proposals seem to fail on empirical grounds it is important to note that other Functionalist theories, notably those that identify consciousness with multiple functional roles remain open. Finally, I propose a radical alternative suggesting that the answer to the Hard Question is to be found not in the uses to which representations are put within a cognitive system, but in the nature of the representations themselves. Regardless of which of the two research agendas individuals chose to pursue, it is clear that we do not have an answer to the Hard Question for the sense of agency nor do we spend enough time thinking about it.

## **ATTENTION**

One potential answer to the Hard Question is *attention*. Such an answer is suggested by well-known cases of inattentional blindness, where subjects fail to see perfectly obvious stimuli (like a woman in a gorilla suit) simply because their attention is directed elsewhere (Mack and Rock, 1998). More specifically,

3The supposed "Hard Problem" of consciousness is the problem of explaining how mental and physical states give rise to conscious experience, given that it seems that no explanation in terms of the structure or function of mental states is sufficient to explain this (Chalmers, 2003). This is a problem closely tied with mysterian and dualistic approaches to consciousness. In contrast the "Hard Question" is a question within the materialist tradition, which works from arguments that a structural or functional explanation of consciousness is possible in principle, and asks what is the difference between a conscious and unconscious mental representation.

let us hypothesize that the difference between an unconscious representation of agency and the conscious sense of agency is that the conscious representation is attended to. If this is true then we would have a clear research agenda: understand how and why a representation of agency is selected or not selected for attention and understand the mechanisms of attention.

Such a view has not been developed in detail for the sense of agency; indeed, I am suggesting here that consideration of the Hard Question with respect to the sense of agency has been neglected almost entirely. Notwithstanding, attention based accounts of consciousness do have some currency in the explanation of perceptual consciousness. Prinz (2000, 2012), for example, advocates such a view. Unfortunately evidence is mounting that attention is not a good answer to the Hard Question, at least not on its own, as attention is not sufficient for consciousness. That is, subjects can attend to things of which they are not conscious. Here I discuss one well-studied example.

Norman et al. (2013) have provided compelling evidence that subjects can visually attend to objects, namely two-dimensional shapes, even when they cannot consciously see those objects. They start from prior observations of the effects of taking twodimensional shapes as the objects of attention in color discrimination tasks. In these tasks, subjects are asked to indicate with a button press the color of a circle. Before the target colored circle appears a supraliminal spatial cue is presented. In the trials of interest here the, cue appears some distance from where the target circle will ultimately appear. However, it may appear in the same shape as the cue or a different shape. See **Figure 1** for an example layout.

When the target appears in the same object as the cue, response times are facilitated (Egly et al., 1994). Norman et al. take this as characteristic of attention to such shapes.

Norman et al. repeated this experiment, but made the shapes invisible. They presented on a screen an array of Gabor patches whose orientation rapidly alternated between vertical and horizontal. Within the array rectangles were defined by Gabor patches flickering out of phase with the remainder of the array (Norman et al., 2013, p. 838). When the background patches were vertical, those defining the rectangle were horizontal, and vice versa. Observing the array subjects reported seeing flickering Gabor patches, but were unable to see the rectangles. Indeed, subjects were no better than chance when asked to guess whether or not such flickering displays contained rectangles (Norman et al., 2013, p. 840). Despite the invisibility of the shapes there was a facilitation effect in the color discrimination task characteristic of attention being directed at the shapes. That is, subjects were faster at responding to targets which appeared in the same shape as the cue, than for targets which appeared the same distance from the cue but in a different shape (Norman et al., 2013, p. 839).

In this study we see an effect characteristic of attention being directed at an object, despite the object being invisible. This demonstrates that subjects can attend to shapes of which they are not conscious. In general, this also suggests that attention is not sufficient for consciousness. Without a reason to think that the sense of agency will be an exception to this, it seems unlikely that attention will answer the Hard Question for the sense of agency.

## **REPORTABILITY**

Often we take it that we can be confident that a subject experiences something if they are able to verbally report it. Although such reports are susceptible to a variety of introspective omissions and commissions (Dennett, 1991, p. 96; Schwitzgebel, 2008), in practise, verbal reports (especially questionnaire responses) are very often treated as the best way to operationalize experience. Indeed the theories of the sense of agency introduced above are built on studies using questionnaires to ask subjects to report their experiences of agency. At the heart of this approach lies an intuition that, however imperfectly, we are able to talk about those things that we experience, but not those things that reside in our unconscious minds. This intuition suggests an approach to the Hard Question: perhaps the difference between conscious and unconscious representations is just that conscious representations are available for report. Although such an approach would be highly controversial (Block, 2007), there is no approach to the Hard Question that is not controversial, and this proposal remains live.

That said, we do have strong reason to doubt that it is reportability that distinguishes conscious and unconscious representations of own agency, as there are many non-verbal animals that display evidence of experiencing a sense of agency. This suggests that being available for verbal report is not necessary for a conscious sense of agency.

Good evidence for this comes from the mirror self-recognition test. This test, first proposed by Gallup (1970), involves marking an animal surreptitiously (usually when anesthetized) with a non-irritating, odorless dye on a part of the animal's body that cannot be seen without a mirror (such as the forehead). An animal is deemed to pass the mirror self recognition test if there is a significant increase in mark directed behavior coincident with the animal observing itself in the mirror (Gallup, 1970, p. 87). Such behavior indicates that the animal has recognized itself in the mirror as it uses the mirror to direct actions towards itself. A sense of agency is needed to pass such tests. To learn to recognize oneself in a mirror one needs to realize that the actions one sees in the mirror are equivalent to the actions one is currently performing (Povinelli, 2001, p. 855). In order to recognize oneself in a mirror, then, one needs to know (amongst many other things) what action one is performing. This is a function of the sense of agency (Povinelli and Cant, 1995). As such, passing the test is good evidence for a sense of agency.

Where this creates a problem for using reportability as an answer to the Hard Question is in the fact that many nonverbal animals pass the mirror self-recognition test. This includes chimpanzees (Gallup, 1970), orang-utans, human raised gorillas (Povinelli and Cant, 1995), bottlenose dolphins (Marten and Psarakos, 1994) and European magpies (Prior et al., 2008). These animals thus show evidence of experiencing a conscious sense of agency. As such, verbal report does not seem necessary for consciousness, and thus investigating how unconscious representations of agency become available for verbal report is a nonstarter as a solution to the Hard Question.

The solutions considered so far to the Hard Question are Functionalist theories. They posit that for a representation to be conscious is for it to be used a certain way, say be being attended to or by being made available for report. On such views it is *use* which constitutes consciousness. Whilst the two options considered here do seem like non-starters, there are other Functionalist theories available. Other accounts, such as Dennett (1991) multiple drafts model Dennett (1991) or Baars (1988) global workspace Baars (1988), suggest that consciousness is not a single use within a cognitive system, but rather a conglomeration of many uses and these options remain live. My point here is not to solve the problem of what distinguishes conscious and unconscious representations, but merely to suggest that in sense of agency research this is a problem we should spend more time on. Next, I turn to a theoretical basis for approaching the Hard Question that offers a fundamentally different kind of solution to the options considered so far.

## **VEHICLE THEORIES**

Vehicle theories of consciousness answer the Hard Question in a rather different way. The key issue we are getting at is: what is the difference between an unconscious and a conscious representation of own agency? The proposals considered thus far have followed Dennett in hypothesizing that this difference is a difference between how unconscious and conscious representations are processed (e.g., are they subject to attention or made available for verbal report). In other words the difference is a matter of what is done with the representation. Such approaches are Functionalist theories in that they consider the particular use of a representation within a cognitive system to constitute that representation's being conscious.

Vehicle theories, in contrast, hypothesize that the difference between conscious and unconscious representations is not how they are processed, but in the nature of the representation itself (O'Brien and Opie, 1999, p. 128). The nature of conscious vehicles of representation (also known as representation bearers) is hypothesized to be different to the nature of unconscious representing vehicles. On such views consciousness is a way of representing the world using different kinds of vehicle than those used by unconscious representations. On this kind of view the answer to the Hard Question is not "and then some additional processing occurs" but rather, "and then the vehicle of representation is changed from one form to another".

O'Brien and Opie propose a general answer to this question making use of distinctions in kinds of representing vehicles offered by Dennett (1982). In particular they focus on a distinction between "explicit" representations which are: "physically distinct objects, each possessed of a single semantic value" (O'Brien and Opie, 1999, p. 133) and "potentially explicit" and "tacit"4 representations which are to be understood in terms of a computational system's *capacity* to make certain information explicit in the above sense. In general, O'Brien and Opie hypothesize that we are conscious of all and only things that are represented in an explicit form. All unconscious representations would then take the form of potentially explicit or tacit representations.

According to this version of a Vehicle Theory, a conscious sense of agency would be an explicit representation of own agency. That is, a discrete vehicle, such as a stable pattern of activity across a layer of neurons (O'Brien and Opie, 1999, p. 138), with that content. An unconscious representation of agency would not be a discrete vehicle, but a disposition in the cognitive system to produce such a representation. To allow for unconscious representations of own agency on such a view, the output of the comparator model or Wegner et al. inference needs to be reconceived. It is not an explicit representation of own agency, but rather a change in the dispositions of a computational system to produce such a representation.

If such an approach is correct then we have a new way to approach the Hard Question for the sense of agency. How is the output of the comparator model, or whichever account we ultimately agree on, made explicit? Why is it the case that it is sometimes not made explicit? Is this a matter of the subject metaphorically "looking for" it, if so, how would that be understood more literally?

The benefit of taking this approach is that it offers a new kind of answer to the Hard Question by offering a new conception of what properties of a computational system distinguish conscious from unconscious representations. With this reconceptualization we can deploy O'Brien and Opie's hypothesis for the sense of

4There are differences between "potentially explicit" and "tacit" representations. These differences become important when we consider the kinds of computations being performed within a cognitive system, but won't play a role in this short statement of O'Brien and Opie's hypothesis regarding consciousness (indeed O'Brien and Opie argue that a Vehicle Theory could only be true for connectionist systems and that the "potentially explicit" versus "tacit" distinction does not apply to such systems).

agency and answer the Hard Question in a way that doesn't seem to be falsified like the other answers considered here. In addition, a research agenda is set: why and how is a representation of own agency sometimes made explicit? Of course this question has not yet been answered. Indeed whichever form of the Hard Question we prefer it is clear that we have not yet answered it, although there seem to be two promising avenues to approach it. And so I implore us, as a community to ask of ourselves, now that we have made progress in understanding how a representation of own agency is elicited, *then what happens?*

### **CONCLUSION**

In this paper I have argued that in order to explain the sense of agency we need to move beyond proximal etiological explanations and consider the Hard Question. Although such accounts, including the comparator model and the inference to apparent mental sate causation, are powerful so far as they go, they fail to distinguish between conscious and unconscious representations of own agency. As a consideration of the occasionality problem suggests, not only is this a real distinction, but such representations can play very different roles in cognition. Finally, I have suggested that there are ways we can approach the Hard Question, and although some of the specifics of certain particular approaches might seem like non-starters on empirical grounds it should be clear that there are alternative approaches, both Functionalist and vehicle, available and specific questions to ask. Now what happens?

### **REFERENCES**

**Conflict of Interest Statement**: The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 01 April 2014; accepted: 29 May 2014; published online: 23 June 2014*. *Citation: Carruthers G (2014) What makes us conscious of our own agency? And why the conscious versus unconscious representation distinction matters. Front. Hum. Neurosci. 8:434. doi: 10.3389/fnhum.2014.00434*

*This article was submitted to the journal Frontiers in Human Neuroscience*.

*Copyright © 2014 Carruthers. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## From action intentions to action effects: how does the sense of agency come about?

#### **Valérian Chambon1,2,3\* † , Nura Sidarus 3\* † and Patrick Haggard 3**

1 Laboratoire de Neurosciences Cognitives, INSERM U960, Paris, France

2 Institut Jean Nicod, Ecole Normale Supérieure-EHESS, CNRS UMR-8129, Paris, France

3 Institute of Cognitive Neuroscience, University College London, London, UK

#### **Edited by:**

James W. Moore, Goldsmiths, University of London, UK

#### **Reviewed by:**

Chloe Farrer, CNRS, France Anouk Van Der Weiden, University Medical Center Utrecht, Netherlands

#### **\*Correspondence:**

Valérian Chambon, Institut Jean Nicod, Ecole Normale Supérieure-EHESS, CNRS UMR-8129, 29, Rue d'Ulm, Paris 75005, France e-mail<EMAIL_ADDRESS>Nura Sidarus, Institute of Cognitive Neuroscience, University College London, Alexandra House, 17 Queen Square, London WC1N 3AR, UK

Sense of agency refers to the feeling of controlling an external event through one's own action. On one influential view, agency depends on how predictable the consequences of one's action are, getting stronger as the match between predicted and actual effect of an action gets closer. Thus, sense of agency arises when external events that follow our action are consistent with predictions of action effects made by the motor system while we perform or simply intend to perform an action. According to this view, agency is inferred retrospectively, after an action has been performed and its consequences are known. In contrast, little is known about whether and how internal processes involved in the selection of actions may influence subjective sense of control, in advance of the action itself, and irrespective of effect predictability. In this article, we review several classes of behavioral and neuroimaging data suggesting that earlier processes, linked to fluency of action selection, prospectively contribute to sense of agency. These findings have important implications for better understanding human volition and abnormalities of action experience.

**Keywords: fluency, action selection, agency, angular gyrus, human volition**

### e-mail<EMAIL_ADDRESS>
†These authors have contributed equally to this work.

## **ACTION-EFFECT LINK AND COMPARATOR MODELS: A RETROSPECTIVE ACCOUNT OF AGENCY**

Agency is a key component of action experience. In a nutshell, agency refers to the sense of controlling one's own actions and, through these actions, events in the outside world. We rarely have an intense, clear phenomenology of agency, but we clearly recognize failures of agency when we experience actions that do not unfold as expected or fail to produce intended effects. One might even say that our sense of "authorship" becomes apparent only when it is falsified, resulting in a break of the flow from intentions to action effects that normally characterize experience. Thus, determining where the sense of agency comes from requires properly specifying where the break may occur along the intention-action-effect chain. Identifying the break may in turn depend on how we choose to specify the chain, and on the causal relation between its constituents (intention, action, effect).

On one influential view, agency implies a control mechanism that causally relates actions to their effects. More specifically, it implies a mechanism that has goals, and that controls actions to achieve them. This mechanism was first, and successfully, formalized as a *comparator model* (Wolpert et al., 1995; Miall and Wolpert, 1996). In its first incarnation, a comparator model translates intentions into outcomes, by continually monitoring whether action consequences occur, or do not occur, as predicted. Though originally formulated as models of motor control (Wolpert et al., 1995), comparator models have also been increasingly used to explain the subjective sense of agency (e.g., Blakemore et al., 2001). On the comparator account, agency is computed by matching predicted and actually experienced consequences of movement. In this framework, action effects are precisely those sensory events that can be predicted from one's intentions, using the specific intermediate mechanism of the comparator model (Wolpert et al., 1995; **Figure 1A**). Thus, the comparator model allows for two specific predictions. First, sense of agency should be strong when there is a close match between the predicted and the actual sensory consequences of an action, and should be reduced when predicted and experienced consequences do not match. Second, sense of agency necessarily occurs *late*, i.e., after an action has been performed, and sensory evidence about the consequences of action becomes available.

This view has received considerable empirical support from studies showing that spatial and temporal discrepancies between making an action and viewing visual feedback of the action reduce the sense that the observed action is one's own. Thus, introducing a spatial transformation between an action and its visual consequences reduces participants' sense of agency in proportion to

the mismatch induced. In one typical task, participants received distorted visual feedback of their hand moving a joystick. When the movement of the virtual hand did not correspond to the subjects' movement (Farrer and Frith, 2002), or when an angular bias was introduced between the subject's and the virtual hand's movement, participants more readily attributed it to another agent (Fourneret and Jeannerod, 1998; Farrer et al., 2003; Synofzik et al., 2006; David et al., 2007). Note that manipulating temporal relations between actions and outcomes had similar effects (Franck et al., 2001; Leube et al., 2003; MacDonald and Paus, 2003; David et al., 2007, 2011; Farrer et al., 2008). The so-called "intentional binding" effect provides another line of evidence for the role of temporal contiguity between action and outcome in the building of agency. The intentional binding effect has been first reported by Haggard et al. (2002): it refers to the subjective compression of the temporal interval between a voluntary action and its external sensory consequences. Thus, actions are perceived as shifted in time towards the outcomes that they cause, while outcomes are perceived as shifted back in time towards the actions that cause them (see Moore and Obhi, 2012, for a review). This temporal attraction is absent in cases of involuntary or passive movement. Equally, when participants simply judge the interval between action and effect, their judgments show a perceptual compression absent for equivalent passive movements (Engbert et al., 2008). The intentional binding effect would constitute an implicit, but reliable, measure of agency, as it only occurs when events in the external environment are precisely recognized as the *consequences* of one's action.

On comparator accounts, a positive sense of agency is the default operation when no mismatch between predicted and current states occurs (see Synofzik et al., 2008). It is the experiential output of sub-personal processes that mostly run outside consciousness. Crucially, although sense of agency relies on realtime motor signals, it can only be computed after those signals are compared with reafferent feedback. Thus, a reliable, explicit sense of agency may only be formed when reafferent (visual, motor, or proprioceptive) signals become available for matching with intentions. Thus, one cannot feel agency over any event until that event has been registered and processed in the brain. As a consequence, agency can only be *retrospectively* attributed, although it is informed by *on-line* signals about motor guidance and control (Chambon and Haggard, 2013).

Note the retrospective account on agency has several advantages. First, it is grounded on several classes of converging behavioral and neuroimaging evidence. Second, it primarily relies on a computational model that provides a convincing explanation for the link between action and effect: action effects are sensory events that can be predicted from one's action plans. However, an alternative possibility, that sense of agency is also generated *prospectively*, in advance of the action itself and before knowing the actual effect of actions, has received recent support (Wenke et al., 2010). On this view, selecting between alternative possible actions might itself generate a sense of agency. This view places a new emphasis on the intention-action, rather than the action-effect, link—i.e., on the process through which intentions are transformed into specific actions, to achieve desired effects. Importantly, this view suggests that agency may depend on realtime, prospective signals arising from internal circuits of action preparation, rather than on a *post-hoc*, retrospective comparison between predicted and current states of the environment.

## **INTENTION-ACTION LINK AND SELECTION FLUENCY: A PROSPECTIVE ACCOUNT OF AGENCY**

Previous studies have shown that judgments of agency tend to be related to how participants think that they perform in a task (Metcalfe and Greene, 2007). Similarly, errors in task performance may lead to a *feeling* of dysfluency during the task, without any explicit awareness of an error, and without the ability to explicitly report the error. Thus, a feeling that something went "wrong" during the control of instrumental action may be sufficient to modulate later judgments of control, even without being able to identify or explicitly report the error. The term "epistemic feeling" has been coined to describe this subjective, on-line, experience of an error (Arango-Muñoz, 2010; Charles et al., 2013). Importantly, such on-line experience strongly influences the sense of agency, as shown by recent priming studies. Thus, Wenke and colleagues showed that the sense of agency could be modulated by using subliminal priming to affect the *fluency* of action selection processes (Wenke et al., 2010; Haggard and Chambon, 2012, for a review). Interestingly, this procedure enabled a manipulation of the subjective sense of agency, without manipulating the *predictability* of action outcomes. In this experiment, participants pressed left or right keys in response to left- or right-pointing arrow targets. Prior to the target, subliminal left or right arrow primes were presented, unbeknownst to the subject. Prime arrow directions were either identical (compatible condition) or opposite (incompatible condition) to the subsequent target (**Figure 1B**). Responding to the target caused the appearance of a color after a jittered delay. The color patch can thus be considered as the action outcome. The specific color shown depended on whether the participant's action was compatible or incompatible with the preceding subliminal prime, but did not depend on the prime identity or the chosen action alternative alone. Unlike previous studies, therefore, the primes did not predict action effects, nor could any specific color be predicted on the basis of the action chosen. Participants rated how much control they experienced over the different colors at the end of each block (Wenke et al., 2010).

Analyses of reaction times (RTs) showed that compatible primes facilitated responding whereas incompatible primes interfered with response selection. More importantly, priming also modulated the sense of agency over action effects: participants experienced more control over colors that followed actions compatible with the preceding primes than over colors that followed prime-incompatible actions. Thus subliminal priming made action selection processes more or less *fluent*, and this modulation of fluency affected the sense of agency over action outcomes.1

These results have several important cognitive implications. First, they suggest that the sense of agency depends strongly on processes of action selection that necessarily occur before action itself. Second, strong sense of agency may be associated with fluent, uncontested action selection. In contrast, conflict between alternative possible actions, such as that caused by incompatible subliminal priming, may reduce the feeling of control over action outcomes. Third, this prospective contribution of action selection processes to sense of agency is distinct from predicting the outcomes of action, since action outcomes were equally (un-) predictable for compatible and incompatible primes. That is, these primes did not prime effects of action as in previous studies (e.g., Wegner and Wheatley, 1999; Aarts et al., 2005; Linser and Goschke, 2007; Sato, 2009). Therefore, participants could not retrospectively base their control judgements on match between primes and effects alone. Rather, their stronger experience of control when primes were compatible could only be explained by the fluency of action selection—i.e., by a signal experienced *before* the action was made, and the effect was displayed.

Finally, participants did not consciously perceive the subliminal primes. Therefore, participants' sense of agency could not be based on (conscious) beliefs about the primes. Instead, action priming itself presumably directly influenced the subjective sense of agency. Pacherie (Pacherie, 2008; see also Synofzik et al., 2008) has suggested that action selection conflict need not necessarily be conscious (Morsella et al., 2009). Such conflict may elicit the feeling "that something is wrong", without necessarily leading to knowledge about *what* is wrong. Wenke et al.'s study shows that subjects can rely on this *implicit feeling* to make judgments about their own control over action effects.2

## **DISSOCIATING PROSPECTIVE SENSE OF AGENCY FROM MOTOR PERFORMANCE**

Wenke et al.'s findings suggest that monitoring fluency signals generated *during* action selection could be an important marker for the experience of agency. However, it is also possible that

1 Subliminal priming would facilitate action selection by reducing conflict between alternative action programmes (Fleming et al., 2009). This facilitation of premotor processing would precisely be experienced as a feeling of action *fluency*, while the opposite effect of conflict between alternative actions would be experienced as *dysfluency*. Note this suggestion is analogous to the wellaccepted way that the feeling of "effort" is seen, as the experiential output of an increased demand in cognitive control (McGuire and Botvinick, 2010). Conversely, the literature usually defines fluency as an experiential consequence of smooth, *effortless* cognitive processing (e.g., Oppenheimer, 2008). In this sense, both terms are interchangeable—a fluent processing is an effortless processing–, and both may prospectively inform agency (e.g., Demanet et al., 2013).

2We take that selection fluency does not require to be *explicitly* represented to inform conscious experience of action. In fact, recent data suggest that fluency signals need to be kept implicit in order to influence agency on compatible trials—i.e., in order to be mistaken for actual control over action effects. Indeed, when primes are consciously perceived (i.e., presented at a supraliminal threshold), the compatibility effect is reversed: sense of agency is higher on *incompatible* trials (Damen et al., 2014).

participants might have estimated agency based on implicit monitoring of their own performance, such as their RTs. Since RTs are lower on compatibly primed trials (Dehaene et al., 1998; Schlaghecken and Eimer, 2000; Schlaghecken et al., 2008), participants would therefore feel more control on compatible trials, because they respond more rapidly. On this second view, agency would depend on *retrospective* monitoring of action execution performance (Marti et al., 2010), not on *prospective* monitoring of premotor fluency signals.

To distinguish between these two accounts of sense of agency, we used an experimental procedure that dissociated fluency of action selection from performance monitoring (Chambon and Haggard, 2012). Specifically, we increased the interval between mask and target to take advantage of a Negative Compatibility Effect (NCE) in priming. Longer mask-target latencies *increase* RTs following compatible primes, relative to incompatible primes (Schlaghecken et al., 2008). By combining this factor with Wenke et al.'s design for assessing sense of agency, it was possible to directly compare the contrasting retrospective (performance monitoring) and prospective (action selection) accounts. Specifically, if sense of agency depends on selection fluency, it should be greater when actions are compatibly (fluent condition) versus incompatibly (dysfluent condition) primed, irrespective of whether priming benefits (faster RTs) or impairs (slower RTs) performance. Alternatively, if sense of agency depends only on performance monitoring, it should be stronger for rapid versus slower responding, irrespective of whether priming is compatible or incompatible with the action executed.

Crucially, reversing the normal relationship between primetarget compatibility and RTs did not alter subjective sense of agency. Thus, in compatible NCE trials, participants experienced *stronger* control despite *slower* response times and higher error rates, compared to incompatible NCE trials (Chambon and Haggard, 2012; see also Stenner et al., 2014). These results suggest that the feeling of control normally experienced by subjects on compatible trials does not depend on retrospectively monitoring performance, thereby strengthening the evidence for a prospective contribution of action selection fluency to sense of agency.

In both Wenke et al.'s (2010) and Chambon and Haggard's (2012), experiments, priming did not influence the actual objective level of control that participants had over the colors presented after their actions. Indeed, the contingency between action and color effect was similar for compatibly-primed and incompatibly-primed trials. Importantly, the prospective sense of control identified in these experiments is therefore an illusion of control, since it is not based on differences in the actual statistical relation between action and effect. In other words, action selection is irrelevant to actual action-effect contingency, and thus to the agent's actual ability to drive external events. Although illusory, this prospective sense of control may nevertheless be a convenient proxy for actual control, because agents often just know what to do and what will happen next in most everyday life situations. In that sense, *fluent* action selection is generally a good *advance predictor* of actual statistical control over the external environment (Haggard and Chambon, 2012; Chambon et al., 2014). Prospective agency might thus reflect a learned experiential metacognition: if we can fluently select an appropriate action, then we are likely to get what we want, or fulfill our intentions.

As suggested above, internal signals of premotor fluency might not produce a strong conscious experience with distinctive content, but might influence the experience of surrounding events. Thus, fluency of action selection would not be experienced as such, but would presumably be experienced as something that goes "right" or "wrong" in the control of instrumental action, and thus seems relevant to sense of agency. In that sense, signals relating to the fluency of action selection would not be perceived for what they really are, but (mis-)attributed to the processes of actually controlling the action. Such a misattribution may foster the subject not to adjust her behavior accordingly. Indeed, it has been shown that behavioral adjustment does not only depend on the presence or absence of an error, but also on its cause (e.g., me vs. not-me) (Steinhauser and Kiesel, 2011). Thus, if participants misattribute dysfluency to lack of control on the selected action, and misattribute fluency to the process of actually controlling the action, then they should adjust their behavior less in the dysfluent, than in the fluent, condition—despite the fact that control is equally illusory in both conditions. Future work is required to test this assumption directly.

## **NEURAL SUBSTRATES OF PROSPECTIVE (FLUENCY-BASED) AGENCY**

Taken together, these findings suggest that neural activity in action preparation circuits *prospectively* informs agency, independent of outcome predictability, and actual performance. Tracking dysfluency in action selection networks (Miele et al., 2011; Nahab et al., 2011) could be the basis for this prospective sense of agency. Recently, we adapted the prospective agency paradigm for functional neuroimaging (Chambon et al., 2013). Specifically, we studied whether the angular gyrus (AG), a parietal brain region which has been shown to compute *retrospective* agency by monitoring mismatches between actions and subsequent outcomes (Farrer et al., 2003, 2008), may also code for a *prospective* sense of agency, by monitoring action selection processes in advance of the action itself, and independently of action outcomes.

Behavioral results replicated those of Wenke and colleagues. Again, participants experienced greater control over action effects when the action was compatibly versus incompatibly primed (Chambon et al., 2013). More importantly, this prospective contribution of action-selection processes to sense of agency was accounted for by exchange of signals between specific frontal action selection areas and the parietal cortex. First, we found that activity in the AG was sensitive to mismatches, but not matches, between prime arrow and actual response to the target arrow. Moreover, this activity due to the prime-target mismatch predicted the magnitude of subsequent sense of agency: for incompatible trials only, activity in the AG decreased as sense of control over outcomes increased. Importantly, this neural coding of non-agency occurred at the time of action selection *only*, as in Wenke et al.'s original experiment.

Second, connectivity analyses (psycho-physiological interaction) revealed that activity in the AG (signaling non-agency) in incompatible trials was negatively correlated with activity in the dorso-lateral prefrontal area (DLPFC; **Figure 2A**). Previous studies of willed action also noticed the same frontoparietal correlation, namely, that increased activity in DLPFC was associated with decreased activity in the AG (Frith et al., 1991). Our results are directly analogous: compatible primes might partly engage circuits for willed action, while prime-target incompatibility might relatively decrease activity in this circuit (Wenke et al., 2010). Thus, DLPFC *deactivation* would signal dysfluency in the selection of willed action, as a consequence of prime-target incompatibility. Decreased DLPFC activity due to incompatible primes would in turn result in a concomitant increase in AG activity and a subjective loss of control. Overall, this suggests that AG may monitor signals relating to fluency or dysfluency of action selection emanating from DLFPC and use them to (pre)construct an experience of agency. Importantly, under this interpretation, this monitoring of fluency signals by AG would occur *prior* to actions and their sensory consequences. This prospective contribution of AG to sense of agency can thus be distinguished from other functions such as action outcome monitoring. Interestingly, Farrer et al. (2008) demonstrated a role of AG in action outcome monitoring, but found a bilateral AG activation, which was slightly more ventral than the AG found here. In Farrer et al.'s study, AG activation varied with mismatch between predicted and actual sensory consequences of an action, while AG activation in Chambon et al.'s study was elicited by a mismatch between a prime-induced intention and response to a target (Chambon et al., 2013). The different localization found in these two studies could thus reflect a subdivision within the inferior parietal cortex, with more dorsal AG being involved in detecting mismatch between intention and action, independent of action consequences (Chambon et al., 2013), while more ventral AG would be involved in retrospectively comparing predicted and actual consequences of an action (Farrer et al., 2008).

Monitoring of fluency signals by AG might provide the subject with an on-line, subjective marker of volition, prior to action itself. As such, this finding sketches an important qualification of recent *post-hoc* determinist views of action control (Ackerman et al., 2010). In its strongest form, determinist views suggest that human behavior is unconsciously determined by subtle changes in the stimulus environment. On this view, individuals are not even aware of how their behavior is shaped and transformed, although they can retrospectively integrate general information about past actions and environmental cues to make inferences and narrative explanations about their own behavior (Wegner, 2002). While participants, in both Wenke's and Chambon's experiments, did not have any conscious experience of the subliminal primes, they did have a real-time *subjective experience* of their own action generation, which reflected the prime's capacity to influence action selection. In this respect, the ability to monitor fluency signals generated *during* action selection in AG might be an important part of what makes our action intentional, and thus a key component of the experience of agency—defined as the feeling that we are intentionally making things happen by our own choices and actions.

## **A CAUSAL EVIDENCE FOR THE ROLE OF AG IN THE CODING OF PROSPECTIVE AGENCY**

Although informative, this fMRI study was nevertheless limited in two key ways. First, the evidence was indirect, because of the correlational nature of fMRI. Secondly, it was not possible to pinpoint the precise *time* at which AG is involved in the prospective coding of agency owing to the relatively poor temporal resolution of fMRI. As we saw, the issue of timing is important for understanding *where* the sense of agency is computed within the intention-action-effect chain.

We recently addressed these two limitations by combining single-pulse transcranial magnetic stimulation (TMS) with subliminal priming of action selection and judgements of control over action effects. On two distinct experiments we assessed the effects of TMS over left AG on action selection processing, by linking TMS to either (i) the presentation of the arrow target; (ii) to action execution; or (iii) to the presentation of the action effect (color patch). We made specific predictions based on our previous fMRI findings. Because AG activation correlated with sense of agency only on incompatible trials, we assumed that this area monitored signals relating to selection fluency generated by DLPFC (Chambon et al., 2013). In this case, applying TMS over AG should prevent this region from monitoring any signals from DLPFC, and hence reduce the tendency for incompatibility primes to influence judgements of control.

Consistent with these predictions, we found that TMS over left AG abolished the compatibility effect (i.e., the difference between compatible and incompatible conditions) on sense of agency at the time of action selection *only* (**Figure 2B**), while TMS delivered shortly after presentation of the action effect did not alter experienced agency. Importantly, TMS had no effect on RTs. This suggests that TMS-induced disruption of AG did not interfere with action selection processing itself, but rather interfered with a circuit that monitors selection fluency to preconstruct the experience of control.

Previously it is has been suggested that the AG is involved in the retrospective construction of sense of agency by monitoring the consistency between predicted and actual sensory consequences of movements (David et al., 2008, for a review). When these predictions are violated sense of agency is reduced, and AG activity is increased. Results from our TMS study do not disagree with this view of AG function, but point to an additional role: by monitoring the consistency between action plans and required actions, the AG is also involved in *earlier* prospective aspects of sense of agency, relating to action selection and action programming.

Note the prospective *and* retrospective mechanisms have some general features in common. Both involve monitoring actionrelated signals or "cues" (such as re-afferent sensory feedback) as they become available, and comparing them with other relevant information for consistency (see Moore et al., 2009). We suggest that monitoring and checking is a very general function of the AG during instrumental action. Initial action intentions, such as those caused by subliminal primes in the series of studies described above, could be checked for compatibility with the action subsequently performed. These action selection cues may provide an important "online" marker of control as the action is unfolding. Not only would this provide an estimate of agency without the need to wait until sensory feedback becomes available but, as we have suggested (Chambon et al., 2013), it may protect against aberrant experiences of agency. For example, the sense of agency in patients with schizophrenia is characterised by excessive reliance on re-afferent sensory information generated by their actions, presumably due to poor, or unreliable, action selection processing (Voss et al., 2010). Prospective signals—such as fluency signals—may indeed provide an important counterweight to reafferent information, and hence may protect against xenopathic experiences (e.g., loss of control over one's actions and thoughts) such as those experienced in passivity symptoms. At the same time, excessive reliance on these prospective signals may produce the opposite delusion of omnipotence, in which the mere decision to act is incorrectly assumed to produce successful action outcomes. This latter illusion appears to be common in historical despots but is interestingly absent in depressed people (Alloy and Abramson, 1979). A robust and reliable sense of agency may thus require a balanced—and probably context-dependent—mixture of both prospective and retrospective components. Future work is required to test whether other (contextual of individual) factors may influence the interplay between these two components. For example, it has been convincingly suggested that priming effects on the experience of agency depend on the *level* at which the agent represents her behavior (van der Weiden et al., 2010). Thus, while some people represent their own behavior at a lowlevel (i.e., the instrumental level: in terms of *how* an action is done), some others represent their behavior at a higher level (i.e., the outcome level: in terms of *why* an action is done). Interestingly, the former may depend more heavily on prospective cues to agency (e.g., selection fluency), whereas the latter may show excessive reliance on retrospective information—i.e., on general information about past actions and outcome-related cues.

## **LINKING FLUENCY TO OUTCOME PREDICTABILITY**

Recent accounts of agency have highlighted that it results from the integration of various cues (Synofzik et al., 2008; Moore and Fletcher, 2012), which may emerge at different times (Farrer et al., 2013). Namely, it has been suggested that several agency cues may be weighted by their reliability in order to obtain a "Bayesian optimal" estimate of true agency (Moore and Fletcher, 2012). This view has received some support as studies have shown, for example, that changes in action-contingency affected the weighting of predictive and postdictive cues (Moore and Haggard, 2008; Wolpe et al., 2013). As outcome predictability was reduced, there was a greater reliance on *post-hoc*, inferential processes.

In the action priming studies described above, outcomes were fully contingent on a given action, in order to hold outcome predictability constant. However, it remained unclear whether action selection fluency would still be a relevant cue to agency in a context of greater uncertainty about action-outcome relations. Given our previous proposal that action selection fluency could serve as an advance predictor of successful action (Chambon and Haggard, 2012), one might predict that reducing action-outcome contingency would reduce the contribution of the prospective (fluency-based) relative to the retrospective (outcome-based) cue, to sense of agency. That is, if the outcome monitoring revealed that the action was in fact unsuccessful—i.e., outcomes did not match expectations, then the fluency of action selection would no longer be relevant.

To test this, we adapted our previous paradigm (e.g., Chambon et al., 2013) to involve a reduced contingency between actions and outcomes (Sidarus et al., 2013). Thus, a given action was associated with two possible colored outcomes on 66% of trials, but these colors would appear after the alternative action on the remaining 33% of trials. This allowed us to create situations in which outcomes could either match or mismatch expectations, given action-outcome contingencies. In addition, these outcomes would follow actions that were either compatibly or incompatibly primed. Therefore, we could assess the relative contribution of a prospective cue—action selection fluency, with a retrospective cue—outcome monitoring.

Results showed that participants' sense of agency was sensitive to manipulations of both the prospective and the retrospective cues. Compatibly primed actions were associated with higher control ratings than incompatibly primed actions. Additionally, participants reported a stronger sense of agency when the outcome was expected, compared to when the outcome was unexpected. More importantly, there was an interaction between the two variables. Incompatible action priming led to a significant reduction in control ratings when outcomes were unexpected, but not when outcomes were expected. At the same time, unexpected outcomes only reduced control ratings significantly when they followed incompatibly primed actions, and not compatibly primed actions (Sidarus et al., 2013). Thus, contrary to our predictions, selection fluency had a larger impact on sense of agency when outcomes were unexpected.

These findings reiterate the importance of action selection processes to the sense of agency. Even though outcomes were less predictable than in previous studies, we still found a similar effect of action priming on control ratings. What is more, the interaction between selection fluency and outcome expectation suggests that the sense of agency does not merely reflect information about action-outcome relations (e.g., Metcalfe and Greene, 2007). The sense of agency was drastically reduced only when both action selection was dysfluent *and* the outcome was unexpected. Prospective cues related to action selection fluency may thus make an independent contribution to the sense of agency from retrospective, outcome-based, cues.

Our findings are also not fully compatible with the cue integration models presently proposed for agency computation (Moore and Fletcher, 2012). Within this framework, it is the reliability of a given cue that determines its impact on the resulting sense of agency. Reliability is, however, a feature of the distribution of events. Thus, changes in cue reliability can only be assessed over a number of trials. Instead, our results suggest that the specific information carried by a given cue in a single trial can alter its weight relative to other cues. More complex Bayesian models of cue integration might be able to encompass these dynamic changes in cue weight. Yet, as mentioned above, perhaps a complete account of the sense of agency cannot be provided by simply maximising information about action-outcome relations.

These results overall support the idea that agency is the "default" assumption, which is only falsified, or reduced, when there is "sufficient" evidence against it.3 In some circumstances, it might be adaptive to maintain a sense of agency in the face of unexpected outcomes. Our environment mostly does not afford us fully predictable and contingent relations between actions and outcomes, but rather these tend to be probabilistic in nature. As such, we can learn these predictive relations, but we must also admit that predictions may be violated either due to the known statistical relations (e.g., when it is 66%), or due to random or outlier events. This type of *expected uncertainty* (Yu and Dayan, 2005) suggests that a mismatch between prediction and outcome does not always imply that the environment has changed, and one is not in control. In these situations, agency may be retained depending on information from other available cues, namely internal signals related to action selection.

## **ACTION SELECTION, AGENCY, AND EXPERTISE**

Interestingly, the experience associated to selection fluency (at least partly) overlaps with the phenomenal properties of what has been formalized as "flow" in positive psychology (Csikszentmihalyi, 1990). The flow is a particular mental state, described by expert people as a feeling of mindfulness and total commitment to the task at hand, often associated with an experience of a dilation of subjective time (Witt and Sugovic, 2010; see also Hagura et al., 2012). In some professional tennis players, for example, this feeling of "flow", resulting from a fluently selected (and perfectly executed) backhand stroke, may be associated with a "premonitory" anticipation of *where* the ball is going to hit the ground (Murphy and White, 1978). Consistently, our findings suggest that people may use the *fluency* (or ease) with which an action is selected as a good advance predictor of actual statistical control over the external environment.

Two hypotheses can be considered to account for the use of fluency signals in daily life. Using these signals adequately could first require learning *stable* relations between actions (e.g., the backhand stroke) and outcomes (e.g., where the tennis ball hit the court on average after that specific backhand). Indeed, simply having a feeling of fluently knowing which action to select does not guarantee the correct action outcome. Thus, fluencybased behaviors might only develop with expertise, once the brain has shifted from supervisory control to automatic or expert control. Under the expert regime, fluency would be used as an

3Whereas subjects experience less control when presented with incompatibly vs. neutrally-primed (using "neutral" primes constructed by superimposing left- and right-oriented primes), they do not feel more control when compatibly vs. neutrally-primed (Chambon and Haggard, 2012). These results are consistent with previous accounts suggesting that agency is a *default* experience. On this view, sense of agency would only really become apparent as a sense of *non-agency*, when the normal flow from our intentions to the effects of our actions is broken (Haggard, 2005; Chambon and Haggard, 2013).

*implicit proxy* for the current status (success or failure) of the action unfolding (Haggard and Chambon, 2012), and would substitute for explicit monitoring of the action-effect link through short-circuiting the process of "checking" the actual consequences of our actions.

In contrast, an alternative hypothesis would propose that we learn in our everyday lives to use fluency of action selection as a reliable cue to agency. Fluency signals may become a heuristics for assessing one's control over the external world, and we might even rely more on this heuristics in novel or uncertain situations. Before we know the statistical contingency between an action and its outcome in a given situation, we still have a sense of agency over what we do. Hence, we might rely on selection fluency to guide this sense of agency, until the more reliable action-outcome contingency cue is available. Although the Sidarus et al. (2013) study may provide some support for this alternative hypothesis, further research is needed to explore how the role of different agency cues may shift over time, during the learning of actionoutcome relations. Similarly, high levels of expertise in complex tasks may involve the recruitment of different processes, and also affect the types of cues that inform the sense of agency.

## **ACKNOWLEDGMENTS**

Valérian Chambon was supported by a Post-doctoral Fellowship of the Région Île-de-France (Paris), and by ANR-10-LABX-0087 IEC and ANR-10-IDEX-0001-02 PSL\*. Nura Sidarus was supported by UCL Impact Scholarship and the Belgian Science Policy Office project "Mechanisms of conscious and unconscious learning" (IAP P7/33). Patrick Haggard was supported by an ESRC Professorial Fellowship, an ESRC/ESF ECRP Research Project, an ERC Advanced Grant (HUMVOL) and by EU FP7 Project VERE, Work Package 1.

## **REFERENCES**

**Conflict of Interest Statement**: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 21 March 2014; accepted: 29 April 2014; published online: 15 May 2014*. *Citation: Chambon V, Sidarus N and Haggard P (2014) From action intentions to action effects: how does the sense of agency come about? Front. Hum. Neurosci. 8:320. doi: 10.3389/fnhum.2014.00320*

*This article was submitted to the journal Frontiers in Human Neuroscience*.

*Copyright © 2014 Chambon, Sidarus and Haggard. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms*.

## What can mirror-touch synaesthesia tell us about the sense of agency?

## *Maria Cristina Cioffi\*, James W. Moore and Michael J. Banissy*

*Department of Psychology, Goldsmiths, University of London, London, UK \*Correspondence<EMAIL_ADDRESS>
*Edited by:*

*Nicole David, University Medical Center Hamburg-Eppendorf, Germany*

*Reviewed by:*

*Bigna Lenggenhager, University Hospital Zurich, Switzerland*

**Keywords: agency, mirror-touch synaesthesia, self-other distinction, ownership, self-other representation**

Sense of agency (SoAg) refers to the feeling of control over one's actions and forms an integral part of our cognitive and social lives (Moore and Fletcher, 2012). For example, it is thought that the recognition of oneself as the agent of an action plays a fundamental role in self-awareness (Jeannerod, 2003). It is also thought that the experience of agency is important for guiding our attributions of responsibility (Haggard and Tsakiris, 2009).

The importance of SoAg is also demonstrated by the striking changes in this experience associated with various psychiatric (e.g., schizophrenia) and neurological (e.g., cortico-basal degeneration) disorders. While in recent years a number of studies have examined SoAg in these clinical groups, one group of individuals that have not yet been examined are those with mirror-touch synaesthesia (MTS). This opinion article seeks to explain why changes in SoAg may occur in MTS and also why mirror-touch synaesthetes could offer unique insights into the neurocognitive basis of SoAg.

For most of us, observing another person being touched activates neural regions in the somatosensory cortex that are also involved in experiencing touch (e.g., Keysers et al., 2004, 2010; Ebisch et al., 2008; Schaefer et al., 2012), however this activation does not lead to overt sensations of the observed event: we typically do not feel any tactile sensation when observing the tactile experience of others. On the contrary, people with MTS, approximately 1.6% of the population (Banissy et al., 2009), do experience overt tactile sensations to the observed event: they feel tactile sensations on their body when simply seeing touch to another's body (Blakemore et al., 2005; Holle et al., 2011; Banissy, 2013). These experiences are reported to be automatic (Banissy and Ward, 2007), enduring (Holle et al., 2011), and may be associated with broader differences in social perception (Banissy and Ward, 2007; Banissy et al., 2011; Goller et al., 2013).

Recent studies (e.g., Aimola-Davies and White, 2013; Holle et al., 2013; Maister et al., 2013) suggest that individuals with MTS have atypical self-other representations. For example, Maister et al. (2013) ran a study using the "enfacement illusion" paradigm. In the typical "enfacement illusion," participants are asked to say to what extent images of faces that were morphed between themselves or another person look like themselves or the other; they then watch a video in which the other person is touched in synchrony and congruent with a felt touch on the participant's face. After experiencing a synchrony between the observed and felt touch, the images that participants had initially perceived as containing equal quantities of self and other became more likely to be recognized as the self (Tsakiris, 2008; Tajadura-Jiménez et al., 2012). Maister et al. (2013) adapted this paradigm in MTS, by removing the physical touch component. That is to say that individuals observed touch to other people, but veridical synchronous touch was not physically applied to the face. They showed that MT synaesthetes experienced the same effect of "enfacement illusion" in the absence of a touch applied to their face, concluding that simply viewing the touch on others evokes changes in self-other representations in MTS. In this regard, MTS may therefore be characterized as bringing more malleable body representations, reflecting a blurring in the self-other distinction processes (Banissy and Ward, 2013; Maister et al., 2013).

This self-other blurring may be significant for SoAg. Experimental work in neurotypical individuals has shown how the deliberate blurring of the boundaries between self and other can have dramatic effects on SoAg. A good example of this is the so-called "Vicarious Agency" illusion, first demonstrated by Wegner et al. (2004).  A cardboard shield is placed behind their back to block their view of the experimenter standing behind them. The experimenter places their arm forward so that it appears where the participant's own arm would have been. This set-up is therefore aimed at engendering self-other confusion. Participants are then asked to look at the mirror in front of them, while the experimenter performs the gestures. Participants also wear headphones on which are played action previews (e.g., "wave your hand," "make the ok gesture"). These previews are either congruent or incongruent with the actions subsequently made by the experimenter. Wegner et al. found that participants experienced a SoAg and ownership over the arm that appeared in the mirror and that their experience of controlling the movements was increased when the previews were congruent with the action the experimenter made. In this way, we can see how an experimentally-induced blurring of the boundaries between self and other has a striking effect on SoAg. A strong prediction from this finding is that individuals with MTS will be more vulnerable to these agency illusions. This is something we are currently testing.