p = 0.279 (see **Figure 2**). In the fourth step of the model, we input the interaction of each individual difference regressor with Stim\_Familiarity to detect their potentially differential effects on outcome-sensitivity across Familiar and Novel conditions, but found no significant contribution from any ADHD-related or daily habit frequency variable (all ps > 0.085, χ 2 (4) = 6.19, p = 0.186, R 2 = 0.15, 1R 2 = 0.03). These results suggest that our sample exhibited outcome-insensitive well-learned habits across the board, but the degree of habitual control as assessed by change in NoGo accuracy was not significantly related to ADHD symptom severity.

### ADHD Symptom Severity and Habit Disruption

Similarly, we altered our pre-registered model to prevent multicollinearity, and performed a linear mixed model to examine the link between ADHD symptomology and habit disruption (see **Table 2**). The pre-registered analysis that

TABLE 2 | Hierarchical mixed model of ADHD symptomology and habit disruption: 1NoGo\_Accuracy.

Top layer of table depicts all regressors included in the hierarchical model. Model Comparisons layer depicts the predictive strength of each model, as compared to its previous step. VIF, Variance Inflation Factor. SE, Standard Error. CI, Confidence Interval. Log likel., Log likelihood. Significant p-values depicted in bold typeface. Analyses have been outlier corrected, with resulting deviations highlighted in the text. 95% confidence intervals were obtained by bootstrapping 1000 samples in each model.

violated assumptions of multicollinearity can be found in the section "**Supplementary Material**." In our corrected model, we input Gender, Phase\_Order, and Driving experience into step one, where none significantly predicted outcome-sensitivity (all ps > 0.142), model R 2 = 0.01. In step two, we added ASRS\_Inattentive, ASRS\_Hyperactive, Diagnosis, and COHS into the model, and found that none of these regressors yielded main effects on outcome-sensitivity (all ps > 0.162), and they did not significantly improve the predictive strength of the model, χ 2 (4) = 3.19, p = 0.526, R 2 = 0.03, 1R 2 = 0.01. We input Feedback as a regressor in step three, which contributed significantly to predicting outcome-sensitivity, βFeedback = −0.28, t(103) = −4.13, p < 0.001, and rendered the model a significant predictor of 1NoGo\_Accuracy, χ 2 (1) = 17.10, p < 0.001, R 2 = 0.11, 1R 2 = 0.08. We performed post hoc pairedsamples t-tests to confirm the beneficial effect of dual feedback.

We found that a significant NoGo accuracy impairment was evident in absence of dual feedback, t(103) = 5.33, p < 0.001, whereas the delivery of feedback yielded no significant accuracy impairments, t(103) = −0.50, p = 0.616 (see **Figure 3**). No individual difference measures' interaction regressor in step four significantly predicted outcome-sensitivity (all ps > 0.391, χ 2 (4) = 1.56, p = 0.815, R 2 = 0.11, 1R 2 = 0.01). These results suggest that the delivery of dual feedback indeed had a protective effect on outcome-sensitivity when managing familiar stimuli,

## Supplementary Analysis of ADHD Symptom Severity and Well-Learned Habits

albeit independent of ADHD symptom severity.

We performed identical analyses using 1Go\_Accuracy as DV and Subject as a random factor to capture the potential association between ADHD symptomology and a supplemental assay of outcome-sensitivity (see **Table 3**). Two participants' data were identified as outliers. Due to changes in statistical significance following outlier correction, we report our outlier-removed dataset below, highlighting any change in statistical significance due to outlier correction. Neither Gender, Phase\_Order, nor Driving experience predicted 1Go\_Accuracy (all ps > 0.323), model R 2 = 0.01. In step two, the Diagnosis regressor, which codes for the presence of a clinical ADHD diagnosis, made a significant contribution, βDiagnosis = 0.17, t(94) = 2.11, p = 0.038 (without outlier correction: βDiagnosis = 0.14, t(96) = 1.80, p = 0.076). Specifically, the presence of a diagnosis predicted more flexible Go actions. No

FIGURE 3 | Dual monetary/performance feedback prevents the incongruency-related impairments in NoGo accuracy, breaking the habit. Participants exhibit no incongruency-related NoGo accuracy impairments after receiving cumulative performance and monetary feedback (p = 616). Without this feedback integration, participants exhibit a significant impairment in NoGo accuracy when the color-response mappings are incongruent with daily experiences (p < 0.001). The habit disruption effect of feedback is independent of ADHD symptom severity (see Table 2 for individual difference measure contributions to habit disruption). Color of bars reflects NoGo stimulus colors.

FIGURE 4 | Familiar stimuli elicit incongruency-related impairments in Go accuracy. Analysis of our supplementary index of outcome-sensitivity, Go accuracy, yields evidence of habitual Go actions when managing familiar stimuli with color-response mappings that are incongruent with daily experiences (p < 0.001). In contrast, newly learned Go/NoGo contingencies evoke no significant change in Go accuracy regardless of color-response mapping, indicating intact goal-directed performance (p = 0.445). The differential habit expression effect across Stim\_Familiarity conditions depicted here is independent from ADHD symptom severity (see Table 3 for individual difference measure contributions to habit expression). Color of bars reflects Go stimulus colors.

other step two regressor significantly predicted 1Go\_Accuracy (all ps > 0.259) The step two model was not significantly improved from step one, χ 2 (4) = 5.56, p = 0.235, R 2 = 0.04, 1R 2 = 0.03. The Stim\_Familiarity regressor in step three served as a significant predictor, βStim\_Familiarity = 0.14, t(101) = 2.07, p = 0.010, improving the predictive strength of the model, χ 2 (1) = 4.44, p = 0.035, R 2 = 0.06, 1R 2 = 0.02. Paired-samples t-tests revealed a significant Go accuracy impairment in the Familiar condition, t(101) = 3.80, p < 0.001, but not the Novel condition, t(101) = −0.77, p = 0.445 (see **Figure 4**). Lastly in step four, other than Diagnosis × Stim\_Familiarity, βDiagnosis × Stim\_Familiarity = 0.19, t(97) = 2.71, p = 0.008, no individual difference measures significantly predicted 1Go\_Accuracy across the Familiar and Novel conditions (all other interaction ps > 0.125, χ 2 (4) = 10.43, p = 0.034, R 2 = 0.10, 1R 2 = 0.05). Because we only had six individuals with an ADHD diagnosis, we refrain from further interpretations of the contribution of the Diagnosis regressor. These results suggest that Go accuracy is differentially affected by whether familiar or novel stimuli serve as Go/NoGo signals, and a significant impairment is evident when familiar contingencies are incongruent with daily experiences. However, the habitual Go actions elicited by our familiar stimuli are independent of ADHD symptom severity.

### Supplementary Analysis of ADHD Symptom Severity and Habit Disruption

We investigated habit disruption via mapping-related changes in Go accuracy using a similar mixed model (see **Table 4**). Our

TABLE 3 | Hierarchical mixed model of ADHD symptomology and habit expression: 1Go\_Accuracy.

Top layer of table depicts all regressors included in the hierarchical model. Model Comparisons layer depicts the predictive strength of each model, as compared to its previous step. VIF, Variance Inflation Factor; SE, Standard Error; CI, Confidence Interval. Log likel., Log likelihood. Significant p-values depicted in bold typeface. Analyses have been outlier corrected, with resulting deviations highlighted in the text. 95% confidence intervals were obtained by bootstrapping 1000 samples in each model.

multicollinearity-corrected model identified two outliers. We report outlier-removed results below, accompanied by any changes in statistical significance following outlier correction. In step one of the mixed model, no controlled regressors predicted 1Go\_Accuracy (all ps > 0.093), model R 2 = 0.02. In step two, COHS was a near significant variable, βCOHS = −0.14, t(94) = −1.95, p = 0.054 (without outliercorrection: βCOHS = −0.08, t(96) = −1.05, p = 0.296), suggesting that a higher frequency of daily habits may predict more outcome-insensitive Go actions. Otherwise, no individual difference regressor served as a significant predictor of 1Go\_Accuracy (all other ps = 0.149), although the inclusion of step two regressors resulted in the Phase\_Order variable to yield a near-significant p-value, p = 0.066. Step two regressors in aggregate yielded only a near-significant contribution on the DV, χ 2 (4) = 8.56, p < 0.073, R 2 = 0.06, 1R 2 = 0.04. In step

incongruent with daily experiences (p < 0.001). The habit disruption effect of feedback is independent of ADHD symptom severity (see Table 4 for individual difference measure contributions to habit disruption). Color of bars

reflects Go stimulus colors.

three, the Feedback regressor significantly predicted outcomesensitivity as indexed by 1Go\_Accuracy, βFeedback = −0.26, t(101) = −4.07, p < 0.001, improving the predictive strength of the model, χ 2 (1) = 16.01, p < 0.001, R 2 = 0.13, 1R 2 = 0.07. This finding suggests that outcome-sensitivity as assessed by 1Go\_Accuracy is differentially impacted depending on the availability of dual feedback. Indeed, a post hoc pairedsamples t-test confirms a significant impairment in Go accuracy when no feedback is delivered, t(103) = 3.85, p < 0.001, whereas with feedback, no such impairment is evident, t(103) = −0.56, p = 0.573 (see **Figure 5**). In step four, we found that COHS × Feedback significantly predicted habit disruption, βCOHS × Feedback = −0.16, t(97) = −2.46, p = 0.016 (without outlier-correction: p = 0.120), suggesting that an increased daily habit frequency predicts a reduction in the beneficial effects of dual feedback in restoring goal-directed control. No other individual difference × Feedback regressor predicted habit disruption (all ps > 0.188, χ 2 (4) = 9.70, p = 0.046, R 2 = 0.16, 1R 2 = 0.04). Similar to our primary measure of outcome-sensitivity using NoGo accuracy, the protective effect of dual feedback on Go accuracy was independent from ADHD symptomology. However, we do observe a significant association between habitual tendencies in daily life and a difficulty in suppressing a well-learned habit.

### Exploratory Analyses: Go RT and Individual Difference Measures

We explored the potential association between prepotency to respond to the familiar Go stimulus and our individual difference

measures of ADHD symptom severity (ASRS\_Inattentiveness and ASRS\_Hyperactivity) and daily habit frequency (COHS). We reasoned that hyperactive individuals may exhibit a more pronounced prepotency to respond to Go stimuli, thus we were especially interested in the hyperactivity scale's association with RT. As hypothesized, we found a significant negative correlation between Go RT to the familiar green-Go colorresponse mapping and ASRS\_Hyperactivity, r = −0.25, p = 0.030, Holm–Bonferroni corrected (**Figure 6**), suggesting that higher hyperactivity scores are associated with faster Go responses. This relationship between hyperactivity and response latency was not apparent when the Go signal was incongruent with lifelong experiences (red-Go r = −0.05, p = 1, Holm–Bonferroni corrected), or when the Novel condition stimuli served as the Go signal (purple-Go r = −0.12, p = 0.630; blue-Go r = −0.10, p = 0.770, Holm–Bonferroni corrected). The association between familiar Go RT and ASRS\_Hyperactivity may suggest that individuals high in hyperactive symptoms may be exhibiting abnormally pronounced prepotency to stimuli that evoke habitual control.

### DISCUSSION

The neurobehavioral evidence of atypical reward-related processes in ADHD, and the scarcity of strategies to restore potential behavioral rigidities, motivated us to examine the expression and disruption of well-learned habits as a

Top layer of table depicts all regressors included in the hierarchical model. Model Comparisons layer depicts the predictive strength of each model, as compared to its previous step. VIF, Variance Inflation Factor; SE, Standard Error; CI, Confidence Interval; Log likel., Log likelihood. Significant p-values depicted in bold typeface. Analyses have been outlier corrected, with resulting deviations highlighted in the text. 95% confidence intervals were obtained by bootstrapping 1000 samples in each model.

function of ADHD symptom severity. To this end, we collected ADHD symptom severity metrics from a wide sample of participants in the general population and administered our Go/NoGo task that capitalizes on familiar green-Go/red-NoGo associations. Importantly, our incorporation of a motivational enhancement manipulation (i.e., cumulative performance and monetary feedback) permitted the study of habit expression and disruption. Our results replicate our recent documentation of familiar Go/NoGo stimuli evoking rigid habitual control, which is also rendered more flexible (i.e., goal-directed) with motivational enhancement (Ceceli et al., 2019). However, we found only modest support for the hypothesis of ADHD symptomology tracking behavioral rigidity and habit disruption. No measure of ADHD significantly predicted outcomeinsensitivity as assayed by color-response mapping-related NoGo or Go accuracy impairments. Our exploratory analyses,

however, supported our hypothesis of a significant association between pre-potency of habitual Go actions (i.e., familiar green-Go RT) and hyperactivity presentation. Furthermore, although not directly associated with ADHD, we also found a link between the frequency of habitual tendencies in daily life and habit disruption as indexed by our supplementary measure of outcome-sensitivity: mapping-related Go accuracy impairments. This significant association between daily habit frequency and difficulty breaking well-learned Go associations lends further credence to the idea that the familiar associations we capitalize on are indeed related to well-established, ecologically relevant habits.

A cardinal indicator of habitual control is the performance of an action regardless of the outcome value (Dickinson and Balleine, 1994). Accordingly, we believe that our Go/NoGo task captures outcome-sensitivity, in that the contingency change requires the agent to update which action produces the desired outcome. An impairment in the ability to override the welllearned habit may cause difficulties in flexibly updating the associations between cues and actions (i.e., the color-response mappings) that yield desirable outcomes (e.g., the value of performing a correct action).

We assert that our familiar stimuli elicit outcome-insensitive habits due to their well-established nature. The newly formed associations (e.g., purple-Go) are more labile, allowing the agent to exert goal-directed control regardless of changes to the color-response mappings. By this logic, these novel associations should eventually elicit habitual control with sufficient exposure – similar to overtraining of S–R associations in rodents (Adams, 1982). The magnitude of training necessary for this switch in motivational control using a change in Go and NoGo contingencies remains unknown. Previous research has suggested that pre-training stimuli over the course of an extra training session can yield stronger S–R execution in comparison to new stimulus sets (McKim et al., 2016). Possibly, extensively training the novel associations in our paradigm may also produce habitual control, albeit not with the behavioral rigidity elicited by the familiar associations that have been associated with go and stop actions over the course of development.

In both scientific reports and diagnostic criteria, ADHD is characterized by pronounced deficits in inhibitory control (Wodka et al., 2007; American Psychiatric Association, 2013). When taken together with the reward-related irregularities, we posited that ADHD may also be associated with an impaired motivational control system favoring habits over goal-directed behaviors. Our results do not support this hypothesis with our primary analyses, which could be due to a few key factors.

First, our study recruited participants from the general population and obtained a normal distribution of ADHDrelated symptom severity, such that most participants in our sample did not reach the clinical threshold for an ADHD diagnosis. This approach contextualizes any potential ADHDrelated impairment in motivational processes to a wider audience, thus expanding the applicability of our research. Consequentially, we are unable to sufficiently represent those who are most debilitated by the symptoms in question: individuals who meet the clinical threshold for ADHD. Any potential ADHD-related effect may therefore be weakened by the large proportion of individuals who present symptoms below the clinical threshold at magnitudes that do not impair daily functioning. Indeed, a study that recruited adults from the general population to examine ADHD symptomology-related inhibitory control disparities found only a modest association between symptom severity and Go/NoGo task accuracy with 440 participants (Polner et al., 2015). A study with a larger sample size (n = 1156) obtained from the general population pinpointed Go/NoGo impairments due to high ADHD-like symptoms, though these effects were sensitive to variations in task structure (e.g., speed and reward structure) (Kuntsi et al., 2009). Taken together with our results, although the ADHD–Go/NoGo impairment association is well-documented in clinical presentations of ADHD, symptom-based approaches may not be sensitive to such effects in the general population. Nonetheless, although there may be disorder-specific factors playing a role in behavioral flexibility that are undetected here, we had reasoned that sampling indiscriminately – that is, without diagnostic cutoffs – could expand the generalizability of potential symptom-related anomalies to the public.

An alternative explanation for the absence of a strong link between motivational control and ADHD symptomology is the notion that individuals with ADHD-like symptoms may also have compensatory mechanisms that promote adaptive behavioral output. For instance, despite the strong evidence of response inhibition deficits in ADHD, attention compensation supported by parietal brain activity has been documented, resulting in comparable Go/NoGo task performance (Ersche et al., 2017). Another possibility is that individuals with ADHD may adopt habitual or goal-directed control in different circumstances. A design that capitalizes on varying task difficulty or cognitive demands may be able to reflect such shifts in habitual and goaldirected processes that are sensitive to individual differences. Brain maturation is another candidate for behavioral similarities in ADHD and NT populations. ADHD is associated with a delayed maturation of the prefrontal cortex (Shaw et al., 2007), a region that is critical for error detection, reversal learning, and conflict monitoring. These processes are crucial for optimal Go/NoGo task performance (Garavan et al., 2002; Zhang et al., 2016), especially one involving changes to colorresponse mappings. Accordingly, adults with ADHD may produce signs of intact Go/NoGo performance due to the maturations in prefrontal regions, compensating for potential impairments that may have been evident with a less mature cortex (Carmona et al., 2012). Another potential compensatory mechanism may be driven by ADHD medications that act on the brain's dopaminergic systems. We did not ascertain whether our participants – with or without ADHD – were taking ADHD medication. Methylphenidate, for instance, has been reported to enhance executive function in individuals with ADHD, as well as in NTs (Schweitzer et al., 2004; Linssen et al., 2014; Moeller et al., 2014). These beneficial effects of ADHD medication on executive function have also been shown to extend beyond methylphenidate (Hosenbocus and Chahal, 2012). Our sample of adults with varying degrees of ADHD-related symptoms may be recruiting similar compensatory mechanisms that aid in maintaining goal-directed control. Future research that captures

developmental and pharmacological aspects of ADHD and goaldirected control may elucidate which of these mechanisms plays a critical role in adaptive motivational control.

We reasoned that because hyperactive ADHD presentation is associated with the number of impulsivity-related items endorsed on the ASRS (Kessler et al., 2005a), participants exhibiting high hyperactivity may execute quicker, impulsive Go actions. Our green-Go RT data supported our hypothesis, in that hyperactivity scores correlated with quicker responses to the well-learned habit eliciting stimulus. It should be noted that this finding was the result of an exploratory analysis. Nonetheless, our finding of a significant response latency and hyperactivity association bridges the fields of motivation and ADHD. Impulsivity, a core element of the hyperactive presentation of ADHD, is also associated with reflexive behaviors to cues and heightened variability in response latency (Kirkeby and Robinson, 2005). The heightened pre-potency to respond to habitual cues tracked by our hyperactivity scale may suggest an overlap in the motivational and inhibitory mechanisms underlying hyperactivity in ADHD, potentially explaining the lapses in behavioral output that result in higher RT and accuracy variability (Kirkeby and Robinson, 2005; Tamm et al., 2012). In other words, if hyperactivity predicts quicker responses to well-learned stimuli and high RT variability, this effect may be due to motivational and motor processes that are activated depending on past experience with the cue at hand. Future research will be imperative in effectively dissociating the motivational, attentional, and inhibitory processes that underlie response latency variability in ADHD.

In addition to the analyses reported here, an alternative method of exploring Go and NoGo performance is via signal detection. In a typical signal detection analysis, hits, misses, false alarms, and correct rejection values are used to derive d' – an estimate of response bias (Stanislaw and Todorov, 1999). Importantly, in each run of our task, a color-response mapping (e.g., green-Go) would only provide two of the four values that comprise a d' score (i.e., hits and misses, but not false alarms or correct rejections for this color). The remaining parameters would need to be extracted from the "incongruent" run (green-NoGo), which would make it difficult to obtain accurate response bias information. However, one can indeed investigate signal detection based on familiarity and congruency of the color-response mappings (e.g., where green-Go and red-NoGo together are coded as d'\_familiar\_congruent, and green-NoGo, red-Go are together coded as d'\_familiar\_incongruent). When performed as such, response bias results mirror our NoGo and Go accuracy findings reported here, in that (1) participants show high response bias when the color-response mappings are familiar and congruent with daily experiences, (2) response bias is significantly lower when familiar stimuli are mapped onto incongruent responses, (3) the two novel color-response mappings are similar in the elicited response bias, and (4) response bias does not show significant associations with ADHDrelated individual difference measures.

### Limitations

We acknowledge several limitations in the present study that should be considered in future investigations.

Although we were able to generalize our findings to a wider audience by recruiting without diagnostic cutoffs, we did not survey participants for history of psychiatric illnesses or psychoactive medication use. Several psychiatric conditions have been documented to affect motivational control (Griffiths et al., 2014). Furthermore, ADHD medications have been shown to improve executive function (Hosenbocus and Chahal, 2012; Linssen et al., 2014; Moeller et al., 2014), which may be related to the expansion of cognitive resources necessary to maintain goal-directed control. An interesting avenue to explore in future ADHD research may be the roles of psychiatric comorbidities and treatment history in the expression of habits.

Our study's primary hypotheses regarding habitual control and ADHD symptomology were motivated by reports of reward circuitry dysfunction in ADHD (Ceceli et al., 2019). However, we did not collect neural data that may speak to the potential links between ADHD symptomology and habitual control as mediated by neural function. The brain systems of reward processing and learning are outside the scope of our study, but the mechanisms underlying motivational control as related to ADHD symptoms may be effectively elucidated by a neurobiological approach. Future research that examines the potential disparities in the ADHD brain related to motivational control may advance our understanding of the disorder's pathophysiology.

We adopted a within-subject design to tackle the expression and disruption of habits over the course of two sessions. This design permitted us to compare habit expression and disruption at an individual basis while improving statistical power. However, it can be argued that administering a task twice to the same set of participants may introduce training effects. Our second session data suggest that participants did not significantly improve their performance in the face of congruent associations by merely undergoing the task in the previous session. However, the definitive method to circumvent potential training effects would be to apply a between-subjects design, in which separate sets of participants undergo the feedback and no-feedback sessions. We report in another study that adopts a between-subject design a similar pattern of results – motivational enhancement indeed disrupts the expression of well-learned habits (Ceceli et al., 2019).

### CONCLUSION

Attention deficit-hyperactivity disorder is a heterogenous psychiatric condition with debilitating consequences to behavior, neural processing, and well-being. In this study, we aimed to reveal the potential irregularities in managing well-learned habits by sampling symptom severity information from the general population. Although we did not find a strong association between motivational control deficits and ADHD-related symptoms, our data replicate a previous report of well-learned habit expression and disruption, and allude to a link between hyperactivity and pre-potency to respond to well-learned Go stimuli. Taken together with previous reports of compensatory mechanisms aiding in Go/NoGo task performance in ADHD, delay in cortical maturation in ADHD yielding differential inhibitory processes across children and adults, and our sample largely comprising subclinical ADHD presentations, a full understanding of the potential link between ADHD and motivational control may require a neurobehavioral and developmental approach.

### DATA AVAILABILITY

fpsyg-10-01997 August 30, 2019 Time: 17:39 # 15

The datasets and scripts associated with this study can be found in the **Supplementary Materials**.

### ETHICS STATEMENT

This study was carried out in accordance with the recommendations of the Rutgers University Institutional Review Board with written informed consent from all subjects. All subjects gave written informed consent in accordance with the Declaration of Helsinki. The protocol was approved by the Rutgers University Institutional Review Board.

### REFERENCES

### AUTHOR CONTRIBUTIONS

AC and ET designed the experiments. AC and GE coded the experimental paradigm and collected the data. All authors contributed to the data analysis and manuscript preparation.

### FUNDING

This work was supported by a grant from the National Science Foundation (BCS1150708) awarded to ET.

### ACKNOWLEDGMENTS

We would like to express our gratitude to the Learning and Decision Making Lab of Rutgers University-Newark for their helpful comments, and Miriam Rosenberg-Lee for guidance during task-design.

### SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpsyg. 2019.01997/full#supplementary-material

survey replication. Biol. Psychiatry 57, 1442–1451. doi: 10.1016/j.biopsych.2005. 04.001

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2019 Ceceli, Esposito and Tricomi. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Binary Theorizing Does Not Account for Action Control

#### *Bernhard Hommel\**

*Cognitive Psychology Unit, Institute of Psychology, Leiden University, Leiden, Netherlands*

Everyday thinking and scientific theorizing about human action control are equally driven by the apparently obvious contrast between will and habit or, in their more modern disguise: intentional and automatic processes, and model-based and model-free action planning. And yet, no comprehensive category system to systematically tell truly willed from merely habitual actions is available. As I argue, this is because the contrast is ill-conceived, because almost every single action is both willed and habitual, intentional and automatic, and model-based and model-free, simply because will and habit (and their successors) do not refer to alternative modes or pathways of action control but rather to different phases of action planning. I further discuss three basic misconceptions about action control that binary theorizing relies on: the assumption that intentional processes compete with automatic processes (rather than the former setting the stage for the latter), the assumption that action control is targeting processes (rather than representations of action outcomes), and the assumption that people follow only one goal at a time (rather than multiple goals). I conclude that (at least the present style of) binary theorizing fails to account for action control and should thus be replaced by a more integrative view.

#### *Edited by:*

*John A. Bargh, Yale University, United States*

#### *Reviewed by:*

*Bas Verplanken, University of Bath, United Kingdom Maurizio Tirassa, University of Turin, Italy*

*\*Correspondence: Bernhard Hommel<EMAIL_ADDRESS>
#### *Specialty section:*

*This article was submitted to Cognitive Science, a section of the journal Frontiers in Psychology*

*Received: 03 May 2019 Accepted: 28 October 2019 Published: 14 November 2019*

#### *Citation:*

*Hommel B (2019) Binary Theorizing Does Not Account for Action Control. Front. Psychol. 10:2542. doi: 10.3389/fpsyg.2019.02542*

Keywords: action control, dual-route models, goal, automaticity and control, intention

## BINARY THEORIZING ON ACTION CONTROL

### Will vs. Habit

The study of action control was driven by binary theorizing right from the start. In his first systematic analysis of the human will, Ach (1910) postulated that will can be best studied by analyzing the degree to, and the conditions under which, it can overcome what Ach considered its natural opponent: acquired habits. To achieve that, he developed what he called the *combined method* ("kombiniertes Verfahren"), which first established a particular habit, defined as a set of stimulus-response associations reflecting a particular stimulus-response rule, and then changed the instruction in such a way that participants were now to respond differently to the previously acquired stimulus set (see Hommel, 2000a). For instance, participants may first learn to read through lists of nonsense syllables that were followed by a rhyme (e.g., zup → tup, tel → mel) over an extended time period and then respond to the same stimulus syllables by changing the order of the letters (e.g., zup → puz, tel → let). As predicted, participants were slower and produced more errors when applying the new instruction to a stimulus set that was previously related to different responses than when working with a new set. The idea was that being exposed to lists constructed according to the first rule created stimulus-response habits that would need to be overcome in order to successfully apply the second rule. Accordingly, the degree to which participants were able to overcome the previously acquired habit (i.e., the difference in performance on old versus new sets) was taken to measure "willpower," which was shown to differ between individuals (which was taken to diagnose individual willpower) and to vary systematically with the practice given on old sets (which was taken to increase the strength of the habit).

It is easy to see that this pioneering approach has survived until today, even though researchers less frequently take the effort to induce habits experimentally anymore: they often exploit existing habits, such as the tendency to read words even if one is to name their color, as in the notorious Stroop task (Stroop, 1935; even though Stroop himself did analyze the impact of experimental training). Like in Ach's studies, the degree to which performance is impaired with stimulus sets that are assumed to activate the hypothetical habit (such as words denoting response-incompatible colors in the Stroop task) as compared to suitable control sets (such as nonwords, non-color words, or words denoting response-compatible colors) is taken to reflect the strength or weakness of willpower, which meanwhile has been relabeled as "cognitive control" or "executive function" presumably in an attempt to get rid of the phenomenological connotations of the will concept (Goschke, 2003).

Working with binary oppositions such as will and habit has been taken to reflect human nature (Newell, 1973; Melnikoff and Bargh, 2018), and so it comes as no surprise that the will/ habit couple has survived in various disguises until today. Its long-standing history tends to be systematically underestimated by available reviews, which for instance have dated back its introduction into theorizing about action control to the work of Tolman (1948, see Dolan and Dayan, 2013), Atkinson and Shiffrin (1968, see Monsell and Driver, 2000), or Dickinson (1985, see Gillan et al., 2015)—thus rather generously neglecting the pioneering study on the phenomenology of will by Michotte and Prüm (1911); the first systematic experimental program on studying will and habit by Ach (1910, 1935), which spanned no less than 30 years; the first approach questioning the goalindependence of habits by Lewin (1922a,b); and the other 200 or so studies on action control summarized in Ach (1935) already.

The basic thought underlying the opposition between will and habit is that some responses are so strongly associated with particular stimuli that encountering the stimulus is sufficient to activate the response. This holds for rhyming in Ach's studies—seeing a nonsense syllables triggers the overlearned rhyming response, reading in the Stroop task—seeing the word is sufficient to trigger some reading tendency, and performing a left or right response in the Simon task (Simon, 1969) processing a left or right stimulus triggers a spatially corresponding action (Kornblum et al., 1990). The basic setup of all tasks investigating the interplay between will and habit puts the two against each other, just as recommended by Ach (1910), by instructing individuals to carry out a relatively uncommon or counterintuitive action B to a particular stimulus that is assumed to be strongly associated with another action A. If then any experimental evidence can be found that action A was activated to at least some measurable degree, the participant is thought to have experienced an action-control problem that was due to the fact that practice established an association between A and the stimulus, so that encountering the stimulus would activate action A even under circumstances where A is not appropriate and not wanted.

Very soon after Ach's claims that stimulus-response associations can challenge and may even outcompete the processes controlled by the actual goal, Lewin (1922a,b) reported findings calling for a more moderate view. On the one hand it was possible to counteract an intense intention with a habit that relied on few, sometimes just one repetition but, on the other, 300 repetitions were insufficient to have any impact. According to Lewin (1928), the key to understand the impact of habits has to do with their specific role in the current action plan. On the one hand, habitual actions do not represent real alternatives to intentional actions, in the sense that people would face difficulties to decide whether they should name the color of a Stroop word or read it. Lewin suggests that the intention to open a door that requires pushing the handle up, rather than down, will not be hindered by the thousands or so previous repetitions of opening doors by pushing the handle down. On the other hand, however, habitual actions do have the potency to interfere if they are embedded into a larger action context, such as if one is to open the door on one's way to get a glass of water from the other room.

The same principle seems to apply to the Stroop effect, which is very pronounced (often >100 ms effect size) if the response set consists of spoken color words (i.e., the responses that reading the words would produce) but often dramatically shrinks or disappears with keypressing responses (e.g., McClain, 1983)—and even the effects that keypressing responses sometimes do produce seem to be artifacts due to task-irrelevant but spontaneously occurring internal naming strategies (Martin, 1978; Mascolo and Hirtle, 1990). In other words, the Stroop effect is likely to depend on introducing an obvious contradiction by requiring participants to attend to, and actually generate color words and at the same time nominally declaring color words task-irrelevant and to-be-ignored. Another obvious contradiction results from the fact that, in the standard Stroop task (as well as in other tasks following the same rationale), violating the instruction by reading the word actually pays off in 50% of the trials. This means that, on average, participants are rewarded for unintentionally or intentionally reading the word, especially given that word-reading is faster and requires less effort—just because of the more elaborate practice. That this is an important ingredient of the task is obvious from the finding that the size of the effect varies systematically with the percentage of the payoff: it becomes stronger if payoff increases and weaker if it decreases (e.g., Logan and Zbrodoff, 1979). This suggests that the impact of habitual action tendencies is anything but non-intentional, and clearly very sensitive to the expected outcome—a theme I will get back to below.

### Controlled vs. Automatic

As pointed out by Goschke (2003), theories on action control have seen a rather dramatic conceptual overhaul since the early days of Michotte, Ach, and Lewin. While the pioneering approaches were still strongly connected to the phenomenology of willing and acting, understanding which was even an explicit theoretical aim of Michotte and Ach, later theorizing preferred a less "subjective" terminology that was inspired by the increasingly popular computer metaphor for the description and analysis of human cognition in the 1950s and 1960s (Broadbent, 1958; Neisser, 1967). This terminological preference favored less colorful concepts like "controlled" versus "automatic" processing over the old-fashioned terms will and habit. Even though the basic idea was the same, the explanations changed in flavor: whereas the older approaches tried to explain the strong impact of habits by referring to an assumed cause—the strong stimulus-response association driving the habitual action, the new generation of processing theories tended to emphasize different degrees of speed and efficiency of the underlying processes (even though some studies still tested the practice-dependency of automaticity directly: e.g., Schneider et al., 1984; Smith and Lerner, 1986; MacLeod and Dunbar, 1988). For instance, the observation that responses are easier to perform in response to particular stimuli than others (e.g., left rather than write keypresses to stimuli appearing on the left) was explained by postulating the existence of a particular "population stereotype" (Fitts, 1951). At the surface, accounts of this sort do not seem to go beyond redescribing the actual finding in theoretically sounding terms, but they often implicitly rely on associationist logic: one way or another, such shared stereotypes must emerge from shared practices and training, which implies that stereotype is just another word for an associative structure linking stimuli to particular responses.

In other approaches the correspondence between controlled versus automatic processes on the one hand and will versus habit on the other is even more opaque. For instance, in their comprehensive model of stimulus-response compatibility, Kornblum et al. (1990) attribute the impact of what previously counted as habit to automaticity. It is automaticity that does the major trick in the explanation of why irrelevant stimuli seem to be able to trigger responses that conflict with the actually intended action, like in a Stroop task. Where automaticity comes from is a topic that the authors explicitly neglect: they briefly consider the possibility that training plays a role but then choose "not to make practice a major focus or concern" in their model (p. 263). Again, this renders the major theoretical contribution to the question of *why* irrelevant stimuli can trigger conflicting responses a mere reformulation of the empirical observation in theoretical terms1 .

These and other theoretical developments indicate that the systematic replacement of the will/habit concept by the controlled/ automatic concept has tempted at least more cognitively oriented theorists as cited above2 to refocus the theoretical attention away from the possible causes of the impact of the relevant information on action control to the consequences—away from the possible role of overlearning to the resulting automaticity. As a consequence, in these approaches automaticity was no longer defined with respect to its origin, such as the amount of training necessary to achieve it, but with respect to its opponent: the intention or control process. Note that this is a dangerous theoretical twist. The explananda targeted by control/automaticity theories derive from empirical observations that some behavior or some aspects of behavior do not fully comply with the instructions given to the investigated participants: they tend to read words rather than naming their color and press the key that spatially corresponds to the stimulus even if they should do the opposite. A certain lack of control is thus inherent in these observations, which renders the attempt to explain the observations by referring to automaticity circular: if automaticity is only defined by the absence of control, and if control is defined by compliance with the experimental instruction, the observed behavior must be automatic. In other words, automaticity cannot be an explanation because it is an integral component of the description of the to-be-explained phenomenon—automaticity is an explanandum, not an explanans!

These terminological confusions aside, it is fair to say that true automaticity has yet to be demonstrated. Kornblum et al. (1990) suggest applying the definition of Kahneman and Treisman (1984, p. 43), according to whom a strongly automatic process is one that is "neither facilitated by focusing attention on [its object] nor impaired by diverting attention from [it]," whereas "a partially automatic process is one that is normally triggered without attention directed at its object but is facilitated by having attention focused on it" (Kornblum et al., 1990, p. 261). "According to this view," Kornblum et al. (1990) continue, "an automatic process could under some conditions be attenuated or enhanced. However, under no conditions could it be ignored or bypassed." I have already mentioned evidence suggesting that even the Stroop effect, thought to be one of the milestones of demonstrating true automaticity, can disappear by simply changing the response set. However, such evidence might be discounted by considering a role of attention, which might be drastically reduced by this change and thus make the automaticity only partial. Moreover, Kornblum et al. claim true automaticity only for feature-overlap between stimuli and responses, which arguably is reduced, in some sense even eliminated by changing the response set in a Stroop task. However, automaticity can be shown to not exist even without changing the responses.

For instance, Valle-Inclán and Redondo (1998) presented participants with a Simon task, in which they responded to red and green colored circles by pressing the left and right response keys, respectively. In one condition, participants received the stimulus-response mapping first and were then presented with the lateralized color circle. Electrophysiological recordings showed that the presentation of the stimulus led to an increased activation in the cortical hemisphere opposite to its location—a classical lateralized readiness potential that is thought to represent response activation of the contralateral response hand (Eimer, 1995). This potential was even seen if the actual response

1 According to Lewin (1931), the idea that categorizing a particular phenomenon is sufficient to explain it is a reflection of what he called Aristotelian psychology (a theoretical attitude that is very typical for stage approaches to studying human information processing: Hommel, in press), which he contrasts with Galilean psychology that seeks to unravel the actual functional mechanisms. 2 This is not to say that attempts to systematically control the degree of automaticity acquired through experimental practice no longer exist. The learning-theoretical tradition to make training/exercise part of the experimental design has survived especially in the cognitive neurosciences (e.g., Schwabe and Wolf, 2009) and applied areas related to lifestyle issues and addiction (e.g., Watson et al., 2014; Lin et al., 2016).

required movement of the other hand, suggesting that it indicated the potency of the stimulus to automatically activate the spatially corresponding response hand. In another condition, the stimulus appeared first, and only thereafter the stimulus-response mapping was presented. If, according to the definition of Kahneman and Treisman and Kornblum et al., the association between stimulus location and response would be strongly automatic, the presentation of the stimulus should generate the same electrophysiological response as in the other condition. If the association would be partially automatic, the stimulus might show a reduced electrophysiological response. However, the findings showed no response whatsoever. If anything, this suggests that implementing the instruction is a precondition for automatic responses to occur, which means that they are neither fully nor partially automatic (cf., Trafimov, 2018) but what Bargh (1989) has called conditionally automatic.

A key problem with dealing with the concept of automaticity is that it remains a moving target in the literature. For instance, some authors (like Kahneman and Treisman, 1984) speak of automatic processes while others speak of automatic actions (e.g., Wheatley and Wegner, 2001). Some authors have argued that automatic processes need to meet all criteria for automaticity to deserve this label (what Moors and de Houwer, 2006, call the "all-or-none view"; e.g., Johnson and Hasher, 1987), while others were more liberal, allowing for various combinations of some of the criteria (e.g., Bargh, 1994; Moors and de Houwer, 2006), and the fact that the discussed criteria themselves vary extensively from author to author (see Melnikoff and Bargh, 2018) did not help to find a broad consensus either. For instance, while Kahneman and Treisman considered a process automatic if it is "neither facilitated by focusing attention on [its object] nor impaired by diverting attention from [it]," Bargh (1994) suggested a combination of a lack of awareness and intentionality, high efficiency, and a lack of motivation (a criterion that appeals to the desire criterion that I will criticize below), and Moors and de Houwer (2006) extend this list to eight criteria, according to which automaticity might refer to processes that are unintentional, uncontrollable, goal independent, autonomous, purely stimulus driven, unconscious, efficient, and fast.

I will not provide point-to-point point reviews of these criteria but do like to set the stage for the following discussion by means of two comments: first, the sheer number and variability of suggested criteria for sorting processes into automatic versus intentional ones, together with the fact that authors increasingly give up the idea that automaticity criteria might converge onto any coherent category (Bargh, 1994; Moors and de Houwer, 2006; Melnikoff and Bargh, 2018), undermine the original idea that cognitive processes can be categorized into two non-overlapping categories. Second, the criteria that have been suggested so far undoubtedly relate to measurable features of processes but there are reasons to doubt whether they even speak to the question of willed vs. non-willed behavior. As I will elaborate below, this is because: (1) goals and intentions control *outcomes* of behavior but not the *processes* producing it, which renders the connection between action control and criteria like controllability or autonomy questionable; (2) selecting an action emerges from the *goal-driven but fully automatic competition* between automatically executed action tendencies, which undermines the very idea that processes might be non-automatic in principle; and (3) the selection value that processes bring to this competition may well refer to the efficiency and speed of the action that this process represents, suggesting that the relevance of these criteria in action selection should be considered a sign of intentionality rather than the opposite.

### Model-Based vs. Model-Free

The most recent version of will/habit thinking comes in the disguise of models contrasting model-based and model-free systems. This contrast refers to two kinds of modeling reinforcement learning (e.g., Sutton and Barto, 2017): modelbased learning is assumed to rely on a state-transition model, which accumulates knowledge about the current state, the possible actions this state allows, and the state that would follow when taking this action, and a reward model that connects end-states with particular rewards. Hence, this kind of learning is based on a kind of model of the environment, which allows forward-planning and reward-maximization even when the environment changes. Model-free learning, in contrast, does not consider sequential dependencies like state-actionoutcome relationships or rewards but relies on stored selection values for all previously experienced state-action contingencies.

It is fair to say that there is no coherent theory integrating the available thoughts about how these systems work and how they interact, and it is also fair to say that quite a bit of confusion exists regarding what the terms model-based and model-free imply. One idea is that the goal-related model-based system stores contingencies between actions and outcomes while the automatic, model-free system stores stimulus-response associations (Dickinson and Balleine, 1994). According to this idea, model-based action implies consideration of the expected outcome whereas model-free action is driven by some contextual cue—a metamorphosis of the traditional habit. Others have criticized this conceptual opposition. For instance, Miller et al. (2019) have argued that the original idea assumes that habits are outcome-blind ("value-free"), whereas modern reinterpretations (e.g., Daw et al., 2005) imply that habits and model-free actions are driven by a reward-maximization process, that is, a process that depends directly on potential outcomes. Given that habit strength, the parameter that conventional habit theorists consider to be crucial for the probability to select a stimulus-response association, can well be considered a kind of selection value, the difference between value-free and value-based modeling might be less dramatic than Miller et al. (2019) assume. However, in their review on habits, Wood and Rünger (2016) question whether habits can be equated with model-free learning in view of suggestions that habits are acquired through model-based processes (Dezfouli and Balleine, 2012) and failures to find relationships between the strength of model-free learning and habit formation in individual-difference studies (Friedel et al., 2014; Gillan et al., 2015). Hence, it is clear that the model-free/ model-based framework is still under development and it remains to be seen whether a systematic connection between model-based/model-free learning on the one hand and will/ habit on the other will emerge. In any case, model-free action is considered to be insensitive to current action goals, whereas model-based algorithms are assumed to compute transition probabilities (e.g., an agent's likelihood of being in a wanted state after having performed a given action), which are used to compute the expected value of actions by comparing the states they are predicted to produce to the states the agent wants to establish. Some approaches assume that the two systems compete for action control (e.g., Gillan et al., 2015), while others assume that they can be integrated (Krueger and Griffiths, 2018). Some authors consider the model-based/model-free approach a strongly advanced version of the original will/habit approach (e.g., Dolan and Dayan, 2013), while others consider the two pairs of concepts basically equivalent (e.g., Friedel et al., 2014).

However, the probably most defining two novelties in the context of the model-based/model-free approach are the contrast between action-outcome contingencies, which are related to the model-based/goal-related system, and stimulus-response associations, which are the main ingredients of the modelfree/habitual system (De Wit and Dickinson, 2009), and the experimental procedure used to test whether a particular action relies on one or the other system. The latter is based on Heyes and Dickinson's (1990) "desire criterion" of voluntary action, which together with the "belief criterion" serves as diagnostic indicator of whether a particular action is based on a goal. The belief criterion requires the voluntarily acting agent to know about the current action-outcome relation and the desire criterion requires him or her to actually want the current outcome. Given that voluntary action is commonly defined as an activity directed toward the creation of some intended effect, the belief criterion is uncontroversial and explicitly or implicitly shared by any approach to voluntary action control (see Hommel and Wiers, 2017). The role and relevance of the desire criterion is less clear, however. The key procedure to assess whether the desire criterion is fulfilled is test after satiation, which reflects the behaviorist heritage of the model-based/model-free approach and the fact that it is mainly based on experiments carried out with rodents. For instance, participants who like popcorn would be tested for popcorn-related actions before and after receiving the opportunity to eat as much popcorn as they like (e.g., Watson et al., 2014). If they would show similar attentional and behavioral biases toward popcorn after the sating procedure as they showed before the procedure, the corresponding behavioral tendency would be considered to rely on the modelfree system and the stimulus-response associations it contains. The rationale for that conclusion seems straightforward: the sating procedure should make sure that participants no longer want popcorn, so if they would still be showing popcornapproaching behavior this cannot rely on an active popcorngetting goal—leaving a previously acquired popcorn-getting habit as the only option.

But is this rationale watertight? Let us consider why a person might eat popcorn. She may like digesting popcorn, feeling popcorn in her mouth, smelling popcorn, listening to the sound of popcorn being chewed, the attention she attracts from other popcorn-loving individuals, the satisfaction of having access to one's favorite food, the entertainment of filling time with a liked activity, and more. Liking popcorn is thus not a simple desire for one single aspect of popcorn-eating behavior but rather a complex compound of what one might call desire aspects or subdesires. Which of those would be sated by eating as much popcorn as one likes? Being stuffed with popcorn might make the digesting aspect less attractive, but would it eliminate the joy experienced by any of the other aspects? How reasonable is it to expect that the intentional component of the behavior of a sated popcorn-lover would be identical to the behavior of a popcorn-hater or of one who just does not care about popcorn? I suggest that the fundamental flaw of satiation logic consists in the idea that agents have just one single goal and that this goal is comprehensively captured by the aspect of the goal that the sating procedure is targeting (Hommel and Wiers, 2017). While it is not impossible that this is indeed the case, it is not very likely either.

Moreover, real human actions do not only rely on more than one goal aspect, they also consist of multiple elements: eating one popcorn consists in locating it in a nearby spot, moving one's hand toward it, opening and closing the hand until the popcorn is being grasped, moving it to one's mouth, opening the mouth, moving the popcorn inside, dropping it, closing the mouth, and starting to chew. Most of the elements of this action pattern have been discussed as the paragon of goal-directed voluntary action in the literature on grasping (e.g., Jeannerod, 1988; Milner and Goodale, 1995), which does not seem to fit with the classification of the entire pattern as a non-intentional stimulus-driven habit. One might object that the grasping part of the action may well be intentional and the popcorn part may not, but this is exactly my point: actions commonly comprise of multiple goals and it is unlikely that any satiation procedure can ever target all of them.

Finally, if all the popcorn-related behavior of the sated popcorn-lover would really be run by the model-free system alone, why would she actually eat the popcorn? Popcornlovers are likely to have done many things with popcorn apart from eating: buying and putting it into the bag, carrying it home and putting it into the cupboard, unpacking it and putting it on the table, offering it to others, cleaning the table from it, and throwing the remains into the trash, and so forth. The stimulus popcorn must thus be associated with many different responses, which raises the question which of the corresponding stimulus-response habits are triggered by the popcorn after satiation. What experiments show is that even the most popcorn-loving participants show contextually appropriate behavior even after satiation: they may eat some if they stand in front of it, but they do not clean the table from it, store them, or do other things that would not fit the experimental context and the social situation it creates. If so, sating the popcorn-lover does not seem to prevent her from showing contextually and socially appropriate popcorn-related behavior, which is not well-covered by calling it model-free.

### MISCONCEPTIONS IN BINARY THEORIZING

This brief and incomplete historical tour through some of the highlights of binary theorizing on action control was intended to show that none of the suggested terminological couples really works. Practicing stimulus-response combinations is likely to change the representations thereof, and presumably makes these representations more available under certain circumstances. However, there is still no evidence that stimuli can do what intentions and goals can: to trigger a particular response. What stimuli are capable of is to trigger misleading action tendencies under circumstances that are dictated by the kind and generality of the action goal, and to the degree that they are primed and enabled by the goal, whereas the actual association strength often fails to predict the degree to which representations of stimulus-response combinations affects action control. The opposition of controlled and automatic processes suffers from similar problems and from the lack of convincing demonstrations of true automaticity. The available demonstrations are consistent with the idea that automatic processes are enabled by the goal (as suggested by Exner, 1879; James, 1890; Bargh, 1989; Gollwitzer, 1993), so that it is the goal that eventually determines whether what is considered to be an automatic process has any impact on action selection. If the model-based/model-free approach goes beyond the will/ habit approach at all, which is not always clear, it does not make a convincing case that satiation procedures are a diagnostic method to tell truly goal-driven from purely stimulus-driven actions. The main problem is that this approach systematically underestimates the complexity of human action planning, a possible reflection of its behaviorist heritage. One complaint about binary theorizing has been that, even though actioncontrol processes can be easily divided into two categories, the various categories that researchers have created so far do not sufficiently overlap to make a convincing coherent story (Melnikoff and Bargh, 2018). Even though I agree, I would even argue that the criteria offered so far have been ill-conceived and failed to allow sorting processes into non-overlapping categories. The reasons for that, I believe, have to do with some fundamental misconceptions regarding (1) the temporal relationship between the operation of processes assumed to reflect the goal and the operation of processes that are assumed to be automatic; (2) the aspects of actions that control operations keep themselves busy with; and (3) the number of goals involved in action control. In the following, I will discuss each misconception in turn.

### The Competition Misconception

When he was laying the ground for modern reaction-timebased analyses of human cognitive processes, Donders (1868/1969) was optimistic to have measured the time demands of what he called the "expression of the will."  More important than the validity of this estimate is the time point at which Donders thought that the will would express itself: between processing the stimulus information and executing the response. Once we replace the outdated terms "will" and "expression of the will" through their modern successors "goal" and "controlled process," we can see that the main function of controlled processes are thus assumed to consist in stimulus-response translation. This scenario perfectly fits with most modern action-control approaches, including the model of Kornblum et al. (1990), where the stimulusguided "identification of the correct response" is actually the only control(led) process. It is this process that is assumed to compete with the habitual, automatic, or model-free process for controlling the eventual action.

Even though Donders' view turned out to provide the basic theoretical template for modern action-control approaches, alternatives were available. In particular, Exner (1879) rejected the idea that the will intervenes between stimulus and response processing. Instead, he argued that preparing for a task or a particular action is accomplished by turning oneself into an automatic system long before the first stimulus appears. It is this automatic state that according to Exner enables humans to act efficiently. Note that the temporal relationship between actual control and automaticity has changed from concurrent competition to a sequence in which control operations set up the stage for automatic processes to take over. Exner's view provides an excellent theoretical framework for understanding the observations of Valle-Inclán and Redondo (1998) discussed above: automaticity can indeed be demonstrated but it depends on the implementation of the action goal, just as the conditionalautomaticity approach has claimed (Bargh, 1989). Hence, rather than competing with habitual, automatic, and model-free processes, goal-related control processes turn the cognitive system into a "prepared reflex," as Woodworth (1938) has called it (see Hommel, 2000b).

### The Process-Control Misconception

One of the oldest theoretical problems that experimental psychology deals with relates to what Turvey (1977) has called "executive ignorance": how is it possible that humans can carry out intentional actions but, if being asked how they did so, have very little of interest to report? The answer favored by ideomotor theorists since Lotze (1852) and James (1890) consists in the assumption of a mechanism that integrates co-activated representations of the sensory consequences of a movement (reafferent information) and the motor patterns generating these consequences. According to this view, infants and other novices start by motor babbling—performing relatively random movements—and integrate the produced motor patterns with the sensory consequences thereof (i.e., action effects). Once they have experienced action effects they like or find functional in achieving a particular goal, they "imagine," "expect," or "predict" these consequences, which functionally translates into reactivating the sensory representations of action effects. Given that these representations have been integrated with the motor patterns that have generated them in the past, reactivating them will prime and eventually activate the associated motor patterns, which is likely to reproduce the (now intended) sensory consequences.

Recent research has provided strong evidence for the existence of such an ideomotor mechanism, unraveled its neural and functional underpinnings, and its role in the development of intentional action (for reviews, see Hommel, 2009; Shin et al., 2010). However, for present purposes, the only important implication of this research relates to the target of control. If it is true that all that an intentionally acting agent has available are representations of past (and now expected) sensory consequences of movements, it is clear that action planning mainly consists in the activation and maintenance of these representations. In other words, action control deals with and operates on representations of expected sensory outcomes. While this might sound obvious, it is important to emphasize that this does not imply that action control is targeting particular processes. It is in fact the inability to intentionally target particular processes—the executive ignorance—that has provided the main impetus for ideomotor approaches to emerge. It thus makes little sense to compare processes that are thought to be controlled with processes that are thought to be not controlled or, as in most approaches, controlled by external stimuli. Instead, it makes more sense to assume that implementing a particular goal establishes a condition that allows representations of action-outcome relations to compete, and the representation with the closest fit to the intended action effect to win, at least under ideal circumstances (see Hommel and Wiers, 2017, for elaboration). If so, it would only be the implementation of the goal that could meaningfully be referred to as intentional or controlled, while the resulting competition would be fully automatic—just as Exner envisioned.

From this perspective, stimuli might be able to activate particular goals but, once a particular goal is implemented, they would not be able to make an agent perform an action that is entirely unrelated to that goal. And this is indeed what all available purported demonstrations of automaticity show: if a participant commits an error in a manual Stroop task, she is very unlikely to actually speak the word out loud—even though this should theoretically be the strongest habit and the most automatic tendency—but rather press the key that corresponds to the color designated by this word. Note that this error is anything but model-free, as it reflects many aspects of the task instruction, actually results from obviously outcompeting the strongest habit, and takes into account the goal of intending to press keys, rather than to say something or do something else. In other words, the error reflects the consideration of almost all aspects of the goal and the task model—something that arguably undermines all available binary accounts.

### The Single-Goal Misconception

Distinguishing between goal-related and automatic processes requires a good understanding of what the current goal actually is. Researchers implicitly or explicitly identify the current goal with reference to the instruction: aspects of the task that were considered relevant in the instruction are assumed to be represented by the goal whereas aspects of the task that were considered irrelevant are not. If thus evidence for processing the latter can be obtained, this is taken as evidence for control leakage and, thus, automatic processing. Importantly, the logic of this rationale presupposes that people have only one goal at a time, which unfortunately is entirely unrealistic. According to Atkinson and Birch (1970), the stream of human behavior is driven by multiple internal response tendencies that continuously vary in strength. Vallacher and Wegner (1987) have suggested that actions can be described at various levels, due to the concrete action plans being commonly nested into more abstract action plans, which are part of even more abstract plans, etc. Indeed, if a student is participating in a Stroop task, she is unlikely to give up her plans to earn some money, to complete her studies in time, to become a famous scientist, to be a sympathetic person, and to lead a happy life when entering the lab. How are all these goals, smallscale and large-scale, long-term and short-term, reflected in current theorizing on action control? I am afraid they are not.

That this has severe consequences for our understanding of action control can be easily shown. As discussed earlier, tasks that are thought to tap into action control give participants mixed messages about the relevance of processing particular information. In the Stroop task, words are explicitly declared to be irrelevant and yet in a substantial portion of the trials, often up to 50%, processing the word or even reading it pays off, and the argument holds for Simon tasks, flanker tasks, and many other versions of them as well. Mixed messages of this kind are likely to undermine the instructed ignorance to the type of information that the instruction has declared irrelevant. Why would a system that is assumed to be attuned to optimizing reward, as the human cognitive system, not be sensitive to the possibility to receive reward in 50% of the trials? Moreover, researchers commonly try to counteract reward-sensitive strategies by varying the irrelevant information in an unpredictable fashion. This however implies considerable variability with respect to the irrelevant stimulus dimension. Variability implies uncertainty, and the human cognitive system is notoriously interested in reducing uncertainty. This has been emphasized in recent predictive-coding approaches (Friston, 2009) but also featured strongly in the approach of Berlyne (1949, 1960). Berlyne has claimed that one of the major human drives consists in curiosity—a chronic goal that is unlikely to be traded for a Stroop instruction. Curiosity is assumed to be attracted to stimulus aspects of maximal uncertainty, which the cognitive system then tries to reduce by improving its expectations (Sokolov, 1963) or, in more fashionable terms, its predictions (Friston, 2009). If we thus assume that participants bring their curiosity goal to our labs, it should not be overly surprising that they are particularly interested in information satisfying it. If they are, this would not indicate a lack of goal-related action control but rather imply that participants satisfy various goals concurrently. Among other things, this predicts that effects hitherto assumed to reflect a leakage of control decrease as irrelevant information becomes less uncertain—which is exactly what Frings et al. (2019) have observed.

### A UNITARY ALTERNATIVE

As I have tried to argue, binary theorizing that divides actions into willed and un-willed categories does not provide us with a useful perspective to understand action control, neither in the disguise of the will/habit opposition, nor in the case of the intentional/automatic opposition, nor with the model-based/ model-free opposition.3 There can be little doubt that practice changes the representation of stimulus and action events, that it creates associations between the codes forming these representations, and that these associations have impact on action control. However, there is no systematic evidence suggesting that the amount of practice can predict which actions people choose, or that people choose actions that are unrelated to their current goals. Rather, it seems that goals set the stage for the competition of various, presumably automatic processes. Given that people control goals, rather than processes, it is always possible that one of the processes being involved in the competition turns out to be less functional than others, but this is a normal outcome of processing in a system that is as competitive as the human brain. As argued and developed in some detail elsewhere (Hommel and Wiers, 2017), the time seems ripe to move on toward a more integrative framework of human action control: a framework that embraces the complexity of action control and that goes beyond mere binary categorization, both in terms of functional explanation and with respect to the neural mechanisms. In the following, I will briefly sketch the core concepts of Hommel and Wiers' Unitary Model of Action Control (UMAC; the interested reader is referred to Hommel and Wiers, 2017, for more detail) and relate them to existing dual-route models.

According to UMAC, selecting an action is biased by multiple goals. Goals are functionally represented by one or many selection criteria that serve to provide top-down support for representations of actions that are expected to meet these criteria. For instance, the decision to grasp a cup of coffee on a table by means of one's right hand might be driven by selection criteria that promote actions that involve grasping, actions that serve reaching a cup, actions that are likely to have positive consequences, actions that are easy to perform, and actions that go fast. The selection criteria might be taken to represent multiple goals, like quenching one's thirst for coffee, moving with little effort, having fun, and pushing one's energy, but UMAC does not require the specification or even the integration of dedicated goals—all that counts are activated selection criteria. Given that many of the criteria will be satisfied by more than one action representation, the (entirely automatic!) competition between suitable representations might be fierce but eventually be gravitating toward the representation of the action that best meets most or all of the criteria. Note that this scenario implies both: that all actions reflect goal states and that all actions are selected automatically. In other words, all actions are both intentional and automatic.

Highly overlearned actions or actions that the agent has preferred to choose under coffee-drinking circumstances may well have a selection benefit in the competition, because they had been learned to have low control demands (i.e., they meet the easy-to-perform criterion particularly well) and to go fast (i.e., they meet the high-speed criterion particularly well). However, it is important to emphasize that the degree of overlearning as such does not render the corresponding action special (or "more automatic") in any way. There would be nothing wrong with calling the corresponding action a habit, simply because the agent tends to prefer this action over others—which is the defining criterion for calling something a "habit" in everyday communication. But the habitual character only exists in the eyes of the observer—the agent simply selects an action that is fast and easy. In other words, the key difference between binary theories and UMAC is that the former assume that particular actions tend to be chosen *because they are habits* that happen to be fast and efficient, whereas the latter (e.g., Moors and de Houwer, 2006) assumes that they are chosen exactly *because they are fast and efficient*. Whereas the former reasoning implies that the selection of a habit is non-intentional, at least under some circumstances, the latter implies that the selection takes place because of the current goals—which of course may involve selection criteria other than my current examples speed and efficiency.

From a UMAC perspective, it makes little sense to develop any binary system to sort actions into two categories. While practicing an action may well increase the likelihood of selecting it in the future, there is no theoretical reason to reserve a dedicated label to overlearned actions. For instance, even if overlearning to open a door by pressing the handle down, to use Lewin's example, will make down-pressing a particularly fast and efficient action that is likely to be a strong competitor for selection under high-speed pressure (a selection criterion that propagates fast and efficient actions), a strong accuracy instruction is likely to render this candidate entirely impotent. Note that this theoretical problem cannot be solved by turning the binary distinction between intentional action and habit into a continuous dimension; it rather highlights the actual status of the word "habit," which should be considered a descriptive term taken from everyday language but not a scientific, and certainly not an explanatory concept (cf., Hommel, in press).

An obvious objection against UMAC might be that it is merely changing the semantics in a way that is impossible to test: every time some seemingly non-intentional behavior can be demonstrated, a new goal might be invented to account for it. This would indeed not do a good service to our

3 Note that what I criticize is the way theorists have sorted actions, actioncontrol operations, and related processes into two categories over the last 150 years or so. It is thus *a particular kind of binary theorizing* that I criticize, and my main argument is that the distinctions being drawn between the binary categories make little sense both theoretically and empirically. I would like to emphasize that I am more interested in the flaws in making these distinctions than in the binary nature of the underlying theorizing. Accordingly, theories that would keep that distinction but add further categories would not escape my criticism. Conversely, binary theories that make other distinctions than between willed and un-willed (and related versions) may well escape it, even though I find it difficult to imagine what kind of distinction that might be and even though I would suspect that it would still tempt researchers to categorize actions and related processes rather than understanding their mechanics (the tendency that I criticize in Hommel, in press).

understanding of action control, but fortunately UMAC is not at all immune to empirical test, as the following examples show. First, a key point of UMAC is that implementing an action goal/intention enables (increases the possible impact of) event representations with features from dimensions that either are or seem to be task-relevant. It is this task-relevance that renders the tendency to say "red" in a Stroop task a potent competitor in action selection. A strict automaticity approach could thus easily disconfirm the corresponding UMAC prediction by demonstrating that people say "red" when being faced with the word "RED" in the absence of any task or in a task that neither requires reading nor otherwise dealing with colors or color words. Second, even though it may be difficult to create conditions under which chronic goals like curiosity or novelty-seeking can be entirely switched off, it is certainly possible to create conditions that make that goal more or less relevant, like in a dual-task paradigm with one task emphasizing or not emphasizing novel information. Demonstrating that such a manipulation has no impact on the processing of novel information whatsoever would be difficult to take for UMAC.

Another interesting issue in the comparison of UMAC and strict automaticity approaches relates to the role of external stimuli. Both approaches assume that action alternatives can be activated by processing external stimuli: the automaticity approach assumes that processes and even actions can be triggered by stimuli—where the latter, as I have argued above, is yet to be demonstrated in humans—and UMAC assumes that stimuli activate all representations that feature-overlap with the stimulus on task-relevant dimensions (Hommel, 2004; Hommel and Wiers, 2017). The critical difference between these two theoretical approaches does thus not relate to the possibility of stimulusinduced activation of internal representations but rather to the question whether the degree of this activation is moderated by task-relevance (which UMAC assumes but the automaticity approach does not) and whether activation can result in action, as the automaticity approach assumes, or in competition for action control according to goal criteria, as UMAC suggests.

Yet another difference refers to the role of the context. Many automaticity accounts imply a rather pure, de-contextualized connection between particular stimuli and overlearned responses to these stimuli (e.g., Dickinson, 1985; De Wit and Dickinson, 2009). In contrast, UMAC assumes that the basic representational unit is an event file (Hommel, 2004), which integrates stimuli, actions, and outcomes, as well as internal and external context conditions. This feature allows UMAC to deal with findings as those reported by Neal et al. (2011). These authors found that participants who are used to eating popcorn in the cinema are likely to eat popcorn even if it is stale and even though they report disliking it, but only if it is offered in the cinema but not in a lab room while watching music videos. Even though more research is required to identify further conditions of such observations, UMAC's assumption that action representations are contextualized and, thus, more likely retrieved and more strongly activated in a context in which they were acquired, is well-equipped to tackle such empirical challenges in principle.

Last but not least, it is important to point out that UMAC does not deny the important role of practice—the key player of automaticity accounts. According to UMAC, practice can change behavior in various ways that have an impact on action control, that is, on the probability that the event file related to a practiced action is eventually selected for execution. For instance, practice is known to increase the speed and efficiency at which an action is carried out. Increasing practice will thus increase the number of event files that satisfy goals that emphasize or imply speed and efficiency, which will make these event files more likely to outcompete others if and to the degree that these goals are activated. Practice will also lead to a more systematic, sharpened integration of other action effects, so that the experienced popcorn-eater, say, will have learned and will thus anticipate a richer and more specific set of sensory outcomes of popcorn eating than the popcorn greenhorn. This in turn will make the resulting event files more potent competitors under conditions in which goals that are satisfied by such outcomes are activated. For instance, it may take some time to register and appreciate social-improvement signals from other popcorn-eaters in the cinema, so that popcorn eating is more likely to satisfy social goals in the more experienced popcorn-eater. Practice may also increase or reduce the role of context, depending on the kind of experience: if 90% of the event files resulting from one's street-crossing experience contain a representation of a green light, encountering a green light is likely to play a stronger role in selecting the appropriate action than if light representations in street-crossing event files would be much more varied. UMAC and automaticity accounts do thus not differ with respect to the assumption that practice and learning can have a strong impact on action control, but they rather differ with respect to why and how this impact is thought to be achieved. If, thus, the popcorn-lover keeps eating popcorn even after having finished an XXL tube, this might reflect the ongoing satisfaction of (e.g., tactile, olfactory, or social) goals that are not yet sated, or simply an attempt to fight boredom, rather than a breakdown of intentionality.

### CONCLUSION

The unitary account to action control shows that there is no need to heed the conventional distinction between will and habit. In this framework, goals still play an important role, as do automatic processes and practice, but goals and automatic processes do not compete but serve complementary purposes. The next challenge will be to better understand how goals and selection criteria constrain the operation of automatic processes, and when and under which circumstances action representations become relevant competitors in the action-selection process. In any case, I believe that theorizing about action control is ready to take the next step, and that this next step should not consist in inventing yet another binary opposition.

### AUTHOR CONTRIBUTIONS

The author confirms being the sole contributor of this work and has approved it for publication.

### FUNDING

This research was funded by an Advanced Grant of the European Research Council (ERC-2015-AdG-694722) to the author.

### REFERENCES

Atkinson, J. W., and Birch, D. (1970). *The dynamics of action*. New York: Wiley.

Berlyne, D. E. (1960). *Conflict, arousal, and curiosity*. New York: McGraw-Hill.

Broadbent, D. (1958). *Perception and communication*. London: Pergamon Press.

### ACKNOWLEDGMENTS

The author is grateful to the reviewers and editors for their efforts to help improving this article.

Fitts, P. M. (1951). "Engineering psychology and equipment design" in *Handbook of experimental psychology*. ed. S. S. Stevens (New York: Wiley), 1287–1340.

Sokolov, E. N. (1963). *Perception and the conditioned reflex*. New York: MacMillan. Stroop, J. R. (1935). Studies of interference in serial verbal reactions. *J. Exp.*

**Conflict of Interest:** The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Copyright © 2019 Hommel. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# Model-Based and Model-Free Social Cognition: Investigating the Role of Habit in Social Attitude Formation and Choice

Leor M. Hackel1 \*, Jeffrey J. Berg2 , Björn R. Lindström3 and David M. Amodio2,3 \*

1 Department of Psychology, University of Southern California, Los Angeles, CA, United States, 2 Department of Psychology, New York University, New York, NY, United States, 3 Department of Psychology, University of Amsterdam, Amsterdam, Netherlands

Do habits play a role in our social impressions? To investigate the contribution of habits to the formation of social attitudes, we examined the roles of model-free and modelbased reinforcement learning in social interactions – computations linked in past work to habit and planning, respectively. Participants in this study learned about novel individuals in a sequential reinforcement learning paradigm, choosing financial advisors who led them to high- or low-paying stocks. Results indicated that participants relied on both model-based and model-free learning, such that each type of learning was expressed in both advisor choices and post-task self-reported liking of advisors. Specifically, participants preferred advisors who could provide large future rewards as well as advisors who had provided them with large rewards in the past. Although participants relied more heavily on model-based learning overall, they varied in their use of modelbased and model-free learning strategies, and this individual difference influenced the way in which learning related to self-reported attitudes: among participants who relied more on model-free learning, model-free social learning related more to post-task attitudes. We discuss implications for attitudes, trait impressions, and social behavior, as well as the role of habits in a memory systems model of social cognition.

#### Edited by:

John A. Bargh, Yale University, United States

#### Reviewed by:

Kent Berridge, University of Michigan, United States Ion Juvina, Wright State University, United States

\*Correspondence:

Leor M. Hackel<EMAIL_ADDRESS>David M. Amodio<EMAIL_ADDRESS>
#### Specialty section:

This article was submitted to Cognitive Science, a section of the journal Frontiers in Psychology

Received: 20 May 2019 Accepted: 31 October 2019 Published: 21 November 2019

#### Citation:

Hackel LM, Berg JJ, Lindström BR and Amodio DM (2019) Model-Based and Model-Free Social Cognition: Investigating the Role of Habit in Social Attitude Formation and Choice. Front. Psychol. 10:2592. doi: 10.3389/fpsyg.2019.02592 Keywords: social, cognition, attitude, learning, habit, model-free, model-based, computational

## MODEL-BASED AND MODEL-FREE SOCIAL COGNITION

Human thriving depends on social relationships, and the impressions we form of new acquaintances are essential guides to our social behavior (Fitzsimons and Anderson, 2013). We befriend people who are kind, hire people who are competent, avoid those who are domineering, or seek counsel from those who are empathic. In this way, impression formation often serves our goals (Brewer, 1988; Fiske and Neuberg, 1990; Bargh and Ferguson, 2000), as we use our knowledge of other people – their traits, mental states, and behaviors – to predict their actions and decide whether to interact with them (Heider, 1958; Tamir and Thornton, 2018).

Yet, while goals drive much of human behavior, this is not always the case. Habits, in particular, are responses that occur automatically and independent of our goals, often representing a highly-repeated behavior that was once goal-directed but that persists and is expressed even when the goal has changed (Wood and Rünger, 2016; Robbins and Costa, 2017). Habits likely

explain many behaviors, from benign compulsions like biting one's nails to more harmful acts like mindlessly reaching for a cigarette. Here, we asked whether habit-like processes may also contribute to social cognition – how we learn about, interact with, and evaluate other people – and thus help explain social behaviors that appear to occur independently of, or in opposition to, one's goals.

### Multiple Systems for Social Learning

Research on impression formation has, to date, primarily emphasized conceptual forms of learning that give rise to goaldirected behavior; that is, acquiring conceptual knowledge about a person's traits and behavior (Uleman and Kressel, 2013). Early theories of impression formation focused on instructed forms of learning, in which we learn about a person from descriptions shared by others (Asch, 1946; Wyer and Carlston, 1979). If we are told that Bob is generous and friendly, we may infer that he's a good person. We can also learn about other people through observation and the use of attributional processing (Heider, 1958; Jones and Davis, 1965; Rydell and McConnell, 2006). If we see Jane offer money to a homeless person, we may infer from her actions that she is generous; if we see Jane choose a highperforming stock, we may infer that she is competent. These conceptual inferences can give rise to goal-directed behaviors, like choosing to spend time with someone who is generous or to hire someone who is competent.

More recent research has shown that social attitudes and impressions can also be formed through reward-based instrumental learning in direct social interaction – trial-anderror learning in which people make choices and receive feedback (Hackel et al., 2015). For instance, one might choose a lunch partner and experience rewards when they share their food, or one might hire a financial advisor and experience rewards when their advice pays off. Through this feedback, one can learn the reward value of an individual while also inferring aspects of their character traits (Hackel et al., 2015). Unlike instructed and observational forms of learning, which are typically passive (e.g., reading about another person), instrumental learning is active: it concerns feedback from another person regarding one's own actions. If, on most days, Bob's greeting to Jane is met with a smile, he will associate reward with his behavior toward Jane in addition to inferring that she is friendly.

Instrumental learning thus represents a distinct mode of learning in social interactions relative to conceptual knowledge (Amodio, 2019). Instead of inferring other people's qualities in order to decide how to interact with them, instrumental learning involves learning the reward value of social interaction through direct action and feedback. That is, in traditional impression formation approaches, Bob learns to interact with Jane because he infers she is friendly, and he wants to be around friendly people. In instrumental learning, Bob learns to interact with Jane because he previously did so and received rewarding outcomes, such as social rewards like smiles and compliments or material rewards like money and food. He may like Jane as a result of those rewards, rather than as a result of qualities he attributes to her. Thus, instrumental learning directly informs how we should interact with others given the rewards they provide. In this way, preferences acquired through instrumental learning may be more directly tied to behavior.

### A Role for Habits in Social Cognition?

Over time, instrumentally learned responses may be automatized into habits (Thorndike, 1911; Robbins and Costa, 2017). Although people may initially perform an action deliberately to achieve a goal, rewards can "stamp in" an association between a stimulus (or context) and a response, such that people later perform the response automatically. In contrast to skills, which are goal-directed action routines triggered intentionally, habits reflect a well-learned response that unfolds even when it is not consistent with a goal, and it persists even when its expression is no longer rewarded (Balleine and Dickinson, 1998; Tricomi et al., 2009; Wood and Rünger, 2016; Wood, 2017). Nevertheless, habits can be adaptive, initiating an important behavior that we might otherwise forget in the pursuit of another goal, such as grabbing our keys when rushing out the door to get to work in the morning.

Habits differ from other forms of unintentional learning that may contribute to impression formation. For example, spontaneous trait impressions (STIs) form when a perceiver is simply asked to read and memorize a set of trait-implying sentences (Winter and Uleman, 1984; Carlston and Skowronski, 1994). People may be unaware that they formed an impression, yet STIs become evident in measures of cued recall and may subsequently influence judgment (Moskowitz and Roman, 1992). There is also evidence that evaluative conditioning, in which a neutral social target is paired repeatedly with either positive or negative images (Walther, 2002; Olson and Fazio, 2006), may even occur when such images are presented subliminally (e.g., De Houwer et al., 1997; Hofmann et al., 2010; but see Sweldens et al., 2014). However, both forms of learning involve passive exposure to stimuli and the formation of conceptual associations, likely supported by a semantic/conceptual associative memory system (Amodio and Berg, 2018; Amodio, 2019), in contrast to the active process of action-outcome learning involved in instrumental habit formation.

### Examining Habit Formation Through Reinforcement Learning

A major challenge in the study of habits in humans is that it is often difficult to discern habits from other, goal-directed processes in behavior. However, this distinction has recently been linked to two forms of behavior within a computational account of reinforcement learning (Daw et al., 2011). Broadly, reinforcement learning algorithms describe how an agent learns the value of different actions with different states of the world by making choices and experiencing rewards (Sutton and Barto, 1998). According to this account, two types of computations can underlie reinforcement learning: Agents can engage in modelbased learning, in which they consider the likely outcomes of their actions given knowledge about their environment, and also in model-free learning, in which they associate actions directly with reward value and repeat previously rewarded actions (Daw et al., 2011; Doll et al., 2015). Model-based learning is thus

prospective and goal-oriented, sensitive to both environmental contingencies (e.g., how to get to a reward) and expected outcomes (e.g., whether a desirable reward will be attained) – like a hungry mouse considering how to navigate a maze to reach the room with the tastiest cheese. In contrast, modelfree learning is retrospective, relying on a past history of rewards for an action; it requires no internal model of one's environment and is insensitive to the outcomes an action will presently bring. A model-free learner stores cached values for previously performed actions and selects actions with the highest cached value.

Because model-free learning is computationally simpler but less flexible than model-based learning, it may give rise to behavior that has features of habits. For instance, an animal might continue to press a food lever despite being fully sated because this action was previously rewarded and thus associated with high reward value (Dickinson and Balleine, 1994; Daw et al., 2011). Although a model-free learner could eventually learn to adapt to the new value, it would persist in pressing the lever until learning takes place in its newly satiated state. In contrast, a model-based learner should not require this learning at all; instead, it should plan ahead to the likely outcome of the lever press, realize that it does not desire that outcome, and avoid the action from the start. Given these characteristics, the model-based/model-free distinction has been used recently to probe the role of habits in a range of learning contexts in humans. For instance, individuals who engage in greater model-based learning show less persistence in a devaluation task – a classic marker of habits (Gillan et al., 2015). Yet, to date, this approach has not been applied to questions on the formation of social impressions through direct social interactions with other people.

### Model-Free Learning in Social Cognition

How might a model-based/model-free account relate to social impressions? When other people provide us with material feedback (like a gift) or social feedback (like a smile or a compliment), we experience this feedback as rewarding; as a result, this feedback can reinforce our social choices and draw us back to the same partners again in the future (Jones et al., 2011; Lin et al., 2011; Lindström et al., 2014; Hackel et al., 2015; Lindström and Tobler, 2018). If people learn from this feedback in a model-free manner, specifically, they might return to interaction partners previously associated with high reward regardless of whether those partners will currently provide desirable outcomes. This pattern would resemble a traditional definition of habit.

Some existing work hints at the possibility that reward feedback gives rise to social preferences that persist in a habitlike manner. In research by Hackel et al. (2015), participants played an economic game in which they chose partners who could share money; partners varied in the average amount they shared (indicating reward value) and average proportion they shared (indicating generosity). During initial learning, it was economically advantageous for participants to prefer individuals who provided large rewards, regardless of their generosity. However, when participants were later asked to choose one of these partners to work with in a non-economic puzzle-solving task – a context where generosity, but not previous reward value, is advantageous – participants' choices were still influenced by partners' past reward value in addition to their generosity. This persistent influence of past reward – even when reward value no longer informed desired outcomes – suggests that participants may have developed model-free reward associations that guided subsequent social preferences. Nevertheless, past work has not directly tested this possibility by dissociating model-based and model-free learning in social interaction.

### Study Overview

The present research was designed to provide initial evidence for model-free learning in social impression formation. To this end, we administered a sequential choice task commonly used to dissociate model-based and model-free learning (Kool et al., 2016; Kool et al., 2017; see also Daw et al., 2011), adapted to examine social partner choice and attitudes. On each round, participants chose financial advisors who had supposedly invested in one of two stocks; participants then received a payout from that advisor's stock. We examined the extent to which participants chose advisors based on model-based and model-free reinforcement, and further examined whether these forms of learning predicted participants' subjective attitude toward each advisor.

### MATERIALS AND METHODS

### Participants

Sixty-nine participants (42 male, 27 female) were recruited via Amazon Mechanical Turk (AMT), in exchange for \$3.50 for study completion, plus a monetary bonus based on their task performance. A sample size of 65 participants was chosen a priori; an additional four participants completed the task due to an error in which an extra set of slots was posted. Data collection was completed before analysis. Participants were eligible if they were located in the United States, completed at least one prior AMT study, and had approval rates of at least 95%. Informed consent was obtained from all participants in accordance with the guidelines of the New York University Committee on Activities Involving Human Subjects. We excluded data from participants who did not respond in time to either the first or second stage of a trial on more than 20% of trials (Kool et al., 2017). This rule excluded data from four participants, leaving data from 65 participants in analyses.

### Procedure

Participation took place via Psiturk, an online platform for cognitive tasks (Gureckis et al., 2016). After providing consent, participants read a self-guided description of the study, which included practice trials, and completed the main experimental task. Next, participants completed self-reported evaluation items and a demographics questionnaire. Lastly, participants were informed of their bonus compensation for participating and then completed a debriefing procedure that included a suspicion probe and an explanation of study goals. All data exclusions, all manipulations, and all measures included in this research are fully reported in this article.

#### Two-Step Task

fpsyg-10-02592 November 19, 2019 Time: 15:41 # 4

We adapted a sequential learning task (Kool et al., 2016, 2017) designed to dissociate model-free and model-based learning (**Figure 1**). In our adaptation, participants were told they would learn about choices made by four AMT workers who previously participated in a financial decision-making study (see **Supplementary Material** for full task instructions). According to this cover story, these previous workers were assigned the role of "Financial Advisor," in which they chose (only) one of two stocks ("Axiom" and "Zephyr") to invest in for the duration of the study. These Advisors then earned money based on the performance of their chosen stock, which fluctuated throughout the study and could change from one round of "dividends" to the next.

Next, participants were assigned to the role of the "Client," in which they would make a series of decisions about which Advisor to hire. Participants learned they would earn points based on the performance of the stock chosen by their hired Advisor on each round. Participants were explicitly told that the performance of the stocks would change over time ("a stock that was bad at the beginning of the game might start performing well, and a stock that initially pays well might perform poorly later on"), and that they should try to hire Advisors with the better performing stock at that particular moment. Moreover, participants were informed that they would receive a monetary bonus for their performance in the task, with better performance (in terms of points earned) equating to a larger bonus.

Return on each trial, participants began in one of two randomly chosen first-stage states. In these states, participants were presented with one of two pairs of Advisors, represented by distinct cartoon avatars (**Figure 1**). Avatars were randomly

FIGURE 1 | Schematic of task design. In the first stage of each round, participants saw one of two sets of advisors and chose an advisor for that round. Participants then viewed the stock that advisor had chosen; after making a button press, participants saw feedback indicating the payout provided by the stock, ranging from zero to nine points. Within each pair of advisors, one advisor always led to the "Axiom" stock and the other always led to the "Zephyr" stock. This feature of the task rendered the two sets of advisors equivalent, such that a model-based learner could apply experiences with one set of advisors to choices involving the other set of advisors.

assigned to different roles across participants (i.e., which stock they were linked with) and were equally likely to appear on the left or right side of the screen. Participants chose one of the two Advisors via button response and then transitioned deterministically to one of the two stocks, which comprised the second-stage states. That is, participants could reach either of the two stocks from each of the first-stage states; one Advisor in each pair always invested in the Axiom stock and the other Advisor in the given pair always invested in the Zephyr stock.

When they reached the second-stage state, participants were instructed to press the spacebar to reveal the performance of the stock in which the chosen Advisor invested. If participants did not respond in time to either the first- or second-stage states, no reward was provided and participants moved to the next trial. The number of points obtained for each stock fluctuated slowly and stochastically over the course of the task, varying according to a Gaussian random walk (SD = 2) with reflecting bounds at 0 and +9 points. The drifting nature of the reward feedback encouraged continuous learning throughout the task.

Importantly, the two first-stage states were equivalent in terms of the stocks they could lead to: within each pair of advisors, one Advisor always invested in the Axiom stock, whereas the other Advisor always invested in the Zephyr stock. This design allows for the separation of model-free and model-based control. Given that both stocks can be reached from each pair of Advisors, the stock reached from one set of advisors can be used by a model-based learner to update preferences regardless of which set of advisors is encountered on the next trial. For instance, if an Advisor in one pair invested in the Axiom stock and this stock paid out a large number of points on that trial, a model-based learner should subsequently be more likely to choose the Advisor in the other pair that also invests in the Axiom stock. That is, a model-based learner can generalize across equivalent first-stage choice options due to its exploitation of the overarching task structure. Conversely, model-free learners would not generalize across equivalent first-stage choice options, as they simply rely on directly-experienced action-outcome associations – the outcomes experienced following a choice in one pair of advisors should not affect preferences for the advisors in the second pair, and vice-versa.

Participants were trained extensively on the deterministic transitions (i.e., which financial advisor in a given pairing invested in which of the two stocks) prior to completing the experimental trials, such that 80% accuracy across 15 consecutive trials was required to advance to the main task. Participants did not receive explicit instructions on which advisor led to which stock, but rather were required to learn these transitions through experience. After this training phase, participants completed 150 trials of the main task, split evenly between the two first-stage states. The response deadline in both stages was 1500 ms and feedback was presented for 1000 ms.

#### Post-task Evaluations

Following the two-step task, participants responded to a series of self-report items which pertained to participants' evaluations (or "liking") of the different Advisors encountered during the two-step task. Participants were presented with the avatar of each financial advisor, one at a time, and rated how much they liked the advisor using a seven-point scale (from 1 = "Do not like them at all" to 7 = "Like them a lot"). Finally, participants were also asked to estimate how valuable, on average, each of the two stocks were over the course of the learning task (see **Supplementary Material**).

### Computational Model

fpsyg-10-02592 November 19, 2019 Time: 15:41 # 5

In order to determine the degree to which participants employed model-based and model-free learning, we fit data from the learning phase to a computational model of reinforcement learning used in previous work (Kool et al., 2017). Doing so allowed us to estimate latent variables related to social learning for each subject (Hackel and Amodio, 2018), which we then used as input in our analyses.

The model contains a hybrid of model-free learning and model-based learning for selecting advisors (see **Supplementary Material** for additional details and **Supplementary Table S1** for parameter fits). The model-free system stores values for advisors at the first stage and for stocks at the second stage based on prior reward feedback. The model-based system computes the value of selecting each advisor at the time of choice, combining knowledge about how advisors lead to stocks with the expected payoff of each stock (acquired through model-free learning at the second stage). A model-based learner thus prospectively plans toward a goal: he or she selects an advisor based on the stock the advisor will lead to, in light of the reward expected from each stock. In contrast, a model-free learner selects advisors based on the rewards those advisors have led to in the past.

Critically, the model includes a weighting parameter (w) that indicates the relative influence of model-based and modelfree learning in choice, ranging between 0 (purely modelfree) and 1 (purely model-based). This parameter can serve as an individual difference measure of the extent to which a participant engaged in model-based or model-free learning. We fit this model for each participant using maximum a posteriori (MAP) estimation, with empirical priors used in previous work (Gershman, 2016; Kool et al., 2017). Doing so allowed us to estimate each participant's w parameter (mean = 0.83), indicating the extent to which they relied on model-based vs. model-free learning. We used this parameter in subsequent analyses examining individual differences in the use of these learning strategies.

### RESULTS

### Model-Free and Model-Based Social Learning

To what extent did participants engage in model-based and model-free social learning? To answer this question, we examined choices in the learning phase, drawing on the following logic of the task. As noted above, the two sets of advisors in the task are equivalent, such that one advisor from each set leads to a particular stock. As a result, a model-based learner would generalize experiences with one set of advisors to the other set. For instance, imagine a participant who sees the first pair of advisors, picks the advisor that leads to the "Axiom" stock, and receives a large reward. On the next round, a model-based learner would try to return to the "Axiom" stock regardless of whether they see the same pair of advisors or a different pair of advisors. In contrast, a model-free learner updates values for individual advisors and chooses advisors based on these values. A modelfree learner would therefore repeat their choice on the next trial if presented with the same advisors but would do so to a lesser extent if presented with different advisors. That is, the model-free learner would fail to generalize across sets of advisors.

Drawing on this task logic, we fit learning phase data to a lagged regression model predicting, on a trial-by-trial basis, whether or not participants repeated their most recent choice of Stage 2 stocks (1 = stay, 0 = switch). This analysis provides a model-agnostic way to test the qualitative behavioral predictions of the model-free/model-based account of learning. Following Kool et al. (2016), predictors included the reward earned on the previous trial (standardized, within-subject, to z-scores), whether or not the previous trial started with the same set of advisors (1 = same, -1 = different), and the interaction of these two predictors. A main effect of reward would indicate model-based learning: people return to a highpaying stock regardless of whether they see the same or different advisors on the next trial to get to that stock (simulated data shown in **Figure 2A**). An interaction of reward and start state would indicate model-free learning: people try to return to a high-paying stock, but particularly do so when presented with the same set of advisors, thus repeating the advisor choice that led to the large reward (**Figure 2B**). Models were fit using the lme4 package in R (Bates et al., 2015; R Core Team, 2016). Random variances were allowed for the intercept and all slopes (see **Supplementary Table S2** for all coefficients.).

This analysis revealed a main effect of reward, b = 1.47, SE = 0.07, z = 19.80, p < 0.001, consistent with model-based learning: overall, participants returned to second-stage stocks after receiving large rewards. However, the analysis also revealed a Reward × Start State interaction, b = 0.22, SE = 0.03, z = 6.45, p < 0.001, indicating the presence of model-free learning: participants were more likely to return to a high-paying stock when starting with the same advisors at the first stage. Although participants in our sample were highly model-based (mean w parameter in the computational model fits = 0.83), these results support the hypothesis that both model-based and model-free reinforcement learning contributed to social choice (**Figure 2C**).

### Post-task Evaluations

If reinforcement learning also gives rise to attitudes, participants might like advisors who can provide reward in the future (model-based value) and advisors associated with past reward (model-free value). To test how learning affects attitudes, we examined participants' self-reported liking of each advisor following the learning task. Using each subject's individual parameter fits in the computational model, we estimated the final model-based and model-free values associated with each advisor for each subject at the end of learning, given the unique series

of stimuli and outcomes viewed by each participant. We then regressed liking ratings simultaneously on each type of value.

Notably, model-based values were identical for advisors who led to the same stock. That is, if the Axiom stock would be expected to deliver 6 points on average at the end of the task, then each advisor who leads to the Axiom stock would have a model-based value of 6 points. If social evaluations reflect model-based learning, participants would therefore like the two advisors who led to the Axiom stock equally. In contrast, model-free values reflect the unique reward history associated with a particular advisor; even for two advisors who led to the Axiom stock, participants might have experienced different reward outcomes with each advisor. If social evaluations reflect model-free learning, people would therefore prefer advisors who provided greater rewards. Finally, this tendency should depend on individual differences in learning, as reflected in the w parameter: individuals who engage in greater model-free learning should especially like advisors associated with high model-free value.

To test these hypotheses, we fit a mixed-effects linear regression predicting post-task liking ratings (**Supplementary Table S3**). Predictors included each participant's final model-free values and model-based values toward each advisor (estimated from the computational model), each participant's w parameter, and the interaction of w with each type of value. Each predictor was standardized to z-scores (within-subject for the value regressors and between-subject for the w parameter). As a result, main effects of value regressors are interpretable relative to the mean level of the w parameter (w = 0.83). We included random variances for the intercept and each predictor. The models were fit using the lme4 package and lmerTest packages (Bates et al., 2015; Kuznetsova et al., 2016) in R (R Core Team, 2016).

This analysis yielded a main effect of model-based values, b = 0.30, SE = 0.14, t(71.46) = 2.17, p = 0.03, and a marginally significant main effect of model-free values, b = 0.16, SE = 0.09, t(162.97) = 1.82, p = 0.07. In other words, at mean levels of the w parameter, attitudes reflected both kinds of learning: people liked advisors who could lead them to more rewarding stocks and also liked advisors who were uniquely associated with greater reward in the past.

We further examined whether the effects of model-based and model-free learning on reported attitudes varied by participants' individual learning tendencies, as indexed by the w parameter. We found that the w parameter, which represents this individual difference variable, interacted with model-free values, b = −0.24, SE = 0.08, t(148.01) = −2.97, p = 0.004. Participants who exhibited relatively greater modelfree learning also expressed greater liking of partners who had provided more reward. Simple effects analysis supported this interpretation: for learners relying relatively more on model-free control (centered at the 25th percentile of the w parameter, or w = 0.70), model-free values were strongly predictive of attitudes toward advisors, b = 0.31, SE = 0.10, t(155.32) = 3.11, p = 0.002, revealing a novel effect of modelfree learning on social evaluation. By contrast, for those relying relative more on model-based control (centered at the 75th percentile of the w parameter, or w = 1), model-free values were not associated with evaluations, b = −0.03, SE = 0.11, t(162.01) = −0.31, p = 0.76. Thus, participants who exhibited model-free learning also liked advisors associated with greater model-free value1 .

Together, these results identify two ways in which reinforcement learning influences social attitudes, one that is goal-directed and one that is habit-like: people like others who are equivalently capable of providing large rewards in the future, and they also like others who have uniquely provided large rewards in the past. Moreover, the influence of past (model-free) reward history depends on individual differences in learning: individuals who weight model-free rewards more strongly during

1 In contrast, we did not observe an interaction between the w parameter and model-based values (see **Supplementary Material**). This finding is consistent with the fact that model-based learning was relatively high across participants, whereas not all participants showed a meaningful degree of model-free learning (i.e., w < 1).

learning also have a stronger preference for advisors associated with past rewards.

### DISCUSSION

fpsyg-10-02592 November 19, 2019 Time: 15:41 # 7

Does habit play role a social impressions? Our findings demonstrate that, indeed, people form impressions through reward-based reinforcement processes that include model-free learning – a form of learning thought to contribute to habitual behavior. In the sequential learning task used here, participants chose financial advisors based on both model-based and modelfree learning. That is, participants chose advisors who could lead them to desirable stocks in the future (model-based) as well as who were associated with high rewards in prior interactions (model-free). Although participants relied far more heavily on model-based (as opposed to model-free learning) in general, this pattern of model-free learning suggests the additional role of a habit-like component of learning and behavior in the context of social impression formation.

Furthermore, participants' learning processes had implications for their explicit social evaluations. Across participants, both model-based and model-free learning predicted self-reported attitudes toward advisors. Moreover, participants varied in their reliance on model-based vs. modelfree processing during the learning task, and this individual difference in learning related to differences in evaluation: participants who exhibited greater model-free learning during the investment task showed an effect of model-free learning on self-reported attitudes. Thus, these findings dissociate two routes through which reinforcement learning contributes to attitudes toward social partners, and they highlight the importance of considering individual differences in learning strategies during social interactions to understand the effects of rewards on social attitudes and decisions.

### Model-Based and Model-Free Social Cognition

Our central finding – of model-free learning in social impression formation – offers novel theoretical implications for social cognition, learning, and attitudes. First, our findings highlight a role for reward-based reinforcement learning in social interactions. Previous impression formation research demonstrates that people learn about the traits of others in order to predict how others will behave (Heider, 1958). For instance, by observing financial advisors, people can form impressions of an advisor's competence and predict that advisor's future performance (Boorman et al., 2013; Leong and Zaki, 2018). Our results introduce a complementary mode of social learning based on reward: people also learn whom to choose and whom to like through instrumental learning, such as directly choosing an advisor and experiencing rewards as a result.

The observation of model-free social learning, in particular, supports the proposed role of habit in social cognition. In modelfree learning, people repeat previously-rewarded choices in a relatively inflexible manner – the hallmark of a habit. Habits may therefore influence social behavior: because habits reflect routinized responses that operate most adaptively in invariable environments, they may fill in the gaps between goal-directed responses to facilitate social behavior. In some cases, habits may have harmful effects; for example, people may persist in interacting with social partners with whom they had positive past experiences, even when other partners might be equally or more relevant to one's current goals. In other cases, habits may be beneficial, leading an individual to approach a previouslyrewarding person while distracted by their pursuit of an unrelated goal – perhaps eliciting help, if needed, or simply avoiding a social faux pas. In both cases, their effects may be subtle, relative to goal-directed responses, yet still crucial to adaptive social function.