# DYNAMICS OF SENSORIMOTOR INTERACTIONS IN EMBODIED COGNITION

EDITED BY: Guillaume T. Vallet, Benoit Riou, Lionel Brunel and Nicolas Vermeulen PUBLISHED IN: Frontiers in Psychology

#### *Frontiers Copyright Statement*

*© Copyright 2007-2016 Frontiers Media SA. All rights reserved. All content included on this site, such as text, graphics, logos, button icons, images, video/audio clips, downloads, data compilations and software, is the property of or is licensed to Frontiers Media SA ("Frontiers") or its licensees and/or subcontractors. The copyright in the text of individual articles is the property of their respective authors, subject to a license granted to Frontiers.*

*The compilation of articles constituting this e-book, wherever published, as well as the compilation of all other content on this site, is the exclusive property of Frontiers. For the conditions for downloading and copying of e-books from Frontiers' website, please see the Terms for Website Use. If purchasing Frontiers e-books from other websites or sources, the conditions of the website concerned apply.*

*Images and graphics not forming part of user-contributed materials may not be downloaded or copied without permission.*

*Individual articles may be downloaded and reproduced in accordance with the principles of the CC-BY licence subject to any copyright or other notices. They may not be re-sold as an e-book.*

*As author or other contributor you grant a CC-BY licence to others to reproduce your articles, including any graphics and third-party materials supplied by you, in accordance with the Conditions for Website Use and subject to any copyright notices which you include in connection with your articles and materials.*

*All copyright, and all rights therein, are protected by national and international copyright laws. The above represents a summary only. For the full conditions see the Conditions for Authors and the Conditions for Website Use.*

ISSN 1664-8714 ISBN 978-2-88919-807-8 DOI 10.3389/978-2-88919-807-8

### About Frontiers

Frontiers is more than just an open-access publisher of scholarly articles: it is a pioneering approach to the world of academia, radically improving the way scholarly research is managed. The grand vision of Frontiers is a world where all people have an equal opportunity to seek, share and generate knowledge. Frontiers provides immediate and permanent online open access to all its publications, but this alone is not enough to realize our grand goals.

### Frontiers Journal Series

The Frontiers Journal Series is a multi-tier and interdisciplinary set of open-access, online journals, promising a paradigm shift from the current review, selection and dissemination processes in academic publishing. All Frontiers journals are driven by researchers for researchers; therefore, they constitute a service to the scholarly community. At the same time, the Frontiers Journal Series operates on a revolutionary invention, the tiered publishing system, initially addressing specific communities of scholars, and gradually climbing up to broader public understanding, thus serving the interests of the lay society, too.

### Dedication to Quality

Each Frontiers article is a landmark of the highest quality, thanks to genuinely collaborative interactions between authors and review editors, who include some of the world's best academicians. Research must be certified by peers before entering a stream of knowledge that may eventually reach the public - and shape society; therefore, Frontiers only applies the most rigorous and unbiased reviews.

Frontiers revolutionizes research publishing by freely delivering the most outstanding research, evaluated with no bias from both the academic and social point of view. By applying the most advanced information technologies, Frontiers is catapulting scholarly publishing into a new generation.

### What are Frontiers Research Topics?

Frontiers Research Topics are very popular trademarks of the Frontiers Journals Series: they are collections of at least ten articles, all centered on a particular subject. With their unique mix of varied contributions from Original Research to Review Articles, Frontiers Research Topics unify the most influential researchers, the latest key findings and historical advances in a hot research area! Find out more on how to host your own Frontiers Research Topic or contribute to one as an author by contacting the Frontiers Editorial Office<EMAIL_ADDRESS>
# **DYNAMICS OF SENSORIMOTOR INTERACTIONS IN EMBODIED COGNITION**

Topic Editors:

**Guillaume T. Vallet,** Centre de Recherche de l'Institut Universitaire de Gériatrie de Montréal, Montreal, Canada **Benoit Riou,** Université Lyon 2, France **Lionel Brunel,** Université Paul Valery, France **Nicolas Vermeulen,** Université Catholique de Louvain, Belgium

Image by Thomas Camus

We interact with our environment through perception and action. Perception is based on sensory components while actions are based on motor components. It is commonly accepted that these sensorimotor components constitute the foundation of knowledge (i.e., percepts and concepts), action and emotion. However, whether or not these components remain part of knowledge, action and emotion is still being debated (see Glenberg, Witt, & Metcalfe, 2013). According to the classical symbolic/abstracted approach of cognition, cognitive processes operate on symbols that are abstracted from these components. Reversely, embodied cognition theory states that knowledge, action and emotion remain grounded in these sensorimotor components (see Wilson, 2002). This embodiment revolution assumes that the interactions between present and absent —but simulated in memory— sensory-motor components determine the emergence of knowledge, action and emotion (Barsalou, 2008). It also implies that perception, memory (in particular conceptual knowledge), action and emotion interact together in a closer way that previously thought (e.g. Riou, Lesourd, Brunel & Versace, 2011; Corveleyn, Lopez-Moliner & Coello, 2012; Vermeulen et al., 2013).

Despite the accumulation of empirical evidence showing that perception, memory, action and emotion interact together, less is known about the dynamics of these interactions. It remains to precise the temporal dynamic (when these interactions occur), the neural underlying networks, and the factors that modulate these interactions. The present research topic focuses on the dynamic relationship between present and absent sensorimotor components across perception, memory, action and emotion in a grounded cognition perspective. This research topic aims 1) to demonstrate the validity of the embodied cognition theories 2) to highlight the dynamics of emergence of conceptual knowledge, action and emotion 3) to provide a comprehensive state-of-the-art theoretical explanation and/or models.

**Citation:** Vallet, G. T., Riou, B., Brunel, L., Vermeulen, N., eds. (2016). Dynamics of Sensorimotor Interactions in Embodied Cognition. Lausanne: Frontiers Media. doi: 10.3389/978-2-88919-807-8

# Table of Contents

*06 Editorial: Dynamics of Sensorimotor Interactions in Embodied Cognition* Guillaume T. Vallet, Lionel Brunel, Benoit Riou and Nicolas Vermeulen *09 Mechanisms of embodiment* Katinka Dijkstra and Lysanne Post *20 What are memory-perception interactions for? Implications for action* Loïc P. Heurley and Laurent P. Ferrier *24 Manipulation gesture effect in visual and auditory presentations: the link between tools in perceptual and motor tasks* Amandine E. Rey, Kévin Roche, Rémy Versace and Hanna Chainay *35 The embodied dynamics of perceptual causality: a slippery slope?* Michel-Ange Amorim, Isabelle A. Siegler, Robin Baurès and Armando M. Oliveira *46 Visiting Richard Serra's "Promenade" sculpture improves postural control and judgment of subjective visual vertical* Zoï Kapoula, Alexandre Lang, Thanh-Thuan Lê, Marie-Sarah Adenis, Qing Yang, Gabi Lipede and Marine Vernet *56 Evidence for the embodiment of space perception: concurrent hand but not arm action moderates reachability and egocentric distance perception.* Stéphane Grade, Mauro Pesenti and Martin G. Edwards *65 For your eyes only: effect of confederate's eye level on reach-to-grasp action* François Quesque and Yann Coello *72 Children's looking preference for biological motion may be related to an affinity for mathematical chaos* Joshua L. Haworth, Anastasia Kyvelidou, Wayne Fisher and Nicholas Stergiou *79 Starting off on the right foot: strong right-footers respond faster with the right foot to positive words and with the left foot to negative words* Irmgard de la Vega, Julia Graebe, Leonie Härtner, Carolin Dudschig and Barbara Kaup *86 When "good" is not always right: effect of the consequences of motor action on valence-space associations* Denis Brouillet, Audrey Milhau and Thibaut Brouillet *94 Affective valence facilitates spatial detection on vertical axis: shorter time strengthens effect* Jiushu Xie, Yanli Huang, Ruiming Wang and Wenjuan Liu *105 Spatial biases during mental arithmetic: evidence from eye movements on a blank screen*

Matthias Hartmann, Fred W. Mast and Martin H. Fischer

*113 Pushing forward in embodied cognition: may we mouse the mathematical mind?*

Martin H. Fischer and Matthias Hartmann

*117 Extending the reach of mousetracking in numerical cognition: a comment on Fischer and Hartmann (2014)*

Thomas J. Faulkenberry and Amandine E. Rey

# Editorial: Dynamics of Sensorimotor Interactions in Embodied Cognition

#### Guillaume T. Vallet 1 \*, Lionel Brunel 2 , Benoit Riou3 and Nicolas Vermeulen4

1 Department of Psychology, Centre de Recherche de l'Institut Universitaire de Gériatrie de Montréal, Montreal, Canada, 2 Laboratoire EPSYLON, Department of Psychology, Université Paul-Valéry Montpellier III, Montpellier, France, 3 Laboratoire d'Etude des Mécanismes Cognitifs, Department of Psychology, Université Lyon 2, Lyon, France, 4 Psychological Sciences Research Institute, Université Catholique de Louvain, Louvain-la-Neuve, Belgium

Keywords: embodied cognition, grounded cognition, situated cognition, sensorimotor interactions, memory, action, perception, emotion

### **The Editorial on the Research Topic**

### **Dynamics of Sensorimotor Interactions in Embodied Cognition**

The concept of incorporating the current situation and the body state within cognitive processes, referred to as embodiment, has revolutionized cognitive research (Glenberg et al., 2013). Interest in this approach has grown substantially in the last few decades. Embodied cognition has now been demonstrated across a wide range of topics, from babies (e.g., Smith and Gasser, 2005) to elderly adults (e.g., Vallet et al., 2013a), from normal cognition to neuropsychology (e.g., Vallet et al., 2013b) as well as in emotion (e.g., Vermeulen et al., 2007), and in neuroscience as a whole (e.g., Pulvermüller, 2013). Nevertheless, there is yet much to discover in order to better understand embodiment.

One of the striking arguments of embodiment is that sensori-motor features are at the core of mental processes (Pecher and Zwaan, 2005; Barsalou, 2010). Their relationship should also be considered to be dynamic. In the embodied cognition approaches, perception, memory, and action are no longer regarded as (relatively) independent functions, but rather as closely interacting components. Some authors even argue for an overlap within these processes (Brunel et al., 2009, 2015; Vermeulen et al., 2009) that would rely on the same neural code (e.g., Hommel). One key direction in embodiment that needs to be further explored is the dynamic interaction between the sensory and motor components across the different cognitive functions. The present Research Topic aims to shed new light on this issue.

The importance of studying this problematic is well presented in Dijkstra and Post's critical review. Their article highlights the crucial role of sensorimotor simulation across several cognitive activities (e.g., reasoning, evaluating) and how the interaction between the current situation and the simulation of past sensorimotor states mediates the emergence of adaptive behavior. This is possible since perception and memory interact closely and share processes and resources (e.g., Vermeulen et al., 2008; Riou et al., 2011; Rey et al., 2014). In other words, "direct" perception of an object and the mental simulation of this object involve common sensorimotor units. Heurley and Ferrier argue in their perspective that these interactions serve to plan and control actions to interact with well-known objects in non-optimal perceptual conditions, thus producing adaptive behavior.

The idea that cognitive activities are strongly influenced by the given social and physical environment is referred to as situated cognition. Rey et al. demonstrate that motor simulation relies on the relevance of the object as a function of the task. Further evidence also supports the hypothesis that sensorimotor simulation is heavily influenced by situational aspects (Kapoula et al.). For instance, Amorim et al. have demonstrated that the visual angle and screen orientation along with contextual information modulates the resulting movement.

Edited and reviewed by: Bernhard Hommel, Leiden University, Netherlands

> \*Correspondence: Guillaume T. Vallet<EMAIL_ADDRESS>
#### Specialty section:

This article was submitted to Cognition, a section of the journal Frontiers in Psychology

Received: 27 November 2015 Accepted: 30 November 2015 Published: 05 January 2016

#### Citation:

Vallet GT, Brunel L, Riou B and Vermeulen N (2016) Editorial: Dynamics of Sensorimotor Interactions in Embodied Cognition. Front. Psychol. 6:1929. doi: 10.3389/fpsyg.2015.01929

The reverse is also applicable, since sensorimotor simulation shapes our perception of, as well as our interaction with, our environment. Grade et al. found that similar action simulation underlies both reachability and egocentric distance perception. This effect can be generalized across individuals and possibly incorporate the role of social environment in cognition. In accordance, Quesque and Coello show that during reach-tograsp actions, participants unconsciously modify their trajectory curvature based on their partner's eye level. This adaptation to biological motion is one the key component of social interaction, but human movements are so complex that they might be described as mathematically chaotic. Nonetheless, children quickly develop the ability to coordinate gaze, and in some respect posture, in response to complex chaotic motion structures (Haworth et al.).

The dynamics of sensorimotor interactions allows for situational behavioral adaptation in reaching and grasping, but also in more complex evaluative processes. For instance, associations between handedness and valence have previously been found (e.g., Casasanto, 2011). de la Vega et al. extend this finding by showing that strong right-footers respond to positive words faster with the dominant foot. The association between valence and laterality, however, becomes less clear when motor fluency is taken into account. Brouillet et al. observe that most participants prefer choosing stable supports for "good" items, regardless of side and handedness. The spacevalence association could also be reported for the vertical axis. Xie et al. demonstrate that the processing of affective valence concepts activated the vertical spatial axis (positive in the up position).

The association between cognition and space in the context of emotion is well documented, but it is not restricted to this cognitive domain. Hartmann et al. report that simple arithmetics are associated with gaze shifting along the vertical axis. Interestingly, operand magnitude partly modulated horizontal gaze position as well. The reciprocal influence of motor components on numerical cognition may also provide the opportunity to assess numerical cognition with methods such as

## REFERENCES

mouse tracking. In their perspective, Fischer and Hartmann point out important insight and methodological considerations on conceptual aspects related to numeric cognition. This perspective has been further elaborated by Faulkenberry and Rey.

These recent reports offer a new perspective on cognitive functioning, one that combines sensorimotor dynamics with contextual and body information. These new concepts provide new opportunities to explore related domains such as motivation (Shalev) and anticipation (Raab) and offer a new framework to interpret well-known and sometimes contradictory results in fields such as short-term memory (Macken et al.) or healthy cognitive aging (Vallet).

### AUTHOR CONTRIBUTIONS

Every author participates to the writing of the editorial article. All of them also significantly contribute to the edition of the articles of the research topic.

### FUNDING

GV is supported by a postdoctoral grant from the Fonds Québécois de la Recherche en Santé (FRQS). LB is supported by ACCEPT ("Assistance Tools and Cognitive Contribution: Embodied Potential of Technology"), French research ministerial mission (MiRe-DREES and CNSA). BR is supported by FS-CH (Formation Universitaire à distance, Suisse) and this work by the LabEx Cortex ("Construction, Function, and Cognitive Function and Rehabilitation of the Cortex," ANR-10-LABX-0042) of Université de Lyon, within the program "Investissements d'Avenir" (ANR-11-IDEX-0007) operated by the French National Research Agency (ANR).

### ACKNOWLEDGMENTS

The authors wish to thank Kristina Aurousseau for her precious help and meticulous proofreading as well as all authors and reviewers involved in the present research topic.

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2016 Vallet, Brunel, Riou and Vermeulen. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# **Mechanisms of embodiment**

### *Katinka Dijkstra\* and Lysanne Post*

*Department of Psychology, Faculty of Social Sciences, Erasmus University Rotterdam, Rotterdam, Netherlands*

This paper is a critical review of recent studies demonstrating the mechanism of sensorimotor simulation in different cognitive domains. Empirical studies that specify conditions under which embodiment occurs in different domains will be discussed and evaluated. Examples of relevant domains are language comprehension (Tucker and Ellis, 1998), autobiographical memory (Dijkstra et al., 2007), gestures (Alibali et al., 2014), facial mimicry (Stel and Vonk, 2010), and problem solving (Wiemers et al., 2014). The focus of the review is on supporting claims regarding sensorimotor simulation as well as on factors that modulate dynamic relationships between sensorimotor components in action and cognitive domains, such as expertise (Boschker et al., 2002). This discussion takes place within the context of currently debated issues, specifically the need to specify the underlying mechanisms of embodied representations (Zwaan, 2014; Körner et al., 2015).

**Keywords: embodied cognition, sensorimotor simulation, memory, cognition, online embodiment, offline embodiment**

## **INTRODUCTION**

#### *Edited by:*

*Guillaume T. Vallet, Centre de Recherche de l'Institut Universitaire de Gériatrie de Montréal, Canada*

#### *Reviewed by:*

*Lionel Brunel, Université Paul Valéry, France Denis Brouillet, Université Paul Valéry, France*

> *\*Correspondence: Katinka Dijkstra<EMAIL_ADDRESS>
#### *Specialty section:*

*This article was submitted to Cognition, a section of the journal Frontiers in Psychology*

*Received: 28 January 2015 Accepted: 22 September 2015 Published: 15 October 2015*

#### *Citation:*

*Dijkstra K and Post L (2015) Mechanisms of embodiment. Front. Psychol. 6:1525. doi: 10.3389/fpsyg.2015.01525* More than two decades after the grounding problem in symbol theories was brought up (Harnad, 1990) embodied cognition approaches have gained a stronghold in the study of cognition (Dijkstra and Zwaan, 2014). Since the first notion that cognition is grounded in perception and action (Glenberg, 1997), an abundance of empirical studies have demonstrated support for this groundedness (see for example Glenberg et al., 2013; Dijkstra and Zwaan, 2014).

Recently, the need to take stock of what all these studies contribute to the concept of grounded and embodied cognition has been expressed (Willems and Francken, 2012). Moreover, discontent with the current state of affairs has been noted. For example, opposing results in different studies have been interpreted as supporting embodied cognition approaches, indicating that the predictions may be too general to be falsified (Willems and Francken, 2012). Also, boundary issues have been raised regarding what phenomena embodied cognition approaches may or may not be able to explain (Mahon and Caramazza, 2008). For example, the issue whether, or to what extent, abstract symbols are also grounded in action and perception is a recurring topic in the debate on embodied versus disembodied approaches. Abstract concepts generally do not have physical or spatial referents, which renders a direct mapping of an abstract concept with a sensorimotor domain problematic at least (Mahon and Caramazza, 2008; Dijkstra et al., 2014).

This criticism does not stand on its own but is complemented by constructive proposals to counter the potential "erosion of the embodiment concept" (Willems and Francken, 2012) and to get out of the "impasse" regarding the discussion around the grounding of language comprehension (Zwaan, 2014). These proposals converge on the issue that the focus of current research should not be on supporting either embodied or disembodied accounts but on *how* sensorimotor systems and cognitive processes interact. The *pluralist view of cognition* proposed by Zwaan (2014) entails that abstract and grounded symbols contribute differently to language comprehension depending on how language is embedded in the environment in which it is used. In this view, research should focus on what the representations consist of and assess when and how they interact (Zwaan, 2014). Others proposals claim that research on embodiment lacks explanative power (Körner et al., 2015), should specify the conditions under which conditions certain phenomena do or do not occur (Willems and Francken, 2012), and integrate existing knowledge regarding embodiment to describe its underlying mechanisms (Körner et al., 2015).

This review is written within the scope of these proposals to more specifically assess conditions under which embodiment occurs or not and to gain insight into underlying mechanisms of embodiment. To accomplish this, studies are drawn from various domains in research in which embodiment effects have been found under clearly defined conditions and that demonstrate the mechanism that underlies these embodiment effects. The selection of studies discussed in this review is by no means exhaustive but illustrates how a limited number of claims that are empirically supported can contribute to a deeper understanding of embodiment effects. "Embodiment" is defined here broadly as the effect that the body or parts of the body (movement, position) can have on cognition or vice versa.

### **MECHANISMS OF EMBODIMENT— SENSORIMOTOR SIMULATION**

Recent reviews on embodied cognition have focused on establishing mechanisms of embodiment (Körner et al., 2015), describing relevant domains in embodiment effects (Dijkstra and Zwaan, 2014), such as language comprehension (Fischer and Zwaan, 2008), or stipulating a theory on embodied simulation (Gallese, 2007, 2009), or grounded cognition (Barsalou, 2008, 2010). A common element in their reviews is that sensorimotor simulation is considered to be a core element in cognitive processing.

Most recently, the idea of sensorimotor simulation as one of the main mechanisms underlying embodiment, has been developed further into specific ways in which this sensorimotor simulation takes place (Körner et al., 2015). The first claim is that the perception of a stimulus automatically triggers the simulation of reenactment with it. Actions are facilitated when they match the simulated actions and impeded when there is a mismatch between the two. Secondly, simulation may be blocked by a concurrent task that involves the same sensorimotor resources. The third claim is that simulation may also work offline, whereas the fourth claim entails that simulation depends on previous experiences and skills. Two other claims are discussed later on in this review.

In this review, both the mechanism of sensorimotor simulation as the claims derived from it are discussed in the context of research domains that illustrate the conditions under which embodiment occurred or failed to occur. Some of these domains have already been studied extensively, such as language comprehension, whereas fewer studies have been dedicated to other domains, such as autobiographical memory, gestures, expertise, facial mimicry, and problem solving. The review will remedy this omission by devoting more attention to these domains.

### **SENSORIMOTOR SIMULATION IN LANGUAGE COMPREHENSION**

The idea of sensorimotor simulation is that there are neural correlates between the content of what is being read and represented (i.e., action words) and the areas in the brain being activated (i.e., actions). Embodied cognition research on language comprehension has focused on examining the simulations being formed when reading sentences that are compatible with certain actions that are performed (Glenberg and Kaschak, 2002; Fischer and Zwaan, 2008; Kaschak et al., 2014). In one study, participants read sentences describing actions toward the body (John gave you a pencil) or away from the body (I closed the drawer) and had to make sensibility judgments regarding the content of the sentence by moving their hand toward or away from their body by pressing a button. Participants responded faster when the direction implied in the sentence (i.e., toward the body) was congruent with their own action (Glenberg and Kaschak, 2002). This supports the claim of congruence effects in sensorimotor simulation facilitating performance for congruent over incongruent conditions.

Other studies have shown how simulation is prevented from happening and how this affects cognitive processing. If the neural system is already engaged with a task at the time that a similar task requires a response and also involves the same neural circuits, interference will occur and embodiment is impeded. Consequently, there will be no facilitation of a congruent bodily manipulation on cognitive processes. This has been demonstrated in a study in which participants listened to sentences that implied upward or downward motion and concurrently watched a display on a screen that would scroll upward or downward (e.g., the cat climbed the tree; Kaschak et at., 2005). Participants had longer response times to decide whether the sentence made sense or not when sentence content and visual display were congruent than when they were incongruent. It was not possible to simulate the motion they represented from the sentence as they were processing the same motion in a different task. In the second experiment (Kaschak et at., 2005), participants responded faster to sentences when the visual stimulus concurrently showed motion in the opposite direction as was implied by the sentence content. Apparently, when neural circuits are already engaged for the motion part of the experiment, it is not possible to simulate sentence content that requires a mental representation of motion information at the same time. Thus, interference occurred for congruent but not for incongruent mappings.

The studies discussed above illustrate how and when the simulation mechanism operates and when it does not. They also support the claim that the perception of stimuli triggers their simulation when they match and the claim that simulation may be blocked by a concurrent task that involves the same sensorimotor resources. Compatibility effects are only present when neural circuits are available for simulation of congruent stimuli materials. If not, interference for processing congruent stimuli materials occurs. The third claim that simulation can work offline is also supported by empirical research and is described in more detail next.

Sensorimotor simulation may happen offline (Niedenthal et al., 2005; Körner et al., 2015) in the absence of a bodily state or action. Research has shown that retrieving an object from memory, and thinking or imagining an object generates responses in the body that show similarities to the responses that would occur if the object were present (Brouillet et al., 2010). Such an offline embodiment effect was demonstrated in a study conducted by Tucker and Ellis (1998). In this study, lists of words were presented to participants. Half the words represented small (penny) and half the words represented large objects (beach ball) and these words could be either natural or artificial. Participants indicated whether the object was natural or artificial with a joy stick and either had to apply a full grasp of the joystick or a precision grasp between thumb and forefinger. The results indicated that congruence in the grasp response with the type of object resulted in faster responses (small object-precision grip) than when this congruence was absent (large object-precision grip). Apparently, participants have a similar response in the absence of the object as when the object would have been actually present.

Another study demonstrating offline embodiment under congruent conditions of sensorimotor simulation was a language comprehension study (Pecher et al., 2009) in which participants responded whether a picture of an object occurred in the sentence preceding the picture. Previous studies have demonstrated that participants tend to simulate the shape of the object, being slower in the yes response of the object having occurred in the sentence when the shape does not entirely match the implied shape of the object in the sentence than when the object matches closely in shape (Stanfield and Zwaan, 2001; Zwaan et al., 2002). These were online effects. In the study by Pecher et al. (2009), however, the matching or mismatching pictures were presented at the end of the experiment when all sentences had been presented or after a 45-min delay. In both retention conditions, a match effect was found. Simulation of relevant aspects of the stimuli (shape) from memory should have occurred because only this mechanism can explain the faster responses under congruent (match) than under incongruent (mismatch) conditions.

The studies discussed above demonstrate sensorimotor simulation while processing the stimuli or considerable time after these stimuli have been processed or reconstructed from memory. Sensorimotor simulation affects the speed with which responses are made and requires that relevant aspects of the stimuli (for example shape or motion direction) are congruent in order to transpire unless neural circuits are already activated in a concurrent sensorimotor simulation and therefore blocking the simulation from happening. These results illustrate what favorable conditions are for sensorimotor simulation. Simulation happens under favorable conditions, such as compatibility, without the requirement that a bodily state or action is present at the same time. Simulation is hindered, however, when the same sensorimotor resources engaged in one task, are also needed for a second, concurrent task, suggesting that simulation may be a default mechanism that is constrained by how resources are engaged. More compatibility effects have been demonstrated in research on language comprehension (see for an overview Fischer and Zwaan, 2008; Kaschak et al., 2014) but it exceeds the purpose of this review to mention them all. Importantly, we aim to illustrate how certain claims regarding the sensorimotor simulation also operate in different domains, such as gestures.

## **SENSORIMOTOR SIMULATION IN GESTURES**

According to the Gesture as Simulated Action (GSA) framework (Hostetter and Alibali, 2008), gestures emerge when premotor activation exceeds a certain threshold and spreads to motor activation. This premotor activation is evoked by embodied (sensorimotor) simulations, which occur, for instance, while thinking about a task that would be facilitated by the use of gestures, such as pointing. The motor activation that results from exceeding the threshold is a gesture. This way, gestures express embodied simulations.

Gestures do not only express embodied simulations, but even have a causal role in the sense that gestures highlight perceptual and motor information which are consequently more likely to be incorporated in reasoning (Alibali et al., 2014). Specifically, the information conveyed in gestures influences the listeners' reasoning, because they refer to present and absent entities through pointing gestures and representational gestures, respectively, which help the listener to comprehend what the speaker is saying. Besides literally seeing what the speaker means because of the pointing gestures, the referential gestures can influence the readers simulations through activation of the motor system, but they can also cause the listener to mimic the speakers' gestures which allows the gestures to influence the listeners' reasoning the same way they influenced the speaker. This influence of gestures on simulations proposed in the GSA framework is therefore in line with the idea of sensorimotor simulation as a major mechanism underlying embodiment as proposed by Körner et al. (2015).

Indeed, several studies suggest that perception of gestures automatically triggers its simulation and that simulation may also work offline. A study by Cook and Tanenhaus (2009), for example, showed that speakers' co-speech gestures while explaining a problem solving task affected listeners' behavior when solving the problem on a computer (later in time, i.e., offline), that is, the mouse trajectories resembled the observed gestures. There is even evidence that speakers' gestures activates listeners' motor systems (i.e., automatic simulation; Ping et al., 2014). This study also showed that, in line with the claims made by Körner et al. (2015), actions are facilitated when they match the simulated actions and impeded when they mismatch. That is, the initially found positive effect on reaction times for pictures after congruent sentences from observing gestures disappeared when listeners moved their arms and hands. In contrast, the effect did not disappear when participants moved their legs and feet, indicating that the speakers' gestures indeed activated the listeners' motor system, that is, the part of the motor system that is involved in gesture related body parts, the arm and hands (Ping et al., 2014).

There seems to be, however, more to gesturing than sensorimotor simulation. Pouw et al. (2014) focus on the mechanism of offloading onto the environment, a claim on embodiment stipulated by Wilson (2002). According to Pouw et al.'s (2014) theory, gestures are external placeholders for internal processes. This offloading on the environment allows more internal processes to take place at the same time. Their point is illustrated by the following two examples. First, gesturing as if one actually rotates an object during a mental rotation task (e.g., Chu and Kita, 2008) can reveal information that is otherwise difficult to mentally compute and provide a physical platform to support internal processes (Pouw et al., 2014). Secondly, pointing gestures can aid internal cognitive processes by helping to keep track of counting (Delgado et al., 2011) or mental calculations (Hatano et al., 1977; Hatano and Osawa, 1983). Thus, Pouw et al. (2014) argue that gestures have a cognitive function, because they are used for and support cognition when the costs of mental computation are high (either by internal or external constraints).

Now, within the scope of the current paper, we ask what can the interaction between gestures and memory tell us about the underlying mechanisms of embodiment? Are gestures an externalization of cognition (Pouw et al., 2014) or are the relationships between gestures and cognition bidirectional, with one influencing the other and vice versa (Alibali et al., 2014)? Even though these two theories are not mutually exclusive, empirical research has mainly shown support for the latter claim (Beilock and Goldin-Meadow, 2010; Hostetter and Alibali, 2010; Post et al., 2013).

Positive effects of gestures on children's memory were found in a substantial number of studies on mathematics (Broaders et al., 2007; Cook et al., 2008), memory of a fictional story (Stevanoni and Salmon, 2005), and word learning (Tellier, 2008; see Macedonia and Von Kriegstein, 2012, for a review). Word learning in a foreign language was also facilitated by gesture observation for adults in a study by Kelly et al. (2009). These results are in line with the current ideas about embodiment. In the different studies, gesture production and observation seem to reflect and even elicit sensorimotor simulations. Because it is theorized that these simulations automatically emerge, gestures should be an effective way to improve memory. Indeed, memory was improved when participants used gestures (Kelly et al., 2009).

The studies discussed above demonstrated positive effects of gestures on learning and memory. But there are also circumstances under which gestures do not enhance learning (Beilock and Goldin-Meadow, 2010; De Nooijer et al., 2013; Post et al., 2013). Does that mean there is no embodiment in those cases? Or do those studies perhaps even provide evidence that embodiment is not characterized by sensorimotor simulations? Not necessarily. Beilock and Goldin-Meadow (2010) had participants gesture while explaining the Tower of Hanoi task. Their gestures reflected disk size and weight, as they used one-handed gestures when referring to the light disks and twohanded gestures when referring to the heavy disks. In a second task, weight was switched in a way that was less compatible with the gestures that were made earlier (i.e., small disks were heavier than large disks). Performance for participants in the gesture condition declined, indicating that the gestures actually represented the weight of the disks. Thus, in this case a detrimental effect of gestures is actually revealing sensorimotor simulations (i.e., embodiment).

Other studies in which negative effects of gestures were found shed light on the boundary conditions of sensorimotor simulation. For example, Post et al. (2013) found that, for children with poor general language skills, gestures increase cognitive load and hamper learning when imitated online during the first encounter with new material. For these children, sensorimotor simulation, and thus embodiment, was prevented by the cognitive overload they experienced. De Nooijer et al. (2013) found that gestures do help when imitated during either learning or testing, but not when imitated during both. It is not entirely clear what caused the effects found by the Nooijer et al., but it is clear that gestures are not always beneficial for memory.

There is also empirical evidence of the influence of cognition on gestures. Gesture frequency during descriptions of images of dot patterns was influenced by participants' physical experience with the patterns (Hostetter and Alibali, 2010). Participants gestured more when they had specific physical experience with the patterns than when they had only learned visually. The experiments show that this was not due to a decrease in verbal rehearsal or motor priming. Hostetter and Alibali conclude that gestures occur when people talk about thoughts that involve action simulations.

In sum, within the domain of gestures, the interaction of action and cognition is bidirectional. Gestures are a form of actions that can influence learning and memory and this is interpreted as support for embodied cognitive processes (Hostetter and Alibali, 2008; Pouw et al., 2014). In line with the claims made by Körner et al. (2015), the described gesture studies support the idea that perception can automatically trigger sensorimotor simulations, that actions can be facilitated or hampered depending on congruency with the simulated actions, and that simulations also work offline. However, there are boundaries as to how far this embodiment of gestures reaches, because gestures are not always supportive for learning, for example when cognitive resources are overloaded (Post et al., 2013).

The domain of gestures is after language comprehension the second cognitive domain that illustrates how sensorimotor simulation operates, under which conditions, and what the boundaries are. Another domain in which relevant aspects of the body can influence memory through simulation is autobiographical memory.

## **SENSORIMOTOR SIMULATION IN AUTOBIOGRAPHICAL MEMORY**

A simulation view of autobiographical memory entails that modality-specific states of perception, action, and introspection that were activated when an event was experienced, are activated again when the experience is represented at a later point in time (Niedenthal et al., 2005, 2014; Dijkstra et al., 2007; Dijkstra and Zwaan, 2014). If autobiographical memory is a simulation and reconstruction of the original experience along with the relevant perceptual, sensorimotor and affective components of the experience, facilitation of the retrieval process should occur if these components of the original experience are triggered when the retrieval process is initiated (Dijkstra and Zwaan, 2014).

Support for this prediction was found in a study in which participants retrieved autobiographical memories in bodycongruent and body-incongruent positions relative to that of the original experience (Dijkstra et al., 2007). Participants were not only faster retrieving the memory when prompted after assuming a body-congruent position (talking about a previous visit to the dentist while being reclined in a chair) than in a body-incongruent position (talking about this dentist visit while standing up with the legs out and the hands in the waist) but also retained the memory better after a period of 2 weeks when they were asked to talk about the memories they retrieved 2 weeks earlier. This study demonstrates sensorimotor simulation under congruent conditions of body position both during retrieval and long-term retention tasks. It seems as if body position is a sensorimotor component of the original experience that acts as an embodied cue to facilitate the reconstruction of the original experience. The ease of retrieval is reflected in faster retrieval times to access a relevant (i.e., body congruent) memory and even strengthens the memory trace so that body-congruent memories are remembered better at a later point in time.

This study supports two of the claims that were discussed earlier, compatibility effects when there is a correspondence between the stimuli condition (body position) and its simulation (the body position that was relevant during the original experience). The other claim is that the study demonstrates offline embodiment because experiences from the past are being reconstructed rather than experiences that happen at this moment. The study also supports the claim regarding embodiment by Wilson (2002) that offline cognition is body based, specifically for memories that are records of spatiotemporally localized events that are relived by the person who remembers them. Studies demonstrating congruence effects with a body manipulation thus provide strong support for claims regarding sensorimotor simulation.

Do these simulations extend to more indirect relationships between bodily states and autobiographical memory simulations? For example, the mapping of certain actions, such as moving hands "up" or "down" with certain emotions, such as "positive" for "up" and "negative" for "down" illustrate such an indirect relationship. This mapping is an example of a conceptual metaphor, abstract concepts that have metaphorical associations with actual experiences (Dijkstra et al., 2014). Conceptual metaphors arise from a pattern of associations of concrete experiences (cheering, jumping out of joy) with certain body movements (upward movement). Conceptual metaphors, such as "positive is up" can be understood in terms of concrete concepts and experiences, such as the times that you cheered when your favorite soccer team scored a goal, or the times you slumped in your seat when the other team scored against your team. Based on the conceptual metaphor that maps verticality with valence, the prediction can be made that the retrieval of positive or negative autobiographical memories should be facilitated when an up or down action is performed, but only under movement-valence congruent conditions (up and positive, down and negative).

This facilitation was demonstrated in a study involving body movement and emotional memory retrieval (Casasanto and Dijkstra, 2010). Participants retrieved autobiographical memories to prompts while moving both hands upward to deposit a marble in each hand in two containers that were placed in a higher location, or downward into containers below. The idea behind the study was that an upward movement triggers the "up is positive" metaphor and facilitates retrieval of positive memories, while a steady downward movement triggers the "down is negative" metaphor and facilitates retrieval of negative memories. The tobe-retrieved memories, were either positive or negative (Tell me of a time you felt proud of yourself/Tell me of a time you felt ashamed of yourself) in Experiment 1, and neutral (Tell me of an event that happened yesterday) in Experiment 2. In Experiment 1, reaction times to congruent (positive is up) and incongruent (positive is down) movement/valence memories were assessed. Participants were faster retrieving memories in congruent than incongruent trials. In the second experiment, participants again moved their hands but first retrieved memories to neutral prompts silently and then retold the memories afterward when their hands did not move. They were more likely to retell memories they later judged as positive when they moved their hands upward.

Just as in the previous study with body position (Dijkstra et al., 2007), body movement seemed to facilitate access to the memory itself by activating a relevant aspect of the experience, the emotion that was experienced at that time. This was an indirect trigger, however, because the emotion arose as a result of the mapping of a motor action and the associated emotion with that action. Both studies demonstrate the role of offline sensorimotor simulation under congruent conditions. The last study also supports the claim by Körner et al. (2015) that simulation may play a causal role in processing emotion. Only the activation of the mapping between valence ("positive is up") and the vertical movement of the hands (up) explains why participants have better access to emotion-movement congruent memories and attribute a certain emotion to memories when they move their hands a certain way.

A similar embodiment effect demonstrating the mapping of body position with valence was demonstrated in a study by Riskind (1983). Participants were instructed to be either in an upright position and smile or in a slumped position with a downcast expression and their head and neck down. While being in this position, they had to retrieve unpleasant and pleasant experiences. The results indicated that participants were faster in retrieval of these experiences in body positionvalence congruent condition (upright and pleasant or slumped and unpleasant) compared to body position-valence incongruent conditions. Again, research demonstrated that manipulations of the body affect cognitive processes under specific conditions: those of body-valence congruence.

A last study that contributes to our insight into sensorimotor simulation in autobiographical memory does not involve the retrieval of emotional experiences but the generation of emotion words, such as "disappointment" and "pride" that are associated with previous emotional experiences in which the word was relevant (Oosterwijk et al., 2009). Given the association of positive words with up movements and negative words with down movements, changes in body posture (straight up or slumped) were expected depending on the emotion that was elicited. Participants' height was measured (with a hidden camera) during the word generation procedure which would give an indication of the posture demonstrating pride (upright posture) or disappointment (slumped posture). The results showed that participants indeed changed their body position to a lower, more slumped position when disappointment words compared to the body position during the generation of pride words (Oosterwijk et al., 2009).

Other than the previous study, this study illustrates a more subtle embodiment effect. Body position was influenced by the simulated sensation that was triggered by the valence of the presented words. A major difference with the autobiographical memory studies, discussed above, is that the effect of perceiving a valenced stimulus triggered the simulation of the valence-matching body position, not the other way around as in the previous studies. Such bidirectional effects were also demonstrated in the discussion of research on gestures. A similarity with the autobiographical memory studies on valence was that all three studies illustrate the claim that simulation plays a causal role in processing emotion. A stimulus or action that is congruent in valence with the simulated stimulus or action can be simulated more readily and easily than when these favorable conditions are absent. Autobiographical memory retrieval and sensorimotor processes tap into long-term memory stores of experiences that are being simulated when triggered appropriately. In other domains, the accumulation of motor experiences builds up to a certain level of motor fluency and a knowledge base that can be tapped into when being triggered. Individuals, who have done this over a long period of time, can tap into this store more easily and efficiently than those who have not. They are considered to be experts.

### **SENSORIMOTOR SIMULATION: EXPERTISE**

What happens when a person has performed complex motor movements and motor sequences so many times that a high level of expertise has been reached? From an embodied cognition perspective, if someone has extensive experience with in an action domain, such as tennis, more grounded representations will be activated when playing or talking about the sport compared with a novice. This fits with the claim by Körner et al. (2015) that automatic simulation depends on previous experiences and skills. Experts are likely to form a full-blown first person mental simulation of the described actions (with many grounded representations) whereas a domain-novice would form shallow, word-like representations instead (Sutton and Williamson, 2014). The motor expertise that has been developed would then facilitate any processing and memory of expertise-related information that is presented to this expert. Several studies have demonstrated support for this assumption.

In one study, expert climbers were tested to see if they would remember climbing routes of wall elements better compared to novices (Boschker et al., 2002). Expert climbers have built up motor fluency of specific climbing movements and should therefore remember these routes better. In addition, they should notice elements of the climbing environment as potential holds for grasping and establishing a route for ascend or descend to a greater extent than novices. The results supported the assumption that experts remembered the climbing routes with the affordances that the wall elements offered whereas the more inexperienced climbers tended to focus on structural features of the climbing wall and did not remember the routes as well. Expertise thus facilitates the way to take in and remember expertise-related information that results in superior performance on cognitive tasks. Here, the sensorimotor system involves an accumulation of experiences that feed into the cognitive domain.

There is another relevant aspect of expertise when considered from an embodied cognition perspective, which may actually hinder performance on certain tasks, and this is loss of attentional control. Expert skills are built up over a substantial period of time, resulting in competencies that are part of procedural knowledge and no longer require explicit attentional control (Beilock et al., 2004). Consequently, if actions have to be performed with unlimited time available, novice learners in a domain should benefit from this whereas it may actually harm an expert because control processes may come into play for a task that only requires procedural skills (Beilock et al., 2004).

These effects of expertise were demonstrated when experts imagined actions that were within their expert motor repertoire (Beilock and Gonso, 2008). Novice golfers had lower accuracy scores in putting when speed was stressed in the instruction and they had to actually make a putt, or imagine putting a certain sequences of actions. In contrast, expert golfers performed better on actual and imagined putting when time was more limited because they tapped into their proceduralized skill under conditions of time pressure but allowed conscious control when more time was available which impaired their performance. Imagery appeared to serve the function of action readiness in these experts. This is similar to the step-by-step unfolding of the action itself and involves the same cognitive and motor parameters. When speed is no issue, other elements become important in the imagery process of experts that involve explicit control of the skill, which gets in the way of the planning of actual steps.

Motor expertise may therefore enhance memory performance and performance accuracy because complex patterns are practiced frequently and readily available. However, superior memory performance in experts tends to be limited to expertiserelated information relative to everyday information and novices when stimuli were encoded through observation, or enactment (Dijkstra et al., 2008). The knowledge and experience base are specific to the domain of expertise and do not necessarily transfer to other domains.

Because the action patterns that form the basis of expert performance are overlearned, motor expertise may also lead to reconstruction bias (Barsalou, 1999) in tasks that may cause some confusion as to whether an item tapping into this fluency was encountered before or not. As a result, it may be more difficult to differentiate between patterns that were or were not presented or imagined previously. This false recognition bias was examined by (Yang et al., 2009) who assessed the effect of the expert skill of typing on recognition rates of letter dyads that would normally be typed with the same finger (j and h with the index finger) or with different fingers (j and l), reflecting motor fluency among expert typists. The idea was that the activation of action plans that are associated with the different letter dyads, such as dyads reflecting higher motor fluency could lead to decision errors on a recognition task among experts but not novices. Experts have more consistent mappings between certain letters and how they type them. The results supported this assumption. Skilled typists recognized different-finger letter dyads that were not presented earlier incorrectly as having been presented before more frequently than novices. Recognition memory was influenced by the covert simulation of repeated action patterns among experts. As a typist, you cannot help but covertly simulate the motor action of typing letters when you are merely processing them visually. Therefore, you think you saw letter dyads before because you have a covert motor trace of activating these letters. In other words, your motor fluency gets in the way of simply performing the cognitive task.

These studies on the role of expertise show how sensorimotor simulation operates in the domain of expertise. The activation of previous experiences relevant for an expert task results in the availability of relevant knowledge for a particular task. This knowledge can be utilized in the performance of motor and memory tasks. If other information is allowed to enter the system when there is ample time available or if processing information in one modality allows for covert simulation of action patterns that are part of motor fluency, expertise can actually get in the way of performance in the cognitive domain. What we learn from these studies is that there are different aspects of expertise that play a differential role in the interaction between sensorimotor systems and cognitive domains. These have to be taken into account when examining the role of expertise from an embodied cognition perspective.

The studies discussed so far were chosen because their results support claims regarding sensorimotor simulations. Language comprehension is facilitated under congruent stimulus-simulation conditions and hindered when the neutral circuits needed for simulation are already engaged with another task. Gestures can be considered as a way to offload information into the environment reducing working memory load and increase availability of cognitive resources for internal cognitive processes. Bodily cues, such as body position, help the reconstruction process of the original experience when retrieving an autobiographical memory. Expertise intensifies the interactions between action and cognitive domains. Repeated action patterns allow experts to take in and remember more relevant aspects of a scene than novices (Boschker et al., 2002) because they have learned to perceive their environment in a different manner. All these effects operate through sensorimotor simulation under congruent conditions and both online and offline. However, once motor fluency has been developed in experts, it cannot be undone easily which suggests that performance may include activated yet incorrect information from a motor repertoire, forming a bias in memory processes.

In the preceding sections, research on offline embodiment effects was discussed in the domains of language comprehension, gestures, autobiographical memory, and expertise. Simulation may occur in the absence of an object, in a later stage of stimuli processing, when retrieving an event from the past, and as an accumulation of previous experiences. Another domain in which offline embodiment effects can be explained with sensorimotor simulation, is problem solving.

### **SENSORIMOTOR SIMULATION: PROBLEM-SOLVING**

Embodied research on problem-solving often demonstrates the use of physical or imagined simulation to facilitate finding the solution for a task. A well-known example in this respect is a study by (Kirsh and Maglio, 1994) in which participants have to rotate and flip falling block shapes quickly to make them fit with the surface and the blocks they fall on. Participants use the strategy of actual rotation of the blocks in order to determine the best fit rather than mental computation to solve the problem. This fits with the idea of sensorimotor simulation and the facilitating role of the body in the simulation process (Dixon et al., 2014).

An example of such simulation is a study that employed a gearsystem problem. Participants have to predict the turning direction of the final gear in a series based on the turning direction of the gear that drives the force to the system. Participants typically solve this problem by employing their body, that is, by manually simulating the movement with their hand of each gear that follows the previous one (Dixon et al., 2014). This manual simulation then helps participants to discover the higher order solution that applies to all these problems which is the insight that alternation occurs of the gear direction.

Simulation based on indirect but commonly occurring mappings between abstract concepts and concrete experiences, as we saw in the studies in the autobiographical memory domain has also been demonstrated in the domain of problem solving. Apart from the "positive is up," mapping, other mappings exist, for example in relation to the mental number line, "many is up" or "few is left." They can also be activated with sensorimotor manipulations and facilitate processing of information if the mapping is present. Numerical magnitude can be represented both horizontally, with left representing a smaller quantity than right (Pinhas and Fischer, 2008), and vertically, with up representing a larger quantity than down (Shaki and Fischer, 2012). We stack coins into piles with higher piles indicating higher quantities and numbers are usually written horizontally with lower numbers being presented on the left and higher numbers being presented on the right.

Wiemers et al. (2014) examined the activation of these two representations, horizontal and vertical, in a mental arithmetic task. Participants performed addition and subtraction tasks while making upward and downward movements, movements to the right and left, or no movement at all with their right arm in Experiment 1. They solved more problems under movement/magnitude congruent than incongruent conditions. In Experiment 2, they did the same arithmetic task but this time the arithmetic problem (and not their arm) moved in the directions described above. Again a compatibility effect of movement with the spatial representation of the task (up and addition with down and subtraction) was shown. This study demonstrates the idea of sensorimotor simulation through compatibility of actual and perceived movement in order to solve the task. Moreover, earlier experiences with arithmetic and the representation of magnitude were activated when the magnitude-spatial mapping occurred which resulted in facilitation of the response.

Werner and Raab (2014) also investigated the mapping between horizontal spatial representations and the abstract concept of magnitude but in a different type of problem solving task. The authors used the water-jar problem which requires participants to obtain a required volume of water when they are given certain empty jars for measure. Participants were primed to a left or right gaze direction in a perception task to mentally compare full jars either to the left or the right of a similar empty jar. This should bias them to either the left or right when being presented with the water jug problem and after they sorted marbles from outer bowls inward (plus group) or the other way around (minus group). Participants indeed demonstrated a spatial bias in gaze direction to the right for the plus group and to the left for the minus group without there being differences in overall problem solving ability.

A similar gaze design was used by (Thomas and Lleras, 2007) who manipulated gaze behavior in participants prior to them being exposed to a problem solving task, known as the Duncker radiation task. Participants have to destroy a tumor in a patient on a computer screen with lasers while keeping the tissue around the tumor healthy. The solution to this task is to have multiple laser beams fire at low intensity at the tumor from various locations around the tumor so that the convergence of these beams can destroy the tumor, yet keep the tissue intact. Participants whose gaze behavior was manipulated to make saccadic eye movements between the tumor and the surrounding locations, which hints at the solution, were more successful in solving the problem later than participants who were instructed to fixate their gaze on the tumor. Here, the practiced eye movements (moving in and out versus fixed) facilitated later problem solving by activating the sensorimotor simulation to tackle the problem.

A last study demonstrating sensorimotor simulation in a problem solving task involved spatial working memory (Thomas, 2013). Again, the underlying assumption was that movement with arms or eyes may embody the solution to an insight problem through simulation. A second assumption was, however, that actions will only affect problem solving when the representations of these actions are active in working memory. If these representations are (no longer) active, no effect will occur. Participants tried to solve the Duncker radiation problem, but different from the study discussed above, participants had to occasionally perform a visual tracking task. Moreover, they held a spatial (a grid filled with dots) or verbal stimulus (a string of digits) active in working memory at the same time. It was expected that holding a spatial stimulus in memory would engage spatial working memory resources and therefore interfere with the problem solving task, even if the eye movements are directed at various locations around the tumor. The results showed that being assigned to an embodied-solution condition (observing different colored stimuli from different locations around the tumor toward the tumor and out again) indeed was not sufficient for better problem solving performance. Only if they also held a verbal stimulus active in working memory did they solve the Duncker problem after fewer attempt intervals. They also did this faster than participants who were in the same embodiedsolution condition but also held a spatial stimulus active. In other words, representations in spatial working memory affected problem solving even if participants were visually primed toward a solution of the problem. This demonstrates the simulation blocking effect that we encountered in the discussion of language comprehension research. If neural circuits are already engaged with a concurrent task, simulation is impeded or blocked.

The studies discussed from the domain of problem solving also support the claims regarding sensorimotor simulation. Under specific conditions of congruence as long as neural circuits are not engaged beyond capacity, sensorimotor simulation may aid problem solving. This simulation may be a physical or mental simulation and can take place during the task or afterward. It may also apply to the last domain to be discussed in this review, facial mimicry. Research conducted within this domain also supports the last claim from Körner et al. (2015)that has not been discussed as of yet, and that is the claim that simulation is important in social interactions.

### **SENSORIMOTOR SIMULATION: FACIAL MIMICRY**

Facial mimicry is the imitation of facial expressions by a person who is observing these expressions in another person.  The mimicry of these emotions on the face is the simulation of what one observes. How does this happen? When individuals observe actions and emotions of others, the same neural circuits are engaged as when the same actions and emotions would be experienced personally (Oosterwijk and Barrett, 2014). We therefore understand the feelings and emotions of another person through the engagement of neural circuits based from a recreation of these feelings in ourselves (Niedenthal et al., 2005). From an embodiment perspective, this is an interesting notion because it illustrates how embodiment is a shared rather than an individual phenomenon and it fits with a sensorimotor simulation account that presupposes a causal role regarding emotion processing and is important in social interactions (Körner et al., 2015).

Researchers on facial mimicry agree that emotions are grounded in online bodily states that arise from a context-specific experience in which the emotion is evoked and distributed across multiple regions of the brain (Niedenthal et al., 2014). Facial mimicry then is the way how the sharing of emotion becomes apparent for the other (Niedenthal et al., 2014). Facial mimicry can help to strengthen social bonds between people because a speaker receives through the facial expression of the listener the feedback that the other understands what is being communicated. If the expression is emotional, facial mimicry may signal the other is feeling the same way. This benefit of understanding the other may not be limited to understanding how the other person is feeling but extend to understanding what this person is communicating.

Empirical support for these assumptions comes from research manipulating how an emotion can be expressed on the face (Niedenthal et al., 2001; Oberman et al., 2007; Stel and Vonk, 2010). An emotional expression on the face can either be induced by placing a pen between the teeth, or blocked by placing a pen between the lips to block the expression of smiling. In one study employing the emotion-blocking procedure, some participants held a pen in the mouth between the lips to hinder the expression of smiling whereas other participants could display their emotions freely. When instructed to detect changes in facial emotion expression in morphed pictures (from sad to happy) participants were slower to recognize the emotion change when their ability to display the emotion was blocked than participants who did not have this restriction (Niedenthal et al., 2001). In another emotionblocking study, participants placing pressure on a pen, showed an impaired ability to recognize happy faces relative to participants who could show facial expressions without being hindered in their expression and thus engaging the relevant muscles (Oberman et al., 2007). Both studies demonstrated better performance on a cognitive task when participants were able to freely express their emotions on the face relative to a condition in which the expression was blocked. Other than the studies discussed above in which simulation was impeded because of engagement of neural circuits with a concurrent task, here simulation is physically blocked by limiting the muscles to contract to express emotion.

These effects of manipulations of the face can also be the result of more invasive procedures, for example when getting the first Botox injection. People have these injections to eliminate wrinkles from their forehead for a certain period of time. A by-effect is that at the peak of the effectiveness of the injection, the corrugator muscle that normally is involved in frowning does not participate in the frowning after Botox is injected. This non-participation of the muscle was used in a study in which participants read sentences with angry, sad, and positive content right before and 3 weeks after the Botox injection when the effectiveness of the injection reaches a peak (Havas et al., 2010). The results indicated that reading times were slower after the injection than before but only for the sentences with negative content. Because participants were no longer able to frown due to the paralysis of the muscles in the corresponding area, facial feedback of expressing the emotion to emotion-relevant neural circuits was disrupted and processing of negative sentences affected because the associated frowning expression could not be made.

The studies discussed so far on facial expression and mimicry illustrate how sensorimotor simulation occurs and under what conditions this may or may not happen. This effect is specific to the emotion that can be displayed on the face and the emotion that is depicted in or activated with the stimuli materials. The effects discussed so far were online effects where the observation and display of emotions occurred while processing emotional expressions on a face, or valenced information in sentences. A few studies have also examined offline effects of facial expression, supporting the claim that simulation may work offline when the bodily state no longer is relevant for a cognitive task to be performed.

Specifically, studies have assessed the effects of manipulations by having participants remember or retrieve emotional information after the task was completed (Schnall and Laird, 2003; Halberstadt et al., 2009). A study by Halberstadt et al. (2009) included pictures of ambiguous facial expressions that were paired with an emotion concept, such as "happy," or with a concept unrelated to emotion, such as "reliable." Afterward, faces encoded with the happy concepts were remembered as happier than in the learning phase whereas no such effect was demonstrated for the concepts that were unrelated to emotion (Halberstadt et al., 2009). The encoding process thus was relevant for later memory when the pictures of the faces and the labels of the concepts were no longer available for comparison. In another study, participants practiced facial expressions and body postures based on emotional cues (Schnall and Laird, 2003). After this practice phase, participants responsive to personal cues recalled more life events that were emotionally congruent (positive memories to an upright body posture and smiling facial expression) than incongruent with the practiced expressions.

Both studies demonstrated offline simulation because once an emotion-congruent association of facial or bodily expression with emotion concepts was established, memory for encoded or newly retrieved materials was higher under congruent than under incongruent conditions. These studies on facial mimicry also underscore the causal role in emotion processing. An emotional stimulus is needed in order for the simulation to occur or be blocked. Otherwise, simulation of the emotional expression on the face would be irrelevant. Moreover, if the emotion cannot be simulated on the face because of a physical (pen) or mental (suppression) prevention of emotion to be shown, the benefits as a result of the compatibility effects disappear. Finally, the simulation mechanism may work offline, after a bodily state has occurred. This was shown in the studies demonstrating benefits of facial mimicry in memory tasks in which earlier encoded information under simulated conditions had to be retrieved at a later point in time when the manipulation of the body was no longer relevant.

Facial mimicry was the last domain that demonstrated sensorimotor simulation, in particular its effect on emotion processing and within a social context. Together, these six domains, language comprehension, gestures, autobiographical memory, expertise, problem solving, and facial mimicry, support an account of sensorimotor simulation to explain embodiment effects in specific ways. Now, it is time to take stock of what this empirical support in those domains has brought us. How stable and general is sensorimotor simulation?

## **CONCLUSION**

Some researchers claim that the degree of simulation required for a task is inversely related to the abstraction needed for it (Myachykov et al., 2014). In this view, the more abstract the process, the more offline it is, and the more stable it is. In contrast, online processes are less stable because of their sensitivity to individual differences in the sensorimotor experience. Although this may be a promising notion that could be developed further, our review shows rather how universal sensorimotor simulation is, as much for abstract and offline processes as for concrete and online processes, by accounting for a wide variety of embodiment effects in various domains. It remains to be seen whether online processes are less stable than offline processes and how sensitive online processes are to individual differences in the sensorimotor experience. Based on the discussion of the studies on online processes in this review, there is no evidence that online processes are less stable than offline processes.

Interference of embodiment was demonstrated in research on language comprehension, gestures, and problem solving. This phenomenon occurs when the same neural circuits are engaged for the sensorimotor as the cognitive processes in an experiment. At this point, it is not entirely clear exactly how the interference occurs and whether it is strictly a capacity problem or a dual engagement problem. It appears that involvement in one task leaves insufficient resources for the other and that this interferes with simulation. This was also found in some of the gesture studies that demonstrated negative effects of gestures among children with poor general language skills (Post et al., 2013). For this group, gestures increased cognitive load and had a negative effect on performance. Further examination of these conditions is essential to a deeper understanding of how sensorimotor simulation operates.

As said before, this overview of research on sensorimotor simulation in various domains is by no means exhaustive. Examples from different domains were chosen to illustrate this mechanism and specific claims operating under this mechanism. This review can be considered a first step toward an approach that further defines conditions under which embodiment occurs or not, and how they depend on previous experiences and skills, affect emotion processing, and operate in social situations. The evidence so far points in the direction of congruence/compatibility being a necessary condition for embodiment to occur and for engagement of similar systems to hinder embodiment. Only in this context can offline simulation and emotion processing be accomplished. The role of simulation in social interactions requires further study. Only support from one domain was discussed in this review.

Future research should also start to address questions regarding individuals who may be less able or susceptible to the effects of

### **REFERENCES**

the body. How does sensorimotor simulation work for individuals with depression, aphasia or dementia? Would children be more or less affected by manipulations of the body, or does it not make a difference? For example, children with poor language skills may not benefit as much from body manipulations, such as gestures, compared to children with average to good language skills (Post et al., 2013). Likewise, older adults may not benefit as much from embodied encoding compared to young adults when their memories that are being retrieved, are more remote (Dijkstra et al., 2007). On the other hand, in the posture study (Dijkstra et al., 2007), older adults demonstrated as much facilitation from congruent postures as younger adults.

With these studies, domains, and claims, we have now a better understanding of one of the major mechanisms underlying embodiment; sensorimotor simulation. The future holds promise for further exploration and specification of the claims associated with this mechanism and gathering empirical evidence to support these claims.

### **ACKNOWLEDGMENT**

During the realization of this work, LP was supported by a grant from the Netherlands Organization for Scientific Research (NWO-PROO; Project number 411-10-907).

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Copyright © 2015 Dijkstra and Post. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# What are memory-perception interactions for? Implications for action

#### *Loïc P. Heurley1 \* and Laurent P. Ferrier 2,3*

*1 Laboratory CeRSM EA2931, Center of Research on Sport and Movement, Université Paris Ouest-Nanterre La Défense, Nanterre, France*

*2 Institut Français des Sciences et Technologies des Transports, de l'Aménagement et des Réseaux, Laboratoire de Recherche Mécanismes d'Accidents, Salon de Provence, France*

*3 Laboratory EPSYLON EA4556, Dynamics of Human Abilities and Health Behaviors, Université Montpellier III Paul Valéry, Montpellier, France*

*\*Correspondence<EMAIL_ADDRESS>
#### *Edited by:*

*Benoit Riou, Université Lyon 2, France*

#### *Reviewed by:*

*Dermot Lynott, Lancaster University, UK*

**Keywords: embodied cognition, memory, knowledge, perception-action interaction, size perception, grasping movements**

Currently, a growing body of studies demonstrates memory-perception interactions (see Barsalou, 2008; Heurley et al., 2012; Lobel, 2014, for reviews). Even if such interactions are highly relevant to support embodied approaches of cognition as well as to better understand memory and perception (e.g., Zwaan, 2008; Versace et al., 2009; Landau et al., 2010; Kiefer and Barsalou, 2013), their functional role remains unclear: Why would perception integrate memory and knowledge while it seems highly efficient without such influences? To understand the functional relevance of these interactions, we assume that it is necessary to take into account two important conditions in which our cognitive systems have evolved during the phylogenesis and continue to evolve during our ontogenesis. More precisely, we develop a view where memory-perception interactions are highly relevant to plan and control actions when we interact with well-known objects in non-optimal perceptual conditions.

It is widely accepted that to properly parameterize action components and to control them during the course of action, it is necessary to perceptually process some object's features (Hommel and Elsner, 2009). As already claimed by Glover (2004), these "action-relevant perceptual features" (ARPF) can be spatial (e.g., shape, orientation) as well as non-spatial (e.g., fragility, weight). Among spatial-ARPF, size is usually recognized as an important cause to its involvement in a great variety of actions. Jeannerod (1984) has for example demonstrated that the magnitude of the grip aperture, a component of the grasping movement, is function to the visual size of objects (see also Ellis et al., 2007; Fagioli et al., 2007; Wykowska et al., 2009). Visual size processing also seems highly important in order to intercept flying objects (Lee, 1976). Nevertheless, very frequently and for various reasons, the perceptual processing of ARPF is far from being optimal especially in "out-of-laboratory" conditions. For instance, some ARPF can't be processed by the available perceptual channels. Indeed, when we want to grasp an object, we are only able to visually perceive it and therefore we are unable to directly perceive its fragility, weight, and temperature whereas they are extremely relevant to plan the force and the velocity of the grasp (Glover, 2004). Even when the right channels are available, some environmental conditions can impair perception. For example, the occlusion of an object by other surfaces can limit our ability to visually perceive it and thus process its shape, size, or distance (Tanaka et al., 2001). Furthermore, short- or long-term injuries to perceptual systems can also induce nonoptimal conditions of perception. Indeed, the eyes can be long-term-impaired by cell aging or short-term-impaired by an intense flash light, but in both cases our ability to process visual features is affected. Accordingly, how do we plan and control actions in conditions where features that are suited to plan and control relevant parameters of action cannot often be optimally perceived? First of all, it is important to note that non-optimal processing of ARPF do not necessarily induce object-recognition problems. Indeed, as mentioned in several models, object recognition can be accurately based on non-ARPF such as the color and/or the texture of objects and the context (Tanaka et al., 2001; Bar, 2009). Therefore, even if some ARPF cannot be processed, objects can be accurately identified in many cases. Secondly, because in everyday life we mainly interact with well-known objects, preserved ability to recognize object identity can automatically induce the retrieval of a myriad of knowledge associated with the recognized-objects including associated ARPF (e.g., shape, size). Thus, we claim that recognition processes used to identify objects during the planning phase of actions involve the retrieval of previous experienced ARPF that are automatically integrated into perception. We also claim that they allow compensating nonoptimally perceived ARPF and so to maintain a high level of action efficiency even in non-optimal conditions of perception. To resume, we assume that the functional relevance of memory-perception interactions (i.e., an embodied cognitive architecture) occurs when humans interact with wellknown objects in degraded-perceptualconditions. We discuss three potential sets of evidence, coming from studies all focusing on the ARPF size in support of this view.

First, numerous experiments suggests that memory would be able to store objects' perceptual features and especially ARPF (see Barsalou, 2008, for a review). For instance, a great variety of studies support the idea that the size of objects is accurately stored in memory and closely matches their real visual organization (Moyer, 1973; Holyoak, 1977; Holyoak et al., 1979; Shoben and Wilson, 1998; Bertamini et al., 2011; Konkle and Oliva, 2011; Linsen et al., 2011). More importantly, it seems that the known size of objects can be automatically retrieved even when objects are briefly perceived suggesting the possible automatic retrieval of ARPF during fast real interactions with objects. Ferrier et al. (2007) have for example demonstrated that a target picture (e.g., an elephant) is easily categorized as an animal when the brief prime picture (150 ms) has a similar known size (e.g., a giraffe or a car) rather than a different (e.g., a bee or a key) while both pictures have the same visual size on the screen (see also Setti et al., 2009; Gabay et al., 2013). It is noteworthy that the size is generally stable across items of a category as well as across experiences. Because all the ladybugs we experienced have approximately the same small size, their size can be easily stored at a conceptual level (i.e., general knowledge, Whittlesea, 1987). However, in some cases, ARPF could be stored in a more specific or short-term format. For example, because the size of your car is not shared by all the exemplars of the "car" category, this feature is undoubtedly stored in a more autobiographic format. Furthermore, some ARPF are so variable that we can only store them for a short period of time like the last position of your car on the supermarket car park or the distance of some objects on a table (see also Borghi, 2013 for a close distinction). Thus, we claim that ARPF can be stored and automatically retrieved from memory but perhaps in various ways according to the stability of the ARPF across experiences.

Moreover, several studies suggest that ARPF are not only stored but can also influence conscious perception. Among others, the case of the size perception has been strongly studied. In a primary study, Paivio (1975) has demonstrated that the comparison of the known sizes of objects is faster when they are congruent with their visual sizes. In others words, it is easier to say that in general an elephant is larger than a mouse when in the experiment the picture of the elephant is presented larger than the picture of the mouse rather than smaller (see also Srinivas, 1996; Rubinsten and Henik, 2002; Konkle and Oliva, 2012, for similar results). The works of Riou et al. (2011) and Rey et al. (2014) go further and suggest the automatic nature of this influence. Riou et al. (2011) have demonstrated that the known size of objects can influence the detection of a visually oddsized stimulus in a visual search task while such an object's feature is absolutely useless to complete the task. Others studies have demonstrated an influence of the known size of objects on the judgment of distance that are often derived from visual size suggesting that the stored size can automatically impact not only the perception of visual size but also the perception of other ARPF derived from it (Epstein, 1965; Predebon, 1992, 1994; Hershenson and Samuels, 1999; Distler et al., 2000). Besides the known size of objects, the perception of visual size can be affected by a more abstract kind of size representation: numbers. Henik and Tzelgov (1982) have replicated the interaction between visualand stored-size reported by Paivio (1975) but with numbers. In a classic bisection task requiring implicit length estimation, de Hevia and Spelke (2009) have found a bias of bisection toward the side of the line where the larger number is printed. In a reproduction task, Viarouge and de Hevia (2013) have demonstrated that large numbers (e.g., 9) presented at each corner of a square induce larger reproduction of this square compared to the condition where smaller numbers are presented (e.g., 2). Altogether, these studies support the possibility that the size stored in memory (i.e., known size of objects or numbers) can directly influence the perception of size or of size-related features (e.g., distance) supporting the possible completion of perception by stored-ARPF when some of them are missing or ambiguous (see Barsalou, 2009, for a similar idea).

A further step is achieved by recent works demonstrating the influence of size stored in memory on more automatic perception-action links (rather than conscious judgments). Indeed, some studies have been able to show an influence of the known size of objects on action parameters that are dependent on the visual size. For instance, Hosking and Crassini (2010) have conducted experiments in which participants have to carry out time-tocontact judgments on stimuli for which a linear or a parabolic trajectory of approach are simulated. Such a judgment is highly important for a great variety of interceptive actions and is mainly based on the online processing of the visual size of the approaching stimulus. In their experiments, the stimuli used have different known sizes (i.e., large: a football vs. small: a tennis ball). Results elegantly demonstrated that this stored feature of objects influences time-to-contact judgments suggesting that it could interfere with our ability to intercept mobiles (see also DeLucia, 2005; Hosking and Crassini, 2011, for similar results). Another set of studies suggest also an influence of the known size of objects on another wellestablished perception-action link: Our ability to adapt our grip aperture according to the visual size of the to-be-grasped objects (Jeannerod, 1984). Several studies demonstrate that participants are faster to carry out a precision grip on typically small objects (e.g., cherry) and a power grip on typically large objects (i.e., eggplant; Ellis and Tucker, 2000; Tucker and Ellis, 2004; Derbyshire et al., 2006; Girardi et al., 2010) even when visual size cannot interfere (Glover et al., 2004; Tucker and Ellis, 2004; Heurley et al., in revision). The same effect on grip aperture is obtained when size-related adjectives are concomitantly processed (e.g., SMALL/LARGE, LONG/SHORT) rather than known objects (Gentilucci and Gangitano, 1998; Gentilucci et al., 2000; Glover and Dixon, 2002). These results are also replicated when numbers are used. More concretely, Moretto and di Pellegrino (2008) have shown that large number processing facilitate power grips while small number processing facilitate precision grips (see also Andres et al., 2004; Lindemann et al., 2007). In addition, some results support that such interactions are highly automatic (Moretto and di Pellegrino, 2008; Namdar et al., 2014) and seems to be restricted to the planning phase of grasping (Glover and Dixon, 2002; Glover et al., 2004; Badets et al., 2007; Andres et al., 2008). Taken together, these works demonstrate that stored ARPF, such as size, can influence automatic perception-action links and not only conscious perception supporting the possibility that perception can be completed by stored ARPF, itself influencing the planning of some action components.

This short review suggests that the ARPF size can be stored in memory, automatically retrieved during object perception, and can influence conscious perception of visual size (or relatedfeatures) as well as the planning of action components mainly based on visual size processing. We used this evidence to support the view that the interactions between present and absent–but simulated in memory–perceptual features are important for action especially in "out-of-laboratory conditions" in which ARPF can't be optimally perceived and in which interactions mainly occur with wellknown objects. Of course, the reported evidence is limited to the size, but several studies have already demonstrated that other ARPF such as distance, position, and weight could be stored and automatically retrieved (Estes et al., 2008; Scorolli et al., 2009; Winter and Bergen, 2012). This strongly suggests that our view can be extended. Even if many questions remain open and a lot of work has to be done to best support this view, it has the advantage to search for the functional relevance of memory-perception interactions (i.e., an embodied cognitive architecture) by taking into account two main constraints in which our cognitive systems have certainly evolved at phylogenetic and ontogenetic scales: Interactions with (i) well-known objects in (ii) more or less degraded-perceptual-conditions.

#### **ACKNOWLEDGMENT**

We are grateful to Gabrielle Chesnoy-Servanin for her revision of the English text.

### **REFERENCES**

memory level: manipulation of the memory size and not the perceptual size. *Exp. Psychol.* 61, 378–384. doi: 10.1027/1618-3169/a000258

knowledge. *J. Exp. Psychol. Learn. Mem. Cogn.* 13, 3–17. doi: 10.1037/0278-73<IP_ADDRESS>

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 08 November 2014; accepted: 15 December 2014; published online: 08 January 2015.*

*Citation: Heurley LP and Ferrier LP (2015) What are memory-perception interactions for? Implications for action. Front. Psychol. 5:1553. doi: 10.3389/fpsyg. 2014.01553*

*This article was submitted to Cognition, a section of the journal Frontiers in Psychology.*

*Copyright © 2015 Heurley and Ferrier. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# Manipulation gesture effect in visual and auditory presentations: the link between tools in perceptual and motor tasks

#### Amandine E. Rey \* † , Kévin Roche † , Rémy Versace and Hanna Chainay

Laboratoire d'Etude des Mécanismes Cognitifs, Université Lumière Lyon 2, Lyon, France

There is much behavioral and neurophysiological evidence in support of the idea that seeing a tool activates motor components of action related to the perceived object (e.g., grasping, use manipulation). However, the question remains as to whether the processing of the motor components associated with the tool is automatic or depends on the situation, including the task and the modality of tool presentation. The present study investigated whether the activation of motor components involved in tool use in response to the simple perception of a tool is influenced by the link between prime and target tools, as well as by the modality of presentation, in perceptual or motor tasks. To explore this issue, we manipulated the similarity of gesture involved in the use of the prime and target (identical, similar, different) with two tool presentation modalities of the presentation tool (visual or auditory) in perceptual and motor tasks. Across the experiments, we also manipulated the relevance of the prime (i.e., associated or not with the current task). The participants saw a first tool (or heard the sound it makes), which was immediately followed by a second tool on which they had to perform a perceptual task (i.e., indicate whether the second tool was identical to or different from the first tool) or a motor task (i.e., manipulate the second tool as if it were the first tool). In both tasks, the similarity between the gestures employed for the first and the second tool was manipulated (Identical, Similar or Different gestures). The results showed that responses were faster when the manipulation gestures for the two tools were identical or similar, but only in the motor task. This effect was observed irrespective of the modality of presentation of the first tool, i.e., visual or auditory. We suggest that the influence of manipulation gesture on response time depends on the relevance of the first tool in motor tasks. We discuss these motor activation results in terms of the relevance and demands of the tasks.

Keywords: embodied cognition, gesture, visual and auditory presentation, perceptual task, motor task, situated cognition

#### Edited by:

Roberta Sellaro, Leiden University, Netherlands

#### Reviewed by:

Filomena Anelli, University of Bologna & Fondazione Salvatore Maugeri, Italy Christine Sutter, RWTH Aachen University, Germany

#### \*Correspondence:

Amandine E. Rey, Laboratoire d'Etude des Mécanismes Cognitifs, EA 308 Université Lumière Lyon 2, 5 Avenue Pierre-Mendès France - 69676 Bron Cedex, Lyon, France

<EMAIL_ADDRESS>
† These authors have contributed equally to this work.

#### Specialty section:

This article was submitted to Cognition, a section of the journal Frontiers in Psychology

> Received: 23 January 2015 Accepted: 06 July 2015 Published: 22 July 2015

#### Citation:

Rey AE, Roche K, Versace R and Chainay H (2015) Manipulation gesture effect in visual and auditory presentations: the link between tools in perceptual and motor tasks. Front. Psychol. 6:1031. doi: 10.3389/fpsyg.2015.01031

## 1. Introduction

Grounded and embodied cognition theories claim that knowledge is assembled in order to prepare for action (Wilson, 2002) and is grounded in sensory-motor systems (Barsalou, 1999). The cognitive processes that underpin the use of knowledge are thought to be deeply rooted in physical action, with close links existing between perception, action and the environment (Glenberg, 1997; Clark, 2008). Consequently, it has been suggested that seeing an object does not involve only the processing of its different sensory properties but also the activation (or simulation) of motor components related to the object's typical action/use (Barsalou, 2008). The present paper focuses on the following question: Does the activation of motor components always result in an influence on the current task or does this influence depend on the relevance of motor activation for the current task? The study reported here focused more specifically on manipulation gestures that are typically associated with a tool (e.g., cutting for a knife, screwing for a screwdriver). We first explored whether the facilitation effect in perceptual and motor tasks depends on the relevance of the manipulation gesture activated by the prime. Second, because real-life experience is inherently multimodal and depends on our knowledge or our environment (Slotnick, 2004; Jääskeläinen et al., 2007; Versace et al., 2014), we compared the facilitation of manipulation gestures in response to visual (static tools in this study) and auditory (dynamic action-related) tool presentations in both perceptual and motor tasks.

Certain data reported by neurological and behavioral studies involving perceptual tasks has partially confirmed the idea that motor components can be automatically activated. It has been suggested that seeing objects typically activates actions that are associated with these objects irrespective of the task. Some neuroimaging studies have lent support to this argument by showing that neural motor areas are activated by a visual presentation of tools even when there is no intention of acting upon them (Chao and Martin, 2000; Vingerhoets, 2008). At the behavioral argument, Ellis and Tucker (2000) argued that visually presented objects activate motor components that are appropriate for grasping these objects. They showed that even if participants did not have to use the objects, the response times in categorization tasks were slower in congruent conditions (when the grip potentiated by an object was the same as that required by the ongoing task) than in incongruent conditions (see also Tucker and Ellis, 1998). Some studies have reported that actions with tools directly activate representations of their typical manipulation and have suggested that knowledge about manipulation gesture is involved in the selection of appropriate action plans (Creem-Regehr and Lee, 2005; Jax and Buxbaum, 2010; Ranganathan et al., 2011). According to these authors, specific memorized movements include action knowledge about manipulation and use that is automatically activated when a tool is seen (Buxbaum and Kalénine, 2010).

However, some studies have reported results that indicate non-automatic motor effects. For instance, the study conducted by McNair and Harris (2012) showed that seeing a tool automatically activates the grasp component rather than the manipulation component of motor activity in order to prepare for possible future use of the tool. They tested this assumption by comparing congruent vs. incongruent grasp and congruent vs. incongruent manipulation gestures between a prime and a target (both presented as pictures on a computer screen). The participants' task was to recall the name of the previously seen tool from a choice of many other tool names. The results showed that only grasp congruency enhanced participants' accuracy when identifying the previously seen tool. Furthermore, Pecher (2013) showed that a concurrent motor task did not interfere with the processing of the motor components of manipulable tools. This author asked participants to perform a perceptual task based on the perceptual or motor components of the stimulus, while also performing a concurrent motor task (i.e., various movements with their free hands). For instance, the participants performed visual tasks on manipulable and non-manipulable objects (e.g., they had to indicate whether a photograph of a tool was the same as or a mirror image of a preceding photograph) while performing a concurrent motor task. The author assumed that if the processing of manipulable tools is based partially on the activation of motor components, a concurrent motor task should interfere with processing. However, this study, like certain others that have been conducted, revealed no difference between the perceptual processing of manipulable and non-manipulable tools in a concurrent motor task paradigm (see also Pecher, 2013; Quak et al., 2014).

One possible explanation of motor activation effects (automatic or not) could lie in the intention to act (Massen and Prinz, 2009). Indeed, it has been suggested that relevant motor components are selected depending on the intention of the actor (Allport, 1987), which may be absent in perceptual tasks. Intention to act determines the nature of the information that is relevant for processing and this information can be processed irrespective of the target of the action (Craighero et al., 1998; Bekkering and Neggers, 2002; Lee et al., 2013; Roche and Chainay, 2014). One possibility is that activation of tool knowledge is not automatic but selectively modulated by the purpose of the action. If this is indeed the case, tool manipulation knowledge will not be activated in full and only those aspects relevant for the present situation will be activated. Contrary to sensorimotor theories, ideomotor theories have proposed that, rather than being automatic, tool knowledge depends on the intention (Prinz, 1997; Hommel, 2009). The intention to act in a given situation activates certain motor components that result from similar tool uses in the past and that are associated with the current environmental (Massen and Prinz, 2009). Thus, the goal of the action could be taken into account at a very early stage during the planning of a movement, i.e., when the relevant information is selected for the planning and execution of the action (van Elk et al., 2010). For example, the study by Lindemann et al. (2006) focused on how tool manipulation knowledge is involved in the preparation for an action. Their results suggest that tool manipulation knowledge is not activated automatically, but is only activated when the subject intends to grasp the object in a typical way instead of just making a finger-lifting movement. In a more recent study by Ranganathan et al. (2011), participants had to interact in three different ways with a glass placed either upright or upside-down; by grasping it, touching it with a clenched fist, or grasping it with a magnetic implement. Shorter initiation times were found in the case of simple grasping and grasping with the magnetic implement when the glass was placed upright as opposed to upside-down. This effect was not present when the participants touched the glass with their fist. These results, together with those obtained by Lindemann et al. (2006), suggest that an object does not activate motor components automatically but instead does so in light of the purpose of the action and the possibilities and intentions of the person performing the action. Consequently, if there is no intention to manipulate the tool, manipulation gesture components remain irrelevant to the task. However, it is unclear whether a tool-associated manipulation gesture component can be activated when it is irrelevant to the task as in a physical identity judgment task.

The second aim of the present study was to examine activation of the manipulation gesture processes as a function of the presentation modality of the tool (i.e., visual or auditory). In the traditional approach in which motor activation is considered to include manipulation gestures, visual information seems: (1) to be the preferred basis for the efficient execution of actions (Jacob and Jeannerod, 2005; Milner and Goodale, 2008) and (2) to rely on processes different from those involved in recognition and action knowledge (Milner and Goodale, 1995; Buxbaum and Kalénine, 2010). Accordingly, the visual processing of a tool would be sufficient in order to select the appropriate manipulation gesture for executing the action (Jax and Buxbaum, 2010). On the other hand, in a grounded cognition perspective, motor components are thought to be activated by another modality such as auditory information. Indeed, in everyday life, some actions are accompanied by a specific sound. Grounded theories suggest that the typical use of a tool is part of our knowledge about it (Gallese, 2005; Barsalou, 2008). This type of activation is consistent with the suggestion made by Gallese (2000) that tools differ from other objects because knowledge about tools includes one particular usage (Creem-Regehr and Lee, 2005). According to sensorimotor theories, knowledge about tools comes from sensorimotor traces that result from previous experience with tools. According to this view: (1) the auditory modality, just like any other sensory modality, could form the basis for the activation of motor components (see also Trumpp et al., 2014) and (2) motor and perceptual processes, including recognition and action knowledge, share common processes (see also Helbig et al., 2006, 2010; Sim et al., 2014).

In the present study, we tested the possible effect of motor component activation as a function of manipulation gesture in perceptual and motor tasks. In both tasks, a first tool was presented to the participants just before the presentation of a second tool on which they had to perform a perceptual and a motor task. In all the experiments, we manipulated the factor of Gesture Similarity between the first and second tool in three conditions. The gesture used for the two tools could be Identical (same tool, same gesture), Similar (different tools but similar gesture) or Different (different tools and gestures). We assumed that the activation of a similar gesture for the two tools would facilitate the subject response (i.e., faster response times or initiation times) if manipulation gesture is activated by the presentation of a tool. In addition, different results should be observed depending on the demands of the motor task. Indeed, if motor components are activated when they are relevant to the task, participants should only respond faster in the Similar than in the Different condition in the motor task since the activation of motor components is not relevant for the perceptual tasks. In addition to the type of the task (perceptual or motor), we also manipulated the presentation modality (visual or auditory) of the first object across the experiments. By using familiar tools associated with a well-known sound during utilization, we assumed that the auditory presentation of tools could activate manipulation gesture components in the same way as a visual presentation. More specifically, the first and second tools were presented visually in a perceptual (Experiment 1) and a motor task (Experiments 1 and 2), whereas the first tool was presented auditorily in Experiment 3 in perceptual and motor tasks.

## 2. Experiment 1

### 2.1. Method

### 2.1.1. Participants

Sixteen participants from the University of Lyon 2 took part in Experiment 1 (13 females, M = 20.06, SD = 2.05) after completing a written consent form. All of them reported themselves as right-handed and with normal or corrected-to-normal vision and audition. The study was approved by the local Ethics Committee.

### 2.1.2. Stimuli

The stimuli consisted of six manipulable objects which were presented as pictures in the perceptual tasks (in order remain consistent with the habitual perceptual paradigms, Labeye et al., 2008; Borghi et al., 2012; Pecher, 2013, for instance) and were physically in front of the participants for the motor tasks. As in the McNair and Harris (2012) study, they were subdivided into three pairs depending on the similarity of the gestures required for their use. The first and the second pairs corresponded to pistol/spraybottle and hammer/maracas (the maracas replaced the bell from the McNair and Harris study, because in the pre-test we found it to be too noisy when manipulated by the experimenter). The third pair—whistle/party blowout—was chosen on the same principle as the first and the second pair (see **Figure 1**).

The pictures were colored photographs of the six tools (2725 × 1187 pixels with a resolution of 300 × 300 dots per inch), taken from the same angle as that at which they were presented in the motor task. The photographs were presented at a distance of 65 cm from the participant's eyes.

### 2.1.3. Tasks Assignment and General Design

The participants were tested individually in each of the tasks. The assignment of participants to the order of presentation of the perceptual and motor tasks was counterbalanced across participants. Before starting the experiment, we made sure that the participants knew the tools and the gesture associated with their use: the participants were first asked to say the name of the tools. If they failed to say the correct name, they were asked

correspond to the pairs with simular utilization gestures.

to describe the context of use (this was particularly useful for the party blowouts and spray cleaner since they have unfamiliar names in French). They were then asked to grasp the tool and demonstrate how to use it. If the gesture they made was only approximate, we told them normally we use it like this, showed them the correct movement and asked them to replicate the gesture (for example, maracas are moved front-to-back and not left-to-right). After this preparatory phase, the participants were asked to start performing the task to which they had been assigned.

In all the experiments presented in this study, we manipulated the factor of Gesture Similarity between the first and second tool over three conditions: (1) Identical: the first tool was identical to the second; (2) Similar: the first tool belonged to the same pair of tools as the second; (3) Different: the first tool was different and did not belong to the same pair of tools as the second (the first tool was chosen pseudo-randomly across the four remaining tools).

### 2.1.4. Material and Procedure **Perceptual task**

### **Material**

The experiment was conducted on a Macintosh IMac. OpenSesame software was used to set up and control the experiment (Mathôt et al., 2012).

### **Procedure**

For the perceptual task, the participants were positioned facing the computer, with their right hand on the keyboard. Following the display of a fixation point, a first tool was presented to the participants for 1000 ms. After an Inter Stimulus Interval (ISI) of 500 ms, a second tool was presented. The second tool was displayed until the subject responded and was followed by an inter-trial interval of 1500 ms. The participants were asked to respond as quickly as possible by pressing the appropriate key on the keyboard (corresponding to the J and K keys, with the key assignment being counterbalanced across participants). After the phase of familiarization with the material, the perceptual task consisted of a physical identity judgment task (e.g., Vingerhoets et al., 2009). The participants had to indicate whether the tools were visually identical or different. Both tools were presented in one of the Gesture Similarity conditions (Identical, Similar, or Different). The first tool was always presented at 45◦ to the right (relative to the participant's midline), while the second tool was presented twice at 0◦ (aligned with the participant's midline) and twice at 90◦ , thus giving a total of 72 trials.

### **Motor task**

### **Material**

A Dell computer equipped with E-prime2 software (Psychology Software Tools, Inc., USA) was used to run the experiment and record initiation times. Liquid-crystal goggles (Plato Translucent Technologies, Toronto, ON, USA) were used to control the subjects vision and a home-made spherical trigger button of 4 cm diameter was connected to the computer and used to collect gesture initiation times. The tools were placed on a board measuring 40 cm by 50 cm.

### **Procedure**

The participants were positioned facing the experimental board, with their right hand on the button. In order to be consistent with the other tasks, the primes and targets, i.e., the first and second tools, respectively, were presented on the experimental board one at a time at a distance of approximately 45 cm from the participant's hand and with their graspable component facing the participant. To avoid the affordance of exactly the same grasp movement between the first and second tools, we always used different orientations for the two tools. The orientations were 0◦ (aligned with the participant's midline) or 90◦ for the second tool, and always 45◦ to the right for the first tool.

After 10 training trials in which all the conditions and tools were presented, each participant performed 72 trials which were identical to those used in the perceptual task (see **Figure 2**). The second tool was oriented at either 0◦ or 90◦ to create a variation in the grasp parameters and thus avoid repetitive grasp movements across trials. The trials were divided into 3 mini-blocks which were counterbalanced across participants.

All the trials started with a beep to remind the participant to place his/her hand on the release button. At the same time, the goggles became opaque for 1500 ms and a first tool was placed on the experimental board during this period. The goggles then became transparent for 500 ms so that the prime was visible, before turning opaque again for a further 1500 ms. During the ISI, the experimenter replaced the first tool on the experimental board with the second or, in the Identical condition, simply changed the orientation of the tool. At the end of the ISI, the goggles became transparent again and a simultaneous go signal indicated to the participant that he/she should grasp the second tool and show how to use it. The next trial then started with a beep. The participants were told to initiate the movement toward the tool as quickly as possible and simulate its use. They were given 3000 ms to do so.

To summarize, there were three differences between the tasks: the time interval between the stimuli (500 ms in the perceptual tasks and 1500 ms in the motor task), the presentation of the stimuli (pictures in the perceptual tasks and real tools in the motor task) and the nature of the task (comparison of two tools

in the perceptual tasks and execution of the utilization gesture in the motor task).

### 2.1.5. Statistical Analyses

We measured Reaction Times (RT) and error rates in the perceptual tasks and Initiation Time (IT) in the motor task (IT corresponded to the time that elapsed between the go signal and the time when the participants removed their hand from the release button). Errors in the motor task were not analyzed due to their very small number (i.e., only three subjects made one or two errors in the motor task of the first experiment). Reaction (or Initiation) times that were greater than 1500 ms or less than 250 ms and also differed by more than 2.5 standard deviations from the individual participants mean for each condition were removed (less than 3% of the data). Preliminary analyses were conducted to check for normality (Shapiro-Wilks test) and sphericity (Mauchleys test) and no violations were found. We used the mean correct RT for the analyses. Separate analyses of variance (ANOVA) were performed for RT and error rates in the Perceptual task and for IT in the Motor task, with subjects as random variableand Gesture Similarity (identical, similar, different) as within-subjects factor.

Given that we tested specific hypotheses, planned comparisons were performed. A significance level of a = 0.05 was used for all the statistical analyses. Means and standard errors of RT/IT for all the tasks and experiments are presented in **Table 1**.

For control purposes, we checked for a possible Tool Pair effect as well as for an interaction with the Gesture Similarity factor. We also checked for a possible Task Order effect and for an interaction between this and the Gesture Similarity factor. The data analyses were performed using STATISTICA (version 8.0, Stat-Soft, Inc.). The same analyses and controls were used for all the data presented in this article.

### 2.2. Results and Discussion 2.2.1. Perceptual Task

The analysis of RT revealed a significant effect of Gesture Similarity, F(2, 30) = 4.49, p = 0.023, η 2 p = 0.23. Planned comparisons showed that RT were faster in the Identical condition (M = 575 ms, SE = 38) than in either the Similar condition (M = 600 ms, SE = 41, p = 0.012) or the Different condition (M = 602 ms, SE = 40, p = 0.018), but no difference was observed between the Similar and

TABLE 1 | Means of reaction (initiation) times (in ms) for Experiments 1, 2, and 3 (between-subjects standard errors in parentheses).

Different conditions (p = 0.91). No simple effect of Task Order or Tool Pair was observed and there was no interaction between either Task Order or Tool Pair and Gesture Similarity (p > 0.1).

The analysis of error rates showed a significant effect of Gesture Similarity, F(2, 30) = 2.40, p = 0.029, η 2 p = 0.14. The participants were more accurate in the Identical condition (M = 1.95, SE = 0.56) than in the Different condition (M = 3.22, SE = 0.59, p = 0.029. No difference was observed between the Similar (M = 2.77, SE = 0.54) and either the Identical, p = 0.20, or Different conditions, p = 0.42.

### 2.2.2. Motor Task

No simple effect of Task Order or Tool Pairs was observed, and neither of these interacted with Gesture Similarity (p > 0.1). No significant effect of Gesture Similarity was observed, F(2, 30) = 0.49, p = 0.62. We did not observe a significant difference between Identical (M = 521 ms, SE = 35 ms), Similar (M = 528 ms, SE = 71 ms) and Different (M = 520 ms, SE = 67 ms) conditions.

In the perceptual task, the fact that the two tools required a similar gesture did not facilitate the subjects response (no difference between the Similar and Different conditions was observed). The processing of the first tool seemed irrelevant for the processing of the second tool even if use of the two tools shared a similar manipulation gesture.

Surprisingly, no effect at all was found in the motor task. A previous study using an identical protocol, but with a grasping task, found priming effects when the prime and the target were identical tools (Roche and Chainay, 2013). One way to explain this difference compared to the present motor task is to consider that the movement is planned and controlled as a function of its purpose and that this determines the different steps involved in the movement, including the grasp (Rosenbaum and Halloran, 2006; Ansuini et al., 2008). If this is indeed the case, then it is possible that a priming effect will be found in a grasping task, whereas no such effect will occur in a task in which a toolspecific gesture guides the entire movement, (e.g., only grasping a toothbrush vs grasping a toothbrush and performing the toothbrushing movement) (Massen and Prinz, 2009). Another possible explanation for this effect might be that the participants had learned that the first tool was irrelevant to the task and that they therefore ignored it. Indeed, Pfannmüller et al. (2012) have shown that visuomotor priming effects depend on the quality of prime processing and its memorization. It is possible that some of the processes involved in grasping are more likely to be activated automatically and are less intentional than those involved in the utilization gesture. As in the (Pfannmüller et al., 2012) study, we changed the protocol used for our motor task in Experiment 2 so that the first tool involved a preparation for a subsequent response. This ensured that the participants could not ignore the first tool as they could in Experiment 1. We asked the participants to grasp the second tool while performing the action corresponding to the first tool. This change of protocol increased the memorization and quality of prime processing (Pfannmüller et al., 2012). Thus, in this experiment, the intention to act was directed toward the second tool, whereas the planned gesture was determined in advance by the first tool.

### 3. Experiment 2

### 3.1. Method

### 3.1.1. Participants

Sixteen self-reported right-handed participants took part in this experiment (8 women, M = 23.25, SD = 5.65). None of them had participated in Experiment 1.

#### 3.1.2. Stimuli and Procedure

The same material and procedure as in the motor task in the previous experiment were used, except that we did not use a visuomotor protocol. In this experiment, the participants were told to grasp the second tool as quickly as possible while reproducing the action corresponding to the first tool, irrespective of the grasped tool.

#### 3.1.3. Statistical Analyses

The same cutoff as in Experiment 1 was used (which eliminated less than 1% of the data). The IT werepre-processed according to the same criteria as in Experiment 1. ANOVAs were performed on the IT with subjects as a random variable and Gesture Similarity as within-subjects factor. In addition, the interactions of Tool Pair and Task order with Gesture Similarity were tested for control purposes.

#### 3.2. Results and Discussion

The results showed a significant effect of Gesture Similarity, F(2, 30) = 11.92, p < 0.001, η 2 p = 0.44. IT were shorter for the Identical condition (M = 543 ms, SE = 23) than for the Similar (M = 565 ms, SE = 27, p = 0.026) and the Different conditions (M = 604 ms, SE = 29, p < 0.001). Moreover, IT were faster for the Similar condition than the Different condition (p = 0.011). The Tool Pair effect did not differ and did not interact significantly with Gesture Similarity (p > 0.1).

To gain a better understanding of the relevance of the first tool for the task, which could explain the different patterns of results in the motor tasks in Experiments 1 and 2, we performed an ANOVA with the relevance of the first tool (relevant/Experiment 1 vs. irrelevant/Experiment 2) as group factor and Gesture Similarity as repeated-measure factor. The analysis revealed a significant effect of Gesture Similarity [F(2, 60) = 7.67, p = 0.002, η 2 p = 0.20] and, more interestingly, showed that the interaction between Experiments and Gesture Similarity [F(2, 60) = 8.55, p < 0.001, η 2 p = 0.22) was significant (see **Figure 3**). Planned comparisons are reported separately in the results section of each experiment.

In Experiment 2, we found an effect of Gesture Similarity. First of all, the results showed that movement IT were faster when the first and second tools were Identical rather than Similar or Different. We interpret this finding in terms of a facilitatory effect which enables subjects to plan their action in advance on the basis of the first tool presented just before manipulating the same (Identical) tool. Secondly, we found shorter IT in the Similar than in the Different condition. In both conditions, although the tools changed between the first and second presentation, their similarity in terms of motor manipulation nevertheless facilitated the initiation of the movement. In addition, and unlike in the motor task in Experiment 1, increasing the memorization and quality of the processing of the first tool enabled us to obtain an effect of Gesture Similarity. It seems possible that, unlike in a perceptual or grasping task, a more complex action such as demonstrating the actual utilization of a tool demands more situated processing. More generally, the results of Experiment 2 suggest that it is the intention to act that determines the processing of motor components (Allport, 1987) in the light of the overall goal of the action (Massen and Prinz, 2009).