# LANGUAGE AND COGNITION

EDITED BY: Kuniyoshi L. Sakai and Leonid Perlovsky PUBLISHED IN: Frontiers in Behavioral Neuroscience

#### *Frontiers Copyright Statement*

*© Copyright 2007-2015 Frontiers Media SA. All rights reserved. All content included on this site, such as text, graphics, logos, button icons, images, video/audio clips, downloads, data compilations and software, is the property of or is licensed to Frontiers Media SA ("Frontiers") or its licensees and/or subcontractors. The copyright in the text of individual articles is the property of their respective authors, subject to a license granted to Frontiers.*

*The compilation of articles constituting this e-book, wherever published, as well as the compilation of all other content on this site, is the exclusive property of Frontiers. For the conditions for downloading and copying of e-books from Frontiers' website, please see the Terms for Website Use. If purchasing Frontiers e-books from other websites or sources, the conditions of the website concerned apply.*

*Images and graphics not forming part of user-contributed materials may not be downloaded or copied without permission.*

*Individual articles may be downloaded and reproduced in accordance with the principles of the CC-BY licence subject to any copyright or other notices. They may not be re-sold as an e-book.*

*As author or other contributor you grant a CC-BY licence to others to reproduce your articles, including any graphics and third-party materials supplied by you, in accordance with the Conditions for Website Use and subject to any copyright notices which you include in connection with your articles and materials.*

*All copyright, and all rights therein, are protected by national and international copyright laws. The above represents a summary only. For the full conditions see the Conditions for Authors and the Conditions for Website Use.*

ISSN 1664-8714 ISBN 978-2-88919-627-2 DOI 10.3389/978-2-88919-627-2

#### About Frontiers

Frontiers is more than just an open-access publisher of scholarly articles: it is a pioneering approach to the world of academia, radically improving the way scholarly research is managed. The grand vision of Frontiers is a world where all people have an equal opportunity to seek, share and generate knowledge. Frontiers provides immediate and permanent online open access to all its publications, but this alone is not enough to realize our grand goals.

#### Frontiers Journal Series

The Frontiers Journal Series is a multi-tier and interdisciplinary set of open-access, online journals, promising a paradigm shift from the current review, selection and dissemination processes in academic publishing. All Frontiers journals are driven by researchers for researchers; therefore, they constitute a service to the scholarly community. At the same time, the Frontiers Journal Series operates on a revolutionary invention, the tiered publishing system, initially addressing specific communities of scholars, and gradually climbing up to broader public understanding, thus serving the interests of the lay society, too.

#### Dedication to Quality

Each Frontiers article is a landmark of the highest quality, thanks to genuinely collaborative interactions between authors and review editors, who include some of the world's best academicians. Research must be certified by peers before entering a stream of knowledge that may eventually reach the public - and shape society; therefore, Frontiers only applies the most rigorous and unbiased reviews.

Frontiers revolutionizes research publishing by freely delivering the most outstanding research, evaluated with no bias from both the academic and social point of view. By applying the most advanced information technologies, Frontiers is catapulting scholarly publishing into a new generation.

#### What are Frontiers Research Topics?

Frontiers Research Topics are very popular trademarks of the Frontiers Journals Series: they are collections of at least ten articles, all centered on a particular subject. With their unique mix of varied contributions from Original Research to Review Articles, Frontiers Research Topics unify the most influential researchers, the latest key findings and historical advances in a hot research area! Find out more on how to host your own Frontiers Research Topic or contribute to one as an author by contacting the Frontiers Editorial Office<EMAIL_ADDRESS>
### **LANGUAGE AND COGNITION**

Topic Editors: **Kuniyoshi L. Sakai,** The University of Tokyo, Japan **Leonid Perlovsky,** Harvard University and Air Force Research Laboratory, USA

Functional evidence of a syntax-related network.

In a given sentence with syntactic structures, the greater depth of merged subtrees, but not the linear order of words, elicited significant activation in the pars opercularis and pars triangularis of the left inferior frontal gyrus (L. F3op/F3t), as well as the left supramarginal gyrus (L.SMG). Activations (red) are projected onto the left (L) and right lateral surfaces of a standard brain. Taken from Ohta, Fukui, and Sakai (2013; doi: 10.3389/fnbeh.2013.00204).

Interaction between language and cognition remains an unsolved scientific problem. What are the differences in neural mechanisms of language and cognition? Why do children acquire language by the age of six, while taking a lifetime to acquire cognition? What is the role of language and cognition in thinking? Is abstract cognition possible without language? Is language just a communication device, or is it fundamental in developing thoughts? Why are there no animals with human thinking but without human language? Combinations even among 100 words and 100 objects (multiple words can represent multiple objects) exceed the number of all the particles in the Universe, and it seems that no amount of experience would suffice to learn these associations.

Since the 19th century we know about involvement of Broca's and Wernicke's areas in language. What new knowledge of language and cognition areas has been found with

fMRI and other brain imaging methods? Every year we know more about their anatomical and functional/effective connectivity. What can be inferred about mechanisms of their interaction, and about their functions in language and cognition? Why does the human brain show hemispheric (i.e., left or right) dominance for some specific linguistic and cognitive processes? Is understanding of language and cognition processed in the same brain area, or are there differences in language-semantic and cognitive-semantic brain areas? Is the syntactic process related to the structure of our conceptual world?

Chomsky has suggested that language is separable from cognition. On the opposite, cognitive and construction linguistics emphasized a single mechanism of both. Neither has led to a computational theory so far. Evolutionary linguistics has emphasized evolution leading to a mechanism of language acquisition, yet proposed approaches also lead to incomputable complexity.

There are some more related issues in linguistics and language education as well. Which brain regions govern phonology, lexicon, semantics, and syntax systems, as well as their acquisitions? What are the differences in acquisition of the first and second languages? Which mechanisms of cognition are involved in reading and writing? Are different writing systems affect relations between language and cognition? Are there differences in languagecognition interactions among different language groups (such as Indo-European, Chinese, Japanese, Semitic) and types (different degrees of analytic-isolating, synthetic-inflected, fused, agglutinative features)? What can be learned from sign languages?

Rizzolatti and Arbib have proposed that language evolved on top of earlier mirror-neuron mechanism. Can this proposal answer the unknown questions about language and cognition? Can it explain mechanisms of language-cognition interaction? How does it relate to known brain areas and their interactions identified in brain imaging?

Emotional and conceptual contents of voice sounds in animals are fused. Evolution of human language has demanded splitting of emotional and conceptual contents and mechanisms, although language prosody still carries emotional content. Is it a dying-off remnant, or is it fundamental for interaction between language and cognition? If language and cognitive mechanisms differ, unifying these two contents requires motivation, hence emotions. What are these emotions? Can they be measured? Tonal languages use pitch contours for semantic contents, are there differences in language-cognition interaction among tonal and atonal languages? Are emotional differences among cultures exclusively cultural, or also depend on languages?

Interaction of language and cognition is thus full of mysteries, and we encouraged papers addressing any aspect of this topic.

**Citation:** Sakai, K. L., Perlovsky, L., eds. (2015). Language and Cognition. Lausanne: Frontiers Media. doi: 10.3389/978-2-88919-627-2

## Table of Contents

Carmelo M. Vicario and Raffaella I. Rumiati

*121 Why open-access publication should be nonprofit—a view from the field of theoretical language science*

Martin Haspelmath

*124 The importance of Open Access publishing in the field of Linguistics for spreading scholarly knowledge and preserving languages diversity in the era of the economic financial crisis*

Nicola L. Bragazzi

**EDITORIAL** published: 16 December 2014 doi: 10.3389/fnbeh.2014.00436

### Language and Cognition

#### *Leonid Perlovsky1 \* and Kuniyoshi L. Sakai 2 \**

*1 Department of Electrical Engineering, School of Engineering and Applied Sciences, Harvard University, Cambridge, MA, USA 2 Department of Basic Science, Graduate School of Arts and Sciences, The University of Tokyo, Tokyo, Japan*

*\*Correspondence<EMAIL_ADDRESS><EMAIL_ADDRESS>
#### *Edited and reviewed by:*

*Nuno Sousa, University of Minho, Portugal*

**Keywords: language, cognition, brain, functional imaging, emotion**

Interaction between language and cognition remains an unsolved scientific problem. What are the differences in neural mechanisms of language and cognition? Why do children acquire language by the age of six, while taking a lifetime to acquire cognition? What is the role of language and cognition in thinking? Is abstract cognition possible without language? Is language just a communication device, or is it fundamental in developing thoughts? Why are there no animals with human thinking but without human language? Combinations even among 100 words and 100 objects (multiple words can represent multiple objects) exceed the number of all the particles in the Universe, and it seems that no amount of experience would suffice to learn these associations.

Since the nineteenth century we know about involvement of Broca's and Wernicke's areas in language. What new knowledge about the brain regions responsible for language and cognition has been found with fMRI and other brain imaging methods? Every year we know more about their anatomical and functional/effective connectivity. What can be inferred about their interactions and functions in language and cognition? Why does the human brain show hemispheric (i.e., left or right) dominance for some specific linguistic and cognitive processes? Is linguistic and cognitive comprehension processed in the same or different regions? Do the syntactic processes affect the structure of our conceptual world?

Such issues regarding brain functions and mind have been increasingly drawing attention from various fields in recent years, and investigations that go beyond the boundaries of previous fields of study are becoming necessary. The need for study spanning the brain and the mind has given birth to a new discipline, such as cognitive neuroscience, neurolinguistics, biolinguistics, etc. We assume that mind is a part of brain function, and we tentatively define the mind as a combination of three main cognitive factors: perception, memory, and consciousness. Language is created by mind, yet, once uttered, words return to the mind, where they are understood. The cycle from the mind to the language and then from the language to the mind, is *recursive*, in that the language produced by the mind comes back to the mind once again. This recursiveness is important when considering the relationship between language and mind.

When viewed language and mind as a whole system, it is evident that the functions of language are part of the brain system at the same time as being involved in the workings of the mind. Moreover, information is exchanged between language and each of perception, memory, and consciousness in both directions. Namely, language is involved in both reciprocal and recursive information exchange with each element of the mind. Since language is tightly linked to the mind, it would be more natural to assume that language is a part of the mind than to think it is an entity which exits outside the mind. The study of language is, in essence, to understand a part of the "human" mind. The more we study the language used by humans, the more we will understand the structure of the mind.

Chomsky has suggested that language is separable from cognition (Berwick et al., 2013), and this notion has been well supported by functional imaging experiments in neuroscience (Sakai, 2005). On the opposite, cognitive and construction linguistics emphasized a single mechanism of both. Neither has led to a computational theory so far, but language is learned early in life with only limited cognitive understanding of the world (Perlovsky, 2009). Evolutionary linguistics has emphasized evolution leading to a mechanism of language acquisition, yet proposed approaches also lead to incomputable complexity. Papers in this volume report new knowledge on interacting language and cognition, still there remains more questions than answers.

In animals, emotional and conceptual contents of voice sounds are fused. Evolution of human language has demanded splitting of emotional and conceptual contents, as well as of their mechanisms, although language prosody still carries emotional content. Is it a dying-off remnant, or is it fundamental for interaction between language and cognition? If language and cognitive mechanisms differ, unifying these two contents requires motivation, hence emotions. What are these emotions? Can they be measured? If tonal languages use pitch contours for semantic contents, are there differences in language-cognition interaction among tonal and atonal languages? Are emotional differences among cultures exclusively cultural, or also depend on languages?

This volume introduces a broad range of research addressing these topics, including three opinion articles, one hypothesis and theory article, eight original research articles, and a pair of an opinion article and a general commentary article. Their summaries are as follows.

First, Perlovsky (2013) introduces joint acquisition, dual hierarchy, and emotional prosody of language and cognition, such that emotional prosody may perform a fundamental function in connecting sounds and meanings of words. Vicario (2013) discusses about FOXP2 gene and language development, which might inform us about the origin of language. Perry and Lupyan (2013) explain that language and thought are different but strongly interacting abilities, based on the online manipulation of linguistic activity.

Next, Ohta et al. (2013) propose computational principles of syntax in the regions specialized for language, thereby integrating theoretical linguistics and functional neuroimaging. Nagels et al. (2013b) present an fMRI study on the neural substrates of figurative language during natural speech perception. De La Cruz et al. (2013) show that finger counting helps cognitive robots to learn words. Straube et al. (2013) suggest that abstract information conveyed by speech and gesture may be processed independent of modality. Tilles and Fontanari (2013) examine reinforcement and inference in cross-situational word learning. Nagels et al. (2013a) indicate the role of semantic abstractness and perceptual category in processing speech accompanied by gestures. Zhong et al. (2013) study a self-organizing pre-symbolic neural model representing sensorimotor information. Shuai and Gong (2013) analyze temporal relationships between top-down and bottom-up processing in lexical tone perception. Vicario and Rumiati (2013) demonstrate how notions of left and right affect processing of trading verbs.

We end the volume with a highly-popular discussion on the role of open access publications in linguistics, contributed by Haspelmath (2013) and Bragazzi (2013).

#### **REFERENCES**

Nagels, A., Kauschke, C., Schrauf, J., Whitney, C., Straube, B., and Kircher, T. (2013b). Neural substrates of figurative language during natural speech perception: an fMRI study. *Front. Behav. Neurosci.* 7:121. doi: 10.3389/fnbeh.2013.00121

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 13 November 2014; accepted: 01 December 2014; published online: 16 December 2014.*

*Citation: Perlovsky L and Sakai KL (2014) Language and Cognition. Front. Behav. Neurosci. 8:436. doi: 10.3389/fnbeh.2014.00436*

*This article was submitted to the journal Frontiers in Behavioral Neuroscience.*

*Copyright © 2014 Perlovsky and Sakai. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

### Language and cognition—joint acquisition, dual hierarchy, and emotional prosody

#### *Leonid Perlovsky\**

*The AFRL and Athinoula A. Martinos Center for Biomedical Imaging, Harvard University, Charlestown, MA, USA \*Correspondence<EMAIL_ADDRESS>
#### *Edited by:*

*Kuniyoshi L. Sakai, The University of Tokyo, Japan*

**Keywords: language, cognition, acquisition, dual hierarchy, prosody, emotion**

#### **FUNCTION OF LANGUAGE AND COGNITION IN THINKING**

Do we think with language, or is it just a communication device used for expression of completed thoughts? What is a difference between language and cognition? Chomsky (1995) suggested that these two abilities are separate and independent. Cognitive linguistics emphasizes a single mechanism for both (Croft and Cruse, 2004). Evolutionary linguistics considers the process of transferring language from one generation to the next one (Cangelosi and Parisi, 2002; Christiansen and Kirby, 2003; Hurford, 2008). This process is a "bottleneck" that forms the language. Brighton et al. (2005) demonstrated emergence of compositional language due to this bottleneck. Still, none of these approaches resulted in a computational theory explaining how humans acquire language and cognition. Here I discuss a computational model overcoming previous difficulties and based on a hypothesis that language and cognition are two separate and closely integrated abilities. I identify their functions and discuss why human thinking ability requires both language and cognition.

Among fundamental mechanisms of cognition are mental representations, memories of objects and events (Perlovsky, 2001, 2006a). The surrounding world is understood by matching mental representations to patterns in sensor signals. However, mathematical modeling of this process since the 1950s met with difficulties. The first difficulty is related to a need to consider combinations of sensor signals, objects, and events. The number of combinations is very large and even a limited number of signals or objects form a very large number of combinations, exceeding all interactions of all elementary particles in a lifetime of the Universe (Perlovsky, 1998). This is known as combinatorial complexity, CC. This difficulty in modeling the mind has been overcome by dynamic logic (Perlovsky, 2001, 2006a,b, 2007a; Perlovsky et al., 2011). Whereas classical logic considers static statements such as "this is a chair," dynamic logic models processes from vague to crisp representations. These processes do not need to consider combinations, an initial vague state of a "chair" matches any object in the field of view, and at the end of the process it matches the chair actually present, without CC.

The second difficulty is similar still even more complex. It is related to the fact that "events" and "situations" in the world do not necessarily exist "ready for cognition." There are many combinations of percepts and objects, a near infinity, events and situations important for understanding and learning have to be separated from those that are just random collections of meaningless percepts or random objects (Perlovsky and Ilin, 2012). Events and situations recognized by non-human animals are very limited compared to human abilities to differentiate events in the world. Human cognitive abilities acquire their power due to language. Language is "easier" to learn than cognitive representations. Language representations: words, phrases exist in the surrounding language "ready made," created during millennia of cultural evolution. Therefore, language could be learned without much real-life experience; only interactions with language speakers are required. Every child learns language early in life before acquiring full cognitive understanding of events and their cognitive meanings. Thus, language is learned early in life with only limited cognitive understanding of the world (Perlovsky, 2009a, 2012c). Cognitive representations of situations and abstract concepts initially exist in vague states. Throughout the rest of life, language guides acquisition of cognitive representations from experience. Vague cognitive representations become more crisp and concrete. Thinking involves both language and cognition, and as we discuss later thinking about abstract ideas usually involves language more than cognition, not too different from thinking by children.

#### **THE DUAL HIERARCHY**

Cognitive representations are organized in mind in an approximate hierarchy (Grossberg, 1988) from sensor-motor percepts near "bottom," to objects "higher up," to situations, and to still more abstract cognitive representations. Language representations are organized in a parallel hierarchy from sounds, and words for objects and situations, to phrases, and to more abstract language representations. Our previous discussion can be described by an integrated mathematical model of language and cognition forming a dual hierarchy (Perlovsky, 2009a), as illustrated in **Figure 1**. Neural evidence suggests that the hierarchy is approximate, not as definite as shown in this figure.

Hierarchical organization of cognition and related brain structures are reviewed in (Badre, 2008). In particular, anterior-posterior axis corresponds to a gradient of abstract-concrete cortex functions. Hierarchical organization of language functions is also well established. However, hierarchical organization of language does not correspond to a particular spatial axis in the brain, it is distributed (Price, 2012). Therefore, the dual hierarchy in **Figure 1** is a functional hierarchy not organized along a spatial axis in the brain as in this figure. A fundamental aspect of acquiring

mental representations is interaction between higher and lower layer representations (top and bottom layers). In this interaction a lower layer representations are organized in more abstract and general concept-representations at a higher layer. These interactions are referred to as bottom-up and top-down signals (BU and TD) indicated in **Figure 1** by vertical arrows.

Mathematical model of the dual hierarchy is described in Perlovsky (2009a, 2012c) and Perlovsky and Ilin (2010, 2012). This model explains many facts about thinking, language, and cognition, which has remained unexplainable and would be considered mysteries, if not so commonplace.

The dual model makes a number of experimentally testable predictions. (1) It explains functions of language and cognition in thinking: cognitive representations model surrounding world, relations between objects, events, and abstract concepts. Language stores culturally accumulated knowledge about the world, yet language is not directly connected to objects, events, and situations in the world. Language guides acquisition of cognitive representations from random percepts and experiences, according to what is considered worth learning and understanding in culture. Events that are not described in language are likely not even noticed or perceived in cognition. (2) Whereas language is acquired early in life, acquiring cognition takes a lifetime. The reason is that language representations exist in surrounding language "ready-made," acquisition of language requires only interaction with language speakers, but does not require much life experience. Cognition on the opposite requires life experience. (3) This is the reason why abstract words excite only language regions of brain, whereas concrete words excite also cognitive regions (Binder et al., 2005). The dual model predicts that abstract concepts are often understood as word descriptions, but not in terms of objects, events, and relations among them. (4) This model explains why language is acquired early in life, whereas cognition takes a lifetime. It also explains why children can acquire the entire hierarchy of language including abstract words without experience necessary for understanding them. (5) Since dynamic logic is the basic mechanism for learning language and cognitive representations, the dual model suggests that language representations become crisp after language is learned (5–7 years of age), however, cognitive representations may remain vague for much longer; the vagueness is exactly the meaning of "continuing learning," this takes longer for more abstract and less used concepts. (6) The dual model gives mathematical description of the recursion mechanism (Perlovsky and Ilin, 2012). Whereas Hauser et al. (2002) postulate that recursion is a fundamental mechanism in cognition and language, the dual model suggests that recursion is not fundamental, hierarchy is a mechanism of recursion.

(7) Another mystery of humancognition, not addressed by cognitive or language theories, is basic human irrationality. This has been widely discussed and experimentally demonstrated following discoveries of Tversky and Kahneman (1974), leading to the 2002 Nobel Prize. According to the dual hierarchy model, the "irrationality" originates from the dichotomy between cognition and language. Language is crisp and conscious while cognition might be vague and ignored when making decisions. Yet, collective wisdom accumulated in language may not be properly adapted to one's personal circumstances, and therefore be irrational in a concrete situation. In the 12th century Maimonides wrote that Adam was expelled from paradise because he refused original thinking using his own cognitive models, but ate from the tree of knowledge and acquired collective wisdom of language (Levine and Perlovsky, 2008).

#### **EMOTIONAL PROSODY AND ITS COGNITIVE FUNCTION**

The dual model implies connections between language and cognitive representations, indicated by a wide horizontal arrow in **Figure 1**. These neural connections have to be developed and maintained. This requires motivation, in other words, emotions. These emotions must be in addition to utilitarian meanings of words, otherwise only practically useful words would be connected to their cognitive meanings. Also these emotions must "flow" from language to cognition, so that language is able to perform its cognitive function of guiding acquisition of cognitive representations, organizing experience according to cultural contents of language. These emotions therefore must be contained in language sounds, before cognitive contents are acquired.

This requirement of emotionality of language sounds is surprising and contradictory to assumed direction of evolution of language. Evolution of the language ability required rewiring of human brain in the direction of freeing vocalization from uncontrollable emotions (Deacon, 1997; Perlovsky, 2009b). Yet, the dual model requires that language sounds be emotional. Emotionality of human voice is most pronounced in songs (Perlovsky, 2010, 2012a,d, 2013b). Emotions of everyday speech are low, unless affectivity is specifically intended. We may not notice emotions in everyday "non-affective" speech. Nevertheless, this emotionality is important for developing the cognitive part of the dual model. If language is highly emotional, speakers are passionate about what they say, however, evolving new meanings might be slow, emotional ties of sounds to old meanings might be "too strong." If language is low-emotional, new words are easy to create, however, motivation to develop the cognitive part of the dual model might be low, the real-world meaning of language sound might be lost. Cultural values might be lost as well. Indeed languages differ in how strong are emotional connections between sounds and meanings. This leads to cultural differences. Thus, the dual model leads to Emotional Sapir-Whorf Hypothesis (Perlovsky, 2007b, 2009b, 2012b). Strength of emotional connections between sound and meaning depends on language inflections. In particular, after English lost most of its inflections, it became a low emotional language, powerful for science and engineering. At the same time English is losing autonomous connections to cultural values that used to be partially inherent in language sounds. Fast change of cultural values during recent past is usually attributed to progress in thinking, whereas effects of change in emotionality of language sounds have not been noticed.

Emotional prosody can be important for overcoming cognitive dissonance. Cognitive dissonance is a discomfort due to holding contradictory cognitions (Festinger, 1957; Harmon-Jones et al., 2009). It is resolved by discarding contradictions. If a new word contradicts existing knowledge its meaning might be discarded. Emotional prosody as well as songs could be fundamental mechanisms that overcome cognitive dissonance and enable keeping new contradictory knowledge (Masataka and Perlovsky, 2012; Perlovsky, 2013a).

#### **CONCLUSION AND EXPERIMENTAL PREDICTIONS**

This article advances a hypothesis about functions of language and cognition in thinking, and possible model of their interactions. This is the only computable model explaining a number of mysteries about language and cognition and overcoming computational difficulties. It makes a number of predictions that could be experimentally tested, including the following: cognitive representations model the world, while language representations only model language; abstract cognitive representations can only be acquired due to language; abstract cognition is more clearly represented in language whereas cognitive representations may remain vague throughout life.

#### **ACKNOWLEDGMENTS**

I am thankful for discussions with my colleagues, Michel Cabanac and Nobuo Masataka.

#### **REFERENCES**

Badre, D. (2008). Cognitive control, hierarchy, and the rostro–caudal organization of the frontal lobes. *Trends Cogn. Sci.* 12, 193–200. doi: 10.1016/j.tics.2008.02.004

(Heidelberg: Springer Verlag), 73–108. doi: 10.1007/978-3-540-73267-9\_5

*Received: 26 August 2013; accepted: 02 September 2013; published online: 19 September 2013.*

*Citation: Perlovsky L (2013) Language and cognition joint acquisition, dual hierarchy, and emotional prosody. Front. Behav. Neurosci. 7:123. doi: 10.3389/ fnbeh.2013.00123*

*This article was submitted to the journal Frontiers in Behavioral Neuroscience.*

*Copyright © 2013 Perlovsky. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

### *FOXP2* gene and language development: the molecular substrate of the gestural-origin theory of speech?

#### *Carmelo M. Vicario\**

*School of Psychology, The University of Queensland, Brisbane, QLD, Australia \*Correspondence<EMAIL_ADDRESS>
*Edited by:*

*Kuniyoshi L. Sakai, The University of Tokyo, Japan*

The view that language evolved from a primarily gestural mode of communication has its roots on the 18th-century philosophers speculations (Vico, 1953/1744; de Condillac, 1971/1756).

Over time, these philosophical thoughts have gradually got consistence, thanks to the research in the field of Psychology and Neuroscience, which has provided exciting evidence in support of the so-called gestural origin-theory of language (Corballis, 2002). This theory recognizes to the gestures a precise role in language development. In particular, in terms of evolution it has been suggested that spoken language evolves from an ancient communication system using arm gestures. Accordingly, it has been suggested that gestures of the mouth might have been added to the manual system to form a combined manuofacial gestural system (Corballis, 2002; Gentilucci and Corbalis, 2006).

The literature on Mirror Neurons has provided a strong support to the gesturalorigin theory thanks to the evidence of a close relationship between arm (Gallese et al., 1996; Rizzolatti et al., 1996) and mouth actions (Ferrari et al., 2003) in the brain of non-human primates. In particular, it has been shown that the area F5 of the monkey premotor cortex includes also a class of neurons that discharge when the animal grasps an object with the either the hand or the mouth (Rizzolatti et al., 1988). However, although a recent meta-analysis of 125 human fMRI studies (Molenberghs et al., 2012) identified a core network of human brain regions that possess mirror properties associated with action observation and execution, there is also a literature that challenges the existence of these neurons in humans. In fact, direct evidence for the existence of mirror neurons in humans is still lacking (Dinstein et al., 2008; Hickok, 2009). On the other hand,

some studies provide data against it. For example, the study of Lingnau et al. (2009) failed in finding adaptation for motor acts that were first executed and then observed (as found for executed motor acts, when these were preceded by execution or observation of the same motor act) in the brain areas that are typically considered as endowed of mirror properties. This implies that the link between motor gestures and spoken language is not necessarily mediated by neurons provided of "mirror" properties.

Clues in support of the gestural-origins theory of the language are also provided by the research on humans. For example, grasping larger objects (Gentilucci et al., 2001) and bringing them to the mouth (Gentilucci et al., 2004) induces selective increases in parameters of lip kinematics and voice spectra of syllables pronounced simultaneously with action execution. Moreover, it has been reported that repetitive transcranial magnetic stimulation of Broca's area affects verbal responses to gesture observation (Gentilucci et al., 2006). This suggests that Broca's area is probably involved in the simultaneous control of gestures and word pronunciation.

Neuroimaging studies on humans provide a further support to this direct link between gestures and verbal language. Sakai et al. (2005) used functional Magnetic Resonance Imaging to examine hemispheric dominance during the processing of signed and spoken sentences. Their study was provided of two conditions: (i) the sign condition with sentence stimuli in Japanese Sign Language (JSL) in which were tested Deaf signers of JSL, hearing bilinguals (children of Deaf adults, CODA) of JSL and Japanese (JPN); (ii) the speech condition in which were tested hearing monolinguals (Mono) of JPN with auditory JPN stimuli alone (AUD), or with an audiovisual presentation of JPN and JSL stimuli (A and V). The authors found that the ventral part of the left inferior frontal gyrus (F3t/F3O) showed no main effects of modality condition, providing evidence in support of the existence on a common area for the processing of linguistic information from both signed and spoken sentences. Moreover, it has been recently documented the common involvement of the left area 7A in the superior parietal lobule while performing a sequenced button presses task or a sequence of different syllables repetition task (Heim et al., 2012). These data demonstrate the existence of a common cortical module in the area 7A while sequencing vocal gestures and hand motor actions.

Finally, a support to the gestural-origin theory of language originates from the study of human infants. For example, Fogel and Hannan (1985) provided evidence of gesture-vocalization synchrony in 2-and 3-months-old human infants. Word comprehension in children between 8 and 10 months and word productions between 11 and 13 months are typically accompanied by deictic gestures (Volterra et al., 1979; Bates and Snyder, 1987). Deictic gestures (referring to an object or location) are particularly important since they allow reference to grow from the immediate context toward abstraction by helping infants understand the link between symbols and referents (De Villiers Rader and Zukow-Goldring, 2010). Moreover, deictic gestures seem able to predict linguistic development in both typical and atypical human populations across many cultures (Iverson and Goldin-Meadow, 2005). All these studies suggest that gestures provide a foundation for each new stage in early linguistic development.

A recent discovery in the field of genetics seems providing new insights in support of the gestural-origin theory. In particular, evidence suggests that the *FOXP2* gene, located on the human chromosome 7 (Fisher et al., 1998), could be the molecular substrate linking speech with gesture. In fact, this gene is involved not only in speech production and comprehension but also in gesture coordination.

In an early work Gopnik (1990) argued that the *FOXP2* gene is involved in the development of morphosyntax. For this motivation this gene has been identified more broadly as the "grammar gene" (Pinker, 1994). However, a subsequent investigation suggested that the core deficit associated with the abnormal expression of this gene is one of articulation, with grammatical impairment as secondary outcome (Watkins et al., 2002). Thus, it was proposed (Corballis, 2004) that this gene may play a role in the incorporation of vocal articulation, but have little to do with grammar itself. In support of this suggestion it has been reported that *FOXP2* shows overlapping expression patterns within brains of zebra finches and fetal human brains, particularly in subcortical regions that play important roles in sensorimotor integration and coordinated movements important for vocalization and speech (Teramitsu et al., 2004). Moreover, recent studies on humans extend the role of *FOXP2* gene to the coordination of upper limb movements. For example, the recent study of Peter et al. (2011) found an influence of the *FOXP2* gene on several language processing tasks such as nonword repetition, real word reading efficiency, rapid oral reading. Interestingly, they documented an effect of this gene also on rapid motor sequencing ability which also included finger movements.

Another recent work (Wilcke et al., 2012) has shown that the Single Nucleus Polymorphism (rs12533005) of the *FOXP2* gene can be associated with congenital dyslexia. Interestingly, the difficulty with sequential finger movements is another type of deficit which may characterize reading disorders (Tiffin-Richards et al., 2004). Furthermore, *FOXP2* mutations seem to account for the childhood apraxia of speech (CAS) (MacDermot et al., 2005; Laffin et al., 2012), which is characterized by problems in saying sounds, syllables, and words. Peter (2012) recently described the CAS *FOXP2* phenotype in multi-generational families as characterized not only by deficits in sequential processing at the level of alternating oral motor movements, which is consistent with the traditional CAS definition as a motor programing disorder, but also by deficits in sequential hand movements.

All these studies provide suggestive evidence that the *FOXP2* gene might be the possible molecular substrate linking gestures with verbal language. However, the research in support of this hypothesis is still limited, although there are promising fields of investigation. For example, it would be interesting to assess the impact of the *FOXP2* gene polymorphism on linguistic and manual skills in healthy adults. This investigation not only would provide a further support to the molecular substrate hypothesis for the gestural-origin theory of speech, but it could have also practical implications for developmental and educational psychology, as it might allow an early assessment of the risk for dyslexia and/or dysgraphia in childhood individuals.

Other potential issues worthy of investigation might refer to the study of the expression of *FOXP2* in individuals with special linguistic and/or manual skills (e.g., polyglot people, painters of talent); the influence played by particular socio-environmental factors on its expression, which in turn might influence linguistic and/or manual skills of healthy individuals.

Finally, it would be intriguing to valuate whether the *FOXP2* genetic variations influence the resilience of linguistic and/or manual functions in patients affected by stroke.

#### **REFERENCES**

*Locke's Essay on the Human Understanding (A facsimile reproduction of the 1756 translation by T. Nugent of Condillac's 1747 essay)*. Gainesville, FL: Scholars' Facsimiles and Reprints.

action: a qualitative look," in *The Emergence of Symbols: Cognition and Communication in Infancy,* eds E. Bates, L. Benigni, I. Bretherton, L. Camaioni, and V. Volterra (New York, NY: Academic Press), 141–222.

*Received: 09 July 2013; accepted: 18 July 2013; published online: 05 August 2013.*

*Citation: Vicario CM (2013) FOXP2 gene and language development: the molecular substrate of the gesturalorigin theory of speech? Front. Behav. Neurosci. 7:99. doi: 10.3389/fnbeh.2013.00099*

*Copyright © 2013 Vicario. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

#### *Lynn K. Perry\* and Gary Lupyan*

*Department of Psychology, University of Wisconsin-Madison, Madison, WI, USA \*Correspondence<EMAIL_ADDRESS>
*Edited by:*

*Leonid Perlovsky, Harvard University and Air Force Research Laboratory, USA*

**Keywords: verbal interference, transcranial direct current stimulation (tDCS), language and thought, linguistic relativity, labeling**

Questions about the relationship between language and thought have long fascinated psychologists, philosophers, and the general public. One specific question is the extent to which verbal labels causally impact cognitive processes—how does calling an object by a particular name influence the way people categorize it; how does knowing words for mental states influence our reasoning about the minds of others; how does learning and using words like *left* influence our navigation behavior? One way to learn how the words we use to label objects, mental states, or locations affect our thoughts is to increase or decrease the ease with which we can use these words and observe outcomes of these manipulations on "non-linguistic" tasks. For example, if the word *left* enables us to remember which way to turn, preventing its activation might be expected to disrupt navigation. Manipulating the labeling process (and the engagement of language more broadly) is therefore very useful in exploring how language influences cognition. In this paper, we review two methodologies for implementing linguistic manipulations: verbal interference and transcranial direct current stimulation (tDCS), and discuss what we can learn about the role of language in cognitive processes from this line of research.

#### **VERBAL INTERFERENCE**

The primary method for manipulating linguistic activity is the use of verbal interference. The logic of verbal interference is well summarized by Winawer et al. (2007) in the context of studying the effects of language on color perception:

"[I]f linguistic processes play an active, online role in perceptual tasks, then a verbal dual task, but not a non-linguistic dual task, should diminish the goluboy/siniy [light blue / dark blue] category advantage found in Russian speakers," (Winawer et al., 2007, p. 7781).

Winawer et al. reported that Russian speakers appear to perceive a larger perceptual distinction between light and dark blues, consistent with a lexical difference between these categories in Russian. One possibility is that these differences stem from long-term perceptual learning caused by years of distinguishing colors in one language (e.g., Özgen and Davies, 2002). An alternative is that the cross-linguistic perceptual differences arise from online top-down influences of language (e.g., Fonteneau and Davidoff, 2007; Lupyan, 2008). If true, then disrupting these top-down effects in some way may disrupt these online effects of language eliminating the cross-linguistic difference—the pattern observed in the study. It is important to note that for verbal interference to have some effect does not require the involvement of language to be strategic. Rather, the involvement could be one of "the spontaneous but unspoken use of lexical codes," (Gilbert et al., 2006, p. 489). On this account, previous associations of using the word *blue* to describe the color blue cause the word to become reactivated when the color is seen (Lupyan, 2012a,b). The automatic recruitment of color labels may temporarily warp the perceptual space, producing cross-linguistic differences of the sort observed by Winawer et al.

Using similar logic, verbal interference has been used to argue for a role of language in number concepts (Frank and Barner, 2012), spatial memory (Hermer-Vazquez et al., 1999), categorization (Lupyan, 2009), and theory of mind (Newton and de Villiers, 2007). A similar logic underlies behavioral *up*-regulation of linguistic processes through redundant presentation of labels (e.g., Lupyan et al., 2007) or overt self-directed speech (Lupyan and Swingley, 2012). If internally generated labels support some cognitive or perceptual process, then redundant externally-presented labels can be thought to *up-regulate* the linguistic contribution and verbal interference to *down-regulate* it.

Verbal interference has also been used to examine mechanisms of developmental change. For example, in a task requiring participants to locate an object hidden in the corner of a room, children rely on the room's shape, while adults use one of the walls—painted in a distinct color from the other walls—to find the object (Hermer and Spelke, 1994; cf. Twyman and Newcombe, 2010). While developments in spatial language correlate with developments in using the wall as a landmark, this cannot tell us whether language causally influences spatial memory. In order to determine whether language supports development of spatial memory, Hermer-Vazquez et al. (1999) used a verbal interference paradigm. When adults performed the task while shadowing speech, their performance mirrored that of children's. Thus, in addition to improving our understanding of how language affects cognition in real-time, verbal interference has improved our understanding of how influences of language on cognition develop.

However, the use of verbal interference is not without problems. First, despite being used for many years across many domains, there is at present no working theory of verbal interference. Put bluntly: no one is sure how it works. Its use is vaguely based on the notion that language is a unitary system that can be disrupted, but this assumption is potentially problematic given the degree to which language-related activity is distributed both anatomically (Jung-Beeman, 2005) and functionally, e.g., syntax and semantics, rather than being entirely dissociable systems, tightly interact (MacDonald et al., 1994). Additionally, there is little agreement as to what constitutes verbal interference. Researchers have used numerous tasks, with little theoretical basis for choosing one over another. Tasks falling under the umbrella of verbal interference have included: rehearsing multi-digit numbers for later memory tests (e.g., Gilbert et al., 2006; Lupyan, 2009), repeating the letters "a,b,c" (e.g., Emerson and Miyake, 2003); alternating between naming months and days (e.g., Baddeley et al., 2001), making rhyme judgments (Roberson et al., 2007), answering factual questions such as "What is your name?" (e.g., Hatano et al., 1977), and repeating text, known as speech shadowing (Hermer-Vazquez et al., 1999; Frank and Barner, 2012). These interference tasks differ both in general difficulty and the ease with which performance can be assessed. For example, an interference task with a memory component gives an indication of how well participants rehearsed information—a proxy for how much effort was put into the verbal task—repetition of "a,b,c" does not. Such inconsistencies makes it hard (1) to infer why verbal interference sometimes interferes with primary task performance and sometimes does not and (2) to assess what aspect of language is recruited in the primary task which, when interfered with, disrupts performance.

A final problem with verbal interference is that it requires participants to perform two tasks simultaneously. It is therefore necessary to use a control interference task to determine which changes in primary task performance stem from manipulation to specifically linguistic processes and which stem from having to perform two tasks. Control tasks also vary widely across experiments: from tests of visuospatial memory (e.g., Gilbert et al., 2006; Lupyan, 2009) to foot tapping (Baddeley et al., 2001; Emerson and Miyake, 2003), and rhythm-shadowing (Hermer-Vazquez et al., 1999). Importantly, unless interference tasks are equated in all ways except for their "verbality," little can be said about the role of language in the primary task. For example, it has been argued that verbal, but not non-verbal interference disrupted performance on a falsebelief task (Newton and de Villiers, 2007). However, when the interference tasks were better equated for difficulty, both were similarly disruptive (Dungan and Saxe, 2012). Additionally, because verbal interference uses a dual-task paradigm, participants can exert different amounts of effort into the tasks. Such differential effort is difficult both to measure and control.

Despite its appeal, verbal interference paradigms have clear shortcomings. Below, we outline an alternative way of perturbing linguistic processes that solve *some* of the shortcomings, and advocate for systematic cross-method comparisons of linguistic perturbation methods to more fully inform our understanding of how language augments cognition and perception.

#### **TRANSCRANIAL DIRECT CURRENT STIMULATION**

One way to avoid some challenges posed by verbal interferences is by manipulating language processing without using secondary tasks through the use of noninvasive brain stimulation. Here, we focus on one such method—tDCS a painless method of regulating cortical excitability through weak electrical current to the scalp—that allows the experimenter to subtly up- and down-regulate neural activity over cortical areas implicated in language processing. For example, using tDCS to up-regulate activity over Wernicke's area (associated with aspects of labeling, particularly comprehension of word meaning; e.g., Price, 2000) is associated with increased ability to map novel words to pictures (Flöel et al., 2008) using it to up-regulate activity over Broca's area (associated with linguistic processes such as speech production; e.g., Gernsbacher and Kaschak, 2003) is associated with increased artificial grammar learning (de Vries et al., 2010).

Similar to using verbal interference, using tDCS to study linguistic influences on cognition assumes that language is a system that can be selectively perturbed. However, tDCS avoids the need to use dual task paradigms. The participant simply performs the main task while undergoing tDCS which, depending on the electrode arrangement, either up- or down-regulates cortical activity in targeted regions. Up-regulating activity is theoretically analogous to behavioral up-regulation through presentation of overt labels (Lupyan, 2008) or behavioral self-directed speech (Lupyan and Swingley, 2012); down-regulating activity is theoretically analogous to verbal interference.

Recently, Lupyan et al. (2012) examined effects of tDCS on non-verbal categorization, showing that down-regulating activity over Broca's area was associated with impairments in the ability to form categories that required selectively representing specific perceptual features to the exclusion of others, e.g., GREEN THINGS, a deficit similar to one shown by individuals performing verbal interference (Lupyan, 2009) or by those with language impairments such as aphasia (Davidoff and Roberson, 2004; Lupyan and Mirman, 2013). In avoiding pitfalls of a dual task design, however, Lupyan and colleagues' tDCS study allows for more definitive conclusions about mechanisms by which language might affect categorization because participants all completed the same task, and between-group differences can be linked to changes in neural activity in a particular cortical region providing at least a foothold for starting to connect behavioral data on the role of language in categorization to particular neural mechanisms.

#### **IMPORTANT FUTURE CONSIDERATIONS**

Although tDCS may be well-suited for manipulating linguistic activity, at present there are no direct comparisons of tDCS to verbal interference. It would be useful to know whether domains in which we have seen effects of verbal interference on performance are affected by tDCS over areas associated with language processes and whether domains in which we have *not* previously seen effects are similarly *unaffected*. For example, tDCS can be used to determine if language-related cortical regions shown to be recruited in non-verbal color-judgment tasks (e.g., Ting Siok et al., 2009) are *causally* implicated by showing perturbations with tDCS have behavioral consequences in colorjudgment, thereby informing the neural basis of language-augmented perception. tDCS also has the potential to open up additional domains to investigate effects of language on cognition without the errorfraught hunt for perfect control interference tasks.

In sum, we advocate: (1) a more systematic comparison of linguistic perturbation methods, specifically comparing behavioral linguistic perturbations to perturbations utilizing stimulation techniques such as tDCS; (2) a more rigorous comparison of different verbal interference tasks; and (3) a theory that elucidates the mechanisms by which verbal interference actually works. Such an examination will be critical to clarifying the contributions of language to cognition, helping us answer such questions as whether the mechanisms by which language affects perception of color categories are in some broad sense similar to mechanisms by which language affects spatial cognition. Performing the cross-method comparisons we advocate will highlight possible contradictions leading to further theoretical refinement. For example, what would it mean for the underlying cognitive and neural processes if one verbal interference method affects a primary task but another does not?

The extant empirical literature on effects of language on cognition and perception takes us considerably beyond the question as it is often phrased: "Does language affect thought," (Boroditsky, 2010a). This literature (e.g., Gentner and Goldin-Meadow, 2003; Casasanto, 2008; Boroditsky, 2010b; Lupyan, 2012a,b), while rich in demonstrations, requires a more rigorous investigation of the mechanisms by which learning and using language augment and perhaps fundamentally alter cognition and perception. We believe significant clarity on this important question can be achieved by combining creative uses of linguistic perturbation techniques with theoretical refinement of their mechanisms.

#### **ACKNOWLEDGMENTS**

We would like to thank Pierce Edmiston for his comments on an earlier version of this paper.

#### **REFERENCES**

evidence for a "category adjustment" model. *Mem. Cogn.* 35, 1814–1829. doi: 10.3758/BF0319 3512

*Received: 15 July 2013; accepted: 29 August 2013; published online: 17 September 2013.*

*Citation: Perry LK and Lupyan G (2013) What the online manipulation of linguistic activity can tell us about language and thought. Front. Behav. Neurosci. 7:122. doi: 10.3389/fnbeh.2013.00122*

*This article was submitted to the journal Frontiers in Behavioral Neuroscience.*

*Copyright © 2013 Perry and Lupyan. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

### Computational principles of syntax in the regions specialized for language: integrating theoretical linguistics and functional neuroimaging

#### *Shinri Ohta1,2, Naoki Fukui 3,4 and Kuniyoshi L. Sakai 1,4,5\**

*1 Department of Life Sciences, Graduate School of Arts and Sciences, The University of Tokyo, Tokyo, Japan*

*2 Japan Society for the Promotion of Science, Tokyo, Japan*

*3 Department of Linguistics, Sophia University, Tokyo, Japan*

*4 CREST, Japan Science and Technology Agency, Tokyo, Japan*

*5 Department of Basic Science, Graduate School of Arts and Sciences, The University of Tokyo, Tokyo, Japan*

#### *Edited by:*

*Leonid Perlovsky, Harvard University and Air Force Research Laboratory, USA*

#### *Reviewed by:*

*Cedric A. Boeckx, Catalan Institute for Research and Advanced Studies, Spain Noriaki Yusa, Miyagi Gakuin*

*Women's University, Japan*

#### *\*Correspondence:*

*Kuniyoshi L. Sakai, Department of Basic Science, Graduate School of Arts and Sciences, The University of Tokyo, 3-8-1 Komaba, Meguro-ku, Tokyo 153-8902, Japan e-mail<EMAIL_ADDRESS>
The nature of computational principles of syntax remains to be elucidated. One promising approach to this problem would be to construct formal and abstract linguistic models that parametrically predict the activation modulations in the regions specialized for linguistic processes. In this article, we review recent advances in theoretical linguistics and functional neuroimaging in the following respects. First, we introduce the two fundamental linguistic operations: Merge (which combines two words or phrases to form a larger structure) and Search (which searches and establishes a syntactic relation of two words or phrases). We also illustrate certain universal properties of human language, and present hypotheses regarding how sentence structures are processed in the brain. Hypothesis I is that the Degree of Merger (DoM), i.e., the maximum depth of merged subtrees within a given domain, is a key computational concept to properly measure the complexity of tree structures. Hypothesis II is that the basic frame of the syntactic structure of a given linguistic expression is determined essentially by functional elements, which trigger Merge and Search. We then present our recent functional magnetic resonance imaging experiment, demonstrating that the DoM is indeed a key syntactic factor that accounts for syntax-selective activations in the left inferior frontal gyrus and supramarginal gyrus. Hypothesis III is that the DoM domain changes dynamically in accordance with iterative Merge applications, the Search distances, and/or task requirements. We confirm that the DoM accounts for activations in various sentence types. Hypothesis III successfully explains activation differences between object- and subject-relative clauses, as well as activations during explicit syntactic judgment tasks. A future research on the computational principles of syntax will further deepen our understanding of uniquely human mental faculties.

#### **Keywords: syntax, universal grammar, recursive computation, inferior frontal gyrus, supramarginal gyrus, fMRI**

#### **INTRODUCTION**

Tree structures are one of the most ubiquitous structures in nature, appearing in the branchings of rivers, lightning, snowflakes, trees, blood vessels, nervous systems, etc., and can be simulated in part by fractal geometry (Mandelbrot, 1977). To properly quantify the complexity of such tree structures, various models have been proposed. The number of nodes would be one of the simplest models; this approach consists of simply counting the total number of non-terminal nodes (branching points) and terminal nodes of a tree structure (**Figure 1A**). This model obviously cannot capture hierarchical levels within the tree (sister relations in linguistic terms). To properly measure the hierarchical levels of a tree structure, we have proposed the Degree of Merger (DoM) as a key computational concept (**Figure 1B**) (Ohta et al., 2013). The DoM is defined as the *maximum depth* of merged subtrees (called Mergers) within a given domain. With this model, the same numbers are assigned to the nodes with an identical hierarchical level. The DoM corresponds to the number of iterations for generating fractal figures, when the tree structures are self-similar.

In this article, we first explain certain universal properties of human language discovered in modern linguistics, and we present hypotheses regarding how sentence structures are processed in the brain. We then introduce our recent functional magnetic resonance imaging (fMRI) study, which demonstrated that the DoM is indeed a key syntactic factor that accounts for syntax-selective activations in the regions specialized for language (Ohta et al., 2013). We also show that the top-down connectivity from the left inferior frontal gyrus to the left supramarginal gyrus is critical for the syntactic processing. Next, we clarify that the DoM can account for activation modulations in the frontal region, depending on different sentence structures. Finally, we hypothesize that the DoM domain changes dynamically in accordance with iterative Merge applications, the distance

required for Search operations (or simply the "Search distance"), and/or task requirements. This hypothesis accounts for activation differences between subject-relative and object-relative clauses, as well as for activations during explicit syntactic judgment tasks.

#### **UNIVERSAL PROPERTIES OF HUMAN LANGUAGE**

#### **THEORETICAL BACKGROUND**

Modern linguistics has clarified universal properties of human language, which, directly or indirectly, reflect the computational power, or engine, of the human language faculty. A sentence is not a mere string of words, but is made of phrase structure (called constituent structure). Moreover, a single phrase contains the key element (i.e., the "head") that determines the basic properties of the phrase. Furthermore, a sentence can be recursively embedded within other sentences, as in, e.g., "*I think that John believes that Mary assumes that. . .* ," and there is in principle no upper bound for the length of sentences. These universal properties can be adequately and minimally expressed by hierarchical tree structures with a set of relevant structural relations defined on such structures (Chomsky, 1957, 1965).

To construct hierarchical tree structures, modern linguistics has proposed the fundamental linguistic operation of *Merge* (capitalized in linguistics to indicate a formal operation). Merge is a structure-building operation that combines two syntactic objects (words or phrases) to form a larger structure (Chomsky, 1995). Merge would be theoretically "costless," requiring no driving force for its application (Saito and Fukui, 1998; Chomsky, 2004; Fukui, 2011). Besides Merge, we have proposed *Search* operation of searching syntactic features, which applies to a syntactic object already constructed by Merge, where Search couples and connects two distinct parts of the same structure, thereby assigning relevant features from one to the other part (Fukui and Sakai, 2003). Various other "miscellaneous" operations that have been employed in the linguistics literature, such as Agree, Scope determination, Copy, etc., are in fact different manifestations of one and the same, i.e., more generalized, operation of Search (Fukui and Sakai, 2003). Human language, therefore, should minimally contain two universal operations, Merge and Search. The total number of Merge and Search applications within an entire sentence are here simply denoted as "number of Merge" and "number of Search," respectively. The number of Merge in a sentence becomes always one less than the number of terminal nodes, *irrespective of sentence structures* (see Appendix S2 of Ohta et al., 2013).

#### **SYMBOL SEQUENCES AND FORMAL LANGUAGES**

In regard to formal symbol sequences beyond the bounds of finite state languages, three specific types of language have been discussed in the linguistics literature: (i) "counter language," (ii) "mirror-image language," and (iii) "copying language" (cf. Chomsky, 1957, p. 21).

The counter language can be handled by a counting mechanism to match the number of each symbol, whereas the mirror-image language contains a mirror-image dependency, requiring more than a mere counter. If the number of symbols is not fixed (i.e., infinite), both of these languages are beyond the bounds of finitestate grammars, and are to be generated by context-free (simple) phrase structure grammars, while the copying language with a cross-serial dependency clearly goes beyond the bounds of even context-free phrase structure grammars, requiring a more powerful device, viz., context-sensitive phrase structure grammars or transformational grammars (Chomsky, 1959; Hopcroft and Ullman, 1979).

It remains a central issue in cognitive sciences whether or not the faculty of language is also shared by animals. Animals have thus been tested with regular symbol sequences such as A*n*B*n* (*n* ≥ 2; i.e., AABB, AAABBB, *...*) and (AB)*n* (*n* ≥ 2; i.e., ABAB, ABABAB, *...*), which differ in *symbol order*. In an animal study, songbirds were trained to discriminate patterns of A*n*B*n* and (AB)*n* in more than ten thousand trials (Gentner et al., 2006). However, this learning can be achieved by tracking symbol repetition or counting strategy alone (Corballis, 2007). There is also a recent report that songbirds seemed to discriminate strings with or without nesting (Abe and Watanabe, 2011), but this learning can be achieved by simply remembering partial strings (Beckers et al., 2012). Along the lines of contrasting A*n*B*n* and (AB)*n*, fMRI studies have tested participants with different symbol sequences, such as A2A1B1B2 vs. A1B1A2B2 (each subscript denotes a matching order), which also differ in matching order (Bahlmann et al., 2008). The difference in activation patterns can be simply explained by differences in any factor associated with matching orders and symbol orders, i.e., temporal order-related factors. It is thus necessary to completely control these general factors when extracting any syntactic factor from a number of cognitive factors involved in actual symbol processing.

Since the number of symbols is inevitably fixed (i.e., finite) in any actual experiment, it should be noted that any symbol sequence can be expressed by a regular (finite state) grammar, i.e., the least powerful grammar in the so-called Chomsky hierarchy. Therefore, one cannot, in principle, claim from the experiments that individual grammars (e.g., context-free phrase structure grammars vs. regular grammars) are differentially represented in the brain. Thus, the neural representation of individual grammars was *not* within the scope of Ohta et al. (2013). In addition to the various models examined, other non-structural and non-symbolic models with simple recurrent networks have been proposed to process some examples of even context-free and context-sensitive phrase structure languages, generalizing to some degree to longer strings than the training set (Rodriguez, 2001). However, these models do not account for any parametric modulation of the activations reported in Ohta et al. (2013), except the length of sentences.

In the previous experiment, we introduced letter strings, which had no lexical associations but had both symbol orders (e.g., AABB and ABAB) and matching orders (e.g., A2A1B1B2). There were two basic types of strings: reverse-order strings (Reverse) and same-order strings (Same). In the Reverse strings, the first and second halves of a string were presented in the reverse order, while in the Same strings the halves were presented in the same order (**Figure 2**). Under these conditions, there was actually no path connecting the non-terminal nodes of symbol pairs (e.g., A1B1 and A2B2), as there was *no* Merge application to connect the multiple pairs. In regard to the symbol orders, both the Reverse and Same strings took the above type (i) of A*n*B*n*. In regard to the matching orders, the Reverse string took the type (ii) of A2A1B1B2 or A3A2A1B1B2B3, while the Same string took the type (iii) of A1A2B1B2 or A1A2A3B1B2B3.

#### **HYPOTHESIS I**

Given a tree structure with a formal property of Merge and *iterativity* (recursiveness) (Fukui, 2011), we propose the following hypothesis (Hypothesis I):

(1) The DoM, which can be defined as the *maximum depth* of merged subtrees within a given domain, is a key computational concept to properly measure the complexity of tree structures.

The DoM can quantify and compare various syntactic phenomena, such as self-embedding, scrambling, *wh*-movement, etc. Furthermore, when Search applies to each syntactic object with its hierarchical structure, the calculation of the DoM plays a critical role. Indeed, from a nested sentence "[[*The boy*2 [*we*3 *like*3]2]1 *sings*1]0" (subscripts denote the DoM for each node), two sentences "[*The boy.*..]1 *sings*1" and "*we*3 *like*3" are obtained, where relevant features (numbers and persons here) are searched and matched between the nodes with the identical DoM. Since such analyses of hierarchical structures would produce specific loads in syntactic computation, we expect that the DoM and associated "number of Search" would affect performances and cortical activations.

Sentences with various constructions have been previously discussed in terms of the acceptability of sentences (cf. Chomsky, 1965, p. 12).

The nested constructions are created by *centrally* embedding a phrase within another phrase (with some non-null element to its left and some non-null element to its right), and the self-embedded constructions are the special case of nested constructions when nesting occurs within the *same* type of phrases (e.g., noun phrases). The multiple-branching constructions are made by conjoining phrases at the same hierarchical level, and the left/right-branching constructions are yielded by merging a phrase in the left-most or right-most phrase. The degrees of nesting and self-embedding have already been proposed to model the understanding of sentences (Miller and Chomsky, 1963). By generalizing this attractive idea in such a way as to include any construction with merged phrases, we introduced the DoM as a key computational concept.

Based on the nested (self-embedded), left/right-branching, and multiple-branching constructions, three basic types of sentences can be distinguished: the nested sentence (Nested), simple sentence (Simple), and conjoined sentence (Conjoined), respectively. The sentences shown in **Figure 3** are some examples in Japanese. Given syntactic structures like the ones shown, the correspondence of each subject-verb pair becomes fixed. Here N and V denote a noun phrase and a verb phrase, respectively. For the sentence shown in **Figure 3A**, an entire sentence is constructed by nesting sentences in the form of [N2[N1V1]V2], where [N*i*V*i*] represents a subject-verb pair of a sentence. Since Japanese is a head-last, and hence an SOV (verb-final) language, a main verb is placed after a subordinate clause. Therefore, Japanese sentences naturally yield nested structures without having to employ, as in English, object-relative clauses (e.g., "*The boy whoi we like ti sings*"), which require "movement" of an object (i.e., with more Merge applications) and thus leave behind a "trace" (*ti*, subscripts denote the same entity). For the sentence shown in **Figure 3B**, a simple sentence is constructed by adding the same number of left/right branches to both Ns and Vs. The last noun (i.e., head) in the branches of Ns made a subject-verb pair with the last verb (i.e., head) of a compound verb. Each simple sentence thus takes the form of [(NN1)(VV1)]. For the sentence shown in **Figure 3C**, an entire sentence is constructed by conjoining sentences in the form of [N1V1][N2V2]. When considering longer sentences like N3N2N1V1V2V3, these constructions have distinct values for DoM.

#### **HYPOTHESIS II**

In any sentence, functional elements, such as inflections, auxiliary verbs, and grammatical particles, serve an essentially grammatical function without descriptive content. In regard to the fundamental role of these functional elements, we propose the following hypothesis (Hypothesis II):

(2) The basic frame of the syntactic structure of a given linguistic expression (e.g., sentence) is determined essentially by functional elements, which trigger Merge and Search operations.

In the non-sense poem "Jabberwocky" by Lewis Carroll, e.g., "'*Twas* ('*It was*') *brillig, and the slithy toves did . . .* ," the basic frames of syntactic structures are indeed determined by the functional elements of "*'Twas*," "*and*," "*the*," "*-s*," and "*did*." In the Japanese language, grammatical particles and morphosyntactic inflections are functional elements. The sentences shown in **Figure 3** actually contain only three kinds of grammatical particles, which represent *canonical* (i.e., in a prototypical use) case markings and syntactic information in Japanese: *-ga*, a nominative case marker; *-no*, a genitive case marker; and *-to*, a

**FIGURE 3 | Japanese sentences with three major constructions.** The figure shows three basic types of sentences in Japanese: the nested sentence, simple sentence, and conjoined sentence. Based on contemporary linguistics, each diagram represents a unique tree structure of each sentence constructed from nouns and verbs. Below each example, word-by-word translations in English are shown. **(A)** A sentence (S) at the lowest hierarchical level was nested into an entire sentence (S') ("*Taro-ga Hanako-ga utau-to omou*," "*Taro thinks that Hanako sings*"). **(B)** A simple sentence was constructed by adding the same number of left/right branches to both nouns and verbs ("*Taro-no ani-ga tabe hajimeru*," "*Taro's brother starts eating*"). **(C)** An entire sentence (S') was constructed by conjoining two sentences ("*Taro-ga utatte Hanako-ga odoru*," "*Taro sings, and Hanako dances*"). Symbols used: S and S', sentence; N, noun phrase; V, verb phrase; *-ga*, nominative case marker; *-no*, genitive case marker; *-to*, complementizer; *-te*, gerundive form; Nom, nominative case; Gen, genitive case; Comp, complementizer.

complementizer. It should be noted that both the nested and simple sentences have the same symbol order of N*n*V*n*, but they have different grammatical particles and syntactic structures. In contrast, both the simple and conjoined sentences have the same tree structures as a result, but they have different symbol orders of N*n*V*n* or (NV)*n* (*n* ≥ 2). It is the grammatical particles and morphosyntactic inflections, but not symbol orders or matching orders themselves, that determine the basic frame of syntactic structures of a sentence.

Following morphosyntactic and phonological features of Japanese verbs (Tsujimura, 2007), Vs take a non-past-tense form (*-ru*), past-tense form (*-ta*), or gerundive form (*-te*); Vs ending with *-to* and *-te* introduce *that*-clauses and *and*-conjunctives, respectively. The gerundive form can be used not only in *and*conjunctives, but in compound verbs (e.g., "*tabete-simau*," "*finish*

syntactic structures of these jabberwocky sentences are same as those of real sentences in **Figure 3**. The digits shown in red and blue denote the DoM for each node and "number of Search," respectively. We also tested the long stimuli with six words.

*eating*"; actual Japanese words will be translated hereafter), much as gerunds can in English. The *-ga*, *-no*, *-to*, and *-te* endings (*green* letters in **Figures 3**, **4**), together with the first verb of a compound verb in an adverbial form (e.g., "*tabe*"), are associated with Merge applications to connect multiple nouns/verbs or sentences, amounting to "number of Merge." The Japanese language lacks the "agreement features" (i.e., number, person, gender, etc.), but it is nevertheless equipped with the general Search procedure that is employed in agreement phenomena in other languages. This Search mechanism is in fact attested for various other phenomena in Japanese (see Fukui and Sakai, 2003 for further discussion). For example, the Japanese language exhibits a phenomenon called "honorification," where a noun phrase denoting an honored person and the form of honorifics in verbs are to be matched (Gunji, 1987; Ivana and Sakai, 2007).

In this section, we provided some theoretical discussions based on modern linguistics, focusing on the two fundamental linguistic operations of Merge and Search. We hypothesized that the DoM is a key computational concept to properly quantify the complexity of tree structures, and that the basic frame of the syntactic structure of a given linguistic expression is determined essentially by grammatical particles and morphosyntactic inflections, which trigger Merge and Search operations.

#### **THE DoM AS A KEY SYNTACTIC FACTOR ELUCIDATED BY AN fMRI EXPERIMENT**

One possible way to elucidate the neural basis of computational properties of natural language is to examine how the brain responds to the modulation of specified syntactic factors. We should not be content with such a general cognitive factor as so-called "syntactic complexity" or "syntactic working memory," which could involve both linguistic and non-linguistic factors. We should instead identify minimal factors that *sufficiently* explain any activation change obtained. In our recent study, we focused on different sentence constructions, and found that the DoM and "number of Search" were the *minimal* syntactic factors associated with phrase structures, which parametrically modulate cortical responses measured with event-related fMRI (Ohta et al., 2013). In this section, we will present the basic paradigm and results of this work.

#### **A PARADIGM TO TEST HYPOTHESES I AND II**

We used jabberwocky sentences, which consist of pseudonoun phrases (Ns) and pseudoverb phrases (Vs) that lack lexical associations, but have grammatical particles and morphosyntactic inflections (**Figure 4**). According to Hypothesis II stated above, these jabberwocky sentences had the same syntactic structures as normal sentences. Under the sentence conditions of Nested, Simple, and Conjoined with the same structures shown in **Figure 3**, the jabberwocky sentences were visually presented in a phrase-by-phrase manner to the participants. We made six pseudonouns by repeating the same syllables with voiced consonants and any one of /a/, /u/, or /o/: *rara*, *zaza*, *mumu*, *gugu*, *yoyo*, and *dodo*. We also made four pseudoverb roots by repeating the same syllables with voiceless consonants and either /i/ or /e/: *kiki*, *hihi*, *sese*, and *tete*. Here, vowel harmony was adopted to change the last, i.e., the second, vowel of the verb root, so that this vowel harmonized with the vowel (i.e., /a/, /u/, or /o/) of the corresponding subject (e.g., "*rara-ga tetaru*" from "*teteru*," underlined vowels within pseudowords). These features of vowels were only *experimentally* introduced, and these pseudoverbs lacked grammatical features, as in the Japanese verbs. In all jabberwocky sentences, the distinction between Ns and Vs was clear without memorizing pseudowords, because Ns, but not Vs, ended with either *-ga* or *-no*, i.e., case markers in Japanese such as -*ga* and -*no* can be generally attached only to nominal phrases.

To test whether participants actually paid attention to the correspondence of each subject-verb pair, we used a matching task, such that the vowel of a subject (N*i* as a sample stimulus) was matched with the last vowel of the corresponding verb root (V*i* as a comparison stimulus), probing the goal with the same vowel as explained above. It follows that the same syntactic structures were constructed from matching stimuli and non-matching stimuli (e.g., "*rara-ga teturu*"), which were both well-formed, i.e., *grammatical*, in Japanese. A matching strategy (counting, for example, the first and the fourth stimuli for matching) was useful in solving the task, but performing the task was *not* prerequisite for constructing syntactic structures. Our matching task is different from classification tasks for symbol orders (e.g., AABB vs. ABAB, where A and B are symbols representing certain sets of stimuli), which can be solved by counting the maximum number of consecutively repeated symbols. The order of the Nested, Simple, and Conjoined was pseudo-randomized without repetition. We further examined whether cortical activations were modulated by the length of sentences: short (S as a subscript, e.g., Conjoined*(S)*; four-word) and long (L as a subscript, e.g., Conjoined*(L)*; sixword) sentences, where the DoM domain spanned four and six relevant words, respectively.

We also used the same matching task under the string conditions of Reverse and Same (**Figure 2**), such that the first half of a string (A*i* as a sample stimulus) was matched with the corresponding second half (B*i* as a comparison stimulus) in the reverse or same order. These string conditions also controlled any involvement of the matching strategy stated above. Between the Nested (N2N1V1V2) and Reverse (A2A1B1B2) conditions, the curved arrows shown in **Figures 2**, **4** represent the *same* matching order of sequentially presented stimuli. The symbol order was also identical among the Nested, Simple, Reverse, and Same conditions in the form of N*n*V*n* or A*n*B*n*. Combining these multiple conditions, we were able to properly examine whether different structures were actually constructed between sentences and strings. The spatial and temporal resolution of fMRI, as well as its sensitivity, has been proven to be high enough to confirm various hypotheses about human cognitive functions like ours.

#### **SYNTAX-SELECTIVE ACTIVATIONS MODULATED BY THE DoM AND THE NUMBER OF SEARCH**

To control both matching orders and symbol orders, we directly compared the Nested with the Reverse condition, using the Simple and Same conditions as respective references, i.e., (Nested − Simple) *>* (Reverse − Same), where we combined the short and long stimuli. This contrast further controlled various linguistic and non-linguistic factors, such as the number of Merge, number of case markers, number of nodes, memory span, and counting. This point is particularly important, because temporal order-related or memory-related factors have often been confused with differences in structure or grammar type. Significant activation was elicited by this contrast in the pars opercularis and pars triangularis of the left inferior frontal gyrus (L. F3op/F3t) [local maximum: (*x*, *y*, *z*) = (−51, 24, 24), *Z* = 5.8], and the left supramarginal gyrus (L. SMG) [(−39, −45, 42), *Z* = 5.7] (**Figure 5A**). Our results are best explained by the linguistic factors associated with the Nested condition, supporting our second hypothesis that basic syntactic structures are constructed when well-formed sentences are given even without lexical meanings.

For these two critical regions, we examined the percent signal changes under the Nested and Simple conditions by subtracting those under the Conjoined condition, which had the simplest tree structures (**Figure 4** and **Table 1**), separately for

**FIGURE 5 | Functional and anatomical evidence of a syntax-related network. (A)** Regions identified by the (Nested − Simple) *>* (Reverse − Same) contrast (see **Figure 4**). Activations were projected onto the left (L) and right lateral surfaces of a standard brain. **(B)** Percent signal changes for Nested − Conjoined and Simple − Conjoined in the L. F3op/F3t and L. SMG. Overlaid red dots and lines denote the values fitted with the estimates (digits in red) for the best models: DoM for the L. F3op/F3t and "DoM + number of Search" for the L. SMG. **(C)**

The results of DCM, testing effective connectivity between the L. F3op/F3t and L. SMG. The best model included a significant top-down connection from the L. F3op/F3t to L. SMG (a thick line). **(D)** Anatomical connectivity between the L. F3op/F3t and L. SMG revealed by DTI. The population probability map is shown on the left lateral and dorsal surfaces of a standard brain with maximum intensity projection. Blue spheres represent seed regions of the L. F3op/F3t and L. SMG. Symbols used: L, long sentences; S, short sentences.

**Table 1 | Estimates of various factors to account for activations in Ohta et al. (2013).**

*Estimates under the Conjoined condition were subtracted from those under the other Nested and Simple conditions [e.g., DoM for Nested(L)* − *Conjoined(L), 5* − *2* = *3], separately for long and short sentences. We regarded "DoM* + *number of Search" (i.e., adding the estimates of two factors) as an additional factor.*

long and short sentences. Since we used the Conjoined*(L)* and Conjoined*(S)* as appropriate references, we examined whether likewise *subtracted* estimates of each factor (e.g., DoM for Nested*(L)* – Conjoined*(L)*; see **Table 1**) directly explained the parametric modulation of activations in the four contrasts of Nested*(L)* – Conjoined*(L)*, Nested*(S)* – Conjoined*(S)*, Simple*(L)* – Conjoined*(L)*, and Simple*(S)* – Conjoined*(S)*. The percent signal changes in the L. F3op/F3t and L. SMG, averaged across significant voxels, indeed correlated exactly in a step-wise manner with the parametric models of the DoM [3, 1, 1, 0] and "DoM + number of Search" [3, 1, 0, −1], respectively (**Figure 5B**). By generalizing the role of Search, we assumed that Search applied to a subject-verb pair, where the relevant features (vowels here) are experimentally "inserted" (Ohta et al., 2013).

We further examined 19 models proposed in theoretical linguistics, psycholinguistics, and natural language processing to verify that the models of the DoM and "DoM + number of Search" best explained the cortical activations (Ohta et al., 2013). All contrasts of Nested*(L)* – Conjoined*(L)*, etc. predicted that the activations should be exactly zero when a factor produced no effect or load relative to the Conjoined. We thus adopted a nointercept model, in which percent signal changes of each region were fitted with a single (thus minimal) scale parameter to a model of each factor using its subtracted estimates. For the four contrasts, a least-squares method was used to minimize the residual sum of squares (RSS) for the four fitted values (i.e., four estimates multiplied by a fitting scale) against the corresponding signal changes averaged across participants (**Table 2**).

The model of the DoM for the L. F3op/F3t, as well as that of "DoM + number of Search" for the L. SMG, indeed produced by far the least RSS (≤0.0020) and largest coefficient of determination (*r*2) (≥ 0.97). Goodness of fit was further evaluated for each model by using a one-sample *t*-test (significance level at α = 0*.*0125, Bonferroni corrected) between the fitted value for each contrast and individual activations. The model of the DoM for the L. F3op/F3t, as well as that of "DoM + number of Search" for the L. SMG, produced no significant deviation for the four contrasts (*P* ≥ 0*.*17). To further take account of interindividual variability, we fitted "linear mixed-effects models" with individual activations, and found that the models of the DoM and "DoM + number of Search" were by far more likely for the L. F3op/F3t and L. SMG, respectively. Even if we took the Simple condition as a reference for subtracted estimates, we obtained the same results of best models. These results directly support Hypotheses I and II, such that the basic frame of syntactic structures are determined essentially by functional elements, whereas the DoM, together with the number of Search, is a key factor to properly quantify the complexity of the syntactic structures.

#### **THE SIGNIFICANCE OF THE CONNECTIVITY BETWEEN THE L. F3op /F3t AND L. SMG**

It has been reported that the L. F3op/F3t is specialized for syntactic processing (Stromswold et al., 1996; Dapretto and Bookheimer, 1999; Embick et al., 2000; Hashimoto and Sakai, 2002; Friederici et al., 2003; Musso et al., 2003; Suzuki and Sakai, 2003; Kinno et al., 2008), suggesting that this region subserves a grammar center (Sakai, 2005). On the other hand, the left angular gyrus and SMG (L. AG/SMG) have been suggested to be important for vocabulary knowledge or lexical processing (Lee et al., 2007; Pattamadilok et al., 2010). To elucidate the relationships between the L. F3op/F3t and L. SMG, we modeled the effective connectivity between these two regions by using dynamic causal modeling (DCM). Our interest was to identify the direction of the connectivity modulated by the Nested condition, which has the largest DoM of all conditions. First, we assumed intrinsic, i.e., task-independent, bi-directional connections, and the models were grouped into three "modulatory families": families with modulation for the bottom-up connection from the L. SMG to L. F3op/F3t, for the top-down connection from the L. F3op/F3t to L. SMG, and for both connections. Each family was composed of three "input models" as regards the regions receiving driving inputs. We found that the model with the modulation for the bottom-up connection, in which the L. F3op/F3t received driving inputs, was the best and most probable model (**Figure 5C**). We further confirmed that the intrinsic top-down connectivity was significantly positive (+0.22; *P <* 0*.*0002), while the bottom-up connectivity was negatively modulated.

A recent DCM study with a picture-sentence matching task has suggested that the L. F3op/F3t received driving inputs (den

**Table 2 | Fittings and likelihood of various models tested in Ohta et al. (2013).**

*Percent signal changes in the L. F3op/F3t and L. SMG were fitted with a single scale parameter to a model of each factor using its subtracted estimates (Table 1) for the four contrasts of Nested(L)* − *Conjoined(L), Nested(S)* − *Conjoined(S), Simple(L)* − *Conjoined(L), and Simple(S)* − *Conjoined(S). The P-values for the t-tests are shown in ascending order. The models with an asterisk resulted in the best fit of 19 models tested (four models are shown here) for explaining activations in the L. F3op/F3t or L. SMG, i.e., with the least residual sum of squares (RSS), largest coefficient of determination (r2), and larger P-values. The likelihood ratio was taken as the ratio of each model's likelihood to the best model's likelihood. The best models were by far more likely than the other models.*

Ouden et al., 2012), which was consistent with our DCM results. Moreover, our previous studies revealed that the functional connectivity between the L. F3t/F3O (pars orbitalis) and L. AG/SMG was selectively enhanced during sentence processing (Homae et al., 2003), and that the L. AG/SMG was also activated during the identification of correct past-tense forms of verbs, probably reflecting an integration of syntactic and vocabulary knowledge (Tatsuno and Sakai, 2005). Considering the role of the L. AG/SMG in lexical processing, the Search operation based on the DoM would be essential in assigning relevant features to the syntactic objects derived from lexical items.

To further confirm the anatomical plausibility of the network between the L. F3op/F3t and L. SMG revealed by DCM, we used diffusion tensor imaging (DTI) with a probabilistic tractography. We observed that a single continuous cluster of the left superior longitudinal and arcuate fasciculi (SLF/AF) connected these regions (cluster size, 3189 mm3), together with much smaller clusters or islands (**Figure 5D**). Moreover, the left SLF/AF was consistently observed in all participants.

The findings of recent DTI studies have been controversial regarding the functional roles of two different pathways in language processes: the dorsal tracts of the SLF/AF, and the ventral tracts of the middle longitudinal fasciculus (MdLF) and extreme capsule (EmC). Both pathways connect the inferior frontal and superior/middle temporal areas (Saur et al., 2008; Wilson et al., 2011; Wong et al., 2011; Griffiths et al., 2013). Our DCM and DTI results indicate that the L. SMG activations reflecting the DoM mirrored a top-down influence from the L. F3op/F3t through the left dorsal pathway of the SLF/AF, revealing the most crucial network and pathway for syntactic computation.

#### **FURTHER CONFIRMATION OF HYPOTHESES I AND II**

#### **A PICTURE-SENTENCE MATCHING PARADIGM**

We further examined whether our hypotheses hold for various cases discussed in previous studies. In our fMRI study (Kinno et al., 2008), we used a picture-sentence matching task with three sentence types in Japanese: active, passive, and scrambled sentences (**Figure 6A**). In the picture-sentence matching task, the participants read a sentence covertly and judged whether or not the action depicted in a picture matched the meaning of the sentence. Each sentence had two noun phrases called *arguments*, each of which assumes a different grammatical relation ("subject, direct object, or indirect object" in linguistic terms) and a semantic role ("agent, experiencer, or patient" in linguistic terms, i.e., an agent who performs the action, and an experiencer/patient who is affected by it); these three conditions were thus called Twoargument conditions. More specifically, the active, passive, and scrambled sentences corresponded to "agent and patient" (subject and direct object), "experiencer and agent" (subject and indirect object), and "patient and agent" (direct object and subject) types, respectively. Pictures consisted of two stick figures, each of which was distinguished by a "head" symbol: a circle (◦), square (-), or triangle (). These sentences excluded the involvement of pragmatic information about word use (e.g., "*An officer chases a thief* " is more acceptable than "*A thief chases an officer*"). To minimize the effect of general memory demands, a whole sentence of a minimal length was visually presented for a longer time than was needed to respond.

In Japanese syntax, the grammatical relations are first marked by grammatical particles (nominative, dative, or accusative), which in turn allow the assignment of semantic roles. In the active sentences we used, a noun phrase with the nominative case marker *-ga* (*green* letters in **Figure 6B**) is associated with an agent, and the one with the accusative case marker *-o* is associated with a patient. For the passive sentences we used, however, a noun phrase with the nominative case marker *-ga* is associated with an experiencer (a person experiencing a situation), whereas a passive bound verb "-(*r*)*areru*" marks passiveness, making a subject-verb pair with the experiencer. In contrast, a noun phrase with the dative marker *-ni* is associated with an agent, whereas an action verb (e.g., "*hik*(*u*)," "*pull*") makes a subject-verb pair with the agent, forming a subordinate clause within the main clause "◦-*ga.*.. *-*(*r*)*areru.*" Note that there exist similar causative

structures in both Japanese and English: "*Hanako-ga kare-ni hikaseta*," "*Hanako made him pull*." Actually, there are two types of passivization in Japanese: *ni* passive (e.g., "*Hanako-ga Taroni hik-areru*," "*Hanako is affected by Taro's pulling her*") and *ni yotte* passive (e.g., "*Hanako-ga Taro-ni yotte hik-areru*," "*Hanako is pulled by Taro*"). According to Kuroda (1992), the *ni* passive involves no noun-phrase movement, while the *ni yotte* passive involves a movement similar to the case in English. For the scrambled sentences, an object moves from its canonical position to higher nodes by undergoing another Merge operation. This type of constructions is perfectly normal, not only in Japanese but in German, Finnish, and other languages. We also tested the One-argument condition, under which each sentence was presented with an intransitive verb and double agents. This condition did not involve two-argument relationships, and was thus syntactically simpler than any of the Two-argument conditions.

#### **HYPOTHESIS III**

Here we present the following hypothesis (Hypothesis III):

(3) The DoM domain changes dynamically in accordance with iterative Merge applications, the Search distances, and/or task requirements.

Since Merge combines two syntactic objects to form a larger structure, Merge always produces a one-level higher node. When Merge applies iteratively to an existing phrase or sentence, the DoM domain becomes thus larger in accordance with the number of Merge applications. The Search distance is the structural distance between two distinct parts to which the Search operation applies, regardless of the nodes that are irrelevant to the Search operation. As observed from **Figure 4**, the DoM domain changes in accordance with the Search distance. On the other hand, for every sentence stimulus in the study of Ohta et al. (2013), the construction of syntactic structures was ensured by task requirements, in which three sentence types had to be distinguished while they were completely mixed. Task requirements include not only certain constraints required by experimental tasks, but detailed parsing naturally required to understand a part of phrases or sentences (e.g., subject-verb relationships and noun-pronoun (coreference) relationships).

In the above mentioned paradigm (Kinno et al., 2008), the four task conditions (three sentence types under the Two-argument conditions, as well as one type under the One-argument condition) were completely mixed (see **Figure 6A**). With such task requirements, the DoM domain spanned three relevant words for all sentence types under the Two-argument conditions. Under the One-argument condition, the action of two stick figures was always identical, and thus a subject (a triangle just below N in **Figure 6B**) is regarded as a unit. Under these four task conditions, participants were required to check at least one of the argument-verb relationships, demanding Search at least once. For the scrambled sentences alone, an additional Search operation should match the identical indices of the moved object and its trace. For the active, passive, and scrambled sentences, the estimates of DoM were 2, 3, and 3, respectively, while those of the DoM was 1 under the One-argument condition.

#### **APPLYING THE DoM TO VARIOUS SENTENCE TYPES**

In the study of Kinno et al. (2008), we directly contrasted passive and active sentence conditions to identify a cortical region that is activated by purely syntactic processes. This stringent contrast resulted in significant activation in the left dorsal F3t (L. dF3t) alone [(−48, 24, 21), *Z* = 3.8] (**Figure 7A**), which was very close to the L. F3op/F3t activation in the study of Ohta et al. (2013). The L. dF3t activation was significantly enhanced under both the passive and scrambled sentence conditions compared to that under the active sentence condition (*P* ≤ 0*.*033) (**Figure 7B**), whereas there was no significant difference between the passive and scrambled sentence conditions (*P* = 0*.*15). Taking the One-argument condition as a reference for subtracted estimates, the signal changes in the L. dF3t were precisely correlated in a step-wise manner with the parametric model of the DoM [1, 2, 2], producing the RSS of 0.0001 and *r*2 of 0.99, without significant deviation for the three contrasts (*P* ≥ 0*.*87). The model of the DoM thus *sufficiently* explains the L. dF3t activations. It should be noted that the parametric model of "the number of nodes" [2, 4, 4] also yielded the same fitting results in this case. The design of experimental paradigms limits the separation of multiple factors.

In a recent fMRI study, only right-branching constructions were examined, and activations in the L. F3t were modulated by the size of constituents (i.e., number of terminal nodes) (Pallier et al., 2011). Since the estimates of the DoM were identical to those of "the number of Merge" or "the number of non-terminal nodes" in this case, it was not possible to separate these factors. Taking their simplest condition (lists of unrelated words) as an appropriate reference, the model of the

reference. Overlaid red dots and lines denote the values fitted with the

DoM actually showed a comparable or better goodness of fit for activations in the L. F3t, when compared with their log-fitting models.

#### **FURTHER CONFIRMATION OF HYPOTHESIS III THE EFFECT OF THE SEARCH DISTANCES ON THE DoM**

Neuroimaging and psycholinguistic studies have reported that English sentences with object-relative clauses have higher processing loads than those with subject-relative clauses (Just et al., 1996; Stromswold et al., 1996; Gibson, 2000). To properly parse the relative clauses, the relative pronoun and its antecedent are coindexed; "*whoi*" and "*the boyi*," respectively, in the example shown in **Figure 8**. In a subject-relative clause, a relative pronoun "*whoi*" was displaced from the *subject* position denoted by a trace *ti* (originally, "*the boyi likes the girl*"), while in an objectrelative clause, a relative pronoun was displaced from the *object* position (originally, "*the girl likes the boyi*"). Following the proposal by Hawkins (1999), we assume that the relative pronoun searches the corresponding trace within tree structures of a sentence (see curved arrows in **Figure 8**). In a subject-relative clause, Search ends at the initiation of the verb phrase, while in an objectrelative clause, Search ends *after* a verb appears within a subordinate clause. In accordance with the Search distances for these examples, the DoM would become one unit larger for the objectrelative clause than the subject-relative one. Higher processing

**FIGURE 8 | The DoM domains varied with the Search distances. (A)** A sentence with a subject-relative clause. **(B)** A sentence with an object-relative clause. In these relative clauses, a relative pronoun *whoi* is displaced from its subject or object position denoted by a trace *ti*. A set of red straight arrows corresponds to the DoM domain. The digits shown in red denote the DoM for each node within the domain. Symbols used: S and S', sentence; N, noun phrase; V, verb phrase; *ti*, trace (subscripts denote the same entity).

estimates (digits in red) for the model of the DoM.

loads observed with object-relative clauses are consistent with this inference about the DoM domain.

#### **THE EFFECT OF TASK REQUIREMENTS ON THE DoM**

If Hypothesis III is correct, then the L.F3op/F3t activations can be different in accordance with task requirements, even when the same sentences are presented. In our previous fMRI study, we compared three explicit linguistic tasks with the same set of normal two-word sentences: syntactic decision, semantic decision, and phonological decision tasks (Suzuki and Sakai, 2003). In the syntactic decision task, the participants judged whether or not the presented sentence was syntactically correct, and this judgment required syntactic knowledge about the distinction between transitive and intransitive verbs (e.g., normal sentence, "*yuki-ga tumoru*," "*snow lies* (*on the ground*)"; anomalous sentence, "*yukio tumoru*," "(*something*) *lies snow*"). In the semantic decision task, lexico-semantic knowledge about selectional restrictions was indispensable. In the phonological decision task, phonological knowledge about accent patterns was required. Neither the semantic decision task nor the phonological decision task, both with *implicit* syntactic processing, elicited significant activations in the L. F3op/F3t (−57, 9, 6), which was significantly activated during *explicit* syntactic processing, even by a direct comparison between the syntactic decision task and the other tasks. These results suggest the presence of the DoM domain in accordance with the task requirements of explicit syntactic processing.

#### **THE MIXED EFFECTS OF THE SEARCH DISTANCES AND TASK REQUIREMENTS ON THE DoM**

In another fMRI study, we directly compared syntactic decision and short-term memory tasks (Hashimoto and Sakai, 2002). In this unique paradigm, we visually presented nested sentences that included two proper nouns, two verbs, and one pronoun, in which either verb or pronoun was underlined. After presenting one complete sentence in a phrase-by-phrase manner, paired phrases including an underlined phrase were shown. In one syntactic decision task (SYN-1), participants were required to judge whether the subject of an underlined verb corresponded to the person in paired phrases (**Figure 9A**). In this case, the Search distance was the structural distance between the subject and verb of the same clause. In the other syntactic decision task (SYN-2), the participants were required to judge whether an underlined pronoun was able to refer to the person in paired phrases (**Figure 9B**). In this case, the Search distance was the structural distance between the coindexed noun and pronoun. In these syntactic decision tasks, the Search distance, and consequently the DoM domain, changed dynamically in accordance with the different task requirements, even when the same sentences were presented. The estimate of the resultant DoM was 2 for both cases. In a short-term memory task with a sentence, the participants memorized the linear order of the phrases, and judged whether the left-hand phrase preceded the right-hand one in the original sequence (**Figure 9C**). With such a task requirement, the factor of DoM would become less effective. Indeed, we found that activations in the L. F3op/F3t were equally enhanced in both syntactic decision tasks when compared with the short-term memory task.

#### **CONCLUSIONS**

In this article, we reviewed recent advances in theoretical linguistics and functional neuroimaging in the following respects. First, we provided theoretical discussions about the hierarchical tree structures of sentences, and introduced the two fundamental linguistic operations of Merge and Search. We also presented our hypotheses that the DoM is a key computational concept to properly measure the complexity of tree structures (Hypothesis I), and that the basic frame of the syntactic structure of a given linguistic expression is determined essentially by functional elements, which trigger Merge and Search operations (Hypothesis II). Second, we presented our recent fMRI studies, which have demonstrated that the DoM, together with the number of Search, is indeed a key syntactic factor that accounts for syntax-selective activations in the L. F3op/F3t and L. SMG (Ohta et al., 2013). Moreover, based on the DCM and DTI results, we revealed the significance of the top-down connection from the L. F3op/F3t to L. SMG, suggesting that information about the DoM is transmitted through this specific dorsal pathway. Third, we further hypothesized that the DoM domain changes dynamically in accordance with iterative Merge applications, the Search distances, and/or task requirements (Hypothesis III). We showed that the DoM sufficiently explains activation modulations due to different structures reported in previous fMRI studies (Kinno et al., 2008; Pallier et al., 2011). Finally, we confirmed that Hypothesis III accounts for higher processing loads observed with object-relative clauses, as well as activations in the L. F3op/F3t during explicit syntactic decision tasks, reported in the previous neuroimaging and psycholinguistic studies (Just et al., 1996; Stromswold et al., 1996; Gibson, 2000; Hashimoto and Sakai, 2002; Suzuki and Sakai, 2003). It is likely that the DoM serves as a key computational principle for other human-specific cognitive capacities, such as mathematics and music, both of which can be expressed by hierarchical tree structures. A future investigation into the computational principles of syntax will further deepen our understanding of uniquely human mental faculties.

#### **ACKNOWLEDGMENTS**

We would like to thank R. Kinno, H. Miyashita, K. Iijima, and T. Inubushi for their helpful discussions, N. Komoro for her technical assistance, and H. Matsuda for her administrative assistance. This research was supported by a Core Research for Evolutional Science and Technology (CREST) grant from the Japan Science and Technology Agency (JST), by Grants-in-Aid for Scientific Research (S) (Nos. 20220005) from the Ministry of Education, Culture, Sports, Science and Technology, and by a Grant-in-Aid for Japan Society for the Promotion of Science (JSPS) Fellows (No. 24·8931).

#### **REFERENCES**

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 30 September 2013; paper pending published: 17 October 2013; accepted: 01 December 2013; published online: 18 December 2013.*

*Citation: Ohta S, Fukui N and Sakai KL (2013) Computational principles of syntax in the regions specialized for language: integrating theoretical linguistics and functional neuroimaging. Front. Behav. Neurosci. 7:204. doi: 10.3389/fnbeh.2013.00204*

*This article was submitted to the journal Frontiers in Behavioral Neuroscience. Copyright © 2013 Ohta, Fukui and Sakai. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

### Neural substrates of figurative language during natural speech perception: an fMRI study

#### *Arne Nagels 1 \*, Christina Kauschke2, Judith Schrauf 2, Carin Whitney3, Benjamin Straube1 and Tilo Kircher 1*

*1 Department of Psychiatry and Psychotherapy, Philipps-University Marburg, Marburg, Germany*

*2 Department of Germanic Linguistics, Philipps-University Marburg, Marburg, Germany*

*3 Department of Psychology and York Neuroimaging Centre, University of York, York, UK*

#### *Edited by:*

*Leonid Perlovsky, Harvard University and Air Force Research Laboratory, USA*

#### *Reviewed by:*

*Seth D. Norrholm, Emory University School of Medicine, USA Yueqiang Xue, The University of Tennessee Health Science Center, USA*

#### *\*Correspondence:*

*Arne Nagels, Department of Psychiatry and Psychotherapy, Philipps-University Marburg, Rudolf-Bultmann-Str. 8, 35039 Marburg, Germany e-mail<EMAIL_ADDRESS>Many figurative expressions are fully conventionalized in everyday speech. Regarding the neural basis of figurative language processing, research has predominantly focused on metaphoric expressions in minimal semantic context. It remains unclear in how far metaphoric expressions during continuous text comprehension activate similar neural networks as isolated metaphors. We therefore investigated the processing of similes (figurative language, e.g., "He smokes like a chimney!") occurring in a short story. Sixteen healthy, male, native German speakers listened to similes that came about naturally in a short story, while blood-oxygenation-level-dependent (BOLD) responses were measured with functional magnetic resonance imaging (fMRI). For the event-related analysis, similes were contrasted with non-figurative control sentences (CS). The stimuli differed with respect to figurativeness, while they were matched for frequency of words, number of syllables, plausibility, and comprehensibility. Similes contrasted with CS resulted in enhanced BOLD responses in the left inferior (IFG) and adjacent middle frontal gyrus. Concrete CS as compared to similes activated the bilateral middle temporal gyri as well as the right precuneus and the left middle frontal gyrus (LMFG). Activation of the left IFG for similes in a short story is consistent with results on single sentence metaphor processing. The findings strengthen the importance of the left inferior frontal region in the processing of abstract figurative speech during continuous, ecologically-valid speech comprehension; the processing of concrete semantic contents goes along with a down-regulation of bilateral temporal regions.

 For example, parts of the human body are used in a multitude of figurative expression (head of department, eye of needle, arm of tree, etc.). Thus, figurative speech use goes far beyond the concept of a mere stylistic device and can be seen as an integral part of day-to-day communication. So far, it remains unclear in how far the comprehension of figurative speech draws on additional resources when presented in a naturally-evolving continuous and coherent story. Increases in activation of the left inferior frontal gyrus (IFG) have previously been reported for isolated sentences that contained figurative as opposed to non-figurative elements (Rapp et al., 2004, 2007; Kircher et al., 2007). However, prior context and repeated exposure to figurative speech—as it appears in more natural environments—can have an impact on how we process figures of speech such as similes. Context and conventionalization may facilitate comprehension (for idiom comprehension see Gibbs, 1992) and therefore neural activation of contextualized similes might not differ from non-simile sentences.

The current study focuses on the processing of figurative speech in form of similes such as "The sun is like the eye of heaven" in a natural context without constraining instructions or predetermined cognitive tasks, e.g., decision tasks; asking the participants to press a button, when either abstract or concrete content was presented.

A simile, such as "The sun is like the eye of heaven," can be divided into three different components: (1) the explained element ("the sun"), (2) the explaining element ("the eye of heaven"), (3) and the term of comparison (TOC; "is like"), which connects the two elements in the simile (Leech, 1969). For successful comprehension, the listener needs to refer back to the explained element and identify similarities or common features with the corresponding explaining element (e.g., "He smokes like a chimney!" stresses that someone smokes heavily). The aspect that the two elements have in common is the so-called tertium comparationis, i.e., the third domain involved in a comparison. According to prevailing theories, similes are strongly linked to metaphors, which can be regarded as similes without the TOC (i.e., elliptical similes; cf. Aristotle).

When metaphors are presented, the listener is confronted with a semantic conflict between the explained and explaining element, which needs to be resolved. The initiator of a figurative utterance selects, emphasizes, suppresses, and organizes features of the explained element by applying characteristics of the explaining elements. Thus, a "mental linkage" between the explained element and the corresponding explaining element is required which goes beyond the usual semantic or word-byword analysis (Rapp et al., 2004). The TOC ("like") in similes may facilitate this linking process, since it explicitly points to the comparative nature of the utterance.

Usually, similes or metaphors do not stand alone but are integrated into a speech or text. This context can alter the ease by which meaning is integrated, including processing of figures of speech (Gibbs, 1992). Prior context facilitates the comprehension of idioms when it is consistent with the specific entailments of the idiom (i.e., their conceptual representation): though there might be different ways of expressing anger in an idiomatic way (e.g., "bite your head-off" as opposed to "blow your stack"), it is easier to comprehend idioms whose specific conceptual representation has been primed by previous information e.g., describing anger in a way that refers to anger as "animal behavior" (Nayak and Gibbs, 1990). A coherent and evolving story can provide such information, and participants should therefore understand figurative expressions that are embedded in a story easier than isolated sentences or sentence pairs with little detail (e.g., Rapp et al., 2007; Schmidt and Seger, 2009). Behavioral studies of other complex semantic operations, such as ambiguity processing, have also shown that cognitive resources can be saved when the target item is embedded in semantically coherent context as opposed to isolated or neutral environments [for a review see Simpson (1994)]. Also, the amount of prior, coherent information can facilitate inference making and improve comprehension of non-figurative material (Zwaan and Radvansky, 1998).

A number of functional Magnetic Resonance Imaging (fMRI) studies have investigated the neural correlates of figurative speech mostly in the form of metaphoric sentences (Rapp et al., 2004, 2007; Eviatar and Just, 2006; Kircher et al., 2007; Mashal et al., 2007, 2009; Shibata et al., 2007; Stringaris et al., 2007; Schmidt and Seger, 2009; Desai et al., 2011; Diaz and Hogstrom, 2011; Diaz et al., 2011), and for similes (Shibata et al., 2012). The brain response to similes in a story context, however, remains unexplored, thus far. The results of these previous studies support the involvement of the left lateral prefrontal cortex [for a critical review on the neural basis of metaphor processing see Schmidt et al. (2010)], as a correlate for increased cognitive demand during the comprehension of figurative language. In particular, Rapp et al. (2004) for the first time reported enhanced cortical activation in the left IFG [Brodmann area (BA) 45/47] for metaphor reading as compared to literal sentences. The left IFG is an integral component of the semantic processing network and has been related to executive aspects of meaning retrieval, such as semantic search, retrieval, selection, and integration (e.g., Thompson-Schill et al., 1997; Wagner et al., 2001; Noppeney et al., 2004; Badre et al., 2005; Badre and Wagner, 2007; Bedny et al., 2008; Binder et al., 2009). Moreover, difficult metaphors in comparison to easy metaphors such as "Political success is a house of cards" vs. "Books are treasure chests of information" (Schmidt and Seger, 2009) as well as anomalous metaphors ["Their (financial) capital has a lot of rhythm' (Ahrens et al., 2007)] were found to selectively activate the left IFG. Similarly, conventional metaphors as compared to novel metaphors were found to engage the left IFG, whereas novel metaphors activated the left middle frontal gyrus (LMFG) during a reading paradigm (Mashal et al., 2007). The graded response in prefrontal cortex, particularly the left IFG, suggests that activation correlates with the level of cognitive-semantic resources required for successful performance (e.g., integration effort). This is in accordance with previous semantic retrieval studies of graded difficulty, which showed enhanced left IFG response during tasks of high vs. low executive-semantic demands (Thompson-Schill et al., 1997; Roskies et al., 2001; Wagner et al., 2001; Badre et al., 2005; Snyder et al., 2007; Zempleni et al., 2007; Kuperberg et al., 2008; Nagel et al., 2008; Ruff et al., 2008; Snijders et al., 2009; Whitney et al., 2009a,b). A recent study found activations in the head of the caudate presenting metaphors in a context (Uchiyama et al., 2012). However, the majority of past studies on figurative language comprehension utilized highly controlled sentence reading paradigms with minimal prior information (Rapp et al., 2004, 2007; Eviatar and Just, 2006; Stringaris et al., 2006, 2007; Mashal et al., 2007; Yang et al., 2009). The observed left IFG activations for metaphors might partly reflect executive-semantic processes that are related to the lack of sufficient contextual priming, as it would occur in naturally evolving texts.

The aim of the current study was therefore to analyze the neural responses to simile processing and determine the involvement of the left IFG when similes were presented within a natural, unconstrained short story context. Naturalistic stimuli were successfully investigated in the context of different experimental fMRI paradigms (Hasson et al., 2004; Skipper et al., 2007; Domahs et al., 2012). However, the neural correlates of understanding similes in a short story have not been investigated, so far. We hypothesized left IFG activation during processing of sentences containing similes (e.g., he jumps like a gazelle) vs. literal sentences of comparable frequency, plausibility, comprehensibility, and length. In addition, we expected significant correlations between unfamiliar as well as highly abstract similes and enhanced bloodoxygenation-level-dependent (BOLD) responses in the left frontal region.

#### **MATERIALS AND METHODS**

#### **PARTICIPANTS**

Initially, 19 male subjects took part in the fMRI study. Due to head movement, data of three subjects had to be discarded from further analysis. For the remaining 16 participants movement was minimal as the maximum change in translation and rotation for each participant was less than one voxel size (i.e., 3.5 mm) and less than 1◦, respectively. All 16 participants (mean age = 27.00 years, *SD* = 6*.*65; mean years of education = 14.50 years, *SD* = 1*.*67) were native speakers of German, right-handed according to the Edinburgh Inventory of Handedness (Oldfield, 1971) and showed average or above average verbal IQ as assessed by the German MWT-B multiple choice vocabulary test (Lehrl et al., 1995; mean estimated verbal IQ = 120.06, *SD* = 17*.*16). Subjects with recent substance use or general MRI incompatibility (e.g., metal implants) were excluded. All subjects gave informed consent and were paid 20 Euros for participation in the study. The local ethics committee at RWTH Aachen University approved the study.

#### **STIMULI**

A slightly modified version of the short story "Der Kuli Kimgun" by Dauthendey (1930) was chosen for this study as in Whitney et al. (2009b). Low-frequent or foreign words were substituted by more familiar or high-frequent words. The final version included a total of 3581 words.

In general, short stories are well-structured narratives restricted to a few protagonists and basic narrative events which, together, are ascribed to a single, central conflict. Our story was written from an omniscient perspective, leaving out elaborate descriptions of the character's emotional or mental states. The sequence of events occurs chronologically, which allows the listener to build up a temporally continuous mental representation of the story. The short story chosen for the current fMRI study contained figurative descriptions of events and situations.

For auditory presentation during fMRI, the story was professionally recorded and spoken in a natural way by a trained, male speech therapist. The duration of the story was 23:32 min.

#### **PROCEDURE**

The story was presented via MRI compatible headphones in two successive runs lasting 14:32 and 9:00 min, respectively. Participants were instructed to close their eyes and listen to the story carefully. To make sure that subjects attended to the content, they were informed at the beginning of the experiment about a short interview after the MRI session about the content of the story. Hereby, 10 questions regarding critical episodes of the short story had to be answered.

#### **FIGURATIVE AND CONTROL SENTENCES**

First, 32 similes as well as 50 randomly chosen control sentences (CS) were extracted from the short story independently by two linguists (Christina Kauschke and Judith Schrauf). In a behavioral test, the isolated similes and CS were rated by 20 volunteers, who did not take part in the fMRI study, according to the dimensions "plausibility," "comprehensibility," and "figurativeness." For this purpose, an analogue scale from 1 to 7 was used. Regarding the plausibility rating the instruction was as follows: "Please rate the subsequent sentences according to their plausibility. Very plausible sentences describe ordinary events being easy to follow." Comprehensibility was rated according to the instruction: "Please rate the subsequent sentences according to their comprehensibility. Very comprehensible sentences are those where the meaning can be understood easily and within a very short period of time." Regarding the dimension figurativeness, the rating instruction was formulated: "Sentences can be distinguished with regard to their properties to evoke inner pictures from the content being conveyed. The figurative content can be perceived faster and be grasped more easily in some sentences. Please rate the degree of figurativeness in the following sentences."