Interactionists sometimes use the metaphor of the tango (e.g., Di Paolo et al., 2013). Just as it takes two to tango, one cannot accomplish interaction by oneself. Just as when two people dance the tango, something dynamic is created that neither one could create on one's own. One might think that the metaphor of the tango involves an overly formal structure and that perhaps something more like a free dance form is more appropriate for how the dynamics of interaction work. But most of our lives are lived within social and intersubjective structures (practices and institutions) that do specify how we relate to one another. In some cases this takes the shape of a norm or rule that requires that we mutually recognize our responsibility to the other. Even within such structures, however, even in those that may support totalizing practices, but perhaps short of war, torture, and solitary confinement, one can find the possibility of transcendence in faceto-face relationships. In that interaction there is a mutual expectation of response, and an expectation that we will continue the interaction to some defined or perhaps ill-defined and imperfect end.

Levinas is right about the face, and about its irreducibility; but the other's face is not an absolute alterity, nor does it lead us beyond what we can find in our daily interactions.

On the one hand, in this realm (and clearly in the realm of some institutions) there are no guarantees that we find the kind of transcendence that Levinas talks about. On the other hand, the transcendence that may be found in interactions can open up a vista of possibilities—possibilities of further interactions that have the potential to take me beyond myself, and that make the other incalculably significant, someone I turn away from at my own risk.

### **ACKNOWLEDGMENTS**

This work is supported the Marie-Curie Initial Training Network, "TESIS: toward an Embodied Science of InterSubjectivity" (FP7-PEOPLE-2010-ITN, 264828), and the Humboldt Foundation, Anneliese Maier Research Award. An earlier version of this paper was presented at the Workshop on Other Minds. St. Hilda's College, Oxford University (12 March 2013).

#### **REFERENCES**

**Conflict of Interest Statement:** The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 26 April 2014; accepted: 18 June 2014; published online: 09 July 2014*. *Citation: Gallagher S (2014) In your face: transcendence in embodied interaction. Front. Hum. Neurosci. 8:495. doi: 10.3389/fnhum.2014.00495*

*This article was submitted to the journal Frontiers in Human Neuroscience*. *Copyright © 2014 Gallagher. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms*.

## Development of holistic vs. featural processing in face recognition

## **Kazuyo Nakabayashi 1\* and Chang Hong Liu2**

1 Department of Psychology, University of Hull, Hull, UK

2 Department of Psychology, Faculty of Science and Technology, Bournemouth University, Poole, UK

#### **Edited by:**

Mark A. Williams, Macquarie University, Australia

#### **Reviewed by:**

Benjamin J. Balas, North Dakota State University, USA Lesley Calderwood, University of the West of Scotland, Scotland

#### **\*Correspondence:**

Kazuyo Nakabayashi, Department of Psychology, University of Hull, Hull, HU6 7RX, UK

e-mail<EMAIL_ADDRESS>
According to a classic view developed by Carey and Diamond (1977), young children process faces in a piecemeal fashion before adult-like holistic processing starts to emerge at the age of around 10 years. This is known as the encoding switch hypothesis. Since then, a growing body of studies have challenged the theory. This article will provide a critical appraisal of this literature, followed by an analysis of some more recent developments. We will conclude, quite contrary to the classical view, that holistic processing is not only present in early child development, but could even precede the development of part-based processing.

**Keywords: holistic processing, part processing, developmental face recognition, children, encoding switch, holistic interference, part-whole paradigm**

The encoding-switch hypothesis came from a series of experiments carried out by Carey and Diamond (Carey and Diamond, 1977; Diamond and Carey, 1977; Carey et al., 1980). It was a developmental perspective on Yin's (1969, 1970) finding that face recognition was more adversely affected by inversion than the recognition of non-face stimuli, such as airplanes or snowflakes. Face inversion is commonly known to disrupt the processing of configural information (i.e., the spatial layout of facial features) that is critical for successful face recognition.

Carey and Diamond found an adult-like inversion effect among 10-year-olds. In contrast, recognition performance by 6- and 8-year-olds was poorer than adults only for upright but not for inverted faces. Based on the assumption that inversion disrupts configural processing, Carey and Diamond reasoned that configural processing is only fully developed at around 10 years of age. The lack of an age difference for inverted faces could be due to all ages using a similar piecemeal encoding strategy. The authors suggested that the emergence of the inversion effect at the age of 10 might reflect the development of the ability to represent upright faces using configural information.

However, a number of subsequent studies have challenged the methodologies and theoretical arguments in Carey and Diamond's earlier studies. For instance, Flin (1985) argued that young children's sensitivity to facial orientation could have been masked by a floor effect in Carey and Diamond's study. If the recognition task is made sufficiently sensitive, age difference in children's recognition of inverted faces should be observed. To reduce the likelihood of floor effects, Flin re-examined the inversion effect using a small number of faces with long inspection time in an old-new recognition task. As expected, all 7- to 16-year-olds children in the study showed a typical inversion effect.

In fact, there is a growing body of evidence demonstrating that the inversion effect emerges much earlier than previously suggested. Infants' sensitivity to face orientation has been reported in a number of studies (e.g., Fagan, 1972; Cohen and Cashon, 2001; Turati et al., 2004; Bhatt et al., 2005; Leo and Simion, 2009; Zieber et al., 2013). For instance, 5-month-olds were sensitive to changes to configural information in upright faces, but not in inverted faces (Bhatt et al., 2005). Moreover, 4 month-olds recognized upright faces better than inverted faces when identifying the target face (Turati et al., 2004). Furthermore, even newborns were sensitive to the orientation of a face as their preference for attractive faces disappeared when the faces were inverted (Slater et al., 2000). Newborns, as young as 1- to 3-daysold (Leo and Simion, 2009), exhibited sensitivity to first-order configural information (i.e., the spatial layout of facial features common to all human faces) as they showed a preference for faces as opposed to non-face stimuli of comparable visual complexity (e.g., Johnson and Morton, 1991). These results provide compelling evidence for configural processing at a very early stage of life.

It is worth noting that the presence of the inversion effect at an early age only demonstrates the existence of holistic or configural processing. However, this does not mean that such processing is already adult-like or that it will not go through further development. In addition, as prior studies used different tasks (e.g., preferential looking vs. a standard recognition task) tailored to the age of participants, it would be difficult to assess whether the observed effects across studies reflect identical holistic/configural processing.

Moreover, some researchers have pointed out that featural and configural manipulations may not differ in a fundamental way. For instance, Riesenhuber et al. (2004) hypothesized that the inversion effect on configural processing could have resulted from separating faces with configural and featural transformations into different blocks. Such a design could encourage a specific recognition strategy for detecting one type of change, rather than provoking generic face recognition strategies. As predicted, they found no difference in the recognition of inverted configural and featural changes when they were presented in the same block. Others have also questioned the featural and configural distinction (e.g., Sekuler et al., 2004; Yovel and Duchaine, 2006). It would be interesting to establish whether these results could be replicated in children.

Another piece of key evidence for the encoding switch hypothesis comes from the paraphernalia-to-fool paradigm in which Carey and Diamond (1977) manipulated facial expressions and paraphernalia (e.g., hat, hairstyle, and glasses). Here, children were first presented with a target face. Next the target was paired with a distractor face. The task was to identify which face in the pair was the target. The target varied either in facial expression (no expression or expression) or paraphernalia (paraphernalia removed or added), whereas the distractor either wore the same paraphernalia or posed the same expression as the target during inspection.

The finding was that 6- and 8-year-olds were highly susceptible to errors when the distractor face wore the same paraphernalia as the target. This tendency declined markedly among 10 year-olds. Young children's reliance on paraphernalia in making identity judgments was also demonstrated when the pair of images were presented simultaneously (Diamond and Carey, 1977). However, when faces were familiar (e.g., classmates), children were able to ignore paraphernalia but used cues that were diagnostic to identity. Diamond and Carey argued that young children represent unfamiliar faces in terms of isolated features when making identity judgements. However, children at the age of around 10 years start to show an adult level of capacity for efficient processing of configural information of a face. The authors suggested that the developmental changes could be due to increased experience with faces in general, but also it may be a result of the maturation of the right hemisphere for dealing configural representations of faces.

Subsequently, Lundy et al. (2001) suggested that the inconsistencies in the literature could be explained by stimulus size. Studies reporting holistic processing in young children used small images (e.g., Flin, 1985; Baenninger, 1994) while others arguing for a change from part to whole processing used larger images (e.g., Schwarzer, 2000). Young children have a tendency to process a stimulus as an undifferentiated whole when the overall image can be perceived in a single glance. Moreover, young children's differentiation of stimulus components may be related to a limitation in the ability to narrow the focus of attention (see Enns and Girgus, 1985). Hence, the influence of paraphernalia is likely to vary depending on stimulus size as well as age related attentional limitations.

Based on these considerations, Lundy et al. conducted a paraphernalia-to-fool study to examine the effects of visual angle size on face recognition across 3-, 7-, and 10-year-olds. As predicted, 10-year-olds performed better than the two younger groups who showed equivalent performance. The size of a visual angle influenced only 7-year-olds' performance, with a significant improvement with a large visual angle. However, unlike Diamond and Carey, Lundy et al., did not treat the children's susceptibility to paraphernalia as evidence for piecemeal processing. Rather, they interpreted it as evidence for younger children's tendency to process a small image more holistically. According to this, the younger children in Carey and Diamond's study treated faces and paraphernalia as undifferentiated wholes whereby paraphernalia is processed as part of the face. Clearly, this was an important shift from Carey and Diamond's interpretation of the same effect. Whilst Carey and Diamond considered holistic processing as the ability to separate irrelevant features from relevant facial information, Lundy et al. treated the effects of irrelevant features as evidence for holistic processing.

Although paraphernalia (e.g., a change of a hair style or addition of glasses) can also affect adults' face recognition, it does not directly influence their configural or featural processing (Righi et al., 2012). It is because adults tend to include paraphernalia as part of a face, which disrupts encoding of relevant holistic information. Consistent with Lundy et al., Freire and Lee's (2001) paraphernalia study found that children have the capacity to process configural information by 4 years of age. Nevertheless, misleading paraphernalia could still hamper recognition as children's memory is susceptible to superfluous information. Therefore, distracting effects of paraphernalia do not necessarily provide evidence for a lack of configural processing as they could be due to a limitation in more general cognitive abilities.

Moreover, as Baenninger (1994) pointed out, the results from Carey and Diamond's study could have been derived from the distinctiveness of paraphernalia that distracted children's attention away from relevant information of a face. Furthermore, Flin (1985) argues that distinctive paraphernalia would be of greater perceptual salience than relevant facial information when targetdistractor similarity is high, making children more susceptible to distracting effects of paraphernalia. Flin showed that when facial information was made salient (target-distractor dissimilar), 6-year-olds accurately judged the identity while ignoring paraphernalia. However, when target-distractor similarly was high, 4- and 6-year-olds made their identify judgements based on paraphernalia. Flin, therefore, suggests that Diamond and Carey's (1977) findings could be explained by the task difficulty in that young children ignored relevant facial information while attending only to salient paraphernalia cues. Subsequently, Carey and Diamond (1994) modified their original encoding hypothesis by stating that even young children process faces holistically.

Studies using the composite-face paradigm also support holistic processing among young children (e.g., Tanaka et al., 1998; Pellicano and Rhodes, 2003; see also Carey and Diamond, 1994; Carey, 1996; for a modification of their original theory). The composite effect was first reported by Young et al. (1987), who studied how adult face recognition could be affected when the top half of a face was combined with the bottom half of another face. Typically, participants had a greater difficulty with identifying the top half of the composite face when the two halves were aligned than misaligned. This provided strong evidence that adults tend to perceive the composite face as an undifferentiated whole (see Rossion, 2013 for recent review of the composite effect).

The composite effect has later been demonstrated among 6-year-olds (Carey and Diamond, 1994) and preschool children (de Heering et al., 2007; Cassia et al., 2009). Like adults, children's identification of the top part of a composite face was better when the face was misaligned than aligned. In fact, Susilo et al. (2009) even reported a lager composite effect for 8- to 13-year-olds than adults when the stimuli were child faces. Subsequently, Turati et al. (2010) showed that infants as young as 3-month-olds exhibit the composite effect, indicating that they are capable of processing faces holistically.

Overall the literature appears to show no fundamental difference in the way children and adults use holistic information to perceive, store, and recognize faces (e.g., Tanaka et al., 1998; de Heering et al., 2007; Mondlock et al., 2007). This leaves an unanswered question: if holistic processing does not separate children's face processing from that of adults, then what might explain the difference in their recognition performance. We suggest that the answer may partly lie in the inverse of the original encoding switch hypothesis. That is, it may be the proficiency of piecemeal, rather than holistic, processing that takes longer to develop. The evidence for this comes mainly from a holistic interference effect on facial part recognition.

The holistic interference effect was studied in a variant of the part-whole paradigm first developed by Tanaka et al. (Tanaka and Farah, 1993; Tanaka and Sengco, 1997). Unlike the earlier paraphernalia studies that examined the effects of an addition or elimination of non-facial cues (e.g., glasses or hat) which are not part of facial components on identify judgments, the partwhole paradigm directly measures holistic vs. featural processing by examining part recognition in or out of the facial context. Tanaka et al. discovered that adults' part recognition was better in a whole face than in isolation. In a subsequent study, Tanaka et al. showed that by 6 years of age, children displayed similar holistic effects as adults (Tanaka et al., 1998). Hence children's face processing does not change from a part to holistic based strategy from this age. Pellicano and Rhodes (2003) later extended the finding to 4-year-olds by showing that children at this age remembered face parts better when tested in a whole face than in isolation.

Leder and Carbon (2005) subsequently provided a fresh approach to the part-whole paradigm using adults. They pointed out that empirical evidence for the whole face advantage had emerged when whole faces were learnt, leaving the gap in our knowledge as to whether the same advantage would still arise when facial parts are learned without the context of a whole face. An underlying assumption was that if learning parts imposes a strict part-based representation, an additional context at test might be ignored. Hence, performance in whole and part conditions may be comparable.

However, the authors found that when isolated parts were learned, presenting the parts in a whole face at test impaired part recognition. This finding was also seen even when participants knew which a critical part was, which implies that it was the unexpected context that hindered part recognition, but not uncertainly about the critical part *per se*. These results suggest that the interaction between facial features and the whole plays a key role in adult face processing. Leder and Carbon argue that the holistic interference is an essence of holistic processing because it demonstrates how difficult it is to ignore irrelevant facial information in a whole face.

Nakabayashi and Liu (2013) have subsequently investigated the holistic interference effect among children. They found that the effect was strongest among the 6-year-olds relative to 9–10-year-olds or adults. Participants in their study judged whether a sequentially presented pair of probe and target eyes were of the same child in four conditions: (1) both probe and test were isolated eyes (part-part); (2) probe was isolated eyes but tested in a whole face (part-whole); (3) probe was a whole face and tested with a part (whole-part); (4) both probe and test eyes were presented in a whole face (whole-whole). The results showed developmental differences when a part was presented in a whole face (i.e., part-whole, whole-part, and whole-whole), with 6-year-olds showing poorer part recognition performance than 9–10-year-olds or adults. In the part-part condition, 6-year-olds were able to identify the parts as well as the two older groups. These findings suggest that holistic processing is already present at 6 years of age. More importantly, the results demonstrate that it is the ability to inhibit the influence of this holistic processing on part recognition that seems to require a longer period of development.

Nakabayashi and Liu suggest that the developmental differences in part recognition may reflect differences in general inhibitory abilities. For instance, research using the go/no-go task procedure reveals that children's ability to inhibit an automatic response continues to develop throughout childhood (e.g., Dowsett and Livesey, 2000; Brocki and Bohlin, 2004; Berwid et al., 2005). The go/no go task typically requires a key-press response to frequently presented "go" stimuli while inhibiting responses to "no-go" stimuli.  As children get older, they become more able to inhibit their responses at an earlier stage of responding. Perhaps the lack of maturity in this ability among 6-year-olds in Nakabayashi and Liu's study led to their susceptibility to holistic interference. Their 6-year-olds were less able to inhibit the tendency to process irrelevant information as part of a face.

Evidence for a slower development of piecemeal, as opposed to holistic, processing can also be found in a study by Liu et al. (2013), who investigated the development of facial feature processing in 8–9-, 13–14-year-olds, and adults. In one experiment, participants learnt whole faces, followed by a standard old-new recognition test whereby they identified one of the following test items: eyes; nose; mouth; inner face; outer face; or whole face. The results showed no age difference in the recognition of whole faces, but unlike adults and 13–14-year-olds, 8–9-year-olds were unable to distinguish between old and new facial features. More importantly, when part recognition was preceded by whole learning even 13–14-year-olds did not seem to naturally encode and recognize isolated parts. Based on these findings, the authors suggest that the processing of isolated facial regions differs in its developmental course from that of holistic processing, and that holistic processing may be more dominant before adulthood relative to featural and configural processing.

Results reported in these studies call for a more radical revision of the classical encoding switch hypothesis. The reverse of it may be a more accurate description of the developmental trajectory. It seems that holistic processing is a default mode of processing from an early age as there appears no qualitative difference in the way young children and adults use holistic information. The real developmental differences seem to lie in the ability to successfully encode and extract a critical part from a whole face while ignoring irrelevant information. It is this successful execution of piecemeal processing that seems to take longer to develop to an adult level.

Finally, we should caution that although this review has focused on the development of face recognition, we are not making any assumption that the developmental pattern is specific to face perception. As few studies have made a direct comparison between face and non-face visual processing in the developmental literature, it would be difficult to ascertain whether the development of face recognition has its own unique course. However, some researchers have examined whether certain mechanisms are unique to faces by using both faces and non-face objects (e.g., Yovel and Duchaine, 2006). This line of enquiry will be important for future research because it addresses the question of whether the current knowledge about face recognition development is domain specific or domain general.

#### **ACKNOWLEDGMENTS**

We would like to thank Toby Lloyd-Jones and the two reviewers for their helpful comments on the manuscript.

#### **REFERENCES**

Tanaka, J. W., and Sengco, J. A. (1997). Features and their configuration in face recognition. *Mem. Cognit.* 25, 583–592. doi: 10.3758/bf03211301

Zieber, N., Kangas, A., Hock, A., Hayden, A., Collins, R., Beda, H., et al. (2013). Perceptual specialization and configural face processing in infancy. *J. Exp. Child Psychol.* 116, 625–639. doi: 10.1016/j.jecp.2013.07.007

**Conflict of Interest Statement**: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 27 April 2014; accepted: 30 September 2014; published online: 20 October 2014*.

*Citation: Nakabayashi K and Liu CH (2014) Development of holistic vs. featural processing in face recognition. Front. Hum. Neurosci. 8:831. doi: 10.3389/fnhum. 2014.00831*

*This article was submitted to the journal Frontiers in Human Neuroscience*.

*Copyright © 2014 Nakabayashi and Liu. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution and reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms*.

## Face recognition: a model specific ability

## **Jeremy B. Wilmer 1\*, Laura T. Germine2 and Ken Nakayama3**

1 Department of Psychology, Wellesley College, Wellesley, MA, USA

2 Psychiatric & Neurodevelopmental Genetics Unit, Massachusetts General Hospital, Boston, MA, USA

3 Department of Psychology, Harvard University, Cambridge, MA, USA

#### **Edited by:**

Mark A. Williams, Macquarie University, Australia

#### **Reviewed by:**

Fiona N. Newell, Trinity College Dublin, Ireland Roberta Daini, Università degli studi di Milano - Bicocca, Italy

#### **\*Correspondence:**

Jeremy B. Wilmer, Department of Psychology, Wellesley College, 106 Central Street, Wellesley, MA 02481, USA e-mail<EMAIL_ADDRESS>
In our everyday lives, we view it as a matter of course that different people are good at different things. It can be surprising, in this context, to learn that most of what is known about cognitive ability variation across individuals concerns the broadest of all cognitive abilities; an ability referred to as general intelligence, general mental ability, or just g. In contrast, our knowledge of specific abilities, those that correlate little with g, is severely constrained. Here, we draw upon our experience investigating an exceptionally specific ability, face recognition, to make the case that many specific abilities could easily have been missed. In making this case, we derive key insights from earlier false starts in the measurement of face recognition's variation across individuals, and we highlight the convergence of factors that enabled the recent discovery that this variation is specific. We propose that the case of face recognition ability illustrates a set of tools and perspectives that could accelerate fruitful work on specific cognitive abilities. By revealing relatively independent dimensions of human ability, such work would enhance our capacity to understand the uniqueness of individual minds.

**Keywords: specific ability, individual differences, face recognition, intelligence, IQ, multiple intelligences, cambridge face memory test, generalist gene**

## **INTRODUCTION**

Most of what we know about human cognitive ability—and by ability, we mean *variation across individuals* in performance or potential—concerns *g*. *g* is the single, broad ability that has been observed to account for a large portion of the variation in any sufficiently large and diverse battery of cognitive tests (Spearman, 1904; Carroll, 1993; Jensen, 1998). Studies of *g* (and of highly g-related tests) have long dominated the human abilities literature, producing the bulk of known genetic (Plomin et al., 2013), neural (Deary et al., 2010), clinical (American Psychiatric Association, 2013), academic (Neisser et al., 1996), professional (Schmidt and Hunter, 2004), and personal (Jensen, 1998; Deary, 2012) correlates of human abilities. In contrast to the literature on *g*, the parallel literature on specific abilities, those abilities that correlate little with *g*, is tiny (Neisser et al., 1996; Jensen, 1998; Schmidt and Hunter, 2004; Deary, 2012).

Why do we know so little about specific abilities? Lack of interest cannot account for this limited knowledge. Theories hypothesizing consequential specific ability dimensions have enjoyed wild popularity in fields as diverse as education and business, as well as in the media (Goleman, 1995; Gardner, 2006). Another possible explanation for the lack of knowledge about specific abilities is that they simply do not play a very important role in our lives (Schmidt and Hunter, 2004). Indeed, upon cursory examination, the sheer size and apparent comprehensiveness of the human abilities literature make it difficult to imagine that important specific abilities could have been missed.

We will argue here, nevertheless, that it is too early to write off specific abilities as unimportant or inconsequential. We propose, on the contrary, that the lack of emphasis on specific abilities is an artifact of (a) traditional test development procedures in the human abilities literature and (b) the bottleneck of human subjects testing; and we suggest that recent methodological advances and discoveries could be harnessed to fundamentally rebalance our broad understanding of human talent toward a greater appreciation of specific abilities. We will base this argument on insights we have gained from researching face recognition ability. Work in our labs and others has recently established face recognition as an exceptionally specific ability (Wilmer et al., 2010, 2012; Wilhelm et al., 2010; Davis et al., 2011; Hildebrandt et al., 2011; Peterson and Miller, 2012; McGugin et al., 2012; Palermo et al., 2013).

To be clear, when we use the terms *specific*, *specific ability*, *specificity*, or *specifically* in this paper, we use them in their classic human variation sense to refer to performance that correlates little across individuals with general intelligence (Spearman, 1904; Carroll, 1993; Jensen, 1998). The term *specific* is frequently used differently in the experimental psychology and human neuroscience literatures. In these literatures, it refers neither to individual differences nor to general intelligence, but it is rather used as a shorthand for domain or process specificity (Gazzaniga, 2004). While studies of individual differences can and do effectively tackle questions of domain and process specificity (Wilmer, 2008), here we focus on the more basic question of whether an ability dissociates from (is specific relative to) general intelligence.

In the first section below, we briefly review the evidence that face recognition varies specifically across individuals. In the second section, we examine two illuminating false starts whereby well-resourced efforts to measure face recognition ability misinterpreted promising evidence for its specificity. These false starts demonstrate how easily a specific ability can be overlooked. In the third and final section, we identify three key factors that fueled the recent discovery that face recognition ability is specific and that, we believe, could likewise fuel the discovery of further specific abilities. These factors were: incorporation of priorities, discoveries, and techniques from experimental psychology and cognitive neuroscience; the development and validation of an excellent test; and a powerful internet-enabled Citizen Science approach to investigating human variation.

## **FACE RECOGNITION VARIES SPECIFICALLY ACROSS INDIVIDUALS**

The core evidence that face recognition varies specifically across individuals comes from two complementary sources. The first source is face recognition's dissociations from other, more general cognitive abilities; the second, equally-critical source is the robust associations observed among assessments that measure face recognition ability in very different ways.

Face recognition, as measured by the widely-used Cambridge Face Memory Test (CFMT; Duchaine and Nakayama, 2006), dissociates strongly from more general abilities. It dissociates almost completely from standardized IQ tests. To date, its mean reported correlation with such IQ tests, weighted by sample size and corrected for range restriction in the IQ tests, is 0.01 (Davis et al., 2011; Peterson and Miller, 2012; Palermo et al., 2013). Face recognition, as measured by the CFMT, also dissociates surprisingly strongly from other recognition abilities. It shares a mere 3% of its variation with the recognition of word pairs (*n* = 3003; 95% CIs 2–4%; Wilmer et al., 2010, 2012); and even *within* the realm of visual recognition, it shares only 7% of its variation with the recognition of abstract art images (*n* = 4475; 95% CIs 5–8%; Wilmer et al., 2010, 2012).

These pervasive dissociations from other abilities are not a result of poor measurement. Not only is the CFMT as reliable per unit time as the most widely-used IQ test (Wechsler, 2008; Wilmer et al., 2010), but it correlates well with tests that measure face identity processing in quite different ways. Two such tests are the Cambridge Face Perception Test (CFPT), which correlates 0.60 with the CFMT (*n* = 124; Bowles et al., 2009), and the Cambridge Famous Faces Memory Test (CFFMT), which correlates 0.52 with the CFMT (*n* = 1219; Wilmer et al., 2010, 2012).

The CFPT and CFFMT differ from the CFMT in multiple ways. The CFMT assesses one's ability to memorize a set of previously unfamiliar faces and then, shortly thereafter, recognize them among distractors. The CFPT, in contrast, assesses one's ability to *rank* several faces by the similarity of their identity to a *simultaneously-viewed* "exemplar" face. The CFFMT, in further contrast, assesses one's ability to *attach names or other identifying information to* celebrity faces *learned haphazardly over one's lifetime*.

The CFMT, CFPT, and CFFMT thus differ starkly in both the task being performed (from visual matching in the CFPT to recognition in the CFMT to recall in the CFFMT) and the duration over which faces must be remembered (from milliseconds in the CFPT to minutes in the CFMT to years or decades in the CFFMT), making their robust intercorrelations a powerful demonstration of valid measurement.

Finally, and perhaps most impressive of all, the CFMT correlates 0.37 with a person's self-rating with the single statement "I can recognize famous celebrities in photos or on TV" (*n* = 190; 95% CI 0.24–0.49; Wilmer et al., 2010). This is substantially larger than the average 0.15 correlation found between objective and self-report measures of memory abilities in a major metaanalysis of 24,897 individuals tested across 169 studies (Beaudoin and Desrichard, 2011).

Associations like these between CFMT and CFPT, CFFMT, and self-reported recognition ability critically distinguish specificity from invalid measurement. As we will see below, such associations, as a counterpoint to face recognition's persistent dissociations, were the missing piece in prior face recognition ability research.

In addition to being specific, human variation in face recognition is highly heritable (Wilmer et al., 2010). This combination of specificity and heritability is rare (Wilmer et al., 2010). Indeed, so consistently has specificity traded off against heritability in past research that a recent behavioral genetic theory, the "generalist gene" theory, posited that most or all cognitive variation results from the same set of genes (Kovas and Plomin, 2006). A major exception to the generalist gene theory (Wilmer et al., 2010; Plomin et al., 2013), face recognition's heritability demonstrates that different sets of genes contribute independently to human cognitive ability. Given the example of face recognition, it is worth considering not only how many other specific abilities exist, but also whether any of them are as strongly heritable as face recognition.

In sum, face recognition, at least when measured via the CFMT, is exceptionally specific. Moreover, it is rare among specific abilities for its high heritability. In the next section, by examining two earlier false starts in the valid measurement of face recognition ability, we illustrate barriers to the discovery of its specificity that could bear importantly on the search for further specific abilities.

## **TWO FALSE STARTS IN THE MEASUREMENT OF FACE RECOGNITION ABILITY**

In this section, we will recount two major efforts to assess face recognition ability. These efforts, begun nearly 70 years apart, are among history's most concerted efforts to measure any social ability (Kihlstrom and Cantor, 2000; Wilmer et al., 2012). In each case, initial promising evidence for face recognition's specificity was misinterpreted as invalid measurement, and development of the test in question was abandoned. These missed opportunities to examine the specific ability of face recognition seem unlikely to us to be isolated examples. The lessons learned from these missed opportunities may therefore provide valuable information on where to search for additional specific abilities.

The first false start in the measurement of face recognition ability involved the George Washington Social Intelligence Test (GWSIT), developed in the late 1920s (Hunt, 1928). The GWSIT consisted of six subtests, two of which involved faces. A face recognition subtest assessed the ability to learn the names for a set of twelve novel target faces; then, presented with a larger group of faces, one was required to pick out the target faces and recall their names. The second subtest involving faces assessed the ability to label the mental states of faces based on their expression. The remaining four subtests verbally assessed other aspects of social knowledge and social judgment (Hunt, 1928).

The initial validation study for the GWSIT clearly showed that, though none of its subtests correlated particularly highly with each other (maximum *r* = 0.44), the face recognition subtest dissociated most strongly of all from the other subtests (mean *r* = 0.22; Hunt, 1928). On the basis of these dissociations, as well as a dissociation from a measure of general intelligence, Hunt (1928) presciently suggested that "the special ability of being able to recognize [faces] is relatively independent of pure 'brains"'.

What happened next is telling. Surprisingly, at least in hindsight, the promising evidence that the GWSIT provided for face recognition's specificity was not eagerly pursued. Quite to the contrary, the GWSIT was roundly criticized for failing to measure a unitary social ability. That is, it was criticized (a) because its subtests dissociated strongly from each other; and (b) because the small amount of overlap between its subtests was ultimately attributed to general intelligence (Thorndike, 1936; Thorndike and Stein, 1937). On this basis, the GWSIT rapidly fell out of favor as a research instrument (Kihlstrom and Cantor, 2000). Moreover, a mere two decades after it was introduced, the GWSIT was cited, in what would soon become the classic paper on test validity, as the classic example of an invalid test (Campbell and Fiske, 1959). In sum, far from inspiring further research, face recognition's clear and persistent dissociations from other abilities were the core inspiration for the rejection of the GWSIT as a valid ability measure.

Lest one be tempted to write off the rejection of the GWSIT as an isolated historical event, let us move forward nearly 70 years to a second, remarkably similar story. This story involved the third edition of the highly influential Wechsler Memory Scale (WMS-III), introduced in 1997 (Wechsler, 1997). The WMS-III added, for the first time in the WMS's history, a face recognition subtest. This subtest assessed the ability to memorize a set of faces and then classify a subsequent series of faces as "old" (seen before) vs. "new" (not seen before) (Wechsler, 1997). As with the GWSIT, the WMS-III's face recognition subtest dissociated persistently from other measures. These other measures included the WMS's own verbal and visual recognition subtests (Wechsler, 1997; Millis et al., 1999; Holdnack and Delis, 2004). Again, such dissociations were viewed as a liability rather than a virtue. The face recognition subtest was criticized for its dissociations (Millis et al., 1999; Holdnack and Delis, 2004), and it was dropped from the WMS-IV (Wechsler, 2009).

Seven decades apart, the story was the same. Face recognition's dissociations fueled a presumption of invalid measurement and an abandonment of measures, with remarkably little work aimed at disentangling specificity from invalid measurement by examining correlations across diverse measures of face recognition ability. The persistence with which face recognition was overlooked in these cases illustrates a blind spot for specificity that we believe is broadly characteristic of traditional test development practices in the human ability literature.

## **FACE RECOGNITION AS A MODEL IN THE SEARCH FOR FURTHER SPECIFIC ABILITIES**

We will now discuss three key factors that fueled the recent discovery that face recognition varies specifically across individuals, and that could plausibly fuel the discovery of further specifically varying abilities. These factors were: incorporation of priorities, discoveries, and techniques from experimental psychology and cognitive neuroscience; the development and validation of an excellent test; and a powerful internet-enabled Citizen Science approach to investigating human variation.

In contrast to the human ability literature's capacity to overlook dissociations, the cognitive neuroscience and experimental psychology literatures have, throughout their history, actively sought out dissociations. A remarkable aspect of the WMS story is that its face recognition subtest was introduced the same year, 1997, as major reports of face-selective activation in the human fusiform face area (FFA; see also Sergent et al., 1992; Kanwisher et al., 1997; McCarthy et al., 1997). Simultaneously, the WMS's dissociations inspired disappointment and rejection, while the FFA's dissociations inspired excitement and followup work. Indeed, the FFA's dissociations, along with other key neural and cognitive dissociations, have played a central role in solidifying the status of face processing as a major model system in studies of mind and brain. Such different reactions to evidence for dissociation are instructive when considering where to look for specific abilities. Perhaps equally valuable inspiration on where to look could be derived from the orphan tests of human abilities research (tests that were reliable yet abandoned due to their persistent dissociations) and the core dissociable model systems of cognitive neuroscience and experimental psychology.

As illustrative examples of ability domains that could plausibly contain additional specific abilities, consider social cognition, navigation, and dynamic visual perception. In the case of social cognition, the dissociations produced by the GWSIT raise the possibility that additional specific social abilities may exist (Hunt, 1928; Kihlstrom and Cantor, 2000; see also Mayer et al., 2008), and several aspects of social cognition, including theory of mind and joint attention, have been associated with distinct neural areas (Saxe, 2006). Navigation and dynamic visual perception, too, each involve well-defined neural areas (Epstein and Kanwisher, 1998; Newsome and Pare, 1988), and appear to dissociate from at least some general abilities (Hegarty et al., 2006; Wilmer and Nakayama, 2007). These are merely a few illustrative examples of the many domains in which orphan tests and/or functional or neural dissociations exist. We expect that there exist tens or hundreds of additional areas were such evidence is compelling enough to consider initiating a search for specific abilities.

The recent discovery of face recognition's specificity owes much to the careful development of a single, high-quality test: the CFMT (Duchaine and Nakayama, 2006). Ironically, it was the cognitive neuroscience and experimental psychology literatures, not the human abilities literature, that inspired the development of the CFMT (Duchaine and Nakayama, 2006). The CFMT's development drew primarily from three scientific areas. First, it drew from the stimulus-control techniques of visual psychophysics to produce well-controlled stimuli. Second, it drew from the dissociation-focused manipulations of cognitive neuroscience and experimental psychology to achieve an effective isolation of face processing mechanisms. Third, it drew from the practical test design methods of patient-based neuropsychological testing to minimize its demands on test-takers' general cognitive resources, including their capacity to attend, interpret, and problem-solve (taxing such general resources likely increases a test's reliance on general intellectual ability) (Duchaine and Nakayama, 2006).

The exceptional specificity of face recognition, as measured by the CFMT, is a case study in the value of incorporating the priorities, discoveries, and techniques of cognitive neuroscience and experimental psychology into efforts to measure human ability. Meaningful progress in the isolation of specific abilities, however, additionally requires a combination of rigorous psychometrics and access to the large, diverse samples of participants that enable iterative development, validation, and norming of high-quality tests.

Fortunately, we live at a time when the internet has opened up unprecedented opportunities for testing large samples. Resources such as Amazon Mechanical Turk®–an online clearing-house for small jobs where psychological research is increasingly conducted—enable the rapid recruitment and testing of large samples. Our own web-based work on face recognition and other abilities has been powered by our *Citizen Science* project TestMy-Brain.org (Germine et al., 2012). As with other citizen science initiatives (Bonney et al., 2009), TestMyBrain.org seeks to actively collaborate with the general public to answer scientific questions. At TestMyBrain.org, we make high-quality tests freely available via the web, and participants complete these tests to learn about themselves. We then aggregate data across participants to further refine the tests we offer and to answer scientific questions. Due to high public interest in self-discovery, the ease of participation across demographic groups, and the near-zero incremental cost of recruiting and testing each additional participant, our studies of face recognition have been able to rapidly collect high-quality data from many thousands of individuals of varied age, sex, occupation, and socioeconomic status (Wilmer et al., 2010, 2012; Germine et al., 2011a,b, 2012). Citizen science projects like Test-MyBrain.org, as well as other large-scale internet-based testing projects like Mechanical Turk, provide the necessary throughput to capture specific abilities and examine their importance in our lives.

#### **CONCLUSIONS**

Here, we have examined face recognition as a model specific ability. First, we reviewed the recent work that documents face recognition's specificity. Second, we recounted two major false starts in the measurement of face recognition ability. These false starts reveal a capacity for specific abilities not only to be missed, but indeed, to be actively avoided by major test development efforts. Third, we discussed three key factors that contributed to the discovery that face recognition ability is specific and that, we believe, could serve as a compass for the discovery of further specific abilities. These factors were: incorporation of priorities, discoveries, and techniques from experimental psychology and cognitive neuroscience; the development and validation of an excellent test; and a powerful internet-enabled Citizen Science approach to investigating human variation. We suggest that the time is right for a renewed effort to investigate specific abilities, and that this effort can be guided by the model example of face recognition ability. By revealing relatively independent dimensions of human ability, such work would enhance our capacity to understand the uniqueness of individual minds.

### **ACKNOWLEDGMENTS**

This work was supported by a Brachman Hoffman Fellowship to Jeremy B. Wilmer.

## **REFERENCES**

Germine, L., Nakayama, K., Duchaine, B. C., Chabris, C. F., Chatterjee, G., and Wilmer, J. B. (2012). Is the Web as good as the lab? Comparable performance from Web and lab in cognitive/perceptual experiments. *Psychon. Bull. Rev.* 19, 847–857. doi: 10.3758/s13423-012-0296-9

Goleman, D. (1995). *Emotional Intelligence.* New York: Bantam Books.

**Conflict of Interest Statement**: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 09 May 2014; accepted: 10 September 2014; published online: 10 October 2014*.

*Citation: Wilmer JB, Germine LT and Nakayama K (2014) Face recognition: a model specific ability. Front. Hum. Neurosci. 8:769. doi: 10.3389/fnhum.2014.00769 This article was submitted to the journal Frontiers in Human Neuroscience*.

*Copyright © 2014 Wilmer, Germine and Nakayama. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution and reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms*.

## A parametric study of fear generalization to faces and non-face objects: relationship to discrimination thresholds

#### **Daphne J. Holt 1,2,3\*, Emily A. Boeke1 , Rick P. F. Wolthusen1,4 , Shahin Nasr 3,5 , Mohammed R. Milad1,2 and Roger B. H. Tootell 2,3,5**

1 The Department of Psychiatry, Massachusetts General Hospital, Boston, MA, USA

2 Harvard Medical School, Boston, MA, USA

3 The Athinoula A. Martinos Center for Biomedical Imaging, Charlestown, MA, USA

4 Department of Child and Adolescent Psychiatry, Faculty of Medicine Carl Gustav Carus of the Technische Universität Dresden, Dresden, Germany

5 The Department of Radiology, Massachusetts General Hospital, Boston, MA, USA

#### **Edited by:**

Aina Puce, Indiana University, USA

#### **Reviewed by:**

Andrew Jahn, Indiana University, USA

Assaf Harel, National Institutes of Health, USA

#### **\*Correspondence:**

Daphne J. Holt, The Department of Psychiatry, Massachusetts General Hospital, 149 13*th* St., Room 2608, Charlestown, Boston, MA 02129, USA e-mail<EMAIL_ADDRESS>
Fear generalization is the production of fear responses to a stimulus that is similar—but not identical—to a threatening stimulus. Although prior studies have found that fear generalization magnitudes are qualitatively related to the degree of perceptual similarity to the threatening stimulus, the precise relationship between these two functions has not been measured systematically. Also, it remains unknown whether fear generalization mechanisms differ for social and non-social information. To examine these questions, we measured perceptual discrimination and fear generalization in the same subjects, using images of human faces and non-face control stimuli ("blobs") that were perceptually matched to the faces. First, each subject's ability to discriminate between pairs of faces or blobs was measured. Each subject then underwent a Pavlovian fear conditioning procedure, in which each of the paired conditioned stimuli (CS) were either followed (CS+) or not followed (CS−) by a shock. Skin conductance responses (SCRs) were also measured. Subjects were then presented with the CS+, CS− and five levels of a CS+-to-CS− morph continuum between the paired stimuli, which were identified based on individual discrimination thresholds. Finally, subjects rated the likelihood that each stimulus had been followed by a shock. Subjects showed both autonomic (SCR-based) and conscious (ratings-based) fear responses to morphs that they could not discriminate from the CS+ (generalization). For both faces and non-face objects, fear generalization was not found above discrimination thresholds. However, subjects exhibited greater fear generalization in the shock likelihood ratings compared to the SCRs, particularly for faces. These findings reveal that autonomic threat detection mechanisms in humans are highly sensitive to small perceptual differences between stimuli. Also, the conscious evaluation of threat shows broader generalization than autonomic responses, biased towards labeling a stimulus as threatening.

**Keywords: fear, faces, emotion, learning, generalization, perception**

## **INTRODUCTION**

Fear generalization is an adaptive process in which a fear response occurs to stimuli that are similar to a threatening stimulus (Lissek et al., 2008; Hajcak et al., 2009; Dunsmoor and Labar, 2013; Haddad et al., 2013). Some generalization of fear responses is presumably crucial for survival, because similar stimuli may well be similarly dangerous.  However, fear generalization processes may be abnormal in some psychopathological states (Lissek, 2012).

The process of stimulus generalization has been studied for decades, using a variety of methods and stimuli, in a range of species including pigeons, goldfish, worms and humans (Ghirlanda and Enquist, 2003). In humans, the generalization of fear-related responses has been studied primarily using Pavlovian fear conditioning paradigms. In these studies, a variety of outcomes have been used to index fear generalization, including electromyography-measured startle responses (Lissek et al., 2008; Hajcak et al., 2009; Haddad et al., 2013), skin conductance responses (SCRs; Vervliet et al., 2010b; Dunsmoor and Labar, 2013) and explicit ratings (ERs) of fear or shock likelihood (Vervliet et al., 2006, 2010b; Lissek et al., 2008; Hajcak et al., 2009). Each of these studies found significantly increased fear-related responses to stimuli that were perceptually similar (compared to those that were less similar) to a conditioned stimulus (CS) that had been paired with an aversive outcome, such as an electrical shock. In other words, conditioned fear responses were found to generalize from a CS (typically an abstract shape such as a circle or a rectangle) paired with a shock (a CS+), compared to a slightly altered version of that CS that was not paired with the shock (a generalization stimulus, GS). Although these studies have described a qualitative association between fear generalization magnitudes and the degree of perceptual similarity of the GSs to the CS+, the precise relationship between discrimination ability and fear generalization in humans has not been systematically studied. One might predict that autonomic measures of fear responses would generalize beyond perceptual discrimination thresholds, i.e., subjects show fear responses to similar but easily distinguishable stimuli. Alternatively, autonomic responses might be more sensitive than perception in some cases, based on prior demonstrations of sub-threshold summation (Kulikowski and King-Smith, 1973; To et al., 2011) and "unconscious" fear responses (Morris et al., 1998; Whalen et al., 1998).

An additional possibility is that fear generalization gradients might narrow or broaden depending on the context or type of stimuli encountered. For example, the ability to both discriminate and extract common features from similar stimuli is important in social contexts. It is often necessary to quickly assess whether an individual is a friend or foe, generalizing from prior experience and erring on the side of a defensive posture when in doubt, until additional information becomes available. However, the benefits of generalization during social interactions are balanced against the advantages of being able to discriminate among specific individuals with whom one has different relationships.

Recognition and discrimination among distinct humans occurs primarily via recognition of faces (McKone et al., 2007). Many lines of evidence suggest that faces are processed in a specialized manner by the brain. For example, psychophysical studies have shown that faces are processed "holistically" (Kemp et al., 1996; Farah et al., 1998; Hole et al., 1999). In contrast, other types of stimuli are processed in a more piecemeal manner, based on their feature components.  Thus, it is possible that these unique aspects of face perception influence the generalization of fear responses across perceptually similar faces.

Thus, in the current study, we aimed to (1) measure the relationship between visual discriminability and fear generalization; and (2) compare fear generalization gradients for faces and non-face control stimuli. First, we predicted that significant fear generalization would occur to stimuli that were indistinguishable from a threatening stimulus (one that had been associatively linked to an aversive experience, an electrical shock). Second, we predicted that fear generalization would be greater to faces, compared to non-face control stimuli.

## **MATERIALS AND METHODS**

#### **OVERVIEW OF EXPERIMENT 1 (FACES) AND EXPERIMENT 2 ("BLOBS")**

We created image morphs between images of (1) two distinct human faces (Experiment 1); and (2) two distinct non-face shapes or "blobs" (Experiment 2) (**Figure 1**). Later in the experiment, one of the two faces or blobs (the conditioned stimuli, CS) was paired with an electrical shock (the CS+) during a Pavlovian fear conditioning procedure.

First, each subject performed a *discrimination task* to identify the image morph that he could distinguish from the CS+ stimulus at a 75% accuracy level. This value was defined as the Just Noticeable Difference (JND) threshold.

Second, subjects underwent a Pavlovian *fear conditioning procedure*, in which the CS+ stimulus was intermittently followed by a shock, and the other stimulus of the pair was not followed by a shock (the CS−). SCRs were measured continuously.

Third, subjects underwent a *fear generalization procedure* during which they were presented with the CS+, CS− and five morphs whose degree of difference from the CS+ was determined by the subject's performance on the discrimination task (i.e., the specific JND for that subject). SCRs were measured continuously.

Fourth, subjects were presented with each of the previously presented stimuli, and asked to rate the likelihood that each stimulus had ever been followed by a shock (*explicit ratings*).

## **PARTICIPANTS**

Seventy-one healthy male volunteers (mean age: 24.61 ± 0.91) were recruited using an on-line advertisement and enrolled in the study (39 and 32 for Experiment 1 and 2, respectively). Only males were included in this initial study in order to minimize SCR heterogeneity related to gender differences in fear responses (Milad et al., 2006). Participants had no history of psychiatric or neurologic illness, as determined by a phone screen and the Structured Clinical Interview for DSM-IV (SCID; First et al., 1995). All subjects had normal or near normal vision, based on Snellen acuity.

The study was approved by the Partners Healthcare Institutional Review Board, and written informed consent was obtained from all subjects at the time of enrollment.

#### **STIMULI**

#### **Experiment 1: Faces**

Four images of human faces (see **Figure 1A**) were generated using FaceGen 3.4 (Singular Inversions, Canada), as described previously (Yue et al., 2011, 2013; Holt et al., 2014). All four faces (A, B, C and D) were male and caucasian, and achromatic (i.e., all color parameters were set to 0). FaceGen was then used to create morphs (99 even steps) between faces A and B and between faces C and D.

#### **Experiment 2: Non-face "blobs"**

Four images of three-dimensional, unfamiliar shapes ("blobs"; see **Figure 1B**) were generated as described elsewhere (Yue et al., 2013). To equate the texture pattern of the blob and face stimuli, a synthesized texture was generated from scrambling the texture of the face images (Portilla et al., 2003) and overlaid onto the blob stimuli. As with the faces, morphs (99 even steps) were created between blobs E and F and between blobs G and H.

#### **DISCRIMINATION TASK**

Participants were assigned one of the two pairs of face (A/B or C/D, Experiment 1) or blob (E/F or G/H, Experiment 2) stimuli.

The assigned face or blob pair was counterbalanced across subjects. Later in the experiment (during the Fear Conditioning procedure, see below), one of the two stimuli (the CS+) was paired with an electrical shock (the unconditioned stimulus, US), while the other stimulus (the CS−) was not paired with a shock. The CS+ and CS− assignment within each pair was counterbalanced across subjects.

Before the Fear Conditioning procedure, the subjects' ability to discriminate between the pair of stimuli assigned to them was evaluated using a forced-choice discrimination task. Prior to any measurements, the subject practiced the task until they confirmed that they understood the procedure (3–5 trials). The task consisted of three runs of 50 trials each. During each trial, participants first viewed the CS+ stimulus for 500 ms. Following an inter-trial interval (ITI) of 500 ms, the participants were presented with a morph and the CS+ stimulus side by side. Subjects were then asked to select which stimulus they had previously seen, by pressing one of two buttons, indicating the image on the right or the left. The positioning of the morph and CS+ stimulus was randomized across trials. Participants had unlimited time to respond (self-paced). The morphs used in the discrimination task were 6%, 12%, 24%, 48%, and 100% different from the CS+ stimulus (100% different = the other stimulus of the pair, the CS−). The participant's response was followed by an ITI of 1 s. Following completion of the task, participant accuracy was plotted against the morph level (the percentage difference from the CS+ stimulus), in order to calculate the JND level (**Figure 2**) for that participant. The JND was the morph level (% difference from the CS+ stimulus, which could fall between the morphs presented during the discrimination task) that could be distinguished from the CS+ stimulus at an accuracy of 75%. For an independent experiment (not shown here), subjects performed this task a second time (3 additional runs) following the completion of the Fear Conditioning and Fear Generalization phases of the experiment.

### **FEAR CONDITIONING AND FEAR GENERALIZATION**

The Coulbourn Instruments Lablink V System (Allentown, Pennsylvania) was used for these two phases of the experiment. Skin conductance levels were measured with the Coulbourn Isolated Skin Conductance Coupler. Before the Fear Conditioning procedure, two electrodes were placed on the palm of the participant's left hand (to record SCRs) and on the index finger and middle finger of the participant's right hand (to deliver the US, a mild electrical stimulus 500 ms in duration). Next, the intensity of the US was set by each participant to a level that was "highly annoying but not painful" (Milad et al., 2005; Holt et al., 2009). Also, prior to these procedures, the subjects were told that, during the experiment, each stimulus may or may not be followed by the US, but one stimulus was more likely to be followed by the US. They were also told that they would be asked questions about what they had observed following the experiment. Throughout these two procedures, subjects were observed through a closed circuit video camera to ensure that they were awake and attentive.

#### **Fear conditioning**

This phase consisted of 8 CS+ trials and 8 CS− trials, each 6 s long, presented in a pseudorandom order (Milad et al., 2005).

ITIs were 9, 12, or 15 s in duration. The CS+ was followed by the US in 5 of the 8 CS+ trials, and the CS− was never followed by a shock.

In the first 17 participants, a pilot version of the Fear Conditioning phase was used (12 trials, 50% reinforcement). Because this version did not produce reliable learning in this group (learning occurred in 12/17 subjects), this phase was modified. (Since the goal of the study was to examine generalization of previously learned fear responses, it was important that subjects demonstrate adequate fear learning initially, see below).

#### **Fear generalization**

This phase began after a 1-min break following Fear Conditioning. During the Fear Generalization procedure, subjects were presented with the CS+, CS− and five morph levels (m1, m2, m3, m4, and m5) whose degree of difference from the CS+ was determined by the subject's performance on the discrimination task (*m1* = 0.125 JND; *m2* = 0.25 JND; *m3* = 0.5 JND; *m4* = 1.0 JND; *m5* = 1.5 JND) (**Figure 3**). This phase consisted of 35 trials, i.e., five trials for each stimulus category. The ITIs were again 9, 12, and 15 s in length and each stimulus was presented for 6 s. For each subject, stimuli were presented in one of two different pseudorandom orders, so that no more than two of the same stimuli were presented consecutively (to avoid habituation of responses due to repetition) (Lissek et al., 2008; Dunsmoor et al., 2009), counterbalanced across subjects. During this phase, the CS+ was always followed by the shock (100% reinforcement), in order to minimize extinction of the association produced by viewing many CS+-like stimuli that were not followed by a shock.

### **EXPLICIT FEAR RATINGS**

Following Fear Generalization, subjects were presented with each of the previously presented stimuli once (in one of two pseudorandom orders, counterbalanced across subjects; stimulus presentation time = 6 s; ITI = 9 s), then asked to rate the likelihood that the stimulus had ever been followed by a shock (on a scale of 0–100% likely).

#### **SKIN CONDUCTANCE DATA PRE-PROCESSING**

During Fear Conditioning and Fear Generalization, skin conductance was recorded continuously. A participant was considered a "responder" if ≥2 of the 16 trials of the Fear Conditioning phase showed a response greater than 0.05 µS (Schnur et al., 1999; Turner et al., 2005). Data from subjects which did not meet this criteria ("non-responders") were excluded from further analysis (see below).

For both Fear Conditioning and Fear Generalization, the SCR to the stimulus was calculated by subtracting the mean skin conductance for the 2 s prior to stimulus onset from the peak of the skin conductance during the 6 s of stimulus presentation. In addition, for the analysis of the Fear Generalization data only, SCRs were calculated in an identical manner (using the 2 s prior to stimulus onset as the baseline) for the first 6 s of the ITI that immediately followed stimulus offset. Thus, we tested for fear generalization during two time intervals: (1) during the stimulus presentation (*immediate* fear generalization, IFG); and (2) following stimulus offset (*delayed* fear generalization, DFG). SCRs were square-root transformed and averaged across each stimulus type for both Fear Conditioning (CS+ and CS−) and Fear Generalization (CS+, m1, m2, m3, m4, m5, CS−), prior to the statistical analyses.

#### **PARTICIPANTS INCLUDED IN THE ANALYSES**

Because our goal was to measure fear generalization in participants who had successfully learned to discriminate the CS+ and CS−, we included data in our analyses from the participants who demonstrated successful learning only. The "learner" criterion for each individual consisted of a difference in shock likelihood ratings between the CS+ and CS− stimuli ≥50%.

Also, data from three subjects were excluded from the analyses because their discrimination task data were unusable; another subject's data were excluded because he fell asleep during the Fear Generalization procedure. Of the remaining 67 subjects, 53 were learners (28 and 25 subjects in Experiment 1 and 2, respectively). Of the 53 learners, 6 were non-responders (4 and 2 subjects in Experiment 1 and 2, respectively). Thus, 47 subjects (24 and 23 subjects in Experiment 1 and 2, respectively) were included in the analyses (mean age: 23.61 ± 0.94).

Lastly, two of the subjects who participated in Experiment 2 had JND values following the discrimination task that were too high to permit assignment of stimuli. These subjects were assigned generalization stimuli that differed maximally from the CS+ (*m1* = 8%, *m2* = 17%, *m3* = 33%, *m4* = 66%, *m5* = 99%). Excluding these two subjects from the analyses did not alter the findings.

## **DATA ANALYSES**

#### **Fear conditioning**

The presence of significant differential fear conditioning (CS+ minus CS− responses, *p* < 0.05) was assessed using paired, twotailed *t*-tests.

#### **Fear generalization**

In the *SCR data*, we tested for fear generalization using a repeated measures ANOVA with three factors: stimulus level (6: m1, m2, m3, m4, m5, CS−), experimental phase (2: during stimulus presentation, following stimulus offset), and stimulus type (2: faces, blobs) as a between-subjects factor. SCR data collected for the CS+ was not included in this analysis (thus, there are six stimulus levels in this ANOVA), since the presence of the shock during the ITI phase confounds the measurement of the CS+ response following stimulus offset. Significant main effects and interactions with stimulus level (*p* < 0.05) were followed up by paired, two-tailed *t*-tests.

In the *ER data*, we conducted a second repeated measures ANOVA with two factors: stimulus level (7: CS+, m1, m2, m3, m4, m5, CS−), and stimulus type (2: faces, blobs) as a betweensubjects factor. Significant main effects and interactions with stimulus level (*p* < 0.05) were followed up by paired, two-tailed *t*-tests.

*Comparison of autonomic and explicit fear generalization*: we compared the amount of fear generalization in the SCRs (independently for the IFG and DFG responses) to the fear generalization in the ERs, using ANOVAs performed on normalized data (normalization permitted comparison of SCR and ratings data). These two ANOVAs included three factors: response type (2: SCR, ratings), stimulus level (seven or six for the IFG and DFG analyses, respectively), and stimulus type (2: faces, blobs) as a betweensubjects factor.

For each subject, each averaged value for a given stimulus was normalized using the following formula: (Value − Minimum)/(Maximum − Minimum), where Minimum is the smallest average value for a given subject (i.e., the subject's average value for the CS+, m1, m2, m3, m4, m5, or CS−, whichever is the smallest), and Maximum is the largest averaged value for the subject (i.e., their average value for the CS+, m1, m2, m3, m4, m5, or CS−, whichever is the largest). Thus, each subject's largest average response was scaled to 1, and the smallest response was scaled to 0.

## **Correlations**

Correlations among fear conditioning, fear generalization (for the morphs to which there was significant fear generalization, see below) and JND levels were examined using Pearsons *r*.

## **RESULTS**

## **FEAR CONDITIONING**

In both Experiment 1 and 2, subjects acquired differential, conditioned fear responses (CS+ > CS−, *p*s < 0.001). We found no difference between the level of differential fear conditioning acquired during the two experiments (*t*(45) = 0.69, *p* = 0.50; mean SCR to CS+ = 0.40 ± 0.07 µS (mean ± SEM); mean SCR to CS− = 0.16 ± 0.08 µS across all subjects (*n* = 47); comparison of the CS+ vs. CS−: *t*(46) = 7.22; *p* = 4 × 10−9 ). During Fear Generalization, this learning was maintained (i.e., SCRs were significantly greater to the CS+ compared to the CS− during Fear Generalization in both experiments (*p*s < 0.004)).

## **FEAR GENERALIZATION: SKIN CONDUCTANCE RESPONSES**

Fear generalization was defined by the presence of a significantly greater SCR to a morph (m1 +/− the other morphs) compared to the SCR to the CS− (Lissek et al., 2008; Haddad et al., 2013). The ANOVA revealed a significant effect of stimulus level (*F*(5,225) = 4.78, *p* < 0.001) and a significant stimulus level by experimental phase interaction (*F*(5,225) = 2.55, *p* = 0.03), with no main effects or interactions with stimulus type (all *p*s > 0.22). Follow-up tests revealed that, across both experiments, during the stimulus presentation, there was significant generalization to m1, compared to the CS− (*t*(46) = 2.08, *p* = 0.043; *p*s > 0.25 for the other morph levels). Following stimulus offset, there was generalization to m1 (*t*(46) = 3.80, *p* = 0.0004) and m2 (*t*(46) = 2.74, *p* = 0.009), and a trend towards generalization to m3 (*t*(46) = 1.93, *p* = 0.06) and m4 (*t*(46) = 1.90, *p* = 0.06), with no generalization to m5 (*p* = 0.29) (**Figure 4**). Thus, there was both immediate (IFG, during stimulus presentation) and delayed (DFG, following stimulus offset) fear generalization to morphs that were perceptually similar to the CS+ (i.e., perceptually closer to and indistinguishable from the CS+, compared to the JND threshold = m4).

The interaction with experimental phase arose from the greater amount of delayed, compared to immediate, fear generalization (DFG > IFG, **Figure 5**). A direct comparison of the differential SCRs (response to the morph − the response to the CS−) during the two phases of the experiment confirmed that the responses were greater for m1 (*t*(46) = 3.32, *p* = 0.002) and m2 (*t*(46) = 2.14, *p* = 0.04) (*p*s for the other morph levels >0.12) following stimulus offset, compared to during stimulus presentation.

Consistent with the absence of an interaction with stimulus type in the ANOVA, the pattern of responses was similar across Experiments 1 (faces) and 2 (blobs), although the effects at the individual morph levels appeared to be slightly (but nonsignificantly) stronger in Experiment 1 (**Figures 4** and **5**).

## **FEAR GENERALIZATION: EXPLICIT RATINGS**

For the ERs, there was a significant effect of stimulus level (*F*(6,270) = 61.29, *p* < 0.001), as well as a significant interaction between stimulus type and level (*F*(6,270) = 2.5, *p* = 0.023). This pattern of results arose from the presence of (1) explicit fear generalization to the morph stimuli; and (2) greater fear generalization in Experiment 1 (faces) compared to Experiment 2 (blobs) (**Figure 6**). Using the CS− as the baseline, comparison condition, we found fear generalization to all morph levels in both experiments (all *p*s < 0.013). However, because the shock likelihood ratings of the CS− were always 0, we also computed fear generalization using the ratings for m5 (the morph that was the most different from the CS+) as the comparison condition. Compared to the m5 ratings, in Experiment 1, subjects showed significantly greater shock likelihood ratings to the CS+, m1, m2, m3 and m4 (all *p*s < 0.0008), whereas in Experiment 2, subjects showed greater shock likelihood ratings to the CS+, m1, m2, m3 (all *p*s < 0.01) but not m4 (*p* = 0.52). Consistent with this, a direct comparison of the ratings across the two experiments at each stimulus level showed that there were significantly higher shock likelihood ratings in Experiment 1 compared to Experiment 2 for m1 (*t*(45) = 2.06, *p* = 0.046) and m3 (*t*(45) = 2.77, *p* = 0.008) (*p*s for the comparisons at the other stimulus levels >0.17).

In summary, although the SCRs showed similar generalization patterns and magnitudes across the two experiments (i.e., to faces and blobs), there was greater *explicit* fear generalization to perceptually similar faces, compared to the non-face control stimuli.

## **DIRECT COMPARISON OF AUTONOMIC (SCR) AND EXPLICIT FEAR GENERALIZATION IN NORMALIZED DATA**

#### **I. Comparison of shock likelihood ratings vs. SCRs during stimulus presentation**

Here again we found a significant effect of stimulus level (*F*(6,270) = 46.10, *p* < 0.001), consistent with the results described above showing fear generalization to the morphs for both the SCRs and ERs, across both experiments (**Figure 7A**). In addition, there was a significant interaction of stimulus level by response type (*F*(6,270) = 17.16, *p* < 0.001), with no significant interactions with stimulus type (*p*s > 0.07). In the normalized data, the ratings values were significantly greater than the SCR values for m1, m2, and m3 (*p*s < 0.002) but not for m4 and m5 (*p*s > 0.21) or CS−, which showed the opposite pattern (*p* = 5 × 10−10), since the ratings of the CS− were always 0.

## **II. Comparison of shock likelihood ratings vs. SCRs following stimulus offset**

Similar results were found following stimulus offset, with a significant effect of stimulus level (*F*(5,225) = 33.02, *p* < 0.001), and an interaction of stimulus level by response type (*F*(5,225) = 14.49, *p* < 0.001), with no interactions with stimulus type (*p*s > 0.24) (**Figure 7B**). The ratings values were significantly greater than the SCR values for m1, m2 and m3 (*p*s < 0.05) but not for m4 (*p* = 0.31). Also, m5 and the CS− showed the opposite pattern (*p* = 0.01 and 1 × 10−8 , respectively).

Thus, this analysis demonstrates statistically that a greater amount of fear generalization was present in the ERs compared to the SCRs in both experiments.

### **CORRELATIONS BETWEEN FEAR LEARNING AND FEAR GENERALIZATION**

In the full sample (*n* = 47), the success of differential fear conditioning (i.e., the magnitude of the difference between SCRs to the CS+ and CS−) predicted the differential SCR to m1 (vs. CS−) during the stimulus presentation (*r* = 0.36, *p* = 0.01) and following stimulus offset (*r* = 0.59, *p* < 0.001), and the differential SCR to m2 following stimulus offset (*r* = 0.48, *p* = 0.001). In

**generalization procedure**. Bar plots of SCRs during fear generalization of the subjects of the two experiments combined (**A,D**; n = 47), Experiment 1 (**B,E**; n = 24) and Experiment 2 (**C,F**; n = 23) are shown. Panels (**A,B** and **C**) show mean maximum SCRs during the 6-s stimulus presentation; panels (**D,E** and **F**) show the mean maximum SCRs following stimulus offset, during the first 6 s of the ITI. Data for the CS+ are omitted from the graphs of the ITI data (panels (**D,E** and **F**), since the responses to the CS+ were likely influenced by the unconditioned stimulus (the electrical shock), which was delivered during the ITI immediately following the presentation of the CS+. A symbol over a CS+, m1 or m2 bar indicates that the mean SCR for this stimulus was significantly

The blue arrows indicate the morph level corresponding to the JND, m4. Error bars represent one standard error from the mean. Overall, these data reveal that a similar pattern of fear generalization occurs in response to perceptually similar face and non-face control stimuli. In Experiment 1 (faces), there was significant fear generalization to m1 (t(23) = 2.44, p = 0.02) and m2 (t(23) = 2.30, p = 0.03) during the stimulus presentation, and to m1 (t(23) = 2.55, p = 0.02) and m2 (t(23) = 2.46, p = 0.02) following stimulus offset (ps for the other morphs > 0.15). In Experiment 2, there was generalization to m1 only, following stimulus offset (t(22) = 2.93, p = 0.008), with no significant fear generalization during the stimulus presentation (all other ps > 0.08).

Experiment 1 only (*n* = 24), similar correlations were found between fear conditioning success and differential SCRs to m1 during the stimulus presentation (*r* = 0.65, *p* = 0.001) and to m1 and m2 following stimulus offset (m1: *r* = 0.81, *p* < 0.001; m2: *r* = 0.60, *p* = 0.002). Similar correlations were found when the SCRs to the CS− were not subtracted from the SCRs to the morphs.

Fear conditioning success was also correlated with the amount of explicit fear generalization to m1 (ratings to m1 vs. m5) in the full sample (*r* = 0.32, *p* = 0.027, *n* = 47) and in Experiment 2 (*r* = 0.47, *p* = 0.02, *n* = 23).

We found no correlations between JND levels and magnitudes of fear learning or fear generalization.

#### **DISCUSSION**

#### **SUMMARY OF FINDINGS**

First, we found that fear generalization is closely linked to perceptual discriminability. Specifically, in all analyses, generalization did not occur above discrimination thresholds. Second, we showed that conscious fear responses, measured as shock likelihood ratings, showed a broader fear generalization gradient than the SCRs. Also, both peripheral and conscious measures of fear generalization correlated with the success of acquisition of conditioned fear responses, suggesting that fear generalization here was not due to poor encoding of the original CS-US association. Lastly, partially confirming our prediction, conscious fear generalization was greater in response to faces than to non-face control stimuli.

#### **GENERALIZATION OF FEAR RESPONSES IS LINKED TO PERCEPTUAL DISCRIMINABILITY**

During stimulus presentation, SCR-based fear generalization occurred to the stimulus morph that was perceptually closest to the CS+ (m1), and then extended further following stimulus offset, to include m2 as well. In the ERs, generalization also occurred to m3 and variably (in Experiment 1 only) to m4, which represented the discrimination threshold, but not to m5.

corrected for all stimulus conditions by adjusting the mean response during the 2-s interval before the stimulus onset to 0.

These findings are in line with many previous studies conducted in non-mammalian species (e.g., pigeons responding to varying frequencies of light) showing a relationship between perceptual similarity and generalization of operant responses, which typically have a Gaussian distribution (Ghirlanda and Enquist, 2003). Here, we provide empirical evidence for this type of relationship in humans, demonstrating that the autonomic fear system in humans is sensitive to quite small perceptual differences between stimuli.

The finding of a broader fear generalization gradient in the post-experiment shock likelihood ratings, compared to the SCRs, is consistent with the results of two previous studies that used on-line shock likelihood ratings (Lissek et al., 2008; Haddad et al., 2013), suggesting that this is a robust phenomenon. This dissociation may at first appear counter-intuitive, since the mechanism(s) generating the conscious appraisal of threat seems to be "throwing away" more accurate information possessed by a lower level system.

However, we speculate that this conservative bias in conscious fear responses may have promoted survival during primate evolution. It may be advantageous, in certain contexts, to be wary of stimuli that are similar, but clearly not identical, to known threats, given that these stimuli may have other common characteristics. In the current study, the autonomic system was not mobilized for the morphs that were similar to, but distinguishable from, the CS+, suggesting that the cost of mobilizing the physiological resources to respond to a threat is outweighed, in the short term, by the benefits of gathering more information about the stimulus. A conscious perception of a potential threat may serve the purpose of directing attentional resources towards gathering this additional information (Ledoux, 2000). If new evidence suggests that the stimulus is indeed threatening, then the autonomic fear system may be recruited at that point.

The neural circuitry responsible for these two types of fear generalization responses has not been fully characterized. However it is known that distinct subfields of the hippocampal formation are involved in the individual coding of (vs. the generalization of features across) similar stimuli or events (Aimone et al., 2011; Newman and Hasselmo, 2014). Other studies have reported that the medial prefrontal cortex and midline thalamus also contribute to these processes (Xu et al., 2012; Xu and Südhof, 2013). During face perception, it is likely that the face-selective areas within the ventral temporal cortex, including the fusiform face area (Kanwisher et al., 1997) and anterior temporal area (Rajimehr et al., 2009; Nasr and Tootell, 2012) communicate with this fronto-thalamic-hippocampal memory network.

Consistent with the work conducted in rodents, functional imaging studies of fear generalization in humans using Pavlovian conditioning procedures have found that the medial prefrontal cortex (Dunsmoor et al., 2011; Greenberg et al., 2013a; Lissek et al., 2013a; Cha et al., 2014b) and hippocampus (Lissek et al., 2013a) show response gradients that are consistent with a fear generalization phenomenon. Similar gradients have also been detected in the responses of regions known to be important in salience detection and fear production, such as the insula (Dunsmoor et al., 2011; Greenberg et al., 2013a; Lissek et al., 2013a), striatum (Dunsmoor et al., 2011; Greenberg et al., 2013a) and ventral tegmental area (Cha et al., 2014a). However the mechanisms responsible for integrating the relevant perceptual and motivational information to produce these response gradients remain unclear. Studies that parametrically vary each component (e.g., the perceptual

**FIGURE 6 | Explicit shock likelihood ratings following the fear generalization procedure**. Bar plots of shock likelihood ratings for the two experiments combined (**A**; n = 47), Experiment 1 (**B**; n = 24) and Experiment 2 (**C**; n = 23) are shown. A symbol over a bar indicates that the mean ratings for this stimulus were significantly greater than the mean ratings for m5, the morph that was the most different perceptually from the CS+ (\*\* p < 0.01; + p < 0.001). The blue arrows indicate the morph level corresponding to the JND, m4. Error bars represent one standard error from the mean. Compared to the m5 ratings, subjects

features and motivational value of the stimuli) may clarify how these distinct types of information are used to inform both automatic and conscious perceptions of threat and resulting behavior.

## **AUTONOMIC FEAR GENERALIZATION HAS AN EXTENDED TIME COURSE**

The generalization gradients observed in our SCR data were larger following stimulus offset than during the presentation of the stimulus. This slow time course is typical of SCRs (Bach et al., 2010). This delayed generalization response may also reflect an interaction between the initial autonomic response and the conscious assessment of threat—top-down processes may augment fear responses over time. Alternatively, subjects may experience an acute increase in fear during the time period when they expect to receive a shock, immediately following stimulus offset. The absence of the shock in the context of an increased expectation for it may produce a "prediction error" signal (Li and Mcnally, 2014), contributing to this late SCR. Future studies that manipulate the predictability of the shock may determine whether this response is indeed linked to prediction error-related mechanisms, or merely reflects the long latency of SCRs.

showed significantly greater fear shock likelihood ratings to the CS+ (t(23) = 9.00, p = 5 × 10−9 ), m1 (t(23) = 7.24, p = 2 × 10−7 ), m2 (t(23) = 7.10, p = 3 × 10−7 ), m3 (t(23) = 8.12, p = 3 × 10−8 ), and m4 (t(23) = 3.87, p = 0.0008) in Experiment 1 (faces), and to the CS+ (t(22) = 5.90, p = 6 × 10−6 ), m1 (t(22) = 4.43, p = 0.0006), m2 (t(22) = 4.50, p = 0.0002), and m3 (t(22) = 2.83, p = 0.01), but not m4 (p = 0.52) in Experiment 2 (blobs). Direct comparisons of the ratings of the two experiments revealed that there was more explicit fear generalization to faces than to blobs (see text).

## **CONSCIOUS FEAR GENERALIZATION WAS GREATER TO FACES THAN TO NON-FACE CONTROL STIMULI**

Conscious fear generalization (shock likelihood ratings) was greater to the face stimuli, compared to the perceptually matched control stimuli. Although we can only speculate regarding the mechanisms underlying this effect, one possibility is that the holistic, configural based (vs. feature-based) processing mechanisms relied upon during face perception promotes generalization of fear responses across similar-appearing faces. This hypothesis could be explored further in follow-up work in which, in addition to faces, inverted or contrast-reversed faces (which are processed in a feature-based manner) are used as generalization stimuli.

It is important to also note that our interpretation of this finding is somewhat limited by the fact that we used unrecognizable shapes ("blobs") as our control stimuli. Fear generalization may be greater for stimuli that are recognizable members of a known category of objects (Dunsmoor et al., 2013) (i.e., clear category membership may facilitate the extraction of general features of objects), compared to stimuli that are unrecognizable and seemingly arbitrary. Future work using non-face, known objects as control stimuli could further test whether fear

generalization to faces differs from that to other objects. However, these experiments would also need to account for disadvantages associated with these types of control stimuli, i.e., they would not be closely matched to the face stimuli in terms of lower level cues.

Another open question is whether the pattern of results seen here would change if faces with emotional expressions, such as fear, were used as stimuli. Dunsmoor et al. conducted several studies in which a morph continuum between a fearful and neutral face were used as generalization stimuli (Dunsmoor et al., 2009, 2011). In these experiments, the CS+ stimulus was a morph that was at the midpoint of the fear-to-neutral continuum. They found an asymmetric generalization gradient, with the most fear generalization in response to a morph on the "fear side" of the continuum. Given these data and the results of the current study, one question remains: is there fear generalization to faces with fearful expressions (or other biologically prepared stimuli) that are above the discrimination threshold (i.e., to those that can be clearly discriminated from the CS+) due to their intrinsic aversiveness? An alternative possibility is that discriminability among perceptually similar fearful faces is lower than that to perceptually similar neutral faces (perhaps because of the evolutionary importance of defending oneself rapidly from any possible threat), which would lead to greater generalization across fearful faces. These competing explanations could be investigated with the approach used here in the current study.

#### **LIMITATIONS**

This study has several limitations. First, we studied only males, in order to minimize heterogeneity in our data in this first study using this paradigm. A similar study in females is currently underway to determine whether the effects seen here differ across genders. Second, the shock likelihood ratings were not collected during the fear generalization procedure but immediately afterwards. This was done in order to avoid suppression of fear responses by evaluative processes (Lange et al., 2003; Taylor et al., 2003), but this aspect of our design may have affected our results. However, because previous studies that used on-line shock likelihood or fear ratings found qualitatively similar results (i.e., more apparent fear generalization in ratings than in physiological measures) (Lissek et al., 2008; Haddad et al., 2013), this seems unlikely to have had a large effect. Third, our findings could have been influenced by the fact that subjects viewed face or blob stimuli during the discrimination task, before undergoing Pavlovian fear conditioning and generalization procedures with some of the same stimuli. This raises the possibility that other types of learning processes, such as latent inhibition (the inhibitory effect of stimulus pre-exposure on fear conditioning and generalization (Vervliet et al., 2010a)) occurred. However, a latent inhibition effect would have led to a reduction in the level of differential fear conditioning achieved. Given that differential fear conditioning was robust in both experiments, and fear generalization magnitudes correlated with the amount of fear conditioning, this effect was likely small or insignificant.

#### **FUTURE STUDIES AND CLINICAL IMPLICATIONS**

The development of quantitative measures of perceptual and emotional processes and their interactions is needed for several reasons. After validating such measures, the mechanisms governing these processes can be explored further, by varying the experimental design and measuring additional outcomes, including the underlying brain mechanisms. Also, although some degree of fear generalization is adaptive, excessive generalization of fear or other types of emotional responses may lead to inappropriate behaviors and responses during social interactions, giving rise, in some cases, to psychopathological states. For example, fear generalization has been shown to be excessive in anxiety disorders (Lissek et al., 2010, 2013b; Greenberg et al., 2013b; Kaczkurkin and Lissek, 2013; Cha et al., 2014b). Thus, a quantitative index of abnormal fear generalization may serve as an intermediate phenotype for these disorders, which can serve as a target of treatment and early intervention. Abnormal fear processes have been demonstrated in depression (Nissen et al., 2010) and schizophrenia (Jensen et al., 2008; Holt et al., 2009; Romaniuk et al., 2010) as well. In light of the evidence for abnormalities in neural systems that span diagnostic categories in psychiatry (Insel et al., 2010), the study of fear-related processes in patients with a wide range of symptom types may clarify the degree to which patients with distinct primary diagnoses share a common vulnerability to negative affect and the experience of inappropriate fear.

response; ER = explicit ratings.

## **ACKNOWLEDGMENTS**

This work was supported by the National Institute of Mental Health (Daphne J. Holt: RO1MH095904) and the National Eye Institute (Roger B. H. Tootell: R01EY017081).

### **REFERENCES**

research on mental disorders. *Am. J. Psychiatry* 167, 748–751. doi: 10.1176/appi. ajp.2010.09091379

**Conflict of Interest Statement**: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 30 May 2014; accepted: 26 July 2014; published online: 05 September 2014*. *Citation: Holt DJ, Boeke EA, Wolthusen RPF, Nasr S, Milad MR and Tootell RBH (2014) A parametric study of fear generalization to faces and non-face objects: relationship to discrimination thresholds. Front. Hum. Neurosci. 8:624. doi: 10.3389/fnhum.2014.00624*

*This article was submitted to the journal Frontiers in Human Neuroscience*. *Copyright © 2014 Holt, Boeke, Wolthusen, Nasr, Milad and Tootell. This is an openaccess article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms*.

## Novel method to classify hemodynamic response obtained using multi-channel fNIRS measurements into two groups: exploring the combinations of channels

## *Hiroko Ichikawa1,2,3\*, Jun Kitazono4, Kenji Nagata4, Akira Manda4, Keiichi Shimamura5,6, Ryoichi Sakuta5,6, Masato Okada4,7, Masami K. Yamaguchi 1,2, So Kanazawa8 and Ryusuke Kakigi 9*

*1 Department of Psychology, Chuo University, Tokyo, Japan*

*2 Research and Development Initiative, Chuo University, Tokyo, Japan*

*3 Japan Society for the Promotion of Sciences, Tokyo, Japan*

*4 Department of Complexity Science and Engineering, The University of Tokyo, Kashiwa, Japan*

*5 Department of Pediatrics, Dokkyo Medical University Koshigaya Hospital, Koshigaya, Japan*

*6 Center for Child Development and Psychosomatic Medicine, Dokkyo Medical University Koshigaya Hospital, Koshigaya, Japan*

*7 RIKEN Brain Science Institute, Wako, Japan*

*8 Department of Psychology, Japan Women's University, Kawasaki, Japan*

*9 Department of Integrative Physiology, National Institute for Physiological Sciences, Okazaki, Japan*

#### *Edited by:*

*Aina Puce, Indiana University, USA*

#### *Reviewed by:*

*Giancarlo Zito, 'S. Giovanni Calibita' Fatebefratelli Hospital, Italy Yosuke Kita, National Center of Neurology and Psychiatry, Japan*

#### *\*Correspondence:*

*Hiroko Ichikawa, Department of Psychology, Chuo University, 742-1, Higashi-Nakano, Hachioji, Tokyo 192-0393, Japan e-mail<EMAIL_ADDRESS>Near-infrared spectroscopy (NIRS) in psychiatric studies has widely demonstrated that cerebral hemodynamics differs among psychiatric patients. Recently we found that children with attention-deficit/hyperactivity disorder (ADHD) and children with autism spectrum disorders (ASD) showed different hemodynamic responses to their own mother's face. Based on this finding, we may be able to classify the hemodynamic data into two those groups and predict to which diagnostic group an unknown participant belongs. In the present study, we proposed a novel statistical method for classifying the hemodynamic data of these two groups. By applying a support vector machine (SVM), we searched the combination of measurement channels at which the hemodynamic response differed between the ADHD and the ASD children. The SVM found the optimal subset of channels in each data set and successfully classified the ADHD data from the ASD data. For the 24-dimensional hemodynamic data, two optimal subsets classified the hemodynamic data with 84% classification accuracy, while the subset contained all 24 channels classified with 62% classification accuracy. These results indicate the potential application of our novel method for classifying the hemodynamic data into two groups and revealing the combinations of channels that efficiently differentiate the two groups.

**Keywords: hemodynamic data, near-infrared spectroscopy (NIRS), support vector machine (SVM), sparse modeling, attention-deficit/hyperactivity disorder (ADHD), autism spectrum disorders (ASD)**

## **INTRODUCTION**

Near-infrared spectroscopy (NIRS) has been utilized to measure brain activity in humans (for review, Ferrari and Quaresima, 2012). Because NIRS is non-invasive and requires less stabilization of participants than other neuroimaging techniques, NIRS is highly suitable for studies with infants, children (for review, Lloyd-Fox et al., 2010; Gervain et al., 2011) and patients with psychiatric symptoms or disorders such as schizophrenia, depression, anxiety disorder, and attention-deficit/hyperactivity disorder (ADHD) (for review, Fukuda, 2009; Ernst et al., 2012). ADHD is characterized by major symptoms of hyperactivity, impulsivity and inattention (American Psychiatric Association, 2000). Children with ADHD (ADHD children) show atypical hemodynamic response in the prefrontal region associated with attention and working memory deficits (Weber et al., 2005; Negoro et al., 2010; Monden et al., 2012, but see also Schecklmann et al., 2010).

The neural response of ADHD children in face processing is different from that of typically developing children (TD children). Tye et al. (2013) used ERP techniques to examine the face-inversion effect and gaze processing in ADHD children, children with ASD (ASD children), children with comorbid ASD+ADHD, and TD children. They found that children with ADHD (ADHD/ADHD+ASD) showed atypical response that reflect early attentional stage of face processing, while children with ASD (ASD/ASD+ADHD) showed atypical response in gaze processing and atypical neural specialization, which are likely to be more relevant to the characteristic social deficits of autism. As far as we know, Tye et al. (2013) was the first study that found the basic face processing in ADHD children different from that of ASD children and no previous studies has yet investigated familiar face processing in children with ADHD. Dawson et al. (2002) demonstrated that ASD children showed no differential ERPs when viewing their mother's face and viewing an unfamiliar female face, while TD children did. Although the hemodynamic response to one's mothers' face has not been tested with ADHD children, we may suppose that an atypical response might also be observed in ADHD children.

Recently our group found that boys with ADHD showed a different hemodynamic response to their own mother's face than typically developing boys (Shimamura et al., 2012, under review). Thorell et al. (2012) tested non-clinical 8.5 year-old children and reported that attachment disorganization and executive functioning were independently related to ADHD symptoms. Furthermore, Carlsson et al. (2008) investigated the endogenous and exogenous factors that predict inattentiveness and hyperactivity in middle childhood and demonstrated that the quality of the caregiving more powerfully predicted inattention and hyperactivity than did early biological or temperamental factors. The quality of the relationship between the child and the caregiver might increase or reduce the amount of communication between the two, even in TD children, which may possibly affect the development of the children's neural basis for processing his/her mother's face. (Incidentally, all adults participated as "caregivers" in Shimura et al.'s study were the mother's of the children.)

Furthemore, we found that boys with ADHD showed a different hemodynamic response to their own mother's face than boys with ASD (Shimamura et al., 2012, under review). Although ASD is characterized by difficulties with social interaction, restricted interests, and repetitive behaviors (American Psychiatric Association, 2000), there is a large overlap of symptoms in ADHD and ASD patients such as hyperactivity, restlessness, and impairments in social cognitive abilities (Yerys et al., 2009; Taurines et al., 2012; van der Meer et al., 2012). It is often difficult in actual clinical practice to distinguish between these patients due to their overlapping symptoms (Yoshida and Uchiyama, 2004). Using fNIRS, we presented boys of both groups with images of their mother's face and measured their cerebral hemodynamics in the bilateral temporal area. Only children with ADHD showed a significantly greater concentration of oxyhemoglobin (oxy-Hb) in the bilateral temporal area than the pre-task baseline; children with ASD showed a decrease of oxy-Hb concentration in the left temporal area.

These findings suggest the possibility of distinguishing the cerebral hemodynamic data of the ADHD participants from those of the ASD participants and the possibility of classifying the data into two separate diagnostic groups. With such classification, we might be able to predict to which diagnostic group an unknown participant belongs by analyzing his/her hemodynamic data.

To classify the hemodynamic data into two groups, a promising method involves the Support Vector Machine (SVM; Vapnik, 1982). The SVM is a multivariate method for binary classification and has recently been introduced to NIRS studies aimed at the development of BCI (Sitaram et al., 2007; Cui et al., 2010).

When using multivariate pattern analysis methods such as SVM, we can improve classification accuracy by selecting informative variables and eliminating uninformative ones (Weston et al., 2001; Guyon and Elisseeff, 2003; Liu and Yu, 2005). This process is called feature selection or variable selection and has been increasingly applied in recent BCI studies (Yamashita et al., 2008; Gottemukkula and Derakhshani, 2011). A number of feature selection methods have been proposed in past studies (Guyon and Elisseeff, 2003; Liu and Yu, 2005). These methods, however, do not necessarily select the subset of variables that gives the best classification accuracy because these methods are intended to be applied to high-dimensional data such as genetic data, and are designed to decrease computation time by using approximate treatment. Here, without approximate treatment, we exhaustively evaluated classification accuracy using all 224 − 1 = 16*,* 777*,* 215 subsets of channels to find the best one, using 5-fold cross validation.

In this study, we applied SVM to real hemodynamic data obtained in Shimamura et al. (2012, under review) and tried to classify the data into two groups: ADHD participants and ASD participants. We exhaustively searched the optimal subset among all subsets of channels and evaluated the classification accuracy using 5-fold cross validation. To compare the effectiveness of SVM with a standard method, we applied two feature selection methods: Lasso (Tibshirani, 1996) and sparse logistic regression (SLR) (Yamashita et al., 2008), and a channel-wise *t*-test, which is one of the most popular statistical analysis to find an activated channel in NIRS measurement.

## **METHOD AND ANALYSIS**

## **PARTICIPANTS**

The participants in this study were nine boys with ADHD (three ADHD-only and six with comorbid ADHD+ASD; mean age = 9y9m, *SD* = 1y7m) and eight boys with ASD (two ASD-only and six with comorbid of ASD+ADHD; mean age = 9y9m, *SD* = 1y4m). Six ADHD boys received methylphenidate, and one ADHD boy received atomoxetine. The mean score of ADHD-Rating Scale was 34.2 (range = 11–52; *SD* = 13*.*9) in the ADHD group, 26.6 (range = 16–38; *SD* = 7*.*6) in ASD group. The twotailed two-sample *t*-test did not show significant difference in their ADHD-RS scores [*t*(15) = 1*.*375, *ns*]. These two diagnostic groups were not different in their age or sex. All diagnoses were based on the DSM-IV-TR (American Psychiatric Association, 2000) and were made by a pediatric neurologist (Ryoichi Sakuta).

## **PROTOCOL**

During the measurement, participants observed the visual stimuli presented on the monitor. A single trial was comprised of a baseline period and test period. Each trial started with a baseline period during which the participants fixated on a black dot displayed on the monitor at the rate of 1 Hz. The duration of the baseline period was at least 20 s. Following this, a test period began. During the test period, the image of the child's mother's face or that of an unknown female face was presented. Either the mothers' face or the unknown female face appeared successively 10 times at the rate of 1 Hz. The duration of test trial was 10 s. The mothers' face and the unknown face were presented alternatively. Each participant performed five trials. In this study, we analyzed the hemodynamic data only from when the boys were passively looking at their mother's face.

The study was approved by the Ethical Committee of the Dokkyo Medical University Koshigaya Hospital and by the Ethical Committee of Chuo University. Written informed consent was obtained from the participants and their parent. The experiments were conducted according to the Declaration of Helsinki.

#### **FUNCTIONAL NIRS RECORDING**

We used a Hitachi ETG-4000 system (Hitachi Medical, Chiba, Japan), which can record from 24 channels simultaneously, with 12 channels for the right temporal area and 12 for the left. This instrument generates two wavelengths of NIR (695 and 830 nm) and measures the time courses of the levels of oxyhemoglobin (oxy-Hb), deoxyhemoglobin (deoxy-Hb), and their sum (total hemoglobin: total-Hb) with 0.1 s time resolution. Based on the previous study, which showed that the oxy-Hb change reflects the task-related neural activity more reliably than deoxy-Hb or total-Hb (Strangman et al., 2003; Cui et al., 2011), we analyzed only the oxy-Hb concentration.

The probes were set on the child's scalp at the bilateral temporal area centered at T5 and T6 according to the International 10–20 system (Jasper, 1958) (**Figure 1**). When the probes were positioned, the experimenter checked to see if the fibers were touching the child's scalp correctly. The Hitachi ETG-4000 system automatically detects whether the contact can adequately measure the emerging photons for each channel. All the trials were rejected from the analysis if adequate contact between the fibers and the child's scalp couldn't be achieved because of hair interference.

### **PREPROCESSING OF DATA**

The raw data on oxy-Hb concentrations from each channel were digitally band-pass-filtered at 0.02–1.0 Hz to remove any noise due to heartbeat pulsations and any longitudinal signal drift.

Although the raw NIRS data were originally relative values, and could not be compared directly across subjects or channels, the normalized data such as the Z-score could be averaged regardless of the unit (Schroeter et al., 2003; Shimada and Hiraki, 2006). This calculation of the Z-score is a reliable analysis for changes in concentration in the children's brains since the analysis is independent of the differential path length factor (DPF).

In order to show the relative change of oxy-Hb concentration during the task period, we standardized the raw data into Z-scores based on the baseline period ahead of the task period. We calculated the Z-scores of oxy-Hb in a time series of 0.1 s time resolutions from 3 s before the test period onset to the test period offset (Ichikawa et al., 2010; Kobayashi et al., 2011, 2012). The Z-score at each time point can indicate the deviation of hemodynamic response during the presentation of faces from the "baseline." The "baseline" for calculating the Z-score was a period of 3 s immediately before the beginning of the each test period, which reflects the activation during the observation of the blank and fixation points. The Z-scores were calculated using the following formula:

$$d = (\mathfrak{x}\_{test} - m\_{baseline}) / s \tag{1}$$

*xtest* represents the raw data [mM mm] at each time point during the test period and *mbaseline* represents the mean of the raw data during the baseline period. *s* represents the *SD* of the raw data during the baseline period.

Consistent with a previous study (Boynton et al., 1996) and our previous studies using NIRS (Ichikawa et al., 2010; Nakato et al., 2011), we found that a response peak lags a few seconds

test period was fixed for 10 s. The baseline period and test period were

between the fibers was set at 3 cm.

behind stimulus onset. Therefore, we performed the following analyses against the mean Z-scores from 3 to10 s after the face stimulus onset.

The Z-scores were calculated separately for each trial. In this study, we eliminated the trials from further analysis if all of 24 channels were not completely recorded due to hair interference or motion artifact. Finally, the number of data we obtained was 50: 25 data from 6 participants of the ADHD group and 25 data from 8 participants of the ASD group. The mean number of valid trials was 4.3 (*SD* = 2*.*7) per child with ADHD and 3.1 (*SD* = 1*.*3) per child with ASD.

#### **SUPPORT VECTOR MACHINE (SVM)**

We trained the SVM to discriminate data by considering to which diagnosis group the owner of the data belongs. We used the mean Z-scores of hemodynamic activities and the diagnosis group of participants as inputs and outputs of the SVM, respectively.

SVMs are state-of-the-art models for classification with a high generalization capability (Vapnik, 1982; Bennett and Mangasarian, 1992; Cortes and Vapnik, 1995). Given input data, an SVM classifies them into two classes. An SVM learns the relationship between the input data and their classes from the training samples. It also predicts the class of unknown data.

$$\left\{ (\mathbf{x}\_i, t\_i) \mid \mathbf{x}\_i \in \mathbb{R}^D, \ t\_i \in \{+1, -1\} \right\}\_{i=1}^N,\tag{2}$$

where *xi* is a D-dimensional feature vector, and *ti* is a class label of *xi*. In this study, the mean Z-scores were calculated for 24 channels (or 12 for hemispheric analysis) for each trial, and were used as a 24-dimensional (or 12-dimensional) feature vector. We had 50 valid trial data sets and 50 feature vectors input to the SVM. We related ADHD participants to *ti* = +1 and ASD participants to *ti* = −1.

Given this data set, an SVM finds a hyperplane in the feature vector space that separates the samples into two groups. The obtained hyperplane is called a decision boundary. New samples are then classified according to which side of the boundary they belong. Classifying samples using a decision boundary is a common practice for other linear classifiers such as logistic regression. What characterizes SVMs is margin maximizing. The margin is the distance between the decision boundary and the closest sample to the decision boundary. By maximizing the margin, SVMs are capable of accurately predicting the classes of new samples. The decision boundary is expressed as a linear equation as follows:

$$\boldsymbol{\chi}(\mathbf{x}) = \boldsymbol{w}^T \boldsymbol{x} + \boldsymbol{b} = \mathbf{0} \tag{3}$$

Here *w* is a weight vector. We want to find w and b that satisfy *y*(*xi*) *>* 0 for *ti* = 1 and *y*(*xi*) *<* 0 for *ti* = −1, that is, *ty*(*x*) *>* 0 for all samples (Vapnik, 1982). The larger value of *w* indicates that the more corresponding *x* contributes to classify data into two groups. A positive sign of *w* indicates that the larger value of a corresponding *x* raises the possibility of *ti* = 1, while a negative sign of *w* indicates that the larger value of a corresponding *x* raises the possibility of *ti* = −1.

#### **CROSS VALIDATION**

Cross validation (CV) is a method for evaluating how the ability of a learning machine such as an SVM is generalized to the unknown data that are not used in the training (Kohavi, 1995). In the CV, the data set is divided into two parts. One part is used for the training of the machine, and the other part is used for testing the machine's ability. This training and testing procedure is repeated using different partitioning. The CV is effective when the number of the data is limited.

We explain the *K*-fold CV for the SVM below. First, we divide the data set into *K* parts *C*1,. . . , *CK*. For each *k* = 1,. . . , *K*, we train the SVM using the data other than the *k*-th part *Ck*. We denote by *y\k* (*x*) the decision boundary. Then, by using this boundary *y\k* (*x*), we predict the classes of the data in *Ck*, and compare them against the true class labels *t*. We repeat this procedure for every *k* = 1,. . . , *K*, and calculate the following cross validation error (CVE):

$$\text{CVE} = \frac{1}{N} \sum\_{k=1}^{K} \sum\_{i \in \mathcal{C}\_k} L(t\_i, \boldsymbol{\upchi}\_k(\boldsymbol{\upchi})),\tag{4}$$

$$L(t, \boldsymbol{\gamma}(\boldsymbol{\alpha})) = \begin{cases} 1 \ (t\boldsymbol{\gamma} \,(\boldsymbol{\alpha}) > 0) \\ 0 \ (t\boldsymbol{\gamma} \,(\boldsymbol{\alpha}) < 0) \end{cases} \tag{5}$$

*L*(*t*, *y*(*x*)) indicates whether the prediction is correct or not. When the prediction is correct, *L*(*t*, *y*(*x*)) = 0, and when the prediction is incorrect, *L*(*t*, *y*(*x*)) = 1. CVE represents the ratio of the number of incorrectly predicted data to the total number of data. If the CVE is small, the generalization capability of the SVM is high. The classification accuracy is defined using the following formula:

$$(1 - \text{CVE}) \times 100\tag{6}$$

We used a 5-fold CV. In this study, we used three data sets; (1) *D* = 12 (12 channels placed on the right temporal area), (2) *D* = 12 (12 channels placed on the left temporal area), and (3) *D* = 24 (24 channels placed on the bilateral temporal areas). The above sets of (1) and (2) were subsets of (3).

We selected a subset *A* of *D* features and set *xiA* := (*xid* )*d*∈*A* ∈ R|*A*| . We then applied the 5-fold CV to the data set *xiS , ti N i* = 1 and calculated the CVE. We carried out this process for all (2*D* − 1) subsets.

Our aim was to find the optimal subsets that classify the data into two groups most correctly. In other words, we wanted to find the combination of NIRS measurement channels that distinctively respond to the experimental stimuli depending on which diagnostic groups the data belong.

#### **RESULTS**

We used the mean Z-scores of hemodynamic response obtained from each channel and the diagnostic group of participants as inputs and outputs for the SVM, respectively. We trained the SVM to find the optimal subset in the data set of channels in order to classify the data into two diagnostic groups. For each subset, we evaluated its classification accuracy using 5-fold CV.

We conducted the exhaustive search of three data sets of Z-scores of hemodynamic data: (1) the 12-dimension dataset obtained from the right 12 channels, (2) the 12-dimension dataset obtained from the left 12 channels, and (3) the 24-dimension dataset obtained from the bilateral 24 channels. For the 12 dimension datasets [(1) and (2)], SVM training and calculation of the CVE was repeated with 212 − 1 = 4095 subsets. For the 24 dimension dataset (3), SVM training and calculation of the CVE was repeated with 224 − 1 = 16*,* 777*,* 215 subsets.

## **RESULTS OF SVM CLASSIFICATION APPLIED ON THE DATA OBTAINED FROM THE RIGHT HEMISPHERE**

We found nine subsets to classify the data more correctly into two groups among the 4095 subsets. The measurement channels which comprised those subsets are listed in **Table 1**. The best classification accuracy for these subsets was 70%. **Figure 2A** shows the classification accuracy for all 4095 subsets.

**Figure 2B** shows the feature subsets corresponding to the classification accuracy represented in **Figure 2A**. **Figure 2B** shows that Ch. 15, 16, and 18 repeatedly appear in the subsets represented on the left side, which have relatively higher classification accuracy. These channels are filled with darker color, indicating these channels have a stronger coefficient than other channels. Furthermore, the colors of these three channels were consistent among all the subsets, respectively. This indicates that their weight vectors have the same sign commonly in all the subsets.

Moreover, **Figure 2B** shows that the red colored cells and blue colored cells are separated. The lower-numbered channels (Ch. 13–17) are constantly red and the higher-numbered channels (Ch. 18, 19, and 23) are constantly blue. This suggests that in the right hemisphere the brain area contributing to classification are separated. On the other hand, other channels were constantly filled with yellowish color. These channels do not have strong weight vectors, thus, they do not contribute to the classification in all of the 4095 subsets.

**Figure 3** shows the classification accuracy of the best 50 subsets (**Figure 3A**) and the corresponding feature subsets (**Figure 3B**).

**Table 1 | The measurement channels those comprised the subsets with best classification accuracya.**

*aThe numbers indicated the channel numbers. In the right hemisphere, the number of the best subset was nine. The percentage represented in the parenthesis indicating the best classification accuracy for each data set.*

Each cell indicates the weight vector of each channel in the feature subset. Ch. 15, 16, and 18 appear in most of subsets. Ch. 15 was in 49 of the 50 subsets. Ch. 16 was in 45 subsets and Ch. 18 was in 40 subsets, respectively. These three channels were more often used in the best 50 subsets and more effectively contributed to the classification. The positions of these channels are illustrated in **Figure 3C**.

The color of the cells corresponding to Ch. 15 and 16 are red, but Ch. 18 is blue. This tendency is consistent through the best 50 subsets. This result indicated that the greater hemodynamic response in Ch. 15 and 16 more often occurred in ADHD participants than in ASD participants, and those at Ch. 18 were more often found in ASD participants than in ADHD participants.

### **RESULTS OF SVM CLASSIFICATION APPLIED ON THE DATA FROM THE LEFT HEMISPHERE**

The best subset for classifying the data into two groups contained the five channels listed in **Table 1**. This subset had the best classification accuracy of 74%. **Figure 4** shows the classification accuracy for all 4095 subsets (**Figure 4A**) and the corresponding feature subsets (**Figure 4B**). Ch. 5 and 6 constantly appear in the subsets with relative higher classification accuracy and had greater weight value than the other channels. Furthermore, the color of their cells is consistent through all the subsets. This indicates the sign of the weight vector is common in all the subsets.

Compared with the result from the right hemisphere (**Figure 3B**), the red colored cells and blue colored cell are scattered among the channels. This suggests that in the left hemisphere the brain areas contributing to classification are not clearly separated.

**FIGURE 2 | (A)** The classification accuracy corresponding with 4095 subsets consisted from the channels in the right hemisphere. Horizontal axis represents the rank-ordered subsets. Vertical axis indicates the classification accuracy. The highest accuracy is represented on the left side. **(B)** The channel numbers in the subsets represented in **(A)**. Horizontal and vertical axes represent the rank-ordered feature subsets and the channel numbers, respectively. Colored cells indicate that the channel was in the subset and black cells indicate that the channel was not in the subset. The color of each cell indicates a value of weight vector corresponding to the Z-score of each channel. Red/yellow spectrum indicates positive value and blue spectrum indicates negative value.

**Figure 5** shows the classification accuracy for all 4095 subsets (**Figure 5A**) and the corresponding feature subsets (**Figure 5B**). Ch. 6 was most often used in the best 50 subsets. The channel was in 43 of the 50 subsets. Following Ch. 6, 3, and 7 were in 32 subsets. It is worth mentioning that Ch. 5 and 6 appear alternately in the top 17 subsets. The position of these channels is illustrated in **Figure 5C**.

Ch. 3, 5, and 7 are red and only Ch. 6 is blue. This tendency is consistent through the best 50 subsets. This result indicated that at Ch. 3, 5, and 7 ADHD participants showed greater hemodynamic response than ASD participants, and only at Ch. 6 did ASD participants show greater hemodynamic response than ADHD participants.

### **RESULTS OF SVM CLASSIFICATION APPLIED ON THE DATA FROM BILATERAL HEMISPHERES**

We found two subsets for classifying the data more correctly into two groups among the 16,777,215 subsets. The measurement channels that comprised those subsets listed in **Table 1**. These subsets had the best classification accuracy of 84%. **Figure 6A** shows the classification accuracy of the subsets with accuracy above 70%. The classification accuracy was higher when using the data set from the bilateral temporal areas than when using data sets from the one-side temporal area. This result indicated that the bilateral temporal areas interacted with each other and that the combined data analysis increased the amount of information about the classification.

**Figure 6B** shows the feature subsets corresponding to the classification accuracy represented in **Figure 6A**. As in the analysis on the data sets from each hemisphere, Ch. 6 appeared constantly and was colored blue.

**Figure 7** shows the classification accuracy for all 4095 subsets (**Figure 7A**) and the corresponding feature subsets (**Figure 7B**). Ch. 6 was in all of the 50 subsets and was most often used in the

best 50 subsets. Ch. 3 and 14 were in 49 subsets. The position of these channels is illustrated in **Figure 7C**.

Ch. 3, 5, and 14 are red and only Ch. 6 is blue. This tendency is consistent through the top 50 subsets. This result indicated that the greater hemodynamic response in Ch. 3, 5, and 14 more often occurred in ADHD participants than in ASD participants, and that those at Ch. 6 are more often found in ASD participants than in ADHD participants.

### **LASSO AND SPARSE LOGISTIC REGRESSION**

To demonstrate the effectiveness of the exhaustive search, we compared the exhaustive search with two existing methods, least absolute shrinkage and selection operator (LASSO, Tibshirani, 1996) and SLR (Yamashita et al., 2008). These are classification methods that incorporate feature selection as a part of the process of classifier training. We applied LASSO and SLR to datasets (1)–(3), and evaluated classification accuracies by using 5-fold CV.

By using LASSO, the classification accuracies were (1)57.5 and (2)66%. These accuracies rank (1)1106th (in the top 27.0%) and (2)147th (in the top 3.59%) of the 4095 accuracies obtained by the exhaustive search. The classification accuracy in dataset (3) was 70%, and this ranks 139,815th (in the top 0.833%) of the 16 million accuracies obtained by the exhaustive search.

By using SLR, the classification accuracies were (1)52 and (2)64%. These accuracies rank (1)1978th (in the top 48.3%) and (2)372th (in the top 9.08%) of the 4095 accuracies obtained by the exhaustive search. The classification accuracy in dataset (3) was 66%, and this ranks 698,955th (in the top 4.17%) of the 16 million accuracies obtained by the exhaustive search.

subsets consisted from the channels in the bilateral hemisphere. **(B)** The channel numbers in the subsets represented in **(A)**. **(C)** The position of the channels referred to in the text.

These results showed that, in datasets (2) and (3), the classification accuracies using LASSO and SLR were high and were located near the top of all the accuracies obtained by the exhaustive search. In comparison, in dataset (1) the classification accuracies were low compared to those in datasets (2) and (3), and were in the middle of all the accuracies obtained by the exhaustive search. These findings highlighted the fact that by using LASSO and SLR, the classification accuracy is not always high compared to the best result obtained by the exhaustive search, depending on the dataset. This fact was not revealed unless performing the exhaustive search. This means that neither LASSO nor SLR could uncover the latent structures relevant to discrimination between ASD and ADHD in the high-dimensional fNIRS data.

#### **CHANNEL-WISE ANALYSIS OF 24 CHANNELS**

With the aim of comparing the effectiveness of SVM with a standard method, we also performed a channel-wise *t*-test on each channel. A two-tailed two-sample *t*-test was conducted for the difference of Z-scores between the ADHD participants and ASD participants during the 3–10 s of the test trials. For all 24 channels, a *t*-test was performed. To reduce the risk of a Type I error, we performed the corrections using the false discovery rate (FDR) (Singh and Dan, 2006).