We plotted voltage topography and scalp current density (SCD) maps of ERPs elicited by deviants and standards, and of the deviant-minus-standard difference potentials. Calculation and plotting was carried out by using the *sphspline* plug-in (Widmann, 2006) for EEGlab (Delorme and Makeig, 2004). As there were no striking differences in the distribution of deviant-related activity between the two target-to-object assignments we collapsed the data obtained in the two conditions. The time window was set to 246–280 ms thus, equally comprising the peaks of deviant-related activity of both target-toobject assignments. Furthermore, we applied Variable Resolution Electromagnetic Tomography (VARETA, Bosch-Bayard et al., 2001) in order to localize cortical generators of deviant-related activity. The VARETA technique uses a discrete spline distributed inverse model to estimate the spatially smoothest intracranial distribution of primary current densities that correspond to the EEG-signals measured at the scalp. In doing so VARETA estimates the smoothing parameter voxel-wise, thus allowing for variable amounts of spatial smoothness and localizing discrete and distributed sources with equal accuracy (Bosch-Bayard et al., 2001; Pizzagalli, 2007). We mapped possible sources on a 3D regular grid model (3244 voxels, 7 mm grid spacing) based on the probabilistic brain tissue maps available from the Montreal Neurological Institute (MNI, Evans et al., 1993) which restricts sources to the gray matter. Significant activations were displayed as 3D-images by computing statistical parametric maps of the estimated primary current densities based on a voxel-by-voxel Hoteling *T*2-test against zero. Random field theory (Worsley et al., 1996) was applied to correct thresholds for spatial dependencies between voxels. To localize deviant-specific activation we contrasted the solutions obtained for deviants with those obtained for standards.

# **RESULTS**

#### **BEHAVIORAL DATA**

When we asked for specifics of the design at the end of the experiment five out of our 16 participants reported that the task-relevant discs and the enclosing ellipses (i.e., the objects) were somehow related: they noticed that the targets could be enclosed in either the same object or in different objects. However, all but one 1 did not report spontaneously that they realized any difference in the frequency of the occurrence of the two types of target-to-object assignments. Even after we presented a figure displaying both target-to-objects assignments and we explicitly inquired whether they occurred with different frequencies none of the participants reported that they realized the occurrence of frequently and infrequently presented assignments within one block, i.e., participants neither realized object-based regularities nor violations of these regularities.

However, results of the repeated measures ANOVA with the factors STIMULUS TYPE (standards vs. deviants) and TARGET-TO-OBJECT ASSIGNMENT (discs belonging to the same object vs. discs belonging to different objects) conducted on the reaction times showed that the performance of the participants was significantly influenced by the (unnoticed) object-based regularities: participants responded significantly faster in trials with frequently presented target-to-object assignments (i.e., in standard trials, mean RT 505 ms ± 14 ms *SEM*) compared with trials with infrequent target-to-object assignments [i.e., in deviant trials, 515 ± 14 ms, main effect of factor STIMULUS TYPE: *F*(1, 15) = 35.5, *p* < 0.001, η2 *p* = 0.70]. Furthermore, participants responded slightly faster when discs belonged to different objects compared with when discs belonged to the same object [507 ± 14 ms vs. 513 ± 14 ms, main effect of factor TARGET-TO-OBJECT ASSIGNMENT: *F*(1, 15) = 5.75, *p* = 0.03, η2 *p* = 0.28]. There was no interaction of the two factors [*F*(1, 15) = 0.35, *p* > 0.5]. On average participants responded correctly in 95.76% ± 0.5 of all trials. Hits rates were not significantly affected by neither the factor STIMULUS TYPE nor TARGET-TO-OBJECT ASSIGNMENT (both *F* < 1). The interaction of the two factors only marginally failed to reach

significance [*F*(1, 15) = 4.52, *p* = 0.051, η2 *p* = 0.23]. However, none of the possible follow-up comparisons reached significance [all *t*(df = 15) < −1.65, all *p* > 0.1 even without correction for multiple comparisons]. Behavioral results are summarized in **Table 1**.

#### **ELECTROPHYSIOLOGICAL DATA**

**Figure 2** displays the grand average ERPs elicited by deviant and standard displays superimposed with the respective deviantminus-standard differences waveforms, separately for the two target-to-object assignments (discs belonging to the same object vs. discs belonging to different objects). Deviant and standard displays of both target-to-object assignments elicited a representative sequence of prominent visual ERP components at posterior electrode sites: P1 peaking at 95 ms, N1 at 150 ms, P2 at 205 ms and N2 at around 260 ms which was followed by a broad-peaked P3b in the 350–550 ms latency range (**Figure 2**). In the P1 and N1 latency range deviant and standard ERPs are nearly perfectly matched. In contrast, in the N2 latency range deviant ERPs clearly show a more negative response than standard ERPs. Visual inspection revealed that deviant-specific responses were most prominent at posterior-temporal electrode sites (**Figure 2**, lower row) whereas there were no deviant-specific responses at frontal electrode sites (**Figure 2**, upper row). The posterio-temporal distribution of deviant-specific responses is also illustrated by the corresponding potential maps and SCD maps (**Figure 3**, upper and middle row). Visual inspection further revealed that there were no differences between deviant and standard ERPs at frontocentral electrode sites at latency ranges around 400 ms poststimulus, i.e., we did not find evidence that deviants elicit the P3a component.

Results of a repeated measures ANOVA conducted on the mean amplitudes in the N2 latency range with the factors STIMULUS TYPE (standards vs. deviants) × TARGET-TO-OBJECT ASSIGNMENT (discs in the same object vs. discs in different objects) × HEMISPHERE (left vs. right)

#### **Table 1 | Behavioral performance. Stimulus type RT (ms) Hit rates (%) Deviants Standards Deviants Standards** Target-to-object assignmentsDiscs belonging to the same object Discs belonging to the different objects

*Reaction times (RT) and hit rates are displayed separately for deviant (red outlines) and standard trials (blue outlines) for the two target-to-object assignments, respectively. SEM are given in parentheses. Cells containing responses given within one experimental condition are marked by identical gray-scale and line-style (dark-gray cells with solid outlines correspond to the "same-objectstandard-condition," light-gray cells with dashed outlines correspond to the "different-object-standard-condition"). Responses given to physically identically deviants and standards are contrasted line-by-line. Asterisks indicate significant differences between deviant- and standard-responses averaged over the two target-to-object assignments (\*\*\*p* < *0.001).*

1One participant reported that displays containing discs belonging to the same object occurred more frequently throughout the whole experimental session.

× ROI (posterio-temporal vs. frontal) confirmed that deviants exhibited significantly more negative amplitudes than standards [main effect of STIMULUS TYPE: *F*(1, 15) = 28.19, *p* < 0.001, η2 *p* = 0.65]. This effect was restricted to the posterior-temporal ROI [interaction of STIMULUS TYPE × ROI, *F*(1, 15) = 53.59, *p* < 0.001, η2 *p* = 0.78]. A significant threefold interaction of STIMULUS TYPE × HEMISPHERE × ROI [*F*(1, 15) = 5.56, *p* = 0.032, η2 *p* = 0.27] suggests that deviant specific responses found at the posterior-temporal ROI were more accentuated in the right hemisphere (−2.6μV ± 0.4 SEM vs. −2.3μV ± 0.4 in the right vs. left hemisphere). Follow-up analyses, however, failed to reach significance [*t*(df = 15) = −1.67, *p* > 0.1]. In general, amplitudes at the posterior-temporal ROI were more negative than amplitudes at the frontal ROI [main effect of ROI, *F*(1, 15) = 20.58, *p* < 0.001, η2 *p* = 0.58]. Amplitudes were not modulated by neither the TARGET-TO-OBJECT ASSIGNMENT [*F*(1, 15) = 3.14, *p* = 0.1] and the HEMISPHERE [*F*(1, 15) = 0.03, *p* = 0.87] itself nor by anyone of the other possible interactions of factors [all *F*(1, 15) < 3.25, all *p* > 0.09]. Mean amplitudes of deviant and standard responses for the two target-to-object assignments are summarized separately for the posterior-temporal ROI and the frontal ROI, respectively, in **Table 2**.

The potential map of the deviant-minus-standard difference waves reveals a broadly distributed occipito-temporal twopeaked negative potential (**Figure 3**, upper row, right column). The corresponding SCD topography exhibits prominent bilateral occipito-temporal sinks accompanied by a weak source over the central occipital region and distributed weak sources over fronto-central areas (**Figure 3**, middle row, right column). Source analyses conducted by the VARETA approach show that brain activity elicited by deviant-trials is generated

in the posterior part of the inferior/middle temporal gyrus (MNI coordinates *X*, *Y*, *Z*: 50/−50, −62, −10) and at the occipital pole (17/−17, −95, −1, **Figure 3**, lower row, left column). Activity elicited by standard-trials is generated more superiorly in the middle temporal gyrus (50/−50, −62, −2) and at the occipital pole (15/−15, −98, −2), too (**Figure 3**, lower row, middle column). We contrasted source localizations obtained for deviants and standards for highlighting regions with **Table 2 | Mean amplitudes (**μ**V) elicited by deviants (red outlines) and standards (blue outlines) at posterio-temporal ROI (electrodes P5/6, P7/8, PO7/8) and frontal ROI (electrodes AF3/4, F3/4, F5/6) in the N2-latency range.**

*Responses are displayed separately for the two target-to-object assignments. SEM are given in parentheses. As in Table 1 cells containing responses given within one experimental condition are marked by identical gray-scale and line-style (dark-gray cells with solid outlines correspond to the "same-objectstandard-condition," light-gray cells with dashed outlines correspond to the "different-object-standard-condition"). Responses given to physically identically deviants and standards are contrasted line-by-line. Asterisks indicate significant differences between deviant- and standard-responses averaged over the two Target-to-object assignments (\*\*\*p* < *0.001).*

*Numerically the vMMN-amplitudes differed between the two target-to-object assignments (*−*2.83* ± *0.4*µ*V vs.* −*2.13* ± *0.4* µ*V when discs belonged to the same vs. different objects). However, within the present data this difference does not reach significance [interaction between the factors STIMULUS TYPE* × *TARGET-TO-OBJECT ASSIGNMENT F*(*1*, *15*) = *0.3, p* = *0.1 when we conducted the ANOVA for the posterior ROI only].*

deviant-specific activation. Deviant-specific activation was generated in the inferior temporal gyrus (50/−50, −62, −10, **Figure 3**, lower row, right column) and showed a right-hemispheric accentuation.

### **DISCUSSION**

In the present study, we investigated automatic visual object formation by testing whether task-irrelevant violations of object-based regularities are capable of (1) influencing implicit behavioral measures and (2) eliciting the vMMN—an automatic ERP-component which indexes the detection of a mismatch between the actual stimulus and a prediction generated on the basis of regularities extracted from the preceding sequence of stimuli (Kimura et al., 2011b). Importantly, in the present design violations of object-based regularities were exclusively related to the (non-salient) assignment of task-relevant elements of the display to the objects, i.e., there were no salient violations of feature-regularities. Indeed, our participants did not notice any object-related regularity or any violation of regularity within the sequence of stimuli. Nevertheless, task-irrelevant violations of object-related regularities resulted in increased reaction times. This result is in line with behavioral studies which showed that the regular organization of task-irrelevant background elements influences the processing of task-relevant items via perceptual grouping (e.g., Driver et al., 2001; Kimchi and Razpurker-Apfeld, 2004; Russell and Driver, 2005; Lamy et al., 2006; Kimchi and Peterson, 2008; Shomstein et al., 2010).

Extending these behavioral indicators, our study provides electrophysiological evidence for automatic visual object formation: compared with physically identical standard displays, those displays violating the regular target-to-object assignment elicited higher negative potentials in the 246–280 ms time window over posterio-temporal electrode positions. Latency and topography of this negative difference potential correspond to the characteristics of the vMMN elicited in an experiment designed to disentangle effects of sensory, N1-refractoriness-based deviance detection from genuine cognitive effects based on the violation of predictions (Kimura et al., 2009). Our results show that in the P1-N1 latency range ERPs elicited by deviant- vs. standarddisplays were nearly perfectly matched. Thus, we could convincingly show that the visual system is capable of detecting violations of higher-level regularities automatically even if those violations are not accompanied by N1-refractoriness-effects. Moreover, as we did not find any evidence for the elicitation of the P3a component—a component which is considered as an indicator of involuntary attention shifts (for a review see, Escera et al., 2000)—we conclude that the irregularities in our design indeed were detected without shifting attention toward the task-irrelevant aspects of the displays. In contrast, in visual studies investigating the processing of salient lower-order regularities/irregularities the elicitation of N1-differences/vMMN was accompanied by the elicitation of the P3a. For this reason in those studies the behavioral impairment observed in the processing of irregular stimuli was ascribed to costs related to involuntary attention shifts toward task-irrelevant aspects of the stimuli (Berti and Schröger, 2001, 2004, 2006; Kimura et al., 2008a,b). In contrast, in our design we observed increased reaction times for irregular displays compared with regular displays without an accompanying P3a (for similar results obtained in a visual multi-deviant design see, Grimm et al., 2009). Thus, the differences in the reaction times could be (at least partly) due to the facilitated processing of regular displays rather than the exclusive impaired processing of irregular displays. However, the elicitation of vMMN in our design suggests that the processing of irregular displays was associated with genuine costs, too.

So far, the automatic detection of higher-level regularities in the visual modality was shown by means of facial emotional expressions only [reviewed in Winkler and Czigler (2012)]. Our results show that the detection of higher-level regularities is not restricted to the ecologically highly important emotional expression of human faces but extends to rather general element-toobject assignments as regularities and irregularities in our design were solely defined on the basis of object-related characteristics. The elicitation of vMMN by object-related irregularities suggests that the process of object formation must have preceded the process of irregularity detection, i.e., our results support the notion of automatic visual object formation based on the Gestalt principle of common region. This conclusion fits to a recently published article reporting the elicitation of vMMN by violations of a conditional rule: task-irrelevant stimuli were presented pairwise in close temporal proximity with regularly both stimuli within one pair had the same color whereas irregularly the second stimulus of a pair took on a different color (Stefanics et al., 2011). As both colors occurred equiprobably within one experimental block regularities/irregularities were defined on the basis of the relation between the two elements of a pair (e.g., if the first stimulus is green then the second stimulus is green, too). Pairs of stimuli in this design can be seen as objects based on the Gestalt principle of temporal proximity. Thus, as in our study, object formation must have preceded the process of irregularity detection which suggests automatic visual object formation to be a more general mechanism. Such automatically formed object representations were recently suggested to be regarded as components of generative models which on the one hand predict the specifics of the upcoming stimulation and which on the other hand are modified by mismatches between the predicted and the actual stimulus (Winkler and Czigler, 2012).

In our study, we identified brain structures related to the violation of object-based regularities by computing SCD maps and applying VARETA. Our SCD maps show a bilateral occipital/occipito-temporal distribution of deviant-specific negative potentials in the 246–280 ms time range. Source analysis carried out by the VARETA technique localized our object-related vMMN to the posterior part of the inferior temporal gyrus (Brodmann's area 37). In numerous articles the inferior temporal gyrus—a structure belonging to the ventral pathway of visual information processing- was shown to be associated with high-order visual object processing in humans or macaques (e.g., Baizer et al., 1991; Goodale and Milner, 1992; Malach et al., 1995; Ishai et al., 1999; Haxby et al., 2001; Grill-Spector and Malach, 2004). The localization of object-related effects of irregularity detection in the inferior temporal gyrus corroborates our recently published localization data (Müller et al., 2012). Here, a combination of object-related and feature (color)-related irregularities generated activation in the inferior temporal gyrus, too. Moreover, also the aforementioned vMMN studies on facial emotional expressions, i.e., on material containing higher-order regularities, found activation related to regularity violation in the inferior temporal gyrus (Kimura et al., 2011a; Stefanics et al., 2012). In contrast, vMMN studies based on feature (orientation and/or color)-related regularities localized deviant-specific activity to earlier anatomical structures of the cortical visual system (occipital lobe—BA 19—Kimura et al., 2010; middle occipital gyrus—Urakawa et al., 2010a,b; occipital fusiform regions—BA 17, 18, 19/7—Yucel et al., 2007). The activation of different feature-/stimulus-specific cortical structures by different types of deviants parallels results from irregularity detection in the auditory modality (e.g., Alain et al., 1999; Rosburg, 2003; Grimm et al., 2006). Interestingly, in all of the vMMN studies cited above deviant-specific activity based on higher-order irregularities or on feature irregularities was additionally found in prefrontal cortical regions (mainly the inferior frontal/medial frontal cortex). It seems plausible to assume that this deviant-specific prefrontal activation indicates involuntary attention shifts toward ecologically relevant irregularities in either facial expression (Kimura et al., 2011a; Stefanics et al., 2012) or hand laterality (Stefanics et al., 2012) and toward salient feature irregularities, respectively (Kimura et al., 2010; Urakawa et al., 2010a,b; Yucel et al., 2007). In contrast, the source localization of our non-salient object-related

irregularities does not show a prefrontal activation, which might again underline that in our design there were no involuntary attentional shift and object-related information indeed was processed automatically. However, there are alternative suggestions regarding the functional role of the frontal generator of the auditory MMN (1) sensitivity tuning for irregularity detection in the auditory modality (e.g., Doeller et al., 2003), (2) inhibiting the tendency to respond to task-irrelevant auditory irregularities (Rinne et al., 2005), or (3) updating predictive models on the nature of upcoming stimuli (e.g., Garrido et al., 2009). The latter alternative is also taken into account for explaining the function of the frontal generators of the vMMN (Kimura et al., 2011a). The vMMN studies reporting combined cortical activation of feature-/stimulus-specific regions as well as of frontal regions suggest that the detection of irregularities in both the visual and the auditory modality works in a comparable hierarchically organized manner (for a model see Garrido et al., 2009). In contrast, our results as well as several studies on irregularity detection in the auditory modality suggest that irregular stimuli can elicit a mismatch response even without an accompanying frontal activation (for a review on the frontal generator of the auditory MMN see, Deouell, 2007). It remains a topic of further studies to investigate under which conditions irregularity detection is indicated by both stimulus-specific and frontal activation.

In conclusion, our results show (1) that object-based irregularities are automatically detected presumably by the visual subsystem encoding and/or processing object-related information. That is, we showed that object formation based on the Gestalt principle of common region must have occurred before the visual input was checked for the occurrence of regularities/irregularities. As the visual regularity extraction process was shown to work automatically (Berti, 2011; Kogai et al., 2011) we concluded that the process of object formation which in our design necessarily preceded the regularity extraction process should work automatically, too. Thus, our results support the notion of automatic visual object formation which parallels findings from the auditory modality for which the occurrence of automatic object formation also has been proved (e.g., Ritter et al., 2000; Atienza et al., 2003; Winkler et al., 2003; Sussman et al., 2007). (2) Although closely connected to our first conclusion we can state additionally that the detection of irregularities within sequences of visual stimuli is not restricted to salient stimulus attributes but also works for non-salient higher-order stimulus attributes thus emphasizing the sensitivity of processes extracting regularities from our environment.

# **ACKNOWLEDGMENTS**

This work was financially supported by the German Research Foundation (No. Schr 375/16). The experiment was realized using Cogent 2000 developed by the Cogent 2000 team at the FIL and the ICN and Cogent Graphics developed by John Romaya at the LON at the Wellcome Department of Imaging Neuroscience. We thank Katrin Bernhard and Jana Eliasova for assistance in data acquisition.

# **REFERENCES**

*J. Psychophysiol.* 21, 224–230. doi: 10.3389/fnhum.2011.00046

doi: 10.1146/annurev.neuro.27.<PHONE_NUMBER>20

*Psychophysiology* 48, 4–22. doi: 10.1111/j.1469-8986.2010.01114.x

*Psychophys.* 67, 606–623. doi: 10.3758/BF03193518

in the perception of objects. *Cogn. Psychol.* 14, 107–141. doi: 10.1016/0010-0285(82)90006-8

*Sci. U.S.A.* 100, 11812–11815. doi: 10.1073/pnas.2031891100

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 16 March 2013; accepted: 23 May 2013; published online: 10 June 2013.*

*Citation: Müller D, Widmann A and Schröger E (2013) Object-related regularities are processed automatically: evidence from the visual mismatch negativity. Front. Hum. Neurosci. 7:259. doi: 10.3389/fnhum.2013.00259*

*Copyright © 2013 Müller, Widmann and Schröger. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in other forums, provided the original authors and source are credited and subject to any copyright notices concerning any third-party graphics etc.*

# Task difficulty affects the predictive process indexed by visual mismatch negativity

# *Motohiro Kimura\* and Yuji Takeda*

*Cognition and Action Research Group, Human Technology Research Institute, National Institute of Advanced Industrial Science and Technology, Tsukuba, Japan*

#### *Edited by:*

*István Czigler, Institute of Cognitive Neuroscience and Psychology, Hungary*

#### *Reviewed by:*

*Toshihiko Maekawa, Kyushu University, Japan János Horváth, Research Centre for Natural Sciences, Hungary*

#### *\*Correspondence:*

*Motohiro Kimura, Cognition and Action Research Group, Human Technology Research Institute, National Institute of Advanced Industrial Science and Technology, Central 6, 1-1-1 Higashi, Tsukuba, Ibaraki 305-8566, Japan e-mail<EMAIL_ADDRESS>
Visual mismatch negativity (MMN) is an event-related brain potential (ERP) component that is elicited by prediction-incongruent events in successive visual stimulation. Previous oddball studies have shown that visual MMN in response to task-irrelevant deviant stimuli is insensitive to the manipulation of task difficulty, which supports the notion that visual MMN reflects attention-independent predictive processes. In these studies, however, visual MMN was evaluated in deviant-minus-standard difference waves, which may lead to an underestimation of the effects of task difficulty due to the possible superposition of N1-difference reflecting refractory effects. In the present study, we investigated the effects of task difficulty on visual MMN, less contaminated by N1-difference. While the participant performed a size-change detection task regarding a continuously-presented central fixation circle, we presented oddball sequences consisting of deviant and standard bar stimuli with different orientations (9.1 and 90.9%) and equiprobable sequences consisting of 11 types of control bar stimuli with different orientations (9.1% each) at the surrounding visual fields. Task difficulty was manipulated by varying the magnitude of the size-change. We found that the peak latencies of visual MMN evaluated in the deviant-minus-control difference waves were delayed as a function of task difficulty. Therefore, in contrast to the previous understanding, the present findings support the notion that visual MMN is associated with attention-demanding predictive processes.

**Keywords: attention, event-related brain potential, perceptual load, predictive process, prediction error, task difficulty, visual mismatch negativity**

# **INTRODUCTION**

**PREDICTIVE PROCESSES INDEXED BY VISUAL MISMATCH NEGATIVITY** The ability to extract sequential rules embedded in the temporal structure of sensory events and to predict upcoming sensory events based on the extracted sequential rules is crucial for successful adaptation to the external environment (e.g., Mumford, 1992; Friston, 2003, 2005). Recent electrophysiological studies have shown that such predictive processes in vision are well reflected by visual mismatch negativity (MMN), an event-related brain potential (ERP) component (for reviews, see Pazo-Alvarez et al., 2003; Czigler, 2007; Kimura et al., 2011; Kimura, 2012; Winkler and Czigler, 2012). Visual MMN is a negative-going ERP component with a posterior scalp distribution that usually emerges at around 150–400 ms after the onset of visual events. This component has been most typically observed in response to infrequent deviant stimuli that are randomly inserted among frequent standard stimuli (i.e., an oddball sequence). Importantly, however, the elicitation of visual MMN is not limited to such physically deviant stimuli, but rather includes a variety of stimuli that violate concrete or abstract sequential rules (e.g., Czigler et al., 2006; Kimura et al., 2010b, 2012; Stefanics et al., 2011). This leads to the notion that visual MMN emerges when a current visual event is incongruent with visual events that are predicted on the basis of extracted sequential rules (i.e., prediction error account of visual MMN; Kimura et al., 2011; Kimura, 2012).

#### **ATTENTION-INDEPENDENT PREDICTIVE PROCESSES**

One of the unique aspects of visual MMN elicitation is its automaticity. In most previous studies, visual MMN has been observed in response to deviant stimuli when oddball sequences are unrelated to the task and are not actively attended by the participant. This indicates that the elicitation of visual MMN is largely automatic and obligatory. This notion is further strengthened by the finding that visual MMN elicited by task-irrelevant deviant stimuli is insensitive to the manipulation of task difficulty (Heslenfeld, 2003; Pazo-Alvarez et al., 2004). Heslenfeld (2003) presented task-irrelevant oddball sequences consisting of deviant and standard grating stimuli with different spatial frequencies at the peripheral visual fields while the participant performed a visuo-motor tracking task that involved a small, continuously moving rectangle presented at the central visual field. The difficulty of the tracking task was manipulated among three levels (easy, moderate, and difficult) by varying the speed and frequency of changes in direction of the moving rectangle. Visual MMN elicited by deviant stimuli did not differ as a function of task difficulty. Pazo-Alvarez et al. (2004) obtained similar results. They presented task-irrelevant oddball sequences consisting of deviant and standard grating stimuli with different directions of motion at the peripheral visual fields while the participant performed a discrimination task that involved small colored digits discretely presented at the central visual field. The difficulty of the discrimination task was manipulated between two levels (easy and difficult) by asking the participant to perform either a task that involved the digit numbers (easy) or a task that involved the combination of both the digit numbers and the color of digits (difficult). Visual MMN elicited by deviant stimuli did not differ between the two task-difficulty conditions. According to the perceptual load theory of attention (Lavie and Tsal, 1994; Lavie, 1995, 2005), the task difficulty in the perceptual discrimination (i.e., perceptual load) of task-relevant information is one of the critical factors that determine the amount of attentional allocation to peripherally presented task-irrelevant information. Therefore, the lack of a task difficulty effect suggests that visual MMN is insensitive to the amount of attentional allocation, which leads to the notion that visual MMN reflects attention-independent predictive processes.

#### **PRESENT STUDY**

Although the results described by Heslenfeld (2003) and Pazo-Alvarez et al. (2004) support the notion that attentionindependent predictive processes underlie visual MMN, this idea needs to be studied further. In these previous studies, visual MMN was evaluated by comparing ERPs elicited by infrequent deviant stimuli to those elicited by frequent standard stimuli (i.e., deviant-minus-standard difference waves). However, more recent studies have questioned the validity of this comparison for the evaluation of visual MMN (see e.g., Czigler, 2007; Kimura et al., 2011; Kimura, 2012). This is because, due to the large difference in probability between deviant and standard stimuli, the state of refractoriness (or the level of habituation) of afferent neurons that specifically respond to the feature value of deviant stimuli can be drastically lower than that of afferent neurons that specifically respond to the feature value of standard stimuli. In other words, the amplitudes of visual evoked potentials (in particular, N1) in response to deviant stimuli can be substantially greater than those of N1 in response to standard stimuli. As a result, the classical visual MMN extracted in deviant-minus-standard difference waves [we refer to this effect as deviant-related negativity (DRN)] can include not only visual MMN elicited by deviant stimuli (i.e., prediction error effects) but also the N1 difference between deviant and standard stimuli (i.e., refractory effects) (for a more detailed discussion, see e.g., Czigler et al., 2002; Kenemans et al., 2003; Kimura et al., 2009). If we consider that these two effects often overlap each other both spatially and temporally in deviant-minus-standard difference waves (see e.g., Maekawa et al., 2005; Kimura et al., 2009, 2010b), it is possible that the effects of task difficulty on visual MMN have been underestimated in previous studies (Heslenfeld, 2003; Pazo-Alvarez et al., 2004). For example, if we assume that visual MMN and N1-difference contribute to DRN, as illustrated in **Figure 1A** (for similar empirical data, see e.g., Kimura et al., 2009, 2010b), neither the reduction of amplitudes (**Figure 1B**) nor the delay of latencies of visual MMN (**Figure 1C**) may be detected, at least with common ERP analyses that focus on the peak of DRN.

By considering this possibility, in the present study, we examined the effects of task difficulty on visual MMN, less contaminated by N1-difference. We used a so-called "equiprobable" protocol that allows for the reliable dissociation of visual MMN and N1-difference (e.g., Kimura et al., 2009; for the original protocol, see Schröger and Wolff, 1996; Schröger, 1997; Jacobsen and Schröger, 2001). While the participant performed a size-change detection task for a small fixation circle that was continuously presented at the central visual field, we presented either (1) typical oddball sequences consisting of the randomized presentation of deviant and standard bar stimuli with different orientations (e.g., 5.0 and 37.7◦ to the right from the horizontal; 9.1 and 90.9%, respectively) or (2) equiprobable sequences consisting of the randomized presentation of 11 types of equiprobable control bar stimuli with different orientations (5.0, 21.4, 37.7, 54.1, 70.5, 86.8, 103.2, 119.5, 135.9, 152.3, and

**FIGURE 1 | (A)** A schematic illustration of DRN, visual MMN, and N1-difference. **(B)** The modeled DRN as the sum of visual MMN with reduced amplitudes and N1-difference. **(C)** The modeled DRN as the sum of visual MMN with delayed latencies and N1-difference.

168.6◦; 9.1% each) at the surrounding visual fields in separate blocks (see **Figure 2B**). In this protocol, while the deviant stimuli should elicit visual MMN, the standard and control stimuli should not. This is because the standard and control stimuli do not violate any sequential rule. In addition, N1 elicited by the deviant stimuli should be equal to (or even smaller than) N1 elicited by the control stimuli, and should be greater than N1 elicited by the standard stimuli. This is because the probability of the deviant and control stimuli is kept the same (9.1%) and is lower than the probability of the standard stimuli (90.9%), and further, the physical separation among control stimuli (ca. 45.0◦, on average) is kept greater than that between deviant and standard stimuli (ca. 32.7◦) (for more detailed information, see Schröger and Wolff, 1996; Schröger, 1997; Jacobsen and Schröger, 2001; Kimura et al., 2009, 2010b). Thus, visual MMN (and possibly a small polarity-reversed N1-difference) should be extracted by comparing ERPs elicited by the deviant stimuli to those elicited by the control stimuli (i.e., deviantminus-control difference waves), while N1-difference should be extracted by comparing ERPs elicited by the control stimuli to those elicited by the standard stimuli (i.e., control-minusstandard difference waves). The difficulty of the size-change detection task for the central fixation circle was manipulated among three levels (easy, moderate, and difficult) by varying the magnitude of the size-change. With this experimental design, we examined the effects of task difficulty on visual MMN as well as DRN and N1-difference, and investigated whether or not the predictive processes reflected by visual MMN are truly attention-independent.

# **METHODS**

#### **PARTICIPANTS**

Twenty-two undergraduate and graduate university students (7 women, 15 men; age range = 20–25 years, mean = 21.5 years) participated in this experiment. Twenty-one participants were right-handed and one was left-handed. All participants had normal or corrected-to-normal vision and were free of neurological or psychiatric disorders. Written informed consent was obtained

for the central fixation circle in three task-difficulty conditions (easy, moderate, and difficult).

from each participant after the nature of the study had been explained. The experiment was approved by the National Institute of Advanced Industrial Science and Technology (AIST) Safety and Ethics committee.

#### **STIMULI AND PROCEDURE**

All stimuli were presented on a 17-inch cathode ray tube (CRT) display (Sony, Trinitron Multiscan G220), which was controlled by programs written in MATLAB (Mathworks, Inc.) with the Psychophysics Toolbox (Brainard, 1997; Pelli, 1997) installed on a computer (Apple, MacPro 1,1; NVIDIA, GeForce 7300GT). **Figure 2A** shows an example of the stimulus display consisting of a central fixation circle and surrounding bars. Eleven types of bar stimuli were used (**Figure 2B**). Each bar stimulus consisted of eight gray bars (luminance of 14.5 cd/m2 and visual angle of 3.0◦ (length) × 0.4◦ (width) from a viewing distance of 70 cm, respectively) at eight surrounding locations (3.3◦ upper, lower, left, and right and 4.6◦ upper-left, upper-right, lower-left, and lower-right from the center of the display to the center of each bar, respectively) against a black background (luminance of 0.1 cd/m2). The 11 types of surrounding bar stimuli differed in the orientation of the bars (5.0, 21.4, 37.7, 54.1, 70.5, 86.8, 103.2, 119.5, 135.9, 152.3, and 168.6◦ to the right from the horizontal (ca. 16.4◦ step), respectively). The exposure duration of the surrounding bar stimuli was fixed at 250 ms and the stimulus onset asynchrony was fixed at 500 ms (i.e., the inter-stimulus interval was fixed at 250 ms) in all conditions.

These surrounding bar stimuli were presented in 23 types of stimulus sequences (**Figure 2B**): 22 oddball sequences and one equiprobable sequence. In the oddball sequences, two types of surrounding bar stimuli (deviant and standard, 11 and 110 times/block, i.e., 9.1 and 90.9%, respectively) were presented in random order, with the exception that a standard stimulus was presented at least 11 times at the beginning of each block and each deviant stimulus was followed by at least one standard stimulus. In the equiprobable sequences, 11 types of surrounding bar stimuli (control, 11 times/block each, i.e., 9.1% each) were presented in random order, with the exception that each control stimulus was followed by at least one control stimulus with a different orientation. Through the use of these stimulus sequences, we could ensure that, on average, the physical properties of deviant, standard, and control stimuli were the same, which allowed us to evaluate visual MMN as well as N1-difference and DRN without any contamination by the effects of physical differences in the eliciting stimuli. Also, in these stimulus sequences, the probability of control stimuli was kept the same as that of deviant stimuli (9.1%), and the physical separation among control stimuli (ca. 45.0◦, on average) was kept greater than that between deviant and standard stimuli (ca. 32.7◦), which guaranteed that the state of refractoriness for control stimuli was equal to (or may be even lower than) that for deviant stimuli.

In addition to the surrounding bar stimuli, a gray fixation circle (luminance of 14.5 cd/m2 and visual angle of 1.1 × 1.1◦) was continuously presented at the center of the display throughout the blocks (**Figure 2A**). From time to time, the size of the fixation circle suddenly became smaller. The mean frequency of the size-change was four times/block (ranging from three to five times/block) and the exposure duration of the size-changed fixation circle was 100 ms in all conditions. To ensure that the timing of the size-change was independent of the surrounding bar stimulation, we segmented the whole period of each block into consecutive 50-ms intervals and randomly selected two consecutive intervals for the size-change (i.e., 100 ms), with the exception that the size-change did not occur within the 5.5-s interval at the beginning of each block (where the first 11 surrounding bar stimuli were presented) and at least a 1.5-s interval was inserted between a size-change and the subsequent size-change. To manipulate the task difficulty, three levels of magnitude of the size-change were used in separate blocks (**Figure 2C**): from 1.1 × 1.1◦ to 0.6 × 0.6◦ in the easy condition, to 0.8 × 0.8◦ in the moderate condition, and to 1.0 × 1.0◦ in the difficult condition.

The experiment consisted of 66 blocks (11 blocks for the easy oddball condition, 11 blocks for the moderate oddball condition, 11 blocks for the difficult oddball condition, 11 blocks for the easy equiprobable condition, 11 blocks for the moderate equiprobable condition, and 11 blocks for the difficult equiprobable condition), each of which consisted of the presentation of 121 surrounding bar stimuli. For half of the participants, oddball sequences #1–11 and the equiprobable sequence (**Figure 2B**) were used, while for the other half of the participants, oddball sequences #12–22 and the equiprobable sequence (**Figure 2B**) were used. The order of these blocks was randomized across participants.

The participant was seated in a reclining chair in a soundattenuated and electrically-shielded dimly lit room. Before the start of the experiment, the participant was instructed to focus on a fixation circle, ignore surrounding bars, and press a button with the right index finger as quickly and accurately as possible when the fixation circle became smaller. The participant was also asked to minimize any eye movement and blinking during each block. Before the start of each block, the participant was informed about the magnitude of the size-change of the fixation circle in the upcoming block (i.e., large, medium, or small).

#### **RECORDINGS**

The electroencephalogram (EEG) was recorded with a digital amplifier (Nihon-Kohden, Neurofax EEG1100) and silver-silver chloride electrodes placed at 26 scalp sites (Fp1, Fp2, F7, F3, Fz, F4, F8, FCz, T3, C3, Cz, C4, T4, T5, P3, Pz, P4, T6, PO7, PO3, POz, PO4, PO8, O1, Oz, and O2 according to the extended International 10–20 System). All electrodes were referenced to the nose tip. To monitor blinks and eye movements, vertical and horizontal electrooculograms (EOGs) were also recorded with two electrodes above and below the right eye and two electrodes at the right and left outer canthi of the eyes, respectively. The impedance of all electrodes was kept below 10 k-. The EEG and EOG signals were digitized at a sampling rate of 1000 Hz and bandpass-filtered at 1–30 Hz with a finite impulse response (FIR) filter. The EEG and EOG signals time-locked to the onset of surrounding bar stimuli were then averaged for nine categories defined by three stimulus types (deviant, standard, and control) and 3 task difficulties (easy, moderate, and difficult). Averaging epochs were 600 ms featuring a 100-ms pre-stimulus baseline. In the averaging procedure, (1) the first three epochs in each block, (2) epochs during which the size-change of the fixation circle occurred and the two subsequent epochs, (3) epochs during which the participant made a button press and the two subsequent epochs, (4) epochs preceded by deviant stimuli, and (5) epochs in which the signal changes exceeded ± 80μV on any of the electrodes, were excluded. As a result, the averaging number for deviant, standard, and control stimuli was, on average, 89, 735, and 921 times for the easy condition, 89, 735, and 919 times for the moderate condition, and 91, 745, and 928 times for the difficult condition, respectively.

#### **DATA ANALYSIS**

#### *Behavioral performance*

Behavioral performance was measured in terms of reaction time (ms), hit rate (%), and false alarm (times/block). Responses were scored as a hit if the button was pressed within 200–1000 ms after the onset of the change in the fixation circle. Responses outside this period were classified as a false alarm. These measures were subjected to repeated-measures ANOVAs with two factors: 2 Sequences (Oddball vs. Equiprobable) and 3 Task difficulties (Easy, Moderate, vs. Difficult). The Greenhouse–Geisser ε correction for the violation of sphericity was applied when appropriate. Effect sizes were calculated as partial eta squared (η2). *Post-hoc* comparisons involved paired *t*-tests with the Bonferroni correction.

#### *ERPs and difference waves*

Grand-average deviant-minus-standard difference waves were calculated for the three task-difficulty conditions. In the difference waves, a bilateral parieto-occipital (PO7 and PO8) maximum negativity (DRN) that peaked at 188 ms (easy condition, PO8), 196 ms (moderate condition, PO8), and 203 ms (difficult condition, PO8) was observed. To decompose DRN into N1-difference and visual MMN (and possibly, small polarityreversed N1-difference), grand-average control-minus-standard and deviant-minus-control difference waves were then calculated for the three task-difficulty conditions, respectively. In the control-minus-standard difference waves, a bilateral parietooccipital (PO7 and PO8) maximum negativity (N1-difference) that peaked at 193 ms (easy condition, PO8), 196 ms (moderate condition, PO8), and 197 ms (difficult condition, PO8) was observed. In the deviant-minus-control difference waves, a right parieto-occipital (PO8) maximum negativity (visual MMN) that peaked at 186 ms (easy condition, PO8), 193 ms (moderate condition, PO8), and 225 ms (difficult condition, PO8) was observed.

#### *Scalp distributions of N1-difference and visual MMN*

To compare the scalp distributions of N1-difference and visual MMN, the mean amplitudes of the control-minus-standard and deviant-minus-standard difference waves (within the 11 ms time-windows including ± 5 ms from the corresponding peak) at 13 posterior electrodes in the three task-difficulty conditions were subjected to repeated-measures ANOVAs with three factors: 2 Difference waves (Control-minus-standard vs. Deviant-minus-control), 13 Electrodes (T5, P3, Pz, P4, T6, PO7, PO3, POz, PO4, PO8, O1, Oz, vs. O2), and 3 Task difficulties (Easy, Moderate, vs. Difficult). Further, the same analysis was performed on the amplitude values that were normalized by vector length, where, for each of the six conditions defined by two difference waves and three task-difficulty conditions, each amplitude value was divided by the square root of the sum of the squared amplitudes over the 13 electrode locations (McCarthy and Wood, 1985). The Greenhouse–Geisser ε correction for the violation of sphericity was applied when appropriate. Effect sizes were calculated as partial η2. *Posthoc* comparisons involved paired *t*-tests with the Bonferroni correction.

#### *Mean amplitudes of DRN, N1-difference, and visual MMN*

To test the significance of the elicitation of DRN, N1-difference, and visual MMN, the mean amplitudes of the deviant-minusstandard, control-minus-standard, and deviant-minus-control difference waves (within the 11-ms time-windows including ± 5 ms from the corresponding peak) at an electrode (PO8, where these components had the maximum amplitudes) in the three task-difficulty conditions were subjected to one-tailed paired *t*-tests. The effect sizes are presented as *d*-values. Further, to compare the mean amplitudes of each component among the three task-difficulty conditions, the mean amplitudes of each component were subjected to repeated-measures ANOVAs with one factor: 3 Task difficulties (Easy, Moderate, vs. Difficult). The Greenhouse–Geisser ε correction for the violation of sphericity was applied. Effect sizes were calculated as partial η2. *Posthoc* comparisons involved paired *t*-tests with the Bonferroni correction.

#### *Peak latencies of DRN, N1-difference, and visual MMN*

To estimate the peak latencies of DRN, N1-difference, and visual MMN, a jackknife method was used (Miller et al., 1998; Ulrich and Miller, 2001; Kiesel et al., 2008). With regard to 22 sub-grandaverage difference waves at PO8 electrode for each component in each task-difficulty condition, the peak latency was evaluated as the time at which the difference waves reached the peak amplitude of each component, within 100–300 ms after stimulus onset. To compare the peak latencies of each component among the three task-difficulty conditions, the evaluated peak latencies of each component were then subjected to repeated-measures ANOVAs with one factor: 3 Task difficulties (Easy, Moderate, vs. Difficult). The Greenhouse–Geisser ε correction for the violation of sphericity was applied. The *F*-values were corrected according to Ulrich and Miller (2001). The effect sizes are shown as partial η2. *Post-hoc* comparisons involved paired *t*-tests with the Bonferroni correction.

#### *Mean amplitudes of visual evoked potentials*

To examine the effects of task difficulty on visual evoked potentials, the mean amplitudes of standard and control ERPs (within each of 20 consecutive 10-ms time-windows from 100 to 300 ms) at the PO8 electrode in the three task-difficulty conditions were subjected to repeated-measures ANOVAs with two factors: 2 Stimuli (Standard vs. Control) and 3 Task difficulties (Easy, Moderate, vs. Difficult). The Greenhouse–Geisser ε correction for the violation of sphericity was applied. Effect sizes were calculated as partial η2. *Post-hoc* comparisons involved paired *t*-tests with the Bonferroni correction.

# **RESULTS**

#### **BEHAVIORAL PERFORMANCE**

The mean reaction time in the oddball condition was 459 ms (*SD* = 79), 463 ms (77), and 480 ms (77), while that in the equiprobable condition was 459 ms (78), 467 ms (76), and 479 ms (79), in the easy, moderate, and difficult conditions, respectively. Two-Way ANOVAs (2 Sequences × 3 Task difficulties) revealed a significant main effect of Task difficulty [*F*(2, 42) = 11.49, *p* < 0.001, ε = 0.97, partial η2 = 0.35]. *Post-hoc* comparisons showed that the reaction time in the difficult condition was longer than those in both the easy (*p* < 0.001) and moderate conditions (*p* < 0.05). The hit rate in the oddball condition was 95.4% (*SD* = 5.3), 94.5% (6.9), and 88.4% (11.9), while that in the equiprobable condition was 95.3% (6.3), 93.4% (8.6), and 87.6% (10.9), in the easy, moderate, and difficult conditions, respectively. Two-Way ANOVAs (2 Sequences × 3 Task difficulties) revealed a significant main effect of Task difficulty [*F*(2, 42) = 21.24, *p* < 0.001, ε = 0.74, partial η2 = 0.50]. *Post-hoc* comparisons showed that the hit rate in the difficult condition was lower than those in both the easy (*p* < 0.001) and moderate conditions (*p* < 0.01). The false alarm was negligible in all conditions (on average, less than 0.1 times/block). Two-Way ANOVAs (2 Sequences × 3 Task difficulties) revealed no significant effects (*F*s < 1.0).

#### **ERPS AND DIFFERENCE WAVES**

**Figure 3** shows the grand-average ERPs and EOGs in response to deviant, standard, and control stimuli in the easy (left column), moderate (middle column), and difficult conditions (right column). **Figure 4A** (left column) shows the traditional, grand-average deviant-minus-standard difference waves in the three task-difficulty conditions. A posterior negativity (DRN) that peaked at 188 ms (easy condition, PO8), 196 ms (moderate condition, PO8), and 203 ms (difficult condition, PO8) was observed. **Figure 4A** (middle column) shows the grand-average control-minus-standard difference waves in the three taskdifficulty conditions. A posterior negativity (N1-difference)

**FIGURE 3 | Grand-average ERPs and EOGs in response to deviant, standard, and control stimuli in the easy (left column), moderate (middle column), and difficult conditions (right column).**

**FIGURE 4 | (A)** Grand-average deviant-minus-standard difference waves (left column), grand-average control-minus-standard difference waves and topographical maps of N1-difference (middle column), and grand-average deviant-minus-control difference waves and topographical maps of visual MMN (right column), in the three task-difficulty conditions (easy, moderate, and difficult). **(B)** Grand-average mean amplitudes of DRN, N1-difference,

and visual MMN in the three task-difficulty conditions (electrode: PO8). Error bars indicate standard errors of the mean. **(C)** Grand-average peak latencies of DRN, N1-difference, and visual MMN in the three task-difficulty conditions (electrode: PO8). Error bars indicate standard errors of the mean with a jackknife method. Asterisks indicate a significant difference (*p* < 0.01).

that peaked at 193 ms (easy condition, PO8), 196 ms (moderate condition, PO8), and 197 ms (difficult condition, PO8) was observed. **Figure 4A** (right column) shows the grand-average deviant-minus-control difference waves in the three

task-difficulty conditions. A posterior negativity (visual MMN) that peaked at 186 ms (easy condition, PO8), 193 ms (moderate condition, PO8), and 225 ms (difficult condition, PO8) was observed; there was no clear sign of polarity-reversed N1.

#### **SCALP DISTRIBUTIONS OF N1-DIFFERENCE AND VISUAL MMN**

**Figure 4A** (middle and right columns) also shows topographical maps of N1-difference and visual MMN in the three task-difficulty conditions (within the 11-ms time-windows including ± 5 ms from the corresponding peak), respectively. The N1-difference had a scalp distribution that peaked at bilateral parieto-occipital electrodes (PO7 and PO8), while the visual MMN had a scalp distribution that peaked at a right parietooccipital electrode (PO8), regardless of the task-difficulty condition. Three-Way ANOVAs (2 Difference waves × 13 Electrodes × 3 Task difficulties) performed on the mean amplitudes of difference waves (within the 11-ms time-windows including ± 5 ms from the corresponding peak) revealed significant main effects of Difference wave [*F*(1, 21) = 12.96, *p* < 0.01, partial η2 = 0.38] and Electrode [*F*(12, 252) = 26.21, *p* < 0.001, ε = 0.19, partial η2 = 0.56], as well as a significant interaction of Difference wave × Electrode [*F*(12, 252) = 7.12, *p* < 0.001, ε = 0.25, partial η2 = 0.25]. Importantly, the significant interaction of Difference wave × Electrode was also present in the same Three-Way ANOVAs performed on the normalized mean amplitudes [*F*(12, 252) = 4.28, *p* < 0.01, ε = 0.26, partial η2 = 0.17]. *Post-hoc* comparisons revealed that the interaction mainly arose from the fact that the scalp distribution of N1-difference was bi-lateralized, while that of visual MMN was more right-lateralized.

#### **MEAN AMPLITUDES OF DRN, N1-DIFFERENCE, AND VISUAL MMN**

**Figure 4B** shows the grand-average mean amplitudes of DRN, N1-difference, and visual MMN in the three task-difficulty conditions (within the 11-ms time-windows including ± 5 ms from the corresponding peak at PO8 electrode). For the DRN, the mean amplitude was −3.04μV (*SE* = 0.34) in the easy condition, −3.14μV (0.37) in the moderate condition, and −2.62μV (0.23) in the difficult condition. One-tailed paired *t*-tests showed that DRN was significantly elicited in the easy [*t*(21) = −8.79, *p* < 0.001, *d* = 1.87], moderate [*t*(21) = −8.59, *p* < 0.001, *d* = 1.83], and difficult conditions [*t*(21) = −11.24, *p* < 0.001, *d* = 2.39]. However, a One-Way ANOVA (3 Task difficulties) revealed no significant effect (*F* = 1.0). For the N1-difference, the mean amplitude was −1.90μV (0.27) in the easy condition, −2.25μV (0.29) in the moderate condition, and −2.15μV (0.26) in the difficult condition. One-tailed paired *t*-tests showed that N1-difference was significantly elicited in the easy [*t*(21) = −6.92, *p* < 0.001, *d* = 1.48], moderate [*t*(21) = −7.80, *p* < 0.001, *d* = 1.66], and difficult conditions [*t*(21) = −8.26, *p* < 0.001, *d* = 1.76]. However, a One-Way ANOVA (3 Task difficulties) revealed no significant effect (*F* = 1.2). For the visual MMN, the mean amplitude was −1.17μV (0.33) in the easy condition, −0.89μV (0.34) in the moderate condition, and −0.93μV (0.31) in the difficult condition. One-tailed paired *t*-tests showed that visual MMN was significantly elicited in the easy [*t*(21) = −3.56, *p* < 0.01, *d* = 0.76], moderate [*t*(21) = −2.64, *p* < 0.05, *d* = 0.56], and difficult conditions [*t*(21) = −2.97, *p* < 0.01, *d* = 0.63]. However, a One-Way ANOVA (3 Task difficulties) revealed no significant effect (*F* = 1.9).

### **PEAK LATENCIES OF DRN, N1-DIFFERENCE, AND VISUAL MMN**

Peak latencies were calculated by a jackknife method (Miller et al., 1998; Ulrich and Miller, 2001; Kiesel et al., 2008). **Figure 4C** shows the grand-average peak latencies of DRN, N1-difference, and visual MMN in the three task-difficulty conditions (PO8 electrode). For the DRN, the peak latency was 188.7 ms (*SE* = 0.36) in the easy condition, 197.2 ms (0.25) in the moderate condition, and 203.5 ms (0.33) in the difficult condition. A One-Way ANOVA (3 Task difficulties) revealed no significant effect (*F*corrected = 1.3). For the N1-difference, the peak latency was 194.3 ms (0.36) in the easy condition, 197.6 ms (0.16) in the moderate condition, and 197.9 ms (0.10) in the difficult condition. A One-Way ANOVA (3 Task difficulties) revealed no significant effect (*F*corrected = 1.0). For the visual MMN, the peak latency was 185.9 ms (0.22) in the easy condition, 194.5 ms (0.81) in the moderate condition, and 226.2 ms (0.24) in the difficult condition. A One-Way ANOVA (3 Task difficulties) revealed a main effect of Task difficulty [*F*corrected(2, 42) = 4.35, *p* < 0.05, ε = 0.64, partial η2 = 0.17]. *Post-hoc* comparisons showed that the peak latency of visual MMN was longer in the difficult condition than in the easy condition [*t*corrected(21) = 5.59, *p* < 0.01].

#### **MEAN AMPLITUDES OF VISUAL EVOKED POTENTIALS**

**Figure 5A** shows the grand-average ERPs and EOGs in response to standard (left column) and control stimuli (right column) in the three task-difficulty conditions. **Figure 5B** shows the results of Two-Way ANOVAs (2 Stimuli × 3 Task difficulties) that were performed on the mean amplitudes of ERPs elicited by standard and control stimuli (within each of 20 consecutive 10-ms time-windows from 100 to 300 ms). Reflecting the larger N1 in response to control stimuli compared to standard stimuli (see the control-minus-standard difference waves shown in the middle panel of **Figure 4A**), a significant main effect of Stimulus was revealed for the 14 consecutive 10-ms time-windows from 140 to 280 ms [*F*s(1, 21) = 6.44–88.01, *p*s < 0.05–0.001, partial η2s = 0.24–0.81]. With regard to the Task-difficulty factor, a significant main effect of Task difficulty was revealed for the 3 consecutive 10-ms time-windows from 120 to 150 ms (i.e., the latency range of P1) [*F*s(2, 42) = 4.04–5.68, *p*s < 0.05–0.001, εs = 0.93–0.96, partial η2s = 0.16–0.21]. *Post-hoc* comparisons showed that P1 elicited by both standard and control stimuli was smaller in the difficult condition than in the easy condition (*p*s < 0.05). Further, a significant interaction of Stimulus × Task difficulty was revealed for the 2 consecutive 10-ms time-windows from 110 to 130 ms (i.e., the latency range of P1) [*F*s(2, 42) = 3.56–4.78, *p*s < 0.05, εs = 0.89–0.92, partial η2s = 0.15–0.19]. *Post-hoc* comparisons showed that P1 elicited by control stimuli was smaller in the difficult condition than in the easy condition (*p*s < 0.05), while P1 elicited by standard stimuli did not differ among the three task-difficulty conditions. Importantly, there was no significant main effect or interaction related to the Task-difficulty factor for the time-windows from 150 to 300 ms (i.e., the latency range of N1 and P2, where DRN, N1-difference, and visual MMN were emerged in the difference waves, see **Figure 4A**).

# **DISCUSSION**

Previous studies have shown that DRN (most likely, consisting of visual MMN and N1-difference) is insensitive to the manipulation of task difficulty (Heslenfeld, 2003; Pazo-Alvarez et al.,

2004), which supported the notion that visual MMN reflects attention-independent predictive processes. By taking into account the possible underestimation of the effect of task difficulty on visual MMN due to the superposition of N1-difference, we examined the effects of task difficulty on visual MMN, less contaminated by N1-difference, and investigated whether or not the predictive processes indexed by visual MMN are truly attention-independent.

#### **EFFECTS OF TASK DIFFICULTY ON VISUAL MMN**

Behavioral performance in the size-change detection task for the central fixation circle deteriorated (i.e., reaction times became slower and hit rates decreased) as the task difficulty increased, although no significant difference was observed between the easy and moderate conditions. These results confirm that the difficulty of the size-change detection task was successfully manipulated by varying the magnitude of the size-change (at least there was a difference between the difficult condition and the other two conditions).

In traditional deviant-minus-standard difference waves, a posterior negativity was observed at around 100–300 ms (DRN). The latency and scalp distribution are highly similar to those of DRN observed in previous studies (see e.g., Pazo-Alvarez et al., 2003; Czigler, 2007; Kimura, 2012). The DRN was then decomposed into N1-difference and visual MMN. In control-minus-standard difference waves, a posterior negativity at around 100–300 ms with no clear hemispheric dominance (N1-difference) was observed, while in deviant-minus-control difference waves, a posterior negativity at around 150–300 ms with clear right hemispheric dominance (visual MMN) was observed; there was no clear sign of polarity-reversed N1 in deviant-minus-control difference waves. The latency and scalp distribution of these two components are similar to those of N1-difference and visual MMN observed in recent studies, respectively (e.g., Kimura et al., 2009, 2010b). The significantly different scalp distributions of these two components are also consistent with the recent finding that N1-difference evaluated in control-minus-standard difference waves and visual MMN evaluated in deviant-minus-control difference waves are generated from distinct cortical areas (Kimura et al., 2010a). Further, the clear right hemispheric dominance observed for the latter posterior negativity is a characteristic of visual MMN (e.g., Kimura et al., 2009, 2010b, 2012). These observations suggest that DRN would be decomposed into N1-difference reflecting refractory effects and visual MMN reflecting prediction error effects.1

Neither the mean amplitudes nor the peak latencies of DRN were affected by the task difficulty, which is consistent with previous studies which showed that task difficulty does not affect DRN (Heslenfeld, 2003; Pazo-Alvarez et al., 2004). Task difficulty also did not affect the mean amplitudes or peak latencies of N1-difference. This result implies that task difficulty did not significantly influence the refractoriness state of afferent neurons that engage in N1. Unlike the findings regarding DRN and N1 difference, while task difficulty did not affect the mean amplitudes of visual MMN, it did affect the peak latencies of visual MMN: the peak latencies were significantly delayed in the difficult condition compared to the easy condition. This result implies that, while task difficulty did not significantly influence visual MMN elicitation itself, it strongly influenced the speed (or efficiency) of visual MMN elicitation.

The delay of peak latencies of visual MMN is not attributable to the modulation of visual evoked potentials elicited by control stimuli as a function of task difficulty. The amplitudes of ERPs elicited by control as well as standard stimuli in the latency range of P1 were slightly smaller in the difficult condition than in the easy condition: 110–150 ms for the control stimuli and 130–150 ms for the standard stimuli. This result is consistent with previous studies which demonstrated that the amplitude of P1 is a reliable index of spatial attention allocation (Hillyard et al., 1995; Mangun and Hillyard, 1995; Hillyard and Anllo-Vento, 1998) and the amplitude of P1 elicited by task-irrelevant peripheral stimuli is reduced as the task difficulty is increased from easy to difficult, via decreasing the amount of spatial attention allocated to the task-irrelevant peripheral stimuli (Handy and Mangun, 2000; Handy et al., 2001). Importantly, unlike the amplitudes of ERPs in the P1 latency range, those of ERPs in the subsequent N1 and P2 latency range were not affected by the manipulation of task difficulty for both control and standard stimuli: 150–300 ms, including the latency range of visual MMN as well as DRN and N1-difference in the difference waves. This result ensures that the delayed peak latency of the posterior negativity in the deviant-minus-control difference waves truly represents the modulation of visual MMN elicited by deviant stimuli.

The result that the peak latencies of visual MMN were delayed with an increase in the task difficulty is compatible with the expectation from the perceptual load theory (Lavie and Tsal, 1994; Lavie, 1995, 2005). This theory proposed that, as the perceptual load of task-relevant central information increases, a greater portion of the attention resources is needed for the perceptual processing of this information, and as a result, fewer residual attention resources are available to be involuntarily allocated for the perceptual processing of task-irrelevant peripheral information. The present effect of task difficulty on visual MMN can be interpreted as follows: as the difficulty of the size-change detection task increased from easy to difficult, a greater portion of the attention resources became necessary for detection of the size-change and fewer residual attention resources became involuntarily allocated to task-irrelevant surrounding bar stimuli, which caused the less rapid (less efficient) elicitation of visual MMN.

The present findings may be in line with a recent finding that visual MMN in response to task-irrelevant deviation is sensitive to the congruency between the feature dimension of task-irrelevant deviation and that of task-relevant target (Czigler and Sulykos, 2010). In that study, the authors presented taskirrelevant oddball sequences consisting of deviant and standard bar stimuli with either different colors or different orientations at the peripheral visual fields in separate blocks while the participant performed either a color- or an orientation-change detection task regarding a continuously presented shape at the central visual field in separate blocks. They found reduced amplitudes and delayed peak latencies for visual MMN in response

1One may argue that the posterior negativity extracted in the deviant-minuscontrol difference waves may still include N1-difference, since the smallest physical separation between control stimuli (ca. 16.4◦) was smaller than the physical separation between deviant and standard stimuli (ca. 32.7◦), and thus, the state of refractoriness for control stimuli might be higher than that for deviant stimuli. To rule out this possibility, we calculated ERPs elicited by control stimuli that were preceded by other control stimuli with at least ca. 32.7◦ physical separation. We found that, even with the newly-calculated control ERPs, the same statistically significant pattern of results regarding the visual MMN and N1-difference were obtained. This indicates that the present equiprobable protocol was sufficient for keeping the state of refractoriness for control stimuli equal to (or even lower than) that for deviant stimuli.

to deviant stimuli when the feature dimensions of deviation and target were congruent (e.g., color deviant stimuli under a color-change detection task) compared to when they were incongruent (e.g., color deviant stimuli under an orientation-change detection task). They interpreted the reduction of amplitudes and delay of peak latencies of visual MMN in terms of the competition for feature-specific attentional resources (Desimone and Duncan, 1995): when congruent, the processing of taskrelevant target and task-irrelevant deviation compete for featurespecific attentional resources, which leads to the suppression of visual MMN in response to the task-irrelevant deviation. More interestingly, although they did not consider the effects of task difficulty on visual MMN, as in the present study, their results showed delayed peak latencies of visual MMN for the difficult task (i.e., the color-change detection task) relative to the easy task (i.e., the orientation-change detection task) (however, their study evaluated visual MMN in deviantminus-standard difference waves, and thus it is possible that the delayed peak latency may represent the modulation of N1 difference). Although the experimental and analysis procedures differed between the present study and that reported by Czigler and Sulykos (2010), in a broad context, the findings in these studies consistently shed new light on the attention-sensitive nature of visual MMN (for another example, see Kimura et al., 2010d; but see also Winkler et al., 2005; Berti, 2011, for contrasting examples).

In summary, we found that visual MMN can be affected by the manipulation of task difficulty. This result suggests that visual MMN is not necessarily insensitive to the amount of attentional allocation. In contrast to the previous understanding, the present study supports the notion that visual MMN involves attentiondemanding predictive processes.

#### **THEORETICAL AND PRACTICAL IMPLICATIONS**

The present findings suggest that at least some portion of predictive processes underlying visual MMN elicitation is attentiondemanding. According to the predictive framework of visual MMN (Kimura et al., 2011; Kimura, 2012), the elicitation of visual MMN requires the contribution of multiple predictive processes: (1) the extraction of sequential rules embedded in the temporal structure of successive visual stimulation, (2) the establishment of a predictive model that encodes the extracted sequential rules, (3) the formation of a temporally-aligned prediction about forthcoming visual events on the basis of the predictive model, and (4) the comparison of the current and predicted visual events. Visual MMN is the output of these predictive processes: when incongruence has been detected via the comparison, visual MMN emerges. According to this framework, the delay of visual MMN elicitation observed in the present study implies that the comparison process required more time as the task difficulty increased from easy to difficult. At present, it is difficult to determine whether the delay represents the direct influence of attention on the comparison process (cf. Berti, 2011) or is the result of attentional influence on processes earlier than the comparison process (cf. Kimura et al., 2010b,c), providing no clue as to which part of the predictive process is attention-demanding. Determination of the attention-sensitivity of each process should be an important challenge in future visual MMN research, which could lead to the establishment of an integrative theory of sensory prediction and attention.

Research in this area should be important not only for theoretical development but also for practical progress. To date, visual MMN has been used in clinical studies as an effective tool for investigating preattentive visual processing, and has shed new light on its abnormality in the elderly (e.g., Tales et al., 2002) and several clinical populations (e.g., Tales and Butler, 2006; Tales et al., 2008; Urban et al., 2008; Chang et al., 2011; Qiu et al., 2011). With regard to such clinical applications, the present findings offer two implications. First, although visual MMN can be reliably regarded as a reflection of automatic visual processing (in that the elicitation of visual MMN does not require attention to be actively directed to visual stimulation), it can no longer be regarded as a reflection of preattentive visual processing (in that not all of the predictive processes that underlie visual MMN elicitation can be considered to be attention-independent). Second, possible attentional influences on visual MMN should always be taken into account: significant between-group differences in visual MMN may represent differences in automatic visual processing itself or may represent differences in attentional influences on automatic visual processing. A better understanding of the attention-sensitivity of the aforementioned predictive processes would be helpful for optimizing experimental design, so that such an ambiguous interpretation can be avoided.

Finally, the present findings suggest that visual MMN may be an effective tool in ergonomics (human factors) studies. In this research field, there has been a substantial interest in the utility of ERPs for the assessment of mental workload in the laboratory or real-world tasks (Donchin et al., 1986; Kramer and Weber, 2000). One of the major ERP procedures for assessing the mental workload is the so-called "probe" technique. In this procedure, while the participant performs a certain primary task, stimuli that are unrelated to the primary task (i.e., probe stimuli) are presented concurrently. To date, it has been suggested that P300, sensory evoked potentials, or other ERPs in response to probe stimuli can be used to assess the mental workload in the primary task (e.g., Kramer et al., 1983, 1995; Wickens et al., 1983; Ullsperger et al., 2001). Although the conditions for the application of visual probe stimuli would be fairly limited compared to those for the application of auditory or somatosensory probe stimuli, the utility of visual MMN in ergonomics applications may deserve more attention, given the unique (automatic but still attention-sensitive) nature of visual MMN.

#### **CONCLUSIONS**

The present study demonstrated that visual MMN can be affected by the manipulation of task difficulty, which suggests that visual MMN is sensitive to the amount of attentional allocation. In contrast to the previous understanding, the present finding supports the notion that visual MMN involves attention-demanding predictive processes.

#### **REFERENCES**

and visuocortical processing: event-related potentials reveal sensory-level selection. *Psychol. Sci.* 12, 213–218. doi: 10.1111/1467- 9280.00338

of the data of Kimura et al. (2009). *Neurosci. Lett.* 485, 198–203. doi: 10.1016/j.neulet. 2010.09.011

characterization of mismatch negativity to a visual stimulus. *Clin. Neurophysiol.* 116, 2392–2402. doi: 10.1016/j.clinph.2005.07.006

of sequential regularity violation. *Front. Hum. Neurosci.* 5:46. doi: 10.3389/fnhum.2011.00046

doi: 10.1097/00001756-200205240- 00014

a psychophysiological analysis of the reciprocity of informationprocessing resources. *Science* 221, 1080–1082. doi: 10.1126/science. 6879207

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 29 March 2013; accepted: 24 May 2013; published online: 12 June 2013.*

*Citation: Kimura M and Takeda Y (2013) Task difficulty affects the predictive process indexed by visual mismatch negativity. Front. Hum. Neurosci. 7:267. doi: 10.3389/fnhum.2013.00267*

*Copyright © 2013 Kimura and Takeda. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in other forums, provided the original authors and source are credited and subject to any copyright notices concerning any third-party graphics etc.*

# **HUMAN NEUROSCIENCE**

# The visual mismatch negativity elicited with visual speech stimuli

# *Benjamin T. Files 1, Edward T. Auer Jr. 2 and Lynne E. Bernstein1,2\**

*1 Neuroscience Graduate Program, University of Southern California, Los Angeles, CA, USA*

*2 Communication Neuroscience Laboratory, Department of Speech and Hearing Science, George Washington University, Washington, DC, USA*

#### *Edited by:*

*István Czigler, Hungarian Academy of Sciences, Hungary*

#### *Reviewed by:*

*Erich Schröger, University of Leipzig, Germany Paula P. Alvarez, University of Santiago de Compostela, Spain*

#### *\*Correspondence:*

*Lynne E. Bernstein, Communication Neuroscience Laboratory, Department of Speech and Hearing Science, George Washington University, 550 Rome Hall, Washington, DC 20052, USA e-mail<EMAIL_ADDRESS>
The visual mismatch negativity (vMMN), deriving from the brain's response to stimulus deviance, is thought to be generated by the cortex that represents the stimulus. The vMMN response to visual speech stimuli was used in a study of the lateralization of visual speech processing. Previous research suggested that the right posterior temporal cortex has specialization for processing simple non-speech face gestures, and the left posterior temporal cortex has specialization for processing visual speech gestures. Here, visual speech consonant-vowel (CV) stimuli with controlled perceptual dissimilarities were presented in an electroencephalography (EEG) vMMN paradigm. The vMMNs were obtained using the comparison of event-related potentials (ERPs) for separate CVs in their roles as deviant vs. their roles as standard. Four separate vMMN contrasts were tested, two with the perceptually *far* deviants (i.e., "zha" or "fa") and two with the *near* deviants (i.e., "zha" or "ta"). Only *far* deviants evoked the vMMN response over the left posterior temporal cortex. All four deviants evoked vMMNs over the right posterior temporal cortex. The results are interpreted as evidence that the left posterior temporal cortex represents speech contrasts that are perceived as different consonants, and the right posterior temporal cortex represents face gestures that may not be perceived as different CVs.

**Keywords: speech perception, visual perception, lipreading, scalp electrophysiology, mismatch negativity (MMN), hemispheric laterazation for speech**

# **INTRODUCTION**

The visual mismatch negativity (vMMN) paradigm was used here to investigate visual speech processing. The MMN response was originally discovered and then extensively investigated with auditory stimuli (Näätänen et al., 1978, 2011). The classical auditory MMN is generated by the brain's automatic response to a change in repeated stimulation that exceeds a threshold corresponding approximately to the behavioral discrimination threshold. It is elicited by violations of regularities in a sequence of stimuli, whether the stimuli are attended or not, and the response typically peaks 100–200 ms after onset of the deviance (Näätänen et al., 1978, 2005, 2007). The violations that generate the auditory MMN can range from low-level stimulus deviations such as the duration of sound clicks (Ponton et al., 1997) to high-level deviations such as speech phoneme category (Dahaene-Lambertz, 1997). More recently, the vMMN was confirmed (Pazo-Alvarez et al., 2003; Czigler, 2007; Kimura et al., 2011; Winkler and Czigler, 2012). It too is elicited by a change in regularities in a sequence of stimuli, across different levels of representation, including deviations caused by spatiotemporal visual features (Pazo-Alvarez et al., 2004), conjunctions of visual features (Winkler et al., 2005), emotional faces (Li et al., 2012; Stefanics et al., 2012), and abstract visual stimulus properties such as bilateral symmetry (Kecskes-Kovacs et al., 2013) and sequential visual stimulus probability (Stefanics et al., 2011).

Speech can be perceived visually by lipreading, and visual speech perception is carried out automatically by hearing as well as by hearing-impaired individuals (Bernstein et al., 2000; Auer and Bernstein, 2007). Inasmuch as perceivers can visually recognize the phonemes (consonants and vowels) of speech through lipreading, the stimuli are expected to undergo hierarchical visual processing from simple features to complex representations along the visual pathway (Grill-Spector et al., 2001; Jiang et al., 2007b), just as are other visual objects, including faces (Grill-Spector et al., 2001), facial expression (Li et al., 2012; Stefanics et al., 2012), and non-speech face gestures (Puce et al., 1998, 2000, 2007; Bernstein et al., 2011). Crucially, because the vMMN deviation detection response is thought to be generated by the cortex that represents the standard and deviant stimuli (Winkler and Czigler, 2012), it should be possible to obtain the vMMN in response to deviations in visual speech stimuli. However, previous studies in which a speech vMMN was sought produced mixed success in obtaining a deviance response attributable to visual speech stimulus deviance detection (Colin et al., 2002, 2004; Saint-Amour et al., 2007; Ponton et al., 2009; Winkler and Czigler, 2012). A few studies have even sought an auditory MMN in response to visual speech stimuli (e.g., Sams et al., 1991; Möttönen et al., 2002).

The present study took into account how visual stimuli conveying speech information might be represented and mapped to higher levels of cortical processing, say for speech category perception or for other functions such as emotion, social, or gaze perception. That is, the study was specifically focused on the perception of the physical visual speech stimulus. The distinction between representations of the forms of exogenous stimuli vs. representation of linguistic categories is captured in linguistics by the terms *phonetic form* vs. *phonemic category*. Phonetic forms are the exogenous physical stimuli that convey the linguisticallyrelevant information used to perceive the speech category to which the stimulus belongs. Visual speech stimuli convey linguistic phonetic information primarily via the visible gestures of the lips, jaw, cheeks, and tongue, which support the system of phonological contrasts that underly speech phonemes (Yehia et al., 1998; Jiang et al., 2002; Bernstein, 2012). Phonemic categories are the consonant and vowel categories that a language uses to differentiate and represent words. If visual speech is processed similarly to auditory speech stimuli, functions related to higher-level language processing, such as categorization and semantic associations, are carried out beyond the level of exogenous stimulus form representations (Scott and Johnsrude, 2003; Hickok and Poeppel, 2007).

This study was concerned with the implications for cortical representation of visual speech stimuli in the case that speech perception is generally left-lateralized. There is evidence for formbased speech representations in high-level visual areas, and there is evidence that they are left-lateralized (Campbell et al., 2001; Bernstein et al., 2011; Campbell, 2011; Nath and Beauchamp, 2012). For example, Campbell et al. (1986) showed that a patient with right-hemisphere posterior cortical damage failed to recognize faces but had preserved speech lip-shape recognition, and that a patient with left-hemisphere posterior cortical damage failed to recognize speech lip-shapes but had preserved face recognition.

Recently, evidence for hemispheric lateralization was obtained in a study designed to investigate specifically the site/s of specialized visual speech processing. Bernstein et al. (2011), applied a functional magnetic resonance imaging (fMRI) block design while participants viewed video and point-light speech and nonspeech stimuli and tiled control stimuli. Participants were imaged during localizer scans for three regions of interest (ROIs), the fusiform face area (FFA) (Kanwisher et al., 1997), the lateral occipital complex (LOC) (Grill-Spector et al., 2001), and the human visual motion area V5/MT. These three areas were all under-activated by speech stimuli. Although both posterior temporal cortices responded to speech and non-speech stimuli, only in the left hemisphere was an area found with differential sensitivity to speech vs. non-speech face gestures. It was named the *temporal visual speech area* (TVSA) and was localized to the posterior superior temporal sulcus and adjacent posterior middle temporal gyrus (pSTS/pMTG), anterior to cortex that was activated by non-speech face movement in video and point-light stimuli. TVSA is similarly active across video and point-light stimuli. In contrast, right-hemisphere activity in the pSTS was not reliably different for speech vs. non-speech face gestures. Research aimed at non-speech face gesture processing has also produced evidence of right-hemisphere dominance for non-speech face gestures, with a focus in the pSTS (Puce et al., 2000, 2003).

The approach in the current study was based on predictions for how the representation of visual speech stimuli should differ for the right vs. left posterior temporal cortex under the hypothesis that the left cortex has tuning for speech, but the right cortex has tuning for non-speech face gestures. Specifically, lipreading relies on highly discriminable visual speech differences. Visual speech phonemes are not necessarily as distinctive as auditory speech phonemes. Visual speech consonants are known to vary in terms of how distinct they are from each other, because some of the distinctive speech features used by listeners (e.g., voicing, manner, nasality, place) to distinguish phonemes are not visible or are less visible to lipreaders (Auer and Bernstein, 1997; Bernstein, 2012). A left posterior temporal cortex area specialized for speech processing, part of an extensive speech processing pathway, is expected to be tuned to represent linguistically useful exogenous phonetic forms, that is, forms that can be mapped to higher-level linguistic categories, such as phonemes. However, when spoken syllables (e.g., "zha" and "ta") do not provide enough visual phonetic feature information, their representations are expected to generalize. That is, the indistinct stimuli activate overlapping neural populations. This is depicted in **Figure 1**, for which the visually *near* (perceptual categories are not distinct) syllables "ta" and "zha" are represented by almost completely overlapping ovals in the box labeled *left posterior temporal visual cortex*. The perceptually far stimulus "fa," a stimulus that shares few visible phonetic features with "zha," is depicted within its own non-overlapping oval in that box. Here, using the vMMN paradigm, a deviance response was predicted for the left hemisphere with the stimuli "zha" vs. "fa," representing a *far* contrast. But the *near* contrast "zha"-"ta," depicted in **Figure 1**, was not predicted to elicit the vMMN response by the left posterior temporal cortex for "zha" or for "ta" syllables.

In contrast, the right posterior temporal cortex, with its possible dominance for processing simple non-speech face motions such as eye open vs. closed, and simple lips open vs. closed (Puce et al., 2000, 2003), was predicted to generate a deviance response to both perceptually *near* and *far* speech stimulus contrasts. The depiction in **Figure 1** for the right posterior temporal cortex shows that the stimulus differences are represented there more faithfully (i.e., there are more neural units that are not in common). The right posterior temporal cortex is theoretically more concerned with perception of non-speech face gestures, for example, gestures related to visible emotion or affect: The representations may even be more analog in the sense that they are not used as input to a generative system that relies on combinations of representations (i.e., vowels and consonants) to produce a very large vocabulary of distinct words.

Even very simple low-level visual features or non-speech face or eye motion in the speech video clips can elicit the vMMN (Puce et al., 2000, 2003; Miki et al., 2004; Thompson et al., 2007). With natural speech production, phonetic forms vary from one production to the next. An additional contribution to variability is the virtually inevitable shifts in the talker's head position, eye gaze, eyebrows, etc., from video recording to recording. Subtle differences are not necessarily so obvious on a single viewing, but the vMMN paradigm involves multiple stimulus repetitions, which can render subtle differences highly salient.

The approach here was to use two recordings for each consonant and to manipulate the stimuli to minimize non-phonetic visual cues that might differentiate the stimuli. The study design took into account the likelihood that the deviance response to speech stimuli would be confounded with low-level stimulus

differences, if it involved a stimulus as standard (e.g., "zha") vs. a different stimulus as deviant (e.g., "fa"). Therefore, the vMMN was sought using the event-related potentials (ERPs) obtained with the same stimulus (e.g., "zha") in its two possible roles of standard and deviant. Stimulus discriminability was verified prior to ERP recording. During ERP recording, participants monitored for a rare target phoneme to engage their attention and hold it at the level of phoneme categorization, rather than at the level of stimulus discrimination.

temporal cortex when the syllables in the pair are perceptually similar

# **METHOD**

# **PARTICIPANTS**

Participants were screened for right-handedness (Oldfield, 1971), normal or corrected to normal vision (20/30 or better in both eyes using a traditional Snellen chart), normal hearing, American English as a first and native language, and no known neurological deficits. Lipreading was assessed with a screening test that has been used to test a very large sample of normal hearing individuals (Auer and Bernstein, 2007). The screening cutoff was 15% words correct in isolated sentences to assure that participants who entered the EEG experiment had some lipreading ability. Forty-nine individuals were screened (mean age = 23 years), and 24 (mean age = 24, range 21–31, 18 female, lipreading score *M* = 28.7% words correct) met the inclusion criteria for entering the EEG experiment. The EEG data from 11 participants (mean age = 23.2, range 19–31, 7 female, lipreading score *M* = 33.0) were used here: One participant was lost to contact, one ended the experiment early, two had unacceptably high initial impedance levels and were not recorded, and nine had high electrode impedances, excessive bridging between electrodes, or unacceptable noise levels. Informed consent was obtained from all participants. Participants were paid. The research was approved by the Institutional Review Boards at George Washington University and at the University of Southern California.

# **STIMULI**

# *Stimulus dissimilarity*

activate non-overlapping populations.

The stimuli for this study were selected to be of predicted perceptual and physical dissimilarities. Estimates of the dissimilarities and the video speech stimuli themselves were obtained from Jiang et al. (2007a), which gives a detailed description of the methods for predicting and testing dissimilarity. Based on the dissimilarity measures in Jiang et al. (2007a), the stimulus pair "zha"—"fa," with modeled dissimilarity of 4.04, was chosen to be perceptually *far*, and the stimulus pair "zha"—"ta," with modeled dissimilarity of 2.28 was chosen to be perceptually *near*. In a subsequent study, Files and Bernstein (submitted) tested whether the modeled dissimilarities among a relatively large selection of syllables correctly predicted stimulus discriminability, and they did.

# *Stimulus video*

Stimuli were recorded so that the talker's face filled the video screen, and lighting was from both sides and slightly below his face. A production quality camera (Sony DXC-D30 digital) and video recorder (Sony UVW 1800) were used simultaneously with an infrared motion capture system (Qualisys MCU120/240 Hz CCD Imager) for recording 3-dimensional (3D) motion of 20 retro-reflectors affixed to the talker's face. The 3D motion recording was used by Jiang et al. (2007a) in developing the dissimilarity estimates. There were two video recordings of each of the syllables, "zha," "ta," and "fa" that were used for eliciting the vMMNs. Two tokens of "ha," and of "va" were used as targets to control attention during the vMMN paradigm. All video was converted to grayscale.

In order to reduce differences in the durations of preparatory mouth motion across stimulus tokens and increase the rate of data collection, some video frames were removed from slow uninformative mouth opening gestures. But most of the duration differences were reduced by removing frames from the final mouth closure. No frames were removed between the sharp initiation of articulatory motion and the quasi-steady-state portion of the vowel.

During the EEG experiment, the video clips were displayed contiguously through time. To avoid responses due to minor variations in the position of the head from the end of one token to the beginning of the next, morphs of 267 ms were generated (Abrosoft's FantaMorph5) to create smooth transitions from one token to the next. The morphing period corresponded to the inter-stimulus-interval.

The first frame of each token was centered on the video monitor so that a motion-capture dot that was affixed at the center of the upper lip was at the same position for each stimulus. Also, stimuli were processed so that they would not be identifiable based solely on the talker's head movement. This was done by adding a small amount of smooth translational motion and rotation to each stimulus on a frame-by-frame basis. The average motion speed was 0.5 pixels per frame (0.87◦ of visual angle/s), with a maximum of 1.42 pixels per frame (2.5◦/s). Rotation varied between plus and minus 1.2◦ of tilt, with an average change of 0.055◦ of tilt per frame (3.28◦/s) and a maximum change of 0.15◦ of tilt per frame (9.4◦ of tilt/s). A stationary circular mask with radius 5.5◦ of visual angle and luminance equal to the background masked off the area around the face of the talker.

# *Stimulus alignment and deviation points*

The two tokens of each consonant (e.g., "zha") varied somewhat in their kinematics, so temporal alignments had to be defined prior to averaging the EEG data. We developed a method to align tokens of each syllable. Video clips were compared frame by frame separately for "zha," "fa," and "ta." In addition, mouth opening area was measured as the number of pixels encompassed within a manual tracing of the vermillion border in each frame of each stimulus.  **Figure 2** shows the mouth-opening area and video of the lips for the three different consonant-vowel (CV) stimuli and the two different tokens of each of them. The stimuli began with a closed neutral mouth and face, followed by the gesture into the consonant, followed by the gesture into the /a/ vowel ("ta," "fa," "zha"). Consonant identity information develops across time and continues to be present as the consonant transitions into the following vowel. The steep mouth opening gesture into the vowel partway through the stimulus was considered a possible landmark for temporal alignment, because it is a prominent landmark in the mouth area trace, but using this landmark in some cases brought the initial part of the consonant into gross misalignment. The frames comprising the initial gesture into the consonant were chosen to be the relevant landmark for alignment across tokens, because they are the earliest indication of the consonant identity (Jesse and Massaro, 2010).

The question was then, when did the image of one consonant (e.g., "fa") deviate from the image of the other (e.g., "zha"). The MMN is typically elicited by stimulus deviation, rather than stimulus onset (Leitman et al., 2009), and this deviation onset point is used to characterize the relative timing of the vMMN. Typically, ERPs to visual stimuli require steep visual energy change (Besle et al., 2004), but visual speech stimulus onset can be relatively slow-moving, depending on the speech phonetic features. Careful examination of the videos shows that differences in the tongue are

visible across the different consonants. The "zha" is articulated by holding the tongue in a quasi-steady-state somewhat flattened position in the mouth. This articulation is expected to take longer to register as a deviation, because of its subtle initial movement. The "ta" and "zha" stimuli vary primarily in terms of tongue position, which is visible but difficult to discern without attention to the tongue inside the mouth aperture. The deviation onset point here was defined as the first frame at which there was a visible difference across consonants. The 0-ms points in this report are set at the relevant deviation point and vMMN times are reported relative to the deviation onset.

#### **PROCEDURES**

#### *Discrimination pre-test*

To confirm the discriminability of the consonants comprising the critical contrasts in the EEG experiment, participants carried out a *same-different* perceptual discrimination task that used "zha"— "fa", and "zha"—"ta" *different* stimulus pairs. The two tokens of each syllable were combined in each of four possible ways and in both possible orders. *Same* pairs used different tokens of the same syllable, so that accurate discrimination required attention to consonant category. This resulted in six unique *same* pairs and 16 unique *different* pairs. To reduce the difference in number of *same* pairs vs. the number of *different* pairs, the *same* pairs were repeated, resulting in 12 *same* pairs and 16 *different* pairs per block, for a total of 28 pairs per block. During each trial, the inter-stimulus interval was filled by a morph transition from the end of the first token to the start of the second lasting 267 ms. Instructions emphasized that the tokens might differ in various ways, but that the task was to determine if the initial consonants were the same or different. Eleven blocks of pseudo-randomly ordered trials were presented. The first block was used for practice to ensure the participants' familiarity with the task, and it was not analyzed.

#### *vMMN procedure*

EEG recordings were obtained during an oddball paradigm in which standard, deviant, and target stimuli were presented. If one stimulus category is used as the standard and a different category stimulus is used as the deviant in deriving the vMMN, the vMMN also contains a response to the physical stimuli (Czigler et al., 2002). In order to compare ERPs to standards vs. deviants, holding the stimulus constant, each stimulus was tested in the roles of deviant and standard across different recording blocks (**Table 1**)1 .

EEG recording comprised 40 stimulus blocks divided across four block types (**Table 1**). Each block type had one *standard* consonant (i.e., "zha," "fa," or "ta"), one *deviant* consonant (i.e., "zha," "fa," or "ta"), and one *target* consonant (i.e., "ha," or "va"). The "zha" served as *deviant* or *standard* with either "fa" or "ta." Thus, four vMMNs were sought: (1) "zha" in the context of "ta" (*near*); (2) "ta" in the context of "zha" (*near*); (3) "zha" in the

*Each block had a standard syllable, a deviant syllable and a target syllable. aDissimilarity measures the difference between the standard and the deviant syllable.*

context of "fa" (*far*); and (4) "fa" in the context of "zha" (*far*). Each vMMN was based on 10 stimulus blocks with the vMMN stimulus in either deviant or standard role. During each block, a *deviant* was always preceded by five to nine *standards*. At the beginning of a block, the *standard* was presented 9 times before the first *deviant*. The inter-stimulus-interval was measured as the duration of the morphs between the end of a stimulus and the beginning of the next, which was 267 ms.

To ensure that the visual stimuli were attended, participants were instructed to monitor the stimuli carefully for a *target* syllable. At the start of each block, the target syllable was identified by presenting it six times in succession. A *target* was always preceded by three to five *standards*. Participants were instructed to press a button upon detecting the target, which they were told would happen rarely. In each block, the *target* was presented four times, and the *deviant* was presented 20 times. In all, 85.4% of stimuli in a block were standards, 12.1% were deviants and 2.4% were targets. This corresponded to 200 *deviant* trials and ∼1400 *standard* trials per contrast per subject. The first *standard* trial following either a *deviant* trial or a *target* trial was discarded from analysis, because a standard following something other than a standard might generate a MMN (Sams et al., 1984; Nousak et al., 1996). This resulted in 1160 *standard* trials for computing the vMMN.

Participants were instructed to take self-paced breaks between blocks, and longer breaks were enforced every 10 blocks. Recording time was ∼4.5 h per participant. After EEG recording, electrode locations recorded were for each subject using a 3-dimensional digitizer (Polhemus, Colchester, Vermont).

#### **EEG RECORDING AND OFFLINE DATA PROCESSING**

EEG data were recorded using a 62-electrode cap that was configured with a modified 10–20 system for electrode placement. Two additional electrodes were affixed at mastoid locations, and bipolar EOG electrodes were affixed above and below the left eye and at the external canthi of the eyes to monitor eye movements. The EEG was amplified using a high input impedance amplifier (SynAmps 2, Neuroscan, NC). It was digitized at 1000 Hz with a 200 Hz low-pass filter. Electrode impedances were measured, and the inclusion criterion was 35 kOhm.

Offline, data were band-pass filtered from 0.5 to 50 Hz with a 12-dB/octave rolloff FIR zero phase-shift filter using EDIT 4.5 software (Neuroscan, NC). Eyeblink artifacts were removed using EDIT's blink noise reduction algorithm (Semlitsch et al., 1986). Data were epoched from 100 ms before video onset to 1000 ms after video onset. Epochs were baseline-corrected by subtracting

1This approach does not account for different refractoriness or adaptation due to different probabilities of stimulus presentation (Schroger and Wolff, 1996; Czigler et al., 2002; Kimura et al., 2009). However, an additional set of control recordings would have been needed to take this into account, and here the focus was not on isolating a unique MMN component. Also, the design of the experiment would have been excessively long (see General Discussion).

the average of the voltage measurements from −100 to +100 ms for each electrode and then average-referenced.

Artifact rejection and interpolation were performed using custom scripts calling functions in EEGLAB (Delorme and Makeig, 2004). Epochs in which no electrode voltage exceeded 50µV at any point in the epoch were included. For those epochs in which only one electrode exceeded the 50µV criterion, the data for that electrode were interpolated using spherical spline interpolation (Picton et al., 2000). This procedure resulted in inclusion of 91% of the EEG sweeps. To correct for variation in electrode placement between subjects, individual subject data were projected onto a group average set of electrode positions using spherical spline interpolation (Picton et al., 2000).

### **ANALYSES OF DISCRIMINATION DATA**

*Same-different* discrimination sensitivity was measured with *d* (Green and Swets, 1966). The hit rate was the proportion *different* responses to trials with different syllables. The false alarm rate was the proportion *different* responses for same pairs. If the rate was zero it was replaced with 1/(2*N*), and if it was one it was replaced by 1–1/(2*N*), where *N* is the number of trials (Macmillan and Creelman, 1991). Because this is a *same-different* design, *z*(*hit rate*)—*z*(*false alarm rate*) was multiplied by √**2** (Macmillan and Creelman, 1991).

Target detection during the EEG task was also evaluated using *d* , but the measure was *z*(*hit rate*)—*z*(*false alarm rate*). A response within 4 s of the target presentation was considered a *hit*, and a *false alarm* was any response outside this window. All non-target syllables were considered distracters for the purpose of calculating a false alarm rate. To assess differences in target detection across blocks, *d* was submitted to repeated-measures ANOVA.

# **ANALYSES OF EEG DATA**

# *Overview*

*A priori*, the main hypothesis was that visual speech stimuli are processed by the visual system to the level of representing the exogenous visual syllables. Previous research had suggested that there was specialization for visual speech stimuli by left posterior temporal cortex (Campbell et al., 2001; Bernstein et al., 2011; Campbell, 2011; Nath and Beauchamp, 2012). Previous research also suggested that there was specialization for non-speech face motion by right posterior temporal cortex (Puce et al., 1998, 2000, 2007; Bernstein et al., 2011). Therefore, the *a priori* anatomical regions of interest (ROI) were the bilateral posterior temporal cortices. However, rather than merely selecting electrodes of interest (EOI) over scalp locations approximately over those cortices and carrying out all analyses with those EOIs, a more conservative, step-by-step approach was taken, which allowed for the possibility that deviation detection was carried out elsewhere in cortex (e.g., Sams et al., 1991; Möttönen et al., 2002).

In order first to test for reliable stimulus deviation effects, independent of temporal window or spatial location, global field power (GFP; Lehmann and Skrandies, 1980; Skrandies, 1990) measures were compared statistically across standard vs. deviant for each of the four different vMMN contrasts. The GFP analyses show the presence and temporal interval of a deviation response anywhere over the scalp. The first 500 ms post-stimulus deviation was examined, because that interval was expected to encompass any possible vMMN.

Next, source analyses were carried out to probe whether there was evidence for stimulus processing by posterior temporal cortices, consistent with previous fMRI results on visual speech perception (Bernstein et al., 2011). Distributed dipole sources (Tadel et al., 2011) were computed for the responses to standard stimuli and for the vMMN waveforms. These were inspected and compared with the previous Bernstein-et-al. results and also with results from previous EEG studies that presented source analyses (Bernstein et al., 2008; Ponton et al., 2009). The inspection focused on the first 500 ms of the source models.

After examining the source models, EOIs were sought for statistical testing of vMMNs, taking into account the ERPs at individual electrode locations. For this level of analysis, an approach was needed to guard against double-dipping, that is, use of the same results to select and test data for hypothesis testing (Kriegeskorte et al., 2009). Because we did not have an independent localizer (i.e., an entirely different data set with which to select EOIs), as is recommended for fMRI experiments, we ran analyses on several different electrode clusters over posterior temporal cortices. Because all those results were highly similar, only one set of EOI analyses are presented here.

A coincident frontal positivity has also been reported for Fz and/or Cz in conjunction with evidence for a vMMN (Czigler et al., 2002, 2004). The statistical tests for the vMMN were carried out separately on ERPs from electrodes Fz and Cz to assess the presence of a frontal MMN. These tests also served as a check on the validity of the EOI selection. Fz and Cz electrodes are commonly used for testing the auditory MMN (Näätänen et al., 2007). If the same results were obtained on Fz and Cz as with the EOIs, the implication would be that EOI selection was biased toward our hypothesis that the posterior temporal cortices are responsible for visual speech form representations. The results for Fz and Cz were similar to each other but different from the EOI results, and only the Fz results are presented here. None of the Cz results were statistically reliable. ERPs evoked by target stimuli were not analyzed, because so few target stimuli were presented.

# *Global field power*

GFP (Lehmann and Skrandies, 1980; Skrandies, 1990) is the root mean squared average-referenced potential over all electrodes at a time sample. The GFP was calculated for each standard and deviant ERP per stimulus and per subject. The analysis window was 0–500 ms post stimulus deviation. Statistical analysis of group mean GFP differences between standard and deviant, within syllable, used randomization testing (Blair and Karniski, 1993; Nichols and Holmes, 2002; Edgington and Onghena, 2007) of the null hypothesis of no difference between the evoked response when the stimulus was a *standard* vs. the evoked response when the stimulus was a *deviant*. The level of re-sampling was the individual trial.

Surrogate mean GFP measures were generated for each subject by permuting the single-trial labels (i.e., *standard* or *deviant*) 1999 times and then computing mean GFP differences (*deviant* minus *standard*) for these permutation samples. These single-subject permutation mean GFP differences were averaged across subjects to obtain a permutation distribution of group mean GFP differences within the ERPs for a particular syllable. To avoid bias due to using a randomly generated subset of the full permutation distribution, the obtained group mean GFP difference was included in the permutation distribution, resulting in a total of 2000 entries in the permutation distribution. The *p*-value for a given time point was calculated as the proportion of surrogate group mean GFP difference values in the permutation distribution that were as or more extreme than the obtained group mean GFP difference, resulting in a two-tailed test.

To correct for multiple comparisons over time, a threshold length of consecutive *p*-values <0.05 was established (Blair and Karniski, 1993; Groppe et al., 2011). The threshold number of consecutive *p*-values was determined from the permutation distribution generated in the corresponding uncorrected test. For each entry in the permutation distribution, a surrogate *p*-value series was computed as though that entry were the actual data. Then, the largest number of consecutive *p*-values <0.05 in that surrogate *p*-value series was computed for each permutation entry. The threshold number of consecutive *p*-values was the 95th percentile of this null distribution of run lengths. This correction, which offers weak control over family-wise error rate and is appropriate when effects persist over many consecutive samples (Groppe et al., 2011), is similar to one used with parametric statistics (Guthrie and Buchwald, 1991) but requires no assumptions or knowledge about the autocorrelation structure of the underlying signal or noise.

#### *EEG distributed dipole source models*

EEG sources were modeled with distributed dipole source imaging using Brainstorm software (Tadel et al., 2011). In lieu of having individual anatomical MRI data for source space and forward modeling, the MNI/Colin 27 brain was used. A boundary element model (Gramfort et al., 2010) was fit to the anatomical model using a scalp model with 1082 vertices, a skull model with 642 vertices, and a brain model with 642 vertices. The cortical surface was used as the source space, and source orientations were constrained to be normal to the cortical surface. Cortical activity was estimated using depth-weighted minimum-norm estimation (wMNE; Baillet et al., 2001).

EEG source localization is generally less precise than some other neuroimaging techniques (Michel et al., 2004). Simulations comparing source localization techniques resulted in a mean localization error of 19.6 mm when using a generic brain model (Darvas et al., 2006), as was done here. Similar methods were used here, so the estimate of localization errors is ∼20 mm. Therefore, the source solutions found here serve as useful visualization tools and for EOI selection but are not intended for making conclusion related to precise anatomical localization.

#### *vMMN analyses*

The vMMN analyses used the same general approach as the approach to the GFP analyses rather than the more pervasive analysis of difference waveforms. To assess the reliability of the vMMNs for each stimulus, the average of the ERP for the EOIs for the token-as-standard was compared with the average of the ERPs for the token-as-deviant using a standard paired-samples permutation test (Edgington and Onghena, 2007) with the subject mean ERP as the unit of re-sampling. A threshold number of consecutive *p*-values <0.05 was established to correct for multiple comparisons using the same criterion (Blair and Karniski, 1993) as described above for the GFP analyses. The EOI cluster results that are presented are from the clusters left P5, P3, P1, PO7, PO5, and PO3, and right P2, P4, P6, PO4, PO6, and PO82 . We also carried out comparisons of the difference waveforms across *near* vs. *far* contrasts. These were a general check on the hypothesis that *far* contrasts were different from *near* contrasts.

In some cases in which a vMMN is observed, a coincident frontal positivity has also been reported for Fz and/or Cz (Czigler et al., 2002, 2004). The statistical tests for the vMMN were carried out separately on ERPs from electrodes Fz and Cz to assess the presence of a frontal MMN.

# **RESULTS**

### **BEHAVIORAL RESULTS**

The purpose of testing behavioral discrimination was to assure that the stimulus pair discriminability was predicted correctly. The 49 screened participants were tested, and the EEG data from 11 of them are reported here. Discrimination *d* scores were compared across groups (included vs. excluded participants) using analysis of variance (ANOVA) with the within-subjects factor of stimulus distance (*near* vs. *far*) and between-subjects factor of group (included vs. excluded). The groups were not reliably different, and group did not interact with stimulus distance.

*Far* pairs were discriminated better than *near* pairs, *F*(1, 47) = 591.7, *p* < 0.001, mean difference in *d* = 3.13. Within the EEG group, mean *d* for the *far* stimulus pairs was reliably higher than for the *near* stimulus pairs, paired-*t*(10) = 12.25, *p* < 0.001, mean difference in *d* = 3.02. Mean *d* was reliably above chance for both *near*, *t*(10) = 8.09, *p* < 0.001, *M* = 1.40, and *far*, *t*(10) = 15.62, *p* < 0.001, *M* = 4.51, stimulus pairs.

Detection *d* of "ha" or "va" during EEG recording was high, group mean *d* = 4.83, range [3.83, 5.91]. The two targets were detected at similar levels, *paired*-*t*(10) = 0.23, *p* = 0.82. For neither target syllable was there any effect of which syllable was the standard in the EEG recording block.

#### *ERPs across vMMN stimulus pairs*

The ERP group mean data sets for the four stimulus pairs were inspected for data quality. **Figures S1–S2** show the montages for each of the vMMN data sets.

# *GFP results*

GFP measures were computed for each standard and deviant syllable. Holding syllable constant, the standard vs. deviant GFP was compared to determine whether and, if so, when a reliable effect of stimulus deviance was present in each of the four stimulus conditions (i.e., "zha" in the *near* context, "zha" in the *far* context,

2The alternate EOI clusters that were analyzed were: left (TP7 CP5 P7 P5), right (CP6 TP8 P6 P8); left (CP5 CP3 CP1 P7 P5 P3 P1 PO7 PO5 PO3 CB1) right (CP2 CP4 CP6 P2 P4 P6 P8 PO4 PO6 PO8 CB2); and left (TP7 CP5 CP3 CP1 P7 P5 P3 P1 PO7 PO5 PO3 CB1), right (CP2 CP4 CP6 TP8 P2 P4 P6 P8 PO4 PO6 PO8 CB2).

"fa" a *far* contrast, and "ta" a *near* contrast). All of the stimulus contrasts resulted in reliable effects. **Figure 3** summarizes the GFP results for each vMMN. The reliable GFP difference for "zha" in the *far* context was 200–500 ms post-deviation onset. For "zha" in the *near* context, there were two intervals of reliable difference, 268–329 and 338–500 ms post-deviation onset. The reliable difference for "fa" was 52–500 ms post-deviation onset. The reliable difference for "ta" was 452–500 ms post-deviation onset.

#### *Distributed dipole source models*

Dipole source models were computed using ERPs obtained with standard stimuli ("zha," "fa," and "ta") in order to visualize the spatiotemporal patterns of exogenously driven responses to the stimuli. **Figures 4**–**6** show the dipole source strength at 20-ms intervals starting from 90 ms after onset of visible motion until 670 ms for the group mean ERPs. The images are thresholded to only show dipole sources stronger than 20 pA·m. The figures show images starting at 90 ms post-stimulus onset, because no suprathreshold sources were obtained earlier. The images continue through 690 ms to indicate that posterior activity rises and falls within the interval, as would be expected in response to a temporally unfolding stimulus.

The right hemisphere overall appeared to have stronger and more sustained responses focused on posterior temporal cortex. Additionally, the right posterior temporal activation was more widespread but with a more inferior focus compared to that in left posterior temporal cortex. Variations in the anatomical locations of the foci of activity across **Figures 4**–**6** suggest that the possibility that activation sites varied as a function of syllable. But these cannot be interpreted with confidence given the relatively low level of spatial resolution of these distributed dipole source models.

The temporal differences across syllable are more interpretable. Variation across syllables is attributed to differences in stimulus kinematics. The "fa" standard (**Figure 4**) resulted in sustained right hemisphere posterior temporal activity from ∼120 to 490 ms relative to stimulus onset and sustained left hemisphere posterior temporal activity from ∼170 to 270 ms. The "zha" standard (**Figure 5**) resulted in sustained right hemisphere posterior temporal activity from ∼190 to 430 ms and sustained left hemisphere posterior temporal activity from ∼190 to 390 ms. The "ta" standard (**Figure 6**) resulted in sustained right hemisphere posterior temporal activity from ∼150 to 250 ms and sustained left hemisphere posterior temporal activity from ∼150 to 230 ms. The shorter period of sustained activity for "ta" vs. "fa" and "zha" can be explained by its shorter (fewer frames) initial articulatory gesture (**Figure 2**).

Some fronto-central and central activity emerged starting 220 to 280 ms post-stimulus onset, particularly with "zha" and "fa." No other prominent activations were obtained elsewhere during the initial periods of sustained posterior temporal activity.

Dipole source models were also computed on the vMMN difference waveforms (**Figures S3**–**S6**), resulting in lower signal strength in posterior temporal cortices in comparison with models based on the standard ERPs. The models support the presence of deviance responses in those cortical areas and higher right posterior activity for *far* contrasts than *near* contrasts. All of the difference waveform models demonstrate patterns of asymmetric frontal activity with greatest strength generally beyond 200 ms post-deviation that seems attributable to attention to the deviant.

**FIGURE 4 | Source images for "fa" standard.** Images show the depth-weighted minimum norm estimate of dipole source strength constrained to the surface of the cortex using a boundary element forward model and a generic anatomical model at 20-ms intervals starting from 90 ms after onset of visible motion for the group mean ERPs for syllable "fa" as *standard*. The time indicated by the cyan bar indicates the time at which "fa" visibly differs from "zha." Images are thresholded at 20 pA·m. Initial activity is in the occipital cortex. At 150 ms after syllable onset, the bilateral posterior temporal activity begins that lasts until 290 ms in the left hemisphere and until 490 ms in the right hemisphere. Activation in the right posterior temporal cortex is more widespread and inferior to that on the left. Fronto-central activity is visible from 250 to 510 ms post-stimulus onset.

#### *vMMN results*

ERPs of EOI clusters for each syllable contrast and hemisphere were submitted to analyses to determine the reliability of the deviance responses. Thus, there were four vMMN analyses per

**FIGURE 5 | Source images for "zha" standard.** Images show the depth-weighted minimum norm estimate of dipole source strength constrained to the surface of the cortex using a boundary element forward model and a generic anatomical model at 20-ms intervals starting from 90 ms after the onset of visible motion for the group mean ERPs for syllable "zha" as *standard*. The cyan bar indicates the time at which "zha" visibly differs from "fa," and the magenta bar indicates the time at which "zha" visibly differs from "ta." Images are thresholded at 20 pA·m. Initial activity is in the occipital cortex. At 190 ms after syllable onset, strong, widespread bilateral posterior temporal activity begins that lasts until 290 ms, with weaker activations recurring through 610 ms post-stimulus onset. Activation in the right posterior temporal cortex is more widespread and inferior to that on the left. Fronto-central activity is visible from 270 to 490 ms post-stimulus onset.

hemisphere. They were for "zha" in its *near* or *far* context, "fa" in the *far* context, and "ta" in the *near* context. Summaries of the results are given in **Table 2**. The duration (begin points to end points) of reliable deviance responses varied across syllables

**FIGURE 6 | Source images for "ta" standard.** Images show the depth-weighted minimum norm estimate of dipole source strength constrained to the surface of the cortex using a boundary element forward model and a generic anatomical model at 20-ms intervals starting from 90 ms after the onset of visible motion for the group mean ERPs for syllable "ta" as *standard*. The magenta bar indicates the time at which "ta" visibly differs from "zha." Images are thresholded at 20 pA·m. Initial activity is in occipital cortex. At 130 ms after syllable onset, bilateral posterior temporal activity begins that fades by 250 ms post-stimulus onset, but then recurs from 330 to 590 ms on the right and from 330 to 470 ms on the left. Fronto-central activity is visible from 270 to 470 ms post-stimulus onset.

(from 50 to 185 ms) and varied in mean voltage (from −0.35 to −0.85µV).

**Figure 7** shows the statistical results for the EOI cluster waveforms for each contrast and hemisphere. The theoretically predicted results were obtained. All of the right-hemisphere contrasts resulted in reliable deviance responses. They were "zha" in the *near* context from 239 to 288 ms post-deviation onset, "zha" in the *far* context from 324 to 500 ms post-deviation onset, "ta" from 449 to 500 ms post-deviation onset, and "fa" from 300 to 442 ms post-deviation onset. Only the *far* contrasts resulted in reliable left-hemisphere deviance responses. They were "zha" in the *far* context from 322 to 497 ms post-deviation onset and "fa" from 251 to 435 ms post-deviation onset.

# *Comparison of far vs. near vMMNs*

Difference waveforms were computed using the standard type of approach to the vMMN, that is, by subtracting the EOI cluster ERPs to standards from the response to deviants for each stimulus contrast and hemisphere on a per-subject basis. The magnitudes of the vMMN waveforms were then compared between *far* and *near* contrasts using the resampling method that was applied to the analyses of standards vs. deviants.

The "zha" *near* and *far* vMMN waveforms were found to be reliably different (**Figure 8**). On the left, the difference wave for "zha" in the *far* context was reliably larger (i.e., more negative) than for "zha" in the *near* context (320 to 443 ms post-deviation onset), not unexpectedly as the *near* context did not result in an observable vMMN. On the right, the difference wave was also reliably larger for "zha" in the *far* context (from 331 to 449 ms post-deviation onset), although both contexts were effective. The results were similar when the vMMN waveforms were compared between "fa" vs. "ta" (**Figure 9**). On the left, the difference wave for "fa" was reliably larger than for "ta" (309–386 ms post-deviation onset). On the right, the difference wave was also reliably larger for "fa" (from 327 to 420 ms post-deviation onset).

### *Fronto-central results*