Auditory Cortex Tracks Both Auditory and Visual Stimulus Dynamics Using Low-Frequency Neuronal Phase Modulation
[['We do not experience the world as parallel sensory streams; rather , the information extracted from different modalities fuses to form a seamlessly unified multi-sensory percept dynamically evolving over time .', 'There is a compelling benefit to multimodal information: behavioral studies show that combining information across sensory domains enhances unimodal detection ability—and can even induce new , integrated percepts [1]–[4] .', 'The relevant neuronal mechanisms have been widely investigated .', 'One typical view posits that multisensory integration occurs at later stages of cortical processing , subsequent to unisensory analysis .', 'This view has been supported by studies showing that higher , “association” areas in temporal , parietal , and frontal cortices receive inputs from multiple unimodal areas [5]–[8] and respond to stimulation in manner that reflects multisensory convergence , for example with amplified or suppressed responses for multimodal over unimodal stimuli [9]–[12] .', 'A growing body of evidence provides a complementary view , suggesting that cross-modal interaction is not restricted to association areas and can occur at early , putatively unisensory cortical processing stages [11] , [13] .', 'For example , non-auditory stimulation ( visual and somatosensory ) has been found to drive auditory cortical activity , as observed in both humans and animals [4] , [14]–[23] .', 'Similarly , visual cortical responses are modulated by inputs from other modalities [24] , [25] .', 'Importantly , independent anatomical evidence also reveals direct connections among early sensory areas [26] , [27] .', 'Therefore , multisensory integration may operate through lateral cross-sensory modulation , and there exist multiple integration pathways beyond purely hierarchical convergence [12] , [28] , [29] .', 'How is early cortical activity coordinated ?', 'Beyond the classical examination of cross-modal influences on neuronal firing rate , recent studies suggest temporal coherence [30] , [31] to underlie multisensory integration [28] , [32] .', 'This view posits that oscillations synchronous across different brain areas might serve an essential role in multisensory binding , similarly as that for feature binding and attentional selection [30] , [33]–[36] .', 'Several EEG/MEG studies in humans implicate oscillations and cross-area coherence in multisensory integration [29] , [37]–[42] .', 'However , most of the studies employed short , transient multisensory stimuli and focused on the evoked transient oscillatory power instead of examining sustained cross-modal modulation for long , naturalistic audiovisual streams .', 'Importantly , with regard to the cross-area modulation mechanism , it has recently been suggested that cross-sensory phase modulation may underlie this interaction [28] , [32] , [43] , [44] .', 'For example , non-auditory inputs ( re ) set the phase of ongoing local neuronal activity in auditory cortex to a high-excitability state ( reflected in phase angle ) , effectively “selecting” or amplifying the response to subsequent auditory inputs [11] , [13] , [20] , [22] , [45] .', 'Whether such a mechanism is implemented in populations of neurons and could mediate the perception of audiovisual speech in human viewers/listeners is completely unknown .', 'In order to test directly the proposal of cross-modal phase modulation of oscillatory neural activity , we investigate online audiovisual interaction , in auditory and visual cortices simultaneously , by recording magnetoencephalography ( MEG ) responses from human participants presented with 30-s-long natural movie clips from the movie “Dumb and Dumber” ( 1994 , New Line Platinum Series ) .', 'These video segments had either “matched” ( congruent audio-visual combinations , V1A1 , V2A2 , V3A3 ) or “mixed” streams ( incongruent audio-visual , V1A3 , V2A1 , V3A2 ) .', 'Building on our previous results showing that the theta-band phase pattern in human auditory cortex reflects the dynamic structure of spoken sentences [46] , we employed a new trial-by-trial phase tracking analysis to explore multi-sensory integration .', 'We conjectured that , in response to naturalistic audio-visual streams ( movies ) , the low-frequency phase of auditory and visual sensory activity in single trials', '( i ) will robustly track and discriminate ( in a classification analysis ) the sensory stream dynamics in each modality ( “within-modality tracking”; i . e . auditory channel tracks auditory , visual tracks visual dynamics ) ,', '( ii ) may carry information about stimulus dynamics in the other modality ( “cross-modality tracking”; e . g . an auditory channel can reflect visual dynamics ) , and', '( iii ) that the efficacy of such cross-sensory phase modulation ( trial-to-trial phase variance ) depends on the relative audiovisual timing , such that a temporally matched audio-visual stream will enhance phase tracking reliability , compared to unmatched ( mixed ) pairs .', 'Our data support these predictions , highlighting the critical role of cross-sensory phase modulation of oscillations in multisensory integration , commensurate with the hypothesis [28] , [44] .', 'We thus argue that multi-sensory integration may use cross-modal phase modulation as a basic mechanism to construct temporally aligned representations that facilitate perceptual decoding of audiovisual speech .'], ['We first assessed whether MEG responses in single trials can reliably track the six movie clips we presented to participants ( three Matched , three Mixed ) .', 'The phase and power pattern of MEG responses to the movies ( see illustration of cross-trial phase coherence analysis in Figure 1a ) and the corresponding discrimination ability were calculated as a function of frequency of the brain response ( 0–50 Hz ) using previously developed methods [46] .', 'We quantified stimulus-specific trial-by-trial phase and power pattern coherence in 20 auditory and 20 visual channels , which were defined in separate auditory ( 1 kHz tone pip ) and visual ( alternating checkerboard ) localizer pretests for each subject ( see Figure S2 ) .', 'As illustrated in Figure 2a , both auditory and visual cortical responses showed good discrimination ability in the delta-theta-band ( 2–7 Hz ) phase pattern ( above zero discrimination score , 2-way ANOVA , main effect of frequency , F ( 24 , 840 ) =\u200a7 . 94 , p<0 . 0001; post-hoc one-sample t test in delta-theta band ( 2∼7 Hz ) , Auditory: t\u200a=\u200a11 . 57 , df\u200a=\u200a35 , p<0 . 0001 , Visual: t\u200a=\u200a11 . 16 , df\u200a=\u200a35 , p<0 . 0001 ) .', 'Critically , phase tracking was not accompanied by comparable power pattern tracking ( Figure 2b , 2-way ANOVA , main effect of frequency , F ( 24 , 840 ) =\u200a0 . 517 , p\u200a=\u200a0 . 97; t test in delta-theta band ( 2∼7 Hz ) , Auditory: t\u200a=\u200a0 . 913 , p\u200a=\u200a0 . 368; Visual: t\u200a=\u200a0 . 698 , p\u200a=\u200a0 . 49 ) .', 'These results demonstrate that the phase of ongoing auditory and visual cortical low-frequency oscillations is reliably modulated by the audio-visual stimuli , and thus conveys information about the rich naturalistic dynamics of these multi-sensory movies .', 'Having established the sensitivity of the low-frequency phase pattern to different audiovisual movie streams using the cross-trial phase coherence ( Figure 1a ) , we next evaluated its modality specificity in auditory and visual cortical responses , by employing a cross-movie coherence analysis ( Figure 1b; Figure S3 schematizes the logic ) .', 'Given the predominantly unisensory characteristics of cortical responses early in the cortical processing hierarchy , the low-frequency phase pattern should be mainly driven by the stimulus in the corresponding sensory modality .', 'We thus tested a double dissociation hypothesis , namely that in auditory channels , movie clips sharing the same auditory input regardless of visual input ( stimuli we call “SameAud” ) should induce a more similar low-frequency phase pattern response ( and display higher cross-movie delta-theta phase coherence ) than those containing the same visual but different auditory input ( stimuli called “SameVis” ) ; analogously , in visual channels , SameVis movies should yield higher cross-movie delta-theta phase coherence compared to SameAud movie pairs .', 'For the three matched clips ( V1A1 , V2A2 , V3A3 ) , we selected the corresponding SameVis and SameAud stimuli ( see Figure 1b and Figure S3 for visualization of the design; e . g . , for matched clip V1A1 , its SameVis counterpart is V1A3 , its SameAud is V2A1 ) ; we then calculated the similarity or coherence between the responses to matched clips and the corresponding SameAud or SameVis mixed clips ( ) , separately for auditory and visual areas .', 'The cross-movie low-frequency phase coherence results ( , ) show a double dissociation ( Figure 3a; condition×place interaction , F ( 1 , 5 ) =\u200a10 . 44 , p\u200a=\u200a0 . 023 ) .', 'This confirms the efficacy of the auditory and visual “functional channel localizers”; more importantly , though , this analysis suggests , plausibly , that the phase patterns over auditory and visual areas are predominantly driven by the sensory stimulus structure in the corresponding modality .', 'Critically , the corresponding power coherence ( , ) did not show the double dissociation pattern ( Figure 3b; condition×place interaction , F ( 1 , 5 ) =\u200a0 . 077 , p\u200a=\u200a0 . 79 ) , confirming that precise timing—as reflected in the phase of delta and theta activity—plays a dominant role in sensory stream representation .', 'The modality-dependent characteristics of the delta-theta phase pattern in all 157 recorded channels were verified by comparing the spatial distribution maps of the cross-movie delta-theta phase coherence ( , ) .', 'We observed a lateral temporal origin of and an occipital origin of in every subject ( Figure 4 ) .', 'The spatial distribution results thus confirm the finding that in response to a multi-sensory audiovisual stream , the low-frequency phase of the auditory and visual cortical activities principally and concurrently tracks the respective sensory stimulus dynamics .', 'We then examined the critical hypothesized cross-modality modulation effects in the low-frequency phase pattern , by studying whether naturalistic visual input can affect the phase of auditory cortical oscillations ( as previously only observed using artificial stimuli and in animal data ) , and similarly whether the auditory dynamic structure influences the phase of ongoing rhythmic activities in visual cortex , to some extent .', 'A cross-movie coherence analysis was again performed ( Figure 1b; Figure S3 schematizes the logic ) , by calculating the coherence or similarity between the responses to matched clips and the corresponding NoSame mixed clips , i . e . movie clip differing in both auditory and visual input ( e . g . , for matched clip V1A1 , V2A2 , V3A3 , their respective NoSame counterpart is V3A2 , V1A3 , V2A1 ) , in auditory and visual areas separately .', 'The logic of this analysis is as follows: If the low-frequency phase pattern in one sensory modality is systematically influenced by the other modality , movies sharing same visual input ( SameVis ) should show more similar low-frequency phase pattern in auditory cortex , compared to movies differing in both visual and auditory inputs ( NoSame ) ; similarly , in visual cortex , the SameAud movies should show higher cross-movie coherence than NoSame movies .', 'Figure 3a shows that the NoSame pair manifested the smallest cross-movie phase coherence ( ) , supporting our hypothesis ( 3-way ANOVA , condition main effect , F ( 2 , 10 ) =\u200a36 . 394 , p<0 . 0001; post-hoc analysis , NoSame versus SameVis , p<0 . 0001 , NoSame versus SameAud , p<0 . 0001; condition×place interaction , F ( 2 , 10 ) =\u200a8 . 467 , p\u200a=\u200a0 . 007 ) .', 'The delta-theta power pattern reflects no such effect ( Figure 3b ) .', 'This suggests that in response to an audio-visual stream ( e . g . , V1A1 ) , the phase of the cortical activity is driven and modulated not only by the input in the corresponding modality ( double dissociation result discussed above ) but also by input from another modality ( cross-sensory phase modulation ) .', 'The above cross-movie coherence results demonstrate that the phase pattern in response to an audiovisual stream carries information about both auditory and visual stimulus structure .', 'We next ask whether multisensory tracking is simply a mixture of passive following responses to unisensory stimuli , or—more interestingly—whether phase-tracking plays an active role in multisensory integration , by establishing a cross-modal temporal context in which a unisensory stimulus unfolds and merges into a coherent perceptual representation .', 'We first examined the similarity in the elicited phase pattern response in auditory and visual areas .', 'Given the congruent temporal structure in matched audiovisual stimuli , together with the observed within-modality phase tracking , we predict that both auditory and visual areas show higher similarity in low-frequency phase responses for the matched conditions .', 'The cross-movie analysis results support the hypothesis ( Figure 5c , paired t test , t ( 9 ) =\u200a2 . 31 , p\u200a=\u200a0 . 046 ) ; the corresponding power coherence revealed no statistical difference ( Figure 5d , paired t test , t ( 9 ) =\u200a1 . 93 , p\u200a=\u200a0 . 086 ) .', 'In light of the observed similarity between the phase response in the two modalities , we next conjecture that the cross-modality phase modulation will occur in a manner “temporally commensurate” to within-modality phase modulation , leading to more temporally reliable integration and consequently achieving a more robust low-frequency-based representation of audio-visual naturalistic stimuli ( enhanced trial-to-trial response reliability ) in both sensory areas ( not between areas ) .', 'Importantly , the cross-trial reliability enhancement hypothesis cannot be derived from a passive following response interpretation .', 'We compared the delta-theta cross-trial phase coherence for the three matched and three mixed movies separately , noting that the three movies in the mixed group contained exactly the same auditory and visual inputs as the matched one—but in incongruent audio-visual combinations ( Figure 1a ) .', 'We observed stronger trial-by-trial delta-theta phase pattern coherence in the matched group than in the mixed group ( 2-way ANOVA , significant main effect of condition , F ( 1 , 9 ) =\u200a7 . 33 , p\u200a=\u200a0 . 024 ) , in both auditory and visual areas ( Figure 5a ) .', 'The cross-trial power coherence revealed no significant difference between the two conditions ( Figure 5b , condition main effect , 2-way ANOVA , F ( 1 , 9 ) =\u200a3 . 64 , p\u200a=\u200a0 . 09 ) .', 'The result that the trial-by-trial phase reliability depends on the relative audiovisual temporal relationship thus supports the “active cross-modal phase modulation” hypothesis for multisensory integration .', 'In our view , sensory cortical activity builds a more efficient and robust continuous representation for a temporally congruent multi-sensory stream by mutually modulating the low-frequency phase of ongoing oscillatory activity in an active manner , perhaps facilitating temporal packaging of information that can then act “predicatively” across modalities .', 'To apply a unified analysis framework to our data , a classification analysis was employed based on the low-frequency ( 2–7 Hz ) phase pattern in single response trials across all six movies .', 'For each of the six movie clips , the delta-theta phase pattern as a function of time for one single trial response under one stimulus condition was arbitrarily chosen as a template response for that movie .', 'The delta-theta phase pattern of the remaining trials of all stimulus conditions was calculated , and their similarity to each of the six templates was defined as the distance to the templates .', 'Responses were then classified to the closest movie template .', 'The classification was computed 100 times for each of the 20 auditory and 20 visual channels in each subject , by randomly choosing template combinations .', 'This classifier analysis shows that the delta-theta phase pattern successfully discriminates among movies .', 'The individual trial data for each condition were predominantly classified as belonging to that condition , for both auditory ( Figure 6a ) and visual ( Figure 6b ) areas .', 'Second , the classification results support the tracking hypothesis for matched versus mixed conditions , revealing higher “self”-classification for matched than mixed movies .', 'Third , the modality-specific characteristics of phase tracking were manifested in the classification in that in auditory areas , each of the six movies was categorized to the movie stimulus sharing the same auditory input ( SameAud ) with larger proportion than to SameVis input , and vice versa for visual areas .', 'Finally , the classification results also support the elevated response reliability by congruent audiovisual stimuli .', 'The response to each movie clip was primarily classified to itself , secondly to the clip sharing the same modality ( e . g . , SameAud for auditory channels ) , and thirdly to the movies sharing the same input in the other modality ( e . g . , SameVis in auditory area ) , which has a significantly better classification proportion than stimuli differing in both inputs ( NoSame ) .', 'A statistical analysis and summary of the classification data ( Figure 6c ) underscores the effect of this cross-sensory phase modulation .', 'The results demonstrate that the low-frequency phase pattern in sensory cortices can be relied on for audiovisual stream discrimination in single trial responses , and that it is modulated by input from multiple sensory domains , reflecting an active cross-sensory integration , dynamically evolving in time .', 'Neurophysiological work in animal preparations suggests that non-auditory inputs can modulate auditory responses towards a preferred excitability state , by aligning the phase of ongoing low-frequency auditory activity with a specific phase angle known to elicit maximal stimulus-driven responses , resulting in the cross-sensory response amplification [20] , [22] .', 'We hypothesize that stimulus-induced temporal regularization leads to robust phase tracking , by resetting the phase of the intrinsic low-frequency rhythmic activity to a preferred phase .', 'We thus expect', '( i ) that the cross-trial delta-theta phase coherence is phase dependent , and the phase values corresponding to high cross-trial phase coherence values are non-uniformly distributed and centered on a preferred phase angle , and', '( ii ) that the matched movie elicits a larger fraction of optimal phase compared to the mixed condition , since a temporally congruent stream would achieve cross-sensory phase tracking enhancement , by regularizing low-frequency phase to the optimal phase angle more robustly in each response trial .', 'We explored the relationship between the cross-trial phase coherence and the corresponding phase angles and observed an increasingly clustered phase angle distribution ( around 0 and ± ) for higher phase coherence in both auditory and visual areas ( Figure 7a , upper and lower panel ) .', 'As shown in Figure 7b , we further quantified the deviation of phase distribution from uniform distribution as a function of cross-trial phase coherence values , and the results confirm that higher phase coherence corresponds to larger deviation from uniform distribution ( 2-way ANOVA , F ( 19 , 95 ) =\u200a67 . 99 , p<0 . 001 ) , thus suggesting a trend of non-uniform phase clustering for the robust phase tracking pattern .', '( Note that the drop in the deviation values for the highest phase coherence ( ∼1 ) may be due to the artifacts produced by small samples and large variance across subjects during such a high coherence regime .', ') The findings demonstrate that it is mainly the stimulus-induced delta-theta phase resetting to the preferred phase angle ( 0 or ± ) that regularizes the low-frequency phase pattern in each response trial to improve the phase tracking reliability .', 'In addition , as shown in Figure 7c , the matched movies showed a larger fraction of optimal phase angle ( 0 or ± ) than mixed movies for higher phase coherence ( >0 . 7 ) in both auditory and visual areas , as hypothesized; statistical testing confirms that phase angle at ± was more relevant to preferred or optimal phase ( 2-way ANOVA , main effect of condition , F ( 1 , 5 ) =\u200a5 . 794 , p\u200a=\u200a0 . 06 ) than phase angle at 0 ( 2-way ANOVA , main effect of condition , F ( 1 , 5 ) =\u200a2 . 856 , p\u200a=\u200a0 . 152 ) , commensurate with optimal phase findings in neurophysiological studies [20] , [22] , [45] .', 'The results support the view that the visual ( auditory ) stream in a matched movie modulates the auditory ( visual ) cortical activity by aligning the phase to the optimal phase angle so that the expected auditory ( visual ) input arrives during a high excitability state , to be amplified and achieve the cross-sensory enhancement .', 'In contrast , mixed , incongruent audiovisual streams cannot benefit from the cross-sensory phase regularization and thus are driven to the preferred phase angle with a significantly smaller fraction than matched movie stimuli .'], ['Unlike pairings of transient artificial stimuli used in most previous audiovisual studies , we examined the cross-modal integration effects in presumptively unimodal areas by employing naturalistic audiovisual movies that are ethologically natural and extended in time ( 30-s film clips ) .', 'Naturalistic stimuli contain complex structure and rich dynamics in the time domain , and it has been suggested that the relevant neural mechanisms are in part shaped by the statistical structure of natural environments [47] , [48] .', 'Our previous MEG studies revealed that the phase pattern of theta-band responses reliably tracks and discriminates natural spoken sentences [46] .', 'Here we build on and extend the previous findings by showing that delta-theta phase tracking exists for multi-sensory streams and that the low-frequency phase response in auditory and visual cortices reliably tracks audio-visual movies concurrently .', 'There is emerging consensus that the signals quantified in neuroimaging ( e . g . , MEG signals ) reflect synchronized large-scale neuronal ensemble activity and have been found to mainly derive from LFP rather than spiking activity [49] .', 'A recent neurophysiological study in monkeys quantified the information different codes carry about natural sounds in auditory cortex and found that spiking responses interpreted with regard to the relative phase of the accompanying slow ongoing LFP are more informative about the properties of the dynamic sound than spiking responses alone [50] .', 'The same encoding scheme has also been observed in visual cortex in response to natural movies [51] .', 'Our results from human neuroimaging converge with these neurophysiological studies on low-frequency phase tracking for naturalistic streams and are commensurate with the observed essential role of brain oscillations in sensory processing , feature integration , and response selection within the various sensory modalities [30] , [34]–[36] , [52] .', 'It has been argued that intrinsic rhythms undergo significant phase resetting in response to stimulus presentation [35] , [53] , [54] , and crucially , some studies demonstrate that neuronal oscillations enhance the response robustness to natural stimulation by modulating the excitability state ( phase resetting ) for spiking activity [55] .', 'Could one argue that the observed delta-theta phase tracking is due to different levels of attention to a given modality , given the important role of attention in multisensory integration [25] , [56] , [57] ?', 'Such a view cannot be a sufficient explanation because the low-frequency phase pattern distinguishes the audio-visual streams belonging to the matched or mixed conditions , both of which elicit similar attentional states .', '( The three matched ( or mixed ) movies should elicit similar attentional states , and therefore the delta-theta phase pattern should not be able to discriminate them only based on attentional state .', ') Interestingly , previous studies show that such cross-sensory interactions occur in anaesthetized animals [19] , [21] .', 'These observations suggest that the general attentional level is not the main source underlying the observed delta-theta phase tracking .', 'Recent studies [56] , [57] revealed that the phase of low-frequency oscillations in auditory and visual cortex entrains to the rhythm of the attended sensory stream amidst multi-sensory inputs and thus could track either a visual or auditory stimulus .', 'They suggest the phase modulation mechanism to underlie temporally based attention .', 'Their results further challenge an attentional-load explanation for the present data , given the observed modality-specific characteristics ( the double dissociation results ) , and support that the observed delta-theta phase tracking is not due to global modality-independent attentional modulation .', 'Uncontrolled eye movements also constitute a possible confounding factor , given previous findings reporting the effect of eye position on the auditory cortical responses [17] .', 'We believe that the eye-movement-related activity may contribute to phase modulation in early sensory activity , but not in a dominant way , given that the cross-modal phase modulation exists under both anesthetized conditions [19] , [21] and controlled eye fixation conditions [22] .', 'Note that eye movements by themselves cannot account for the observed stronger modulation for matched over mixed audiovisual stimuli; both carry the same visual stream; which should result in a comparable pattern of eye movements .', 'More generally , during the free viewing of movies , eye movements are argued to be tightly correlated with stimulus dynamics , which in turn induces phase tracking in brain signals , and therefore the phase modulation mechanism may also be integral to the temporally based attention .', 'Fries [35] recently proposed a rhythmic input gain model to link attention to brain oscillations and suggested that the strength of gamma-band synchronization ( binding by synchronization ) is modulated with the theta rhythm , the phase of which makes or breaks selections of input segments , thus constituting a strong link to the “biased competition” modal in visual attention [33] .', 'We found that low-frequency phase patterns were sufficiently reliable to continuously track the naturalistic audiovisual streams .', 'The crucial relevance of low-frequency oscillations to perceptual analysis has been observed in several studies [20] , [22] , [46] , [50] , [51] .', 'The acoustic structure of both natural sounds and movies contain rich dynamics on multiple time scales , but with power dominance in the low-frequency range [48] , [58]–[60] .', 'Accumulating evidence demonstrates that a coarse representation suffices for the comprehension of natural streams [61] .', 'For example , from the perspective of speech processing , a temporal window of ∼200 ms corresponds to mean syllable length across languages , and such a temporal window has been suggested as a fundamental unit for speech perception [62] , [63] .', 'The observed tracking ability of slow quasi-rhythmic ( and aperiodic ) activity may be simply driven by the input temporal pattern , but we conjecture that it reflects an internal stable processing rhythm [64] that is ideally suited to match the gross statistical temporal structure of natural streams .', 'Recent data [65] demonstrate robust temporal correspondence in the delta-theta range ( 2∼7 Hz ) between visual and auditory streams in multisensory speech signals , supporting this interpretation .', 'In addition to the essential role of long-duration time scales in natural stimuli , the dynamic structure at other biologically relevant scales , especially the short windows ( e . g . , ∼25 ms ) corresponding to gamma band oscillation , also carries important information [62] , [64] .', 'Several previous studies show the relevance of gamma oscillations to multisensory integration , but in contexts of transient or evoked responses [40] , [42] , which is a very different approach from ours .', 'In the current work , we examine the sustained response pattern to natural complex audiovisual scenes and the relevance to multisensory integration .', 'A possible factor accounting for the absence of evidence for fast , gamma rhythms in tracking might lie in the task demands; subjects were only asked to passively view and listen to the audiovisual streams , without requiring their focused , selective attention to fast transitions , phonemes , any aspect of sublexical information , etc .', 'Crucially , both unimodal and multimodal naturalistic streams contain various temporal scales that are nested within each other .', 'For example , in human speech , high-frequency events ( e . g . , formant transitions ) are temporally nested within low-frequency structures ( e . g . , syllables , phrases ) .', 'Correspondingly , human cortical oscillations at different frequencies also manifest similar temporally nested relationships and tend to be phase-amplitude coupled [66] .', 'Such cross-scale coupling in both naturalistic extended stimuli and brain oscillations are consistent with the “sampling window hypothesis” for speech perception [62] , and further indicate a general cross-scale modulation mechanism underlying multi-sensory interaction [56] .', 'The central finding concerns the hypothesis of active cross-modality phase modulation of endogenous oscillations in a multi-sensory context .', 'Specifically , we observed that the auditory and visual modalities can mutually and actively modulate the phase of the internal low-frequency rhythms in early sensory cortical regions and that such cross-sensory driving efficiency depends on the relative audiovisual timing .', 'A study recording A1 in awake macaques [20] revealed phase modulation in multi-sensory interaction: somatosensory inputs enhanced auditory processing by resetting the phase of ongoing neuronal oscillations in A1 so that the accompanying auditory input arrived during a high-excitability phase .', 'A further neurophysiological experiment exploring the impact of visual stimulation on auditory responses demonstrated that visual stimuli modulated auditory cortex activity , at the level of both LFP and single-unit responses [22] .', 'Importantly , they too found that the observed cross-sensory enhancement correlated well with the resetting of slow oscillations to an optimal phase angle , and the multi-sensory interactions were sensitive to the audiovisual timing .', 'Moreover , they discovered that matched audiovisual stimuli enhanced the trial-to-trial response reliability in auditory cortex of alert monkeys [45] , precisely like one of our central findings of a tight link between cross-sensory modulation efficacy and relative audiovisual timing congruency .', 'Our results in humans are thus in good agreement with these animal data and also implicate neural mechanisms accounting for previous behavioral results showing temporally matched visual amplification of auditory processing , in both monkeys [67] and human subjects [4] , [68] .', 'Given the simple binary design here ( matched versus mixed ) , further studies need to be executed by continuously jittering the temporal relationship between auditory and visual stimuli and investigating the influences in both behavior and cross-modal low-frequency phase modulation in a more systematic way .', 'Recently , Schroeder et al . [44] proposed a phase-resetting-based mechanism to solve the “cocktail party” problem using such a mechanism and hypothesized that the visual amplification of speech perception is operating through efficient modulation or “shaping” of ongoing neuronal oscillations .', 'Our results support such a model and indicate that multi-sensory integration is at least in part based on a cross-modal phase resetting mechanism in early cortical sensory regions .', 'The phase patterns of the ongoing rhythmic activity in early sensory areas help construct a temporal framework that reflects both unimo
['Introduction', 'Results', 'Discussion', 'Materials and Methods']
['Integrating information across sensory domains to construct a unified representation of multi-sensory signals is a fundamental characteristic of perception in ecological contexts .', 'One provocative hypothesis deriving from neurophysiology suggests that there exists early and direct cross-modal phase modulation .', 'We provide evidence , based on magnetoencephalography ( MEG ) recordings from participants viewing audiovisual movies , that low-frequency neuronal information lies at the basis of the synergistic coordination of information across auditory and visual streams .', 'In particular , the phase of the 2–7 Hz delta and theta band responses carries robust ( in single trials ) and usable information ( for parsing the temporal structure ) about stimulus dynamics in both sensory modalities concurrently .', 'These experiments are the first to show in humans that a particular cortical mechanism , delta-theta phase modulation across early sensory areas , plays an important “active” role in continuously tracking naturalistic audio-visual streams , carrying dynamic multi-sensory information , and reflecting cross-sensory interaction in real time .']
['When faced with ecologically relevant stimuli in natural scenes , our brains need to coordinate information from multiple sensory systems in order to create accurate internal representations of the outside world .', 'Unfortunately , we currently have little information about the neuronal mechanisms for this cross-modal processing during online sensory perception under natural conditions .', 'Neurophysiological and human imaging studies are increasingly exploring the response properties elicited by natural scenes .', 'In this study , we recorded magnetoencephalography ( MEG ) data from participants viewing audiovisual movie clips .', 'We developed a phase coherence analysis technique that captures—in single trials of watching a movie—how the phase of cortical responses is tightly coupled to key aspects of stimulus dynamics .', 'Remarkably , auditory cortex not only tracks auditory stimulus dynamics but also reflects dynamic aspects of the visual signal .', 'Similarly , visual cortex mainly follows the visual properties of a stimulus , but also shows sensitivity to the auditory aspects of a scene .', 'The critical finding is that cross-modal phase modulation appears to lie at the basis of this integrative processing .', 'Continuous cross-modal phase modulation may permit the internal construction of behaviorally relevant stimuli .', 'Our work therefore contributes to the understanding of how multi-sensory information is analyzed and represented in the human brain .']
['neuroscience/cognitive neuroscience', 'neuroscience/sensory systems']
How is naturalistic multisensory information combined in the human brain? Based on MEG data we show that phase modulation of visual and auditory signals captures the dynamics of complex scenes.