in many circumstances people experience external events by a number of different sensory modalities . for example , when someone is talking , there is auditory and visual information that is initially processed by specialized neural pathways . ultimately , though , the different sensory signals are integrated into a coherent multimodal percept of the speaker .
many behavioural and neurophysiological studies have emphasized the importance of spatial co - localisation and temporal synchrony for intersensory pairing to occur ( e.g. , welch and warren 1980 ; bedford 1989 ; stein and meredith 1993 ; radeau 1994 ; bertelson 1999 ; welch 1999 ) .
however , there is accumulating evidence that some intersensory phenomena may not require spatial alignment ( welch et al . 1986 ; scheier et al . 1999 ; morein - zamir et al .
; teder - salejarvi et al . 2005 ; vroomen and keetels 2006 ; keetels et al . 2007 ) . in the present study
, we explored the importance of spatial alignment for audio  visual ( av ) temporal recalibration .
temporal recalibration refers to the phenomenon that the brain adapts itself to ( small ) temporal asynchronies . in a multi - modal percept
this occurs , despite the fact that there are natural asynchronies between the senses caused by differences in signal transduction time through air and differences in neural transmission time .
at least two options are available to handle these asynchronies : one is concerned with immediate corrections , the other is important for adaptation on a longer time scale . as concerns the immediate effect , several studies have shown that the brain corrects for small av temporal asynchronies by shifting one or both modalities on the time scale so that the temporal discordance is reduced .
for example , when a sound and a light are presented at slightly different onset times ( usually in the order of 100  ms ) , the temporal asynchrony is reduced by a capturing effect of the light by the sound ; a phenomenon called temporal ventriloquism ( scheier et al . 1999 ; fendrich and corballis 2001 ; morein - zamir et al . 2003 ; vroomen and de gelder 2004 ; stekelenburg and vroomen 2005 ; vroomen and keetels 2006 ) .
temporal ventriloquism can , for example , be demonstrated by the use of a visual temporal order judgment ( toj ) task in which participants are presented two lights at various stimulus onset asynchronies ( soas ) and judge which light came first . by presenting a sound before the first and after the second light
, the just noticeable difference ( jnd ) improves ( i.e. participants become more sensitive ) , presumably because the two sounds attract the temporal occurrence of the two lights , and thus effectively pull the lights further apart in time ( scheier et al .
there are also long - term effects reflecting an adaptive change to av asynchrony , a phenomenon called temporal recalibration ( fujisaki et al .
for example , vroomen et al . studied temporal recalibration by exposing participants to 3  min of sound and light flashes with a constant time lag , after which an av toj or av simultaneity task was performed .
following exposure , observers were given av test stimuli and judged whether the sound or the light came first , or whether the sound and light were simultaneous or successive .
the results showed that the point of subjective simultaneity ( pss ) , the point of perceived temporal alignment between the sound and the light , was shifted in the direction of the exposure lag .
so , following exposure to a train of sound - first stimulus pairs , participants perceived sound - first trials as more simultaneous than after light - first exposure .
( 2004 ) demonstrated similar findings and also provided somewhat mixed evidence that temporal recalibration may generalize to different test stimuli than the ones presented during exposure .
the authors adapted participants to asynchronous tone - flash stimulus pairs and later tested them on the  bounce  illusion ( sekuler et al .
two visual targets that move across each other can be perceived either to bounce off or to stream through each other .
a brief sound presented at the moment that the visual targets coincide generally biases visual perception in favour of a bouncing motion , while without sound observers tend to report a streaming percept .
light pairs , the optimal delay for obtaining the bounce illusion was shifted in the same direction , but in other conditions , the magnitude of the after - effect was smaller for some of the cross - adaptation conditions .
 tactile temporal recalibration by exposing participants to streams of brief auditory and tactile stimuli presented in synchrony , or else with the auditory stimulus leading by 75  ms . rather than a shift in the pss , they observed that the jnd to resolve audio 
tactile temporal order was larger after exposure to the desynchronized streams than after exposure to the synchronous streams .
the authors argued that the temporal window for integration was widened due to audio  tactile asynchrony .
the goal of the present study was to explore whether spatial disparity between a sound and light affects temporal recalibration . according to the 
common notion  of intersensory pairing , intersensory effects should be bigger when the individual components of a multisensory stimulus come from the same location ( e.g. welch and warren 1980 ; bedford 1989 ; stein and meredith 1993 ; radeau 1994 ; bertelson 1999 ; welch 1999 ) .
however , vroomen and keetels ( 2006 ) demonstrated that , at least for temporal ventriloquism , spatial correspondence between sound and light is not important . in their study ,
a visual toj task was used with a sound presented before the first and after the second light .
temporal ventriloquism manifested itself as an improvement in the jnds but , crucially , the improvement was unaffected by whether the sounds came from the same or a different position as the lights , whether the sounds were static or moved , or whether the sounds and lights came from the same or opposite sides of fixation .
( 2007 ) further examined how principles of auditory grouping ( bregman 1990 ) relate to intersensory pairing .
they embedded two sounds that normally enhance sensitivity on the visual temporal order judgement task in a sequence of flanker sounds , which either had the same or different frequency , rhythm , or location . in all experiments ,
temporal ventriloquism only occurred when the two capture sounds differed from the flankers , thus demonstrating that intramodal grouping of the sounds in the auditory stream took priority over intersensory pairing . by combining principles of auditory grouping with intersensory pairing
, they also demonstrated that the capture sounds could , counter - intuitively , be more effective when their locations differed from that of the lights rather than when they came from the same position , thus demonstrating that sound location mattered for auditory grouping , but not intersensory pairing . here we examined whether , like in temporal ventriloquism , spatial disparity is ignored when temporal recalibration is at stake .
participants were exposed for 3  min to a train of asynchronous sounds and lights that came either from the same or a different location .
following exposure , participants performed an av toj task with sounds and lights from either the same or different location .
first , we could test whether temporal recalibration is affected by spatial disparity between the sounds and lights .
recalibration is usually considered to be a low - level perceptual learning phenomenon necessary for re - alignment of the senses ( bertelson and de gelder 2004 ) .
observing an after - effect following exposure to spatially disparate sound  light pairs would provide strong evidence that spatial co - occurrence is , even at this early stage , not necessary for intersensory pairing to occur .
secondly , the use of an exposure  test design allowed us to introduce a change between the exposure and test stimulus so that we could test whether after - effects generalize to different test stimuli .
here we tested whether spatial similarity between the exposure and test sound affects after - effects .
if spatial co - location plays no role in intersensory pairing , one would expect stimulus generalization across space to be complete .
thirty students from tilburg university received course credits for their participation . all reported normal hearing and normal or corrected - to - normal vision .
the study was carried out along the principles laid down in the helsinki declaration and informed consent from the participants was obtained .
visual stimuli were presented by a green led , positioned at central location , at 70  cm from the subject s eyes ( diameter of 0.5  cm , luminance of 40  cd / m ) .
auditory stimuli were 88  db sound bursts presented by one of two loudspeakers ; one directly behind the green led and the other placed laterally at 70  cm distance on either the far left or the far right of the subject ( i.e. , 90  degrees of spatial separation between the sound and light ) .
see fig .  1 for a schematic view of the experimental set - up .
a small red led , placed 2  cm below the green led , was constantly lit during the experiment and served as fixation point . 

, the subject was exposed to a sound  light pair with 100  ms temporal offset ( either sound - first or light first ) . during exposure , sounds were either presented from central ( a , c ) or lateral location ( b , d ) . in the test phase ,
light pairs were presented with a particular soa ranging between 240 and 240  ms , with negative values indicating that the sound was presented first .
sounds of the test - stimulus pair either came from central ( a , b ) or lateral location ( c , d ) schematic illustration of the experimental conditions . in the exposure phase
, the subject was exposed to a sound  light pair with 100  ms temporal offset ( either sound - first or light first ) . during exposure ,
sounds were either presented from central ( a , c ) or lateral location ( b , d ) . in the test phase , sound 
light pairs were presented with a particular soa ranging between 240 and 240  ms , with negative values indicating that the sound was presented first .
sounds of the test - stimulus pair either came from central ( a , b ) or lateral location ( c , d ) three within - subjects factors were used : exposure lag during the exposure phase ( 100 and + 100  ms , with negative values indicating that the sound was presented first ) , location of the sound during exposure ( exposure - sound central or lateral ) and soa between the sound and light of the test stimuli ( 240 , 120 , 90 , 60 , 30 , 0 , + 30 , + 60 , + 90 , + 120 , and + 240  ms , with negative values indicating that the sound came first ) .
the location of the test sound ( central or lateral ) was a between - subjects variable .
half of the participants were tested with central test sounds , the other with lateral test sounds .
these factors yielded 44 equi - probable conditions for each location of the test sound ( 2    2    11 ) , each presented 12  times for a total of 528 trials .
the location of the exposure sound were constant within a block , while the soa between sound and light varied randomly .
the order of the blocks was counterbalanced across participants . in half of the blocks with a lateral exposure sound ,
each block started with an exposure phase consisting of 240 repetitions ( 3  min ) of a sound  light stimulus pair ( isi  =  750  ms ) with a constant lag ( 100 or + 100  ms ) between the sound and the light .
after a 2,500  ms delay , the first test trial then started . to ensure that participants were fixating the light during exposure , they had to detect the occasional occurrence of the offset ( 150  ms ) of the fixation light ( i.e. , a catch trial ) .
the test phase consisted of two parts : a short av re - exposure phase followed by three av test trials of which the temporal order of the sound and light had to be judged .
the re - exposure phase consisted of a train of ten sound - light pairs with the same lag , isi , and sound location as used during the immediately preceding exposure phase .
after 1  s , the three av test trials were presented with a variable soa between the sounds and lights .
the participant s task was to judge whether the sound or the light of the test stimulus was presented first .
an unspeeded response was made by pressing one of two designated keys on a response box .
the next test stimulus was presented 500  ms after a response , and the re - exposure phase of the next trial started 1,000  ms after the response on the third test stimulus . to acquaint participants with the toj task ,
experimental blocks were preceded by four practice blocks in which no exposure preceded the test trials .
the first two practice blocks were to acquaint participants with the response buttons , and consisted of 16 trials in which only the largest soas were presented ( 240 and 120 ) . during this part , participants received verbal feedback (  correct  or  wrong  ) about whether they gave the correct response or not .
the next two practice blocks consisted of 66 trials in which all soas were presented 6  times randomly without verbal feedback .
thirty students from tilburg university received course credits for their participation . all reported normal hearing and normal or corrected - to - normal vision .
the study was carried out along the principles laid down in the helsinki declaration and informed consent from the participants was obtained .
visual stimuli were presented by a green led , positioned at central location , at 70  cm from the subject s eyes ( diameter of 0.5  cm , luminance of 40  cd / m ) .
auditory stimuli were 88  db sound bursts presented by one of two loudspeakers ; one directly behind the green led and the other placed laterally at 70  cm distance on either the far left or the far right of the subject ( i.e. , 90  degrees of spatial separation between the sound and light ) .
see fig .  1 for a schematic view of the experimental set - up .
a small red led , placed 2  cm below the green led , was constantly lit during the experiment and served as fixation point . 

, the subject was exposed to a sound  light pair with 100  ms temporal offset ( either sound - first or light first ) . during exposure ,
sounds were either presented from central ( a , c ) or lateral location ( b , d ) . in the test phase ,
light pairs were presented with a particular soa ranging between 240 and 240  ms , with negative values indicating that the sound was presented first .
sounds of the test - stimulus pair either came from central ( a , b ) or lateral location ( c , d ) schematic illustration of the experimental conditions . in the exposure phase , the subject was exposed to a sound  light pair with 100  ms temporal offset ( either sound - first or light first ) . during exposure , sounds were either presented from central ( a , c ) or lateral location ( b , d ) . in the test phase ,
light pairs were presented with a particular soa ranging between 240 and 240  ms , with negative values indicating that the sound was presented first .
sounds of the test - stimulus pair either came from central ( a , b ) or lateral location ( c , d )
three within - subjects factors were used : exposure lag during the exposure phase ( 100 and + 100  ms , with negative values indicating that the sound was presented first ) , location of the sound during exposure ( exposure - sound central or lateral ) and soa between the sound and light of the test stimuli ( 240 , 120 , 90 , 60 , 30 , 0 , + 30 , + 60 , + 90 , + 120 , and + 240  ms , with negative values indicating that the sound came first ) .
the location of the test sound ( central or lateral ) was a between - subjects variable .
half of the participants were tested with central test sounds , the other with lateral test sounds .
these factors yielded 44 equi - probable conditions for each location of the test sound ( 2    2    11 ) , each presented 12  times for a total of 528 trials .
the location of the exposure sound were constant within a block , while the soa between sound and light varied randomly .
the order of the blocks was counterbalanced across participants . in half of the blocks with a lateral exposure sound ,
each block started with an exposure phase consisting of 240 repetitions ( 3  min ) of a sound  light stimulus pair ( isi  =  750  ms ) with a constant lag ( 100 or + 100  ms ) between the sound and the light .
after a 2,500  ms delay , the first test trial then started . to ensure that participants were fixating the light during exposure , they had to detect the occasional occurrence of the offset ( 150  ms ) of the fixation light ( i.e. , a catch trial ) .
the test phase consisted of two parts : a short av re - exposure phase followed by three av test trials of which the temporal order of the sound and light had to be judged .
the re - exposure phase consisted of a train of ten sound - light pairs with the same lag , isi , and sound location as used during the immediately preceding exposure phase .
after 1  s , the three av test trials were presented with a variable soa between the sounds and lights .
the participant s task was to judge whether the sound or the light of the test stimulus was presented first .
an unspeeded response was made by pressing one of two designated keys on a response box .
the next test stimulus was presented 500  ms after a response , and the re - exposure phase of the next trial started 1,000  ms after the response on the third test stimulus . to acquaint participants with the toj task ,
experimental blocks were preceded by four practice blocks in which no exposure preceded the test trials .
the first two practice blocks were to acquaint participants with the response buttons , and consisted of 16 trials in which only the largest soas were presented ( 240 and 120 ) . during this part
, participants received verbal feedback (  correct  or  wrong  ) about whether they gave the correct response or not .
the next two practice blocks consisted of 66 trials in which all soas were presented 6  times randomly without verbal feedback .
trials of the practice session were excluded from analyses . the proportion of  light - first 
responses was for each participant calculated for each combination of exposure lag ( 100 , + 100  ms ) , location of the exposure sound ( central , or lateral ) , location of the test sound ( central , or lateral ) and soa ( ranging from 240 to + 240  ms ) .
performance on catch trials was flawless , indicating that participants were indeed looking at the fixation light during exposure . for each combination of exposure lag , location of the exposure sound and location of the test sound , an individually determined psychometric function
was calculated over the soas by fitting a cumulative normal distribution using maximum likelihood estimation .
the mean of the resulting distribution ( the interpolated 50% crossover point ) is the point of subjective simultaneity ( henceforth the pss ) , and the slope is a measure of the sharpness with which stimuli are distinguished from one another . the slope is inversely related to the just noticeable difference ( jnd ) and represents the interval ( absolute soa ) at which 25 and 75% visual - first responses were given .
the pss and the jnd data are shown in fig .  2 and table  1 .
temporal recalibration was expected to manifest itself as a shift of the pss in the direction of the exposure lag .
the temporal recalibration effect ( tre ) was therefore computed by subtracting the pss following auditory - first exposure from visual - first exposure . 

2the proportions of visual - first responses ( v - first ) for each exposure lag ( 100  ms sound - first , 100  ms light - first ) for each combination of location of exposure sound ( central , lateral ) and location of the test sound ( central , lateral)table  1mean points of subjective simultaneity ( psss ) in ms , and mean just noticeable differences ( jnd ) in parentheseslocation test sound location of the exposure soundcentral lateralav - lag ( ms)pss ( jnd)trepss ( jnd)trecentral10012.5 ( 39.3)14.59.9 ( 37.8)6.41002.0 ( 40.8)3.5 ( 38.6)lateral1006.1 ( 38.2)14.214.3 ( 36.4)16.810020.3 ( 36.1)2.5 ( 42.0)exposure stimulus pairs were presented with an auditory  visual lag ( av - lag ) of 100 and + 100  ms with sounds either central or lateral ; the location of the test stimulus sound was either central or lateral .
the temporal recalibration effect ( tre ) reflects the difference in psss between the 100 and + 100  ms audio  visual lags the proportions of visual - first responses ( v - first ) for each exposure lag ( 100  ms sound - first , 100  ms light - first ) for each combination of location of exposure sound ( central , lateral ) and location of the test sound ( central , lateral ) mean points of subjective simultaneity ( psss ) in ms , and mean just noticeable differences ( jnd ) in parentheses exposure stimulus pairs were presented with an auditory  visual lag ( av - lag ) of 100 and + 100  ms with sounds either central or lateral ; the location of the test stimulus sound was either central or lateral . the temporal recalibration effect ( tre ) reflects the difference in psss between the 100 and + 100  ms audio  visual lags an overall 2    2    2 anova with
as within - subjects factors exposure lag , location of the exposure sound and as between - subjects factor location of the test sound was run on the jnds .
none of the effects was significant ( all p  >  0.08 ) , except for a second - order interaction between exposure lag , exposure location and test location , f(1,28 )  =  4.6 , p  =  0.041 .
inspection of table  1 shows that the differences between the jnds ( on average 38.7  ms ) were rather small and unsystematic .
the anova on the tres only showed a significant effect of exposure lag , f(1,28 )  =  23.0 , p  <  0.001 , demonstrating , as predicted , that the exposure phase shifted the pss such that there were more visual - first responses after sound - first exposure than after light - first exposure ( i.e. the tre ) .
the overall size of this effect corresponds well with previous reports ( fujisaki et al .
an average tre of 6.7% ) . there were , furthermore , no main effects of location of the exposure and test sound , and the crucial interaction between the location of the exposure and test sound was non - significant ( all f  <  1 ) .
temporal recalibration thus manifested itself no matter whether exposure sounds came from central or lateral location , and whether the location of the exposure and test sounds was changed or not .
the goal of the present study was to address whether spatially co - located av asynchronous stimulus pairs induce temporal recalibration as much as spatially dislocated stimuli do , and whether spatial correspondence between the exposure and test sound affects the size of this effect .
results showed that in all cases , there were clear temporal recalibration effects as subjective simultaneity was shifted in the direction of the adapted audiovisual lag .
the shift was equally big for spatially separated and spatially co - located exposure stimuli .
stimulus generalization across space was also complete , as the shift in temporal alignment was equally big for when the exposure and test sound came from the same or different positions .
the results therefore support the notion that spatial alignment between the senses is unimportant for av pairing in the temporal domain .
the results are also in line with previous reports on temporal ventriloquism ( keetels et al . 2007 ; vroomen and keetels 2006 ) where it was shown that spatial separation does not affect the capturing effect of a light by a sound .
taken together , these findings provide strong evidence that spatial co - occurrence is , even at early perceptual stages , not a necessary constraint for intersensory pairing .
one might object , though , that spatial ventriloquism has diminished the potential effects of spatial discordance .
it is well - known that the apparent location of a sound can be shifted towards a visual stimulus that is presented at approximately the same time ( howard and templeton 1966 ; radeau and bertelson 1978 ; welch 1978 ; bertelson and radeau 1981 ; bertelson 1994 , 1999 ; radeau 1994 ) .
could it be , then , that the av spatial discordance in our set - up was diminished , if not became unnoticeable due to spatial ventriloquism ?
if so , one may not observe an effect of spatial separation on temporal recalibration .
this argument , though , seems highly unlikely because it is known that spatial ventriloquism dramatically declines whenever spatial separation exceeds approximately 15  degrees ( slutsky and recanzone 2001 ; godfroy et al .
given that we maximized the spatial separation between the sound and light ( i.e. , at 90  degrees azimuth ) , and that informal testing indeed confirmed that spatial separation was clearly noticeable , it seems safe to assume that spatial ventriloquism did not diminish the effect of spatial discordance .
one might also ask whether the visual task as used during the exposure phase ( i.e. , detection of the offset of visual fixation ) resulted in an attentional shift towards the visual modality . according to the  law of prior entry  ( titchener 1908 ) , attending to one sensory modality speeds up the perception of stimuli in that modality , resulting in a change in the pss ( see also shore et al . 2001 , 2005 ; spence et al . 2001 ; schneider and bavelier 2003 ; zampini et al .
our visual task might thus result in a shift of the pss towards more  visual - first 
responses . however , this shift should be uniform for all conditions , and given that temporal recalibration is expressed as a difference in the pss between exposure lags , the possible role of attention will be subtracted out .
a remarkable aspect of the data is that previous studies have demonstrated that av temporal order judgements become more sensitive ( i.e. smaller jnd ) when the sound and light of the test stimuli are spatially separated ( see also bertelson and aschersleben 2003 ; spence et al .
, there was a small trend in this direction ( average jnd of 39.1 vs. 38.2  ms , for spatially co - located vs. separated test stimuli , respectively ) , but the effect was non - significant .
possibly , we might have picked up this difference if the effect were measured as a within - subjects factor . for the current purpose ,
though , this was considered to be unpractical because it would have doubled individual testing time . despite
that we did not observe an effect of av spatial separation on the jnds , the data speak on the interpretation on this effect .
at least two explanations have been brought up for the improved temporal sensitivity when the locations of test sound and light differ .
one is that there is more intersensory integration with as a consequence that the temporal discordance is fused ; the other is that there are extra spatial cues that help toj performance ( spence et al . 2003 ) . given that our results show that intersensory pairing occurs independent of a spatial mismatch ( see also vroomen and keetels 2006 ; keetels et al .
2007 ) , it seems more likely that the previously observed effects of spatial separation on temporal sensitivity were induced by the availability of redundant spatial cues rather than fusion per se . to conclude ,
our results provide strong evidence for the claim that commonality in space between a sound and light is not relevant for av pairing in the temporal domain .
this may , at first sight , seem unlikely , because after all , most natural multisensory events are spatially and temporally aligned .
however , a critical assumption that underlies the idea of spatial correspondence for cross - modal pairing is that space has the same function in vision and audition .
this notion , though is arguable , as it has been proposed that the role of space in hearing is to steer vision ( heffner and heffner 1992 ) , while in vision it is an indispensable attribute ( kubovy and van valkenburg 2001 ) . if one accepts that auditory spatial perception evolved for steering vision , but not for deciding whether sound and light belong together , there is no reason why cross - modal interactions would require spatial co - localization .
our results therefore have also important implications for designing multimodal devices or creating virtual reality environments , as they show that the brain can ignore cross - modal discordance in space .
 it is known that the brain adaptively recalibrates itself to small ( 100  ms ) auditory  visual ( av ) temporal asynchronies so as to maintain intersensory temporal coherence . 
 here we explored whether spatial disparity between a sound and light affects av temporal recalibration . 
 participants were exposed to a train of asynchronous av stimulus pairs ( sound - first or light - first ) with sounds and lights emanating from either the same or a different location . following a short exposure phase , 
 participants were tested on an av temporal order judgement ( toj ) task . 
 temporal recalibration manifested itself as a shift of subjective simultaneity in the direction of the adapted audiovisual lag . 
 the shift was equally big when exposure and test stimuli were presented from the same or different locations . 
 these results provide strong evidence for the idea that spatial co - localisation is not a necessary constraint for intersensory pairing to occur . 
Introduction
Method
Participants
Stimuli
Design
Procedure
Results
Discussion