multisensory integration multisensory integration multisensory integration, also known as multimodal integration, is the study of how information from the different sensory modalities (such as sight, sound, touch, smell, self-motion, and taste) may be integrated by the nervous system. a coherent representation of objects combining modalities enables animals to have meaningful perceptual experiences. indeed, multisensory integration is central to adaptive behavior because it allows animals to perceive a world of coherent perceptual entities. multisensory integration also deals with how different sensory modalities interact with one another and alter each other's processing. general introduction multimodal perception is how animals form coherent, valid, and robust perception by processing sensory stimuli from various modalities. surrounded by multiple objects and receiving multiple sensory stimulations, the brain is faced with the decision of how to categorize the stimuli resulting from different objects or events in the physical world. the nervous system is thus responsible for whether to integrate or segregate certain groups of signals. multimodal perception has been widely studied in cognitive science, behavioral science, and neuroscience. stimuli and sensory modalities there are four attributes of stimulus: modality, intensity, location, and duration. the neocortex in the mammalian brain has parcellations that primarily process sensory input from one modality. for example, primary visual area, v1, or primary somatosensory area, s1. these areas mostly deal with low-level stimulus features such as brightness, orientation, intensity, etc. these areas have extensive connections to each other as well as to higher association areas that further process the stimuli and are believed to integrate sensory input from various modalities. however, recently multisensory effects have been shown to occur in primary sensory areas as well. binding problem the relationship between the binding problem and multisensory perception can be thought of as a question – the binding problem, and potential solution – multisensory perception. the binding problem stemmed from unanswered questions about how mammals (particularly higher primates) generate a unified, coherent perception of their surroundings from the cacophony of electromagnetic waves, chemical interactions, and pressure fluctuations that forms the physical basis of the world around us. it was investigated initially in the visual domain (colour, motion, depth, and form), then in the auditory domain, and recently in the multisensory areas. it can be said therefore, that the binding problem is central to multisensory perception. however, considerations of how unified conscious representations are formed are not the full focus of multisensory integration research. it is obviously important for the senses to interact in order to maximize how efficiently people interact with the environment. for perceptual experience and behavior to benefit from the simultaneous stimulation of multiple sensory modalities, integration of the information from these modalities is necessary. some of the mechanisms mediating this phenomenon and its subsequent effects on cognitive and behavioural processes will be examined hereafter. perception is often defined as one's conscious experience, and thereby combines inputs from all relevant senses and prior knowledge. perception is also defined and studied in terms of feature extraction, which is several hundred milliseconds away from conscious experience. notwithstanding the existence of gestalt psychology schools that advocate a holistic approach to the operation of the brain, the physiological processes underlying the formation of percepts and conscious experience have been vastly understudied. nevertheless, burgeoning neuroscience research continues to enrich our understanding of the many details of the brain, including neural structures implicated in multisensory integration such as the superior colliculus (sc) and various cortical structures such as the superior temporal gyrus (gt) and visual and auditory association areas. although the structure and function of the sc are well known, the cortex and the relationship between its constituent parts are presently the subject of much investigation. concurrently, the recent impetus on integration has enabled investigation into perceptual phenomena such as the ventriloquism effect, rapid localization of stimuli and the mcgurk effect; culminating in a more thorough understanding of the human brain and its functions. history studies of sensory processing in humans and other animals has traditionally been performed one sense at a time, and to the present day, numerous academic societies and journals are largely restricted to considering sensory modalities separately ('vision research', 'hearing research' etc.). however, there is also a long and parallel history of multisensory research. an example is the stratton's (1896) experiments on the somatosensory effects of wearing vision-distorting prism glasses. multisensory interactions or crossmodal effects in which the perception of a stimulus is influenced by the presence of another type of stimulus are referred since very early in the past. they were reviewed by hartmann in a fundamental book where, among several references to different types of multisensory interactions, reference is made to the work of urbantschitsch in 1888 who reported on the improvement of visual acuity by auditive stimuli in subjects with damaged brains. this effect was also found later in individuals with undamaged brains by krakov and hartmann, as well as the fact that the visual acuity could be improved by other type of stimuli. it is also noteworthy the amount of work in the early 1930s on intersensory relations in the soviet union, reviewed by london. a remarkable multisensory research is the extensive work of gonzalo in the forties on the characterization of a multisensory syndrome in patients with parieto-occipital cortical lesions. in this syndrome, all the sensory functions are affected, and with symmetric bilaterality, in spite of being a unilateral lesion where the primary areas were not involved. a feature of this syndrome is the great permeability to crossmodal effects between visual, tactile, auditive stimuli as well as muscular effort to improve the perception, also decreasing the reaction times. the improvement by crossmodal effect was found to be greater as the primary stimulus to be perceived was weaker, and as the cortical lesion was greater (vol i and ii of reference). this author interpreted these phenomena under a dynamic physiological concept, and from a model based on functional gradients through the cortex and scaling laws of dynamical systems, thus highlighting the functional unity of the cortex. according to the functional cortical gradients, the specificity of the cortex would be distributed in gradation, and the overlap of different specific gradients would be related to multisensory interactions. multisensory research has recently gained enormous interest and popularity. example of spatial and structural congruence when we hear a car honk, we would determine which car triggers the honk by which car we see is the spatially closest to the honk. it's a spatially congruent example by combining visual and auditory stimuli. on the other hand, the sound and the pictures of a tv program would be integrated as structurally congruent by combining visual and auditory stimuli. however, if the sound and the pictures did not meaningfully fit, we would segregate the two stimuli. therefore, spatial or structural congruence comes from not only combining the stimuli but is also determined by our understanding. theories and approaches visual dominance literature on spatial crossmodal biases suggests that visual modality often influences information from other senses. some research indicates that vision dominates what we hear, when varying the degree of spatial congruency. this is known as the ventriloquist effect. in cases of visual and haptic integration, children younger than 8 years of age show visual dominance when required to identify object orientation. however, haptic dominance occurs when the factor to identify is object size. modality appropriateness according to welch and warren (1980), the modality appropriateness hypothesis states that the influence of perception in each modality in multisensory integration depends on that modality's appropriateness for the given task. thus, vision has a greater influence on integrated localization than hearing, and hearing and touch have a greater bearing on timing estimates than vision. more recent studies refine this early qualitative account of multisensory integration. alais and burr (2004), found that following progressive degradation in the quality of a visual stimulus, participants' perception of spatial location was determined progressively more by a simultaneous auditory cue. however, they also progressively changed the temporal uncertainty of the auditory cue; eventually concluding that it is the uncertainty of individual modalities that determine to what extent information from each modality is considered when forming a percept. this conclusion is similar in some respects to the 'inverse effectiveness rule'. the extent to which multisensory integration occurs may vary according to the ambiguity of the relevant stimuli. in support of this notion, a recent study shows that weak senses such as olfaction can even modulate the perception of visual information as long as the reliability of visual signals is adequately compromised. bayesian integration the theory of bayesian integration is based on the fact that the brain must deal with a number of inputs, which vary in reliability. in dealing with these inputs, it must construct a coherent representation of the world that corresponds to reality. the bayesian integration view is that the brain uses a form of bayesian inference. this view has been backed up by computational modeling of such a bayesian inference from signals to coherent representation, which shows similar characteristics to integration in the brain. cue combination vs. causal inference models with the assumption of independence between various sources, the traditional cue combination model is successful in modality integration. however, depending on the discrepancies between modalities, there might be different forms of stimuli fusion: integration, partial integration, and segregation. to fully understand the other two types, we have to use causal inference model without the assumption as cue combination model. this freedom gives us general combination of any numbers of signals and modalities by using bayes' rule to make causal inference of sensory signals. the hierarchical vs. non-hierarchical models the difference between two models is that hierarchical model can explicitly make causal inference to predict certain stimulus while non-hierarchical model can only predict joint probability of stimuli. however, hierarchical model is actually a special case of non-hierarchical model by setting joint prior as a weighted average of the prior to common and independent causes, each weighted by their prior probability. based on the correspondence of these two models, we can also say that hierarchical is a mixture modal of non-hierarchical model. independence of likelihoods and priors for bayesian model, the prior and likelihood generally represent the statistics of the environment and the sensory representations. the independence of priors and likelihoods is not assured since the prior may vary with likelihood only by the representations. however, the independence has been proved by shams with series of parameter control in multi sensory perception experiment. principles the contributions of barry stein, alex meredith, and their colleagues (e.g."the merging of the senses" 1993,) are widely considered to be the groundbreaking work in the modern field of multisensory integration. through detailed long-term study of the neurophysiology of the superior colliculus, they distilled three general principles by which multisensory integration may best be described. the spatial rule states that multisensory integration is more likely or stronger when the constituent unisensory stimuli arise from approximately the same location. the temporal rule states that multisensory integration is more likely or stronger when the constituent unisensory stimuli arise at approximately the same time. the principle of inverse effectiveness states that multisensory integration is more likely or stronger when the constituent unisensory stimuli evoke relatively weak responses when presented in isolation. perceptual and behavioral consequences a unimodal approach dominated scientific literature until the beginning of this century. although this enabled rapid progression of neural mapping, and an improved understanding of neural structures, the investigation of perception remained relatively stagnant, with a few exceptions. the recent revitalized enthusiasm into perceptual research is indicative of a substantial shift away from reductionism and toward gestalt methodologies. gestalt theory, dominant in the late 19th and early 20th centuries espoused two general principles: the 'principle of totality' in which conscious experience must be considered globally, and the 'principle of psychophysical isomorphism' which states that perceptual phenomena are correlated with cerebral activity. just these ideas were already applied by justo gonzalo in his work of brain dynamics, where a sensory-cerebral correspondence is considered in the formulation of the "development of the sensory field due to a psychophysical isomorphism" (pag. 23 of the english translation of ref.). both ideas 'principle of totality' and 'psychophysical isomorphism' are particularly relevant in the current climate and have driven researchers to investigate the behavioural benefits of multisensory integration. decreasing sensory uncertainty it has been widely