Multimodal Interactions: Neural Basis

Our sensory world is made up of stimuli in the form of various types of environmental energy. To deal with the vast array of information we are continually bombarded with, specialized sensory systems have evolved, each of which is tuned to a different form of energy (e.g., light, chemical, mechanical). Although each of our sensory systems provides us with a unique perspective on the world, it is only through the synthesis of information between the different senses that we can ultimately form a coherent perceptual unity. Various examples serve to illustrate both the ubiquitous nature of such multisensory interactions as well as the compelling way in which they shape our behaviors and perceptions. Thus, the ventriloquist's act highlights the ability of visual cues (i.e., movements of the dummy's mouth and head) to effectively shift the perceived location of a sound source (i.e., the voice of the ventriloquist). For those who haven't seen the vaudeville act, a similar multisensory “illusion” is evident (or actually, not so evident) whenever we go to a movie theater, where the auditory sources are often far away from the screen. Thus, the actors' voices are typically coming from speakers located far from any actor's position on the screen. Despite the significant spatial mismatch between the visual and auditory cues, we have no trouble in ascribing the appropriate voice to a given char [Page 586]acter on the screen. One of the most common multisensory perceptual effects happens every time we have a meal, and in which gustatory, olfactory, tactile, and visual information from the food and drink is combined into a wonderful multisensory mélange. In a simpler context, multisensory interactions are also evident when examining behavioral responses, such as simple reaction times, in which participants are asked to press a button as soon as they detect a light or a sound. When the light and sound are presented simultaneously, responses are significantly faster than they are for either of the stimuli when presented alone. Such a speeding underscores the adaptive significance of multisensory processes, and points to the strong evolutionary pressures that likely shaped our ability to make use of redundant (and nonredundant) information from multiple senses. In order for such multisensory interactions to take place, information from the different sensory systems must come together somewhere within the brain. Indeed, the convergence of inputs from multiple senses takes place at various sites within the central nervous system, ranging from the brain stem to the complex neocortical domains that make us uniquely human. Within these various multisensory brain regions, many individual neurons, the building blocks of the nervous system, receive input from two (or more) sensory systems. Rather than simply serving as passive filters to relay these inputs to the next processing stages, these multisensory neurons actively transform their different sensory inputs, often in ways that give rise to outputs that differ markedly from expectations. This entry will discuss the superior colliculus as a model, multisensory integration, the principles of multisensory integration, multisensory integration from neurons to networks, and beyond the traditional multisensory designations.

...