Intersensory Perception

Nature of Information

Most objects and events are multimodal as they can be seen, heard, touched, and so on, and the question becomes how do we arrive at a unified perception of the world? There are two broad categories of information available for perception. One [Page 1205]category of information is what is termed amodal. Amodal information refers to that information, or property, that can be perceived in more than one sensory system or modality. For example, the texture of an object can be seen and felt. Affective or emotional information can be conveyed in a person’s auditory speech and in the person’s visual facial expression. Properties such as the tempo, rate, or intensity of an object impacting a surface can be perceived in both the visual and auditory sensory systems. Thus, amodal information is information that is not specific or bound to one sensory system. In contrast, modality-specific information is that information that is bound to a specific property. For example, the color of an object or the pitch an object makes when impacting a surface are tied, or bound, to the visual or auditory system, respectively.

Intersensory Perception: Human Infants

Human infants, from near birth, are sensitive to a variety of intersensory properties. For instance, 3-week-olds are sensitive to the temporal synchrony between a bouncing object and the timing or synchrony of its impact sounds. By 6 weeks of age, infants can match the composition of an object in terms of its visual appearance (e.g., a single large metal nut or a cluster of smaller metal nuts) and the appropriate corresponding impact sounds (e.g., a thud or a crashing sound). By 2 months of age, infants match one of two faces with the appropriate voice on the basis of temporal synchrony. Two-month-olds also match visible speech with auditory speech. Specifically, they detect which of two women is articulating an /i/ sound and which is articulating an /a/ speech sound by matching the movement of the mouth with the speech sound being heard. By 7 months of age, infants match a happy or sad vocalized affective expression with an appropriate moving or static happy or sad facial expression. Seven-month-olds also match affective vocal and facial expressions even when the facial expressions consist only of moving points of light. Five- to 7-month-olds also perceive properties such as the age, gender, and identity of the speaker and can use the intersensory relationship in the perception of self and other. In each of the aforementioned examples, intersensory perception is achieved by perceiving what is common across the visual and auditory domains and, in the case of perceiving the self, the proprioceptive and the visual domains.

...