Multimodality

Through its prominent use in dance, opera, film, and theater, music is widely recognized as an integral aspect of multimodal artistic experiences. Yet, music by itself is typically regarded as an acoustic endeavor, one that is captured by recordings focused on the sound alone (i.e., radio broadcasts, CDs, and MP3s). Consequently, musical analysis typically focuses on sound or visual representations of sound—such as a notated score. Nonetheless, there is now a growing realization that the musical experience is not fully explained by sound alone. Extra-acoustic information, such as seeing a performers' gestures (both sound-producing and otherwise), observing musicians' facial expressions, and even feeling one's body movement, can affect the experience of music listening.

It is tempting to regard such influences as extraneous, a musical equivalent of the well-known ventriloquist illusion. Watching clever performers trick audiences into “hearing” puppets speak is amusing as a parlor trick, but would not generally be regarded as serious art. The unconscious use of extra-acoustic information stems from the deeply interconnected way in which the human brain processes information. The multimodal nature of music is “hard-wired” into one's listening experience. Because auditory areas of the brain are affected by what one sees, visual information plays an unavoidable role in hearing. Listening to music can even activate motor regions of the brain when stationary, demonstrating that music and movement are intertwined on a neural level. Although formal study of these extra-acoustic factors is relatively recent, they have been affecting the perception of music ever since humans first began performing and listening.

In fact, prior to the advent of recording technology, music was nearly always experienced with concurrent visual input, often in group settings with rich social dynamics. The first audio-alone representations of music in the phonograph's early days were often equivocally greeted. Therefore, “new” evidence showing visual and motor influence on music listening merely rediscovers well-known truisms. Schuman once remarked that a well-known pianist of his day, “must be heard and seen; for if Liszt played behind a screen, a great deal of poetry would be lost.”

[Page 721]

Visual Influences on Music Perception and Cognition

Visual influences on music listening can be divided into two groups: those affecting cognitive evaluations of music, and those affecting perception of the acoustic signal. Although differentiating between the two is in some cases difficult, the distinction is helpful both in understanding the degree to which the cross-modal influence is obligatory, and in guiding efforts to connect this research with broader inquiries of audiovisual integration within the psychological sciences.

The fact that vision affects a broad range of musical evaluations is recognized by normative standards for concert dress and stage performance. Many orchestras employ “blind auditions” in which committees sit behind a screen to avoid extra-acoustic influences on their evaluations. Studies of visual influences use a wide range of approaches, including comparing differences in judgments of a performance's expressivity, quality, or musicality when using either restrained or elaborate body movements. Participants then rate these performances under audiovisual and audio-alone conditions, allowing for direct exploration of vision's influence.

Figure 1Levels of visual influence. Visual information can affect the musical experience by either (a) affecting one's perception of acoustic information or (b) altering one's cognitive evaluations of this information. The distinction between these two types of influence is helpful in placing these effects within a larger framework of audio-visual integration—an issue of relevance and interest to a wide community of researchers within the psychological sciences.

...