Facial Expression

The Complexity of the Human Face

With 52 distinct muscles, humans have one of the most complex and well-developed systems of facial muscles in the animal kingdom. The human face conveys a wide range of information about people, including identity, age, gender, visual and social attention, ethnicity, attractiveness, pathology, emotional intentions and state, linguistic and musical information, as well as indications of personality, traits, and attitudes. While many of these qualities likely affect the relationship between performers and listeners, only some have been examined in the context of music. Of these areas, emotional expression has received the most attention.

Emotional Expression

A primary goal of music is the expression and experience of emotion. Musicians use a variety of techniques to convey and elicit emotion in listeners. These techniques include aspects of the sounded performance, as well as visual elements that encompass facial expressions, eye gaze, and body gestures. The performance of drummer Brian Viglione of the Dresden Dolls illustrates some of these visual elements; he produces an intense facial expression while performing. These visual displays can occur whether a performer is vocalizing, clapping, playing an instrument, or waiting for their moment to play again. In contemporary music, visual displays have become an important element in the concert experience of many musicians.

In 1993, one of the first accounts of the role of visual information on perception in music performance had violinists perform several musical [Page 448]works, each with different expressive intentions. Observers reliably distinguished the expressive intent of the performers from audio-only, video-only, and audio-video recordings. While the author intentionally occluded the performer's facial expressions, the study highlighted that visual information could play an important role in musical performance.

Later research in 2004 examined the influence of performers' facial expressions on listeners' emotional experience. In this work, still images were taken of singers performing individual vowels with different emotional intentions. Performers' intended emotions were identified more accurately from their facial expressions than from their sounded performance. Observers were most accurate at identifying emotion from combined face and voice displays. This last finding highlighted the audiovisual nature of music performance. Extending this, another study in 2005 revealed how audio and visual information were integrated during the perception of music. Video images of vocalists singing major and minor thirds, thought to express happiness and sadness, respectively, were paired with audio channels in emotionally congruent and incongruent combinations. While observers were told to focus only on the acoustic channel, their ratings of emotion were strongly affected by the presence of facial expressions and the congruency between the two modalities. Subsequent work in 2008 showed that this integration occurred preattentively, below an observer's conscious attention. These findings confirm the integrated and complex nature of audiovisual music perception and point to a deeper psychological basis for the enjoyment of music as an audiovisual medium.

...