Avatar Technology

Background

Including captioning or other text-based options in movies, television broadcasts, or web-based media does not provide satisfactory access for people whose preferred language is a signed language. Signed languages are independent natural languages in their own right, and they have structures that do not correspond to those in spoken language. Signed language has multiple articulators that can co-occur, which means that a signer’s hands, face, and spine can all be simultaneously contributing to the meaning of the sentence being signed. For these reasons, there is no widely accepted written form of signed languages, and there is little correspondence between signed language and written forms of spoken language. For example, British Sign Language is a language different from English, and American Sign Language (ASL) is also different from English.

For people in the United States who use ASL as their preferred language, English is a second language. Although there are some members of the Deaf community who are bilingual in ASL and English, many are not. Thus captions written in English, which can be useful for late-deafened adults, are of limited use to people whose primary language is a signed language.

Differences Between Avatars and Movie Characters

Since 1995, animated movies such as Toy Story and Avatar and video games such as The Sims and Grand Theft Auto have demonstrated that computer animation technology has advanced to the point where it is possible to create three-dimensional graphics that portray humanlike characters convincingly. The technology for building a sign language avatar is similar to the technology used for movies and video games, but it needs to provide more precision and realism. Video game or movie characters are perfectly acceptable to viewers even when they have only four fingers, but sign language avatars need all five fingers to properly produce hand shapes when signing. Avatars also need facial expressivity for conveying nonmanual signals, which are an essential part of signed language. For example, it is not possible to pose a question without nonmanual signals. Figure 1 shows an avatar asking a yes-no question about a cup of coffee. The hands are in the process of producing the sign for coffee, and the raised eyebrows and inclined head indicate that the utterance is a yes-no question. The avatar has a set of simulated facial muscles that allow the face to assume poses that correspond to facial nonmanual signals.

Automatic Translation

Researchers are currently developing systems to translate spoken languages to signed languages. In these systems, a person can speak into a microphone connected to a computer. The computer recognizes the speech and directs an avatar to produce the corresponding signed language. Several prototype systems have been created, including one for transactions at post offices in the United Kingdom and another for airport security in the United States.

...