Perceptual-Motor Integration

The Fusion Problem

This problem of combining information from different sources is referred to as “sensor fusion” in computer science and robotics and as “cue combination” in psychology. First of all, we can approach it by asking what benefit is there in combining cues from different sensory modalities? The theoretical justification for doing so is statistical: Any sensory estimate is inherently uncertain and, by combining multiple estimates from different sensory modalities, the resulting composite estimate can be more reliable (less variable) than any of the individual estimates that go into it. Michael Landy, Laurence Maloney, and colleagues summarize the theory and pointed out that, because sensory estimates from any modality can vary in reliability, the rule for combining them to get the best composite estimate should also change. If we are judging the shape of an object through vision and active touch in a dimly lit room, the nervous system should give more weight to the haptic information. If the same judgment is carried out under bright illumination, we should give more weight to visual information.

Marc Ernst and Marty Banks carried out an elegant experiment to test the claim that the perceptual system integrates visual information and haptic information so as to produce a composite estimate as reliable as possible. They asked a subject to judge the size of a small object by viewing and also grasping it between his index finger and thumb. However, the experiment was carried out in a virtual environment. The subject in the experiment “saw” the object presented binocularly and, when he moved to grasp it, he “felt” the object as well. However, the pressure that he felt in grasping was exerted by robotic arms attached to his index finger and thumb. The visual impression of the objects was equally unreal, a product of computer graphics.

...