audiovisual speech perception definition

These results are more consistent with theories of speech perception in which contextual information across modalities is used to parse incoming speech, than either unisensory ( 15) or motor ( 16, 17) theories. There is a large amount of evidence for a ubiquitous role for sensorimotor systems in auditory and audiovisual speech perception ( 13). One possibility is that visual speech cues are mapped onto an internal representation of the speech articulators before being combined with compatible representations derived from auditory information.

Here, we address the questions of whether and how visual speech signals might provide information that is compatible with acoustic speech signals. While the neural loci of audiovisual speech integration have been studied extensively ( 6– 11), the exact nature of the visual representation utilized in audiovisual speech integration remains unclear ( 12). For example, integration has been framed as a temporally focused lexical competition, in which visual information is used to constrain the corpus of words indicated by the acoustic signal ( 5). Audiovisual speech integration is predominantly discussed in the context of comparing high-level acoustic and visual representations. These visual cues become increasingly important when the acoustic signal is degraded, for example, in individuals with hearing problems ( 1, 2), or when the signal needs to be extracted from noise ( 3) or mixed competing speech sources, as may occur at a cocktail party ( 4).

While speech is predominantly an acoustic signal, visual cues can provide valuable information about the timing and content of the signal. In addition, the facial and vocal tract regions that are important for reconstruction are those that are used to generate the acoustic speech signal. Our data reveal that there is sufficient information in the face to recover vocal tract shape during speech. A “Bubbles” technique was used to identify which areas of the face were important for recovering information about the vocal tract, and vice versa, on a frame-by-frame basis. Reconstruction fidelity was significantly higher when images from the two sequences corresponded in time, and including implicit temporal information by combining contiguous frames also led to a significant increase in fidelity. The joint variation in the face and vocal tract was extracted using an application of principal components analysis (PCA), and we demonstrate that MR image sequences can be reconstructed with high fidelity using only the facial video and PCA. Here, we investigate whether facial and vocal tract movements are linked during speech production by comparing videos of the face and fast magnetic resonance (MR) image sequences of the vocal tract. One suggestion is that integration can occur as both visual and auditory cues arise from a common generator: the vocal tract. While we have a good understanding of where speech integration occurs in the brain, it is unclear how visual and auditory cues are combined to improve speech perception. It is well established that speech perception is improved when we are able to see the speaker talking along with hearing their voice, especially when the speech is noisy.

Audiovisual speech perception definition