Hearing lips and (not) seeing voices: Audiovisual integration with the suppressed percept Manuel Vidal
1,2
and Victor Barrès
1,3
Laboratoire de Physiologie de la Perception et de l’Action, Collège de France, Paris 2 Institut de Neurosciences de la Timone, Marseille, France 3 Neuroscience department, University of Southern California, USA
[email protected]
1
Introduction Munhall and colleagues2 designed a dynamic version of the classical face/vase illusion in which the vase contour changed in time to make the face talk.
The suppressed percept had no influence on sound: when the vase interpretation was dominant, the moving lips of the unseen faces could not produce the McGurk effect on the heard voice. We also used speech stimuli for it involves robust audiovisual interactions at several cortical levels, however we designed competing animated lips with perceptual suppression based on binocular rivalry. Both
Results
Right eye
=
/aba/
/aba/
Left eye
Right eye
=
/aga/
or
/aga/
O
1.85s
+RT /aba/
/aba/ phoneme
/ /ada/
· Task: single audiovisual perceptual report (forced choice)
Selected subjects
90%
90%
80%
80%
70%
70%
60%
60%
50%
50%
40%
40%
30%
30%
20%
20%
10%
10%
0%
0% 7
12 14
3
8 10 Subject
1
13
2
6
9
100%
11
/aba/
Left eye
Wait for
Right eye
Left eye /aba/
Right eye
/ Instructions
O
Esc
+tS
+1.85s
Space
/aba/
Target viseme stabilization McGurk
+RT
/
/ada/
No McGurk
Group
8 subjects report /ada/ when seeing /aga/, while the other 6 disregard the lips motion and always report /aba/
In the rivalry conditions, when the viseme was seen with the dominant eye this proportion is not different from the baseline in which there is no ambiguity. In contrast, when the viseme was seen in the non-dominant eye this proportion seems to differ for both conditions, leading to:
/ada/
80% 70% 60% 50% 40% 30% 20% 10%
/aba/
/aga/
/aba/
/aga/
/aba/
/aga/
0% in both eyes
in dominant eye
· Subjects were instructed to signal if a perceptual alternation occurred before the end of the animation, the trial was then restarted
When the unseen stimulus is displayed in the dominant eye, it can sometimes integrate with the auditory input either to produce or cancel the McGurk effect that would be expected with the seen stimulus. This finding indicates that the suppressed stimulus is not fully deactivated and can be processed, before rivalry is resolved, to produce a conscious audiovisual percept.
in non-dominant eye
Viseme seen
~16% of integration toward McGurk (p=0.10) ~24% of integration toward congruent (p=0.11)
/aga/
{ {
McGurk effect
1120ms 1280ms 1440ms 1600ms
AV speech integration in binocular rivalry
100%
5
960ms
Main protocol
Resuts
4
800ms
90%
/ada/
Left eye
640ms
/aga/ viseme
Auditory
/aba/
480ms
100%
/aba/
320ms
{ {
(auditory reports)
Right eye
McGurk effect
160ms
AV integration with suppressed percept
/ada/
0ms
Regular AV integration
/aba/
/aba/ viseme
Baseline without rivalry
/ada/
Audiovisual stimuli
Proportion of trials in which the percept corresponds to the expected audiovisual integration (dark gray) given the conscious viseme seen either binocularly (baseline) or during rivalry (dominant and non-dominant eye).
AV fusion distribution
· Baseline measurement · Effect tested: A/aba/ + V/aga/ → AV/ada/ · 14 naïve subjects
Left eye
Main experiment
/aba/
Materials and methods McGurk sensitivity experiment
interpretations remaining in the speech perception domain, this could increase chances of observing integration with the suppressed percept. The perceived phoneme when presenting a synchronous voice saying /aba/ together with rivaling faces saying /aba/ and /aga/ was recorded for 8 McGurk sensitive participants.
/aba/
In binocular rivalry, sensory input remains the same yet subjective experience fluctuates irremediably between two mutually exclusive representations. We investigated whether a suppressed viseme can still produce the McGurk effect1 with an auditory input, showing that unconscious lips motion can sometimes modulate the perceived phoneme through a well-known multisensory integration mechanism.
This observation was made possible using natural stimuli of talking faces presented in binocular rivalry instead of the figure/ground illusion as described in2.
Discussion
Feature binding & awareness Is conscious emergence possible without binding? In other words, is it possible to be aware of two features of rivaling items that are not coming from the same item (i.e. independent feature competition)? Our results indicate that some features of the audiovisual items are not necessarily bound together before conscious emergence, which suggests that these two mechanisms are to some extent independent.
· The observer would be conscious of the lips color from viseme A and the lips motion from viseme B, the latter being integrated with the auditory input (feature binding between but not within modality) · The observer would be conscious of lips color and motion from viseme A, but the suppressed motion from B would be integrated with the auditory input (feature binding within but not between modality)
Conscious Visual percept
Dominant
0.8
0.6
(lips color, motion, viseme...)
(physical sound, phoneme...)
0.4
1.0
0.8
~80%
0.6
0.4
0.2
0.2
0.0
0.0
Sampling
~20%
Follow-up studies (undergoing)
Left eye vs.
/aga/
Left eye
/aba/
Right eye vs.
/aga/
Seen
Seen
?
Heard /aba/ /ada/ /abga/ /aga/
Sampling
STS
Viseme
Color
L ...
1
/aba/
· Rivalry is distributed along visual pathway5 · Outcome resulting from a probabilistic draw on the possible distribution6
Associative areas
Phoneme 2
R /aga/
Visual stream
Design integrating:
/aba/
/aga/
?
Heard /aba/ /ada/ /abga/ /aga/
/aba/
/aga/
Ears ...
/aba/
/ada/
Auditory stream
...
CFS vs. viseme Is it possible to find AV integration with a suppressed visual input, when the rival stimulus does not provide an alternative input to integrate with? Viseme vs. viseme Replicate the initial experiment with a stronger control for correct suppression and perceptual alternations report during the video sequence.
References 1. McGurk & MacDonald (1976). Hearing lips and seeing voices. Nature 264, 746–748 2. Munhall, ten Hove, Brammer & Paré (2009). Audiovisual Integration of Speech in a Bistable Illusion. Curr. Biol. 19, 735–739 3. van Wassenhove, Grant & Poeppel (2005). Visual speech speeds up the neural processing of auditory speech. Proc. Natl. Acad. Sci. U.S.A. 102, 1181–1186
Right eye
3
Eyes
Modeling the AV interactions
Conscious auditory percept
1.0
competitio n
The suppressed visual input interacts with the auditory input at an early processing stage, supporting the idea of automatic (preconscious) audiovisual integration for the construction of high-level speech perception (early models of integration as in3,4).
· Audiovisual integration for speech3,4 · Neurophysiological plausibility: role of STS7,8
Distributed
Speech perception & AV integration
Two scenarios according to the locus where binding is accidentally disrupted:
4. Arnal, Morillon, Kell & Giraud (2009). Dual neural routing of visual facilitation in speech processing. J. Neurosci. 29, 13445–13453 5. Tong, Meng & Blake (2006). Neural bases of binocular rivalry. Trends Cogn Sci. 10, 502–511 6. Moreno-Bote, Knill & Pouget (2011). Bayesian sampling in visual perception. Proc. Natl. Acad. Sci. U.S.A. 108, 12491–12496 7. Beauchamp, Argall, Bodurka, Duyn & Martin (2004). Unraveling
multisensory integration: patchy organization within human STS multisensory cortex. Nat.Neurosci. 7, 1190 –1192 8. Beauchamp, Nath & Pasalar (2010). fMRI-guided TMS reveals that the STS is a Cortical Locus of the McGurk Effect. J Neurosci 30, 2414–2417
