Speech Perception NACS April 2009

Transcription

1 Speech Perception NACS April 2009

2

3

4

5

6

7 power/amplitude

8

9 frequency

10

11 +

12 + =

13

14 Tonotopic Organization

15 Speech...

16 Source-Filter Model

17 Source-Filter Model

18 Source-Filter Model

19 Source-Filter Model

20 Frequency Time

21

22

23

24

25 Stop Consonants: [p b t d k g]

26 Fricatives: [!! f v s z " #]

27 The Problem of Speech Perception

28

29

30

31 Hypothesized Representational Format

32 Hypothesized Representational Format

33 Hypothesized Representational Format

34 How do we get from here to there

35 How do we get from here to there

36 How do we get from here to there

37

38 The simplest theory Hypothesis: There is a one-to-one relationship between pieces of acoustic information and the segmental information stored in our head

39 The simplest theory

40 The simplest theory

41 29

42 30

43 30

44 30

45 Different Acoustic Input: Same percept! 30

46

47

48

49 Front Back High Low

50 Front Back High Good! Low

51

52 37

53 38

54 39

55 40

56 41

57 42

58 Peterson & Barney (1952)

59 Obscured by phonetic context and speaker differences...

60 Simple One-to-One Mapping between acoustic cue and phoneme doesn t seem to exist...

61 From vibrations in the ear to abstractions in the brain

62 From vibrations in the ear to abstractions in the brain sounds words

63 From vibrations in the ear to abstractions in the brain sounds words

64 From vibrations in the ear to abstractions in the brain sounds words Continuously varying waveform with information on multiple time- and frequency scales must be encoded

65 From vibrations in the ear to abstractions in the brain sounds words Continuously varying waveform with information on multiple time- and frequency scales must be encoded

66 From vibrations in the ear to abstractions in the brain sounds words Continuously varying waveform with information on multiple time- and frequency scales must be encoded and decoded to make contact with the long-term linguistic representations in memory WORD

67 From vibrations in the ear to abstractions in the brain sounds words Continuously varying waveform with information on multiple time- and frequency scales must be encoded word and decoded to make contact with the long-term linguistic representations in memory WORD WORD

68 sincetherearenowordboundarysignsinspokenlanguagethedifficultywefeelinreading andunderstandingtheaboveparagraphprovidesasimpleillustrationofoneofthemaind ifficultieswehavetoovercomeinordertounderstandspeechratherthananeatlyseparat edsequenceofletterstringscorrespondingtothephonologicalformofwordsthespeech signalisacontinuousstreamofsoundsthatrepresentthephonologicalformsofwordsin additionthesoundsofneighboringwordsoftenoverlapwhichmakestheproblemofident ifyingwordboundariesevenharder

69 Why speech perception should not work

70 Why speech perception should not work linearity no straightforward mapping between stretches of sound and phonemes

71 Why speech perception should not work linearity no straightforward mapping between stretches of sound and phonemes

72 Why speech perception should not work linearity invariance no straightforward mapping between stretches of sound and phonemes no (obvious) invariant features identify a given phoneme in all contexts

73 Why speech perception should not work linearity invariance no straightforward mapping between stretches of sound and phonemes no (obvious) invariant features identify a given phoneme in all contexts

74 Why speech perception should not work linearity invariance perceptual constancy no straightforward mapping between stretches of sound and phonemes no (obvious) invariant features identify a given phoneme in all contexts we reliably identify speech despite tremendous variation across speakers (pitch, rate, accent, affect )

75 Why speech perception should not work linearity invariance perceptual constancy no straightforward mapping between stretches of sound and phonemes no (obvious) invariant features identify a given phoneme in all contexts we reliably identify speech despite tremendous variation across speakers (pitch, rate, accent, affect )

76 Why speech perception should not work linearity invariance perceptual constancy no straightforward mapping between stretches of sound and phonemes no (obvious) invariant features identify a given phoneme in all contexts we reliably identify speech despite tremendous variation across speakers (pitch, rate, accent, affect ) Halle and Stevens 1962 Chomsky and Miller 1963

77 Varies across: speakers, phonetic context, rate, etc. Stable across: speakers, phonetic context, rate, etc.

78 Varies across: speakers, phonetic context, rate, etc. What set of perceptual/ neural mechanisms mediate the mapping between acoustic input and long term memory representations? Stable across: speakers, phonetic context, rate, etc.

79 The Problem of Speech Perception [+ voiced] [+ continuant]

80 The Problem of Speech Perception [+ voiced] [+ continuant] What s involved in this mapping?

81 The Problem of Speech Perception Time (s) Time (s) Time (s)

82

83 Questions Cognitive Neuroscience can help answer: 1. What is the nature of stored mental representations? 2. What types of mechanisms are involved in mapping from acoustics to memory? 3. What brain areas are implicated in the perception of speech?

84 Questions Cognitive Neuroscience can help answer: 1. What is the nature of stored mental representations? 2. What types of mechanisms are involved in mapping from acoustics to memory? 3. What brain areas are implicated in the perception of speech?

85 Levels of Representation Acoustics: Variation in air pressure; Analog input to auditory system Phonetics: Language-specific categorization of different acoustic tokens; phonetic tokens Discriminability of different acoustic tokens relatively preserved Phonology: Abstract symbolic representations; Fine-grained distinctions irrelevant; All or nothing category membership phonemes English [p h at] pot [spat] spot Hindi [p h $l] fruit [p$l] moment English /p/ Hindi /p h / /p/

86 Phonetic Categories Map acoustic tokens into a multidimensional space There still may be speech specific processing t t t t t t t t t t t t t t But... representations are not discrete, abstract, etc. Store fine phonetic detail d d d d d d d d d Dennis Klatt, Stephen Goldinger, Peter Jusczyk, Jessica Maye, Keith Johnson

87 Voice Onset Time

88 Voice Onset Time The dot The tot

89 Voice Onset Time The dot The tot

90 Voice Onset Time The dot The tot Short VOT Long VOT

91 Voice Onset Time /da/ /ta/ 20 Nb of tokens produced [d] [t] VOT (in ms)

92 Voice Onset Time

93 Categorical Perception [da] VOT: 20ms [ta] VOT: 80ms same different Discrimination Task

94 Categorical Perception [da] VOT: 20ms /t/ /d/ Identification Task

95 Voice Onset Time Identification RT of Identification from Phillips et al (2000)

96 Voice Onset Time Identification RT of Identification from Phillips et al (2000)

97

98 MMN = Mismatch Negativity ERP (event related potential) that reflects sensory discrimination Elicited by repeated presentation of a sound stimulus (standard) which is sometimes changed into a different sound (deviant): X X X X X X Y X X X X X Y X X X X Y

99 MMN = Mismatch Negativity ERP (event related potential) that reflects sensory discrimination Elicited by repeated presentation of a sound stimulus (standard) which is sometimes changed into a different sound (deviant): Elicited pre-attentively!

100 from Näätänen (1999)

101 = Standard - Deviant NOTICE: negative voltage up positive voltage down from Näätänen (1999)

102 N1 or N100 = Standard - Deviant NOTICE: negative voltage up positive voltage down from Näätänen (1999)

103 Obligatory ERP Reflects sensory encoding of auditory stimulus attributes

104 Discriminability (Methods) % Behavioral level: Categorical Perception % Electrophysiological level: MMN

105 Discriminability of phones by VOT % Behavioral level: Categorical Perception & % Electrophysiological level: MMN?

106 Looking at VOT: [dæ] vs [tæ] Behavioral data EEG: N1 (sensory encoding) EEG: MMN (sensory discrimination)

107 Sharma & Dorman 1999 Behavioral Experiment:

108 Sharma & Dorman 1999

109 Sharma & Dorman 1999 Discrimination: AX task

110 Sharma & Dorman 1999 Discrimination: AX task Performing at chance level

111 Sharma & Dorman 1999 MMN Experiment 30-50ms 60-80ms

112 Sharma & Dorman 1999 MMN Experiment

113 Level of representation Acoustics: % Variation in air pressure; % Analog input to auditory system Phonetics: % Language-specific categorization of different acoustic tokens; % Discriminability of different acoustic tokens relatively preserved Phonology: % Abstract symbolic representations; % Fine-grained distinctions irrelevant; % All or nothing category membership

114 Questions What kinds of representation is the MMN sensitive to? % Acoustic? % Phonetic? % Phonemic? How can we be sure it s not just acoustics?

115 Potential problem How can we be sure it s not just acoustics? There seems to be a difference between the and the MMN response; BUT, what if this difference has nothing to do with the phonetic category people perceive? Could it be that there is something special about the 30-50ms gap, for instance?

116 Potential problem How can we be sure it s not just acoustics? Could it be that there is something special about the 30-50ms gap, for instance?

117 Perception of VOT Identification RT of Identification from Phillips et al (2000)

118

119 Potential problem How can we be sure it s not just acoustics? Could it be that there is something special about the 30-50ms gap, for instance? If Chinchillas can show the same Categorical Perception behavior for the VOT continuum, this response is probably not based on phonetics

120 Potential problem Neuroscience evidence: VOT < 30ms and > 60ms have different neuronal population encoding in mammalian auditory system than VOT in the the 30ms-60ms range

121 Potential problem Could it be that there is something special about the 30-50ms gap, for instance? There is, apparently. How can we be sure it s not just acoustics? With these results alone, we can t.

122 Suggestions? Can we come up with ways to test whether or not we can test the MMN response to see if it is sensitive to the phonetic and phonological level of representations? Requirement: Many-to-one ratio XXXXY

123 Look at sounds that are phonemically in one language, but not in the other. % Näätänen et al (1997)

124 Na!a!ta!nen et al (1999) Looking for language-dependent memory traces for sounds Vowels Finnish Estonian

125

126 Vowels varying only in F2

127 Vowels varying only in F2 Estonian extra vowel F2 values

128 Vowels varying only in F2

129 MMN = Standard - Deviant NOTICE: negative voltage up positive voltage down from Näätänen (1999)

130 Pure Tones with freq = F2

131 Pure Tones with freq = F2

132 F2 Pure Tones vs Vowels

133 F2 Pure Tones vs Vowels Nonmonotonic increase; Drop Linear increase; No drop

134 Vowels: Finns vs Estonians

135 Vowels: Finns vs Estonians Drop No Drop

136 Finns vs Estonians MMN peak amplitude at Fz Finns (! blue) Estonians (' purple)

137 Finns vs Estonians MMN peak amplitude at Fz Finns (! blue) Estonians (' purple) Drop

138 MEG data - Dipole Model

139 MEG data - Dipole Model Drop

140 Conclusions Tone vs Vowel data is dissimilar for Finnish speakers, even though what s being varied in the two conditions is the exact same acoustic quantity % Finnish judge a vowel with an F2 of 1,311Hz to be a very bad instance of /ö/ % Estonians have a vowel /õ/, and judge a vowel with an F2 of 1,311Hz as a good instance of that /õ/ vowel

141 Conclusions % Finnish judge a vowel with an F2 of 1,311Hz to be a very bad instance of /ö/ % Estonians have a vowel /õ/, and judge a vowel with an F2 of 1,311Hz as a good instance of that /õ/ vowel % Estonian vowel MMN data is more in line with Finnish Tone data

142 Any problems? Are you convinced? Does this show that the MMN is indeed sensitive to phonemic categories? Could these results be explained on a purely acoustic basis? Could these results be explained on a purely phonetic basis?

143 Level of representation Acoustics: % Variation in air pressure; % Analog input to auditory system Phonetics: % Language-specific categorization of different acoustic tokens; % Discriminability of different acoustic tokens relatively preserved Phonology: % Abstract symbolic representations; % Fine-grained distinctions irrelevant; % All or nothing category membership

144 Phillips et al (2000) Question: Is the MMN sensitive to Phonological Categories? % Abstract symbolic representations; % Fine-grained distinctions irrelevant; % All or nothing category membership

145 Phillips et al (2000) Template of MMN design: X X X X X X Y X X X X X Y X X X X Y Sharma & Dorman (1999) - VOT values:

146 Phillips et al (2000) Template of MMN design: X X X X X X Y X X X X X Y X X X X Y Sharma & Dorman (1999) - VOT values: Many-to-one ratio at all levels

147 Phillips et al (2000) Template of MMN design: X X X X X X Y X X X X X Y X X X X Y Sharma & Dorman (1999) - VOT values: Many-to-one ratio at all levels Let s try to do the many-to-one ratio only at the phonological level

148 Phillips et al (2000) Template of MMN design: X X X X X X Y X X X X X Y X X X X Y Sharma & Dorman (1999) - VOT values: Phillips et al. (2000) - VOT values:

149 Perception of VOT Identification RT of Identification from Phillips et al (2000)

150 Many-to-one only at P level Phillips et al. (2000) - VOT values: A: P: D D D D D T D D D D D D T D D T D

151 Results Exp1

152 What if not PhonCat, but... What if the results are not due to Phonological categories, but to something prosaic as the VOT difference between adjacent sounds? From standard to standard, the VOT difference could span 0 to 24ms (mean 12) From standard to deviant, the VOT difference could go from 14 to 72ms (mean 40) How can we address this?

153 Exp. 2 - Acoustics Add 20ms VOT in all sounds, such that the relative distance between them remains the same, but the proportion of sounds falling on each side of the boundary change:

154 Exp. 2 - Acoustics Add 20ms VOT in all sounds, such that the relative distance between them remains the same, but the proportion of sounds falling on each side of the boundary change: No longer many-to-one relations at P level

155 What if not PhonCat, but...

156

157 No MMN for acoustic condition

158 Phillips et al (2000) Conclusion: MMN here is driven by phonological category membership, not acoustics.

159 Question Are you convinced? Can we be sure this result does not stem from acoustics? What about phonetic categories?

160 No Abstract Categories You simply map acoustic tokens into a multidimensional space There still may be speech specific processing t t t t t t t t t t t t t t But... representations are not discrete, abstract, etc. Store fine phonetic detail d d d d d d d d d Dennis Klatt, Stephen Goldinger, Peter Jusczyk, Jessica Maye, Keith Johnson

161 VOT Distribution DISTRIBUTION OF VOT Frequency More Voice Onset Time (in ms)

162 Do We Even Have Categories? Perhaps we should not even be asking if infants have well-formed phonetic categories, separated by boundaries, but rather if any language users do. In other words, the very concept of categories, and even more so of boundaries, needs to be reconsidered...we have no evidence that boundaries exist in the natural world, or any account of how or why they may have evolved by natural selection. To extend to them any degree of psychological reality is unsupportable, and deleterious to efforts to understand how phonetic structure is indeed instantiated and retrieved from the speech signal. Nittrouer (2001)

163 A Phonetic Explanation for Phillips, et al MMN could be induced by sampling from the statistical distribution of phonetic categories No need to rely on abstract phonological categories if this is how we conceive of the phonetic space t t t t t t t t t t t t t t Sampling from/mapping into different distribution could elicit MMN d d d d d d d d d

164 A Phonetic Explanation for Phillips, et al MMN could be induced by sampling from the statistical distribution of phonetic categories No need to rely on abstract phonological categories if this is how we conceive of the phonetic space t t t t t t t t t t t t t t t Sampling from/mapping into different distribution could elicit MMN d d d d d d d d d

165 A Phonetic Explanation for Phillips, et al MMN could be induced by sampling from the statistical distribution of phonetic categories No need to rely on abstract phonological categories if this is how we conceive of the phonetic space t t t t t t t t t t t t t t t Sampling from/mapping into different distribution could elicit MMN d d d d d d d d d

166 A Phonetic Explanation for Phillips, et al MMN could be induced by sampling from the statistical distribution of phonetic categories No need to rely on abstract phonological categories if this is how we conceive of the phonetic space t t t t t t t t t t t t t t t Sampling from/mapping into different distribution could elicit MMN d d d d d d d d d

167 A Phonetic Explanation for Phillips, et al MMN could be induced by sampling from the statistical distribution of phonetic categories No need to rely on abstract phonological categories if this is how we conceive of the phonetic space t t t t t t t t t t t t t t t Sampling from/mapping into different distribution could elicit MMN d d d d d d d d d

168 A Phonetic Explanation for Phillips, et al MMN could be induced by sampling from the statistical distribution of phonetic categories No need to rely on abstract phonological categories if this is how we conceive of the phonetic space t t t t t t t t t t t t t t t Sampling from/mapping into different distribution could elicit MMN d d d d d d d d d

169 A Phonetic Explanation for Phillips, et al MMN could be induced by sampling from the statistical distribution of phonetic categories No need to rely on abstract phonological categories if this is how we conceive of the phonetic space t t t t t t t t t t t t t t t Sampling from/mapping into different distribution could elicit MMN d d d d d d d d d

170 A Phonetic Explanation for Phillips, et al MMN could be induced by sampling from the statistical distribution of phonetic categories No need to rely on abstract phonological categories if this is how we conceive of the phonetic space Sampling from/mapping into different distribution could elicit MMN t t t t t t t t t t t t t t t MMN! d d d d d d d d d

171 Kazanina, et al. (2006)

172 Kazanina, et al. (2006)

173 Kazanina, et al. (2006)

174 Kazanina, et al. (2006)

175 Dupoux et al Phonotactics seems to influence how people perceive phonetic sounds. % Look at these Japanese borrowed words:

176 Dupoux et al Japanese has a restricted syllabic inventory when compared to languages such as English and French % V, CV, CVNasal, CVQ (Q = first half of a geminate consonant)

177 Dupoux et al Look at these Japanese borrowed words: Production? or is it Perception? Orthography?

178 Dupoux et al Exp 1 Use non-ambiguous stimuli, manipulate native language Hypothesis is that vowel epenthesis is perceptual phenomenon When presented with items like Ebzo, native French speakers would be ok, but Japanese speakers should report hearing a [u] sound.

179 Dupoux Dupoux et al. et 1999 al Exp 1 Japanese speaker recorded pseudo words of the structure VCuCV middle [u] was spliced out to different degrees (from virtually erased to just a little) Subjects had to hear stimuli and say whether or not they heard [u]

180 Dupoux Dupoux et al. et 1999 al Exp 1

181 Dupoux Dupoux et al. et 1999 al Exp 1

182 Dupoux Dupoux et al. et 1999 al Exp 1 Japanese participant reported many more [u]s when there was little or no [u] information in the signal, unlike French speakers BUT % Japanese speaker % Coarticulation cue in the preceding consonant?

183 Dupoux Dupoux et al. et 1999 al Exp 1 Japanese speaker: Coarticulation cue in the preceding consonant? [u] is often reduced or devoiced in Japanese Japanese might be extra sensitive to subtle coarticulation cues indicating [u]

184 Dupoux Dupoux et al. et 1999 al Exp 2 Japanese speaker: Coarticulation cue in the preceding consonant? -- Get a French speaker! Japanese might be extra sensitive to subtle coarticulation cues indicating [u] -- Make French speaker articulate true VCCVs as well as VCiCV The rest is the same as in Exp 1

185 Dupoux Dupoux et al. et 1999 al Exp 2

186 Dupoux Dupoux et al. et 1999 al Exp 2

187 Dupoux Dupoux et al. et 1999 al Exp 2

188 Dupoux Dupoux et al. et 1999 al Exp 2 Even with no coarticulation cue, Japanese speakers were reporting hearing [u] in VCCV nonwords.

189 How Early? (ERPs)

190 How Early? (ERPs)

191 Dehaene-Lambertz, et al. (2000) 164 ms

192 Dehaene-Lambertz, et al. (2000) 315 ms

193 Dehaene-Lambertz, et al. (2000) 531 ms

194 Dehaene-Lambertz, et al. (2000)

195 Dehaene-Lambertz, et al. (2000)

196 Dehaene-Lambertz, et al. (2000)

197 A quick word on cortical connectivity in speech perception...

198 Geschwind Model

199 Geschwind Model

200 Hickok & Poeppel (2007)

201

202 To wrap up...

203 To wrap up Speech perception involves a complex mapping between acoustic input and long term memory.

204 To wrap up Speech perception involves a complex mapping between acoustic input and long term memory. 2. Can use cognitive neuroscience methods to ascertain representational nature of speech segments.

205 To wrap up Speech perception involves a complex mapping between acoustic input and long term memory. 2. Can use cognitive neuroscience methods to ascertain representational nature of speech segments. 3. Understand how brain encodes speech representations.

206 To wrap up Speech perception involves a complex mapping between acoustic input and long term memory. 2. Can use cognitive neuroscience methods to ascertain representational nature of speech segments. 3. Understand how brain encodes speech representations. 4. Auditory cortex seems to store speech segments in phonemic form (at least in addition to phonetic representations).