This lecture Speech production and articulatory phonetics. Mel-frequency cepstral coefficients (i.e., the input to ASR systems) Next week: 3 lectures Some images from Jim Glass course 6.345 (MIT), the Jurafsky & Martin textbook, the Rolling Stones. CSC401/2511 Spring 2017 2
The vocal tract Nasal cavity Many physical structures are co-ordinated in the production of speech. Velum Tongue Lips Generally, sound is generated by passing air through the vocal tract. Glottis Lungs Jaw Sound is modified by constricting airflow in particular ways. CSC401/2511 Spring 2017 3
Place of articulation The location of the primary constriction can be: Alveolar: constriction near the alveolar ridge (e.g., /t/) Bilabial: touching of the lips together (e.g., /m/, /p/) Dental: constriction of/at the teeth (e.g., /th/) Labiodental: constriction between lip and teeth (e.g., /f/) Velar: constriction at or near the velum (e.g., /k/). CSC401/2511 Spring 2017 4
Phonetic features Phonemes can have several features, e.g.: Place: Location of primary constriction. Voicing: Manner: High/low: Front/back: {"#$%&#"', )*#")*"#, +%,-"#, #")*&+%,-"#, $%#"', } Whether the glottis is vibrating. {+$&*1%, $&*1%, } The class of articulation type. {3-&4, $&5%#,,"3"#, 6'*1"-*$%, "44'&7*8",-, '%-'&6#%7, } Anterior position of the tongue. {h*;h, 8*+, #&5, } Ventral position of the tongue. {6'&,-, 1%,-'"#, )"1<, } voicing as in silence CSC401/2511 Spring 2017 5
Spectrograms Spectrograms are 3D representations of any acoustic signal where a co-ordinate represents the amplitude of a particular sinusoidal frequency at a particular time. CSC401/2511 Spring 2017 6
Formants and phonemes Formant: n. A large concentration of energy within a band of frequencies (e.g., = >, =?, = @ ). = @ =? = > CSC401/2511 Spring 2017 7
Phonemic alphabets There are several alphabets that categorize the sounds of speech. The International Phonetic Alphabet (IPA) is popular, but it uses non-ascii symbols. The TIMIT phonemic alphabet will be used by default in this course. Other popular alphabets include ARPAbet, Worldbet, and OGIbet, usually adding special cases. E.g., /pcl/ is the period of silence immediately before a /p/. TIMIT IPA e.g. /iy/ /i y / beat /ih/ /ɪ/ bit /eh/ /ɛ/ bet /ae/ /æ/ bat /aa/ /ɑ/ Bob /ah/ /ʌ/ but /ao/ /ɔ/ bought /uh/ /ʊ/ book /uw/ /u/ boot /ux/ /u/ suit /ax/ /ə/ about CSC401/2511 Spring 2017 8
Phonemes Now we re going to discuss each of these manners of articulation: Vowels: open vocal tract, no nasal air. Fricatives: noisy, with air passing through a tight constriction (e.g., shift ). Stops/plosives: complete vocal tract constriction and burst of energy (e.g., papa ). Nasals: involve air passing through the nasal cavity (e.g., mama ). Semivowels: similar to vowels, but typically with more constriction (e.g., wall ). Affricates: Alveolar stop followed by fricative. CSC401/2511 Spring 2017 9
Vowels (1/6) There are approximately 19 vowels in Canadian English, including diphthongs in which the articulators move over time. Vowels are distinguished primarily by their formants. other /er/ /axr/ e.g. Bert butter diphthong /ey/ /ow/ /ay/ /oy/ /aw/ /ux/ e.g. bait boat bite boy bout suit Monophthong /iy/ /ih/ /eh/ /ae/ /aa/ /ao/ /ah/ /uh/ /uw/ /ax/ /ix/ e.g. beat bit bet bat Bob bought but book boot about roses CSC401/2511 Spring 2017 10
The uniform tube Closed, vibrating end glottis 17 cm Open, radiating end lips The positions of the tongue, jaw, and lips change the shape and cross-sectional area of the vocal tract. CSC401/2511 Spring 2017 11
Uniform tubes in practice Many musical instruments are based on the idea of uniform (or, in many cases, bent) tubes. Longer tubes produce deeper sounds (lower frequencies). A tube ½ the length of another will be 1 octave higher. CSC401/2511 Spring 2017 12
Vowels as concatenated tubes The vocal tract can be modelled as the concatenation of dozens, hundreds, or thousands of tubes. /iy/ /uw/ CSC401/2511 Spring 2017 13
Aside waves in concatenated tubes We model the volume velocity A B and the pressure variation 4 B at position 7 in the < CD lossless tube (whose area is E B ) at time - A B 7, - = A B G - 7 1 A B H - + 7 1 4 B 7, - = I1 E B A B G - 7 1 + A B H - + 7 1 where 1 is the speed of sound, I is the density of air. CSC401/2511 Spring 2017 14
Waves in concatenated tubes Because of partial wave reflections that occur at tube boundaries, we can generate spectra with particular resonances. CSC401/2511 Spring 2017 15
Spectrograms of vowels Tongue height is correlated with the first formant, = >. Tongue frontness is correlated with the second formant, =?. CSC401/2511 Spring 2017 16
The vowel trapezoid = > increases =? increases CSC401/2511 Spring 2017 17
Formants and front/back, high/low Front/ low Front/ high Back/ high CSC401/2511 Spring 2017 18
Fricatives (2/6) Fricatives are caused by acoustic turbulence at a narrow constriction whose position determines the sound. Fricatives can be voiced (i.e., the glottis can be vibrating). Labiodental dental alveolar palatal /f/ /th/ /s/ /sh/ CSC401/2511 Spring 2017 19
Fricatives Fricatives have four places of articulation: Labio-dental ( labial ) Interdental ( dental ) Alveolar Palato-alveolar ( palatal ) Unvoiced Every place of articulation has both a voiced and an unvoiced version. Voiced Labial /f/ fee /v/ Vendetta Dental /th/ thief /dh/ Thee Alveolar /s/ see /z/ Zardoz Palatal /sh/ she /zh/ Zha-zha CSC401/2511 Spring 2017 20
Unvoiced fricatives Note in these examples that /f/ has more energy in the word-final position, and /sh/ excites more of the spectrum than /s/ when word-initial. CSC401/2511 Spring 2017 21
Voiced versus unvoiced fricatives CSC401/2511 Spring 2017 22
Voiced versus unvoiced fricatives Spectra here look similar, but voiced fricatives often include significant energy < 150 Hz. CSC401/2511 Spring 2017 23
Plosives (3/6) Plosives build pressure behind a complete closure in the vocal tract. This closure can be associated with voiced excitation. A sudden release of this constriction results in brief noise. labial /b/ alveolar /d/ velar /g/ CSC401/2511 Spring 2017 24
Plosives Plosives have three places of articulation: Unvoiced Voiced Labial /p/ porpoise /b/ baboon Alveolar /t/ tort /d/ dodo Velar /k/ kick /g/ Google Voiced stops are usually characterized by a voice bar during closure, indicating the vibrating glottis. Formant transitions are very informative in classification. CSC401/2511 Spring 2017 25
Voicing contrasts The voice bar CSC401/2511 Spring 2017 26
Formant transitions among plosives Despite a common vowel, the motion of =? and = @ into (and out of) the vowel helps identify the plosive. CSC401/2511 Spring 2017 27
Voice onset time and voicing cues Voice onset time: n. the time from stop release (i.e., the start of the sound burst) to the start of vocal periodicity. There are at least 6 features that indicate voicing in plosives. CSC401/2511 Spring 2017 28
Voice onset time among plosives Unvoiced plosives have longer voice onset times CSC401/2511 Spring 2017 29
Nasals (4/6) Nasals involve lowering the velum so that air passes through the nasal cavity. Closures in the oral cavity (at same positions as plosives) change the resonant characteristics of the nasal sonorant. labial alveolar velar /m/ /n/ /ng/ CSC401/2511 Spring 2017 30
Formant transitions among nasals Nasals often appear as two formants Despite a common vowel, the motion of =? and = @ before and after each nasal helps to identify it. CSC401/2511 Spring 2017 31
Semivowels (5/6) Semivowels act as consonants in syllables and involve constriction in the vocal tract, but there is less turbulence. They also involve slower articulatory motion. Laterals involve airflow around the sides of the tongue. /w/ /y/ /r/ /l/ CSC401/2511 Spring 2017 32
Semivowels Semivowels are often sub-classified as glides or liquids. Glides Liquids Semivowel Nearest vowel /w/ Wow /uw/ /y/ yoyo /iy/ /r/ rear /er/ /l/ Lulu /ow/ Semivowels are more constricted versions of corresponding vowels. Similar formants, though generally weaker. CSC401/2511 Spring 2017 33
Semivowels Note the drastic formant transitions which are more typical of semivowels. CSC401/2511 Spring 2017 34
Affricates and aspirants (6/6) There are two affricates: /jh/ (voiced; e.g., judge) and /ch/ (unvoiced; e.g., church). These involve an alveolar stop followed by a fricative. Voicing in /jh/ is normally indicated by voice bars, as with plosives. There s only one aspirant in Canadian English: /h/ (e.g., hat) This involves turbulence generated at the glottis, In Canadian English, there is no constriction in the vocal tract. CSC401/2511 Spring 2017 35
Affricates and aspirants CSC401/2511 Spring 2017 36
Putting it all together Phonemes are grouped together in syllables which must contain a non-obstruent nucleus. Consonants can be clustered in particular ways in the onset and coda, but with constraints. E.g., you usually can t start a word with /kt/, nor end one with /tk/. CSC401/2511 Spring 2017 37
Alternative pronunciations Pronunciations of words can vary significantly, but with observable frequencies. The Switchboard corpus is a phonetically annotated database of speech recorded in telephone conversations. CSC401/2511 Spring 2017 38
Known effects of pronunciation Speakers tend to drop or change pronunciations in predictable ways in order to reduce the effort required to co-ordinate the various articulators. Palatalization generally refers to a conflation of phonemes closer to the frontal palate than they should be. Final t/d deletion is simply the omission of alveolar plosives from the ends of words. CSC401/2511 Spring 2017 39
Variation at syllable boundaries CSC401/2511 Spring 2017 40
Recall our input to ASR Frame Spectrum Amplitude We want to transform the spectrum into something more useful Frequency (Hz) CSC401/2511 Spring 2017 41
1. The Mel-scale filter bank To mimic the response of the human ear (and because it empirically improves speech recognition), we often discretize the spectrum using J triangular filters. Uniform spacing before 1 khz, logarithmic after 1 khz CSC401/2511 Spring 2017 42
2. Source and filter The acoustics of speech are produced by a glottal pulse waveform (the source) passing through a vocal tract whose shape modifies that wave (the filter). The shape of the vocal tract is more important to phoneme recognition. We to separate the source from the filter in the acoustics. CSC401/2511 Spring 2017 43
2. Source and filter (aside) Since speech is assumed to be the output of a linear time invariant system, it can be described as a convolution. Convolution, 7 L, is beyond the scope of this course, but can be conceived as the modification of one signal by another. For speech signal 7[,], glottal signal ;,, and vocal tract transfer $[,] with spectra O[P], Q[P], and R[P], respectively : 7, = ;, $, O P = Q P R P log O[P] = log Q P + log R[P] We ve separated the source and filter into two terms! CSC401/2511 Spring 2017 44
2. The cepstrum We separate the source and the filter by pretending the log of the spectrum is actually a time domain signal. the log spectrum log O[P] is a sum of the log spectra of the source and filter, i.e., a superposition; finding its spectrum will allow us to isolate these components. Cepstrum: n. the spectrum of the log of the spectrum. Fun fact: ceps is the reverse of spec. Instead of filters we have lifters log log log CSC401/2511 Spring 2017 45
2. The cepstrum Spectrum Log spectrum Cepstrum The domain of the cepstrum is quefrency (a play on the word frequency ). CSC401/2511 Spring 2017 46
2. The cepstrum Spectrum Pictures from John Coleman (2005) Cepstrum This is due to the vocal tract shape This is due to the glottis CSC401/2511 Spring 2017 47
Mel-frequency cepstral coefficients Mel-frequency cepstral coefficients (MFCCs) are the most popular representation of speech used in ASR. They are the spectra of the logarithms of the mel-scaled filtered spectra of the windows of the waveform. Speech signal window DFT Mel filterbank log DFT MFCC CSC401/2511 Spring 2017 48
Advantages of MFCCs The cepstrum produces highly uncorrelated features (every dimension is useful). This includes a separation of the source and filter. In practice, the cepstrum has been easier to learn than the spectrum for phoneme recognition. There is an efficient method to compute cepstra called the discrete cosine transform. CSC401/2511 Spring 2017 49
MFCCs in practice An observation vector of MFCCs often consists of The first 13 cepstral coefficients (i.e., the first 13 dimensions produced by this method), An additional overall energy measure, The velocities (V) of each of those 14 dimensions, i.e., the rate of change of each coefficient at a given time The accelerations (VV) of each of original 14 dimensions. The result is that at a timeframe - we have an observation MFCC vector of (13+1)*3=42 dimensions. This vector is what is used by our ASR systems CSC401/2511 Spring 2017 50
Next week