This lecture. Speech production and articulatory phonetics. Mel-frequency cepstral coefficients (i.e., the input to ASR systems) Next week: 3 lectures

Size: px

Start display at page:

Download "This lecture. Speech production and articulatory phonetics. Mel-frequency cepstral coefficients (i.e., the input to ASR systems) Next week: 3 lectures"

Oswin Carroll
6 years ago
Views:

2 This lecture Speech production and articulatory phonetics. Mel-frequency cepstral coefficients (i.e., the input to ASR systems) Next week: 3 lectures Some images from Jim Glass course (MIT), the Jurafsky & Martin textbook, the Rolling Stones. CSC401/2511 Spring

Velum Tongue Lips Generally, sound is generated by passing air through

3 The vocal tract Nasal cavity Many physical structures are co-ordinated in the production of speech. Velum Tongue Lips Generally, sound is generated by passing air through the vocal tract. Glottis Lungs Jaw Sound is modified by constricting airflow in particular ways. CSC401/2511 Spring

g., /th/) Labiodental: constriction between lip and teeth (e.g., /f/) Velar: constriction at or near the velum (e.

4 Place of articulation The location of the primary constriction can be: Alveolar: constriction near the alveolar ridge (e.g., /t/) Bilabial: touching of the lips together (e.g., /m/, /p/) Dental: constriction of/at the teeth (e.g., /th/) Labiodental: constriction between lip and teeth (e.g., /f/) Velar: constriction at or near the velum (e.g., /k/). CSC401/2511 Spring

5 Phonetic features Phonemes can have several features, e.g.: Place: Location of primary constriction. Voicing: Manner: High/low: Front/back: {"#$%&#"', )*#")*"#, +%,-"#, #")*&+%,-"#, $%#"', } Whether the glottis is vibrating. {+$&*1%, $&*1%, } The class of articulation type. {3-&4, $&5%#,,"3"#, 6'*1"-*$%, "44'&7*8",-, '%-'&6#%7, } Anterior position of the tongue. {h*;h, 8*+, #&5, } Ventral position of the tongue. {6'&,-, 1%,-'"#, )"1<, } voicing as in silence CSC401/2511 Spring

6 Spectrograms Spectrograms are 3D representations of any acoustic signal where a co-ordinate represents the amplitude of a particular sinusoidal frequency at a particular time. CSC401/2511 Spring

7 Formants and phonemes Formant: n. A large concentration of energy within a band of frequencies (e.g., = >, =?, ). =? = > CSC401/2511 Spring

8 Phonemic alphabets There are several alphabets that categorize the sounds of speech. The International Phonetic Alphabet (IPA) is popular, but it uses non-ascii symbols. The TIMIT phonemic alphabet will be used by default in this course. Other popular alphabets include ARPAbet, Worldbet, and OGIbet, usually adding special cases. E.g., /pcl/ is the period of silence immediately before a /p/. TIMIT IPA e.g. /iy/ /i y / beat /ih/ /ɪ/ bit /eh/ /ɛ/ bet /ae/ /æ/ bat /aa/ /ɑ/ Bob /ah/ /ʌ/ but /ao/ /ɔ/ bought /uh/ /ʊ/ book /uw/ /u/ boot /ux/ /u/ suit /ax/ /ə/ about CSC401/2511 Spring

9 Phonemes Now we re going to discuss each of these manners of articulation: Vowels: open vocal tract, no nasal air. Fricatives: noisy, with air passing through a tight constriction (e.g., shift ). Stops/plosives: complete vocal tract constriction and burst of energy (e.g., papa ). Nasals: involve air passing through the nasal cavity (e.g., mama ). Semivowels: similar to vowels, but typically with more constriction (e.g., wall ). Affricates: Alveolar stop followed by fricative. CSC401/2511 Spring

10 Vowels (1/6) There are approximately 19 vowels in Canadian English, including diphthongs in which the articulators move over time. Vowels are distinguished primarily by their formants. other /er/ /axr/ e.g. Bert butter diphthong /ey/ /ow/ /ay/ /oy/ /aw/ /ux/ e.g. bait boat bite boy bout suit Monophthong /iy/ /ih/ /eh/ /ae/ /aa/ /ao/ /ah/ /uh/ /uw/ /ax/ /ix/ e.g. beat bit bet bat Bob bought but book boot about roses CSC401/2511 Spring

11 The uniform tube Closed, vibrating end glottis 17 cm Open, radiating end lips The positions of the tongue, jaw, and lips change the shape and cross-sectional area of the vocal tract. CSC401/2511 Spring

12 Uniform tubes in practice Many musical instruments are based on the idea of uniform (or, in many cases, bent) tubes. Longer tubes produce deeper sounds (lower frequencies). A tube ½ the length of another will be 1 octave higher. CSC401/2511 Spring

13 Vowels as concatenated tubes The vocal tract can be modelled as the concatenation of dozens, hundreds, or thousands of tubes. /iy/ /uw/ CSC401/2511 Spring

14 Aside waves in concatenated tubes We model the volume velocity A B and the pressure variation 4 B at position 7 in the < CD lossless tube (whose area is E B ) at time - A B 7, - = A B G A B H B 7, - = I1 E B A B G A B H where 1 is the speed of sound, I is the density of air. CSC401/2511 Spring

15 Waves in concatenated tubes Because of partial wave reflections that occur at tube boundaries, we can generate spectra with particular resonances. CSC401/2511 Spring

16 Spectrograms of vowels Tongue height is correlated with the first formant, = >. Tongue frontness is correlated with the second formant, =?. CSC401/2511 Spring

17 The vowel trapezoid = > increases =? increases CSC401/2511 Spring

18 Formants and front/back, high/low Front/ low Front/ high Back/ high CSC401/2511 Spring

19 Fricatives (2/6) Fricatives are caused by acoustic turbulence at a narrow constriction whose position determines the sound. Fricatives can be voiced (i.e., the glottis can be vibrating). Labiodental dental alveolar palatal /f/ /th/ /s/ /sh/ CSC401/2511 Spring

20 Fricatives Fricatives have four places of articulation: Labio-dental ( labial ) Interdental ( dental ) Alveolar Palato-alveolar ( palatal ) Unvoiced Every place of articulation has both a voiced and an unvoiced version. Voiced Labial /f/ fee /v/ Vendetta Dental /th/ thief /dh/ Thee Alveolar /s/ see /z/ Zardoz Palatal /sh/ she /zh/ Zha-zha CSC401/2511 Spring

21 Unvoiced fricatives Note in these examples that /f/ has more energy in the word-final position, and /sh/ excites more of the spectrum than /s/ when word-initial. CSC401/2511 Spring

22 Voiced versus unvoiced fricatives CSC401/2511 Spring

23 Voiced versus unvoiced fricatives Spectra here look similar, but voiced fricatives often include significant energy < 150 Hz. CSC401/2511 Spring

24 Plosives (3/6) Plosives build pressure behind a complete closure in the vocal tract. This closure can be associated with voiced excitation. A sudden release of this constriction results in brief noise. labial /b/ alveolar /d/ velar /g/ CSC401/2511 Spring

25 Plosives Plosives have three places of articulation: Unvoiced Voiced Labial /p/ porpoise /b/ baboon Alveolar /t/ tort /d/ dodo Velar /k/ kick /g/ Google Voiced stops are usually characterized by a voice bar during closure, indicating the vibrating glottis. Formant transitions are very informative in classification. CSC401/2511 Spring

26 Voicing contrasts The voice bar CSC401/2511 Spring

27 Formant transitions among plosives Despite a common vowel, the motion of =? and into (and out of) the vowel helps identify the plosive. CSC401/2511 Spring

28 Voice onset time and voicing cues Voice onset time: n. the time from stop release (i.e., the start of the sound burst) to the start of vocal periodicity. There are at least 6 features that indicate voicing in plosives. CSC401/2511 Spring

29 Voice onset time among plosives Unvoiced plosives have longer voice onset times CSC401/2511 Spring

30 Nasals (4/6) Nasals involve lowering the velum so that air passes through the nasal cavity. Closures in the oral cavity (at same positions as plosives) change the resonant characteristics of the nasal sonorant. labial alveolar velar /m/ /n/ /ng/ CSC401/2511 Spring

31 Formant transitions among nasals Nasals often appear as two formants Despite a common vowel, the motion of =? and before and after each nasal helps to identify it. CSC401/2511 Spring

32 Semivowels (5/6) Semivowels act as consonants in syllables and involve constriction in the vocal tract, but there is less turbulence. They also involve slower articulatory motion. Laterals involve airflow around the sides of the tongue. /w/ /y/ /r/ /l/ CSC401/2511 Spring

33 Semivowels Semivowels are often sub-classified as glides or liquids. Glides Liquids Semivowel Nearest vowel /w/ Wow /uw/ /y/ yoyo /iy/ /r/ rear /er/ /l/ Lulu /ow/ Semivowels are more constricted versions of corresponding vowels. Similar formants, though generally weaker. CSC401/2511 Spring

34 Semivowels Note the drastic formant transitions which are more typical of semivowels. CSC401/2511 Spring

35 Affricates and aspirants (6/6) There are two affricates: /jh/ (voiced; e.g., judge) and /ch/ (unvoiced; e.g., church). These involve an alveolar stop followed by a fricative. Voicing in /jh/ is normally indicated by voice bars, as with plosives. There s only one aspirant in Canadian English: /h/ (e.g., hat) This involves turbulence generated at the glottis, In Canadian English, there is no constriction in the vocal tract. CSC401/2511 Spring

36 Affricates and aspirants CSC401/2511 Spring

37 Putting it all together Phonemes are grouped together in syllables which must contain a non-obstruent nucleus. Consonants can be clustered in particular ways in the onset and coda, but with constraints. E.g., you usually can t start a word with /kt/, nor end one with /tk/. CSC401/2511 Spring

38 Alternative pronunciations Pronunciations of words can vary significantly, but with observable frequencies. The Switchboard corpus is a phonetically annotated database of speech recorded in telephone conversations. CSC401/2511 Spring

Palatalization generally refers to a conflation of phonemes closer to the frontal palate than they

39 Known effects of pronunciation Speakers tend to drop or change pronunciations in predictable ways in order to reduce the effort required to co-ordinate the various articulators. Palatalization generally refers to a conflation of phonemes closer to the frontal palate than they should be. Final t/d deletion is simply the omission of alveolar plosives from the ends of words. CSC401/2511 Spring

40 Variation at syllable boundaries CSC401/2511 Spring

41 Recall our input to ASR Frame Spectrum Amplitude We want to transform the spectrum into something more useful Frequency (Hz) CSC401/2511 Spring

42 1. The Mel-scale filter bank To mimic the response of the human ear (and because it empirically improves speech recognition), we often discretize the spectrum using J triangular filters. Uniform spacing before 1 khz, logarithmic after 1 khz CSC401/2511 Spring

43 2. Source and filter The acoustics of speech are produced by a glottal pulse waveform (the source) passing through a vocal tract whose shape modifies that wave (the filter). The shape of the vocal tract is more important to phoneme recognition. We to separate the source from the filter in the acoustics. CSC401/2511 Spring

44 2. Source and filter (aside) Since speech is assumed to be the output of a linear time invariant system, it can be described as a convolution. Convolution, 7 L, is beyond the scope of this course, but can be conceived as the modification of one signal by another. For speech signal 7[,], glottal signal ;,, and vocal tract transfer $[,] with spectra O[P], Q[P], and R[P], respectively : 7, = ;, $, O P = Q P R P log O[P] = log Q P + log R[P] We ve separated the source and filter into two terms! CSC401/2511 Spring

2. The cepstrum We separate the source and the filter by pretending the log

the log spectrum log O[P] is a sum of the log spectra of the source and

45 2. The cepstrum We separate the source and the filter by pretending the log of the spectrum is actually a time domain signal. the log spectrum log O[P] is a sum of the log spectra of the source and filter, i.e., a superposition; finding its spectrum will allow us to isolate these components. Cepstrum: n. the spectrum of the log of the spectrum. Fun fact: ceps is the reverse of spec. Instead of filters we have lifters log log log CSC401/2511 Spring

46 2. The cepstrum Spectrum Log spectrum Cepstrum The domain of the cepstrum is quefrency (a play on the word frequency ). CSC401/2511 Spring

47 2. The cepstrum Spectrum Pictures from John Coleman (2005) Cepstrum This is due to the vocal tract shape This is due to the glottis CSC401/2511 Spring

They are the spectra of the logarithms of the mel-scaled filtered spectra of the

48 Mel-frequency cepstral coefficients Mel-frequency cepstral coefficients (MFCCs) are the most popular representation of speech used in ASR. They are the spectra of the logarithms of the mel-scaled filtered spectra of the windows of the waveform. Speech signal window DFT Mel filterbank log DFT MFCC CSC401/2511 Spring

49 Advantages of MFCCs The cepstrum produces highly uncorrelated features (every dimension is useful). This includes a separation of the source and filter. In practice, the cepstrum has been easier to learn than the spectrum for phoneme recognition. There is an efficient method to compute cepstra called the discrete cosine transform. CSC401/2511 Spring

50 MFCCs in practice An observation vector of MFCCs often consists of The first 13 cepstral coefficients (i.e., the first 13 dimensions produced by this method), An additional overall energy measure, The velocities (V) of each of those 14 dimensions, i.e., the rate of change of each coefficient at a given time The accelerations (VV) of each of original 14 dimensions. The result is that at a timeframe - we have an observation MFCC vector of (13+1)*3=42 dimensions. This vector is what is used by our ASR systems CSC401/2511 Spring

51 Next week

On the Formation of Phoneme Categories in DNN Acoustic Models

On the Formation of Phoneme Categories in DNN Acoustic Models Tasha Nagamine Department of Electrical Engineering, Columbia University T. Nagamine Motivation Large performance gap between humans and state-