INTERSPEECH 2014 Enabling Controllability for Continuous Expression Space Langzhou Chen, Norbert Braunschweiler Toshiba Research Europe Ltd., Cambridge, UK langzhou.chen,norbert.braunschweiler@crl.toshiba.co.uk Abstract A continuous expression space assumes that each utterance contains individual expressions. Thus, it can be used to model detailed expression information in speech data. However, since an infinite number of different expressions can be contained in the continuous expression space, it is very difficult to manually label them. That means, these expressions are very hard to identify and to extract for synthesising expressive speech. A mechanism to control the continuous expression space is missing. In the discrete expression space though, only a few emotions are defined, thus users can easily choose from these emotions, but the range of expressivity is limited. This work proposes a method to automatically annotate expressions in the continuous expression space based on the cluster adaptive training (CAT) method. Using the proposed method, complex emotion information can be associated to the individual expressions in the continuous space. These emotion labels can be used as indexes of the expressions in the continuous space to enable users to select desired expressions at synthesis time, i.e. enable the controllability for the continuous expression space. Meanwhile, the rich expressive information in the continuous space is kept so that more expressive speech can be generated compared to the discrete space. Index Terms: expressive speech synthesis, hidden Markov model, cluster adaptive training, audiobook 1. Introduction Expressive text-to-speech (TTS) synthesis systems have been developed very fast in recent years. Traditional expressive TTS systems maintain a discrete expression space which contains several pre-defined emotions, e.g. happy, sad, angry, etc. Since the discrete expression space methods only support a limited number of pre-defined emotions, the training data related to different emotions can be collected separately and various methods have been presented to model the emotions, e.g. the model interpolation method [1], the decision tree based method [2], etc. In order to solve the data fragmentation problem, the adaptive training framework has been introduced into the emotion modelling in which the emotion information and the phonetically relevant information in speech data are modelled by linear transforms and the canonical model respectively [3, 4]. The advantage of the discrete space methods is that each emotion in the discrete space has a very clear definition. Thus, users can choose the desired emotion to synthesise the expressive speech directly. That means, the discrete systems provide a high degree of control to users. However, in the discrete space, the range of expressivity is limited and the expressions in human s speech are too rich to be represented by a few emotions. To address the problem of the limited range of expressivity in the discrete expression space, a continuous expression space method was presented in [5]. In the continuous expression space, each utterance is assumed to contain individual expressions. Thus, the very detailed expressive information in each utterance can be modelled. Unlike the discrete expression space which was widely used to model the acted speech corpus with pre-defined emotions, the continuous expression space was used to model the rich expressions in very large, highly diverse speech corpora like audiobooks [4, 6, 7, 8]. The continuous expression space can be used to model the very rich expressions in human s speech. However, it is non-trivial to identify and extract the detailed expressive information in each utterance. Manually annotating the unique expressive information in each utterance is expensive and yields poor inter-annotator agreement. To use the unlabelled expressive information in the continuous expression space, in [9], an automatic text to expression prediction method was presented to automatically assign an expression in the continuous expression space to a given text. However, in some applications, the users need to define the emotions in synthetic speech by themselves. In this scenario, the unlabelled expressions in the continuous expression space can not be used directly, i.e. they are not controllable. This work aims at enabling users to control the continuous expression space. The continuous expression space was constructed based on the CAT model presented in [10]. This model is a subspace based method which represents the very high dimensional synthesis parameters as a point in a very lowdimensional subspace (typically 4 to 8 dimensions). By using the CAT model to construct the continuous expression space, an expression subspace is built and the expression in each utterance is defined as a point in the expression subspace. To enable users to select the desired expressions from the continuous expression space, the expressions in the space must be annotated with user understandable labels. The labels should be complex enough to describe the rich expressive information in the continuous space. In this work, an automatic expression annotation method is proposed by using the emotion information in the discrete space. The speech data with discrete emotion is projected into the continuous expression space. This way, each discrete emotion, e.g. happy, sad, etc., is represented as a point in the continuous expression space as well. Given an expressive utterance, the expressions in this utterance can be represented as a vector of the posterior distribution of the discrete emotions. This provides a way to identify the emotions in an utterance and their degree. At synthesis time, the continuous expression space forms an expression database, while the expression labels generated by the proposed method can be used as indexes to access the database. Users input the desired emotion information as query, the best matched expression in the continuous space is selected and used to generate expressive synthetic speech. The advantage of the proposed method is that it enables the continuous expression space to be used to synthesise speech with expressions defined by the user. Thus a controllable TTS system with a large range of expressivity can be achieved. Copyright 2014 ISCA 2912 14-18 September 2014, Singapore
2. CAT Model and Expression Space The CAT model [10] is a subspace based method. It consists of a set of cluster models, each of which contains a set of Gaussian mean parameters while the Gaussian variances are shared over all clusters. When this CAT model is used to calculate the likelihood of an observation vector, the mean vector to be used is a linear interpolation of all the cluster means, i.e. p(o t λ, M (m), Σ (m) )=N (o t; M (m) λ, Σ (m) ) (1) where M (m) is the matrix of P cluster mean vectors i for component m, i.e.m (m) = hμ (m,1)... μ (m,p ) and λ is the CAT weight vector. As in the standard CAT approaches the first cluster is specified as a bias cluster, thus λ =[1 λ 2... λ P ] T (2) CAT cluster models can be viewed as a space of expressive synthesis parameters, effectively an expression-space. Two types of cluster models are defined in a CAT model: one bias cluster and multiple non-bias clusters. The bias cluster model is used to capture the common factors across all the expressions and its CAT weight is fixed to 1 for all the expressions. The nonbias clusters, are used to model the expression dependent information. The CAT weights for non-bias clusters are different for different expressions. The cluster models form a subspace of expression synthesis parameters. The synthesis parameters with different expressions can be projected into this subspace, while the CAT weights are the coordinates of this projection. Thus, based on CAT, the synthesis parameters for each expression are represented as a unique CAT weight vector. 2.1. Discrete Expression Space In the discrete expression space, only a limited number of expressions are modelled. That means, the utterances are classified into several groups, each group of data is assumed to contain the same expression. To estimate the expression dependent CAT weight vectors, the auxiliary function of CAT weight training can be written as Q( ˆΛ; Λ) = (3) = X ˆλ(e)T (e) k disc 1 2 ˆλ (e)t G (e) ˆλ (e) disc + C e where e is the discrete expression state index, and C represents the terms independent of the CAT weight vectors. The sufficient statistics G (e) and k (e) are given by G (e) disc = X m k (e) disc = X m X X u e t M (m)t Σ (m)-1 M (m) (4) X t (o t μ (m,1) ) (5) M (m)t Σ (m)-1 X u e where u e indicates that the summing over u is performed for all the utterances belonging to a particular expression e; T u represents the duration of utterance u; μ (m,1) is the mean vector of component m from the bias cluster. Based on the discrete expression space, a controllable expressive TTS system can be built in a straightforward way. That means, to classify the utterances into a set of pre-defined emotions, e.g. happy, sad, etc. Thus each discrete expression state in the space is related to a particular emotion. Users can use the emotions as indexes to access the desired expression state in the discrete space and synthesise the expressive speech. However a limited number of expression states are not enough to represent the richness of human s speech. To model more detailed expressive information in each utterance, the continuous expression space was proposed. 2.2. Continuous Expression Space The continuous expression space assumes that each utterance contains an individual expression. Using the CAT model, the difference between a discrete or a continuous expressive space representation is simply at what level the sufficient statistics for parameter estimation are accumulated. In the case of a discrete space, as shown in equation 4 and 5, the summation over t is for all frames that are allocated to a particular discrete expressive state. When a continuous expressive space is used, the summation over t is only for the frames of an utterance, i.e. G (u) cont = X X t M (m)t Σ (m)-1 M (m) (6) m k (u) cont = X M (m)t Σ X (m)-1 t (o t μ (m,1) ) (7) m These estimates allow each utterance u to have a separate expressive representation. The only constraint is that it will sit within the continuous expressive space defined by CAT. To construct the continuous expression space, the data sparseness is the problem which needs to be addressed since the sufficient statistics are accumulated at the utterance level. This problem can be alleviated by using the statistics in the discrete space as prior to smooth the utterance level statistics. The details can be found in [5]. 3. Adding controllability for the continuous expression space This work presents a framework for a user controllable expressive TTS system using the continuous expression space. In the controllable expressive TTS system with the continuous expression space, users can define the expressions in synthetic speech by the following vector: q =[q 1 q 2... q k ] T (8) where each component i represents a particular emotion, and q i represents the degree of emotion i chosen by the user. Equation 8 yields a more complex expression representation than the emotions in the discrete space. This representation does not only consider the combination of different emotions, but also includes the degree of each emotion. Thus, it can be used to represent richer expressions than the isolated emotions in the discrete space. Given the user defined complex expression representation, the process of enabling the controllability for the continuous expression space is actually a process of looking for the expressions in the continuous space which match the user defined expression. It can be divided into two steps, annotating the human understandable emotion information to the expressions in the continuous space and looking up the desired expression according to the annotations. 3.1. Posterior probability based expression annotation The continuous expression space contains very rich expressive information in human s speech. To use this information for 2913
building a human controllable expressive TTS, all the expressions in the continuous expression space need to be labelled with human comprehensible emotion information. More detailed emotion labels are desired, so that users of the expressive TTS system can select the accurate expressions at synthesis time. However, manually identifying and extracting the detailed expressive information in each utterance is non-trivial. In this paper, the method of automatic expression annotation is investigated. The expressions in the continuous expression space are represented as a posterior probability distribution of the predefined emotions, given an utterance. Each expression in the continuous space represents the particular expressive information for an utterance. Thus the posterior probabilities of different emotions given the utterance u can be viewed as the degrees from different emotions in the expression related to this utterance. Thus the expressions in the continuous expression space can be represented as the following vector p (u) =[p(e 1 u) p(e 2 u)... p(e k u)] T (9) where p(e i u) is the posterior probability of emotion e i given the utterance u,i.e. p(e i u) = 1 p(ou H; M, ei) Tu 1 Pj p(ou H; M, ej) Tu (10) where O u represents the observation vectors for utterance u, H represents the transcripts of the utterance u and M represents the acoustic model parameters. p(o u H; M, e i) can be calculated using a standard forward-backward algorithm. 3.2. Controllable Expressive Speech Synthesis Using the Continuous Expression Space A controllable expressive speech synthesis system means that the users can define the expressions for the synthetic speech by themselves. In this work, the continuous expression space was viewed as a large expression database which contains very rich expressive information in human s speech. The emotion annotation for each expression which are represented in the form of equation 9 is used as index to access the expressions in the database. At the synthesis stage, the user provides the desired expression information as equation 8. Then, the desired emotion representation is compared to the complicated emotion annotations for the expressions in the continuous space one by one to find the matched emotion annotations. Finally, using the best matched emotion annotation, the ideal expression in the continuous space can be found and used to generate the synthetic speech with the same expression. This process is shown in figure 1. Figure 1: Controllable TTS with continuous expression space To evaluate the distance between the user defined expression and the expressions in the continuous space, the user defined expression is converted as a posterior probability vector of emotions as well, i.e.» q γ t = P 1 P qγ 2... P qγ k T (11) For different users, the scale of the emotion is different, that means the same emotion description, e.g. very angry, may imply different degrees for different users. To adjust this difference, a user dependent parameter γ is introduced into equation 11. With equation 11 and equation 9, the distance between the user defined expression and the expressions in the continuous space can be calculated as the Kullback Leibler divergence between these two posterior distributions. In the process in figure 1, there may be multiple expressions which match users definition, since the continuous expression space contains a large amount of expressions. These provide an opportunity for users to select the most suitable one by listening. Note, in the discrete space, users do not have the opportunity to choose alternative expressions since the available emotions are pre-defined and of limited amount. 4. Experimental Results The training speech corpus used in this work is a studio recorded audiobook-like corpus consisting of 50 Hans Christian Anderson fairytales including 6k sentences ( 11h speech). This corpus is very expressive since the professional female speaker interpreted the stories and used multiple expressions to enhance the appropriate emotions for the content. Many of the characters occurring in the fairytales were realised by distinct character voices chosen by the speaker. The sampling rate of the synthetic speech samples was 16kHz and acoustic features consisted of 40 mel-cepstral coefficients, logf0, 21 (approximately bark scaled) BAP plus their delta and delta-delta information. The models were 5 state left-to-right multi-space probability distribution hidden semi-markov models. A CAT model was trained based on the fairytales corpus to construct the continuous expression space. This model includes one bias cluster model and 4 non-bias models. The cluster models formed the basis of the expression space. Then each utterance in the training speech data was projected onto this basis and the CAT weight vector, i.e. the coordinates in this projection represent the expression information in the utterance. The basis of the expression space, i.e. cluster models and the expressions in the space, i.e. the CAT weight vectors were alternately updated to maximize the likelihood of the training data. The training process of the CAT model was described in [5]. Since the continuous expression space assumes that each utterance contains an individual expression, the expressions in the training corpus form an expression database with 6k entries, and each entry represents an expression from an utterance. In order to make the continuous expression space controllable, the expression annotations need to be added to each entry as the indexes to access the expression database. In this work, a discrete emotion corpus from the same speaker was used to annotate the expressions in the continuous space. The discrete emotion corpus contains 6.9k sentences ( 5h speech) and consists of 6 subcorpora, 5 of them including the emotions angry, fear, happy, sad, tender (each of them including roughly the same amount of speech, i.e. less than 1 hour) and 1 sub-corpus including neutral speech ( 2h). The emotion dependent speech data from the discrete emotion corpus were projected into the continuous expression space by maximizing the likelihood of the sub-corpus of each emotion. This way, each individual emotion e.g. happy 2914
sad, etc. is represented as a point (i.e. a CAT weight vector) in the continuous expression space. Based on the CAT weight vectors of the discrete emotion, the posterior probability vectors of emotions given by equation 9 can be calculated for each expression in the continuous space and they were used as the annotations of the expressions in the continuous space. Meanwhile, the CAT weight vectors for the discrete emotions also form a discrete expression space in contrast to the continuous space in this work. Although in this work, the training data to construct the continuous expression space and the acted emotion corpus used to generate the discrete emotion CAT weight vectors are from the same speaker, it is not an essential requirement. Using the speaker and emotion factorization methods [11, 12], the discrete emotions from one speaker, can be used to annotate the continuous expressions from another speaker. Since the purpose of this work is to enable the users to define the expressions for the synthetic speech, it is important to evaluate if the synthetic speech contains the expressions consistent with the user s definition. For this, a listening test was designed which asked subjects to identify one speech sample (out of three), which best expressed a textually described targetexpression, i.e. first the listeners read some emotion description as reference, e.g. very angry and slightly fearful. Then, they listened to 3 synthesized speech sentences from different systems and were asked to indicate which speech file contains the same emotion as the reference emotion description. Synthetic speech samples from the following 3 systems were compared in this target-expression-identification test: system cont. used the expression from the continuous space according to the user s definition, system rand. used random expressions from the continuous space, and system neut. used the neutral speech which was assumed to be the most emotionless realisation. The reference emotion description was generated from the user defined emotion vector by quantizing the value of each emotion into 3 degrees: slightly, moderately and very. The evaluation sentences included 10 utterances with various expressions. The result is shown in figure 2. Figure 2: Percent preference for the 3 systems in the identifytarget-expression listening test Figure 2 indicates that in general, the synthetic speech based on the proposed method, contains more of the user defined emotions than both the random system and the neutral system. That says, the controllability of the TTS system with the continuous expression space is improved. To investigate the results further, the reference emotion descriptions were divided into two parts: strong expressions and weak expressions. Strong expressions include the reference emotion descriptions in which at least one of the emotion is set as very. Strong expressions are related to the condition that the user s desired emotions are very clear. All the reference emotion descriptions other than strong expressions are defined as weak emotions which are related to the condition that the user s desired emotions are ambiguous. Figure 2 shows that, when the expression definition is a strong one, the synthetic speech reflects the user s definition clearly. However, when the expression definition is weak, the expressions in the synthetic speech are more ambiguous. Another listening test using paragraphs was conducted to evaluate the expressiveness of synthetic speech. In this preference test, the expressive TTS system with the controllable continuous expression space (system cont. ) was compared to two contrast systems. The first one is the TTS system based on the discrete expression space (system disc. ). The expressive synthetic speech was generated based on the continuous expressive space and the discrete expressive space separately, given the user defined expression description. The second contrast system randomly selects the expressions of the training utterances from the continuous space (system rand. ). It represents the performance of the continuous expression space without the controllability. Thus, through this comparison, the effectiveness of enabling the controllability for the continuous expression space can be evaluated. The preference test included 12 paragraphs with an average of 4 sentences per paragraph. The listeners listened to two paragraphs from different systems and were asked to indicate which of two English speech files expressed an appropriate emotion for the content of the paragraph. The results are shown in table 1. Table 1: Preference test for paragraph reading cont. disc. rand. p 64.6% 35.4% 0.022 56.0% 44.0% 0.124 The results in table 1 show that the TTS system based on the controllable continuous expression space achieved a better performance than both the random system and the system based on the discrete expression space. This indicates that based on the proposed method, the continuous expression space becomes controllable, similar as the discrete space. Meanwhile, more expressive speech can be generated due to the richer expressive information in the continuous space. 5. Conclusion The continuous expression space contains very rich expressive information. However, the expressive information is complex and highly diverse. Thus it is hard for the user to access the desired expressions from the continuous expression space to synthesise expressive speech. On the other hand, the discrete expression space is highly controllable. However, the range of the expressions in the discrete space is limited. This work presented a method to enable the controllability for the continuous expression space. The proposed method automatically annotates the expressions in the continuous space using complex emotion labels which can describe the rich expressive information. These annotations are used as index to access the expressions in the continuous expression space. This way, users can define their desired expressions for a sentence and find the best matched expressions from the continuous space for synthesis. Experimental results indicate that based on the proposed method, the continuous expression space can be as controllable as the discrete space. Meanwhile more expressive synthetic speech can be generated compared to the discrete expression space. 2915
6. References [1] M. Tachibana, J. Yamagishi, T. Masuko, and T. Kobayashi, Speech synthesis with various emotional expressions and speaking styles by style interpolation and morphing, IEICE Trans. on information and systems, vol. 88, no. 11, pp. 2484 2491, 2005. [2] J. Yamagishi, K. Onishi, T. Masuko, and T. Kobayashi, Acoustic modeling of speaking styles and emotional expressions in HMMbased speech synthesis, IEICE Trans. on information and systems, vol. E88-D, pp. 503 509, 2005. [3] J. Yamagishi, T. Kobayashi, M. Tachibana, K. Ogata, and Y. Nakano, Model adaptation approach to speech synthesis with diverse voices and styles, in Proc. of ICASSP, 2007. [4] F. Eyben, S. Buchholz, N. Braunschweiler, J. Latorre, V. Wan, M. J. F. Gales, and K. Knill, Unsupervised clustering of emotion and voice styles for expressive TTS, in Proc. of ICASSP, 2012. [5] L. Chen, M. J. F. Gales, V. Wan, J. Latorre, and M. Akamine, Exploring rich expressive information from audiobook data using cluster adaptive training, in Proc. of INTERSPEECH, 2012. [6] Y. Zhao, D. Peng, L. Wang, M. Chu, Y. Chen, P. Yu, and J. Guo, Constructing stylistic synthesis databases from audio books, in Proc. of Interspeech, 2006. [7] K. Prahallad, A. Toth, and A. Black, Automatic building of synthetic voices from large multi-paragraph speech databases, in Proc. of Interspeech, 2007. [8] N. Braunschweiler, M. J. F. Gales, and S. Buchholz, Lightly supervised recognition for automatic alignment of large coherent speech recordings, in Proc. of Interspeech, 2010, pp. 2222 2225. [9] L. Chen, M. J. F. Gales, N. Braunschweiler, M. Akamine, and K. Knill, Integrated automatic expression prediction and speech synthesis from text, in Proc. of ICASSP, 2013. [10] H. Zen, N. Braunschweiler, S. Buchholz, M. J. F. Gales, K. Knill, S. Krstulovic, and J. Latorre, Statistical parametric speech synthesis based on speaker and language factorization, IEEE Trans. on Audio Speech and Language Processing, vol. 20, no. 5, 2012. [11] L. Chen and N. Braunschweiler, Unsupervised speaker and expression factorization for multi-speaker expressive synthesis of ebooks, in Proc. of INTERSPEECH, 2013. [12] J. Latorre, V. Wan, M. J. F. Gales, L. Chen, K. Chin, K. Knill, and M. Akamine, Speech factorization for HMM-TTS based on cluster adaptive training, in Proc. of Interspeech, 2012. 2916