Natural Language Processing

Transcription

1 Lecture 18 Natural Language Processing Marco Chiarandini Department of Mathematics & Computer Science University of Southern Denmark Slides by Dan Klein at Berkeley

2 Course Overview Introduction Artificial Intelligence Intelligent Agents Search Uninformed Search Heuristic Search Uncertain knowledge and Reasoning Probability and Bayesian approach Bayesian Networks Hidden Markov Chains Kalman Filters Learning Supervised Decision Trees, Neural Networks Learning Bayesian Networks Unsupervised EM Algorithm Reinforcement Learning Games and Adversarial Search Minimax search and Alpha-beta pruning Multiagent search Knowledge representation and Reasoning Propositional logic First order logic Inference Plannning 2

3 Outline Statistical MT Rule-based MT 3

4 : Sequential data 4

5 : Filtering 5

6 : State Trellis State trellis: graph of states and transitions over time Each arc represents some transition x t 1 x t Each arc has weight Pr(x t x t 1 ) Pr(e t x t ) Each path is a sequence of states The product of weights on a path is the seq s probability Can think of the Forward (and now Viterbi) algorithms as computing sums of all paths (best paths) in this graph 6

7 : Forward/Viterbi 7

8 : Particle Filtering Particles: track samples of states rather than an explicit distribution 8

9 Natural Language years ago humans started to speak years ago humans started to write Machines process natural language to: acquire information communicate with humans 9

10 Natural Language Processing Speech technologies Automatic speech recognition (ASR) Text-to-speech synthesis (TTS) Dialog systems Language processing technologies Machine translation Information extraction Web search, question answering Text classification, spam filtering, etc. 10

11 Outline Statistical MT Rule-based MT 11

12 Digitalizing Speech Speech input is an acoustic wave form 12

13 Spectral Analysis 13

14 Acoustic Feature Sequence 14

15 State Space Pr(E X ) encodes which acoustic vectors are appropriate for each phoneme (each kind of sound) Pr(X X ) encodes how sounds can be strung together We will have one state for each sound in each word From some state x, can only: Stay in the same state (e.g. speaking slowly) Move to the next position in the word At the end of the word, move to the start of the next word We build a little state graph for each word and chain them together to form our state space X 15

16 HMM for speech 16

17 Transition with Bigrams 17

18 Decoding While there are some practical issues, finding the words given the acoustics is an HMM inference problem We want to know which state sequence x 1:T is most likely given the evidence e 1:T : From the sequence x, we can simply read off the words 18

19 Outline Statistical MT Rule-based MT 19

20 Fundamental goal: analyze and process human language, broadly, robustly, accurately... End systems that we want to build: Ambitious: speech recognition, machine translation, information extraction, dialog interfaces, question answering... Modest: spelling correction, text categorization, language recognition, genre classification. 20

21 Language Models Language defined by a sequence of strings and rules called grammars. Formal languages also need semantics that define meaning. Natural Languages: 1. not definitive: is disagreement with grammar rules Not to be invited is sad To be not invited is sad 2. ambiguous: Entire store 25% off I will bring my bike tomorrow if it looks nice in the morning. 3. large and constantly changing 21

22 n-gram sequence of n characters or sequence of n words, syllables n-gram models: define probability distributions for these sequences n-gram model is defined as a Markov chain of order n 1. For a trigram: p(c i c 1:i 1 ) = p(c i c i 2:i 1 ) N N p(c 1:N ) = Pr(c i c 1:i 1 ) = Pr(c i c i 2:i 1 ) i=1 i=1 100 chars millions of entries with words even worse Corpus body of text 22

23 Language identification Learned from corpus: p(c i c i 2:i 1, l) Most probable language: l = argmax l p(l c 1:N ) = argmax l p(l)p(c 1:N l) (Bayes) N = argmax l p(l) p(c i c i 2:i 1, l) (Markov property) i=1 Computers can reach 99% accuracy 23

24 Rough translation: gives the main point but contains errors Pre-edited translation: original text written in constrained language easier to translate automatically Restricted-source translation: fully automatic but only on technical content as e.g. weather forecast 24

25 Systems Very simplified there are three types of machine translation Statistical machine translation (SMT) learn relational dependencies of features such as grams, lemmas, etc. Requires large data sets Example: google translate Relatively easy to implement Rule-based machine translation (RBMT) use grammatical rules and language constructions to analyze syntax and semantics Use moderate size data sets Long development time and expertise Hybrid machine translation either construct from RBMT and use SMT to post-process and optimize the result Or use grammatical rules to derive further features to then be fed in the statistical learning machine New direction of research. 25

26 Brief History 26

27 Interlingual model: the source language, i.e. the text to be translated is transformed into an interlingua, i.e., an abstract language-independent representation. The target language is then generated from the interlingua. Transfer model: the source language is transformed into an abstract, less language-specific representation. Linguistic rules which are specific to the language pair then transform the source language representation into an abstract target language representation and from this the target sentence is generated. Direct model: words are translated directly without passing through an additional representation. 27

28 Levels of Transfer Interlingua Semantics Attraction(NamedJohn, NamedMary, High) English Semantics Loves(John, Mary) French Semantics Aime(Jean, Marie) English Syntax S(NP(John), VP(loves, NP(Mary))) French Syntax S(NP(Jean), VP(aime, NP(Marie))) English Words John loves Mary French Words Jean aime Marie Vauquois pyramid 28

29 Levels of Transfer 29

30 The problem with dictionary look ups 30

31 Statistical machine translation Data driven MT 32

32 e sequence of strings in English f sequence of strings in French f = argmax f Pr(f e) = argmax f Pr(e f ) Pr(f ) Pr(e f ) learned from bilingual (parallel) corpus made of phrases seen before 33

33 with 100 French phrases for a 5-gram English there are different 5-gram and 5! reorderings. 34 e 1 e 2 e 3 e 4 e 5 There is a smelly wumpus sleeping in 2 2 f 1 f 3 f 2 f 4 f 5 Il y a un wumpus malodorant qui dort à 2 2 d 1 = 0 d 3 = -2 d 2 = +1 d 4 = +1 d 5 = 0 Given English sentence e find French sentence f : 1. break English e into phrases e 1,..., e n 2. e i choose the French f i : Pr(f i e i ) 3. choose a permutation of phrases f 1,..., f n f i choose distortion d i : num. of words that phrase f i has moved wrt f i 1 n Pr(f, d e) = Pr(f i e i ) Pr(d i ) i=1

34 Learn probabilities 1. Parallel corpus: parliamentary debates, web pages 2. Segment into sentences. Periods are good indicators with some care. 3. Align sentences. length of sentences is an indicator, landmarks another 4. Align phrases within sentence: iterative process,aggregation of evidence, no other pair appear so frequently in the corpus. Pr(f i e i ) 5. Extract distortions: count how often distortions appear in the corpus after phrase alignment (smoothing) 6. Improve estimates of Pr(f e) and Pr(d) with EM. 35

35 Learning to translate 36

36 An HMM model 37

37 Machine translation systems 39

38 Grammars Grammars: set of rules (from left to right) that describe how to form strings from the language s alphabet that are valid according to the language s syntax (Language generator). Parsing is the process of recognizing a string in natural languages by breaking it down to a set of symbols and analyzing each one against the grammar of the language, ie, determining whether the string belongs to the language or is grammatically incorrect. The result is a parse tree. context free grammars (see probabilistic context free grammars lexicalized probabilistic context free grammars 40

39 Parsing as search 41

40 Probabilistic Context Free Grammars 42

41 Hybrid Systems The translated sentence can be checked against a monolingual corpus. 43

42 Translate text from one language to another Recombines fragments of example translations Challenges: What fragments? [learning to translate] How to make efficient? [fast translation search] 44

43 After a first bubble now full speed in the sector In spite of the economical crisis 7% growth on world basis Commercial and technological focus Danish is a marginal language and existing systems cannot be applied reliably and search development in collaboration with research institutions (SDU, CBS, ASB) 45

44 Announcement Need for human resources, possibilties for thesis and individual study activities together with: Visual Interactive Syntax Learning project at the Institute for Language and Communication of SDU Eckhard Bick project leader If interested contact me. 46