Lecture 2: Mixing Compositional Semantics and Machine Learning

Transcription

1 Lecture 2: Mixing Compositional Semantics and Machine Learning Kyle Richardson April 14, 2016

2 Plan main paper: Liang and Potts 2015 (conceptual basis of class) secondary: Mooney 2007 (semantic parsing big ideas), Domingos 2012 (remarks about ML) 2

3 Classical Semantics vs. Statistical Semantics (caricature) Logical Semantics: Logic, algebra, set theory compositional analysis, beyond words, inference, brittle. Statistical Semantics: Optimization, algorithms, geometry distributional analysis, word-based, grounded, shallow. The types of approaches share the long-term vision of achieving deep natural language understanding... 3

4 Montague-style Compositional Semantics Principle of Compositionality: The meaning of a complex expression is a function of the meaning of its parts and the rules that combine them. Example: John studies. john John (λx.(study x)) studies 4

5 Montague-style Compositional Semantics Principle of Compositionality: The meaning of a complex expression is a function of the meaning of its parts and the rules that combine them. Example: John studies. john John (λx.(study x)) studies (λx.(study x))(john) (study john ) {True, False} john John (λx.(study x)) studies 4

6 A mini functional interpreter (python) Principle of Compositionality: The meaning of a complex expression is a function of the meaning of its parts and the rules that combine them. Example: John studies. john John (λx.(study x)) studies >>> students studying = set([ john, mary ]) >>> study = lambda x : x in students studying >>> fun application = lambda fun, val : fun(val) 5

7 A mini functional interpreter (python) Principle of Compositionality: The meaning of a complex expression is a function of the meaning of its parts and the rules that combine them. Example: John studies. john John (λx.(study x)) studies >>> students studying = set([ john, mary ]) >>> study = lambda x : x in students studying >>> fun application = lambda fun, val : fun(val) >>> fun application(study, bill ) ## What will we get? 5

8 A mini functional interpreter (python) Principle of Compositionality: The meaning of a complex expression is a function of the meaning of its parts and the rules that combine them. Example: John studies. john John (λx.(study x)) studies >>> students studying = set([ john, mary ]) >>> study = lambda x : x in students studying >>> fun application = lambda fun, val : fun(val) >>> fun application(study, bill ) ## What will we get? >>> False 5

9 A mini functional interpreter (python) Principle of Compositionality: The meaning of a complex expression is a function of the meaning of its parts and the rules that combine them. Example: John studies. john John (λx.(study x)) studies >>> students studying = set([ john, mary ]) >>> study = lambda x : x in students studying >>> fun application = lambda fun, val : fun(val) >>> fun application(study, mary ) ## What will we get? 6

10 A mini functional interpreter (python) Principle of Compositionality: The meaning of a complex expression is a function of the meaning of its parts and the rules that combine them. Example: John studies. john John (λx.(study x)) studies >>> students studying = set([ john, mary ]) >>> study = lambda x : x in students studying >>> fun application = lambda fun, val : fun(val) >>> fun application(study, mary ) ## What will we get? >>> True 6

11 Montague-style Compositional Semantics Principle of Compositionality: The meaning of a complex expression is a function of the meaning of its parts and the rules that combine them. Example: Bill does not study. bill Bill (λx.(study x)) study (λf.λx.(not (f x))) does not 7

12 Montague-style Compositional Semantics Principle of Compositionality: The meaning of a complex expression is a function of the meaning of its parts and the rules that combine them. Example: Bill does not study. bill Bill (λx.(study x)) study (λf.λx.(not (f x))) does not (λx.(not (study x)))(bill) bill (λx.(not (study x))) Bill (λf.λx.(not (f x))) does not (λx.(study x)) study 7

13 A mini functional interpreter (python) Principle of Compositionality: The meaning of a complex expression is a function of the meaning of its parts and the rules that combine them. Example: Bill does not study. bill Bill (λx.(study x)) study (λf.λx.(not (f x))) does not >>> students studying = set([ john, mary ]) >>> study = lambda x : x in students studying >>> fun application = lambda fun, val : fun(val) >>> neg = lambda F : (lambda x : not F(x)) 8

14 A mini functional interpreter (python) Principle of Compositionality: The meaning of a complex expression is a function of the meaning of its parts and the rules that combine them. Example: Bill does not study. bill Bill (λx.(study x)) study (λf.λx.(not (f x))) does not >>> students studying = set([ john, mary ]) >>> study = lambda x : x in students studying >>> fun application = lambda fun, val : fun(val) >>> neg = lambda F : (lambda x : not F(x)) >>> neg(study)( bill ) # True 8

15 A mini functional interpreter (python) Principle of Compositionality: The meaning of a complex expression is a function of the meaning of its parts and the rules that combine them. Example: Bill does not study. bill Bill (λx.(study x)) study (λf.λx.(not (f x))) does not >>> students studying = set([ john, mary ]) >>> study = lambda x : x in students studying >>> fun application = lambda fun, val : fun(val) >>> neg = lambda F : (lambda x : not F(x)) >>> neg(study)( bill ) # True >>> fun application(neg,study)( bill ) 8

16 A mini functional interpreter (python) Principle of Compositionality: The meaning of a complex expression is a function of the meaning of its parts and the rules that combine them. Example: Bill does not study. bill Bill (λx.(study x)) study (λf.λx.(not (f x))) does not >>> students studying = set([ john, mary ]) >>> study = lambda x : x in students studying >>> fun application = lambda fun, val : fun(val) >>> neg = lambda F : (lambda x : not F(x)) >>> neg(study)( bill ) # True >>> fun application(neg,study)( bill ) >>> fun application(fun application(neg,study), bill ) 8

17 A mini functional interpreter (python) Principle of Compositionality: The meaning of a complex expression is a function of the meaning of its parts and the rules that combine them. Example: Bill does not study. bill Bill (λx.(study x)) study (λf.λx.(not (f x))) does not >>> students studying = set([ john, mary ]) >>> study = lambda x : x in students studying >>> fun application = lambda fun, val : fun(val) >>> neg = lambda F : (lambda x : not F(x)) >>> neg(study)( bill ) # True >>> fun application(neg,study)( bill ) >>> fun application(fun application(neg,study), bill ) >>> neg(neg(sleep))( bill ) 8

18 Montague-style Compositional Semantics: What s needed Principle of Compositionality: The meaning of a complex expression is a function of the meaning of its parts and the rules that combine them. Example: Bill does not study. bill Bill (λx.(study x)) study (λf.λx.(not (f x))) does not Grammar rules for building syntactic structure. Interpretation rules to composing meaning. Decoding algorithm for generating structures 9

19 Montague-style Compositional Semantics: Issues Principle of Compositionality: The meaning of a complex expression is a function of the meaning of its parts and the rules that combine them. Example: Bill does not study. bill Bill (λx.(study x)) study (λf.λx.(not (f x))) does not Features and (Computational) Issues: compositional, provides a full analysis. supports further inferencing 10

20 Montague-style Compositional Semantics: Issues Principle of Compositionality: The meaning of a complex expression is a function of the meaning of its parts and the rules that combine them. Example: Bill does not study. bill Bill (λx.(study x)) study (λf.λx.(not (f x))) does not Features and (Computational) Issues: compositional, provides a full analysis. supports further inferencing issue: Does not provide an analysis of words (not grounded). 10

21 Montague-style Compositional Semantics: Issues Principle of Compositionality: The meaning of a complex expression is a function of the meaning of its parts and the rules that combine them. Example: Bill does not study. bill Bill (λx.(study x)) study (λf.λx.(not (f x))) does not Features and (Computational) Issues: compositional, provides a full analysis. supports further inferencing issue: issue: Does not provide an analysis of words (not grounded). Is brittle, cannot handle uncertainty. 10

22 Montague-style Compositional Semantics: Issues Principle of Compositionality: The meaning of a complex expression is a function of the meaning of its parts and the rules that combine them. Example: Bill does not study. bill Bill (λx.(study x)) study (λf.λx.(not (f x))) does not Features and (Computational) Issues: compositional, provides a full analysis. supports further inferencing issue: issue: issue: Does not provide an analysis of words (not grounded). Is brittle, cannot handle uncertainty. Says nothing about how the translation to logic works. 10

23 Statistical Approaches to Semantics Statistical semantics hypothesis: Statistical patterns of human word usage can be used to figure out what people mean Turney et al. (2010) corpus word-context matrix The furry dog is walking outside... furry walking shiny driving The shiny car is driving... dog A furry cat is walking around... cat A shiny bike is driving... car bike

24 Statistical Approaches to Semantics Statistical semantics hypothesis: Statistical patterns of human word usage can be used to figure out what people mean Turney et al. (2010) corpus word-context matrix furry walking shiny driving dog cat car bike

25 Example Tasks and Applications: Turney et al. (2010) Statistical semantic models are often used in downstream classification or clustering tasks/applications. Term-document matrices Document retrieval/clustering/classification. Question Answering and Retrieval. Essay scoring. Word-Context Matrices Word similarity/clustering/classification Word-sense disambiguation Automatic thesaurus generation/paraphrasing Pair-pair matrices Relational similarity/clustering/classification. Analogy comparison. 13

26 Statistical Approaches to Semantics Statistical semantics hypothesis: Statistical patterns of human word usage can be used to figure out what people mean Turney et al. (2010) corpus word-context matrix The furry dog is walking outside... furry walking shiny driving The shiny car is driving... dog A furry cat is walking around... cat A shiny bike is driving... car bike Features and Issues (caricature): Robust, requires little manual effort, grounded Can provide rich analysis of content words. 14

27 Statistical Approaches to Semantics Statistical semantics hypothesis: Statistical patterns of human word usage can be used to figure out what people mean Turney et al. (2010) corpus word-context matrix The furry dog is walking outside... furry walking shiny driving The shiny car is driving... dog A furry cat is walking around... cat A shiny bike is driving... car bike Features and Issues (caricature): Robust, requires little manual effort, grounded Can provide rich analysis of content words. issue: Hard to scale beyond words. 14

28 Statistical Approaches to Semantics Statistical semantics hypothesis: Statistical patterns of human word usage can be used to figure out what people mean Turney et al. (2010) corpus word-context matrix The furry dog is walking outside... furry walking shiny driving The shiny car is driving... dog A furry cat is walking around... cat A shiny bike is driving... car bike Features and Issues (caricature): Robust, requires little manual effort, grounded Can provide rich analysis of content words. issue: Hard to scale beyond words. issue: In general, hard to model logical operations, shallow. 14

29 Mixing compositional and statistical semantics Desiderata: Want a model of semantics that is robust, reflects real-word usage and learnable, but one that is also compositional. 15

30 Mixing compositional and statistical semantics Desiderata: Want a model of semantics that is robust, reflects real-word usage and learnable, but one that is also compositional. Generalization 15

31 Mixing compositional and statistical semantics Desiderata: Want a model of semantics that is robust, reflects real-word usage and learnable, but one that is also compositional. Generalization Logical semantics: generalize using composition and abstract recursive structures. 15

32 Mixing compositional and statistical semantics Desiderata: Want a model of semantics that is robust, reflects real-word usage and learnable, but one that is also compositional. Generalization Logical semantics: generalize using composition and abstract recursive structures. Machine Learning (classification): learns generalizations through real-world examples (e.g. target input-output) 15

33 Mixing compositional and statistical semantics Desiderata: Want a model of semantics that is robust, reflects real-word usage and learnable, but one that is also compositional. Generalization Logical semantics: generalize using composition and abstract recursive structures. Machine Learning (classification): learns generalizations through real-world examples (e.g. target input-output) Bridge: get our learning to target compositional structures. 15

34 A simple model: Liang and Potts Model: a simple discriminative learning framework. compositional model: (semantic) context-free grammar. learning model: linear classification and first-order optimization. 16

35 Compositional Model: Linguistic Objects: < u, s, d > u: utterance s: semantic representation (symbolized as ˆuˆ) d: denotation (symbolized as s ) 17

36 Compositional Model: Linguistic Objects: < u, s, d > u: utterance s: semantic representation (symbolized as ˆuˆ) d: denotation (symbolized as s ) Example: < seven minus five, (- 7 5), 2 > 17

37 Compositional Model: Linguistic Objects: < u, s, d > u: s: d: utterance semantic representation (symbolized as ˆuˆ) denotation (symbolized as s ) Example: < seven minus five, (- 7 5), 2 > < minus times, (* (- 2 2) 2), 0 > 17

38 Compositional Model: Linguistic Objects: < u, s, d > u: s: d: utterance semantic representation (symbolized as ˆuˆ) denotation (symbolized as s ) Example: < seven minus five, (- 7 5), 2 > < minus times, (* (- 2 2) 2), 0 > semantic parsing: u s 17

39 Compositional Model: Linguistic Objects: < u, s, d > u: s: d: utterance semantic representation (symbolized as ˆuˆ) denotation (symbolized as s ) Example: < seven minus five, (- 7 5), 2 > < minus times, (* (- 2 2) 2), 0 > semantic parsing: u s interpretation: s d 17

40 Computational Modeling: The full picture Standard processing pipeline input List samples that contain every major element Semantic Parsing sem (FOR EVERY X / MAJORELT : T; (FOR EVERY Y / SAMPLE : (CONTAINS Y X); (PRINTOUT Y))) Knowledge Representation Interpretation world sem ={S10019,S10059,...} Lunar QA system (Woods (1973)) 18

41 Compositional Model: Context-free grammar provides the background grammar and interpretation rules 19

42 Compositional Model: Context-free grammar provides the background grammar and interpretation rules example: u = times plus three N: (plus (mult 2 2) 3) N : (mult 2 2) R : plus N : 3 R : mult plus three times 20

43 Compositional Model: Context-free grammar provides the background grammar and interpretation rules example: u = times plus three N: (plus (mult 2 2) 3) N : (mult 2 2) R : plus N : 3 R : mult plus three times >>> plus = lambda x,y : x + y >>> mult = lambda x,y : x * y 20

44 Compositional Model: Context-free grammar provides the background grammar and interpretation rules example: u = times plus three N: (plus (mult 2 2) 3) N : (mult 2 2) R : plus N : 3 R : mult plus three times >>> plus = lambda x,y : x + y >>> mult = lambda x,y : x * y >>> plus(2,2) # 4 20

45 Compositional Model: Context-free grammar provides the background grammar and interpretation rules example: u = times plus three N: (plus (mult 2 2) 3) N : (mult 2 2) R : plus N : 3 R : mult plus three times >>> plus = lambda x,y : x + y >>> mult = lambda x,y : x * y >>> plus(plus(2,3),2) # 7 21

46 Compositional Model: Context-free grammar provides the background grammar and interpretation rules example: u = times plus three N: (plus (mult 2 2) 3) N : (mult 2 2) R : plus N : 3 R : mult plus three times >>> plus = lambda x,y : x + y >>> mult = lambda x,y : x * y >>> plus(mult(2,2),3) # 7 22

47 Compositional Model: Components Components: Grammar rules for building syntactic structure. 23

48 Compositional Model: Components Components: Grammar rules for building syntactic structure. Interpretation rules to composing meaning. 23

49 Compositional Model: Components Components: Grammar rules for building syntactic structure. Interpretation rules to composing meaning. Decoding algorithm for generating structures (later lecture) 23

50 Compositional Model: Components Components: Grammar rules for building syntactic structure. Interpretation rules to composing meaning. Decoding algorithm for generating structures (later lecture) Rule extraction (later lecture) 23

51 Compositional Model: Components Components: Grammar rules for building syntactic structure. Interpretation rules to composing meaning. Decoding algorithm for generating structures (later lecture) Rule extraction (later lecture) Issues: 23

52 Compositional Model: Components Components: Grammar rules for building syntactic structure. Interpretation rules to composing meaning. Decoding algorithm for generating structures (later lecture) Rule extraction (later lecture) Issues: example: u = times plus three N: (plus (mult 2 2) 3) N: (plus (plus 2 2) 3) N : (mult 2 2) R : plus N : 3 N : (plus 2 2) R : plus N : 3 R : mult plus three R : plus plus three times times 23

53 Compositional Model: Components Components: Grammar rules for building syntactic structure. Interpretation rules to composing meaning. Decoding algorithm for generating structures (later lecture) Issues: example: u = times plus three N: (plus (mult 2 2) 3) N: (mult (plus 2 2) 3) N : (mult 2 2) R : plus N : 3 N : (plus 2 2) R : mult N : 3 R : mult plus three R : mult plus three times times 24

54 Compositional Model: Components Components: Grammar rules for building syntactic structure. Interpretation rules to composing meaning. Decoding algorithm for generating structures (later lecture) Issues: example: u = times plus three N: (plus (mult 2 2) 3) N: (mult 2 (plus 2 3)) N : (mult 2 2) R : plus N : 3 R : mult N : (plus 2 3) R : mult plus three times R: plus N : 3 times plus three 25

55 Learning Model Goal: Helps us learn the correct derivations and handle uncertainty (word mappings, composition). Classifier: a system that inputs a vector of discrete and/or continuous feature values and outputs a single discrete value, the class. Domingos (2012). 26

56 Learning Model Goal: Helps us learn the correct derivations and handle uncertainty (word mappings, composition). Classifier: a system that inputs a vector of discrete and/or continuous feature values and outputs a single discrete value, the class. Domingos (2012). 26

57 Learning Model Goal: Helps us learn the correct derivations and handle uncertainty (word mappings, composition). Classifier: a system that inputs a vector of discrete and/or continuous feature values and outputs a single discrete value, the class. Domingos (2012). Components 26

58 Learning Model Goal: Helps us learn the correct derivations and handle uncertainty (word mappings, composition). Classifier: a system that inputs a vector of discrete and/or continuous feature values and outputs a single discrete value, the class. Domingos (2012). Components training data D = {(xi, y i ) i...n} 26

59 Learning Model Goal: Helps us learn the correct derivations and handle uncertainty (word mappings, composition). Classifier: a system that inputs a vector of discrete and/or continuous feature values and outputs a single discrete value, the class. Domingos (2012). Components training data D = {(xi, y i ) i...n} feature representation of data 26

60 Learning Model Goal: Helps us learn the correct derivations and handle uncertainty (word mappings, composition). Classifier: a system that inputs a vector of discrete and/or continuous feature values and outputs a single discrete value, the class. Domingos (2012). Components training data D = {(xi, y i ) i...n} feature representation of data scoring and objective function 26

61 Learning Model Goal: Helps us learn the correct derivations and handle uncertainty (word mappings, composition). Classifier: a system that inputs a vector of discrete and/or continuous feature values and outputs a single discrete value, the class. Domingos (2012). Components training data D = {(xi, y i ) i...n} feature representation of data scoring and objective function optimization procedure 26

62 Training data Goal: Find the correct derivations and output using our compositional model 27

63 Training data Goal: Find the correct derivations and output using our compositional model Logical forms (more information) (u = minus times, s = (* (- 2 2) 2)) 27

64 Training data Goal: Find the correct derivations and output using our compositional model Logical forms (more information) (u = minus times, s = (* (- 2 2) 2)) Denotations (less information) (u = minus times, r = 0) 27

65 Training data Goal: Find the correct derivations and output using our compositional model Logical forms (more information) (u = minus times, s = (* (- 2 2) 2)) Denotations (less information) (u = minus times, r = 0) Weakly Supervised: the learner. In both cases, details are still hidden from 27

66 Learning from Semantic Representations example: ( times plus three,(plus (mult 2 2) 3)) N: (plus (mult 2 2) 3) N: (plus (plus 2 2) 3) N : (mult 2 2) R : plus N : 3 N : (plus 2 2) R : plus N : 3 R : mul plus three R : plus plus three times times 28

67 Learning from Semantic Representations example: ( times plus three,(plus (mult 2 2) 3)) N: (plus (mult 2 2) 3) N: (plus (plus 2 2) 3) N : (mult 2 2) R : plus N : 3 N : (plus 2 2) R : plus N : 3 R : mul plus three R : plus plus three times times Trade off: More information (good) but more annotation (bad) 28

68 Learning from Denotations example: ( times plus three,7) N: (plus (mult 2 2) 3) N: (plus (plus 2 2) 3) N : (mult 2 2) R : plus N : 3 N : (plus 2 2) R : plus N : 3 R : mul plus three R : plus plus three times times 29

69 Learning from Denotations example: ( times plus three,7) N: (plus (mult 2 2) 3) N: (plus (plus 2 2) 3) N : (mult 2 2) R : plus N : 3 N : (plus 2 2) R : plus N : 3 R : mul plus three R : plus plus three times times Trade off: Less annotation (good) but less information (maybe bad) 29

70 Weak Supervision Goal: Find the correct derivations and output using our compositional model Logical forms (more information) (u = minus times, s = (* (- 2 2) 2)) Denotations (less information) (u = minus times, r = 0) Current learning methods for NLP require annotating large corpora with supervisory information...[e.g. pos tags, syntactic parse trees, semantic role labels]... Building such corpora is an expensive, arduous task. As one moves towards deeper semantic analysis the annotation task becomes increasingly more difficult and complex. Mooney (2008) 30

71 Feature Representations: General Remark At the end of the day, some machine learning projects succeed and fail. What makes the difference? Easily the most important factor is the features used. Domingos (2012) 31

72 Feature selection and overfitting What if the knowledge and data we have are not sufficient to completely determine the correct classifier? Then we run the risk of just hallucinating a classifier (or parts of it) that is not grounded in reality.. This problem is called overfitting. Domingos (2012) Bias: Tendency to consistently learn the wrong thing. Variance: Tendency to learn random things irrespective of the real signal. 32

73 Good vs. Bad Feature Selection 33

74 Feature Extraction Example input: x = times plus three. y 1 = N: (plus (mult 2 2) 3) y 2 = N: (plus (plus 2 2) 3) N : (mult 2 2) R : plus N : 3 N : (plus 2 2) R : plus N : 3 R : mult plus three R : plus plus three times times φ(x,y 1 ) = R : mult [ times ] 1 R : plus [ plus ] 1 top [ R : plus ] 1... φ(x,y 2 ) = R : plus [ times ] 1 R : plus [ plus ] 1 top [ R : plus ]

75 Scoring Function (Linear) Score Function Score w (x,y) = w φ(x, y) = d j=1 w jφ(x, y) 35

76 Scoring Function (Linear) Score Function Score w (x,y) = w φ(x, y) = d j=1 w jφ(x, y) weight vector w = [w 1 = 0.1 w 2 = 0.2 w 3 = ] 35

77 Scoring Function (Linear) Score Function Score w (x,y) = w φ(x, y) = d j=1 w jφ(x, y) weight vector w = [w 1 = 0.1 w 2 = 0.2 w 3 = ] φ(x,y 2 ) = w 1 R : plus [ times ] 1 w 2 R : plus [ plus ] 1 w 3 top [ R : plus ] 1... score w (x, y 2 ) = w φ(x, y 2 ) = ( ) + ( ) + ( ) 35

78 Scoring Function (Linear) Score Function Score w (x,y) = w φ(x, y) = d j=1 w jφ(x, y) weight vector w = [w 1 = 0.1 w 2 = 0.2 w 3 = ] prediction: arg-max y Y Score w (x, y) 36

79 Objectives: What do we want to learn? (informal) General Idea: want to learn a model (or weight vector) that can distinguish correct and incorrect derivations. y 1 = N: (plus (mult 2 2) 3) y 2 = N: (plus (plus 2 2) 3) N : (mult 2 2) R : plus N : 3 N : (plus 2 2) R : plus N : 3 R : mult plus three R : plus plus three times times φ(x,y 1 ) = R : mult [ times ] 1 R : plus [ plus ] 1 top [ R : plus ] 1... φ(x,y 2 ) = R : plus [ times ] 1 R : plus [ plus ] 1 top [ R : plus ]

80 Objectives: What do we want to learn? (informal) General Idea: want to learn a model (or weight vector) that can distinguish correct and incorrect derivations. y 1 = N: (plus (mult 2 2) 3) N: (mult 2 (plus 2 3)) N : (mult 2 2) R : plus N : 3 R : mult N : (plus 2 3) R : mult plus three times R: plus N : 3 times plus three φ(x,y 1 ) = R : mult [ times ] 1 R : plus [ plus ] 1 plus [ R : mult ] 1... φ(x,y 2 ) = R : plus [ times ] 1 R : plus [ plus ] 1 mult [ R : plus ]

81 Objectives: What do we want to learn? (formal) hinge loss: (learning from logical forms) min w R d n max y Y [Score w (x, y )+c(y, y )] Score w (x, y) (x,y) D ( minus times, s = (* (- 2 2) 2)) 39

82 Objectives: What do we want to learn? (formal) hinge loss: (learning from logical forms) min w R d n (x,y) D max y Y [Score w (x, y )+c(y, y )] Score w (x, y) ( minus times, s = (* (- 2 2) 2)) In English: select parameters that minimize the cumulative loss over the training data. 39

83 Objectives: What do we want to learn? (formal) hinge loss: (learning from logical forms) min w R d n (x,y) D max y Y [Score w (x, y )+c(y, y )] Score w (x, y) ( minus times, s = (* (- 2 2) 2)) In English: select parameters that minimize the cumulative loss over the training data. Missing: A decoding algorithm for generating Y (not trivial, Y might be very large). 39

84 Optimization: How do I achieve this objective? Stochastic gradient descent: An online learning and optimization algorithm (more about this in future lectures). 40

85 Optimization: Illustration 41

86 Learning Model Components training data: D = {(xi, y i ) i...n} 42

87 Learning Model Components training data: D = {(xi, y i ) i...n} feature representation of data 42

88 Learning Model Components training data: D = {(xi, y i ) i...n} feature representation of data scoring and objective function 42

89 Learning Model Components training data: D = {(xi, y i ) i...n} feature representation of data scoring and objective function optimization procedure 42

90 Learning Model Components training data: D = {(xi, y i ) i...n} feature representation of data scoring and objective function optimization procedure Important Ideas What kind of data do we learn from? (differs quite a bit) What kind of features do we need? 42

91 Experimentation and Evaluation Training Set: Test Set: A portion of the data to train model on. An unseen portion of the data to evaluate on. Dev Set : (optional) An unseen portion of the data for analysis, tuning hyper parameters,.. 43

92 Experimentation and Evaluation Training Set: Test Set: A portion of the data to train model on. An unseen portion of the data to evaluate on. Dev Set : (optional) An unseen portion of the data for analysis, tuning hyper parameters,.. Evaluation1: Given unseen examples, how often does my model produce the correct output semantic representation? Evaluation2: Given unseen examples, how often does my model produce the correct output answer? 43

93 Conclusions and Take Aways Presented a simple model that mixes machine learning and compositional semantics. Conceptually describes most of the work in this class. Technically describes many of the models we will use. Fundamental Problem: Which semantics representations do we use, and what do we learn from? 44

94 Conclusions and Take Aways Presented a simple model that mixes machine learning and compositional semantics. Conceptually describes most of the work in this class. Technically describes many of the models we will use. Fundamental Problem: Which semantics representations do we use, and what do we learn from? Question: Does this particular actually work? 44

95 Conclusions and Take Aways Presented a simple model that mixes machine learning and compositional semantics. Conceptually describes most of the work in this class. Technically describes many of the models we will use. Fundamental Problem: Which semantics representations do we use, and what do we learn from? Question: Does this particular actually work? Yes! Liang et al. (2011) (lecture 5), Berant et al. (2013); Berant and Liang (2014) (presentation papers) 44

96 Roadmap Lecture 2: Lecture 3: Lecture 4: Lecture 5: rule extraction, decoding (parsing perspective) rule extraction, decoding (MT perspective) structured classification and prediction. grounded learning (might skip). 45

97 References I Berant, J., Chou, A., Frostig, R., and Liang, P. (2013). Semantic parsing on Freebase from question-answer pairs. In in Proceedings of EMNLP-2013, pages Berant, J. and Liang, P. (2014). Semantic parsing via paraphrasing. In ACL (1), pages Domingos, P. (2012). A few useful things to know about machine learning. Communications of the ACM, 55(10): Liang, P., Jordan, M. I., and Klein, D. (2011). Learning dependency-based compositional semantics. In Proceedings of ACL-11, pages Mooney, R. (2008). Learning to connect language and perception. In Proceedings of AAAI Turney, P. D., Pantel, P., et al. (2010). From frequency to meaning: Vector space models of semantics. Journal of artificial intelligence research, 37(1): Woods, W. A. (1973). Progress in natural language understanding: an application to lunar geology. In Proceedings of the June 4-8, 1973, National Computer Conference and Exposition, pages