Text Classification and Sentiment Analysis

Transcription

1 Text Classification and Sentiment Analysis Muhammad Atif Qureshi National University of Ireland, Galway and University of Milano-Bicocca, Italy

2 Is this spam?

3 Who wrote which Federalist papers? : anonymous essays try to convince New York to ra1fy U.S Cons1tu1on: Jay, Madison, Hamilton. Authorship of 12 of the lelers in dispute 1963: solved by Mosteller and Wallace using Bayesian methods James Madison Alexander Hamilton

4 Male or female author? 1. By 1925 present- day Vietnam was divided into three parts under French colonial rule. The southern region embracing Saigon and the Mekong delta was the colony of Cochin- China; the central area with its imperial capital at Hue was the protectorate of Annam 2. Clara never failed to be astonished by the extraordinary felicity of her own name. She found it hard to trust herself to the mercy of fate, which had managed over the years to convert her greatest shame into one of her greatest assets S. Argamon, M. Koppel, J. Fine, A. R. Shimoni, Gender, Genre, and Wri1ng Style in Formal WriLen Texts, Text, volume 23, number 3, pp

5 Posi8ve or nega8ve movie review? unbelievably disappoin1ng Full of zany characters and richly applied sa1re, and some great plot twists this is the greatest screwball comedy ever filmed It was pathe1c. The worst part about it was the boxing scenes. 5

6 What is the subject of this ar8cle? 6 MEDLINE Article? MeSH Subject Category Hierarchy Antogonists and Inhibitors Blood Supply Chemistry Drug Therapy Embryology Epidemiology

7 Text Classifica8on Assigning subject categories, topics, or genres Spam detec1on Authorship iden1fica1on Age/gender iden1fica1on Language Iden1fica1on Sen1ment analysis

8 Text Classifica8on: defini8on Input: a document d a fixed set of classes C = {c 1, c 2,, c J } Output: a predicted class c C

9 Classifica8on Methods: Hand- coded rules Rules based on combina1ons of words or other features spam: black- list- address OR ( dollars AND have been selected ) Accuracy can be high If rules carefully refined by expert But building and maintaining these rules is expensive

10 Classifica8on Methods: Supervised Machine Learning Input: a document d a fixed set of classes C = {c 1, c 2,, c J } A training set of m hand- labeled documents (d 1,c 1 ),...,(d m,c m ) Output: a learned classifier γ:d à c 10

11 Classifica8on Methods: Supervised Machine Learning Any kind of classifier Naïve Bayes Logis1c regression Support- vector machines k- Nearest Neighbors

12 Text Classification and Naïve Bayes The Task of Text Classifica1on

13 Text Classification and Naïve Bayes Naïve Bayes (I)

14 Naïve Bayes Intui8on Simple ( naïve ) classifica1on method based on Bayes rule Relies on very simple representa1on of document Bag of words

15 The bag of words representa8on I love this movie! It's sweet, but with satirical humor. The dialogue is great and the adventure scenes are fun It manages to be whimsical and romantic while laughing at the conventions of the fairy tale genre. I would recommend it to just about anyone. I've seen it several times, and I'm always happy to see it again whenever I have a friend who hasn't seen it yet.! γ( )=c

16 The bag of words representa8on I love this movie! It's sweet, but with satirical humor. The dialogue is great and the adventure scenes are fun It manages to be whimsical and romantic while laughing at the conventions of the fairy tale genre. I would recommend it to just about anyone. I've seen it several times, and I'm always happy to see it again whenever I have a friend who hasn't seen it yet.! γ( )=c

17 The bag of words representa8on: using a subset of words x love xxxxxxxxxxxxxxxx sweet xxxxxxx satirical xxxxxxxxxx xxxxxxxxxxx great xxxxxxx xxxxxxxxxxxxxxxxxxx fun xxxx xxxxxxxxxxxxx whimsical xxxx romantic xxxx laughing xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxx recommend xxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xx several xxxxxxxxxxxxxxxxx xxxxx happy xxxxxxxxx again xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxx! γ( )=c

18 The bag of words representa8on great! 2! love! 2! γ( )=c recommend! 1! laugh! 1! happy! 1!...!...!

19 Bag of words for document classifica8on Test document parser! language! label! translation!! Machine Learning! learning! training! algorithm! shrinkage! network...! NLP! parser! tag! training! translation! language...!? Garbage! Collection! garbage! collection! memory! optimization! region...! Planning! planning! temporal! reasoning! plan! language...! GUI!...!

20 Text Classification and Naïve Bayes Naïve Bayes (I)

21 Text Classification and Naïve Bayes Formalizing the Naïve Bayes Classifier

22 Bayes Rule Applied to Documents and Classes For a document d and a class c P(c d) = P(d c)p(c) P(d)

23 Naïve Bayes Classifier (I) c MAP = argmax c!c = argmax c!c = argmax c!c P(c d) P(d c)p(c) P(d) P(d c)p(c) MAP is maximum a posteriori = most likely class Bayes Rule Dropping the denominator

24 Naïve Bayes Classifier (II) c MAP = argmax c!c P(d c)p(c) = argmax c!c P(x 1, x 2,, x n c)p(c) Document d represented as features x1..xn

25 Naïve Bayes Classifier (IV) c MAP = argmax c!c P(x 1, x 2,, x n c)p(c) O( X n C ) parameters Could only be es1mated if a very, very large number of training examples was available. How often does this class occur? We can just count the relative frequencies in a corpus

26 Mul8nomial Naïve Bayes Independence Assump8ons P(x 1, x 2,, x n c) Bag of Words assump8on: Assume posi1on doesn t maler Condi8onal Independence: Assume the feature probabili1es P(x i c j ) are independent given the class c. P(x 1,, x n c) = P(x 1 c) P(x 2 c) P(x 3 c)... P(x n c)

27 Mul8nomial Naïve Bayes Classifier c MAP = argmax c!c c NB = argmax c!c P(x 1, x 2,, x n c)p(c) " x!x P(c j ) P(x c)

28 Applying Mul8nomial Naive Bayes Classifiers to Text Classifica8on positions all word posi1ons in test document c NB = argmax c j!c " i! positions P(c j ) P(x i c j )

29 Text Classification and Naïve Bayes Formalizing the Naïve Bayes Classifier

30 Text Classification and Naïve Bayes Naïve Bayes: Learning

31 Sec.13.3 Learning the Mul8nomial Naïve Bayes Model First alempt: maximum likelihood es1mates simply use the frequencies in the data ˆP(c j ) = doccount(c = c j ) N doc ˆP(w i c j ) = count(w i,c j ) count(w,c j ) " w!v

32 Parameter es8ma8on ˆP(w i c j ) = count(w i,c j ) count(w,c j ) " w!v frac1on of 1mes word w i appears among all words in documents of topic c j Create mega- document for topic j by concatena1ng all docs in this topic Use frequency of w in mega- document

33 Sec.13.3 Problem with Maximum Likelihood What if we have seen no training documents with the word fantas&c and classified in the topic posi8ve (thumbs- up)? ˆP("fantastic" positive) = count("fantastic", positive) " count(w, positive) Zero probabili1es cannot be condi1oned away, no maler the other evidence! w!v c MAP = argmax c ˆP(c)! ˆP(xi c) i = 0

34 Laplace (add- 1) smoothing for Naïve Bayes ˆP(w i c) = = " w!v # % $ count(w i, c)+1 count(w, c) +1 ( ) ) count(w i, c)+1 & " count(w, c) ( + V ' w!v

35 Mul8nomial Naïve Bayes: Learning From training corpus, extract Vocabulary Calculate P(c j ) terms For each c j in C do docs j all docs with class =c j docs P(c j )! j total # documents Calculate P(w k c j ) terms Text j single doc containing all docs j For each word w k in Vocabulary n k # of occurrences of w k in Text j n P(w k c j )! k +! n +! Vocabulary

36 Laplace (add- 1) smoothing: unknown words Add one extra word to the vocabulary, the unknown word w u count(w ˆP(w u c) = u,c)+1 # & % " count(w, c) ( + V +1 $ w!v ' 1 = # & % " count(w, c) ( + V +1 $ ' w!v

37 Text Classification and Naïve Bayes Naïve Bayes: Learning

38 Text Classification and Naïve Bayes Naïve Bayes: Rela1onship to Language Modeling

39 Genera8ve Model for Mul8nomial Naïve Bayes c=china X 1 =Shanghai X 2 =and X 3 =Shenzhen X 4 =issue X 5 =bonds 39

40 Naïve Bayes and Language Modeling Naïve bayes classifiers can use any sort of feature URL, address, dic1onaries, network features But if, as in the previous slides We use only word features we use all of the words in the text (not a subset) Then 40 Naïve bayes has an important similarity to language modeling.

41 Sec Each class = a unigram language model Assigning each word: P(word c) Assigning each sentence: P(s c)=π P(word c) 0.1 I 0.1 love 0.01 this 0.05 fun 0.1 film Class pos I love this fun film P(s pos) =

42 Sec Naïve Bayes as a Language Model Which class assigns the higher probability to s? Model pos Model neg 0.1 I 0.1 love 0.01 this 0.2 I love 0.01 this I love this fun film fun 0.1 film fun 0.1 film P(s pos) > P(s neg)

43 Text Classification and Naïve Bayes Naïve Bayes: Rela1onship to Language Modeling

44 Text Classification and Naïve Bayes Mul1nomial Naïve Bayes: A Worked Example

45 ˆP(w c) = 45 Priors: P(c)= P(j)= count(w, c)+1 count(c)+ V ˆP(c) = N c N Condi8onal Probabili8es: P(Chinese c) = P(Tokyo c) = (5+1) / (8+6) = 6/14 = 3/7 (0+1) / (8+6) = 1/14 P(Japan c) = (0+1) / (8+6) = 1/14 P(Chinese j) = (1+1) / (3+6) = 2/9 P(Tokyo j) = (1+1) / (3+6) = 2/9 P(Japan j) = (1+1) / (3+6) = 2/9 Doc Words Class Training 1 Chinese Beijing Chinese c 2 Chinese Chinese Shanghai c 3 Chinese Macao c 4 Tokyo Japan Chinese j Test 5 Chinese Chinese Chinese Tokyo Japan? Choosing a class: P(c d5) 3/4 * (3/7) 3 * 1/14 * 1/ P(j d5) 1/4 * (2/9) 3 * 2/9 * 2/

46 Naïve Bayes in Spam Filtering SpamAssassin Features: Men1ons Generic Viagra Online Pharmacy Men1ons millions of (dollar) ((dollar) NN,NNN,NNN.NN) Phrase: impress... girl From: starts with many numbers Subject is all capitals HTML has a low ra1o of text to image area One hundred percent guaranteed Claims you can be removed from the list 'Pres1gious Non- Accredited Universi1es' hlp://spamassassin.apache.org/tests_3_3_x.html

47 Summary: Naive Bayes is Not So Naive Very Fast, low storage requirements Robust to Irrelevant Features Irrelevant Features cancel each other without affec1ng results Very good in domains with many equally important features Decision Trees suffer from fragmentagon in such cases especially if lille data Op1mal if the independence assump1ons hold: If assumed independence is correct, then it is the Bayes Op1mal Classifier for problem A good dependable baseline for text classifica1on But we will see other classifiers that give bewer accuracy

48 Text Classification and Naïve Bayes Mul1nomial Naïve Bayes: A Worked Example

49 Text Classification and Naïve Bayes Precision, Recall, and the F measure

50 The 2- by- 2 con8ngency table correct not correct selected tp fp not selected fn tn

51 Precision and recall Precision: % of selected items that are correct Recall: % of correct items that are selected correct not correct selected tp fp not selected fn tn

52 A combined measure: F A combined measure that assesses the P/R tradeoff is F measure (weighted harmonic mean): F 2 1 (" + 1) PR = = # + (1!#) " P + R P R The harmonic mean is a very conserva1ve average; see IIR 8.3 People usually use balanced F1 measure i.e., with β = 1 (that is, α = ½): F = 2PR/(P+R)

53 Text Classification and Naïve Bayes Precision, Recall, and the F measure

54 Text Classification and Naïve Bayes Text Classifica1on: Evalua1on

55 More Than Two Classes: Sets of binary classifiers Sec.14.5 Dealing with any- of or mul1value classifica1on A document can belong to 0, 1, or >1 classes. For each class c C Build a classifier γ c to dis1nguish c from all other classes c C Given test doc d, Evaluate it for membership in each class using each γ c d belongs to any class for which γ c returns true 55

56 More Than Two Classes: Sets of binary classifiers Sec.14.5 One- of or mul1nomial classifica1on Classes are mutually exclusive: each document in exactly one class For each class c C Build a classifier γ c to dis1nguish c from all other classes c C Given test doc d, Evaluate it for membership in each class using each γ c d belongs to the one class with maximum score 56

57 57 Evalua8on: Classic Reuters Data Set Most (over)used data set, 21,578 docs (each 90 types, 200 toknens) 9603 training, 3299 test ar1cles (ModApte/Lewis split) 118 categories An ar1cle can be in more than one category Learn 118 binary category dis1nc1ons Average document (with at least one category) has 1.24 classes Only about 10 out of 118 categories are large Common categories (#train, #test) Earn (2877, 1087) Acquisitions (1650, 179) Money-fx (538, 179) Grain (433, 149) Crude (389, 189) Trade (369,119) Interest (347, 131) Ship (197, 89) Wheat (212, 71) Corn (182, 56) Sec

58 Reuters Text Categoriza8on data set (Reuters ) document Sec <REUTERS TOPICS="YES" LEWISSPLIT="TRAIN" CGISPLIT="TRAINING-SET" OLDID="12981" NEWID="798"> <DATE> 2-MAR :51:43.42</DATE> <TOPICS><D>livestock</D><D>hog</D></TOPICS> <TITLE>AMERICAN PORK CONGRESS KICKS OFF TOMORROW</TITLE> <DATELINE> CHICAGO, March 2 - </DATELINE><BODY>The American Pork Congress kicks off tomorrow, March 3, in Indianapolis with 160 of the nations pork producers from 44 member states determining industry positions on a number of issues, according to the National Pork Producers Council, NPPC. Delegates to the three day Congress will be considering 26 resolutions concerning various issues, including the future direction of farm policy and the tax law as it applies to the agriculture sector. The delegates will also debate whether to endorse concepts of a national PRV (pseudorabies virus) control and eradication program, the NPPC said. A large trade show, in conjunction with the congress, will feature the latest in technology in all areas of the industry, the NPPC added. Reuter 58 &#3;</BODY></TEXT></REUTERS>

59 Confusion matrix c For each pair of classes <c 1,c 2 > how many documents from c 1 were incorrectly assigned to c 2? c 3,2 : 90 wheat documents incorrectly assigned to poultry Docs in test set Assigned UK Assigned poultry Assigned wheat Assigned coffee Assigned interest True UK True poultry True wheat True coffee True interest Assigned trade 59 True trade

60 Sec Per class evalua8on measures Recall: Frac1on of docs in class i classified correctly:! j c ii c ij 60 Precision: Frac1on of docs assigned class i that are actually about class i: Accuracy: (1 - error rate) Frac1on of docs classified correctly:! j! i i c ii!! j c ii c ji c ij

61 Sec Micro- vs. Macro- Averaging If we have more than one class, how do we combine mul1ple performance measures into one quan1ty? Macroaveraging: Compute performance for each class, then average. Microaveraging: Collect decisions for all classes, compute con1ngency table, evaluate. 61

62 Sec Micro- vs. Macro- Averaging: Example Class 1 Class 2 Micro Ave. Table Truth: yes Classifier: yes Truth: no Classifier: no Truth: yes Classifier: yes Truth: no Classifier: no Truth: yes Classifier: yes Truth: no Classifier: no Macroaveraged precision: ( )/2 = 0.7 Microaveraged precision: 100/120 =.83 Microaveraged score is dominated by score on common classes 62

63 Development Test Sets and Cross- valida8on Training set Development Test Set Test Set Metric: P/R/F1 or Accuracy Unseen test set avoid overfiƒng ( tuning to the test set ) more conserva1ve es1mate of performance Cross- valida1on over mul1ple splits Handle sampling errors from different datasets Pool results over each split Compute pooled dev set performance Training Set Dev Test Training Set Dev Test Dev Test Training Set Test Set

64 Text Classification and Naïve Bayes Text Classifica1on: Evalua1on

65 Text Classification and Naïve Bayes Text Classifica1on: Prac1cal Issues

66 Sec The Real World Gee, I m building a text classifier for real, now! What should I do? 66

67 No training data? Manually written rules Sec If (wheat or grain) and not (whole or bread) then Categorize as grain Need careful cra ing Human tuning on development data Time- consuming: 2 days per class 67

68 Sec Very little data? Use Naïve Bayes Naïve Bayes is a high- bias algorithm (Ng and Jordan 2002 NIPS) Get more labeled data Find clever ways to get humans to label data for you Try semi- supervised training methods: Bootstrapping, EM over unlabeled documents, 68

69 Sec A reasonable amount of data? Perfect for all the clever classifiers SVM Regularized Logis1c Regression You can even use user- interpretable decision trees Users like to hack Management likes quick fixes 69

70 Sec A huge amount of data? Can achieve high accuracy! At a cost: SVMs (train 1me) or knn (test 1me) can be too slow Regularized logis1c regression can be somewhat beler So Naïve Bayes can come back into its own again! 70

71 Sec Accuracy as a function of data size With enough data Classifier may not maler 71 Brill and Banko on spelling correc1on

72 Real- world systems generally combine: Automa1c classifica1on Manual review of uncertain/difficult/"new cases 72

73 Underflow Preven8on: log space Mul1plying lots of probabili1es can result in floa1ng- point underflow. Since log(xy) = log(x) + log(y) BeLer to sum logs of probabili1es instead of mul1plying probabili1es. Class with highest un- normalized log probability score is s1ll most probable. c NB = argmax c j!c Model is now just max of sum of weights " i! positions log P(c j )+ log P(x i c j )

74 Sec How to tweak performance Domain- specific features and weights: very important in real performance Some1mes need to collapse terms: 74 Part numbers, chemical formulas, But stemming generally doesn t help Upweigh1ng: Coun1ng a word as if it occurred twice: 1tle words (Cohen & Singer 1996) first sentence of each paragraph (Murata, 1999) In sentences that contain 1tle words (Ko et al, 2002)

75 Text Classification and Naïve Bayes Text Classifica1on: Prac1cal Issues

76 Posi%ve or nega%ve movie review? unbelievably disappoin+ng Full of zany characters and richly applied sa+re, and some great plot twists this is the greatest screwball comedy ever filmed It was pathe+c. The worst part about it was the boxing scenes. 2

77 Google Product Search a 3

78 Bing Shopping a 4

79 Twi;er sen%ment versus Gallup Poll of Consumer Confidence Brendan O'Connor, Ramnath Balasubramanyan, Bryan R. Routledge, and Noah A. Smith From Tweets to Polls: Linking Text Sen+ment to Public Opinion Time Series. In ICWSM- 2010

80 Twi;er sen%ment: Johan Bollen, Huina Mao, Xiaojun Zeng TwiXer mood predicts the stock market, Journal of Computa+onal Science 2:1, /j.jocs

81 Bollen et al. (2011) CALM predicts DJIA 3 days later At least one current hedge fund uses this algorithm Dow Jones CALM 7

82 Target Sen%ment on Twi;er TwiXer Sen+ment App Alec Go, Richa Bhayani, Lei Huang TwiXer Sen+ment Classifica+on using Distant Supervision 8

83 Sen%ment analysis has many other names Opinion extrac+on Opinion mining Sen+ment mining Subjec+vity analysis 9

84 Why sen%ment analysis? Movie: is this review posi+ve or nega+ve? Products: what do people think about the new iphone? Public sen1ment: how is consumer confidence? Is despair increasing? Poli1cs: what do people think about this candidate or issue? Predic1on: predict elec+on outcomes or market trends from sen+ment 10

85 Scherer Typology of Affec%ve States Emo%on: brief organically synchronized evalua+on of a major event angry, sad, joyful, fearful, ashamed, proud, elated Mood: diffuse non- caused low- intensity long- dura+on change in subjec+ve feeling cheerful, gloomy, irritable, listless, depressed, buoyant Interpersonal stances: affec+ve stance toward another person in a specific interac+on friendly, flirta1ous, distant, cold, warm, suppor1ve, contemptuous AGtudes: enduring, affec+vely colored beliefs, disposi+ons towards objects or persons liking, loving, ha1ng, valuing, desiring Personality traits: stable personality disposi+ons and typical behavior tendencies nervous, anxious, reckless, morose, hos1le, jealous

86 Scherer Typology of Affec%ve States Emo%on: brief organically synchronized evalua+on of a major event angry, sad, joyful, fearful, ashamed, proud, elated Mood: diffuse non- caused low- intensity long- dura+on change in subjec+ve feeling cheerful, gloomy, irritable, listless, depressed, buoyant Interpersonal stances: affec+ve stance toward another person in a specific interac+on friendly, flirta1ous, distant, cold, warm, suppor1ve, contemptuous AGtudes: enduring, affec%vely colored beliefs, disposi%ons towards objects or persons liking, loving, ha1ng, valuing, desiring Personality traits: stable personality disposi+ons and typical behavior tendencies nervous, anxious, reckless, morose, hos1le, jealous

87 Sen%ment Analysis Sen+ment analysis is the detec+on of agtudes enduring, affec+vely colored beliefs, disposi+ons towards objects or persons Holder (source) of aftude 2. Target (aspect) of aftude 3. Type of aftude From a set of types Like, love, hate, value, desire, etc. Or (more commonly) simple weighted polarity: posi1ve, nega1ve, neutral, together with strength 4. Text containing the aftude Sentence or en+re document

88 Sen%ment Analysis Simplest task: Is the aftude of this text posi+ve or nega+ve? More complex: Rank the aftude of this text from 1 to 5 Advanced: Detect the target, source, or complex aftude types

89 Sen%ment Analysis Simplest task: Is the aftude of this text posi+ve or nega+ve? More complex: Rank the aftude of this text from 1 to 5 Advanced: Detect the target, source, or complex aftude types

90 Sentiment Analysis What is Sen+ment Analysis?

91 Sentiment Analysis A Baseline Algorithm

92 Sentiment Classification in Movie Reviews Polarity detec+on: Is an IMDB movie review posi+ve or nega+ve? Data: Polarity Data 2.0: Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan Thumbs up? Sen+ment Classifica+on using Machine Learning Techniques. EMNLP- 2002, Bo Pang and Lillian Lee A Sen+mental Educa+on: Sen+ment Analysis Using Subjec+vity Summariza+on Based on Minimum Cuts. ACL, hxp:// review- data

93 IMDB data in the Pang and Lee database when _star wars_ came out some twenty years ago, the image of traveling throughout the stars has become a commonplace image. [ ] when han solo goes light speed, the stars change to bright lines, going towards the viewer in lines that converge at an invisible point. cool. _october sky_ offers a much simpler image that of a single white dot, traveling horizontally across the night sky. [... ] snake eyes is the most aggrava+ng kind of movie : the kind that shows so much poten+al then becomes unbelievably disappoin+ng. it s not just because this is a brian depalma film, and since he s a great director and one who s films are always greeted with at least some fanfare. and it s not even because this was a film starring nicolas cage and since he gives a brauvara performance, this film is hardly worth his talents.

94 Baseline Algorithm (adapted from Pang and Lee) Tokeniza+on Feature Extrac+on Classifica+on using different classifiers Naïve Bayes MaxEnt SVM

95 Sen%ment Tokeniza%on Issues Deal with HTML and XML markup TwiXer mark- up (names, hash tags) Capitaliza+on (preserve for words in all caps) Phone numbers, dates Emo+cons Useful code: 21 Christopher PoXs sen+ment tokenizer Brendan O Connor twixer tokenizer PoXs emo+cons [<>]? # optional hat/brow! [:;=8] # eyes! [\-o\*\']? # optional nose! [\)\]\(\[ddpp/\:\}\{@\ \\] # mouth! #### reverse orientation! [\)\]\(\[ddpp/\:\}\{@\ \\] # mouth! [\-o\*\']? # optional nose! [:;=8] # eyes! [<>]? # optional hat/brow!

96 Extrac%ng Features for Sen%ment Classifica%on How to handle nega+on I didn t like this movie! vs I really like this movie! Which words to use? Only adjec+ves All words All words turns out to work bexer, at least on this data 22

97 Nega%on Das, Sanjiv and Mike Chen Yahoo! for Amazon: Extrac+ng market sen+ment from stock message boards. In Proceedings of the Asia Pacific Finance Associa+on Annual Conference (APFA). Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan Thumbs up? Sentiment Classification using Machine Learning Techniques. EMNLP-2002, Add NOT_ to every word between nega+on and following punctua+on: didn t like this movie, but I! didn t NOT_like NOT_this NOT_movie but I!

98 Reminder: Naïve Bayes c NB = argmax c j!c " i! positions P(c j ) P(w i c j ) ˆP(w c) = count(w,c)+1 count(c)+ V 24

99 Binarized (Boolean feature) Mul%nomial Naïve Bayes Intui+on: For sen+ment (and probably for other text classifica+on domains) Word occurrence may maxer more than word frequency The occurrence of the word fantas1c tells us a lot The fact that it occurs 5 +mes may not tell us much more. Boolean Mul+nomial Naïve Bayes Clips all the word counts in each document at 1 25

100 Boolean Mul%nomial Naïve Bayes: Learning From training corpus, extract Vocabulary Calculate P(c j ) terms Calculate P(w k c j ) terms For each c j in C do Text Remove j duplicates single doc in containing each doc: all docs j docs j all docs with class =c j For For each each word word w type k in Vocabulary w in doc j docs P(c j )! j n k Retain # of only occurrences a single instance of w k in of Text w j n total # documents P(w k c j )! k +! n +! Vocabulary

101 Boolean Mul%nomial Naïve Bayes on a test document d First remove all duplicate words from d Then compute NB using the same equa+on: c NB = argmax c j!c " i! positions P(c j ) P(w i c j ) 27

102 Normal vs. Boolean Mul%nomial NB Normal Doc Words Class Training 1 Chinese Beijing Chinese c 2 Chinese Chinese Shanghai c 3 Chinese Macao c 4 Tokyo Japan Chinese j Test 5 Chinese Chinese Chinese Tokyo Japan? Boolean Doc Words Class Training 1 Chinese Beijing c 2 Chinese Shanghai c 3 Chinese Macao c 4 Tokyo Japan Chinese j Test 28 5 Chinese Tokyo Japan?

103 Binarized (Boolean feature) Mul%nomial Naïve Bayes B. Pang, L. Lee, and S. Vaithyanathan Thumbs up? Sen+ment Classifica+on using Machine Learning Techniques. EMNLP- 2002, V. Metsis, I. Androutsopoulos, G. Paliouras Spam Filtering with Naive Bayes Which Naive Bayes? CEAS Third Conference on and An+- Spam. K.- M. Schneider On word frequency informa+on and nega+ve evidence in Naive Bayes text classifica+on. ICANLP, JD Rennie, L Shih, J Teevan Tackling the poor assump+ons of naive bayes text classifiers. ICML 2003 Binary seems to work bexer than full word counts This is not the same as Mul+variate Bernoulli Naïve Bayes MBNB doesn t work well for sen+ment or other text tasks Other possibility: log(freq(w)) 29

104 Cross- Valida%on Iteration Break up data into 10 folds (Equal posi+ve and nega+ve inside each fold?) For each fold Choose the fold as a temporary test set Train on 9 folds, compute performance on the test fold Report average performance of the 10 runs Test Training Test Training Training Test Training Training Test Training Test

105 Other issues in Classifica%on MaxEnt and SVM tend to do bexer than Naïve Bayes 31

106 Problems: What makes reviews hard to classify? Subtlety: Perfume review in Perfumes: the Guide: If you are reading this because it is your darling fragrance, please wear it at home exclusively, and tape the windows shut. Dorothy Parker on Katherine Hepburn She runs the gamut of emo+ons from A to B 32

107 Thwarted Expecta%ons and Ordering Effects This film should be brilliant. It sounds like a great plot, the actors are first grade, and the suppor+ng cast is good as well, and Stallone is axemp+ng to deliver a good performance. However, it can t hold up. Well as usual Keanu Reeves is nothing special, but surprisingly, the very talented Laurence Fishbourne is not so good either, I was surprised. 33

108 Sentiment Analysis A Baseline Algorithm

109 Sentiment Analysis Sen+ment Lexicons

110 The General Inquirer Philip J. Stone, Dexter C Dunphy, Marshall S. Smith, Daniel M. Ogilvie The General Inquirer: A Computer Approach to Content Analysis. MIT Press Home page: hxp:// List of Categories: hxp:// Spreadsheet: hxp:// Categories: Posi+v (1915 words) and Nega+v (2291 words) Strong vs Weak, Ac+ve vs Passive, Overstated versus Understated Pleasure, Pain, Virtue, Vice, Mo+va+on, Cogni+ve Orienta+on, etc Free for Research Use

111 LIWC (Linguis%c Inquiry and Word Count) Pennebaker, J.W., Booth, R.J., & Francis, M.E. (2007). Linguis+c Inquiry and Word Count: LIWC Aus+n, TX Home page: hxp:// words, >70 classes Affec%ve Processes nega+ve emo+on (bad, weird, hate, problem, tough) posi+ve emo+on (love, nice, sweet) Cogni%ve Processes Tenta+ve (maybe, perhaps, guess), Inhibi+on (block, constraint) Pronouns, Nega%on (no, never), Quan%fiers (few, many) $30 or $90 fee

112 MPQA Subjec%vity Cues Lexicon Theresa Wilson, Janyce Wiebe, and Paul Hoffmann (2005). Recognizing Contextual Polarity in Phrase-Level Sentiment Analysis. Proc. of HLT-EMNLP Riloff and Wiebe (2003). Learning extraction patterns for subjective expressions. EMNLP Home page: hxp:// words from 8221 lemmas 2718 posi+ve 4912 nega+ve Each word annotated for intensity (strong, weak) GNU GPL 38

113 Bing Liu Opinion Lexicon Minqing Hu and Bing Liu. Mining and Summarizing Customer Reviews. ACM SIGKDD Bing Liu's Page on Opinion Mining hxp:// lexicon- English.rar 6786 words posi+ve 4783 nega+ve

114 Sen%WordNet Stefano Baccianella, Andrea Esuli, and Fabrizio Sebas+ani SENTIWORDNET 3.0: An Enhanced Lexical Resource for Sen+ment Analysis and Opinion Mining. LREC Home page: hxp://sen+wordnet.is+.cnr.it/ All WordNet synsets automa+cally annotated for degrees of posi+vity, nega+vity, and neutrality/objec+veness [es+mable(j,3)] may be computed or es+mated!pos 0 Neg 0 Obj 1! [es+mable(j,1)] deserving of respect or high regard!pos.75 Neg 0 Obj.25!

115 Disagreements between polarity lexicons Christopher PoXs, Sen+ment Tutorial, 2011 Opinion Lexicon General Inquirer Sen%WordNet LIWC MPQA 33/5402 (0.6%) 49/2867 (2%) 1127/4214 (27%) 12/363 (3%) Opinion Lexicon 32/2411 (1%) 1004/3994 (25%) 9/403 (2%) General Inquirer 520/2306 (23%) 1/204 (0.5%) Sen%WordNet 174/694 (25%) LIWC 41

116 Analyzing the polarity of each word in IMDB How likely is each word to appear in each sen+ment class? Count( bad ) in 1- star, 2- star, 3- star, etc. But can t use raw counts: Instead, likelihood: Make them comparable between words Scaled likelihood: PoXs, Christopher On the nega+vity of nega+on. SALT 20, P(w c) = P(w c) P(w) " w!c f (w,c) f (w, c)

117 Analyzing the polarity of each word in IMDB PoXs, Christopher On the nega+vity of nega+on. SALT 20, POS good (883,417 tokens) amazing (103,509 tokens) great (648,110 tokens) awesome (47,142 tokens) 0.28 l 0.27 l Scaled likelihood P(w c)/p(w) Pr(c w) l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l Rating NEG good (20,447 tokens) depress(ed/ing) (18,498 tokens) bad (368,273 tokens) terrible (55,492 tokens) 0.28 l Scaled likelihood P(w c)/p(w) l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l

118 Other sen%ment feature: Logical nega%on PoXs, Christopher On the nega+vity of nega+on. SALT 20, Is logical nega+on (no, not) associated with nega+ve sen+ment? PoXs experiment: Count nega+on (not, n t, no, never) in online reviews Regress against the review ra+ng

119 Po;s 2011 Results: More nega%on in nega%ve sen%ment a Scaled likelihood P(w c)/p(w)

120 Sentiment Analysis Sen+ment Lexicons

121 Sentiment Analysis Learning Sen+ment Lexicons

122 Semi- supervised learning of lexicons Use a small amount of informa+on A few labeled examples A few hand- built paxerns To bootstrap a lexicon 48

123 Hatzivassiloglou and McKeown intui%on for iden%fying word polarity Vasileios Hatzivassiloglou and Kathleen R. McKeown Predic+ng the Seman+c Orienta+on of Adjec+ves. ACL, Adjec+ves conjoined by and have same polarity Fair and legi+mate, corrupt and brutal *fair and brutal, *corrupt and legi+mate Adjec+ves conjoined by but do not fair but brutal 49

124 Hatzivassiloglou & McKeown 1997 Step 1 Label seed set of 1336 adjec+ves (all >20 in 21 million word WSJ corpus) posi+ve adequate central clever famous intelligent remarkable reputed sensi+ve slender thriving 679 nega+ve contagious drunken ignorant lanky listless primi+ve strident troublesome unresolved unsuspec+ng

125 Hatzivassiloglou & McKeown 1997 Step 2 Expand seed set to conjoined adjec+ves nice, helpful nice, classy 51

126 Hatzivassiloglou & McKeown 1997 Step 3 Supervised classifier assigns polarity similarity to each word pair, resul+ng in graph: helpful brutal nice corrupt irrational 52 fair classy

127 Hatzivassiloglou & McKeown 1997 Step 4 Clustering for par++oning the graph into two + - brutal helpful nice corrupt irrational 53 fair classy

128 Output polarity lexicon Posi+ve bold decisive disturbing generous good honest important large mature pa+ent peaceful posi+ve proud sound s+mula+ng straigh orward strange talented vigorous wixy Nega+ve ambiguous cau+ous cynical evasive harmful hypocri+cal inefficient insecure irra+onal irresponsible minor outspoken pleasant reckless risky selfish tedious unsupported vulnerable wasteful 54

129 Output polarity lexicon Posi+ve bold decisive disturbing generous good honest important large mature pa+ent peaceful posi+ve proud sound s+mula+ng straigh orward strange talented vigorous wixy Nega+ve ambiguous cau%ous cynical evasive harmful hypocri+cal inefficient insecure irra+onal irresponsible minor outspoken pleasant reckless risky selfish tedious unsupported vulnerable wasteful 55

130 Turney Algorithm Turney (2002): Thumbs Up or Thumbs Down? Semantic Orientation Applied to Unsupervised Classification of Reviews 1. Extract a phrasal lexicon from reviews 2. Learn polarity of each phrase 3. Rate a review by the average polarity of its phrases 56

131 Extract two- word phrases with adjec%ves First Word Second Word Third Word (not extracted) JJ NN or NNS anything RB, RBR, RBS JJ Not NN nor NNS JJ JJ Not NN or NNS NN or NNS JJ Nor NN nor NNS RB, RBR, or RBS VB, VBD, VBN, VBG anything 57

132 How to measure polarity of a phrase? Posi+ve phrases co- occur more with excellent Nega+ve phrases co- occur more with poor But how to measure co- occurrence? 58

133 Pointwise Mutual Informa%on Mutual informa%on between 2 random variables X and Y I(X,Y ) =! P(x,y) P(x, y) log 2 P(x)P(y) x! y Pointwise mutual informa%on: How much more do events x and y co- occur than if they were independent? PMI(X,Y ) = log 2 P(x,y) P(x)P(y)

134 Pointwise Mutual Informa%on Pointwise mutual informa%on: How much more do events x and y co- occur than if they were independent? PMI(X,Y ) = log 2 P(x,y) P(x)P(y) PMI between two words: How much more do two words co- occur than if they were independent? PMI(word 1, word 2 ) = log 2 P(word 1,word 2 ) P(word 1 )P(word 2 )

135 How to Es%mate Pointwise Mutual Informa%on Query search engine (Altavista) P(word) es+mated by hits(word)/n! P(word 1,word 2 ) by hits(word1 NEAR word2)/n 2! PMI(word 1, word 2 ) = log 2 hits(word 1 NEAR word 2 ) hits(word 1 )hits(word 2 )

136 Does phrase appear more with poor or excellent? Polarity( phrase) = PMI( phrase,"excellent")! PMI( phrase,"poor") = log 2 hits(phrase NEAR "excellent") hits(phrase)hits("excellent") = log 2 hits(phrase NEAR "excellent") hits(phrase)hits("excellent")! log 2 hits(phrase NEAR "poor") hits(phrase)hits("poor") hits(phrase)hits("poor") hits(phrase NEAR "poor") 62! hits(phrase NEAR "excellent")hits("poor") $ = log 2 # & " hits(phrase NEAR "poor")hits("excellent")%

137 63 Phrases from a thumbs- up review Phrase POS tags Polarity online service JJ NN 2.8! online experience JJ NN 2.3! direct deposit JJ NN 1.3! local branch JJ NN 0.42! low fees JJ NNS 0.33! true service JJ NN -0.73! other bank JJ NN -0.85! inconveniently located JJ NN -1.5! Average 0.32!

138 64 Phrases from a thumbs- down review Phrase POS tags Polarity direct deposits JJ NNS 5.8! online web JJ NN 1.9! very handy RB JJ 1.4! virtual monopoly JJ NN -2.0! lesser evil RBR JJ -2.3! other problems JJ NNS -2.8! low funds JJ NNS -6.8! unethical prac+ces JJ NNS -8.5! Average -1.2!

139 Results of Turney algorithm 410 reviews from Epinions 170 (41%) nega+ve 240 (59%) posi+ve Majority class baseline: 59% Turney algorithm: 74% Phrases rather than words Learns domain- specific informa+on 65

140 Using WordNet to learn polarity WordNet: online thesaurus (covered in later lecture). Create posi+ve ( good ) and nega+ve seed- words ( terrible ) Find Synonyms and Antonyms Posi+ve Set: Add synonyms of posi+ve words ( well ) and antonyms of nega+ve words Nega+ve Set: Add synonyms of nega+ve words ( awful ) and antonyms of posi+ve words ( evil ) Repeat, following chains of synonyms Filter 66 S.M. Kim and E. Hovy Determining the sen+ment of opinions. COLING 2004 M. Hu and B. Liu. Mining and summarizing customer reviews. In Proceedings of KDD, 2004

141 Summary on Learning Lexicons Advantages: Can be domain- specific Can be more robust (more words) Intui+on Start with a seed set of words ( good, poor ) Find other words that have similar polarity: Using and and but Using words that occur nearby in the same document Using WordNet synonyms and antonyms

142 Sentiment Analysis Learning Sen+ment Lexicons

143 Sentiment Analysis Other Sen+ment Tasks

144 Finding sen%ment of a sentence Important for finding aspects or axributes Target of sen+ment The food was great but the service was awful! 70

145 Finding aspect/a;ribute/target of sen%ment M. Hu and B. Liu Mining and summarizing customer reviews. In Proceedings of KDD. S. Blair- Goldensohn, K. Hannan, R. McDonald, T. Neylon, G. Reis, and J. Reynar Building a Sen+ment Summarizer for Local Service Reviews. WWW Workshop. Frequent phrases + rules Find all highly frequent phrases across reviews ( fish tacos ) Filter by rules like occurs right a er sen+ment word great fish tacos means fish tacos a likely aspect Casino Children s Barber Greek Restaurant Department Store casino, buffet, pool, resort, beds haircut, job, experience, kids food, wine, service, appe+zer, lamb selec+on, department, sales, shop, clothing

146 Finding aspect/a;ribute/target of sen%ment The aspect name may not be in the sentence For restaurants/hotels, aspects are well- understood Supervised classifica+on Hand- label a small corpus of restaurant review sentences with aspect food, décor, service, value, NONE Train a classifier to assign an aspect to asentence Given this sentence, is the aspect food, décor, service, value, or NONE 72

147 PuGng it all together: Finding sen%ment for aspects S. Blair- Goldensohn, K. Hannan, R. McDonald, T. Neylon, G. Reis, and J. Reynar Building a Sen+ment Summarizer for Local Service Reviews. WWW Workshop Sentences & Phrases Sentences & Phrases Sentences & Phrases Reviews Final Summary Text Extractor Sentiment Classifier Aspect Extractor Aggregator 73

148 Results of Blair- Goldensohn et al. method Rooms (3/5 stars, 41 comments) (+) The room was clean and everything worked fine even the water pressure... (+) We went because of the free room and was pleasantly pleased... (- ) the worst hotel I had ever stayed at... Service (3/5 stars, 31 comments) (+) Upon checking out another couple was checking early due to a problem... (+) Every single hotel staff member treated us great and answered every... (- ) The food is cold and the service gives new meaning to SLOW. Dining (3/5 stars, 18 comments) (+) our favorite place to stay in biloxi.the food is great also the service... (+) Offer of free buffet for joining the Play

149 Baseline methods assume classes have equal frequencies! If not balanced (common in the real world) can t use accuracies as an evalua+on need to use F- scores Severe imbalancing also can degrade classifier performance Two common solu+ons: Resampling in training Random undersampling 2. Cost- sensi+ve learning Penalize SVM more for misclassifica+on of the rare thing

150 How to deal with 7 stars? 1. Map to binary Bo Pang and Lillian Lee Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. ACL, Use linear or ordinal regression Or specialized models like metric labeling 76

151 Summary on Sen%ment Generally modeled as classifica+on or regression task predict a binary or ordinal label Features: Nega+on is important Using all words (in naïve bayes) works well for some tasks Finding subsets of words may help in other tasks Hand- built polarity lexicons Use seeds and semi- supervised learning to induce lexicons

152 Scherer Typology of Affec%ve States Emo%on: brief organically synchronized evalua+on of a major event angry, sad, joyful, fearful, ashamed, proud, elated Mood: diffuse non- caused low- intensity long- dura+on change in subjec+ve feeling cheerful, gloomy, irritable, listless, depressed, buoyant Interpersonal stances: affec+ve stance toward another person in a specific interac+on friendly, flirta1ous, distant, cold, warm, suppor1ve, contemptuous AGtudes: enduring, affec+vely colored beliefs, disposi+ons towards objects or persons liking, loving, ha1ng, valuing, desiring Personality traits: stable personality disposi+ons and typical behavior tendencies nervous, anxious, reckless, morose, hos1le, jealous

153 Computa%onal work on other affec%ve states Emo%on: Detec+ng annoyed callers to dialogue system Detec+ng confused/frustrated versus confident students Mood: Finding trauma+zed or depressed writers Interpersonal stances: Detec+on of flirta+on or friendliness in conversa+ons Personality traits: Detec+on of extroverts

154 Detec%on of Friendliness Friendly speakers use collabora+ve conversa+onal style Laughter 80 Less use of nega+ve emo+onal words More sympathy That s too bad More agreement I think so too! Less hedges Ranganath, Jurafsky, McFarland I m sorry to hear that! kind of sort of a little!

155 Sentiment Analysis Other Sen+ment Tasks