Deep Learning in Natural Language Processing. Tong Wang Advisor: Prof. Ping Chen Computer Science University of Massachusetts Boston

Transcription

1 Deep Learning in Natural Language Processing Tong Wang Advisor: Prof. Ping Chen Computer Science University of Massachusetts Boston

2 Outline Natural Language Processing Deep Learning in NLP My Research Projects My Path in Computer Science My Experience to Find Internship

3 What is Natural Language Processing Natural Language Processing is related to the area of human-computer interaction. Natural language understanding Natural language generation

4 Natural Language Processing

5 Natural Language Processing

6 NLP Applications Information Extraction Name Entity Recognition Machine Translation Question Answering Topic Model Summarization

7 Information Extraction

8 Name Entity Recognition Classify elements in text into categories such as location, time, name of person, organization. Jim worked in Google corp. in 2012 (Jim)[person] worked in (Google corp.) [organization] in (2012)[time]

9 Machine Translation

10 Machine Translation Difficulties Words together are more than the sum of their parts. Can not translated word by word E.g, Fast food, Light rain Need a big dictionary with grammar rules in both languages, large start-up cost Require computer to understand

11 Question Answering IBM Watson won Jeopardy on 02/16/2011

12 Question Answering

13 Question Answering

14 Question Answering

15 NLP Tasks

16 Why NLP is hard Basically text is not computer-friendly Many different ways to represent the same thing Order and context are extremely important Language is very high dimensional and sparse. Tons of rare words. B4 (before), IC (I see), cre8(create) Ambiguity

17 Ambiguity At last, a computer understands you like your mother It understands you as well as your mother understands you It understands (that) you like your mother It understands you as well as it understands your mother

18 Ambiguity at Syntactic Level

19 DEEP LEARNING IN NATURAL LANGUAGE PROCESSING

20 Deep Learning (Representation learning) in NLP

21 Deep Learning in NLP Word Level Application: Word Embedding, word2vec Sentence/paragraph Level Application: Neural Machine Translation, doc2vec, etc.

22 Word Representation The majority of rule-based and statistical NLP work regarded words as atomic symbols In vector space terms, this is a vector with one 1 and many zeros, it is called one-hot representation Condo: [0,0,0,0,1,0,0, 0] Apartment: [0,1,0,0,0,0,0, 0] These two vectors are orthogonal, no similarity

23 Word2vec

24 Word Embedding From word2vec Parameter Learning Explained

25 Word Embedding From Distributed Representations of Words and Phrases and their Compositionality

26 Word Embedding W( woman ) W( man ) W( queen ) W( king )

27 Sentence Embedding From Paragraph Vector - Stanford Computer Science

28 Recurrent Neural Network

29 Neural Machine Translation

30 MY RESEARCH ROJECT

31 Text Simplification Text simplification (TS) aims to simplify the lexical, grammatical, or structural complexity of text while retaining its semantic meaning It can help various groups of people, including children, non-native speakers, and people with cognitive disabilities

32 Lexical Simplification Substitute long and infrequent words with shorter and more frequent words Candidate selection Semantic similarity Syntax and grammar correct The meaning of the sentence remains the same Disadvantage: On word level

33 Lexical Simplification Lexical Simplification webpage:

34 LS System For each word w in text: Check part of speech tagging of w Retrieve top 20 most similar words from word2vec For c in 20 candidate words: If c is the same pos with w If c is not a different form of w, e.g, past tense. If w is more difficult than c: Put c in the sentence, compute sentence similarity and n- gram Otherwise continue

35 TS using Neural Machine Translation Original English and simplified English can be thought of as two different languages. TS would be the process to translate English to simplified English.

36 Text Simplification using Neural Machine Translation AAAI 2016, Student abstract

37 Steps Collecting training data Pairs of sentences: original sentence and simplified sentence From English Wikipedia and Simple English Wikipedia Build RNN Encoder Decoder Model Evaluation

38 Use Sentence Similarity to Collect Training Data From Siamese Recurrent Architectures for Learning Sentence Similarity

39 Other projects Extended topic model for word dependency Opinion mining for chemical spill in West Virginia Compression and data mining

40 My Path in Computer Science Huazhong Agricultural University, Information and Computing Science, BS, China, Bioinformatics lab, Huazhong Agricultural University, Northeastern University, Computer Systems Engineering, MS, IoMosaic, Software Engineer, University of Massachusetts Boston, Computer Science, PhD, present

41 Keep Healthy Play badminton almost every day from Monday to Friday Run 5 miles in weekend

42 Keys to find internship Good resume Did a lot of projects Networking (Very important!) Go to conference Ask for job reference from professors, friends, alumni, strangers from Linkedin

43 Prepare interview Know that company Behavior questions Technical questions You must start to practice programming in your favorite language at least 1 month before the interview. (Leetcode)

44 Thank you!