deep learning for natural language processing - … · ji et al., arxiv, ... •noah’ ark lab is...

Deep Learning for Natural Language Processing

- Research at Noah’s Ark Lab

Hang Li

Noah’s Ark Lab

Huawei Technologies

Institute of Software, CAS Beijing

Jan 15, 2016

DL for NLP @Noah Lab

Zhengdong Lu

Lifeng Shang Lin Ma Zhaopeng Tu Xin Jiang

Research on Deep Learning for Natural Language Processing

• Goal – Develop advanced deep learning technologies for

natural language processing

• Tasks We Are Working on – Language Representation Learning

– Semantic Matching

– Image Retrieval

– Machine Translation

– Natural Language Dialogue

Outline

• Language Representation Learning

• Semantic Matching

• Image Retrieval

• Machine Translation

• Natural Language Dialogue

Learning of Sentence Representation

• Start from word embedding

• In a specific task

• Learn a neural network model to generate sentence representation

• Previous approach: Using RNN

John loves Mary

Mary is loved by John

Mary loves John

Using high-dimensional real-valued vectors to represent the meaning of sentences

Convolutional Neural Network (CNN)

• “Robust parsing” • Shared parameter on same level • Fixed length, zero padding

the cat sat on the mat

Concatenation

……

the cat sat on the mat

cat sat

the cat sat

the cat

sat on

cat sat on

cat sat

on the

sat on the

sat on

the mat

on the mat

on the

sat on

the cat sat

the cat

the mat

on the mat

sat on

convolution

max pooling

Hu et al., NIPS 2014

Convolution

Filter feature map neuron

TT

hi

T

i

T

ii

i

l

i

flfl

fl

i

l

fll

i

flfl

i

xxxz

iz

liz

lfbw

lifz

Ffbzwz

],,[

location for vectors wordedcancatenat frominput is

1layer from location for neuron ofinput is

function sigmoid is

layer in typeofneuron of parameters are ,

layer in location for typeofneuron ofoutput is

,,2,1 )(

11

)0(

)0(

)1(

),(),(

),(

),()1(),(),(

)1( liz

),( flb

+1

),( flw

),( fliz

…

convolution

Max Pooling

lifzz

lifz

zzz

fl

i

fl

i

fl

i

fl

i

fl

i

fl

i

layer in location for typeof pooling ofinput are ,

layer in location for typeof pooling ofoutput is

),max(

),1(

2

),1(

12

),(

),1(

2

),1(

12

),(

max pooling

Outline

• Sentence Representation Learning


• Image Retrieval



Semantic Matching

• Question Answering： A language = question, B language = answer

• Information Retrieval： A language = query, B language = document

Semantic Representation

Syntactic Representation

Phrase

Word

Semantic Representation

Syntactic Representation

Phrase

Word

A Language B Language

Word Matching

Phrase Matching

Syntactic Matching

Semantic Matching

Deep Match CNN - Architecture I

• First represent two sentences as vectors, and then match the vectors

11

Hu et al., NIPS 2014

…

…

MLP

Matching Degree

sen

ten

ce S

X

sen

ten

ce S

Y

Deep Match CNN - Architecture II

• Represent and match two sentences simultaneously

• Two dimensional model

12

…

MLP

Matching Degree

max-pooling

2D convolution

more 2D convolution & pooling

1D convolution

sentence SX

sen

ten

ce S

Y W

…

Layer-1 (1D convolution)

Layer-2 (2D-pooling) Layer-3 (2D-convolution)

Deep Match Tree • Based on dependency parsing

• Deep neural network for matching, with first layer representing mined matching patterns

13 Wang et al., IJCAI 2015

Outline



• Image Retrieval



Image Retrieval

Find the picture that I had dinner with my friends at an Italian restaurant in Hong Kong

• Scenario

– Image search on smartphone

– Key: matching text to images

• Technology

– Deep model for matching text and image

query representation

image representation

index of images

Matching model

Deep Match Model for Image and Text

• Represent text and image as vectors and then match the two vectors

• Word-level matching, phrase-level matching, sentence-level Matching

• Our models (CNN) work better than state of the art models (RNN)

Ma et al., ICCV 2015

Word-level Matching and Phrase-level Matching

Word Level Matching

Phrase Level Matching

Sentence-level Matching

Sentence Level Matching

Outline



• Image Retrieval



Statistical Machine Translation and Neural Machine Translation

NMT: RNN Search

model language gram-)(, of word(aligned) affiliated

),,,,,,,|()|()

2

1()

2

1(

11

||

1

mnts

ssstttPSTP

ia

ma

ama

nii

T

i

i

i

ii

i

SMT: Joint Language Model as feature

CNN based Joint Language Model

• Capture local structures of source sentence using CNN

• tagCNN uses aligned word

• inCNN uses `attention signal’

Wang et al. ACL 2015

CNN based Language Model • Capture local structures of sentence using CNN

• Recursively connect αCNN and βCNN

Meng et al., ACL 2015

Deep Memory

• For sequence to sequence learning

• Stack of Neural Turing Machines

• Transformation of representations to representations

• Flexibility from reading and writing on memories

• Suitable for translation between distant languages

• Contains RNN Search and conventional DNN as special cases

Meng et al. 2015

Deep Memory Model

Deep Memory

RNN Search

• Layer 1: word embedding • Layer 2: location based reading from layer 1 • Output Layer: content based reading from layer 2 (attention)

• Layer 1: word embedding • Layer 2: location based reading from layer 1 • Layer 3: content based reading from layer 2 and location based reading from layer 1 • Output Layer: content based reading from layers 2 and 3 (attention)

Global re-ordering

Outline



• Image Retrieval



Natural Language Dialogue • Single turn vs multi-

turn dialogue

• Single turn

– Retrieval based approach

– Generation based approach

– Generative question answering

• Multi turn

– Simple inference

– Dialogue management

Alan Turing

Natural Language Dialogue System - Retrieval based Approach

index of messages and responses

matching

ranking

message

retrieval

retrieved messages and responses

ranked responses

matching models

ranking model

online

offline

best response

matched responses

Retrieval based Dialogue System

• Matching Models (Features)

– Deep Match CNN

– Deep Match Tree

– Vector Space Model

– Translation Model

• Ranking Model

– Ranking SVM

Ji et al., arXiv, 2014

Natural Language Dialogue System - Generation based Approach

• Encoding messages to intermediate representations

• Decoding intermediate representations to responses

• Recurrent Neural Network (RNN)

Message

Response

Encoder

Txxx 21x

Decoder

tyyy 21y

c

h

Context Generator

Shang et al., ACL 2015

Encoders and Decoder

Decoder Global Encoder

Local Encoder Concatenated Encoder

Generative Question Answering

(Yao-Ming, spouse, Ye-Li) (Yao-Ming, born, Shanghai) (Yao-Ming, height, 2.29m) … … (Ludwig van Beethoven, place of birth, Germany) … …

Knowledge Base

Q: How tall is Yao Ming? A: He is 2.29m tall and is visible from space. (Yao Ming, height, 2.29m) Q: Which country was Beethoven from? A: He was born in what is now Germany. (Ludwig van Beethoven, place of birth, Germany)

Neural Question Answering System

Q: How tall is Liu Xiang? A: He is 1.90m

Learning System

GenQA

• Interpreter: creates representation of sentence using RNN

• Enquirer: retrieves top k triples with matching scores using linear matching model

• Attention model: controls generation of answer

• Generator: generates answer based on question and retrieved triples using attention based RNN

Short Term Memory

Long Term Memory

(Knowledge Base)

Question

Interpreter

Enquirer

Generator

Answer

Attention Model

Key idea: • Generation of answer based on question and retrieved result • Combination of neural processing and symbolic processing

Yin et al. 2015

Simple Inference • Fact1: John travelled to the hallway.

• Fact2: Mary journeyed to the bathroom.

• Question: Where is Mary?

• Answer: bathroom, relies on 2

• Fact1: The triangle is above the pink rectangle.

• Factor2.The blue square is to the left of the triangle.

• Question: Is the pink rectangle to the right of the blue square?

• Answer: Yes, relies on 1 and 2

Inferring and Answering System

Question Answer

Facts

Neural Reasoner

• Encoding layer: representing question and facts using RNN

• Reasoning Layers:

Question fact interaction using DNN,

Pooling: selection of new question representation

• Answering Layer: selection of answer from question representation using classifier

Peng et al., 2015

Summary

• Noah’ Ark Lab is working on deep learning for natural language processing

• Significant progresses have been made in – Language Representation Learning

– Semantic Matching

– Image Retrieval

– Machine Translation

– Natural Language Dialogue

• Future of NLP: combination of neural processing and symbolic processing

References • Zhengdong Lu, Hang Li. A Deep Architecture for Matching Short Texts.

NIPS'13, 1367-1375, 2013.

• Baotian Hu, Zhengdong Lu, Hang Li, Qingcai Chen. Convolutional Neural Network Architectures for Matching Natural Language Sentences. NIPS'14, 2042-2050, 2014.

• Mingxuan Wang, Zhengdong Lu, Hang Li, Qun Liu. Syntax-based Deep Matching of Short Texts. IJCAI’15, 2015.

• Zhao, Han, Zhengdong Lu, and Pascal Poupart. "Self-Adaptive Hierarchical Sentence Model." IJCAI’15, 2015.

• Lifeng Shang, Zhengdong Lu, Hang Li. Neural Responding Machine for Short Text Conversation. ACL-IJCNLP'15, 2015.

• Fandong Meng, Zhengdong Lu, Mingxuan Wang, Hang Li, Wenbin Jiang, Qun Liu. Encoding Source Language with Convolutional Neural Network for Machine Translation. ACL-IJCNLP'15, 20-30, 2015.

• Mingxuan Wang, Zhengdong Lu, Hang Li, Wenbin Jiang, Qun Liu. GenCNN: A Convolutional Architecture for Word Sequence Prediction. ACL-IJCNLP'15, 1567-1576, 2015.

References • Baotian Hu, Zhaopeng Tu, Zhengdong Lu, Hang Li, Qingcai Chen. Context-

Dependent Translation Selection Using Convolutional Neural Network. ACL-IJCNLP'15, 536-541, 2015.

• Lin Ma, Zhengodng Lu, Lifeng Shang, Hang Li . Multimodal Convolutional Neural Networks for Matching Image and Sentence, ICCV’15, 2015.

• Lin Ma, Zhengdong Lu, Hang Li. Learning to Answer Questions From Image Using Convolutional Neural Network. Proceedings of the Thirtieth AAAI Conference (AAAI’16), 2016.

• Fandong Meng, Zhengdong Lu, Zhaopeng Tu, Hang Li, Qun Liu. A Deep Memory-based Architecture for Sequence-to-Sequence Learning. arXiv, 2015.

• Baolin Peng, Zhengdong Lu, Hang Li, Kam-Fai Wong. Towards Neural Network-based Reasoning. arXiv, 2015.

• Pengcheng Yin, Zhengdong Lu, Hang Li, Ben Kao. Neural Enquirer: Learning to Query Tables. arXiv, 2015

• Jun Yin, Xin Jiang, Zhengdong Lu, Lifeng Shang, Hang Li, Xiaoming Li. Neural Generative Question Answering. arXiv, 2015.

Thank you!

[email protected]

deep learning for natural language processing - … · ji et al., arxiv, ... •noah’ ark lab is...

Documents