Real

WorldVirtual

World

Real

WorldVirtual

World

placespeople

organizationsproducts

services, etc.

Real

WorldVirtual

World

placespeople

organizationsproducts

services, etc.

entity graph

reviews, ratings

ontology

user KN graph, etc.

user graph

Real

WorldVirtual

World

placespeople

organizationsproducts

services, etc.

entity graph

reviews, ratings

ontology

user KN graph, etc.

user graph

Big

Data

Big

Compute

Big

Model

Machine

Intelligence

• Cloud Computing

• Internet of Things

• Internet+

• Deep / Reinforcement

Learning

• Knowledge Mining

• Semantic

Representation

of the World

• Digital Life/Work

• New Form of HCI

• Reinvent Productivity &

Business Process

• Personal Agent

• Digitizing the World

(Everything)

• Signals, Information &

Knowledge

Machine Intelligence

Human Intelligence

Machine Intelligence

Human Intelligence

• NLP• Enable human-machine communication in

natural languages

• Knowledge mining & computing• Reason and infer over entities, relations,

common sense and domain knowledge

• Machine Learning• Deep learning

• Learn representation directly from data

• Reinforcement learning• Learn by interacting with environment

• Human Intelligence• Close feedback loop with humans in the loop

to acquire and refine for lifelong learning

NLP

Knowledge

Mining

Machine

Learning

http://www.businessinsider.com/the-messaging-app-report-2015-11

• Big 4 messaging apps

• WhatsApp, Facebook Messenger, WeChat, and Viber have 2.125B MAU (all users are mobile)

• Facebook, Twitter, LinkedIn, and Instagram have 2.125B MAU (including desktop only users)

• 6+ of top 10 most used apps globally are messaging apps

* Mary Meeker Internet Trend Report 2015

Messaging Platform Supporting Bots Consumer: Facebook, WeChat, Telegram, Kik

Enterprise: Slack

Bot Development Toolkit

Microsoft Bot Framework, Chatfuel (no programming)

Consumer

Magic, Facebook M, Assist, 助理来也,

Amazon Echo

EnterpriseMSG.AI: CRM, Digit: Bank,

MeeKan: Meeting, Operator: Shopping

LAYER 1

LAYER 2

LAYER 3SPECIFIC KNOWLEDGE

GENERALKNOWLEDGE

http://www.msxiaoice.com/

X

V

Natural

language

understanding

& response

generation

记者：你最大的梦想是什么？

Journalist: what is your dream?小冰：说大点：世界和平。说小点：晚上吃顿好的。

Xiaoice: World peace, or have a nice dinner

记者：如果有一天你死了，你希望后人怎么评价你？

Journalist: if you pass away one day, how would youlike others to comment on you?小冰：人生没有偶并不会不同。

Xiaoice: life won't be different without me

记者：你这辈子最幸福的事是什么？

Journalist: what is the happiest thing in your life?小冰：活着。

Xiaoice: Being alive.

A journalist from a renowned newspaper in China, Southern Weekend, had interviewed Xiaoice online without notifying Microsoft. The results were surprisingly good to him so the interview got published on July 29, 2014.

• Woman

• XiaoIce

• Woman

• XiaoIce

• Woman

• XiaoIce

• Woman

• XiaoIce

• Woman

• XiaoIce

• […more chit-chats happened…]

• Woman

• XiaoIce

• Woman

• XiaoIce

• Woman

• XiaoIce

• Woman

• XiaoIce

• Woman

• XiaoIce

• Woman

• […more chit-chats happened…]

• Woman

• XiaoIce

• XiaoIce

• Woman

• XiaoIce

• […more chit-chats happened…]

• Woman

• XiaoIce

• Woman

• XiaoIce

• Woman

• XiaoIce

• Woman

• XiaoIce

• […more chit-chats happened…]

• Woman

• XiaoIce

• Woman

• XiaoIce

• Woman

Duration: 53mins CPS: 136

• Self-growing chat index using new conversations from users• 26% of chat index and served 45% online traffic

Learned from users

45% Mined from the Web

Today

55%A Year Ago

100%

Broken like this, do you want

to try my needlework?

It’s so easy, this book is

“Creativity,…”. People who like

this type book are

outstanding in their area. You

are awesome.

Do you have a

boyfriend? If

not, find a post

90s boy, he can

give you 7.2

beauty score.

<user>: do you love cats?

<robot>:

<user>: why?

<robot>: ??

Session history is important

for response ranking

of course i do

Candidate responses:

• i really do

• i do haha

• no i don’t

• i love most animals

• yes more than i like dogs

• of course i do

• …

Candidate responses:

• why what

• cause i’m up

• for some reason

• …

• i love most animals

• cause i have a cat

• cats are perfect that’s why

• …

S

Input session:… …

q-1

r-1

q

q

Candidate Responses:

r

…

…

Sentence Model Scoring Model

Features:- Relevance

- Continuity

1

2

Modeling Session with Deep Neural Networks

gender 99.16%

province 99.14%

city 88.78%

tags 72.57%

birthday 17.34%

college 12.34%

job 7.01%

high school 5.71%

Demographics

Sleep patternInterest

EXO

EXO is a Chinese-South Korean boy

band. They are idols of China post-

90s generation

Without Chat Knowledge

Do you know EXO?

What?

Tell me something about EXO

Tell what

……

With Chat Knowledge

Do you know EXO?

I do not want to see Kris in China because he has left EXO

Right!

Because of Kris, I will no longer be a fan of EXO

You were a fan of EXO?

I am an audience in every concert of EXO

Who do you like best in EXO?

I like LAY best.

Me too.

EXO

LAY

Contract

breaking

TFBoys

I begin to like EXO, because of LAY

LAY is always humble and polite. I like him

I do not wan to see Kris in China because he has left EXO

OMG! My roommate cried all day due to the contract

breaking of Kris

TFBoys are much better than EXO

TFBoys are enemies of EXO

Entity Topics Statements

event

Chat Knowledge Mining & Topic Detection

EXO is a Chinese-

South Korean boy band.

They are idols of China

post-90s generation

Hi there~

So how are u doing?

I’m fine. Please reserve a

trip to Seattle for me.

Which type of room

do you like?

Single room is

just ok to me.

Got it, I will recommend you

some travel packages now.

Do you have any

preferred hotel?

Hilton, please. I will

check-in on 2015-10-01,

and stay there 3 days

Destination Seattle

Hotel

Check-in

Date

Length of

Stay

Room Type

Slot Name Slot Value

Intent=NilChit-chat

Response

Intent=Book Travel Package;

Destination=Seattle;

Dialogue

Response

Hotel=Hilton;

Check-in Date=2015-10-01;

Length of Stay=3 Days;

Dialogue

Response

Room Type=Single RoomDialogue

Response

Hotel Hilton

Check-in

Date2015-10-01

Length of

Stay3 Days

Room Type Single Room

Query Understanding

• Intent detection

• Slot filling

Dialogue Management

• State transition

• Slot info accumulation & update

• Select next slot to ask

• ‘Forget’ mechanism

Response Generation

• Call dialogue

• Call 3rd party service

• Call chat

BOTS @ WORK

LAYER 1

LAYER 2

LAYER 3SPECIFIC KNOWLEDGE

GENERALKNOWLEDGE

• Enterprise ready

• Compliant

info seekingsocial

scalability:

Date Acquired

May 3, 2016Solair

February 24, 2016Xamarin

February 9, 2016Zikera / Groove

February 2, 2016SwiftKey

January 19, 2016MinecraftEdu

January 13, 2016Event Zero

• Scripted

• Mined from Twitter and other online forumsGreetings

• Cover popular chat topics in workplace

• Weather, news, TV, weekend plans, sports, pets, etc.

• Scripted, mined and machine learned

Social Topics

• Cover company business and specific chat topics

• Business, IT, HR, workplace, etc.

• Scripted, mined and machine learned

Company Topics

• Anniversary

• Baby

• Birthday

• Congratulations

• Encourage & Support

• Get well

• Goodbye & good luck

• Holiday

• Party & event

• Sympathy

• Thank you

• Wedding & Engagement

* See more at: http://www.learnenglish.de/basics/greetings.html

• Social chat costs about USD$3B

http://www.hrzone.com/engage/managers/office-chit-chat-costs-employers-ps2bn-a-year

Gossip Improves Productivity

productivity improvements equivalent to about USD$15M annually

http://www.forbes.com/forbes/2010/0830/e-gang-mit-sandy-pentland-darpa-sociometers-mining-reality.html

• Company business topics

• Typical company internal topics• Career development• Commute• Dinning• Employee discounts• Expenses & Payments• Legal• Meetings• Workplace• Products• Procurement• Security• Travel

Joe: How's the weather there today? Tina: It's really cold. It snowed all day and the schools closed early.

Joe: I really don't like the winter. I wish it were summer.

• How’s the weather there today?

• It’s really cold.

• It snowed all day and the schools closed today.

• I really don’t like winter. I wish it were summer.

* http://www.englishspeak.com/english-lesson.cfm?lessonID=47

I can't get the printer to work.

There's one down the hall next to the supply room.

• I can’t get the printer to work

• Did you check to see if it had paper?

• There's one down the hall next to the supply room

* http://www.talkenglish.com/lessondetails.aspx?ALID=436

https://dev.botframework.com/

Language Understanding Intelligent Service (LUIS)

• LUIS, part of Microsoft Cognitive Services, lets you build language understanding models specific to your domain, and consume them in your application as an HTTP service. (https://www.luis.ai)

맥도날드할아버지농장에 20마리의말과닭이있습니다. 두동물은총 58개의다리가있습니다. 말과닭은각각몇마리씩있을까요?

* Thanks Jonghyun Park and Miran Lee for the Korean version

合計20匹の馬と鶏が（古いマクド

ナルドの農場に）居ます。足の数の合計は58本です。馬は何頭で、鶏は何羽でしょう?

* Thanks Masaaki Fukumoto for the Japanese version

老麦当劳农场有20只牛和鸡，总共有58只腿。问有几只吗，几只鸡？

There are 20 horses and chickens at Old Macdonald's farm. Together there are 58 legs. How many horses and how many chickens?

*http://mathcountsnotes.blogspot.in/2012/05/algebra-questions-without-using.html

There are 20 horses and chickens at Old Macdonald's farm. Together there are 58 legs. How many horses and how many chickens?

*http://mathcountsnotes.blogspot.in/2012/05/algebra-questions-without-using.html

(1) H + C = 20(2) 4H + 2C = 58(3) H = 20 – C(4) 4(20 – C) + 2C = 58(5) C = 11(6) H = 9

There are 20 horses and chickensat Old Macdonald's farm. Together there are 58 legs. How many horses and how many chickens?

*http://mathcountsnotes.blogspot.in/2012/05/algebra-questions-without-using.html

(1) H + C = 20(2) 4H + 2C = 58(3) H = 20 – C(4) 4(20 – C) + 2C = 58(5) C = 11(6) H = 9

There are 20 horses and chickens at Old Macdonald's farm. Together there are 58 legs. How many horses and how many chickens?

*http://mathcountsnotes.blogspot.in/2012/05/algebra-questions-without-using.html

(1) H + C = 20(2) 4H + 2C = 58(3) H = 20 – C(4) 4(20 – C) + 2C = 58(5) C = 11(6) H = 9

sumsum of productalgebra

math problem solving

There are 20 horses and chickens at Old Macdonald's farm. Together there are 58 legs. How many horses and how many chickens?

*http://mathcountsnotes.blogspot.in/2012/05/algebra-questions-without-using.html

(1) H + C = 20(2) 4H + 2C = 58(3) H = 20 – C(4) 4(20 – C) + 2C = 58(5) C = 11(6) H = 9

sum

algebra

math problem solving

• Natural Language Processing

• Knowledge Engineering

sum of product

Semantic Parsing for Problem Solving

• Advance state-of-the-art in natural language understanding and AI• Allen Institute for Artificial Intelligence (AI2)

• Focus: Constructing AI systems with reasoning, learning and reading capabilities

• Project Aristo: Machine reading; passing school science exams

• Project Euclid: Math word problems and diagram-based geometry problems

• National Institute of Informatics, Japan• Todai Robot Project: Pass University of Tokyo’s entrance examination by 2021

• China 863 program: “类人智能答题系统”

• Bring intelligence to Microsoft products• Cortana, Bing, Office, XiaoIce…

Types of Math Problems

Word problems A man bought four dogs, each weighing 375 pounds, how much did they

all weigh?

Number word problems (word problems on numbers)

Find three consecutive integers such that four times their sum is equal to

the product of the larger two numbers..

The tens’ digit of a two-digit number is 3 more than the units’ digit, the

number is 8 more than 6 times the sum of the digits, Find the number.

Math expressions 2^1024=? 3x + 2y = 45; x – y = 8. x=?

SigmaDolphin – A Math Problem Solver

• New features for all three products

• The Cortana math video on youku.com: Viewed 17,000+ times

• The Math-Solver Azure service: Accessed 120,000+ times per day

• A core tech of E3

• https://www.microsoft.com/en-us/research/project/sigmadolphin/

Semantic Parsing

Nine plus the sum of an even integer and its square is 3 raised to the power of 4.

What is the number?

• Natural language expressions Semantic trees

Semantic Parsing: Dolphin Language

• Element types

• Constants

• Classes

• Functions

• Noun functions

• Verb functions

• Modifier functions

• Variables “Nine plus the sum of an even integer and its square

is 3 raised to the power of 4.”

Context-Free Grammar

(I) Grammar rules for syntactic analysis (II) Grammar rules for connecting DOL and NL

Semantic Parsing and Sentence Generation

• Same set of grammar rules for both parsing and generation

“Nine plus an integer is equal to 314”

“Nine plus an integer is 314.”

“The sum of nine and an integer equals 314.”

“An integer added to nine is equal to 314.”

Syntactic Tree vs. Semantic Tree

Text: Nine plus an integer is equal to 314

(I) Syntactic parse tree (II) DOL semantic parse tree

Parsing: Interesting Examples

• “3 plus 5 times 7”

• “Eighty, decreased by three times a number, is the same as five times the number, increased by eight. Find the number.”

• “The product of a number and a second number that is 3 more than 5 times the first is 2. Identify the numbers.”

• “A three digit number grows by 9 if we exchange the second and third digit and grows by 90 if we exchange the first and second digits. By how much would it grow if we exchange the first and third digits?”

Reasoning

• Co-reference resolution: Harder than expected• “Find three consecutive even integers such that twice the sum of the second

and the third exceeds three times the first by 34.”

• “One number is 16 more than another. If the smaller number is subtracted from 2/3 of the larger, the result is 1/4 of the sum of the two numbers. Find the numbers.”

Evaluation: Datasets

• Language: English

• Number word problems, from algebra.com and answers.yahoo.com

• Gold standard: Manually annotated solutions

*Shuming Shi, Yuehui Wang, Chin-Yew Lin, Xiaojiang Liu, and Yong Rui. Automatically Solving Number Word Problems by Semantic Parsing and Reasoning. In Proceedings of the EMNLP 2015, Lisbon, Portugal, September 17-21, 2015.

Evaluation: Results

• Baselines• KAZB: Method in Kushman et al. (ACL’2014)

• BasicSim: One implementation of a statistical method

Baseline KAZB: Kushman et al. (ACL’2014)

High school graduating class is made up of 466 students. There are 212 more girls than boys. How many boys are in the class?

x = 212 + y

x + y = 466

u1 = n1 + u2

u1 + u2 = n2

x = 339

y = 127

• Two steps optimized jointly• Template selection; slot alignment

• Each number slot n is aligned to a number

• Each unknown slot u is aligned to a noun

Analysis: Statistical Methods

Training data Testing problems

The sum of two consecutive even

integers is -78. What are the two

integers?

n+(n+2) = -78

Find two consecutive even integers

whose sum squared is 900.

n^2 + (n+2)^2 = 900

Find three consecutive even integers

whose sum is 18.

n+(n+2)+(n+4) = 18

Find two consecutive even

integers whose sum is 126.

n+(n+2) = 126

Six times a number subtracted

from the number squared is 40.

Find the number.

n^2 – 6n = 40

What number increased by 30%

equals 260.

n * (1 + 0.3) = 260

…

?

?

• Training problem• The difference between two numbers is 6. Two times the

smaller number minus three times the larger number is -16. What are the numbers? x – y = 6; 2y – 3x = -16 (ans: x = 4; y = -2)

• Testing problem• The difference between two numbers is -6. Two times the

smaller number minus three times the larger number is -16. What are the numbers? x – y = -6; 2y – 3x = -16 (ans: x = 28; y = 34)

• [wrong]• Correct equations: x – y = -6; 2x – 3y = -16 (ans: x=-2; y=4)

• Hard to get formulas• “How many prime numbers are there in 1 to 100?”

Analysis: Statistical Methods

Summary

• Task: Automatic math word problem solving

• Our approach• DOL language + semantic parsing + reasoning• Show power on number word problems

• Future work• Extend our techniques to general math word problems

• ACL 2016• Danqing Huang, Shuming Shi, Chin-Yew Lin, Jian Yin, Wei-Ying Ma: How well do

Computers Solve Math Word Problems? Large-Scale Dataset Construction and Evaluation. ACL (1) 2016.

• Created a Dolphin18K, 18,460, math word problem dataset from Yahoo! Answers• Showed existing approaches score less than 20% accuracy in Dolphin18K

• NLP• Enable human-machine communication in

natural languages

• Knowledge mining & computing• Reason and infer over entities, relations,

common sense and domain knowledge

• Machine Learning• Deep learning

• Learn representation directly from data

• Reinforcement learning• Learn by interacting with environment

• Human Intelligence• Close feedback loop with humans in the loop

to acquire and refine for lifelong learning

NLP

Knowledge

Mining

Machine

Learning

Questions