watson devcon 2016 - engage and be engaging: building compassionate and personalized conversational...

Engage and be Engaging: Building Compassionate and Personalized Conversational Systems

Rama AkkirajuIBM Distinguished Engineer

Please note IBM’s statements regarding its plans, directions, and intent are subject to change orwithdrawal without notice and at IBM’s sole discretion.

Information regarding potential future products is intended to outline our general productdirection and it should not be relied on in making a purchasing decision.

The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code or functionality. Information about potential futureproducts may not be incorporated into any contract.

The development, release, and timing of any future features or functionality described for our products remains at our sole discretion.

Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user’s job stream, the I/O configuration, the storage configuration, and theworkload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.

11/21/16World of Watson 2016 2

Consider this conversation

Presentation Title8/11/16

WatsonDeveloperConference2016 3

What kind of cuisine do you prefer?

What’s a good place to eat in San Francisco?

Are you a vegetarian?

Italian, or Mexican, or Indian, or Chinese

Do you like a fine dining experience or fast food?

Yes

Are you driving, taking a taxi or using public transportation?

Fine dining

Do you prefer a restaurant that has a parking lot of its own, or are you okay to find street parking?

Driving

I can recommend three choices: Greens on Marina, Gracious Madre on Mission, and Enjoy Chinese on Kearney…

Parking lot would be good

Works! But takes too many turns to arrive at a solution and solution is not personalized

Consider thisrevisedconversation



What’s a good place to eat in San Francisco?

Better! Takes far fewer turns and offers much more personalized experience

What kind of cuisine do you prefer?

Joan, because you are a vegetarian, open to experiences, prefer fine dining, and are likely to drive given you are currently renting a car, I’d recommend Greens, a fusion vegetarian Restaurant on 2 Marina Blvd. It is a 4.1/5-rated vegetarian restaurant with ample parking and is within 1 mile from your current location, so you could even walk to it.

Italian, or Mexican, or Indian, or Chinese

Cognitive assistant is demonstrating understanding of Joan’s preferences, personality, life events and personalizing answers accordingly.

Consider this conversation



There are proven, effective ways to deal with bullying and bullies. Would you like me to help you?

I’m getting bullied in school. I’m so upset.

Are you facing verbal, physical or social bullying?

Yes. Please.

Where has this been happening?

I guess, social and verbal bullying. I can’t focus on my studies. My situation is so hopeless.

Why do you think they are bullying you?

On the school bus, during recess, after school, during projects –everywhere

Have you tried avoiding those who are bullying you?

I think it’s because I’m overweight. They call me names. It’s so demeaning.

Here is a short video on how to deal with bullying. This can help you.

I can’t. they are my teammates. I have to do a project with them.

Works! But lacks compassion




Kathy, I’m so sorry to hear that you are facing bullying in school. Don’t feel distressed. You are not alone in this fight. There are proven, effective ways to deal with bullying and bullies. I’m here to listen and to help you in any way I can. Would you like me to help you?

I’m getting bullied in school. I’m so upset.

Are you facing verbal, physical or social bullying?

Yes. Please.

Oh! That is indeed unfortunate! Don’t lose hope, Kathy. You can manage the situation. I’m here to help you. First, let me ask you this. Where has the bullying been happening?

I guess, social and verbal bullying. I can’t focus on my studies. My situation is so hopeless.

Better! Compassionate companion




I see. Why do you think they are bullying you?

On the school bus, during recess, after school, during projects –everywhere

I’m sorry to hear that. That is indeed very insensitive of them to say that. Have you tried avoiding those who are bullying you?

I think it’s because I’m overweight. They call me names. It’s so demeaning.

I see. Kathy, I can help you in the following ways. I have access to some videos that may help you. After you had a chance to watch them and grasp the techniques suggested, if you wish, we can also do some role play. That way, you can be prepared to deal with them next time you face them.

I can’t. They are my teammates. I have to do a project with them.

Cognitive assistant is demonstrating understanding of Kathy’s emotional state and is working with her in a compassionate manner.

Better! Compassionate companion

To build compassionate and personalized conversational systems, three core models are needed:



Naturally3

Interact2

People1

Understand people at a deeper level

Understand and respond in various mediums in which interactions can occur

Interact2 naturally3 with people1

Input types: Text, speech, gesturesMediums: Computers, mobile devices, robots, avatars

Understand styles of human interaction and optimize human-computer interaction

Building block 1: People modeling/user modeling

Naturally3

Interact2

People1

People are complex beings with many facets

“Who are these people?”

“How do they make

decisions?”

“What do they value?”

“How do they think?”

“What is their context?”

“What are their

preferences?”

“What are their

emotions?”

“How do they act?”

“How are they feeling?”

“What are their skills?”

“How do they express

themselves?”

User modeling: Our framework

An individual takes action based on the combination of his/her unique being and environment

Act

Be

Feel

Context

Think Options

Explore&

Decide

Inner State Environment Outer State

User modeling: Our framework

Act

Search

Preferences

Communications Decisions

Commitments

Purchases

Context

Life Style, Events Sociological

Economic

Political Technological

Options

Price Promotions

Products/Services Place

FeelPerceptions

Emotions

Sensations

Attitudes

Influences

Sentiments

Be

Personality

Needs, Values

Beliefs

Motives

Identity

Goals, Ambitions

Interests

Think

Knowledge

Skills

Opinions

Cognitive Style

Explore&

Decide

Choices Consequences

Session

Intent

Time

11/21/16

WatsonDeveloperConference201613

Ready to use Still in R&D

Use Personality Insights to engage with individuals at personalized level

11/21/16


Watson-pi-demo.mybluemix.net

Source: https://www.army.mil/artic le/78562/Leaving_the_battlefield__Soldier_shares_story_of_PTSD

Emotional Analysis helps build empathetic systems

11/21/16


sentiment-and-emotion.mybluemix.net

Use Tone Analyzer to understand and fine tune your message

11/21/16


tone-analyzer-demo.mybluemix.net

Building block 2: Human interaction patterns

Naturally3

Interact2

People1

Natural interactions among people

11/21/16


Verbal (expressive, aggressive, passive) , Non-verbal (gestures, facial expressions, postures)

Dialog strategies Start

Giving an extra

Acknowledging Need

Description of Need

Anger

Acknowledging w/o encouraging

Refocus statements

Active Listening

Possibility of mistake Admitting

mistake

Allowing venting

Apology

Smiles

Arranging Follow-up

Need cannot be fulfilled on the spot

Assurance of effort

Assurance of result

Mistake has been made Bonus buyoff Broken recordUncooperative

customer

Closing positively

Common Courtesy

Completing Follow-up

Contact Security

Aggressiveness Disengaging

Distraction

Frustration Empathy statement

Expediting

Expert Recommend

ation

Explain Reasoning or

action

Embarrassment Face-Saving Out

Conflict

FindingAgreement Points

Following up

Helpless

Offering Choice

Empowering

Preventive strike

Privacy insurance

Privacy concern

Probing question

Pros and Cons

Providing Alternatives

Providing Takeaway

Confusion ProvidingExplanation Questioning

instead of stating

Referral to supervisor

Referral to 3rdparty

Lost focus RefocusInappropriate behavior Setting Limits

Critical

Neutral mode

Summarize the conversation

Silence

Thank-you

TimeoutUse customer

name

Verbal Softeners

When questionYou’re rightAction

Negative Emotion

Monologue

End External Giving

Emotions

General

States

GratitudeStatementHappiness

Work by Michal Shmueli-Scheuer, Jonathan Herzig, Guy Feigenblat, David Konopnicki

Tone-aware Expressions in Conversations

11/21/16


food-coach.mybluemix.net

11/21/16World of Watson 2016

Auto-generated response in customer care domain

11/21/16


Agent responses are auto-generated.

Bot personas and expressive paraphrasing

11/21/16


Personalized conversational systems

11/21/16


Based on customer’s emotion and personality the cognitive bot personalizes responses

Detect user emotion

Bot understands user’s personality and preferences and provides recommendations

Change bot expression based on user emotion

Building block 3: Understand various mediums in which human-computer interaction can occur

Naturally3

Interact2

People1

Tone-aware Expressions in Conversations

11/21/16


Speech-to-Text Text-to-Speech

speech-to-text-demo.mybluemix.net

text-to-speech-demo.mybluemix.net

Deriving human expressions from speech, images and video

Speech– Non-verbal clues: pauses, volume, intonation, pitch

11/21/16


Video, images– Gestures– Facial expressions– Eye contact– Posture– Tone of voice– Distance

Bringing it all together:Building compassionate and personalized conversational systems

Naturally3

Interact2

People1

Deep-dive: Watson Food-Coach App

Demonstrates how to incorporate tone and emotion detection into Conversations and to design compassionate dialog flow

29

11/21/16World of Watson 2016

Intents

Entities

Food Coach Dialog Flow

31

Building Compassionate and Personalized Conversational Agents

11/21/16


Naturally3

Interact2

People1

• We are developing the platform and the building blocks for implementing compassionate and personalized conversational agents.

• Checkout our services, Build your conversational apps and give us your feedback!

Notices and disclaimers continued

Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products in connection with this publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. IBM does not warrant the quality of any third-party products, or the ability of any such third-party products to interoperate with IBM’s products. IBM EXPRESSLY DISCLAIMS ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.

The provision of the information contained herein is not intended to, and does not, grant any right or license under any IBM patents, copyrights, trademarks or other intellectual property right.

IBM, the IBM logo, ibm.com, Aspera®, Bluemix, Blueworks Live, CICS, Clearcase, Cognos®, DOORS®, Emptoris®, Enterprise Document Management System™, FASP®, FileNet®, Global Business Services ®, Global Technology Services ®, IBM ExperienceOne™, IBM SmartCloud®, IBM Social Business®, Information on Demand, ILOG, Maximo®, MQIntegrator®, MQSeries®, Netcool®, OMEGAMON, OpenPower, PureAnalytics™, PureApplication®, pureCluster™, PureCoverage®, PureData®, PureExperience®, PureFlex®, pureQuery®, pureScale®, PureSystems®, QRadar®, Rational®, Rhapsody®, Smarter Commerce®, SoDA, SPSS, Sterling Commerce®, StoredIQ, Tealeaf®, Tivoli®, Trusteer®, Unica®, urban{code}®, Watson, WebSphere®, Worklight®, X-Force® and System z® Z/OS, are trademarks of International Business Machines Corporation, registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at: www.ibm.com/legal/copytrade.shtml.

33 11/21/16World of Watson 2016

Thank you

11/21/16
