lecture 6: watson and the social web (2014), chris welty

© 2011 IBM Corporation

Watson and the Social Web

Chris Welty IBM Watson Group ibmwatson.com

Do Not Record. Do Not Distribute.


What is Cognitive Computing?

§  Increasingly, machines are being asked to add their computational power to problems which are not inherently solvable

§ Traditionally, these problems came from AI – The hardest AI problems are the easiest for human intelligence:

vision, speech, natural language – these are not actually associated with “being intelligent”

– Human intelligence provides solutions, but does not scale § Cognitive Computing is founded on four principles

Learn & improve. Cognitive computing systems focus on inexact solutions to unsolvable problems that utilize machine learning and improve over time. Often they combine multiple approaches and must integrate them effectively. They must learn from humans, in more and more seamless ways.

Speed&Scale. Cognitive computing harnesses the clear advantage machines have over humans in their ability to perform mundane tasks of arbitrary complexity repeatedly, whether it is the scale of the data or the complexity of the task.

Interact in a natural way. Cognitive computing provides technologies that support a higher level of human cognition by adapting to human approaches and interfaces...over the next several decades it will incorporate essentially all the ways humans sense and interact.

Assist & augment human cognition. Cognitive computing addresses problems that lie squarely in the province of human intelligence, but where we can't handle the volume of information, penetrate the complexity or otherwise extend our reach (physically). The goal is to be useful, not universally correct.

or Computers can be incorrect and still prove useful!


Examples of Cognitive Computing

§ Web Search

§ Image Search

§ Event Search

§ Recommendations

§ Natural Language Processing


What is Watson?

§ Open Domain Question-Answering Machine § Given

– Rich Natural Language Questions – Over a Broad Domain of Knowledge

§ Delivers – Precise Answers: Determine what is being asked & give precise response – Accurate Confidences: Determine likelihood answer is correct – Consumable Justifications: Explain why the answer is right –  Fast Response Time: Precision & Confidence in <3 seconds – At the level of human experts

– Proved its mettle in a televised match – Won a 2-game Jeopardy match against

the all-time winners –  viewed by over 50,000,000

4


What is Jeopardy?

§ Jeopardy! is an American quiz show

– 1964 – Today – Household name in U.S.

§ answer-and-question format – contestants are presented with

clues in the form of answers – must phrase their responses in

question form. – Open domain trivia questions,

speed is a big factor §  Example

–  Category: General Science –  Clue: When hit by electrons, a

phosphor gives off electromagnetic energy in this form

–  Answer: What is light?


Social Computing: What’s the connection?

§ Social Web as Data Source: – The vast majority of sources Watson

used to answer questions came from community-created data

–  Adapting Watson to a new problem requires the same kind of information about that problem

§  Social Machines: – Watson combined with people is a

powerful proposition

§  Social Web as Application: – Watson’s major advance is in

understanding natural language, the technology can be useful to augment social interaction


$200 If you are looking at the wainscoating, you are looking in

this direction.

$1000 The first person

mentioned by name in ‘The Man in the Iron

Mask’ is this hero of a previous book by the

same author.

7

The Jeopardy! Challenge Hard for humans, hard for machines

Broad/Open Domain

Complex Language

High Precision

Accurate Confidence

High Speed

$600 In cell division, mitosis

splits the nucleus & cytokinesis splits this liquid cushioning the

nucleus

$800 The conspirators against

this man were wounded by each other while they

stabbed at him

But hard for different reasons.

For people, the challenge is knowing the answer For machines, the challenge is understanding the question

What is down? Who is

D’Artagnan?

What is cytoplasm?

Who is Julius Caesar?


The Winner’s Cloud What It Takes to compete against Top Human Jeopardy! Players

Winning Human Performance

2007 QA Computer System

Grand Champion Human Performance

Top human players are remarkably

good.

Each dot – actual historical human Jeopardy! games

More Confident Less Confident

Develop against a metric!


2007 QA Computer System

In 2007, we committed to making a Huge Leap!

More Confident Less Confident

Each dot – actual historical human Jeopardy! games

Computers? Not So Good.

Winning Human Performance

Grand Champion Human Performance

The Winner’s Cloud What It Takes to compete against Top Human Jeopardy! Players


DeepQA: The Technology Behind Watson An example of a new software paradigm

. . .

Answer Scoring

Models

Answer & Confidence

Question

Evidence Sources

Models

Models

Models

Models

Models Primary Search

Candidate Answer

Generation

Hypothesis Generation

Hypothesis and Evidence Scoring

Final Confidence Merging & Ranking

Synthesis

Answer Sources

Question & Topic

Analysis

Question Decomposition

Evidence Retrieval

Deep Evidence Scoring



Learned Models help combine and

weigh the Evidence

DeepQA generates and scores many hypotheses using an extensible collection of Natural Language Processing, Machine Learning and Reasoning Algorithms. These gather and weigh evidence over both unstructured and structured content to

determine the answer with the best confidence. Content from Community Resources!


Example Question

In 1894 C.W. Post created his warm cereal drink Postum in this Michigan city

Related Content (Structured & Unstructured)

Primary Search

1985

Post Foods

aramour

General Foods

Grand Rapids

…

Battle Creek

…

…

Candidate Answer Generation

1)  Battle Creek (0.85) 2)  Post Foods ( 0.20) 3)  1985 (0.05)

Merging & Ranking

Evidence Retrieval

Question Analysis

Keywords: 1894, C.W. Post, created … Lexical AnswerType: (Michingan city) Date(1894) Relations: Create(Post, cereal drink) …

[0.58 0 -1.3 … 0.97]

[0.71 1 13.4 … 0.72]

[0.12 0 2.0 … 0.40]

[0.84 1 10.6 … 0.21]

[0.33 0 6.3 … 0.83]

[0.21 1 11.1 … 0.92]

[0.91 0 -8.2 … 0.61]

[0.91 0 -1.7 … 0.60]

Evidence Scoring

Need thousands of Q/A pairs for training!


Planet Fitness

Role of Answer Typing in QA

Type Information - a crucial hint to get the correct answer

ASTRONOMY: In 1610 Galileo named the moons of this planet for the Medici brothers

Telescope

Giovanni Medici

Sidereus Nuncius

Jupiter

Ganymede Telescope (Instrument)

Giovanni Medici (Person)

Sidereus Nuncius (Book)

Jupiter (Planet)

Ganymede (Moon)

Terms Associated with Clue Context (e.g. via Keyword Search)

Planet Fitness (Planet)


§  This fish was thought to be exLnct millions of years ago unLl one was found off South Africa in 1938

§  Category: ENDS IN "TH" §  Answer:

§  When hit by electrons, a phosphor gives off electromagneLc energy in this form

§  Category: General Science §  Answer:

§  Secy. Chase just submiXed this to me for the third Lme-‐-‐guess what, pal. This Lme I'm accepLng it

§  Category: Lincoln Blogs §  Answer:

The type of thing being asked for is often indicated but

can go from specific to very vague

coelacanth

light (or photons)

his resigna4on

13

Answer Typing for Jeopardy!?


Broad Domain

Our Focus is on reusable NLP technology for analyzing vast volumes of as-is text. Structured sources (DBs and KBs) provide background knowledge for interpreting the text.

We do NOT attempt to anticipate all questions and build databases.

We do NOT try to build a formal model of the world


Sources for typing evidence

§ DbPedia & Freebase – Wide coverage of well-known entities – Taxonomy (MountainsOfNepal → Mountain) – Good type coverage, but not many synonyms

• E.g. what about “summit”

§  Wikpedia Categories – Wide coverage of entities and type name synonyms – Noisy (many errors)

§  Wikipedia Intro –  First sentence always indicates the most common type of the entity – Highly reliable, low coverage of types

Communities can scale data collection!


Typing Impact on Jeopardy! clues

61.5% 62.0% 62.5% 63.0% 63.5% 64.0% 64.5% 65.0% 65.5% 66.0% 66.5%

An ensemble of TyCor components

+ ~10%


Many sources of evidence

In 1894 C.W. Post created his warm cereal drink Postum in this Michigan city

Related Content (Structured & Unstructured)

Primary Search

1985

Post Foods

aramour

General Foods

Grand Rapids

…

Battle Creek

…

…

Candidate Answer Generation

1)  Battle Creek (0.85) 2)  Post Foods ( 0.20) 3)  1985 (0.05)

Merging & Ranking

Evidence Retrieval

Question Analysis

Keywords: 1894, C.W. Post, created … Lexical AnswerType: (Michingan city) Date(1894) Relations: Create(Post, cereal drink) …

[0.58 0 -1.3 … 0.97]

[0.71 1 13.4 … 0.72]

[0.12 0 2.0 … 0.40]

[0.84 1 10.6 … 0.21]

[0.33 0 6.3 … 0.83]

[0.21 1 11.1 … 0.92]

[0.91 0 -8.2 … 0.61]

[0.91 0 -1.7 … 0.60]

Evidence Scoring


Watson as part of a social machine

§ Watson makes mistakes: – This woman was the first to witness her husband resign from the U.S. Presidency.

– This U.S. City’s largest airport is named for a world-war II hero; its second largest for a world-war II battle.

§ These mistakes are typically obvious to people – Even when they don’t know the answer – Watson isn’t stupid, it solves problems differently – Often these multiple perspectives can combine productively

•  E.g. add a “dismiss” button to the answer interface

Richard Nixon Dolly Madison

Pat Nixon

Watson can adapt and learn from its users!


Cut to the chase….. Watson emerges victorious


Technology marches forward…


Adapt Watson

Models

Answer & Confidence

Question

Evidence Sources

Models

Models

Models

Models

Models

Answer Sources

. . .

Answer Scoring

Primary Search

Candidate Answer

Generation



Final Confidence Merging & Ranking

Synthesis Question &

Topic Analysis

Question Decomposition

Evidence Retrieval

Deep Evidence Scoring



Learned Models help combine and

weigh the Evidence

What does it take to use Watson in a new domain? (medical diagnosis, call centers, etc...)

Gathering significant numbers of question-answer pairs is proving to be one of the most significant challenges for adapting Watson. Can the social web help?

Community created!


Integrating Watson in Social Interaction?

Did you hear about Bob?

No

He’s taking a year off to climb the tallest mountain!

The tallest mountain is Mount Everest.

Wow.

me

me

Jeff

Watson

Jeff


Privacy – a blessing and a curse

Need to protect our data, but… Crime on the web, the social web, is very real

Identity theft Credit card, bank, insurance fraud Terrorist networks

Medical diagnosis

Monitoring your profile for health-related information ICT for depression

Calendar, appointments, traffic, spreading disease


The arrival of Cognitive Computing

Learn & improve. The core of Watson is a group of over 100 independent algorithms that approximate a solution to the “is this the right answer to the question” problem. Achieving winning (human expert) performance, required two hallmarks of cognitive computing systems: a metric to measure improvements to the system (the winners cloud), and a significant ground truth (over 200K Q-A pairs).

Speed&Scale. Watson used big data, as well as a 3000 node cluster for massive computation to get answering speeds down into the 2s range.

Interact in a natural way. Watson was a significant step forward in natural language understanding, the most basic interface for humans. Say goodbye to your mouse…

Assist & augment human cognition. Watson depended on primarily a set of background documents (the corpus). The value of having access to this kind of fact-finding power over a large (and possibly changing) corpus provides a clear augmentation to human abilities.




0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% % Answered




UTI

Diabetes

Influenza

hypokalemia

Renal failure

esophogitis

Diagnosis Models Confidence

Most Confident Diagnosis: UTI

Symptoms

Tests/Findings Medica4ons

Family History

Notes/Hypotheses

Huge Volumes of Texts, Journals, References, DBs etc.

Pa4ent History


…and for Social Web

§ First and foremost, social web analytics (e.g. recommendations) and Social Computing in general lie clearly in the realm of Cognitive Computing

– Uncertainty, natural language, human intelligence –  Inexact solutions that can improve with time, training – Problems & solutions need metrics to be solvable

§ All cognitive computing systems require ground truth data – This data is expensive to collect – Crowdsourcing is a key new technology/approach

§ The user interface moving closer to people – Natural language, speech, gestures –  In addition, integrating the collection of training data seamlessly into the interface

is a key development § Cognitive computing systems require integration of multiple, disparate, data

sources – Structured, unstructured, semi-structured – curated, crowdsourced