watson api use case demos for the nittany watson challenge

Watson Use Case Demosfor the Nittany Watson Challenge

January 2017Mike Pointer, Watson Sr. Solution Architect

IBM WATSON CAPABILITIES

2

Watson Microservices

Language Services

Speech Services

Vision Services

Data Services

Embodied Cognition

Watson Knowledge Studio

25+ Services

Discovery

25+ WATSON MICROSERVICES

3

Language ServicesAlchemyLanguageConversationDocument ConversionLanguage TranslatorNatural Language ClassifierPersonality InsightsRetrieve and RankTone AnalyzerEntity ExtractionSentiment AnalysisEmotion AnalysisKeyword ExtractionConcept TaggingRelation ExtractionTaxonomy ClassificationAuthor Extraction

Language DetectionText ExtractionMicroformats ParsingFeed DetectionLinked Data SupportSpeech ServicesSpeech to TextText to SpeechVision ServicesVisual RecognitionSimilarity SearchData Insight ServicesAlchemyData NewsDiscoveryTradeoff AnalyticsEmbodied Cognition ServicesIntu

Watson’s APIs are the cognitive building blocks that harness our data.

Vision Recognition

Conversation

Use Cases and Demos

Speech to Text – Multiple SpeakersChatbot – School NavigatorWatson Conversation ServiceWatson Discovery ServiceWatson Knowledge Studio

Speech to Text

5

SPEECH TO TEXT WITH DIARIZATION – MULTIPLE SPEAKERS

6© Copyright IBM Corporation 2016 https://speech-to-text-demo.mybluemix.net/

https://speech-to-text-demo.mybluemix.net/

Speech to SpeechSpeech to Text Language Translation Text to Speech

7

SPEECH TO SPEECH – WITH LANGUAGE TRANSLATION

8© Copyright IBM Corporation 2016

https://speech-to-speech-app.mybluemix.net/?cm_mc_uid=75059567007614843110528&cm_mc_sid_50200000=

https://speech-to-speech-app.mybluemix.net/?cm_mc_uid=75059567007614843110528&cm_mc_sid_50200000

Watson Virtual AgentEngagement Your Customers, Students, Faculty, Citizens

9

NYC School FinderMatching Personality with Assisted Decision Making (Tradeoff Analytics)

11

NYC SCHOOL FINDER

12

https://nyc-school-finder.mybluemix.net/?cm_mc_uid=75059567007614843110528&cm_mc_sid_50200000=

https://nyc-school-finder.mybluemix.net/?cm_mc_uid=75059567007614843110528&cm_mc_sid_50200000

School NavigatorWatson Conversation Service Chatbot

13

©IBM 2016

Choosing a graduate school is BIG decision, let Watson help…

14

Many students are overwhelmed when deciding which graduate school to attend. Many do not know who to ask, and turn to costly advisors who do not know what is best for their specific needs and goals.

THE OPPORTUNITY: Watson can help aspiring graduate students find the best graduate school for them by understanding their preferences, and previous academic scores to build a candidate profile to match to relevant schools. Watson will also educate the user on everything from the admissions process to the best practices for the GMAT exam. The School Navigator uses an interactive interface to engage and advise aspiring students in one of their most important decisions.

©IBM 2016

Link: http://schoolnavigator.mybluemix.net/#/username: watsonpassword: p@ssw0rd

SchoolNavigator Demo

15

http://creditcardnavigator.mybluemix.net/

http://creditcardnavigator.mybluemix.net/#/

Expertise Finder

16

EXPERTISE FINDER

17

ChallengeOrganizations often staff projects with available resources without certainty that the resource is the best fit. That’s because it can be extremely time consuming to align relevant personnel and expertise to projects and tasks. This can lead to under-utilization of resources, delivery challenges, cost overruns and missed opportunities for organic growth. Especially relevant for Financial Services, Law, Research & Development, Consulting & Engineering organizations.

Watson inspired solution to the problem Watson can enable enterprises to efficiently locate and identify expertise across the firm. By matching needed expertise with relevant employee experience, Watson streamlines a company’s internal discovery process for resource matching to projects.

Expertise Finder - Legal

Username: watson

Password: w@ts0n

http://legal-expertise-finder.mybluemix.net/login

Multimedia EnrichmentGetting concepts from video and audio

18

©IBM 2015 IBM Confidential IBM Internal ONLY

Live Demo

1/24/2017 19

http://cnn-media.mybluemix.net/#/Username – cnnPassword – cnn123

http://cnn-media.mybluemix.net/#/


The Accelerator provides processing of multimedia content to acquire, ingest and enrich it

• Input can be any type of multimedia content (images, social media text, video transcripts, audio etc.) and Output is the enriched metadata based on processing them using relevant Watson API’s.

• Leverages relevant Watson APIs

• Index the enriched content in an efficient storage based on an canoncial schema.

1/24/2017 20


Multimedia Enrichment Pipeline for Audio enrichment

1/24/2017 21

Audio Ingestion (optional)• Extract Audio from the

videos• Preserve metadata

Transcribe Audio (optional)• Transcribe Audio• Maintain start & end times• Capture word alternatives• Speech to Text

Entity Extraction• Invoke AlchemyLanguage

and/or Custom WKS AlchemyLanguage Model

Keywords• Highly relevant terms &

phrases• AlchemyLanguage

Expand Keywords• Word2Vec as a API• Trained on Wikipedia and

other generally available corpus

Tone Analysis• Extract the top emotion,

social tone and writing tone• Tone Analysis

Sources

Pull / Poll

Taxonomy & Relationships• Classify or categorize content• Different types of relations

between detected entities• AlchemyLanguage

Visual Recognition• Use random frames / samples• Class & Face Detection• Visual Recognition

Store in Enrichment DB’s(Cloudant, IBM Graph)

Closed Caption Transcripts


Multimedia Enrichment Pipeline for Video files

1/24/2017 22

Enrichment Pipeline Service• Pass through all text related

pipelines


VideoProcessorService• Transcribe• ImageCapture/Visual

Recognition

Video Workload Manager • UI to select file/URI and have

‘processed’, indicate progress, See Results

Video URI

Status/Complete events

Start Event w/ json ID in DB.

Insert JSON/Update JSON

Read JSON/Update JSON

A workload manager would drive workload. We can horizontally scale the VideoProcessor and Enrichment Pipeline.

VideoProcessorService• Transcribe• ImageCapture/Visual

Recognition

Enrichment Pipeline Service• Pass through all text related

pipelines


Multimedia Enrichment Pipeline - Video Processor

1/24/2017 23

Video Ingestion• Preserve metadata• Extract Audio from the videos• Generate Sample Frames

Transcribe/Normalize Audio • Transcribe Audio• Maintain start & end times• Capture word alternatives• Speech to Text

Visual Recognition• Use random frames / samples• Class & Face Detection• Visual Recognition


jsonInitial

metadata

A URI is passed to the Video Ingestion and we use ffmpeg to ingest the video:1. We instantiate a metadata

object and save any existing metadata in the file.

2. a) As the video is read, we pass audio to STT (if we don’t have a transcript)b) Every TIME_INTERVAL ffpmeg saves a Screen capture of the file.

3. We save Transcript, the filename and time taken to the Metadata.

4. Send image URI to VR5. Update JSON Metadata in

DB w/ VR Results.

URI

PNGimagePNG

imagePNGimagePNG

image

1

2a

2b

3

4

5

The main idea here is that we must process the file 1 time (at least) and it will take as long as it takes to play the file (for the most part) This process will extract all data possible during the initial processing PRIOR to handing it off to the TranscriptionAnalyzer. One possibility is if we already have a transcription and can skip 2a, we may be able to process video faster for just images.


Multimedia Enrichment Pipeline - Text Enrichment

1/24/2017 24

Entity Extraction• Invoke AlchemyLanguage

and/or Custom WKS AlchemyLanguage Model

Keywords• Highly relevant terms & phrases• AlchemyLanguage

Expand Keywords• Word2Vec as a API

• Trained on Wikipedia and other generally available corpus

Tone Analysis• Extract the top emotion,

social tone and writing tone

• Tone Analysis

Taxonomy & Relationships• Classify or categorize content• Different types of relations between

detected entities• AlchemyLanguage


Retrieve JSON Document for processing, Update when complete

Enrichment Pipeline• Lookup document and retrieve results

from Enrichers

Each stage will pass on its enriched JSON file to the next stage which will work on the data as it sees fit. At the end of each stage an even to the Enrichment Pipeline indicating the stage is finished will be generated.

A set of node.js Streams

event: ‘start’Id: docID

event: ‘’finished’Id: docID

Watson Knowledge Studio

25

WKS RULES

27

Thank [email protected]

January 2017

Backup Slides

January 2017

watson api use case demos for the nittany watson challenge

Education