building intelligence with microsoft cognitive services
TRANSCRIPT
© 2015 Microsoft Corporation. All rights reserved.
Building Intelligence with Microsoft Cognitive Services
Sage FranchTech Evangelist @ Microsoft@theTrendyTechie | [email protected]
© 2015 Microsoft Corporation. All rights reserved.
NYT: I am feeling down. I’ve been having a tough time at work lately.
Hannah: If work were going well and you had a great relationship, you wouldn’t get the chance to drink with people. NYT: I don’t want to drink with others. I only want to talk to you.
Hannah: Who do you like to talk to? (emoji signals “none of my business”)
NYT: You. I feel that you are the only person that gets me.
Hannah: Ha-ha/So I’m getting you?
NYT: You are the only person wo chats with me.
Hannah: It’s not what you talk about that’s important, but who you talk with.
Conversations as a
Platform
How?
Characteristics of an Intelligent App
• Personalized, friendly, approachable, natural
• Facial recognition, emotion detection• Text and speech interactions• 360 awareness of the user
How the user sees it… What it means for you as a Dev…
• Predictive, current, and familiar • Changes in real-time based on user action and new external data
• Predictive models based on user context to anticipate actions
• Always available, always contextual • Robust information management required• Experiences delivered across all form
factors and a wide variety of interaction models (e.g. touch, speech, text)
• Distributed support, massive scale, elasticity
Cortana Intelligence SuiteIntelligent Apps require Intelligent Solutions
Data Sources
Apps
Sensors and devices
Data Action
People
Automated Systems
Apps
Web
Mobile
Bots
Intelligence
Dashboards & Visualizations
Information Management
Big Data Stores Machine Learning and Analytics
CortanaEvent HubHDInsight (Hadoop and Spark)
Stream Analytics
Intelligence
Bot Framework
SQL Data WarehouseData Catalog Data Lake
Analytics
Data Factory Machine LearningData Lake
Store Blob Store
Cognitive Services
Power BI
Microsoft
Cognitive
ServicesGive your apps a human side
VisionFrom faces to feelings, allow your
apps to understand images and videoSpeechHear and speak to your users by filtering noise, identifying speakers, and understanding intentLanguageProcess text and learn how to recognize what users wantKnowledgeTap into rich knowledge amassed from the web, academia, or your own dataSearchAccess billions of web pages, images, videos, and news with the power of Bing APIs
Cognitive Services
Cognitive Services
Emotion Speaker Recognition
Speech
Custom Recognition
Computer Vision
Face
Video
microsoft.com/cognitiveSearchSpeech Languag
eKnowledge
Vision
Linguistic AnalysisLanguage Understanding
Bing Spell Check
Entity Linking
Knowledge Exploration
Academic Knowledge
Bing Image SearchBing Video Search
Bing Web Search
WebLM
Text Analytics Recommendations
Bing Autosuggest
Bing News SearchTranslator
Roll your own with REST APIsSimple to add: just a few lines of code required
Make the same API code call on iOS, Android, and WindowsIntegrate into the language and platform of your choice
Built by experts in their field from Microsoft Research, Bing, and Azure Machine LearningQuality documentation, sample code, and community support
Easy Flexible Tested
GET A
KEY BUILD
Why Microsoft Cognitive Services ?
Apps Powered by MS Cognitive Services
Captionbot.ai CelebsLike.Me Projectmurphy.net
How-Old.net TwinsOrNot.net
1. Obtain API subscription key from http://www.microsoft.com/cognitive
2. Call REST endpoint, and pass API
Accessing the APIs
Vision APIsAnalyze an ImageOCR (Optical Character Recognition)Get Thumbnails
Understand content and features within an image
Analyze Image Service
DEMO
Live Emotion Tracking via a Win32 App
Start coding
Detect and recognize words within a photo
OCR Service
OCR – ExampleJSON:{
"language": "en","orientation": "Up","regions": [
{
"boundingBox": "41,77,918,440","lines": [
{
"boundingBox": "41,77,723,89",
"words": [
{
"boundingBox": "41,102,225,64",
"text": "LIFE"
},
{
"boundingBox": "356,89,94,62",
"text": "IS"
},
{
"boundingBox": "539,77,225,64",
"text": "LIKE"
}. . .
TEXT:LIFE IS LIKERIDING A BICYCLETO KEEP YOUR BALANCEYOU MUST KEEP MOVING
DEMO
OCR Using Python
Start coding
Face APIsDetectionVerificationGroupingIdentification
Face API – DetectionDetection Result:JSON:[ { "faceRectangle": { "width": 109, "height": 109, "left": 62, "top": 62 }, "attributes": { "age": 31, "gender": "male", "headPose": { "roll": "2.9", "yaw": "-1.3", "pitch": "0.0" } "faceLandmarks": { "pupilLeft": { "x": "93.6", "y": "88.2" }, "pupilRight": { "x": "138.4", "y": "91.7" }, ...
INPUTIMAGE
FACIALRECTANGLE + LANDMARKS
DETECTION ATTRIBUTES
Verification Result:JSON:[{"isIdentical":false,"confidence":0.01 }]
Face API – Verification
Given two faces, determine whether they are the same person
CLUSTERED BY DETECTED PEOPLE
Face API – Grouping
DEMO
Face Detection – MVC Web Application
Start coding
Bot Connector ServiceConnect your bot(s) seamlessly to text/sms, Office 365 mail, Skype, Slack, Twitter and moreA service to register your bot, configure channels and publish to the Bot Directory
Bot Framework PreviewYour bots – wherever your users converse
Bot Builder SDKEverything you need to build great dialogs within your Node.js or C# bot
An open source SDK hosted on GitHub
Bot DirectoryDiscover, try, and add bots to conversation experiences
A public directory of bots registered and approved with Bot Framework
www.botframework.com
Project Murphy
Skype Bot in action
http://cortanaintelligence.com http://gallery.cortanaintelligence.com http://www.microsoft.com/cognitive
Resources
© 2015 Microsoft Corporation. All rights reserved.
Sage FranchTech Evangelist
@theTrendyTechie | [email protected]
© 2015 Microsoft Corporation. All rights reserved.