hey cortana, set a reminder to …. hey cortana, send email to jonas brandell …. hey cortana, call...

Adina Trufinescu - Principal Program ManagerKhuram Shahid - Senior Developer

Cortana Extensibility and Speech Platform In-Depth

3-716

Hey Cortana, set a reminder to ….

Hey Cortana, send email to Jonas

Brandell ….

Hey Cortana, call Jonas Brandell

Hey Cortana, text Jonas Brandell …

Hey Cortana, what’s up with

Jonas Brandell on Facebook

Hey Cortana, create a station for Hozier on Pandora

Hey Cortana, send a message to Jonas Brandell on Viber

Hey Cortana, Tweet flying into San

Francisco tonight

• Differentiate your app with voice commands

• Delight your users with natural interactions

• Increase user engagement • Increase productivity• Increase discovery

Developer Opportunity

Voice Commands in the Foreground

Speech Recognition in your app

Voice Commands in the Background

Agenda

Voice Commands in the Foreground

• Windows Universal Apps can register voice commands for users to say to Cortana

• Cortana launches the app to complete the task

• Available for Windows Universal Apps built with XAML and JavaScript

Voice Commands for Universal Windows AppsForegroundVoice Commands

Demo:Voice Commands for Adventure Works

Foreground Voice Commands Architecture

2

Windows SpeechPlatform

Call

Text

Remind

Email

Voice Command

3 6

4

5

XApplication

Voice Command Definition

7

1

Microsoft SpeechRecognition Service

Adventure Works, show my trip to London

Foreground Voice Commands - How ToCreate the Voice Command Definition (VCD)

Register the VCD XML on App Startup

Handle Voice Command Activation

• Determine the task to enable and what should the user say

• Start a chat with {contact}• Send a message to {contact}

• Determine what Cortana should say as the app is launched

• Starting a chat with {contact}

• Provide an example of what the user can say for Cortana to show on “What can I do?”

• Start a chat with Jonas

Voice Command Intent

• Users need to say the app name as part of the voice command• Viber, start a chat with Jonas• Facebook, what’s up with Jonas• Adventure Works, show my trip to London

• By default uses the app’s display name

• Provide a alternate such as short form when the app name is long or hard to pronounce

Voice Command & App Name

• Explicitly specify the app name where feels most natural• Start a chat with Jonas on/with/using Viber• What’s up with Jonas on Facebook• Show my Adventure Works trip to London

• Increase the recognition accuracy by providing a phonetically-rich phrase• Don’t: Get Jonas on Viber• Do: Start a chat with Jonas on Viber

• Avoid using a Cortana reserved command, e.g. Call, Text, Start, Open, Find. If you have to, use the app name as a prefix• Viber, text Jonas I’m on my way

Voice Commands Guidance

Voice Command Registration

• Register the Voice Command Definition on app startup

• Use VoiceCommandDefinitionManager API in Windows.ApplicationModel.VoiceCommands namespace

• Update the VCD with application data

Handle the Voice Command Activation• Override App.OnActivated method

• Check for ActivationKind.VoiceCommand

• Use VoiceCommandActivatedEventArgs.Result to determine what the user said

• Navigate to page and continue the interaction

Demo:Code walkthrough for“Show my trip to London for AdventureWorks”

• Websites can be submitted to Windows Store

• Cortana can launch Web-Hosted Apps with Voice Commands

• Same Voice Command Definition schema and activation model as for JavaScript apps

Voice Commands for Web-Hosted Apps

Voice Commands for Web-Hosted Applications

Voice Activation for Web-Hosted Apps

2


Call

Text

Remind

Email

Voice Command

3 6

4

5

1

7

XApplication

Package.appxmanifest

default.html

<meta name="msapplication-cortanavcd" content="https://contoso.com/vcd.xml"/>

<script src="voice.js"></script>

Vcd.xml


Contoso.com/default.html

if (typeof Windows != 'undefined') { var activation = Windows.ApplicationModel.Activation; Windows.UI.WebUI.WebUIApplication.addEventListener("activated", function (args) { if (args.kind === activation.ActivationKind.voiceCommand) { var speechRecognitionResult = args.result; var properties = speechRecognitionResult.SemanticInterpretation.Properties; //... } });

<Application Id="App" StartPage="https://flightarcade.azurewebsites.net">

<uap:VisualElements ... </uap:VisualElements>

<uap:ApplicationContentUriRules> <uap:Rule Type="include" WindowsRuntimeAccess="all“ Match="https://flightarcade.azurewebsites.net"/> </uap:ApplicationContentUriRules>

</Application>

Package.appxmanifest

Voice.js

Voice Commands for Web-Hosted AppsCreate the Voice Command Definition and host on web service

Register the VCD on the default HTML page

Handle Voice Command Activation in .js

Demo:Voice Activation for Web-Hosted App

Speech Recognition in your app

• Check if the interaction was input through voice or text

• For voice, use Speech to continue the user interaction in the app

• Use Windows.Media.SpeechSynthesis APIs to talk to the user

• Use Windows.Media.SpeechRecognition for speech recognition

In-App Speech Recognition

• Use SpeechRecognizer.RecognizeWithUIAsync to present the user with System listening GUI that can be customized

• Use SpeechRecognizer.RecognizeAsync for on demand recognition and provide a listening UI experience

• System predefined grammars: Dictation, Web Search, Form Filling

• App grammars: a list of strings or SRGS file

On-Demand Speech Recognition

• SpeechRecognizer.ContinuousRecognitionSessionenables listening continuously

• Can be used with Dictation grammar for scenarios where users speak for long periods

• Can be used with app-provided grammars for always-listening in-app commands

Continuous Speech Recognition

Demo: Adventure Works walkthrough for Speech Recognition

Demo:Voice Commands in the Background

• Universal App can register voice commands to run in an AppService

• AppService takes action on the voice command

• May use Cortana’s canvas and voice to present feedback to the user

• Available for Store Apps built with XAML and JavaScript


Background Voice Activation

Background Voice Commands Architecture

2


36

4

5

Isolated Storage

Voice CommandApp Service

Render XAML

7

Web Service


Adventure Works, what trips do I have coming up? VCD

Your App

8

Go to Adventure Works

Here are your upcomingtrips

A

YosemiteAugust 15th

San FranciscoMay 29th

9

1

Background Voice Commands - How To

Create the Voice Command Definition (VCD)

Register the VCD XML on App Startup

Handle the Voice Command Activation in an App Service

<VoiceCommands xmlns="http://schemas.microsoft.com/voicecommands/1.2"> <CommandSet xml:lang="en-us" Name=“en-us-CommandSet"> <AppName> Adventure Works </AppName> <Example> Show my trip to San Francisco </Example> <Command Name=“showTripToDestination"> <Example>Show my trip to San Francisco</Example> <ListenFor >show [my] trip to {destination}</ListenFor> <Feedback>Looking for trip to {destination}</Feedback> <VoiceCommandService Target=“AdventureWorksService"/> </Command>

Voice Command Definition

Register the Voice Command Definition on app startup

<Extensions>

<Extension

Category="windows.appService"

EntryPoint="AdventureWorks.AdventureWorksService">

<AppService Name=“AdventureWorksService"/>

</Extension>

</Extensions>

Create an AppService•Inherit from IBackgroundTask and implement Run()

• Register the AppService in the package.appxmanifest

public async void Run(IBackgroundTaskInstance taskInstance)

See Session 765 For Details on AppServices

using Windows.ApplicationModel.AppService;

using Windows.ApplicationModel.Background;

using Windows.ApplicationModel.VoiceCommands;• VoiceCommandDefinitionManager• VoiceCommandDefinition• VoiceCommandServiceConnection• VoiceCommand• VoiceCommandUserMessage• VoiceCommandContentTile• VoiceCommandResponse

WinRT Namespaces and APIs

VoiceCommandServiceConnection• Request actions from

Cortana:• RequestConfirmationAsync(…)• RequestDisambiguationAsync(…)• RequestAppLaunchAsync(…)

• Send updates to Cortana:• ReportProgressAsync(…)• ReportFailureAsync(…)• ReportSuccessAsync(…)

• Handle Events:• VoiceCommandCompleted

Cortana Canvas Interactions from Background• Present UI on

Cortana’s canvas

• Speak a message using Cortana’s voice

Which trip would you like to cancel?

May 14th 2015Las Vegas

May 28th 2015San Diego


August 24th 2015Phoenix

Content Area

App GUI Text

App Icon

Link to your app

App Screen

Text-To-Speech (TTS)

Content Tile

Which one do you wanna

cancel?

A

VoiceCommandResponse







cancel?

A







cancel?

A

VoiceCommandResponse

VoiceCommandResponse{ VoiceCommandUserMessage Message; VoiceCommandUserMessage RepeatMessage; List<VoiceCommandContentTile> VoiceCommandContentTiles; string AppLaunchArgument;}

VoiceCommandContentTile

VoiceCommandContentTile{ string Title; string TextLine1; string TextLine2; string TextLine3; Object AppContext; string AppLaunchArgument;

Windows.Storage.StorageFile Image; VoiceCommandContentTileType ContentTileType;}

Which one do you want to add to your trip?

Title NameText Line OneText Line TwoText Line Three

A

VoiceCommandContentTile – Text

Listening…

Listening…

Icon, Title


Listening…

Icon, Title, Line1, Line2, Line3

Title Name OneLong text lorem ipsum dolor sit amet, Consec tetuer amias Sed do eiusmod

Title Name One

Title Name Two

Title Name Three

Title Name FourTitle Name TwoLong text lorem ipsum dolor sit amet, Consec tetuer amias Sed do eiusmod

Title Name ThreeLong text lorem ipsum dolor sit amet, Consec tetuer amias Sed do eiusmod

Title Name Four



AA

VoiceCommandContentTile– Icons68x92xIcon, Title, Line1, Line2, Line3


Listening…

Title Name One

Long text two lines lorem ipsum consec tetuer amias

Title Name Two


Title Name Three


Title Name Four


240x140 Icon, Title, Line1, Line2, Line3


Listening…

Title Name OneOne line text

Title Name TwoOne line text

Long text two lines lorem ipsum consec tetuer amias odoran

68x68 Icon, Title, Line1, Line2


Listening…

Title Name One

Long text lorem ipsum

Title Name Two


Title Name Three


Title Name Four


Title Name Five



A A A

Demo: Code walkthrough for “When’s my trip to Las Vegas with AdventureWorks”

Which one do you want to cancel?

May 14th 2015

Vegas Tech Conference

August 18th 2015

Party in Phoenix

Listening…

Disambiguation


A

Cancel this trip?

Listening…

May 14th 2015


Yes No

Confirmation

Yes

A

Cancelling this trip…

Cancel

May 14th 2015



Progress

A

Cancelled this trip.

May 14th 2015



Completion

A

Which ‘Vegas’ trip do you wanna cancel?

Vegas Tech Conference or Party

in Vegas.

Do you wanna cancel ‘Vegas Tech

Conference’?Cancelling this trip… I’ve cancelled it.

[success-earcon]

Cancel my Adventure Works

trip

Which one do you want to cancel?

May 14th


August 18th

Party in Phoenix

Listening…

A

Which trip do you wanna cancel?

Disambiguation


The first one

The one on August 18th

• The user may select an option via speech or UI input.

• Cortana automatically listens for utterances that uniquely identify a content tile.

• RequestDisambiguationAsync() returns only after a selection has been made.

Do you wanna cancel ‘Vegas Tech Conference’?

Confirmation

Yes

No

• Always confirm before taking destructive or irreversible actionsCancel this trip?

Listening…

May 14th 2015


Yes No

A

Progress

• Text To Speech is optional

• You may send multiple progress updates. Updates are coalesced where possible.

• Send progress during long-running operations. If there are no updates for 5 seconds, Cortana may cancel the flow.


Cancel

May 15th 2015




A

Cortana Supplied Progress Screens

Listening…

Working on that…

Cancel

I’ll get Adventure Works on that.

Adventure WorksLooking for trip to San Francisco

Cancel

Search “Cancel my Adventure Works trip to Vegas”

Looking for your trip to San Francisco…

A

AMay 14th 2015



• Shown if launching the App service, or the initial response takes > 500ms

• Feedback string from VCD, if any, is spoken out loud.

• Shown after RequestConfirmation or RequestDisambiguation if the App does not respond quickly (500ms)

• Content derived from the previous Confirmation/Disambiguation Response

• Ask the user to confirm before taking action on the voice command

• Ask the user to disambiguate if there are more than one action or target of the action

• Show progress as the voice command is handled

Interaction Summary:

Canvas Interactions from Background

Cortana Your AppServiceAdventure Works, cancel

my trip! I’ll get Adventure Works on that!

Inspect Voice Command

Las Vegas

yes

Invoke IBackgroundTask::Run()

RequestDisambiguationAsync

RequestConfirmationAsync

VoiceCommandConfirmationResult

VoiceCommandDisambiguationResult

Interaction Flow

Which one?Las VegasCancun//build/

check SelectedItem

Are you sure?

yes no

Process request

(Talk to web service,

save dataetc.)

Cortana Your AppAdventure Works, cancel

my trip! I’ll get Adventure Works on that!

Inspect Voice Command

Which one?Las VegasCancun//build/

Are you sure?

yes no

Las Vegas

yes

check SelectedItem

Invoke IBackgroundTask::Run()

RequestDisambiguationAsync

RequestConfirmationAsync

VoiceCommandConfirmationResult

VoiceCommandDisambiguationResult

Process request

(Talk to web service,

save dataetc.)

Canceling… ReportProgressAsync

foreground

Event: VoiceCommandCompletedTeardown

App Launch

Event: VoiceCommandCompleted

UI Interaction

Demo:Authoring & Debugging an App that supports background Voice Commands

• Be efficient. Less is more! • Be relevant. Provide information

that’s relevant to the given task, content, and context.

• Be clear. Avoid ambiguity. Use everyday language.

• Be trustworthy. Present accurate information. Be transparent. Respect privacy.

Conversational Writing Guidance


• For each supported language, localize the CommandSet in the Voice Command Definition

• When launched, inspect the value of Language property of the VoiceCommandServiceConnection

• Localize the dialog on the app provided screens to match Cortana’s language

Localization


Create a Voice Command Definition

Register the VCD when the app starts

Handle the Voice Activation in the app or AppService

Voice Commands for Universal Apps - Recap

• for tasks that require complex interactions

• for tasks that demand the user’s attention for a long time

Voice Command Handling Guidance

• for simple tasks that don’t require additional input beyond Confirmation and Disambiguation

• for tasks that complete within seconds

• to initiate longer tasks that execute remotely

• may use RequestAppLaunchAsync method to launch the app and complete remainder of the flow

Foreground Background

• Launch and deep link into your app from Cortana• Extend Cortana with your app’s capabilities through

voice commands in the background• Add speech recognition as an input modality to your

app

• References:• See AppServices Session 765• See Web-Hosted Apps Session 665• See Adventure Works Sample Code• See Design Guidelines for extending Cortana

Call to Action

https://github.com/Microsoft/Windows-universal-samples/tree/master/cortanavcds

https://msdn.microsoft.com/en-us/library/windows/apps/dn974233.aspx



hey cortana, set a reminder to …. hey cortana, send email to jonas brandell …. hey cortana, call...

Documents