developers.io 2017 iphoneによるalexa/lex/pollyを利用した...

iPhoneによるAlexa/Lex/Pollyを利⽤した⾳声対応クライアントの作成⽅法

⾃⼰紹介

クラスメソッド（株）アプリケーションエンジニア平内真⼀

・モバイルアプリサービス部（iOSエンジニア）・フリーソフト BlackJumboDog・Microsoft MVP (2013/1〜)

アジェンダ

1. Overview2. Alexa3. Lex4. Polly5. Summary

1.概要

Overview

1.概要

SkillSkillSkillSkill

Lambda

Skill

ゴール

1.Amazonの各種⾳声サービスの概要2.⾳声対応クライアントの実装⽅法3.要件に応じたサービスの選択

1.概要

Alexa

Alexa アーキテクチャー

今⽇の東京の天気は？

今⽇の東京の天気は、晴れ時々曇り最⾼気温は２７度・・・・

Alexa アーキテクチャー

AWSre:Invent 2016:HowCapitalOneBuiltaVoice-BasedBankingSkillforAmazonEcho(ALX201)

Alexa Alexa Voice Service

https://developer.amazon.com/public/solutions/alexa/alexa-voice-service/content/avs-api-overview

Alexa Alexa Voice Service

http://dev.classmethod.jp/smartphone/alexa-client-friendly-voice-assistant/

Alexa

• Endpoints/Protocol• Interface•Registration•Authorization

Alexa Endpoints/Protocol

Region SupportedCountries

URL

NorthAmerica US https://avs-alexa-na.amazon.com

Europe UK,Germany https://avs-alexa-eu.amazon.com


:method=GET:scheme=https:path=/{{APIversion}}/directivesauthorization=Bearer{{YOUR_ACCESS_TOKEN}}

ダウンチャンネルストリームタイマー・アラームなど接続後１０秒以内クライアントからの半閉状態で開いたまま接続の存続期間中はAVSから開く⻑い休⽌があることは珍しいことではない v20160207


:method=GET:scheme=https:path=/pingauthorization=Bearer{{YOUR_ACCESS_TOKEN}}

PingandTimeout5分に１回


:method=POST:scheme=https:path=/{{APIversion}}/eventsauthorization=Bearer{{YOUR_ACCESS_TOKEN}}content-type=multipart/form-data;boundary={{BOUNDARY_TERM_HERE}}


https://developer.amazon.com/public/solutions/alexa/alexa-voice-service/docs/avs-http2-requests

JSONイベント・ディレクティブ

Audio⾳声


audioencoded16bitLinearPCM(LPCM16)16kHzsamplerateSinglechannelLittleendianbyteorder

Alexa


Alexa Interface

Interface Ddescription

SpeechSynthesizer AlexaのスピーチインターフェースSpeechRecognizer AVSのコアインターフェース各ユーザ発話は、認識イベントを利⽤するSpeaker ミュートやミュート解除を含む、デバイスやアプリケーションの⾳量コントロールの

ためのインターフェイスSettings ロケールなど、製品のAlexa設定を管理するためのインタフェースPlaybackController

ボタンアフォーダンスを介して再⽣キューをナビゲートするためのインターフェイスAudioPlayer オーディオ再⽣を管理および制御するためのインターフェイスAlerts タイマーとアラームの設定、停⽌、および削除のためのインターフェースSystem Alexaにクライアント情報を提供するためのインタフェース

Alexa Interface

SpeechRecognizer Interface

Event/Directive

RecognizeEvent ユーザ⾳声の送信StopCapture Directive 録⾳停⽌指⽰ExpectSpeech Directive 録⾳開始指⽰（会話継続）ExpectSpeechTimedOut Event タイムアウト

Alexa Interface

SpeechSynthesizer Interface

Event/Directive

SpeakDirective Alexaの送話SpeechStarted Event 再⽣開始SpeechFinished Event 再⽣終了

Alexa Interface

https://developer.amazon.com/public/solutions/alexa/alexa-voice-service/reference/speechrecognizer

Alexa Interface

Event Code Directive1 録⾳開始2 録⾳終了3 SpeechRecognizer.Recognize

+ 録⾳⾳声200 SpeechSynthesizer.Speak

+ Alexa⾳声4 SpeechSynthesizer.SpeechStarted 204 会話が継続する場合

SpeechRecognizer.ExpectSpeech会話が継続しない場合DIRECTIVEなし

5 再⽣開始6 再⽣終了7 SpeechSynthesizer.SpeechFinished 204

SpeechRecognizer.ExpectSpeechが返っている場合は、１に戻る

Alexa


Alexa Registration

https://developer.amazon.com

Alexa Registration

Alexa Registration

ApplicationTypeID認証で使⽤されます

Alexa Registration

Alexa Registration

Key/BundleId認証で使⽤されます

Alexa Registration

Alexa Registration

新規

Alexa Registration

Alexa Registration

必要なのは以下の３つ＊ApplicationTypeID（変更できない）＊BundleID（追加可能）＊Key （追加可能）削除できない

Alexa


Alexa Authorization

https://developer.amazon.com/sdk-download

Alexa Authorization

Alexa AuthorizationSwiftから利⽤するためには、Bridging-Headerを作成#import <LoginWithAmazon/LoginWithAmazon.h>

Alexa Authorization

リダイレクトをスキームで受けれるように、URLSchemesで、BundleIdentifireの前にamzn- を付けたスキームを指定します。

amzn-jp.classmethod.us.ios.Friendly

Alexa Authorization

開発者ポータル

Alexa Authorization

class AppDelegate:UIResponder,UIApplicationDelegate {

func application(_application:UIApplication,openurl:URL,

sourceApplication:String?,annotation:Any)->Bool{

return AIMobileLib.handleOpen(url,sourceApplication:sourceApplication)

}

Alexa Authorization

@IBAction func tapLoginButton(_sender:Any){

letSCOPE_DATA= "{alexa:all:{productID:プロダクトID,productInstanceAttributes:{

{deviceSerialNumber:デバイスシリアル番号}}

}AIMobileLib.authorizeUser(forScopes:["alexa:all"],

delegate:self,options:[kAIOptionScopeData:SCOPE_DATA])

}

開発者ポータル

Alexa Authorization

Alexa Authorization

https://www.amazon.com

Alexa Authorization

1.概要

Lex

Lex アーキテクチャー

AWSConsoleLex()


AmazonPollyandAmazonLexWorkshop

Lex デモアプリ

http://dev.classmethod.jp/smartphone/ios-lex-tap/

Lex

• Endpoints/API•AWSmobileSDK•Authorization•SessionAttribute

Lex Endpoints/API

Service Region URL

Model buildingservice

USEast(N.Virginia)

https://models.lex.us-east-1.amazonaws.com

Runtimeservice

USEast(N.Virginia)

https://runtime.lex.us-east-1.amazonaws.com

Lex Endpoints/API

CreateBotDeleteBotDeleteBotAliasGetBotGetBotAliasGetBotAliasesGetBuiltinIntentGetBuiltinSlotTypesPutBotPutBotAliasPutIntentPutSlotTypeEtc.

AmazonLexModelBuildingService

Lex Endpoints/API

PostContentPostText

AmazonLexRuntimeService

Lex Endpoints/APIPostContent Sendsuserinput(textorspeech)toAmazonLex

PostText Sendsuserinput(text-only) toAmazonLex

POST/bot/botName/alias/botAlias/user/userId/contentHTTP/1.1x-amz-lex-session-attributes:sessionAttributesContent-Type:contentTypeAccept:acceptinputStream

POST/bot/botName/alias/botAlias/user/userId/textHTTP/1.1Content-type:application/json

{"inputText":"string","sessionAttributes":{"string":"string"

}}

Lex


Lex AWS mobile SDK for iOS


http://dev.classmethod.jp/smartphone/amazon-lex-ios-sdk/

1. pod install2. マイク利⽤許可（info.plist）3. CognitoによるIdentity発⾏とLexの初期化4. AWSLexVoiceBottn


source'https://github.com/CocoaPods/Specs.git’target'BotSampleApp'doplatform:ios,'9.0’use_frameworks!pod'AWSLex’pod'AWSCognito’

end


<key>NSMicrophoneUsageDescription</key><string>ForinteractionwithAmazonLex</string>


let poolId ="us-east-1:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx"let credentialsProvider =AWSCognitoCredentialsProvider(regionType:.USEast1,identityPoolId:poolId)let configuration =AWSServiceConfiguration(region:.USEast1,credentialsProvider:credentialsProvider)AWSServiceManager.default().defaultServiceConfiguration =configuration

let BotName =“LexSample"let BotAlias =“$LATEST"let chatConfig =AWSLexInteractionKitConfig.defaultInteractionKitConfig(withBotName:BotName,botAlias:BotAlias)

AWSLexInteractionKit.register(with:configuration!,interactionKitConfiguration:chatConfig,forKey:"AWSLexVoiceButton")AWSLexInteractionKit.register(with:configuration!,interactionKitConfiguration:chatConfig,forKey:"chatConfig")

cognito

Lex


http://dev.classmethod.jp/smartphone/amazon-lex-ios-sdk/


etc.


•AWSLexInteractionDelegate•AWSLexAudioPlayerDelegate•AWSLexMicrophoneDelegate


Fullfillment

switchMode

InteractionDelegate


started

finished

AudioPlayerDelegate


start

end

SoundLevelChaned

MicrophoneDelegate

Lex


Lex Authorization

Lex


Lex SessionAttribute


{"currentIntent":{"name":"intent-name","slots":{"slot-name":"value",},"confirmationStatus":"None, Confirmed, or Denied (intent confirmation, if configured)",},"bot":{"name":"bot-name","alias":"bot-alias","version":"bot-version"},"userId":"UserIDspecified inthe POSTrequest to AmazonLex.","inputTranscript":"Textused to process the request","invocationSource":"FulfillmentCodeHook or DialogCodeHook","outputDialogMode":"Textor Voice, based onContentType request header inruntime APIrequest","sessionAttributes":{"key1":"value1","key2":"value2"

}}

LambdaInput

sessionAttribute

http://docs.aws.amazon.com/ja_jp/lex/latest/dg/lambda-input-response-format.html


{"sessionAttributes":{"key1":"value1","key2":"value2"},"dialogAction":{"type":"Close","fulfillmentState":"FulfilledorFailed","message":{"contentType":"PlainText orSSML","content":"Messagetoconveytotheuser.Forexample,Thanks,yourpizzahasbeenordered."},"responseCard":{"version":integer-value,"contentType":"application/vnd.amazonaws.card.generic","genericAttachments":[{"title":"card-title","subTitle":"card-sub-title","imageUrl":"URLoftheimagetobeshown",

LambdaResponse

sessionAttribute

http://docs.aws.amazon.com/ja_jp/lex/latest/dg/lambda-input-response-format.html


POST/bot/botName/alias/botAlias/user/userId/text HTTP/1.1Content-type:application/json

{"inputText":"string","sessionAttributes":{"string":"string"

}}

ClientPOST

sessionAttribute

http://docs.aws.amazon.com/ja_jp/lex/latest/dg/API_runtime_PostText.html


HTTP/1.1200Content-type:application/json

{"dialogState":"string","intentName":"string","message":"string","responseCard":{"contentType":"string","version":"string"

},"sessionAttributes":{"string":"string"

},"slots":{"string":"string"

},"slotToElicit":"string"

}

ClientResponse

sessionAttribute

http://docs.aws.amazon.com/ja_jp/lex/latest/dg/API_runtime_PostText.html


"sessionAttributes":{"cards":[{"title":"Mocha","subtitle":"Pleaseenjoythefragrantmocha","body":"$4.15","imageUrl":"https://exsample.com/mocha.png","slotName":"CoffeeType","slotValue":"Mocha"

},

タップされた場合は、Textで送る

Polly アーキテクチャー

https://www.slideshare.net/AmazonWebServices/amazon-polly?qid=c9c8e2c4-1c27-41c1-bc5f-9b1e0c651c21&v=&b=&from_search=11

Polly

•AWSmobileSDK•Authorization

Polly AWS mobile SMD

source'https://github.com/CocoaPods/Specs.git’target'BotSampleApp'doplatform:ios,'9.0’use_frameworks!pod'AWSPolly’pod'AWSCognito’

end


letinput=AWSPollySynthesizeSpeechURLBuilderRequest()input.text=textView.textinput.outputFormat=AWSPollyOutputFormat.mp3input.voiceId=AWSPollyVoiceId.joannaletbuilder=AWSPollySynthesizeSpeechURLBuilder.default().getPreSignedURL(input)

builder.continueOnSuccessWith{(awsTask:AWSTask<NSURL>)->Any?inleturl=awsTask.result!self.audioPlayer.replaceCurrentItem(with:AVPlayerItem(url:urlasURL))self.audioPlayer.play()returnnil

}

http://dev.classmethod.jp/smartphone/amazon-polly-sdk/

AWSPollySynthesizeSpeechURLBuilderRequest

テキスト

声の種類

AVFoundation.AudioPlayer

Polly

•AWSmobileSDK•Authorization

Polly Authorization


⾔語24

声の種類

Summary

Summary

SkillSkillSkillSkill

Intent, Slots

Lambda

Skill

Intent, Slots,SessionAttributes

Intent, Slots,SessionAttributes

Summary

1.Amazonの各種⾳声サービスの概要2.⾳声クライアントの実装⽅法3.要件に応じた選択

これからもどうぞ宜しくお願い申し上げます。