© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Markku Lepisto, Principal Technology Evangelist, AWS

Olivier Klein, Emerging Technologies SA, AWS

Glen Fields, VP GTM and Strategy, ThingWorx

November 2016

IOT403: ROBOTSThe Fading Line Between Real and Virtual Worlds

What to Expect from the Session

• Connect Bluetooth devices with AWS IoT and Lambda

• Integrate an Amazon Alexa Skills and have it

communicate with AWS IoT

• Get a glimpse of the Amazon Lumberyard game

engine and use Cloud Canvas to integrate

a virtual world with AWS IoT

• Peak into the world of Augmented Reality

• Have fun with robots!

The Physical World

Intel Edison

AWS IoT Device SDK &CylonJS Robot SDK

for Node.JS

MQTT

AWS IoTPlatform

Bluetooth

Demo

Robot controller Edison – Thing in AWS IoT

Intel Edison - AWS IoT & Cylon SDKs, Bluetooth

AWS IoT Device SDK &CylonJS Robot SDK

for Node.JS

Bluetooth connections for Edison and Sphero

Intel Edison - AWS IoT Thing & Thing Shadow

Register Edison Thing & Thing

Shadow clients, with keys and

certificates

Intel Edison – MQTT Messages & Shadow Updates

Connect to AWS IoT & Publish

(robot) connected & (initial) language

MQTT message

Message received. If the Action is

‘roll’, command robot to <roll> in

<direction> at <speed>

Shadow update received.

Command robot to change its color to

the <newcolor>

Intel Edison – Driving the Robot

Read keypresses on Edison,

send driving commands

via Bluetooth to Sphero

and publish actions as

MQTT messages

Intel Edison – Detecting Robot Collisions

On Collision event from the robot,

publish <collision> MQTT message

Let’s Give our Robot a Voice

Speaker

Raspberry Pi 2Intel Edison

AWS IoT Device SDK &CylonJS Robot SDK

for Node.JS

AWS IoT Device SDK for Node.JS

MQTT MQTT

AWS IoTPlatform

Bluetooth

Intel Edison

Roll:

<direction, speed>

Raspberry Pi 2

Action: roll Action: roll

AWS IoTPlatform

Speaker

Gogogo !

Intel Edison Raspberry Pi 2

Action: collision Action: collision

AWS IoTPlatform

Event:

<collision>

Speaker

Ouch !

Demo

Raspberry Pi - AWS IoT Device SDK & Raspbian Sound Player

Register AWS IoTclient

Subscribe to all MQTT topics for sphero

Function for executing audio player on

Raspbian OS

Raspberry Pi – Playing Sounds for Action Messages

On receiving a message, play corresponding sound files in <current language>

Example JSON Messages

{ 'Action': ‘connect’, ‘Type’: ‘sphero’} // connected to Sphero robot

{ 'Action': ‘connect’, ‘Type’: ‘lumberyard’} // connected to Lumberyard cloud canvas

{ 'Action': ‘disconnect’ , ‘Type’: ‘sphero’} // disconnected from Sphero robot

{ 'Action': ‘disconnect’ , ‘Type’: ‘lumberyard’} // disconnected from Lumberyard canvas

{ 'Action': roll’, ‘Speed’: 30, ‘Direction’: 0} // impulse to roll at speed 30 straight ahead

{ 'Action': ‘speed’, ‘Speed’: 50} // change robot speed to 50

{ 'Action': ‘collision’} // collision event detected by robot

{ 'Action': ‘language’, ‘Language’: ‘korean’} // change robot language to Korean

Let’s Talk to our Robot

The Power of Speech: Alexa

• Alexa, the voice service that powers

Echo, provides capabilities, or skills,

that enable customers to interact with

devices using voice

• Alexa Skills Kit (ASK) allows everyone

to build and publish their own skills

• Skills can be powered by AWS

Lambda

AWS Lambda

Skill:

Sphero

Intel Edison

Color: <orange> Color: <orange>

AWS IoTPlatform

Amazon AlexaSDK

Alexa, tell Sphero to change color to orange

Color:

<orange>

I changed the color for you

Demo

Alexa Skill - Trigger for AWS Lambda function

AWS Lambda - Implement Alexa Skill for Color Change

Lamdba Response to Alexa – what should Alexa say to the user

Get the current shadow of Edison thing

AWS Lambda - Update Edison Shadow with New Color

The Virtual World

A free AAA game engine deeply integrated with

AWS and Twitch – with full source

Lumberyard - Cloud Canvas

• Tools integrated into Lumberyard to

directly access AWS resources

• Storage, Database, Compute,

Identity, Queuing, Notification

• Integrated into the FlowGraph of the

IDE for easy visual scripting

• Lumberyard handles the backend

communication with AWS

Let’s connect the Virtual and Physical World

• Idea is to send robot movements from

Lumberyard to AWS IoT

• Glue is the Cloud Canvas, which calls

and AWS Lambda function via the

Lumberyard FlowGraph

• Sphero robot works with “impulses”,

Lumberyard with vectors changing at

high frequency – sending every

movement to AWS IoT is not scalable!

Amazon Lumberyard

AWS Lambda

Cloud

Canvas

Solution: Calculate ∆ on Coordinate System

• Camera / Player position exposes X,

Y, Z coordinates in Lumberyard

• Send coordinates to AWS Lambda in

a certain frequency (e.g. 1 second)

and calculate the ∆ between each

function call

• ∆ can then be used to calculate

impulses and sent it to the MQTT

topic

Math to the help - Coordinates to Impulse?

• Impulse = Direction * Speed

• Trigonometry helps to calculate

direction (0-360º)

• Pythagorean Theorem helps to

calculate distance

• Speed is distance over time

(frequency of AWS Lambda

function call)

• Z left out, Sphero can’t jump

X1,Y1

0º

90º

180º

270º

X2,Y2

AWS Lambda – IoT MQTT Protocol Bridge

Import AWS SDK for Python

Publish Lambda event payload to MQTT topic

Amazon Lumberyard

AWS LambdaCalculates Impulse

based on delta of coordinates

Intel Edison

Action: roll<direction, speed>

Action: roll<direction, speed>

AWS IoT

Action: Roll<direction, speed>

Cloud Canvas <X, Y, Z>

Amazon

LumberyardAWS Lambda

TopicsThingIntel Edison

Alexa ASK

Sphero

Robot

Amazon

Echo

The Final Architecture

Raspberry

PiSpeaker

Go Go Go!

Thing

© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Glen Fields, VP GTM and Strategy

November, 30th, 2016

ThingWorxBridging the Digital and Physical Worlds

BRIDGING THE DIGITAL AND PHYSICAL WORLDS

DIGITAL & PHYSICAL WORLDS COMBINED

INTERNET OF THINGS

DIGITAL WORLD PHYSICAL WORLD

Augmented Reality: Superimposing computer graphics onto a live view of the world. Also known as “AR”

Internet of Things

ThingHuman

Data

Collect Data

Send DataThe ChallengeHow does the human

process and interact

with all of that data?

Monitor & Control Thing

IoT : Ability to digitally talk &

listen to physical things to

monitor and control

45

1000101011000011110100010011110111111011101101011000011101101110111101101111010000

0000001111110101011010001110011000101001001101010101111101000011000101011000011110

1000100111101111110111011010110000111011011101111011011110100000000001111110101011

0100011100110001010010011010101011111010000110001010110000111101000100111101111110

1110110101100001110110111011110110111101000000000011111101010110100011100110001010

0100110101010111110100001100010101100001111010001001111011111101110110101100001110

1101110111101101111010000000000111111010101101000111001100010100100110101010111110

1000011000101011000011110100010011110111111011101101011000011101101110111101101111

0100000000001111110101011010001110011000101001001101010101111101000011000101011000

0111101000100111101111110111011010110000111011011101111011011110100000000001111110

1010110100011100110001010010011010101011111010000110001010110000111101000100111101

1111101110110101100001110110111011110110111101000000000011111101010110100011100110

0010100100110101010111110100001100010101100001111010001001111011111101110110101100

0011101101110111101101111010000000000111111010101101000111001100010100100110101010

1111101000011000101011000011110100010011110111111011101101011000011101101110111101

1011110100000000001111110101011010001110011000101001001101010101111101000011000101

01100001111010001001111011111101110110001111000

10001010110

00011110100

01001111011

11110111011

01011000011

10110111011

11011011110

10000000000

11111101010

11010001110

01100010100

10011010101

01111101000

01100010101

10000111101

00010011110

11111101110

11010110000

11101101110

11110110111

10100000000

00111111010

10110100011

10011000101

00100110101

01011111010

00011000101

01100001111

IoT Data Growth (ZETTABYTES)

2014

9.82016

18.52018

34.32020

70.0

Big Data

Analytics

Visual

Processing

via Augmented Reality

JOV-0386-2DA-2N

Internet of

ThingsAugmented

Reality

Ability to digitally talk to

physical things to

monitor and control

Ability to see and

experience the digital

attributes of physical things

APPLICATION AREAS47

Market & Sell Create Operate ServiceTrain

Market & Sell Create48

Service Train Operate

144

215

303

FY14 FY15 FY16

App Installs (million)

Source: Aeon Mobile., August 2016. Includes CV AR apps found in the Apple App Store and Google Play;

Vuforia SDK is the de facto standard

Market Share

270K+ 300M+80%+

121k

181k

272k

FY14 FY15 FY16

Registered Developers

Meet Thingworx studio

Simple UI DesignEasy, codeless “drag and drop” user

interface for ease of authoring AR

experiences

Content IntegrationSeamlessly integrate existing 3D

content, SLM and other enterprise

systems data

IIoT ReadyQuickly integrate IIoT sensor data and

enterprise system data from

ThingWorx

AR AnimationsPresent step-by-step animations with

Creo Illustrate

Designed for the Industrial Enterprise

Unique IdentifierQuickly identifies the unique thing and its relevant experiences

Real World AlignmentAligns 3D information contextually and spatially

Ability to ScaleTracking vast numbers of products, dynamically updated in the Cloud

Unique Identifier

Real World Alignment

Ability to Scale

S/N 1005 S/N 1006 S/N 1007 S/N 1008

S/N 1001 S/N 1002 S/N 1003 S/N 1004

S/N 1009 S/N 1010 S/N 1011 S/N 1012

ThingMark

Create Publish Scan Experience

• Codeless UI

• Import 3D CAD

and external data

• Integrate IoT

• Extend with Java

Script and CSS

• Publish to AWS

Cloud

• Saves all

experiences

• Access control

• Unique marker

• Provides Real

World Alignment

• Scalable

• Connects ‘Thing’

to Experience

• Window to the

Experience

• One app = All

experiences

Architecture

Experience

Creator

experience

servicefoundation

Publish

ThingMark

external data

sources

&

enterprise systems

experiences

thing model & thing

Sensor

Data

PLM CRM Service

experience

service

foundation

ThingMark

experience

AWS IoT

Connector

<Domain ID> + <Instance ID>

Solves two fundamental IoT business problems

1. Collect and Connect

• AWS IoT collects data from the edge into the cloud securely, at scale, and at a low cost

• AWS Cloud Services provides compute, storage, and security of your data

2. Interact

• ThingWorx uses data to analyze, create, and experience the IoT in a meaningful way.

• Contextually see and experience the digital data in the physical world through the power of Augmented Reality

Why partner with AWS as IoT cloud provider

CONNECT

ANALYZE

EDGE

CREATE

EXPERIENCE

AWS IoT Amazon

EC2

Amazon

DynamoDBAmazon Kinesis

Streams

Thingworx - AWS connector

• AWS IoT – Ingestion Layer

• AWS IoT Rule that forwards

data to Kinesis

• Kinesis is buffer between

AWS IoT and ThingWorx

Connector

• ThingWorx Connector pulls

data from the stream and

ingest into ThingWorx

platform

AWS IoT

Amazon

Kinesis

Streams

IoT

action

ThingWorx

AWS IoT

Connector

ThingWorx

Core

AWS Iot Edge

Node.js

AWS IoT ThingWorx

ThingWorx AR

Experience

Service

Thing Shadow Format

Shadow REST API

Thing Model Format

Amazon Kinesis

Client Library

Build Experiences Fast with Engineering Data

Digital Eyewear Support

Thank you!

Markku Lepisto, Principal Technology Evangelist, AWS

Olivier Klein, Emerging Technologies SA, AWS

Glen Fields, VP GTM and Strategy, ThingWorx

tinyurl.com/thingworx-studio

Remember to complete

your evaluations!