selecting home appliances with smart glass based on

Osaka UniversityAdvanced Telecommunications Research

Institute InternationalOsaka UniversityAdvanced Telecommunications Research

Institute International

*Quan Kong(Osaka University)Takuya Maekawa(Osaka University)

Taiki Miyanishi(ATR)Takayuki Suyama(ATR)

Selecting Home Appliances with Smart Glass based on Contextual Information Ubicomp 2016



Introduction- Control home appliances

Free your hands

AndIn more direct way



Approaches for Home Appliances Control

Voice Gesture

Wearable CameraIR

Wearable Camera

Shi F., 06

“Alex, open the curtain on the kitchen’s north wall.”

IR Emitter

Ben Z., SUI ’14

Amazon echo



Our Approach

“ON”

+Living RoomTelevision

Home Network

・Activity・Indoor position



Feature of Our Approach – Context-aware appliance selectionGoogle Glass

Camera ・Orientation・Accelerator・Light sensor…

Context Information

Indoor positions

Activities

non-parametric unsupervised learning

Why context information ?Related to the home appliances Distinguish between different appliances



System Overview- Sensors Used in Our System

ScreenLight sensor

Orientation

Google Glass SmartphoneWi-Fi

Microphone

Accelerator

Camera

Appliance Image(First person view)

Camera

Activity Information

Microphone

Accelerator

Light sensor

Wi-Fi

Indoor position Information

Orientation

Head direction



Update the model by using the collected history daily data

Use the trained appliance selection model in the daily life

Select the prepared name of appliance in glass application

Stand in front of the appliance and take a 10 seconds video

System Overview- Initialization and Model Update

1

2

Images extracted from the video used as the training data for initializing 3

4

5

wrong estimation is corrected by the user and used as training data

Initi

aliz

atio

n



Proposed Method

Extract the attention time’s image feature and estimate the activity & position (IGMM)

Detect the user’s attention using orientation data 1

2Extracted above information as the input of appliance selection model (MKL) 3

Appliance Image Sensor Data

Position

Activity

+

Deep Convolutional

NeuralNetwork

Extract CNN fc6

CNN featurefc6:4096

dimensional

Feature Extraction

Train(ed)ActivityModel

Train(ed)Positional

ModelTrain Data(test data) Feature Data

0.981 Act10.765 Act20.543 Act3… …

MultipleKernel

Learning

Classification

Estimated Appliance

Context feature

Clustering

0.923 Pos10.865 Pos20.443 Pos3… …

Attention Detection



Proposed Method – Image feature extraction with DCNN

image feature

CaffeNetReference Model: ILSVR-2012



Proposed Method-Unsupervised activity recognition and indoor positioning

Wi-Fi

- light- Acc- microphone

PositionIGMM

ActivityIGMM

FeatureExtraction

Test data

𝐷 : the invers of the distance between the test data and each cluster

𝐷

𝐷Bedroom Kitchen

Cook Sleep- Wi-Fi- microphone- light- Acc

Feature extraction of IGMM input

Accelerator

Microphone Wi-Fi

Light sensor3-axis combination signal

Average MFCC components

Average of illumination

Signal strength values

Learning Activity and Position Model・Use non-parametric learning approach IGMM for activity and position clustering



Proposed Method – Appliance selection using MKL

Cameraimage

Wi-Fi

- light- Acc- microphone

Selected Appliance

PositionIGMM

ActivityIGMM

Deep convolutional

neuralnetwork

Multiple kernel

learning

Image features

context features

A linear combination of multiple base kernels for image and context feature

describe a different property of the datawith multiple kernels

Multiple Kernel Learning𝒌𝒊𝒎𝒈,∗: 𝑝𝑜𝑙𝑦𝑛𝑜𝑚𝑖𝑎𝑙 𝑘𝑒𝑟𝑛𝑒𝑙 𝑓𝑜𝑟 𝑖𝑚𝑎𝑔𝑒 𝒌𝒄𝒐𝒏𝒕𝒆𝒙𝒕,∗: 𝑟𝑎𝑑𝑖𝑎𝑙 𝑏𝑎𝑠𝑖𝑠 𝑓𝑢𝑛𝑐𝑡𝑖𝑜𝑛（𝑓𝑜𝑟 𝑐𝑜𝑛𝑡𝑒𝑥𝑡）

Decision Function : f 𝑥∗ 𝑎 𝑒 𝑘 ,∗ 𝑒 𝑘 ,∗ 𝑏



Device for data collection Google Glass, Nexus 5 (in pocket) Sampling rate : 30HzSemi-naturalistic collection

protocol Activities follow the instruction 3 users X 10 sessions activities

prepare meals

eat meals

wash dishes

watch TV

…

sleep

Random

Activities

floor plan and appliances

toilet faucet

bedroom air conditioner

bedroom lighting

lounge lighting

TV

lounge air conditioner

kitchen faucet

kitchen curtain lounge curtain

kitchen lighting

drawer

front door

fan

Evaluation – Data set



Evaluation Result - Leave-one-session out cross validation

Proposed: activity + position + camera

Proposed w/o pos:activity+ camera

Proposed w/o act: position + camera

Proposed w/ cam: only camera

SVM w/ cam: only camera

SVM all: activity + position + camera

Effect of context

F-measureRecallPrecision

10%

0.845

0.813

0.857

0.928

0.894

0.955

0.844

0.812

0.862

0.897

0.878

0.936

0.844

0.812

0.859

0.912

0.886

0.945

SVM all

SVM w/ cam

Proposed w/ cam

Proposed w/o act

Proposed w/o pos

Proposed



Evaluation Result – Confusion MatrixVisual confusion matrix of Proposed w/ cam Visual confusion matrix of Proposed

• air conditioner and lighting were relatively poor • can’t distinguish between kitchen lighting and

bedroom lighting • drawer performed not well

• air conditioner and lighting were increased about 14% on average of F-measure

• F-measure improved by about 10% on total average



0.4

0.5

0.6

0.7

0.8

0.9

1

0 2 4 6 8 10 12

Proposed

Proposed (reuse)

Proposed (imgaenet)

Evaluation Result – Transition of Average F-measures

sessions

F-m

easu

re

Reusing other users’ data


Position

Activity

+


Position

Activity

+


Position

Activity

+

User1User2

User3

-collect online images for each category

-find top-k similar images for each appliance

Utilizing online image database

32% 28%



Conclusion

We proposed a new method of appliance selection with a smart glass based on position and activity contextual information

The effectiveness of contextual information in an appliance selection task has been confirmed in a real experiment environment.

Context based method can also be used to enhance the performance of such other appliance selection approaches as speech, gaze direction, and beacon- based approaches

selecting home appliances with smart glass based on

Documents