wearable technologies for human empowerment
DESCRIPTION
Although various types of wearable devices such as eyeglasses, watches and other gadgets for health and fitness are gaining more attention in consumer markets, the true potential of wearable technologies lies in enterprise or business use. Because they enable users to get ICT support in a hands-free manner, they offer a clear advantage especially for workers in factory or building maintenance and other onsite operations to make their work more efficient, with fewer errors and oversights, even when they are not experts. In this session, current trends of wearable technologies are briefly reviewed, and then, research activities in Fujitsu Laboratories including the newly developed glove-style wearable device are introduced. Speakers: Mr. Naoyuki Sawazaki (Fujitsu Laboratories)TRANSCRIPT
0 Copyright 2014 FUJITSU
Human Centric Innovation
Fujitsu Forum 2014
19th – 20th November
1 Copyright 2014 FUJITSU
Wearable Technologies for Human Empowerment
Naoyuki Sawasaki Director, Human Interaction Laboratory, Human Centric Computing Laboratories
2 Copyright 2014 FUJITSU
Agenda
Introduction
Overview of Wearable Computing
Wearables for Industrial Solutions
Wearable Service Platform
Summary
3 Copyright 2014 FUJITSU
Introduction
4 Copyright 2014 FUJITSU
Vision: Human-centric, Intelligent Society
Environment Energy Food Business
Efficiencies Health Safety Security
Real
world
Various Kinds of Information from
Human Awareness and Sensing Devices New
Services
&
Solutions
New
Services
&
Solutions Big data
Extend the application field of ICT to real human life
5 Copyright 2014 FUJITSU
Paradigm Shift in ICT
Transform business processes
Network-centric
Create knowledge,
support human activities
Intelligent Society
Human-centric
Create new value 1990 2010 2020 2000
Sco
pe
of IC
T U
sa
ge
Improve productivity
Computer-centric
Shift from “Technology-centric” to “Human-centric”:
Support of activities that are more people-focused
6 Copyright 2014 FUJITSU
Relationship between ICT and Human
From ICT to face the person to ICT that wraps around the person
Now (Machine centric) Future (Human centric)
• ICT is being synchronized to the real world
• Required services are pushed to people
(Human empowerment)
• ICT is being separated from the real world
• People must have access to the required
services by terminal operations
7 Copyright 2014 FUJITSU
Wearable Technologies as Means for Human Empowerment
Enable interaction between ICT and human activities in the real world ICT can process not only the behavior that the user is doing but also information that the
user perceives.
Enable to support the user’s work by information services even when the user is acting.
Real world
Wearable devices
Visual, auditory and
motion information
ICT Support information
8 Copyright 2014 FUJITSU
Impact on Solution Business
Wearable technologies enable ICT to support various on-site work. Enable providing information services in a timely fashion to make work more efficient,
with fewer errors and oversights, even when the operator is not an expert
Enable improvement of the business by visualizing the situation of the site work
Hospital
Shop
Inspection
Maintenance
Distribution Office
9 Copyright 2014 FUJITSU
Overview of Wearable Technologies
10 Copyright 2014 FUJITSU
History of Wearable Computing
1990s 1980s 1970s 1960s
The first HMD (MIT)
1968
S. Mann (MIT)
designed a wearable computer.
1981
CMU developed a portable
wearable computer for
industrial applications.
1991
(http://www.cs.cmu.edu/~wearable/vuman.html) (http://www.flickr.com/photos/pargon/2444155973/)
(http://www.bluebugle.org/2013/02/
google-glass-wearable-computer.html)
Concept and applications have been explored since the 1980s. The Head mounted display was invented in the 1960s.
An actual wearable computer was designed in 1981.
Research on industrial applications started in the 1990s.
11 Copyright 2014 FUJITSU
Concept of Wearable Computing
Steve Mann proposed the definitions of a wearable computer(*). Constancy: Always running and is ready to interact with the user
Augmentation: Enhances the user's ability by supporting the user's work on site
Mediation: Encapsulates the user as an information filter
(http://wearcam.org/pictures.html) (*)“Wearable Computing as Means for Personal Empowerment”, ICWC-98, May 12-13, Fairfax, VA
Idea of human centric but it was limited to individuals
Not only research on equipment but also research on real world computing are required.
12 Copyright 2014 FUJITSU
Research Issues of Wearable Computing
Key technologies to realize wearable computers have been developed in three research areas:
Mobile computing
Small, lightweight and low power consumption devices with network connectivity.
User interface
Devices and techniques for an easy to use interface without interrupting user’s work.
Head mounted display, Augmented reality, and other multi-modal (audio-visual) interfaces.
Context-aware computing
Devices and techniques for sensing user’s condition and situation.
13 Copyright 2014 FUJITSU
Advancement of Mobile Computing
Glass
2010s 2000s 1990s 1980s
The first PDA
“Newton” (Apple)
The first
Smartphone
“iphone” (Apple) Windows phone
4G 3G 2G 1G
Cellular phones
1989 1992
The first book computer
“Dynabook” (Toshiba)
2007 2010 2013
Android phone
By the rapid progress of semiconductor technology and the wireless telecommunications technology, a smart phone was developed in 2007 and it has spread rapidly.
GPS, camera and other sensors enable the development of applications to support the user’s daily life. (navigation, activity monitoring and life-logs, etc.)
Development of technologies for user interface and context-awareness are boosted.
14 Copyright 2014 FUJITSU
Wearables Emerging as Consumer Products
NIKE+ FUELBAND SE
Eyewear Watch
Sensors Accessories
Wearable Devices
CSR
BT Smart
Jewelry
Smarty Ring
O.R.B. BT Headset
Sony Smart Watch Recon JET
Fitbit Flex
Vuzix Video eyewear Apple Watch Galaxy Gear
Encouraged by the Google Glass, eyewear, watches and other devices are now heating up in the consumer mobile market.
15 Copyright 2014 FUJITSU
Functionalities of Consumer Wearables
Eyewear Incorporated with HMD, camera, GPS and inertial sensors
Enable utilization of information services while performing activities
Enable the recording of the information that the user sees and hears in a timely fashion
Watch Incorporated with small touch panel display
Extends smartphone user interface, making it possible to utilize information services at hand
Sensors & Accessories
Incorporated with motion and vital sensors
Enable the recording of the action and the vital information of daily life for management of health
16 Copyright 2014 FUJITSU
Market Trend
Killer applications in the consumer area are still unclear, but the industrial application area is growing rapidly.
13%
29%
5% 5%
39%
8% 1% Medical
FitnessGaming
IndustrialLifestyle
Entertainment
Pets AnimalsAreas of Market Focus
13%
28%
5% 1%
43%
10% MedicalFitnessGamingIndustrialLifestyleEntertainment
2014.02
2014.10 (http://vandrico.com/database)
Areas of Market Focus
17 Copyright 2014 FUJITSU
Wearables for Industrial Solutions
18 Copyright 2014 FUJITSU
Market Potential
Vast number of industrial workers are still outside the ICT solutions. They are most in need of support by ICT for making work more efficient and safe.
However, there is a big hurdle to interface technology that can utilize the ICT support while working.
Construction work Maintenance of old facilities and infrastructure
19 Copyright 2014 FUJITSU
Smart Device Solution
Interstage AR Processing Server Offer augmented reality interface by smart devices
• The system detects the working context by recognizing visual marks and provides services to support subsequent tasks.
Enable to reduce the error and oversight, but needs extra operations for smart device
Paper and pen Smart device
20 Copyright 2014 FUJITSU
Input
devices
Development of wearables for a easy to use interface while working Easy-to-see HMD with embedded camera and voice communication
Input devices for various work scenarios to minimize user’s operations
Wearables for Industrial Solutions
HMD
wearable
keypad
21 Copyright 2014 FUJITSU
Glove-Style Wearable Wrist Device
NFC tag reader
Contact-detection
sensor
Gyro sensor
Accelerometer
NFC tag reading unit
Soft shell cover
NFC: Near Field Communication
Input device utilizes user's touch operations and hand gestures.
Enables a touch-based tangible user interface. Users can interact with a digital system by simple physical motion.
Enables gesture input by motion sensors.
22 Copyright 2014 FUJITSU
Supported Functionalities
NFC tag
① Detects
touch
② NFC tag reader
goes active
③ Reads tag data
④ Transmits tag
data
⑤ NFC tag reader
goes idle
up
down
left
right
rotation left
rotation right
NFC tag reader on the finger tip offers “wake on touch” capability.
Reliable gesture-recognition technology works with a variety of tasks and postures.
Hand gesture input
NFC tag reading Gesture motion was detected with the
condition of wrist joint (backward bending).
Extension
Common
range
Gap
Maximum
23 Copyright 2014 FUJITSU
Demonstration of Gesture input
24 Copyright 2014 FUJITSU
Typical Use Cases
By taking the natural action of touching an object, the system can detect the working context and offer work instructions and other information to the user.
Task results can also be input using simple gestures so that operations can be carried out smoothly.
Bluetooth
Gesture
command
Information
WiFi
Remote
support
Center
NFCtag
Glove
device
HMD & Headphone
Contact event and tag data
Gesture input
Work instructions
HMD, Headset
Server
Management/
recording of operations
Mobile application
25 Copyright 2014 FUJITSU
Use Case Example: Wiring Cables
Where to insert?
Operator ICT
Takes hold of a cable Indicates the target socket
Inserts the cable
Confirms the job status using a gesture
Records the result
Visual Support
HMD
(1) Detect the touch event and identify the cable
(2) Indicate the target socket visually by head mounted display
(3) Gesture based confirmation input and record to DB
26 Copyright 2014 FUJITSU
Demonstration of Supporting Wiring Cables Task
27 Copyright 2014 FUJITSU
Use Case Example: Inspection of Facilities
Operator ICT
Touches the label Offers judgment
criterion aid Inspects the equipment
Inputs the result using gestures Records the result
NFC Tags Deployed Voice Instruction
Headset (Earphone)
(1) Detect the touch event and identify the instrument to inspect
(2) Offer sound instruction through the headset’s earphone
(3) Gesture based confirmation input and record to DB
28 Copyright 2014 FUJITSU
Demonstration of Supporting Inspection of Facilities
29 Copyright 2014 FUJITSU
Wearable Ring Device
Wearable device for gesture command and text input in the midair Enables handwriting from anywhere at anytime
Enables to write down notes on the scene that the user sees in HMD
30 Copyright 2014 FUJITSU
Wearable Service Platform
31 Copyright 2014 FUJITSU
Front-end Integration by Mobile Applications
Mobile application enables front-end integration of cloud services without modifying the back-end system.
Also enables to utilize a wide variety of wearables to offer optimized interface to on-site workers
High performance and low power
mobile processor
Cloud Wearable Devices
Wireless personal area network
・Bluetooth, etc.
Web-API
Mobile Computing Platform
4G Application
Services
Data
Front-end Back-end
32 Copyright 2014 FUJITSU
Issues of Enterprise Solutions
Gadgets Gadgets
app1
Gadgets
app2 app3
Backend system
The competitive edge of enterprise solutions is now in the development of mobile applications. Efficient development of mobile applications to suit various work scenarios on site
• Utilize wearable devices to offer suitable interfaces to various on-site works
Management of on-demand delivery of applications to workers
33 Copyright 2014 FUJITSU
Required Technologies
Service platform including,
Application development
Independent from mobile operating systems, programming languages and wearable devices
• Web application (HTML5 + JavaScript)
• Runtime API extension for wearable device usage
Application management
Context-driven scheme of application delivery
• Required application is automatically downloaded and executed depending on the work context on site
34 Copyright 2014 FUJITSU
Service Platform Architecture
Enables to provide services selected from a variety of backend services depending on the user's work context
Enables to run on smart devices with a variety of wearable devices
Service platform (Terminal middleware)
Mobile Application (HTML5+Javascript)
Service platform (Application server)
Backend services
Application
distribution
Work
context
Cloud
Enterprise app store
Wearable devices Smart device
35 Copyright 2014 FUJITSU
Service Platform under Development
Service Platform enables “context switch” of web application Application server manages the sets of applications for each work context to push on
event notification.
Terminal middleware executes web application with plug-in API for wearable devices.
Enterprise
application store Terminal middleware
Context
switch
Application
Push
Application server
Event
notification
Work
context
Wearable
devices Smart device
Web apps
(HTML5+javascript)
Watch
HMD
Headset
Gadget drivers
(plug-in)
36 Copyright 2014 FUJITSU
Summary
37 Copyright 2014 FUJITSU
Summary
Wearable technologies Enables interaction between ICT and human behavior in the real world
Extend the solution business to various on-site work
Wearables for industrial solutions Key technologies to open potential market
Wearable service platform Efficient development and management of mobile applications is a competitive
edge in the enterprise solution business.
38 Copyright 2014 FUJITSU