MULTI MODAL INTERFACE WITH AUGMENTED

REALITY FOR INDUSTRIAL APPLICATIONS

• Presenter:

Dinh Quang Huy, IGS

• Supervisor: Assoc. Prof. Seet Gim Lee,Gerald, MAE

• Co-supervisors: Prof. Nadia Thalmann, IMI-IGS

• Mentor: Assoc. Prof. Lin Weisi, SCE

4/21/2015 1

OUTLINE

• MOTIVATION

• LITERATURE REVIEW

• PROPOSED SOLUTION

• PRELIMINARY RESULTS

• FUTURE PLAN

4/21/2015 2

MOTIVATION

Table 1. Annual growth rates of U.S.

companies for the period 2004-2011[1]

4/21/2015 3

• Robots are now being used in many fields such as Manufacturing, Medical

Treatment, Healthcare, Service Applications, Space or Defencse.

Figure 1. Worldwide annual supply of

industrial robots 2003 – 2016 [2]

PROBLEM STATEMENT

Robots are not popular in small medium enterprises (SMEs)

There are two main reasons

4/21/2015 4

High -

Investment

User-Interface

Problems

Figure 2. Example of small lot size processes: welding(left), cutting(right)

PROBLEM STATEMENT

High - investment

o The high – investment is rapidly vanishing (25% from 1990 to 2006 [5])

Industrial user – interface limitations

4/21/2015 5

o User-unfriendly and unnatural

Figure 4. An operator wearing

protective equipments Figure 3. Smart robot Teach Pendant

PROBLEM STATEMENT

4/21/2015 6

o Programming methods are time-consuming and require a lot user experience

which is difficult to apply to high mix, low volume production

OperatorGenerated Program

Specify points and orientations

Add process-related events

Robot Path

Figure 5. Traditional approach to program a robot o Cannot generate and return any type of visual feedback from the robot to the user

Figure 6. Human – robot cooperation

Task simulation

Task monitoring

Trust and accountability are important

considerations for cooperation[17]

Shift from robot – centric approach to

human - centric approach

MOTIVATION

4/21/2015 7

Adaptable and Reconfigurable interface

Perception for Unstructured Environments

Human must be able to recognize and interpret robot’s understanding

Image taken from A roadmap for US robotics: From Internet to Robotics,2013 edition

PROBLEM SUMMARY

• What prevent robots from becoming popular in

industrial contexts?

High - investment

User – interface limitations

User-unfriendly and unnatural

Programming methods are time-consuming and require a lot user

experience

Cannot generate and return any type of visual feedback from the robot

to the user

4/21/2015 8

LITERATURE REVIEW

o Human – robot interaction using Computer – Aided Design tool [6][7]

• The robot is controlled and programmed using a computer with a 3D model simulation

software

4/21/2015 9

Figure 7. CAD tool

o Intuitive with simulation

capability

o Require user - experience

o CAD calibration error

o Costly

Pros

Cons

LITERATURE REVIEW

Human – robot interface using kinesthetic teaching [8][9]

• Human user pilots the robot through the intended trajectory

4/21/2015 10

Figure 8. Operator piloting the manipulator t a

coordinate[8]

o Simple and more natural

o Time – consuming

o No visual feedback

Pros

Cons

RESEARCH GAP AND CONTRIBUTIONS

Research Gap

• The lack of a formal framework for Human-Robot interaction has the following features:

Enhance productivity and adapt to different industrial working scenarios.

Programming needs to be more intuitive, secure with a higher level of abstraction.

The human needs to “say less and the robot do more”.

Contributions

• Develop a completed framework for human-machine interaction for industry which includes

• A multimodal single-handed device that is user – friendly, natural and flexible for

industrial contexts.

• An AR system is capable of increasing the understanding between human and robot using

visual feedback.

• Refine and evaluate the proposed framework to identify essential interface’s factors that are

critical for industrial needs.

4/21/2015 11

Crane Tasks

Ship Hull Tasks

Proposed HRI in Mixed Reality with Multi

Modal Interface

12

Human User Robot

Mapping Hand-

Motion Gestures &

Actions to Robot

Actions

Computer

Generated Objects

Superimposed

Over Real World

Industrial Environment

Computer Graphics

Real World Image

Finger Action & Hand Gesture

Laser Mark

Visual Feedback

with Task

Simulation

4/21/2015

Human & Robot Interactions In Mixed Reality

13Human User Industrial Environment

Task SpaceComputer Graphics

Real World Image

Finger Action & Hand Gesture

Laser Mark

Visual Feedback with Task Simulation Computer Generated Objects,

Superimposed Over Real World

Mapping Hand-Motion Gestures & Actions to Robot Actions

Interaction

Human request Robot’s attention,

Robot confirms availability,

Human requests Robot to perform a task

Robot elaborates on Task requirements

and requests Human approval

Human amends Task details or Confirms

Robot executes Task

Human Monitors Robot Action

• A Robot Partner must recognize the Human’s intention and to add implicit constraints defined by the human.

• The human defines an incomplete Task, and the Robot elaborates on the Task.

• The High Level Robotic Assistance will enhance productivity at the risk of an appropriate proposal.

• Human confirmation/selection of Task options, is required of the Human.

4/21/2015

Proposed HRI Block Diagram

14

Computer

Graphics

Real World

Image

Finger Action &

Hand Gesture

Laser Mark

Human User

Task Space

IndustrialEnvironment

Graphics & Text

Generator

Finger Action &

Hand Gesture

Interpreter

Robot Motion

Command Generator

Robot/Environment

Status Parser

Knowledge

Decision Making

Task Constraints

Task SupportUser Interface

Viewer Head

Motion

Compensation

Mode?Direct

Assisted

4/21/2015

Proposed Secure Multimodal User Interface

15

Vision

Hand Action

Gesture & Command Recognition

10-DOF IMU Data

Analog Paddle & Buttons Status

Laser Pointer Track Recognition

Task Recognition

Task Simulation

Task ExecutionInteractions & ProcessesUser Command & Feedback

Task Space

Industrial

Environment

Transparent Display Glasses with Front

Facing Camera

Naked eye view

Augmented Image

Multi-modal paddle hand motion gesture and laser pointing

Paddle Action

Hand Motion

Laser Pointing

Hand Motion

4/21/2015

Multimodal Hand-controller

• What is multimodal interface?

Multimodal interfaces allow users to interact with computers using

multiple different modes or channels of communication (pointer, haptic

buttons, gesture recognition)

• Why multimodal interaction?

Communication between humans is a multimodal process [15].

Multi modality is believed to produce more reliable semantic meanings

out of error-prone input models and remove vague information [16].

• Why handcontroller?

Hand gesture communication channel is natural for human.

The operator can perform another task while wearing the device.

4/21/2015 16

Hand-controller input models Laser pointer

Laser pointer is useful when it comes to defining spatial

information such as robot task trajectory or object selection.

Haptic buttons

Provide tangible feeling while wearing gloves

Simple , high accuracy

4/21/2015 17

Figure 15. A clickable word

Hand-controller input models

Motion gesture mapping and recognition using inertial

measurement unit (IMU).

Vision-based gesture recognition often requires constant camera human

spatial information and consistent lightning [23].

If robots have too little autonomy, human operators will waste time

attending to robot instead of attending to their work tasks. If robot are

highly autonomous, situation awareness of activity is diminished[17].

The controlling mode could be a combination between these input

models

For example, haptic buttons can be useful to move the robot forward or

backward while gesture mapping is more convenient to rotate the robot.

4/21/2015 18

Augmented Reality System

Head Mounted Display (HMD)

Can perform functions similar to the teach pendant and be able to align

digital information to real information.

Task creation and simulation

Feedback Visualization

Input model mapping

4/21/2015 19

Figure 14. Task simulation example[18] Figure 15. Task creation example[19]

PREMINARY RESULTS

Hand-controller

4/21/2015 20

Figure 21. Hand-controller schematic

PREMINARY RESULTS

Hand-controller

Can control the simulated

robot using

• Haptic buttons

• Gesture

4/21/2015 21

Figure 22. Hand-controller prototype

FUTURE PLAN

Identify and design a AR module with importants components, and factors

that are critical for a sucessful human – machine interface for industrial

purposes

Interface Evaluation

• Use ten design heuristics for evaluation by Jacob Nielsen[22]

• Compare the performance of experienced users with novice users

• Calculate the average time needed to complete a task using the

interface for a set of users

4/21/2015 22

4/21/2015 23

Reference[1] A roadmap for US robotics: From Internet to Robotics,2013 edition

[2] World Robotics 2013 – Industrial Robot, IFR 18, September, 2013

[3] Image taken from http://www.autoalliance.org/images/dmImage/SourceImage/AdvancedTech-Cobots.png

[4] World Robotics 2004. UNCE, IFR. United Nations, Geneva, 2004

[5] R. D. Schraft, Christian Meyer, ‘The need for an intuitive method for small and medium enterprises’, ISR – Robotics, 2006

[6] Pedro Neto, J.Norberto Pires, A. Paulo Moreira, ‘CAD-Based off-line robot programming’, 2010 IEEE Conference on

Robotics Automation and Mechatronics (RAM)

[7] Pedro Neto, ‘Off-line Programming and Simulation from CAD Drawings: Robot-Assisted Sheet Metal Bending’, Industrial

Electronics Society, IECON 2013 - 39th Annual Conference of the IEEE

[8] c. Sehoul, J.S. Damgaard, S. B0gh, O. Madsen, ‘Human-Robot Interface for Instructing Industrial Tasls using Kinesthetic

Teaching’, 44th International Symposium on Robotics (ISR), 2013

[9] Petar Kormushev, Dragomir N. Nenchev, Sylvain Calinon and Darwin G. Caldwell, ‘Upper-body Kinesthetic Teaching of a

Free-standing Humanoid Robot’, 2011 IEEE International Conference on Robotics and Automation (ICRA)

[10] Azin Aryania, Balazs Daniel, Trygve Thomessen and Gabor Sziebig, ‘New Trends in Industrial Robot Controller User

Interface’, 3rd IEEE International Conference on Cognitive Infocommunications, 2012

[11] Balazs Daniel, Peter Korondi, Trygve Thomessen, ‘New Approach for Industrial Robot Controller User Interface’,

Industrial Electronics Society, IECON 2013 - 39th Annual Conference of the IEEE

[12] Image taken from http://www.windowscentral.com/nokia-maps-get-augmented-reality-functionality

[13] Image taken from http://www.gizmag.com/ikea-augmented-reality-catalog-app/28703/

[14] Gunther Reinhart, Wolfgang Vogl, Ingo Kresse, ‘A Projection-based User Interface for Industrial Robots’, IEEE Symposium

on Virtual Environments, Human-Computer Interfaces and Measurement Systems, 2007. VECIMS 2007.

[15] T. Brick and M. Scheutz. Incremental natural language processing for HRI. Proceeding of the ACM/IEEE international

conference on Human-robot interaction – HRI ’07, page 263, 2007.

4/21/2015 24

Reference[16] S. L. Oviatt. Ten myths of multimodal interaction. In CACM, volume42(11):pages74–81, 1999.

[17] Clint Heyer, ‘Human-robot Interaction and Future Industrial Robotics Applications’, 2010 IEEE/RSJ International

Conference on Intelligent and Systems, Taipei, Taiwan

[18] Jens Lambrecht, Martin Kleinsorge, Martin Rosenstrauch and Jörg Krüger, ‘Spatial Programming for Industrial Robots

Through Task Demonstration’’, International Journal of Advanced Robotic Systems, 2013, Vol 10

[19] Jens Lambrecht and Jörg Krüger, ‘Spatial Programming for Industrial Robots based on Gestures and Augmented Reality’

2012 IEEE/RSJ International Conference on Intelligent Robots and Systems,October 7-12, 2012. Vilamoura, Algarve,

Portugal

[20] G. Falcao, N. Hurtos, J. Massich, D. Fofi, ‘Projector-Camera Calibration Toolbox Report’,

http://code.google.com/p/procamcalib, 2009

[21] Samuel Audet and Masatoshi Okutomi, “A User-Friendly Method to Geometrically Calibrate Projector-Camera Systems”,

IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, 2009. CVPR Workshops 2009,

Page(s):47 – 54

[22] Nielsen, J., and Molich, R. (1990). Heuristic evaluation of user interfaces, Proc. ACM CHI'90 Conf. (Seattle, WA, 1-5

April), 249-256.

[23] Michael T. Wolf, Christopher Assad, Matthew T. Vernacchia, Joshua Fromm, and Henna L. Jethani, “Gesture-Based

Robot Control with Variable Autonomy from the JPL BioSleeve”, 2013 IEEE International Conference on Robotics and

Automation (ICRA) Karlsruhe, Germany, May 6-10, 2013.

4/21/2015 25