A Wearable Visuo-Inertial Interfacefor Humanoid Robot Control

Junichi SugiyamaDept. of Computer Science and Engineering

Toyohashi University of TechnologyToyohashi, Aichi, Japan

Email: sugiyama@aisl.cs.tut.ac.jp

Jun MiuraDept. of Computer Science and Engineering

Toyohashi University of TechnologyToyohashi, Aichi, Japan

Abstract—This paper describes a wearable visuo-inertial in-terface for humanoid robot control, which allows a user to controlmotion of a humanoid robot intuitively. The interface composedof a camera and inertial sensors and estimates body motion of theuser: movement (walk), hand motion, and grasping gesture. Thebody motion (walk) estimation is performed by a combinationof a monocular SLAM and a vision-inertial fusion using anextended Kalman filter. The hand motion is also estimated byusing the same motion model and sensor fusion as the bodymotion estimation. The estimated motion was used to operate themovement or arm motion of the humanoid robot. We conductedthe experiment on robot operation. The results revealed thatthe user intuitively controlled the robot and it responded to theoperator commands correctly.

Keywords—human robot interaction; humanoid robots

I. INTRODUCTION

Service robots are expected to support human in variouseveryday situations. However it is difficult to give a robot acomplete set of required skills and knowledge to achieve tasksin advance. On-site teaching is an effective way to transfer theskills and the knowledge to the robot.

In this paper, we adopt human body motion to teach robotmotion as a natural and intuitive method. Such a methodfor supervising the robot’s motion can be realized throughcapturing the human motion. There are several methods formeasurement of human motion: by motion capture system, byenvironmental cameras [1], or through attachment of manysensors to the user’s body [2] [3]. These approaches haveseveral limitations such as narrow field of view and equipmentthat is difficult to wear.

For estimation of user’s body motion we developed awearable interface with a camera and two inertial sensors. Theinformation from these sensors is used to estimate body move-ment (walk) and hand motion of the user. The estimated bodymovement and hand motion is used to operate the movementand arm motion of a humanoid robot. We implemented anexperimental system and successfully applied it to controllingthe humanoid robot.

II. SYSTEM OVERVIEW

The robotic system is composed of the body motionestimation interface and the mobile humanoid robot system.

Fig. 1. User with the body motion estimation interface.

Kinect

Laser range finders

HIRO

Omnidirectional mobile base

(a) Humanoid robot, a Hiroon an omni-directional mo-bile base.

Interface MobileHiro HiroGate

To RTC on Hiro

OmniLRFVehicle

VelocityFilter

Target velocity

Vehicle pose

OmniVehicle

Hiro operation service

Estimation result (motion, hand pose, and grasping state)

(b) System diagram of the robot system usingOpenRTM, with the major inputs, outputs, andservices.

Fig. 2. Robot and software system.

Fig. 1 shows a user with the proposed interface. Thewearable interface consists of the head-mounted part and thehand-worn part. The head-mounted part has a camera and a 9-axis orientation sensor attached to a monocular head-mounteddisplay. The orientation sensor includes a 3-axis accelerometer,a 3-axis gyroscope, and a 3-axis magnetometer. The hand-wornpart has an inertial sensor with a 3-axis accelerometer and a3-axis gyroscope.

Fig. 2a shows the humanoid robot, Hiro by KawadaIndustries on an omni-directional mobile base by KER. Inthis robotic system, the functional software modules (Fig.2b) are constructed as RT components (RTC) with Open-RTM (http://www.openrtm.org/). InterfaceGate receives theestimated poses of the user’s body, the hand motion, andgrasping gesture from the interface. Then, this RTC transformsthese data into the head tilt angle, the pose of right hand ofHiro, and the pose of the mobile base. These transformed datais sent to MobileHiro as target inputs. MobileHiro controlsHiro and the mobile base: Hiro is controlled through HiroGatewhich serves as a gateway to its motion services; the velocity

978-1-4673-3101-2/13/$31.00 © 2013 IEEE 235 HRI 2013 Proceedings

of the mobile base is controlled by OmniVehicle. OmniLRFand VehicleVelocityFilter are used for detection of collisionwith obstacles by using the laser range finders (Fig. 2a).

III. BODY MOTION ESTIMATION INTERFACE

The interface (Fig. 1) is based on the body motion esti-mation method which we have developed [4]. This methodcombines a monocular SLAM [5] with a vision-inertial fusionusing an extended Kalman filter [6] for both movement andhand motion estimation. By the vision-inertial sensor fusion, adrift-less and robust movement estimation is realized withoutany environmental sensors.

The previous system used a Wii remote with a visualmarker for hand motion estimation. In this system, we estimatethe hand motion without any markers. The new method detectsthe hand in the camera image based on skin color and pairs ofedges which lie in a predetermined finger width [7]. It can alsodetect the grasping gesture by analyzing the edge intensity inthe hand region (Fig. 3c, d).

IV. EXPERIMENTAL RESULT

We carried out an experiment of the robot operation usingthe interface. Fig. 3 shows the experimental procedure. In thisexperiment, the user wearing the interface walks and moves hishand simultaneously. The interface estimated the user’s motionsuccessfully (see the middle column of Fig. 3) and sent theestimated poses and the grasping gesture to the robot. Therobot then followed the user’s motion correctly (the left andthe right column of Fig. 3).

V. CONCLUSIONS

We proposed a wearable visuo-inertial interface for con-trolling the arm/hand/mobile platform motion of the robotby the human body movement. The interface has a cameraand two inertial sensors; the streams of information fromthose sensors are fused on-line by an EKF for fast androbust body motion estimation. The interface enables us tocontrol a humanoid robot naturally and intuitively without anyenvironmental sensors.

The future work will be devoted to the application of thesystem to mobile manipulation tasks in, for example, remote-control and/or telepresence applications. This will requirevarious sensory feedback from the remote robot such as visualand tactile information for a safe and easy robot control.Integration of autonomous and manual control modes forperforming complex tasks is also the scope of research interest.

REFERENCES

[1] T. Matsuyama, X. Wu, T. Takai, and T. Wada, “Real-time dynamic 3dobject shape reconstruction and high-fidelity texture mapping for 3dvideo,” IEEE Trans. on Circuits and Systems for Video Technology, vol.CSVT-14, no. 3, pp. 393–434, 2004.

[2] T. Harada, T. Gyota, Y. Kuniyoshi, and T. Sato, “Development ofwireless networked tiny orientation device for wearable motion captureand measurement of walking around, walking up and down, and jumpingtasks,” in Proc. of IEEE/RSJ 2007 Int. Conf. on Intelligent Robots andSystems, pp. 4135–4140, 2007.

[3] T. Maeda, H. Ando, M. Sugimoto, J. Watanabe, and T. Miki, “Wearablerobotics as a behavioral interface – the study of the parasitic humanoid–,” in Proc. of 6th Int. Symp. on Wearable Computers, pp. 145–151,2002.

(a) 5sec: initial pose.

(b) 10sec: the user walks.

(c) 11sec: the user moves his right hand (hand open) and the robot followshis motion.

(d) 17sec: the user stops (hand close).

(e) 45sec: the user moves backward and stops (hand open).

Fig. 3. Result of the robot operation experiment.

[4] J. Sugiyama and J. Miura, “A wearable robot control interface based onmeasurement of human body motion using a camera and inertial sensors,”in Proc. of the 2011 IEEE Int. Conf. on Robotics and Biomimetics, pp.565–570, 2011.

[5] J. Montiel, J. Civera, and A. Davison, “Unified inverse depth parametriza-tion for monocular slam,” in Proc. of Robotics: Science and Systems,2006.

[6] L. Armesto, J. Tornero, and M. Vincze, “Fast ego-motion estimation withmulti-rate fusion of inertial and vision,” The Int. Journal of RoboticsResearch, vol. 26, no. 6, pp. 577–589, 2007.

[7] N. Petersen and D. Stricker, “Fast hand detection using posture invariantconstraints,” in Proc. of the 32nd annual German conference on Advancesin artificial intelligence, pp. 106–113, 2009.

978-1-4673-3101-2/13/$31.00 © 2013 IEEE 236 HRI 2013 Proceedings