immersion type virtual environment for human …onishi-lab.jp/riken/pdf/596_wb3_5_1622.pdfimmersion...
TRANSCRIPT
Immersion type virtual environment forhuman-robot interaction∗
Tadashi OdashimaBio-Mimetic Control Research Center
RIKENNagoya, Japan
Masaki OnishiBio-Mimetic Control Research Center
RIKENNagoya, Japan
Zhiwei LuoBio-Mimetic Control Research Center
RIKENNagoya, Japan
Shigeyuki HosoeBio-Mimetic Control Research Center RIKEN,
Nagoya UniversityNagoya, Japan
Abstract – With the development of information sci-ence and robotic technology, it becomes more importantto generate human interactive robots. The design plat-form for developing such robots should satisfy three ba-sic conditions: (1) It can test safely the performanceof the robot through the physical interaction with hu-man, (2) Human subject can estimate subjectively theoutside appearance of the robot, and (3) It can simu-late the dynamic human interactive robot motion withinreal-time. This paper proposes our immersion type dy-namic simulation platform. An application to estimatethe robot’s performance when performing cooperative ob-ject lifting task with human subject is chosen in order toshow the effectiveness of our system. The analysis ofthe recorded data is useful to design the novel humaninteractive robots.
Keywords: Immersion projection technology, 3D dy-namic simulator, Subjective estimation, Human inter-active robot.
1 IntroductionWith the development of information science and
robotic technology, it becomes more important to gen-erate human interactive robots. One of the most desiredspecifications for the robot is to cooperate with humanin general environment such as streets or homes. Thefinal goal of our research is to construct such a robotwhich can interact physically with human. We call ita “human cooperative robot” in this paper. There aremany different aspects between the traditional indus-trial robots and the human cooperative robot. One of
∗0-7803-7952-7/03/$17.00 c� 2003 IEEE.
them is the basic requirement of the robots. It is re-quired for the industrial robot to perform operationswith high speed and high accuracy, whereas the highsafety and affinity are more important for the coopera-tive robot. The criteria for the cooperative robot tendto be more subjective than that for the industrial robot.Another difference is the evaluation method. The sit-uation where the cooperative robot is evaluated mustalso include human. This fact derives difficulties forevaluating the cooperative robot. The experiment us-ing directly the real cooperative robot is very danger-ous, especially, when the new algorithm or mechanicalsystem is applied. In this sense, it is considered thatcomputer simulation is the best way for the evaluation.However to build the precise human model in the sim-ulator is much more difficult. Even if the model canbe constructed, it is hard to evaluate the impressionsreceived from the robot. It is also important for thehuman subject to feel the robot just as if the humansubject and the robot exist in the same space.
Considering above aspects, novel design platform isnecessary for developing the final human cooperativerobot. The basic conditions for the environment are:1) It is possible to interact with the robot safely. 2)The human subject can experience the robot with highreality for subjective evaluation, 3) It can simulate thedynamic robot motion within real-time. By now, severalrobot simulators are proposed [3, 4, 6]. In reference [6],a simulator satisfying conditions 1) and 3) is proposed.The simulator can also provide a reaction force from thevirtual space by haptic interface, but the visual imageis displayed just on a usual monitor as 2-dimensionalimage.
Immersion TypeDisplay
3D Dynamic Simulator
Motion Capture
Figure 1: System overview
In this research, we construct an effective develop en-vironment that satisfies all three conditions by combin-ing the virtual reality techniques and 3-dimensional dy-namic simulator[5]. In this paper, we express our systemsetup in detail and show the experimental results.
2 Outline of the systemAs shown in Figure 1,the robot simulator which we
propose consists of three parts. They are: (1) An im-mersion type display, (2) A 3-dimensional dynamic sim-ulator, and (3) A motion capturing part, respectively.Each part consists of two or more PC computers, andall computers can communicate with each other viaTCP/IP.
The immersion type display part presents human withaudio-visual information and tactile sensation, whichmakes the human subject feel as if he/she is interactingwith the virtual robot directly. For vision display, it isnecessary to give the subject the feeling as if he/she isfacing an actual robot. For audition, on the other hand,the location of the sound source such as the robot’s foot-step and voice should be set so that the subject can hearwhere the sound actually comes from. In addition, whena subject touches the robot, it is necessary to display theforce, although it has not realized and is under researchnow.
The 3-dimensional dynamic simulator part controlsa virtual robot to follow the given set of target angleswhile interacting with human subject. Here, it is nec-essary for the 3-dimensional dynamic simulator part toconsider the external force applied to the robot fromhuman as well as the gravity in its dynamic calculation.As a result of this calculation, the instruction torqueis calculated based on the target angle and the torqueadded by the subject. The position and orientation ofeach link are then calculated in real time for every sam-pling period.
The robot’s form takes as a humanoid-type, its targetjoint angle is measured from the motion of an actualhuman subject by using the motion capture system.
PC/AT Clones
Front screenSide screen
Projector
Side screen
Speaker
Figure 2: Backside of the PC-CAVE
3 Immersion type displayAs shown in the Figure 2, the immersion type display
part presents the human subjects with high presence ofthe virtual robot. The visual and auditory informationplaying an important part for perception are installed inthis system. In addition, this part measures the motionof a human subject in order to reflect his/her motioninto the virtual space.
3.1 Displaying visual information
Immersive Projection Technology (IPT) uses parallelimage display system to show the 3-dimensional stere-ography. The displayed images are projected on thescreens arranged as the wall surrounding the humansubject. The high resolution of image and a large visualfield angle provide a high feeling of immersion to hu-man. Some types of IPT have been reported in the past.CAVE (CAVE Automatic Virtual Environment) [1] hasfour screens (front, right, left, and platform screen), andthe basic projection structure of our immersion type dis-play is the same. Immersion type display processes andrenews the images of a virtual space by sensing the po-sition and orientation of the human subject’s head withmagnetic sensor. The processed images are projectedonto 4 screens simultaneously so as to be observed asthe 3D image from the human subject.
The conventional IPT is constituted with the expen-sive supercomputer, whereas we use only four PC/AT-clone computers, which reduces the total cost of IPTand realizes the easy use without special skill. Fromour experiments, it is found that, although the calcu-lation capability of a PC/AT-clone computer is inferiorto an expensive supercomputer, it is possible to coverthe lack of performance by reducing the complexity ofthe displayed contents. Moreover, considering the rapidincrease of the calculation and/or drawing capability ofPC/AT-clone computer and ease of maintenance, thissetup can be used to realize our research objective. Inthe following paper, we simply call this projection sys-tem “PC-CAVE”.
3.2 Displaying auditory information
The display of auditory information is realized usingHuron of Lake Technology. Huron is the virtual realitytechnology that can generate the virtual sound space
Sensor Posiotion
(a) Subject in thePC-CAVE
(b) Virtual subjectmodel
Figure 3: Subject and virtual subject
with high reality by eight speakers array. Each speakeris set at every corner of the PC-CAVE structure. Theapplications of the Huron here are to play the soundfrom a virtual robot, for example, the utterance made byspeech synthesis, the sound of footsteps and the soundwith tumbling down.
3.3 Sensing subject’s motion
It is necessary to send the subject’s motion to the3D dynamic simulator in order to realize the interac-tion between a human subject and the virtual robot.Therefore immersion type display performs not only pre-senting visual-audio information but also measuring thesubject’s motion simultaneously. To achieve accuracysimulation results, it is better to use sensors as manyas possible for sensing the subject’s position and orien-tation precisely. The increase of the sensors causes theenlargement of the system scale as well as the calcu-lation load, which then leads to the serious problem inpreventing the subject from receiving visual informationwith high presence. Therefore we limit the number ofmagnetic sensors measuring the position and orientationup to four. These sensors are attached on the subject’shead, two wrists and body as shown in Figure.3(a). Thevisual image presented in immersion type display is re-freshed based on the information comes from the sensorattached on head. The measured data from other sen-sors are sent to 3D dynamic simulator for controlling avirtual subject.
4 3D dynamic simulatorThe 3-dimensional dynamic simulator calculates the
position and orientation of every part of the virtualrobot. This dynamic simulator is coded by applyingthe programming library Vortex of CM-Labs Simula-tions Inc.. Vortex has functions to process the collisioncheck between the links and the dynamic motion calcu-lations of each link in real time. The calculated positionand orientation of all links is transmitted to the immer-sion type display part using TCP/IP. Owing to the higheffectiveness of the processing routine, the time delayfor rendering on the immersion type display part is verysmall. The measurement data of two human motionsare gathered into this simulator for controlling virtual
15 D.O.F.
Figure 4: Virtual robot
objects. One is the motion of the human subject in theimmersion type display mentioned in 3.3. This data isused for controlling the virtual subject. Another is themotion captured in motion capture system described in5, which is used for deciding the target joint angle ofthe virtual robot.
4.1 Virtual subject
A virtual model of the human subject in PC-CAVEis build in virtual space, so that he/she interacts witha virtual robot. In the virtual space, he/she can addforce to virtual object by controlling the virtual subject.Since we limit the interaction on only the upper half ofthe subject’s body, as shown in Figure.3(b), the virtualsubject is constructed from a torso including head, up-per arms, and forearms including hands. Each Link isconstrained with next by free ball-socket joint. It is bet-ter to construct a detailed model copied human struc-ture and to control it by the captured human motion inorder to acquire accurate simulation results. As men-tioned before, however, we can use only four magneticsensor, the torso and forearms are controlled directory,and the upper arms are moved indirectly by the me-chanical constrains. The length and size of every link isset based on the size of human subject.
4.2 Virtual robot
Since it is important for the robot interacting with hu-man to have a high personal affinity design, the robot’sform is better to be a humanoid. The main objectiveof our robot research here is not focused on the leggedlocomotion, but on the cooperation with human. Asshown in Figure.4, the wheel is used for simplifying thelocomotion mechanism, and only the upper half of thebody is designed as humanoid. The virtual robot hastotally 15 d.o.f., including wheels’ rotation. Each armpart has 5 d.o.f., 3 at the shoulder, 1 at the elbow and1 at the wrist. The neck, waist and knee has 1 d.o.f.,respectively. The size of every part is designed based onthe human parameters.
This simulator not only enables us to estimate theoutside appearance of the robot subjectively but alsomakes it possible to evaluate the specification of therobot numerically. Using the real motor parameters for
(a) Real human movements
(b) Results of motion capture
(c) 3D dynamic simulation
(d) Interaction between human subject and a robot in the PC-CAVE
Figure 5: Experimental scene
controlling the virtual robot motion may help the se-lection of actuators for designing the real robot. Inaddition, many kinds of information about the robotcan be measured easily in the simulator. For example,the force and torque at any contact point is measuredby setting virtual sensor function, and the visual datafrom any point is obtained by setting the camera func-tion. These functions are useful for the later analysisof the robot motion. The simulation results about themeasured data are presented in 6.
The joint angles of the virtual robot are controlled byPD control, and the desired angle is decided from themotion data captured in the motion capture system.The simulation result consists of the data of position,orientation and index label of all links except the linksconstructing the virtual subject, and is sent to the im-mersion type display for presenting 3D image.
5 Motion captureConsidering the affinity of human interactive robot,
we selected the humanoid design for our robot and builtthe motion capture part for getting the human motion inorder to acquire the desired angle for the virtual robot.
In our system, the capture data is used to controlthe virtual robot in the simulator. Here, VICON with
six high-speed infrared cameras is used to observe themarkers attached on the human body with the samplingperiod of 120Hz. From the precise point of view, it isdesired to attach markers on the whole body of the sub-ject, however, since the biped walk is not easy, in thisresearch, we focus only on the motion of upper bodypart, especially the movement of both arms. In orderto capture the motion of both arms, 16 markers are at-tached on specific points between the shoulder and handof both arms. These makers cover totally the motion upto 10 d.o.f., that is, 3 d.o.f. of each shoulder, 1 d.o.f. ofthe elbow, and 1 d.o.f. of the wrist, respectively. Themeasured joint angles are sent to the 3-dimensional dy-namic simulator part explained in 4 via TCP/IP andare used as the target joint angles of the virtual robot.
6 Experimental resultsThe experiments using the proposed robot simulator
have been performed to show the effectiveness of ourresearch.
6.1 Interaction in PC-CAVE
Figure 5 shows the results of an experiment using oursystem. Figure 5 (a) shows the motion of a human sub-ject who was attached with 16 markers, and then 10
Figure 6: Example of whole body manipulation
joint angles of the upper half body of the subject arecalculated from these markers’ position data. (b) showsthe calculated result of motion capturing by the VI-CON. The set of calculated angles are sent to the 3Ddynamic simulator as a set of target angles. The virtualrobot shown in (c) basically follows the target angles byPD control in 3D dynamic simulator, and the virtualsubject (in (c) left) follows the human subject’s motionmeasured in the immersion type display. The stereo-scopic glasses are set on the side of the video camerawhen the images in (d) are recorded so as to keep theconsistency of these images as seen from the human sub-ject.
The subject in the immersion type display can recog-nize the real scale of the virtual robot and add forces bypushing or pulling some parts of the virtual robot. InFigure 5(d), the human subject tries to stop the swingof the virtual robot arm. It is clear that the added forceworks as the external force, and the force makes dif-ference between the posture of robot arm and that ofcaptured motion shown in (b). The robot falls downwhen a large force is added, and a tumbling sound isgenerated from the falling point by sound effect.
Figure 6 shows the experiment of a cooperative task tocarry the human-formed object from the human subjectto the virtual robot. Such kind of cooperation is consid-ered as an essential task for the robot in the human careand welfare areas. The subject can get an impressionagainst the robot through this task, and the subjectiveevaluation is important for designing the real robot’sshape and motions. Of course, numerical data can also
-100
-50
0
50
100
150
200
0 [ ]sec
Nm[ ]
2 4 6 8 10
(a) Torque of the shoulder joint
-100
-50
0
50
100
150
200
0 [ ]sec
[ ]Nm
2 4 6 8 10
(b) Torque of the elbow joint
Figure 7: Torque of the each joint
be monitored in this simulator. Recently, our researchteam has also studied on the Whole Body Manipulation(WBM)[2] for human-robot cooperation. The numericaldata derived from our system is useful for analyzing thealgorithm of the WBM and for estimating the efficiencyof the WBM.
6.2 Sensor simulator
In virtual space created by the 3D dynamic simulator,we can monitor easily any value by only implementingvirtual sensory functions, even if it is difficult to monitorthe values practically in real space. Currently, threetypes of virtual sensory functions have been developed.They are torque sensor, force sensor and cameras.
The reaction force at any collision point and/or theforce and torque added at specific constraint is calcu-lated in the 3D dynamic simulator for deciding motionof every object. The virtual force/torque sensor hasfunctions to get these values.
Figure 7 shows the time series of the joint torque dur-ing the cooperative task mentioned in previous section(Figure.6). The weight of the human formed object isset to 47.5[kg] in this experiment. According to thisresult, a peak appears at the moment when the robotreceive the object.
Responses of the force sensor function is shown in Fig-ure.8. The human subject hits the virtual robot handthree times, twice from up side, and once from below.Each graph illustrates each direction corresponding toeach axis in a coordinate system fixed on ground. Con-sidering the upper direction is a positive on y axis, thehitting direction is clear according to (b).
The subject cannot feel reaction force from virtualspace. The development of the force display device isour future work. The monitored force on the body ofthe virtual subject can be used as the target value forthe device.
-20
-40
0
20
40
0 2 4 6 8 10 sec[ ]
N[ ]
(a) Force of x-axis
20 6 8 104 sec[ ]
40
20
0
-20
-40
N[ ]
(b) Force of y-axis
0 2 4 6 8 10 sec[ ]
-40
-20
0
20
40N[ ]
(c) Force of z-axis
Figure 8: Input data of force sensor
Rendering the visual image from specific point in vir-tual space corresponds to set the camera in real space.The images input from the virtual robot’s left and righteyes are shown in Figure 9 (b) and (c), respectively. Weunderstand that the virtual robot is looking at the hu-man formed object, the virtual human subject and itshands. Since the virtual subject image means the pres-ence of the human subject, the virtual robot can detectthe human subject in real space and makes an actionto the human subject, which means the bilateral inter-action could be realized. In order to detect the humansubject, the virtual robot can select two ways. One isto get the information about the virtual subject directlyfrom 3D dynamic simulator, and another is to processthe image. The later is more realistic situation for ap-plying to the real robot. Figure 9(d) and (e) show theedge detection as an example of image processing.
7 ConclusionsIn this paper, we express the 3-dimensional dynamic
simulation environment for evaluating the human coop-erative robot which can interact physically with human.This environment consists of three parts, the immersiontype display part, the 3-dimensional dynamic simulatorpart, and the motion capturing part. The subject caninteract with a virtual robot, which follows a human mo-tion, using advanced virtual reality technology. In thisexperiment, we confirmed the performance of the pro-posed system by monitoring the virtual robot motionunder an external force from the subject. In addition,we developed the virtual sensor functions. These are aforce sensor, torque sensor, and cameras.
(a) Image of left eye
(c) Edge image
(b) Image of right eye
(d) Edge image
Figure 9: Input image of virtual visual sensors
Currently, the development of force display devicewhich represents reaction forces from virtual space tothe human subject is not finished. It is our futureresearch subject. Moreover, the measurement of jointtorque of human motion capturing part is also underworking. The torques are measured by implementingmulti channels of EMG, 8 force plates as well as multichannels of accelerometer set in the motion capturepart. It is our future subject for the virtual robot tomimic the skillful and dynamic movement of human sub-jects.
References[1] C. Cruz-Neira, et al., “Surround-Screen Projection-
Based Virtual Reality: The Design and Implementa-tion of the CAVE,” Proc. SIGGRAPH 93, pp.135–142,1993.
[2] F. Asano, et al., ”Dynamic Modeling and Control forWhole Body Manipulation,” Proc. IEEE/RSJ Inter-national Conference on Intelligent Robots and Sys-tems(IROS), 2003, to appear.
[3] F. Gravez, et al., “Dynamic Simulation of a Hu-manoid Robot with Four DOFs Torso,” Proc. IEEEInternational Conference on Robotics and Automation(ICRA2002), vol.1, pp.511–516, May 2002.
[4] F. Kanehiro, et al., “Open Architecture HumanoidRobotics Platform,” Proc. IEEE International Confer-ence on Robotics and Automation (ICRA2002), vol.1,pp.24–30, May 2002.
[5] M. Onishi, et al., “Development of PC-based 3D Dy-namic Human Interactive Robot Simulator,” Proc.IEEE International Symposium Computational Intelli-gence in Robotics and Automation (CIRA), 2003, toappear.
[6] O. Khatib, et al., “Robotics and Interactive Simula-tion,” Communication of the ACM, vol.45, no.3, pp.46–51, March 2002.