Training wheels for the robot: Learning from demonstration using simulation

Nathan KoenigOpen Source Robotics Foundation

Mountain View, CA

Maja MataricUniversity of Southern California

Los Angeles, CA

IntroductionLearning from demonstration (LfD) is a promising tech-nique for instructing/teaching autonomous systems based ondemonstrations from people who may have little to no expe-rience with robots (Billard et al. 2008). A key aspect of LfDis the communication method used to transfer knowledgefrom an instructor to a robot. The communication methodaffects the complexity of the demonstration process for theinstructor, the range of tasks the robot can learn, and thelearning algorithm itself.

We have designed a graphical interface and an instruc-tional language to provide an intuitive LfD teaching system.The challenge of simplifying the teaching interface is thatthe resulting demonstration data are less structured, addingcomplexity to the learning process. In our approach, this ad-ditional complexity is handled through the combination ofusing a minimal set of predefined behaviors and a task repre-sentation capable of learning probabilistic policies over a setof behaviors. The predefined behaviors consist of finite ac-tions a robot can perform, such as move to, pick up, and putdown. Behaviors act as building blocks for more complextasks, and are parametrized by features, objects the robotcan observe and manipulate.

A series of behaviors and features from a group of instruc-tors is used to produce a generalized policy for a task. Thepolicy is represented by a set of influence diagrams, an ex-tension of Bayesian networks, that incorporate the ability toprobabilistically choose actions based on state information.We allow for error in the teaching and learning process byproviding a mechanism to refine influence diagrams duringautonomous operation. This technique effectively reducesthe number of complete demonstrations required for a robotto accurately learn a task.

Teaching Interface and Task LearningTeaching interfaces use a variety of forms, includingmanual manipulation (Hersch et al. 2008), teleoperationvia joysticks (Grollman and Jenkins 2007), graphical in-terfaces (Chernova and Veloso 2008), external observa-tions (Schaal, Ijspeert, and Billard 2004), and sensors placedon the instructor (Ijspeert, Nakanishi, and Schaal 2002).

Copyright c© 2012, Association for the Advancement of ArtificialIntelligence (www.aaai.org). All rights reserved.

These methods necessarily trade off teaching complexityand complexity of the learning algorithm.

Our approach emphasizes ease of use for instructors andgenerality of use with multiple robot platforms. The recon-figurable graphical interface meets these needs by providingan experience that is familiar to people with some computerexperience. The instructor uses the graphical interface toobserve the state of the world and send instructions to therobot, as shown in Figure1.

The instructor commands the robot by building sentencesusing an instructional language that is composed of a behav-ior, a feature, and an optional modifier-feature combinationthat specifies how the behavior is performed relative to thefeature. An example instruction is put down the salt on thetable, where put down is the behavior, salt is the feature, andon the table is the optional modifier-feature combination.

The graphical interface guides the instructor through theprocess of constructing a sentence by sequentially askingfor the next part of the instruction. An undo option allowsthe user to easily fix mistakes. Once the instruction is com-plete, it is sent to the robot for execution. During execution,the user observes the robot’s progress and any textual feed-back from the robot. The textual feedback consists of sen-tences that describe the robot’s internal state and responseto instructions. Feedback examples include, ”Moving tosaucepan”, ”Unable to reach bowl”, or ”Waiting 5 seconds”.The process of building instructions continues until the in-structor decides the task demonstration is complete.

During the demonstration, the robot records the instruc-tions and world state information. After one or more in-structors have provided demonstrations, the recorded infor-mation is used to learn a task policy. The task is representedby a set of influence diagrams constrained to non-realtimeand discrete sequences of behaviors. The influence diagramlearning algorithm and example influence diagrams are de-tailed in (Koenig, Takayama, and Mataric 2010).

Experimental SetupThe graphical interface and learning system were used toteach a robot the process of preparing mushroom risotto.The complete set of predefined behaviors consisted of moveto, pick up, put down, stir, chop, turn on, turn off, and wait,which represent the minimum set needed to complete therisotto task.

Figure 1: Simulated kitchen environment with teaching in-terface. 1. Robot camera view, 2. Contents of saucepan, 3.Contents of bowl, 4. Left gripper, 5. Right gripper, 6. Robotfeedback, 7. Instruction interface, 8. Change view left/right,9. Demonstration complete.

The demonstration environment used the Gazebo(Koenig and Howard 2004) simulation environment, withdynamics disabled for improved speed. The environmentconsisted of a kitchen, a PR2 robot, ingredients, andutensils. Simulation was used to reduce teaching time,provide a stable environment, and facilitate error correctionthrough an undo feature that reverses instructions.

A fifteen-step tutorial guided participants through thegraphical interface. Upon completion of the tutorial, par-ticipants received a printed mushroom risotto recipe. At thispoint, each participant was free to instruct the robot. Follow-ing the demonstration, each participant completed a surveythat collected demographic information.

ResultsThirty three participants completed the demonstration, 26male and 7 female, with an age range of 23 to 67. The av-erage time to complete the demonstration was 27 minutes,with a standard deviation of 10 minutes. In contrast, a PR2robot took roughly three to four times as long to completethe task due to the pace of perception and slowness of themovements for safety.

As expected, the instructions sent to the robot were simi-lar across participants in the beginning of the demonstration.The instructions diverged over time as participants chose tocomplete the recipe using a different order of instructions.This divergence made it more difficult to learn correct in-fluence diagrams. As a result, a few influence diagramsproduced incorrect behavior when used on an autonomousrobot.

The incorrect diagrams can be discovered by the systemitself or by a human observer. An incorrect influence dia-gram can be self-discovered when multiple behaviors havethe same probability. In these cases, the system is incapableof choosing the best behavior and must ask for help. In allother instances, the system chooses what it believes to bethe best behavior, which may not in fact be what a humanobserver would select.

In both cases, a human may intervene and provide thesystem with the correct behavior. This new information is

incorporated directly into the influence diagram through anupdate process.

Discussion and Future WorkWe have developed an approach that provides an intuitive in-terface for instructing a robot in time-extended tasks. The in-terface requires no prior knowledge of robotics. Teaching insimulation provides a less time consuming teaching experi-ence with the ability to gracefully fix errors. The demonstra-tion data are used to generate influence diagrams that repre-sent the task being taught. These diagrams can be refinedwhen the system operates autonomously.

Due to the influence diagram representation, we are con-strained to high-level tasks that are categorized as a time-extended series of behaviors. The process of solving influ-ence diagrams does not operate on the time scale necessaryfor robot joint angle control. We are also limited by therange of features and behaviors available to the robot. Thediversity of features is ever increasing, but is still limited bycurrent work in object perception and recognition.

Our future work will explore incorporating knowledgetransfer across tasks to allow the robot to utilize influencediagrams from one task to reduce teaching time for a newtask. We will also study the use of graphical interfaces as ateaching tool, and determine how best to provide two-waycommunication between an instructor and a robot.

ReferencesBillard, A.; Calinon, S.; Dillmann, R.; and Schaal, S. 2008.Robot programming by demonstration. In Handbook ofRobotics. MIT Press. chapter 59.Chernova, S., and Veloso, M. 2008. Teaching collabora-tive multi-robot tasks through demonstration. In IEEE-RASInternational Conference on Humanoid Robots.Grollman, D. H., and Jenkins, O. C. 2007. Learning robotsoccer skills from demonstration. In International Confer-ence on Development and Learning, 276–281.Hersch, M.; Guenter, F.; Calinon, S.; and Billard, A.2008. Dynamical system modulation for robot learning viakinesthetic demonstrations. IEEE Transactions on Robotics24(6):1463–1467.Ijspeert, J. A.; Nakanishi, J.; and Schaal, S. 2002. movementimitation with nonlinear dynamical systems in humanoidrobots. In international conference on robotics and automa-tion (icra2002).Koenig, N., and Howard, A. 2004. Design and useparadigms for gazebo, an open-source multi-robot simulator.In IEEE/RSJ International Conference on Intelligent Robotsand Systems, 2149–2154.Koenig, N.; Takayama, L.; and Mataric, M. 2010. Learningfrom demonstration: A study of visual and audiory commu-nication and influence diagrams. In International Sympo-sium on Experimental Robotics (ISER’10).Schaal, S.; Ijspeert, A.; and Billard, A. 2004. Computationalapproaches to motor learning by imitation. In the neuro-science of social interaction, number 1431. Oxford Univer-sity Press. 199–218.