From Virtual Demonstration to Real-World Manipulation ... ?· From Virtual Demonstration to Real-World…

Download From Virtual Demonstration to Real-World Manipulation ... ?· From Virtual Demonstration to Real-World…

Post on 26-Jul-2018

212 views

Category:

Documents

0 download

TRANSCRIPT

  • From Virtual Demonstration to Real-World Manipulation Using LSTM and MDN

    Rouhollah Rahmatizadeh, Pooya Abolghasemi, Aman Behal, Ladislau BoloniDepartment of Computer Science

    University of Central Florida, United Statesrrahmati,pooya.abolghasemi,lboloni@cs.ucf.edu, abehal@ucf.edu

    Abstract

    Robots assisting the disabled or elderly must perform com-plex manipulation tasks and must adapt to the home environ-ment and preferences of their user. Learning from demonstra-tion is a promising choice, that would allow the non-technicaluser to teach the robot different tasks. However, collectingdemonstrations in the home environment of a disabled useris time consuming, disruptive to the comfort of the user, andpresents safety challenges. It would be desirable to performthe demonstrations in a virtual environment.In this paper we describe a solution to the challenging prob-lem of behavior transfer from virtual demonstration to a phys-ical robot. The virtual demonstrations are used to train a deepneural network based controller, which is using a Long ShortTerm Memory (LSTM) recurrent neural network to generatetrajectories. The training process uses a Mixture Density Net-work (MDN) to calculate an error signal suitable for the mul-timodal nature of demonstrations. The controller learned inthe virtual environment is transferred to a physical robot (aRethink Robotics Baxter). An off-the-shelf vision componentis used to substitute for geometric knowledge available in thesimulation and an inverse kinematics module is used to allowthe Baxter to enact the trajectory.Our experimental studies validate the three contributions ofthe paper: (1) the controller learned from virtual demonstra-tions can be used to successfully perform the manipulationtasks on a physical robot, (2) the LSTM+MDN architecturalchoice outperforms other choices, such as the use of feedfor-ward networks and mean-squared error based training signalsand (3) allowing imperfect demonstrations in the training setalso allows the controller to learn how to correct its manipu-lation mistakes.

    IntroductionAssistive robotics, whether in the form of wheelchairmounted robotic arms or mobile robots with manipulators,promises to improve the independence and quality of lifeof the disabled and the elderly. Such robots can help usersto perform Activities of Daily Living (ADLs) such as self-feeding, dressing, grooming, personal hygiene and leisureactivities. Almost all ADLs involve some type of objectmanipulation. While most current systems rely on remote

    Copyright c 2018, Association for the Advancement of ArtificialIntelligence (www.aaai.org). All rights reserved.

    control, there are ongoing research efforts to make assis-tive robots more autonomous (Endres, Trinkle, and Burgard2013; Miller et al. 2012; Bollini, Barry, and Rus 2011). Oneof the challenges of assistive robotics is the uniqueness ofevery home and the need to adapt to the preferences anddisabilities of the user. As the user is a non-programmer,learning from demonstration (LfD) is a promising way toadapt the robot behavior to the specific environment, objectsand preferences. However, collecting large numbers of phys-ical demonstrations from a disabled user is challenging. Forinstance, the manipulated objects and the environment canbe fragile and the wheelchair-bound users might have dif-ficulty recovering from a failed manipulation task such as adropped cup. Similar considerations hinder from-the-scratchreinforcement learning performed in a home environment.

    In this paper we propose an approach where the usersdemonstrate the tasks to be performed in a virtual environ-ment. This allows the safe collection of sufficient demon-strations to train a deep neural network based robot con-troller. The trained controller is then transferred to the phys-ical robot. The general flow is illustrated in Figure 1.

    In the remainder of this paper we discuss related work,describe the approach in detail, and through an experimen-tal study validate the three contributions outlined in the ab-stract.

    Related workVirtual training to physical execution. The desirability oftransferring learning from a simulated robot to a physicalone had been recognized by many researchers. This need isespecially strong in the case of reinforcement learning-basedapproaches, where the exploration phase can be very expen-sive on a physical robot. Unfortunately, policies learned insimulation do not work well when transferred navely to aphysical robot. First, no simulation can be perfect, thus thephysical robot will inevitably have a suboptimal policy leav-ing to a slightly different state from the expected one. Thiswould be acceptable if the policy corrects the error, keep-ing the difference bounded. In practice, it was found thatif a policy was learned from demonstrations, initially smalldifferences between the simulation and the real world tendto grow larger and larger as the state diverges farther andfarther from the settings in which the demonstrations tookplace an aggravated version of the problem that led to the

    arX

    iv:1

    603.

    0383

    3v4

    [cs

    .RO

    ] 2

    2 N

    ov 2

    017

  • Virtual world: training the network Physical world: inference from the network

    Demonstration of the task by user in the simulation

    Robot performs the task in real-world based on the trajectory

    generated by the networkTraining an LSTM network

    on demonstrations

    Dataset of trajectories for training

    Current state of the environment

    and gripper

    Multilayer LSTM NN

    Gripper state at next

    time-step

    Current state of the environment

    and gripper

    Gripper state at next

    time-step

    Inverse kinematics

    Multilayer LSTM NN

    Figure 1: The general flow of our approach. The demonstrations of the ADL manipulation tasks are collected in a virtualenvironment. The collected trajectories are used to train the neural network controller.

    development of algorithms such as DAgger (Ross, Gordon,and Bagnell 2011).

    A number of different approaches had been developed todeal with this problem. One approach is to try to bring thesimulation closer to reality through learning. This approachhad been taken by Grounded Simulation Learning (Farchyet al. 2013) and improved by Grounded Action Transforma-tion (Hanna and Stone 2017) on the task of teaching a Naorobot to walk faster.

    Another approach for improving the virtual to physicaltransfer is to increase the generality of the policy learnedin simulation through domain randomization (Tobin et al.2017). (Christiano et al. 2016) note that the sequence ofstates learned by the controller in simulation might be rea-sonable even if the exact controls in the physical worldare different. Their approach computes what the next statewould be, and relies on learned deep inverse dynamicsmodel to decide on the actions to achieve the equivalentreal world state. (James, Davison, and Johns 2017) train aCNN+LSTM controller to perform a multi-step task of pick-ing up a cube and dropping it to a table. The training isbased on demonstrations acquired from a programmed op-timal controller, with domain randomization in the form ofvariation of environmental characteristics, lighting and tex-tures.

    Yet another possible approach is to learn an invariant fea-ture space that can transfer information between the domainsthrough a sort of analogy making (Gupta et al. 2017).

    In the work described in this paper, we took a differentapproach. We had to accept that the simulation is nowhereclose to the physical environment, thus the robot will makemistakes. Instead of domain randomization, we rely on thenatural imperfections of demonstrations done by (possiblydisabled) humans, but also on the ability of humans to cor-rect the errors they made. Using this training data, we foundthat the physical robot also learned to correct its mistakes.

    Autonomous trajectory execution through Learning fromDemonstration (LfD). LfD extracts policies from exam-ple state to action mappings (Argall et al. 2009) demon-strated by a user. Successful LfD applications include au-tonomous helicopter maneuvers (Abbeel, Coates, and Ng2010), playing table tennis (Kober, Oztop, and Peters 2011;Calinon et al. 2010), object manipulation (Pastor et al.2009), and making coffee (Sung, Jin, and Saxena 2015).

    The greatest challenge of LfD is generalization to un-seen situations. One obvious way to mitigate this problemis by acquiring a large number of demonstrations cover-ing as many situations as possible. Some researchers pro-posed cloud-based and crowdsourced data collection tech-niques (Kehoe et al. 2013; Forbes et al. 2014; Crick et al.2011), and some others proposed to use simulation envi-ronments (Fang, Bartels, and Beetz 2016). Another direc-tion is to use smaller number of demonstrations, but changethe learning model to generalize better. One possible tech-nique is to hand-engineer task-specific features (Calinon,Guenter, and Billard 2007; Calinon et al. 2009). Using deepneural networks might help to eliminate the need to hand-engineer features. For instance, (Levine et al. 2016) usedfeed-forward neural networks to map a robots visual inputto control commands. The visual input is processed usinga convolutional neural network (CNN) to extract 2D fea-ture points, then it is aggregated with the robots currentjoint configuration, and fed into a feed-forward neural net-work. The resulting neural network will predict the next jointconfiguration of the robot. A similar neural network archi-tecture is designed to address the grasping problem (Pintoand Gupta 2016). In (Mayer et al. 2006), LSTMs are usedfor a robot to learn to autonomously tie knots after a pre-processing step to reduce the noise from the training data.

  • Our approachCollecting demonstrations in a virtual environmentTo allow users to demonstrate their ADLs, we designed inthe Unity3D game engine a virtual environment modeling atable with an attached shelf that can hold various objects,and a simple two-finger gripper that can be opened andclosed to grasp and carry an object. The user can use themouse and keyboard to open/close the gripper, as well as tomove and rotate it, giving it 7 degrees of freedom. Unity3Dsimulates the basic physics of the real world including grav-ity, collisions between objects as well as friction, but there isno guarantee that these will accurately match the real world.

    We represent the state of the virtual environment as thecollection of the poses of the M movable objects q ={o1 . . . oM}. The pose of an object is represented by thevector containing the position and rotation quaternion withrespect to the origin o = [px, py, pz, rx, ry, rz, rw]. Dur-ing each step of a demonstration, at time step t we recordthe state of the environment qt and the pose of the end-effector augmented with the open/close status of the gripperet. Thus a full demonstration can be recorded as a list ofpairs d = {(q1, e1) . . . (qT , eT )}.

    For our experiments we considered two manipulationtasks that are regularly found as components of ADLs: pickand place and pushing to a desired pose.

    The pick and place task involves picking up a small boxlocated on top of the table, and placing it into a shelf abovethe table. The robot needs to move the gripper from its initialrandom position to a point close to the box, open the gripper,position the fingers around the box, close the gripper, movetowards the shelf in an orientation where it will not collidewith the shelf, enter the shelf, and finally open the gripperto release the box. This task is a clearly segmented multi-step task, where the steps need to be executed in a particularorder. On the other hand, the execution of the task dependsvery little on the details of the physics: as long as the boxcan be picked up by the gripper, its weight or friction doesnot matter.

    The pushing to desired pose task involves moving and ro-tating a box of size 10 7 7cm to a desired area only bypushing it on the tabletop. In this task, the robot is not al-lowed to grasp the object. The box is initially positioned ina way that needs to be rotated by 90 to fit inside the desiredarea which is 3cm wider than the box in each direction. Therobot starts from an initial gripper position, moves the grip-per close to the box and pushes the box at specific points atits sides to rotate and move it. If necessary, the gripper needsto circle around the box to find the next contact point. Thistask does not have a clearly sequenced set of steps. It is un-clear how many times does the box needs to be pushed. Fur-thermore, the completion of the task depends on the physics:the weight of the box and the friction between the box andtable impacts the way the box moves in response to pushes.

    The demonstrations were collected from a single user, inthe course of multiple sessions. In each session, the user per-formed a series of demonstrations for each task. The qualityof demonstrations varied: in some of them, the user couldfinish the task only after several tries. For instance, some-

    times the grasp was unsuccessful, or the user dropped theobject in an incorrect position and had to pick it up again.After finishing a demonstration, the user was immediatelypresented with a new instance of the problem, with randomlygenerated initial conditions. All the experiments had beenrecorded in the trajectory representation format presentedabove, at a recording frequency of 33Hz. However, we foundthat the neural network controller can be trained more effi-ciently if the trajectories are sampled at a lower rate, with arate of 4Hz to giving the best results.

    To improve learning, we extended our training data by ex-ploiting both the properties of the individual tasks and tra-jectory recording technique. First, we noticed that in the pickand place task the user can put the object to any locationon the shelf. Thus we were able to generate new synthetictraining data by shifting the existing demonstration trajecto-ries parallel with the shelf. As the pushing to desired posetask requires a specific coordinate and pose to succeed, thisapproach is not possible for the second task.

    The second observation was that by recording the demon-stration at 33Hz but presenting the training trajectories atonly 4Hz, we have extra trajectory points. These trajectorypoints can be used to generate multiple independent trajecto-ries at a lower temporal resolution. The process of the trajec-tory generation by frequency reduction is shown in Figure 2.Table 1 describes the size of the final dataset.

    ...

    Original trajectory

    ...

    ...

    ...

    ...

    .

    .

    .

    8 generated trajectories

    Waypoints

    Figure 2: Creating multiple trajectories from a demonstra-tion recorded at a higher frequency.

    Task Pick andplace

    Push topose

    Raw demonstrations 650 1614Demonstrations after shift 3900 -Low frequency demonstrations 31,200 12,912Total no. of waypoints 645,198 369,477Avg. demonstration waypoints 20.68 28.61

    Table 1: The size of the datasets for the two studied tasks

    The neural network based robot controllerThe robot controller takes as input the pose of the objectsinvolved and the pose and open/close status of the grip-

  • per at time t and outputs a prediction of the pose and theopen/closed status of the gripper at time t+1. During train-ing, this prediction is used to generate the error signal. Dur-ing the deployment of the trained network, the predictionrepresents the desired pose of the end actuator which therobot needs to achieve through its inverse kinematics cal-culations. The controllers for the pick and place and thepush to desired pose tasks have the same network archi-tecture but were trained on the specific tasks.

    Our architecture uses an LSTM recurrent neural networkand relies on mixture density networks (MDNs) to predictthe probability density of the output. The error signal, in thiscase, is based on the negative logarithm likelihood of thenext target waypoint given the probability density impliedby the MDN.

    Let us now describe the intuitions that led to thesechoices. The solution to both manipulation tasks containa series of individual movements which need to be exe-cuted in a specific sequence. Although both tasks can besolved in several different ways, the individual movementsin them cannot be randomly exchanged. In order to success-fully solve the task, the robot needs to choose and committo a certain solution. While it is technically possible thatthis commitment will be encoded in the environment out-side the robot, we conjecture that a robot controller whichhas a memory to store these commitments will perform bet-ter. The requirement of a controller with a memory leads usto the choice of recurrent neural networks, in particular, oneof the most widely used models, the LSTM (Hochreiter andSchmidhuber 1997). We are using three LSTM layers with50 nodes each as shown in Figure 3.

    The second intuition applies to the choice of the outputlayer and error signal. Both tasks allow multiple solutions.For instance, for the push to pose task, the robot might needto move the box in a diagonal direction. This can be achievedby either (a) first pushing the shorter side of the box fol-lowed by a push on the longer side or (b) the other wayaround. However, by averaging these two solutions we reacha solution where the gripper would try to push the corner ofthe box, leading to an unpredictable result. This leads us toconjecture that a multi-modal error function would performbetter than the unimodal MSE. The approach we chose isbased on Mixture Density Networks (MDN) (Bishop 1994)which is to use the output of the network to predict the pa-rameters of a mixture distribution. The output of the network(which, in our case, is the next pose of the gripper), will be asample drawn from this distribution. Unlike the model withMSE cost which is deterministic, this approach can modelstochastic behaviors to be executed by the robot. The prob-ability density of the next waypoint can be modeled using alinear combination of Gaussian kernel functions

    p(y|x) =mi=1

    i(x)gi(y|x) (1)

    where i(x) is the mixing coefficient, gi(y|x) is a multi-variate Gaussian, and m is the number of kernels. Note thatboth the mixing coefficients and the Gaussian kernels areconditioned on the complete history of the inputs till current

    timestep x = {x1 . . . xt}. This is because the concatenationof the output of all layers which is used to estimate the mix-ing coefficients and Gaussian kernels is itself a function ofx. The Gaussian kernel is of the form

    g(y|x) = 1(2)

    c/2i(x)

    exp

    {y i(x)

    2

    2i(x)2

    }(2)

    where the vector i(x) is the center of ith kernel. We do notcalculate the full covariance matrices for each component,since this form of Gaussian mixture model is general enoughto approximate any density function (McLachlan and Bas-ford 1988).

    The parameters of the Gaussian kernels i(x), i(x) andmixing coefficients i(x) are represented by the layer M inFigure 3. To accomplish this, layer M needs to have one neu-ron for each parameter. Thus layer M will have a width of(c + 2) m, containing c m neurons for i(x), m neu-rons for i(x), and another m neurons for i(x). This layeris fully connected to the concatenation of layers H1, H2 andH3.

    To satisfy the constraintm

    i=1 i(x) = 1, the corre-sponding neurons are passed through a softmax function.The neurons corresponding to the variances i(x) are passedthrough an exponential function and the neurons corre-sponding to the means i(x) are used without any furtherchanges. Finally, we can define the error in terms of nega-tive logarithm likelihood

    EMDN = ln

    {mi=1

    i(x)gi(y|x)

    }(3)

    The network is unrolled for 50 time steps. All the pa-rameters are initialized uniformly between -0.08 to 0.08 fol-lowing the recommendation by (Sutskever, Vinyals, and Le2014). Stochastic gradient descent with mini-batches of size10 is used to train the network. RMSProp with initial learn-ing rate of 0.001 and decay of 0.99-0.999 (based on numberof examples) is used to divide the gradients by a runningaverage of their recent magnitude. In order to overcome theexploding gradients problem, the gradients are clipped in therange [-1, 1]. We use 80% of the data for training and keepthe remaining 20% for validation. We stop the training whenthe validation error does not change for 20 epochs.

    Transferring the controller to the physical robotThe last step of the process is to transfer the trained con-troller to a physical robot (a Rethink Robotics Baxter). Asthe controller provides only the next pose of the end effec-tor, the controller had been augmented with an inverse kine-matics module to calculate the trajectory in the Baxter robotarms joint space.

    Another challenge is that while in the virtual world wehad perfect knowledge of the pose of the effector and all theobjects in the environment, we needed to acquire this infor-mation through sensing. As our controller architecture onlyperforms robot arm control, in order to supplant the miss-ing vision component, we relied on a Microsoft Kinect sen-sor and objects annotated with markers to track their pose.One of the problems with this approach is that the robot

  • Training the LSTM-MDN network unrolled through time LSTM-MDN network performing the task in a closed loop

    et, qt

    Mixture density parameters

    LSTM

    LSTM

    LSTM

    Multivariate Mixture of Gaussians

    et+1

    draw a sample

    Inverse kinematics

    Joint angles

    Negative log likelihood cost

    et, qt

    Mixture density parameters

    LSTM

    LSTM

    LSTM

    et+1

    Negative log likelihood cost

    et+1, qt+1

    Mixture density parameters

    LSTM

    LSTM

    LSTM

    et+2

    Negative log likelihood cost

    et+49, qt+49

    Mixture density parameters

    LSTM

    LSTM

    LSTM

    et+50

    Dataset

    M

    C

    H1

    H2

    H3

    Softmax ExpSoftmax ExpSoftmax Exp

    Figure 3: The training and evaluation phase. During the training the LSTM network is unrolled for 50 time-steps. The gripperpose and status (open/close) et and the pose of relevant objects qt at time-step t is used as input and output of the network tocalculate and backpropagate the error to update the weights. During the evaluation phase, the mixture density parameters areused to form a mixture of Gaussians and draw a sample from it. The sample is used to control the robot arm.

    arm might occlude the view of the sensor. The Kinect sensorwas placed close to the table to reduce the chance of occlu-sion, however, occlusions may still occur if the robots arm isplaced between the object and the Kinect. Another challengeis the fact that the waypoints generated by the controller arerelatively far away, leading to a jerky motion. To smooth therobots movement we use interpolation in joint space to fillin the gap between the current waypoint and the previousone.

    Finally, we found that the trajectory described by the con-troller can not always be executed by the Baxter arm at thesame speed as in the virtual environment. Therefore, we usea dynamic execution rate to wait between execution of eachwaypoint. Concretely, the algorithm waits for 0.2sec andchecks if the difference between the current pose of the grip-per and the predicted one is below a certain threshold. If yes,it commands the robot to go to the next waypoint, otherwiseit waits in a loop until the end-effector reaches the desiredpose or timeout occurs which means that the end-effectorcannot reach that pose (either because the inverse kinematicfailed or a collision occurred).

    Experimental studyIn the following we describe a series of experiments and ob-servations justifying the three claims we made in the abstractof this paper.

    The benefits of LSTM and MDNOur network architecture uses LSTM layers and an MDN-based error signal. These choices need to be validated, asmany researchers reported success on similar tasks with asimpler network - without recurrent layers and using meansquared error (MSE) as an error signal. For instance, (Levineet al. 2016) use convolutional layers (for extracting poses

    from images) followed by feedforward layers to give a uni-modal prediction about the next waypoint in the trajectory.As our controller is not end-to-end (in real world experi-ments we are using an off-the-shelf solution for computervision), we can only compare our controller with the partof the controller from (Levine et al. 2016) that follows theconvolutional layers used for image processing. In order tobe able to perform a rough comparison, we implemented afeedforward network that closely matches that controller, butreplaces the convolutional layers with a direct position input.

    With this, we have four choices for the controller struc-ture:

    FeedForward-MSE: 3 layers of fully connected feedfor-ward network with 100 neurons in each layer and meansquared error as the cost function.

    LSTM-MSE: 3 layers of LSTM with 50 memory states ineach layer and mean squared error as the cost function.

    FeedForward-MDN: Mixture density network containing3 fully connected feedforward layers with 100 neurons ineach layer. The mixture contains 20 Gaussian kernels.

    LSTM-MDN: Mixture density network containing 3 lay-ers of LSTM with 50 memory states in each layer. The mix-ture contains 20 Gaussian kernels.

    The LSTM-MDN network is described in Figure 3, whilethe architectures of the other approaches are shown in Fig-ure 5. Each network had been separately trained for the pickand place and the push to desired pose respectively, in effectcreating 8 different controllers. The resulting controllers hadbeen tested in the virtual environment by requiring the robotto perform randomly generated tasks 20 times. If it can notcomplete the task in a limited time (1 minute for the first taskand 2 minutes for the second one), we count the try as a fail-ure and reset the position of the box. The numerical successrates are shown in the following table:

  • Figure 4: A sequence of images showing the autonomous execution of pick and place in simulation (first row), pick and placein real world (second row), pushing in simulation (third row), and pushing in real world (fourth row). The robot is controlledby a mixture density network with 3 layers of LSTM.

    Controller Pick and place Push to pose

    Feedfoward-MSE 0% 0%LSTM-MSE 85% 0%Feedforward-MDN 95% 15%LSTM-MDN 100% 95%

    The results allow us to derive several conclusions. Clearly,the Feedforward-MSE combination does not work for thisparticular set of problems and training data, failing to com-plete either task even once. Another conclusion is that thepick-and-place task is clearly the easier from the two, evenwhen tested in the virtual environment where physical mod-eling errors dont play a role. The presence of either LSTMor MSE in the network provided enough improvement in thecontroller to allow the finishing of the task in the majority ofsituations, while the presence of both lead to 100% successrate. On the other hand, the harder push to pose task requiresboth components to have a reasonable success rate of 95%.

    We conclude from these experiments that both addingLSTM layers, and using an MDN output provide significantbenefits individually to the robot controller, and that thesetwo techniques can be combined for more individual bene-fits.

    Evaluating the transfer to the physical robotTo verify the ability of the controller trained in the virtualworld to perform in the physical one, we subjected both thevirtual and the physical robots to the same tasks. The se-quence of images in the Figure 4 shows the controller actingautonomously for the pick and place and pushing to pose

    Feedforward-MDNLSTM-MSE

    Training the LSTM-MDN network unrolled through time

    LSTM-MDN network performing the task in a closed loop

    et, qt

    Mixture density parameters

    LSTM

    LSTM

    LSTM

    Multivariate Mixture of Gaussians

    et+1

    draw a sample

    Negative log likelihood cost

    et, qt

    Mixture density parameters

    LSTM

    LSTM

    LSTM

    et+1Negative log

    likelihood cost

    et+1, qt+1

    Mixture density parameters

    LSTM

    LSTM

    LSTM

    et+2

    Negative log likelihood cost

    et+49, qt+49

    Mixture density parameters

    LSTM

    LSTM

    LSTM

    et+50

    Dataset

    M

    C

    H1

    H2

    H3

    Softmax ExpSoftmax ExpSoftmax Exp

    et, qt

    Predicted et+1

    LSTM

    LSTM

    LSTM

    MSE cost

    Feedforward-MSE

    et, qt

    Predicted et+1

    FC

    FC

    FC

    MSE cost

    et, qt

    Mixture parameters

    FC

    FC

    FC

    Negative log likelihood cost

    et+1

    Softmax Exp

    et+1 et+1

    Figure 5: Alternative network architectures used in thecomparison study: Feedforward-MSE, LSTM-MSE andFeedforward-MDN

    tasks in the virtual and physical environments respectively.We have found that indeed, in most cases, the physical robotwas successful in executing both tasks. This is not becausethe virtual and physical worlds are highly similar. The sizeof the gripper of the Baxter robot is different from the onein the virtual world. The friction coefficients are very differ-ent, and the physics simulation in the virtual world is also oflimited accuracy. Even the size and shape of the box used in

  • the physical experiments is not an exact match of the ones inthe simulation, and the physical setup suffered from cameracalibration problems. Overall, the number of things that cango wrong, is much higher in the physical world.

    What we found remarkable is that, in fact, during the ex-periments many things went wrong or changed from thetraining data - however, the robot was often able to recoverfrom them. This shows how the model is robust in handlingdeviation from what it has seen. The network contains dif-ferent solutions for a case that can apply if others fail. For in-stance, in the pushing task, to rotate the box, it tries to touchthe corner of the box and push it. If the box did not movesince the gripper passed it without a touch, it tries again butthis time from a point closer to the center of the box. Thisgives the network some tolerance to slight variations in thesize of the box from the simulation to the real world.

    To compare the success ratio in the virtual and physi-cal world, we repeated the 20 experiments on the physi-cal world, using the LSTM-MDN controller (the only onethat had significant success rate on the virtual robot on bothtasks). The success rates in the virtual and physical worldsare compared as follows:

    Environment Pick and place Push to pose

    Virtual world 100% 95%Physical world 80% 60%

    The first conclusion we can draw from these values is thatthe approach successfully demonstrated the ability to trans-fer an unchanged controller trained in the virtual world to aphysical robot. As expected, the success rate was lower inthe physical world for both tasks. Some of the reasons be-hind the lower success rate is obvious: for instance, in thephysical world there is an inevitable noise in the positionof the objects and the end effector. Some of the noise is aconsequence of limited sensor accuracy (such as the calibra-tion of the Kinect sensor) and effector performance. Anothersource of inaccuracy is due to the way in which we acquiredthe positional information through a Kinect sensor: if duringthe manipulation the robot arm occluded the view of the ob-ject to the Kinect sensor, we temporarily lost the ability totrack the object.

    Another reason for the lower performance in the physicalworld is due to the differences in the size, shape, physicalattributes such as friction, etc. of the gripper and objects be-tween the simulation and real world. As we discussed whenintroducing the problems, the push-to-pose tasks is more de-pendent of the physics (such as the friction between the ob-ject and the table determines the way the object moves whenpushed). This creates a bigger difference between the vir-tual and the physical environment compared to the pick andplace task, where after a successful grasp the robot is essen-tially in control of the environment. Thus, the push to posetask shows a stronger decrease in success rate when movingto the physical world.

    Imperfect demonstrations teach the robot how toself-correctThe conventional wisdom in any learning from demonstra-tion system is that the better the demonstrations, the better

    the solution. For instance (James, Davison, and Johns 2017)use a programmed controller in the virtual world. Our exper-iments show that this is not always the case.

    As the objective of our project was to develop methodsthrough which disabled users can demonstrate their pre-ferred way to perform activities of daily living, in our experi-ments demonstrators were done by imperfect human demon-strators. The demonstrations contained errors such as failureto grasp, bumping the arm into shelves, dropping the ob-ject from the side of the shelf, overpushing, underpushing,and accidentally rotating the pushed object. In all cases, thedemonstrator tried to correct the error by retrying, or throughcompensatory actions. In some cases, the demonstrator didnot manage to complete the task. All these demonstrations,even the failed ones, were included in the training data.

    In the behavior of the trained robot, both in the virtualand the physical ones show that the robot learned to correctits mistakes. See, for instance, the experiments shown in thevideo1 from about 2:30. This is an obvious performance im-proving factor, especially in the physical world, where therobot inevitably makes more errors. In fact, arguably, therobot would rarely be able to complete a task without thisself-correcting behavior.

    Conclusions and future workIn this paper we have developed a technique through whicha robotic arm can be taught to perform certain manipulationtasks. We focused on two tasks that are frequently requiredof robots that assist disabled users in activities of daily liv-ing: pick and place and push to desired pose. As disabledusers can not generate large numbers of demonstrations onphysical robots, we designed an approach where the userdemonstrates the task in a virtual environment. These vir-tual demonstrations are used to teach a deep neural networkbased robot controller. Then, the controller is transferredto the physical robot. We found that the best performancewas obtained using a network with LSTM layers and a mix-ture density network based error signal. We also found thathaving imperfect demonstrations, where users occasionallymake mistakes but correct them, allows the controller to cor-rect the inevitable mistakes it makes when transferred to aphysical environment.

    Our team is working to improve these results along sev-eral directions. Multi-task learning might reduce the neces-sary number of demonstrations as many features are likelyshared between tasks. We are extending the tasks to morecomplex, multi-step tasks involving multiple objects. Weare working on acquiring demonstrations from multiple dis-abled users, who only demonstrate each task only two orthree times. We are also extending the controller to an end-to-end, vision-to-control model.Acknowledgments: We would like to gratefully thankSergey Levine, Navid Kardan, and Amirhossein Jabalamelifor their helpful comments and suggestions. This work hadbeen supported by the National Science Foundation underGrant Number IIS-1409823.

    1https://youtu.be/9vYlIG2ozaM

    https://youtu.be/9vYlIG2ozaM

  • References[Abbeel, Coates, and Ng 2010] Abbeel, P.; Coates, A.; andNg, A. Y. 2010. Autonomous helicopter aerobatics throughapprenticeship learning. International Journal of RoboticsResearch 29(13):16081639.

    [Argall et al. 2009] Argall, B. D.; Chernova, S.; Veloso, M.;and Browning, B. 2009. A survey of robot learningfrom demonstration. Robotics and autonomous systems57(5):469483.

    [Bishop 1994] Bishop, C. M. 1994. Mixture density net-works. Technical report, Aston University.

    [Bollini, Barry, and Rus 2011] Bollini, M.; Barry, J.; andRus, D. 2011. Bakebot: Baking cookies with the PR2. InProc. of the IROS PR2 workshop.

    [Calinon et al. 2009] Calinon, S.; Dhalluin, F.; Caldwell,D. G.; and Billard, A. 2009. Handling of multiple con-straints and motion alternatives in a robot programming bydemonstration framework. In Proc. of IEEE Intl Conferenceon Humanoid Robots, 582588.

    [Calinon et al. 2010] Calinon, S.; Dhalluin, F.; Sauser, E. L.;Caldwell, D. G.; and Billard, A. G. 2010. Learning and re-production of gestures by imitation. IEEE Robotics & Au-tomation Magazine 17(2):4454.

    [Calinon, Guenter, and Billard 2007] Calinon, S.; Guenter,F.; and Billard, A. 2007. On learning, representing, andgeneralizing a task in a humanoid robot. IEEE Transactionson Systems, Man, and Cybernetics 37(2):286298.

    [Christiano et al. 2016] Christiano, P.; Shah, Z.; Mordatch,I.; Schneider, J.; Blackwell, T.; Tobin, J.; Abbeel, P.; andZaremba, W. 2016. Transfer from simulation to realworld through learning deep inverse dynamics model. arXivpreprint arXiv:1610.03518.

    [Crick et al. 2011] Crick, C.; Osentoski, S.; Jay, G.; andJenkins, O. C. 2011. Human and robot perception in large-scale learning from demonstration. In Proc. of Intl confer-ence on Human-Robot Interaction, 339346. ACM.

    [Endres, Trinkle, and Burgard 2013] Endres, F.; Trinkle, J.;and Burgard, W. 2013. Learning the dynamics of doors forrobotic manipulation. In IEEE/RSJ International Confer-ence on Intelligent Robots and Systems (IROS), 35433549.

    [Fang, Bartels, and Beetz 2016] Fang, Z.; Bartels, G.; andBeetz, M. 2016. Learning models for constraint-based mo-tion parameterization from interactive physics-based simu-lation. In IEEE/RSJ International Conference on IntelligentRobots and Systems (IROS), 40054012.

    [Farchy et al. 2013] Farchy, A.; Barrett, S.; MacAlpine, P.;and Stone, P. 2013. Humanoid robots learning to walk faster:From the real world to simulation and back. In Proc. of theIntl Conf. on Autonomous Agents and Multi-Agent Systems(AAMAS-2013), 3946.

    [Forbes et al. 2014] Forbes, M.; Chung, M. J.-Y.; Cakmak,M.; and Rao, R. P. 2014. Robot programming by demon-stration with crowdsourced action fixes. In Proc. of SecondAAAI Conf. on Human Computation and Crowdsourcing.

    [Gupta et al. 2017] Gupta, A.; Devin, C.; Liu, Y.; Abbeel, P.;and Levine, S. 2017. Learning invariant feature spaces to

    transfer skills with reinforcement learning. arXiv preprintarXiv:1703.02949.

    [Hanna and Stone 2017] Hanna, J. P., and Stone, P. 2017.Grounded action transformation for robot learning in simu-lation. In Proc. of the 31st AAAI Conf. on Artificial Intelli-gence, 38343840.

    [Hochreiter and Schmidhuber 1997] Hochreiter, S., andSchmidhuber, J. 1997. Long short-term memory. Neuralcomputation 9(8):17351780.

    [James, Davison, and Johns 2017] James, S.; Davison, A. J.;and Johns, E. 2017. Transferring end-to-end visuomotorcontrol from simulation to real world for a multi-stage task.arXiv preprint arXiv:1707.02267.

    [Kehoe et al. 2013] Kehoe, B.; Matsukawa, A.; Candido, S.;Kuffner, J.; and Goldberg, K. 2013. Cloud-based robotgrasping with the Google object recognition engine. InIEEE International Conference on Robotics and Automation(ICRA), 42634270.

    [Kober, Oztop, and Peters 2011] Kober, J.; Oztop, E.; andPeters, J. 2011. Reinforcement learning to adjust robotmovements to new situations. In 2010 Robotics: Scienceand Systems Conference (RSS 2010), 3340. MIT Press.

    [Levine et al. 2016] Levine, S.; Finn, C.; Darrell, T.; andAbbeel, P. 2016. End-to-end training of deep visuomotorpolicies. Journal of Machine Learning Research 17(39):140.

    [Mayer et al. 2006] Mayer, H.; Gomez, F.; Wierstra, D.;Nagy, I.; Knoll, A.; and Schmidhuber, J. 2006. A systemfor robotic heart surgery that learns to tie knots using recur-rent neural networks. In IEEE/RSJ International Conferenceon Intelligent Robots and Systems (IROS), 543548.

    [McLachlan and Basford 1988] McLachlan, G. J., and Bas-ford, K. E. 1988. Mixture models. inference and applica-tions to clustering. Statistics: Textbooks and Monographs,New York: Dekker 1.

    [Miller et al. 2012] Miller, S.; Van Den Berg, J.; Fritz, M.;Darrell, T.; Goldberg, K.; and Abbeel, P. 2012. A geomet-ric approach to robotic laundry folding. The InternationalJournal of Robotics Research 31(2):249267.

    [Pastor et al. 2009] Pastor, P.; Hoffmann, H.; Asfour, T.; andSchaal, S. 2009. Learning and generalization of motor skillsby learning from demonstration. In IEEE International Con-ference on Robotics and Automation (ICRA), 763768.

    [Pinto and Gupta 2016] Pinto, L., and Gupta, A. 2016. Su-persizing self-supervision: Learning to grasp from 50k triesand 700 robot hours. In IEEE International Conference onRobotics and Automation (ICRA), 763768.

    [Ross, Gordon, and Bagnell 2011] Ross, S.; Gordon, G. J.;and Bagnell, D. 2011. A reduction of imitation learning andstructured prediction to no-regret online learning. In Proc.of Intl Conference on Artificial Intelligence and Statistics(AISTATS), 6.

    [Sung, Jin, and Saxena 2015] Sung, J.; Jin, S. H.; and Sax-ena, A. 2015. Robobarista: Object part-based transfer ofmanipulation trajectories from crowd-sourcing in 3d point-

  • clouds. In International Symposium on Robotics Research(ISRR).

    [Sutskever, Vinyals, and Le 2014] Sutskever, I.; Vinyals, O.;and Le, Q. V. 2014. Sequence to sequence learning with neu-ral networks. In Advances in neural information processingsystems (NIPS), 31043112.

    [Tobin et al. 2017] Tobin, J.; Fong, R.; Ray, A.; Schneider, J.;Zaremba, W.; and Abbeel, P. 2017. Domain randomizationfor transferring deep neural networks from simulation to thereal world. arXiv preprint arXiv:1703.06907.

    IntroductionRelated workOur approachCollecting demonstrations in a virtual environmentThe neural network based robot controllerTransferring the controller to the physical robot

    Experimental studyThe benefits of LSTM and MDNEvaluating the transfer to the physical robotImperfect demonstrations teach the robot how to self-correct

    Conclusions and future work

Recommended

View more >