ON-BOARD SIMULATOR FOR AUTONOMY ENHANCEMENT IN ROBOTIC SPACEMISSIONS

Domınguez, Raul1, Schwendner, Jakob1, and Kirchner, Frank2

1DFKI GmbH, Robotics Innovation Center, Robert-Hooke-Straße 1, 28359 Bremen, Germany2DFKI GmbH, Robotics Innovation Center and University of Bremen, Robotics Research Group, Robert-Hooke-Straße

1, 28359 Bremen, Germany

ABSTRACT

Space robotics missions are subjected to hard challengeson deployment time and the higher the level of autonomythe larger the magnitude of these difficulties. Its final be-havior is always dependent of the current states of theenvironment and of the system itself. In order to pro-vide the required level of reliability, accurate predictionsof these behaviors is mandatory for planning. This docu-ment proposes an approach based on the use of complexsimulations involving the models of the environment, ofthe robot and of the whole control software as a tool toprovide this predictions and improve the available plan-ners without modifying them internally. The approachpursues affecting the planner’s behavior through differ-ent external means (e.g. modification of it inputs, pa-rameter adaptation) so that it will produce more reliablesolutions. An application for improving the efficiency ofa naive planner aiming to solve a salesman travel missionscenario is presented. The connections between the dif-ferent locations to visit are assumed by the planner to beconnected but the final scenario and the navigation lim-itations of the robot do not allow the traverse of all thepaths. Through a simulation based on an aerial image thevalid paths are found and a plan is generated which is notleading to a failure state.

Key words: On-Board Simulation, Internal Simulator,Planning, Autonomy, Space Robotics, Validation, For-ward Models.

1. INTRODUCTION

Space exploration demands robotic systems that performreliably and autonomously complex missions in unstruc-tured environments (e.g. exploration of caves in Mars).As the demands on autonomy increase, the complexityof the software that control the robot does so, as well asthe physical complexity of the system. The combinationof these three factors: Behavioral, dynamical and envi-ronmental complexity pose a hard challenge when aim-ing for a reliable and intelligent system. A key factor for

achieving operational safety is the capacity of the sys-tem to predict with realism the outcomes of its actions. Itis here proposed an approach in this direction based onhighly complex physical simulations in which it is pur-sued to reproduce the whole system’s behaviors mirror-ing as much as possible its real execution.

The concept of the Internal Simulator is present in cog-nitive science theories to explain high level cognitive ac-tivity (thinking)[Bar99, Hes12] . These theories have in-spired robotics research where theories about imagina-tion functions are proved functional[MH09] and useful.On the other hand, Artificial Intelligence literature ex-ists in which learning from an internal model and notonly from reality enhances efficiency [SB12] and appli-cations of the same principle have been proposed in themanipulation field [Mel88]. Some examples already existwhere on-board simulations are used in complex roboticssimulations of space missions [RmGA+14]. Though,in the literature real missions applications with complexrobots are scarce. More applications are envisioned in ourgroup, mainly related to in operational safety and auton-omy. Two crucial aspects in space missions because com-munications might be impossible for long periods and er-ror costs are very high.

An experiment aimed to point out is the capacity of theInternal Simulator to improve available modules withoutthe need of human intervention is presented. In partic-ular, the experiment shows two features that an InternalSimulator can be useful for: efficiency enhancement anddesign limitations overcoming.

In the experiment, a planner of a simulated robot will en-hance its efficiency in solving a salesman problem withnon traversable paths. Initially, these non traversablepaths are assumed traversable by the planner and thusproposed as valid solutions that fail at execution. The In-ternal Simulator will use additional knowledge of the en-vironment (a 2D overhead image of the area) to generateand execute simulations of parts of the mission. Detect-ing in advance the failures and avoiding them (see [VZ06]for a similar experiment). Furthermore, the knowledgeobtained from the simulation will be integrated in a struc-tured representation that will be integrated along the orig-inal planner. Eventually, it will not be necessary to simu-

late in order to determine whether a path is traversable ornot.

Thanks to the advances in computer science and engi-neering it is now feasible to integrate complex simulatorsin the control software architectures of robotic systems.Meanwhile, the physics simulation engines and the simu-lation models are increasingly efficient and realistic. Forinstance, models for terramechanics simulation is an ac-tive research area in the field of space robotics. In thiswork, it is aimed to study the applications that an on-board simulator can have in space missions performed byrobotic systems. Reliability, efficiency and autonomy en-hancement are some of the envisioned uses.

In Section 2 the ideas on how to integrate the tool in therobotic controller are explained in detail. In Section 3an illustrative experiment is presented in which a roboticsystem enhances its chances to success on a navigationmission thanks to the Internal Simulator. Finally Section4 summarizes the most important ideas and explains thenext research objectives.

2. PROPOSED APPROACH

For introducing the concepts a first brief look on the cog-nitive paradigm in which the Internal Simulator is intro-duced from a functional point of view is presented. Thenin the Second part of this section a more close to roboticsengineering vision of the approach is provided.

2.1. Cognition and the Internal Simulator

The Internal Simulator is a high level tool, which can per-form different tasks. In general, it brings robustness andsafeness to the system but it can also provide a tool forlearning new behaviors. Following the Levels of Behav-ior model of cognition [KRS+12] (Figure 1), the InternalSimulator takes the highest position, here the challengeis to detect failures and limitations in the plans and themodels, explain them and apply this knowledge to im-prove the system.

The cognitive hypothesis is that some intelligent behav-ior is performed automatically, when for instance the taskis repetitive or known. But to achieve some tasks an ex-pansion of this automatic thinking is often required andthere is where the Internal Simulator as highest cogni-tive feature is used. Its task is to simulate acting, conse-quences and furthermore learn to predict and adapt withthis knowledge the automatic thinking.

This idea is being brought to the field of space robotics.The planners and models that the system uses for per-forming a task must reduce the complexity of the spaceof search (i.e. the environment model) to a computablyfeasible one. Furthermore, they may not account withthe whole interaction of other components (e.g. reactive

Internal Simulator

Reward

Reaction

Decision

Perceptors Actuators

Model PolicyReward

Figure 1: The Levels of Behavior Model is chosen asthe architectural paradigm upon which the modules of therobotic system are presented. In this figure, the area out-side the circle represents the environment.

behaviors). Thus, the space for possible failures in ex-ecution time becomes large. On the other hand, simu-lations can represent the reality and the robot itself withits all complete behavior accurately but its computationalcost and the high dimensionality of the space of searchmakes its use inefficient for planning. The Internal Sim-ulator that is envisioned pursues to get the best from bothelements by validating the plans and performing only ex-pensive simulations in the cases where the plan success isnot guaranteed.

2.2. Modular Robotics and the Internal Simulator

The Internal Simulator is therefore closely related to thePlanners, the Models and finally to the Environment rep-resentation. This last component includes more informa-tion than those present in the Models about the environ-ment and the system itself but it would be impossible toaccount with it all for real time planning in general. Onthe other hand, when a limitation on the system is de-tected some solution might be possible if the informationincluded in this environment were taken into account.

An Internal Simulator can affect the behavior of a plannerin several ways. Here we present an example in which bychanging the order of the goals, it is achieved the genera-tion of valid solutions. Other manners in which an Inter-nal Simulator could affect the planner is by modifying itspolicies or by introducing non existing elements in the en-vironment representation so that the generated models forplanning encode this artificial information (e.g. [CM09]).

For an Internal Simulator to provide useful results fourmain requirements are identified:

• Generative: Capability to generate, from the exist-

ing information about the environment a simulationenvironment.

• Executive: Capability to execute, based on the par-ticular constraints of the mission to be validated, thenecessary simulations.

• Adaptive: Capability to alter the existing softwareso that the information from the simulation can beintegrated in the execution loop.

• Supervisory: Capability to supervise the executionof the mission either in real world and in simulationto enable the detection of failures and successes,

Given these requirements the Internal Simulator providesthe tools to detect the regions in the space of possiblesolutions generated by the running components which ei-ther for limitations on the software side or hardware lim-itations will not produce the desired results.

• Software that attempts solutions which can not bephysically executed.

• Software that attempts solutions which are not de-sired by the designer.

• Software and hardware that generate an undesiredbehavior.

• Hardware features that are not exploited by the soft-ware and that are not desired.

• Hardware features that are not exploited by the soft-ware and that are desirable.

3. EXPERIMENTS

The experimental results here presented were imple-mented using the Rock, the Robot Construction Kit[Roc]and the simulation environment Mars [Mar]. Rock is aframework for programming and controlling robotic sys-tems. It is based on the principle of having modular com-ponents (i.e. tasks), each with an specific function thatare connected through ports. Mars is an application forsimulating robots and the environment where they are in-mersed, it is integrated in the Rock framework but canalso be used independently. Mars internally uses OpenDynamics Engine [ODE] for computing the physical in-teraction between the components of the simulation.

To exemplify the potential capabilities of the approach, acase of the classical salesman problem is presented. Therobot has the mission to visit certain positions of an un-explored area, of which aerial imagery is available.

Figure 2: The experiment was performed with two robotsof different morphologies. In the picture the real and itscorrespondent simulation are shown.

3.1. Robotic Systems

The experiments were performed with two simulatedrobots. Both robots are equipped with a laser sensor forenvironment perception. Crex robot is a 6 legs robots de-signed for negotiating complicated surfaces such as cavesand crater surfaces. The Asguard robot has a combina-tion of wheels and legs, that are more energy efficientthan legs and allows the traverse on more unstructuredsurfaces than wheels (e.g. rocks surface).

The components that enable the robot to perform thenavigation task are depicted in the diagram in Figure 3.The localization is performed in this experiment basedon odometry. There is no slam algorithm or global modelof the environment. The perception of the environmentsis based on the data provided by a tilting laser sensor lo-cated at the front part of each robots. The sensor pro-duces a pointcloud from which a local traversability mapis built. This map is then converted to a grid upon whichthe local planner plans a trajectory avoiding the obsta-cles. The local trajectory pursues to follow the globalorientation which is a straight vector between the currentposition and the target position.

The environment representation module uses data fromdifferent sensors such as laser sensors and inertial mea-surement units to generate a model of the environment.This module also incorporates the geo-referenced aerialimage. The environment representation is used by differ-ent tasks that generate models in which the planning al-gorithms search for action chains in order to arrive to thegoal state. Furthermore, the Environment Representationmodule includes external information about the environ-ment where the mission will be performed. For this case

Local PlannerLocal Map

ActuatorsTilt Scan

IMU

Odometry

Laser Scanner Leg / Wheel JointsDynamixel

EnvironmentRepresentation

Global Map Global Planner

Goal Positions

Internal Simulator

Figure 3: Modules that control the navigation of therobots. The Internal Simulator uses the EnvironmentRepresentation to generate the simulation environment.The Internal Simulator overrides the goals input to theglobal planner. In this way, only validated outputs will begenerated.

a post processed aerial image is available from which asimulation can be generated (see Figure 4).

(a) Possible post processed aerialimage of the deployment sce-nario. The stars represent the po-sitions to visit. The circled one isthe start position. The gray areascorrespond to heightmaps.

(b) Automatically generatedsimulation scene. In Mars[? ]this 3D object is used as terrainobject.

Figure 4: From the available information about the envi-ronment, the Internal Simulator should generate a realis-tic as possible simulation.

3.2. Problem Description

The goals are provided to the global planner which deter-mines using a global model the global path to follow. Thelimitation that is proposed here to overcome with the In-ternal Simulator is one of the global model. This task as-sumes that all paths are traversable but reality is different.The system can get stuck in certain regions due to char-acteristics of the environment not accounted in the designof the model. In this particularly simplified case the mostrepeated failures is that the robot arrives to a local min-imum (dead end in the straight trajectory between twopoints). Nevertheless other cases of failure have takenplace due to a not enough fine parametrization of some ofall the tasks (Figure 5).

(a) ASGUARD arrives to a Dead End Corridor. The Figure in the lowright part shows in green where the robot can plan to navigate. Theplanner cannot find any trajectory towards the goal (yellow dot).

(b) CREX falls when walking down from a too high obstacle.

Figure 5: Two cases of failure. The first was caused bylimited environment model (5a). The second one due tobad parametrization of the navigation components (5b).

3.3. Proposed Solution

The solution using the Internal Simulator is as follows.First, all the potentially valid connections (paths) are sim-ulated in both directions. This has to be done in bothdirections because the failure might only occur when go-ing from state i to j but not in the transition from j toi. The Internal Simulator then executes a simulation ofthe navigation of each path. The simulation is evaluatedand a cost for each path ci,j is assigned based on a costfunction Φ

ci,j = Φ(Σ(i, j)) | i 6= j and i, j ∈ S (1)

Where S is the set of states or in this case the subgoals,ci,j the cost of going from state i to j. Σ(i, j) is the sim-ulation of the execution of the transition from state i toj and Φ, the function that evaluates the cost of the sim-ulated execution. For simplicity, the cost can be set toinfinity for the plans which failed in simulation and to 1for those which succeeded.

Φ =

{1 if success∞ otherwise (2)

Once all the simulations have been performed and eval-uated, a Validation Graph G(S,C) is built. Where thedirected edges are denoted by

C = {ci,j} ∀i, j | i 6= j, i, j ∈ S and ci,j 6=∞ (3)

and the vertices correspond with S the states.

This graph can be used to find a validated solution usinga graph search algorithm. The solution is then executed.

In the generated Validation Graph, the nodes representthe different initial and final points of a path (i.e. initialand final states of the proposed plan). Each directed edgeencodes the cost of executing that particular path. The re-sulting graphs after removing the edges with infinite costare directed graphs. In particular the graphs may containparts connected only with only one edge (i.e. either leav-ing or arriving to them is impossible) and unconnectedcomponents (i.e. unaccessible from other components)

In Figure 6 the graphs resulting from running the exper-iments with both robots are presented. In both graphsthere is an unconnected node (16, 9). This is because noexecution was able to neither reach the subgoal from an-other subgoal nor get from that node to any other. Assum-ing that the start position is (7, 4.5), Asguard for robot itis possible to visit all the subgoals with exception of theone that correspond to the unconnected node. In the caseof Crex, two more states (i.e. subgoals) are unreachablefrom the initial position. Those states are only connectedthrough an incoming path to the symmetric componentwhere the initial state is.

A comparison of the system with and without the Inter-nal Simulator would show how, in this experiment therobot would enter a failure state. One of the goals (16,-9) is neither reachable from any of the other goal nor theinitial position. Thus, a global planner which assumesthat every two goals are reachable will fail in this specificmission (independently of the policy of search). Further-more, the failure could be critical because some of thepaths proposed will likely make the robot fall. The solu-tion using the Internal Simulator will neither reach to allthe goals, but those paths which were found to generatea failure in simulation won’t be executed increasing thechances of reaching more goals.

4. CONCLUSIONS

In space robotics robustness while performing au-tonomous missions is a crucial and hard problem. Thesystems are deployed in an environment where they are

-16, 15

2.5, 15 7, 4.5

-15, -7

-2, -9

16, 15

16, -9

12

3

4 5

6

78

(a) Validated paths for thewheeled robot ASGUARD.

-16, 15

2.5, 15 7, 4.5

-15, -7

-2, -9

16, 15

16, -9

12

3 4

5

(b) Validated Paths for the leggedrobot CREX.

Figure 6: The Internal Simulator executes the plan gen-erated between every two subgoals in both directions tofind out with paths are valid. The results are stored in adirected graph. The numbers on the edges indicate vali-dated solutions for the mission.

expected to work for long periods of time without main-tenance. Ideally, the system would adapt its software tothe current physical reality and overcome previously un-known design limitations. Autonomous adaptations, un-known environments and changes in the physical robotare potential sources of failures at execution time in spacemissions. Validation of the system behavior in an On-board simulation before execution is proposed to mini-mize these problems.

The Internal Simulator will operate in close relation withthe planners, the environment representation and robotrepresentation. Using this information to pursue to pre-dict where failures might occur. The challenges for aneffective Internal Simulator can be summarized into gen-erative (i.e. realistic simulations production), evaluative(i.e. performance supervision), executive (i.e. computa-tional and time constraints) and adaptive (i.e. integratingknowledge from the simulation). The Internal Simula-tor is conceptually general enough to work with differentplanners and in different time ranges ( Prediction Hori-zons in [RmGA+14]).

A simulation of an exploration mission is presented asconceptual experiment to show the potential of the Inter-nal Simulator. The robot must reach a set of goal pointsgiven its 2D position in a non visited area. The systemhas a serious limitation when attempting to execute themission: no global map is built. The global map onlycontains the positions of the goals and the current posi-tion of the robot and only the robot position updates onexecution. This map is used to generate straight trajecto-ries towards the subgoals under the assumption that theywill be traversable. This assumption is often wrong andthe robot fails in execution time.

Assuming available a post processed aerial image of the

surface, it is presented how an Internal Simulator wouldimprove the behavior of the system. In the proposed solu-tion, the final path through as many as possible subgoalshas to be obtained from a directed graph, which encodeswhich sub plans have been validated by the Internal Sim-ulator (Validation Graph).

This structure represents known states for the InternalSimulator of the planner and whether its transition is ac-tually prune to fail. The structure is then analyzed, in thisparticular case to go generate a plan that passes throughas many nodes as possible as efficiently as possible. Thissame structure could also be used for different tasks (e.g.get to certain position as fast as possible). Furthermore,it is appropriate to be used with any state based planner

The experiment aims to emphasize that an Internal Sim-ulator to improve the behavior of an state based plan-ner without modifying it by exploiting information of theenvironment representation not accounted, analyzing theexecutive features of the whole system (e.g. detect thatsome transitions might fail due to inaccurate parametriza-tion of subtasks)

In future works it is intended to improve the generativecapabilities of the Internal Simulator so that realistic sim-ulations are generated automatically from the environ-mental knowledge available and using the experiences ofthe robots. The Internal Simulator will be tested with dif-ferent real robots in scenarios that resemble challenges ofextraterrestrial missions (e.g. lava tubes). It is also envi-sioned the use with other types of planners (e.g. manip-ulation) or even combinations of various (e.g. navigateand grasp).

It is envisioned a future were robots will be able to testthemselves and discover new strategies and behaviorswhile reassuring the success of the missions.

ACKNOWLEDGMENTS

The Entern project is funded by the Space Agency of theGerman Aerospace Center with federal funds of the Fed-eral Ministry of Economics and Technology (BMWi) inaccordance with the parliamentary resolution of the Ger-man Parliament, grant no. 50RA1407.

REFERENCES

[Bar99] Lawrence W Barsalou. Perceptual symbolsystems. Behavioral and Brain Sciences,22(4):577–609; discussion 610–660, Au-gust 1999.

[CM09] Antonio Chella and Irene Macaluso. Theperception loop in CiceRobot, a museumguide robot. Neurocomputing, 72(4-6):760–766, January 2009.

[Hes12] Germund Hesslow. The current status ofthe simulation theory of cognition. Brainresearch, 1428:71–9, January 2012.

[KRS+12] Tim Kohler, Christian Rauch, MartinSchroer, Elmar Berghofer, and FrankKirchner. Concept of a Biologically In-spired Robust Behaviour Control System.In Proceedings of International Confer-ence on Intelligent Robotics and Applica-tions 2012, volume 7507, pages 486–495.Springer Berlin Heidelberg, 2012.

[Mar] Mars. An Open-Source, flexible 3D phys-ical simulation framework. http://rock-simulation.github.io/mars/, Last visited:2015-30-04.

[Mel88] Barlett W. Mel. MURPHY: A robot thatlearns by doing. Neural information pro-cessing systems, pages 544–553, 1988.

[MH09] Hugo Gravato Marques and Owen Hol-land. Architectures for functional imagina-tion. Neurocomputing, 72(4-6):743–759,January 2009.

[ODE] ODE. Open Dynamics Engine.http://www.ode.org/, Last visited: 2015-30-04.

[RmGA+14] Jurgen Roß mann, Eric Guiffo Kaigom, Li-nus Atorf, Malte Rast, Georgij Grinshpun,and Christian Schlette. Mental Modelsfor Intelligent Systems: eRobotics EnablesNew Approaches to Simulation-Based AI.KI - Kunstliche Intelligenz, 28(2):101–110,March 2014.

[Roc] Rock. The Robot Contruction Kit.http://rock-robotics.org, Last visited:2015-30-04.

[SB12] Richard S Sutton and Andrew G Barto.Reinforcement Learning : An Introduction(Second Edition). 2012. Chapter 8: Plan-ning and Learning with Tabular Methods.

[VZ06] Richard Vaughan and Mauricio Zuluaga.Use your illusion: Sensorimotor self-simulation allows complex agents to planwith incomplete self-knowledge. From An-imals to Animats 9, 4095:298–309, 2006.