ai and robotics: lessons learned from the race project · ai and robotics: lessons learned from the...

AI and Robotics: LessonsLearned from the RACEProject

Federico Pecora?

[email protected]

Center for Applied AutonomousSensor Systems (AASS)

Örebro University, Sweden © Joshua Ellingson

? Contributors:

A. Saffiotti, T. Cohn, K. Dubba, J. Hertzberg,L. Hotz, Š. Konecný, J. Lehamnn,L. Lopes, M. Mansouri, B. Neumann,M. de Oliveira, S. Rockel, S. Stock, L. Zhang

[email protected]

How is AI Useful for Robots? Knowledge Representation Learning from Few Examples Planning & Execution Conclusions

AI + Robotics — Why?

Making robots intelligent has beena common goal of AI andRobotics since Shakey

“The general purpose robot is amirage”[Debate on failure of AI, BBC“Controversy” series, 1973]

But what does AI contribute toRobotics? And vice-versa?

© 2013 F. Pecora / Örebro University – aass.oru.se 2 / 29

aass.oru.se


Model-Centered Approach

“Robotics is closely related to AI [as] intelligence isrequired for manipulation, navigation, localization,mapping, motion planning”[Wikipedia page on Artificial Intelligence]

Model-centered approach is a major contribution of AI toRoboticsModels capture environment, capabilities, tasks

“competent behavior” results from reasoningmodels have formal propertiesmodels can be changed to suit different environments,physical capabilities, tasks

Which models are useful for autonomous robots?


aass.oru.se


Scenario: a Busy Restaurant

knife

fork

dish

time

Spatial models: cappuccinosshould be served “in front of”guests

Resource models: cappuccinoswon’t fit if other dishes arepresentAction models: dishes can becleared to make space forcappuccinosOntological models: dishes andcutlery should be cleared, saltand pepper should stayTemporal models: cappuccinosshould be served before they getcold


aass.oru.se



knife

fork

dish

time

Spatial models: cappuccinosshould be served “in front of”guestsResource models: cappuccinoswon’t fit if other dishes arepresent

Action models: dishes can becleared to make space forcappuccinosOntological models: dishes andcutlery should be cleared, saltand pepper should stayTemporal models: cappuccinosshould be served before they getcold


aass.oru.se



knife

fork

dish

time

Spatial models: cappuccinosshould be served “in front of”guestsResource models: cappuccinoswon’t fit if other dishes arepresentAction models: dishes can becleared to make space forcappuccinos

Ontological models: dishes andcutlery should be cleared, saltand pepper should stayTemporal models: cappuccinosshould be served before they getcold


aass.oru.se



knife

fork

dish

time

Spatial models: cappuccinosshould be served “in front of”guestsResource models: cappuccinoswon’t fit if other dishes arepresentAction models: dishes can becleared to make space forcappuccinosOntological models: dishes andcutlery should be cleared, saltand pepper should stay

Temporal models: cappuccinosshould be served before they getcold


aass.oru.se



knife

fork

dish

time

Spatial models: cappuccinosshould be served “in front of”guestsResource models: cappuccinoswon’t fit if other dishes arepresentAction models: dishes can becleared to make space forcappuccinosOntological models: dishes andcutlery should be cleared, saltand pepper should stayTemporal models: cappuccinosshould be served before they getcold


aass.oru.se



knife

fork

dish

time

Action+Resource models: traycapacity determines number oftrips to clear a table

Ontological+Spatial models: typeof meal affects spatial layout. . .


aass.oru.se



knife

fork

dish

time

Action+Resource models: traycapacity determines number oftrips to clear a tableOntological+Spatial models: typeof meal affects spatial layout. . .


aass.oru.se


Key Challenges

? What modeling languages areappropriate?

How do we jointly reason aboutdifferent models?How do we learn relevant modelsfrom few examples?How to we obtain symbolic models(appropriate for reasoning) fromexperience?


aass.oru.se


Key Challenges

? What modeling languages areappropriate?How do we jointly reason aboutdifferent models?

How do we learn relevant modelsfrom few examples?How to we obtain symbolic models(appropriate for reasoning) fromexperience?


aass.oru.se


Key Challenges

? What modeling languages areappropriate?How do we jointly reason aboutdifferent models?How do we learn relevant modelsfrom few examples?

How to we obtain symbolic models(appropriate for reasoning) fromexperience?


aass.oru.se


Key Challenges

? What modeling languages areappropriate?How do we jointly reason aboutdifferent models?How do we learn relevant modelsfrom few examples?How to we obtain symbolic models(appropriate for reasoning) fromexperience?


aass.oru.se


Robustness by Autonomous Competence Enhancement

Goal of RACE

Enable robots to operate more robustly by exploiting experiences

Methodology in RACE

Experiences are transformed into sub-symbolic andsymbolic models

Hand-coded and learned models are refined over time

Hand-coded and learned models are used for reasoning


aass.oru.se


RACE Partners

University of Hamburg (DE)Centre of Intelligent Systems and Robotics

University of Leeds (UK)School of Computing

Örebro University (SE)Center for Applied Autonomous SensorSystems

University of Osnabrück (DE)Institite for Computer Science

University of Aveiro (PT)Inst. for Electronics and TelematicsEngineering

HITeC (DE)Hamburger Informatik Technologie-Center,eV

EU-FP7 STREP, Objective 2.1: CognitiveSystems and Robotics, 2011–2015Funding: 2,997,298 e, 3 years


aass.oru.se


The Metaphor of Description Logics[S. Rockel et al., “An Ontology-based Multi-level Robot Architecture for Learning from Experiences”]

T-Box: the world’s rules — “general knowledge”

A-Box: individuals and their properties — “currentknowledge”


aass.oru.se


“Mental State” as a Collection of Fluents

end−boundsstart−bounds

fluent

is−a

Concept

has−ahas−a

[ls , us ] [le , ue ]

!FluentClass_Instance: [On, on1]Starttime: [10, 10]FinishTime: [11, ?]Properties:

[hasPhysicalEntity, PhysicalEntity, mug1][hasArea, Area, placingAreaWestRightTable1]


aass.oru.se




fluent

is−a

Concept

has−ahas−a

[ls , us ] [le , ue ]

!FluentClass_Instance: [PlacingAreaWestRight,

placingAreaWestRightTable1]StartTime: [0, 0]FinishTime: [INF, INF]Properties:

[hasManipulationArea, ManipulationArea,manipulationAreaSouthTable1]


aass.oru.se




fluent

is−a

Concept

has−ahas−a

[ls , us ] [le , ue ]

!FluentClass_Instance: [Mug, mug1]StartTime: [0, 0]FinishTime: [INF, INF]Properties:

[hasBoundingBox, BoundingBox, boundingBoxMug1][hasPose, Pose, poseMug1]


aass.oru.se


Fluents as the Mental State of the Robot


aass.oru.se


Leveraging Ontological Models

Expressing the robot’s knowledge in OWL-2-DL providessome useful services

consistency of robot’s knowledgescene interpretationlearning from few examples


aass.oru.se


Generalizing from Positive Examples

Autonomous concept creation for ServeCoffee activity from a singleexample

guest1

mug1

table1 table2

robot

counter1

“Move to counter1, grasp mug1, move to south of table1, place mug1 atplacement area west — this is a ServeCoffee”


aass.oru.se



Generalization of ServeCoffee concept to cover current and previousexamples

mug1

guest2 table2

robot

counter1

table1

“Move to counter1, grasp mug2, move to north of table1, place mug2 atplacement area east — this is also a ServeCoffee”


aass.oru.se



Further generalization of ServeCoffee concept and application to a newsituation

mug1

table1

guest3

robot

table2

counter1

“Do a ServeCoffee to guest3 at table2”


aass.oru.se


Conceptualization, Generalization and Unseen States[L. Hotz, B. Neumann, S. von Riegen, N. Worch. “Using Ontology-based Experiences for Supporting Robots Tasks”]

1 Conceptualize ServeCoffeeA

2 Conceptualize ServeCoffeeB

3 MLCS(ServeCoffeeA ,ServeCoffeeB )⇒ ServeCoffeeAB

4 MCSh (episode, initialStateC ) =ServeCoffeeAB

5 MLCS(ServeCoffeeAB ,initialStateC )⇒ ServeCoffeeABC


aass.oru.se


Using the Most Appropriate KR&R Approach

Not all reasoning tasks fall within the scope of OWL-2-DLnot all types of knowledge are easy to represent inOWL-2-DLnot all reasoning tasks are amenable to DL inference

Reified constraints

Class: SceneLayoutEquivalentTo: Occurrence

AND (hasPassiveObject SOME PassiveObject)AND (hasLayoutConstraint EXACTLY 1 LayoutConstraint)

Class: TableLayoutEquivalentTo: SceneLayout ...


aass.oru.se


Spatial Knowledge: Qualitative & Metric Constraints

A BB

A

B

A

A B

A

B

A

B

A B A B

y

B−x B+x A−x A+

x

B−y

B+y

B

A

A+y

A−y

x

N

W

A PO B A TPP−1 B A NTPP−1 BA EQ B

A DC B A EC B A TPP B A NTPP B

NW NE

E

SESSW

〈Before, Before〉

DC

Representation: Region Connection Calculus, CardinalDirection Calculus, (Augmented) Rectangle Algebra, ARA+

Reasoning: qualitative and metric constraint propagation


aass.oru.se


Temporal Knowledge: Qualitative & Metric Constraints

Representation: (Augmented) Allen’s Interval Algebra,(disjunctive) temporal constraints

Reasoning: search (TCSP) and 3-consistency (STP)


aass.oru.se


Resource Knowledge: Multi-Capacity Reusable Resources

timetime

reso

urc

e usa

ge

max capacity

Representation: can model constraints like max weight thatthe robot can carry

Reasoning: precedence constraint posting approach


aass.oru.se


Building Blocks for Solving “Pixels to Predicates” ExistLearning/recognizing spatio-temporal

relations from video

Allen’s temporalrelations

Spatial relations

Objects

before meets

P

meets

PO DR

[M. Sridhar, A.G. Cohn, D.C. Hogg, 2011. “Benchmarkingqualitative spatial calculi for video activity analysis”]

Learning object categories (objectrecognition, tracking, anchoring)

[A. Chauhan, Z. Lu, L. Seabra Lopes, 2011.“Manhattan-Pyramid Distance: A Solution to an Anomaly

in Pyramid Matching by Minimization”]


aass.oru.se


Reasoning with Diverse Knowledge[M. Mansouri, F. Pecora, 2013. “A Representation for Spatial Reasoning in Robotic Planning”]

Goal: place cup1 on table2 so that the table is well set

This requires hybrid reasoning

causal reasoning (planning)

temporal reasoning

spatial reasoning

ontological reasoning

resource reasoning


aass.oru.se


Spatio-Temporal-Resource Planning[M. Mansouri, F. Pecora, 2014. “Planning with Space, Time and Resources for Robots”]


aass.oru.se


Hybrid Search Space

...

...

...

time

1: pick(fork1,table1)

1: place(fork1,tray1)

2: pick(knife3,table1)

3: place(knife3,table1)

4: pick(fork1,tray1)

5: place(fork1,table1)

max capacity

time

nu

mb

ero

f ar

ms

Sol

utio

nsto

caus

alsu

b-pr

oble

m

Solut

ions to

resou

rcesu

b-pr

oblem

Solutions to temporal sub-problem


aass.oru.se


Hybrid Search SpaceS

olut

ions

toca

usal

sub-

prob

lem

Solut

ions to

resou

rcesu

b-pr

oblem



aass.oru.se


Hybrid Search SpaceS

olut

ions

toca

usal

sub-

prob

lem


Solut

ions to

resou

rcesu

b-pr

oblem

How toexplorethis searchspace?


aass.oru.se


Towards a General Hybrid Reasoning Schema

Meta−CSP

Meta−values

Meta−variables

Ground−CSP

Ground−values

Ground−variables

Meta−constraints

Ground−constraintsd ⊆ D

δ ⊆ LD

(expressed in LD )

High-level decisions High-level requirements

Low-level decisions Low-level requirements

Sources and binaries onMaven Central and Google Code

metacsp.org


metacsp.org

aass.oru.se


Semantic Planning & Execution Monitoring

Can we use temporal/spatial/ontological models duringexecution?

Task: bring a mug from counter1 to table1

Initial condition (known to the planner): On(mug1, counter1)Plan (HTN):

1 go to counter12 grasp mug13 bring mug1 to table1

What could go wrong?


aass.oru.se


Temporal/Ontological Execution Models[Š. Konecný et al., 2014. “Planning Domain & Execution Semantics: a Way Towards Robust Execution?”]

"object dropped

while grasping"

time

"object not grasped"

timetime

time

Observe(table1) Pick(mug1,table1)

holding(mug1)on(mug1,table1)

Move(counter,table1)

at(counter) at(table1)

holding(mug1)

Pick(mug1,table1)

Failure (contains)

Failure (before)

Pick(mug1,table1)

effect OWLprecondition

Pick(mug1,table1)

Success (meets ∨ overlaps)

{before}

{meets, overlaps}{meets, overlaps}

{meets, overlaps}

{meets, overlaps} {meets, overlaps, before}

holding(mug1)

Execution modelis-a

DrinkingVessel

Pick(cup22,table1)

cup22

Success (mug1 ↔ cup22)

holding(mug1)

holding(cup22)


aass.oru.se


Simulation[S. Rockel et al., 2014. “An Hyperreality Imagination based Reasoning and Evaluation System”]

Simulation as a tool for failure detection . . .

. . . and for use during planning?


aass.oru.se


What AI Can Do for Robots. . .

AI has studied how to reason within specific KR schemasfor decades

planning (STRIPS/PDDL, HTN, CBP, . . . )temporal reasoning (Allen’s Algebra, STP, TCSP, DTP, . . . )spatial reasoning (RCC, CDC, RA, ARA, . . . )scheduling (reusable resources, consumable resources,timetabling, . . . )logics (FOL, DL, . . . )

Reasoning can lead to competent behavior

Models can be adapted to changing conditions,requirements, tasks, goals

Models can facilitate robot programming


aass.oru.se


. . . and What Robots Can Do for AI

Each KR schema makes different restricitve assumptionsrobots will erode these restrictive assumptions

Few and naïve efforts in integrating different AI problemsolving techniques

robots will push integration of different branches of AI

AI researchers take perception for grantedrobots will require us to solve the pixels to predicatesquestion

AI problem solving is increasingly self-referentialrobots will finally provide a meaningful benchmark for AItechniques


aass.oru.se


ああありりりがががとととううう !


aass.oru.se

ai and robotics: lessons learned from the race project · ai and robotics: lessons learned from the...

Documents