knowledge processing for autonomous robots
TRANSCRIPT
Knowledge Processing for AutonomousRobots
Moritz Tenorth
Intelligent Autonomous Systems GroupTechnische Universitat Munchen
November 5, 2010
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
Outline
1. Why Knowledge Processing for Autonomous Robots?
2. Introduction to Prolog
3. Overview of KnowRob
4. Asking Queries
5. Extending KnowRob
6. Integrating Perception Components
7. Inferring Control Decisions with KnowRob
8. Outlook
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
Outline
1. Why Knowledge Processing for Autonomous Robots?
2. Introduction to Prolog
3. Overview of KnowRob
4. Asking Queries
5. Extending KnowRob
6. Integrating Perception Components
7. Inferring Control Decisions with KnowRob
8. Outlook
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
Knowing what you are looking at
I Can you eat it or drink it?
I Is it perishable?
I Is it fragile?
I Can you recognize it?
I Where would you put it?
I How much is it?
I Have you seen anythingsimilar before?
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
Knowing what you are looking at
Not only labels, butI Sub-class relations: electrical
appliance, household item, coolingdevice
I properties of objects: fragile,heavy, perishable, edible
I relations to other objects: storagelocation, similarity
I integration of knowledge sources:common sense, environment,perception, human dialog
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
Knowing where to look for something
I Basic:Fixed controller for opening everything with a handle
I Parameterizable:Look up the positions of all containers in the map
I Knowledge-based:Infer the right container based on the types of the object andthe container
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
Knowing where to look for something
I Basic:Fixed controller for opening everything with a handle
I Parameterizable:Look up the positions of all containers in the map
I Knowledge-based:Infer the right container based on the types of the object andthe container
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
Knowing where to look for something
I Basic:Fixed controller for opening everything with a handle
I Parameterizable:Look up the positions of all containers in the map
I Knowledge-based:Infer the right container based on the types of the object andthe container
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
Knowing where to look for something
storagePlaceFor
instanceOf
instanceOf
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
Outline
1. Why Knowledge Processing for Autonomous Robots?
2. Introduction to Prolog
3. Overview of KnowRob
4. Asking Queries
5. Extending KnowRob
6. Integrating Perception Components
7. Inferring Control Decisions with KnowRob
8. Outlook
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
Why Prolog for autonomous robots?
I Can be read declaratively
I Can be programmed (prodecurally)
I There are textbooks and tutorials
I Industrial-strength open-source implementations
I Foreign language interfaces
I Result interpretation: similar to a DB query language
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
Knowledge base for the milk example
jmilk.plj
cup(cup0).
dairyProduct(milk1).
meatProduct(ham2).
cupboard(cupboard3).
refrigerator(fridge4).
perishable(Prod) :- dairyProduct(Prod); meatProduct(Prod).
storagePlaceFor(Loc, Item) :- refrigerator(Loc), perishable(Item).
storagePlaceFor(Loc, Item) :- cupboard(Loc), cup(Item).
searchForIn(Item,Loc) :- storagePlaceFor(Loc, Item).
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
Example queries
Two main types of queries:
I Query with unbound variables → variable bindings
I All variables bound → true/false result
Example queries
$ prolog -f milk.pl
?- refrigerator(A).
A = fridge4.
?- refrigerator(fridge4).
true.
?- searchForIn(milk1, A).
A = fridge4 .
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
Example queries
Example queries
?- storagePlaceFor(A, milk1).
A = fridge4 .
?- storagePlaceFor(fridge4, A).
A = milk1 ;
A = ham2 ;
?- storagePlaceFor(A, B).
A = fridge4,
B = milk1 ;
A = fridge4,
B = ham2 ;
A = cupboard3,
B = cup0.
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
Under the hood: the tracer
Tracing a query
?- trace, searchForIn(milk1, P).
Call: (7) searchForIn(milk1, G369) ? creep
jmilk.pljsearchForIn(Item, Loc) :- storagePlaceFor(Loc, Item).
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
Under the hood: the tracer
Tracing a query
?- trace, searchForIn(milk1, P).
Call: (7) searchForIn(milk1, G369) ? creep
. Call: (8) storagePlaceFor( G369, milk1) ? creep
jmilk.pljstoragePlaceFor(Loc, Item) :- refrigerator(Loc), perishable(Item).
storagePlaceFor(Loc, Item) :- cupboard(Loc), cup(Item).
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
Under the hood: the tracer
Tracing a query
?- trace, searchForIn(milk1, P).
Call: (7) searchForIn(milk1, G369) ? creep
. Call: (8) storagePlaceFor( G369, milk1) ? creep
. . Call: (9) refrigerator( G369) ? creep
jmilk.pljrefrigerator(fridge4).
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
Under the hood: the tracer
Tracing a query
?- trace, searchForIn(milk1, P).
Call: (7) searchForIn(milk1, G369) ? creep
. Call: (8) storagePlaceFor( G369, milk1) ? creep
. . Call: (9) refrigerator( G369) ? creep
. . Exit: (9) refrigerator(fridge4) ? creep
jmilk.pljrefrigerator(fridge4).
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
Under the hood: the tracer
Tracing a query
?- trace, searchForIn(milk1, P).
Call: (7) searchForIn(milk1, G369) ? creep
. Call: (8) storagePlaceFor( G369, milk1) ? creep
. . Call: (9) refrigerator( G369) ? creep
. . Exit: (9) refrigerator(fridge4) ? creep
. . Call: (9) perishable(milk1) ? creep
jmilk.pljdairyProduct(milk1).
perishable(Prod) :- dairyProduct(Prod); meatProduct(Prod).
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
Under the hood: the tracer
Tracing a query
?- trace, searchForIn(milk1, P).
Call: (7) searchForIn(milk1, G369) ? creep
. Call: (8) storagePlaceFor( G369, milk1) ? creep
. . Call: (9) refrigerator( G369) ? creep
. . Exit: (9) refrigerator(fridge4) ? creep
. . Call: (9) perishable(milk1) ? creep
. . . Call: (10) dairyProduct(milk1) ? creep
jmilk.pljdairyProduct(milk1).
perishable(Prod) :- dairyProduct(Prod); meatProduct(Prod).
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
Under the hood: the tracer
Tracing a query
?- trace, searchForIn(milk1, P).
Call: (7) searchForIn(milk1, G369) ? creep
. Call: (8) storagePlaceFor( G369, milk1) ? creep
. . Call: (9) refrigerator( G369) ? creep
. . Exit: (9) refrigerator(fridge4) ? creep
. . Call: (9) perishable(milk1) ? creep
. . . Call: (10) dairyProduct(milk1) ? creep
. . . Exit: (10) dairyProduct(milk1) ? creep
jmilk.pljdairyProduct(milk1).
perishable(Prod) :- dairyProduct(Prod); meatProduct(Prod).
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
Under the hood: the tracer
Tracing a query
?- trace, searchForIn(milk1, P).
Call: (7) searchForIn(milk1, G369) ? creep
. Call: (8) storagePlaceFor( G369, milk1) ? creep
. . Call: (9) refrigerator( G369) ? creep
. . Exit: (9) refrigerator(fridge4) ? creep
. . Call: (9) perishable(milk1) ? creep
. . . Call: (10) dairyProduct(milk1) ? creep
. . . Exit: (10) dairyProduct(milk1) ? creep
. . Exit: (9) perishable(milk1) ? creep
jmilk.pljdairyProduct(milk1).
perishable(Prod) :- dairyProduct(Prod); meatProduct(Prod).
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
Under the hood: the tracer
Tracing a query
?- trace, searchForIn(milk1, P).
Call: (7) searchForIn(milk1, G369) ? creep
. Call: (8) storagePlaceFor( G369, milk1) ? creep
. . Call: (9) refrigerator( G369) ? creep
. . Exit: (9) refrigerator(fridge4) ? creep
. . Call: (9) perishable(milk1) ? creep
. . . Call: (10) dairyProduct(milk1) ? creep
. . . Exit: (10) dairyProduct(milk1) ? creep
. . Exit: (9) perishable(milk1) ? creep
. Exit: (8) storagePlaceFor(fridge4, milk1) ? creep
jmilk.pljstoragePlaceFor(Loc, Item) :- refrigerator(Loc), perishable(Item).
storagePlaceFor(Loc, Item) :- cupboard(Loc), cup(Item).
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
Under the hood: the tracer
Tracing a query
?- trace, searchForIn(milk1, P).
Call: (7) searchForIn(milk1, G369) ? creep
. Call: (8) storagePlaceFor( G369, milk1) ? creep
. . Call: (9) refrigerator( G369) ? creep
. . Exit: (9) refrigerator(fridge4) ? creep
. . Call: (9) perishable(milk1) ? creep
. . . Call: (10) dairyProduct(milk1) ? creep
. . . Exit: (10) dairyProduct(milk1) ? creep
. . Exit: (9) perishable(milk1) ? creep
. Exit: (8) storagePlaceFor(fridge4, milk1) ? creep
Exit: (7) searchForIn(milk1, fridge4) ? creep
jmilk.pljsearchForIn(Item, Loc) :- storagePlaceFor(Loc, Item).
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
Under the hood: the tracer
Tracing a query
?- trace, searchForIn(milk1, P).
Call: (7) searchForIn(milk1, G369) ? creep
. Call: (8) storagePlaceFor( G369, milk1) ? creep
. . Call: (9) refrigerator( G369) ? creep
. . Exit: (9) refrigerator(fridge4) ? creep
. . Call: (9) perishable(milk1) ? creep
. . . Call: (10) dairyProduct(milk1) ? creep
. . . Exit: (10) dairyProduct(milk1) ? creep
. . Exit: (9) perishable(milk1) ? creep
. Exit: (8) storagePlaceFor(fridge4, milk1) ? creep
Exit: (7) searchForIn(milk1, fridge4) ? creep
P = fridge4 .
jmilk.pljsearchForIn(Item, Loc) :- storagePlaceFor(Loc, Item).
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
Prolog syntax
I Predicate names and constants start with a lowercase letter orare inside single quotes
I Variables start with an uppercase letter
I Formulas end with a full stop ’.’
I Stepping through all results of a query with ’;’
I Logical operators:
Operator Prolog equivalent Example∧ , type(A, ’Cup’), on(A, T).∨ ; type(A, ’Food’); type(A, ’Drink’).
jExamplej
dairyproduct(milk).
perishable(Stuff) :- storedIn(Stuff, Ref), refrigerator(Ref).
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
Outline
1. Why Knowledge Processing for Autonomous Robots?
2. Introduction to Prolog
3. Overview of KnowRob
4. Asking Queries
5. Extending KnowRob
6. Integrating Perception Components
7. Inferring Control Decisions with KnowRob
8. Outlook
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
The KnowRob knowledge base
Machine learning
Clustering,Classification,Segmentation
KNOW-ROB
Reasoners
DL, MLN, BLN
Natural languageprocessing
Physicalreasoning
Action-relatedconcepts
Actionabstraction
Encyclopedicknowledge
Common-senseknowledge
Queryinterface
Visualizationmodules
ROS SQL
human
Computable classes and properties
map vision human human
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
Launching KnowRob
I KnowRob is very modular: Dependencies are automaticallyloaded by rosprolog
I General syntax: rosrun rosprolog rosprolog <pkg>I starts a Prolog environmentI checks for dependencies of <pkg>I locates them on the diskI loads the prolog/init.pl file for each of these packages
I For today we use:rosrun rosprolog rosprolog knowrob tutorial
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
Outline
1. Why Knowledge Processing for Autonomous Robots?
2. Introduction to Prolog
3. Overview of KnowRob
4. Asking Queries
5. Extending KnowRob
6. Integrating Perception Components
7. Inferring Control Decisions with KnowRob
8. Outlook
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
Querying KnowRob from your program
I Tell/ask interface via a ROS serviceI tell: add statementsI ask: send a query
I Send queries as string like you would do from a console
I Result serialized using json
I Client libraries for C++, Python, Java, Lisp available
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
Querying KnowRob from Python
jknowrob/json prolog/examples/test json prolog.pyj
#/usr/bin/env python
import roslib; roslib.load manifest(’json prolog’)
import rospy
import json prolog
if name == ’ main ’:
rospy.init node(’test json prolog’)
prolog = json prolog.Prolog()
query = prolog.query("member(A, [1, 2, 3, 4]), B = [’x’, A]")
for sol in query.solutions():
print ’Found solution. A = %s, B = %s’ % (sol[’A’], sol[’B’])
query.finish()
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
Querying KnowRob from C++
jknowrob/json prolog/examples/test json prolog.cppj
#include <string>
#include <iostream>
#include <ros/ros.h>
#include <json prolog/prolog.h>
using namespace std;
using namespace json prolog;
int main(int argc, char *argv[]) {
ros::init(argc, argv, "test json prolog");
Prolog pl;
PrologQueryProxy bdgs = pl.query("member(A, [1, 2, 3]), B = [’x’, A]");
for(PrologQueryProxy::iterator it=bdgs.begin(); it != bdgs.end(); it++) {
PrologBindings bdg = *it;
cout << "Found solution: " << (bool)(it == bdgs.end()) << endl;
cout << "A = "<< bdg["A"] << endl;
cout << "B = " << bdg["B"] << endl;
}return 0;
}
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
The KnowRob Taxonomy
I Defines the basic concepts of the household and roboticdomain
I Hierarchy of classes and their properties
I Stored in the Web Ontology Language OWL as (S,P,O) triples
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
The KnowRob Taxonomy
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
Scaling up
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
Class Restrictions
Asserted vs. inferred class based on restrictions
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
TBOX
I The TBOX contains terminological statements: abstractdescriptions of classes and properties
I owl subclass of(Sub, Super). reads all classes derived from Super
I rdfs subproperty of(SubProp, Prop). lists sub-properties of Prop
I owl individual of(Restr, owl:’Restriction’), owl subclass of(Class,
Restr). reads all restrictions that are defined on Class
I owl satisfies restriction(Res, Restr). verifies whether Ressatisfies the restriction Restr
I owl description(RestrOWL, RestrProlog). translates the OWLidentifier of a restriction into a Prolog term
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
Example queries: TBOX
Example
$ rosrun rosprolog rosprolog knowrob tutorial
?- owl subclass of(A, knowrob:’FoodOrDrink’).
A = ’http://ias.cs.tum.edu/kb/knowrob.owl#FoodOrDrink’ ;
A = ’http://ias.cs.tum.edu/kb/knowrob.owl#Drink’ ;
A = ’http://ias.cs.tum.edu/kb/knowrob.owl#Coffee-Beverage’ ;
A = ’http://ias.cs.tum.edu/kb/knowrob.owl#InfusionDrink’ ;
A = ’http://ias.cs.tum.edu/kb/knowrob.owl#Tea-Beverage’ ;
A = ’http://ias.cs.tum.edu/kb/knowrob.owl#Tea-Iced’
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
ABOX
I The ABOX contains assertional statements: concreteinstances of the abstract classes and relations between them
I owl individual of(Ind, Class) is true if Ind is an instance of Class
I owl has(S, P, O) is a general query predicate for relations Pbetween a subject S and an object O
Important: Most predicates can be used with differentcombinations of free and bound variables:
I owl individual of(Ind, knowrob:’Cup’) reads all instances of classCup
I owl individual of(knowrob:’cup0’, Class) reads all classes cup0belongs to
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
Example queries: ABOX
Example
?- owl has(A, rdf:type, knowrob:’Cupboard’).
A = ’http://ias.cs.tum.edu/kb/knowrob.owl#Drawer1’ ;
A = ’http://ias.cs.tum.edu/kb/knowrob.owl#Drawer103’ ;
?- owl has(knowrob:’Drawer1’, P, O).
P = ’http://www.w3.org/1999/02/22-rdf-syntax-ns#type’,
O = ’http://ias.cs.tum.edu/kb/knowrob.owl#Drawer’ ;
P = ’http://ias.cs.tum.edu/kb/knowrob.owl#widthOfObject’,
O = literal(type(xsd:float, ’0.58045006’)) ;
P = ’http://ias.cs.tum.edu/kb/knowrob.owl#properPhysicalParts’,
O = ’http://ias.cs.tum.edu/kb/knowrob.owl#Door4’ ;
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
Visualizing knowledge
I A java-based visualization canvas can display objects and theirlocations
I visualization canvas(C). starts the canvas and stores a handle inthe variable C
I $C refers to the last binding of the variable C
I add object(Identifier, $C). sends an object instance to thecanvas in order to visualize it
I API documentation: http://ias.in.tum.de/kb/api/mod vis.pl.html
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
Example queries: Visualization
Example
?- owl has(A, rdf:type, knowrob:’Cupboard’),
add object with children(A, $C).
A = ’http://ias.cs.tum.edu/kb/ias semantic map.owl#closet0’ ;
A = ’http://ias.cs.tum.edu/kb/ias semantic map.owl#cupboard0’ ;
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
Exercise: Querying KnowRob
Exercise sheet – Querying the Knowledge Base
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
Optional queries
I Sub-classes of CookingUtensil
I Everything that has a handle
I What to use for heating food?
I What needs electricity?
I What needs to be put into the fridge?(hint: predicate storagePlaceFor)
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
Outline
1. Why Knowledge Processing for Autonomous Robots?
2. Introduction to Prolog
3. Overview of KnowRob
4. Asking Queries
5. Extending KnowRob
6. Integrating Perception Components
7. Inferring Control Decisions with KnowRob
8. Outlook
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
Editing Ontologies with Protege
I Graphical editor for OWL ontologies
I Download and tutorials available at http://protege.stanford.edu/
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
Exercise: Adding knowledge to KnowRob
Exercise sheet – Extending the Knowledge Base
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
Outline
1. Why Knowledge Processing for Autonomous Robots?
2. Introduction to Prolog
3. Overview of KnowRob
4. Asking Queries
5. Extending KnowRob
6. Integrating Perception Components
7. Inferring Control Decisions with KnowRob
8. Outlook
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
Reasoning on robot data
I Robot knowledge processing = dynamic knowledge processing
I Knowledge base needs to make decisions based on the mostcurrent information
Use the robot’s data structures as extended knowledge base!
I Create instances on demand (e.g. recognized objects)
I Compute properties of and relations between them
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
Computables: Computation On Demand
Computables: Attachcomputational methods tosemantic relations
Two components:
I Prolog predicate tocompute the relation
I OWL file to link thepredicate to the OWLrelation
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
Calling computables
I Take computables into account only when needed
I Special query predicates: rdf triple, rdfs instance of
I return all normal solutions (owl has, owl individual of)I search for all computables for the class/relationI return all results of all computables
Useful predicates
rdf triple(+Pred, ?Subj, ?Obj)
rdfs instance of(?Inst, ?Class)
Note the different order of arguments for rdf triple
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
Task 1: Retrieve object information
I Java client to interface with ROS:src/edu/tum/cs/ias/knowrob/tutorial/ROSClient.java
I Use JPL to call the object detection service from Prolog/object detection
Useful predicates
jpl new(+Class, +Arguments, -Value)
jpl call(+Class, +Method, +Args, -Result)
jpl call(+Object, +Method, +Args, -Result)
jpl get(+Object, +Field, -Value)
jpl array to list(+JArray, -PlList)
See also: http://www.swi-prolog.org/packages/jpl/prolog api/api.html
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
Task 1: Retrieve object information
I Extract object information from the TabletopDetectionResult
I Pass these values to the predicates for creating the objectinstance, the perception instance, and for linking those
Useful predicates
create perception instance(-Perception)
create object instance(+ObjTypes, +ID, -ObjInst)
set object perception(+ObjInst, +Perception)
set perception pose(+Perception, [+M00, +M01, +M02, +M03,
+M10, +M11, +M12, +M13,
+M20, +M21, +M22, +M23,
+M30, +M31, +M32, +M33])
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
Task 1: Object representation
VisualPerception-42objectActedOn: icetea2eventOccursAt: homography9startTime: timep_1.572
KNOWROBhomography9
1 0 0 2.560 1 0 1.320 0 1 0.930 0 0 1
compCoptarget:VisualPerception
TIM
E
CoP
homography8
1 0 0 2.340 1 0 1.750 0 1 0.730 0 0 1
icetea2type: TetraPakwidth: 0.1depth: 0.1height: 0.3
VisualPerception-34objectActedOn: icetea2eventOccursAt: homography3startTime: timep_1.243
VisualPerception-37objectActedOn: icetea2eventOccursAt: homography5startTime: timep_1.295
VisualPerception-41objectActedOn: icetea2eventOccursAt: homography8startTime: timep_1.493
homography3
1 0 0 3.040 1 0 2.250 0 1 0.930 0 0 1
homography5
1 0 0 0.940 1 0 1.050 0 1 1.530 0 0 1
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
Task 2: Reason about the perceived objects
Have a look at the KnowRob ontology and formulate the followingqueries on the just perceived set of objects:
I Everything that can contain other things
I All food items
I Perishable things
I The positions (x,y,z) of all drinks
Useful predicates
rdfs instance of(A, knowrob:’HumanScaleObject’)
owl individual of(Inst, Class)
owl subclass of(Sub, Super)
rdf triple(knowrob:orientation, Obj, Or)
rdf triple(knowrob:m03, Or, X)
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
Optional tasks
I Query for items that belong into the fridge
I Determine the pose of the handle of the storage place for alldetected perishable items
I Visualize your results in mod vis
I Write a program in Python/C++ that sends the abovequeries via ROS
I Have a look at owl/tabletop obj.owl to see how the computableis realized
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
Outline
1. Why Knowledge Processing for Autonomous Robots?
2. Introduction to Prolog
3. Overview of KnowRob
4. Asking Queries
5. Extending KnowRob
6. Integrating Perception Components
7. Inferring Control Decisions with KnowRob
8. Outlook
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
Integration into the executive
Lorenz
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
Outline
1. Why Knowledge Processing for Autonomous Robots?
2. Introduction to Prolog
3. Overview of KnowRob
4. Asking Queries
5. Extending KnowRob
6. Integrating Perception Components
7. Inferring Control Decisions with KnowRob
8. Outlook
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
Reasoning on objects
jcomp copj
Listen to topics and add theinformation to the knowledge base
jcomp spatial, comp temporalj
Compute qualitative spatial andtemporal relations (in, on, during)
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
Reasoning on actions
jcomp ehowj
Translate natural-languageinstructions into OWLrepresentations and robot plans
jcomp fipm, mod hamj
Segment and abstract humanmocap data
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
Reasoning
jsrldb, mod probcogj
Probabilistic inference usingMarkov Logic Networks andBayesian Logic Networks
jias prolog addons/classifiers.plj
Weka and Mallet libraries forclassifiers and clusterers
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth
Motivation Prolog KnowRob Query Extend Integrate Use Outlook
Reasoning
jmod similarityj
Compute the semantic similaritybetween concepts in the ontology
jias naive physics simj
Naive physics reasoning forcomputing relations in theknowledge base
November 5, 2010
Knowledge Processing for Autonomous Robots Moritz Tenorth