artificial intelligence & human-robot interaction · • multi-robot support pnpgen generates...

Artificial Intelligence & Human-Robot Interaction

Luca IocchiDept. of Computer Control and Management Eng.

Sapienza University of Rome, Italy

Robotic Applications• Industrial/logistics/medical robots

• Known environment• Minimal interaction with expert users

• Rescue robots• Unknown/partially known environment• Minimal interaction with expert users

• Home service robots• Known environment• Long-term interaction with a few (trained) users

• Service robots in public environments• Known and dynamic environment• Short-term interaction with many naïve users

AIRO 2017 - AI & HRI 2

AI&HRI Motivations• Interacting with people requires more "intelligence" than

interacting with the environment• Difficulty in modeling and perception (Unpredictability)

• Difficulty in decision making (Social norms)

• Some examplesIn HRI

• proper decisions about when to start an interaction are required

• failures have more severe consequences

• wrong assumptions or guesses may be socially unacceptable

• …


ICAPS 2017 Tutorial: AI Planning for Robotics and Human-Robot InteractionAAAI 2017 Fall Symposium: AI for Human-Robot InteractionRSS 2017 Workshops

AI & HRI

Robotic basic skills commonly used in HRI

Computer Vision fairly used for simple recognition tasks

Machine Learning used in subtasks (perception, speech recognition) and in Learning by Demonstrations

Natural Language Processing less used in robotics

KR & Decision making (autonomous behaviors) are less investigated


HRIapplication domain

(i.e., problem generator)

AI&RO methodologies and techniques

to solve complex problems(i.e., solution generator)

HRI Experimental Methodology

Wizard-of-Oz: an operator (hidden from the user evaluatingthe HRI system) replaces perception and decision making of the robot

Issues• Over-estimation of actual abilities of the robot

• Partial evaluation of the HRI system (does not measure"intelligence in interaction")

• Expectations not met by current technology, preventing or reducing possibilities of deployment of actual systems


HRI in public environments

Main challenges• Interaction with naïve users• Follow complex social norms• Assumption of having

complete information is not realistic

• Perception involving humans is generally more difficult

• Guessing missing information may bring to socially unacceptable behaviors

• Need of very robust perception and decision-making


Analysis of planning techniques RoboCup@Homelargest worldwide competition on service robots

Planning in the 24 teams in RoboCup@Home 2016

Complex robotic systems integrating:• Robotics: navigation, manipulation, …• Computer Vision: object/face/person detection, recognition and tracking• HRI: Speech recognition and natural language processing

Plan GenerationHigh-level Behavior

N/A Total

2 (ASP, MDP) 9 13 24


RoboCup@Home Teams

WrightEagle: ASP Markvito: MDP/SPUDD

Golem: cognitive architecture. Dialogue models specified in SitLogHomer: state machinesLeon: state machine (BICA)Pumas: HTN / High-level Petri NetsSepanta: SMACHSocRob: POMDP + Discrete Event System / SMACHTech United: SMACHToBI : BonSAI + State Chart XML = SMACHWalking Machine: SMACH


Decision making in HRIDecision making in noisy and partially known real environments is still a challenging open issue in AI

HRI researchers not expert in AI & decision making do not find a practical and easy way to apply these techniques

Manual behavior programmingand Wizard-of-Oz experimentsare commonly used


TODO

• Extend applicability

• Improve usability

• Improve robustness

Automatic planning in HRI

AIRO 2017 - AI & HRI 10

Planning improves scalability wrt complexity and flexibility• Manual plan writing is error prone and not scalable

• Each task must be defined explicitly/no reuse of components

• Verification/validation is not possible

• Explanations are very difficult

Advantages• Compact representations

• Less effort in generating many plans

• High-level domain specification language for non-expert designers

Service robot has to make sure user needs are satisfied. User needs are not known in advance.

AIRO 2017 - AI & HRI 11

Example

Classical planning(complete knowledge about initial state)

• Guess a user need

• Plan with this guess

• Execute the plan

• If guess is wrong, adjust conditions and replan

When guess is wrong, behavior is not socially acceptable.

• The robot does not move, but the user needs something.

• The robot prepares food or drink that is not requested by the user.

AIRO 2017 - AI & HRI 12

Example

Plan with explicit sensing action• Go to person

• Ask if s/he needs something // Sensing action

• if (need_food AND need_drink)• Go to the kitchen

• Prepare food and drink

• Serve food and drink to person

• if (need_food)• …

• else• Do nothing

AIRO 2017 - AI & HRI 13

Example

Pro-active behavior in which the robot does not

just wait for orders.

Conditional Planning for service robots

Advantages• Does not require complete knowledge about the initial state

at planning time (can model situations where some user needs are not known)

• Allows for minimal execution of sensing (reduces wrong behaviors due to incorrect perception)

Issues• Plan generation more complex • Conditional Planners less developed• Writing domains is still difficult• Loop generation (e.g., while conditions) not available

AIRO 2017 - AI & HRI 14

Not-only planning

AIRO 2017 - AI & HRI 15

Robot planning

Planning expert

Robot/plan expert user

Robot planning

Domain expert

Naïve user

Robust plans

• Plans generated by planners are usually no robust to unmodelledevents

• Interaction with naïve users requires increased robustness of plans

AIRO 2017 - AI & HRI 16

Robot planning

Domain expert

Naïve user

Generation of robust plans

DomainPlanner

???

Execution

π

Goal

Planning and Execution Component

π*

UILearning

AIRO 2017 - AI & HRI 17

???

Proposed solution

ROSPlanContingent-FF

Robustification

PN

P-R

OS

π

Planning and Execution Component

Execution

Rules

PNP

AIRO 2017 - AI & HRI 18

Domain

Goal

[Sanelli et al., ICAPS 2017]

ROSPlan and Contingent-FF

ROSPlan,

Contingent-FF

http://kcl-planning.github.io/ROSPlan/

AIRO 2017 - AI & HRI 19

[Cashmore et al., ICAPS 2015]

[Hoffmann and Brafman, ICAPS 2005]

Petri Net PlansFormalism for high-level plan representationbased on Petri Nets

• Ordinary and sensing actions• Conditions and loops• Interrupts• Parallel execution (fork and join operators)• Multi-robot support

PNPGen generates PNP from several planners (MDP solver, ROSPlan, HATP, …)

PNP-ROS uses ROS actionlib to run plans including ROS actions

pnp.dis.uniroma1.it

AIRO 2017 - AI & HRI 20

[Ziparo et al., JAAMAS 2011]

Execution rules

Adding to the conditional plan• interrupt (special conditions that activate recovery paths)• recovery paths (how to recovery from unexpected events)• social norms• parallel execution (multi-modalities)

Main feature• Easy definition• Execution variables are generally different from the ones

in the planning domain (thus not affecting complexity of planning)

AIRO 2017 - AI & HRI 21

Examples

if personhere and closetotarget during goto do skip_actionif personhere and not closetotarget during goto do

say_hello; waitfor_not_personhere; restart_actionif lowbattery during * do recharge; fail_planafter receivedhelp do say_thanksafter endinteraction do say_goodbyewhen say do display

AIRO 2017 - AI & HRI 22

Execution rules

Petri Net Plans generation

PNP Generation 1. Translation of conditional plan to PNP

2. Introduction of execution rules

Robust plan with sensing and loops

Algorithm is linear in the size of the plan and of the execution rules (average case)

AIRO 2017 - AI & HRI 23

Robot Office AssistantOutput: PNP with 17 actions, 45 places, 52 transitions, 104 edges

AIRO 2017 - AI & HRI 24

Robot Office Assistant

AIRO 2017 - AI & HRI 25

Other demos

Advertising in many shops (MDP)

AIRO 2017 - AI & HRI 26

Human-Robot Collaboration (HATP)

COACHES project - https://coaches.greyc.fr/

[Sebastiani et al., ICAPS 2017][Iocchi et al., ICAPS 2016]

Domain description

• In previous examples planning domains written by planner experts

• HRI domain experts (not expert in planning) need high-level framework for interaction design

AIRO 2017 - AI & HRI 27

Robot planning

Domain expert

Naïve user

MODIM

Multi-modal Interaction Manager• Formalism for high-level description of multiple modalities interactions• Based on interaction templates that can be instantiatedto generate many

actual interactions• Represent robotic and interaction actions in a unique framework• Not yet a framework to specify planning domains• Actions types:

• robotic actions (e.g., move, approach, goto, deliver)• interactions (e.g., ask, state, answer)

• Operators:• conditional (e.g., evaluate an answer)• choice (e.g., choose a question)

AIRO 2017 - AI & HRI 28

MODIM IMPLEMENTATION

Interaction template

AIRO 2017 - AI & HRI 29


Interactions based on multi-modal interaction function

AIRO 2017 - AI & HRI 30


AIRO 2017 - AI & HRI 31

MODIM EXAMPLEInput <LABEL1; ask < Quiz1; GetChoices() >; answer < Result1 >; Result1 = wrong?state < Answer; “wrong" >; GOTO LABEL1 : state < Answer; “right" >; ….>

AIRO 2017 - AI & HRI 32

Output

user answer < Result1==disappears >

user answer < Result1==remains the same >

Very well! ….

disappears

remains the same

MODIM user studies

MODIM was used by a school teacher to design and realize a physics lesson and to perform HRI studies

AIRO 2017 - AI & HRI 33

Conclusions• AI Decision making techniques useful in HRI tasks

• Conditional planning + robustification improve robustness of robot plans for HRI applications

• High-level frameworks for interaction design improve usability by non-expert designer and developer

• More work is needed to improve usability and applicability

• More experiments in real environments with real users and real limitations (no Wizard-of-Oz)

Thank you for your attention

AIRO 2017 - AI & HRI 34

[Hoffmann and Brafman 2005] Hoffmann, J., and Brafman, R. Contingent planning via heuristic forward search with implicit belief states. In Proc. of ICAPS, 2005.

[Cashmore et al., 2015] Cashmore, M.; Fox, M.; Long, D.; Magazzeni, D.; Ridder, B.; Carrera, A.; Palomeras, N.; Hurtos, N.; and Carreras, M. Rosplan: Planning in the robot operating system. In Proc. of ICAPS, 2015.

[Iocchi et al., 2016] Luca Iocchi, Laurent Jeanpierre, Maria Teresa Lazaro, Abdel-IllahMouaddib: A Practical Framework for Robust Decision-Theoretic Planning and Execution for Service Robots. In Proc. of ICAPS, 2016.

[Sanelli et al., 2017] V. Sanelli, M. Cashmore, D. Magazzeni, L. Iocchi. Short-Term Human Robot Interaction through Conditional Planning and Execution. In Proc. of ICAPS 2017

[Sebastiani et al. 2017] E. Sebastiani, R. Lallement, R. Alami, L. Iocchi. Dealing with On-line Human-Robot Negotiations in Hierarchical Agent-based Task Planner. In Proc. ICAPS 2017.

[Ziparo et al., 2011] V. A. Ziparo, L. Iocchi, P. U. Lima, D. Nardi, P. F. Palamara. Petri Net Plans - A framework for collaboration and coordination in multi-robot systems. In Autonomous Agents and Multi-Agent Systems, 23 (3), 2011.

AIRO 2017 - AI & HRI 35

References

artificial intelligence & human-robot interaction · • multi-robot support pnpgen generates...

Documents