Plan Recognition in Virtual Laboratories

Ofra Amir and Ya’akov (Kobi) Gal

Ben-Gurion University of The NegevDepartment of Information Systems Engineering

2

Background

• Educational software in the sciences is including open-ended “construction” environments

• Students build and analyze models using these software

3

Virtual Labs– An interactive simulation of a chemistry lab.

4

Virtual Labs – Dilution Problem

“ Your supervisor has just asked you to prepare

500mL of 3M HNO3 for tomorrow's experiment.

In the stockroom explorer, you will find a cabinet

called Stock Solutions. Open this cabinet to find a

2.5L bottle labeled 15.4M HNO3. ”

5

The Problem

Please prepare a flask containing 500 ml of a 3M

solution…

I’m confused Boring… This is

easy!

During class:

6

The Problem

After class:

I wonder how each student performed…

7

The Goal

• Develop automatic support for teachers in their analysis of student performance

– Approach: use plan recognition to infer students’ problem solving plans

8

Related Work

• Bayesian networks to model students’ interactions with intelligent tutors [Conati et al., 2002]

• Complete algorithms for plan recognition (CSP) [Reddy et al, 2009; Quilici et al, 1998]

• Heuristic algorithm for recognizing students’ activities in pedagogical software for statistics [Gal et al, 2008]

9

Solution Building blocks

• User actions

• Recipes

• Plans

10

User Actions – Log Files

11

Recipes [Pollack, 1990]

• Basic actions are rudimentary (log actions)

• Complex actions are abstract

• A recipe for a complex action describes

– A series of sub-actions for completing the action

– Constraints on these actions

12

• Representing actions:

• Recipe structure:

The Recipe Language

Action Name An action parameter

with value constraint

D[dt:H20, sid_1,did_1][ 0 ]

Parameter without value

constraint

Action Pre-conditions

D[sid_1,did_2] -> D[sid_1,did_1][ ] C[sid_2,did_2][ ]

The complex action

sub-actions did_1=sid_2

Constraints on sub-actions parameters

13

Plans in Virtual Labs

• A plan for a complex action is– A hierarchy of recipes towards completing the

action

• The plan represents students’ activity with the software

• Plan recognition– Infer students’ activities based on their actions

with the software, given a set of recipes

14

Plan Recognition Approach

Plan Recognition Algorithm

Student Plan

15

Dilution Problem Recipes

• To solve the problem, the student should:– Pour H20 to the destination flask

– Pour HNO3 to the destination flask

• Students can solve this problem in many different ways

16

Dilution Problem - Example Solution 1

2 1

17

Dilution Problem - Example Solution 2

18

Challenges in The Virtual Labs Domain

• Indefinite repetition of activities

• Interleaving activities

• Trial-and-error, mistakes

• Conclusion:– Complete approach intractable for this domain

19

Build Plan Greedy Algorithm• Input:

– A set of Virtual Labs basic actions, A set of recipes• Algorithm steps:

– initialize open list with the actions from the log– for each recipe in order of increasing depth

• find match (Recipe, open list)• while match exists

– add complex action to the open list – create branches from the complex action to its sub-actions– remove sub-actions from open list– Call find match with the updated open list

20

( SM , did=ID5 , sid=ID1 )

( SM , did=ID5 , sid=ID1 )

( MO , id=ID5 )

( MO , id=ID5 )

( SM , did=ID6 , sid=ID5 )

( MO , id=ID1 )

( AS , id=ID6 )

( MO , id=ID6 )

( MO , id=ID5 )

( FC , did=ID6 ,sid=ID5 )

( SM , did=ID6 , sid=ID5 )

( C , did=ID5 , sid=ID1 )

( MO , id=ID5 )

( MO , id=ID5 )

( AS , id=ID6 )

( MO , id=ID6 )

( MO , id=ID5 )

( FC , did=ID6 ,sid=ID5 )

( C , did=ID6 , sid=ID5 )

( C , did=ID5 , sid=ID1 )

( MO , id=ID5 )

( MO , id=ID5 )

( AS , id=ID6 )

( MO , id=ID6 )

( MO , id=ID5 )

( FC , did=ID6 ,sid=ID5 )

( C , did=ID6 , sid=ID5 )

( D , did=ID6 , sid=ID1 )

P1 P2P3

Recipe: C -> SM SM Recipe: D -> C C

21

Find Match• Find match searches for actions in the open list which fulfill the recipe for the complex action

• Actions in the match can be free ordered as long as they satisfy the constraints in the recipe

• Was implemented as depth first search, but can be implemented in other ways

• Find Match is complete, given a recipe and an open list of actions

22

Dilution Problem – Partial Plan

MSC – Mixing solution componentMSI – Mixing solution through intermediate flaskSDP – Solve Dilution Problem

23

Empirical Evaluation• The algorithm was run on 20 log files taken from real

student interactions– 6 different problems– logs ranged in size from 20 actions to 187 actions– Plans ranged in depth from 3 to 14 levels

• The plans were validated by a domain expert

24

Contributions

• A new computationally efficient plan recognition algorithm that can cope with interleaving activities, mistakes, indefinite repetition.

• The algorithm can be integrated with real pedagogical software

• Shown to succeed on real-world data

25

Questions