Constraint-based tutoring

CPI 494 Feb 17, 2009Kurt VanLehn

ASU

Outline

• SQL-Tutor (today)• ER-Tutor & Normit (next class)• Technical details (next class)• Evaluation & Dissemination (next class)

Framework for comparing step-based tutoring systems

Feature Constraint based Cognitive

Step granularity & user interface

Interpreting a student’s step

Suggesting good steps

Feedback and hints on steps

Task selection

Assessment

Evaluations

Dissemination

In class-exercises• Install SQL tutor• Try at least 3 problems• Check our your student model periodically• Try different levels of help• Try different ways of expressing the same step• Try different ordering of steps• What steps does it suggest?• Try to game it• Try fancy ways to say simple things: E.g., instead of

referring to Stanley Kubrick by name, retrieve the director of “Clockwork Orange”

Step granularity & user interface

• Type of user interface? Canvas, forms (i.e., goal boxes), text, simulation based?

• Learnability? Every problem the same UI? • In general, what are the steps? The Answer?• On the problem “Display the titles of movies

that are comedies or dramas”, what are the steps?– Would a finer granularity make sense? For who?

Framework for comparing step-based tutoring systems

Feature Constraint based Cognitive

Step granularity & user interface Large, fixed-size goal boxes Goal boxes, granularity varies

Interpreting a student’s step

Suggesting good steps

Feedback and hints on steps

Task selection

Assessment

Evaluations

Dissemination

Interpreting student steps: How does tutor know what is correct?• Is it example-based or knowledge-based or

both?• “Constraint-based modeling (CBM) is…not

interested in the exact sequence of states in the problem space…”

• “CBM does not require an executable domain model, and is applicable in situations in which such a model would be difficult to construct.”

• “CBM eliminates the need for bug libraries”

A syntactic constraint

• Msg: "The SELECT clause is a mandatory one. Specify the attributes/expressions to retrieve from the database.“

• Clause applied to: SELECT• Relevance condition: t• Satisfaction condition: (not (null (select-clause ss)))

A semantic constraint• Msg: "Check whether you should have ascending or descending

order!“• Clause: “ORDER BY“• Relevance: (and (not (null (order-by is)))

(not (null (order-by ss)))(bind-all ’?n (names (order-by ss)) bindings)(match ’(?*d1 ?n "DESC" ?*d2) (order-by ss) bindings)(not (qualified-name ’?n)))

• Satisfaction: (match ’(?*d3 ?n "DESC" ?*d4) (order-by is) bindings)

• Comment: “verifies that if the results of a query are sorted in decreasing order in the ideal solution (is), the the student’s solution (ss) uses the same ordering.”

Interpreting a student’s step:Details of the matching

• How is an individual step interpreted?• Spaces? Capitalization? • Ordering of conjuncts and disjuncts?• Ordering of steps?• Fancy ways to say simple things? E.g., instead

of referring to Stanley Kubrick by name, retrieve the director of “Clockwork Orange”

Framework for comparing step-based tutoring systems

Feature Constraint based Cognitive

Step loop: Granularity & user interface Large, fixed-size goal boxes

Goal boxes, granularity varies

Step loop: Interpreting student steps Constraints + ideal solution; CB matching

EB or KB; Exact, intervals, regular expressions

Step loop: Suggesting good steps

Step loop: Feedback and hints

Task selection

Assessment

Evaluations

Dissemination

Suggesting good steps

• What does its method/ordering appear to be?• What if there are multiple correct solutions?

Framework for comparing step-based tutoring systems

Feature Constraint based Cognitive

Step granularity & user interface Large, fixed-size goal boxes

Goal boxes, granularity varies

Interpreting a student’s step String match Exact, intervals, regular expressions

Suggesting good steps Next blank step Next step in order

Feedback and hints on steps

Task selection

Assessment

Evaluations

Dissemination

Feedback and hints

• What are the hint levels?• How can one game?• Does it detect cut-and-paste?• Does it detect common mistakes e.g., quotes,

“WHERE” as first word of WHERE clause…• Are the hints pedagogical?

Framework for comparing step-based tutoring systems

Feature Constraint based Cognitive

Step granularity & user interface Large, fixed-size goal boxes

Goal boxes, granularity varies

Interpreting a student’s step String match Exact, intervals, regular expressions

Suggesting good steps Next blank step Next step in order

Feedback and hints on steps Few bugs; selectable hint from sequence

Few bugs; hint sequence

Task selection

Assessment

Evaluations

Dissemination

Task selection

• What happens if you select next task in order?• Anyone graduate from novice to higher?• Do clauses make a good index for student task

selection?• Does student have enough info to do task

selection?

Framework for comparing step-based tutoring systems

Feature Constraint based Cognitive

Step granularity & user interface Large, fixed-size goal boxes

Goal boxes, granularity varies

Interpreting a student’s step String match Exact, intervals, regular expressions

Suggesting good steps Next blank step Next step in order

Feedback and hints on steps Few bugs; selectable hint from sequence

Few bugs; hint sequence

Task selection Some student choice Mastery learning

Assessment

Evaluations

Dissemination

Assessment

• What are the KCs? Clauses or constraints?– Displayed?– Learning curves? – How many?

• Are the colored sections of the bars informative?

• How updated?• Is history window useful?

Framework for comparing step-based tutoring systems

Feature Constraint based Cognitive

Step granularity & user interface Large, fixed-size goal boxes

Goal boxes, granularity varies

Interpreting a student’s step String match Exact, intervals, regular expressions

Suggesting good steps Next blank step Next step in order

Feedback and hints on steps Few bugs; selectable hint from sequence

Few bugs; hint sequence

Task selection Some student choice Mastery learning

Assessment ? Knowledge tracing

Evaluations

Dissemination

Outline

• SQL-Tutor (today)• ER-Tutor & Normit (next class)• Technical details (next class)• Evaluation & Dissemination (next class)