QA on Anderson et al. 1995Intro to CTAT

CPI 494 & 598 Jan 27, 2009Kurt VanLehn

Initial Framework• Step loop– User interface– Interpreting student actions– Suggesting good actions– Feedback and hints

• Task selection• Assessment• Authoring and the software development• Evaluations• Dissemination

NEXT

Describe the user interface of the Lisp Tutor

• What do students type? What are the <..> things? Steps?

Describe the user interface of the Geometry tutor

• What does student do? Steps?

Initial Framework• Step loop– User interface– Interpreting student actions– Suggesting good actions– Feedback and hints

• Task selection• Assessment• Authoring and the software development• Evaluations• Dissemination

NEXT

For the Lisp tutor

• What counts as a correct step?• How did the tutor determine if the student’s

step is correct?• Would the tutor accept both– (plus X 5) and (plus 5 X)– (times X 0.5) and (divide X 2)– How does it know?

• Can students choose their own variable names?

For the geometry tutor

• What steps count as correct?– True but useless propositions?– Top down vs. bottom up?

• How does the tutor determine if the student’s step is correct?

Initial Framework• Step loop– User interface– Interpreting student actions– Suggesting good actions– Feedback and hints

• Task selection• Assessment• Authoring and the software development• Evaluations• Dissemination

NEXT

For the Lisp tutor

• When the student asks for a hint, how does the tutor determine what step to hint?

• If the tutor allowed students to keep their incorrect steps on the screen, what step would the tutor suggest when asked?

• What if the tutor did not give immediate feedback and were asked for a hint?

For the Geometry tutor

• How does it decide what step to suggest?• What does ACT theory say about this policy?• Would it be better to recognize the student’s

plan and help them along by suggesting its next step?

Initial Framework• Step loop– User interface– Interpreting student actions– Suggesting good actions– Feedback and hints

• Task selection• Assessment• Authoring and the software development• Evaluations• Dissemination

NEXT

For the Lisp tutor

• Does the “normal” version of the tutor give immediate feedback, delayed…?

• What other kinds of feedback policies did they try? What were the results?

• How was hinting usually done? • Did the tutor ever ask a question when asked

for a hint?

For the Geometry tutor

• Does the “normal” version of the tutor give immediate feedback, delayed…?

• What other kinds of feedback policies did they try? What were the results?

• How was hinting usually done? • Did the tutor ever ask a question when asked

for a hint?

Initial Framework• Step loop– User interface– Interpreting student actions– Suggesting good actions– Feedback and hints

• Assessment• Task selection• Authoring and the software development• Evaluations• Dissemination

NEXT

Lisp tutor: Assessment

• What is assessed? Motivation? Knowledge? What kind of …?

• What granularity?• Give examples of what is assessed?• What’s the name of the technique?• When students enter correct step, incorrect

step or hint-request, what happens to the assessment?

• IF the goal is to get the second element of the list

• THEN code car and set as a subgoal to pass to car an argument that is the tail of the list

Geometry tutor: Assessment

• What is assessed?• Give examples• How is the assessed stuff different from

Lisp’s?

• IF the goal is to prove two triangles are congruent

• THEN set as a subgoal to prove that corresponding parts are congruent.

Initial Framework• Step loop– User interface– Interpreting student actions– Suggesting good actions– Feedback and hints

• Assessment• Task selection• Authoring and the software development• Evaluations• Dissemination

NEXT

Lisp tutor: Task selection

• What are the tasks to be selected from?• Are they authored or generated dynamically?• Who does the selection? Tutor or student?• What is the procedure for selection?• What information does the procedure need?• How does this procedure fit with ACT theory?

Geometry tutor: task selection

• What are the tasks to be selected from?• Are they authored or generated dynamically?• Who does the selection? Tutor or student?• What is the procedure for selection?• What information does the procedure need?• How does this procedure fit with ACT theory

Initial Framework• Step loop– User interface– Interpreting student actions– Suggesting good actions– Feedback and hints

• Assessment• Task selection• Authoring and the software development• Evaluations• Dissemination NEXT

Lisp stutor: Evaluations

• List the evaluations done• For each, is it: Formative? Summative?

Hypothesis testing?• What are the results of each?

• a fixed set of programming exercises either with the tutor or in a standard LISP environment. Students using the tutor completed the exercises 30% faster and performed 43% better on a posttest.

• using the tutor completed the exercises 64% faster and scored 30% higher on posttests than did students using a standard LISP environment.

What did this graph show?

What did this graph show?

Geometry tutor: Evaluations

• List the evaluations done• For each, is it: Formative? Summative?

Hypothesis testing?• What are the results of each?

• In Peabody HS, one standard deviation• An ethnography• Again, but only got positive result with

teacher.