1 the pépite project Élisabeth delozanne, paris universitas, upmc d. prévit, b. grugeon, f....
Post on 21-Dec-2015
214 views
TRANSCRIPT
![Page 1: 1 The Pépite project Élisabeth Delozanne, Paris Universitas, UPMC D. Prévit, B. Grugeon, F. Chenevotot Automatic Multi-criteria Assessment of Open-Ended](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649d6b5503460f94a4a80d/html5/thumbnails/1.jpg)
1
The Pépite project
Élisabeth Delozanne, Paris Universitas, UPMCD. Prévit, B. Grugeon, F. Chenevotot
Automatic Multi-criteria Assessment of Open-Ended Questions: a case study in
school algebra
ITS’2008
![Page 2: 1 The Pépite project Élisabeth Delozanne, Paris Universitas, UPMC D. Prévit, B. Grugeon, F. Chenevotot Automatic Multi-criteria Assessment of Open-Ended](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649d6b5503460f94a4a80d/html5/thumbnails/2.jpg)
Cognitive modeling authoring tool Problem
Multi-step reasoning, multiple equivalent reasonings Our approach
1.An expert teacher (or a researcher) defines diagnosis exercises
2.A cognitive engineer implements templates that generalize these particular diagnosis exercises
3.A teacher clones these diagnosis exercises by filling template forms
4.A domain specific application • generates the clone and a set of plausible
correct and incorrect anticipated solutions• matches the student’s reasoning with
anticipated solutions2
![Page 3: 1 The Pépite project Élisabeth Delozanne, Paris Universitas, UPMC D. Prévit, B. Grugeon, F. Chenevotot Automatic Multi-criteria Assessment of Open-Ended](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649d6b5503460f94a4a80d/html5/thumbnails/3.jpg)
OutlineAn introductory example
Pépite : a specific diagnosis toolPépiGen : a system to clone Pépite
Author’s and Student’s points of view Automatic Diagnosis
How does it work ? Pépinière *
•Formal processing of expression treesConclusion
3* in French : tree nursery
![Page 4: 1 The Pépite project Élisabeth Delozanne, Paris Universitas, UPMC D. Prévit, B. Grugeon, F. Chenevotot Automatic Multi-criteria Assessment of Open-Ended](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649d6b5503460f94a4a80d/html5/thumbnails/4.jpg)
Blandine
Validity Incorrect V3
Use of letters Incorrect L3
Translation Step-by-step with incorrect chains T4
Algebraic Expressions writing
Incorrect use of parentheses with memory of meaning
EA31
Justification By algebra using incorrect rules J3
![Page 5: 1 The Pépite project Élisabeth Delozanne, Paris Universitas, UPMC D. Prévit, B. Grugeon, F. Chenevotot Automatic Multi-criteria Assessment of Open-Ended](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649d6b5503460f94a4a80d/html5/thumbnails/5.jpg)
Aliou
Validity Incorrect V3
Use of letters No L5
Translation Step-by-step T2
Algebraic writing No EA?
Justification By example J2
![Page 6: 1 The Pépite project Élisabeth Delozanne, Paris Universitas, UPMC D. Prévit, B. Grugeon, F. Chenevotot Automatic Multi-criteria Assessment of Open-Ended](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649d6b5503460f94a4a80d/html5/thumbnails/6.jpg)
Definitions Diagnosis exercise
An exercise (statement and user interface) an analysis grid to assess every plausible solution
anticipated by experts Clone
A similar exercise • has the same kind of statement and user
interface• gives the same kind of information on students’
competence an analysis grid
• to assess every plausible solution • automatically generated by the system 6
![Page 7: 1 The Pépite project Élisabeth Delozanne, Paris Universitas, UPMC D. Prévit, B. Grugeon, F. Chenevotot Automatic Multi-criteria Assessment of Open-Ended](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649d6b5503460f94a4a80d/html5/thumbnails/7.jpg)
PépiGen A system to clone the Pépite diagnosis tool An author (a teacher)
Chooses an exercise to be cloned Enters the statement of the clone
PépiGen generates The student’s interface Each plausible solution (correct or incorrect) and
its assessment on several dimensions
7
![Page 8: 1 The Pépite project Élisabeth Delozanne, Paris Universitas, UPMC D. Prévit, B. Grugeon, F. Chenevotot Automatic Multi-criteria Assessment of Open-Ended](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649d6b5503460f94a4a80d/html5/thumbnails/8.jpg)
The Author’s interface
8
![Page 9: 1 The Pépite project Élisabeth Delozanne, Paris Universitas, UPMC D. Prévit, B. Grugeon, F. Chenevotot Automatic Multi-criteria Assessment of Open-Ended](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649d6b5503460f94a4a80d/html5/thumbnails/9.jpg)
The Student’s interface
9
![Page 10: 1 The Pépite project Élisabeth Delozanne, Paris Universitas, UPMC D. Prévit, B. Grugeon, F. Chenevotot Automatic Multi-criteria Assessment of Open-Ended](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649d6b5503460f94a4a80d/html5/thumbnails/10.jpg)
The Automatic Diagnostic
10
![Page 11: 1 The Pépite project Élisabeth Delozanne, Paris Universitas, UPMC D. Prévit, B. Grugeon, F. Chenevotot Automatic Multi-criteria Assessment of Open-Ended](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649d6b5503460f94a4a80d/html5/thumbnails/11.jpg)
Outline An introductory example
Pépite : a specific diagnosis tool PépiGen : a system to clone Pépite
How does it work ? Pépinière Expanding the tree of plausible steps of
correct and incorrect algebraic transformations Walking through the tree to anticipate different
solutions and their assessment Diagnosing the student’s reasoning
Conclusion
11
![Page 12: 1 The Pépite project Élisabeth Delozanne, Paris Universitas, UPMC D. Prévit, B. Grugeon, F. Chenevotot Automatic Multi-criteria Assessment of Open-Ended](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649d6b5503460f94a4a80d/html5/thumbnails/12.jpg)
Plausible steps
(x+6)*3-3x
-2x+18
18
3x+18-3x
x*3+6*3-3x
x+6*3-3x
3x+18-3x
18x
21x-3x
R1
R3
R3
R2
R4
R3
R3
21x-3x
18x
R5
Correct rules
R1 : (A+B)C AC+BC
R3 : AB+AC A(B+C)
R2: (A+B)C A+BC
R4: AB+C B(A+C)
R5: A+B*C (A+B)*C
Incorrect rules
18
R3
R4
V1,EA1 V3,EA42 V3,EA31
V3,EA31EA42
V3,EA32
![Page 13: 1 The Pépite project Élisabeth Delozanne, Paris Universitas, UPMC D. Prévit, B. Grugeon, F. Chenevotot Automatic Multi-criteria Assessment of Open-Ended](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649d6b5503460f94a4a80d/html5/thumbnails/13.jpg)
Analysis grid generation PépiGen
1. sends the algebraic expression to Pépinière that returns a tree of plausible steps• Validity and Algebraic Expression Writing
2. completes the plausible solutions set with• Non optimal algebraic
3. completes each solution assessment on the 5 dimensions• V, EA, L, T, J
4. saves each algebraic solution and its assessment• XML file : solution analysis grid
Note : arithmetic reasonings are analyzed by the
diagnosis system 13
![Page 14: 1 The Pépite project Élisabeth Delozanne, Paris Universitas, UPMC D. Prévit, B. Grugeon, F. Chenevotot Automatic Multi-criteria Assessment of Open-Ended](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649d6b5503460f94a4a80d/html5/thumbnails/14.jpg)
Analysis grid (extract)<UnexpectedCorrectSolutions> (…) <Comment>Algebraic proof ; the student interprets
the statement as an equation</Comment> <Solution>
<Assessment>V2,EA1,L1,T1,J1</Asssessment> <Expression>(x+6)*3-3*x = 18</Expression><Expression>x*3+6*3-3*x = 18</Expression><Rule>C,3</Rule><Expression>x*3+18-3*x = 18</Expression> <Expression>18 = 18</Expression>
</Solution> </UnexpectedCorrectSolution>
14
![Page 15: 1 The Pépite project Élisabeth Delozanne, Paris Universitas, UPMC D. Prévit, B. Grugeon, F. Chenevotot Automatic Multi-criteria Assessment of Open-Ended](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649d6b5503460f94a4a80d/html5/thumbnails/15.jpg)
Automatic diagnosis
XM L
Diagnosis system
loaded
ExpressionsTree
processor
Pépinière
Equivalent expression
tree ?True/False
save
XM L
Analysis grid
XM L
Student’s reasonin
g
loaded
Student’s reasoning+ assessment
![Page 16: 1 The Pépite project Élisabeth Delozanne, Paris Universitas, UPMC D. Prévit, B. Grugeon, F. Chenevotot Automatic Multi-criteria Assessment of Open-Ended](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649d6b5503460f94a4a80d/html5/thumbnails/16.jpg)
Diagnosis algorithm Numerical or algebraic approach? Loop on each expression of the student’s reasonning
Build the expression tree (ST) Loop on each Plausible solution in the analysis grid
• Build the expression tree (PT)• If numerical approach
- substitute the numerical value in PT• If ST PT
- keep :PT, the rule and the comment and stop At the end
walk through PT to set up the final assessment save the final assessment, the comment and the
applied rules16
![Page 17: 1 The Pépite project Élisabeth Delozanne, Paris Universitas, UPMC D. Prévit, B. Grugeon, F. Chenevotot Automatic Multi-criteria Assessment of Open-Ended](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649d6b5503460f94a4a80d/html5/thumbnails/17.jpg)
Results and tests On going work A demonstration prototype implements a complex
exercise cloning Authoring clones Solving Diagnosing
Preliminary Tests assessment of a corpus of 141 students’ solutions
• Multi-step reasoning• Multiple equivalent reasonings
3 teachers tested it in the lab
17
![Page 18: 1 The Pépite project Élisabeth Delozanne, Paris Universitas, UPMC D. Prévit, B. Grugeon, F. Chenevotot Automatic Multi-criteria Assessment of Open-Ended](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649d6b5503460f94a4a80d/html5/thumbnails/18.jpg)
Discussion Diagnosis
compared with model tracing≈Tree of plausible steps: correct and incorrect
rules≠Emphasis : whole reasoning/step-by-step≠Several student’s types of reasoning derived
from a single solution branch≠Multidimensional assessment
Authoring Filling template forms
- Limited to specified exercisesAutomatic multidimensional diagnosis validated
by expertsNo programming, no modeling for teachers 18
![Page 19: 1 The Pépite project Élisabeth Delozanne, Paris Universitas, UPMC D. Prévit, B. Grugeon, F. Chenevotot Automatic Multi-criteria Assessment of Open-Ended](https://reader035.vdocuments.site/reader035/viewer/2022081519/56649d6b5503460f94a4a80d/html5/thumbnails/19.jpg)
Automatic Multi-criteria AssessmentOur proposal
Teachers clone a diagnosis tool previously designed by experts
The cloning process relies on• A preliminary educational study in the
domain• An implementation of templates of
diagnosis exercises• A specific application to analyze
reasonings that are not pre-formatedDemo: Friday afternoonhttp://pepite.univ-lemans.fr 19