evidence-based practice chapter 3 ken koedinger based on slides from ruth clark 1
TRANSCRIPT
![Page 1: Evidence-based Practice Chapter 3 Ken Koedinger Based on slides from Ruth Clark 1](https://reader036.vdocuments.site/reader036/viewer/2022062321/56649ef95503460f94c0b565/html5/thumbnails/1.jpg)
Evidence-basedPracticeChapter 3
Ken KoedingerBased on slides from Ruth Clark
1
![Page 2: Evidence-based Practice Chapter 3 Ken Koedinger Based on slides from Ruth Clark 1](https://reader036.vdocuments.site/reader036/viewer/2022062321/56649ef95503460f94c0b565/html5/thumbnails/2.jpg)
2
Chapter 3 objectives
• Apply evidence-based practice • Identify
– research approaches to study instructional effectiveness
– features of good experiments– reasons for no effect– research relevant to your organization
• Interpret significance in statistics
![Page 3: Evidence-based Practice Chapter 3 Ken Koedinger Based on slides from Ruth Clark 1](https://reader036.vdocuments.site/reader036/viewer/2022062321/56649ef95503460f94c0b565/html5/thumbnails/3.jpg)
1.Know what to do AND WHY
2.Factor evidence into educational decisions
3.Participate in a community of practice
Features of a professional learning engineer
![Page 4: Evidence-based Practice Chapter 3 Ken Koedinger Based on slides from Ruth Clark 1](https://reader036.vdocuments.site/reader036/viewer/2022062321/56649ef95503460f94c0b565/html5/thumbnails/4.jpg)
Evidence
Politics
IdeologyFads
OpinionsDesign
Decisions
Sources for e-learning design decisions
![Page 5: Evidence-based Practice Chapter 3 Ken Koedinger Based on slides from Ruth Clark 1](https://reader036.vdocuments.site/reader036/viewer/2022062321/56649ef95503460f94c0b565/html5/thumbnails/5.jpg)
Research Question Example Research Method
What works? Does an instructional method cause learning?
Experimental comparison
When does it work? Does an instructional method work better for certain learners or environments?
Factorial experimental comparison
How does it work? What learning processes determine the effectiveness of an instructional method
ObservationInterview
Three roads to instructional research
![Page 6: Evidence-based Practice Chapter 3 Ken Koedinger Based on slides from Ruth Clark 1](https://reader036.vdocuments.site/reader036/viewer/2022062321/56649ef95503460f94c0b565/html5/thumbnails/6.jpg)
VALIDTEST
Mean = 80% Mean = 75%
Random Assignment
Standard deviation = 5 Standard deviation = 8
Treatment 1: Text + Graphics
Treatment 2: Text Only
Sample size = 25 in each version
Experimental comparison
![Page 7: Evidence-based Practice Chapter 3 Ken Koedinger Based on slides from Ruth Clark 1](https://reader036.vdocuments.site/reader036/viewer/2022062321/56649ef95503460f94c0b565/html5/thumbnails/7.jpg)
Graphics No GraphicsMen
Women
Factorial experimental comparison
Examples ProblemsLow VariabilityHigh Variability
![Page 8: Evidence-based Practice Chapter 3 Ken Koedinger Based on slides from Ruth Clark 1](https://reader036.vdocuments.site/reader036/viewer/2022062321/56649ef95503460f94c0b565/html5/thumbnails/8.jpg)
Examples of Process ObservationEd Tech Logs
Others: video, think aloud, physiological measures, brain imaging …
Eye Tracking
Student
Step (Item)
Skill (KC)
Opportunity
Success
S1prob1ste
p1Circle-
area 1 0
S1prob2ste
p1Circle-
area 2 1
S1prob2ste
p2Square-
area 1 1
S1prob2ste
p3 Compose 1 0
S1prob3ste
p1Circle-
area 3 0
![Page 9: Evidence-based Practice Chapter 3 Ken Koedinger Based on slides from Ruth Clark 1](https://reader036.vdocuments.site/reader036/viewer/2022062321/56649ef95503460f94c0b565/html5/thumbnails/9.jpg)
No effect
Graphics No Graphics
Test
Sco
res
![Page 10: Evidence-based Practice Chapter 3 Ken Koedinger Based on slides from Ruth Clark 1](https://reader036.vdocuments.site/reader036/viewer/2022062321/56649ef95503460f94c0b565/html5/thumbnails/10.jpg)
Reasons for no effect?
10
![Page 11: Evidence-based Practice Chapter 3 Ken Koedinger Based on slides from Ruth Clark 1](https://reader036.vdocuments.site/reader036/viewer/2022062321/56649ef95503460f94c0b565/html5/thumbnails/11.jpg)
Reasons for no effect
• instructional treatment did not influence learning• insufficient number of learners• learning measure is not sensitive enough to detect
differences in learning• treatment & control groups are not different enough
from each other• learning materials were too easy for all learners so
no additional treatment was helpful• other variables confounded the effects of the
treatment
![Page 12: Evidence-based Practice Chapter 3 Ken Koedinger Based on slides from Ruth Clark 1](https://reader036.vdocuments.site/reader036/viewer/2022062321/56649ef95503460f94c0b565/html5/thumbnails/12.jpg)
Num
ber
of S
tude
nts
Test Scores
80 90 100
Lesson withMusicMean = 80%
Lesson withoutMusicMean= 90%
Means for test and control groups
![Page 13: Evidence-based Practice Chapter 3 Ken Koedinger Based on slides from Ruth Clark 1](https://reader036.vdocuments.site/reader036/viewer/2022062321/56649ef95503460f94c0b565/html5/thumbnails/13.jpg)
Num
ber
of S
tude
nts
Test Scores
80 90 100
Lesson withMusicMean = 80%
Lesson withoutMusicMean= 90%
Standard
Deviation = 10
Standard
Deviation = 10
Means and standard deviations
![Page 14: Evidence-based Practice Chapter 3 Ken Koedinger Based on slides from Ruth Clark 1](https://reader036.vdocuments.site/reader036/viewer/2022062321/56649ef95503460f94c0b565/html5/thumbnails/14.jpg)
Statistical significance
The probability that the results could haveoccurred by chance.
p < .05
![Page 15: Evidence-based Practice Chapter 3 Ken Koedinger Based on slides from Ruth Clark 1](https://reader036.vdocuments.site/reader036/viewer/2022062321/56649ef95503460f94c0b565/html5/thumbnails/15.jpg)
Num
ber
of S
tude
nts
Test Scores
80 90 100
Effect Size = 90-80 = 1 10
Lesson withMusicMean = 80%
Lesson withoutMusicMean= 90%
Standard
Deviation = 10
Standard
Deviation = 10
Effect size
![Page 16: Evidence-based Practice Chapter 3 Ken Koedinger Based on slides from Ruth Clark 1](https://reader036.vdocuments.site/reader036/viewer/2022062321/56649ef95503460f94c0b565/html5/thumbnails/16.jpg)
![Page 17: Evidence-based Practice Chapter 3 Ken Koedinger Based on slides from Ruth Clark 1](https://reader036.vdocuments.site/reader036/viewer/2022062321/56649ef95503460f94c0b565/html5/thumbnails/17.jpg)
Research relevance
Similarities of the learners to your learners.
![Page 18: Evidence-based Practice Chapter 3 Ken Koedinger Based on slides from Ruth Clark 1](https://reader036.vdocuments.site/reader036/viewer/2022062321/56649ef95503460f94c0b565/html5/thumbnails/18.jpg)
Features of a good experimental design (starting with most important)
Test group Control group Representative sample Post test Pre test Random assignment
Research relevance
![Page 19: Evidence-based Practice Chapter 3 Ken Koedinger Based on slides from Ruth Clark 1](https://reader036.vdocuments.site/reader036/viewer/2022062321/56649ef95503460f94c0b565/html5/thumbnails/19.jpg)
Replication
External validity: Does principle generalize to different content, students, context, etc.?
Review ofEducational
Research
Research relevance
![Page 20: Evidence-based Practice Chapter 3 Ken Koedinger Based on slides from Ruth Clark 1](https://reader036.vdocuments.site/reader036/viewer/2022062321/56649ef95503460f94c0b565/html5/thumbnails/20.jpg)
In most contexts, it is what a person can do, not what they say that really matters.
Learning Measures
Recall
Or
Application?
Research Relevance
![Page 21: Evidence-based Practice Chapter 3 Ken Koedinger Based on slides from Ruth Clark 1](https://reader036.vdocuments.site/reader036/viewer/2022062321/56649ef95503460f94c0b565/html5/thumbnails/21.jpg)
Significance? p < .05
Effect Size ≥ .5
Research Relevance
Nothing magical about these numbers!• Poor treatments can look good by chance
– P=.05 => 1 in 20 chance that treatment just happened, by chance, to be better.
• Good treatments may not– Small p & effect size values can be associated with reliable &
valuable instructional programs • Look for results across multiple contexts (external
validity)
![Page 22: Evidence-based Practice Chapter 3 Ken Koedinger Based on slides from Ruth Clark 1](https://reader036.vdocuments.site/reader036/viewer/2022062321/56649ef95503460f94c0b565/html5/thumbnails/22.jpg)
KLI learning processes & instructional principles
22
![Page 23: Evidence-based Practice Chapter 3 Ken Koedinger Based on slides from Ruth Clark 1](https://reader036.vdocuments.site/reader036/viewer/2022062321/56649ef95503460f94c0b565/html5/thumbnails/23.jpg)
KLI: More complex learning processes are needed for more complex knowledge
![Page 24: Evidence-based Practice Chapter 3 Ken Koedinger Based on slides from Ruth Clark 1](https://reader036.vdocuments.site/reader036/viewer/2022062321/56649ef95503460f94c0b565/html5/thumbnails/24.jpg)
Instructional Principles
![Page 25: Evidence-based Practice Chapter 3 Ken Koedinger Based on slides from Ruth Clark 1](https://reader036.vdocuments.site/reader036/viewer/2022062321/56649ef95503460f94c0b565/html5/thumbnails/25.jpg)
Can interactive tutoring of rule KCs be improved by adding examples?
• No by “desirable difficulties” & testing effect– Eliciting “retrieval practice” is better when students succeed– Feedback provides examples when they do not
• Yes by cognitive load theory & worked examples– Examples support induction & deeper feature search – Early problems introduce load => shallow processing & less
attention to example-based feedback
• Test with lab & in vivo experiments …
![Page 26: Evidence-based Practice Chapter 3 Ken Koedinger Based on slides from Ruth Clark 1](https://reader036.vdocuments.site/reader036/viewer/2022062321/56649ef95503460f94c0b565/html5/thumbnails/26.jpg)
Ecological Control = Standard Cognitive Tutor Students solve problems step-by-step & explain
![Page 27: Evidence-based Practice Chapter 3 Ken Koedinger Based on slides from Ruth Clark 1](https://reader036.vdocuments.site/reader036/viewer/2022062321/56649ef95503460f94c0b565/html5/thumbnails/27.jpg)
Worked out steps with calculation shown by Tutor
Treatments: 1) Half of steps are given as examples
2) Adaptive fading of examples into problems
Student still has to self explain worked out step
![Page 28: Evidence-based Practice Chapter 3 Ken Koedinger Based on slides from Ruth Clark 1](https://reader036.vdocuments.site/reader036/viewer/2022062321/56649ef95503460f94c0b565/html5/thumbnails/28.jpg)
d = .73 *
Lab experiment: Adding examples yields better conceptual transfer & 20% less instructional time
![Page 29: Evidence-based Practice Chapter 3 Ken Koedinger Based on slides from Ruth Clark 1](https://reader036.vdocuments.site/reader036/viewer/2022062321/56649ef95503460f94c0b565/html5/thumbnails/29.jpg)
Course-based “in vivo” experiment
Result is robust in classroom environment: adaptive fading examples > problem solving
problem solving fixed fading adaptive fading0
4
8
12
Delayed Post-Test
experimental condition
perf
orm
ance
in %
![Page 30: Evidence-based Practice Chapter 3 Ken Koedinger Based on slides from Ruth Clark 1](https://reader036.vdocuments.site/reader036/viewer/2022062321/56649ef95503460f94c0b565/html5/thumbnails/30.jpg)
30
Similar results in multiple contexts
• LearnLab studies in Geometry, Algebra, Chemistry– Consistent reduction in time to learn– Mixed benefits on robust learning measures
![Page 31: Evidence-based Practice Chapter 3 Ken Koedinger Based on slides from Ruth Clark 1](https://reader036.vdocuments.site/reader036/viewer/2022062321/56649ef95503460f94c0b565/html5/thumbnails/31.jpg)
“KLI dependency” explanation: Target Knowledge => Learning processes => Which kinds of instruction are optimal
Worked examples
Worked examples
Testing effect
Testing effect
Eliciting recall supports
Aids fact learning, but suboptimal for rules
Many examples support
Aid rule learning, but suboptimal for facts
![Page 32: Evidence-based Practice Chapter 3 Ken Koedinger Based on slides from Ruth Clark 1](https://reader036.vdocuments.site/reader036/viewer/2022062321/56649ef95503460f94c0b565/html5/thumbnails/32.jpg)
Self-explanation prompts: Generally effective?
![Page 33: Evidence-based Practice Chapter 3 Ken Koedinger Based on slides from Ruth Clark 1](https://reader036.vdocuments.site/reader036/viewer/2022062321/56649ef95503460f94c0b565/html5/thumbnails/33.jpg)
Is prompting students to self-explain always effective?
Risks:• Efforts to verbalize may interfere with
implicit learning– E.g., verbal overshadowing (Schooler)
• Time spent in self-explanation may be better spent in practice with feedback– English article tutor (Wylie)
![Page 34: Evidence-based Practice Chapter 3 Ken Koedinger Based on slides from Ruth Clark 1](https://reader036.vdocuments.site/reader036/viewer/2022062321/56649ef95503460f94c0b565/html5/thumbnails/34.jpg)
KLI: Self-explanation is optimal for principles but not rules
Self-explain
Self-explain
Prompting students to self explain enhances
Supports verbal knowledge & rationale
Impedes non-verbal rule induction
![Page 35: Evidence-based Practice Chapter 3 Ken Koedinger Based on slides from Ruth Clark 1](https://reader036.vdocuments.site/reader036/viewer/2022062321/56649ef95503460f94c0b565/html5/thumbnails/35.jpg)
KLI Summary
• Fundamental causal chain: Changes in instruction yield changes in learning yield changes in knowledge yield changes in robust learning measures.
Observed Inferred
• Design process starts at the end– What is the knowledge students are to acquire?– What learning processes produce those kinds of KCs?– What instruction is optimal for those learning processes?
• Bottom line: Which instructional methods are effective depend on fit with knowledge goals