in-vivo research on learning

1

In-vivo research on learning

Charles Perfetti

PSLC Summer School 2009

PSLC summer school 2009 2

In-vivo experiments In Vitro

In Vivo


Features of in-vivo experiments in learning “On-Line” course? An Intelligent Tutoring System? A real class; real students; an

intervention that counts.


The value of in-vivo experiments in learning Noisy, uncontrolled environment Content of intervention is validated by

course goals So: Built in generalization to classroom

learning


Problems faced by an in-vivo researcher Noisy, uncontrolled environment As for your experiment:

Students have other things to do Instructors have other things to do


Examples of in-vivo studies Algebra, Physics, Chemistry, Geometry,

French, Chinese,English Some with computer tutors in major role

ITS Practice tutors

Some without tutors or tutors in minor role


Pre-requisites for an in-vivo experiment Knowledge components analysis Mapping of KCA to a learning or

instructional hypothesis Theory based Empirical precedent

Mapping instructional hypothesis to specific intervention

Knowledge Components vs. curriculum topic

Single Topic (Area) as unit

12 separate KCs as units

Enabled by Data Shop

Mapping a KCA onto an instructional hypothesis The case of Chinese characters


zao3

Whole Character = early morning

Radical = sun

Mapping an instructional hypothesis to an instructional intervention

Learning event space


11

Instructional Event Space

Learning Events

InstructionalEvents

AssessmentEvents

Performance

Explicit or implicitFocus on Valid FeaturesMake Knowledge AccessiblePromote Active ProcessingSchedule events effectivelyCoordinate multiple events

12

Knowledge Components Analysis 2 (+2) Knowledge Components:

1. the character as a whole; (plus its meaning)2. the radical that is part of the character (plus its meaning)

Two approaches based on this analysis (1) Dunlap, Liu, & Perfetti; (2) Pavlik

Two different Instructional Events manipulations

Illustrate 1 here: Feature focus

1. Learning meanings of Chinese characters

zao3

Whole Character = early morning

Radical = sun

13


Associate character form with meaning

AssessmentEvents

Performance

Whole Character

means x

Default (typical) Instructional event

Early morning

14

Associate radical with x’ and

whole character with x

Part of character means x’

AssessmentEvents

PerformanceDunlap et al: Instructional event manipulation: semantic radical instruction

Early morning

Highlighted radical = sun/day


15

Learning English Spelling (Background knowledge and feature focusing themes)

Transcriptions of Recorded Speaking Activities (Excluding Form Errors and Garble)

0%

20%

40%

60%

80%

100%

3 4 5 3 4 5 3 4 5 3 4 5 3 4

Arabic Chinese Korean Spanish Taiwanese

Proportion of Words in Each Coding Category

Correct

Vowel Error

Consonant Error

Multiple C/V Errors

Dunlap, Juffs, Friedline, Perfetti

KC analysis of English spelling phonology—orthography /breit/--brate /hiyl/--heel /hiyl/--heal So: phonology-semantics-orthography

16

17

Feature focusing interventions 130 students in levels 3 4, & 5 Interventions:

“Pure” feature focus: form only (pronunciation-spelling pairs)

Meaning mediated focus: form + meaning (pronunciation-meaning-spelling triads)

7 sessions, 30 minutes per session over 7 weeks

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

1.00

Pre-Test Post-Test

All 7 Sessions n = 12

Audio Dictation Task

Target Items

Control Items

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

1.00

Pre-Test Post-Test

All 7 Sessions n = 12

Spell Check Task

Target Items

Control Items

Dunlap, Juffs, Friedline, Perfetti


Control conditions for in-vivo experiments

Typical control conditions Existing classroom instruction

Textbook & exercise problems For cog tutors:

Another tutoring system Human tutoring

A control intervention; 2 plausible interventions—which is more effective


Learning Assessments1. Immediate Learning2. Long-term retention3. Transfer

Over content, form, testing situations

4. Accelerated Future Learning New content; learning measure

June 2009 NSF Site Visit


Learning Events

InstructionalEvents

AssessmentEvents

Performance

Explicit or implicitFocus on Valid FeaturesMake Knowledge AccessiblePromote Active ProcessingSchedule events effectivelyCoordinate multiple events

•Learning•Long term retention•Transfer•Accelerated future

learning


Transfer illustrated: Liu, Wang, Perfetti Chinese tone

perception study In-vivo study

Traditional classroom (not online) Materials from students’ textbook New materials each week for 8 weeks of term 1 Term 2 continued this, and added novel syllables unfamiliar to the

student 3 instructional conditions

tone number + pin yin, contour + pin-yin; contour only Hint system

(CTAT) Tutors presented materials in 3 different instructional interfaces, according to the 3 conditions

Data shop logged individual student data


Illustration of 2 conditions from Liu et al

shi


Data from Liu et al tone study

Lesson No.

20151050

0.8

0.6

0.4

0.2

0.0

20151050

0.8

0.6

0.4

0.2

0.0

Pinyin+Contour Pinyin+Number

Contour Only

01

Novel

Scatterplot of Pinyin+Conto, Pinyin+Numbe, Contour Only vs Lesson No.

Learning curves week-by-week


Multiple kinds of transfer Liu et al shows 2 kinds of materials

transfer Within term 1, learning sessions, each

syllable to be learned was different but familiar. So transfer of learning to familiar items

At second term, there were unfamiliar syllables. So transfer of learning to unfamiliar items. (Not so good.)


Example of acceleration of future learning (Min Chi & VanLehn)

First probability, then physics. During probability only, Half students taught an explicit strategy Half not taught a strategy (normal instruction)

Pre PostProbability Training

Sco

re

Pre PostPhysics Training

Sco

re

Accelerated future

learning

Ordinary transfer


Creating assessments General strategy:

Guided by cognitive task analysis (pre-test as well) including learning goals and specific knowledge components

Include some items from the pre-test Check for basic learning

Some items similar to training items Measures near-transfer

Some problems dissimilar to training problems Measures far-transfer


Mistakes to avoid in test design Tests that are

Too difficult Too easy Too long

Tests that Fail to represent instructed content

Missing content; over sampling from some content Depend too much on background knolwedge

Notice problems in test means

Notice variances


Interpreting test results as learning

Post-test in relation to pre-test. 2 strategies: ANOVA on

gain scores First check pre-test equivalence Not recommended if pre-tests not equivalent

Pre-test, post test as within-subjects variable (t-tests for non-independent samples)

ANCOVA. Post-tests scores are dependent variable; pre-test scores are co-variate


Plot learning results Bar graphs for instructional conditions

Differences due to conditions Learning Curves

Growth over time/instruction


Bar graphs (with error bars!)

0

10

20

30

40

50

CompleteParaphrase

CompleteSelf-explain

IncompleteParaphrase

IncompleteSelf-explain

Number of help requests


Learning Curves

Weekly sessions over 2 terms

Error rate

A final word on experiments In-vivo limitations

The role of (in-vitro) laboratory studies

QuickTime™ and a decompressor

are needed to see this picture.

41

The end

in-vivo research on learning

Documents