Download - Evaluation In E Learning
Evaluation in E-learningEvaluation in E-learning(or “How can we know what works?”)(or “How can we know what works?”)
Mark JohnsonMark Johnson
Once upon a time…Once upon a time…
The ‘killer app’ interventionThe ‘killer app’ intervention
And now…?And now…?
Systems everywhere…Systems everywhere…
Questions…Questions…
How can we know anything works? (can How can we know anything works? (can we..?)we..?)
How can the whole sector benefit from How can the whole sector benefit from particular instances of good practice? (can particular instances of good practice? (can it..?)it..?)
What should we be doing technologically? What should we be doing technologically? (hasn’t it all been done..?)(hasn’t it all been done..?)
Is ‘the project’ the only way..?Is ‘the project’ the only way..?
Evaluation in the social worldEvaluation in the social world
Cause and EffectCause and EffectObjectivity and SubjectivityObjectivity and SubjectivityKnowledge and JudgementKnowledge and JudgementParticulars and UniversalsParticulars and Universals
… … what are the features of the ‘successful what are the features of the ‘successful project’project’
A history of causes…A history of causes…
Aristotelian cause: a cause of something Aristotelian cause: a cause of something real which is generated by the nature of real which is generated by the nature of thing itself (thing itself (generative causalitygenerative causality))
The elements of a substance tend to move The elements of a substance tend to move towards one another (i.e. gravity)towards one another (i.e. gravity)
GalileoGalileo
Galileo: the cause of the motion of planets Galileo: the cause of the motion of planets is not inherent in their nature, but due to is not inherent in their nature, but due to external ‘laws of motion’external ‘laws of motion’
Hume and Successionist CausalityHume and Successionist Causality
A cause is not real; it is inferred through A cause is not real; it is inferred through the conduct of scientific investigation into the conduct of scientific investigation into regular successive events (regular successive events (successionist successionist causalitycausality))
A sceptical approach to causality…A sceptical approach to causality…
But are causes ‘real’???But are causes ‘real’???
What is the cause of learning?What is the cause of learning?
Generative?Generative?Successionist?Successionist?Both?Both?
Objectivity?Objectivity? Various philosophical positions:Various philosophical positions:
Idealism: nothing is real (Berkeley) – just ideas in my Idealism: nothing is real (Berkeley) – just ideas in my mindmind
Post-modernism (Baudrillard, Derrida, Barthes)Post-modernism (Baudrillard, Derrida, Barthes) No objectivityNo objectivity
Empirical Realism: Reality can only be known through Empirical Realism: Reality can only be known through scientific experiment scientific experiment
Objectivity through experimentObjectivity through experiment Metaphysical Realism: reality is completely Metaphysical Realism: reality is completely
independent of conceptual schemesindependent of conceptual schemes Objectivity possible through correspondence of ideas with Objectivity possible through correspondence of ideas with
realityreality Critical Realism: Reality comprises generative Critical Realism: Reality comprises generative
mechanisms responsible for producing phenomena: mechanisms responsible for producing phenomena: transitive and intransive mechanisms.transitive and intransive mechanisms.
Objectivity possible through uncovering mechanismsObjectivity possible through uncovering mechanisms
Knowledge and JudgementKnowledge and Judgement
Knowledge: codified, rational, embodied, Knowledge: codified, rational, embodied, habitualhabitual
Judgement: ‘of the moment’, uncodified Judgement: ‘of the moment’, uncodified (uncodifiable??), habitual, instinctive(uncodifiable??), habitual, instinctiveUsually depends to some extent on codified Usually depends to some extent on codified
knowledgeknowledge
Universals and ParticularsUniversals and Particulars
The project and the particularThe project and the particularThe institution, sector (JISC), government, The institution, sector (JISC), government,
etc and the Universaletc and the UniversalThe particular instant and the judgement The particular instant and the judgement
of the individualof the individual ‘‘The project’ is the means by which work The project’ is the means by which work
in e-learning is progressed.in e-learning is progressed.How to go from the project to Universal How to go from the project to Universal
guidance for the sector…guidance for the sector…
Modes of reasoning in evaluationModes of reasoning in evaluation
P
P PP
Preality
Uknowledge
Induction (no theory approach)
Deduction (single theory approach)
P
P PP
P
U
U
UU
U
knowledge
outcomes
reality
retroduction
No shit, Sherlock!No shit, Sherlock!
Differences between induction, deduction Differences between induction, deduction and retroduction are subtleand retroduction are subtle
Retroduction attempts a multi-theory, Retroduction attempts a multi-theory, multi-methodology approach. Attempts not multi-methodology approach. Attempts not to be constrained by existing theory, to be constrained by existing theory, culture, etc..culture, etc..
Much like the work of a detective…Much like the work of a detective…
Context, Mechanism, OutcomeContext, Mechanism, Outcome
What outcome do you observe?What outcome do you observe?What mechanisms do you think might What mechanisms do you think might
produce that outcome?produce that outcome?Within what context do the mechanisms Within what context do the mechanisms
operate?operate?
Mechanisms…Mechanisms…
Cumulating Context, Mechanisms Cumulating Context, Mechanisms and Outcomes…and Outcomes…
Theory
Data
abstraction specification
C M O C M O C M O C M O
C M O C M O C M O
C M O C M O C M O
C M O Programmes
Middle-range theory
O OO Project outcomes
projects
C M O Methodology
Understanding mechanisms in Understanding mechanisms in educationeducation
Educational contexts, learning contextsEducational contexts, learning contextsEducational mechanisms, learning Educational mechanisms, learning
mechanismsmechanismsEducational outcomes, learning outcomesEducational outcomes, learning outcomes
Understanding Projects in e-Understanding Projects in e-LearningLearning
Projects contribute to context, mechanism Projects contribute to context, mechanism and outcomesand outcomes (as do the technologies of projects)(as do the technologies of projects)
Challenge is to describe how the new Challenge is to describe how the new relationships between context, mechanism relationships between context, mechanism and outcome are affected by a projectand outcome are affected by a project
ConclusionConclusion The challenge of evaluation is not necessarily The challenge of evaluation is not necessarily
finding out what works, but examining what does finding out what works, but examining what does and doesn’t work as a way of understanding the and doesn’t work as a way of understanding the mechanisms of education, and planning for mechanisms of education, and planning for future viability.future viability.
The understanding of the institution is crucial: The understanding of the institution is crucial: Realistic Evaluation focuses on understanding Realistic Evaluation focuses on understanding the institution. Positivist evaluation judges the institution. Positivist evaluation judges projects on a pre-given understanding of the projects on a pre-given understanding of the institution.institution.
Through cumulating realist descriptions, Through cumulating realist descriptions, objectivity is possible.objectivity is possible.