Data-Based Decision Making, Assessment for Learning and Diagnostic

Testing in Formative AssessmentAEA-Europe 2013 Paris

Fabienne Van der Kleij Jorine Vermeulen Kim Schildkamp

Theo Eggen

Formative assessment

…lack of a uniform definition

Formative assessment is an umbrella term

informed teaching

peer assessment

feedbackresponding to data

progress

reflectionauthentic

classroom dialogues

student ownership

diagnosis

observationrubrics

understandingattaining standards

self-regulation

targeted instruction

Evaluation or assessment

Different aggregation levels

Formative evaluation

the use of assessment data to make decisions concerning the quality of education at a higher aggregation level than the level of the learner or the class

Formative assessment

concerns decisions at the levels of the learner and the class to accommodate the pupils’ individual educational needs

Three approaches to formative assessment

Data-based decision making

DBDMAssessment for learning

AfL

Diagnostic testing

DT

Attaining results and targets

Focus on the quality of the

learning process

Detailed insights into learning

processes and outcomes

• Educational practice implements a mix of these approaches.

• Terminology and definitions are inappropriately used interchangeably theoretical ambiguity.

• Discussions tend to revolve around finding evidence of what works. “quick fixes” (Elwood, 2006).

• “Definition is important because if we can’t clearly define an innovation, we can’t meaningfully document its effectiveness” (Bennett, 2011, p. 8).

• Need for prescriptions on why, how, and when assessment should be used by learners, teachers, and schools to enhance learning.

Three approaches to FA

… lessons from the literature on formative assessment

• Initial exploration: Recognise the fundamental differences amongst these approaches blend these approaches in a meaningful way.

• Contribute to a more coherent research agenda.

Note: We do not intend to make any claims about which assessment approach is most effective.

Aims and scope

Comparative theoretical study

• What are the similarities and differences in the theoretical underpinnings of DBDM, AfL, and DT?

• What are the consequences of these similarities and differences for implementing DBDM, AfL, and DT in educational practice?

Research questions

Starting point for implementing a system-wide formative assessment approach: An understanding of the learning theories behind currently dominant approaches (Elwood, 2006)

• Neo-behaviourism• Cognitivism• Meta-cognitivism• Social cultural theory• (Social) constructivism

Learning theories and feedback

• Neo-behaviourism: Introduction of the formative concept. memorisation of facts, feedback is intended to reinforce correct recall of facts.

• Cognitivism: Highlights information processing and knowledge representation. Feedback is intended to correct incorrect responses.

• Meta-cognitivism: Emphasis on learning how to learn and regulating the learning process. Feedback is about how to learn, rather than about what to learn.

• Social cultural theory: Feedback in the form of scaffolding (ZPD) for acquiring knowledge and skills, focus on what students are able to learn.

• (Social) constructivism: Learning is a cyclic process in which new knowledge and skills are continuously built on prior ones. Active engagement, realistic contexts, learning in interaction, frequent elaborated feedback.

Learning theories in a nutshell

Theoretical underpinnings and implementation

Data-based decision making

DBDMAssessment for learning

AfL

Diagnostic testing

DT

• Schildkamp and Kuiper (2010): "systematically analyzing existing data sources within the school, applying outcomes of analyses to innovate teaching, curricula, and school performance, and, implementing (e.g., genuine improvement actions) and evaluating these innovations" (p. 482).

• Data in the context of schools is "information that is systematically collected and organized to represent some aspect of schooling" (Lai & Schildkamp, 2013, p. 10).

e.g., assessment results, student background characteristics, questionnaire results.

Theoretical underpinnings

DBDM

Process of data use in DBDM

Feedback loop

Data use

Identify data

Collect data

Analyse data

Interpret data

• Early initiatives: Based on neo-behaviourism and cognitivism.

• Move towards social cultural theory and constructivism.• DBDM for formative assessment:

Set learning goals, assess and monitor student achievement, and if necessary, adjust instruction.

• DBDM for formative evaluation Policy development, school improvement planning,

teacher development, and monitoring the implementation of the school's goals.

DBDM

Learning theories and processes

• Aggregation levelSchool, class, and student levels.

• Assessment methodsObjective data from standardised tests, e.g., from a PMS. Formally gathered data. Less frequently available. + Data from standardised assessments and (structured) observations.

• Feedback loopsFeedback based on assessment data. Educators transform data into actions for educational improvement. Length improvement cycles and feedback loops varies. Retroactive in nature.

Implementation of DBDMAggregation Level, Assessment Methods, and Feedback Loops

Theoretical underpinnings and implementation

Data-based decision making

DBDMAssessment for learning

AfL

Diagnostic testing

DT

• ARG "… the process of seeking and interpreting evidence for use by learners and their teachers to decide where the learners are in their learning, where they need to go and how best to get there" (2002, p. 2).

• Klenowski (2009) "part of everyday practice by students, teachers and peers that seeks, reflects upon and responds to information from dialogue, demonstration and observation in ways that enhance ongoing learning" (p. 264).

• Hargreaves (2005): two approaches within AfL

Theoretical underpinnings

AfL

Measurement Inquiry

• Underpinned by meta-cognitivism, social cultural theory, and social constructivism.

• Learning is a social activity; learning occurs through interaction.

• Students have a central role in the learning process; active participation.

• Increase learner autonomy, motivation, and reflection, by facilitating an inquiry-oriented and interactive classroom climate.

AfL

Learning theories and processes

• Aggregation levelClassroom and student levels.

• Assessment methodsPaper-and-pencil tests, dialogues, practical demonstrations, portfolios, peer assessment, or self-assessment. Qualitative & quantitative. Planned & unplanned. Formal & informal. Assessment quality actionable information for formative purposes over the short term.

• Feedback loopsContinuous dialogues and feedback loops, (immediate) feedback to direct further learning.Assessments integrated into the learning process, feedback loops are usually short. continuous adaptation.Interactive, sometimes retroactive or proactive.

Implementation of AfLAggregation Level, Assessment Methods, and Feedback Loops

Theoretical underpinnings and implementation

Data-based decision making

DBDMAssessment for learning

AfL

Diagnostic testing

DT

• How a task is solved is indicative of the developmental stage of the learner.

• Identify the learner's reasoning styles, and skipped or wrongly executed procedural steps caused by misconceptions and prior knowledge.

• Principles from cognitive psychology, subject pedagogy, and learning theories are combined. inferences about student learning based on a student's response patterns.

Theoretical underpinnings

DT

• Underpinned by principles from cognitivism.• Vygotsky's social cultural learning theory ZPD.

• Identify the learner's developmental stages by obtaining action-oriented, fine-grained assessment data (process data).

• Cognitive theories are used to interpret process data to identify misconceptions and knowledge associated with the learner's developmental stage.

• Small-grain size of the measurements in DT exceptionally useful for formative purposes.

DT

Learning theories and processes

• Aggregation levelStudent level.

• Assessment methods

Assessment tasks make inferences about how task behaviour relates to student thinking.

Inferential chain from empirical knowledge from information processing theories, cognitive psychology, and learning trajectories.

Assessment results: indicative of the developmental stage.

Use of CAT and dynamic assessments.• Feedback loops

Short feedback loops in are preferred.

Retroactive, sometimes interactive or proactive.

Implementation of DTAggregation Level, Assessment Methods, and Feedback Loops

Comparison of the three approaches

Theoretical underpinnings of DBDM, Afl, and DT

Approach

Theoretical aspect DBDM AfL DT

Learning theories

Neo-behaviourism Social cultural

theory (Social)

constructivism

Meta-cognitivism Social cultural theory Social constructivism

Cognitivism Social cultural theory

Goals Improve the quality of education and the quality of instruction by using data to monitor and steer practices to reach intended goals (e.g., increased student achievement).

Improve the quality of the learning process by engaging learners to evaluate and reflect on their own learning and steering the learning process through continuous feedback.

Collect fine-grained data about a student’s zone of proximal development, prior knowledge, and reasoning styles that can inform decisions on adapting the learning environment to the learner’s needs.

• Aggregation levels

• Assessment methodsDiversion in the goals: large variety in assessment methods associated with the nature of data, purposes and stakes regarding the use of these data.

• Feedback loopsTake place at various levels and frequencies.

Comparison of the three approaches

Implementation of DBDM, AfL, and DT

Student Class School

DBDM

AfL

DT

Assessment

Evaluation

• Original theoretical underpinnings of the approaches differ.

• Various assessment methods that are underpinned by different learning theories are needed to fully grasp the complexity of learning at all levels.

• Integrating the three assessment approaches can lead to more valid formative decisions.

• Potential to enable school staff to continuously provide feedback at the school, class, and individual levels, to guide and enhance student learning.

Conclusion

Differences, similarities, and possibilities

• DBDM: School, class and student levels: Monitor learning goals, group students, improve the quality of education. Feedback loops extend over a longer time span.

• AfL: Class and individual levels: Improve quality of the learning process by engaging learners. Continuous feedback in short cycles.

• DT: individual level: Collect fine-grained data that can inform decisions on adapting the learning environment. Feedback loops occur irregularly; depends on learner’s needs.

• At different points in the education process, retroactive, interactive, or proactive feedback loops can be used to optimise students’ learning processes.

Conclusion (continued)

Blending the approaches

Towards a balanced assessment system

Alignment of formative with summative practices

“The field will need to marry information from the different assessments to produce a union that a

stronger than any of the parts in isolation” (O’Malley, Lai, McClarty, & Way, 2013, p. 146).

Thank you for your attention!

or

f.m.vanderkleij@utwente.nl

fabienne.vanderkleij@cito.nl