Leading (and Assessing) a Learning Intervention

IMPACT Lunch and Learn Session

August 6, 2014

Facilitated By

Ozgur Ekmekci, EdD

Interim Chair, Department of Clinical Research and Leadership

School of Medicine and Health Sciences

The George Washington University

1

Overview

Leading a Learning Intervention

In today's complex social environment, anyone who has been assigned the task of leading a learning

initiative has to approach the process in a systematic way in order to succeed. The crucial success

factor is often creating alignment amongst components that constitute a well-articulated learning

intervention proposal. This session is offered for those who would like to better understand the four

key questions that allow educational leaders to effectively envision, design, implement, and assess

a meaningful learning intervention, whether this intervention takes place in the classroom or in the

boardroom. 

Learning Objectives

•Discuss challenges associated with implementing a learning intervention

•Analyze major components of an effective learning intervention proposal

•Analyze four key questions that drive a learning intervention

•Discuss how assessment fits into the larger context of a learning intervention proposal

•Review six major assessment methods used in health sciences

2

• Learning Community Context (overview of the learners’ location, background, roles, and interests)

• The Problem Statement (current state of the learners – in measurable outcomes - and why this is a problem)

• The Statement of Significance (why it is important that the intervention be implemented and who suffers in what way, if not implemented)

• The Proposed Solution (envisioned learning objectives - including the statement of tangible, measurable objectives, which tie back to the problem statement - i.e., current state)

• Implementation Time Line (a graphical depiction of the implementation plan, with major milestones along the way)

• Evaluation Method(s) (how to measure progress against learning objectives and how it will be known when the learners finally get to the envisioned future state)

• Conclusion (summary of key messages with which to leave the audience)

ALIGNMENT

KEY SUCCESS FACTOR

Alignment of Major Components

3

Define Measurable Learning Objectives:

- Individual Level- Change in behavior- Change in perception- Change in performance

- Organizational Level- Change in outcomes

- Societal Level- Change in environment

What do I want to change, for whom, by how much, and by when? In other words, what will success look like?

Learning Intervention Time Line

Plan Learning Intervention:

- Identify learners - Select content- Develop materials- Design delivery method- Construct learning environment- Develop delivery schedule

How will I change what, for whom, by how much, and by when? In other words, how will I achieve success?

Design Learning Evaluation:

- Identify key variables to be measured at the appropriate level of analysis (i.e., individual, organizational, societal)

- Decide on the most appropriate method(s) for conducting formative and summative assessments that measure the key variables

- Determine frequency and timing for formative assessment and summative assessments

- Plan delivery format, medium, frequency, and timing of feedback to learners

- Select quantitative statistical methods that will help determine the effectiveness of intervention

How will I know what I am

changing, for whom, by how much, and by when? In other words, how will I monitor progress against set objectives that define success?

Implement Learning Intervention and Conduct Learning Evaluation:

- Deliver content- Assess learning - Provide feedback to learners- Analyze data using statistical methods- Determine effectiveness of learning intervention- Declare success!

Has what I wanted to change, for those whom I wanted to change it, actually changed by the amount I had wanted it to change?

Conduct Formative

Assessment

Conduct Summative Assessment

Review and Revise Content

Method or Schedule

Deliver Content

START

Provide Learner

Feedback

Provide Learner

Feedback

STOP

Delivery Complete

?

Conduct CAT

Determine Effectiveness of Intervention

YES

NO

Learning Intervention Timeline

4

PHASE I PHASE II PHASE III PHASE IV

Ekmekci, O. (2013). Being there: Establishing instructor presence in an online learning environment, Higher Education Studies. 3(1). 29-38.

Design Learning Initiative Evaluation

How will I know what I am changing, for whom, by how much, and by when? In other words, how will I monitor progress against set objectives that define success?

By asking this question, the leader will be able to identify key variables to be measured, decide on methods and frequency for formative and summative assessment, plan delivery format, medium, frequency, and timing of feedback to participants, and select statistical methods that will help determine the effectiveness of the initiative

Reflecting on Four Key Questions

5

Define Measurable Learning Objectives

What do I want to change, for whom, by how much, and by when? In other words, what will success look like?

By asking this question, the leader will be able to contemplate and quantify the envisioned type of change, its scope (in terms of breadth, depth, and timeline), and the level at which change is to take place (i.e., individual, organizational or societal) – which will help ability to monitor progress against objectives

PHASE IPHASE I

Plan Learning Initiative

How will I change what, for whom, by how much, and by when? In other words, how will I achieve success?

By asking this question, the leader will more easily identify participants, select content, develop materials, design delivery method, construct learning environment, and develop delivery schedule

PHASE IIPHASE II PHASE IIIPHASE III

Implement Learning Initiative & Conduct Evaluation

Has what I wanted to change, for those whom I wanted to change it, actually changed by the amount I had wanted it to change?

By asking this question, the leader will be able to effectively deliver content, assess progress, provide feedback to participants, and analyze data using statistical methods to ultimately determine how successful the initiative has been

PHASE IVPHASE IV

Ekmekci, O. (2013). Being there: Establishing instructor presence in an online learning environment, Higher Education Studies. 3(1). 29-38.

Assessment as a Process

6

Assessment is the ongoing process that allows one to:

•establish clear, measurable expected outcomes of learning;

•ensure learners have sufficient opportunities to achieve those outcomes;

•systematically gather, analyze, and interpret evidence to determine how well

learning matches expected outcomes of learning; and

•using the resulting information to understand and improve learning.

Downing, S.M. & Yudkowsky, R. eds. (2009). Assessment in health professions education. New York: Routledge. p. 4.

Threats to Validity

7

Construct Underrepresentation (CU)

Not measuring the right things: under-sampling of the achievement domain – such as tests that are too short to support legitimate inferences to the domain, trivial questions written at low-levels of the cognitive domain, or mismatch of sample to domain

Construct Irrelevant Variance (CIV)

Not measuring (the right) things in the right manner:erroneous inflation or deflation of test scores due to certain types of uncontrolled or systematic measurement error – such as poorly crafted test questions, rater bias, or environmental conditions

Downing, S.M. & Yudkowsky, R. eds. (2009). Assessment in health professions education. New York: Routledge. 

Levels of Assessment and Matching Methods

8

Behavior

Cognition

DOESRequires methods

that provide an assessment of routine “real” performance for

the learner

SHOWS HOWRequires methods that allow learner to demonstrate

the integration of knowledge and skills that will translateinto successful routine performance

KNOWS HOWRequires methods that provide an assessment of how well learner uses

knowledge in the acquisition, analysis, and interpretation of data and the development of an approach or plan

KNOWSRequires methods that provide an assessment of how well learner has acquired the fundamental knowledge

on which to build higher levels of knowledge and behavior

Written TestOral Examination

Observational Assessment

Performance Test

Simulation

Adapted from: Downing, S.M. & Yudkowsky, R. eds. (2009). Assessment in health professions education. New York: Routledge. 

Portfolio

Wrap Up

9

Questions?