program evaluation luke’s perspective. ideal framework context: what needs to be done? (before)...

Program Evaluation

Luke’s Perspective

Ideal Framework

• Context: What needs to be done? (before)• Input: How should it be done? (before)• Process: Is it being done? (during)• Product: Is it succeeding? (during/after)

• Real world: We focus on the Product first, and then work backward.

• Example – NCLB / AYP

Case Study: New York City

• Purpose of the program evaluation: – Identify why schools are failing (NCLB/AYP

background)

• Expected program evaluation outcome: – list of findings and recommendations to remedy

the situation– organized by curriculum, assessment, and

professional development (arrived at through covisioning)

Mixed Method• Qualitative

– Stakeholders identified as district level, principals, teachers, and aides

– Stratified random interviews

– Complete stakeholder focus groups

– Key document review– Recorded, coded,

emergent themes identified, data analyzed, data summarized

• Quantitative– Teacher/classroom

observations (random mts)– Student performance data

(tests)– Surveys of Enacted

Curriculum (SEC) (next)

Review: Two Major Approaches

Qualitative

• Attempts to figure out why things are the way they are, to describe cultures, groups, events

• Microscopic look at phenomena

Quantitative

• Attempts to figure out the extent to which something is related to something else; to generalize to a larger population

• Wide-angle look at phenomena

In program evaluation we often want both, we want the specific, but also need the big picture

New York Exemplar

• SEC instrument and results (next slides)

• Sample methodology• Sample report• Sample data collection• Sample data analysis• Final report

• Use links on website for more examples

What is Program Evaluation

• In a program evaluation we are interested in evaluating outcomes - at the program level.

• We are not as interested in individual student performance as we are interested in how well a program drives student performance.

• In schools, student outcomes are usually aligned with goals, objectives, and performance standards.

• There are many paths to evaluating the relative worth of a program, but a nice way to conceptualize these alternative methods is on a continuum from absolutely qualitative to absolutely quantitative.

Program Evaluation for You

• Critical component of Program Evaluation is data triangulation toward some purpose

• You might choose to get info from…– Classroom curriculum (akin to action research)– Schoolwide program (behavior? Technology?)– District initiatives

Brainstorm

• (this isn’t a commitment, just a start)

• Take 5 mins and make a list…– What program might you be interested in

evaluating? (can be class, school, district)– What are the key variables?

Brainstorm

• (this isn’t a commitment, just a start)

• Take 5 mins and make a list…– What program might you be interested in

evaluating? (can be class, school, district)– What are the key variables?

– Share in groups of 2-3

Review

• Program evaluation is what it sounds like – evaluating something at the program level, and often calls for mixed methods

• No matter what you choose to evaluate, you will likely want different sources of data

Example Data Sources

• Performance• Attendance• Behavior• Fiscal• Professional development participation• Parent survey data

Conceptual Dimensions of Datasources of question making

1. Organization – How does the school function? Where do we put our money?

2. Demographics – Who are we? Who is here?3. Academics – How do students perform?4. Conation – What is school climate like?5. Behavior – How do students misbehave?6. Time – How does it change over time?

Conceptual Model of Data DimensionsOrganization

Conation

BehaviorAcademics

Time Demographics

Example of a 4 Dimensional Question

Demographics – who are the low SES students?Conation – which students like school?Academics – who scored above average in math?Organization – which students are in AP math?

Question: In terms of SES, how do above average math students who report liking school distribute in tracked math classes?

AD

C O

Program: School practice of tracking students


In terms of SES, how do above average math students who report liking school distribute in tracked math classes?

Basic Advanced AP

Low SES 91 48 26High SES 39 58 45

All of these students are above average math achievers who like school


Basic Advanced AP

Low SES 55.2 29.1 15.8

High SES 27.5 40.8 31.7

In terms of SES, how do above average math students who report liking school distribute in tracked math classes?

Low

High

High

HighLow

Low

0%

10%

20%

30%

40%

50%

60%

Basic Advanced AP

Ultimately, we will want to track this data over time

OrganizationDemographicsAcademicsConationBehaviorTime

Example

Question: Based on who we have as students and how they are being taught, what are the differences in the reading scores?


Program: Current teaching practice


Common Example

Question: Do different groups of students perform the same on standardized tests?


Program: District ‘closing the achievement gap’ initiative


Better Example

Question: Over the last five years, have different groups of students performed the same on standardized tests?


Program: District ‘closing the achievement gap’ initiative

Review


• 6 dimensions of data• 6 common data collection methods

Collecting Data

• Six common methods– Surveys (questionnaires)– Interviews (1:1)– Document review (examine an organization’s literature)– Observations (watch something happening at an organization)– Focus groups (group interviews)– Data mining (using data that has already been collected)

Break into 6 groups and identify advantages and disadvantages to each method.

I’ll present my ideas. Everyone we agree with you get a point – to a maximum of 6.

Collecting Data• Method: surveys, questionnaires, checklists• Purpose: when need to quickly and/or easily get lots of information from people in

a non threatening way• Advantages

– anonymous– inexpensive to administer– easy to compare and analyze– administer to many people / can get lots of data– many sample questionnaires already exist

• Disadvantages– might not get careful feedback– wording can bias client's responses– are impersonal– in surveys, may need sampling expertise– doesn't get full story – only the part you target

Collecting Data• Method: interviews• Purpose: when want to fully understand someone's impressions or experiences, or

learn more about their answers to questionnaires• Advantages

– get full range and depth of information– develops relationship with client– can be flexible with client

• Disadvantages– can take much time– can be hard to analyze and compare– can be costly– interviewer can bias client's responses

Collecting Data• Method: document review• Purpose: when want impression of how program operates without interrupting

the program; review of curriculum, texts, finances, memos, minutes, etc.• Advantages

– get comprehensive and historical information– doesn't interrupt program or routine in program– information already exists– few biases about information

• Disadvantages– often takes much time– info may be incomplete – you probably wont find everything– need to be quite clear about what looking for– not flexible means to get data; data restricted to what already exists

Collecting Data• Method: observation• Purpose: to gather accurate information about how a program actually operates• Advantages

– view operations of a school as they are actually occurring – real and accurate data!– can adapt to events as they occur

• Disadvantages– can be difficult to interpret seen behaviors– can be complex to categorize observations– Your presence can influence behaviors of program participants– can be expensive / time consuming

Collecting Data• Method: focus groups• Purpose: explore a topic in depth through group discussion, e.g., about reactions

to an experience or suggestion, understanding common complaints, etc.• Advantages

– quickly and reliably get common impressions – can be efficient way to get much range and depth of information in short time– can convey key information about programs

• Disadvantages– can be hard to analyze responses– need good facilitator for safety and closure– one or two people can monopolize discussion – steer the discussion– difficult to schedule lots of people together

Collecting Data• Method: data mining• Purpose: use extant data to form a picture• Advantages

– Data does not need collecting– Usually easy to analyze– Tends to be reliable

• Disadvantages– Human rights – permission issues when you use data for another purpose– Gaining access to data sometimes difficult– Aggregating data / merging data sets sometimes difficult

Review


• 6 dimensions of data• 6 common data collection methods• 3 perspectives when answering questions

3 Perspectives (Referents)• In measurement the question we often need to ask is,

to what will we compare our data?• There are three corresponding referents,

• This organization is not always perfect, but reminds us to address a referent in questioning and analysis

IndividualIndividual

NormNorm

CriterionCriterion

Norm Referenced

Criterion Referenced

Individually Referenced

5th grade individual student report - ORF

NormCriterionIndividual

Example

Question: Compared to last year, what are the differences in student reading scores because of attitudes related to whom students have as teachers?



Check for Understanding

Question: How does Luke like the new scheduling system in the middle school?


Check for Understanding

Team up, describe a norm, criterion, and individual reference for the following…

Review


• 6 dimensions of data• 6 common data collection methods• 3 perspectives when answering questions• The purpose leads to questions which become

refined over time

Program Evalation Questions

• Program evaluation begins with a broad purpose– What are you going to evaluate and why?

• Program evaluation then moves to more refined measureable questions– Dictates the research design (the method)– Guides the gathering of data– Usually dictates the kind of analyses– Constrains the kinds of conclusions that can be drawn

The Prog Eval has a purposePurpose• What is the state of

our curriculum in NYC? How is the curriculum broken? What are the good parts?

Questions are raisedPurpose• What is the state of

our curriculum? How is the curriculum broken? What are the good parts?

Questions• What is curriculum?• What is our curriculum?• What does a good

curriculum look like?

Questions are refinedPurpose• What is the state of

our curriculum? How is the curriculum broken? What are the good parts?

Questions• What is curriculum?• What is our curriculum?• What does a good

curriculum look like?

Measureable Questions: (examples)• At the elementary level, what is

the quality of the taught curriculum measured by the SEC to teachers?

• At the middle-school level, to what degree is the social studies curriculum aligned with provincial standards?

Local ExamplePurpose• Is the Coquitlam

Master’s Program (CMP) succeeding?

Questions Measureable Questions

What questions are relevant? How can we make them measureable?

Local ExamplePurpose• Is the Coquitlam

Master’s Program (CMP) succeeding?

Questions Measureable Questions:

Alone or team up, 10 minutes then share a little

Team scenarios (optional)

• Read your scenario• Identify the purpose and guiding questions for

the program evaluation• Identify measureable questions that cross

data dimensions • Identify where you will find the data, and to

what you will compare it (referents). • What data collection method will you use?

Alone or team up, 20 minutes then share

program evaluation luke’s perspective. ideal framework context: what needs to be done? (before)...

Documents