ASC 607Seminar in Program Evaluation

Sometimes, you get it right.

Analysis

Action

Change

The big three…

Evaluations analyze evidence about a program’s effectiveness by examining strengths and weaknesses.

Using the weaknesses as a framework to guide future work, evaluations develop concrete action plans to remedy weaknesses.

Action steps lead to new learnings in the organization…all change is learning.

Session Outcomes

Frame every decision with evidence.

1

• Self grading• Piagetian programs• Providing formative evaluation• Acceleration• Microteaching• Classroom behavior• Comprehensive interventions for

learning disabled students• Teacher clarity• Reciprocal teaching• Feedback• Teacher-student relationship• Spaced vs. massed practice• Metacognitive strategies• Prior achievement• Vocabulary programs

• Repeated reading• Creativity programs• Self-verbalization/self-

questioning• Professional development• Not labeling students• Problem-solving teaching• Phonics instruction• Teaching strategies• Study skills• Direct Instruction• Cooperative vs. individualistic

learning• Comprehension programs• Tactile stimulation programs• Mastery learning

What do these things have in common?

Develop or adopt

2a process for monitoring

and evaluating decisions.

Continuous Improvement

Logical Reasoning Approach

Bicycle Safety and Helmets

1. What would you do to make sure students wore helmets (use yellow stickies)?

2. What would you need to do the things you listed in question one (use green stickies)?

3. What would happen as a result of each thing you did (use blue stickies)?

4. How would you know it happened and how might you capture information along the way (use orange stickies)?

5. What would be your standard for success that would allow you to say that the results were important (use white stickies)?

• Why would you spend time with a group of stakeholders from your school answering these questions?

• How would the answers to these questions help you know if you got it (i.e., helmet safety) right? How would they help you know if you got any decision right?

Reflect and Share

Know what the problem

3looks and sounds like

before you attempt to solve it.

Evaluations and Problem Identification

• What was the ideal state for the helmet safety situation?

• What was the current reality?

• Why was action needed?

Note: The answer to this question is the problem and we identify the problem once we have analyzed why a gap exists between the ideal state and the current reality. In education, this gap often exists because of issues with cultural, learning, or economic systems.

Do gaps exist at your site?

• What would be the ideal state of this issue in my school?

• What is the current reality of this issue in my school?

• What is action needed? What needs to happen to close the gap between the ideal state and the current reality?

Think and Reflect

• What are your concerns about evaluation and continuous improvement?

• How would you convince your staff that evaluation is important?

Facilitate a process

4that clearly articulatesthe way the program works and its intended outcomes.

Keeping the Logic Straight

Case Study Logic Model

Read the case study and discuss two questions.– Discuss the pros and cons of

each the faculty members’ proposals to reach chronically failing students.

– Which proposal might be most successful in reaching chronically failing students?

Use flip charts to complete the steps.

Nine Step Planning Process

Step 1: Determine the ideal state. For example, if we got this right, what would it look and sound like?Step 2: Describe the current reality. What does the situation look and sound like now.Step 3: Assess the gap. Why does a gap exist between the current reality and ideal state?Step 4: Determine intended outcomes. What long term changes would we expect to see and hear? (These are changes in the school environment.)Step 4a: What intermediate changes would we expect? How would teachers’/students’ actions look and sound?Step 4b: What initial changes would we expect to see and hear? How would teachers’/students’ attitudes and knowledge change?

Step 5: Indicators. How will we monitor these changes and provide feedback?Step 6: Success Criteria. What will constitute success?Step 7: Identify high leverage actions. What must we do to create the change?Step 8: Outputs. What tangible evidence will our actions produce that is countable?Step 9: Resources. What do we need to act?

Nine Step Planning Process

Field Work