Developing an Effective Evaluation to Check for Understanding

Susan E. Schultz, Ph.D.

Evaluation Consultant

PARK Teachers

PARK Teachers

Overview

Part 1: Components of Program Development

Part 2: Goals and Types of Evaluation

Part 3: Creating an Effective Evaluation Plan

Developing an Effective Evaluation Plan

PARK Teachers

PART 3: Creating an Effective Evaluation Plan

PARK Teachers

Developing an Effective Evaluation Plan

Identify Program

Goals

Create anEvaluation

Matrix

Use variety of AssessmentTechniques

Select AssessmentStrategies

Pilot and Revise

Assessments

PARK Teachers

“What are the three key ideas or messages that I want all participants to know when they complete the program?”

“What overarching question or statement brings all of your goals together?”

Identify the program goals and essential question

PARK Teachers

The PARK program goals focused on pre-service teachers being able to:• Improve their geology content knowledge, • Engage in quality instructional practices,• Experience national park as a research education site,• Learn about educational resources in national parks, • Transfer their knowledge to their local community.

The essential question was, “How do you recognize evidence of geologic change in your environment?”

Identify the program goals and essential question

PARK Teachers

Check for Understanding Task 3: Outcomes

Part 1: Using your group’s essential question and program goals, write your group’s outcomes on a piece of poster paper.

Part 2: Share your group’s outcomes with the whole group.

PARK Teachers

Create an evaluation matrix based on program goals

An evaluation matrix is an outline of the evaluation plan.

The matrix includes the: • Program Outcomes, • Task to achieve the outcomes, and• Assessment strategies to measure the outcomes.

Let’s look at some specific examples from the PARK program on the next two slides.

PARK Teachers

Educational Outcome Tasks Measurement

Develop a model for creating a partnership to focus on pre-service education

-Develop observation protocol and procedures for tracking decision-making process.-Make observations and record decisions made at the partnership meetings.

Document the formation and continuation of the partnership between constituents.

Design a pre-service teacher graduate course module on geology

Evaluating content of curriculum using a specific set of criteria such as:-Identify explicitly stated goals for curriculum-Alignment of goals with best practice in scientific community on topic of geology-Identify key teaching strategies to teach geology with an inquiry focus.-Examine transitions within and between topics in the curriculum-Examine assessment strategies-Identify connections to home communities

Evaluate each component of the geology curriculum based on the criteria and make suggestions for revision to curriculum developers.

Sample of PARK Program Evaluation Matrix

PARK Teachers

Educational Outcome Tasks Measurement

Enhance teachers’ knowledge of geology content related to the natural setting provided by GGNRA

- Create instruments to assess geology content- Pilot instruments & revise assessments- Administer tests- Analyze data / Write report

Pre and post content tests to determine change in pre-service teachers’ knowledge of geology.

- Develop teachers’ awareness of informal science education resources, particularly at National Parks, available to supplement and enhance science instruction- Develop teachers’ ability to use National Parks as labs for innovative, relevant place-based science instruction, including collaborating with informal educators

- Create survey instruments to assess NPS informal science resources- Pilot survey instruments & revise- Administer surveys- Analyze data / Write report

- Pre and post surveys to determine whether participants’ learned about informal science resources and how they might incorporate them into their teaching.

-Interviews with sub samples of pre-service teachers about how they might use NP as a part of their curriculum in the future. (depends on funding)

Sample of PARK Program Evaluation Matrix

PARK Teachers

Check for Understanding Task 4: Evaluation Matrix

Part 1: Complete your group’s evaluation matrix for your project

A. Break down each program outcome and generate a list of tasks that need to be completed to achieve the specific outcome.

B: Identify how you plan to measure or evaluate if you have achieved the outcome.

Part 2: Present your evaluation matrix to the whole group.

PARK Teachers

Select the Appropriate Assessment Strategies

It is imperative that you select the appropriate assessment strategy when developing your evaluation plan.

The type of assessment used must match the: • program outcomes and• type of knowledge being evaluated.

13

PARK Teachers

Assessing Different Types of Knowledge

Essential to select the appropriate assessment instrument to evaluate different types of knowledge.

Most programs focus on a combination of content and performance-based knowledge.

Content Knowledge

Performance or Skill Knowledge

PARK Teachers

Part 1: Select the Appropriate Assessments

A. Identify how you plan to measure or evaluate if you have achieved the outcome.

B: Think about the desired outcome and the type of knowledge being evaluated when selecting the

assessment instrument.

Part 2: Present your assessment selection to the whole group.

Check for Understanding Task 5: Selecting Assessments

PARK Teachers

Pilot and Revise Assessment Instruments

Piloting and Revising the Assessment Instruments is Essential!

The revision process helps you to: clarify what you are trying to assess, make revisions to other program components

(content, instructional activities, or course planning)