Southeastern Association of Educational Opportunity Program Personnel

38th Annual Conference

January 30 – February 3, 2010

Upward Bound Upward Bound Internal & External Program Internal & External Program

EvaluationEvaluation

PresenterPresenter

Elizabeth C. Retamozo, Executive Director

NOSOTROS Education Center NOSOTROS Education Center www.nosotrosedu.org

Phone: 469-450-7137

Email: eretamozo@nosotrosedu.org

2

Presentation MaterialsPresentation Materials

• PowerPoint Slide Presentation & Handout

• Located at: www.nosotrosedu.org– Go to Resources Page– Register Your Email Address– Click the Presentations & Links Tab– Click SAEOPP 2010 Folder – UB Internal & External Program Evaluation

3

4

Types of TRIO Evaluation PlansTypes of TRIO Evaluation Plans

Outcome-Based Evaluation

Goals-Based Evaluation

Process-Based Evaluation

5

Common Evaluation Plan TerminologyCommon Evaluation Plan Terminology

Qualitative Values Expressed by impressions, feelings & perceptions

Ex. Surveys, Interviews, Focus Groups, etc.

Quantitative Values Expressed in numerical terms, counted, or

compared on a scale

Ex. Grade Point Averages, Test Scores, etc.

6

Common Evaluation Plan TerminologyCommon Evaluation Plan Terminology

Formative Methods Continuous (daily, weekly, monthly, quarterly,

semester) assessment to track progress

Ex. Monthly Reports

Summative Methods End of year assessment that measures the

results or outcomes of a Project

Ex. Annual Performance Report

7

Sample TRIO Evaluation PlansSample TRIO Evaluation Plans

Outcomes Based Evaluation

• External & Summative

Looks at impacts/benefits/changes to your

students as a result of your project’s efforts after

their participation in your Project (PE

OBJECTIVES).

8

Sample TRIO Evaluation PlansSample TRIO Evaluation Plans

Goals Based Evaluation

• Internal & Formative

Evaluates the extent to which projects are

meeting predetermined internal goals or

objectives.

9

Sample TRIO Evaluation PlansSample TRIO Evaluation Plans

Process Based Evaluation

• External & Summative)

Geared to fully understanding how a Project works

How does it produce that results that it gets?

10

Outcome Based Evaluation Outcome Based Evaluation

Outcomes Based Evaluation (External & Summative)

Measures performance on the four (4) established UB Program objectives by the end of each year (summative).

DE Mandated Objectives = Prior Experience

11

Outcome Based Evaluation Plan Outcome Based Evaluation Plan

1) Identify Major Outcomes (Mandatory Objectives)

2) What Types Of Data Will Be Collected?

3) When Will Various Types Of Data & Reports Be Collected?

4) What Instruments Will Be Developed?

5) How Will The Data Be Analyzed?

6) How Will The Data Collected Determine Accomplishment?

12

Outcome Based Evaluation Outcome Based Evaluation

UB Outcome Based Evaluation Plan

13

Goals Based Evaluation Goals Based Evaluation

Goals Based Evaluation (Internal & Formative)

Evaluates the extent to which Project is meeting predetermined internal goals

Project Developed INTERNAL GOALS based on DE Objectives

Includes quantifiable & qualitative measures to track progress during the Project year (formative)

Monitored and reviewed weekly/monthly

14

Goals Based Evaluation Goals Based Evaluation

1) What Project goals were established?

2) What is the status of the project’s progress towards achieving the goals?

3) Will the goals be achieved according to the timelines specified in the program implementation or operations plan?

4) Do personnel have adequate resources (supplies, facilities, training, etc)?

15

Goals Based Evaluation Goals Based Evaluation

5) How should priorities be changed to focus more on achieving the goals?

6) How should timelines be changed?

7) How should goals be established in the future?

16

Goals Based Evaluation Plan Goals Based Evaluation Plan

1) Develop Internal Goals based on External Objectives

2) Activities & Services Needed to Achieve Each Goal

3) Project Staff Responsible to Carry Out Activities & Services

4) Data that Needs to be Collected

5) Instruments (Forms) that Need to be Developed and Collected

17

Goals Based Evaluation Plan Goals Based Evaluation Plan

6) When is Data Collected?

7) How is Data Collected Analyzed?

8) Was the Goal Met? Yes/No

9) Needed or Recommended Revisions

10) Unanticipated Outcomes

18

Outcome Based Evaluation Outcome Based Evaluation

Sample UB Internal Goals

UB Goals Based Evaluation Plan

19

Process Based EvaluationProcess Based Evaluation

Process Based Evaluation (External & Summative)

Geared to fully understanding how a project works

How does it produce the results that it gets?

Usually External Conducted by Program Evaluator

20

External EvaluationExternal Evaluation

• Recommended in last grant proposal competitions

• Must be conducted if included in grant proposals

• Allowable cost for the project

• May be paid by the grantee agency/institution

• May be provided free of charge – in-kind/donation

21

External EvaluationExternal Evaluation

• May be conducted by person working/employed by agency/institution BUT not for the Project

• Must have experience conducting evaluations for educational opportunity programs & TRIO like programs

• Must state what type of evaluation will be conducted by the evaluator.

22

Process Based Evaluation Plan Process Based Evaluation Plan

1) How does the Project identify, recruit, & select participants?

2) How does the Project decide what specific services & activities to provide to its participants?

3) How does the Project monitor participants’ academic progress?

4) How does the Project deliver services to its participants?

23

Process Based Evaluation Plan Process Based Evaluation Plan

5) How does the Project document services provided to participants?

6) What is the Project’s goals based evaluation plan?

7) What is the Project’s outcome based evaluation plan?

8) How does the Project plan for the next program year?

24

Process Based Evaluation Process Based Evaluation

UB Process Based Evaluation Checklist

External Process Based Evaluation Report

25

Basic Guide to Program Evaluation

Written by Carter McNamara, MBA, PhD, Authenticity Consulting, LLC. Copyright 1997-2007.

Adapted from the Field Guide to Nonprofit Program Design, Marketing and Evaluation.

Online Guide: A Basic Guide to Program Evaluation.pdf

Bibliography & ResourcesBibliography & Resources

26

Basic Guide to Goals-Based Evaluation for Nonprofit Organizations with Very Limited Resources

Written by Carter McNamara, MBA, PhD, Authenticity Consulting, LLC. Copyright 1997-2007.

Online Guide: Basic Guide to Outcomes-Based Evaluation for Nonprofit Organizations with Very Limited Resources.mht

Bibliography & ResourcesBibliography & Resources

27

US Department of Health & Human Services Administration for Children & Families - Office of Planning, Research & Development

The Program Manager's Guide to Evaluation

ACF OPRE The Program Manager’s Guide to Evaluation website.mht

This guide explains program evaluation -what it is, how to understand it, and how to do it. It answers your questions about evaluation and explains how to use evaluation to improve programs and benefit students, families, and staff.

Bibliography & ResourcesBibliography & Resources

Questions & AnswersQuestions & Answers

28