Annual Seminar with Department of Education

Using Empowerment Evaluation and Data to Strengthen Your Talent Search Program

Maggie Cahalan and Stephaine MillerPell Institute, COEMarch 9 2011

Presentation Topics1. Overview of Empowerment Evaluation Concepts

and Steps with example from COE’s new I-3 project testing new models for Talent Search (Maggie)

2. Using Data for Evaluation Program Improvement and Student Success (Stepahanie)

3. Breakout Session (Speed Dating Process to move through ideas about steps in empowerment evaluation of your program)—(Everyone--35 minutes)

4. Group empowerment evaluation of our session (what you liked and did not like and how we could improve (Everyone--10 minutes)

Personal Journey in Thinking About Evaluation

Contractor Project Director National Evaluation of Student Support Services

National Evaluation of Talent Search

TRIO performance reporting support contracts

Department of Education National Evaluation of Upward Bound study final technical monitor

Came to see need for taking a more participatory approach involved and began Designing Next Generation of GEAR UP

Next Generation of Gear Up Evaluations

Grew out of challenges (problems with meaningful comparison group, validity issues, length of time to obtain results, black box up or down studies not always useful for program improvement, lack of involvement of stakeholders meant work not viewed as very useful to them)

New Approach (18 month design work—sessions with grantees, expert papers, systemic review of past evaluation methods and challenges, issued call for entries open to all grantees to design evaluations of what they viewed as promising practices, funded 44 grantees proposals and provided technical assistance, and funded 12 to implement the project led evaluations of rigorous designs)

Definition

Empowerment evaluation is the use of evaluation concepts, techniques and findings to foster improvement and self determination. It employs both qualitative and quantitative methodologies.

Empowerment evaluation: Knowledge and Tools for Self-Assessment and Accountability Fetterman, Kaftaraian, and Wandersman 1996

Empowerment Evaluation

Aims to increase the probability of program success by:

1 Providing program stakeholders with tools for planning and implementation, and self evaluation of their program, and

2 Mainstreaming evaluation as part of the planning and management of the program organization (Wandersman, et. Al, 2005, p. 25)

Empowerment Evaluation

Widely usedJoint Committee for Education Program Standards have included conceptsAmercian Evaluation Association and AERA workshops Public Health FieldInternational Development workEducation school systems

ContrastsTraditional Evaluation

External

Expert

Dependency

Independent judgment

Developed when data not available to all— elite with resources to collect and skill to analyze

Empowerment Evaluation

Internal

Coach or Critical friend

Self-determination & capacity building

Collaboration

Makes use of Data Revolution—internet, web, real time interactive sharing of knowledge—all publish—face book, blog, twitter

Internal or External Evaluation

The dilemma of whether to use external or internal evaluation is as false as that between qualitative and quantitative methods. The solution is always to use the best of both, not just one or the other (Michael Scriven)

Learn by Doing

Organizations and their staff evaluate their own strategies

May hire an evaluator to work with them

Rather than presenting a report card, empowerment evaluators coach staff through an evaluation of their own strategies by providing them with the knowledge, skills and resources they need to conduct evaluation

10 Key Principals (Wandersman et. Al 2005)

1. Community Ownership—primary responsibility with organization and not outside evaluator

2. Inclusion—involves representation of key stakeholders

3. Democratic participation—highly collaborative—opportunity to voices questions—every stakeholders voice is heard

4. Community knowledge-promotes growth of knowledge in community—stakeholders are considered to be in the best position to understand the issues and generate solutions to problems

5. Evidence Based strategies-promotes use of strategies with high quality evidence—research evidence of effectiveness—evidence strategies contextualized to fit community

10 Key Principals (cont)6. Accountability-provides data that can be used to determine

whether a strategy has achieved its goals—negative results are not punished;rather are use to inform change in a strategy or the selection of a new strategy for the purpose of producing better outcomes

7. Improvement—Helps organizations improve strategies so that they are more likely to achieve stated goals—process and outcome evaluation (Rossi 1999)

8. Organizational learning—fosters a culture of learning—view positive and negative feeedback as valuable information and believe that all strategies can be improved

9. Social justice—increase capacity to reduce disparities that affect marginalized by persecution, discrimination, prejudice and intolerance

10. Capacity building—builds capacity of organizations to conduct their own evaluations, understand results and use them to continuously improve organization

10 Steps (Chinman, Imm & Wandersman, 2004)

1. Conduct needs and resource assessment

2. Develop goals and outcomes sought

3. Selecting evidence based strategies

4. Assessing strategies fit

5. Building capacity

6. Finalizing plan

7. Evaluate process

8. Evaluate outcomes

9. Implementing continuous quality improvement

10.Sustaining efforts

Fetterman (2001)

Define mission

Taking stock-

Planning for future

Example---COE Investing in Innovation (I-3) project

ED wishes to build capacity at all levels to use data and build in internal and external evaluations as way to improve programming on an on-going basis

October COE won I-3—Investment in Innovation--Using Data to Inform College Access—Power Knowledge

Build on and help validate GE/COE college access work in 2 cities (Erie and Louisville)

Enhanced whole school college access services—college coaches in school

Intensive service Learning Communities

COE Building large data warehouse Power Knowledge Tool for use by all

Contribute to innovative adaptive models for established Federal College Access programs such as Talent Search, GEAR UP, Upward Bound

COE (I-3) project Testing and Validating ---Model-

--Increased access to, skill, and use of data by all stakeholders (students, parents, school, college coaches, community) will lead to increased high school and college participation and success among students in high schools

Internal Empowerment Evaluation and

External evaluators both using the DATA TOOL

DICAP--Builds and Enhances on GE-COE Model

Multiple Power Knowledge Learning CommunitiesUsing data among all stakeholders—key

District and school staffCollege access service providers and planners--College CoachesStudentsParentsCommunityInternal and External Evaluators

Goals of UN Education for Sustainable DevelopmentWhat if every person benefited from an education promoting

development that is environmentally sound, socially equitable, culturally sensitive and economically just?

What if learning was about knowledge and also about doing, being, interacting with others and changing the world?

What if formal learning was enjoyable, hands-on and relevant to life outside school while addressing the problems of our world?

What if every person benefited from genuine learning opportunities throughout life, in the workplace, and within the community?

What if education systems prepared learners to enter the workforce as well as handle a crisis, be resilient, become responsible citizens, adapt to change, recognize and solve local problems with global roots, meet other cultures with respect, and create a peaceful and sustainable society?

Partnership—Collaborative-Empowerment Model

All stakeholders full partners—those most impacted –strongest roles

StakeholdersHelp set goals

Identify assets, challenges, describe current baseline situation and data

Take responsibility for identifying and implementing steps (actions) to meet goals

Take responsibility for analysis, reflection on results and become learning community

Transparency among all stakeholders

Partnership—Collaborative-Empowerment Model

All stakeholders full partners—those most impacted –strongest roles

StakeholdersHelp set goals

Identify assets, challenges, describe current baseline situation and data

Take responsibility for identifying and implementing steps (actions) to meet goals

Take responsibility for analysis, reflection on results and become learning community

Transparency among all stakeholders

Smaller Intensive Learning Community

Power knowledge Learning Community Leadership Group

Summer Bridge

On-going school year workProject to participate in building and using the data tool

Project to communicate to whole school—lead to goals

Project to evaluate and assess efforts and to improve the program –internal empowerment evaluation

Theory of Participatory Action/Change

Participatory goal setting

Identify Inputs

(assets, baselines, challenges

and barriers)

Identify Steps/actions

needed to meet goals

Identify indicators and monitor fidelity

of implementation/challenges/cou

nterfactuals

Power Data Tool

Reflection and

Communicate/disseminate

findings

Learning Community Prepares various

reporting's/feedback for

stakeholders

Impact assessment

and recommendat

ions for future

Assessment of intermediate indicators/out

comes

Logic of Evaluation

Goal or purpose

Baseline –where before intervention

Program or intervention

Measurement (including criteria)

Process

Outcomes and impacts

Speed Dating Stations Idea Generating

1. Goal and Strategy setting1. Setting Goals/Mission—what do want to change or improve

—how much?

2. Evidence based strategy for improvement—what concrete steps do you need to take that you think might work? Now? Next year? Next five years?

2. Taking Stock1. Baseline—where are you now—take stock of assets and

barriers?

2. How can you assess or measure if what you are trying is working-----choose an evaluation design and indicators

3. Provide for gathering credible evidence---Data

3. Plan for Future—Learning Organization1. Use data to analyze and reflect on what you find

2. Improve organizational capacity and the particular strategy evaluated

Resources and contact information

Contact---margaret.cahalan@pellinstitute.org

Empowerment evaluation resourceshttp://eevaluation.blogspot.com/

www.cdc.gov/violenceprevention/pdf/evaluation_improvement-a.pdf

http://wwwstatic.kern.org/gems/region4/DavidFettermanPresentation.pdf