high quality performance measures what is performance measurement? what makes a high quality...

High Quality Performance Measures


What is Performance Measurement?

What makes a high quality performance measure?


Learning Objectives

By the end of this module, you will be able to:•Describe basic performance measurement concepts•Describe how to strengthen performance measures through:

o Alignment with theory of changeo Alignment of outputs and outcomeso Selection of a meaningful outcome

2


Module Overview

Performance Measurement •Concept and definitions•Comparison to impact evaluation

Strengthening Performance Measures through: • Alignment with the Theory of Change• Alignment of outputs and outcomes• Selection of a meaningful outcome

3


What is Performance Measurement?

Systematic process of regularly measuring:

Outputs• Amount of service provided (e.g. people served,

products created, or programs developed)

Outcomes• Reflect the changes or benefits that occur as the

result of the service• Can reflect changes in individuals, organizations,

communities, or the environment• Address changes in attitudes/beliefs,

knowledge/skills, behavior, or conditions

4


Performance Measurement and Impact Evaluation

Similarities•Systematic data collection •Measurement of progress•Collect data on:

• What took place (intervention)

• Outputs (amount of service)

• Outcomes (changes)

5


Performance Measurement and Impact Evaluation

Differences: Key Areas

Performance Measurement

Impact Evaluation

Causality Assumes causality; does not “prove” theory of change

Seeks to show causality and “prove” theory of change

Implementation Tracks outputs and outcomes on a regular, ongoing basis

May occur from time to time, but not on a regular, ongoing basis

Time Focus Shorter term focus; what can be measured within one year

Longer term focus

6


Performance Measurement and Impact Evaluation (continued)

Differences: Key Areas

Performance Measurement

Impact Evaluation

Design Balances rigor* with practicality; measurement implemented by program staff with limited resources and expertise

Uses most rigorous evaluation design and methods that are right for program (e.g. quasi-experimental design)

7

*Capable of producing valid (accurate) results.


Learning Exercise

Check the best answer Performance Measurement

Impact Evaluation

Both

1. Seeks to provide evidence of the existence of a causal relationship between the intervention and the observed outcomes, that is, it attempts to “prove” the theory of change

2. The information collected can be used for decision-making on programmatic improvement or expansion

3. Usually planned and implemented by program staff

4. Careful and systematic and quality data collection is very important

8


Learning Exercise

Check the best answer Performance Measurement

Impact Evaluation

Both

1. Seeks to provide evidence of the existence of a causal relationship between the intervention and the observed outcomes, that is, it attempts to “prove” the theory of change

X

2. The information collected can be used for decision-making on programmatic improvement or expansion

X

3. Usually planned and implemented by program staff

X

4. Careful and systematic and quality data collection is very important

X

9


Ensuring Quality

Performance Measure

•Intervention

•Output

•Outcome

Align output

and outcome

Review for

meaningful outcome

Planning

Theory of Change (ToC)•Problem/Need•Intervention•Outcome

Align elements

within ToC

10


Performance Measurement and Alignment

Alignment in Performance Measurement: Logical flow and connection between the different elements

•Alignment within the theory of change

•Alignment between output and outcome

11


Alignment within Theory of Change

12

Evidence•Guides choice of intervention•Supports cause-effect relationship

Evidence•Guides choice of intervention•Supports cause-effect relationship

Community Need/Problem

Community Need/Problem

Specific Intervention

“cause”

Specific Intervention

“cause”

Intended Outcome“effect”

Intended Outcome“effect”

A theory of change defines a cause-and-effect relationship between a specific intervention and an intended outcome.


Which outcome is aligned with the intervention?

NEED: Documented high rates of chronic absence from school lead to other problems, e.g. low school engagement, poor academic performance (Social Work Practices, 2010).

INTERVENTION: Weekly individual and group mentoring to 6th grade students with attendance problems over 10 months. Focus on promoting re-engagement with school through social development and leadership skills.

OUTCOME: ________

Students participate in the mentoring program

Students increase academic performance

Students improve school attendance

Mentoring Example*: Alignment within Theory of Change

*Descriptions in examples are abbreviated.


Which outcome is aligned with the intervention?



OUTCOME: ________

Students participate in the mentoring program

Students increase academic performance

Students improve school attendance

Mentoring Example: Alignment within Theory of Change




OUTCOME: Students improve school attendance.

Mentoring Example: Alignment within Theory of Change


NEED: 35% of young veterans (18-24 year olds) are unemployed (Department of Veteran Affairs, 2011). Economists cite a lack of marketable civilian skills and the need for education degrees, vocational certifications and experience in professional settings.

Intervention

1.Veterans participate in a drop-in job clinic offering job fairs, resume writing classes and skills in internet job searches.

2.Veterans supported in completing training programs through assistance in finding appropriate programs and financial aid, with tutoring resources, in internship placement.

3. Military families receive job placement counseling.

Outcomea.Veterans report increased confidence in using the internet for a job search.

b.Veteran family member report completion of skill classes.

c.Veterans find or are placed in jobs.

Learning Exercise



Intervention

1.Veterans participate in a drop-in job clinic offering job fairs, resume writing classes and skills in internet job searches.



Outcomea.Veterans report increased confidence in using the internet for a job search.

b.Veteran family member report completion of job skills classes.

c.Veterans find or are placed in jobs.

Learning Exercise (continued)Select the outcome.



Intervention1.Veterans participate in a drop-in job clinic offering job fairs, resume writing classes and skills in internet job searches.



Outcome

Veterans find or are placed in jobs.

Learning Exercise (continued)Select the intervention.



Intervention

Veterans supported in completing training programs through assistance in finding appropriate programs and financial aid, with tutoring resources, in internship placement.

Outcome

Veterans find or are placed in jobs.

Learning Exercise (continued)Summary: Alignment within Theory of Change


Alignment of Outputs and Outcomes

• Output results from the intervention• Outcome is likely to result from output• Output and outcome measure the same

intervention and beneficiaries

20


INTERVENTION: Construction teams assess housing request, plan and execute housing upgrades and repairs and complete inspection requirements for individuals with disabilities.

OUTPUT: Individuals with disabilities receive housing service (upgrades and repairs).

OUTCOME: Construction team members improve construction skills.

Do the intervention and output align? Yes No

Why or why not?

Housing Example: Output-Outcome Alignment





Do the intervention and output align? Yes No

Why or why not?

Logical to assume if well-designed and implemented



INTERVENTION: Construction teams assess housing request, plan and execute housing upgrades and repairs, and complete inspection requirements for individuals with disabilities.

OUTPUT: Individuals with disabilities receive housing services (upgrades and repairs).


Do the output and outcome align? Yes No

Why or why not?






Do the output and outcome align? Yes No

Why or why not?

2 different groups of beneficiaries



INTERVENTION: National Service participants provide one-on-one tutoring to preschool children in math readiness activities, 5 times a week over 9 months.

Output

1.Children complete participation in the early childhood program.

2.Teachers complete training with expert math teachers.

3.Children complete participation in preschool art group.

Outcome

1.Children demonstrate gains in school readiness in terms of literacy skills.

2.Children demonstrate gains in school readiness in terms of numeracy (math) skills

3.Teachers implement new math curriculum in the classroom.

Learning ExerciseOutput-Outcome Alignment



Output1.Children complete participation in the early childhood program

2.Teachers complete training with expert math teachers.

3.Children complete participation in preschool art group.

Outcome

1.Children demonstrate gains in school readiness in terms of literacy skills.






Output

Children complete participation in the early childhood program .

Outcome1.Children demonstrate gains in school readiness in terms of literacy skills.





Examples of Aligned Output-Outcome National Performance Measures

Focus Area Output Outcome

Education Number of children that completed participation in CNCS-supported early childhood education programs. (ED21)

Number of children demonstrating gains in school readiness in terms of social and/or emotional development.(ED23)

Healthy Futures

Number of individuals receiving support, services, education and/or referrals to alleviate long-term hunger (H11)

Number of individuals that reported increased food security of themselves and their children (household food security) as a result of CNCS-supported services (H12)

National performance measures at the Resource Center: www.nationalserviceresources.org/npm/home


Identifying a Meaningful Outcome

The Outcome should:

•Be central to your theory of change

•Match the type of change (attitude, knowledge, behavior, or condition) you want to measure

•Be ambitious but manageable

29


Central to the Theory of Change

30

How is the proposed outcome supported by the theory of change?

• Evidence for intervention should support the outcome

• Evidence supports cause-and-effect relationship between intervention and outcome


Central to the Theory of Change

31

How is the proposed outcome central to the theory of change?

•Addresses the need/problem

•Likely change given the intervention

•A key, not preliminary or peripheral, outcome


Which outcome is central?



1. Families are satisfied with the mentoring program

2. Students improve engagement with peers

3. Students improve school attendance

Mentoring Example: Central to Theory of Change


Type of outcome: Accurate and Clear

33

Is the outcome getting at what you wish to measure? •Articulates the question you really want to answer

Has the outcome type been identified: attitude, knowledge/skills, behavior or condition?•Outcome type should be same as theory of change•Outcome type will inform instrument


Which type of outcome?

NEED: 35% of young veterans (18-24 year olds) are unemployed (Department of Veteran Affairs, 2011). Economists cite a lack of marketable civilian skills and the need for education degrees, vocational certifications…

INTERVENTION: National service participants support veterans in completing training programs by assisting in locating appropriate programs, securing financial aid, and by providing tutoring resources and internship placements.

1. Veterans report increased confidence about finding employment. (attitude)

2. Veterans demonstrate new technical skills. (knowledge/skills)

3. Veterans are placed in jobs. (condition)

Veteran Example: Type of Outcome


Ambitious and Manageable

35

Is the outcome meaningful AND manageable? Consider: •Staff time and expertise •Reporting constraints


Ambitious and Manageable

36

Is the outcome too modest? Consider:•Outcome should reflect the main change expected from the intervention

Is the outcome too ambitious? Consider:•Program timeframe•Scope of the intervention•Severity of the problems being addressed


Which outcome is meaningful and manageable?

NEED: Documented high rates of chronic absence from school…

INTERVENTION: National Service participants provide individual and group mentoring to 6th graders with documented attendance problems…

1. Student increase outdoor teamwork skills

2. Students increase graduation rates from high school

3. Students improve school attendance

Mentoring Example: Ambitious and Manageable


Learning Exercise

1. Which outcome is too ambitious? (select one)

2. Which outcome is too modest? (select one)

3. Which outcomes look ambitious but manageable? (select two)

Outcomesa. Students know location

of the library

b. Student improve reading ability

c. Students graduate from high school

d. Students increase school attendance

Intervention: Year long mentoring project with 4th graders– literacy focus.


Learning Exercise




Outcomesa. Parents are interested in

learning more about the program.

b. There is reduced gang violence in the area.

c. Donations for sustaining activities at the two sites are secured.

d. Trained community volunteers are serving at the new sites.

Intervention: An afterschool program for children ages 6-12 is expanded to two new sites.


Learning Exercise




Outcomesa. Youth develop long-

term healthy eating habits.

b. Youth increase healthy food intake.

c. Youth improve cooking skills.

d. Youth are motivated to eat more vegetables.

Intervention: 6-month nutrition and health program for high school students.


Learning Exercise




Outcomes

a. Youth increase knowledge of native stream bank plant species.

b. 250,000 residents in the county benefit from restored watershed.

c. Youth increase interest in protecting the environment.

d. Water quality is improved over 6 mile clean-up area.

Intervention: Year-long riparian (river) restoration project involving youth volunteers.


Learning Exercise

43

The most meaningful and manageable outcome for this intervention is:

a)Number of individuals satisfied with quality of the work.

b)Number of individuals securing mortgage loan approval for house purchase within 5 years.

c)Number of individuals transitioned into safe housing.

INTERVENTION: National Service participants assess the housing up-grade requests from the community members and address dilapidated, unsafe or inaccessible housing. They plan and then execute housing remodels, upgrades and repairs for economically disadvantaged individuals and individuals with disabilities.


Summary of Key Points

• Performance measurement is a systematic process of measuring progress (outputs and outcomes)

• Performance measurement does not seek to “prove” a theory of change, but can provide snapshots.

• Impact evaluation can determine if results occurred because of the intervention.

• Strong performance measures align with the theory of change (need, intervention and outcome)

– The intervention is based on evidence that supports a cause-and-effect relationship between the intervention and an intended outcome.

– The theory of change helps identify an appropriate outcome to measure.

44


Summary of Key Points

• Strong performance measures are aligned

– Output results from the intervention

– Outcome is likely to result from output

– Output and outcome measure the same intervention and beneficiaries

• Strong outcomes should:

– Be central to the theory of change

– Match the type of change you want to see (attitude, knowledge/skills, behavior, condition)

– Be ambitious but manageable

45


Resources

• CNCS Priorities and Performance Measures: http://

www.nationalservice.gov/resources/npm/home

• Program Specific Notices of Funding Opportunities and Application Instructions: http://http://www.nationalservice.gov/for_organizations/funding/nofa.asp

Link to Evaluation

46

high quality performance measures what is performance measurement? what makes a high quality...

Documents

academic performance

alignment alignment

theory of change x

outcome effect

practicality measurement

quality data collection

intended outcome

choice of intervention