07/25/2018

1

Arlington County, VirginiaNOFA WorkshopJuly 24, 2018

Mary K. WinklerSenior Research Associate, Urban Institute

Center on Nonprofits and Philanthropymwinkler@urban.org

Agenda

• Landscape of Nonprofit Measurement and Evaluation• Evaluation v. Performance Measurement• Overview of Logic Models• Outcomes v. Indicators• Small Group Activity• Q&A

07/25/2018

2

Current State of Affairs

• Funders/donors and nonprofits want better information.• Nonprofits lack data/ability to produce outcome measures.• Funders/donors/nonprofits don’t know how to use information.• Competing/cumbersome reporting requirements.• Resources are becoming more competitive.• More often than not a “top down” approach.• Culture wars between compliance and learning.

07/25/2018

3

What is a learning culture?

“….one that constructively responds to new information, adapts based on experience with both successes and failures, and becomes strong and more effective over time…

…A learning culture is largely created and reinforced by leaders, and is supported by organizational processes and incentives…”

Source: Evaluation for Strategic Learning: Assessing Readiness and Results, Anna Williams, www.evaluationinnovation.org

Performance vs. Compliance

Look what we accomplished? We followed all the rules

I wonder why we missed We’ve been doing things this our target? way for years.

Let’s ask the team what else Let’s show these results to thewe can try to get better results? director and ask what to do

What can we do to better You’ll just need to try harder in support you in your work? the future.

How can we encourage families Parents are already too busyto share in our success? and probably won’t want to

attend another meeting.

07/25/2018

4

Emerging and Current Initiatives

Leap of Reason Ambassador Community

07/25/2018

5

Measure4Change

• Small Grants• One on One Technical Assistance• Knowledge Briefs• Community of Practice

07/25/2018

6

Evaluation Defined

“A structured process of assessing the success of (a program or) project in meeting it’s goals and to reflect on the lessons learning.”

excerpted from the Evaluation Toolkit –- http://evaluationtoolbox.net.au

Many Different Types of Evaluation

• Performance monitoring or measurement• Process evaluation• Implementation evaluation• Outcomes evaluation• Non-experimental design• Quasi-experimental design• Random control trial

07/25/2018

7

13

Comparison of Two Approaches to Evaluation

PERFORMANCE MEASUREMENT

IN-DEPTH EVALUATIONS

1. Depth of Information: Only tells “the score”, not WHY

Can identify reasons for poor performance

2. Timeliness of Findings: Relatively quickly Can require months and even years

3. Skills Required: Considerably less needed Special skills needed

4. Quality of Analysis: Usually limited and weak Strong

5. Cost: Low High

6. Utility: “Minor” program decisions; Program Improvement

Major program decisions

What is Performance Measurement

“Regular measurement of progress toward specified outcomes is a vital component of any effort at managing for results.

A customer-oriented process that focuses on maximizing benefits and minimizing negative consequences for customers of services and programs.”

(Harry Hatry, Performance Measurement: Getting Results, p.3)

07/25/2018

8

Key Steps in Performance Management

1. Identify mission, objectives and clients.2. Identify the results (outcomes) the program seeks.3. Select specific indicators for measuring each outcome.4. Identify appropriate data sources for each indicator and how to

collect this information.5. Develop an analysis plan, including presentation formats.6. Collect and analyze the date, develop reports.7. Share data with staff and stakeholders8. Use information to make changes or improvements.

Performance Cycle

• Source: Isaac Castillo

07/25/2018

9

07/25/2018

10

Logic Models & Theories of Change

• An opportunity to communicate within and outside your organization organizational direction and to link core assumptions about your work with intended results.

• A means of articulating organizational goals and purpose (communications and outreach) with short- and long-term statements of progress (measurement and evaluation).

Logic Models

A tool to help convey a scheme, program, or project in a brief, visual format.

• Common language among stakeholders• Participatory learning opportunities• Document and emphasize results or outcomes• Clarify knowledge about what works and why• Provide a credible reporting framework• Improved design, planning, and management

Retrieved from: http://www.sagepub.com/upm-data/23937_Chapter_1___Introducing_Logic_Models.pdf, on 10/15/2013

07/25/2018

11

Logic Model ComponentsSource: Fieldstone Alliance.org

Logic Model Examples

07/25/2018

12

DOD War Strategy

Bridging OutcomesBridgingOutputs

Student-Level• Program

enrollment• Program

completion• Participation in

offered activities

• Completion of offered activities

Program-Level• Counselors &

instructors continually use data to support & assess learning.

Inputs

• Service delivery model

• Curricula• Service

delivery partners

• Technology & tracking systems

• Staff/Personnel

• Matching dollars

• Technical assistance

• Opportunity youth

Functional ActivitiesActivities necessary to set up & run the program, which feed

back into inputs• Local adaptation of Back on

Track intervention(s)• Develop procedures• Identify and organize service

delivery partners• Identify funding sources• Hire staff/initiate contracts• Train staff• Identify & reach out to youth• Screen/enroll youth• Determine & establish

technology & tracking systems• Monitor tracking system data to

improve processes & services

Implementation Activities

Programmatic ActivitiesService delivery activities to achieve outputs & outcomes

• Support dual enrollment & provide simulated college experience before enrollment

• Develop college knowledge & success strategies

• Provide personalized guidance & connection to best bets (navigation)

• Current students receive mentorship from program graduates

• Transition youth to college

Medium-Term OutcomesConfirmatory• Students apply to training programs, including college• Students enroll in training programs, including college• Students earn college credits• Students enroll for more than one semester• Student enroll for more than one year• Students earn college or industry-recognized credentials• Students complete 12 credits of college workExploratory• Students do not experience subsequent arrests or

incarceration

Short-Term OutcomesSoft Skills for Success • Students gain postsecondary & career navigation skills• Students exhibit career-ready skills & behaviorsAccess College/Career Coursework• Students have access to postsecondary facilities• Students enroll in credit-bearing courses appropriate to

reinforcing essential skillsSubject Knowledge Gains• Students demonstrate proficiency in key skills & subject

areasSupport Systems• Students have a sufficient care network• Students build social networks supportive of

postsecondary success

Long-Term Outcomes• Students are not disconnected from work and education• Students become employed• Students are employed in quality jobs• Students’ earnings are higher than they would be

otherwise

Contextual factors: Local labor market, service infrastructure for youth, juvenile justice system, availability of adult and postsecondary education options, policies affecting youth (e.g., GED changes, Pell Grant availability)

07/25/2018

13

Transitional Housing Program Description To facilitate the movement to permanent housing of the homeless who are willing and able to work towards self-sufficiency by providing shelter and other services that may include meals, laundry, transportation, child care, and counseling/case management for housing, employment, substance abuse, public benefits, and referral to outside services. Also known as transitional shelter. Outcome Sequence Chart Intermediate Outcomes End Outcomes Indicators

Satisfaction with program services is an outcome that occurs within almost every program area, yet does not necessarily have a sequential placement. The indicator may be: Number and percent of clients satisfied with transitional housing programs and services.

Sources Consulted: Developing Community-wide Outcome Indicators for Specific Services by Hatry and Lampkin (2003); Volunteers of America National Program Outcome Measures and Program Standards FY 2005 (Volunteers of America, 2004); Core Performance Indicators for Homeless-Serving Programs Administered by the U.S. Department of Health and Human Services by Trutko and Barnow (2003).

Clients complete registration/ intake

Clients participate in in-house and referral programs

Clients complete service plan or ‘graduate’ the program

Clients’ long-term housing needs are met

Clients maintain stable and secure housing

2. Number and percent of new clients accepted for services during the reporting period.

4. Number and percent of clients who have met with counselor/case manager and developed a treatment/service plan within X days of entry.

Clients develop a program/ service plan

5. Number and percent of clients who, as a result of their service plan, participate in supportive services (in-house or by referral) within X days of entry.

7. Number and percent of clients who complete service plan/ shelter program.

8. Number and percent of clients who move to long-term housing within 24 months of entry.

9. Number and percent of clients who maintain safe and permanent housing for six months or more. 10. Number and percent of clients who do not re-enter the homeless system within one year of obtaining permanent housing.

Participant satisfaction

6. Number and percent of clients in need of mental health/ addiction treatment who enter into such treatment within X days of entry.

Clients apply to transitional housing

1. Number of clients applying for transitional housing during the reporting period.

3. Number and percent of households/ persons in transitional housing during the reporting period.

Youth Development Logic Model

•

After School Program Description An array of safe, structured programs that provide children and youth a range of supervised activities designed to encourage learning and development outside of the typical school day. Intermediate Outcomes End Outcomes Indicators

Increased enrollment in after school programs

Improved program completion by participants.

Academic Achievement

Character Development

Prevent undesired behaviors

Increased parental involvement in children’s education and development.

Full-time employment

Workforce Development

1. Number of participants who enroll in after school program.

2. Number and percent of participants who completed the program.

6. Number and percent of participants who improve their test scores or grades. 7. Average number of educational levels advanced per program participant. 8. Number and percent of participants who receive a high school diploma or GED.

9. Number and percent of participants who enroll in postsecondary education or occupational skills training program.

3-5. Number and percent of parent participants who (a) regularly help children with their school work, (b) attend parent meetings or school activities, (c) read to their children daily.

13. Number and percent of participants who obtain a full-time job after completing school.

10-11. Number and percent of participants who (a) obtain a job within 3 months of program completion, or (b) remain employed 12 months after program completion.

12. Number and percent of participants who enroll in college or voc education program.

Enrollment in post-secondary education

Good citizenship

07/25/2018

14

Additional Tips

• Engage staff and stakeholders (“buy in”).• Look to other program examples/models.• Consider starting with outcomes and working your way backwards.• Avoid jargon and use of subjective terms.

(e.g., adequate, sufficient, appropriate)• Periodically revisit and revise.• Tailor formats and level of detail to audience.

(e.g., layered approaches, visualization)• Keep it “logical”.• Resource: “W.K. Kellogg Foundation Logic Model Development Guide,”

https://www.wkkf.org/resource-directory/resource/2006/02/wk-kellogg-foundation-logic-model-development-guide

07/25/2018

15

Outcomes vs. Outcome Indicators

Outcomes Indicators

The benefit or change for intended beneficiaries or target groups.

Identifies what is being measured to track the program’s success on the outcome. The observable and measurable evidence that tells you whether program is achieving its desired outcomes.

Usually expressed as a number, percentage or rate.

07/25/2018

16

Outcomes vs. Outcome Indicators

Outcomes Indicators

Example 1: Improved Parental Involvement in School

Indicator 1: #/% of parents attending school meetings.

Indicator 2: #/% of parents who volunteer at school activities.

Indicator 3: #/% of parents who help their children with their homework.

Outcomes vs. Outcome Indicators

Outcomes Indicators

Example 2: Children are ready for kindergarten

Indicator 1: #/% of children who meet basic school expectations for kindergarten readiness (e.g., can write first name, recite alphabet, count to 10.)Indicator 2: #/% of children who meet or exceed baseline test scores for K-readiness.Indicator 3: #/% of children demonstrating age-appropriate social skills.

07/25/2018

17

General Principles for Indicator Selection

• Some outcomes are easy to measure, others are not (e.g., graduation rates versus health condition or well-being).

• There are usually multiple possible indicators for a single outcome.

• Sometimes you have to settle for an indicator that “best represents” the outcome – these are sometimes called “proxy” indicators.

• Indicator selection may need to take into account ease access or availability of data, cost to obtain, timeframe/lags.

Data Collection Considerations & Tradeoffs

07/25/2018

18

PerformWell Demonstrationperformwell.org

Hands-on Activity

• Post-it note exercise

Organize post-it note items in a logical order and group by indicator type: (a) input, (b) output, (c) short-term outcome/result, and (d) long-term outcome/result.

07/25/2018

19

Additional Resources

• Measure4Change Knowledge Briefs:http://www.urban.org/measure4change

• Outcome Indicators Project and Nonprofit Series on Outcomes Measurement:http://www.urban.org/center/cnp/Projects/outcomeindicators.cfm

• PerformWell:performwell.org

• East of the River Initiative – Evaluation Matters Report:http://www.urban.org/publications/411961.html

• The Performance Imperative (videos, resources, PIOSA):http://leapambassadors.org/