arlington county, virginia nofa workshop july 24, 2018€¦ · key steps in performance management...
TRANSCRIPT
07/25/2018
1
Arlington County, VirginiaNOFA WorkshopJuly 24, 2018
Mary K. WinklerSenior Research Associate, Urban Institute
Center on Nonprofits and [email protected]
Agenda
• Landscape of Nonprofit Measurement and Evaluation• Evaluation v. Performance Measurement• Overview of Logic Models• Outcomes v. Indicators• Small Group Activity• Q&A
07/25/2018
2
Current State of Affairs
• Funders/donors and nonprofits want better information.• Nonprofits lack data/ability to produce outcome measures.• Funders/donors/nonprofits don’t know how to use information.• Competing/cumbersome reporting requirements.• Resources are becoming more competitive.• More often than not a “top down” approach.• Culture wars between compliance and learning.
07/25/2018
3
What is a learning culture?
“….one that constructively responds to new information, adapts based on experience with both successes and failures, and becomes strong and more effective over time…
…A learning culture is largely created and reinforced by leaders, and is supported by organizational processes and incentives…”
Source: Evaluation for Strategic Learning: Assessing Readiness and Results, Anna Williams, www.evaluationinnovation.org
Performance vs. Compliance
Look what we accomplished? We followed all the rules
I wonder why we missed We’ve been doing things this our target? way for years.
Let’s ask the team what else Let’s show these results to thewe can try to get better results? director and ask what to do
What can we do to better You’ll just need to try harder in support you in your work? the future.
How can we encourage families Parents are already too busyto share in our success? and probably won’t want to
attend another meeting.
07/25/2018
5
Measure4Change
• Small Grants• One on One Technical Assistance• Knowledge Briefs• Community of Practice
07/25/2018
6
Evaluation Defined
“A structured process of assessing the success of (a program or) project in meeting it’s goals and to reflect on the lessons learning.”
excerpted from the Evaluation Toolkit –- http://evaluationtoolbox.net.au
Many Different Types of Evaluation
• Performance monitoring or measurement• Process evaluation• Implementation evaluation• Outcomes evaluation• Non-experimental design• Quasi-experimental design• Random control trial
07/25/2018
7
13
Comparison of Two Approaches to Evaluation
PERFORMANCE MEASUREMENT
IN-DEPTH EVALUATIONS
1. Depth of Information: Only tells “the score”, not WHY
Can identify reasons for poor performance
2. Timeliness of Findings: Relatively quickly Can require months and even years
3. Skills Required: Considerably less needed Special skills needed
4. Quality of Analysis: Usually limited and weak Strong
5. Cost: Low High
6. Utility: “Minor” program decisions; Program Improvement
Major program decisions
What is Performance Measurement
“Regular measurement of progress toward specified outcomes is a vital component of any effort at managing for results.
A customer-oriented process that focuses on maximizing benefits and minimizing negative consequences for customers of services and programs.”
(Harry Hatry, Performance Measurement: Getting Results, p.3)
07/25/2018
8
Key Steps in Performance Management
1. Identify mission, objectives and clients.2. Identify the results (outcomes) the program seeks.3. Select specific indicators for measuring each outcome.4. Identify appropriate data sources for each indicator and how to
collect this information.5. Develop an analysis plan, including presentation formats.6. Collect and analyze the date, develop reports.7. Share data with staff and stakeholders8. Use information to make changes or improvements.
Performance Cycle
• Source: Isaac Castillo
07/25/2018
10
Logic Models & Theories of Change
• An opportunity to communicate within and outside your organization organizational direction and to link core assumptions about your work with intended results.
• A means of articulating organizational goals and purpose (communications and outreach) with short- and long-term statements of progress (measurement and evaluation).
Logic Models
A tool to help convey a scheme, program, or project in a brief, visual format.
• Common language among stakeholders• Participatory learning opportunities• Document and emphasize results or outcomes• Clarify knowledge about what works and why• Provide a credible reporting framework• Improved design, planning, and management
Retrieved from: http://www.sagepub.com/upm-data/23937_Chapter_1___Introducing_Logic_Models.pdf, on 10/15/2013
07/25/2018
12
DOD War Strategy
Bridging OutcomesBridgingOutputs
Student-Level• Program
enrollment• Program
completion• Participation in
offered activities
• Completion of offered activities
Program-Level• Counselors &
instructors continually use data to support & assess learning.
Inputs
• Service delivery model
• Curricula• Service
delivery partners
• Technology & tracking systems
• Staff/Personnel
• Matching dollars
• Technical assistance
• Opportunity youth
Functional ActivitiesActivities necessary to set up & run the program, which feed
back into inputs• Local adaptation of Back on
Track intervention(s)• Develop procedures• Identify and organize service
delivery partners• Identify funding sources• Hire staff/initiate contracts• Train staff• Identify & reach out to youth• Screen/enroll youth• Determine & establish
technology & tracking systems• Monitor tracking system data to
improve processes & services
Implementation Activities
Programmatic ActivitiesService delivery activities to achieve outputs & outcomes
• Support dual enrollment & provide simulated college experience before enrollment
• Develop college knowledge & success strategies
• Provide personalized guidance & connection to best bets (navigation)
• Current students receive mentorship from program graduates
• Transition youth to college
Medium-Term OutcomesConfirmatory• Students apply to training programs, including college• Students enroll in training programs, including college• Students earn college credits• Students enroll for more than one semester• Student enroll for more than one year• Students earn college or industry-recognized credentials• Students complete 12 credits of college workExploratory• Students do not experience subsequent arrests or
incarceration
Short-Term OutcomesSoft Skills for Success • Students gain postsecondary & career navigation skills• Students exhibit career-ready skills & behaviorsAccess College/Career Coursework• Students have access to postsecondary facilities• Students enroll in credit-bearing courses appropriate to
reinforcing essential skillsSubject Knowledge Gains• Students demonstrate proficiency in key skills & subject
areasSupport Systems• Students have a sufficient care network• Students build social networks supportive of
postsecondary success
Long-Term Outcomes• Students are not disconnected from work and education• Students become employed• Students are employed in quality jobs• Students’ earnings are higher than they would be
otherwise
Contextual factors: Local labor market, service infrastructure for youth, juvenile justice system, availability of adult and postsecondary education options, policies affecting youth (e.g., GED changes, Pell Grant availability)
07/25/2018
13
Transitional Housing Program Description To facilitate the movement to permanent housing of the homeless who are willing and able to work towards self-sufficiency by providing shelter and other services that may include meals, laundry, transportation, child care, and counseling/case management for housing, employment, substance abuse, public benefits, and referral to outside services. Also known as transitional shelter. Outcome Sequence Chart Intermediate Outcomes End Outcomes Indicators
Satisfaction with program services is an outcome that occurs within almost every program area, yet does not necessarily have a sequential placement. The indicator may be: Number and percent of clients satisfied with transitional housing programs and services.
Sources Consulted: Developing Community-wide Outcome Indicators for Specific Services by Hatry and Lampkin (2003); Volunteers of America National Program Outcome Measures and Program Standards FY 2005 (Volunteers of America, 2004); Core Performance Indicators for Homeless-Serving Programs Administered by the U.S. Department of Health and Human Services by Trutko and Barnow (2003).
Clients complete registration/ intake
Clients participate in in-house and referral programs
Clients complete service plan or ‘graduate’ the program
Clients’ long-term housing needs are met
Clients maintain stable and secure housing
2. Number and percent of new clients accepted for services during the reporting period.
4. Number and percent of clients who have met with counselor/case manager and developed a treatment/service plan within X days of entry.
Clients develop a program/ service plan
5. Number and percent of clients who, as a result of their service plan, participate in supportive services (in-house or by referral) within X days of entry.
7. Number and percent of clients who complete service plan/ shelter program.
8. Number and percent of clients who move to long-term housing within 24 months of entry.
9. Number and percent of clients who maintain safe and permanent housing for six months or more. 10. Number and percent of clients who do not re-enter the homeless system within one year of obtaining permanent housing.
Participant satisfaction
6. Number and percent of clients in need of mental health/ addiction treatment who enter into such treatment within X days of entry.
Clients apply to transitional housing
1. Number of clients applying for transitional housing during the reporting period.
3. Number and percent of households/ persons in transitional housing during the reporting period.
Youth Development Logic Model
•
After School Program Description An array of safe, structured programs that provide children and youth a range of supervised activities designed to encourage learning and development outside of the typical school day. Intermediate Outcomes End Outcomes Indicators
Increased enrollment in after school programs
Improved program completion by participants.
Academic Achievement
Character Development
Prevent undesired behaviors
Increased parental involvement in children’s education and development.
Full-time employment
Workforce Development
1. Number of participants who enroll in after school program.
2. Number and percent of participants who completed the program.
6. Number and percent of participants who improve their test scores or grades. 7. Average number of educational levels advanced per program participant. 8. Number and percent of participants who receive a high school diploma or GED.
9. Number and percent of participants who enroll in postsecondary education or occupational skills training program.
3-5. Number and percent of parent participants who (a) regularly help children with their school work, (b) attend parent meetings or school activities, (c) read to their children daily.
13. Number and percent of participants who obtain a full-time job after completing school.
10-11. Number and percent of participants who (a) obtain a job within 3 months of program completion, or (b) remain employed 12 months after program completion.
12. Number and percent of participants who enroll in college or voc education program.
Enrollment in post-secondary education
Good citizenship
07/25/2018
14
Additional Tips
• Engage staff and stakeholders (“buy in”).• Look to other program examples/models.• Consider starting with outcomes and working your way backwards.• Avoid jargon and use of subjective terms.
(e.g., adequate, sufficient, appropriate)• Periodically revisit and revise.• Tailor formats and level of detail to audience.
(e.g., layered approaches, visualization)• Keep it “logical”.• Resource: “W.K. Kellogg Foundation Logic Model Development Guide,”
https://www.wkkf.org/resource-directory/resource/2006/02/wk-kellogg-foundation-logic-model-development-guide
07/25/2018
15
Outcomes vs. Outcome Indicators
Outcomes Indicators
The benefit or change for intended beneficiaries or target groups.
Identifies what is being measured to track the program’s success on the outcome. The observable and measurable evidence that tells you whether program is achieving its desired outcomes.
Usually expressed as a number, percentage or rate.
07/25/2018
16
Outcomes vs. Outcome Indicators
Outcomes Indicators
Example 1: Improved Parental Involvement in School
Indicator 1: #/% of parents attending school meetings.
Indicator 2: #/% of parents who volunteer at school activities.
Indicator 3: #/% of parents who help their children with their homework.
Outcomes vs. Outcome Indicators
Outcomes Indicators
Example 2: Children are ready for kindergarten
Indicator 1: #/% of children who meet basic school expectations for kindergarten readiness (e.g., can write first name, recite alphabet, count to 10.)Indicator 2: #/% of children who meet or exceed baseline test scores for K-readiness.Indicator 3: #/% of children demonstrating age-appropriate social skills.
07/25/2018
17
General Principles for Indicator Selection
• Some outcomes are easy to measure, others are not (e.g., graduation rates versus health condition or well-being).
• There are usually multiple possible indicators for a single outcome.
• Sometimes you have to settle for an indicator that “best represents” the outcome – these are sometimes called “proxy” indicators.
• Indicator selection may need to take into account ease access or availability of data, cost to obtain, timeframe/lags.
Data Collection Considerations & Tradeoffs
07/25/2018
18
PerformWell Demonstrationperformwell.org
Hands-on Activity
• Post-it note exercise
Organize post-it note items in a logical order and group by indicator type: (a) input, (b) output, (c) short-term outcome/result, and (d) long-term outcome/result.
07/25/2018
19
Additional Resources
• Measure4Change Knowledge Briefs:http://www.urban.org/measure4change
• Outcome Indicators Project and Nonprofit Series on Outcomes Measurement:http://www.urban.org/center/cnp/Projects/outcomeindicators.cfm
• PerformWell:performwell.org
• East of the River Initiative – Evaluation Matters Report:http://www.urban.org/publications/411961.html
• The Performance Imperative (videos, resources, PIOSA):http://leapambassadors.org/