The Narrative Core Reference GuideImpact Evaluation Discussion

July 2013

1

"Measurement is the first step that leads to control and eventually to improvement. If you can't measure something, you can't understand it. If you can't understand it, you can't control it. If you can't control it, you

can't improve it."- H. James Harrington

Mission

The Numbers

The Comments

The Reactions

IMPACT

2

Table of Contents

• Executive Summary• Background• Why Measure?• Why don’t we measure?• Can we get to a single metric for all?• Reflection Questions – What are the top 5 questions every organisation should be

able to answer?• ABC – About Being a Charity – Our IMPACT Glossary• Next Steps

• Backup Material:– Measurement tools– Additional Resources– Impact Case Studies

3

Executive Summary

• Within the framework of the Narrative, it is important to address « impact evaluation » • We need to ensure we are able to measure and describe our performance• Although there are some things we can never control, the economy, downloading etc., we

can try to understand our impact better through our actions and modify our programs accordingly

• There are many tools available to do this and our purpose within this Guide is to provide you with information on what is available – not recommend which one should be used. Although this is beyond the scope of this report, the sector would benefit if we could determine which tools is best to use in which circumstance

• We should begin capturing « innovation » as a concrete result of our work with supporting examples

• We cannot use « one size fits all » reporting for all organizations when speaking about impact

4

Background

When constructing the Narrative Reference Guide, a discussion surrounding the appropriate way to approach the question of impact measurement has generated considerable conversation.

• How to we address this key area?• What is the appropriate amount of information to include?• How far down the road do we go on this question within the Narrative initiative? This could

conceivably be a project all on it’s own• There are many ways to look at this question given the diversity of the nonprofit sector• We wish to maintain complete transparency, without loosing people « in translation » i.e.

explanations needed for the various measures• We would like to provide the appropriate « framework » for this very important issue• How do we measure “Innovation” and how do we talk about it? Measurement of innovation is

difficult but we can create the conditions that encourage/ enable innovation

5

Background

So….what do we want to do….

• Give helpful and thought provoking reflection points to help in the journey to understanding and explaining « what success looks like »

• Explain various methods that can be used within the organization to « Measure » • Help frame conversations to support discussions concerning “Impact” with partners and

stakeholders

Stress the importance of Measuring Impact

6

Why Measure?

• Strategy and performance measurement should focus on what output (funders have us focus on outputs, we need to get beyond this) and outcomes the organization intends to achieve, not what programs and initiatives are being implemented

• Strategy is not only what the organization intends to do, but also what it decides not to do, a message that is particularly relevant for nonprofits

• Attempting to be everything for everyone virtually guarantees organizational ineffectiveness

• The nonprofit groups that report the most success in developing performance measures all have developed specific, actionable, and measurable goals to bridge the gap between their lofty missions and their near-term operating objectives – these are often called SMART objectives

S = SpecificM = MeasureableA = AchievableR = RealisticT = Timely

7

Why don’t we measure?

A lack of funding or resources is seen by charities as the greatest barrier. And in many cases this is true as some forms of evaluation can be very expensive and funding is often not available for this purpose.There are three more major barriers cited by a majority of respondents:

• not having the right skills and expertise (61% of respondents); • not knowing how to measure (53% of respondents); and • not knowing what to measure (50% of respondents).

Source: Making an Impact (UK Survey October 2012)

We need to determine ways to work with funders to ensure we are both collecting the right kind of data to impact our program delivery

8

Can we get to single metrics for all???

• We recognize that we can’t

• Metrics are as individual as the organization's mission. Therefore the « one size fits all » solution cannot be applied however, there could be groups of organizations working on the same topic area (i.e. homelessness) who could share best practices

• Here’s what CAN be applied to all of us:– We can build a framework of common language to explain what we are measuring to ALL

audiences– We can increase our awareness of the various measurement tools and apply them to our

organization– We must link our measures with our mission– We should start demonstrating concrete examples of “innovation” in our results– We need to « lay our language » so that a Grade 5 student will understand it (this should

resonate with Funders and the Public alike)– The more abstract the mission is, the more difficult it is to develop meaningful measures of

outcome or mission impact – so let’s be really clear with our “raison d’être”

9

Reflection questions…

– What is our organization aiming to accomplish? How do we know whether we are accomplishing what we set out to do?

– How can we make informed decisions about the ongoing use of resources?– What are our strategies for making this happen? – What are our organization’s capabilities for doing this? – How will our organization know if we are making progress? – What have and haven’t we accomplished so far? – How can we test and convince others of what we have achieved and show the

value of our social outputs in terms that would be understood by those financing us? Those who need us? Those who support us (employees, volunteers, future stakeholders)?

What are the TOP 5 questions all organisations should be able to answer?

10

Reflection questions…more

– What services are we willing to defer in the short-term so that we can invest in building the capacity necessary to serve more people and/or have greater impact in the long-term?

– Do we have a plan to diversify revenues beyond the government grants and traditional philanthropy on which we’ve depended?

– In the course of pursuing our mission have we created assets that have a marketplace value and could therefore be leveraged and monetized?

– Do we have clarity and precision around the question of what success looks like and, if so, are our resources aligned with our strategy to achieve a specific and measurable outcome?

– What changes in public policy are essential to achieving our organization’s mission, and have we communicated that as effectively as possible to the appropriate stakeholders?

11

ABC – About Being a Charity – Our IMPACT Glossary

Key definitions in impact measurement

• Impact: The overall difference an organization, programme or intervention makes. We use the terms ‘impact’ and ‘results’ interchangeably throughout this report.

• Outcomes: The results of a programme or intervention. What has changed as a result of your intervention?

• Impact measurement: The set of practices through which an organization establishes what difference its work makes. We use the term to define a broad set of practices that includes measuring both outputs and outcomes. This does not mean we equate measuring outputs with measuring impact—it is just part of the overall process.

• Monitoring: A systematic way of collecting and recording information to check progress against plans and enable evaluation. We need to better understand the value of our data

• Evaluation: The use of information from monitoring and elsewhere to judge and understand the performance of an organization or project.

12

ABC – About Being a Charity – Our IMPACT Glossary

Types of data and different collection methods

• Outputs: The products of an activity—for example, number of people attending a workshop.

• Outcomes: The results of an activity—for example, improved confidence resulting from a workshop.

• Case studies: Participants’ stories about their experiences of an activity. • Customer satisfaction forms: Forms asking participants how happy they were with an

activity. • Questionnaires designed by your organization: Questionnaires tailored to your

activities. • Standardised questionnaires: Published questionnaires, usually tested and validated. • Standardised sector tools: Common frameworks in use by multiple organizations. • Standardised government-produced tools: Common frameworks developed by

government.

13

ABC – About Being a Charity – Our IMPACT Glossary

Explanation of advanced evaluation practices

• Before and after measures: Measuring the same indicators before and after an intervention to assess what has changed. Benchmarking is also important.

• Economic analysis: Assessing the financial value of an intervention through methods including cost-effectiveness analysis, cost-benefit analysis or Social Return on Investment.

• Long-term follow up: Measuring the same indicators over a longer period to assess what changes have been sustained.

• Studies with control groups: Comparing results between those who have received an intervention and those who have not.

• Randomised control trials: Providing an intervention to part of a group that is randomly selected, and measuring changes across the whole group.

14

Next steps…

Obtain input from small “Evaluation "working team as to their thoughts on addressing the « measurement of impact » subject. (July 5, 2013)

Present to the Narrative Mobilisation Team (July 12, 2013) Adjust as appropriate Integrate into the Narrative Resource Guide tool Build an interactive way to collaborate and build on Impact Case Studies i.e. wiki

15

Backup Material

• Description of various measurement tools• Resources for additional information• Impact Case Studies

16

Which measurement tool to use???

There are several models that are explained here, however the « right » answer is only in the hands of the organization and their constituents.

These are provided as possible paths that will lead to the ability to quantify, demonstrate and communicate your organization’s performance and alignment to your Mission.

17

Balanced Scorecard (BSC)

Pioneered by Kaplan and Norton (1996), the BSC is a holistic performance management tool that is intended to support the pursuance of an organization's mission by focusing management attention on controlling the performance of internal activities perceived as causally related to achieving the stated goals or objectives derived from it. The corporate model of the BSC includes a financial perspective, customer perspective, internal process perspective, and a learning and growth perspective. Understandably, the model used in the for-profit world put emphasis on financial performance among all other domains. The non-profit model of the BSC amended Kaplan and Norton's original model to match the needs of SEs. Bull's (2007) model accommodated the social and environmental returns into the financial perspective and reduced the predominance of the financial performance to provide a more balanced weight of all domains.

A strength of the BSC model is that it has the capability to establish causal links between non-financial measures to overall mission success. The BSC can ensure that SE managers make decisions that are strategy-led rather than reactions to short-term conditions in the marketplace. However, in common with the other largely qualitative metrics, it lacks a comparative dimension and therefore is of most value in internal decision making rather than demonstrating public accountability.

18

Social Audit

Social auditing reports are basically longitudinal assessments of an organization’s internal performance using descriptive metrics. It enables organizations to evaluate and demonstrate the social, environmental, and economic benefits as well as limitations of its projects. In this manner, organizations can measure the degree to which it has fulfilled its missions and objectives. This performance metric is adapted from Zadek’s (1998) principles of social accountability. Social auditing focuses on measuring the impact of organizational performance particularly on how it has met non-financial objectives through a systematised and regular monitoring of its internal performance and stakeholder views and opinions.

This performance metric is strong in providing the organization with a thorough narrative of the specific objectives and activities that the organization can utilize to demonstrate how it has progressed over time. The impact of projects is described in “social accounts” which map progress measured against strategic objectives. An example would be to track down 1) the number of people who gained employment or 2) the number of beneficiaries of health care programs.

A weakness of the social audit is that it fails to quantify how an organization has actually accomplished its social purpose. Like the balanced scorecard, it lacks comparative value and its nature as a marketing strategy for stakeholders make it a poor measure of performance for organizations.

19

Triple Bottom Line

The simplest of the qualitative social metrics is the triple bottom line (Elkington, 1997, 2001). This model requires an enterprise’s accounting system to incorporate not only the traditional measures of financial performance, but also social and environmental outcomes. However, unlike financial accounts, the social and environmental audits are typically descriptive, rather than quantitative, and partial and subjective rather than complete and objective. Any external comparative dimension is also typically lacking (although internal, longitudinal comparison is possible). This is primarily the consequence of the lack of agreed social and environmental performance benchmarks. Finally, in this model, the three bottom lines are not weighted or integrated into any final statement of performance.

The triple bottom line is very useful for conventional businesses, as it reminds them to consider the social and environmental outcomes of their commercial behaviours and audit their progress in all three areas. However, for a social venture the model has little value. Since such activity is intrinsically concerned with generating social and environmental outcomes, as well as commercial returns, there is little need to encourage an acknowledgement of the role of all three in assessing business performance.

20

Family of Measures

As a reaction to the perceived lack of focus of existing social metrics, Sawhill and Williamson (2001) developed the Family of Measures model to better articulate a venture’s progress towards its mission objectives. As they commented, ‘Every nonprofit organization should measure its progress towards fulfilling its mission, its success in mobilizing its resources, and its staff’s effectiveness on the job’. The key point they made was that not-for-profit organizations needed to devise metrics that reflect the detail of their mission objectives, rather than their organizational performance. Thus, a social venture would need to focus less on its overall level of sales growth or profitability and more on the social impact it is achieving.

The model was built around three sets of linked metrics: impact measures; activity measures; capacity measures. Impact measures aim to map the progress towards fulfilling the organization’s mission and the long-term objectives that drive it. For these to work, the elements of the mission must be broken down into quantifiable and specific objectives. It may be that detailed research is required for this to be effective. Activity measures chart the progress towards goals and programme implementation that drive organizational behaviour. These would include the success or otherwise of mobilising resources toward programme implementations. Capacity measures capture the effectiveness and progress of all levels of the organization that keep it operational. These might include resource allocation and fund raising or market share.

21

Social Return on Investment

Social Return on Investment (SROI) is a measure that captures the value of social benefits. It is a relatively new measure developed by Jed Emerson and the Roberts Enterprise Development Fund (REDF). SROI represents a development of traditional cost-benefit analysis as a way of translating some of the social objectives of organizations into financial measures (generally gains or losses to public expenditure).

It represents the most sophisticated attempt to measure the impact of social ventures that is currently available. NEF comments: Social Return on Investment mirrors the standard financial measure of economic return but shows how organizations of all kinds create value beyond the economic. This is particularly true for those organizations in the social economy that may search for either economic and social value, or just social value. When compared to mainstream businesses, they may or may not achieve similar levels of financial return, but even if they do not, the value to society of the social or environmental returns that they create may well be equal or higher

The SROI model is in three parts: the first calculates the full blended value of a social venture (combing its enterprise value and social purpose value); the second establishes the investment in the project; the third then calculates the blended return on investment (combining the enterprise and social returns).

22

The Logic Model

A logic model (also known as a logical framework, theory of change, or program matrix) is a tool used most often by managers and evaluators of programs to evaluate the effectiveness of a program. Logic models are usually a graphical depiction of the logical relationships between the resources, activities, outputs and outcomes of a program.

While there are many ways in which logic models can be presented, the underlying purpose of constructing a logic model is to assess the "if-then" (causal) relationships between the elements of the program; if the resources are available for a program, then the activities can be implemented, if the activities are implemented successfully then certain outputs and outcomes can be expected. Logic models are most often used in the evaluation stage of a program, they can however be used during planning and implementation.

23

Developmental Evaluation

Developmental evaluation has emerged fairly recently as a way to support adaptive learning in complex and emergent initiatives. Combining the rigour of evaluation with the flexibility and imagination required for development, this new form of evaluation brings critical thinking to bear on the creative process in initiatives involving high levels of uncertainty, innovation, emergence, and social complexity (Gamble, 2008).

• A Developmental Evaluation Primer – McConnell Foundationhttp://www.mcconnellfoundation.ca/assets/Media%20Library/Publications/A%20Developmental%20Evaluation%20Primer%20-%20EN.pdf

• Abc de l’evaluation evolutive – La Fondation McConnell http://www.mcconnellfoundation.ca/assets/Media%20Library/Publications/A%20Developmental%20Evaluation%20Primer%20-%20FR.pdf

• A Practitioner’s Guide to Developmental Evaluation http://vibrantcanada.ca/files/development_evaluation_201_en.pdf

24

Additional Resources –sources for tools

With thanks to Ms. Becky RossiMeasuring & Managing the Social Performanceof Nonprofit Organizations:Results of a 2011 Survey of Canadian Nonprofit Executives

25

Some Other Resources

• Harvard: The Limits of Nonprofit Impact: A Contingency Framework for Measuring Social Performance (2010) http://www.hbs.edu/faculty/Publication%20Files/10-099.pdf

• Stanford University - Stanford Social Innovation Blog – Measuring Social Impact (Spring 2013)http://www.ssireview.org/topics/category/measuring_social_impact

• Charity Village Resource: What’s your nonprofit’s impact? (2012)https://charityvillage.com/Content.aspx?topic=What_s_your_nonprofit_s_impact

• Global Innovation Centre - Innovation Performance Measurement (2013)http://innovationcenter.deteconusa.com/articles/innovation-performance-measurement-assessing-and-driving-the-innovation-performance-of-companies/

• Centre for Impact Investing Knowledge Hubhttp://impactinvesting.marsdd.com/knowledge-hub/resources/

26

Case Studies

• Case examples of best practices in each area: (Boston Consulting Group)– http://www.bcg.com/expertise_impact/impact_stories/default.aspx– Specific story on Pathways to Education impact:

http://www.bcg.com/about_bcg/social_impact/Education/ImpactStoryDetail.aspx?id=tcm:12-24705

– Junior Achievement Impact Study (2011)http://www.jaimpact.com/pdf/BCGReport.pdf