gates fdn measuring impact presentation nov 30 2010

Prove It: How the Gates Foundation Ties Strategy to Results

Jodi Nelson, Interim Director

Emily Parker, Sr. Strategy Officer

Impact Planning and Improvement

September 30, 2010

Schwab Charitable Philanthropy Speaker SeriesCenter for Nonprofit and Public Leadership

Haas School of Business

What are your questions about

strategy and measurement

at the Bill & Melinda Gates

Foundation?

2© 2010 Bill & Melinda Gates Foundation |

Today we would like to:

Collect your questions

Share our aspiration: is it about “proving” it?

Share what we do in practice

Share our big questions

Ask for your ideas from other experiences and sectors


Fast Facts about the Bill & Melinda Gates Foundation

4

(1) Number of countries in which the foundation has granteesSource: Bill & Melinda Gates Foundation Fact Sheet, September 2010, gatesfoundation.org

© 2010 Bill & Melinda Gates Foundation |

Year founded 1994

Employees 858

Offices 5

Number of strategies 25

Geographic reach (1) 100+ countries

Total endowment $33B

Grants since inception $23B

Grants in 2009 $3B

Changes over the past decade set the context for strategy and measurement

5

From To

Foundation growing rapidly

Pay out requirement hard to meet

A few strategies (i.e., high school education, US libraries)

No consistent approach to strategy development or measuremento little guidanceo let a thousand flowers bloom

Foundation at a stable size

Focus on use of limited resources

25 strategies across 3 broad sectors: global health, development, US education

Approach to strategy maturing

Actionable Measurement beginning to take shape o in learning mode—from inside and outside


Imagine…

You lead the Agricultural Development program at the Bill & Melinda Gates Foundation.

You have to decide how the foundation should invest in agriculture to help alleviate poverty in Sub-Saharan Africa and parts of Asia.

What will the foundation’s goals be? How will we achieve them? How will we know if they are achieved?

You have a meeting with Bill & Melinda in six months and need to tell them your plan.

What do you do?

© 2010 Bill & Melinda Gates Foundation | 6

Why is it so tough?

The world is complex — change is caused by many different things.

We are just one actor — governments, NGOs, other donors, private citizens are also trying to improve people’s lives

We are not on the “ground” — we are strategic donors, but still just donors.

Data can be hard to come by — we may know we need it, but collecting meaningful data is hard to do.



So what do we do?

Use tools of strategy and measurement to plan our workdespite the complexity

8

Our aspiration is to create feedback loops so we can plan, execute, measure, learn, adjust, plan …

9

Execute

Develop/ refresh strategy

Track progress and evaluate what works

Review strategy

Learn, adapt, improve

data &

experience

data &

experience

data

&

expe

rienc

e

data

&

expe

rienc

e


Let’s break it down….


What is a strategy at the foundation?

11

Causal pathway to impact. Outlines the investments and programmatic activities aligned with that pathway. Defines the results of these investments and activities over time.


Global Development Program Agricultural Development Financial Services for the Poor Global Libraries Water, Sanitation, and Hygiene Policy and Advocacy

Global Health Program Discovery Enteric Diseases and Diarrhea Family Planning HIV Malaria Maternal, Newborn, and Child

Health Neglected and Other Infectious

Diseases Nutrition Pneumonia Policy and Advocacy Tobacco Tuberculosis Vaccine Preventable

Diseases/Delivery

United States Program College Ready Early Learning Pacific Northwest Postsecondary Success U.S. Advocacy U.S. Libraries

Bill & Melinda Gates Foundation Approved Strategies

Policy and Government Affairs Charitable Sector Support

Our vision of success Strategies are constantly improving, based on measurement and learning

12

“No strategy survives first contact with the enemy.”

Measurement and learning—coupled with systematic review, reflection, and intellectual dialogue—drives improvement.


intended strategy

unrealized strategy

deliberate strategy

emergent strategy

realized strategy

Figure created by Henry Mintzberg, strategy theorist.

Our business process reflects this philosophy

13

Strategy development: creating a strategy “from scratch”

Strategy review: annual update for the CEO and co-chairs on progress against an approved strategy

Strategy refresh: re-examining an approved strategy with the expectation there will be a significant change in strategic direction

What When

Once: after a period of learning

Every strategy, every year

Every three to five years


Strategy development and refresh is an iterative, phased process with progressive analyses from broad to narrow


Three concepts underpin strategy development and refresh

15

segmentation(s) of the problem/ space and identification of opportunities for change

hypothesis about what needs to happen for the stated goal to be achieved

What must the world do?

what we will do (directly or through others) to help achieve the stated goal

What must we do? How will our actions

maximize leverage?

What it is

Key question


Opportunity map Theory of changeTheory of action solution leverage partner leverage

Where is the leverage?


“Learning initiative”

established

Limited grantmaking

begins

Strategy developed & approved--

the sky’s the limit

Endless possibilities;

limited resources

Strategy review—

strategy is adjusted

???

What about measurement?

What is it anyway?

Why is it so difficult to do well?

What is our philosophy and approach?



What is measurement?

By definition: Obtaining the magnitude of a quantity

Performance measurement o Assessing the achievement of pre-determined goals and objectives

o Produce objective, relevant information on program or organization performance

o Developing measurable indicators that can be tracked to assess progress

What is unfortunate about the word “measurement” o Not everything is about counting or “tracking”

o Under emphasizes evaluation, mixed methods, asking and answering questions about how things happen and why — not just what happens

18

What's our philosophy? Actionable Measurement

Direction from the co-chairs in 2008

Measurement should be strategic and actionable

We should not measure everything:

o Rely on grantee reporting as much as possible

o Align our data requests with those of other funders

o Learn from others

o Leverage technology for data collection and viewing results


Actionable Measurement

20

Measure to inform decisions and actions. How you do it (the approach, methodology, evaluation design) is determined by the purpose. There is no one measurement approach that works all the time, for all purposes.

Why is it unique to be pragmatic?

Where do we weigh in on the philosophical debates?

The “gold standard” is use



Tracking results across time/space

o Tracking execution is good for accountability

o But tough to do well but can describe change over time

Evaluation designs

o Impact evaluation purpose: demonstrating effectiveness of a model/approach, good

for replication, informing the field, testing assumptions to check strategy

o Process evaluation purpose: to improve implementation/execution at key times

o Participatory evaluation purpose: to empower participants with information to act

o Developmental evaluation: to identify components of a model or approach in real

time

Getting useful data requires choosing the right methods or

approaches to measurement given what you need to do with it

Example of purpose driven measurement

It is an approach to measure results of our strategies

22

Inputs Activities Outputs Outcomes Impacts

Strategy

Initiative Sub-Initiative

Grant

Sub-Grant

How is this different from the measurement challenge facing implementingorganizations (grantees)?



What’s the essence of the approach?

Measure impact (change in people’s lives) to track collective progress toward targets, not to attribute impact to the foundation’s actions or investments.

Measure key output and outcome results to track progress of the foundation’s work; use evaluation to inform decision making by testing assumptions, learning what works, how and why.

Selectively track results of grants at critical points for accountability; evaluate to learn about implementation of key investments; use impact evaluation when it seeks to answer a strategic question for an initiative not as an accountability tool.

Strategy

Initiatives

Grants

23

Measuring investment types

24

Priority for evaluation resources

Research Product Development

Model Development

and Demonstration of

Effectiveness

Delivery at Scale

Systems Change

Policy and Advocacy

Measure attribution where it is

technically feasible and ethical

Track execution and

coverage/reach

Measure desired outcomes, track progress, and focus on shorter-term feedback

Measure success or failure and extent of fit with target

product profile

Monitor outputs and

track process

Types of Investment

Measurement Guidelines Specific to Investment Type

Use multiple measurement methods to draw conclusions


Our questions…

25

Execute

Develop/ refresh strategy

Track progress and evaluate what works

Review strategy

Learn, adapt, improve

data &

experience

data &

experience

data

&

expe

rienc

e

data

&

expe

rienc

e


How do we identify, articulate, and evaluate

alternative paths to impact?

What results do we aggregate? What are the

most promising interventions to evaluate?

How do we adjust our strategies while maintaining

continuity in our partnerships?

How do we keep the bar high and hold people accountable

for results, while leaving room for failure and learning?

How do we develop theories of change where evidence

of what works is slim?

gates fdn measuring impact presentation nov 30 2010

Technology

gates foundation ties

measurementtofrom foundation

approved strategy

strategy review

strategy refresh

strategy theorist

use tools of strategy

actionable measurement