nfe april 2006 initiative/portfolio evaluation: a four-step model huilan yang, ph.d. new frontiers...
Post on 18-Dec-2015
215 views
TRANSCRIPT
NFE April 2006
Initiative/Portfolio Evaluation:Initiative/Portfolio Evaluation:A Four-Step ModelA Four-Step Model
Huilan Yang, Ph.D.New Frontiers in Evaluation
Vienna, AustriaApril 25, 2006
NFE April 2006
OverviewOverview
• Overview of Evaluation at the W.K. Kellogg Foundation
• What is an initiative?• What are the characteristics of an initiative
evaluation?• Description of the Four-Step Model• What are the caveats of using this model?• Does the model work?
– Some practical implications– Some after-thoughts and reflections
NFE April 2006
Some HumorSome Humor• DEMOCRATIC
– You have two cows.– Your neighbor has none.– You feel guilty for being
successful.– Barbara Streisand sings for
you.
• REPUBLICAN– You have two cows.– Your neighbor has none.– So?
• COMMUNIST– You have two cows.– The government seizes both
and provides you with milk.– You wait in line for hours to
get it. It is expensive and sour.
• CAPITALIST, AMERICAN STYLE– You have two cows.– You sell one, buy a bull, and
build a herd of cows.
NFE April 2006
Part IPart I
Overview of Evaluation at the Kellogg Foundation
NFE April 2006
The W.K. Kellogg The W.K. Kellogg FoundationFoundation
• $7.3B assets, established in 1930• Independent entity from the Kellogg Co.• Annual grantmaking amount $200M+ in US, Latin
America & the Caribbean, & Southern Africa• Five domestic programming areas
– Food System & Rural Development, Philanthropy & Volunteerism, Health, Youth & Education, local Battle Creek hometown
• Six Impact Services– Evaluation, Communication, Policy, Technology, Meeting
Services, Learning
NFE April 2006
Geographic LocationGeographic Location
Battle Creek
NFE April 2006
Historical ContextHistorical Context
• Cluster evaluation started in late 1980’s• Overall purpose: learn from and determine
impact of national social change work– Common issue, but local variation encouraged
• Over time – many definitions, applied in many different types of programs
NFE April 2006
Cluster Evaluation and Multi-Site Cluster Evaluation and Multi-Site EvaluationEvaluation
Multi -site Evaluation“Evaluation for confirmation”
Cluster Evaluation“Evaluation for learning”
Single intervention model, centrally designed, implemented at different sites.
Multiple intervention models, designed by different sites, according to local needs, resources, and constraints.
Specifics of model known, pre-tested, fixed. Specifics unknown; “cutting edge” and evolving models.
Limited number of narrowly defined goals that lead to dependent variables, common across sites.
Multiple possible goals, broadly defined, somewhat site specific; not all goals or benefits known in advance.
Good framework for testing hypotheses, causal linkages, and generalizability.
Good framework for strengthening programs trying to operationalize guiding philosophy or set of principles at local level.
Top-down project management and evaluation. Autonomous, locally driven project management; dual levels of evaluation.
Assumes controls can be established to maintain reliability and validity; believes in value of “generic model.”
Assumes some common goals, questions, experiences; believes that sharing information increases knowledge about “what” and “how”; values practical knowledge.
NFE April 2006
Along Came Initiatives…Along Came Initiatives…
• Late 90’s – WKKF and other foundations increasing focused on systems change
• WKKF distinguished between – CLUSTERS -- exploratory, designed to learn
about new field of work, 3 – 5 year funding)– INITIATIVES – systems change, driven by theory,
developed in stages, funding for up to 10 years
NFE April 2006
How Does Funding Strategy Influence How Does Funding Strategy Influence Evaluation?Evaluation?
DIMENSIONS Multi-Site Evaluation Cluster Evaluation Initiative Evaluation
Purpose of Evaluation Testing hypotheses, causal linkages, and generalizability
Generating Theories of Change (documenting)
Learning about Theories of Change
Focus of Evaluation On intervention model, which is centrally designed, implemented at different sites. Specifics of model known, pre-tested, fixed.
On process & learning from variability of implementation and outcomes in individual sites
On systems, evaluating systems change
Degree of Variability on Intervention & Outcomes
Limited number of narrowly defined goals that lead to dependent variables, common across sites.
Learning from particular experiences given sites’ specific contexts; learning from naturally occurring variability.
Generalization of findings across all sites
NFE April 2006
Influence…Influence…
Level of Rigor in Evaluation Design
Assumes controls can be established to maintain reliability and validity; “generic model.”
Heavily relies on qualitative measures to identify common trends among sites. Learn from natural variation.
Common criteria of success and measures across all sites; longitudinal studies; use of mixed methods.
Relationship Dynamics (Central / Project Evaluation)
Top-down project management and evaluation.
Project-level evaluations relatively independent; data generated aggregated by Cluster Evaluators.
Project-level evaluations are heavily influenced by the Initiative-level evaluation; information generated at local sites will directly inform success of initiative.
Role of Learning Evaluation emphasis is summative.
Learning takes central role; emphasis upon formative evaluation.
Learning is focused on Theory of Change. Evaluation provides feedback loop.
DIMENSIONS Multi-Site Evaluation Cluster Evaluation Initiative Evaluation
NFE April 2006
Influence…Influence…DIMENSIONS Multi-Site Evaluation Cluster Evaluation Initiative Evaluation
Alignment with project level evaluation activities
Centrally devised and mandated data collection assures alignment
Uses project level evaluation results as building blocks
Using initiative TOC to guide alignment:TOC; Systems ModelsLogic Model OutcomesAggregated impact indicators (could set common data elements)
Use of Systems Theories Optional Optional Yes
Scope of Outcomes / Impact (to be measured)
Narrowly defined depending on the particular intervention.
Focus on outcomes of projects within cluster
Changes in systems that will lead to different outcomes. (Final outcomes may be very long-term.)
NFE April 2006
Part IIPart II
What is an Initiative?
NFE April 2006
Define “Initiatives”Define “Initiatives”
• At the Kellogg Foundation, an initiative is defined as a strategically designed systemic change effort:– with clear outcomes and long-term, sustainable impact– with an intensive focus on sustainability– requiring a clear theory of change and/or logic model– target system and population clearly defined– involving multiple projects
NFE April 2006
Part IIIPart III
Initiative Evaluation Characteristics
NFE April 2006
Initiative Evaluation*Initiative Evaluation*
1. Uses multiple sites (ranges from 5 to 30) that vary in implementation approaches and/or outputs;
2. Assumes that situations are dynamic, changing, and continuously evolving;
3. Is conducted in a way that is philosophically congruent with the change theory of the initiative;
4. Includes a focus on learning from the different strategies and contexts of the funded projects;
5. Obtains multiple perspectives throughout data collection, analysis, synthesis, interpretation, communication, and use;
6. Uses networking conferences or other gatherings of the initiative’s project teams for group analysis of initiative evaluation data and reflection on its meaning.
* The document from which these are cited is not yet published by WKKF. So please do not reference or quote.
NFE April 2006
Part IVPart IV
The Four Step Model
NFE April 2006
Some HumorSome Humor
• AMERICAN CORPORATION– You have two cows.– You sell one, lease it back to
yourself and do an IPO on the 2nd one.
– You force the two cows to produce the milk of four cows. You are surprised when one cow drops dead. You spin an announcement to the analysts stating you have down-sized and are reducing expenses. Your stock goes up.
• FRENCH CORPORATION– You have two cows.– You go on strike because you want
three cows. You go to lunch and drink wine. Life is good.
• ITALIAN CORPORATION– You have two cows but you don't
know where they are. While ambling around, you see a beautiful woman. You break for lunch. Life is good.
• GERMAN CORPORATION– You have two cows.– You engineer them so they are all
blonde, drink lots of beer, give excellent quality milk, and run a hundred miles an hour. Unfortunately they also demand 13 weeks of vacation per year.
NFE April 2006
Content: What is Being Content: What is Being Evaluated?Evaluated?
• Philanthropy is defined as the giving of time, money, and know-how to advance the common good.
• Philanthropy and volunteerism grantmaking at the W.K. Kellogg Foundation is aimed at increasing the ranks of new givers and nurturing emerging forms of philanthropy. Through the program activities, we seek to unleash resources by supporting the emergence of new leaders and donors, creating and sharing knowledge, and building tools that advance the effectiveness of the philanthropic sector.
NFE April 2006
Unleashing Resources InitiativeUnleashing Resources Initiative
NFE April 2006
Figure 1. The URI as a Comprehensive Intervention Program
The URI Initiative
The Leaders Cluster
The KnowledgeCluster
The ToolsCluster
Project 1Project 2Project 3
……
Project 1Project 2Project 3
……
Project 1Project 2Project 3
……
Initiative Level
ClusterLevel
Project Level
Note: This is only a conceptual simplification of the URI, which is more complicated in structure than illustrated here. For an authentic description of the program, go to www.wkkf.org to get the details.
NFE April 2006
Projects
ClustersInitiative
Programming Talk
NFE April 2006
Evaluation Talk
Project EvaluationProject Evaluation
Cluster EvaluationCluster Evaluation Initiative EvaluationInitiative Evaluation
NFE April 2006
Initiative Evaluation Initiative Evaluation Means…Means…
• Establish common high level goals or outcomes
• Coordinate data collection methods, data sources, process, and/or instruments
• Identify major stakeholder needs• Build productive relationships among
evaluators at different levels
NFE April 2006
Unleashing Resources Initiative Unleashing Resources Initiative EvaluationEvaluation
• Evaluation is conducted at all three levels– Initiative level– Cluster level– Project level
• Nested, multi-layed, multi-dimensional nature– Project: its contribution to cluster & initiative success, plus its
unique aspects– Cluster: its contribution to initiative success, its interdepenency
with other clusters, plus its unique aspects– Initiative: overall initiative success, unique contributions from
projects, clusters, plus learning about strategies and informing future grantmaking
NFE April 2006
Without Alignment….Without Alignment….
• Evaluation efforts at different levels will become fragmented in many different directions, making the evaluation simply a collection of independent projects at the best.
• Duplicated data collection will waste human and financial resources.
• Some data sources might be tapped into more than once for information, thus adding tremendously burden and causing loss of good will.
NFE April 2006
Birth of the Four-Step-Birth of the Four-Step-ModelModel
• Alignment of all levels of evaluation– Exhaustive data sources– Efficient, yet effective data collection
• Minimize duplicated efforts• Meet all stakeholders’ needs: project, project
constituents and WKKF
– Best use of exiting data: annual reports, evaluation reports, networking meetings, etc.
NFE April 2006
The Four Step ModelThe Four Step Model
A Four-Step Model of the URI for Evaluation Alignment
The URI Initiative
The Logic Model
Evaluation Questions
Data Indicators
Evaluation Sub-Questions
Data Collection and Analysis
NFE April 2006
EvaluationQuestions
Data
Theory of Changeor Logic Model
Indicators
NFE April 2006
The Logic Model as the Guide & The Logic Model as the Guide & Foundation for EvaluationFoundation for Evaluation
• A logic model is a systematic and visual way to present and share your understanding of the relationships among the resources you have to operate your program, the activities you plan to do, and the changes or results you hope to achieve (W.K. Kellogg Foundation, 2001).
• www.wkkf.org -> knowledgebase -> toolkits -> evaluation -> Logic Model Development Guide
NFE April 2006
LOGIC MODEL for Unleashing Resources InitiativeBuilding Mutually Responsible & Just Society in which All Have the Ability & Will to Contribute to the Common Good
Issues and Opportunities Strategies Outcomes Impacts
LEADERS - LEADERS - LEADERS - LEADERS - LEADERS
1. New resources are leveraged for the common good.2. New resources are leveraged for work toward a mutually just society.3. New and emerging donors are engaged in the field of philanthropy and are working toward a mutually just society.4. Individuals from traditionally underrepresented groups are in key leadership positions in the field of philanthropy.5. Philanthropy learns from, values and uses diverse perspectives and forms for giving (time, money and know-how).6. Relevant knowledge is being produced, is accessible and used to inform philanthropic practice.7. Philanthropy uses appropriate, useful and effective tools to engage new and emerging leaders and donors.8. Nonprofits become more sustained and foundations are more effective.
Significant innovation is occurring in philanthropy and volunteerism. Connecting these innovators will catalyze that momentum and lead to new ways to give.
Exchange between emerging leaders and donors and established philanthropy can invigorate and guide philanthropic learning and practice.
Investment in the capacity of emerging, yet fragile, non-traditional individuals, institutions, and networks will solidify and sustain their ability to lead and shape the future philanthropic agenda.
Support and strengthen emerging leaders (the primary groups)Connect the six key populations
Connecting any population group(s) to critical marketplace opportunities.
1. Increased visibility and engagement through skills development and opportunities available2. Increased access of the primary groups to real opportunities for social change and impact for facilitating groups3. Demonstration of engagement with and influence of mainstream philanthropy by the primary groups
4. Increased sharing, cooperation and collaboration through networking of the six population groups5. Demonstrated high level of resources, ideas, energy, and commitment provided to the primary groups by facilitating groups
KNOWLEDGE - KNOWLEDGE - KNOWLEDGE - KNOWLEDGE
There remain large gaps in the body of knowledge about good practice in philanthropy and volunteerism.Resources in all sectors are highly fragmented which causes duplication, unnecessary competition, and waste in general. (KNOWLEDGE/TOOLS)
Strengthen the use of knowledge to increase capacity for the common good
6. Increased accessibility, usefulness and application to practice in philanthropy through synthesis and management of knowledge
TOOLS - TOOLS - TOOLS - TOOLS - TOOLS - TOOLS
The outdated support infrastructure for organized philanthropy is not built for nor aligned with knowledge management needs of today. This “disconnection” is further exacerbated by lack of perspectives from diverse populations and sectors. New infrastructure or re-invention of current/old structure is needed.
Improve philanthropy work through application of tools built upon knowledge
7. Increased accessibility and use of tools developed8. Advanced innovative ways of giving
NFE April 2006
From Logic Model to Evaluation QuestionsFrom Logic Model to Evaluation Questions
Initiative
Level
ClusterLevel
Evaluation QuestionsEvaluation Questions ProjectLevel
The The UnleashinUnleashin
g g ResourceResource
ssInitiativeInitiative
LeadersLeaders
__________________KnowledgKnowledg
ee
________________ToolsTools
1. To what extent are the primary groups 1. To what extent are the primary groups (including women, youth, and community of (including women, youth, and community of
color) becoming more visible and engaged in the color) becoming more visible and engaged in the field of philanthropy?field of philanthropy?
2. What level of access do the primary groups 2. What level of access do the primary groups have to real opportunities for social change and have to real opportunities for social change and
impacting the facilitating groups?impacting the facilitating groups?3. In what ways do the primary groups engage 3. In what ways do the primary groups engage
with and influence the mainstream philanthropy? with and influence the mainstream philanthropy? 4. Does the networking of the six population 4. Does the networking of the six population
groups succeed in promoting sharing, groups succeed in promoting sharing, cooperation and collaboration?cooperation and collaboration?
5. Does the networking of the six population 5. Does the networking of the six population groups succeed in promoting sharing, groups succeed in promoting sharing,
cooperation and collaboration?cooperation and collaboration?________________________________________________________________________________________
6. How accessible, useful and applicable is the knowledge generated to practice in philanthropy?____________________________________________
7. How accessible, useful and applicable are the 7. How accessible, useful and applicable are the tools developed to practice in philanthropy?tools developed to practice in philanthropy?
8. Are innovative ways of giving determined and 8. Are innovative ways of giving determined and advanced?advanced?
Grantees in Grantees in the Leaders the Leaders
ClusterCluster
______________________Grantees in Grantees in
the the Knowledge Knowledge
ClusterCluster______________________
Grantees in Grantees in the Tools the Tools ClusterCluster
NFE April 2006
From Evaluation Questions to From Evaluation Questions to IndicatorsIndicators
• Evaluation questions are too abstract.• To create a common ground for all grantees,
clusters and initiative evaluators, indicators are developed.– Indicators serving as working definitions for each
evaluation question– Evaluation question = outcome
• Two types of indicators– Quantitative– Qualitative
NFE April 2006
From Indicators to Evaluation Sub-From Indicators to Evaluation Sub-questionsquestions
• For each indicator, an evaluation sub-question is generated to guide data collection.
• Sub-questions need to be:– Directly based on the indicators– Different sets of questions for grantees in different clusters to
triangulate data– Blueprint for instrument development– Cooperation and collaboration across all evaluation efforts– Balance between qualitative and quantitative information
NFE April 2006
Aligning Data Collection and Aligning Data Collection and Analysis Efforts with Aligned Analysis Efforts with Aligned
Evaluation Sub-QuestionsEvaluation Sub-Questions• Make evaluation sub-questions available to evaluators
at different levels to:– Help the evaluators understand the interrelations among the
sub-questions– Provide useful guidance to evaluators as to how to collect and
analyze data to address each sub-question and, when needed, where to find additional help
NFE April 2006
To Sum UpTo Sum Up
1. Operationalize TOC/LM – making sense of the theoretical framework to assess initiative’s evaluability
2. Turn outcomes into overarching evaluation questions3. For each question, identify a list of feasible and desired
indicators4. Generate evaluation sub-questions from indicators
identified5. Clusters and projects “pick” indicators and use evaluation
sub-questions as a guide to collect data6. Initiative evaluators
• Aggregate data from cluster/project evaluations• “Fill the blank” with its own data collection
NFE April 2006
A Successful Case*A Successful Case*
• Tools Cluster Evaluation– Logic Model was developed using the URI LM as
a guide, keeping in mind the needs of the cluster– Guidebook developed by URI evaluators is
referenced when identifying data points– “It (the Guidebook) saves me a lot of time trying
to figure out what I need to do to ‘make Kellogg happy’ in terms of data needs…..In addition, I know the data I collect will have dual purposes.”
* Handouts of Tools Cluster Logic Model and Evaluation Plan chart
NFE April 2006
• Once upon a time, there was a prince….• …..the crystal slipper fits her perfectly. So
Cinderella got onto the horse-drawn carriage and was driven to the Palace, where the prince proposed to her. She accepted the proposal. They got married and a grand ball was given to celebrate the wedding.
• And they lived happily ever after.
NFE April 2006
1. Increased access, visibility and engagement of the three primary groups in giving
2. More collaboration among the six population groups to support giving
4. Improved access to useful knowledge about giving
5. Improved access to and use of tools
3. More innovative ways of giving growing out of Outcomes 1 and 2
6. Greater influence on mainstream giving by the primary groups
7. Increased giving among the six population groups on WKKF social change interests
Social changes of WKKF interests in communities
Supporting Leaders & DonorsCreating & Sharing KnowledgeBuilding Tools for NP Sustainability & Innovative Giving
Interrelationship Among Expected URI OutcomesInterrelationship Among Expected URI Outcomes
NFE April 2006
1. To what extent have the outcomes 1, 2, 4, and 5 been achieved as a direct result of URI approaches? We consider this as evaluation of immediate URI outcomes.
2. How has the achievement of outcomes 1, 2, 4, and 5 helped promote the recognition of the traditions of giving among the three primary groups in society? And how has the achievement of outcomes 1, 2, 4, and 5 promoted innovative ways of giving (outcome 3)? We consider these as evaluation of intermediate URI outcomes.
3. Have such recognition and promotion improved the practice of mainstream giving? If yes, in what ways has the practice of mainstream giving been influenced by the primary groups (outcome 6)? And has giving been increased among the six population groups as a result of recognition and promotion (outcome 7)? We consider these as evaluation of long-term URI outcomes.
4. What social changes of WKKF interests are advanced by URI? We consider this as evaluation of URI impact.
Four Sets of Evaluation Questions
NFE April 2006
Desired Outcome 1 Increased access, visibility and engagement of the three primary groups in giving
Programming Strategy[ ] Supporting Leaders & Donors[ ] Creating & Sharing Knowledge[ ] Building Tools for NP Sustainability & Foundation Effectiveness
Indicators(What will success look like if the desired outcome is achieved?)For this outcome, the questions could be:1.What have you done differently to engage primary groups?2.What difference does that make in the community?
ScopeIs this work reaching enough people to have desired outcome(s)?
SustainabilityHow will this work promote the recognition of giving among primary groups?
SustainabilityHow will this work promote innovative ways of giving?
So what?Will such recognition and promotion improve the practice of mainstream giving?
ImpactWhat social changes of WKKF interests does this work intend to advance?
For Example….
NFE April 2006
Indicator List• Representation on boards and staff• # of attendance at critical sector meetings (Substantive contribution on panels)• Contracts/Partnerships profile• Types of research, content focus, perspectives provided/sought after• Number of fellowships awarded to primary groups (e.g. ARNOVA)• Ways research on primary groups’ engagement was made visible (e.g. ARNOVA)• Degree of influencing visibility of research on primary groups’ engagement
(things are done differently) [ripple effect]• Topics, themes of conferences, seminars• Profile of new givers not there before• More culturally relevant tools to engage donors of color• Perception of the population
– Do people in communities of color feel engaged?– Do they think themselves as actors/givers?
• Amount given ($, time, know-how) by primary groups• More structures supporting giving of new funds, etc. – women, youth and
communities of color (are there more?)• New/additional structures for giving that are unique & appropriate to cultures• New practices emerged in giving• Innovations in giving (what are they?)• Changes in ways of working in grantee organizations
NFE April 2006
Philanthropy & Volunteerism Theory of Change
Vision: A mutually responsible and just society in which all have the ability and will to give time, money and know-how to the common good
Goal: Increased engagement in social change by youth, women, and people of color for effective and innovative impact
•Strengthened leaders and organizations•Increased access to decision-making•Improved access to and control of time,
money, and know-how•New solution and innovations in giving
•Supported collective and representative leadership
•Linked infrastructure organizations and support systems
Outcomes
Program Strategy•Value, honor, celebrate, promote, and connect
traditional & new approaches to giving•Push networks and support systems to be
more effective
•Support leaders and innovators•Create and share knowledge•Develop tools for effective
giving
NFE April 2006
Part VPart V
Reflections
NFE April 2006
NFE April 2006
Words of CautionWords of Caution
• Initiative evolution evaluation modification• Stay flexible and open-minded, looking for
unintended results• Always keep eyes on the ball, transcend trivial stuff• Intent: not establish cause-&-effect relationship
– Evaluation: improving, plausible relationship between what is done and what is achieved
– Research: proving, with emphasis on making judgments
NFE April 2006
RevelationsRevelations
• Traditional thinking & approaches of evaluation are not appropriate for developmental, evolving, or organic initiatives (where parameters are not clearly defined).– When initiatives come into existence based largely on program
staff’s intuition, evidence-based evaluation no longer serves the purpose.
– Formative evaluation may not be alternative, either.• Timing is off, when TOC might be changed; thus data collected based
on “original” evaluation design becomes outdated.– In resource-rich and time-poor environments, there exist
competing/conflicting priorities for evaluation:• Accountability (outcomes), Improvement (process), Learning (for what?)
NFE April 2006
After ThoughtsAfter Thoughts
• Portfolio/initiative evaluation & foundation (institution) evaluation– $30B/year in grantmaking by US foundations– Requires $5M/year to assess foundation
effectiveness, yet not happening– What can we do to “transfer” experiences in
evaluating portfolios/initiatives to evaluating institutions?
NFE April 2006
2. Feedback: the system pushes back or amplifies your efforts
ActionsSteps we take to improve the system
Intermediate indicatorsThe changes we expect to see over the next 1-3 years
OutcomesChanges we want to see in the long term
1. Existing causal theory: How actions lead to outcomes
4. External forces push on the outcomes
3. Mindsets slow or accelerate
change
5. Opportunities for learning and action
NFE April 2006
Working ProgressWorking ProgressEvaluation Designs Evaluation Questions
Exploratory What patterns are evident in seemingly random areas of the system?
Predictability What is the evidence that the intervention has led to the predicted changes in the system?
Emerging Change What principles and valued practices can be identified by observing patterns in self-organizing areas of the system?
Systems Adaptability How does the system adapt to its environment and adjust its random, predictable, self-organizing territories?
NFE April 2006
Pondering QuestionsPondering Questions
• What should evaluation do when the evaluand is “drastically” changed, or when the evaluand is an organic developing entity?
• What approach of evaluation is more appropriate for an evolving and nonlinear evaluand?
• How do evaluators balance between meeting the clients’ needs (even when the needs are somewhat whimsical) and staying true to the articulated body of work?
NFE April 2006
Social System – Systems Social System – Systems PerspectivePerspective
Predictable
Orderly
Controlled
Random No Patterns
Self-organizing Emergent PatternsCoherent But NotPredictable