Using Developmental Evaluation to Support Prototyping:A Workshop
Leading Boldly Network:Calgary’s Network for Collaborative
Social Innovation
Workshop Agenda
• Getting Grounded: – The Collaborative Social Innovation Process– Your own experience with or questions about
prototyping
• What is Prototyping & How Is It Unique?
• What is Developmental Evaluation (DE)? – Using DE to Support Prototyping
Collaborative Social Innovation is…… the process of developing social innovations
that are both social in their ends and collaborative in their means. Specifically, collaborative social innovations include new ideas (services, models and resource flows) that simultaneously address complex social issues and create new social relationships or collaborations. In other words they are innovations that are both good for society and enhance society’s capacity to act.
Adapted from the Young Foundation
The LBN Process of Collaborative Social Innovation*
Secure resources
1. Co-Initiate – Set the right conditions and culture
2. Co-Sensing – Frame the issue or design question
Secure resources
3. Co-Presence – Generate ideas
4. Prepare for Prototyping – Planning
5. Co-create / Prototype – Experiment to test ideas
If prototype results prove to be promising, secure resources
6. Co-Evolve - Implement
7. Scale Out – Share innovations with a broader set of stakeholders
8. Scale Up – Systems Change
* This is a working model
What is Prototyping?
Prototype literally means first impression (proto = first, typos = impression)
• A series of small-scale experiments, in simulated or real environments, that test a ‘first impression’ or ‘hunch’ regarding how to positively influence change within a defined “problem space”
What is Prototyping? • A core activity within the
collaborative social innovation process
• Action– and learning–oriented
• Undertaken with the purpose of designing, testing and rapidly iterating new and unverified processes, services or programs in order to discover how they truly influence the complex problem you are trying to address
• A practice for situations that are non-routine and unpredictable, and therefore different than what is required for ongoing operations
• Best done collaboratively with diverse stakeholders
What is Prototyping & How is it Unique?
4 Prototyping Principles: 1. It is used at the beginning, or developmental
phase, of a new process, service or program, where the way to the desired outcome is unknown
2. It anticipates failure. Indeed a key objective of prototyping is to create the conditions to be able to ‘fail safe’, namely endure failure that is survivable (hence small scale) and provides learning to inform the next iteration.
What is Prototyping & How is it Unique?
4 Prototyping Principles3. It embraces rapid iteration informed by
feedback from the system (stakeholders, end-users, etc.) it is trying to change
4. It includes reflection-in-action, the intentional recording and documenting of what is being learned as projects are implemented
What is Prototyping & How is it Unique?
The focus is on learning:• Using evidence to move from “wild
guesses” to “reliable forecasts”–The focus is not on results, but on
the overall trajectory of the initiative
What is Prototyping & How is it Unique?
Moving from Wild Guesses to Reliable Forecasts
Wild guesses
Informed estimates
Reliable forecasts
Prediction
Learning
Time
How is Prototyping a Unique Practice?
This practice enables us to:• “Think with our hands” as a way to investigate the
merit of our proposed solution;• collect user feedback; and • observe how our actions influence (for better or
worse) change within the domain we are seeking to transform.
“The key to dealing with complexity is to focus on having good conversations about assumptions.”
Why Is Prototyping an Important Practice?
• In a time of tight budgets, prototyping can be used to test an unverified process, service or program before considerable resources are invested into a large-scale initiative. – It helps to expose flaws and limitations in an
approach before significant investment has been made
• Can help leaders/Boards manage the risks involved with trying something different or new
• Can help us develop better products and services
How to Prototype
1. Know when a prototype project is the right course of action
– Developmental phase / uncertain outcomes
2. Develop a plan – Write down what you plan
to do, what you expect to happen and why (Hypothesis of Record)
– “In an innovative initiative the plan is a hypothesis”
3. Develop a clear learning agenda (Developmental Evaluation)
– Compare results to explicit set of assumptions
4. Execute the Plan – Analyze differences
between what you thought would happen and what actually happened
5. Based on what you learned, revise the plan
How to Prototype
Small Group Conversation
What stood out for you?
What is becoming clearer?
What questions are you carrying?
Developmental evaluation is a small but demanding niche
in the world of evaluation.
Five Niches
DE: 7 Things You Need to Know
1. The Essence of DE
• Developmental evaluation isn’t some particular methods of recipe-like steps to follow. It doesn’t offer a template of standard questions. It’s a mindset of inquiry into how to bring data to bear on what’s unfolding so as to guide and develop the unfolding. What that means and the timing of the inquiry will depend on the situation, context, people involved, and the fundamental principle of doing what makes sense for program development (Patton 2010: pp.75-6).
Development is about creative
thinking.
Evaluationis about criticalthinking.
Developmental Evaluationholds theseIn balance.
The Essence of DE
2. Purpose
• Developmental evaluation focuses on developmental questions: What’s being developed? How is what’s being developed and what’s emerging to be judged? Given what’s been developed so far and what has emerged, what’s next?
TRADITIONAL EVALUATIONS DEVELOPMENTAL EVALUATIONS
Render definitive judgments of success or failure.
Provide feedback, generate learnings, support direction or affirm changes in direction.
Measure success against pre-determined goals. Develop new measures and monitoring mechanisms as goals emerge and evolve.
Position the evaluator outside to assure independence and objectivity.
Position evaluation as an internal, team function integrated into action and ongoing interpretive processes.
Design the evaluation based on linear cause-effect logic models.
Design the evaluation to capture system dynamics, interdependencies, and emergent interconnections.
Aim to produce generalizable findings across time and space.
Aim to produce context-specific understandings that inform ongoing innovation.
Accountability focused on and directed to eternal authorities and funders.
Accountability centered on the innovators’ deep sense of fundamental values and commitments.
Accountability to control and locate blame for failures.
Learning to respond to lack of control and stay in touch with what’s unfolding and thereby respond to strategically.
Evaluator controls the evaluation and determines the design based on the evaluator’s perspective on what is important.
Evaluator collaborates in the change effort to design a process that matches philosophically and organizationally.
Evaluation engenders fear of failure.
Evaluation supports hunger for learning.
3.1 The Evaluator
• The developmental evaluator inquires into developments, tracks developments, facilitates interpretation of developments and their significance, and engages with innovators, change agents, program staff, participants in the process, and funders around making judgments about what is being developed, how it is being developed, the consequences and impacts of what has been developed, and the next stages of development (Patton 2010, p. 227).
3.2 Evaluator Roles
3.2.1 Surfacing Issues
Raise critical questions, tensions, debates that emerge as part of the developmental process.
3.2.2 Framing Concepts
Flush out and flesh out problem definitions, lenses, frameworks and ‘working models’ underlying the conception and design of the intervention.
3.2.3 Testing Quick Iterations
Surface –and interpret, draw conclusions and judge -- real time data through techniques such as rapid reconnaissance, secondary evidence, modeling, outcome mapping, simulations, etc.
3.2.4 Tracking Developments
Documenting concrete developments – e.g. learning, new models, design features, forks in the road – in the evolution of a group’s work.
3.3 The Evaluator-User Relationship
Traditional Evaluation
• Positioned outside the action to assure independence and objectivity.
Developmental Evaluation
• Positioned as team member integrated into the action and ongoing process of gathering and interpreting data, framing issues, surfacing and testing model developments. May be situated externally or internally.
3.4 Competencies
Traditional Evaluator Competencies
Reflective Practice. Technical Practice Situational Practice. Management Practice. Interpersonal Practice.
Extra Competencies for DE
Comfort with adaptive processes
High tolerance for ambiguity Can react/adapt quickly Effective communication with
hyperactive, short-attention-span, action-oriented innovators
Larger tool kit of evaluative methods
4. Methods
• [I]t’s worth emphasizing that no definitive list of developmental evaluation inquiry approaches can or should be constructed. Developmental evaluation creatively adapts whatever approaches and methods fit the complexities of the situation and are responsive, appropriate, and credible to social innovators in opening up new understandings and guiding further development.
• In being creative, the developmental evaluator is also practical and pragmatic, doing the best job possible within available resources and other constraints. Constraints always exist and do what constraints do -- constrain. Our ability to think of alternatives is limited. Resources are always limited. Time is of the essence. We do what we can. Part of what we can do is adapt other inquiry traditions to the purposes of developmental evaluation.
Methods continued
Key Features
• Contingency
• Real Time
• Flexible
Adaptive/Evolutionary
• Evaluation methods need to be continually adapted to reflect the changes in the emerging innovation and the learnings and questions of the innovators.
5. Steps
6. Reports
• Dynamic complexities don’t slow down or wait for evaluators to write their reports, get them carefully edited, and then approved by higher authorities. Any method can be used but will have to be adapted to the necessities of speed, real-time reporting and just-in-time, in-the-moment decision-making.
• That is a major reason the developmental evaluator is part of the innovation team, to be present in real time as issues arise and decisions have to be made […] Contrary to the usual practice in evaluation of fixed designs that are implemented as planned, developmental evaluation designs can change as the innovation unfolds and changes (Patton 2010, p. 335-6).
Emerging Practices on Reports
Ongoing Real Time Feedback• A variety of formats that
capture key evaluation questions, results and learnings, and new developments (e.g. memos, PPT, visual diagrams, etc.) in a way that is fits the pattern of information flow and decision-makers of the evaluation users?
Periodic “Developmental Accounts”
• A record of the major developments that have emerged in a developmental process so far (interim accounts) and/or at the end of significant developmental period (e.g. a shift into formative evaluation).
The Tracking the EvolutionOf the Intervention
7. Accountability
To Whom
• People responsible for developing and/or adapting an intervention.
For What
• Developing a better understanding of the issue being addressed, leverage points for change, and developing an intervention.
• This may – or may not – develop into a workable, stable or ultimately successful intervention.
For How
• A rigorous, data-based, user-friendly and responsive, process of reality testing that balances creative and critical thinking.
• Guided by standard evaluation principles and ethics.
Tamarack Interview with Michael Quinn Patton on DEhttp://tamarackcommunity.ca/g3s61_CC4I5.html
Utilization-FocusedEvaluation: 4th Ed.Michael Quinn Patton
The DE Primer. Jamie Gamble.http://www.imprintinc.ca/
Getting to Maybe. Frances Westley Brenda ZimmermanMichael Quinn Patton
Resources
DE 201: Practitioners Guide Dozois, Langlois, Blanchet-CohenJW McConnell Site
Developmental Evaluation Michael Quinn Patton
Extras for Prototyping
types of prototyping
•exploratory prototyping – testing the demand, feasibility & viability of an idea, quickly and inexpensively.
•developmental prototyping – testing the demand, feasibility & viability of an idea more detail, to see if its works in practice.
•slow prototyping – a concurrent focus on developing an idea or practice AND testing the capacity of the group to carry out the idea.
•fast prototyping – an exclusive focus on testing an idea or practice.
fast prototyping
slow prototyping
exploratory prototyping
developmental prototyping
testing a collaborative fundingmodel (simulation)
exploring a new mission and design for employment program
(scenarios)
testing the efficacy of different campaign messages before roll out
(random controlled trials)
testing the buy-in, viabilityfeasibility of a fundingstrategy (role playing)
developing & adapting practices for inter-agency collaboration
(after action review)
Questions
• What questions – if any – emerge for you about DE?
• What is alive for you about DE might contribute to your work?
Key Resources
• Nesta (UK)• The Other Side of Innovation: Solving the
Execution Challenge • The Open Book of Social Innovation• The Austin Centre for Design• IDEO • Human Centred Design