michael quinn patton may, 2008 evaluating the complex: getting to maybe michael quinn patton oslo,...

Michael Quinn Patton May, 2008

Evaluating the Complex:Getting to Maybe

Michael Quinn Patton

Oslo, Norway

29 May, 2008


Evolving Understandings

I keep changing what I said. Any person who is intellectually alive changes his ideas. If anyone at a university is teaching the same thing they were teaching five years ago, either the field is dead, or they haven’t been thinking.

Noam Chomsky

“The Professor Provaocateur,” The New York Times Magazine, Nov. 2, 2003: 13.


Interpretive Frameworks

• May 2003 Harvard Business Review "The High Cost of Accuracy." Kathleen Sutcliffe and Klaus Weber.

They concluded that "the way senior executives interpret their business environment is more important for performance than how accurately they know their environment."


They further concluded that it is a waste of resources to spend a lot of money increasing the marginal accuracy of data available to senior executives compared to the value of enhancing their capacity to interpret whatever data they have. Executives were more limited by a lack of capacity to make sense of data than by inadequate or inaccurate data. In essence, they found that interpretive capacity, or "mind-sets," distinguish high-performance more than data quality and accuracy.


Evaluation’s TraditionalInterpretive Framework

In the beginning…


Original Primary Options

Formative

and

Summative

Evaluation(Mid-term and End-of-Project Reviews)


Evidence-based Practice

Evaluation grew up in the “projects” testing models under a theory of change that pilot testing would lead to proven models that could be disseminated and taken to scale:

The search for best practices

and evidenced-based practices


Fundamental Issue:How the World Is Changed

Top-down dissemination of

“proven models”

versus

Bottoms-up adaptive management


Models vs. Principles

Identifying proven principles for adaptive management

(bottoms-up approach)

versus

Identifying and disseminating

proven models

(top down approach)


Conditions that challenge traditional model-testing evaluation

• High innovation

• Development

• High uncertainty

• Dynamic

• Emergent

• Systems Change

AdaptiveManagement


Mintzberg on StrategyTwo types of strategy: Intended & Emergent

Unrealized

Strategy

Intended

Strategy Deliberate Strategy

Realized

EmergentStrategy

Strategy


Re-conceptualizing Use

• Use is a process not a event

• Use involves an interaction not just a report

• Use involves training for use not just delivery of results


Some premises:• Evaluation is part of initial program design, including

conceptualizing the theory of change• Evaluator’s role is to help users clarify their purpose, hoped-

for results, and change model.• Evaluators can/should offer conceptual and methodological

options.• Evaluators can help by questioning assumptions.• Evaluators can play a key role in facilitating evaluative

thinking all along the way.• Interpretative dialogue is critical. • Designs can be emergent and flexible.


Three ways of conceptualizingand mapping theories of change

Linear Newtonian causality Interdependent systems

relationships Complex nonlinear dynamics


Linear Logic ModelINPUTS (people, materials)ACTIVITIES (processes)

OUTPUTS OUTCOMES

CHANGES IN PEOPLES LIVES IMPACTS

CHANGES IN COMMUNITIES


Pushing Force

(Non-directional)

Dissatisfaction with the

Status Quo

(Inertia)

a.k.a. “Cost of Change”

Resistance to Change

Pulling Force (Directional)

a.k.a. “Desirability of the end state”

Compelling Vision


Pushing Force

(Non-directional)

Dissatisfaction with the

Status Quo

(Inertia)

a.k.a. “Cost of Change”

Resistance to Change

Pulling Force (Directional)

a.k.a. “Desirability of the end state”

Compelling Vision

First StepsBelievability

Causal Diagram of Beckhard’sChange Formula


Output / ProductEssential Attributes

Attributes required to meet of exceedcustomer needs:

"Do the Right Thing"EfficacyAppropriate

Characteristics to meet or exceedcustomer wants and expectationsof excellence

"Do the Right Thing Well":EfficiencyDignity and RespectEffectivenessTimelinessReduce WasteSafetyContinuityAvailability

What inputs need to go into theprocess to make the productthat produces the desiredresult?

What steps need to be taken tocreate the product that achievesthe desired result?

What features / characteristics should the producthave?

Systems Logic Model

CustomerOutcomes

&Satisfaction

MeasureEffectiveness

MeasureSatisfaction

InformImprovementneeds

Effect

Inputs

Staff ResourcesFinancial resourcesInternal StandardsExternal Requirements

and InformationEquipment/Materials

Key Processes & Functions

Inputs organized and utilizedProceduresStepsKey processes

Measure VariabilityAssess Process ControlAssess fidelity to planned

proceduresAssess impact of variationEvaluate opportunity to raise the

bar

Cause

Feedback into process

What is the desired result?What should customerexperience?

Planning

Implementation

Structure Process OUTCOMES

Feedback


Division workplace that:· Offers a healthy

work environment· Recognizes

excellence· Provides quality

training and management

· Includes effective systems, procedures, and communication (Goal 5)

Increased adoption, reach, implementation, and sustainability of recommended public health strategies to achieve strategic plan goals:· Prevent risk factors for

heart disease and stroke (Goal 1)

· Increase detection and treatment of risk factors (Goal 2)

· Increase early identification and treatment of heart attacks and strokes (Goal 3)

· Prevent recurring cardiovascular events (Goal 4)

Division for Heart Disease and Stroke Prevention Evaluation Planning Logic Model

Internal

Workforce that is:· Diverse· Skilled

Resources that are:· Available· Timely

Division Leadership that provides sufficient:· Infrastructure· Policies· Strategic Planning

Leadership

Disparities

Surveillance

Research

Evaluation

Program

Translation and dissemination of the current knowledge base, and identification of ways to improve that knowledge base

Effective:· Management· Coordination· Staff

development

Enhanced competency of public health workforce

Enhanced ability of programs to apply findings to improve public health

Enhanced external application of Division goals and strategies

Increased advocacy and “activated constituency”

Engaged network of states and partners

Enhanced integration among chronic disease programs

Increased focus on heart disease and stroke prevention efforts by states and partners, especially with regard to disparities

Policy

Increased knowledge of signs and symptoms

Improved emergency response

Improved quality of care

Reduced risk factors

Reduced economic impact of heart disease and stroke

Eliminated preventable strokes and risks

Reduced levels of disparities in heart disease and stroke

Reduced morbidity and mortality of heart disease and stroke

External

Planning Activities Translation,

Dissemination Adoption, Practice, Sustainability Impact

WHAT WHYHOW

Communication

Collaboration


Step Two: THEN A MIRACLE OCCURS

“I think you should be more explicit here in step two.”


Systems

• Parts are interdependent such a change in one part changes all parts

• The whole is greater than the sum of the parts

• Focus on interconnected relationships

• Systems are made up of sub-systems and function within larger systems


Systems Concepts in Evaluation – An Expert Anthology. 2006.Bob Williams and Iraj Imam

AEA Monograph,

EdgePress/AEA Point Reyes CA.


Understanding the

Elephant

from

a Systems Perspective


The relationship between what goes in and what comes

out What conceptual framework informs front-end evaluation work?


Teen Pregnancy Program Example


Logic Model for Pregnant Teens Program1. Program reaches out to pregnant teens

2. Pregnant teens enter and attend the program (participation)

3. Teens learn prenatal nutrition and self-care (increased knowledge)

4. Teens develop commitment to take care of themselvesand their babies (attitude change)

5. Teens adopt healthy behaviors: no smoking, no drinking,attend prenatal clinic, eat properly (behavior change)

6. Teens have healthy babies (desired outcome)


Systems web showing possible influence linkages to a pregnant teenager

Teachers/ other adults

Youngpregnantwoman's

attitudes &behaviors

Herparents &

other familymembers

Child'sfather &

peers

Prenatal program

staff

Her peer group


Program systems web showing possible institutional influences affecting pregnant teenagers:

SCHOOL SYSTEM

Youngpregnantwomen's

attitudes &behaviors

PrenatalClinic andHospitalOutreach

Church

Prenatal program

Other community-based youth

programs

Other Systems-- welfare-- legal -- nutrition programs-- transportation-- child protection-- media messagesContext factors-- politics-- economic incentives-- social norms-- culture-- music

YouthCulture


Using Different System Lenses to Understand a “particular” System

Biologic System• Emergence• Coordination/synergy• Structure, Process, Pattern• Vitality

Sociologic System• Relationships• Conversations• Interdependence• Loose-tight coupling• Meaning/sense

Mechanical / Physical System• Flow• Temporal Sequencing• Spatial Proximities• Logistics• Information

Economic System Inputs/Outputs Cost/Waste/Value/Benefits Customers/Suppliers

Political System• Power• Governance• Citizenship• Equity

Anthropologic System• Values• Culture/Milieu

Information System•Access•Speed•Fidelity/utility•Privacy/security•Storage

Psychological System•Organizing•Forces Field•Ecological/Behaviour Settings

SYSTEMDIMENSIONS


5

Map Systems as Webs

Source: Digital Capital: Harnessing the Power of Business Webs,By Don Tapscott, David Ticoll and Alex Lowy


Complex Dynamic Systems Configuration


HIV/AIDS Example• Hits every system: health, family,

social, religious, economic, political, community, international

• Requires multiple interventions on multiple fronts in all subsystems simultaneously

• Resulting reactions, interactions, consequences dynamic, unpredictable, emergent, and ever changing


Challenges:

Situation Recognition

and Appropriate

Evaluation Designs


The nature of

EXPERTISE


Contingency-basedEvaluation

• Situational analysis & responsiveness• Context sensitivity• Clarify and focus on intended users:

stakeholder analysis• Clarify and focus on intended uses• Methodological appropriateness• Criteria for evaluating the evaluation:

credibility, meaningfulness


Seeing Through A Complexity Lens

“You don't see something until you have the right metaphor to let you perceive it”. Thomas Kuhn


Complex Nonlinear Dynamics• Nonlinear: Small actions can have large

reactions. “The Butterfly Wings Metaphor”

• Emergent: Self-organizing, Attractors

• Dynamic: Interactions within, between, and among subsystems and parts within systems can volatile, changing

• Getting to Maybe: Uncertainty, unpredictable, uncontrollable


Michael Quinn Patton TEI 2008

SNOWDEN’s CYNAFIN FRAMEWORK

Simple, Complicated, Complex, Chaotic

and Disordered Behaviours



Linear contextualisation: 1

Mostchaotic

Mostordered

orderunorder

complexchaotic simplecomplicated


Conceptual Options

•Simple

•Complicated

•Complex


Types of Community IssuesThe Stacey Matrix

Certainty

Ag

reem

en

t

Close to Far from

Far

from

Clo

se to


Simple

Certainty

Ag

reem

en

t

Close to Far from

Far

from

Clo

se to Simple

Plan, control



Simple (Known)Inter-relationships between elements :

Tight, centralised connections. Anyone can see the things the way they are.

Very simple linear cause and effect.

Everyone knows the right answer within the current context (which of course may

not be self-evident or known to others – hence importance of context).


Following a Recipe A Rocket to the Moon Raising a Child

Complicated Complex

The recipe is essential

Recipes are tested to assure replicability of later efforts

No particular expertise; knowing how to cook increases success

Recipes produce standard products

Certainty of same results every time

Simple



Complicated (Knowable)Inter-relationships between elements:

Relationships are looser but still clustered around a central core. Cause and effect is non-linear.

Relationships able to be modelled and able to predicted.

An expert would know the right answer(s)


Technically Complicated

Certainty

Ag

reem

en

t

Close to Far from

Far

from

Clo

se to Simple

Plan, control

Technically Complicated Experiment, coordinate expertise


Following a Recipe A Rocket to the Moon Raising a Child• Formulae are

critical and necessary

• Sending one rocket increases assurance that next will be ok

• High level of expertise in many specialized fields + coordination

• Rockets similar in critical ways

• High degree of certainty of outcome

Complicated Complex






Simple


Socially Complicated

Certainty

Ag

reem

en

t

Close to Far from

Far

from

Clo

se to Simple

Plan, control


SociallyComplicated Build relationships, create common ground


Socially complicated

Implementing human rights agreements, like gender equity or outlawing child labor

Environmental Initiatives Many different and competing

stakeholders Diverse vested interests High stakes


Socially complicatedsituations

pose the challenge

of coordinating and

integrating

many players


Stakeholder Mapping

High Interest/

Low Power

THE INVOLVED

High Interest/

High Power

THE PLAYERS

THE CROWD

Low interest/

Low Power

CONTEXT SETTERS

Low Interest/

High Power



ComplexInter-relationships between elements:

Centre is loosely connected to network. Cause effect difficult to understand

in current setting. Situation understandable only in retrospect.

Not predictable.


Know When Your Challenges Are In the Zone of Complexity

Certainty

Ag

reem

en

t

Close to Far from

Far

from

Clo

se to Simple

Plan, control

Zone of Complexit

y



Systems ThinkingRelationship BuildingCollaborationGood Enough VisionChunking Around Drivers Minimum SpecificationsMultiple Actions Adaptability & Organic


Following a Recipe A Rocket to the Moon Raising a Child

Sending one rocket increases assurance that next will be ok

High level of expertise in many specialized fields + coordination

Rockets similar in critical ways

High degree of certainty of outcome

• Formulae have only a limited application

• Raising one child gives no assurance of success with the next

• Expertise can help but is not sufficient; relationships are key

• Every child is unique

• Uncertainty of outcome remains

Complicated Complex






Simple



Chaotic (Unordered)Cause and effect unknowable,

unattributable even in retrospect.No right answer(s)


Know When Your Challenges Are In the Zone of Complexity

Certainty

Ag

reem

en

t

Close to Far from

Far

from

Clo

se to Simple

Plan, control

Zone of Complexity



ChaosMassive

Avoidance


Following a Recipe A Rocket to the Moon Raising a Child• Formulae are

critical and necessary

• Sending one rocket increases assurance that next will be ok

• High level of expertise in many specialized fields + coordination

• Separate into parts and then coordinate

• Rockets similar in critical ways

• High degree of certainty of outcome

• Formulae have only a limited application

• Raising one child gives no assurance of success with the next

• Expertise can help but is not sufficient; relationships are key

• Can’t separate parts from the whole

• Every child is unique

• Uncertainty of outcome remains

Complicated

Complex




Recipe notes the quantity and nature of “parts” needed



Simple


Global Economic Complexity

Arthur Greenspan, Final speech to world’s Central Bankers, Jackson Hole, Wyoming

August 26, 2005


• “In the absence of a single variable, or at most a few, that can serve as a reliable guide, policymakers have been forced to fall back on an approach that entails the interpretation of the full range of economic and financial data.”


• “Despite extensive efforts to capture and quantify what we perceive as the key macroeconomic relationships, our knowledge about many critical linkages is far from complete and, in all likelihood, will remain so. Every model, no matter how detailed or how well conceived, designed, and implemented, is a vastly simplified representation of the world, with all of the intricacies we experience on a day-to-day basis.”


• “We all temper the outputs of our models and test their results against the ongoing evaluations of a whole array of observations that we do not capture in either the data input or the structure of our models. We are particularly sensitive to observations that appear inconsistent with the causal relationships of our formal models.”


Oct 8, 2007 on The Daily Show“I was telling my colleagues the other day...I’d been dealing with these big mathematical models for forecasting the economy, and I’m looking at what’s going on the last few weeks and I say, “Y’know, if I could figure out a way to determine whether or not people are more fearful, or changing to euphoric... I don’t need any of this other stuff. I could forecast the economy better than any way I know. The trouble is, we can’t figure that out. I’ve been in the forecasting business for 50 years, and I’m no better than I ever was, and nobody else is either.” Alan Greenspan.


The Age of Turbulence: Adventures in a New World, 2007, by Alan Greenspan

Two evaluation locations

for

Evaluating the Complex:

Prospective

and

Retrospective



Prospective Example

The McGill-McConnell Leadership Program Example

Simple elements

Complicated elements

Complex elements

Simple outcomes

• Increase knowledge and skills of participants

Evaluation: Pre-post data and documentation of learning


Complicated Impacts

• Change participants’ organizations

Evaluation:

Case studies

of

organizational change


Complex Vision

• Infuse energy into the moribund not-for-profit (voluntary) sector

• Make the sector more dynamic

• Create network of leaders who actively engage in change


Evaluating the Complex

• Real time follow-up of network connections and actions

• Follow-up is an intervention

• Rapid feedback of findings permits infusion of resources in support of emergent outcomes



Process Use

Infusing evaluative thinking as a primary type of process use.

Capacity-building as an evaluation focus of

process use.


Complex Interdependencies


Insert action into the system


EMERGENCE


Retrospective Example

Advocacy Evaluation

Final Push Campaign

to overthrow the

Juvenile Death Penalty

In late 2003 several petitions on behalf of juvenile offenders facing the death penalty were filed with the U.S. Supreme Court. On January 26, 2004, the Court granted certiorari in Roper v. Simmons and the case was argued before the Court on October 13, 2004. The decision was announced March 1, 2005. The Court ruled 5-4 that capital punishment for juveniles was unconstitutional.

Michael Quinn Patton

May, 2008

In the brief window of time between when the Court agreed to hear the case and the case was argued, roughly nine months, a coordinated campaign was organized and funded aimed at overturning the juvenile death penalty. Organizing, public education, networking & communications continued through to the Court's ruling in March, 2005.


Retrospective Evaluation

To what extent, if at all, was the Court’s decision influenced by the campaign?

Modus Operandi

or

General Elimination Method



1. Strong high capacity coalitions. Working through coalitions is a common centerpiece of advocacy strategy.

2. Strong national-state-grassroots coordination. Effective policy change coalitions in the United States have to be able to work bottoms-up and top-down, with national campaigns supporting and coordinating state and grassroots efforts, while state efforts infuse national campaigns with local knowledge and grassroots energy. Strengthening strong national-state coordination is part of coalition development and field building.

3. Disciplined and focused messages with effective communications. Effective communications must occur within movements (message discipline) and to target audiences (focused messaging). Strengthening communications has been a key a key component of advocacy coalition building.

4. Solid research and knowledge base. The content of effective messages must be based on solid research and timely knowledge. In the knowledge age, policy coalitions must be able to marry their values with relevant research and real time data about dynamic policy environment.

5. Timely, opportunistic lobbying and judicial engagement. The evaluation findings emphasize that effective lobbying requires connections, skill, flexibility, coordination, and strategy.

6. Collaborating funders engaged in strategic funding. Effective funding involves not only financial support, but infusion of expertise and strategy as part of field building.


Targeted StateCampaigns & GrassrootsOrganizing

OmnibusCoordinatedIntegrated

Strategy and Implementation

Effective Litigation Credible, Useful and Amicus Up-to-date Research Briefs

United Focused Coalition Communications Partners Campaign

Knowledgeable Funders


Overall Lesson Learned for Effective Advocacy

In essence, strong national/state/grassroots coordination depends on having a high capacity coalition. A solid knowledge and research base contributes to a focused message and effective communications. Message discipline depends on a strong coalition and national-state coordination, as does timely and opportunistic lobbying and judicial engagement. To build and sustain a high capacity coalition, funders must use their resources and knowledge to collaborate around shared strategies. These factors in combination and mutual reinforcement strengthen advocacy efforts. In classic systems framing, the whole is greater than the sum of parts, and the optimal functioning of each part is dependent on the optimal integration and integrated functioning of the whole.


Strong

Timely,National/

GrassrootsCoordination

OpportunisticLobbying &

JudicialEngagement

StrongHigh Capacity

Coalitions

EFFECTIVEADVOCACY

DisciplinedFocusedMessage/Effective

Communications Collaborating Funders/ Strategic Funding

SIX INTERCONNECTED FACTORS,DYNAMICALLY INTERACTING,

THAT STRENGTHEN ADVOCACY

SolidKnowledge & Research Base


Strong, high capacity coalition

Collaborating funders/ strategic funding

Targeted timely

lobbying

National state

grassroots coordinatio

n

Disciplinedfocused message

Relevantresearch

The interdependent system of factors that contribute to effective advocacy and change

Dealing with the Unexpectedand Unpredicted



Photo by Lynsey Gornick


Creative Challenge

Situational adaptability: Contingency-based evaluation Appropriateness

--Using standard forms of evaluation and

-- Going beyond standard forms when appropriate and useful


Being Open:How hard is this to do?

My Colorado experience….


“The range of what we think and do is limited by what we fail to notice. And because we fail to notice that we fail to notice, there is little we can do to change, until we notice how failing to notice shapes our thoughts and deeds.”

Scottish psychiatrist, R. Laing

Paradigms and Lenses

• The importance of interpretive frameworks

• Complexity as an interpretive framework



Getting to Maybe: How the World Is Changed? 2006Frances Westley, Brenda Zimmerman, Michael Q. Patton

Random House Canada,


Complex Situations• Highly emergent (difficult to

plan and predict)

• Highly dynamic, rapidly changing

• Relationships are interdependent and non-linear rather than simple and linear (cause-effect)


Certainty

Ag

reem

en

t

Close to Far from

Far

from

Clo

se to

?Zone of Complexity


Certainty

Ag

reem

en

t

Close to Far from

Far

from

Clo

se to

Search for root cause


Certainty

Ag

reem

en

t

Close to Far from

Far

from

Clo

se to

Search fo

r syste

mic feat

ures


Contingency-basedDevelopmental

Evaluation


Improvement

versus

Development


BeyondjustSummativeand Formative


Beyond

Static Accountability

Models


Example of an emergent option:

Developmental

Evaluation

DEVELOPMENTAL EVALUATION DEFINED

Evaluation processes, including asking evaluative

questions and applying evaluation logic, to support program, product, staff and/or organizational development. The evaluator is part of a team whose members collaborate to conceptualize, design and test new approaches in a long-term, on-going process of continuous improvement, adaptation and intentional change. The evaluator's primary function in the team is to elucidate team discussions with evaluative questions, data and logic, and facilitate data-based decision-making in the developmental process.



Other names

Real time evaluation Emergent evaluation Action evaluation Adaptive evaluation


CONTRASTS

Traditional evaluations…

• Testing models

Complexity-based, Developmental

Evaluation…

• Supporting innovation and adaptation


Traditional

Evaluation…

• Render definitive judgments of success or failure

Developmental Evaluation…

• Provide feedback, generate learnings, support direction or affirm changes in direction in real time


Traditional

Evaluation…• Render definitive judgments of

success or failure

• Measure success against predetermined goals

Developmental

Evaluation…• Provide feedback, generate

learnings, support direction or affirm changes in direction

• Develop new measures and monitoring mechanisms as goals emerge & evolve


Traditional DevelopmentalEvaluation… Evaluation…

• Evaluator external, independent, objective

• Evaluator part of a team, a facilitator and learning coach bringing evaluative thinking to the table, supportive of the organization’s goals



• Evaluator determines the design based on the evaluator’s perspective about what is important. The evaluator controls the evaluation.

• Evaluator collaborates with those engaged in the change effort to design an evaluation process that matches philosophically and organizationally.



• Design the evaluation based on linear cause-effect logic models

• Design the evaluation to capture system dynamics, interdependencies,

and emergent interconnections



• Aim to produce generalizable findings across time & space

.

• Aim to produce context-specific understandings that inform ongoing

innovation



• Accountability focused on and directed to external authorities and funders.

• Accountability centered on the innovators’ deep sense of fundamental values and commitments –

and learning.



• Accountability to control and locate blame for failures

• Learning to respond to lack of control and stay in touch with what’s unfolding

• And thereby respond strategically



• Evaluation often a compliance function delegated down in the organization

• Evaluation a leadership function:

Reality-testing, results-focused, learning-oriented

leadership



• Evaluation engenders

fear of failure.

• Evaluation supports hunger for learning.


Conditions

• High innovation

• Development

• High uncertainty

• Dynamic

• Emergent


SenseMaker software • Dave Snowden, Founder of Cognitive

Edge, former Director of Knowledge Management at IBM

• SenseMaker can code and map 95,000 stories in 24 hours

• See the world as others see it; anti-terror applications.

• See the quantitative patterns in the meta-data with qualitative context and meaning


PANARCHY MODEL


Routine Change

Fore

loop

Growth r

ConservationK


Turbulent Change

ReleaseBackloop

Reorganization


Adaptive Cycle

Fore

loop

Growth r

ConservationK

Release

BackloopReorganization

Stored

Leadership/initiative

Exploration phase

System Capacity Building

Mature product and scaling up

Phases of Technological and Social Innovation

The birth, growth, destruction and renewal of a forest

Weak CONNECTEDNESS StrongLit

tle C

AP

ITA

L S

TO

RED

M

uch

Creative Destruction

1


Weak CONNECTEDNESS StrongLit

tle C

AP

ITA

L S

TO

RED

M

uch

Creative Destruction

1

2

Renewal/Exploration

Renewal/Exploration


Weak CONNECTEDNESS Strong

Exploitation

3

Lit

tle C

AP

ITA

L S

TO

RED

M

uch


Weak CONNECTEDNESS Strong

Exploitation

3

Lit

tle C

AP

ITA

L S

TO

RED

M

uch

Conservation

4

Stored

HARVESTING LESSONS

DEVEOPMENTAL EVALUATION

FORMATIVE

SUMMATIVE

Phases of Technological & Social Innovation


Taking Emergence Seriously

• Beyond “unanticipated consequences” to genuine openness


SURPRISES

The psychologist Baruch Fischhoff wrote:

"The occurrence of an event increases its reconstructed probability"—in other words, surprises are psychologically untenable in some ways, and we reshape our memories and expectations until we believe that the surprising event was, in fact, likely.


The Epistemology of Surprise

MALCOLM GLADWELL

We hate surprises. We try to erase them from our memory. This is part of what keeps us sane. If, after all, we were always fully aware of the possibility of completely unpredictable events, would we be able to walk out the front door in the morning? Would we ever invest in the stock market? Would we have children? Generally speaking, people who have an accurate mental picture of why and how things happen tend to occupy mental hospitals—or, at the very least, a psychiatrist's office….


Information is like a stream teeming with fish, and if you stick out a net you'll collect something—but to decide what information is consequential. How does one do that?


At the heart is the process of distinguishing signal from noise.

How is that done? I have no idea, nor does anyone, I think, who isn't a seasoned analyst. Pattern recognition is something that comes only with experience. It's a matter of intuition, as much as anything.

People always want to reduce this sort of thing to a formula, or a system, and I'm not sure you can do that. I suspect that there are some artificial-intelligence systems that can help to sort through certain kinds of data. But that could only be a first cut, and eventually human judgment has to be involved.


What role can evaluation play with complex dynamic innovations?


Challenge:Matching the evaluation process and design to the nature of the situation:

Contingency-based

Evaluation


References

Getting to Maybe: How the World Is Changed?

Frances Westley, Brenda Zimmerman, Michael Q. Patton, Random House Canada, 2006

Utilization-Focused Evaluation, 4th ed.,

Michael Quinn Patton, Sage, 2008.

michael quinn patton may, 2008 evaluating the complex: getting to maybe michael quinn patton oslo,...

Documents

michael quinn patton

approach slide

adaptive management

strategy deliberate

types of strategy

dissemination of proven

reconceptualizing use

emergent unrealized