matching evaluation to the nature & conceptualization of the problem simple, complicated and...
TRANSCRIPT
Matching Evaluation to the Nature &
Conceptualization of the Problem
Simple, Complicated and Complex Problem Framings and the Implications for Evaluation
Brenda Zimmerman, Schulich School of BusinessYork University, Toronto, Canada
Presentation to the Joint Canadian Evaluation Society/American Evaluation Association Conference
Toronto Oct 29, 2005
Social Transformation – our definition
An alteration of what is established by the introduction of new elements or forms (including new ideas, practices, or resource flows). In particular the alteration of social relationships to allow for an improvement in or transformation of intransigent and broadly based social problems
Our approach to social innovation
Connecting understanding of large scale patterns or dynamics to action at the level of individuals or groups
Concerned with how to understand these interactions in such a way that we could help those trying to make a difference
The Decision to Use a Complexity Lens
Complexity theory leads us to look at the relationship between the micro and the macro
It suggests how individuals act effectively in arenas which they cannot control (complex is more than complicated)
Complexity theory points to what is happening “in the between”; in the relationships between things.
Implications for evaluation??
Following a Recipe A Rocket to the Moon Raising a Child
Complicated Complex
The recipe is essential
Recipes are tested to assure replicability of later efforts
No particular expertise; knowing how to cook increases success
Recipe notes the quantity and nature of “parts” needed
Recipes produce standard products
Certainty of same results every time
Simple
Following a Recipe A Rocket to the Moon Raising a Child• Formulae are critical
and necessary
• Sending one rocket increases assurance that next will be ok
• High level of expertise in many specialized fields + coordination
• Separate into parts and then coordinate
• Rockets similar in critical ways
• High degree of certainty of outcome
Complicated Complex
The recipe is essential
Recipes are tested to assure replicability of later efforts
No particular expertise; knowing how to cook increases success
Recipes produce standard products
Certainty of same results every time
Simple
Following a Recipe A Rocket to the Moon Raising a Child
• Formulae are critical and necessary
• Sending one rocket increases assurance that next will be ok
• High level of expertise in many specialized fields + coordination
• Rockets similar in critical ways
• High degree of certainty of outcome
• Formulae have only a limited application
• Raising one child gives no assurance of success with the next
• Expertise can help but is not sufficient; relationships are key
• Can’t separate parts from the whole
• Every child is unique
• Uncertainty of outcome remains
Complicated Complex
The recipe is essential
Recipes are tested to assure replicability of later efforts
No particular expertise; knowing how to cook increases success
Recipes produce standard products
Certainty of same results every time
Simple
Following a Recipe A Rocket to the Moon Raising a Child
• Formulae are critical and necessary
• Sending one rocket increases assurance that next will be ok
High level of expertise in many specialized fields + coordination
• Separate into parts and then coordinate
• Rockets similar in critical ways
• High degree of certainty of outcome
• Formulae have only a limited application
• Raising one child gives no assurance of success with the next
• Expertise can help but is not sufficient; relationships are key
• Can’t separate parts from the whole
• Every child is unique
• Uncertainty of outcome remains
Complicated Complex
The recipe is essential
Recipes are tested to assure replicability of later efforts
No particular expertise; knowing how to cook increases success
Recipe notes the quantity and nature of “parts” needed
Recipes produce standard products
Certainty of same results every time
Simple
• Sources for charts of next three slides
Begun, Zimmerman and Dooley, “Health Care Organizations as Complex Adaptive Systems”, in S. M. Mick and M. Wyttenbach (eds.), 2003 Advances in Health Care Organization Theory San Francisco: Jossey-Bass, pp 253-288
Also can be found at www.change-ability.ca
Complexity Science Established Science --- “complicated”
Holism Reductionism
Indeterminism Determinism
Relationships among entities Discrete entities
Nonlinear relationships– critical mass thresholds
Linear relationships– marginal increases
Quantum physics– influence through iterative nonlinear feedback– expect novel and probabilistic world
Newtonian physics– influence as direct result of force from one object to another– expect predictable world
Understanding; sensitivity analysis Prediction
Focus on variation Focus on averages
Local control Global control
Behavior emerges from bottom up Behavior specified from top down
Metaphor of morphogenesis Metaphor of assembly
Adapted from Dent 1999
Established Perspectives*
Complexity Science Perspective
TEMPORAL FRAMING
VIEW OF THE FUTURE
Relatively knowable Relatively unknowable
RELEVANCE OF HISTORY
None (transaction cost) to high (institutional). When high, history is deterministic.
High, but history may or may not be deterministic.
SPATIAL FRAMING
DOMAIN OF STUDY
Reified organization in the environment
Relationships among individuals, subsystems, systems
VIEW OF THE ENVIRONMENT
Outside the organization; evolves separately
Part of the domain of study; coevolves with the organization
LEVELS OF ANALYSIS
Single to few, relatively independent
Multitude of nested levels
CONSTRUCT FRAMING
STRATEGY Relatively designed Relatively emergent
STRUCTURE Equilibrium; relatively centralized
Non-equilibrium; relatively decentralized
KEY INFO. FOR THE ORGANIZATION
External environmental intelligence
Functioning of relationships
INFORMATION PROCESSOR
Reified organization Individuals; complex systems of individuals
Complexity Science Perspective
Implications for Research
TEMPORAL FRAMING
VIEW OF THE FUTURE Relatively unknowable Patterns may repeat, but without predictive power. Anticipate surprise. Study emergence.
RELEVANCE OF HISTORY
High, but history may or may not be deterministic.
Requisite to study history (vs. cross-sectional only); conduct longitudinal analysis
SPATIAL FRAMING
DOMAIN OF STUDY Relationships among ind., subsystems, systems
Study patterns of interaction among agents.
VIEW OF THE ENVIRONMENT
Part of the domain of study; coevolves with the org.
Study co-evolution of organization and environment
LEVELS OF ANALYSIS Multitude of nested levels View issue from multiple, nested levels of systems
CONSTRUCT FRAMING
STRATEGY Relatively emergent Study changes in strategy and conditions that facilitate change
STRUCTURE Non-equilibrium; relatively decentralized
Assess flexibility of structures; simple rules; min specs
PURPOSE OF ORGANIZATIONAL RELATIONSHIPS
Learning; co-creation of meaning; functioning of relationships
Assess degrees of co-participation, learning, sharing; study quality of relationships
KEY INFORMATION FOR THE ORGANIZATION
Functioning of relationships
Study quality of relationships
INFORMATION PROCESSOR
Individuals; complex systems of individuals
Study individuals and coalitions, vs. reified organization
The Brazil Case Study
annual per capita income is less than $5000
In the 1980s, Brazil’s AIDS problem was worse than South Africa’s
By 2000, South Africa’s HIV infection rate was 25% whereas Brazil’s was 0.6%
In 1992, the World Bank predicted that Brazil would have 1.2 million AIDS cases by 2000
…but the actual count was 0.5 million.
Brazil Analysis is from Begun, Zimmerman and Dooley, 2002
The Brazil Complex Problem
How do you respond to an AIDS epidemic in a developing country?
No money for expensive drugs problems of affordability of drugs
A very iffy health care system problems of treatment
High levels of illiteracy problems of compliance even if there were drugs
High levels of poverty and hunger problems of nutritional needs for drugs
Making the Brazil Problem Complicated
“What will drug costs be for our infected population?”
“What resources are needed to manage drug therapies for illiterate patients?”
“What resources are needed to assure compliance with drug associated nutrition in this population?”
“What are the resources needed for an effective prevention program?”
World Bank Responds to AIDS as Complicated
Meaningful solutions require sophisticated, integrated national health care systems
We cannot provide treatment to all when the drug costs are so high
We cannot afford resources to manage treatment compliance
With our limited resources, we should focus more on prevention than treatment
It will therefore take a long time for the problem to work itself through
Brazil Implicitly Recognized the Complexity
Began to address the AIDS issue head-on in 1994
Were unwilling to accept the answers of the World Bank
Hence – had to change the questions… Changing the questions changes the focus,
changes what is “analyzed”, changes what is seen as possible/impossible
The Brazil Questions Assume Complexity
“What will drug costs be for our infected population?”
“What resources are needed to manage drug therapies for illiterate patients?”
“How can we reduce costs so that we can provide treatment to all who need it?”
“What methods of
communication will work to convey the drug therapy routine to a patient – even a homeless, illiterate patient?”
World Bank Questions
Brazil Questions
The Brazil Questions
“What resources are needed to assure compliance with drug associated nutrition in this population?”
“What are the resources needed for an effective prevention program?”
“If food is an issue, how can we ensure greater compliance with the routine by linking up with charities that can provide food at the right times of day?”
“How can we achieve our prevention goals while treating all of those currently infected?”
World Bank Questions
Brazil Questions
The Brazil Conclusions
Meaningful solutions require sophisticated, integrated national health care systems
We cannot provide treatment to all when the drug costs are so high
We cannot afford resources to manage treatment compliance
With limited resources, focus more on prevention than treatment
It will therefore take a long time for the problem to work itself through
Find ways to use the resources we have to respond to the problem
Provide drugs to all by finding ways to reduce drug costs
Use our informal system to train people to care for themselves
Prevention will be part of the treatment
Seek short and long term results
World Bank Conclusions
Brazil Conclusions
The Brazil Responses
A stable container: Brazil built on existing infrastructure (natural network existed) A somewhat shaky health system of
hospitals and clinics Added to by 600 NGOs, churches (hubs)
Free drugs to all AIDS patients: faced down drug companies in pursuit of national interest
The Brazil Responses - cont’d
Patients managed their own drugs Illiterate people were taught by local
“trustworthy” folks (hubs were sought)
Free treatment spread prevention ideas No labeling of those affected Prevention information readily available
Brazil saw the value in the relationships as the key to change
The Brazil AIDS story - optimism COURAGE– to challenge
WTO, USA, large pharmaceuticals, World Bank
COMPLEXITY – lived with the complex nature of the society, and the AIDS problem
RELATIONSHIPS – used the power of existing relationships to learn and enhance their connectedness
Matching Evaluation to Cycles of
Intervention & Program Development
Ecocycle & Panarchy as a Framework
The ecocycle as a metaphor for successful social innovation
Based on the work of ecologists studying natural systems (C.S. Hollings in particular)
Suggests four distinct stages in a “life cycle”, linked to two key dynamics: connectedness or sameness and stored capital or potential
BIRTH
GROWTH
MATURITY
The birth, growth, destruction and renewal of a forest
Variety CONNECTEDNESS Sameness
Exploitation
1
Rele
ased
P
OTEN
TIA
L S
tore
d
04/18/23
exploitation
The Psychosocial place called “exploitation” Triggers: Choice! Groups of people have converged on some promising
products or processes and are devoting time and energy to these. Decision to invest in one or more and build capacity
People: the implementers come into their own. Invest in people who love to take ideas and develop them into reality, the team-builders and the engineers come into their own.
Experience: Excitement, flow, high energy, doors opening, long hours, time flying by….often correlated with peak experiences. A time of intense activity, flow and productivity with a steep organizational “learning curve”
Evaluation: “Formative” – look for a continuous feedback systems as the start-up phase moves towards greater goal clarity and efficiency in delivery
Management: Supporting capacity building. Careful attention to the need for training, hiring and developing systems to support bringing a program “to market” and moving it up the scale. Evaluation geared to providing feedback for increasing effectiveness and efficiency.
Traps: Failure to build sufficient capacity or sufficient “lock-in” to achieve economies of scale and delivery efficiency
The birth, growth, destruction and renewal of a forest
Variety CONNECTEDNESS Sameness
Exploitation
1
Rele
ased
P
OTEN
TIA
L S
tore
d
04/18/23
conservation
The Psycho-social place called “conservation” Triggers: Reaching a level of efficiency and learning where
improvements become incremental. The “mature” phase, productivity high and measurable, peak “earnings” and efficiency.
People: Invest in those excellent at management, system design excel, others may be getting restless, or even a little bored.
Experience: satisfaction at success, the pride in the well oiled machine, anxiety about mushrooming demand, loss of momentum
Evaluation: Summative evaluation – success based on goals should be measurable by quantitative or qualitative means.
Management: awareness of vulnerability, standing still (again), openess to creative destruction. Encourage “letting go”
Traps: rigidity, loss of peripheral vision, over-commitment. Fear of change.
The birth, growth, destruction and renewal of a forest
Variety CONNECTEDNESS Sameness
Exploitation
1
Rele
ased
P
OTEN
TIA
L S
tore
d
Creative Destruction
Conservation
Creative destruction
Not devastation of “soil” Crisis…. from root word “to sift” Creation requires destruction To honor creation… open to destruction Externally imposed Internally driven
04/18/23
Creativedestruction
04/18/23
Creativedestruction
The psychosocial space called “creative destruction” Triggers: most clearly signaled by the discontinuation, destruction or
collapse of a given program or process and the simultaneous release of resources and energies for other things.
People: those who thrive on crisis, on new beginnings are happy here – others may be depressed or in mourning
Experience: of confusion, identity crisis,, change in relationships, anxiety, elation.
Management: providing reassurance, “standing still”, ‘active” reflection, contemplation, listening.
Evaluation: organizations should be “harvesting”: identifying lessons learned and thinking about implications for the future.
Traps: Fear and guilt can act to isolate – communication is needed.
Outputs: Focus on new ideas, investment of capital in exploration.
RenewalAlso known as Exploration orReorganization
04/18/23
Renewal/exploration
The psychosocial space called “exploration” Triggers: The need for innovation. Development of multiple “random walks”,
experiments, initiatives which lead to little in the way of measurable outcomes for considerable time
People: people who learn by doing are happy here; invest in individuals who are entrepreneurs/ innovators with a track record.
Experience: Reflection moving to experimentation, lots of false starts and sometimes frustration and mounting anxiety about inputs/output ratios
Management: Generate resources and connections and exchange of information, encourage experimentations and learning, have faith in the self-organizing properties of information and groups. Introduce new ingredients as required. Exert increasing pressure for output.
Evaluation: geared to capturing learning, building transparent accounts of innovation.
Traps: “spinning wheels”, the incapacity to select a course and commit resources to it to the required degree. Ungovernable competition.
Conservation
2
The birth, growth, destruction and renewal of a forest
Weak CONNECTEDNESS Strong
Exploitation
1
Renewal
4
3
Creative Destruction
Lit
tle C
AP
ITA
L S
TO
RED
M
uch
The birth, growth, destruction and renewal of a forest
Variety CONNECTEDNESS Sameness
Exploitation
1
Rele
ased
P
OTEN
TIA
L S
tore
d
Creative Destruction
ConservationRenewal
POVERTYTRAP
RIGIDITYTRAP
VarietySameness
Released
Stored
The psycho-social space of poverty traps
Organization, processes, initiatives: a sense of spinning wheels, initiatives and interactions are infrequent or are not generative
Experience: “chronic disaster”, increasing hopelessness or helplessness, as sense of the poverty of the very ground of being
Psycho-social experience of rigidity traps
Organizations, systems initiatives: rule-bound, unresponsive, resistant to change, too may resources tied up in non-productive systems.
Experience: a kind of rigor mortis, nothing new seems to happen, creativity discouraged, emphasis on evaluation and performance
Stored
Released
Variety Sameness
Do we evaluate Resiliency? Connectedness is an indicator of how much similarity
vs. variety characterizes the behavior/composition of the system
Potential (or capital) is an indicator of the degree to which resources (time, energy, money, skill, knowledge) are committed to existing projects as opposed to available to attach to new projects
Resilience is a measure of the ability of the system to avoid “traps”, i.e. getting stuck in any phase
Developmental Evaluation
Formative Evaluation
Summative Evaluation