Evaluation and the
Science of Complexity
Evaluating Complexity Conference
NORAD
29th -30th May 2008
Agenda
Evaluations – some common issues Complexity science – origins and ideas Implications for evaluations Summary
The history of M&E in the international development sector is in four distinct phases 1960s to 1979: Early developments
1979-1984: Rapidly growing interest
1984 to 1988: M&E matures
1988 to the present: the crossroads
Wealth of tools, techniques and approaches are now available Logical framework analysis Results-based management Needs Assessments, Impact Assessments Ex ante and ex post assessments ZOPP GANTT Social Network Analysis Appreciative Enquiry Most Significant Change Outcome Mapping Many many more!!
For many organisations, evaluations are at the centre of a vicious circle...
Increased competition
Increased pressure to show results and impact
Lack of professional norms and standards
Poor learning and accountability
Growing need for high profile
fundraising and advocacy work
...causing problems for M&E
Evaluations are still largely focused on reports as opposed to changed behaviours, ways of thinking and attitudes
Reflection, learning and analysis are threatened by existing agency cultures and processes
Org. Learning
Existing culture & process
Evaluation
Accountability
Evaluations, like other similar initiatives, often sit on top of existing organisational silos, inefficiencies and power imbalances, rather than resolving them
Silos
Evaluation
KM
Agencies plough the same evaluation “field”, but stick to their own furrows
Understanding of the effectiveness and use of evaluations is weak at best…
Where’s the data??!
Evaluations tend to be based on wish lists, not strategies, and therefore are often overloaded
Leadership and political buy-in to evaluation is rare and unreliable, with two common reactions
...all of which means that (1) evaluation efforts resemble this iceberg...
What is planned
What actually happens
And (2) evaluators spend most of their time feeling like this...
working against this...
Agenda
Evaluations – some common issues Complexity science – origins and ideas Implications for evaluations Summary
“Exploring the science of complexity”
Primary aim was to explore the potential value of complexity science for those who work on change and reform initiatives within the aid sector
Drew on scientific and experimental literature – physiology, physics, mathematics, public sector reform, sociology, economics, organisational theory, plus case studies, reports and evaluations from the aid sector
Reviewed over 250 articles, books, reports and evaluations
10 peer reviewers, including 5 directors of leading research institutes
Published February 2008 Available to download from www.odi.org.uk
A man was walking home one dark and foggy night. As he groped his way through the murk he nearly tripped over someone crawling around by a lamp post.
“What are you doing?” asked the traveler.
“I’m looking for my keys” replied the other.
“Are you sure you lost them here?” asked the traveler.
“I’m not sure at all,” came the reply, “but if I haven’t lost them near this lamp I don’t stand a chance of finding them.”
A (well-known) story…
A closer inspection of the light under the lamp revealed…
INPUTS
ACTIVITIES
OUTPUTS
OUTCOMES
IMPACTS
Logical frameworks / results chains articulate a clear world view and theory of change: “The light under the lamp”
The machine metaphor - universe as clockwork
The future is knowable given enough data points
Phenomena can be reduced to simple cause & effect relationships
Dissecting discrete parts would reveal how the whole system worked; science was the search for the search for the basic building blocks
The role of scientists, technologists & leaders was to predict and control - increasing levels of control (over nature, over people, over things) would improve processes, organisations, quality of life, entire human societies
Key Assumptions
Assumptions about systems Ordered Reductionist - parts would reveal the whole
Assumptions about how systems change Linear relationships influence as direct result of force from one object to
another - predictable Simple cause & effect
Assumptions about human actions Rational choice Behavior specified from top down Certainty and “knowability”
...Reality of aid is a little different...
But a new light is being turned on (slowly, unevenly, using a dimmer switch)…
Complexity science is a science of understanding change A loosely bound collection of ideas, principles and
influences from a number of other bodies of knowledge, including chaos theory fractal geometry cybernetics complex adaptive systems postmodernism systems thinking
Discovery of similar patterns, processes and relationships in a wide variety of phenomena related to the nature and dynamics of change
From microscopic chemical reactions…
...to the evolution of galaxies...
Complexity scientists use a range of ideas and concepts (familiar, challenging and baffling) to make distinctions between simple, complicated and complex systems and phenomena
Following a Recipe A Rocket to the Moon Raising a Child Formulae are critical
and necessary
Sending one rocket increases assurance that next will be ok
High level of expertise in many specialized fields + coordination
Separate into parts and then coordinate
Rockets similar in critical ways
High degree of certainty of outcome
Formulae have only a limited application
Raising one child gives no assurance of success with the next
Expertise can help but is not sufficient; relationships are key
Can’t separate parts from the whole
Every child is unique
Uncertainty of outcome remains
Complicated ComplexSimple
The recipe is essential Recipes are tested to
assure replicability of later efforts
No particular expertise; knowing how to cook increases success
Recipe notes the quantity and nature of “parts” needed
Recipes produce standard products
Certainty of same results every time
The claims of complexity scientists
The complexity of real world systems is (usually) not recognised or acknowledged by scientists and policy makers
Or, that if it is not acknowledged, they don’t deal with them
Or, that if they do deal with them, they don’t do so effectively
Or, that if they do deal with them effectively, it’s because they used an specific approach / framework
...that is also available to you, dear client, at a reasonable daily rate plus a per diem
[JOKE]
There have been diverse efforts to apply ideas to social, economic and political analysis and practice Arthur, Ormerod - Economics Stacey, Snowden - Organisations Jervis, Urry, Cutler - Intl relations De Mancha - History Gilchrist - Community development Education policy - Sanders and McCabe Health policy - Zimmerman Government reform - Chapman Strategic thinking – Saunders Evaluation - Williams
Complexity and systems approaches have already proved useful in re-thinking aid and development issues Uphoff, 1990s Chambers, 1997 Sellamna, 1999 IDRC, Outcome Mapping, 2001 Warner, 2001 Rihani, 2002 Lansing and Miller, 2003 Inclusive Aid, 2004 ECDPM, 2004-06 Eyben, 2006 Guijt, various Davies, Network Analysis, various
`
10 key concepts and implications
8 Adaptive Agents
10 Co-Evolution
4 Non-Linearity
3 Emergence from
SimpleRules
6 Phase space and attractors
5 Sensitivity to initial
conditions
1 Interconnected and interdependent
elements and dimensions
2 Feedback processes
7 Strange
attractors and the edge of
chaos
9 Self organisation
Features of systems
Dynamics of change
Behaviours and relationships
Complex systems
Collection of parts, which collectively have a range of dimensions
Parts share an physical or symbolic environment / space
Action by any part can affect the wholeE.g. individuals, families, communities, cities,
markets, societies, populations, economies, nations, planets
Complex systems are interconnected and interdependent to different degrees Interconnectedness may occur between any elements,
dimensions, systems and environments
This interconnectedness leads to interdependence between the elements and the dimensions of a system, and gives rise to complex behaviour
Complex systems can be tightly or loosely coupled, internally and with their environment, giving rise to different kinds of complex behaviours Tightly coupled: Global FOREX markets Loosely coupled: US University system, global construction industry
Systems can be understood via mapping techniques, followed by analysis to understand the dynamics and interactions of the system
Foot and mouth disease: an example of failure caused by focusing on one part of the system and ignoring the links between sub-systems (biology, geography, economics)
Economic rationalisation of abattoirs and EU subsidies increased the interconnectedness of herds to a critical point.
Changes to foot and mouth reporting rules delayed the isolation of infectious animal
The relationship between these actions and the epidemiology of F&M was not appreciated in advance where it mattered because the livestock industry was not viewed as a interconnected, interdependent system
Complexity also means that systems need to be understood at different scales
Atom
Molecule
Cell
Tissue
Organs
Organisms
Communities
E.g. evaluating the effectiveness of child health programmes
A.N. NGO
Other NGOs
Private Sector
Community and
Family
Church
Civil Society
DevelopingCountry Govmts
Local partners
Example: evaluating resource flows in the humanitarian system
Issues of interconnectedness, interdependence and scale usually do not become apparent until a crisis... Foot and mouth disease, UK
Economics, cattle management, disease September 11th
Globalisation and terrorism Climate change
Western consumerism and Southern disasters Credit crunch
US mortgage market mis-selling and the world economy Food prices
Biofuels and food consumption Vulnerability to natural disasters
Sichuan earthquakes and dams Human trafficking
Desire, economics and rights abuses
...indicating that we have biases in the way we view the world...
Three different kinds of problems have been identified, along with some common biases in dealing with them
They are:“Messes”“Problems” “Puzzles”
“Messes” are issues that do not have a well defined form or structure. NB not a value statement, but a description There is often not a clear understanding of the problem faced
Messes often involve economic, technological, ethical and political issues It has been suggested that all of the really important issues in the world
start out as messes.
For example, how was rising HIV/AIDS incidence in Brazil dealt with in the 1990s? concerned money, technology, ethics, social relations, politics, gender
relations, poverty all of these dimensions of the problem had to be dealt with
simultaneously, and as a whole
Many of the major problems we face are “messes”! Credit crunch
US mortgage market mis-selling and the world economy Food prices
Biofuels and Vulnerability to natural disasters
Sichuan earthquakes and dams Climate change
Western consumerism and Southern disasters September 11th
Globalisation and terrorism Human trafficking
Sexual preferences and human rights abuses Arms trade
Economics and war Etc, etc, etc
“Problems” are issues that have a known or knowable form or structure
The key dimensions and variables of a problem are known and the interaction of dimensions may also be understood, even if only partially.
With problems, there is no single clear cut way of doing things there are many alternative solutions, depending on the constraints faced Expertise matters
For example, dealing with the sewage system in a particular city may rely on amount of money available, technology, political stance of leaders, climatic conditions, urban development, the road system, and so on
Puzzles are well defined and well structured known problems with a specific “best” solution Solutions can be worked out and improved
Solutions are replicable - “best practices” are possible
Policy and traditional science is biased towards puzzle-solving Real-world, complex, messy nature of
systems is frequently not recognised
Simple puzzle-based solutions are applied to complex messesE.g. Global War on Terror has been applied as
the single best solution to the mess of terrorism
“Some of the greatest mistakes have been made when dealing with a mess, by not seeing its
dimensions in their entirety, carving off a part, and dealing with this part as if it were a problem, and then solving it as if it were a puzzle, all the while ignoring the linkages and connections to
other dimensions of the mess”
Or to put it another way: dividing a cow in half does not give you two smaller cows
Implications: analyse and deal with the reality of the system Multidimensionality, interdependence and interconnectedness of
poverty and humanitarian crises (and responses to them) should be recognised by those designing, managing and evaluating aid interventions
Analysis may need to be in line with historical research - not ‘did x cause y?’ but ‘what happened and why?’, building narratives about events and processes.
The task of selection and synthesis of data becomes as important as analysis
Different perspectives on what the system is need to be taken into account, especially when these perspectives differ as to the nature, interconnectedness and scale of the system
Messes, problems and puzzles need to be identified and dealt with using appropriate approaches
Interconnectedness and interdependence gives rise to a range of phenomena and
behaviours
`
To find out more, read the paper!
8 Adaptive Agents
10 Co-Evolution
4 Non-Linearity
3 Emergence from
SimpleRules
6 Phase space and attractors
5 Sensitivity to initial
conditions
1 Interconnected and interdependent
elements and dimensions
2 Feedback processes
7 Strange
attractors and the edge of
chaos
9 Self organisation
Features of systems
Dynamics of change
Behaviour of agents
Agenda
Evaluations – some common issues Complexity science – origins and ideas Implications for evaluations Summary
Evaluation of Complex Systems is NOT new Educational systems, social initiatives and
government interventions are complex social systems where effective evaluation is seen as a key process in measuring success
But there is an increasing recognition that for evaluation of complex social systems to be more effective, evaluations may need to take into account the theoretical understanding of complex systems
Implications for evaluation are at four levels
1. Implications for evaluation methodologies and approaches
2. Implications for the focus of evaluations
3. Implications for the purpose and scope of evaluations
4. Implications for evaluations as a complex system in their own right
Implications for evaluation methodologies and approaches
Linear models dominate...
If you deliver the
product and / or service to the extent
Intended, then your participants
will benefit in certain ways
If these benefits to participants
are achieved, then certain changes in
communities, organisations and systems
might be expected to occur
If you accomplish your planned
activities, then hopefullyyou will deliver the amount of the product
and / or service that
you intended
Certain resources
are needed to operate
your program
If you haveaccess to
them, then you can use
them to achieve your
plannedactivities
Outcomes / Purpose
Impacts / Goal
OutputsInputs Activities
MONITORING EVALUATION
Many evaluation “results chains” visualize change as linear, based on multiple cause-effect logic models
X Y
Linear, Predictable
Focused on the end result
The program (X) gets the credit!
Inputs Activities Outputs Outcomes Impact
When applied to development, “the results chain” can lead to Faulty thinking Misguided data collection Misleading reporting of results Gives a false sense of reality to senior
managers and donors who are far from where change is taking place
Complexity science sees change as…
Interconnected (multiple actors and factors)
Non-linear (unexpected results occur)
Incremental, cumulative, with tipping points
Beyond the control of the project / programme
Two-way (program also changes)
Continuous (not limited to the life of the project)
Therefore
Develop new theories of change, adapt existing theories of change to challenge assumptions of linearity
Design evaluations to allow for interdependencies and interconnections in the system the program is trying to influence, and capture the resulting dynamics
Implications for evaluation focus
Focus of “traditional” evaluations
Formal project / programme / organisation
Environment is outside the organisation evolves separately until programme is implemented
Level of analysis is single or at most a few, relatively independent levels
Implications of complexity Features of systems: evaluate from perspective of
multiple, nested levels of interconnected systems, study feedback between the organisation and its environment, look for emergent rather than planned change
Dynamics and nature of change: Look for non-linearity, anticipate surprises and unexpected outcomes, analyse the system dynamics over time and frame the “space for possible change”, look for changes in conditions that facilitate systemic change, and how well matched the programme is to the wider system
People, motivations and relationships: Study patterns of incentives and interactions among agents, study quality of relationships, study individuals and informal / shadow coalitions, vs. formal organisation, study co-evolution of organisation and environment
Implications for evaluation purpose and scope
M&E is seen as standing in contrast to creative dynamism of field work
Purpose and scope of traditional evaluations vs complexity-oriented evaluations
Traditional Complexity-oriented
Measure success against predetermined goals
Develop new measures and monitoring mechanisms as goals emerge & evolve
Render definitive judgments of success or failure
Provide feedback, generate learning, support direction or affirm changes in direction
Aim to produce generalisable findings across time & space
Aim to produce context-specific understandings that inform ongoing innovation
Creates fear of failure Supports hunger for learning
Implications for evaluation as a complex system
Key Assumptions
Assumptions about systems Ordered Reductionist - parts would reveal the whole
Assumptions about how systems change Linear relationships influence as direct result of force from one object to
another - predictable Simple cause & effect
Assumptions about human actions Rational choice Behavior specified from top down Certainty and “knowability”
Goals ResultsActivities
Monitoring
Evaluation
Evaluations traditionally seen as a rational, technical, information-generating process
...Reality is a little different...
Evaluation systems are themselves complex systems, with many interconnected parts and dimensions; no two evaluations are the same
Focus of evaluation Policy / Guidelines Scope Project vs Policy Demand, goals Timing, quantity Preparation, TOR Management and team selection Methods and tools Engagement with stakeholders Dissemination of findings and utilisation Costs Quality maintenance mechanisms
“...To be effective an evaluation program must match the dynamics of the system to
which it is applied....”
Eoyang and Berkas (1998)
Implications of evaluations as a complex systemTraditional Complexity-oriented
Position the evaluator outside to assure independence and objectivity
Position evaluation as an internal, team function integrated into action and ongoing interpretive processes
Accountability to control and locate blame for failures
Learning to respond to lack of control and stay in touch with what’s unfolding and thereby respond strategically
Accountability focused on and directed to external authorities and funders
Accountability centred on fundamental values and commitments
Evaluator controls the evaluation and determines the design based on the evaluator’s perspective about what is important
Evaluator collaborates in the change effort to design a process that matches philosophically and organisationally
All systems can be placed on a spectrum between extremes of ordered and chaotic E.g. solids and gases
In solids, atoms are locked into place In gases they tumble over one another at random
In between the two extremes, at a phase transition, a phenomenon called the ‘edge of chaos’ occurs This phenomenon describes systems behaviours where the evolution of
the system never quite locks into place and never quite dissolve into turbulence either.
In human organisations, the simplest example is of a system that is neither too centrally controlled (order) nor too bottom-up and therefore disorganised (chaos)
Physiology on the Edge of Chaos
Healthy Heart – on the edge of chaos
Severe Congestive Heart Failure – orderly
Cardiac Arrhythmia, Atrial Fibrillation - chaotic
Dynamics
Ad
ap
tab
ility
Ordered Disordered
ZONE of HEALTH
Point of Maximum Adaptability
Threshold Threshold
High
Low
Dynamic adaptability is the key to system health
Changelessness is a sign of death,transformation a sign of life.- Commentary on the I Ching
Evaluation at the
edge of chaos?
Agenda
Evaluations – some common issues Complexity science – origins and ideas Implications for evaluations Final points
Complexity science gives additional weight to calls for re-thinking
The wider contexts of aid work The nature of the processes involved in aid work The dynamics of change involved in aid work The real influence of aid work The role of partner organisations and
beneficiaries in aid work The tools and techniques for planning,
monitoring, learning and evaluating aid work
Given the resistance to change in the power dynamics of aid, real world applications may continue to be “innovative”, “under the radar”
and outside the mainstream of aid policy and practice...
There are a number of common criticisms of complexity... Theoretical: adds nothing new
E.g. the ideas of complexity science have all been identified elsewhere But complexity brings them together
Practical: doesn’t specify what should be done E.g makes no specific recommendations as to how best to act in complex systems See next slide
Supports managerial “snake-oil” / “complexologists” / re-warmed ideas E.g. is abused and misused What isn’t?
Political: emergence and self organisation support neo-liberal stances Just because self-organisation happens, doesn’t mean it is good
Credit crisis Rwandan genocide
Complexity is more centre-ground, for example, “edge of chaos” systems are seen as most robust and resilient because are the optimally combination of control and flexibility
...and different perspectives on the value of complexity science Deep paradigmatic insights
Champions
Interesting parallels and useful approaches, but not the only way to see things Pragmatists
Meaningless coincidences Critics
Source of weakness is also the source of strength These ideas are not about “what you dos”, but about
“how you dos” Not “solutions for problems”, but “approaches to problems” Tools for furthering understanding, for opening up new ways of
seeing and thinking
They point to the personal, professional, institutional, political mindsets, attitudes and conditions which need to be in place to work effectively in and with complex systems
Complexity concepts can support the intuition and navigation of practitioners
INPUTS
ACTIVITIES
OUTPUTS
OUTCOMES
IMPACTS
Four suggestions
Develop collective intellectual openness to ask a new, potentially valuable, but challenging set of questions of our mission and their work
Develop collective intellectual and methodological restraint to accept the limitations of a new and potentially valuable set of ideas not misuse or abuse or let them become part of the ever-swinging pendulum of
aid approaches
Need to be humble and honest about the scope of what can be achieved through ‘outsider’ interventions, about the kinds of mistakes that are so often made, and about the reasons why such mistakes are repeated
Need to develop the individual, institutional and political courage to face up to the implications of complexity
...We can't solve problems by using the same kind of thinking
we used when we created them..
Final points (1)
Everybody thinks to change the
world; nobody thinks to
change themselves
Final points (2)
Thank you!
Get in [email protected]