Four Rules of the Art

AND SOME POKING AROUND FOR PRACTICAL SOLUTIONS TO THE CHALLENGES OF

JOINT PEACE INTERVENTION EVALUATIONS

Emery BrussetNUPI/IPI Workshop, New York

7 & 8 May 2009

Channel Research 2

Four Theoretical Challenges to the evaluation of conflict interventions

1. Seizing Opportunities

• Programming requires a response to constraints and opportunities as they arise.

• These cannot be planned for in advance and plans are not a good evaluation reference

1. Seizing Opportunities

• Programming requires a response to constraints and opportunities as they arise.

• These cannot be planned for in advance and plans are not a good evaluation reference

2. Attribution

• It is particularly difficult to separate the impact of peace-building projects from other impacts on conflict

• The time required to cover other interventions is limited

2. Attribution

• It is particularly difficult to separate the impact of peace-building projects from other impacts on conflict

• The time required to cover other interventions is limited

3. Data Quandaries

• Selecting evidence requires prioritisation and demands a good understanding of the nature of the conflict

• The verification methods must be applicable in a conflict situation

3. Data Quandaries

• Selecting evidence requires prioritisation and demands a good understanding of the nature of the conflict

• The verification methods must be applicable in a conflict situation

4. End State?

• Problems in defining terms such as peace and conflict.

• Usually depends on the identification of an end-state.

• This may not be explicit or shared by all stakeholders or those within a single programme

4. End State?

• Problems in defining terms such as peace and conflict.

• Usually depends on the identification of an end-state.

• This may not be explicit or shared by all stakeholders or those within a single programme

Options

Channel Research 3

The Practical Challenge of Joint Evaluation: Evaluation needs to provide a convincing set of conclusions to a growing diversity and size of audience

“Ideal single actor evaluation” • TOR enable the

evaluator to identify unplanned and unintended effects

• A way is found to identify comparative weight of intervention

• Good conflict analysis is possible

• The nature of “peace” is agreed

“Reality of Joint Evaluation”

• Expectations evolve

• The questions focus on the management of interventions

• There is no strategic baseline or agreed assessment

• Peace is defined generically, or not at all

4 Rules of Good Evaluation:

1. Independence of team to focus on situation as it emerged

2.Team has means to analyse societal interface

3.Team is capable of participatory assessments

4.Conflict objectives are reconstructed

toFrom

Channel Research 4

• Challenge 1: Capturing unintended and unexpected aspects

Rule 1: The team is enabled to define what issues to focus on, and allows for Steering Committee consultation milestones to secure alignment of all stakeholders.

Assess

Adapt Act

Ongoing Analysis Rolling Assessment

Intensive ManagementOngoing Decision Making and

Documentation

RISK

Challenge 2: Evaluation questions tend to focus on institutional issues

Evaluation, particularly real time evaluation, tends to be linked to decisions and the need to understand and support the management of an intervention. This risks reducing the time available to analyse the impact .

Conflict Analysis Channel Research 6

Rule 2: Evaluations are given time to enable an iterative but systematic process (six months to a year?) – scope is contained. There is agreement that attribution of impact is replaced with contribution.

Threat

Threat

Threat

Threat

Threat

Threat

?$$

Success depends on sufficient resources

Challenge N° 3: Get the readers and audience to agree on a consistent analysis of the conflict, and on key drivers of peace

Map of situation

Intervention footprint

Situation withintervention factored in –

how does it affectsituation, how doessituation affect it?

Channel Research

Channel Research 8

Participatory Mapping of Immediate Conflict Environments (MICE) works by looking at linkages between key trends and events = drivers

Event: occurs ata point in time

Trend: changein a condition orcharacteristic

Arrow of change: increased probability orintensity

The elements of conflict mapping

GrowingUnemployment

Coup

Corruption more apparent

Officers inc. disgruntled

Battle in whichwar was lost

Growing leftistradicalisation

Dissentincreasinglysuppressed

Many insurgent Attacks in XX

Hypothetical and partial example: Country withmilitary regime in conflict with leftist insurgents

Channel Research 9

The aim is the identification of key issues that need to be addressed to mitigate conflict or instability

Growing unemployment

Coup

Corruption more apparent

Officers inc.disgruntled

Battle in whichwar was lost

Growing leftistradicalisation

Dissent increasinglysuppressed

Many insurgent Attacks in XX5

4

4

4

To identify priority issues, we count the linkages leading toand from relevant events and trends – those with the mostlinkages have the most reverberation in the situation

Priority issues:

1. Leftist radicalisation

2. Militarism, state repression, military dictatorship

CAVEAT

Conflict mapping is not root cause or systems analysis. Although corruption and unemployment might be major causes of leftist radicalisation, we do not identify them as priority issues – ranking is not intuitive at all

Channel Research 10

Representation is qualitative and culturally neutral. For example…

Militarycoup Leftist

radicalisation

Leftistuprising

MilitaryPoliticisation increases

Militaryrepression

Labouractivism

More Poverty

Peasantrevolt

Leftist – peasantcooperation

US companyoil contract

Leftistperception of

collusion

Companyunpopularityon ground

Rebel threatsto company

Strikes amongproject staff

Increasingemployment

Increasingstate revenues

Military regimecorruption

Human rightsNGO scrutiny

Shareholderscrutiny

US gov’tcriticism of

military regime

Pressure fordemocratisation

Military steps upproject security

Generalstrikes

Oil productionincreases

_

_

_

Channel Research 11

An example from a real mapping exercise in a post-conflict African country

Channel Research 12

Mapping of Cross-Border Issues in the Conflict in NIWorkshop 1 Glencree Centre for Reconciliation

Workshop 2 Monaghan

Workshop 3 Derry

Workshop 4 East Antrim

Workshop 5Letterkenny

Unionists feel under threat Republic’s population losing interest in Northern Ireland

Belfast/Good Friday Agreement

Creation of cross-border bodiesBritain and Ireland joining the EEC

Launch of the peace process in 19921970s further increase in violence

Partition (1920/21) Fear emerged (late 1960s/early 1970s)

Border region becomes established Belfast/Good Friday

Agreement1994 IRA ceasefire and negotiations /peace process

Crushing of civil rights movementIncreased disengagement by Irish Government

Abandonment of the border regionNorthern Ireland seen as a foreign country

Protestant / Unionist security and law enforcement

Civil rights movement/ marches (1960s)

B Specials introduced

Increasing involvement of people in paramilitary activity (especially hunger strikes (1981)

Violent British government response (late 1960s)

Closure of border roads (1970s)

Establishing checkpoints

Increasing segregation of housing – late 1960s/ early 1970s

Unionism increasingly associated with Loyalism and negative images

Working class ‘cut off’, paramilitaries

DUP and UUP responding to availability of peace money, e.g. Gatekeeping of the funding

Nationalist movement portrayed as freedom fighters

Belfast/Good Friday Agreement

92 miles of border in Donegal

Increased intimidation of businesses in Donegal

Maturity in cross-community relations

Both countries have run away from EU commitments on tariffs and trade agreements

Celtic Tiger exceptional performance of the economy

Channel Research 13

Challenge N° 4: objectives and their connection to peace are unclear or outdated

Project summary Verifiable indicators Means of verification Important assumptions

Gen. Objective/impact: Overall goal which this project will help to achieve

The evidence (quantitative / qualitative) which will be used to measure / judge the achievement of goal

Sources of information / data which will be used to assess the indicator(s)

(Goal to super goal)Main external factors necessary to sustain objectives in the long run

Purpose/Specific Objective/outcome: Immediate impact on the project area or target group, that is, the change or benefit to be achieved by the project

The evidence (quantitative / qualitative) which will be used to measure / judge the achievement of the purpose

Sources of information / data which will be used to assess the indicator(s)

(Purpose to goal) Main external factors necessary if project purpose is to contribute to reaching project goal

Outputs: The specific, deliverable results expected from the project to attain the purpose

The evidence (quantitative / qualitative) which will be used to measure / judge the achievement of the outputs

Sources of information / data which will be used to assess the indicator(s)

(Outputs to purpose) Main external factors necessary for outputs to achieve project purpose

Complexity in PLANNING is not well captured by prevailing tools

“Project”

Channel Research

LogFrame

“Strategy”

???

Rule 3: allow the team to innovate and interpret in mapping objectives and their meaning in terms of peace

“Project”

Channel Research

LogFrame

“Strategy”

“Effect Diagram”

Channel Research 17

The intervention tree in its simplest form – it is an adapted logic or decision tree

We need to achieve this, and to achieve this we need to have obtained…

… this, and to see this, we need to do…

… this, and to see this, we need to do…

… this, and to see this, we need to do…

This

This

This

This

This

This

This

This

This

Each new branch needs to be a separate and complete initiative to the extent possible, toensure no redundancy, and to spot any gaps

Poverty reduced

Alternative rural livelihoods created

Equitable access to productive assets, markets and services (e.g. microfinance) established

EC activities Outputs / Outcomes Impacts

Rural economy strengthened

Poppy production reduced or stabilizedState of law and order

improved (incl. drug enforcement

Sustainable management of natural resources achieved / improved

Vulnerability of rural population decreased

Rural production and gainful employment strengthened

More effective provincial / community based institutions developed

Governmental / admin. structure strengthened

Civil Service System improved (including police)

Governmental physical infrastructure re-created

Gov / ATA budget stabilized

National payment system created

Revenue position of Gov strengthened

Reform of civil service promoted

Access to BHCP improved / increased

Health sector management on central and provincial level improved

Positive tendency in health status of population (especially U5 and maternal mortality)

Rural Developm

entPublic Adm

inistration Reform

Foundation for public administration reform created (NOTE: in 2nd NIP, increased focus on sub-national level)

(Participation of) rural communities in dev. Strengthened

Health

Example: EC Effect Diagram Afghanistan

Channel Research 19

Actions OutcomesOutputs Global Impact

Conflicts in partner countries prevented

Regional co-operation and integration strengthened

Support measures for regional integration mechanisms and trade co-operation implemented

Direct Impact

CS

P used as central instrum

ent for m

ainstreaming conflict prevention

Trade links between countries strengthened.

Macro-economic environment in at risk countries stabilized.

Equal participation of men & women in social, economic & political life promoted (14).

1) Elements of Structural Stability: sustainable economic development, democracy & respect for Human Rights, viable political structures, healthy environmental & social conditions, capacity to manage change without resorting to conflict.

Potential for more favourable democratic environment increased(14).

Rule of Law, Good Governance & Civil Society strengthened & (13)

Security sector prepared for post-conflict situation

Peace in post-conflict countries consolidated (16).

Potential for renewed conflicts in post-conflict countries reduced (16).

Stru

ctu

ral S

tab

ility im

pro

ved

1)

Support to security sector increased (I.e. police training, HR training) (14).

Rehabilitation and reconciliation programmes implemented (16).

Targeted actions implemented to support electoral process, parliamentary activities, administration of justice (14).

Macro-economic assistance operations implemented (12).

Integ

rated A

pp

roach

Ow

nership of process by partner country established

Cross-cutting issues efficiently addressed in EC activities

Mechanisms developed to appropriately address cross-cutting issues in relevant areas:- social conditions (drugs, small

arms, Health / communicable diseases (HIV / Aids), Population flows / trafficking)

- environment & natural resources (water)

- economy (Private sector in unstable areas)

Early-warning mechanism & procedures created

EU decision-making & operational centres alerted to imminent crisis

EC enabled to formulate appropriate response to developing crisis situation (incl. preventive sanctions)

information on cause & potential consequences of conflict provided

EC enabled to react quickly to developing crisis situation

Rapid Reaction Mechanism (RRM) introduced EC enabled to carry out

wide-ranging short term missions on short noticeCo-ordination of Member States on

common political line intensified (incl. sanctions)(23).

More focused, time-flexible & robust political dialogue & diplomacy enabled (23)

Degeneration of political difficulties / tension into armed conflict prevented

Traditional community instruments optimised to respond to conflict (including ESDP)

Reform of aid management allows more responsiveness to needs in line with the performance of partner countries

Coherence of approaches for conflict prevention improved among partners (countries, international organisations, NGOs)

Dialogue with partners intensified & focused on conflict prevention (incl. NGOs)(25)

Common early warning systems, monitoring procedures & conflict indicators developedAppropriate mechanisms &

procedures for coordination with international partners developed.

Cooperation with international partners intensified

Tensions between countries reduced

Poverty reduced (14).

Economic growth stimulated

Societal post-conflict tension reduced

Potential for new conflicts in at at-risk countries reduced.

Potential for conflicts between partner countries reduced

Analysis of Effect Diagrams

What is their purpose?• Visualize hierarchy of objectives / expected effects• Reveal logical links, help to define certain Evaluation Questions

& assess internal coherence of strategy

What are their weaknesses?• Do not capture changes over time or feedback loops• “Other actor” / “Implementation” / “Context” not captured

Conclusion?• For complex interventions: can be used for sub-diagrams• Shows missing elements • Does not provide the final leap from objective to situation

change

Channel Research

The diagram can be turned into a narrative: theories of change

• Theories of change can take the simple format: “We believe that by doing X (action) successfully, we will produce Y (movement towards peace).”

• Or the chain of logic might involve multiple steps: “if we do X, that will lead to Y, which will encourage Z, which will exert an influence on key actor A.”

• In order to create, uncover or test a theory of change, it is essential to conduct a comprehensive analysis of the conflict, identify the key factors of conflict and the relationship between them, and spell out how the intervention (project, programme or policy) will create change.

• Making theories of change explicit is a central part of planning and programme design and can facilitate evaluation.

Channel Research 21

List of Identifiable Theories of Change:

• The Individual Change Theory: Peace comes through transformative change of a critical mass of individuals, their consciousness, attitudes, behaviors and skills. [Methods: investment in individual change through training, personal transformation/consciousness-raising workshops or processes; dialogues and encounter groups; trauma healing.]

 • The Healthy Relationships and Connections Theory: Peace emerges out of a process of

breaking down isolation, polarization, division, prejudice and stereotypes between/among groups. Strong relationships are a necessary ingredient for peacebuilding. [Methods: processes of inter-group dialogue; networking; relationship-building processes; joint efforts and practical programs on substantive problems.]

 • The Withdrawal of the Resources for War Theory: Wars require vast amounts of material

(weapons, supplies, transport, etc.) and human capital. If we can interrupt the supply of people and goods to the war making system, it will collapse and peace will break out. [Methods: anti-war campaigns to cut off funds/national budgets; conscientious objection and/or resistance to military service; international arms control; arms (and other) embargoes and boycotts.]

 

Channel Research 22

• The Reduction of Violence Theory: Peace will result as we reduce the levels of violence perpetrated by combatants or their representatives. [Methods: cease-fires, creation of zones of peace, withdrawal/retreat from direct engagement, introduction of peacekeeping forces/interposition, observation missions, accompaniment efforts, promotion of nonviolent methods for achieving political/social/economic ends.]

• The Root Causes/Justice Theory: We can achieve peace by addressing the underlying issues of injustice, oppression/exploitation, threats to identity and security, and peoples’ sense of injury/victimization. [Methods: long-term campaigns for social and structural change, truth and reconciliation; changes in social institutions, laws, regulations, and economic systems.]

 • The Institutional Development Theory: Peace is secured by establishing stable/reliable social

institutions that guarantee democracy, equity, justice, and fair allocation of resources. [Methods: new constitutional and governance arrangements/entities; development of human rights, rule of law, anti-corruption; establishment of democratic/equitable economic structures; economic development; democratization.]

 • The Political Elites Theory: Peace comes when it is in the interest of political (and other) leaders

to take the necessary steps. Peacebuilding efforts must change the political calculus of key leaders and groups. [Methods: raise the costs and reduce the benefits for political elites of continuing war and increase the incentives for peace; engage active and influential constituencies in favor of peace; withdraw international support/funding for warring parties.]

 

Channel Research 23

Channel Research 24

• The Grassroots Mobilization Theory: “When the people lead, the leaders will follow.” If we mobilize enough opposition to war, political leaders will have to pay attention. [Methods: mobilize grassroots groups to either oppose war or to advocate for positive action. Use of the media; nonviolent direct action campaigns; education/mobilization effort; organizing advocacy groups; dramatic events to raise consciousness.]

 • The Economics Theory: As a politician once said, “It’s the economy, stupid!” People

make personal decisions, and decision-makers make policy decisions based on a system of rewards and incentives and punishment/sanctions that are essentially economic in nature. If we can change the economies associated with war-making, we can bring peace [Methods: use of government or financial institutions to change supply and demand dynamics; control incentive and reward systems; boycotts.]

 • The Public Attitudes Theory: War and violence are partly motivated by prejudice,

mispercep tions, and intolerance of difference. We can promote peace by using the media (television and radio) to change public attitudes and build greater tolerance in society. [Methods: TV and radio programs that promote tolerance; modeling tolerant behavior; symbolic acts of solidarity/unity; dialogues among groups in conflict—with subsequent publicity.]

25OTI HQ

mission

OTI+I.P.

StateNGO

StateNGO

StateNGO

StateNGO

StateNGO

StateNGO

MoH Police CommunityLeaders

Families Banks PCs

Outcome Mapping: Focusing on actors and behaviour

Channel Research 26

Challenge N°5: Bringing it all back together

E T

E

E

E

TT

T

Linkage mapping Issue 1

Issue 2

Issue 3

Outcomes

Outcomes

Outcomes

Outcomes

Relevance

x

Extent

x

Duration

Focus of evaluationThe Significance of Impact methodology has been designed to provide solid evidence on the contribution which an actor makes to a particular issue

Issues identified with the mapping tools can provide the starting point

Priority issues

a. Poor relations with local community

b. Rivalry between central and provincial administration

c. Criminal economy

d. Ethnic cleavages lead to growing fear of threats

Extent and Duration

a. What proportion of the target population is covered?

b. Were the key people accessed?

c. How long did the influence last?

d. Can the influence be sustained after the ending of the intervention?

Duration

Extent

a

b

c

d

SIGNIFIC

ANCE

Priority risks

Leading to three dimensional assessments…

Priority issues

a. Poor relations with local community

b. Rivalry between central and provincial administration

c. Criminal economy

d. Ethnic cleavages lead to growing fear of threats

Extent and Duration

a. What proportion of the target population is covered?

b. Were the key people accessed?

c. How long did the influence last?

d. Can the influence be sustained after the ending of the intervention?

Relevance:

- What do others do?

- Are there other options?

- Could a clearer objective have been designed?

- Does the intervention correspond to the need assessed?

Degreeof

relevance

Extent and Duration

Extent

Duration

Significance

D

C

B

A

D. . . C. . .

B. . .

A. . .

The tools is designed to highlight what has been done and test its connection to the issues which are recognised as issues to be addressed to achieve ‘progress’

Channel Research 29

All levels of the result chain can yield important results or outcomes: demobilisation

Activities and process deliver outputOutputs are defined and easily verified

Example: “a workshop is held”

First levelOperational level

Specific objectives are the results of activities and ideally are how outputs are to be used by the beneficiary groups – user interface

Example: “skills and tools are taught”

Second levelProgramme

level

General objectives are the statement of overall verifiable change intended

Example: “Armed groups are able to design a new force concept”

Third Strategy Level

Society & State

Process can be the output

The issue can be that groups have lost contact and need to build mutual respect

Example: “Familiarity is achieved”.

New capacities can lead to changes in the command structure and the reintegration of groups.

Example: “Specialisation and opportunities are required for conversion to civilian life”

Changes in the overall environment can lead to a better notion of security based on smaller forces

Example: “Security imperatives require larger armed groups”.

Strategy Outcome

The Table of Contents of the Evaluation Report!

1. Issues / Entry Points have been identified

2.Outcomes achieved have been identified

3.Assessment of the value of the contribution is possible:• Relevance and efficiency• Effectiveness • Sustainability• Coherence with other interventions• Overall conclusions on « posture »• Recommendations

Channel Research 30