Monitoring and evaluating capacity-building at multiple levels

Monitoring and evaluating capacity-building at multiple levels

UNFCCC Meeting on monitoring and evaluating CB, Rio, Brazil

6-7 November 2008

David Watson

Consultant

UNFCCC Meeting on monitoring and evaluating CB, Rio, Brazil

6-7 November 2008

David Watson

Consultant

Scope of PresentationScope of Presentation• Background: perspectives and

approach• The story so far: M&E of CB under

UNFCCC• How other global programmes tackle

M&E of CB• The big picture: M&E of CB in

international development• Alternative paradigms: esp. systems

thinking• Towards a tool-kit for practitioners

• Background: perspectives and approach

• The story so far: M&E of CB under UNFCCC

• How other global programmes tackle M&E of CB

• The big picture: M&E of CB in international development

• Alternative paradigms: esp. systems thinking

• Towards a tool-kit for practitioners

BackgroundBackground

• Sorry about late distribution of paper! (See Section X for a potted summary!)

• Personal ‘points of departure’: building institutions and governance

• A touch of scepticism about M&E practices; pragmatism; incrementalism; practicality; process

• Broadening debate constructively• Not ‘selling’ any particular approaches• Cognisant of the seriousness of the challenges

of slowing climate change, and of enhancing the effectiveness of CB to that end

• Sorry about late distribution of paper! (See Section X for a potted summary!)

• Personal ‘points of departure’: building institutions and governance

• A touch of scepticism about M&E practices; pragmatism; incrementalism; practicality; process

• Broadening debate constructively• Not ‘selling’ any particular approaches• Cognisant of the seriousness of the challenges

of slowing climate change, and of enhancing the effectiveness of CB to that end

Levels of Capacity and CBLevels of Capacity and CB

• May be helpful to bear in mind two complementary dimensions:

• ‘Horizontal’: from individuals, through organisations, to social / national systems or networks

• ‘Vertical’ : from local, to regional, to national, to global.

• May be helpful to bear in mind two complementary dimensions:

• ‘Horizontal’: from individuals, through organisations, to social / national systems or networks

• ‘Vertical’ : from local, to regional, to national, to global.

Other Global Programmes’ Experience with M&E of CBOther Global Programmes’ Experience with M&E of CB

• PRSPs

• GAVI / Roll Back Malaria

• UNAIDS

• Programme Based Approaches (PBAs)

• WFP

• Lessons pertinent to CB to address Climate Change

• PRSPs

• GAVI / Roll Back Malaria

• UNAIDS

• Programme Based Approaches (PBAs)

• WFP

• Lessons pertinent to CB to address Climate Change

Lessons from other global progs.Lessons from other global progs.• Several (GAVI and RBM) had specialised

groups focussed on M&E, plus Task Forces on specialised subjects (e.g. CB)

• At country level: a Secretariat may have a M&E Cell; use existing M&E structures where they exist; keep it simple; minimise number of indicators.

• Empirical evidence of enhanced capacities is often scarce

• ‘Strategic incrementalism’ attracted PBA-ers attention: pursue ‘quick wins’ and ‘peer-learning’

• Several (GAVI and RBM) had specialised groups focussed on M&E, plus Task Forces on specialised subjects (e.g. CB)

• At country level: a Secretariat may have a M&E Cell; use existing M&E structures where they exist; keep it simple; minimise number of indicators.

• Empirical evidence of enhanced capacities is often scarce

• ‘Strategic incrementalism’ attracted PBA-ers attention: pursue ‘quick wins’ and ‘peer-learning’

..lessons continued…..lessons continued…• Clarify objectives of CB, based on thorough

needs assessments, as a basis for a simple results framework

• Try to avoid a ‘missing middle’: indicators for the steps between outputs and outcomes;

• Reflect on whether the M&E process should feed more into national political processes, (engaging e.g. with parliaments, media, civil society) AND be a precondition of partnerships nationally and with the international community.

• Don’t be naïve in your ‘theory of political change’: …’participation’ is not enough.

• Clarify objectives of CB, based on thorough needs assessments, as a basis for a simple results framework

• Try to avoid a ‘missing middle’: indicators for the steps between outputs and outcomes;

• Reflect on whether the M&E process should feed more into national political processes, (engaging e.g. with parliaments, media, civil society) AND be a precondition of partnerships nationally and with the international community.

• Don’t be naïve in your ‘theory of political change’: …’participation’ is not enough.

A poignant quotationA poignant quotation‘M&E practices do not provide a

framework for re-presenting (or making meaning from) the complex and multiple processes of institutional and individual learning and change that are sought in strengthening institutional capacity’

Source: Unitar ‘Challenges and Constraints in M&E of Capacity Building’ presentation to first (Antigua) workshop Nov 2007

‘M&E practices do not provide a framework for re-presenting (or making meaning from) the complex and multiple processes of institutional and individual learning and change that are sought in strengthening institutional capacity’

Source: Unitar ‘Challenges and Constraints in M&E of Capacity Building’ presentation to first (Antigua) workshop Nov 2007

M&E of CB : recent insightsM&E of CB : recent insights• Theme paper for ECDPM Study on Capacity

Change and Performance 2003-8 (Baser Morgan + Watson on Theme Paper)

• On M&E of CD tended to agree with UNITAR!• Often, ‘performance improvement’ indicators

are used as proxy for capacity increases• …not many e.g.s of ‘capacity’ being monitored• Accountability (to donors) the main ‘driver’• Most public sector CB experience

disappointing…political and institutional factors seem important

• Donors have a poor record on M&E of CB: ..including their own capacities and incentives

• Theme paper for ECDPM Study on Capacity Change and Performance 2003-8 (Baser Morgan + Watson on Theme Paper)

• On M&E of CD tended to agree with UNITAR!• Often, ‘performance improvement’ indicators

are used as proxy for capacity increases• …not many e.g.s of ‘capacity’ being monitored• Accountability (to donors) the main ‘driver’• Most public sector CB experience

disappointing…political and institutional factors seem important

• Donors have a poor record on M&E of CB: ..including their own capacities and incentives

‘Capacitated’ organisations:‘Capacitated’ organisations:1. ..carry out tasks effectively

2. ..engage, decide, act

3. ..relate, attract resources & support

4. ..adapt and self-renew

5. ..balance diversity, and build coherence across the organisational system

Capacity: ‘that emergent combination of individual competences and collective capabilities which enables a human system to create public value

1. ..carry out tasks effectively

2. ..engage, decide, act

3. ..relate, attract resources & support

4. ..adapt and self-renew

5. ..balance diversity, and build coherence across the organisational system

Capacity: ‘that emergent combination of individual competences and collective capabilities which enables a human system to create public value

..and implications for M&E of CD..?..and implications for M&E of CD..?• Agree on the nature of capacity to be

monitored!• ..need to focus on more complex issues…• Information on change and progress should

not be sent ‘up’ …but shared internally for purpose of learning

• Pay more attention to the context…and inter-relationships in capacity processes..

• to the changes taking place (intended or not) and their contributions to ‘capacity’…

• ..i.e. need more participative approaches to M&E… and to learning and reflection..

• Agree on the nature of capacity to be monitored!

• ..need to focus on more complex issues…• Information on change and progress should

not be sent ‘up’ …but shared internally for purpose of learning

• Pay more attention to the context…and inter-relationships in capacity processes..

• to the changes taking place (intended or not) and their contributions to ‘capacity’…

• ..i.e. need more participative approaches to M&E… and to learning and reflection..

Alternative Paradigms to M&E (of CB)

Alternative Paradigms to M&E (of CB)

• Critical reactions to ‘monitoring abilities to perform…and meeting pre-determined objectives’

• Wheatley: ‘measurement’ = anathema (when looking at human behaviour)

• ‘Cause and effect’ logic inapplicable• ..change processes are complex..involve inter-

dependencies…and are not ‘linear’ or stable, nor necessarily visible

• ..but may well be long-term, and more political than technical

•

• Critical reactions to ‘monitoring abilities to perform…and meeting pre-determined objectives’

• Wheatley: ‘measurement’ = anathema (when looking at human behaviour)

• ‘Cause and effect’ logic inapplicable• ..change processes are complex..involve inter-

dependencies…and are not ‘linear’ or stable, nor necessarily visible

• ..but may well be long-term, and more political than technical

•

Systems Thinking’s relevanceSystems Thinking’s relevance• Growth in interest in ST as an analytical

framework for development and natural resource management

• Explicitly examined as part of ECDPM study: helps address implications mentioned

• Human institutions seen as complex adaptive systems: e.g. climate!

• See Box 1 for a summary of basic concepts• ‘Mess’ = problems unbounded in scope, time

and resources; no clear agreement on optimal solution or how to achieve it; goals and strategies are contested; perspectives differ; no clear cause and effect; uncertainty prevails.

• Growth in interest in ST as an analytical framework for development and natural resource management

• Explicitly examined as part of ECDPM study: helps address implications mentioned

• Human institutions seen as complex adaptive systems: e.g. climate!

• See Box 1 for a summary of basic concepts• ‘Mess’ = problems unbounded in scope, time

and resources; no clear agreement on optimal solution or how to achieve it; goals and strategies are contested; perspectives differ; no clear cause and effect; uncertainty prevails.

What about ‘Capacity’? What about ‘Capacity’?

• ‘Capacity is an emergent property of human systems’ (ECDPM Study)

• (Emergent properties are those which have no meaning in terms of parts making up the whole system. They are the outcome of system behaviour or synergy.)

• Two examples: ENACT Jamaica (Box 2)• And IUCN Asia (Box 3)

• ‘Capacity is an emergent property of human systems’ (ECDPM Study)

• (Emergent properties are those which have no meaning in terms of parts making up the whole system. They are the outcome of system behaviour or synergy.)

• Two examples: ENACT Jamaica (Box 2)• And IUCN Asia (Box 3)

ENACT IUCN ENACT IUCN• Formal performance

monitoring system abandoned

• Empowerment of frontline staff for rapid response

• Absence of a ‘model’ to assess performance

• Let partners adapt and adopt measures

• Donor modified its approach..more ‘learning-friendly’ faced with diversity

• Formal performance monitoring system abandoned

• Empowerment of frontline staff for rapid response

• Absence of a ‘model’ to assess performance

• Let partners adapt and adopt measures

• Donor modified its approach..more ‘learning-friendly’ faced with diversity

• Unusually diverse membership

• Flexibility demonstrated by funding agencies

• Permitted experimentation, innovation and creativity

• Evolution of IUCN learning processes

• ‘Teaming’ process• Ownership by

governments + IUCN credibility / legitimacy

• CB= continuous process ‘no road map, only a goal’

• Unusually diverse membership

• Flexibility demonstrated by funding agencies

• Permitted experimentation, innovation and creativity

• Evolution of IUCN learning processes

• ‘Teaming’ process• Ownership by

governments + IUCN credibility / legitimacy

• CB= continuous process ‘no road map, only a goal’

Cases’ commonalities c.f. STCases’ commonalities c.f. ST• Identification and recognition of goals• Emphasis on values to be reflected• Clarity and awareness of mission amongst clients too• Leadership encourages experimentation• Opportunities for learning from experience: self-

assessment and ‘stories’ of positive experience or changes or errors

• Flexibility to adapt (thru new skills-building) to new needs / priorities, thru OJT ‘hands-on’

• Informality of M&E systems: responsive to needs of clients / network members

• Ability to learn from experience is crucial

•

• Identification and recognition of goals• Emphasis on values to be reflected• Clarity and awareness of mission amongst clients too• Leadership encourages experimentation• Opportunities for learning from experience: self-

assessment and ‘stories’ of positive experience or changes or errors

• Flexibility to adapt (thru new skills-building) to new needs / priorities, thru OJT ‘hands-on’

• Informality of M&E systems: responsive to needs of clients / network members

• Ability to learn from experience is crucial

•

BUT ‘Reductionist’ approaches are still relevant…where:

BUT ‘Reductionist’ approaches are still relevant…where:

• It is possible to define required capabilities unambiguously and specifically

• ..and to assess existing ones = ‘gap’• Therefore easy to define indicators• Where stakeholders able and willing to define

their shortfalls and ‘sign up’• Incentives exist to improve performance• Leadership, and all above combine into

‘ownership’• BUT ..this combination of circumstances is

rare! (e.g. public financial management IMF and WB forged consensus)

• It is possible to define required capabilities unambiguously and specifically

• ..and to assess existing ones = ‘gap’• Therefore easy to define indicators• Where stakeholders able and willing to define

their shortfalls and ‘sign up’• Incentives exist to improve performance• Leadership, and all above combine into

‘ownership’• BUT ..this combination of circumstances is

rare! (e.g. public financial management IMF and WB forged consensus)

Towards a ‘Tool Box’ for M&E of CD in Climate Change

Towards a ‘Tool Box’ for M&E of CD in Climate Change

• Principles: pragmatism: acknowledge weaknesses in all M&E systems: main aim = ‘sense-making’

• Seek out ‘what M&E exists, and what works, already’

• Acknowledge that the best M&E systems are ‘customised’: adapted by participants, based on local conditions

• Principles: pragmatism: acknowledge weaknesses in all M&E systems: main aim = ‘sense-making’

• Seek out ‘what M&E exists, and what works, already’

• Acknowledge that the best M&E systems are ‘customised’: adapted by participants, based on local conditions

Some existing frameworksSome existing frameworks• GEF Resource Kit on M&E• GEF Indicators for 5 key capacities:

– Engagement; generate access to and use of information; policy and legislation development; management and implementation; monitor and evaluate

– Scoring / indicator system for each;– Obliges actions / next steps / link to outcomes

• UNEP ‘Lessons Trees’– Trying to improve quality of learning (especially

about common problems) and application of learning to future programmes

• GEF Resource Kit on M&E• GEF Indicators for 5 key capacities:

– Engagement; generate access to and use of information; policy and legislation development; management and implementation; monitor and evaluate

– Scoring / indicator system for each;– Obliges actions / next steps / link to outcomes

• UNEP ‘Lessons Trees’– Trying to improve quality of learning (especially

about common problems) and application of learning to future programmes

Self-AssessmentSelf-Assessment• Examples from CB in research and development

organisations• SA workshops after applying qualitative and

quantitative tools• Managers, staff and stakeholders identify

strengths and weaknesses, and set new directions ….Advantages = (1) those with knowledge of and interest in the

organisation gain in-depth insights: what’s working and why..where improvements needed

(2) Well-prepared to address the changes needed

• E.g. M&E of past CB in Mekong Farming R&D Systems Institute (Box 4)– Preparation of ‘work stories’ on past CB efforts– + interviews with key staff on changes and challenges

• Examples from CB in research and development organisations

• SA workshops after applying qualitative and quantitative tools

• Managers, staff and stakeholders identify strengths and weaknesses, and set new directions ….Advantages = (1) those with knowledge of and interest in the

organisation gain in-depth insights: what’s working and why..where improvements needed

(2) Well-prepared to address the changes needed

• E.g. M&E of past CB in Mekong Farming R&D Systems Institute (Box 4)– Preparation of ‘work stories’ on past CB efforts– + interviews with key staff on changes and challenges

A ‘Balanced Approach’ to M&E of CB

A ‘Balanced Approach’ to M&E of CB

• Framework generated to address M&E of CB a la ECDPM Study dimensions of ‘Capacity’

• Piloted in PNG legal and judicial reform• Time-consuming and exhaustive: but

national practitioners did find it helpful in conceptualising all dimensions of capacities being built up.

• See extra handout

• Framework generated to address M&E of CB a la ECDPM Study dimensions of ‘Capacity’

• Piloted in PNG legal and judicial reform• Time-consuming and exhaustive: but

national practitioners did find it helpful in conceptualising all dimensions of capacities being built up.

• See extra handout

Appreciative Enquiry in Formative Evaluation

Appreciative Enquiry in Formative Evaluation

• Evaluators form more of an understanding of the political, cultural and historical ‘landscape’

• Encouraging organisations to develop their relationships with primary stakeholders

• Community Development Corporations in US – Seeking out ‘what enabled effectiveness, and what

hindered it’– Evaluator regularly listening to ‘stories’– Regular visits and sharing of reports– Recipients able to ‘co-create’ the initiative and

develop OWN capacity for assessment– Evaluator chosen by NGO not donor: NGO

employed not as expert but for its ability to learn collectively

• Evaluators form more of an understanding of the political, cultural and historical ‘landscape’

• Encouraging organisations to develop their relationships with primary stakeholders

• Community Development Corporations in US – Seeking out ‘what enabled effectiveness, and what

hindered it’– Evaluator regularly listening to ‘stories’– Regular visits and sharing of reports– Recipients able to ‘co-create’ the initiative and

develop OWN capacity for assessment– Evaluator chosen by NGO not donor: NGO

employed not as expert but for its ability to learn collectively

Most Significant Change (MSC)Most Significant Change (MSC)• First applied in evaluation of a complex RD

programme in Bangladesh– Process managers identify domains of change

which are important to evaluate– Stories (descriptions of changes deemed

significant: with reasons: why significant?) periodically collected from stakeholders

– Analysed and filtered up thru committees– Criteria for choosing stories are collated and fed

back to stakeholders;– Final selection made (annually?) with reasons– Circulated to all– Site visits to check; deepen understanding of

changes

• First applied in evaluation of a complex RD programme in Bangladesh– Process managers identify domains of change

which are important to evaluate– Stories (descriptions of changes deemed

significant: with reasons: why significant?) periodically collected from stakeholders

– Analysed and filtered up thru committees– Criteria for choosing stories are collated and fed

back to stakeholders;– Final selection made (annually?) with reasons– Circulated to all– Site visits to check; deepen understanding of

changes

MSC featuresMSC features• Focuses attention and direction of work in

programme towards valued directions• Dialogue and deliberations crucial• Takes place over time: responsive to changing

contexts• Policy makers, funders, field engaged looking at

the value of changes• Stories help all relate to information• Non-experts (the story-tellers) involved in

evaluation• Dialogue based on real experience and concrete

outcomes not abstract indicators• …MSC positively evaluated as technique in Laos

(Willetts 2004)

• Focuses attention and direction of work in programme towards valued directions

• Dialogue and deliberations crucial• Takes place over time: responsive to changing

contexts• Policy makers, funders, field engaged looking at

the value of changes• Stories help all relate to information• Non-experts (the story-tellers) involved in

evaluation• Dialogue based on real experience and concrete

outcomes not abstract indicators• …MSC positively evaluated as technique in Laos

(Willetts 2004)

Annotated Bibliography Annotated Bibliography

• ..use as a part of the ‘tool-kit’!• Sections on:

– Climate Change-related sources– Broader CB literature and M&E– Other Global Programmes’ Materials– Civil Society CB and M&E– Systems Thinking and Complexity

literature and M&E examples

• ..use as a part of the ‘tool-kit’!• Sections on:

– Climate Change-related sources– Broader CB literature and M&E– Other Global Programmes’ Materials– Civil Society CB and M&E– Systems Thinking and Complexity

literature and M&E examples

Summary: why ST has potential for M&E of CB in Climate Change

Summary: why ST has potential for M&E of CB in Climate Change

• Avoids pitfalls of logical framework in what is an amorphous field: CB

• Emphasises clarity of objective-setting, AND of learning collectively from reality

• Generates, relates to, and values ‘stories’• Potential to enhance team-work and inter-

relationships of hitherto disparate institutions and groups in their ‘ecosystem’ context;

• Climate change context – politically-charged, formal and informal, amenable to negotiation

• Evidence indicates ST M&E approaches ‘can work for and strengthen social change’ (Guijt 2007 IDS)

• Avoids pitfalls of logical framework in what is an amorphous field: CB

• Emphasises clarity of objective-setting, AND of learning collectively from reality

• Generates, relates to, and values ‘stories’• Potential to enhance team-work and inter-

relationships of hitherto disparate institutions and groups in their ‘ecosystem’ context;

• Climate change context – politically-charged, formal and informal, amenable to negotiation

• Evidence indicates ST M&E approaches ‘can work for and strengthen social change’ (Guijt 2007 IDS)