1

Welcome to RELM!

Agenda: Announcements MSC Presentation Controlled Chaos Networking

2

Announcements

The RELM HAPPY HOUR! - October 17th - GUSTOs? TITLE: Problematizing the Nonsensical Lexicography of

the M&E FieldOR I don’t think that word means what you think it means…

Keep an eye out for a short survey that will ask you to name Some piece of M&E / Development Jargon What people really mean by it

MOST SIGNIFICANT CHANGE (MSC)

Presentation for the RELM working group

25 September 2014

Kampala, Uganda

4

Outline

Background and basics of the method Getting MSC monitoring/evaluation off the ground Recommendations for implementing MSC Practical examples/lessons learned

5

What is MSC?

The most significant change (MSC) technique is a form of participatory monitoring and/or evaluation.

Participatory – It is participatory because many programme stakeholders  are involved both in deciding the type of changes to be recorded and in analysing the data.

Monitoring – It is a form of monitoring because it occurs throughout the programme cycle and provides information to help people manage the programme.

Evaluation – It contributes to evaluation because it provides examples and explanations of programme effectiveness that can be used to help assess the performance of the programme as a whole.

6

MSC It’s In The Name

Many times, names communicate the obvious and/or hidden nature of something, a process, an item, or programme. In the same way, the words used to name this approach give us a ‘peek’ into its nature and requirements; very simply: the must dos! Most …makes someone think about words or phrases like highest,

implies a process of identifying the highest; maybe ranking, comparison, choice, and reason for any choice made.

Significant

…communicates that something is important to someone.

Change …implies the existence of a “before” state and an “after” state for a person or situation. Many times it also communicates a theory (i.e. when…then…) of cause and effect. A shift from one situation to another, as a result of an experience, event, or action. Changes can be positive or negative.

7

Components of the MSC process

The MSC process involves a few simple components Collecting stories of significant changes from programme

participants Sharing and analysis of stories among implementation

teams Selection of significant stories Sharing stories and feedback from the process

….With a few intermittent steps between

8 Ten steps to implementing MSC

Step 1: How to start and raise interest

Pilot MSC on a smaller scale Identify people who can be involved in the design

and implementation process Invite opinions and discussion through internal focus

groups

Steps 2 & 3:Define domains of change & reporting periods

Broad categories under which similar stories are grouped.

Can be predetermined domains of change.

Participatory process to determine domains

3-5 recommended *Useful to have an open domain*

Most often the monitoring part of M & E: quarterly or biannually

Immediately for evaluations

Domains of change Reporting periods

Step 4: Collecting SC stories

Key questions: Time frame: ‘looking back over the past month…’ Personal Judgment: ‘what do you think was…’ Selectivity: ‘the most significant…’ Change: ‘in the quality of people’s lives’ or ‘in this community’

*Quality of stories; they make or break of the MSC process*

12

Steps 5 & 6: Selection of stories and feedback

Step 1: Collecting stories from target group Stories collected from implementation areas

 

Step 2: First level story analysis/selection Based on agreed-upon and documented criteria, stories are grouped in domains of change and best story in each domain is selected. Stories not selected are stored for analysis

Step 3: Second level analysis/selectionStories selected by first tier are subjected to second phase of analysis to identify THE most significant change story.

MSC

Story

Reason for selection and non selection shared between tiers.

Communities informed about selection process and selected MSC

STEP 7: Verification

Helps to avoid the following: Fictional accounts Descriptions of real events based

on misunderstanding The exaggeration of significant

events Clarifies unclear story details (who,

when, what, story interpretations etc.)

Should verify when: ….a field worker documents a

reported change …recommending change in

the organizations’

Verification is optional when: When beneficiaries are

represented in selection process

Person performing the selection process has the confidence of other stakeholders.

Steps 8: Quantification

Ways qualitative information can be analyzed:1. Quantifying factors within individual stories

• How many people were involved• Number of activities taken place• Types of activities

2. Verification of novel significant change (associated with step 6: feedback)• One-time only query

3. Secondary analysis (or step 9)• Examine full set of collected SC stories• Count the number of times a specific type of change

happens

Step 9: Secondary analysis & meta-monitoringMeta-monitoring is performed continually or periodically and focuses more on attributes of stories, i.e., origin & fate, who identified/ selected SC stories .

*useful for staff, board of trustees and donors

Monitoring of 4 measures:

1. Number of stories written in each reporting period and respective changes over time

2. Demography of SC story authors- gender, age, ethnicity, etc.

3. Whose stories are selected across above variables

4. What was the outcome of those SC stories?• Generated recommendations• How many recommendation were acted upon• Were discovered variables analyzed.

Step 10: Revising the system

It is normal to change the implementation of MSC during and after the introductory phase. It is part of the learning curve.

Most common changes are: Change in domain names to more inconspicuous terms, i.e.,

negative change to lessons learned. Change in frequency of reporting, i.e., monthly to quarterly Change in type of participants, i.e. from only beneficiaries to

include staff Change in structure of meetings for selection processes

17

Recap of 10 steps

Standard steps with guided flexibility. The 10 steps make it MSC: anything less is change or

success stories Steps build into each other Steps are not extremely distint: some happen at the

same time: E.g. Selection and Feed back

18

MSC in a Monitoring, Evaluation, and Learning Setting

Learning•  Can influence the population of values in an organization• Creates horizontal, vertical, and inclusive dialogue• Fosters shared learning and capacity building• Celebrate success

Monitoring• Captures unexpected outcomes• Uses inductive approach• Provides context and allows focus to change

Evaluation • Provides the why• Identifies and aggregates stakeholder needs• Identifies relationships between program activities and changes

Issues of Validity, Bias, and Voice in MSC MSC and validity:

Systematic process of selection

Transparency Verification Participation

MSC and Bias Success Subjectivity Popular Views Story Tellers

Whose voice counts? Make sure review panels

are representative and diverse

Combine with other techniques to get the views of limited participants and critics

Those at the top must choose from options selected by the bottom

20

Best Method Depends on the Questions

What planned and unplanned changes is our

programme contributing

to?

 

Did we implement our

programme efficiently and

achieve maximum Value for

Money (VfM)? 

How did our programme

contribute to changes?

 

When and When Not to Use MSC

USE MSC There are complex, diverse and

emergent outcomes There are large and numerous

organizational layers An organization is focused on

social change An org is participatory in nature There will be repeated interaction

with staff and beneficiaries

An org is looking to capture expected change

Solely for PR Conduct a retrospective eval

for a completed program Understand average

experience of participants Complete a quick and cheap

eval

USE MSC Not for you if…..

22

MSC as a monitoring technique Accountability Can Transform Health Project (ACT

Health) Bugiri District.

Most Significant Change: GOAL Uganda

23

ACT Health Theory of Change

changes within society

(empowerment of individuals)

changes within state

(inclusive and responsive institutions)

changes in the state -

society interface

Responsibility

Responsiveness

Relationships

24

MSC in ACT Health project

Step 1: Collecting StoriesOfficers collect significant stories from target communities: minimum of two stories from each intervention HC every six months.

 

Step 3: 2nd Level Analysis/SelectionStories selected at level 1 are sent for 2nd Level Analysis/Selection. From 4-5 domains 4 stories are selected (max 10). This level reviews the stories and selects the one (1) MSC

Step 2: 1st Level Analysis/Selection Implementation teams bring together stories twice a year. Stories are organized into 4-5 domains of change and ½ stories are selected from each domain.

Applied selection criteriaCredibility of the story

Story presentation

Potential for ripple effects

Clear evidence of change

Sustainability of the change

25 Most Significant Change: Higa Ubeho, Rwanda

26

USAID Higa Ubeho

Multi-intervention livelihoods program in Rwanda

Funded by USAID, implemented in part by Global Communities, Partners for Good

MSC selected by USAID for the purposes of midterm evaluation

Evaluate health and nutrition outcomes

27

MSC in USAID Higa Ubeho Program

Step 1: Collecting StoriesWe conducted interviews in the field with program beneficiaries. We

carried out nearly 100 interviews to collect SC stories. We were working with Global Communities field staff to do this 

Step 3: 2nd Level Analysis/SelectionField staff re-sent stories to our team, we reviewed their process, then we sent a grouping of 10 for final review. After Final review stories were shared back to communities.

Step 2: 1st Level Analysis/Selection Interview team worked together to pare down 96 stories to just 30 for the second level of selection. Those 30 stories were packaged and sent to the field staff for selection.

Applied selection criteriaCredibility of the story

Story presentation

Carry the voice of others

Clear evidence of change

Sustainability of the change

28

Resources on MSC

1. The Most Significant Change (MSC) Technique: A guide to Its Use, Rick Davies and Jess Dart (Version 1.00 – April 2005) www.mande.co.uk/docs/MSCGuide.pdf

2. Introduction to qualitative lens: Karina Kiel Mann, Fabian Cataldo and Janet Seeley (Funded by DFID and Evidence for Action)3. Target 10 Evaluation Stories 4. Quick-Start Guide: A self-help guide for implementing the Most Significant Change technique (MSC)

Jessica Dart & Rick Davies

29

Controlled Chaos!

Your Chance to: • Request assistance with specific and timely issues

facing an organization or study. • Volunteer to help and meet together with the

member requesting assistance during the tea break.