The Most Significant The Most Significant Change Technique (MSC) Change Technique (MSC)

Dr Jessica DartDr Jessica DartClear HorizonClear Horizon

MSCMSC

– Form of qualitative, participatory M&E– Based on ‘stories’ of significant change– Developed by Davies 1996 - Bangladesh – Now used in numerous development

programs and in the public sector

QualitativeQualitative vs quantitative monitoringvs quantitative monitoringQuantitative

• Focus on measurement• Closed questions • About ‘proving’• Easy to aggregate• Deductive• Static• Goal displacement can be

a problem

Qualitative

• Focus on questioning• Open questions • About learning• Hard to aggregate• Inductive• Dynamic• Goal displacement is not

an issue

• Goal displacement • Creaming• Not about learning• Don’t tell you what you don’t know you

need to know

Limitations of indicator Limitations of indicator based monitoringbased monitoring

Qualitative monitoringQualitative monitoring

• Can be used in conjunction with conventional output monitoring

• Is usually more aimed at learning than accountability

Why stories?Why stories?• People tell stories naturally - indigenous• Stories can deal with complexity and

context• People remember stories• Stories can carry hard messages

/undiscussables• But stories not known for accuracy/truth

Use of stories in MSCUse of stories in MSC• Collection of stories + systematic, collective

interpretation = storytelling can be effectively harnessed for participatory evaluation

• Because interpretations tell another story & process has beneficial outcomes for evaluation utilisation

Overview of MSCOverview of MSC

1. Determine sorts of change to monitor2. Collect stories3. Review & filter stories regularly4. Collate ‘selected’ stories for funders review 5. Monitor the process and verify the stories

Overview of MSCOverview of MSC

1. Determine sorts of change to monitor2. Collect stories3. Review & filter stories regularly4. Collate ‘selected’ stories for funders review 5. Monitor the process and verify the stories

ExampleExample

• Target 10 Dairy Extension Project• Four regions in Victoria, 50 staff• 1999-2000 trail of the approach• Still continues today

Step 1- Selection of ‘domains’ of Step 1- Selection of ‘domains’ of changechange

• 3 broad ‘domains’ of changes to be monitored at the project level:

• Changes in on-farm practice • Changes in farmer-decision making skills• Changes in profitability • Any other type of change

• * Not precisely defined

Step 2 - Collect storiesStep 2 - Collect stories• “During the last month, in your opinion, what do you think was the most significant change that took place as a result of the project?”

•The respondent (farmer, extension worker or industry rep) answers in 2 parts:

• 1) descriptive 2) explanatory

Step 3 –Step 3 –Review & filtering processReview & filtering process

• The stories were reviewed by:– The regional committees (every 2-3 months)– Statewide Executive (every 2-3 months)

• The stories are reviewed using a facilitated process at the state and funder levels

Funder meeting

State meetings

Region 1 Region 2 Region 3 Region 4

Story tellers

feedbackflow ofstories

Step 4 - Collate & review Step 4 - Collate & review selected storiesselected stories

1. In total 134 stories were collected - 80% from extension staff

2. A booklet containing 24 selected stories accompanied by the selection criteria & comments

3. Purchasers provide feedback to project after reviewing booklet

Step 5 – Step 5 – Monitor process & verify storiesMonitor process & verify stories

• A database was developed to keep track of all stories

• Secondary analysis at end of reporting period

• In this case selected stories were not verified

• Storytellers were asked to check final stories in report

Impact of MSCImpact of MSC• Staff gained more fully shared vision• Process boosted their morale • Process saw farmers, staff, collaborators

sitting together and interpreting qualitative data & casting evaluative judgements

• Project committees became better at conceptualising impact

Use of storiesUse of storiesIn addition to reporting, stories were used:

– To improve planning– To help explain a point to a farmer– To recruit new participants – To help explain a point to another member

of staff– For PR

MSCMSC vs quantitative monitoringvs quantitative monitoringMSC• Focus on measurement• Closed questions • Project out• About ‘proving’• Deductive• Static• Inclusive• Central tendencies

Qualitative• Focus on questioning• Open questions • Context in• About learning• Inductive• Dynamic• Selective• Outer edges of experience

Goals

Goal-based evaluation

change

change

change

change

change

change

From the viewpoint

of the Participants

Extent to which they were achieved

From the view point of the program staff + consultation

Program out Context in

From the viewpoint of the Participants

Program

Purpose of MSC in M&E Purpose of MSC in M&E Primary purpose to facilitate improvement by:

focusing direction of work towards explicitly valued directions

eg. what do we really want to achieve and how will we produce more of it?

Contributes to summative evaluation: Information about unexpected outcomes Performance information concerning very best

success stories Can inform criteria used to judge projects

MSCMSC• Creates space for stakeholders to reflect, to make sense of

complex changes

• Provides dialogue to help make sense of each other’s values

• Facilitates dynamic dialogue ie. “what do we really want to achieve and how will we produce more of it?”

• Excellent for participatory programs with diverse, complex outcomes, & multiple stakeholders