Searching for outcomes in rural Tanzania: Harvesting directly from those influenced

Implications for evaluating behavioural change initiatives with communities

John Mauremootoo, Richard Smith, Dunstan KishekyaPresentation given at the American Evaluation Association on 12th November 2015 in the session entitled: The strengths and challenges of Outcome Harvesting for Evaluating in Complex Situations. Experiences from around the World.

The Project

A vehicle for tackling polarizing

social issues

The ProjectThe Team Tanzania (The Team) one of 15 completed or ongoing projects implemented in Africa and Asia by Search for Common Ground (SFCG) using The Team concept – The Team is a vehicle for tackling polarizing societal issues by stimulating learning in a persuasive, but non-confrontational manner.

For Tanzania the issue chosen was gender equality

Using an ‘edutainment’ approach (TV and radio soap operas) the project planned to reach and influence a wide audience.

Goal To contribute to strengthening the implementation and enforcement of gender-sensitive legislation in Tanzania by influencing changes in gender attitudes and behaviour of the general public

Priority gender equality themes targeted in the 18 month project

1. Inheritance & women’s consideration in inheritance issues

2. Women’s leadership

3. Gender-based violence

4. Retention of girls in secondary school

5. Rape

The Project Theory of Change

Reach

Response

Resonance

Numbers who watch and listen to the show

The attitude and knowledge changes at the individual and institutional levels

The actions that have been triggered by the programmes

Evaluation Method

Identification, description and interpretation of outcomes through Outcome Harvesting (using an OM-inspired definition of Outcome as changes in actions, relationships, policies or practices of one or more social actors influenced by the intervention).

With especial emphasis on harvesting outcomes of social actors the project had been seeking to influence by mobile cinema screenings and follow up discussions.

Evaluation challengesThe “Blank Slate of:• Poor understanding of outcome concept among project implementation team• Little or no knowledge of outcomes among the project implementation team• No documentation of outcomes in reports

Short duration of assignment given the fact that all outcome data had to be collected from scratch

Data collection via focus group discussion

Focus Group Discussion Guiding questions explained issues such as:• The definition of

outcomes• The definition of

contribution• How to determine how

importance of The Team’s contribution• Sources of

substantiation• Similar outcomes• Negative outcomes

Data collection via focus group discussion

“Treatment” “Control”

Harvesting outcomes through focus group discussions with social actors the project had been seeking to influence by mobile cinema screenings and follow up discussions

Harvesting outcomes from ‘control groups’ – target group representatives who had not participated in the mobile screenings and discussions

Follow up

Follow-up work to finalise and substantiate results descriptions

Results: The Team Contribution54 outcomes and 10 “proto-outcomes”

Series1

0 5 10 15 20 25 30 35 40 45

Useful (25)

Very important (25)

Important (14)

percentage

Strengthened and / or brought forward changes that may have happened to some extent anyway

Either essential to the change or greatly accelerated it

Strengthened and / or brought forward changes that may have happened to some extent anyway

Helped realise a change that may not otherwise have happened or would have happened very much more slowly

Results: Outcomes directly related to priority issues

Inheritance & women’s consideration in inheritance issues

Results relating to shared ownership of goods and property

Women’s leadership

Women taking more responsibility in work, home and in political life

Gender-based violence

Men stopping beating their wives and others encouraging men to stop beating their wives

Retention of girls in secondary school

Parents prioritising girls education and others encouraging parents to prioritise girls education

Rape

No results provided

Results: Other Outcomes

Financial implications of gender equality Participatory decision making

Implications for evaluating behavioural change initiatives with communities

1. Outcome Mapping Concepts

OM concepts of outcomes as behavioural change and contribution and the differences between knowledge, attitude and behaviour were easy and intuitive to grasp for the participating communities.

2. Knowledge & Attitude Changes

Claimed attitude changes even without behavioural change are important and can be readily assessed using OH

These “proto-outcomes” helped:• To understand the claimed process of change even where there have been no

tangible changes in behaviour• To compare the quality and process of change between “treatment” and “control”• Inform the change agent

3. Utility of Control Groups

In situations where there is a well-defined theory of change the control group provides a counter-factual

4. Focus Group Format

The focus group format is effective for discussing less sensitive topics but may not be the best way of evaluating results concerning sensitive issues and totally inappropriate for the most sensitive such as rape

5. The value of Follow up

The importance of an iterative follow-up process to enrich the result description and enhance its credibility.

6. The Timing of the Evaluation

a) It was surprising there were outcomes from a very ambitious project when its 18 months had not yet finished,

b) It was a snap shot & provided credible evidence of changec) Follow up was needed, e.g. after say 2 yearsd) It ideally should really have been a formative evaluation.

Now What?

If we consider the evaluation as an intervention then both its findings and the process need to be built on to help maximise positive change.

This was not formally done for a variety of reasons which in our opinion was a lost opportunity as is often the case for evaluations.

• Ansila Marandu (evaluation assistant)• Paul Glick (Search for Common Ground)• Cornelia Wamba (Search for Common Ground)• Stella Msami (Search for Common Ground)• VanessaCorlazzoli (Search for Common Ground)• The Mvomero Organizations Coalition• Women and Girls Fight illiteracy and Poverty

Organisation Save the Children Tanzania• Kate Dyer (AcT and KPMG)• Layla Ghaid (AcT and KPMG)

Acknowledgements

ImagesScreenshots from The Team Tanzania on YouTube

ImagesStock Images from Shutterstock © All rights reserved

ImagesPublic domain

All Other ImagesJohn Mauremootoo (Creative Commons Attribution: CC-BY)