Slide 1

Walking the line between impartiality and participationSarah ButlerKate Nichols, Jess Dart, Mark BouletSome Green Steps wisdom...After requesting a some participation...."don't imagine you can just give the brief to the consultant and they will come back later with evaluation. You need to commit time and resources to this, otherwise the participatory approach is not as effective"The story today

The beginning...Academia...research, statistics....Monitoring and evaluation consultancyCore business has often involves participatory approachesStrength in qualitative in data collectionLearning and development and capacity building

My gems....LanguageUnderstandingShared data analysisMany brains make light work Using and making time for the recommendationsRealising evaluation is a starting point

Balancing participation with independent evaluation

Often asked to do participatory evaluationAt same time independence is neededOften for fundersHow do you balance the two?

Participation vs independenceParticipatoryIndependentValues and beliefs of program/stakeholdersJudgements made by those close to the programDecision making by those close to the programCan introduce bias in interpretation of dataLots of learning through participation in the process

Unbiased weight on different types of dataValues of the evaluator onlyUnbiased comparison can be made between programsRecommendations provided by evaluators independent of program participantsData collection and analysis reliant on the skills and attention of evaluatorLittle involvement or learning by program participantsTheoretical underpinningsAction research or action learning (e.g. Wadsworth 1993)

Theory continued...Utilisation focused evaluation (Patton 2002)

Theory continued...Empowerment evaluation (Wadersman et al. 2005)

ImprovementStakeholder ownershipINCLUSIONSHARED DECISION MAKINGSocial justiceCommunity knowledgeTheoretical underpinnings contAppreciative inquiry (Cooperrider & Whitney, 2005)

Independent (e.g. Scriven 1991, goal-free evaluation)Contrast to participatory approachesIndependent of managementBenefits to end users or beneficiariesEvaluator makes judgement based on evidenceOften suits funders

Case study: Green StepsHands-on education and leadership programEmpowerment of participantsCommunity feel and high level of participationThose who have participated have shaped the program

The situation10 years had gone byFunders were asking What is the point? How are you going to support this program?Need for independent data collection to see what had occurred and the impactExplicit request for participation in the process

The culture

The processThe processStep in the evaluationLevel of participation (low, medium or high)Who was involved e.g. who participated in the evaluation stepStep 1: PlanhighProgram staff, past participants, reference committee with consultants facilitatingStep 2: DiscoverlowConsultantsStep 3: Analyse preliminary data analysislowConsultantsStep 3: Analyse summit workshophighProgram staff, past participants, reference committee with consultants facilitatingStep 4: ReportlowConsultantsStep 5: Re-designhighProgram staff, reference committee with consultants facilitatingThe processStep in the evaluationLevel of participation (low, medium or high)Who was involved e.g. who participated in the evaluation stepStep 1: PlanhighProgram staff, past participants, reference committee with consultants facilitatingStep 2: DiscoverlowConsultantsStep 3: Analyse preliminary data analysislowConsultantsStep 3: Analyse summit workshophighProgram staff, past participants, reference committee with Consultants facilitatingStep 4: ReportlowConsultantsStep 5: Re-designhighProgram staff, reference committee with Consultants facilitatingSteps...Participatory planning

Independent data collection & analysisIn many ways this was also participatory.... We talked with Green Steps all the timeThey knew where we were up toBUTInternet survey spreadIn-depth semi-structured interviews depthDocument reviewAnalysis thematic and descriptive statsSteps...Summit workshop participatory data analysis

Steps...Re-design/recommendations workshop

What happenedUsed evaluation straight away to talk with fundersUsed it to engage with senior executivesUsed it to do new Green Steps activitiesDesign an M&E planIt directly informed their 2 year business planMany recommendations thrown away, modified or re-worded

As Mark said, It has given them good grounding to move on from

A little more Green Steps wisdomFor a training program, we are often asked to demonstrate our outcomes and have struggled to do so with more traditional evaluation approaches. Participatory evaluation and collecting stories of change was the most effective approach. AND we got some nice quantitative outcomes.

What we learnt - positivesYou say tomato I say tomato....Culture of participation is invaluable when its not there its really tough!Summit or analysing data together was also a critical point & kept the evaluation fun Bringing key executives into the re-design = evaluation directly informed business planning

It works!!

What we learnt"don't imagine you can just give the brief to the consultant and they will come back later with evaluation. You need to commit time and resources to this, otherwise the participatory approach is not as effectiveIt really isnt for everyone

Even with the most participatory team it is toughdedication and passion for the program is important

What do I think....

Your questions....

http://www.monash.edu.au/research/sustainability-institute/green-steps/