Focus docs: Crowdsourcing your evaluation

Steve Powell, UNEP brown-bag, Nairobi March 2015steve@pogol.net

What’s the problem?

• Greedy ToRs• … mean evaluation is the act of balancing

narratives• Evaluation matrix: way to go around the

houses• Recommendations out of thin air?

Evaluation matrixTeam Web

surveyDocument analysis

Youth Youth in villages

…

QA: relevance

QA1: stakeholder relevanceQA1x: stakeholder relevance of component x

x x

…

QM: what shall we do about ….

x x

…

x x

Around the houses

Eval

uatio

n qu

estio

ns

Evaluation

matrix

Ask q’s

Find other

evidence

Synthesise Compare, contrast opinions

Dream up

recommen

dationsGenerate

new

evidence

Evaluation

arithmetic1-5

judgments

Write report!

Focus Docs: the steps

• Preparation• Clarify Privacy• Face to face• Inititialise questions• Invite• Moderate• Summarise• Write

Evalcrowd discussion The evaluator

Results: strategy 1

Stakeholders

Results: strategy 2

Conclusions: strategy 1

Conclusions: strategy 2

“These are the mandatory questions for the report…”

post questions give answersask evaluator for new evidenceadd evidenceadd comments &”likes”

“.. But you can add your own”

posts some questions,moderates,challenges,adds new evidence, synthesises into final report

Using Focus Docs to crowdsource your evaluation

Some other advantages of the Focus Doc procedure

• Break down answers by type of respondent e.g. by gender• discussion between all respondents almost simultaneously• Respondents quote and/or link to additional evidence • allows respondents to flag up where evidence seems to be

missing, i.e. where the evaluator can help• frees up some evaluator time to carry out some more in-depth

investigations where really needed• allows the evaluator to focus on issues which are critical and/or

controversial for contributors and/or minority opinions• makes it easy to bring in subject experts • allows social network analysis of the responses

Disadvantages of Focus Docs?

• Low familiarity with web tools Security/Privacy

• Potential data loss – need for backups, etc ….• Fear of “on the record” contributions

Evaluator role (1): Focus on addressing hidden topics

• What themes are we not talking about?• … would we rather not talk about?• some themes, some groups?• Do some groups get left out? Why?• Rewards / sanctions for saying …

• (Focus Docs disadvantage = advantage!)

Evaluator role

2. The spirit and the letter3. Irreplaceable!4. Still face-to-face, but less5. Address bias6. Moderating the process and interaction7. Ensuring participation & equal contribution