(1)

The democratisation of evaluation

David Gough

Democratisation of Research Methods, ESRC Methods Festival, St Catherine’s College, Oxford, 10th July 2014

EPPI-CentreSocial Science Research UnitInstitute of EducationUniversity of London18 Woburn SquareLondon WC1H 0NR

Tel +44 (0)20 7612 6397Fax +44 (0)20 7612 6400Email eppi@ioe.ac.ukWeb eppi.ioe.ac.uk/

The EPPI-Centre is part of the Social Science Research Unit at the Institute of Education, University of London

(2)

What is democracy?

A system of government by the whole population or all the eligible members of a state, typically through elected representatives Oxford Dictionaries

A form of government in which all eligible citizens participate equally—either directly or indirectly through elected representatives—in the proposal, development, and creation of laws. It encompasses social, religious, cultural, ethnic and racial equality, justice, and liberty. The term originates from the Greek δημοκρατία (dēmokratía) “ rule of the people“ Wikipedia

(3)

Democracy and evaluation1. The evaluation of democracy – not

addressed by this presentation2. Being democratic about how you evaluate:

i. To be more productive (instrumental)ii. To enable those affected to be involved

3. Evaluation as a democratic activity / shared examination of values and data

4. Evaluation activity as a change agent

(4)

Issues• Perspectives:

• Ideology, social values, theory and priorities• Research questions and methods

• Participation• Engagement• Power and control

• Research questions and methods required or preferred by particular approaches to democratic evaluation (and the extent of implicit/explicit value specification )

(5)

Four types of DE (J.C Greene 2006)

1. Democratic evaluation: engaging with participants (rather than Bureaucratic by ministries or Autocratic by universities)

2. Deliberative DE: explicitly democratic / ensure that engage with all including weaker groups

3. Participative E: partners in the construction of knowledge / equity

4. Critical E: critical political analysis of macro structural issues

Social values?Research methods?

(6)

Example 1: Systematic review• PHSE students• Their review – we were technicians• We guessed that their choice would be

‘How to prevent and stop bulling’• It was in fact: …..

Participative E: Technical methods + user perspectives + user control

(7)

Example 2:NICE Guidance

• Topic for guidance from government• Consultation and stakeholder meeting• Stakeholder Guidance Advisory Group• Commission evidence and develop guidance• Explicit social values (SV) policy (e.g.cost

effectiveness/equity/rigour/transparency)DE + PE ++: Deliberative process + technical

methods + user power + built in SV perspectives + SV of process

(8)

Example 3: Participative process evaluationA. Cornwall (2014): IDS Working Paper 2014: 437To understand the dynamics of change in a

nutrition education programme in Kenya1. Stakeholder analysis: roles & relationships2. Expectations and experiences of the prog.3. Analysis by probing +ve and –ve

experiences with stakeholdersPE of process + exploratory + critique of ‘technocratic evaluation methods

(9)

Components of evaluationNot just questions and methods. Also:1. Perspectives2. Participation3. Power and control4. Other implicit or explicit valuesDemocracy in research involves all of these.Democratic values suggest that they should therefore all be explicit transparent methods and part of evaluation standards

(10)

WebsitesEPPI-Centre Website http://eppi.ioe.ac.ukEIPPEE Website: http://www.eippee.euEvidence and Policy: http://www.policypress.co.uk/journals_eap.asp

Twitter @EPPICentre@ProfDavidGough @EIPPEEnet

Emaild.gough@ioe.ac.uk

Thank you for your attention

EPPI-CentreSocial Science Research UnitInstitute of EducationUniversity of London18 Woburn SquareLondon WC1H 0NR

Tel +44 (0)20 7612 6397Fax +44 (0)20 7612 6400Email eppi@ioe.ac.ukWeb eppi.ioe.ac.uk/

The EPPI-Centre is part of the Social Science Research Unit at the Institute of Education, University of London

(11)

Gough D, Thomas J, Oliver S (2012) Clarifying differences between review designs and methods. Systematic Reviews Journal.

http://www.systematicreviewsjournal.com

An introduction to systematic reviews: Sage Publications Ltd

Gough D, (2013) Meta-narrative and realist reviews: guidance, rules, publication standards and quality appraisal. BMC Medicine, 11:22

http://www.biomedcentral.com/1741-7015/11/22

Gough D, Oliver S, Thomas J (2013) Learning from Research: Systematic Reviews for Informing Policy Decisions: A Quick Guide. London: Alliance for Useful Evidence., Nesta.http://www.alliance4usefulevidence.org/assets/Alliance-FUE-reviews-booklet-3.pdf