Government Social Research Unit

Quality and purpose: issues and Quality and purpose: issues and approachesapproaches

Teresa Williams

Head, Government Social Research Unit

Research Methods Festival 2008

2 July 2008

www.gsr.gov.uk

Why does quality matter?

Quality

Credibility

Opinion

‘Evidence-based opinion’

‘Stakeholder opinion’

‘Person on the street’s

view’

‘Good qualitative research’

‘Good quantitative research’

‘Good RCT’

‘Good systematic review’

‘Trusted source’ view

Slow or inconclusive research

Adapted from J. A. Muir Gray Evidence-based healthcare, 1997, Bell and Bain Ltd, Glasgow.

www.gsr.gov.uk

Who does it matter to?

• Research community – badge of professionalism, common language, sense of identity

• Research funders – value for money, continued investment

• Research users – confidence in the results, belief that they are relevant

• Research respondents – ethical considerations, cooperation

www.gsr.gov.uk

Dimensions of quality….

• Rigour – scientific method

• Respect – ethical frameworks

• But also Relevance – Range of stakeholders– Different stages in policy/implementation

process– Tomorrow’s questions as well as today’s

www.gsr.gov.uk

Frameworks for quality

• Principles setting out broad expectations

• Specific standards (can be independently assessed) on what ‘best’ means in different circumstances

• Clear processes to ensure the best design is identified and executed

www.gsr.gov.ukwww.gsr.gov.uk

GSR Code: principles for quality and relevance in govt• What is it?

– Addendum to civil service code– 7 principles under 2 key themes (People and

Products– Specific guidance and resources

• What’s it for?– Cementing our identity– Communicating our standards– Assessing how well we’re doing

www.gsr.gov.ukwww.gsr.gov.uk

www.gsr.gov.uk

www.gsr.gov.uk

STANDARDS - Research ReviewsStandard Description

Level 1 Narrative or descriptive literature review with no specified purpose, summarising research without critical appraisal and no details about sources searched.

Level 2 Narrative or descriptive literature review based on what was available quickly, with no critical appraisal of research summarised and no details about sources searched.

Level 3 Narrative or descriptive literature review based on search of published data, with information on sources searched and some critical appraisal of primary studies.

Level 4 Systematic review or Rapid Evidence Assessment (i.e. interim systematic review). Clear and robust criteria for inclusion/exclusion or research: adheres to Campbell standard that outcome studies must be level 3 or above to include . Details of search terms and sources.

Level 5 International language systematic review. Clear and robust criteria for inclusion/exclusion: adheres to Campbell standard that outcome studies must be level 3 or above to include. Details of search terms and sources, including attrition at each stage of search.

www.gsr.gov.uk

STANDARDS: Survey Methods Standard Description

Level 5 Representative random sample of sufficient size to provide robust estimates for main measures of interest including most or all sub-groups. Good response rate (average or above for similar studies), well developed and tested data collection instruments and competent execution.

Level 4 Representative random sample of sufficient size to provide robust estimates for whole sample and some sub-groups or interaction effects. Minor weaknesses on other aspects such as question wording. Response rate around average for similar studies.

Level 3 Random sample size where response rate is too low to provide robust estimates for the whole sample for the main questions (less than 50%). Some weightings to improve representativeness. Weaknesses on question wording or other aspects.

Level 2 Quota (non-random) sample, in which quotas are set for numbers of individuals in each of the sub-classes. Representative on some key attributes (such as sex, age, or ethnicity) but cannot be fully representative. Or random sample obtaining a very low response rate (30% or less) or failure to take account of differential response rates (e.g. by weighting data).

Level 1 Not random sample. No effort to ensure representativeness, or, where complete, poor reporting of sampling strategy so unable to determine likely representativeness. Alternatively, no sampling frame used (i.e. the list of individuals in the population from which the sample was selected), or serious bias in the sampling frame or sampling procedure. Critical weaknesses on other aspects, e.g. the true level of response are unknown.

www.gsr.gov.uk

Processes• Main tools

– Publication: Inputs (research objectives, design protocols etc) as well as outputs (reports) to be in public domain for general scrutiny

– peer review: ‘independent’ and ‘expert’ scrutiny to help inform decisions

• Not a one-off event. Good QA should be applied throughout project life cycle– Project design– Procurement – Project Delivery – Publication – Post project review

www.gsr.gov.uk

Some key issues• Building in and weighting non-methods aspects of

quality?– Appropriateness of research questions– Ethical considerations– Research relevance/impact– Time/cost considerations

• Can we work towards minimum standards? • Who should (and indeed can) judge quality

– What/who counts as ‘independent’– Methodological vs subject area expertise– Role of practitioners/end users as well as research experts

• Quality assurance processes as tools for assisting decision-making rather than a bureaucratic process

www.gsr.gov.uk

Format for sessionPart 1: Methodological quality• Jane Lewis (qualitative methods)• Angela Dale (quantitative methods)Part 2: Quality from the User perspective• Sandra Nutley (how knowledge is used)• Juliet Mountford (capacity building in the

Third Sector)Discussion – led by Ceridwen Roberts