Gyte Ga, Grant-Pearce Cb, Henderson Sa, Horey Da, Oliver Sc and Sakala Ca

a Cochrane Pregnancy and Childbirth Group, University of Liverpool (UK); b PREST, University of Manchester; c SSRU, Institute of Education, University of London.

Does consumer refereeing improve the quality of systematic reviews of healthcare interventions?

Perspective of editors and authors

Acknowledgements: This research was supported by a ‘Discretionary Funding’ grant from the Cochrane Collaboration Steering Group

Objective: To determine how editors and review authors view consumer refereeing within the editorial process for preparing systematic reviews of effects of healthcare interventions; in particular, their assessment of the impact of consumer involvement on the quality of Cochrane reviews, and lessons for consumer involvement in healthcare research more generally. This information was sought to help plan a more extensive evaluation. Design: An independent researcher undertook semi-structured telephone interviews with editors, review authors, consumers, consumer co-ordinators and the co-ordinator of a Cochrane review group. The researcher examined routine editorial documentation and undertook mapping interviews to understand aims of involving consumers in research and the Cochrane Collaboration’s rationale for involving consumers as referees. A short questionnaire, asking for overall views of consumer input into the editorial process, identified review authors and consumers for telephone interview. This presentation reports results from interviews with five review authors selected to give diverse views, along with four editors and the Group’s co-ordinator. Consumer views are reported elsewhere. Interviews were transcribed, and the main issues, impressions and themes from each were summarised, with resulting data explored to identify themes.  Results: Key points identified were:

quality of consumer input was perceived to be positive; those with an overview of the review process considered that consumer input improved the final review; earlier consumer input may be beneficial

 Conclusions: This evaluation has identified key issues surrounding consumer refereeing of systematic reviews undertaken within the Cochrane Collaboration. Consumers were considered to provide important contributions, and suggestions for improvements in process were made. Further research is planned to assess more specifically what additional contribution consumers make, and whether objectively consumers improve the quality of Cochrane systematic reviews of healthcare interventions.

AbstractAbstract

Background:Background:

Cochrane Pregnancy and Childbirth Group involves consumers as referees in the production of systematic reviews of healthcare interventions to:

ensure the reviews are targeted at problems important to people

take into account outcomes important to people

ensure accessibility to people making decisions

adequately reflect variability in values, conditions & healthcare in different countries

The PCG Consumer Panel (PCGCP) refereeing process:

Panel includes part-time paid coordinator, 2 volunteer regional coordinators,

72 volunteer consumers

Consumers choose of the 60 protocols/reviews a year on which they wish to comment

Consumer comments are collated and summarised by the Consumer Panel Coordinator

Consumers, in coming from a different perspective, add an important and valued contributionConsumers, in coming from a different perspective, add an important and valued contribution

The next phase of the evaluation should proceed, seeking more objective measures of qualityThe next phase of the evaluation should proceed, seeking more objective measures of quality

Aims:Aims: To determine how editors and review authors view consumer refereeing within the editorial process for preparing systematic reviews of effects of healthcare interventions; in particular, their assessment of the impact of consumer involvement on the quality of Cochrane reviews.

This information was sought to help plan a more extensive evaluation.

Contact:: Gill Gyte at ggyte@cochrane.co.uk Acknowledgements: This research was supported by a ‘Discretionary Funding’ grant from the Cochrane Collaboration Steering Group

Design:Design:

2 mapping interviews were undertaken to understand aims of consumer involvement

A preliminary questionnaire was sent to all review authors and consumers with the group (around 300), to select for interview. 70 replied

5 authors and 5 consumers chosen to give a range of views. Consumer views are reported

elsewhere. 4 editors with the longest experience with the group were chosen

Interview questions were developed by the independent researcher based on an understanding of

the review group processes, the aims of involving consumers as referees and the aims of the project

Interviews were transcribed, and the main issues, impressions and themes from each were

summarised, with resulting data explored to identify overall themes

18 semi-structured one-hour telephone interviews were undertaken with 4 editors, 5 review authors, 3 consumer coordinators,

5 consumers and the review group coordinator

1.1. Consumer input was viewed positively, Consumer input was viewed positively, especially by especially by editorseditors

2.2. Consumers brought added value in the Consumers brought added value in the form of a different perspectiveform of a different perspective

3.3. Consumers provided suggestions for: Consumers provided suggestions for:

clarity of languageclarity of language

more meaningful outcomesmore meaningful outcomes

more comprehensive enquirymore comprehensive enquiry

“So it’s to do with clarity of language, but it’s also to do with the centrality

of, the importance of trying to put their views right at the centre, and

their perspective.”

“…it’s important that the Consumer Panel continues to teach us about the

importance of the persons, of the patients’ view, of adverse effects, and

of an accessible language”authorauthor

“I think the summary comments from our

consumer coordinator,

they are by far the biggest influence”

editoreditor

“I think I can speak for all [the] editors on this because it comes up at every single

one of our Editorial meetings, that the input in general is fantastic. It’s the highest quality we get. Its certainly the most detailed

and thoughtful”

“…a very important role for the

consumers in recommending

what issues need to be addressed in

future trials”

“…it was very long when it came. I think it was about 18 pages…it took me a while to get over the heart-sink. I mean it was

enormously valuable

editoreditor

“… they thought particular outcomes were important, and we hadn’t actually considered those

as important outcomes and in fact when we thought about it, they

were…..”

authorauthor

ResultsResults

ConclusionConclusion

authorauthor

authorauthor

“…it works well because we’ve got a … a dedicated

person, that can co-ordinate and work with consumers,

who understands what they are saying, at the same time understands what the review

needs.”

editoreditor

editoreditor