comments on cochrane reviews

COMMENTS ON COCHRANE REVIEWSApproaches to managing feedback

WORKSHOP

22 September 2013

Cochrane Colloquium, Quebec City

Introductions

We are:• John Hilton Editor, Cochrane Editorial Unit

• Gavin Stewart Associate Editor, Wiley

• Jo Garner Senior Business Analyst, Wiley

• Toby Lasserson Senior Editor, Cochrane Editorial UnitFeedback Editor, Airways Group + 19 other roles

Disclosures:

Management of comments is part our jobs as employees of our respective organisations but otherwise we have no actual or potential conflict of interest in relation to this presentation.

Who are you?

Introductions

We are:• John Hilton Editor, Cochrane Editorial Unit

• Gavin Stewart Associate Editor, Wiley

• Jo Garner Senior Business Analyst, Wiley

• Toby Lasserson Senior Editor, Cochrane Editorial UnitFeedback Editor, Airways Group + 19 other roles

Disclosures:

Management of comments is part our jobs as employees of our respective organisations but otherwise we have no actual or potential conflict of interest in relation to this presentation.

Who are you?

Introductions

We are:• John Hilton Editor, Cochrane Editorial Unit

• Gavin Stewart Associate Editor, Wiley

• Jo Garner Senior Business Analyst, Wiley

• Liz Dooley Managing Editor, ARI GroupME Support + 21 other roles

Disclosures:

Management of comments is part our jobs as employees of our respective organisations but otherwise we have no actual or potential conflict of interest in relation to this presentation.

Who are you?

Workshop• Part 1: What’s the latest?

• Overview, data, guidance, support• What do you need? What do you need to know?

• Part 2: Case studies• Some real-life examples• What would you do? And why? Any barriers, hurdles or brick walls?

• Part 3: The way ahead• Improving the process and the display• How can you help?

• Part 4: Is it worth it?• Isn’t everyone using Twitter these days?• What do you think?

Submitting comments

Submitting comments

Submitting comments

Submitting comments

Comments are also submitted via other channels:

• Direct contact to CRG

• From OVID

• Direct contact to Editor in Chief or CEU

• Summaries website (not at the moment)

Processing comments

Comment submission to Wiley &

CEU

Comment triage by

Wiley

Comment assessment by ME & FE

Comment management

What does CEU do?

• Comments go to Editor in Chief, and two editors• EiC looking if intervention required (with CRG)• Editors looking for cross-CRG issues (e.g. multiple submissions)• Support• Tracking

Processing comments

Comment submission to Wiley &

CEU

Comment triage by

Wiley

Comment assessment by ME & FE

Comment management

What gets filtered out:

• Comments or enquiries about the website functionality, access, display, etc

• Comments that are offensive, nonsensical, or inconsequential• Comments that arise from system errors or system testing• Duplicates

Processing comments

Comment submission to Wiley &

CEU

Comment triage by

Wiley

Comment assessment by ME & FE

Comment management

CRG-led process

• Managing Editor• Feedback Editor• Co-ordinating Editor(s)• Feedback contributor(s)

Processing comments

Comment submission to Wiley &

CEU

Comment triage by

Wiley

Comment assessment by ME & FE

Comment management

CRG-led process

• Managing Editor• Feedback Editor• Co-ordinating Editor(s)• Feedback contributor(s)• Author(s)• Editor(s)

Processing comments

Comment submission to Wiley &

CEU

Comment triage by

Wiley

Comment assessment by ME & FE

Comment management

CRG-led process

• Managing Editor• Feedback Editor• Co-ordinating Editor(s)• Feedback contributor(s)• Author(s)• Editor(s)• Editor-in-Chief• CEU Editor(s)• Steering Group, CEO• Interfering third parties

Acting on commentsQuestions, questions, questions

• Is it in English?• Is it coherent and not clearly wrong?• Is it about the content of the review?• Should it be published?• How much editing does it need?• Are the authors going to respond? And when?• If authors won’t or can’t respond, what then?• How long will the whole process take?• Is anyone in editorial base conflicted?• Has contributor declared conflicts?• Does it trigger an amendment?• Does it trigger an update?• Does it raise broader issues?

Acting on commentsQuestions, questions, questions

• Is it in English?• Is it coherent and not clearly wrong?• Is it about the content of the review?• Should it be published?• How much editing does it need?• Are the authors going to respond? And when?• If authors won’t or can’t respond, what then?• How long will the whole process take?• Is anyone in editorial base conflicted?• Has contributor declared conflicts?• Does it trigger an amendment?• Does it trigger an update?• Does it raise broader issues?

What have I missed?

Process

Judgement call?

Workflow?

Comment outcomes1. Validity

1. Valid: take action

2. Not valid: take no further action

2. Usefulness1. Translate

2. Edit

3. Drastically cut or ask for resubmission

3. Suitability for publication1. Publish

2. Don’t publish

4. Author response1. Not needed

2. Needed but not forthcoming

3. Received and published

5. Action on review1. No action needed

2. Review amendment

3. Review update

4. Changes at next updateC

om

mu

nic

atio

n

How many comments?

2011 Q1

2011 Q2

2011 Q3

2011 Q4

2012 Q1

2012 Q2

2012 Q3

2012 Q4

2013 Q1

2013 Q2

0

5

10

15

20

25

30

35

40

45

Which groups?• In the first half of 2013…

Which groups?• In the first half of 2013…

4 Anaesthesia Group

Heart Group

Pregnancy & Childbirth Group

Wounds Group

Which groups?• In the first half of 2013…

4 Anaesthesia Group

Heart Group

Pregnancy & Childbirth Group

Wounds Group

6 Airways Group

Which groups?• In the first half of 2013…

4 Anaesthesia Group

Heart Group

Pregnancy & Childbirth Group

Wounds Group

6 Airways Group

8 Acute Respiratory Infections Group

Which groups?• In the first half of 2013…

4 Anaesthesia Group

Heart Group

Pregnancy & Childbirth Group

Wounds

6 Airways Group

8 Acute Respiratory Infections Group

And 32 other CRGs received at least 1 comment

Characteristics of comments• Study inclusion/exclusion• Interpretation of findings• Conflict of interest• Scope• Methods• Typos, errors

Characteristics of comments• Study inclusion/exclusion• Interpretation of findings• Conflict of interest• Scope• Methods• Typos, errors

One sentence10 pages

Detailed, well-written referenced letterSemi-coherent scribble

Focused on single aspect of review Multiple issues, not all related to review

Characteristics of comments• Study inclusion/exclusion• Interpretation of findings• Conflict of interest• Scope• Methods• Typos, errors

One sentence10 pages

Detailed, well-written referenced letterSemi-coherent scribble

Focused on single aspect of review Multiple issues, not all related to review

• Retired academics• Cochranites• Students• Clinicians• Pharmaceutical companies or manufacturers• Persistent single-issue enthusiasts

Where do comments come from?

What about the unapproved comments?

Example 1

Review

Speed cameras for the prevention of road traffic injuries and deaths

ContributorMaller

Comment

When there is accident happened, who will take responsibility? After several day or long time, how to know the scene when it happened? Car video recorder could record accident and store video data with gps information on memory card,  people could get accident reconstruction, it is very easy. more view: http://www.cheapcarcamera.com

What about the unapproved comments?

Example 2

Review

Contributorwillywonka@gmail.com

Comment

What about the unapproved comments?

Example 3

Review

Probiotics for the prevention of Clostridium difficile‐associated diarrhea in adults and children

ContributorRachel Bock

Comment

Test comment - please delete

Known unknowns?• Who is submitting comments: students, clinicians,

Cochranites, pharma companies? (and why?)• How many comments are published, and how long after

submission?• How many reviews amended or updated as a result of

feedback?• What aspects are commentors focusing on? (study

inclusion, interpretation, outcomes, etc?)• Any more? What would it be useful to know?

• ANY VOLUNTEERS?

Cochrane Editorial & Publishing Policy Resource

http://www.cochrane.org/editorial-and-publishing-policy-resource

Cochrane Editorial & Publishing Policy Resource

http://www.cochrane.org/editorial-and-publishing-policy-resource

Cochrane Editorial & Publishing Policy Resource

http://www.cochrane.org/editorial-and-publishing-policy-resource

Cochrane Organisational Policy Manual

http://www.cochrane.org/organisational-policy-manual

Cochrane Handbook

http://handbook.cochrane.org

Cochrane Editorial Unit website

http://www.editorial-unit.cochrane.org/cochrane-library-feedback

So what’s in those new guidelines?• Guidance on how feedback should be managed• When to publish, when not to publish• Guidance on reasonable timescales• Roles of ME and FE• Meeting expectations of contributors• Where to get support• What to do with non-English comments

Your feedback will be very welcome.

Feedback editors: neglected?

• Communication to and between FEs has been lacking• Need Role description (purpose, expectations)• Induction package?• Page on Cochrane Training page?• Independence of role• Cochrane.org feature• Forum

So what else do you/we need?

Case studies• 1. • 2.• 3. • 4.

What’s in that CEU proposal?• Use the term ‘comments’• Decouple comments from review, but link to version• Display comments parallel to review version• Make comments citable (DOI)• Expectations about reasonable turnaround time• Transparency and tracking• Archie workflow for comments

So that was a while ago. What happened?....

Cochrane-Wiley Roadmap

• A set of work streams fulfilling agreed contractual projects

• Next 18 months• Feedback scheduled for Q2 2014• Project will involve:

• CEU, Wiley, Cochrane Tech team• CRGs• You?

Goals for feedback project

• Make process more efficient and transparent• Phased approach to first improve the user experience for:

• the Contributor• the Reader of feedback

• De-couple publishing feedback from the review or protocol• Make the individual comment citable in its own right

Requirements for user feedback

1. A Contributor is able to submit a comment2. The system shall save and acknowledge the comment3. Wiley Editorial may moderate the comment 4. The system shall trigger an editorial workflow that may result

in publication of the comment along with a response5. A Reader will be able to view comments and responses6. A Reader will be able to cite the comment7. A Search User will be able to search comments and

responses

Proposed approach – submission

Proposed approach – submission

Proposed approach – display

Improve processing

• A system for moderation• Separate the publishing of feedback from the review, retain a

link between them:• Feedback as a new Content type?• Changes to Archie, through XDPS and to Online Library

• Add the ability to cite individual comments & author responses• Improve the robustness of the feedback-response loop

• Feedback items having their own workflow?

Why all the bother?

• So we can “…amend reviews in the light of new evidence…to reflect the emergence of new data, valid feedback, solicited or unsolicited, from whatever source”

• Chalmers I, Haynes B. Reporting, updating, and correcting systematic reviews of the effects of health care. BMJ 1994;209:862-5.

But where are all the comments?

Not just Cochrane

http://jasonpriem.org/2011/01/has-journal-article-commenting-failed/

It can work…

Even then, will authors engage?Peter Gøtzsche, Andreas Lundh and BMJ editors looked at BMJ rapid responses

Even when comments could “invalidate research or reduce reliability” over half the time authors couldn’t be bothered to respond.

http://www.bmj.com/content/341/bmj.c3926

Are they using the ‘shadow system’ - scribbled marginalia, chats in labs, peer review, discussions at conferences, journal clubs

And Twitter, of course.

So, why no comments?“Why is it that comment forums provided by online scholarly journals for post-publication peer review are so consistently underutilized, even though online forums concerning countless subjects, from culture to politics, are booming?”

• Technophobia? Time?• Worry about appearing to be ‘unacademic’• “Commenting on scientific articles does not advance my career”• Prefer a published letter (despite delays)• Disincentives: fear of upsetting, being wrong, giving away ideas

Is the question not whether have comments failed, but whether are they succeeding somewhere else?

What’s different about Cochrane?• Updating! Comments are part of a continuous process of

improvement and updating• Culture of acceptance of criticism• Expectations of authors

So what could we or you do?

• Proactive solicitation• More encouragement on website – expose comments,

prompt for comments• Altmetrics• Activate specific social networks and establish rapport