2268 cochrane structure and function reviews - external … · 2015-10-02 · cochrane structure...

www.technopolis-group.com

30 August 2015

Cochrane Structure and Function Reviews ‐ External Consultation A report to Cochrane

Cochrane Structure and Function Reviews ‐ External Consultation

technopolis |group|, August 2015

Contact name:

Dr Peter Varnai

3 Pavilion Buildings

Brighton BN1 1EE

T: +44 1273 204320

E: [email protected]

E: [email protected]

Cochrane Structure and Function Reviews ‐ External Consultation i

Table of Contents 1. Introduction 4!

2. Methodology and limitations 4!2.1 Global survey 4!2.2 In-depth interviews 5!

3. Profiles of survey respondents 5!

4. Cochrane evidence 6!4.1 Relevance 6!4.2 Timeliness 8!

5. Priority setting practices 8!

6. Role of the consumer 10!

7. Cochrane methodology 11!

8. Accessibility and usability 13!8.1 Translation of Cochrane evidence 14!

9. Advocacy 15!

10. Organisational structure 17!10.1 Partnerships 18!10.2 Membership organisation 19!10.3 Structure and governance 20!

11. Training initiatives 21!

12. Concluding remarks and recommendations 22!Appendix A External stakeholder survey results 25!Appendix B External stakeholder consultation questionnaire 49!

Table of Figures Figure 1 Relevance and timeliness of Cochrane evidence (n=374-379) ......................... 6!Figure 2 Priority setting and external consultation (n=364 – 375) ................................ 9!Figure 3 Involvement of patients, patient advocates and healthcare consumers in the production of Cochrane evidence and in Cochrane activities (n=371 – 374) ................. 11!Figure 4 Cochrane’s methods (n=375; n=283-285) ..................................................... 12!

ii Cochrane Structure and Function Reviews ‐ External Consultation

Figure 5 Please rate the importance of the following factors relating to accessibility and usability of Cochrane evidence. (n=361 – 363) ....................................................... 13!Figure 6 Language-related questions (n=359 – 362) .................................................... 15!Figure 7 Based on your experience in your region or in your field of speciality, please rate the effectiveness of the following activities of Cochrane (n=344 – 346) ............... 16!Figure 8 Awareness and views on Cochrane’s local presence (n=335 – 337, 223) ...... 18!Figure 9 Please rate the extent to which the following features and benefits of a new membership scheme would be of interest to you. (n=324 – 329) ................................ 20!Figure 10 Based on your experience, please indicate the extent to which you agree with the following statements (n=328 - 331) ................................................................. 21!Figure A11 What country do you live in? (n=377) ........................................................ 25!Figure A12 What is your native language? (n=372) ..................................................... 26!Figure A13 Which of the below best describes your current organisation/employer? (n=378) ........................................................................................................................... 26!Figure A14 Which of the below best describes your current position? (n=373) ........... 27!Figure A15 Are you involved in or aware of Cochrane’s activities? (n = 379)* ............. 27!Figure A16 Relevance and timeliness of Cochrane evidence (n = 374-379) ................ 28!Figure A17 Priority setting and external consultation (n = 364 – 375) ....................... 30!Figure A18 Involvement of patients, patient advocates and healthcare consumers in the production of Cochrane evidence and in Cochrane activities (n = 371 – 374) ........ 31!Figure A19 Cochrane’s methods (n = 375; n = 283-285) ............................................. 32!Figure A20 What might attract the best methodologists to join Cochrane’s evidence synthesis instead of other applied research areas? ....................................................... 33!Figure A21 Accessibility and useability of Cochrane evidence (n = 363; n = 358) ...... 34!Figure A22 Please rate the importance of the following factors relating to accessibility and usability of Cochrane evidence. (n = 361 – 363) .................................................... 35!Figure A23 Language-related questions (n = 359 – 362) ............................................ 36!Figure A24 Please rate the importance of translating the following sections of Cochrane Reviews for you. (n = 349 – 355) ................................................................... 37!Figure A25 In your view, which top 3 languages should Cochrane prioritise? (n = 313) ......................................................................................................................................... 37!Figure A26 Based on your experience in your region or in your field of speciality, please rate the effectiveness of the following activities of Cochrane (n = 344 – 346) .. 38!Figure A27 Based on your experience, please rate Cochrane’s influence in promoting evidence-informed decision making for the following stakeholders (n = 340 – 344) . 39!Figure A28 Sources of high-quality systematic reviews in support of decision–making needs in health ............................................................................................................... 40!Figure A29 Awareness and views on Cochrane’s local presence (n = 335 – 337, 223) 42!Figure A30 Potential partner organisations for Cochrane, by country ........................ 44!Figure A31 Please rate the extent to which the following features and benefits of a new membership scheme would be of interest to you. (n = 324 – 329) .............................. 45!Figure A32 Based on your experience, please indicate the extent to which you agree with the following statements (n = 328 – 331) .............................................................. 46!

Cochrane Structure and Function Reviews ‐ External Consultation 1

Acknowledgements The project team would like to thank the many stakeholders who contributed their knowledge and views to this study, by responding to our survey, participating in interviews, and replying to our e-mailed questions. We are grateful to Chris Champion (Cochrane) for his assistance throughout the study. The project team included Peter Varnai, Maike Rentel, and several colleagues at Technopolis who helped prepare the multi-lingual survey.

2 Cochrane Structure and Function Reviews ‐ External Consultation

Executive Summary Cochrane, an independent network of volunteers interested in health and evidence-based medicine, is in the process of reviewing its organisational structure and ways of working as part of its new Strategy to 2020. A global online survey and in-depth interviews have been carried out on behalf of Cochrane by Technopolis Group in 2015 to collect data independently from external stakeholders and support the review process. This report includes relevant information from all stakeholders to help to reflect on their needs and inform Cochrane about ways to improve current processes.

Stakeholders expressed an overwhelming appreciation for what Cochrane stands for and the evidence it produces. They considered the majority of Cochrane reviews relevant and timely, helping decision makers and practitioners in their daily work. It will however be important to continue to improve on achievements and respond to the changing needs of the user community.

There is an increasing requirement to consider not only RCT data but also other clinical and contextual information such as health economics and real world data from different healthcare settings. Although the rigorous methodology adopted by Cochrane is still considered the ‘gold standard’, it needs to continuously innovate to stay cutting edge.

A very large proportion of survey respondents indicated that Cochrane evidence is easily accessible and usable which goes to show the results of the many years of effort Cochrane has invested in making its evidence user-centred. Nevertheless, shorter structured reviews with narrative synthesis, graphical content, contextual information and emphasis on the interpretation of the evidence (including metrics used by practitioners) would further enhance the usability of Cochrane evidence.

Cochrane’s recent efforts to develop and publish its first Priority Review List across all areas was welcomed by interviewees, however its existence was not widely known in the community. Priority setting should not only be an internal exercise for Cochrane Review Groups but involve diverse stakeholders’ input (i.e., consumers, clinicians, funders, and guideline developers) on their needs, objective data on burden of disease and healthcare, and identified gaps in published evidence. Encouragingly, there was a significant interest from survey respondents and interviewees to engage with Cochrane on this important task.

Setting up partnerships with other organisations across the stakeholder spectrum would therefore help inform priority setting processes, eliminate potential duplication of efforts, contribute to wider dissemination of the Cochrane evidence and ultimately serve as an important channel for advocacy. Although advocating for evidence has only recently been part of Cochrane’s strategy, it is already recognised by stakeholders that Cochrane promotes evidence-based medicine effectively (in certain countries) to research funders, healthcare policy makers, and clinical practitioners. Cochrane, however, needs to increase its advocacy efforts and learn from other organisations that campaign more effectively for integrity and transparency of clinical research.

Cochrane’s planned transition to a (non-fee paying) membership-based organisation was considered to be a good choice, with the potential to enhance engagement of diverse stakeholders with Cochrane, open up new dissemination channels, and strengthen Cochrane’s support base. Highest rated features of a membership scheme were the potential for online training, learning about evidence-based medicine and access to Cochrane’s software and tools for dissemination and advocacy. Cochrane’s potential training initiatives on methodology drew considerable interest, however, it may prove more resource-efficient to target users of the evidence, including policy makers, thereby essentially advocating evidence-based medicine and influencing health policy at the highest levels.

Cochrane is a very large network of volunteers with relatively complex organisational structure, with centres and branches, fields and networks, methods and review groups all part of the Cochrane system. Although the structure has proved somewhat fuzzy for most respondents and interviewees, the two major dimensions, geographical (and hence socio-cultural) regions and disease area/ clinical specialisation, appear to work reasonably well. About 70% of survey respondents were aware of a local Cochrane Centre and close to two


thirds of these felt that it allows them to engage with Cochrane. However, one of the biggest challenges for Cochrane is to reduce variability of approaches and processes across its review groups and centres and make those more efficient to produce consistently relevant, high quality and timely systematic reviews. This should not mean to impose uniformity on the entire network but to introduce unity and consistent identity toward the outside world.

Finally, a central feature of Cochrane was considered to be independence from any conflict of interest and thus a trusted source of impartial medical evidence for the community. In this respect, Cochrane has been devoutly stressing independence from the pharmaceutical industry but it would also need to find ways to screen for intellectual conflicts of interest among its contributors worldwide.

The following key recommendations have been made by the study team, based on the data collected during the study:

1. Review the skills, strengths and weaknesses (and its distribution) of current Cochrane methodologists and contributors to better prepare for users’ needs.

2. Accelerate adoption of new methods and/or recruit and reward innovative methodologists.

3. The process of priority setting has to be transparent and must target research questions where there is high potential to change practice.

4. Priority setting must involve a range of stakeholder representations (i.e., health professionals, consumers, funders, policy makers and guideline developers) through workshop and/or using online list. Systematic use of priority setting partnerships with organisations such as the James Lind Alliance would be beneficial.

5. Systematically grade all data reviewed and score resulting recommendations for practitioners accordingly.

6. Optimise Cochrane’s resources by focussing on smaller number of complex reviews.

7. Consider evidence as a ‘living document’ and update reviews if and when new data become available in an area of unmet needs.

8. Increase the number of rapid reviews with concise summaries (translated to Spanish, Chinese and French) and data relevant to practitioners.

9. Increase involvement of patients and consumers in the entire research cycle and Cochrane’s governance.

10. Communicate Cochrane evidence through the relevant channels – customised according to target group (commercial media channels, partnerships with medical journals, briefing papers for policy makers, using professional and community networks, high profile bloggers, etc).


1. Introduction

Cochrane Collaboration is a global, independent network of researchers, professionals, patients, carers, and people interested in health. It is in the process of reviewing its organisational structure and ways of working as part of its new Strategy to 2020.1 Cochrane aims to understand the effectiveness of its current outreach and governance structure from the viewpoint of all of its current and potential partners and stakeholders.

In March 2015, Cochrane Central Executive commissioned the Technopolis Group2, an independent policy research and consulting organisation, to support this work and carry out an external stakeholder consultation at a global scale, including a broad survey and in-depth interviews.

This report has been prepared to document the data gathered, describe the methodology, summarise the results of the study about stakeholders needs and expectations, and provide recommendations to support timely decisions about future strategic actions.

The scope of the study covers the following aspects of the four high-level objectives of Cochrane’s Strategy to 2020 as perceived by external stakeholders:

1. Producing evidence that is relevant, and up-to-date using pioneering methods and efficient production technology.

2. Making evidence accessible through user-centred design and delivery and translating content into multiple languages.

3. Advocating for evidence in health decision-making by offering evidence-informed products and services, contributing to debates on transparency and integrity of research and building local and international partnerships with other organisations to achieve global impact.

4. Building an effective and sustainable organisation that is inclusive and open, global and diverse, governed effectively and transparently.

The report is structured to respond to the above high-level objectives by synthesising the results from the survey and interviews and discussing the major findings and recommendations going forward. The complete survey data have been aggregated, anonymised and analysed, and are available in the Appendices.

2. Methodology and limitations

The study methodology followed a simple linear approach: First, scoping interviews with key internal and external stakeholders were conducted to frame the research questions for data collection, followed by the development and implementation of a global online survey. After a preliminary analysis of survey data, in-depth telephone interviews with a cross-section of stakeholders were conducted before all data were fully analysed and synthesised for final reporting. The target audience for the external stakeholder consultation were funders, policy makers, medical journal editors, practitioners, consumer groups, guideline bodies, partner organisations, and license purchasers who currently or potentially make use of Cochrane’s evidence and other services.

2.1 Global survey The multi-lingual survey (English, French, German, and Spanish) was developed in close collaboration with Cochrane and was proof-read and piloted before launch. The full text of the final surveys included a mix of closed and open questions and are available in the Appendices. After a short profiling section in the survey to help segment responses based on

1 http://www.cochrane.org/about-us/our-strategy 2 http://www.technopolis-group.com


stakeholder categories (level and type of involvement with Cochrane, geographic and lingual localisation of respondents), the survey explored the following topics from stakeholders’ point of view:

• Priority setting practices, and relevance/ timeliness of the evidence Cochrane produces.

• Development and application of Cochrane’s methods to support the production of evidence. (Completion of this section was conditional on interest/ knowledge of methodological aspects of respondent)

• Accessibility/ usability of Cochrane’s evidence and ways to improve those.

• Cochrane’s communication of its goals and principles to external partners and stakeholders through its local/ regional network, and across specialised fields.

• Cochrane’s organisational structure and partnerships, including features and benefits of a new membership scheme.

The survey was implemented in SurveyMonkey and was sent to 480 contacts from a database constructed for this project using information from Cochrane, Technopolis expert network, and other public information, after approval from Cochrane. The introductory email included a link to a landing page where respondents had the option to choose one of the four languages they wished to complete the survey in. This email also requested that individuals distribute the survey link through their networks in order to reach the relevant stakeholders as widely as possible (snowballing strategy). The identity and total number of individuals who received the survey is therefore unknown. As a result, the survey population represents a non-random sample of self-selecting respondents: it cannot be assumed that the views reported in this study represent the views of the entire stakeholder community.

The survey was open to record responses between 20 April and 31 May 2015. A total of 452 individuals responded to the survey (English: 364, French: 33, Spanish: 44: German: 11). Of these, 379 respondents provided answers beyond the profiling section; only these were included in the survey analysis. All data (including open box texts) were pooled together and analysed regardless of the language of the surveys. All graphs in this report show % of all who responded to the specific question; data labels within the chart or in parentheses within the text (unless labelled otherwise) indicate the actual number of respondents.

2.2 In-depth interviews A key part of the data collection was to conduct in-depth telephone interviews with stakeholders to understand the main issues and challenges in more detail. Semi-structured telephone interviews provided a suitable means to probe and draw out insight about emerging themes. We consulted with 22 individuals across the stakeholder landscape from various countries including the UK, Norway, The Netherlands, Spain, USA, Canada, Australia, Chile, and Japan; and the WHO.

3. Profiles of survey respondents

Most respondents lived in the UK (38%), followed by the US (9%), France (8%), Canada (5%), and Germany (5%) but countries such as Chile, Argentina, and Cuba as well as Japan, China and Australia also had representation (for more details, see Figure A11). The predominant native language was English (52%), followed by Spanish (11%), French (10%), and German (8%) (Figure A12).

The largest number of respondents indicated that they worked for a university (44%), a hospital or healthcare facility (17%) or a research institute (10%) (Figure A13). 10% of respondents described their employer as ‘other’; of this group, 32% (13) work for industry (pharma, CRO, trade association, private healthcare provider). Most respondents identified as ‘Researcher (non-clinical)’ (28%), followed by ‘Clinician’ (20%) (Figure A14). Many respondents chose ‘other’ to describe their role (18%); of these, 40% (32) indicated that they were students or student nurses.


Most respondents knew about Cochrane through using the Cochrane Library or website (67%) (Figure A15). Around 30% had colleagues involved with Cochrane, or knew the local Cochrane centre. A further 25% were or had been affiliated with a Cochrane group, and 23% were or had been involved with Cochrane as a contributor.

4. Cochrane evidence

Cochrane produces systematic reviews and other synthesised research evidence to inform decision making in health. A large proportion (80%) of survey respondents indicated that Cochrane evidence tackled the right topics, and answered questions that are well-framed and relevant (Figure 1). Cochrane evidence was considered sufficiently up-to-date by 60% of respondents, only 16% considered that it was not the case. Within this latter group, clinicans were somehwat overrepresented: 31% of respondents were clinicians who felt that the evidence was not sufficiently up-to-date; they constituted 23% of respondents to this question. 44% of those who indicated that Cochrane methods did not meet their needs (see Figure A19) also felt that the data were not sufficiently up-to-date (compared to 16% of those who indicated that Cochrane methods met their needs).

Qualitative interviews confirmed that Cochrane’s evidence is ‘fantastic’ and these are used regularly in decision making. They also highlighted the importance for Cochrane to improve on the relevance and timeliness of its evidence and make efforts to reduce the variability of the quality of its published reviews. Increasingly, there is a need to include evidence ‘lower down the hierarchy’ and take the most useful available data, including qualitative and economic data and those from observational studies that may provide a better explanation of the context and help interpret the results. These aspects will be discussed further in the sections below when discussing relevance, timeliness, and priority setting practices.

Figure 1 Relevance and timeliness of Cochrane evidence (n=374-379)

4.1 Relevance Cochrane’s primary evidence, its Systematic Reviews was overwhelmingly considered relevant today and expected to maintain its relevance going toward 2020.

It was acknowledged the highest quality evidence comes from Randomised Controlled Trials (RCT) of drugs/ surgical interventions and as such Cochrane focuses on those data. In many real life cases, however, data from RCT are not available and other dimensions of a health

327!

291!

306!

230!

47!

18!

14!

62!

5!

68!

54!

84!

0%! 20%! 40%! 60%! 80%! 100%!

Do!you!use!Cochrane!evidence!in!your!work!or!in!decision>making!on!health?!

Do!you!believe!that!Cochrane!evidence!tackles!the!right!topics!in!health!and!health!

care?!

Do!you!believe!that!Cochrane!evidence!answers!quesGons!that!are!well>framed!and!

relevant?!

Do!you!find!that!Cochrane!evidence!is!sufficiently!up>to>date?!

Yes! No! I!don't!know!


intervention such as contextual information, ethical, legal and social considerations, resource availability, and cost effectiveness may significantly alter potential advice and recommendations for implementation. It was repeatedly discussed in the consultation that physicians face important decisions on a daily basis and they expect to make those on the best available evidence. One respondent commented that practitioners would like guidance on what to do in the absence of sufficient evidence for an intervention. The reason why researchers may find insufficient published evidence to conduct the review or provide useful recommendations was, according to respondents, due to many research questions having an overly narrow focus. Interviewees suggested fewer reviews but broader in scope (i.e., complex reviews) relevant to health services.

Respondents were unclear about how Cochrane made decisions on which topics to address and the large number of review groups appeared to have different approaches to topic selection. It was noted that topic choice seemed to be researcher-driven, rather than user-driven, and that it was unclear if all stakeholders were consulted in the process. For example, a respondent commented that “many of the topics addressed seem to be those that are ‘researchable’, rather than those which are of interest/use to patients and clinicians”. This sort of skew across research topics in the broader scientific literature is however well known and it calls for action to change from the entire community from funders to researchers.

Specifically mentioned as currently not (or not sufficiently) addressed topics were diagnostics, rare diseases, obesity in adults and nursing care. One survey respondent would like to see “more 'research on research' reviews, along the lines of existing reviews, such as the analysis of selective reporting, or the implementation of CONSORT”. This type of topic would fit with Cochrane’s wider goal of promoting transparency in medical research and evidence-based medicine.

The expectation is that all available evidence is used and the quality of evidence and strength of recommendations are graded systematically in Cochrane evidence, following the work of the GRADE Working Group.3 (Note also the US Preventive Services Task Force4 grade definition, suggestions for practice, and levels of certainty) Ultimately, a ‘bottom line’ recommendation would be useful for health decision makers as part of all Cochrane evidence.

There have been an expectation that focus will not only be given to treatment but increasingly to diagnosis also and specific thematic areas were highlighted as being underrepresented in the evidence: emergency medicine; environmental health; mental health; medical devices; health services research; pre-hospital care; gerontology; and public health. Other more specific topics are listed in Section A.2 including living with disabilities, obesity in adults, and diagnosis of ADHD.

Although consultations has shown that most of Cochrane’s evidence are relevant today, there were reports that some systematic reviews are not fit for purpose, and future reviews may require broader focus and timely update to continue to produce relevant evidence. It was noted that there was some “lack of equilibrium in quantity and specificity of reviews per topic”, and that questions predominantly addressed topics of developed (rather than developing) countries.

Finally, it is noted that the NIHR (a major funder of Cochrane in the UK) is running a specific Cochrane Review Incentive Scheme with criteria for review nominations including the number of patients affected by the clinical condition; the degree of uncertainty that exists in the absence of an up-to-date systematic review, the likelihood of a review assisting or changing policy or practice; the likely importance and priority that would be given to the topic by consumers, and their involvement in the review; and an indication as to whether the review includes complex methodologies. These factors may be considered by Cochrane to determine the relevance and to set priorities (see below) of their evidence in general.

3 http://www.gradeworkinggroup.org 4 http://www.uspreventiveservicestaskforce.org/Page/Name/grade-definitions


4.2 Timeliness New drugs and medical technologies regularly appear on the market and hence health practitioners are expected to inform their work by utilising effectiveness reviews as early as possible in the process. National health technology assessment agencies and guideline developers usually lead the way to provide such documentation to healthcare professionals.

Cochrane wants to use the highest quality data analysed with its rigorous methodologies to produce ‘gold standard’. The evidence however must be published in time (about 6 months from posing the research question) so that relevant decisions can be made by HTA agencies, guideline developers, and healthcare professionals. Out-dated evidence may in some sense be considered a wasted opportunity to influence the community and its practice.

The predominant view expressed in the survey open boxes regarding the timely update of Cochrane evidence was that reviews were often but not always up to date (44), and many of these respondents (24) noted variation between different groups/topics. Several survey respondents (10) indicated that the administrative nature of the review process is at the heart of the problem and interviewees agreed. It was suggested that it should be made clear when a review was last updated, how the update was performed, and that some reviews would not be further updated. Some more timely contextual information could also be added to the review, along the lines, e.g. ‘this review was produced prior to the introduction of x-class of drugs’. Although Cochrane is outstanding in its rigour in producing systematic reviews, one respondent considered the update processes of UpToDate5 more reactive to users’ needs.

The increased production of Rapid Reviews (versus comprehensive reviews) in this context would be important to reduce opportunity cost since Rapid Reviews have sharper focus and delivery times are shorter to produce timely evidence. In terms of the frequency of update of the published evidence, and extending Rapid Reviews to comprehensive reviews, there can be no general guidelines. Cochrane evidence should be considered as a ‘living review where publication of new data are constantly monitored with the latest technologies deployed and depending on the extent of the accumulated body of new evidence, decision will be made on a case-by-case basis considering internal and external priorities and the availability of financial and human resources.

5. Priority setting practices

External stakeholders, including those that work closely with Cochrane, conveyed that very little is visible about Cochrane’s priority setting practices. Survey responses to questions indicated that knowledge of, and engagement in, priority setting processes is low. 71% of respondents could not comment if Cochrane was receptive to external views, and 84% had never been involved in priority setting (Figure 2). For example, neither respondents from Germany nor Spain had been involved in priority setting for Cochrane. 14% of respondents from the UK had been involved. Canada, with 28% of respondents, had the highest participation of any country. Despite the current low level of engagement, there was significant interest to engage in the future, as signalled by 51% of respondents.

Most people consulted in this exercise assumed that Cochrane’s priorities are strongly influenced by its own researchers and found that those are more relevant to academics than healthcare practitioners. External stakeholders indicated that they hear about new systematic reviews when the protocol is published by the study team. This however may be too late for them to effectively provide input to the research questions. There has been no mention during the external stakeholder consultation of the various topic lists that are discussed and published online by editorial review groups.

5 http://www.uptodate.com


Figure 2 Priority setting and external consultation (n=364 – 375)

It was welcomed by interviewees that Cochrane published for the first time its Priority Review List on 1 February 20156 comprising 200 priority review topics (either new or to be updated) across all areas, which are expected to best match the needs of health policy makers. At the time of conducting the interviews in May/June 2015, however, the external stakeholders asked were not aware of this document. This clearly points to the need for more intense dissemination of important Cochrane news via communication channels and networks used by these stakeholders; and presenting the Priority Review List in an accessible and easily searchable format rather than as a downloadable Excel spreadsheet. The transition to a membership organisation (discussed below) may create the broader community needed to implement efficient communication strategies.

Interviewees remarked that a transparent system is needed where research topics/ questions are identified that have the potential to change the way clinicians practice medicine. An open priority setting system would also reduce the opportunity cost of conducting reviews for Cochrane. Currently, there may be a tendency to produce systematic reviews in the areas where more data/ evidence exist (or where contributors have experience), and not necessarily where users’ needs lie or health burden is. Therefore priority setting needs to involve external stakeholders, including practitioners, policy makers, funders, and consumers. It was also highlighted that a review of skills, strengths and weaknesses (and its distribution) of the current Cochrane workforce would help to better prepare to meet users’ needs.

There was a sense of enthusiasm that Cochrane’s priority setting process may become more open as experience showed that the processes and response from various editorial review groups were highly varied. It was suggested that beyond consulting objective administrative data on disease burden7, medical journal editors could be requested to feed into Cochrane’s priority setting processes as they have a good overall view of the existing research landscape and available published data. Guideline developers, major funders and international organisations could also be consulted to obtain a list of the topics they are preparing to address in the future. Specifically mentioned was the potential for collaboration with a number of existing initiatives, such as the Guidelines International Network (G-I-N), the Haute Autorité de Sante (HAS) in France, the Health Search database in Italy, the Canadian

6 http://www.cochrane.org/news/cochrane-identifies-priority-topics-review 7 http://www.healthdata.org/gbd/data

87!

47!

186!

22!

316!

178!

265!

12!

0%! 20%! 40%! 60%! 80%! 100%!

Based!on!your!experience,!is!Cochrane!recepGve!to!external!views!when!seRng!

prioriGes?!

Have!you!ever!been!involved!with!priority!seRng!for!systemaGc!reviews!in!Cochrane?!

Would!you!like!to!engage!with!Cochrane!on!seRng!its!review!prioriGes!in!future?!



Agency for Drugs and Technology in Health (CADTH) and the UK National Institute for Health Research. Cochrane working in partnerships with other organisations and associations would also ensure that no duplication of effort takes place. It was recognised however that Cochrane’s processes must remain independent and fit with its global mandate.

In addition, engagement of professionals and lay people from local and national settings were considered important as that can serve as the beginning of a longer stakeholder engagement journey. It was suggested that taking on researchers from local settings (even if not at the suitable level to carry out systematic reviews independently) also in the research phase may help uncover new data sources, help with language difficulties, increase the understanding of contextual issues, and not only help future dissemination activities but, by the end of the journey, build vital research capacity often lacking in resource poor settings.

In terms of engagement practices of priority setting with Cochrane, respondents would prefer to participate via workshops and consultation on topic lists (preferably using online consultation). Additional suggestions were Delphi rounds, including Cochrane representatives in meetings at other organisations, provision of feedback at the end of every guideline update cycle, and adopting the James Lind Alliance8 or the TREKK9 method (see below).

Priority setting at TREKK

Translating Emergency Research Knowledge for Kids (TREKK) Prioritization Committee was formed in January 2013 to review and prioritise the results of the needs assessment and give direction to knowledge mobilization projects and work plans of TREKK. Members include medical directors, managers and frontline clinicians from general emergency departments, along with TREKK directors and key researchers from Paediatric Emergency Canada (PERC), Paediatric Emergency Research Network (PERN) and Knowledge Translation Canada.

Prioritisation exercise at TREKK involves looking for gaps, perceived and unperceived needs. In this process, first clinicians and patients are surveyed in the highly specialised field of paediatric emergency. In a second stage, administrative data (i.e. mortality rate) are used to cross-check findings before priorities are established.

6. Role of the consumer

When asked about patient, patient advocate and healthcare consumer involvement in the production of Cochrane evidence and in Cochrane activities (Figure 3), the various dimensions were all rated as ‘important’ on average, with ‘Setting priorities for research’ seen as the most important and ‘Co-production of research’ the least important.

This result is somewhat at odds with interviews where it was advised to involve patients and patient groups in the priority setting exercise with caution in order for Cochrane to remain free from potential conflict of interest or being influenced by specific and predictable agendas. Interviewees emphasised the role for patients to be involved in setting relevant outcome measurers in the research questions and crucial role in disseminating the results of the research as widely as possible. ‘Co-production in research’ being poorly rated in the survey may be explained by the large respondent cohort of non-clinical researchers who consider conducting research requiring a highly specialised training.

8 http://www.lindalliance.org 9 http://trekk.ca/teams/prioritization-team


Figure 3 Involvement of patients, patient advocates and healthcare consumers in the production of Cochrane evidence and in Cochrane activities (n=371 – 374)

7. Cochrane methodology

Three quarters of survey respondents had an interest in evidence synthesis methods, and the vast majority of those felt that the methods used by Cochrane met their needs; 16% did not feel this was the case (Figure 4). The awareness of Cochrane Methods groups was at 62% of those with interest in evidence synthesis methods.

It is widely recognised that Cochrane has developed a rigorous methodology to collect, synthesise, and report primary data to provide authoritative evidence about the effectiveness of health interventions. The real question according to respondents is whether Cochrane’s methodology has remained cutting edge and innovative concerning the diversity of research topics relevant to health professionals today. Several external stakeholders mentioned that Cochrane appears to be a ‘club of methodologists’ and questioned whether the evidence produced is suitable to clinicians and healthcare professionals. Over 25 survey respondents and stakeholders interviewed explicitly called for a broader methodological approach to be adopted by Cochrane regarding evidence synthesis beyond data form Randomised Controlled Trials. To illustrate this point, one comment was that “we need a methodology that is robust for the synthesis of non-RCT data, that particularly enables the description of the context and presumed mechanisms for complex interventions.”

Interviewees agreed that there appears to be resistance in some review groups to develop and apply new methods. They highlighted that “qualitative evidence is not lower quality evidence, just different” and more use of narrative synthesis was essential. Nevertheless, it was recognised that evidence synthesis is not for the gifted amateurs and Cochrane provides a very high professional standard for its systematic reviews.

Suitable methodological approach for new types of study design is requested that synthesise data from observational studies, health economics, cultural and ethical contexts and other qualitative data. Analyses of complex health issues (e.g. for public heath interventions) often require systems approach and cannot adequately be captured by simple, focussed methodologies. The systematic use of GRADE to highlight the quality of evidence, risk of bias

2%!

3%!

12%!

3%!

6%!

14%!

22%!

34%!

20%!

21%!

34%!

37%!

35%!

33%!

36%!

48%!

36%!

16%!

41%!

31%!

2%!

4%!

4%!

3%!

6%!

0%! 20%! 40%! 60%! 80%! 100%!

SeRng!prioriGes!for!research!

InterpreGng!the!relevance!of!outcomes!

Co>producGon!of!research!

DisseminaGon!of!research!

Are!represented!in!the!governance!of!Cochrane!

Not!at!all!important! Somewhat!important! Important! Highly!important! No!view!


and the strength of the ensuing recommendations is considered essential part of the methodology.

It is remarked by the authors of this report that members of the Cochrane Bias Methods Group and the Cochrane Non-Randomised Studies Methods Group used the Cochrane Methods Innovation Fund (2011) to develop the ACROBAT-NRSI tool (A Cochrane Risk Of Bias Assessment Tool for Non-Randomized Studies of Interventions). It is unclear how widely this tool is known in the community but the software documentation and resources are available through a Google site.

It appears however to be difficult to find the right balance between rigorous, tried and tested Cochrane methodologies and more innovative approaches that exist elsewhere in the research community and could be adapted to respond to specific research questions (specific comments on methodological issues are listed in Section A.4 ). The methods requirements may at times be too complex and resource-intensive for new reviewers to join the study team, especially from developing countries.

Figure 4 Cochrane’s methods (n=375; n=283-285)

Two thirds of survey respondents believe that Cochrane has high enough reputation and thus has the ability to attract the best methodologists. There may however be potential limitations to attract innovative methodologists who are not within the scope of Cochrane methods currently applied. One respondent felt that top methodologists preferred primary research.

A range of factors were suggested to continue to attract the best methodologists to join Cochrane (Figure A20), including clarity on how to approach and participate; collaboration opportunities; innovative methodologies developed; faster review processes; demonstrated impact on health care/ policy; and providing salary or stipend. The need for both innovation within Cochrane as well as maintaining consistency across the review groups were highlighted by interviewees as necessary aspects to consider when changes are made to current practice.

287!

201!

178!

189!

46!

45!

107!

27!

42!

37!

69!

0%! 20%! 40%! 60%! 80%! 100%!

Do!you!have!an!interest!in!evidence!synthesis!methods?!

Do!the!methods!used!in!Cochrane!Reviews!meet!your!needs?!

Are!you!aware!of!the!various!methodological!groups!that!comprise!Cochrane!Methods!and!support!method!development!for!systemaGc!

reviews?!!

Do!you!believe!Cochrane!has!high!enough!profile!to!aYract!the!best!methodologists!for!

its!SystemaGc!Reviews?!!

Yes! No! No!view!


8. Accessibility and usability

Cochrane’s new Strategy to 2020 explicitly states the goal to make Cochrane evidence accessible and user-centred. Three quarters of respondents indicated that Cochrane evidence was easily accessible and usable (Figure A21). When asked about the importance of a range of factors relating to accessiblitiy and usability (customised search facility for different audiences; presentation format and layout; adapted language for different audiences) respondents rated most factors on average as ‘important’. Respondents however mentioned issues regarding accessibility of full text of the evidence especially by stakeholders in countries (including Brazil, France, Portugal, Canada and USA) where only restricted institutional subscriptions were available. Notwithstanding the access to Cochrane Library provided via the HINARI and INASP programmes, request for Open Access was commonly mentioned as ‘highly important’ in the stakeholder consultation (

Figure 5). Opening up the Cochrane library is therefore seen as a necessary and progressive step towards open science and responsible research. It is remarked that interviewees were not aware of Cochrane’s commitment10 to universal open access to new and updated reviews by the end of 2016.

Figure 5 Please rate the importance of the following factors relating to accessibility and usability of Cochrane evidence. (n=361 – 363)

Interface to Cochrane Library was also mentioned as one challenge to find relevant reviews. Some interviewees mentioned that they use the PubMed interface to access Cochrane evidence as the Cochrane site was too complex to use and filters were not relevant to many stakeholder groups. It was pointed out that ‘searchability’ needed to be improved, so that protocols and reviews could be located more easily. In this respect, additional piloting and testing of the site was suggested.

Stakeholders appreciated the need to record the full body of evidence collected during the systematic review, however, presented in an overly long format significantly reduces

10 http://www.cochrane.org/about-us/open-access

5%!

5%!

2%!

17%!

8%!

14%!

12%!

36%!

40%!

35%!

84%!

38%!

49%!

45%!

1%!

4%!

2%!

1%!

0%! 20%! 40%! 60%! 80%! 100%!

Open!access!

Customised!search!facility!for!different!audiences!

PresentaGon!format!and!layout!of!the!reviews!

Language!adapted!for!different!audiences!(scienGfic,!guideline!makers,!consumers)!

Not!at!all!important! Somewhat!important! Important! Highly!important! No!View!


usability of the evidence. Stakeholders also asked for consistent structure across reviews with shorter and clearer summaries (with suggestions for a format similar to Critical Appraised Topics, or a Wiki-Cochrane) tailored to the audience (e.g. clinicians), inclusion of a summary of findings table as well as a table of the studies’ patient characteristics to assist in the practical interpretation of the results. Usability of the evidence for clinical and non-medical applications could also be enhanced by clearer and unambiguous recommendations, the ‘clinical bottom line’. It was requested that the popular CONSORT reporting guideline11 is implemented for Cochrane evidence. Other suggestions to make Cochrane reviews more accessible and usable included more graphics in reviews, more publicity around Cochrane and a stronger communication strategy, and links to other websites where people are accessing evidence.

The generality of the above comments were confirmed in qualitative interviews and suggested that more emphasis was needed on the interpretation of ‘rigorous science’ with metrics used by practitioners (e.g., confidence intervals for NNTB / NNTH presented in SoF tables, risk of bias tables).

8.1 Translation of Cochrane evidence When asked about language preferences for Cochrane publications, almost all survey respondents indicated that they were comfortable using English to read Cochrane evidence (Figure A23). However, over a third of the non-English speaking respondents thought they would use Cochrane evidence more often if it were available in their native language. These views varied between different countries and languages: while more than half of all native French and native Spanish speakers indicated they would use Cochrane evidence more often if it were available in their native language (61% and 59%, respectively), a lower proportion of native Portuguese (25%), Italian (22%), and German (16%) held this view. In addition, three quarters of the the non-English speaking survey respondents thought that others in their region would use Cochrane evidence more often if publications were translated into the national language. Broken down by language, native speakers of French (95%), Spanish (94%), Portuguese (85%), and German (79%) held this view, as well as very high proportions of respondents living in France (97%, n=30), Germany (94%, n=16), Spanish-speaking countries (93%, n=43; this includes Argentina, Chile, Costa Rica, Cuba, Mexico, Spain, and Uruguay), and Portuguese-speaking countries (92%, n=12, Portugal and Brazil).

11 http://www.consort-statement.org


Figure 6 Language-related questions (n=359 – 362)

When asked to rate the importance of translating various sections of Cochrane Reviews, survey respondents gave an average rating of ‘important’ to all sections of Cochrane publication (Figure A24). Translation of the abstract received the highest score, followed by the plain-language summary and the conclusions. Translation of the background/objectives and methods sections were seen as having lower importance. Overall, English emerged as the top priority language, followed by Spanish, Chinese and then French (Figure A24).

One survey respondent felt that the strict requirement on English language skills was a significant barrier for some investigators to participate in review groups. One respondent (Spanish-speaking) expressed the desire to contribute to the priority setting process, but pointed out that she/he would only be able to do so in his/her native language.

9. Advocacy

Cochrane considers it critical not only to produce relevant and timely evidence but also to advocate for the use and uptake of the evidence in health policymaking and contribute to public debates on scientific transparency and integrity. Advocating for evidence is part of Cochrane’s Goal Three of its new Strategy approved in late 2013. Real impact of advocacy requires many years before it can be correctly measured. Therefore stakeholders were not only asked to rate the current emerging practices but also to express their views where they think Cochrane should focus on in order to continue to develop this area over the coming years.

When asked to rate the effectiveness of Cochrane’s advocacy and its campaign for transparency, survey respondents considered these, on average, to be less somewhat than effective (scores of 2.7 and 2.8, respectively, with ‘effective’ represented by a score of 3, Figure 7). Most of the survey respondents who explained their answers pointed out that they did not know about these activities or that Cochrane was relatively unknown in their field/ country. In some cases, especially in the South American context, it was noted that the limitations to impact often originate from the environment Cochrane Centres operate in, citing the challenge of inappropriate influence of industry on clinical practice, and policy makers’ low level of understanding of evidence-based medicine. Respondents also pointed to other, effective campaigning groups, e.g. AllTrials and Sense About Science. Interviewees agreed that Cochrane needs to go beyond being a technical group and actively advocate its principles and evidence. It was also suggested that respected clinicians and medical bloggers could advocate on behalf of Cochrane to reach the professional community more effectively.

350!

67!

154!

12!

109!

51!

185!

154!

0%! 20%! 40%! 60%! 80%! 100%!

Are!you!comfortable!using!English!to!read!Cochrane!Reviews?!

Cochrane!evidence!were!increasingly!available!in!your!naGve!language,!would!you!

use!it!more!oben?!!

Do!you!think!others!in!your!region!would!use!Cochrane!evidence!more!oben!if!they!were!

translated!into!the!naGonal!language?!

Yes! No! Not!applicable!


Figure 7 Based on your experience in your region or in your field of speciality, please rate the effectiveness of the following activities of Cochrane (n=344 – 346)

Respondents rated Cochrane’s influence in promoting evidence-informed decision making on average between ‘influential’ and ‘somewhat influential’ with regards to various stakeholders (Figure A27): Clinical practitioners were considered to be (relatively) most influenced by Cochrane, followed by research funders, healthcare policy makers, health service managers, and finally consumers. This result is somewhat surprising as Cochrane has been increasingly focussing on patients and carers through its Cochrane Consumer Network since its establishment in 1995. Although this result shows that Cocrane’s advocacy is not yet as influential as it could be, it should be noted, that the survey results are not based on a representative sample, and close to half of the respondents are non-clinical researchers and clinicians, while funders, patients and carers are in minority.

In terms of strategic approach to influence health policy, the consultations pointed to the need for both ‘bottom-up’ approaches where clinicians and other healthcare professionals are targeted and ‘top-down’ approaches where funder, guideline developers and policy makers are targeted. It is clear that in terms of ‘value for money’, the top-down approach is more effective to propagate change in healthcare practice through multiplier effects. Nevertheless, partnering with organisations and networks where large number of members are already present provide opportunities for Cochrane to engage a broader group of stakeholders. In terms of practical recommendations, it was suggested that engagement of high-level decision makers in the prioritisation exercise, increased high-level meetings between Cochrane and government officials (in particular in Spanish-speaking countries) would lead to better advocacy. In this respect, establishing a liaison officer for each country would be warranted. There were however voices that would like to see Cochrane remain as the source of evidence rather than ‘selling’ its products and services and suggested to partner with professional associations in various areas of health who could then lobby policy makers directly. Stakeholders emphasised that Cochrane evidence in its current format may not be best for policy makers and it needs to be adapted for ease of use. Policy makers also require more contextual information and integration of different types of evidence, such as cost comparison. In addition, tools and guides should be made available (along with training activities) for policymakers to help them implement recommendations in practice.

Enhanced visibility and broader accessibility of evidence and better advocacy are inherently intertwined. Most stakeholders agree that Cochrane could ‘more aggressively’ push its content via popular social media platforms (including twitter and kudos12), mobile apps13

12 https://www.growkudos.com/about/institutions

6%!

3%!

22%!

21%!

34%!

39%!

12%!

15%!

25%!

22%!

0%! 20%! 40%! 60%! 80%! 100%!

Cochrane’s!campaign!for!transparency!and!integrity!in!scienGfic!conduct!

Cochrane’s!engagement!in!debates!advocaGng!for!the!use!of!evidence>based!

methods!and!high!quality!evidence!

Not!at!all!effecGve! Somewhat!effecGve! EffecGve! Highly!effecGve! No!View!


and podcasts that summarise the latest reviews briefly. It is commendable that over 300 reviews already have such podcasts on Cochrane’s website and also available from the iTunes library. It was suggested that individual Cochrane groups should include designated individuals responsible for their dissemination activities.

It was suggested that Cochrane should increasingly track via ‘observational research’ the impact of its evidence so that a link between the review and its impact can be established and success stories better communicated. Cochrane findings could also be disseminated through weekly/monthly short reports on key reviews with relevance to healthcare for publication in journals of local medical associations, as well as via blogs and information services like MedScape. Other formats for sharing Cochrane evidence were also mentioned, including briefing papers to commissioners, press releases to consumers, and workshops with health professionals. Interviewees confirmed that these sorts of activities are happening sporadically already, but a more strategic approach would be needed.

10. Organisational structure

Cochrane is a global network with representatives across 41 countries. It has a complex organisational structure with Centres and Branches, Fields and Networks, and Methods and Review Groups. Almost no interviewee could explain the functioning of Cochrane as an organisation. Despite this fact, about 40% of survey respondents were aware of and/or interacted with Cochrane groups globally, and 70% were aware of the local presence of Cochrane in their region or country (Figure 8). 73% of those who were aware of Cochrane’s local presence felt that it allowed them to engage with Cochrane. In terms of partnership between Cochrane and the external stakeholders’ organisations: one thirds had a partnership and about one thirds did not have a partnership, and surprisingly about one thirds of the the respondents did not know if there was a partnership with Cochrane.

There were however large variations between countries: While respondents from France (20%, n=30) and Spain (31%, n=13) were less aware of Cochrane globally, respondents from UK (38%, n=122), Germany (44%, n=16), US (48%, n=29), Chile (50%, n=12), and Canada (80%, n=15) were more aware of or interacted with Cochrane globally, compared to the overall average of 40%. For awareness of local presence of Cochrane, the US came out the lowest (45%, n= 29), while Canada (93%, n=15) and Spain (100%, n=13) at the top. It must be noted here again that the survey population represents a non-random sample of self-selecting respondents and hence these ratios may not reflect the views of the entire stakeholder community in a given country.

13 For an example, see: http://www.imedicalapps.com/2015/02/med-pearls-app


Figure 8 Awareness and views on Cochrane’s local presence (n=335 – 337, 223)

It is notable that France had the highest proportion of respondents who indicated that Cochrane’s local presence did not allow them to engage (63%, 12 of 19 respondents living in France who indicated they were aware of Cochrane’s local presence). Half of the respondents from Italy (50%) also found it hard to engage with local centres, but with a smaller number of responses to this question (n = 6). The issue of strong university leadership and internal protective politics influencing engagement with Cochrane locally was also touched on in open box comments: “In my country, participation (in Cochrane) centres around one university, and it is not possible for others to participate” and “I haven’t found it easy to become involved in reviews as many are ‘reserved’ by existing teams and it sometimes seems like a closed shop.” It would be important for stakeholders to see Cochrane as an open and easy-to-engage organisation so that barriers encountered by some of these respondents would disappear.

Few respondents answered the question of how their local Centre could better meet their needs. Suggestions included enhanced local dissemination of information, interaction with the broader public and patients, offering courses/meetings, and appointment of an in-country Cochrane representative (where currently not yet available).

10.1 Partnerships A third of respondents (33%) stated that their organisations had a partnership with Cochrane and another about third (38%) that there was no such partnership. Surprisingly, 30% of the respondents did not know if their organisations had partnership with Cochrane or not. This points to the possibility that Cochrane’s brand and message are not clearly visible to the individual even if they are engaged.

Respondents expressed broadly positive views when asked to provide further detail on the value of their institution’s partnership with Cochrane. They reported that the affiliation was “really good for the University’s reputation and impact”, that the “partnership with Cochrane maintains a channel of communication between our profession and evidence based health care”, or that the partnership was a “great help with search strategies and the process of performing systematic reviews for relevant disease areas”. Two respondents noted that while their institutions’ links with Cochrane was good for driving local initiatives, such as training, it did not lead to an engagement with the broader, global Cochrane priorities and activities. One respondent with policy analyst background explicitly commented that “I don’t get much value from the partnership with Cochrane. The meetings are more information sharing and not really an opportunity to inform priorities.”

There have been a number of suggestions about how to establish and strengthen the partnership including visible feedback from Cochrane to the organisation so that funders can

142!

234!

162!

110!

193!

103!

61!

126!99!

0%! 20%! 40%! 60%! 80%!

Are!you!aware!of!and/or!interact!with!Cochrane!Groups!globally?!

Are!you!aware!of!the!local!presence!of!Cochrane!in!your!region/country?!

If!Yes,!does!the!local!presence!of!Cochrane!allow!you!to!engage!with!Cochrane?!

Does!your!organizaGon!have!a!partnership!with!Cochrane!(either!formal!or!informal)?!



see the link, enhanced media outreach, regular meetings and dialogue with policy makers, guideline developers, health authorities, practitioners and scientific societies.

A summary of recommendations for potential partner organisations in specific countries is presented in Figure A30.

10.2 Membership organisation Cochrane plans to change its way of functioning and to become a membership‐based organisation. This would be a non-fee paying membership scheme and its aim is to enhance engagement with and formally recognise links to a diversity of stakeholders: individuals and organisations alike. The study consulted external stakeholders about the key features of a Cochrane membership scheme.

When asked to rate the extent to which various features and benefits of a new Cochrane membership scheme would be of interest, respondents rated the following features and benefits the highest (Figure 9):

• Access to Cochrane online training (66% of respondents interested or highly interested)

• Access to Cochrane software, tools for dissemination and advocacy (63% of respondents interested or highly interested)

• Learning about Cochrane and evidence-based medicine (63% of respondents interested or highly interested)

• Undertaking continued professional development with Cochrane accreditation (59% of respondents interested or highly interested).

Involvement in crowd-sourcing projects was of least interest with 39% of respondents interested or highly interested.

Overall, three quarters of respondents (77%, n=322) would consider becoming a Cochrane member (as an individual or as an institution) in Cochrane’s new membership scheme. However, only few explained their answer, and those who did were respondents that would decline membership; for them membership would not be a priority or that they were just too busy to join another organisation. One respondent from industry, and one respondents from a funder pointed out that membership may represent a conflict of interest to them.

Interviewees were broadly supportive of the idea of Cochrane as a membership organisation. They emphasised the need to see how users would benefit from such a scheme and how to make it work in practice. A strong suggestion was to highlight Cochrane’s commitment to evidence and independence and communicate those in a clear way.


Figure 9 Please rate the extent to which the following features and benefits of a new membership scheme would be of interest to you. (n=324 – 329)

10.3 Structure and governance Cochrane’s new Strategy to 2020 recognises that an effective and sustainable organisation requires transparent governance and inclusiveness with regards to contribution and leadership opportunities by its members. We studied external stakeholders’ views on how effectively such measures have been implemented and their awareness of relevant issues.

On average, respondents agreed with survey statements on Cochrane’s structure and governance (scores between 4.3 and 3.3, where a score of 4 represents ‘agree’ and a score of 3 represents ‘neither agree nor disagree’) (Figure 10). They were most in agreement with the statement that Cochrane is an independent organisation (score: 4.4), and that Cochrane provides useful evidence for their health-decision making needs (score: 4.2). It is notable that about a third of the UK respondents had no view14 on the various statements (29%-38%). The lowest average scores were received by the statements “I would know how to become involved in Cochrane” and “I would be prepared to be involved in the governance of Cochrane in the future” (scores: 3.4 and 3.3, respectively).

Interviewees emphasised that need for Cochrane to continue to stay independent from various interest groups, primarily from the pharmaceutical industry but also patient advocacy groups with specific agenda in order to provide credible evidence to health decision makers. This however should not manifest itself as Cochrane being ‘anti-industry’ and oppose whatever

14 It is possible that some respondents interpreted the rating option ‘neither agree nor disagree’ as ‘no view’ which was a distinct option in the survey. In that case, it would appear that many respondents would not know how to rate these statements.

19%!

12%!

10%!

10%!

20%!

12%!

17%!

7%!

9%!

12%!

28%!

28%!

19%!

21%!

26%!

34%!

30%!

22%!

18%!

20%!

24%!

32%!

34%!

33%!

23%!

28%!

23%!

32%!

31%!

26%!

20%!

23%!

31%!

30%!

16%!

16%!

20%!

31%!

35%!

33%!

8%!

6%!

6%!

6%!

15%!

10%!

10%!

7%!

6%!

8%!

0%! 20%! 40%! 60%! 80%! 100%!

Building!a!personal!Cochrane!profile!

Access!to!a!Cochrane!virtual!community!for!networking!and!forums!

Accessing!Cochrane!sobware!

Accessing!tools!for!disseminaGon!and!advocacy!

Involvement!in!crowd>sourcing!projects!

Tailored!communicaGons!

Discounts!at!events!

Learning!about!Cochrane!and!evidence>based!medicine!

Accessing!Cochrane!online!training!

Undertaking!conGnued!professional!development!with!Cochrane!accreditaGon!

Not!at!all!interested! Somewhat!interested! Interested! Highly!interested! No!View!


important clinical research they conduct. Interviewees also noted the need for more focus on academic independence within the Cochrane network. Intellectual conflict of interest of contributors would be equally harmful for the reputation and credibility of Cochrane and thus implementation of an effective screening system would be important in this regard.15

Cochrane will need to find the balance between being an interest-led academic network and a more centralised corporate entity in order to deliver practice-changing recommendations based on the evidence it produces. As the data collected in this study show, Cochrane has built up a good brand identity and trust within the community that needs to be maintained. Interviewees noted the need for clear rules when and where individual members may use their Cochrane affiliation. There have been examples cited where individuals had expressed their own views or sought funding in the name of Cochrane however these were not properly coordinated and the resulting research findings were not shared in the Cochrane library.

Figure 10 Based on your experience, please indicate the extent to which you agree with the following statements (n=328 - 331)

11. Training initiatives

Cochrane provides access to a range of online material aimed at supporting authors and lay contributors of Cochrane reviews. We wanted to find out if specifically external stakeholders would be interested in Cochrane providing free training initiatives for their organisation?

The majority of respondents (74%, n=326) indicated that they would be interested in Cochrane providing free training initiatives, and those who elaborated further were interested in methodological training on the design and interpretation of systematic reviews and meta-analyses (46), with some respondents (4) asking specifically for online training to

15 For a recent discussion on the topic, see http://www.bmj.com/content/350/bmj.h3176

18%!

15%!

6%!

16%!

17%!

18%!

18%!

9%!

15%!

26%!

45%!

37%!

32%!

28%!

29%!

51%!

34%!

28%!

40%!

13%!

10%!

11%!

12%!

30%!

15%!

12%!

7%!

29%!

38%!

36%!

35%!

8%!

11%!

14%!

0%! 20%! 40%! 60%! 80%! 100%!

Cochrane!is!an!independent!organizaGon!

Cochrane!is!run!effecGvely!and!efficiently!

Cochrane!is!governed!effecGvely!

Cochrane's!governance!is!open!and!transparent!

Cochrane's!conflict!of!interest!policies!are!clear,!well!balanced!and!effecGve!

Cochrane!provides!useful!evidence!for!my!health!decision>making!needs!

I!would!know!how!to!become!involved!in!Cochrane!

I!would!be!prepared!to!become!involved!in!the!governance!of!Cochrane!in!future!

Strongly!disagree! Disagree! Neither!agree!nor!disagree!

Agree! Strongly!agree! No!View!


increase participation. The importance of training tailored to users of reviews was also highlighted. Respondents were also interested in training on evidence-based practice (16), on how to use and contribute to Cochrane (6), using software (2), evidence bias rating (3), using reviews in policy decision making (2), reporting standards (2), standards for plain language translation, quality assessment and critical appraisal, and evidence search/literature review and analysis (11). Mentioned were also an accreditation scheme for members/other organisations, an award scheme for outstanding group members, and training for individuals to promote use of evidence in decision making.

Training initiatives would no doubt benefit engagement with external stakeholders and could provide potential new members of the Cochrane network. Interviewees noted that training targeting policymakers and government officials would be useful for effective advocacy of Cochrane’s principles and facilitate the uptake of Cochrane’s evidence.

12. Concluding remarks and recommendations

Cochrane has initiated various reviews of its processes and engagement practices with external stakeholders as part of its new Strategy to 2020. A global online survey and in-depth interviews have been carried out on behalf of Cochrane by Technopolis Group to collect data independently from external stakeholders and support the review process. This report includes relevant information from external stakeholders to help to reflect on their needs and inform Cochrane about ways to improve current processes.

Overall, external stakeholders of Cochrane expressed an overwhelming appreciation for what Cochrane stands for and the evidence it produces. They considered the majority of Cochrane reviews relevant and timely, helping decision makers and practitioners in their daily work. It will however be important to continue to improve on achievements and respond to the changing needs of the user community.

There is an increasing requirement to consider not only RCT data but also other clinical and contextual information such as health economics and real world data from different healthcare settings. Although the rigorous methodology adopted by Cochrane is still considered the ‘gold standard’, it needs to continuously innovate to stay cutting edge. It may be that Cochrane does not currently have all the relevant expertise among its methodologists to analyse and combine diverse data, so Cochrane either need to develop those capabilities internally or perform such studies in partnership with other specialised organisations. In either case, there currently appears to be a gap in terms of expectations of external stakeholders and Cochrane’s current practices.

A very large proportion of survey respondents indicated that Cochrane evidence is easily accessible and usable which goes to show the results of the many years of effort Cochrane has invested in making its evidence user-centred. Nevertheless, shorter structured reviews with narrative synthesis, graphical content, contextual information and emphasis on the interpretation of the evidence (including metrics used by practitioners) would further enhance the usability of Cochrane evidence.

Cochrane’s recent efforts to develop and publish its first Priority Review List across all areas was welcomed by interviewees, however its existence was not widely known in the community. Priority setting should not only be an internal exercise for Cochrane Review Groups but involve diverse stakeholders’ input (i.e., consumers, clinicians, funders, and guideline developers) on their needs, objective data on burden of disease and healthcare, and identified gaps in published evidence. Encouragingly, there was a significant interest from survey respondents and interviewees to engage with Cochrane on this important task.

Setting up partnerships with other organisations across the stakeholder spectrum would therefore help inform priority setting processes, eliminate potential duplication of efforts, contribute to wider dissemination of the Cochrane evidence and ultimately serve as an important channel for advocacy. Advocating for evidence has only recently been part of Cochrane’s strategy and hence one cannot expect high-level awareness in the community and substantial impact on the ground. Nevertheless, Cochrane is already recognised to promote


evidence-based medicine effectively to research funders, healthcare policy makers, and clinical practitioners. If Cochrane wants to focus on advocacy more in the future, it would need to learn from other (sometimes much younger) organisations that campaign effectively for integrity and transparency of clinical research.

Cochrane’s planned transition to a (non-fee paying) membership-based organisation was considered a good one, with the potential to enhance engagement of diverse stakeholders with Cochrane, open up new dissemination channels, and strengthen Cochrane’s support base. Highest rated features of a membership scheme were the potential for online training, learning about evidence-based medicine and access to Cochrane’s software and tools for dissemination and advocacy. Cochrane’s potential training initiatives on methodology drew considerable interest among external stakeholders, however it may prove more resource-efficient to target users of the evidence, including policy makers, thereby essentially advocating evidence-based medicine and influencing health policy at the highest levels.

Cochrane is a very large network of volunteers with relatively complex organisation structure, with centres and branches, fields and networks, methods and review groups all part of the Cochrane system. Although the structure has proved somewhat fuzzy for most respondents and interviewees, the two major dimensions, geographical (and hence socio-cultural) regions and disease area/ clinical specialisation, appear to work reasonably well. About 70% of respondents were aware of the local Cochrane Centre and close to two thirds of these felt that it allows them to engage with Cochrane. However, one of the biggest challenges for Cochrane is to reduce variability of approaches and processes across its review groups and centres and make those more efficient to produce consistently relevant, high quality and timely systematic reviews. This should not mean to impose uniformity on the entire network but to introduce unity and consistent identity toward the outside world.

Finally, a central feature of Cochrane was considered to be independence from any conflict of interest and thus a trusted source of impartial medical evidence for the community. In this respect, Cochrane has been devoutly stressing independence from the pharmaceutical industry but it would also need to find ways to screen for intellectual conflicts of interest among its contributors worldwide.

As one interviewee has put it, now is the time for Cochrane to implement its new Strategy to 2020, and therefore a number of key recommendations have been made by the study team, based on the data collected during the study:

1. Review the skills, strengths and weaknesses (and its distribution) of current Cochrane methodologists and contributors to better prepare for users’ needs.

2. Accelerate adoption of new methods and/or recruit and reward innovative methodologists.

3. The process of priority setting has to be transparent and must target research questions where there is high potential to change practice.

4. Priority setting must involve a range of stakeholder representations (i.e., health professionals, consumers, funders, policy makers and guideline developers) through workshop and/or using online list. Systematic use of priority setting partnerships with organisations such as the James Lind Alliance would be beneficial.

5. Systematically grade all data reviewed and score resulting recommendations for practitioners accordingly.

6. Optimise Cochrane’s resources by focussing on smaller number of complex reviews.

7. Consider evidence as a ‘living document’ and update reviews if and when new data become available in an area of unmet needs.

8. Increase the number of rapid reviews with concise summaries (translated to Spanish, Chinese and French) and data relevant to practitioners.

9. Increase involvement of patients and consumers in the entire research cycle and Cochrane’s governance.


10. Communicate Cochrane evidence through the relevant channels – customised according to target group (commercial media channels, partnerships with medical journals, briefing papers for policy makers with, using professional and community networks, high profile bloggers, etc).


Appendix A External stakeholder survey results

The multi-lingual survey (English, French, German, and Spanish) was implemented in SurveyMonkey and was sent to 480 contacts from a database constructed for this project using information from Cochrane, Technopolis expert network, and other public information, after approval from Cochrane. The survey was open to record responses between 20 April and 31 May 2015. A total of 452 individuals responded to the survey (English: 364, French: 33, Spanish: 44: German: 11). Of these, 379 respondents provided answers beyond the profiling section; only these were included in the survey analysis. All graphs in this report show % of all who responded to the specific question; data labels within the chart or in parentheses within the text (unless labelled otherwise) indicate the actual number of respondents.

For some questions, answers were converted into numerical values, assigning the following values:

• 1 to ‘not at all important’, ‘not at all effective’, ‘not at all influential’, ‘not at all interested’, ‘strongly disagree’;

• 2 to ‘somewhat important’, ‘somewhat effective’, somewhat influential’, ‘somewhat interested’, ‘disagree’;

• 3 to ‘important’, ‘effective’, ‘influential’, ‘interested’, ‘neither agree nor disagree’;

• 4 to ‘highly important’, ‘highly effective’, highly influential’, ‘highly interested’, ‘agree’;

• 5 to ‘strongly agree’.

The values were multiplied by the number of responses, summed up and divided by the total number of responses. ‘No view’ responses were not included in the analysis.

A.1 Survey characteristics and respondents Figure A11 What country do you live in? (n=377) UK 38% (145)

United States 9% (34)

France 8% (31)

Canada 5% (18)

Germany 5% (17)

Switzerland 4% (16)

Chile, Spain 3% (13)

Australia, Belgium, Portugal, Argentina, Italy, Croatia, Cuba

2% (6-9)

Brazil, India, Finland, Mexico, Netherlands

1% (3-4)

Costa Rica, Denmark, Greece, Japan, South Africa, Austria, Bosnia Herzegovina, China, Estonia, Ireland, New Zealand, Norway, Singapore, Sweden, Uganda, United Arab Emirates, Uruguay

0.5% (1-2)


Figure A12 What is your native language? (n=372)

Figure A13 Which of the below best describes your current organisation/employer? (n=378)

193!

42!

37!

31!

13!

10!

9!

0%! 20%! 40%! 60%!

English!

Spanish!

French!

German!

Dutch!

Portuguese!

Italian!

167!

63!

39!

29!

18!

12!

10!

40!

0%! 20%! 40%! 60%!

University!

Hospital!or!healthcare!facility!

Research!insGtute!

Government!/!Public!administraGon!

NGO!/!charity!

Research!funder!

Consumer!or!paGent!organisaGon!

Other!


Figure A14 Which of the below best describes your current position? (n=373)

Figure A15 Are you involved in or aware of Cochrane’s activities? (n = 379)*

*Note that multiple responses were allowed; answers do not add up to 100%.

A.2 Producing evidence that is relevant and up-to-date The majority of respondents (85%) use Cochrane evidence in their work or in decision-making on health (Figure A16). A large proportion (around 80%) also indicated that Cochrane evidence tackled the right topics, and answered questions that are well-framed and relevant.

122!

88!

20!

16!

14!

11!

9!

9!

3!

81!

0%! 10%! 20%! 30%!

Researcher!(non!clinical)!

Clinician!

InformaGon!specialist!

Policy!officer!

Research!funder!

PaGent!advocate!

Guideline!developer!

Health!service!manager,!commissioner!

PaGent,!service!user!and!carer!

Other!

95!

86!

119!

125!

255!

16!

0%! 20%! 40%! 60%! 80%!

I!am!/!have!been!affiliated!with!a!Cochrane!Group!

I!am!/!have!been!involved!with!Cochrane!as!a!contributor!

Colleagues!within!my!organisaGon!are!involved!with!Cochrane!

I!know!about!Cochrane!through!its!local!Centre!or!Group!acGviGes!

I!know!about!Cochrane!through!using!the!Cochrane!Library!or!its!website!

I!am!not!involved!in!or!aware!of!Cochrane!


However, 16% of respondents did not find Cochrane evidence sufficiently up-to-date. Within this group, clinicans were somehwat overrepresented: 31% of respondents were clinicians who felt that the evidence was not sufficiently up-to-date; they constituted 23% of respondents to this question. 44% of those who indicated that Cochrane methods did not meet their needs (see Figure A19) also felt that the data were not sufficiently up-to-date (compared to 16% of those who indicated that Cochrane methods met their needs).

Figure A16 Relevance and timeliness of Cochrane evidence (n = 374-379)

Survey respondents commented on a range of issues regarding the choice of topics for Cochrane evidence and the questions they addressed:

• Decisions on which topics to address:

A number of survey respondents were unclear about how Cochrane made decisions on which topics to address, and indicated that the selection appeared somewhat haphazard (6). One respondent pointed out that Cochrane encompassed a large number of groups, all of which appeared to have different approaches to topic selection. It was repeatedly noted that topic choice appeared to be researcher-driven, rather than user-driven, and that it was unclear if all stakeholders were involved in the process (specifically mentioned were carers and service users). One respondent was of the view that Cochrane “tackles a minority of major issues in the field”; another felt that he/she had “seen some dubious reviews of little clinical benefit and probable waste of resource”. One respondent noted that the prioritisation process was improving but that there was “a tension between topics being relevant to the NHS and Cochrane's global mandate”. Two respondents suggested a prioritisation for updates, e.g. by asking users to indicate which areas they most need updates on via the website.

Topics specifically mentioned as currently not addressed/ not sufficiently addressed were environmental health, medical devices, and health services research. One respondent also pointed out that there were currently no sources of information on systematic reviews for genetics and molecular medicine. More specifically, individual respondents were interested in the topics of emergency medicine (2); cardiac arrhythmias, oral health, anaesthesia, resuscitation and pain; mental health; living with disabilities; diagnosis of ADHD and treatment with methylphenidate, obesity in adults, and alternative medicines. One respondent would like to see “more 'research on research' reviews, along the lines of existing reviews, such as the analysis of selective reporting, or the implementation of CONSORT”.

327!

291!

306!

230!

47!

18!

14!

62!

5!

68!

54!

84!

0%! 20%! 40%! 60%! 80%! 100%!

Do!you!use!Cochrane!evidence!in!your!work!or!in!decision>making!on!health?!

Do!you!believe!that!Cochrane!evidence!tackles!the!right!topics!in!health!and!health!

care?!

Do!you!believe!that!Cochrane!evidence!answers!quesGons!that!are!well>framed!and!

relevant?!

Do!you!find!that!Cochrane!evidence!is!sufficiently!up>to>date?!



• Focus of questions:

Several survey respondents (5) explained that Cochrane reviews were too strongly focused on RCTs and drug/surgical interventions. Related to this view, one respondent pointed out that Cochrane would need to take account of the increasing volume of data becoming available along the entire ‘evidence cascade’, from basic research through to clinical practice. Another respondent felt that Cochrane work was too narrowly focused on clinical efficacy, effectiveness and safety, and failed to address key issues needed to inform health care funding, e.g. concerning value for money and ethics, legal, social and implementation issues. It was also noted that there was a “lack of equilibrium in quantity and specificity of reviews per topic”, and that sometimes questions were not relevant (“questions that no one in healthcare asks”; “questions a little academic and missing practical relevance”; “topics too narrowly focused”, “questions so obtuse that it is no surprise that there is insufficient published evidence to conduct a review”), and that questions predominantly addressed topics of developed (rather than developing) countries.

Other comments included that reviews were a “predictable aggregation of current evidence”, and that “some reviews just summarise silly outcomes in a silly way”. Several respondents commented that they would like guidance on what to do in the absence of evidence for an intervention; one respondent called the frequent outcome of ‘inconclusive’ frustrating. This is supported by the comment of another respondent: “I feel that both the format and the questions could be more relevant at times. There has been some fantastic progress with this recently, particularly leveraging the Cochrane Consumers Group to influence the research question, but many researchers need to be confident with their interpretation as the standard 'more research is needed' is a decision to stick with the status quo, which may not be supported by what (limited) evidence is available.”

Regarding the timely updating of Cochrane evidence, the predominant view expressed in the available text box was that reviews were often but not always up to date (44), and many of these respondents (24) pointed to variation between different topics and groups. Several respondents (10) pointed out that even up-to-date reviews tended to lag behind, often by around 2 years, due to the slow, bureaucratic nature of the review process (both methodological and editorial). To address this issue, a number of respondents (3) would like to see further attention given to rapid reviews (versus comprehensive reviews) – they felt that while Cochrane methods represented the gold standard, there was a cost in that “real world decisions often cannot wait for a review that takes longer than 6 months to complete”. One respondent compared the ‘slow’ Cochrane update process to that of Up-to-date, which was felt to be much more reactive. Another respondent strongly recommended moving towards a 'living review' format, underpinned with (publically available) data for the meta-analysis.

A number of respondents (7) commented on a lack of clarity regarding updates: Three respondents felt that is was not always clear when a review was last updated, and how the update was performed, or that some reviews would not be further updated. Two others thought it should be made clear if a review is not updated because no relevant new evidence was available or because of editorial delays. Another respondent recommended including a note with some reviews, setting out recent developments in the field not reflected in the review, e.g. ‘this review was produced prior to the introduction of x-class of drugs’. The comment was made that “many Cochrane reviews are not updated for years, and many reviews (which could still be the most current views of the evidence) are withdrawn due to not being updated rather than clearly being wrong or known to be outdated.”

One respondent felt that the strict requirement on English language skills was an insurmountable barrier for some investigators to participate in review groups.

A.3 Priority setting and external consultation Survey responses to questions in this section indicated that knowledge of, and engagement in, priority setting processes is relatively low. 71% of respondents could


not comment if Cochrane was receptive to external views, and 84% had never been involved in priority setting (Figure A17). For example, neither respondents from Germany (16 respondents) nor Spain (13 respondents) had been involved in priority setting for Cochrane. 14% of respondents from the UK had been involved. Canada, with 28% of resopndents, had the highest participation of any country.

Despite the current low level of engagement, there was significant interest to engage in the future, as signalled by 51% of respondents.

Figure A17 Priority setting and external consultation (n = 364 – 375)

The majority of respondents who elaborated further on their answer indicated that they had not tried to engage with Cochrane to inform priority setting, or did not know how to engage, or that they had simply not had the opportunity to engage (30).

Respondents preferred to engage with Cochrane via workshops (46) and consultation on topic lists (55, some specifically preferred online consultation). Additional suggestions were Delphi rounds, including Cochrane representatives in meetings at other organisations, provision of feedback at the end of every guideline update cycle, and adopting the James Lind Alliance method. Specifically mentioned was the potential for collaboration with a number of existing initiatives, such as the Guidelines International Network (G-I-N), the Haute Autorite de Santé (HAS) in France, the Health Search database in Italy, and the Canadian Agency for Drugs and Technology in Health (CADTH, currently establishing a priority setting process for HTA). One respondent (Spanish-speaking) expressed the desire to contribute to the priority setting process (consumer group), but pointed out that she/he would only be able to do so in his/her native language.

When asked about patient, patient advocate and healthcare consumer involvement in the production of Cochrane evidence and in Cochrane activities (Figure A18), the various dimensions were all rated as ‘important’ on average [for scoring methodology, see start of this section], with ‘Setting priorities for research’ seen as the most important (score of 3.3, between ‘important’ and ‘highly important’) and ‘Co-production of research’ the least important (score of 2.6, between ‘somewhat important’ and ‘important’). Other dimensions had the following score: ‘Patients/ consumers are represented in the governance of Cochrane’ 3.0; ‘Patients / consumers are involved in interpreting the relevance of outcomes’ 3.1; and ‘Patients / consumers are involved in dissemination of research’ 3.2.

87!

47!

186!

22!

316!

178!

265!

12!

0%! 20%! 40%! 60%! 80%! 100%!

Based!on!your!experience,!is!Cochrane!recepGve!to!external!views!when!seRng!

prioriGes?!

Have!you!ever!been!involved!with!priority!seRng!for!systemaGc!reviews!in!Cochrane?!

Would!you!like!to!engage!with!Cochrane!on!seRng!its!review!prioriGes!in!future?!



Figure A18 Involvement of patients, patient advocates and healthcare consumers in the production of Cochrane evidence and in Cochrane activities (n = 371 – 374)

A.4 Cochrane’s methods 77% of survey respondents had an interest in evidence synthesis methods, and 71% felt that the methods used by Cochrane Reviews met their needs, while 16% did not feel this was the case (Figure A19). 62% were aware of Cochrane Methods groups.

2%!

3%!

12%!

3%!

6%!

14%!

22%!

34%!

20%!

21%!

34%!

37%!

35%!

33%!

36%!

48%!

36%!

16%!

41%!

31%!

2%!

4%!

4%!

3%!

6%!

0%! 20%! 40%! 60%! 80%! 100%!

SeRng!prioriGes!for!research!

InterpreGng!the!relevance!of!outcomes!

Co>producGon!of!research!

DisseminaGon!of!research!

Are!represented!in!the!governance!of!Cochrane!

Not!at!all!important! Somewhat!important! Important! Highly!important! No!view!


Figure A19 Cochrane’s methods (n = 375; n = 283-285)

Several survey respondents commented that Cochrane methods were too focused on randomised controlled trials (RCTs) (12), and that current methods did not meet needs beyond this type of data, e.g. qualitative data. However, one respondent felt that adopting a common practice within Cochrane that would combine data from RCT and non-RCTs was not appropriate.

A number of survey respondents (6) felt that Cochrane methods were often too rigid and rigorous, and that it was difficult to use innovative or ‘non-standard’ complex statistical methods, which was often required to most appropriately address a specific question. The point was also raised that there was a need to consider opportunity costs, e.g. for questions that could also be addressed through rapid review approaches.

Other respondents would welcome engagement in other types of synthesis, independently of effectiveness reviews and meta-analysis, e.g. qualitative and mixed methods, and guidance on methods for interrupted time series and observational data (these were comments throughout different parts of the survey; hence difficult to count). Two respondents flagged that this was particularly important for non-clinical questions / public health.

One respondent pointed out that the methods requirements were too resource-intensive for his / her research setting, another that the increasing complexity of Cochrane reviews presents a barrier for new reviewers to join (respondent from South America).

Further comments on Cochrane’s methods used included:

• too simplistic, “methods geared towards an audience with at best cursory familiarity with statistics and epidemiological methods”

• “out-of-date, as RevMan unable to carry out complex statistical analysis”

• “more detail is needed on how GRADE was applied to body of evidence”

287!

201!

178!

189!

46!

45!

107!

27!

42!

37!

69!

0%! 20%! 40%! 60%! 80%! 100%!

Do!you!have!an!interest!in!evidence!synthesis!methods?!

Do!the!methods!used!in!Cochrane!Reviews!meet!your!needs?!

Are!you!aware!of!the!various!methodological!groups!that!comprise!Cochrane!Methods!and!support!method!development!for!systemaGc!

reviews?!!

Do!you!believe!Cochrane!has!high!enough!profile!to!aYract!the!best!methodologists!for!

its!SystemaGc!Reviews?!!

Yes! No! No!view!


• “too rigid and too slow”

• “need to supplement with STATA, e.g. meta-regression and EGGER test

• “data on within-group changes in many meta-analyses is absent; and dichotomous variables (e.g., response rates) are used even when continuous variable data are available. This reduces power and can distort outcomes”

• “The conclusions from studies with high or low risk of bias should be reported separately – it does not make sense to first categorise, and then to combine studies anyway as if they carried equal weight.”

Regarding Cochrane’s reputation and ability to attract the best methodologists, 66% of respondents believed that Cochrane had a high enough profile to do so (Figure A19); however, a large propotion had no view on this question (24%), and 9% believed the profile was not sufficiently high. (it was tested but there was no obvious clustering of respondents for this answer category).

Survey respondents detailed a range of views with regards to this question, with many acknowledging the high profile Cochrane holds in the community. While some respondents agreed that top methodologists joined Cochrane, they also pointed out that this could be limited to those who worked within the scope of methods that are currently applied, but may not be the case for other methods, e.g. most relevant to observational data, and more innovative methods. As one respondent put it: “There is a view that the Cochrane way is the only way - why would that attract people interested in developing other ways? I believe that the organisation does attract excellent methodologists but it is difficult to see their views and experiences being incorporated into standard Cochrane guidance or indeed for a unified approach to different methodological views across groups.”

Several respondents (3) pointed out that there existed an ‘us vs. them’ mentality, and that methodologists outside the Cochrane community often viewed it as a ‘closed shop’. Another respondent felt that the top methodologists preferred primary research.

In response to the question “What might attract the best methodologists to join Cochrane’s evidence synthesis instead of other applied research areas?”, survey respondents expressed a range of factors that could be improved. The various points are summarised in Figure A20.

Figure A20 What might attract the best methodologists to join Cochrane’s evidence synthesis instead of other applied research areas?

“Attractiveness” factors Number of respondents

Reputation / Quality of outputs 21

Demonstrated impact on health care / policy 14

Interesting questions / methodologies 17

Salary / stipends 19

Authorship / recognition of contributions 13

Collaboration opportunities and technical support 11

Less heavy review process / faster process 6

Increasing awareness of the Cochrane network / clarity of how to approach and participate

11

A.5 Accessibility and usability of Cochrane evidence Around 75% of respondents indicated that Cochrane’s evidence was easily accessible and usable, compared to around 15% for whom it was not (Figure A21).


Figure A21 Accessibility and useability of Cochrane evidence (n = 363; n = 358)

Regarding accessibility, some survey respondents (20) pointed out that Cochrane’s reviews were not open access, and that hence they could only access abstracts as their institutions did not have a subscription (i.e., this included respondents from Brazil, French hospitals (2), Portugal NHS, Canada, USA). One respondent thought that the web interface was complex; others commented that it was unclear what the best way was to locate Cochrane publications.

Regarding usability, some respondents commented that reviews were presented in an overly long, complex format (21), and four respondents pointed out a need for more plain language. Other respondents asked for shorter and clearer summaries (with suggestions for a format similar to Critical Appraisal Topics, or a Wiki-Cochrane), inclusion of a summary table of the studies’ patient characteristics, and more tools to assist in the interpretation of the results. One respondent suggested Summary of Findings tables and Plain Language Summary to help quick decisions whether a systematic review was relevant; another asked for a consistent structure across reviews to make it easier to locate specific information (e.g. risk of bias tables).

Twelve respondents felt the evidence was not presented in a way that was relevant to clinical/ practical application, or relevant to non-medical professionals. Called out in particular were ambiguous conclusions which could not be used by practitioners, e.g. “in cases where the evidence permits, clearer clinical recommendations should be put forward, based on existing evidence - even if these might change when new data is incorporated in the future. This would make the reviews much more useful for (and used by) clinicians”.

One respondent pointed out that the format Wiley offered was much better than the open library format.

Three respondents commented positively on editorials and special collections and the summaries that the Cochrane Consumers and Communication Review Group provide.

282!

270!

53!

48!

28!

40!

0%! 20%! 40%! 60%! 80%! 100%!

Is!Cochrane’s!evidence!easily!accessible!to!you?!

Is!Cochrane’s!evidence!easily!usable!to!you?!

Yes! No! No!view!


When asked about the importance of a range of factors relating to accessiblitiy and usability, respondents rated all as ‘important’ for Cochrane evidence, with ‘Open access’ scoring as ‘highly important’ (score of 3.8, with ‘highly important’ scoring 4).

Figure A22 Please rate the importance of the following factors relating to accessibility and usability of Cochrane evidence. (n = 361 – 363)

When asked about language preferences for Cochrane publications, almost all survey respondents (97%) indicated that they were comfortable using English to read Cochrane Reviews (Figure A23). However, 38% of respondents who answered ‘yes’ or ‘no’ (rather than ‘not applicable’) thought they would use Cochrane evidence more often if it were available in their native language. These views varied between different countries and languages: while more than half of all native French and native Spanish speakers indicated they would use Cochrane evidence more often if it were available in their native language (61% and 59%, respectively), a lower proportion of native Portuguese (25%), Italian (22%), and German (16%) held this view..

Overall, 75% of survey respondents who responded ‘yes’ or ‘no’ (rather than ‘not applicable’) thought that others in their region would use Cochrane evidence more often if publications were translated into the national language. Broken down by language, native speakers of French (95%), Spanish (94%), Portuguese (85%), and German (79%) held this view, as well as very high proportions of respondents living in France (97%, n=30), Germany (94%, n=16), Spanish-speaking countries (93%, n=43; this includes Argentina, Chile, Costa Rica, Cuba, Mexico, Spain, and Uruguay), and Portuguese-speaking countries (92%, n=12, Portugal and Brazil).

1%!

5%!

1%!

5%!

2%!

17%!

8%!

14%!

12%!

36%!

40%!

35%!

84%!

38%!

49%!

45%!

1%!

4%!

2%!

1%!

0%! 20%! 40%! 60%! 80%! 100%!

Open!access!

Customised!search!facility!for!different!audiences!

PresentaGon!format!and!layout!of!the!reviews!

Language!adapted!for!different!audiences!(scienGfic,!guideline!makers,!consumers)!



Figure A23 Language-related questions (n = 359 – 362)

Translating sections of Cochrane publications

When asked to rate the importance of translating various sections of Cochrane Reviews, survey respondents gave an average rating of ‘important’ (represented by a score of 3) to all sections of Cochrane publication (Figure A24). Translation of the abstract received the highest score (3.3), followed by the plain-language summary and the conclusions (both 3.1). Tranlsation of the background/objectives and methods sections were seen as having lower importance (scores: 2.6).

Overall, English emerged as the top priority language (29% of respondents), followed by Spanish (26%), Chinese (19%) and then French (13%) (Figure A24). This was consistent when responses were broken down in order of choices: the majority of respondents designated English as the language of top priority (70%, n = 313), followed by Spanish (43%, n = 295). Chinese or Spanish tied in third place (28% and 26% respectively, n = 277).

350!

67!

154!

12!

109!

51!

185!

154!

0%! 20%! 40%! 60%! 80%! 100%!

Are!you!comfortable!using!English!to!read!Cochrane!Reviews?!

Cochrane!evidence!were!increasingly!available!in!your!naGve!language,!would!you!

use!it!more!oben?!!

Do!you!think!others!in!your!region!would!use!Cochrane!evidence!more!oben!if!they!were!

translated!into!the!naGonal!language?!

Yes! No! Not!applicable!


Figure A24 Please rate the importance of translating the following sections of Cochrane Reviews for you. (n = 349 – 355)

Figure A25 In your view, which top 3 languages should Cochrane prioritise? (n = 313)

* equal weighting given to all 3 choices

When asked what additional changes would help Cochrane become an accessible, useful and trusted 'Home of Evidence' for health, most respondents (22) thought that outputs needed to more clearly presented and readable, tailored to the audience (e.g. clinicians), with well thought-through implications and information on how to use the information (e.g., by including a “section describing the ideal trial that would address the issue investigated”). Other respondents (17) were in favour of strengthened communication activity to raise the profile of Cochrane more broadly, including

11%!

13%!

15%!

16%!

16%!

15%!

13%!

6%!

10%!

19%!

22%!

13%!

17%!

9%!

14%!

15%!

26%!

22%!

23%!

26%!

18%!

49%!

43%!

19%!

19%!

27%!

22%!

41%!

20%!

20%!

21%!

21%!

21%!

21%!

19%!

0%! 20%! 40%! 60%! 80%! 100%!

Abstract!

Plain>language!summary!

Background!/!objecGves!

Methods!

Results!

Discussion!

Conclusions!


251!

231!

169!

115!

34!

30!

3%!

2%!

0%! 5%! 10%! 15%! 20%! 25%! 30%! 35%!

English!

Spanish!

Chinese!

French!

Arabic!

German!

Portuguese!

Russian!


through engagement with students, and recognition by government institutions. Specific suggestions for improving ease of use included:

• production of podcasts summarising the latest reviews in 5-10 min,

• faster updating of the iPad version,

• linking the Cochrane review information to easily accessible web-tools/Apps (3) and other websites providing access to evidence (e.g.: “Additional information to know when valid evidence is available that has not been incorporated yet into the Cochrane review, such as links to DynaMed (for curated summaries), and links to MEDLINE with Cochrane search strategy and filter set to items after search date of Cochrane review search (for raw data for further consideration)”),

• improving the search options (e.g. indexed keywords) (7), and

• developing a common web portal for independent analyses networks similar to that of the Haute Autorité de Santé (HAS).

It was also mentioned that Cochrane should make all raw data/ versioned scripts and tools available. One respondent felt that Cochrane needed to be better represented globally, with representatives in each country speaking the local language (respondent from Chile).

A.6 Cochrane’s advocacy for evidence When asked to rate the effectiveness of Cochrane’s advocacy activities and the Cochrane campaign for transparency and integrity in scientific conduct, survey respondents considered these to be less than effective (scores of 2.7 and 2.8, respectively, with ‘effective’ represented by a score of 3) (Figure A26).

Figure A26 Based on your experience in your region or in your field of speciality, please rate the effectiveness of the following activities of Cochrane (n = 344 – 346)

Most of the survey respondents who explained their answers pointed out that they did not know about these activities or that Cochrane was relatively unknown in their field/ country. Four respondents pointed out that Cochrane had a relatively low profile in some of the current debates, e.g. transparency and open data access. Two respondents commented that the work of the Iberoamerican Centre was very impressive and proving effective; the limitations were not the Centre, but the environment it operated in, citing inappropriate influence of industry on clinical practice, and policy makers’ low level of understanding of evidence-based medicine. This view was mirrored by two other respondents from South America, who pointed out that corporate activities lacked transparency. For this reason, it was suggested that Cochrane reviews should clearly state the source of funding for RCTs included in the analyses.

6%!

3%!

22%!

21%!

34%!

39%!

12%!

15%!

25%!

22%!

0%! 20%! 40%! 60%! 80%! 100%!

Cochrane’s!campaign!for!transparency!and!integrity!in!scienGfic!conduct!

Cochrane’s!engagement!in!debates!advocaGng!for!the!use!of!evidence>based!

methods!and!high!quality!evidence!

Not!at!all!effecGve! Somewhat!effecGve! EffecGve! Highly!effecGve! No!View!


Respondents rated Cochrane’s influence in promoting evidence-informed decision making between ‘influential’ (score of 3) and ‘somewhat influential’ (score of 2) with regards to various stakeholders (Figure A27). Clinical practitioners were considered to most influenced (score: 2.9), followed by research funders (score: 2.8), healthcare policy makers (score: 2.7), health service managers (score: 2.4), and consumers or patients (score: 2.0).

Figure A27 Based on your experience, please rate Cochrane’s influence in promoting evidence-informed decision making for the following stakeholders (n = 340 – 344)

In response to the question of how Cochrane could have more impact on healthcare policy and practice, respondents highlighted a number of opportunities. These included the following categories:

• Funding: Respondents (6) suggested either coordinating selection of topics and priority setting with the main funders, e.g. the NHS and research councils; or for Cochrane to provide independent funding.

• Uptake of findings: Various approaches to linking with policy makers were put forward. These included offering services directly to policy makers, involving policy makers in the Cochrane priority setting process, partnering with policy makers to conduct reviews, participating in ministerial/government meetings (particularly from Spanish-speaking respondents), offering training/workshops for policy makers, or engaging with professional associations in health who could then lobby policy makers. It was also suggested that Cochrane engage with guideline developers, such as NICE and AHRQ. One respondent commended Cochrane’s connection to policy makers, especially through the inclusion of the GRADE approach.

In addition, respondents felt that Cochrane outputs needed to be adapted to the needs of policy makers and clinicians by making them more concise. This included the suggestions of a) providing a comparison with current healthcare guidelines of other countries to assist policy makers, and b) presenting results in terms of NNT (number-needed-to-treat), rather than only group comparison to support clinicians. One respondent also proposed more tools and guides on how to implement conclusions in policy and practice.

• Data: Some respondents felt that integrating different types of evidence, i.e. beyond clinical data, was necessary to impact policy. Suggestions were made to partner with organisations such as the HTA, INAHTA, or NICE to allow bringing in this information.

3%!

8%!

11%!

27%!

7%!

24%!

27%!

34%!

32%!

22%!

37%!

32%!

21%!

16%!

34%!

23%!

17%!

11%!

5%!

19%!

13%!

17%!

24%!

20%!

19%!

0%! 20%! 40%! 60%! 80%! 100%!

Clinical!pracGGoners!

Healthcare!policy!makers!

Health!service!managers!

Consumers!or!paGents!

Research!funders!

Not!at!all!influenGal! Somewhat!influenGal! InfluenGal! Highly!influenGal! No!View!


To further inform implementation, one respondent recommended linking the results of a review with observational research on the impact this has in practice. Such a link would also allow identification and tracking of ‘success stories’, which would bolster communications and increase the visibility of Cochrane. Two respondents would like to include health economics in Cochrane studies, to provide cost comparison information, as they felt this was a key area for policy makers.

• Communication and awareness: Several respondents thought that increased visibility of Cochrane would enhance its impact on policy, e.g. through stronger links with the media. Cochrane findings could also be disseminated through weekly/monthly short reports on key reviews with relevance to healthcare for publication in journals of local medical associations, as well as via information services like MedScape. Other formats mentioned included briefing papers to commissioners, press releases to consumers, workshops with health professionals, and social media e.g. twitter/kudos. It was suggested that Cochrane groups should include designated individuals responsible for dissemination activity.

• Other suggestions included:

− Patient reviewers, as per BMJ − Enhanced clinician and patient engagement − E-learning courses for policy makers to increase health literacy of decision

makers (and hence enable use of Cochrane findings) (2) − Evidence updates and rapid reviews − Target areas of need in developing countries, e.g. public health − “Establish a contact person for each country whose job it is to be a known

front person who can solidify relationships with key personnel in government (this would not be the most senior person but rather a liaison officer type role)”

− Expert training for authors and review group staff to devise specific dissemination plans

− Improving research recommendations so that these can be commissioned by funders

Two comments in this section questioned the Cochrane’s key target audience. One respondent asked: “What are exactly Cochrane's aims? To influence health policies or to help health care professionals in making evidence-based decisions? These are not the same objectives at all. Concerning the impact on health policies, Cochrane have to keep close to institutional organisations (WHO, EUPHA, etc.) and regional ones (see for example the way the NIVEL University is close to European DG Sanco and OECD leaders to get future research orientations). Concerning health care professionals, Cochrane may participate in the initial training of health care professionals and in congresses.” The other recommended: “Focus on 1 stakeholder group. [Don’t be] all things to all people.”

When asked about other provider(s) available to obtain high-quality systematic reviews in support of decision-making needs in health, respondents provided a long list of options (Figure A28). Most frequently cited were reviews published in medical /scientific journals (36), followed by NICE (19).

Figure A28 Sources of high-quality systematic reviews in support of decision–making needs in health

Medical / scientific journals

36 Joanna Briggs, TRIP database (all 3)

Epistemonikus, Campbell Collaboration, GIN, NIH, EBCTCG, Dare, Dynamed, NHS website (all 2)

In-house research; ECRI; Wikipedia; TARTeams; Bazian; Healthevidence.org; Healthsystemsevidence.org; Sax Institute; PEN (online nutrition resource, Canada;

NICE 19

HTA groups 14

Commissioned work (academic

8


groups / independent consultancies

professional associations); PDQ-Evidence; Fisterra; PROSPER; Tripdatabase; King’s Fund; Public Health Agency website; IQWIG; Clinical Knowledge summaries; Google Scholar; WHO bulletin; SPIP (Belgium); CMA; IHME; NHG standaarden; US National Guidelines Clearinghouse; SumSearch; CAT; MacMaster; BMJ DTB, BMJ Clinical Evidence, Cancer Drug Fund, CRUK-NCIN, NHS

Up-to-Date 7

SIGN 6

AHRQ 4

Very few comparisons between these sources and Cochrane were provided. They included:

• UpToDate: “UpToDate is much more useable in a clinical setting than Cochrane reviews”; “primarily addresses issues of internal medicine”; “more up to date and addresses very specific questions, but the review is less rigorous than Cochrane’s (same for Dynamed)”; “more accessible and useable in a clinical setting, including in areas of uncertainty (same for Fisterra)”

• TRIP Database: “in addition to evidence-based medicine, also includes clinical guidelines, question-answer services, electronic textbooks, medical imaging, patient information bulletins, articles indexed journals”

• SUMSearch: “provides a mixture of systematic reviews and original articles”

• CATS: “more practical, but requires a methodological improvement”

• Non-Cochrane reviews in medical journals: “sometimes have a more global perspective that better answer clinical questions”

• HTA assessments: “include aspects not covered by Cochrane”

• Epistemonikus: “offers a matrix of systematic reviews”

A.7 Cochrane’s organisational structure and partnerships 42% of survey respondents were aware of and/or interacted with Cochrane groups globally, and 69% were aware of the local presence of Cochrane in their region or country (Figure A29). 73% of those who were aware of Cochrane’s local presence felt this allowed them to engage. 33% of respondents’ organisations had a partnership with Cochrane, whereas 38% did not (and in 30% of responses, the individuals did not know).

There was substantial variation between countries: While 77% of respondents from France and 69% from Spain indicated they were not aware of or interacted with Cochrane globally, this proportion was around 50% for the UK, Chile and Germany, 44% for the US, and 17% for Canada.

However, all respondents from Spain were aware of a local presence, and only 6% of respondents from Canada were not aware (compared to 24% from Germany, 31% from Chile, France and the UK, 44% from Switzerland, and 47% from the US).

It is notable that France had the highest proportion of respondents who indicated that Cochrane’s local presence did not allow them to engage (63%, 12 of 19 respondents living in France who indicated they were aware of Cochrane’s local presence.) This was also the case for 50% of respondents from Italy, but only a small number of responses to this question were received (n = 6).


Figure A29 Awareness and views on Cochrane’s local presence (n = 335 – 337, 223)

When asked who they had interacted with, respondents mentioned a broad range of Cochrane groups:

• Anesthesia • Applicability and recommendation methods • Australasian Cochrane Centre • Back review group • Back pain, OHS, Pain • Brazilian Cochrane Centre • Centro Iberoamericano • Child Cancer group • Cochrane Canada • Cochrane Croatia • Cochrane France • Cochrane Heart Group • Cochrane Oral Health Group (2) • Consumer group/network (4) • Cystic Fibrosis, Stats methods • Dementia and Cognitive Improvement Group • Drugs and Alcohol • Dutch Cochrane DTA Group • Effective Practice and Organisation of Care Group • Emergency Medicine Group • Endocrine and metabolic group • EPOC (6) • Haematology group • Hepatobiliary • HIV/AIDS • Incontinence group • Infectious diseases review group • Italian Cochrane groups • Metabolic disorders • Movement disorder group (2) • Muscoskeletal group • Neonatal • NRS, Rapid Reviews • Nursing priorities • Papas (2)

142!

234!

162!

110!

193!

103!

61!

126!99!

0%! 20%! 40%! 60%! 80%!

Are!you!aware!of!and/or!interact!with!Cochrane!Groups!globally?!

Are!you!aware!of!the!local!presence!of!Cochrane!in!your!region/country?!

If!Yes,!does!the!local!presence!of!Cochrane!allow!you!to!engage!with!

Does!your!organizaGon!have!a!partnership!with!Cochrane!(either!formal!



• Paris 5 Ravaud • Patient-reported outcomes group • Pregnancy and childbirth (2) • Pre-hospital care • Primary care group • Prognosis/diagnosis (2) • Public Health • Renal • Stroke (2) • UK Cochrane Centre (2) • Wounds group Very few respondents answered the question of how their local Centre could better meet your needs. Suggestions included enhanced local dissemination of information (10), interaction with public and patients (2) offering courses/meetings (1), and appointment of an in-country Cochrane representative (1). However, this question brought up a number of critical comments relating to unhelpful attitudes of some local Cochrane groups:

• “[…], it is not clear whether Cochrane’s groups want to support systematic reviews. I asked once about making a systematic review and I was answered that it was a long and difficult process. So I gave up the idea....”

• “I’m not sure. I haven’t found it easy to become involved in reviews as many are ‘reserved’ by existing teams and it sometimes seems like a closed shop.”

• “I get the impression that participation with Cochrane in my country is that of a small group who attend Cochrane congresses, while those who conduct the reviews work in isolation.”

• “In my country, participation centres around one university, and it is not possible for others to participate.”

• “Put up speakers when requested instead of declining each time.”

• “University politics, if people are in they are very protective not to share.”

Respondents expressed a range of views when asked to provide further detail on the value of their institution’s partnership with Cochrane. While some respondents reported that the affiliation was “really good for the University’s reputation and impact”, that the “partnership with Cochrane maintains a channel of communication between our profession and evidence based health care”, or that the partnership was a “great help with search strategies and the process of performing systematic reviews for relevant disease areas”, another respondent commented that “I don’t get much value from the partnership with Cochrane. The meetings are more information sharing and not really an opportunity to inform priorities.” Two respondents noted that their institutions’ links with Cochrane was good for driving local initiatives, such as training, but did not lead to an engagement with the broader, global Cochrane priorities and activities.

Suggestions to strengthen the relationship included: “Cochrane has a high standing, but funding for Cochrane work is always somewhat uncertain. Perhaps more visible acknowledgements from “central” Cochrane (that our funders saw and heard) might increase the prestige of our “local” Cochrane work.” Another respondent commented: “Some Cochrane review groups value me enough to pay for my input which is great – all other work via [CCNet and Methods Group] is given free. What really bugs me […] is the lack of feedback on reviews where I have had input and the short lead time for that input.”

When asked what partnership Cochrane could establish in their country or region to support better health decisions, respondents put forward a number of general suggestions (e.g. enhanced engagement and dialogue with policy makers, guideline development offices, health authorities, practitioners in the field, national scientific


societies and physician bodies, teaching hospitals, universities that are not host to the lead Cochrane group).

Two comments focused on patients and consumers, recommending partnering with INVOLVE, bloggers, and individuals who are already summarizing and commenting on reviews. Mentioned were enhanced media outreach and public engagement, e.g. through provision of accessible websites in local languages.

A partnership with the Sylvia Lawry Centre was suggested, to provide evidence based decision support system in Multiple Sclerosis (EBDiMS), including safety information. Another potential partner mentioned was Health Search, “a database covering a total of 1.5 million patients”.

Respondents of the survey in Spanish highlighted increased meetings, links, and agreements with national governments as well as international organisations such as the Organizacion Panamericana de la Salud (e.g., support for EvipNET), UNASUR, ORAS-CONHU, and Mercosur. There were also calls for continued expansion of the Latinamerica and Caribbean networks.

A summary of recommendations for potential partner organisations in specific countries is presented in Figure A30.

Figure A30 Potential partner organisations for Cochrane, by country

Country Organisations

Australia - Australian Commission for Safety and Quality in health care, certain areas of the Department of Health (Cmwth) i.e. MBS, PBAC, enHealth

- Health professional organisations (Dietitians Association of Australia; Australian Medical Association; Public Health Association of Australia)

France HAS/ANSM; the French academies; La Revue Prescrire, the bulletin BIP 31 of the ISDB (International Society of Drug Bulletins), Ministry of Health, CISS (Collectif interassociative pour la santé)

Germany GB-A (Gemeinsamer Bundesausschuss), BMG (Bundesministerium fuer Gesundheit), GKV (Gesetzliche Krankenkassen, BÄK (Bundesaerztekammer); IQWIG (Institut fuer Qualitaet and Wirschaftlichkeit im Gesundheitswesen), Sylvia Lawry Centre

Italy Drug agency / Ministry of Health; universities (include evidence synthesis in medical education programmes); Health Search of the Italian College of General Practitioners

Mexico Universities

New Zealand Ministry of Health

Netherlands Health Council, Insurance Board

Switzerland Swiss Academy of Medical Sciences, the Swiss School of Public Health and the Ecole Romande de Santé Publique, the different University Hospitals, the “sécurité des patients” association, the ANQ, the OFSP

UK NICE, HTA, INVOLVE

USA US government agencies such as AHRQ, NIH, FDA, USPSTF; JAMA, American College of Physicians, American Geriatrics Society, C-PATH, ABIM/ChoosingWisely.org.

South America

Organizacion Panamericana de la Salud (e.g., support for EvipNET), UNASUR, ORAS-CONHU, and Mercosur.

International INAHTA


A.8 Cochrane membership scheme When asked to rate the extent to which various features and benefits of a new Cochrane membership scheme would be of interest, respondents gave an average rating of between ‘interested’ (score: 3) and ‘somewhat interested’ (score: 2) (Figure A31). The highest rated were access to Cochrane online training (score: 3.0) and access to Cochrane software, tools for dissemination and advocacy, learning about Cochrane and evidence-based medicine, and undertaking continued professional development with Cochrane accreditation (scores: 2.9). Involvement in crowd-sourcing projects (score: 2.4) was of least interest.

77% of respondents would consider becoming a Cochrane member (as an individual or as an institution) in Cochrane’s new membership scheme (n = 322). Few explained their answer –the main comments provided were that they were already members of a Cochrane group, that membership would not be a priority, or that they were too busy to join another organisation. Two individuals mentioned that they would not be willing to pay for membership. One respondent from industry, and one respondents from a funder pointed out that membership may represent a conflict of interest to them.

Figure A31 Please rate the extent to which the following features and benefits of a new membership scheme would be of interest to you. (n = 324 – 329)

On average, respondents agreed with statements on Cochrane’s structure and governance (scores between 4.3 and 3.7, where a score of 4 represents ‘agree’ and a score of 3 represents ‘neither agree nor disagree’) (Figure A32). They were most in agreement with the statement that Cochrane is an independent organisation (score: 4.3), and that Cochrane provides useful evidence for their health-decision making needs (score: 4.2). All other statements received an average score of 3.7 or 3.8; it is

19%!

12%!

10%!

10%!

20%!

12%!

17%!

7%!

9%!

12%!

28%!

28%!

19%!

21%!

26%!

34%!

30%!

22%!

18%!

20%!

24%!

32%!

34%!

33%!

23%!

28%!

23%!

32%!

31%!

26%!

20%!

23%!

31%!

30%!

16%!

16%!

20%!

31%!

35%!

33%!

8%!

6%!

6%!

6%!

15%!

10%!

10%!

7%!

6%!

8%!

0%! 20%! 40%! 60%! 80%! 100%!

Building!a!personal!Cochrane!profile!

Access!to!a!Cochrane!virtual!community!for!networking!and!forums!

Accessing!Cochrane!sobware!

Accessing!tools!for!disseminaGon!and!advocacy!

Involvement!in!crowd>sourcing!projects!

Tailored!communicaGons!

Discounts!at!events!

Learning!about!Cochrane!and!evidence>based!medicine!

Accessing!Cochrane!online!training!

Undertaking!conGnued!professional!development!with!Cochrane!accreditaGon!

Not!at!all!interested! Somewhat!interested! Interested! Highly!interested! No!View!


however notable that a large proportion of respondents had no view on these features. It is possible that some respondents interpreted the rating option ‘neither agree nor disagree’ as ‘no view’ which was a distinct option in the survey. In that case, it would appear that many respondents would not know how to rate these statements.

The lowest average scores were received by the statements “I would know how to become involved in Cochrane” and “I would be prepared to be involved in the governance of Cochrane in the future” (3.4 and 3.3, respectively).

Figure A32 Based on your experience, please indicate the extent to which you agree with the following statements (n = 328 – 331)

The majority of respondents (74%) indicated that they would be interested in Cochrane providing free training initiatives for their organisation (e.g., training on how to interpret Cochrane evidence to make informed health decisions) (n = 326). Those who elaborated further were interested in methodological training on the design and interpretation of systematic reviews and meta-analyses (46), with some respondents (4) asking specifically for online training to increase participation. The importance of training tailored to users of reviews was also highlighted, and included specific mention of training for clinicians/ researchers new to SRs/ nurses, physiotherapists, dentists; and even a request for an ‘Idiot’s Guide to Cochrane reviews’. Respondents were also interested in training on evidence-based practice (16), on how to use and contribute to Cochrane (6), using software (2), evidence bias rating (3), using reviews in policy decision making (2), reporting standards (2), standards for plain language translation, quality assessment and critical appraisal, and evidence search/literature review and analysis (11). Mentioned were also an accreditation scheme for members/other organisations, an award scheme for outstanding group members, and training for individuals to promote use of evidence in decision making.

18%!

15%!

6%!

16%!

17%!

18%!

18%!

9%!

15%!

26%!

45%!

37%!

32%!

28%!

29%!

51%!

34%!

28%!

40%!

13%!

10%!

11%!

12%!

30%!

15%!

12%!

7%!

29%!

38%!

36%!

35%!

8%!

11%!

14%!

0%! 20%! 40%! 60%! 80%! 100%!

Cochrane!is!an!independent!organizaGon!

Cochrane!is!run!effecGvely!and!efficiently!

Cochrane!is!governed!effecGvely!

Cochrane's!governance!is!open!and!transparent!

Cochrane's!conflict!of!interest!policies!are!clear,!well!balanced!and!effecGve!

Cochrane!provides!useful!evidence!for!my!health!decision>making!needs!

I!would!know!how!to!become!involved!in!Cochrane!

I!would!be!prepared!to!become!involved!in!the!governance!of!Cochrane!in!future!

Strongly!disagree! Disagree! Neither!agree!nor!disagree!

Agree! Strongly!agree! No!View!


A.9 Alternative providers of systematic reviews At the time Cochrane Collaboration was established in 1993, there were few if any contenders to produce high-quality evidence based systematic reviews of randomized controlled trials of health care. More than 20 years later, there are a number of organisations and networks that recognised the need for systematic reviews to provide rigorous evidence to decision makers and support evidence-based medicine.

Cochrane must stay up-to-date with a changing society where technology and global interconnectedness have transformed lives and the way health decisions are made. Cochrane has a new strategy and it needs to be implemented in order to enhance the effectiveness of the large and committed volunteers the entire organisation relies upon. This is the ‘time of change’ and Cochrane must seize this unique opportunity to transform itself and maintain its lead in a highly competitive environment.

The survey extended to ask respondents where else they might look for systematic reviews; such providers are listed below:

• Public or private research organisations

− Joanna Briggs (joannabriggs.org)

− The Campbell Collaboration (www.campbellcollaboration.org)

− Guidelines International Network (www.g-i-n.net)

− DynaMedPlus (www.dynamed.com)

− ECRI Institute (www.ecri.org.uk)

− Scottish Intercollegiate Guidelines Network (SIGN) (www.sign.ac.uk)

− Agency for Healthcare Research and Quality (AHRQ) (http://www.ahrq.gov)

− Enhance Reviews (enhance-reviews.com)

− UpToDate (www.uptodate.com)

− Bazian (www.bazian.com)

− PEN (www.pennutrition.com)

− IQWiG (www.iqwig.de)

− Early Breast Cancer Trialists’ Collaborative Group (EBCTCG)

− US Preventive Services Task Force (www.uspreventiveservicestaskforce.org)

• Health Technology Assessment groups

− UK NIHR Technology Assessment Review (TAR) teams

− INAHTA (www.inahta.org)

• Various academic groups and consultancies

− University of York, Centre for Reviews and Dissemination (www.york.ac.uk/crd)

− University College London, The EPPI Centre (eppi.ioe.ac.uk)

− McMaster Health Forum/ Health Evidence (healthevidence.org)

− The Sax Institute (saxinstitute.org.au)

• Medical / scientific journals

− BMJ Evidence (clinicalevidence.bmj.com)

− BMJ Drug and Therapeutics Bulletin (dtb.bmj.com)


− Systematic Reviews (www.systematicreviewsjournal.com)

• Databases and search engines

− TRIP database (www.tripdatabase.com)

− Epistemonikus (www.epistemonikos.org)

− NICE Evidence search (www.evidence.nhs.uk)

− BestBETs (bestbets.org)

− PDQ Evidence (www.pdq-evidence.org)

− Database of Abstracts of Reviews of Effects (DARE) (crd.york.ac.uk/CRDWeb)

− National Guidance Clearinghouse (www.guideline.gov)

− SumSearch2 (sumsearch.org)

− Fisterra (fisterra.com)

− WebMD (www.webmd.com)

Some feedback on comparison with the above sites and Cochrane Library were provided. For example respondents stated that UpToDate is a “much more useable in a clinical setting than Cochrane reviews” and “more up to date and addresses very specific questions, but the review is less rigorous than Cochrane’s (same for Dynamed)”; “more accessible and useable in a clinical setting, including in areas of uncertainty (same for Fisterra)”. In general about non-Cochrane reviews in medical journals it was stated that those “sometimes have a more global perspective that better answer clinical questions” and HTA assessments “include aspects not covered by Cochrane”.


Appendix B External stakeholder consultation questionnaire

Introduction

Cochrane strategy survey - external stakeholder consultation

Cochrane is a global, independent network of researchers, professionals, patients, carers, and people interested in health. It is in

the process of reviewing its structure and ways of working as part of its new Strategy to 2020. Cochrane aims to understand the

effectiveness of its current organisational outreach and governance structure from the viewpoint of all of its current and potential

partners and stakeholders.

Cochrane has commissioned Technopolis, an independent policy research and consulting organisation, to support this work and

carry out a global survey. We are therefore seeking the views of people who are involved in or aware of Cochrane’s activities.

Your voluntary participation in the survey will help gather evidence and provide an opportunity for you to contribute to the ongoing

discussions on Cochrane’s organisational changes and help achieve its mission of promoting evidence-informed health decision-

making.

We estimate that it will take 20 minutes to complete the survey. All responses and associated personal information will be treated in

the strictest confidence, in line with legislation on data protection. Information will only be reported in an aggregate or anonymised

form to Cochrane.

If you have any questions related to this survey, please contact the independent study team at cochrane-study@technopolis-

group.com or call Dr Peter Varnai at +44 1273 204320. If you would like to find out more about the Cochrane’s organisational

review, please contact Chris Champion at [email protected].

Thank you for taking the time to complete the survey – your participation is extremely important to the success of the

study.

Before you begin, please make sure that your browser is maximised. It is easy to navigate through the questionnaire: just click on

your answer or answers for each question. You may need to use the scroll bar to see the next question. To continue, click on the

next button at the bottom of each page. While your browser is open you can go backward and forward in the survey but you will not

be able to return to your survey once the browser has been closed or when you have submitted the survey.

Please click ‘next’ to enter the survey.

1

http://www.cochrane.org/about-us/our-strategy

mailto:[email protected]

mailto:[email protected]

We would like to learn about where you work and what you do so that we understand better the

responses in the following sections.

A. About you


1. Which of the below best describes your current organisation/employer?

University

Research institute

Hospital or healthcare facility

NGO / charity

Government / Public administration

Research funder

Consumer or patient organisation

Other (please specify):

Country

2. What country do you live in?

Native

language

3. What is your native language?

2

4. Which of the below best describes your current position?

Researcher (non clinical)

Clinician

Policy officer

Guideline developer

Research funder

Health service manager, commissioner

Patient, service user and carer

Patient advocate

Information specialist

Other (please specify):

5. Are you involved in or aware of Cochrane’s activities? Select all that apply:

I am / have been affiliated with a Cochrane Group

I am / have been involved with Cochrane as a contributor

Colleagues within my organisation are involved with Cochrane

I know about Cochrane through its local Centre or Group activities

I know about Cochrane through using the Cochrane Library or its website

I am not involved in or aware of Cochrane

3

We would like to gather your views on Cochrane’s current priority setting practices and

timeliness of evidence it produces (i.e., Cochrane systematic reviews).

B. Producing evidence that is relevant and up-to-date


If No, please explain your reasons:

6. Do you use Cochrane evidence in your work or in decision-making on health?

Yes

No

I don't know

Please explain your response:

7. Do you believe that Cochrane evidence tackles the right topics in health and health care?

Yes

No

I don't know


Do you believe that Cochrane evidence answers questions that are well-framed and relevant?

Yes

No

I don't know

4


8. Do you find that Cochrane evidence is sufficiently up-to-date?

Yes

No

I don't know


9. Based on your experience, is Cochrane receptive to external views when setting priorities?

Yes

No

I don't know


Have you ever been involved with priority setting for systematic reviews in Cochrane?

Yes

No

I don't know

If Yes, please explain how you would prefer to engage (e.g., participation in priority setting workshops or contributing to high-priority

topic list via consultation) and where you think you could add most value?

Would you like to engage with Cochrane on setting its review priorities in future?

Yes

No

5

Not at all

important

Somewhat

important Important

Highly

important No view

Patients / consumers are involved in setting

priorities for research

Patients / consumers are involved in interpreting

the relevance of outcomes

Patients / consumers are involved in co-

production of research

Patients / consumers are involved in

dissemination of research

Patients / consumers are represented in the

governance of Cochrane

10. Cochrane aims to involve patients, patient advocates and healthcare consumers in the production of

Cochrane evidence its various activities. Please rate the importance of the following:

6

We would like to gather your views on Cochrane’s activities in the development and application

of methods that support systematic reviews.

C. Cochrane’s methods


11. Do you have an interest in evidence synthesis methods?

Yes

No

No view

7


If No, please explain (e.g., which types of methods and why):

12. Do the methods used in Cochrane Reviews meet your needs?

Yes

No

No view

13. Are you aware of the various methodological groups that comprise Cochrane Methods and support

method development for systematic reviews?

Yes

No


14. Do you believe Cochrane has high enough profile to attract the best methodologists for its

Systematic Reviews?

Yes

No

No view

15. What might attract the best methodologists to join Cochrane’s evidence synthesis instead of other

applied research areas? Please explain your response:

8

We would like to gather your views on the accessibility of Cochrane evidence (whether you can

easily get hold of relevant information) and ways to improve its usability (whether you can easily

understand and interpret the relevant information in Cochrane Reviews, Cochrane Clinical

Answers, Summary of Findings tables, etc.).

D. Accessibility of Cochrane’s evidence


If No, please explain:

16. Is Cochrane’s evidence easily accessible to you?

Yes

No

No view

If No, please explain (e.g., presentation, interpretation):

Is Cochrane’s evidence easily usable to you?

Yes

No

No view

Not at all

important

Somewhat

important Important

Highly

important No View

Open access

Customised search facility for different audiences

Presentation format and layout of the reviews

Language adapted for different audiences

(scientific, guideline makers, consumers, etc)

17. Please rate the importance of the following factors relating to accessibility and usability of Cochrane

evidence.

18. The main focus of Cochrane’s language translation activity is to make Abstracts and Plain-

Language Summaries of its Reviews available in selected languages. Please respond to the

following questions regarding the language of Cochrane Reviews.

9

Are you comfortable using English to read Cochrane Reviews?

Yes

No

If Cochrane evidence were increasingly available in your native language, would you use it more often?

Yes

No

Not applicable

Do you think others in your region (colleagues, clinicians, patients, researchers and policy-makers)

would use Cochrane evidence more often if they were translated into the national language?

Yes

No

Not applicable

Not at all

important

Somewhat

important Important

Highly

important No View

Abstract

Plain-language summary

Background / objectives

Methods

Results

Discussion

Conclusions

Please rate the importance of translating the following sections of Cochrane Reviews for you.

1 2 3

Language

Other language(s) (please specify)

In your view, which top 3 languages Cochrane should prioritise?

10

19. What additional changes would help Cochrane become an accessible, useful and trusted 'Home of

Evidence' for health? Please specify:

11

We would like to hear your views on whether Cochrane communicates effectively its goals and

principles to external partners and stakeholders through its local network of centres and groups,

and across specialised fields.

E. Cochrane’s advocacy for evidence


Not at all

effective

Somewhat

effective Effective

Highly

effective No View

Cochrane’s campaign for transparency and

integrity in scientific conduct

Cochrane’s engagement in debates advocating

for the use of evidence-based methods and high

quality evidence

Please explain your answer:

20. Based on your experience in your region or in your field of speciality, please rate the effectiveness of

the following activities of Cochrane:

Not at all

influential

Somewhat

influential Influential

Highly

influential No View

Clinical practitioners

Healthcare policy makers

Health service managers

Consumers or patients

Research funders

21. Based on your experience, please rate Cochrane’s influence in promoting evidence-informed

decision making for the following stakeholders

22. In your view, how could Cochrane have more impact on healthcare policy and practice? Please

provide one example (e.g., by describing another organisation’s practice):

12

23. What other provider(s) could you use to obtain high-quality systematic reviews in support of your

decision-making needs in health? Please briefly explain how they compare with Cochrane:

13

Cochrane is a very large network dedicated to improving health decisions globally. We would

like to gather your views about its structure and governance model.

F. Cochrane’s organisational structure and partnerships


If Yes, please note which Cochrane Group:

24. Are you aware of and/or interact with Cochrane Groups globally?

Yes

No

25. Are you aware of the local presence of Cochrane in your region/country?

Yes

No

If No, how could your local Centre better meet your needs?

If Yes, does the local presence of Cochrane allow you to engage with Cochrane?

Yes

No

If Yes, what value does the partnership with Cochrane have for you and how would you want Cochrane to further strengthen the

relationship?

26. Does your organization have a partnership with Cochrane (either formal or informal)?

Yes

No

I don't know

27. In your opinion, what partnership could Cochrane establish in your country or region to support

better health decisions?

14

Not at all

interested

Somewhat

interested Interested

Highly

interested No View

Building a personal Cochrane profile

Access to a Cochrane virtual community for

networking and forums

Accessing Cochrane software

Accessing tools for dissemination and advocacy

Involvement in crowd-sourcing projects

Tailored communications

Discounts at events

Learning about Cochrane and evidence-based

medicine

Accessing Cochrane online training

Undertaking continued professional development

with Cochrane accreditation

28. Cochrane is planning to establish a membership scheme for individuals and institutions to formalise

and acknowledge their contribution to Cochrane’s work. Please rate the extent to which the following

features and benefits of a new membership scheme would be of interest to you.

If No, please explain your answer:

Would you consider becoming a Cochrane member (as an individual or as an institution) in Cochrane’s

new membership scheme?

Yes

No

15

Strongly

disagree Disagree

Neither

agree or

disagree Agree

Strongly

agree No View

Cochrane is an independent organization

Cochrane is run effectively and efficiently

Cochrane is governed effectively

Cochrane's governance is open and transparent

Cochrane's conflict of interest policies are clear,

well balanced and effective

Cochrane provides useful evidence for my health

decision-making needs

I would know how to become involved in

Cochrane

I would be prepared to become involved in the

governance of Cochrane in future

29. Based on your experience, please indicate the extent to which you agree with the following

statements:

30. Would you be interested in Cochrane providing free training initiatives for your organisation (e.g.,

training on how to interpret Cochrane evidence to make informed health decisions)?

Yes

No

What types of training should Cochrane offer? Please explain your response:

16

G. Closing section


We would like to follow-up this survey with some of the respondents to gain a better understanding of

Cochrane’s engagement practices and/or provide you the opportunity to engage with Cochrane. Would

you be willing to participate further in this study?

Yes

No

Name:

Email Address:

If Yes, please provide your details below:

Please feel free to provide any further comments or suggestions you may have for Cochrane:

Thank you for completing the survey!

17

technopolis |group| United Kingdom 3 Pavilion Buildings Brighton BN1 1EE United Kingdom T +44 1273 204320 E [email protected] www.technopolis-group.com

2268 cochrane structure and function reviews - external … · 2015-10-02 · cochrane structure...

Documents