front page tech interim report 2010 wp7appendices · eunethta ja on hta – wp 7 1/1 2011-02-28...

The EUnetHTA JA (2010-2012) has received funding from the European Union, in the framework of the Health Programme

E U R O P E A N N E T W O R K F O R H E A L T H T E C H N O L O G Y A S S E S S M E N T

EUnetHTA

INTERIM TECHNICAL REPORT – WP7 Appendices YEAR 2010

EUnetHTA JA on HTA – WP 7 1/1

2011-02-28

Joint Action on HTA 2010-2012 Work Package 7 – “New Technologies”

Interim Technical Report 2010

APPENDIX WP7.1_AB

1st WP 7 face-to-face meeting, Dublin - Minutes

Thursday, 10 June 2010 EUnetHTA JA WP7

1

Organised by: HAS and LBI-HTA (logistical assistance provided by HIQA) Address: Health Information and Quality Authority (HIQA) offices at George’s Court, George’s Lane DUBLIN 7 IRELAND

EUnetHTA JA WP7 New Technologies

1st Work Package 7 Meeting Minutes

Meeting Dublin

June 10, 2010


2


TABLE OF CONTENTS

PARTICIPANTS

PAGE 3

AGENDA

PAGE 4

MEETING OBJECTIVES

PAGE 5

DATABASE FOR ADDITIONAL EVIDENCE GENERATION ON NEW AND SELECT

TECHNOLOGIES

PAGES 6-7

EXPLORING A DATASET FOR TRIALS IN DEVELOPMENT

PAGES 8-9

CRITERIA TO SELECT NEW TECHNOLOGIES FOR ADDITIONAL EVIDENCE

GENERATION

PAGES 10-11

PROTOCOL WP 7/B

PAGES 12 -13

PICTURES FROM THE FACE-TO-FACE MEETING

PAGES 14-15

SOCIAL EVENT

PAGE 16

SURVEY RESULTS

PAGES 17-20


3


PARTICIPANTS

Country

Organisation

Member

Lead Partner

France HAS, Haute Autorité de Santé

Sun-Hae Lee Robin Cédric Carbonneil Laura Zanetti Nathalie Merle

Co Lead Partner

Austria LBI-HTA (Ludwig Boltzmann Institute for Health Technology Assessment)

Claudia Wild Gerda Hinterreiter Marisa Warmuth

Associated Partners

Czech Republic Ministry of Health No representative present

Denmark SDU No representative present

Finland THL, National Institute for Health and Welfare Sirpa-Liisa Hovi

Hungary EMKI, Institute for Healthcare Quality Improvement and Hospital Engineering

Lajos Kovacs

Ireland HIQA, Health Information and Quality Authority Martin Flattery Patrick Moran

Italy Age.na.s (Agenzia) Marina Cerbo Maria Rosaria Perrini

Italy AIFA, Italian Medicine Agency Pietro Folino Gallo

Italy Regione del Veneto Teresa Gasperetto

Lithuania VASPVT Gintare Pakolkaite

Malta Ministry of Health Renzo Pace Asciak

Netherlands CVZ Sarah Kleijnen

Norway NOKC (Norwegian Knowledge Centre for the Health Services) Inger Norderhaug

Poland AHTAPol (Agency for Health Technology Assessment) Jadwiga Czeczot

Portugal INFARMED, National Authority of Medicines and Health Product Isaura Viera

Spain AETS, Agencia de Evaluación de Tecnologias Sanitarias Setefilla Luengo

Sweden SBU, Swedish Council on Technology Assessment in Health Care Monica Hultcrantz

United Kingdom

NICE, National Institute for Health and Clinical Excellence Bhash Naidoo

United Kingdom

NETSCC, NIHR Evaluation, Trials and Studies Coordinating Centre Andrew Cook Eleanor Guegan

Collaborating Partners

Spain AETSA Andalusian Agency for Health Technology Assessment Aurora Llanos-Méndez

Switzerland SNHTA, Swiss Network for Health Technology Assessment Sylvie Bailat

Other EUnetHTA Partners

Austria GOG Elisabeth Breyer

Belgium KCE, Belgian Health Care Knowledge Centre Patrice Chalon

Denmark DACEHTA Finn Børlum Kristensen morning visitor


4


AGENDA

Thursday June 10th 09:00 – 16:00

1. Introduction & Welcome Sun Robin (HAS, France)

9:00 – 9:15

2. Database for additional evidence generation on new and

select technologies

• Presentation : “Feedback from Eiffel prototype ; exploring areas of need and improvement” by Laura Zanetti (HAS, France):

• Working groups discussions • Working groups feedback and general discussion

9:15 – 10:45

Coffee break

10:45 – 11:00

3. Exploring a dataset for trials in development

• Presentation by Andrew Cook (NETSCC, United Kingdom) • Discussion about the dataset & accessing trial funders

4. Criteria to select new technologies for additional

evidence generation

• Presentation: criteria to select new technologies for additional evidence generation by Cédric Carbonneil (HAS, France)

• Working groups discussions • Working groups feedback and general discussion

11:00 – 11:45

11:45 – 13:00

Lunch

13:00- 14:00

5. Introduction & Welcome Claudia Wild (LBI-HTA, Austria)

6. WP7 B: Part 1 • WP 7B workplan 2010-2012, POP results, classification & alert

system, POP workroom training by Gerda Hinterreiter, Marisa Warmuth (LBI-HTA, Austria)

• POP database development, needs and functions by Patrice Chalon (KCE, Belgium)

14:00 – 14:45

Coffee break

14:45 – 15:00

7. WP7 B: Part 2

• Special new techs with high potential for collaboration: Claudia Wild

• Fields and ways to collaborate: working groups discussions • Working groups feedback and general discussion 8. Final Words & Thanks Sun Robin & Claudia Wild

15:00 – 16:00

16:00


5


MEETING OBJECTIVES

WP7A

Eiffel Database Objective 1: Presentation of the new proposals/improvements for Eiffel web based tool-kit for 2010-2012 ; Objective 2: Obtain feedback from meeting participants about these new orientations.

Data Set Objective 1: Obtain feedback from meeting participants about the proposed dataset Objective 2: Identify participants interested in being involved in the project, including nomination of appropriate organisations from their countries Objective 3: Gain feedback from participants about studies appropriate for testing the dataset (eg breast cancer)

Criteria Selection Objective 1: Present the context of the selection/prioritization criteria; Objective 2: Obtain feedback from meeting participants about orientations; Objective 3: Define the scope of the development of selection criteria by EUnetHTA WP7.

WP7B

POP

Goal: Reduction of Duplication Objective 1: Providing information about the work of WP 7B Objective 2: Bring people together: identifying concrete fields and projects for collaboration Objective 3: Training to use the POP workroom

� Please note that all supporting documents used during the face-to-face meeting

may be found in the workroom.


6


DATABASE FOR ADDITIONAL EVIDENCE GENERATION ON NEW AND SELECT

TECHNOLOGIES1

The aims of the meeting were to briefly present the lessons learned from the pilot phase of the Eiffel prototype use and receive feedback from all EUnetHTA members on new orientations for the upgraded version of the database. The reasons for the database’s limited use during the pilot phase have been analysed and presented. Based on the results, the following new orientations have been proposed for the database’s evolution and improvement:

• The scope needs to be revised according to the initial objectives of the database (i.e. support European collaboration on studies, avoid duplication and promote global analysis of results). The scope of the database should therefore be limited to sharing systematic identification of evidence gaps in the context of an HTA and exchanging information on additional data requirements, planned or ongoing prospective data collection.

• We suggest restricting the content of the database to NEW and EVALUATED HT that would require the generation of additional evidence. The initial list of items in the database has been revised to include the description of the HT, the HTA status, the coverage/reimbursement status, the rationale for additional evidence generation, the key research question posed, the minimum dataset for each study (NETSSC), the study’s status and results when available and the resultant decision taken.

• We suggest a restructuring of the database for simplification purposes and to improve user-friendliness with new IT functionalities.

• Finally, a process for data submission has also been suggested. Four working groups were assigned to discuss the database’s scope, content, structure and functionality.

Results from the working groups:

Question 1: Are you able to require and/or fund additional data generation? Results from the Green and Blue groups: Not all partners will be able to contribute equally in the provision of information on AEG following an HTA due to differences in their overall missions or remits. Indeed, four categories of partners were identified; those that:

- are unable to require/request or recommend further research [Ministry of Health (MT), VASPVT (LI), EMKI (HU)];

- may request/recommend further research to decision makers [HIQA (IR), NOKC (NO), DACEHTA (DA)];

- may require further research [HAS (FR), Agenas, Regione del Veneto, AIFA (IT), NETSCC (UK), SNHTA (SV), CVZ (NL), AETSA (SP), Infarmed (PT)]

- have allocated funds for further research (AIFA (IT), AETS (SP), NETSCC (UK) • All group participants would be interested in:

o receiving information on potential studies that may be conducted in the future o identifying opportunities for collaboration

1 Please refer to WP7 workroom for corresponding documents (presentation and questions)


7


Question 2: Will the new orientations proposed for the database match your agency/organization needs? Results from the Green and Blue groups:

• Two elementary needs were expressed: - Identifying ongoing studies and the need for additional research - Identifying areas for collaboration and sharing information spontaneously to mutually interested parties.

It was also emphasized that the database should be easy in use in order to generate wide use of it. • Most participants agreed that a clear definition of NEW HT would be needed. Some

participants considered that it would be useful to also consider new INDICATION for technologies that are already on the market. On the other hand, some countries consider that it is important to be focused on the content of the database.

Question 3: Are the proposed items satisfactory? Results from the Yellow and Red groups:

• Overall, all EUnetHTA members agreed on the list of items proposed. Some suggestions were performed (e.g. use of generic name, ATC code for drugs,..). It was highlighted that we should standardize/harmonize the choice the coding within all EUnetHTA tools/database to ensure exchangeable information. A need for collaboration with WP6 and WP7B was also expressed.

• For some agencies, handling questions related to research gaps and uncertainties may present some difficulty (e.g. may require contacting others institutions,..).

Question 4: What operational aspects of information would be required for an optimal/regular use of the database? Results from the Yellow and Red groups:

• A unique password for all EUnetHTA tools would be optimal • Alerts are essentials. For each new entry (inquiry), a 1st global email alert, linked to a

deadline for member responses, should be implemented; a 2nd global email alert should then be diffused when the inquiry has been completed by all organizations/agencies.

• Time to complete an inquiry should be quick (less than 15 min). • Adding a search engine would be helpful (e.g. alphabetical, by categories of

technology, by name etc.)

Overall

Sharing information on additional studies requested/funded by EUnetHTA members is considered of great interest and is expected to facilitate European collaboration on HT projects. Overall, the new orientations for the EIFFEL database in terms of scope, use and content have been agreed and contributing comments will be taken into account for the next steps of Eiffel development process.

For additional information concerning this section of the WP7a, please contact Laura Zanetti at [email protected]


8


EXPLORING A DATASET FOR TRIALS IN DEVELOPMENT2

• Participants were given handouts; a copy of the draft dataset and HTAi poster. • Participants were asked to register their interest in the project & to notify NETSCC of

appropriate contacts in their countries with respect to policy relevant studies funded by or of direct interest to public sector bodies.

Presentation:

• Andrew Cook gave the presentation, ‘What are they doing? Exploring a dataset for studies in development’.

• This outlined the problem – the possibility of similar, but not identical studies, being performed at the same time but which are not comparable therefore missing the chance of collaboration (multinational or comparable studies) and possibly waste of money.

• This was demonstrated with the example of Herceptin – it was not possible for 3 similar studies to identify each other before commitment and therefore outcome measures were not directly comparable. Early identification of trials in the planning stage could lead to multinational studies, or at least comparable outcomes.

• It is important to know what other funders are planning – so we can work with them for multinational studies / comparable outcomes. Reviewers can plan a schedule for doing / updating systematic reviews – ie if you know that there is going to be a major trial it may influence the timing of your HTA. Systematic reviewers would also have the chance to ask for modification to the trial design & timing.

• We want to develop something simple, such as a registry of studies funders are considering. Ideally the system would alert you to studies others are thinking about when you enter your new study. This will be useful for; Funders, Reviewers/HTAers/Policymakers - facilitates determining workplans. To control the volume, at least initially, the contents will be policy relevant studies funded by or of direct interest to public sector bodies ie trial funded by NIHR HTA, studies requested by HAS but done in the private sector etc.

• The work will be in 3 strands; Agreeing a minimum dataset (NETSCC, M1-M12), testing the dataset (NETSCC, M13-M18) and implementing the database (HAS, M19-M36).

• Dataset; we have a 1st draft – we will conduct 2-3 Delphi surveys of this. We want to identify appropriate European participants from EUnetHTA. As the tool is more useful the wider the dataset is circulated, appropriate partners outside EUnetHTA will also be involved.

• A copy of the proposed 1st draft dataset was circulated to meeting participants & discussed.

• Testing the dataset; asking funders to submit details of trials in specific clinical areas & using the dataset to check it can spot overlaps.

• HAS will then develop the database in EIFFEL. Discussion Questions:

Andrew Cook then led a discussion, based on the following questions;

• Who are the funders and reviewers from your country who should be involved? • How should the dataset be changed? • Do we need native language, or will English suffice? • Is MESH the right coding system?



9


Discussion Points:

• MESH is the correct coding system (LBI-HVB) • Asked for clarification about how transcription errors would be dealt with (HAS). May

need to insist on double-entry (NETSCC) • MESH terms would be better than MESH codes because they are more easily used by

operators & computers. (KCE) • All EUnetHTA tools should use English and there is a multilingual MESH term dictionary

available (KCE) • Is this just for EUnetHTA participants? (THL) It will be wider, however all EUnetHTA

participants can see the data. Those from outside EUnetHTA would only be able to see the data if they populated it. (NETSCC). Some clarification about these points to HAS. Suggestion that those outside EUnetHTA should be charged (HAS). This is a matter for WP8.

• ‘This will be of great benefit to all agencies in influencing primary research design’ (HAS)

• Asked for clarification about how many users outside EUnetHTA might be involved because this will have implications for Secretariat creating them a profile & Secretariat should be advised about this (KCE).

• Asked for clarification of timelines of studies to be considered (HAS). Studies affecting - reimbursement or coverage decisions, such as pragmatic trials with patient-centred outcomes. (NETSCC).

• Asked for clarification about these studies and ‘traditional HTA’ (HAS). When doing evidence synthesis such as literature reviews & economic modelling it would be possible to horizon-scan what will be available in the future, so that a timeline can be planned. This will be later than trial registries, such as clinicaltrials.gov (NETSCC). This registry will identify trials that are planned – they may or may not then be funded.

• Asked for clarification about funding for the database (LBI-HVB). This is already funded as part of the EUnetHTA project WP7 and HAS will develop the database into EIFFEL. (NETSCC)

• There is conditional reimbursement with industry in Portugal & this is policy relevant because it is filling evidence gaps. Would be interested in taking part in the project – please contact the agency (INFARMED).

For additional information concerning this section of the WP7a, please contact Eleanor Guegan at [email protected]


10


CRITERIA TO SELECT NEW TECHNOLOGIES FOR ADDITIONAL EVIDENCE

GENERATION3

The aims of the meeting were to present to partners the context and the rationale for the development of criteria to select and prioritize new health technologies for which the generation of additional evidence is of utmost importance. Thus, a review of criteria used to select and prioritize promising technologies requiring additional evidence (WP7A milestone) was presented during the meeting. Based on this review, four main domains were identified:

• Domain 1: Healthcare needs; • Domain 2: Interest of the new technology • Domain 3: Types of evidence gaps • Domain 4: Appropriateness of additional studies

Two working groups were assigned to discuss the scope of the development of selection criteria within EUnetHTA and the relevance of the identified main domains in the further development of selection criteria. Results from the working groups:

Question 1: What could be the main impacts of the development of selection/ prioritization criteria? Results from group 1 (Red and Blue):

• The development of selection/prioritization criteria in the context of EUnetHTA is relevant for all partners

• It was deemed necessary to clarify the stage at which the criteria should be implemented along the entire HTA process. Therefore, it was re-emphasized that the list of criteria is intended to select and prioritize promising technologies requiring additional evidence. The selection criteria would be of greater value at the end of the assessment, when evidence gaps have already been identified and not at the horizon scanning stage used to define HTA programmes.

• Considering the heterogeneity of the European health care systems and their respective coverage mechanisms, criteria should be independent from the local context.

Results from group 2 (Green and Yellow):

• The development of selection/prioritization criteria is of particular interest for HTA doers in order to improve the transparency of decision making.

• It may be helpful for funders whose selection process is often “ad hoc”. It could help to formalise the process and to make transparent decisions.

• Prioritization may prove to be a challenging task, as seen with other databases such as Euroscan and Glasgow system.

Question 2: What is your feedback on the proposed domains and categories of criteria? Results from group 1 (Red and Blue):

• The most important domain to consider is the type of evidence gaps (Domain 3). • Appropriateness of additional studies (Domain 4) is also important as it is of little use

to select/prioritize a technology for which additional studies cannot be performed.



11


• However, the title of Domain 4 should be renamed “feasibility of additional studies” instead of “appropriateness of additional studies”, to minimize confusion. Indeed, the word “appropriateness” is inadequate, as it implies a value judgment that could be a consequence of the application of the selection/prioritization criteria.

• It may be interesting to consider the “interest of the technology” (Domain 2) • As healthcare needs (Domain 1) have been considered in a prior HTA assessment, and

considering that it may be redundant with some criteria of Domain 2 (the characterization of the interest of a new technology involves the consideration of major healthcare needs), the use of Domain 1 may be unnecessary.

Results from group 2 (Green and Yellow): The 4 proposed domains are deemed acceptable.

• The title of Domain 2 should be renamed as “importance of the new technology” instead of “interest of the new technology”.

• The title of Domain 4 should be renamed to “feasibility of additional studies” • Some domains may be more relevant for HTA doers (Domains 1,2 and 3), while

Domain 4 may be more relevant for funders. Question 3: In which of these domains, selection criteria should be developed? Results from group 1 (Red and Blue):

• In order to be user-friendly, the number of selection/prioritization criteria should be limited.

• Developed criteria should be weighted according to their relative meaning and to the level/type of uncertainties previously assessed.

Results from group 2 (Green and Yellow):

• In order to be user-friendly, the number of selection/prioritization criteria should be limited.

• Redundancies should be avoided. • Criteria should mainly answer to 2 main questions: (i) are there any uncertainties? (ii)

What is the clinical impact of resolving such uncertainties? • Clinical criteria should be privileged in the prioritization process than other criteria

Overall • The aim of these criteria is NOT to plan HTA but rather to select/prioritize new health

technologies for which the generation of additional evidence might be necessary, as established by HTA.

• Selection/prioritization criteria should be independent from local context and not include items related to e.g. coverage mechanisms.

• The development of selection/prioritization criteria is of utmost interest for all partners and is considered particularly important to improve the transparency of decision making.

• The development of selection criteria should be performed in three domains: 1. type of evidence gaps 2. importance of the new technology (involving some criteria relative to healthcare

needs) 3. feasibility of additional studies

• A prioritization/ranking process should be considered.

For additional information concerning this section of the WP7a, please contact Cedric Carbonneil at [email protected]


12


PROTOCOL WP 7/B4

Address of welcome and introduction by Claudia Wild/ LBI-HTA

Presentation of POP: categories and workroom access

by Marisa Warmuth (LBI-HTA) & Gerda Hinterreiter (LBI-HTA) Discussion and comments:

Sun (HAS) remarks: Consider changing the title of the Strand B from “Pre-market/pre-assessment of new non-pharmaceutical health echnologies” to another one that would include a broader scope of POP i.e. including also pharmaceuticals. This change should also be mentioned in the work plan update and in the annual report. Teresa (Reg Veneto) asks: Access to POP database is restricted to those who contribute, but all/several researchers in one agency or only one access per agency? LBI-HTA answers: Several persons of an (contributing/providing) agency can have access. In general, Gerda can give access rights to the POP workroom to individual EUnetHTA Members-only user. But a Members-only account is required first (please contact the secretariat). In case a person already has a Members-only account, please contact Gerda directly to get access to the POP workroom. Elisabeth (GÖG) asks: Projects stay as long in POP list till they are published in the CRD-database? LBI-HTA answers: Yes, because published reports only stay in the POP list for 1 request period (3 months). Sun (HAS) remarks: In the POP list are also primary research projects and registries. LBI-HTA answers: From now onwards primary research projects and registries are (re)moved from the POP list (to WP 7/ A database). Marina (AGENAS) asks: Is there a definition or a standard for “similar” projects for alerts? How are “similars” identified”? LBI-HTA answers: The categories “similar” for alerts are broadly defined comprehensively.

Presentation of POP database development & IT aspects by Patrice Chalon (KCE) Conclusion of discussion: Expected time of publishing should be included in POP list (in a separate column) Four working groups were assigned to discuss specific topics on possible

collaborations:

Group 1 (Rheumathoid Arthritis Drugs): AHTAPOL; NICE; Reg Veneto

• Difficulties in the first appraisal due to national timeframes, but updates • Language problems • Abstract sharing • Provision of contact person- project leader • Problems with confidentiality



13


Group 2 (Cochlear Implants, Kyphoplasty, spine implants, PET)

• LBI-HTA will send document on kyphoplasty to DACEHTA, SBU, CVZ– Claudia • CVZ will send document on cochlear implants to NOKC, Asp Lazio, CAHTA – Sarah • FinOHTA/THL will send document on cochlear implants to NOKC, Asp Lazio, CAHTA,

CVZ – Sirpa-Liisa • CVZ and LBI-HTA will exchange documents on cervical/ lumbar spine implants –

Sarah, Claudia • NOKC will coordinate PET in oncology assessments - Inger

Group 3 (Growth Hormones and Radiotherapy) – HAS/ HIQA

• Look into POP: from now on • Wait for starting depends on timeliness • Exchange: all items on checklist are interesting, but not easy to provide (e.g.

extraction tables in English), research questions will always be slightly different, scoping

• POP is good tool for identifying topics for CORE • Commitment to provide project leader details to LBI-HTA – Sun and Martin

Group 4 (Telemedicine, HIFU, Colorectal Cancer Screening)

• HAS (Cédric), LBI-HTA (Marisa), AGENAS (Marina, Maria Rosaria), CVZ (Sarah) will exchange all already existing documents on HIFU

• HAS (Cédric), FinOHTA/THL (Sirpa-Liisa), HIQA (Patrick) will exchange all already existing documents on virtual CT colonoscopy

• HAS (Cédric), LBI-HTA (Marisa), HIQA (Patrick) will exchange all already existing documents on colorectal cancer screening

Timeframe for group commitments: At least an English summary by week 24!!

For additional information concerning the WP7b, please contact Gerda Hinterreiter at [email protected]


14


PICTURES FROM THE FACE-TO-FACE MEETING AT HIQA


15


PICTURES FROM THE FACE-TO-FACE MEETING AT HIQA


16


SOCIAL EVENT ON WEDNESDAY JUNE 9TH AT THE PIG’S EAR RESTAURANT

The WP7 social event, organised by HIQA, took place Wednesday evening in central Dublin at the Pig’s Ear Restaurant. The entire second floor of this cosy, little restaurant was reserved for the entire WP7 partners where we could comfortably mingle while enjoying good Irish cooking.

This was no doubt an excellent way to kick off the first face-to-face meeting for 2010!


17


SURVEY RESULTS

WP7 Face-to-Face Dublin Meeting

SURVEY RESULTS

• A total of 24 meeting delegates responded out of the 30 present. • 100% of the 24 delegates were satisfied with the meeting overall. • On average, 85% of the 24 delegates felt that the meeting met overall expectations. • On average, 12.5% felt that the meeting exceeded overall expectations. • The need to provide meeting documents in advance was most commonly cited for the

preparation of next year’s meeting in Malta.

We would like to thank all delegates for their feedback!!

Please read below for the detail of the individual responses.


18



19



20



2011-02-28



APPENDIX WP7.2_AB

1st WP 7 face-to-face meeting, Dublin - Signed lists of attendees


2011-02-28



APPENDIX WP7.3_A

Review of criteria used to select and prioritize new health technologies requiring additional study (July 2010)

E U n e t H T A Review of criteria used to select and prioritize promising health

technologies requiring additional studies

W ORK PACKAGE 7 JULY 2010

The EUnetHTA-Joint Action is supported by a grant from the European Commission


2

EUnetHTA

“Review of criteria used to select and prioritize promising health technologies requiring additional studies”

was developed by

Work Package 7 : “New technologies”

Work Package 7 Lead Partner: HAS, High Authority for Health, France

July 2010

3

CONTENT

CONTENT ................................................................................................................................3

ABBREVIATIONS ......................................................................................................................4

I. INSTITUTIONS..................................................................................................................4

II. OTHER ABBREVIATIONS ..................................................................................................5

MAN/ WOMANPOWER................................................................................................................6

INTRODUCTION.........................................................................................................................7

I. EUNETHTA PROJECT 2006-2008:IDENTIFICATION OF PROBLEMS AND PROPOSAL OF CONCEPT........................................................................................................................7

II. EUNETHTA JOINT ACTION 2010-2012: IMPLEMENTING THE CONCEPT...............................9

OVERVIEW OF THE LITTERATURE.............................................................................................11

I. SELECTION/PRIORITIZATION CRITERIA IN CANADA (ONTARIO)..........................................11

II. SELECTION/PRIORITIZATION CRITERIA IN AUSTRALIA.......................................................12

III. SELECTION/PRIORITIZATION CRITERIA IN THE USA..........................................................13

III.1. Coverage with Evidence Development (CED) by CMS ...........................................................13

III.2. Use with Evidence Development (UED) by CMTP...................................................................14

IV. SELECTION/PRIORITIZATION CRITERIA IN SPAIN ..............................................................15

V. SELECTION/PRIORITIZATION CRITERIA IN ENGLAND/WALES .............................................16

VI. SELECTION/PRIORITIZATION CRITERIA IN FRANCE ...........................................................17

VII. SELECTION/PRIORITIZATION CRITERIA IN ITALY ...............................................................18

VIII. ANOTHER APPROACH: FILTRATION/PRIORITIZATION CRITERIA BY EUROSCAN ...................19

IX. ANOTHER APPROACH: POST-INTRODUCTION OBSERVATION (PIO) OF HEALTH TECHNOLOGIES IN SPAIN...............................................................................................20

CONCLUSIONS .......................................................................................................................22

I. SYNTHESIS ...................................................................................................................22

II. DOMAINS OF SELECTION CRITERIA .................................................................................23

III. USE OF SELECTION DOMAINS AT THE SINGLE AEG LEVEL................................................25

RATIONALE AND SCOPE OF THE DEVELOPMENT OF SELECTION/PRIORITIZATION CRITERIA BY EUNETHTA WP7A: POSITION OF PARTNERS ................................................................26

I. RATIONALE ..................................................................................................................26

II. SCOPE .........................................................................................................................26

REFERENCES.........................................................................................................................28

4

ABBREVIATIONS

I. Institutions

AETSA: Andalusian Agency for Health Technology Assessment

AETS: Spanish Agency for Health Technology Assessment

AIFA: Italian Medicines Agency

Avalia-t: Galician Agency for Health Technology Assessment

CMS: Centers for Medicare and Medicaid Services

CMTP: Center for Medical Technology Policy

EUnetHTA: European network for health technology assessment

EuroScan: European Information Network on New and Changing Health Technologies

FDA: American Food and Drug Administration

HAS: French National Authority for Health

MOHLTC: Ontario Ministry of Health and Long Term Care

MSAC: Australian Medical Services Advisory Committee

NICE: English National Institute for Health and Clinical Excellence

NIH: National Institutes of Health

OHTAC: Ontario Health Technology Advisory Committee

OSTEBA: Basque Office for Health Technology Assessment

5

II. Other abbreviations

AEG: Access with Evidence Generation

AP: Associated Partner

CAP: Coverage with Appropriateness Determination

CED: Coverage with Evidence Development

CFFE: Conditionally Funded Field Evaluation

CSP: Coverage with Study Participation

EAAS: Early Awareness and Alert Systems

HTA: Health Technology Assessment

IF: Interim Funding

IR: Independent Research

MU: Monitored Use

OIR: Only In Research

PIO: Post-introduction observation

PLS: Post-listing studies

RCT : Randomized Controlled trial

UED: Use with Evidence Development

WP: Work Package.

6

MAN/ WOMANPOWER

The review and analysis of the literature was performed by Cedric Carbonneil PhD (Researcher, HAS) under the supervision of Sun Hae Lee-Robin MD (Head of department, HAS).

7

INTRODUCTION

I. EUnetHTA project 2006-2008:identification of problems and proposal of concept

During the EUnetHTA project 2006-2008, a common policy framework for evidence generation was developed for “access with evidence generation (AEG)” mechanisms (1). This generally applicable five step policy framework comprising:

(i) a first assessment identifying knowledge gaps; (ii) a public policy decision conditional to evidence generation; (iii) the generation of evidence as requested; (iv) a re-assessment integrating the new evidence; (v) a revised decision.

Figure 1: EUnetHTA common policy framework for access with evidence generation (AEG)

8

Among 23 countries (20 European countries, USA, Canada (Ontario) and Australia), a wide heterogeneity of implementation of AEG mechanisms were reported, mostly due to the presence of barriers, such as:

(i) difficulty in agreeing on data requirements and study design; (ii) evidence generated not meeting quality criteria and therefore unable to inform a

decision; (iii) lack of coordination among the partners and bodies overseeing data collection; (iv) limited funds to finance the generation of evidence that meet Health Technology

assessment (HTA) and decision-maker requirements; (v) poorly defined regulatory framework governing coordination and financing.

Nevertheless, some critical success factors were also identified in order to facilitate the implementation of AEG :

i) coordinating body overseeing the contributions and collaboration of all participants;

ii) scientific leadership and clear guidance in key methodological issues (e.g. study design) for relevant and high quality evidence;

iii) dedicated funding, regardless of source, for data collection and analysis; iv) a regulatory framework.

9

II. EUnetHTA Joint Action 2010-2012: implementing the concept

As stated by AEG policy framework (1), limited funding is an important barrier to the implementation of AEG mechanisms. As resources are limited, even the existence of dedicated funding does not allow the generation of evidence for all promising technologies, at the single country level. Indeed, AEG mechanisms imply costly studies which cannot be funded for all promising technologies. Therefore a selection/prioritization process is often implemented in order to fund additional studies only for the most relevant promising health technologies. Another issue is the lack of transparency of AEG funding decisions. In absence of explicit selection/prioritization criteria, the decision to conditionally fund a promising health technology is generally taken in a non formalized ad hoc manner (2). Therefore, the development of selection/prioritization criteria is of particular interest for HTA doers, funders and stakeholders in order to improve the transparency of decision making through a formalized process. The timing of selection is also of utmost importance. Selection/prioritization criteria are intended to select and prioritize promising technologies requiring additional evidence. Therefore, selection criteria would be of greater value subsequently to HTA, when evidence gaps have already been identified (Figure 1: between step 1 and step 2) and not at the horizon scanning stage used to define HTA programmes. However, nowadays, some challenging issues regarding these selection/prioritization criteria are still under debate :

• which criteria are relevant? • What about the level of uncertainty? • Has the feasibility of additional evidence generation been taken into account? • What about the timeliness and duration of additional data collection?

Together, these elements stress the need to develop explicit, transparent and applicable selection/prioritization criteria, at the international level. Therefore, one of the main objectives of Work Package 7A (WP7A) during the EUnetHTA Joint Action 2010-2012, is to develop selection/prioritization criteria of new promising health technologies for which the generation of additional evidence is of utmost importance. Interestingly, the generation of additional evidence is critical for AEG but is also relevant in other areas requesting additional studies/registries, independently from coverage decisions. As examples, some technologies may be already reimbursed and diffused, but their impacts on a specific disease, on public health or their “in real life” use way remain to be analyzed. Therefore, the relevance of defining common selection/prioritization criteria involve a broader area than AEG. These selection/prioritization criteria will be developed in a 5-step collaborative manner involving all WP7A partners and applicable by all countries. The 5 steps of the development of selection/prioritization criteria by WP7A are the following:

(i) a review of existing selection/prioritization criteria in the context of AEG mechanisms (identified during the EUnetHTA 2006-2008 project) to outline a description of available selection/prioritization processes at the international level;

10

(ii) in order to complete this review and to precise the modalities of AEG decision making, a survey will be performed among WP7A partners able to request/funds additional studies (common survey with Eiffel database development);

(iii) based on the review and subsequent survey results, a first version of selection/prioritization criteria will be developed by WP7A partners, through survey/DELPHI consultations;

(iv) this first version of selection/prioritization criteria will be tested by WP7A partners to identify 3 relevant promising health technologies;

(v) based on this real life experiment, a final and commonly validated version of the selection/prioritization criteria will be established.

The present document corresponds to the first step of the development process of selection/prioritization criteria. Based on our previous work during the EUnetHTA project, we focused our literature review on established AEG mechanisms, in order to identify the most relevant and practiced process of selection/prioritization. Therefore, we have investigated such selection/prioritization process implemented in 7 countries including Canada (Ontario), Australia, USA, Spain, England/Wales, France and Italy. Finally a parallel description of other approaches of selection/prioritization process is also provided, including the selection/prioritization process for new emerging technologies established by Euroscan (European Information Network on New and Changing Health Technologies) and for the post-introduction observation of health technologies in Spain. Based on the review of the literature, the rationale and scope of the development of selection/prioritization criteria by EUnetHTA were discussed and stated by EUnetHTA partners (23 partners from 18 countries) during the WP7 face-to face meeting hold in Dublin the 10th June 2010.

11

OVERVIEW OF THE LITTERATURE

I. Selection/prioritization criteria in Canada (Ontario) In Ontario, the decision of conditionally funded field evaluation (CFFE) is taken by the ministry of health and long-term care (MOHLTC) following recommendations by the Ontario Health Technology Advisory Committee (OHTAC). CFFE can be selected when medical devices, procedures or public health interventions encounter the following criteria (3-4):

(i) there is uncertainty (low quality of evidence) about effectiveness, cost-effectiveness or safety;

(ii) there is uncertainty regarding the outcomes due to poorly conducted pre-market trials while a demonstrated impressive patient outcome;

(iii) there is uncertainty regarding the generalisability of the RCT results due to differences in patterns of practice;

(iv) there is a need for additional evidence to establish a trade-off between benefits/risks/burdens;

(v) some potentially disruptive effects were reported during the HTA, requiring a controlled and slower diffusion of the promising technology;

(vi) there is a high potential for immediate off-label use; (vii) the establishment of a technology-related quality control prior to uncontrolled

diffusion has been required; (viii) the diffusion of the promising technology requires a large potential investment

which needs to be anticipated; (ix) increasing costs attributed to the diffusion of the promising technology for which

there was little a priori evidence of effectiveness or cost-effectiveness. Together, CFFE decision is taken by MOHLTC/OHTAC, based on four major categories of selection criteria:

• identified uncertainty, as established by HTA; • need to establish the benefit/risk balance; • need for a controlled diffusion of the technology; • need for the anticipation of large investments required for the diffusion of the

technology.

12

II. Selection/prioritization criteria in Australia In Australia, the decision of interim funding (IF) is taken by the Ministry of Health and Aging following recommendations from the Medical Services Advisory Committee (MSAC). The selection of interim funding is performed based on two levels of selection/prioritization criteria (5): Primary criteria:

(i) significant impact on morbid-mortality; (ii) some short-term evidence on effectiveness; (iii) adequate evidence on safety (short to medium-term); (iv) potentially cost-effective (as effective as the index technology but less expensive) (v) potentially cost-effective (more effective than the index technology but also more

expensive). Secondary criteria

(i) There is a cost-benefit to conduct additional studies (versus a direct positive or negative funding decision);

(ii) There is a lack of information elsewhere (at the international level); (iii) There is a lack of alternative funding sources (e.g. manufacturers); (iv) There is a potential for additional studies to reduce uncertainty.

In summary, a promising technology can be covered through interim funding following a selection based on four main categories of selection criteria :

• Identified uncertainty, as established by HTA (primary criteria); • Need to establish the effectiveness/safety/cost-effectiveness • Potential benefit to conduct additional studies (secondary criteria); • No other possibility to fund the generation of requested evidence (secondary

criteria).

13

III. Selection/prioritization criteria in the USA

III.1. Coverage with Evidence Development (CED) by CMS In the USA, the decision of coverage with evidence development (CED) is taken by the Centre for Medicare/Medicaid Services (CMS). Similar to the Australian selection process, in the USA, the decision to cover a promising health technology using CED is based on a two step process. First, some primary selecting criteria have to be fulfilled (6). CED primary selection criteria

• Evidence complementary to existing medical evidence is required on effectiveness, safety and cost-effectiveness;

• Considering the available evidence, no other type of coverage is suitable; • CED leads in general in expand access to promising technologies; • CED must not assume, duplicate or replace the role of other US authorities (e.g. FDA,

NIH) in assessing promising technologies nor fostering, managing or prioritizing clinical trials (primary research);

• The burden of disease should concern Medicare/Medicaid Beneficiaries; • The treatment of disease should be the costliest for Medicare/Medicaid; • CED should have a potent impact on subsequent decision.

Subsequently, based on the type of uncertainty identified by HTA, two different sub-types of CED may be selected (6): In the presence of uncertainty about efficacy and/or safety, “coverage with study participation” (CSP) is used, involving clinical trials. Conversely, in the presence of uncertainty about “real-life use” of the promising technology, “coverage with appropriateness determination” (CAP) is used, implementing “real life” studies, registries, health economic studies etc. Once the choice between CSP and CAP has been made according to uncertainty secondary selection criteria, specific for CSP and CAP, have to be fulfilled (6). CSP secondary criteria

• Additional data in clinical setting is required; • The coverage of the promising technology is reasonable and necessary only in

clinical research settings; • Clinical evidence was only available for a sub-group of patients; • Available evidence does not allow the determination of the benefit/risk balance; • Relevant outcomes were not evaluated; • HTA has established preliminary evidence regarding basic safety and high potential

of the promising technology. CAP secondary criteria

• There is uncertainty regarding a potential misuse by the patients; • There is a concern among clinicians regarding a potential misuse by healthcare

professionals; • There is a need to restrict the use of the promising technology to a specific group of

healthcare professionals; • There is a need to restrict the use of the promising technology to a specific group of

patients;

14

• The “in real life use” of the promising technology will lead to major changes in patient management.

In summary, a promising technology can be covered through CED following a selection based on four main categories of selection criteria :

• Identified uncertainty, as established by HTA • No other possibility to fund the generation of requested evidence; • Need to establish the effectiveness/safety/cost-effectiveness; • Need for a controlled diffusion of the technology.

III.2. Use with Evidence Development (UED) by CMTP Since 2006, a private non-profit organization, the Centre for Medical Technology Policy (CMTP) aims to facilitate the design and the funding of complementary data collection through public-private partnerships (7). CMTP has developed a prioritization process using 7 selection/prioritization criteria (7):

• HTA have identified promising but unproven clinical indications; • There are evidence needs for clinicians, payers and/or patients; • There is no secured alternative governmental funding; • Required studies must be short time; • The study population must not exceed 5000 patients; • The design of the studies must be “in real life”; • There is a reasonable expectation of conclusive results.

In summary, a promising technology can be covered through UED following a selection based on three main categories of selection criteria :

• Identified uncertainty, as established by HTA; • No other possibility to fund the generation of requested evidence; • Study design.

15

IV. Selection/prioritization criteria in Spain The AEG of a promising technology through the Monitored Use (MU) is decided by the Ministry of Health, following HTA performed by regional HTA agencies (AETS, AETSA, Osteba, Avalia-t…) (8). During a pilot phase, the selection of promising technology for MU was performed using the following criteria:

• Uncertainty on effectiveness and/or efficacy, safety, cost effectiveness (no or low level of evidence);

• Potential high economical impact on the national health system; • Potential risk for healthcare professionals, patients or environment; • Substantial innovation to the diagnostic, therapeutic or rehabilitation of diseases,

leading to a decrease of mortality, morbidity and/or disability; • New indications of existing technology; • The diffusion/use of the promising technology requires new specific equipments; • The diffusion/use of the promising technology involve major organisational changes in

patient healthcare; • The diffusion/use of the promising technology will affect some specific groups of

population. In summary, promising technology can be covered through MU following a selection based on four main categories of selection criteria :

• Identified uncertainty, as established by HTA; • Need to establish the effectiveness/safety/cost-effectiveness; • Need for a controlled diffusion of the technology; • High potential impact on healthcare.

16

V. Selection/prioritization criteria in England/Wales In England and Wales, the decision of “Only in Research” (OIR) is taken by the National Institute for Health and Clinical Excellence (NICE). The selection of OIR is performed based on the following selection/prioritization criteria (2,9):

• There is uncertainty (not enough robust evidence) regarding effectiveness, safety and/or cost-effectiveness;

• What is the potential evolution of the uncertainty without OIR?; • What is the potential impact of positive coverage decision if inappropriate? • Is there any available alternative technologies? • The access to a promising technology under OIR should be guaranteed for all

patients in the country; • The use of a promising technology under OIR should be ethical and transparent; • OIR should have value for money; • Is additional data collection through OIR feasible (number of patients, costs duration

of study)? • Is a sponsorship available?

In summary, a promising technology can be covered through UED following a selection based on four main categories of selection criteria :

• Identified uncertainty, as established by HTA; • Need to establish the effectiveness/safety/cost-effectiveness; • Impact of alternative coverage decisions; • Feasibility of OIR.

NB: These selection/prioritization criteria of OIR are currently under revision by NICE.

17

VI. Selection/prioritization criteria in France In France, post-listing studies (PLS) may be requested by the French High Authority for Health (HAS). There is not a real selection process of new medicines or medical devices requiring post-listing studies, considering that the realization of these studies are mandatory for the manufacturers (10-11). Nevertheless, post-listing studies may be requested if:

• there is uncertainty (not enough robust evidence) regarding effectiveness and/or safety;

• there is uncertainty regarding the generalisability of the RCT results due to differences in patterns of practice;

• there is some concerns regarding the safety of the technology in “real life use”; • there is a potent impact of the technology on organisational aspects of the health care

system; • there is a concern among clinicians regarding a potential misuse by healthcare

professionals and/or patients. In summary, a promising technology can be covered with post-listing studies following a selection based on four main categories of selection criteria :

• Identified uncertainty, as established by HTA; • Need to establish the effectiveness/safety; • Appropriate use; • High potential impact on healthcare. • Need for a controlled diffusion of the technology..

18

VII. Selection/prioritization criteria in Italy In Italy, Independent research (IR) for medicines may be requested by the Italian Medicines Agency (AIFA). The selection is based on call for proposal for Italian public institutions only. Proposal must be related to three specific research area, periodically defined by AIFA (12). Relevant proposals are subsequently analyzed by national and international reviewers to select the most relevant projects. In 2008, research area and subsequent selection criteria were: Orphan drugs for the treatment of rare diseases and drugs for non responders

• There is uncertainty (not enough robust evidence) regarding the benefit/risk balance • Off-label use.

Head to Head comparison and therapeutic strategies

• Considering the lack of comparative evidence, additional studies (head to head comparative studies, health economics studies..) are requested to clearly define the therapeutic strategy in diseases with a high burden (cancer, stroke, diabetes, asthma…).

Strategies to improve the appropriateness of drug use and pharmaco-epidemiology studies In summary, a promising technology can be covered with post-listing studies following a selection step based on four main categories of selection criteria :

• Identified uncertainty, as established by HTA; • Orphan drugs; • Therapeutic strategies; • Appropriate use of medicine.

19

VIII. Another approach: filtration/prioritization criteria by EuroScan

The International Information Network on New and Emerging Health Technology (EuroScan) is a collaborative network of member agencies aimed to exchange information on new and emerging technologies at an early stage of development of health technology, using Early Awareness and Alert Systems (EAAS) (13). Thus, the filtration/prioritization process of EuroScan is used earlier than AEG. Nevertheless, Euroscan’s prioritization tool can be very informative for our current task. EuroScan’s filtration/prioritization criteria are the following: Filtration criteria

• the technology is appropriate for stakeholders and relevant to the health system; • the technology is new/innovative; • the technology is within the timeframe of the EAAS; • the burden of the associated disease; • existing treatment for this condition? • anticipated clinical impact • economic impact; • budgetary impact; • other.

Prioritization criteria

• patient group related; • potential impact; • external emphasis; • potential for inappropriate diffusion given available evidence.

20

IX. Another approach: post-introduction observation (PIO) of Health Technologies in Spain

In 2007, a collaborative work led by the Galician HTA Agency Avalia-t has developed a structured methodological framework for the observation of new health technologies following their introduction in clinical practice (14). Thus, this framework was not strictly applicable for AEG but was rather intended to follow a new health technology following an approval/funding decision, leading to its introduction in daily clinical practice. Therefore, there is no more uncertainty regarding the effectiveness of the technology. One the main objectives of this work was to develop a prioritization tool which can be very informative for our current task. This prioritization tool was based on 14 prioritization criteria, pooled in 4 domains. Domain 1: population/end users

• Frequency of use : when it is known or considered that the technology may be applied to a large number of patients;

• Disease burden : the condition or indication for which the technology is being used entails elevated mortality, morbidity, disability or affects the patient’s quality of life to a major extend;

• Impact on end user/ population : the technology may bring about important improvement in the state of health/wellbeing of the subjects or population to which it is applied (e.g. population screening);

• Vulnerable population: The technology has been essentially designed to be used on a highly sensitive group of patients (e.g. pregnant women…)

Domain 2: technology • Innovative technology: where the design, materials or their operation are totally new

or very different to other existing technologies and/or without any previous technological alternatives for the clinical indication;

• Invasive technology: any technology requiring surgical intervention or aggressive medical procedure (including implantable medical devices);

• Different expectation of use: the health technology may have a broader diffusion than originally approved (e.g. additional indication in clinical practice).

Domain 3: Safety/adverse effects

• Safety: there is evidence in the literature of potent or potential adverse effects; • Undetected potential adverse effects : available evidence is deemed insufficient for

supplying information on infrequent adverse effects: • Risks: there is a possibility of harm for healthcare professionals (e.g radiation) and/or

for environment (e.g. dangerous waste), as a consequence of the use of the new technology.

Domain 4: Organisation/costs and other implications

• Need for training: technology requiring an intense period of training (learning curve); • Financial impact: necessary investment in infrastructure, equipment and/or cost of

fungible goods, maintenance or human resources; • Organisational or structural impact: technologies that demand multidisciplinary

actions, creation of new units, coordination among different units, etc. • Other implications: potential impact of the technology on the ethical, social, cultural

and/or legal domains.

21

In summary, a promising technology can be followed-up a the post-introduction stage following a selection step based on four main categories of prioritization criteria :

• Population/end users; • Characteristics of the technology; • Safety/adverse effects; • Organisation/costs and other implications.

22

CONCLUSIONS

I. Synthesis Overall, in spite of an heterogeneity of systems and criteria across investigated countries, several common categories could be identified.

First, the most frequently reported criteria was the identified uncertainty as established by HTA. Indeed, in all investigated countries (Ontario, USA, Australia, Spain, UK, France and Italy), uncertainty may concern the main HTA domains such as effectiveness, safety and cost-effectiveness.

Second, in nearly all investigated countries, uncertainty may also concerned the appropriateness of “in real life” use of the technology (in terms of safety, generalisability of RCT etc.) which may leading, in case of misuse, to the restricted use of the technology.

Third, most of investigated countries have considered as criteria the impacts on healthcare, mostly at the organisational and financial levels. Overall, these criteria aimed at identifying health technology requiring an anticipation of major investments or organisational changes.

Forth, some countries have considered the technical feasibility of additional studies, in terms of study design, funding, number of patients, timeframe, value for money or transparency.

Fifth, some countries have reported the use of technical criteria related to the health technology itself, regarding its degree of innovation, frequency of use, diffusion, or its potential impact on healthcare.

Sixth, some countries have also considered the burden of the disease (mortality, morbidity, disability, size of population, disease management etc) as a relevant selection criteria.

Last, some criteria were related to the implementation of the local AEG itself, as illustrated by some primary selection criteria for CED (impact on revised decision, dedicated funding, coordination between institutions (e.g CMS, FDA and NIH). As these criteria corresponds to key elements of success or steps of AEG, they cannot be considered as relevant selection/prioritization criteria, as they are (i) specific of AEG and thus non relevant for other areas of additional studies or (ii) included in the AEG definition for all AEG associated studies and thus non selective.

23

II. Domains of selection criteria Interestingly, each category of criteria provides a type of information which can be collected at different times during the process of assessment/appraisal (Fig 2).

Figure 2: Domains of selection of relevant additional studies

Indeed, before or at the beginning of HTA, the identification of potent healthcare needs is performed in order to characterize the context of the assessment and also, in some countries, to prioritize HTA. Healthcare needs are usually associated with diseases exhibiting a heavy burden, such as diabetes, cancer or chronic heart diseases, or with diseases associated with vulnerable population (e.g. rare diseases). The disease management is also critical in terms of existing alternative diagnostics and therapeutics. Politic pressure resulting form lobbying from patients, manufacturers or healthcare professionals may also be taken into account. Therefore, this initial step defines the context of HTA and by consequence, the context of related additional studies.

In a second time, during HTA, the value of a new health technology to cover healthcare needs are investigated, according to several domains (see Core Model), including mostly effectiveness, safety cost-effectiveness and organisational/legal aspects. The degree of innovation of the assessed health technology, its frequency of use, diffusion, or its potential impact on healthcare are also characterized during HTA.

In a third time, rather during the appraisal process following HTA, an identification of evidence gaps is performed, in order to characterize the uncertainty and the reasons why additional studies must be performed (lack of data on effectiveness? Large potential of misuse? Large investment to anticipate?). This step is directly linked to step 2, as evidence

24

gaps are strictly dependant of available evidence analyzed during HTA.

In a forth time, still during the appraisal process, the feasibility of performing additional studies may be discussed, regarding study design, funding, number of patients, timeframe, value for money or transparency.

Subsequently, for each domain, specific categories of selection criteria may be relevant, as illustrated in Figure 3.

Figure 3 Spreading of main categories of identified selection criteria.

25

III. Use of selection domains at the single AEG level Overall, this review of the literature suggests that an optimal selection of health technologies requiring the generation of evidence through additional studies needs to cover 4 domains:

1. what is the context in terms of healthcare needs? 2. what is the value of a new technology in covering these healthcare needs? 3. what are the evidence gaps ? 4. are the conduction of additional studies appropriate to raise uncertainty?

However, the use of all 4 domains is not mandatory. The use of 2 or 3 domains may be sufficient. Indeed, as illustrated in Figure 4, at the single AEG level, none of the investigated countries have selection criteria involving 4 selection domains.

Figure 4 Use of selection domains at the single AEG level

Together, these data stressed the need for a discussion on the scope of the development of selection/prioritization criteria by EUnetHTA WP7.

26

RATIONALE AND SCOPE OF THE DEVELOPMENT OF SELECTION/PRIORITIZATION CRITERIA BY EUNETHTA WP7A: POSITION OF PARTNERS

During the 1st WP7 face-to face meeting (Dublin, 10th June 2010), two working groups were assigned to discuss the rationale and the scope of the development of selection criteria within EUnetHTA, including the relevance of the four identified main domains in the further development of selection criteria:

• Domain 1: Healthcare needs; • Domain 2: Interest of the new technology • Domain 3: Types of evidence gaps • Domain 4: Appropriateness of additional studies

I. Rationale The development of selection/prioritization criteria in the context of EUnetHTA is relevant for all partners, particularly in order to improve the transparency of decision making. It was deemed necessary to clarify the stage at which the criteria should be implemented along the entire HTA process. Therefore, it was re-emphasized that the list of criteria is intended to select and prioritize promising technologies requiring additional evidence. The selection criteria would be of greater value at the end of the assessment, when evidence gaps have already been identified and not at the horizon scanning stage used to define HTA programmes. Considering the heterogeneity of the European health care systems and their respective coverage mechanisms, selection criteria should be independent from the local context.

II. Scope Regarding, the relevance of identified domains, EUnetHTA partners have agreed that the most important domain to consider is the type of evidence gaps (Domain 3). Appropriateness of additional studies (Domain 4) is also important as it is of little use to select/prioritize a technology for which additional studies cannot be performed. However, the title of Domain 4 should be renamed “feasibility of additional studies” instead of “appropriateness of additional studies”, to minimize confusion. Indeed, the word “appropriateness” is inadequate, as it implies a value judgment that could be a consequence of the application of the selection/prioritization criteria. It may be interesting to consider the “interest of the technology” (Domain 2). However, its title of should be renamed as “importance of the new technology”. As healthcare needs (Domain 1) have been considered in a prior HTA assessment, and considering that it may be redundant with some criteria of Domain 2 (the characterization of the importance of a new technology involves the consideration of major healthcare needs), the use of Domain 1 may be unnecessary. Finally, EUnetHTA partners have also considered the development of a prioritization/ranking process. However, based on previous international experiences (e.g. EuroScan), the development of a prioritization process may be a challenging task. Several methods of

27

prioritization/ranking can be investigated, such as : (i) a weighting according to their relative meaning and to the level/type of uncertainties previously assessed, (ii) privileging clinical criteria than other criteria, (iii) ranking domains instead of ranking criteria. Overall, based on the review of literature and working group discussion, EUnetHTA partners have stated that:

• The development of selection/prioritization criteria is of utmost interest for all partners and is considered particularly important to improve the transparency of decision making.

• The aim of these criteria is NOT to plan HTA but rather to select/prioritize new health technologies for which the generation of additional evidence might be necessary, as established by HTA.

• Selection/prioritization criteria should be independent from local context and not include items related to e.g. coverage mechanisms.

• The development of selection criteria should be performed in three domains: -type of evidence gaps -importance of the new technology (involving some criteria relative to healthcare needs) -feasibility of additional studies

• A prioritization/ranking process should be considered.

28

REFERENCES

1. Carbonneil C, Quentin F, Lee-Robin SH for the EUnetHTA. a common policy framework for evidence generation on promising health technologies. Int J Technol Assess Health Care 2009 25 (Suppl 2) 56-67. 2. Dhalla IA, Garner S, Chalkidou K, Littlejohns P. Perspectives on the National Institute for Health and Clinical Excellence’s recommandations to use health technologies only in research. Int J Technol Assess Health Care 2009 25:3 272-280. 3. Programs for Assessment of Technology in Health (PATH), Goeree R, Tarride JE, et al. Using conditionally funded field evaluations (CFFE) for evidence development, uncertainty reduction and reimbursement decision making: case studies from Ontario. HTAi 2008 Conference, Montréal, Québec, Canada. [Présentation]. Toronto, Canada. http://www.cmtpnet.org/recent-articles/topic-coverage-with-evidence-development/HTAi%202008%20-%20Goeree.pdf/view. Accessed January 15, 2009. 4. Ontario Ministry of Health and Long-Term Care (MOHLTC). Current Indications for Conditionally Funded Field Evaluation. 2009. http://www.health.gov.on.ca/english/providers/program/mas/field/field_cffe.html 5. Medical Services Advisory Committee (MSAC). Funding for new medical technologies and procedures: application and assessment guidelines. 2005. http://www.msac.gov.au 6. Centers for Medicare and Medicaid Services (CMS). National Coverage Determinations with data collections as a conditional coverage: coverage with evidence development. 2006. https://www.cms.hhs.gov/mcd/ncpc_view_document.asp?id=8 7. Center for Medical Technology Policy (CMTP), Schaffer DM. A multi-stakeholder approach to designing studies of emerging medical technologies. America's Health Insurance Plans (AHIP) Medical Leadership Forum, October 2007, Phoenix.[Présentation]. Baltimore, United-States. http://www.cmtpnet.org/cmtp-presentations/CMTP%20AHIP%20presentation%20v3%20%20092607%20dms.ppt/view 8. Ministerio de sanidad y consumo. El uso tutelado como mecanismo de actualizacion de las prestaciones: resultados de la experiencia piloto. 2007. http://www.msc.es/profesionales/prestacionesSanitarias/publicaciones/UsoTutelado.htm 9. National Institute for Health and Clinical Excellence (NICE), Citizen Council. Only in Research. 2007. http://www.nice.org.uk/media/129/29/OIRReport300407.pdf 10. Degos L. Conditionnal coverage of promising technologies. The French experience. HTAi 2008 Conference, Montréal, Québec, Canada. [Présentation]. Saint-Denis, France. http://www.htai2008.org/download.php?f=bc2eadd23f147d6b0e7ff1748477ccc0&countonly=1. Accessed January 15, 2009. 11. Stamenkovic.S, Maugendre.P, Xerri.B, Meyer.F. Post-listing studies of medicinal product in France. HTAi 2008 Conference, Montréal, Québec, Canada. [Présentation]. Saint-Denis, France. http://www.htai2008.org/ 12. Italian Medicines Agency (AIFA). Independent research on drugs. 2007. http://www.agenziafarmaco.it/en/content/independent-research-drugs 13. EuroScan International network. A toolkit for the identification and assessment of new and emerging health technologies. 2009 http://wwweuroscan.org.uk 14. Galician Agency for Health Technology Assessment (Avalia-t), Post-introduction observation of health technologies. Methodological guideline. 2007. http://avalia-t.sergas.es


2011-02-28



APPENDIX WP7.4_A

Presentation of EIFFEL during the WP7 face to face meeting in Dublin (June 2010)

EUNETHTA | European Network for Health Technology Assessment | www.eunethta.netEUnetHTA | European network for Health Technology Assessment | www.eunethta.net European network for HTA, EUnetHTA | www.eunethta.net

WP7- New TechnologiesWP7A – Facilitating evidence generation

“Feedback from Eiffel prototype ; Exploring areas of need and improvement”

HASCédric CARBONNEIL

Laura ZANETTI

EUNETHTA | European Network for Health Technology Assessment | www.eunethta.netEUnetHTA | European network for Health Technology Assessment | www.eunethta.net

Summary

• Background (2006-2008 EUnetHTA project)• Toolkit demonstration• Toolkit use during 2009• Plan for 2010-2012 joint action


Background (2006-2008 EUnetHTA project) :

Development of a Web-based toolkit to facilitate European collaboration

on evidence generation on promising health technologies


Web-based toolkit for information sharingEUnetHTA Interface to Facilitate Furthering of Evidence Level

� Helping HTA agencies to exchange and share information on evidence generation

• Structured and standardized forms for requesting or posting information on promising technologies

• An online queryable database, fed automatically when filling the forms:– information on the requests made by partners– information posted (spontaneously or following a request)

�Providing easy, user-friendly, and quick access to the information exchanged


(EIFFEL) toolkit provides information on:

• Level of diffusion of the technology in different healthcare systems

• Status of technology assessment (planned, ongoing, completed, report available)

• Status of monitoring actions and measures for evidence generation (planned, ongoing, or completed, results available)

• Protocols and results of clinical studies or registries

• Effective use of the new evidence (results of clinical studies or registries) for reassessment and/or revising a decision


Toolkit use demonstration

http://www.eiffel.eunethta.has-sante.fr/


Toolkit use results - 2009


Interest and use of toolkit by the partners

• 35 password attributed

• 68 % of the agencies requested a password (18 agencies)

• 29 technologies were entered in the database by 7 agencies (39 %)

• 7 technologies interested more than 1 agency

• 6 technologies have both requests and posted answers

�� LIMITED USE OF EIFFEL


Reasons of limited use of EIFFEL

• Information available in EIFFEL were poorly accessible and informative

�need to focus EIFFEL on the database function rather than on the information exchange function (request/post)

� Increase use-friendliness of EIFFEL

• Inability of EIFFEL to associate relevant similar topics�develop upgraded database functions including more

efficient queryable abilities


Reasons of limited use of EIFFEL

• Too many items to complete, some with redundancy�need to a restriction to the most relevant and non

redundant items

• Requests and posts not always focused on the aims of EIFFEL (some requests relevant to planned or ongoing HTA)

�creation of the POP database � more focused use of EIFFEL

�� OVERALL, NEED FOR A MODIFIED VERSION OF EIFFEL (EIFFEL 2.0)


Plan for 2010-2012 joint action


Eiffel database: Exploring areas of need and improvement (1)

�« Agency has an urgent need to have whatever information about …. »

�« Survival, quality of life data »

�« We’d like to know if there are complementary data collection (planned, on going or finished) on confirmation of effectiveness, appropriateness of use. We would like to have information on any kind of studies or data collection (registries...) »

The questions that emerged during the pilot phase can be divided into 2 main groups:

FURTHER CLINICAL INFORMATION

NEEDED FOR AN HTA REPORT

� « We‘d like to know if others Agencies are also evaluating the technology ».

�« Is this technique approved for use or reimbursed in any country? »

�« Who perform ? Where? Is there some restriction for use? »

�« It would also be useful to know how spread is such technology now. »

- TECHNOLOGY STATUS

- COVERAGE- USE


Eiffel database: Exploring areas of need and improvement (2)

• � Questions posted onto the Eiffel website were :– Not always aligned with EIFFEL’s underlying objectives– Too broad in nature requiring:

• Multiple responses• Additional inquiries to respond to the original question

• � Need to clarify the objectives of the web-based toolkit with :– WP 7 Goals i.e. « collaboration on the GENERATION of

additional evidence on new health technologies of common interest »

– Participation of all EUnetHTA members as highlighted in the pilot phase


Eiffel database: Proposal for improvement

• Design a proper database to meet the following objectives:– systematic identification of evidence gaps in the context

of an HTA (= rationale for the conduction of a study in a given country).

– information sharing on additional data requirements, planned or ongoing prospective data collection (i.e. additional studies that have been requested and/or funded by HTA Agencies ) with especially for each study, the key items identified by the NETSCC as the minimum dataset and the study results when available

• � support European collaboration on studies• � avoid duplication and promote global analysis of results


Eiffel database : The content (1)

All assessed HTAssessed NEWHT

AssessedHT requiring the

generation of additional evidence

AssessedNEW HT requiring the generation

of additional evidence



• Proposals for field items to be included in the database :• � Country, partner organization name• � New technology • � Type : device, drug, procedure (diagnostic,

therapeutic, screening)• � Name• � Therapeutic domain• � HTA status : yes (specify date of the HTA report and

e-link for retrieval) / ongoing / planned / no• � Status of the technology (Reimbursement/Coverage) :

covered/ conditional coverage / not covered



• Proposals for fields items to be included in the database :• � Explicit rationale for additional evidence generation:

– Document the gaps/uncertainties emphasized during the HTA into the database (clinical effectiveness, safety, condition of use,…)

– Clarification about why AEG is needed ? (e.g. insufficient level of evidence, inconclusive/not methodologically robust results, lack of appropriate comparator, difference between the population studied and the one who will really benefit, too few patients studied, …)

• � Explicit key research question • � Minimum dataset for each study (NETSSC)• � Status of the study requested/funded : planned, ongoing, ...

• � Potential Impact for the HTA re-assessment• � Study results when available and decision taken


Eiffel database : The structure

• Three areas of improvement :

• � Information on HT:– clear and simple – more ticked items and less free text– focused on key fields to avoid redundancy– easy retrieval

• � user-friendly IT functionalities

• � “visual” structure with rapid information access within each Eunethta member


Eiffel Database : The functionality

• Increase search capacities and information access • � Provide direct access via the home page to:

– type of HT (drug, device, procedures)– name of HT with an alphabetic listing and index : A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

• � Provide the opportunity to perform request into the database via the home page – by name of HT (direct)– by country– by study objectives – Etc …


Eiffel database: Process of data submission

New HT

1st assessment by an HT Agency

Information sharing for all members

Completion of the database

Etc…

Stay up-to-date

Evidence gaps additional data collection


Discussion

At this phase of the database’s development :

• � Identify and understand agency/organization needs

• � Receive feedback on the database’s scope, content, and functionalities before further technical development

• � Receive feedback on the process for data submission


Working Groups

� Brainstorming sessions on 4 questions :

• Are you able to request/ fund additional data generation ?

• Do new orientations proposed for the database match your agency/organization needs ?

• Are the field items proposed satisfactory ? • What operational aspects would you require for

an optimal/ regular use of the database ?

RED GROUP

BLUE GROUP

GREEN GROUP

YELLOW GROUP


Next steps

Planning for the database’s development:• � The development will be subcontracted to external IT

consultant• � Timelines:

• M10-M18: operational toolkit implementation• M19-M33: Upgrade web-based toolkit with all integrated functions• M33 : Operational web-based toolkit including database (Eiffel)

Next steps : After this meeting, • either we will consider that feedbacks from face-to-

face meetings will be sufficient• either we will integrate questions about Eiffel in the

already planned survey concerning selection criteria.


Working Groups

Please ask if any questions &

Thank you for your collaboration !


2011-02-28



APPENDIX WP7.5_A

WP7A survey on AEG and selection criteria (Sept 2010)

EUnetHTA WP7 (HAS Strand A) Questionnaire 1 – 2010 09 17 1/18

WP7

Strand A : Facilitating Evidence Generation of New Health Tech nologies:

Questionnaire n°1 :

Web-based toolkit (Eiffel Database) and Selection/P rioritization Criteria

Please e-mail the completed questionnaire by October the 4th to Laura Zanetti and Cédric Carbonneil

[email protected] / [email protected]

If needed, additional documents may be directly mailed to:

Sun ROBIN Service Evaluation des actes professionnels

Haute Autorité de santé - 2 avenue du Stade de France 93218 SAINT-DENIS LA PLAINE Cedex FRANCE

Thank you


Objectives of this survey The assessment of a new technology may highlight the interest of the technology but also the evidence gaps and the necessity of further research to reduce remaining uncertainties about its benefit for the patient and the healthcare system. In this case, the most promising ones may benefit from measures and further investment to collect new evidence in order to answer remaining research questions. The generation of new evidence may be performed within a specific framework. Those frameworks may comprise mechanisms linking decision to provide access to a given technology (coverage, reimbursement or funding) to requirements of additional evidence. This kind of mechanism is referred here as Access with Evidence Generation (AEG). The generation of new evidence is referred here as additional data collection (ADC) and may comprise trials, observational studies, registries, data on appropriateness of use, etc. The general objective of the WP7 strand A is to support collaboration on generation of additional evidence on new technologies and to share timely and useful information about data requirements and about planned or ongoing prospective data collection. Three deliverables are associated:

A. Development of a set of criteria to select or prioritize technologies for which further research is mostly relevant (needed and useful).

B. Implementation of a web-based database for structured exchange and storage of information on additional data collection on new technologies.

C. Definition of an agreed dataset to facilitate exchange of information on planned or ongoing data collection.

During the WP7 face-to-face meeting held in Dublin on June 10th, 2010, all partners expressed a real interest in developing such tools but it seemed that not all would be able to contribute equally to the development of the WP7 strand A activities. Therefore, subgroups of partners will be solicited specifically, according to their ability to provide input in different phases of the development of deliverables. Indeed, only a subgroup of partners from countries having experience on applying AEG mechanisms may provide specific information into the Eiffel database. Those partners were identified as being able to recommend, request, and/or fund the generation of new evidence or ADC linked to coverage decisions. They will constitute the group 1 (see below).

GROUP 1 INFARMED (PT),

CVZ (NL), Agenas, Regione del Veneto, AIFA (IT),

NETSCC, NICE (UK), AETS (SP), HAS (FR) SBU (SW)

SNHTA (SZ)


Your organisation/agency is within this group and may provide significant contributions to the development of deliverables A and B at this stage. To identify common interest and needs as well as who would have an active role in providing input to the database, we need first to understand the AEG mechanisms and ADC processes applied in different countries, as well as the key organisations/bodies/stakeholders involved. This first survey will followed by a second one more focused on identifying which information is shareable, of common interest, can be easily provided and retrievable from the Eiffel database. Your input is of utmost importance to develop deliverables useful and applicable by all EUnetHTA partners. Therefore, you are kindly requested to provide information concerning your country/agency by responding to this survey. The questionnaire contains 40 questions divided onto the following 6 specific areas:

1) General information on ADC process 2) Decision making process for ADC 3) Use of explicit criteria to select and prioritize 4) Transparency of ADC process 5) ADC or study protocols 6) Impact of additional data collected

This survey takes approximately 20 minutes to be filled in. A summary of the survey results will later be send to all WP7 partners. Specific guidance for answering the questionnaire

• Please remember that multiples answers are possible for all questions; • If your agency/organization is involved in the assessment of different types of

technologies, please specify in your response whether it concerns all types of HT assessments (all HT) or just a particular type (D= Drugs; MD = Medical Devices; P = Procedures; S = screening).

By responding to the survey, you will contribute to the achievement of the WP7A objectives and we thank you very much for your participation . PLEASE INDENTIFY YOURSELF BELOW: Name: …………………………………………………………………………………………………… Position: ….……………………………………………………………………………………………… E-mail address: .………………………………………………………………………………………... Telephone number: …………………………………………………………………………………… Agency/Institution: ……………………………………………………………………………………... ……….…………………………………………………………………………………………………… …………………………………………………………………………………………………………… Country/region:………………………………………………………………………………………….


1) GENERAL INFORMATION ON ADDITIONNAL DATA COLLECTI ON PROCESS Please remind that multiple answers are possible fo r all questions.

1. Please specify, for each type of technology, to which category your agency/organization belong:

D MD P S All

HT My Agency/organisation is able to recommend * ADC My Agency/organisation is able to request ** ADC My Agency/organisation is able to fund*** ADC This is the role/mission of other organization in my country (please specify) : …………………………………………………………………… ……………………………………………………………………

D (=drugs); MD (= medical device); P (=procedures); S (= screening program). * Recommend = simply proposing ADC. No strings attached. Decision makers may follow the recommendation and request ADC ** Request = formally asking for ADC with special conditions or restrictions to provide coverage or reimbursement. *** Fund= financial support for the required studies, registries, other data collection etc)

2. What coverage/reimbursement mechanisms in your country/region are used to recommend/ request/fund additional data collection (ADC)?

D MD P S All

HT Conditional coverage mechanisms Coverage with specific restrictions: Please describe (e.g. agreement to perform the study, sanction in case of study not conducted,…): …………………………………………………………………….. ……………………………………………………………………

Coverage without conditions No coverage until results of requested study are made available

Others, please specify : ……………………………………………………………………. ……………………………………………………………………

D (=drugs); MD (= medical device); P (=procedures); S (= screening program).


3. In which situation your Agency is able to recommend/ require/ fund/ ADC? D MD P S All

HT Only new and innovative health technologies New health technologies (e.g. new indication) Health technologies already used/on the market (e.g. in the context of extension of indication,…)

Whatever health technology assessed Other, please specify : …………………………………………………………………… ……………………………………………………………………

4. Your Agency’s recommendation or requirement for ADC concerns mainly:

a specific request/recommendation concerning a given HT in the context of/during a single HTA

a common request concerning different HT in the context of/ during multiple HTA (e.g. comparative studies, or data regarding a class of drugs)

Other, please specify………………………………………………………………… ………………………………………………………………………………………………

5. In your country, please identify the different sources of funding for studies/data collection requested for each type of technology.

D MD P S All

HT My agency/organisation’s own funds Private (e.g. industry), please specify in what situation: ….. …………………………………………………………………………………………………………………………………………

Public (e.g. other Governmental bodies, public research organisation) please specify in what situation:……………... …………………………………………………………………………………………………………………………………………

Public/Private Partnership (e.g. foundation), please specify in what situation: ……………………………………............... …………………………………………………………………………………………………………………………………………



2) DECISION MAKING PROCESS FOR ADDITIONNAL DATA COL LECTION

6. Who are the key players involved in discussing the relevance/needs for additional data collection in your country/region?

My HTA agency Other institution(s), e.g. Ministry of health, other regulatory

authority, others external committees. Please specify:………………. ……………………………………………………………………………………………………………………………………………………………………………………

7. If other institution(s) are involved, does a collaboration/coordination exist between your agency/organization and them? (e.g in France, a coordination exists between French Health Products Safety Agency (Afssaps) and HTA agency (HAS) to request additional studies for drugs)

YES NO Not applicable

Please specify:................................................................................................. ……………………………………………………………………………………………………………………………………………………………………………………..

8. Which elements are needed to inform the discussion about the relevance/needs of ADC in your country/region?

Explicit identification of evidence gaps Explicit definition of research questions Explicit objectives and rationale for ADC Draft protocol of required studies Pre-identified possible sources of funding (if relevant) others: Please specify:…………………………………………….

……………………………………………………………………………………….. ………………………………………………………………………………………..

9. Once the relevance/needs for ADC have been stated, who takes the final decision of ADC recommendation/requirement in your country/region?

My HTA agency External appraisal committee Ministry of health Other(s) institution(s) Please specify:...........................................

……………………………………………………………………………………………………………………………………………………………………………………


10. In case of other institution(s)/bodies take the final decision, could you approximately estimate the time between your recommendation/request and the final decision for ADC in your country/region?

Decision is taken at the end of the discussion on the relevance for ADC (same time)

< 15 days 15 days- 1 month 1-3 months 3-6 months >6 months

11. Could you provide the chronology of the different steps of the process, as established in your country region? (just fill the boxes on the right side with the number of the step in chronological order; if not applicable, fill N.A.). The order of the steps presented below is according to the French ADC process. 1. Assessment of the technology (HTA) 2. Explicit identification of evidence gaps/research needs 3. Discussion on the opportunity of ADC (based or not on explicit

selection/prioritization criteria) 4. Explicit research question for the requested ADC 5. Decision of coverage with or without ADC 6. Proposal of the study protocol 7. Validation of the study protocol 8. Implementation of requested study 9. Reassessment of the HT integrating new evidence 10. Revised decision on coverage 11. Others, please specify:………………………………………………..


3) USE OF EXPLICIT CRITERIA TO SELECT AND PRIORITIZ E

12. Is there in your country/region a process to select and prioritize of HT for which ADC is relevant and or needed?

No at all No, but reflexion about such a process is currently initiated Yes, a formal selection process is in place Yes, an informal selection process is in place

Please specify:..…………………………………………………………… ………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………

If you answered NO to this question, please go to Question 16.

13. If a selection/prioritization process is in place, who is in charge of selection/prioritization of health technologies for which ADC is needed in your country/region?

HTA agency Specific appraisal committee Ministry of health Manufacturers Healthcare professionals National insurance funders Private insurance funders Specific funders Other institution(s) Please specify:…………………………………..

……………………………………………………………………………………………………………………………………………………………………………………

14. Within the process currently in place, is the selection of technologies for which ADC is the most relevant, based on explicit criteria?

yes no if no please specify the reasons.....................................................

……………………………………………………………………………………………………………………………………………………………………………………

15. If you answered YES to the previous question, please describe these criteria : ………………………………………………………………………………………………. ……………………………………………………………………………………………… ……………………………………………………………………………………………… ………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………


16. Are there any specific needs or interest you can identify for your

agency/organisation regarding the development of selection/prioritization criteria by EUnetHTA? Please specify:………………………………………………………………………. …………………………………………………………………………………………………………………………………………………………………………………… ………………………………………………………………………………………… …………………………………………………………………………………………


4) TRANSPARENCY OF ADC PROCESS

17. If a decision for ADC is taken by your Agency, are your specific recommendations/ requests for ADC made public?

Yes, always (e.g. dedicated section of HT report) Yes, sometimes (no specific rules) No, never Not applicable

If, yes, please specify which public documents state the need for ADC (e.g. HT assessment report, other documents…)? Is this information easily accessible? (e.g. published on the website) ………………………………………………………………………………………… ……………………………………………………………………………………………………………………………………………………………………………………

18. If the final decision for ADC is taken by other institutions/organization (e.g. external appraisal committee, Ministry of health,…), is the information made public?

Yes, always (e.g. dedicated section of HT report) Yes, sometimes (no specific rules) No, never Not applicable

If, yes, please specify which public documents state the need for ADC (e.g. Ministry statement…)? Is this information easily accessible? (e.g. published in the website) ………………………………………………………………………………………… ……………………………………………………………………………………………………………………………………………………………………………………

19. Is the rationale leading to ADC decision clearly stated in these documents? Yes, always (e.g. remaining uncertainties on effectiveness, need

to highlight evidence gaps clearly stated.) Yes, but not systematically No, no explanation provided Not applicable


5) ADDITIONNAL DATA COLLECTION/ STUDY PROTOCOLS

20. In your country, who is involved in the definition of the research question (i.e. study objectives) to be answered by the ADC?

My HTA agency External appraisal committee Ministry of health Other(s) institution(s) Please specify:……………………………….

……………………………………………………………………………………………………………………………………………………………………………………

………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………

21. Which objectives are usually meant to be answered by the ADC? D MD P S All

HT Clinical efficacy Safety Condition of use Effectiveness Comparative Effectiveness Patient reported outcome (e.g. quality of life) Health Economic including cost-effectiveness Organisational aspects Others (e.g. ethical, societal aspects) : please specify: ……………………………………………………………………... ……………………………………………………………………...


22. From your experience, what are the main types of studies and/or data collection conducted to answer to the research question?

D MD P S All

HT Studies under experimental clinical conditions including randomized clinical trials

Studies in real life setting including observational studies such as registry, cohort, case-control,…

Randomized pragmatic trials Others (e.g. descriptive epidemiology studies, models) , please specify: …………………………………………………... …………………………………………………………………….. ……………………………………………………………………..



23. Which of the following entities in your country/region develop the study protocol ?

D MD P S All

HT My HTA agency Healthcare professionals organisations Manufacturers Specific funders Other(s) institution(s): please specify: ……………………………………………………………………... ……………………………………………………………………..


24. Is your Agency involved in the assessment of the adequacy of the study protocol and the research question in your country/region?

D MD P S All

HT Yes (e.g. Scientific advice or Agreement with the sponsors or Formal Approval) please specify: ……………………………………………………………………... ……………………………………………………………………...

No Not applicable


25. Is your agency/organization involved in the collection of the required data ? D MD P S All

HT Not at all Yes, data collection is always performed by the agency/organisation. Please specify:………………………... ……………………………………………………………………...

Yes, data collection is sometimes performed by the agency/organisation. Please specify in which situation(s): ……………………………………………………………………... ……………………………………………………………………...

Not applicable D (=drugs); MD (= medical device); P (=procedures); S (= screening program).


26. Is your agency/organization involved in the statistical analysis of the data? D MD P S All

HT Not at all Yes, statistical analysis is always performed by the agency/organisation. Please specify:…………….…………... ……………………………………………………………………...

Yes, statistical analysis is sometimes performed by the agency/organisation. Please specify in which situation(s):………………………….......................................... ……………………………………………………………………...

Other : please specify: ……………………………………………………………………………………………………………………………………………………………………………………………………………….


27. In your country, once the final decision for ADC has been taken, is there any regulatory timeframe for the study implementation?

D MD P S All

HT Yes, please specify: ……………………………………………………………………... ……………………………………………………………………...

No Not applicable


28. In your country, what is the average time allowed between the final decision for ADC and the study’s implementation?

D MD P S All

HT 0-6 months 6-12 months 12-24 months > 24 months Not applicable



29. In your country, once the final decision for ADC has been taken, is there any regulatory timeframe for study results’ submission?

D MD P S All


No Not applicable


30. In your country, what is the average time allowed between the final decision for ADC and the study results’ submission?

D MD P S All

HT 1-2 years 2-3 years 4-5 years > 5 years Not applicable


31. Are there any penalties in case of the study has not been conducted? D MD P S All


No Other: please detail : ……………………………………………………………………... ……………………………………………………………………...


32. Are intermediary results or other relevant information regularly submitted to your agency or organization during the conduction of such studies? (e.g. inclusion follow-up, intermediary results,…)

D MD P S All


No, only final results. Not applicable



33. In your country, who is in charge of the assessment of study or data collection results?

D MD P S All

HT My HTA agency Healthcare professionals organisations External appraisal committee Ministry of Health Specific funders Other institution(s): please specify: ……………………………………………………………………... ……………………………………………………………………..


34. In case your agency/organization is involved, when these results are taken into account for the HTA update?

D MD P S All

HT Systematically at the time of the pre-planned re-assessment of the HT (e.g. in France, ADC results are taken into account at the time of every 5-year reassessment of drugs)

As soon as the results are received, a re-assessment of the HT is systematically initiated

As soon as the results are published, a re-assessment of the HT is systematically initiated

Anytime an important issue is raised by the results Not applicable



6) IMPACT OF ADDITIONAL DATA COLLECTION/STUDIES:

35. How long the AEG mechanism has been in place in your country? …………………………………………………………………………………………………. ………………………………………………………………………………………………….

36. In your country/region, how many studies/data collection are recommended/ requested/ funded on average per year for each type of technology?

__________ for drugs __________for procedures __________ for medical devices __________ for screening programs

37. In your country/region, how many studies/data collection results are available

on average per year for each type of technology? __________ for drugs __________ for procedures __________ for medical devices __________ for screening programs

38. From your experience, do you think the results of these studies have had a

real impact on coverage decision/recommendation in your country/region? Yes, results can contribute to a revised decision leading to non

coverage or delisting Yes, results can contribute to a revised decision leading to change

in coverage conditions (e.g. coverage with restriction for specific population, indication)

No, no contribution regarding final decision (please comment, e.g. difficulty to modify initial decision) ………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………

Other, Please specify: ………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………

39. From your experience, which are the main difficulties encountered in the current ADC process in your country/region? (e.g. lack of dedicated funds, difficulties to request head to head studies..)

Please specify: ……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………… …………………………………………………………………………………………………

40. Please provide an explicative scheme summarising the 5 steps of AEG process in your country/region for each type of technology concerned (A4 format): As example, the AEG process in France is presented hereafter:


GLOBAL ORGANIZATION OF ADDITIONNAL EVIDENCE GENERATION FOR

DRUGS AND MEDICAL DEVICES IN FRANCE (HAS)

*Gentleman agreement for drugs

**NB: For drugs: in case of non conduction of the study, possible

sanctions to company (e.g. price revision)

HAS

Post-listing

studies working group

2 HAS Appraisal

Committees:

- Drugs

- Medical Devices

and procedures

Ministry of Health Economic committee for health products

Pharmaceutical companies

Coverage decision

for drugs, MD, and

innovative procedures

HAS

2 Appraisal Committees

HAS Recommendation (HTA Report:):

- Coverage recommendation

- Evidence gaps identification

- Rationale and need for ADC

- Definition of study objectives (research

question)

ADC request for drugs & medical devices

Revised coverage decision

For drugs** & medical devices

Price negociation

HTA

Agency (HAS)

Ministry of Health Economic committee for health products

Revised Price

Negotiation

Confirmation and

notification for ADC

request for drugs* & medical devices

Device companies

Step 2:

coverage decision

Step 1: Initial

Health

technology assessment

HAS

Post-listing


Protocol

submission for approval

- Protocol elaboration

- Conduction of the Study

- Statistical Analysis - Results submission

Every 5 years or every 3 to 5 years:

- Reassessment with review of the studies

results

- Revised coverage recommendation

-

Step 3: Generation

of additional

evidence

Step 4: Health

technology re-

assessment

integrating new evidence

Step 5: Revised coverage decision


GLOBAL ORGANIZATION OF ADDITIONNAL EVIDENCE GENERATION FOR

INNOVATIVE PROCEDURES IN FRANCE (HAS)

HAS

Post-listing


1 HAS Appraisal

Committee:

- Medical

Devices and procedures

Ministry of Health

Conditional

coverage for

innovative

procedures

HAS

1 Appraisal Committee

HAS Recommendation (HTA Report:):

- Coverage recommendation (Conditional

coverage only for innovative procedures &

devices - new legislative framework)

- Evidence gaps identification

- Rationale and need for ADC

- Definition of study objectives (research question)

Public Sponsors/

Healthcare professionals

ADC request for innovative procedures

HTA

Agency (HAS)

Step 2:

coverage decision

Step 1: Initial

Health

technology assessment

HAS

Post-listing

studies

working group

Protocol

submission for approval

- Protocol elaboration

- Conduction of the Study

- Statistical Analysis

- Results submission

- Reassessment with review of the studies

results - Revised coverage recommendation

Payers (Health Insurance)

Standard coverage

decision and tariff Negotiation

If positive coverage recommendation

Confirmation and

notification for ADC

request for innovative

procedures

Step 3: Generation

of additional

evidence

Step 4: Health

technology re-

assessment

integrating new evidence

Step 5: Revised coverage decision


2011-02-28



APPENDIX WP7.6_A

WP7A survey on AEG and selection criteria: main results (Dec 2010)

E U n e t H T A Summary of main results of WP7A survey n°1:

Additional Evidence Generation process and selection criteria

W ORK PACKAGE 7 DECEMBER 2010

The EUnetHTA-Joint Action is supported by a grant from the European Commission


2

EUnetHTA

Summary of main results of WP7A survey n°1:

Additional Evidence Generation process and selectio n criteria

was developed by

Work Package 7 : “New technologies”

Work Package 7 Lead Partner: HAS, French National Authority for Health

December 2010

3

Objectives of the survey The general objective of the WP7 strand A is to support collaboration on generation of additional evidence on new technologies and to share timely and useful information about data requirements and about planned or ongoing prospective data collection. Three deliverables are associated:

A. Development of a set of criteria to select or prioritize technologies for which further research is mostly relevant (needed and useful).

B. Implementation of a database on additional evidence generation or data collection on new technologies.

C. Definition of an agreed dataset to facilitate exchange of information on policy relevant planned studies.

During the WP7 face-to-face meeting held in Dublin on June 10th, 2010, all partners expressed a real interest in developing these deliverables, but it seemed that not all would be able to contribute equally to the development of the WP7 strand A activities. Therefore, subgroups of partners will be solicited specifically, according to their ability to provide input in different phases of the development of deliverables. Only a subgroup of WP7 partners from countries having experience on applying Access with Evidence Generation (AEG) mechanisms may provide specific inputs into the Eiffel database. Those partners were identified as being able to recommend, request, and/or fund the generation of new evidence or Additional data collection (ADC) linked to coverage decisions. They constitute the group 1 (11 partner organisations; see below).

GROUP 1 (n=11) INFARMED (PT),

CVZ (NL), Agenas (IT),

Regione del Veneto (IT), AIFA (IT),

NETSCC (UK), NICE (UK), AETS (SP), HAS (FR) SBU (SW)

SNHTA (SZ) This survey was performed to :

i) understand the AEG mechanisms and effective data collection processes applied in different countries, as well as the key organisations/bodies/stakeholders involved and

ii) to identify common interest and needs as well as who would have an active role in providing input to the database, .

4

The results of the survey will lead to an adaptation of the EIFFEL database (structure and items) and contribute to the development of criteria to select technologies requiring further research. All eleven partners from group 1 have received a questionnaire containing 40 questions, divided onto the following 6 specific areas:

1) General information on ADC process 2) Decision making process for ADC 3) Use of explicit criteria to select and prioritize 4) Transparency of ADC process 5) study protocols for ADC 6) Impact of additional data collected

All partners (11/11) from group 1 have answered to the survey between September and November 2010. Considering that this survey has collected an important amount of information, and that some additional clarifications are needed, the complete results will be presented in the next WP7 face to face meeting in La Valetta, Malta, on March 2011. However, in order to provide to WP7 partners a first feedback on the survey, the present document provides a summary of the main results of the survey. 1. Remit of partner organisations in additional evi dence generation or data collection (ADC) process. Among the 11 EUnetHTA WP7 partners from group 1 (including 9 HTA agencies and 2 medicines agencies):

• All partners are able to recommend or request ADC; • But only 4 partners are able to fund ADC.

2. Type of technologies considered for ADC • All types of technologies can be considered for ADC (medicines, medical

devices, procedures, screening); • Only the two medicine agencies having specific remit are able to fund

additional evidence generation on medicines; • All four organisations having ability to fund additional research, may consider

funding for medical devices procedures, and screening tests; • Additional evidence generation or data collection are mostly requested for new

health technologies already on the market (9/11).

3. Type of additional evidence required

• Clinical efficacy; • Comparative effectiveness; • Conditions of use; • Health economics.

5

4. Role of partner organisations in the ADC process • All but one partner (10/11) are key players in discussing the relevance/needs

for ADC for a given health technology; • Half of them (5/11) share this role with other institutions (Ministry of Health,

other regulatory authorities, etc); • 9/11 partner organisations are involved in the definition of the research

question; • 7/11 participate in the decision making process to fund ADC, in collaboration

with other institutions (Ministry of Health in general); • 2/11 (the two medicine agencies) are the single decision makers for ADC; • In general, partner organizations are NOT involved in the:

• design definition of the required study (7/11). Overall, this step is mostly performed by manufacturers (7/11), other academic partners (7/11) or healthcare professional’s organizations (5/11)

• effective data collection (7/11) and • statistical analysis of collected data (8/11)

• 8/11 are involved in the assessment of collected additional data. 5. Process to select or prioritize health technolog ies for ADC

• A formal process of selection/prioritization is implemented in only one third of group 1 partners (4/11)

• Implementing such formal process is currently being considered or have been initiated in half (5/11) of group 1 partners

• In countries where a formal process of selection/prioritization is implemented and/or is under discussion (8/11), the institutions in charge of such process are HTA Agencies (5/8), or the Ministry of Health (2/8) and/or others institutions or committee (4/8).

6. Elements considered when considering the relevan ce and need of ADC

• Explicit identification of evidence gaps (11/11); • Explicit objectives and rationale for ADC (10/11); • Explicit definition of research questions (7/11); • Draft protocols of the required study (2/11); • Source of funding (1/11).

7. Source of funding for ADC

• ADC could be financed either by private or public funds (7/11) • If an assessment is performed under the submission of a dossier by a private

company in the view of obtaining coverage for their technology, the required ADC is performed by private funds;

• If there is a public interest and no private source of funding available, the required ADC is performed by public funds;

• 3 organizations can finance ADC with their own funds.

6

8. Information on ADC recommendations/request are publicly available

• 9/11 of group 1 partners systematically publish their ADC recommendations, mostly within their HTA reports or on their organisations’ websites;

• The rationale leading to ADC decision is systematically reported on a public document in 7/11 of group 1 partners.

9. ADC mostly concerns studies in usual “real life ” setting

• All group 1 partners have reported to recommend, request or fund further research through studies in “real life” settings.

• Some of them (6/11) may also consider studies under experimental conditions; • But only 3 of them may consider randomized pragmatic trials.

Further detailed and additional results will be pre sented in the next face to face meeting in March 2011 including

• Timeframes for study implementation, • impact of additional data collected on revision of coverage decisions

Dataset and registry for clinical studies in developmentDataset and registry for clinical studies in developmentDataset and registry for clinical studies in developmentDataset and registry for clinical studies in development

Aim of the Dataset

To create a registry for primary clinical research studies which are being considered either by a public funder, or on the advice of a

national appraisal body.

Such a registry would allow research funders and appraisers to consider changes to planned studies-eg to enable easier meta-analysis.

For further information click here to access the introductory note.

Delphi Instructions

1. Please click here to review the dataset.

2. Then please answer the following questions about the dataset.

There are 4 sections; The WHOLE dataset, SPECIFIC data items, Missing information and Possible implementation of a registry.

Please click NEXT when you have answered all questions on a page. You can click PREVIOUS should you wish to go back to the page

before. Please click FINISH at the end to submit your answers. Hyperlinked sites will open in new pages.

Please submit your responses by 29th October 2010.

Many thanks for your help with this project!

1. Introduction


The dataset suggests responses in 2 languages; *‘Natural language’ - the native language of the organisation considering a study. *'Common language’ - either a default human language or a coding system (eg MeSH). English is suggested as the default human language as it is the most common language used in the HTA community, HTAi and INAHTA.

1. What language(s) should be used in the dataset?

2. If a COMMON language is to be used, what should this be?

2. The WHOLE Dataset

ONLY natural language

nmlkj

ONLY common language

nmlkj

BOTH natural language AND common language

nmlkj

Please comment if you wish;

55

66

English

nmlkj

Coding system (controlled vocabulary)

nmlkj

Both English & coding system

nmlkj

Not applicable - I don't think a common language should be used

nmlkj

Other

nmlkj

If 'Other' please specify what should be used;

55

66

Dataset and registry for clinical studies in developmentDataset and registry for clinical studies in developmentDataset and registry for clinical studies in developmentDataset and registry for clinical studies in development3. The dataset suggests MeSH as the most appropriate coding system (controlled

vocabulary). click here for MeSH information

Is MeSH the correct choice?

4. If MeSH should be used, would MeSH terms or codes be better?

5. The dataset has been based on PICO; Patient, Intervention, Control, Outcome.

Should the 'Intervention & Control' be combined into one section as 'Technologies'?

Or would they be better separated?

MeSH is appropriate

nmlkj

Other would be better

nmlkj

If 'Other would be better', please specify what would be better and why;

55

66

MeSH terms

nmlkj

MeSH codes

nmlkj

Either MeSH terms or codes

nmlkj

Not applicable - MeSH is not appropriate

nmlkj

Please comment, if you wish;

55

66

COMBINE 'Intervention & Control' in 1 section

nmlkj

Keep 'Intervention' & 'Control' as 2 SEPARATE sections

nmlkj

Please explain any advantages of keeping Intervention and Control as separate sections;

55

66


1. Contact email

Is EMAIL the best default contact method?

2. Study title

We suggest recording the study's title in English.

A native language title may capture subtleties lost in translation, although these

would only be apparent to speakers of that language.

Should a description in native language also be included?

3. SPECIFIC Data Items

YES

nmlkj

No

nmlkj

If 'No' please specify what would be better & why;

55

66

Yes

nmlkj

No - an English version is sufficient

nmlkj


55

66

Dataset and registry for clinical studies in developmentDataset and registry for clinical studies in developmentDataset and registry for clinical studies in developmentDataset and registry for clinical studies in development3. Internal Reference

Does your organisation use internal reference numbers for projects, that could be

made public?

4. Apart from an internal reference number, are there any other points of identification

which should be used to identify studies?

5. Research question

Is there added value in recording the research question in native language as well as

English?

Yes

nmlkj

No

nmlkj

If 'No', please explain;

55

66

No

nmlkj

Yes

nmlkj

If 'Yes', please specify what else should be used;

55

66

Yes - would like native language too

nmlkj

No - happy with English alone

nmlkj


55

66

Dataset and registry for clinical studies in developmentDataset and registry for clinical studies in developmentDataset and registry for clinical studies in developmentDataset and registry for clinical studies in development6. Source of Research Question

Early discussion has shown that some organisations are interested in where a

research idea has come from eg Government Report, HTA Organisation Report, etc.

Should we record this as a specific referenced source (eg NICE IPG 262, click here

for NICE IPG 262 ), or according to a classification scheme?

7. Outcome

We think that records should be closed once a decision has been made on whether

or not to fund a project.

This set of outcomes are suggested;

* Did not proceed - finished

* Did not proceed - alternative question identified

* Commissioned as RCT

* Comissioned as Observational Study

Should any other outcomes be included?

Full referenced source

nmlkj

Classification scheme

nmlkj

If 'Classification scheme', please recommend which scheme would be best;

55

66

No

nmlkj

Yes

nmlkj

If 'Yes' - please specify what other outcomes should be included;

55

66

Dataset and registry for clinical studies in developmentDataset and registry for clinical studies in developmentDataset and registry for clinical studies in developmentDataset and registry for clinical studies in development8. Should extra information be included, such as the record number in a clinical trials

registry (when this becomes available)?

No

nmlkj

Yes

nmlkj

If 'Yes', please specify what extra information should be included;

55

66


1. We aim to keep this dataset small - to allow users to identify potential overlaps

between planned studies then discuss these in greater depth with the appropriate

organisation(s).

However, is anything missing which would help identify studies?

4. Missing Information

No

nmlkj

Yes

nmlkj

If 'Yes', please specify what is missing;

55

66


1. Pre-registration

Should the dataset ‘pre-register’ participants – ie should participating organisations

be coded, to include contact details?

Items 1 & 2 would therefore be replaced by a single code.

This would also imply one contact per organisation rather then one per study.

Eligibility criteria

It is currently anticipated the following organisations would be accredited to enter policy relevant primary research on the register (and to

use the register to match their research plans against those already registered);

Public appraisal and research funding agencies (organisations that supply funds for research or who have approval of research

protocols) – who produce policy relevant primary research.

2. Can establishing the precise eligibility details be left until the 'register

implementation phase', to be led by HAS from mid-2011?

Unit of Registration

A given grant (or potential grant or application) may contain a number of sub projects – eg a literature review, a survey and a trial.

We suggest that the unit of registration should be an individual research question (as each sub project should have it’s own research

question).

Therefore, in the above example only the trial and possibly the survey would be eligible to register. The literature review would not be - it

should be registered in the registry for literature reviews click here for further information.

5. Possible implementation of a register

Yes

nmlkj

No

nmlkj


55

66

Yes

nmlkj

No

nmlkj

If 'No', please explain why not;

55

66

Dataset and registry for clinical studies in developmentDataset and registry for clinical studies in developmentDataset and registry for clinical studies in developmentDataset and registry for clinical studies in development3. Is a separate research question an appropriate unit of entry?

Yes

nmlkj

No

nmlkj

If 'No', please explain what would be better;

55

66


1. Do you have any other comments on the dataset or the overall project?

Many thanks for your responses.

Now please click FINISH to submit your responses.

6. End

55

66


2011-02-28



APPENDIX WP7.7_A

Survey on Dataset for trials in development: first Delphi round (Oct 2010)

of 4

Dataset for Trials in Development – a report on Delphi round 1 – October/November 2010

Background 1) In the summer of 2010, NETSCC and HAS set out to recruit organisations to

contribute to a Delphi process to develop a dataset to inform a registry for policy relevant primary studies in development.

2) Thirteen organisations from nine countries agreed to participate in the first round.

3) The first round survey was sent to these participants in October 2010. By 8th November 12 (92%) had responded.

Survey Responses

Dataset as a whole

Issues of language 4) As the conductors of this survey are distinctly Anglophone, we wished to explore

and get consensus on which language(s) should be used within the database.

5) Responders from other Anglophone countries exhibited similar anxieties – that an English only (or English first) dataset might not be manageable for organisations where English is not the first language.

6) All organisations from countries where English is not the common language of everyday use supported English being the main language used in the dataset. We shall therefore adopt English as the default language for the dataset.

7) ¼ of respondents suggested some kind of native (ie non-‐English) language involvement would be useful. We shall explore this further in the next Delphi round.

Coding Systems 8) One half of respondents recommended that a coding system / controlled vocabulary

be used in addition to English.

9) Of the 10 respondents who expressed opinion on a coding system to use, all supported the use of MeSH. One commented that in some cases a code may not exist within an adopted coding system, in which case the human readable language would be the appropriate fall back.

10) Given 8) and 9) we will allow entry of both English and MeSH, but see 11) and 12) below.

11) All 10 respondents who expressed an opinion agreed that MeSH terms could be used, only one was also supportive of MeSH codes. Therefore, we will suggest that any database using this dataset support entry and display of MeSH terms. The

of 4

ability to do this will however depend on the implementation platform. One respondent indicated that using codes would require more work than using terms.

12) One respondent suggested we consider the use of ATC (the Anatomical Therapeutic Classification System -‐ http://en.wikipedia.org/wiki/Anatomical_Therapeutic_Chemical_Classification_System) , a system currently in use by the European Medicines Agency. We will explore the added value of adding this to MeSH in the next Delphi round.

PICO 13) We were able to identify PICO as a structure for recording information, which

would allow easy matching of overlapping planned studies. We asked the group if it is sensible to separate intervention and control under different headings.

14) 70% of responders favoured keeping them separate. The suggested advantages related to

a) clarity in the designs of trials, and the use of technologies in different countries

b) searching within the database for technologies which are being compared

c) the absence of comparator indicating that a study may not be comparative

15) We will therefore maintain intervention and control as separate fields. There is an implication for a database however that the records should be searchable by either of these fields individually, or both combined.

Specific Dataset Items

Contact 16) All respondents indicated that email is an appropriate default contact method. We

will therefore adopt email as the default contact method.

Study Title and Research Question 17) We wondered whether, even if we were recording most data in English, there might

be some added value of recording a study title in its native language – eg French for a study originating from France. We considered that there might be some subtlety of language which other co-‐language speakers might benefit from.

a) The majority of responders did not see a need for a native language title or research question.

b) The two responders who thought it might be useful were not native English speakers

c) We will therefore give consideration to making a native language title and research question an optional part of the dataset.

Unique Identifiers 18) It seemed to us that a unique identifier for each record in the database would be

essential, to facilitate discussion between organisations once a study of interest had been identified.

of 4

a) We were surprised to discover that half of responding organisations do not use a unique identifier which they are able to share – this group includes organisations with an active primary research portfolio.

b) Some organisations indicated that collections of characteristics could be used to identify projects. One responder indicated that they fund ‘collections of projects’, and hence one unique identifier would refer to a collection, rather than a single research question.

c) We will explore options for uniquely identifying projects with the database in the next Delphi round.

Source of research idea 19) Of 10 responders to this question, 8 favoured giving the full referenced source of a

research idea. We will therefore include a field for a full referenced source for a research idea in the data set.

Outcomes 20) Responders were in favour in including outcome information in the dataset. It was

particularly stressed that a record should not be removed once an outcome is known.

a) Some suggestions were made for specific outcomes to be included in addition to those suggested in the question

i) Did not proceed due to lack of finance

ii) Did not proceed due to lack of researcher capacity

iii) Did proceed – likely timescale for results to be available

b) We will develop a menu of outcomes. This menu may need to be expanded based on experience with a functional database. We will include a field in the outcomes section for a trial or other registry number.

Other information 21) We received a number of suggestions for other information which could be included

a) A summary project description

b) Country details

c) Number of planned centres for recruitment

22) There was one comment that the dataset was not sufficiently flexible for all studies which might be included.

23) We will explore these responses further in Delphi round 2

Implementation of a register 24) Half of responders thought that organisations should be pre-‐registered. One

suggested the ability to provide an additional project level contact could be appropriate. We will follow this up with HAS, who will be implementing any register which arises from this work.

of 4

25) All except one responder considered that leaving the precise eligibility details could wait for implementation. We will follow this up with HAS, who will be implementing any register which arises from this work.

26) While 8 responders agreed that the research question is an appropriate unit of entry, two raised issues. One in particular, which funds programme of research comprised of multiple questions, felt that they would find this too onerous. We will explore the implications of using a study title as a unit of registration.

Other comments 27) “We should not add more hurdles in the way of people doing research”

28) “there is a difficult balance to be drawn between (a) keeping the database very simple, but alerting funders to possible overlap, and (b) a fuller register with sufficient detail such that an alerted agency finds enough information to not to need to get further information from the second funder. “


2011-02-28



APPENDIX WP7.8_A

Survey on Dataset for trials in development: report of first Delphi round (Nov 2010)

Dataset and registry for clinical studies in development #2Dataset and registry for clinical studies in development #2Dataset and registry for clinical studies in development #2Dataset and registry for clinical studies in development #2

Aim of the Dataset

This dataset aims to allow the creation of a registry for primary research studies which are being considered (either by a public funder or on

the advice of a national appraisal body). Such a registry would allow research funders and appraisers to consider making changes to

planned studies (eg to enable easier meta-analysis).

The registry would contain studies which funders are considering, but have not yet made a final decision on – ie in a state where the study

could potentially still be changed. Once a study progresses sufficiently that it is funded, the record on this database should be updated

with this relevant information.

Such a registry would be used by entering particular characteristics about a study - eg the population, intervention and comparator. The

registry would then identify similar studies. It would contain contact details and sufficient information to allow the enquirer to contact the

potential funder of the matched study for further discussion.

1st Delphi round survey report

We appreciate your response to the 1st Delphi round survey.

Please click here for the survey report.

2nd Delphi Survey Instructions

Following the 1st survey we have updated the dataset.

1. Please click here to review the updated dataset before answering the questions.

2. Then please answer the following questions about the dataset.

These are in sections; dataset as a whole, coding, unique identifiers, outcomes, validation, unit of registration and the overall project.

Please click NEXT when you have answered all questions on a page. You can click PREVIOUS should you wish to go back to the page

before. Please click FINISH at the end to submit your answers. Hyperlinked sites will open in new pages.

Please submit your responses by 20th December 2010.

Many thanks for your help with this project!

1. Introduction - 2nd Delphi round survey


In the 1st survey about ¼ of responders favoured inclusion of some information in native language, as well as in English. We have updated the dataset to include OPTIONAL fields for native language recording of the study title and research question.

1. Is this sufficient?

2. The WHOLE Dataset

Yes

nmlkj

Too complicated - adds little value for increased complexity

nmlkj

No - suggest additional information

nmlkj

If applicable please identify what extra native language information you want included and why;

55

66


In the previous survey respondents agreed that MESH is the most appropriate coding system. However, one respondent suggested ATC codes would add value for pharmaceuticals (this is used by EMEA etc). For further information about ATC codes click here .

1. Are you familiar with ATC coding?

2. Do you see any advantages in using ATC (or any other technology/disease related

coding) in addition to MeSH?

3. Is your organisation able to provide ATC coding for its projects within its current

resources?

3. Coding

Yes

nmlkj

No

nmlkj


55

66

Yes

nmlkj

No

nmlkj

Don't Know

nmlkj

Please explain your answer;

55

66

Yes

nmlkj

No

nmlkj


55

66


Unique identifiers would serve two roles within the database; 1. To uniquely identify individual records - this can be left to the implementation phase.

2. To improve the database for its users. The system should enable a user to identify a possible overlap. They can then contact the owner of the identified study, saying ‘I would like to talk to you about your study’ and be easily able to identify the exact study which they want to discuss.

Different organisations use different ways to identify their projects. Some use code numbers, others use project titles, or other project characteristics.

We propose that two points of identification taken together will uniquely identify a project for database users - 1 AND (2 or 3) below;

1. The 'owning organisation' – which plans the study and entered the record on the database AND 2. A piece of information which is unique to the project within the organisation. This may be a code, or the title (if the institution is confident this will act as a point of identification as outlined above)OR 3. If the organisation is unable to identify an appropriate identifier for (2) above, the database will automtically supply one (much like a clinical trials registry supplies a registry number) – but the owning institution would need to ensure this number was noted on its own records.

1. Other organisations may identify projects from your portfolio via the database and

contact your organisation to discuss them with you.

Would this system enable your institution to identify which projects from your

portfolio they wanted to discuss with you? 1 AND (2 or 3) above.

4. Unique Identifiers

Yes

nmlkj

No

nmlkj

Don't Know

nmlkj


55

66

Dataset and registry for clinical studies in development #2Dataset and registry for clinical studies in development #2Dataset and registry for clinical studies in development #2Dataset and registry for clinical studies in development #22. If your organisation does not use a unique code for your planned projects, each

project would automatically be assigned one by the database (#3 above).

Will you be able to incorporate this registration number into your organisation's own

records? (this will be necessary so that you can identify which project a caller wishes

to discuss)

Yes

nmlkj

No

nmlkj

Don't Know

nmlkj


55

66


The 1st survey emphaised the importance of keeping a record about what happened to each planned study; 1. If they went ahead - where further information can be found 2. If they did not go ahead - why this did not happen. We suggest a menu of choices for what happens to a project. We are interested in your views about what should be included;

1. Studies that WENT AHEAD.

Which outcomes should be included for planned studies that went ahead?

2. Studies that DID NOT PROCEED.

Which outcomes should be included for planned studies that did not go ahead?;

5. Outcome for the planned study

Yes No Don't Know

Funded as a trial (include trial reference number/other reference) nmlkj nmlkj nmlkj

Funded as another type of study (include reference number/other reference) nmlkj nmlkj nmlkj

Results likely to be available by date (note:does not mention trial registration) nmlkj nmlkj nmlkj

Cofunded with a specified public sector funder nmlkj nmlkj nmlkj

Cofunded with a specified private sector funder nmlkj nmlkj nmlkj

Yes No Don't Know

Lack of clinical, or other, need (eg no one cared sufficiently about the results to

make it woth doing)nmlkj nmlkj nmlkj

Lack of researcher capacity(ie a capable researcher could not be found to deliver

the project)nmlkj nmlkj nmlkj

Lack of research funding nmlkj nmlkj nmlkjLack of other funding (eg the trial costs would be paid, but no one would pay for

the drugs involved)nmlkj nmlkj nmlkj

Please add your comments;

55

66

Please add your comments;

55

66

Dataset and registry for clinical studies in development #2Dataset and registry for clinical studies in development #2Dataset and registry for clinical studies in development #2Dataset and registry for clinical studies in development #23. Should any other outcome measures be included?

Please specify them, and their value;

55

66


The 1st survey identified other information that could be included in the registry. Please indicate whether you think they would be valuable.

1. A short summary of the planned research (eg up to 200 words).

This may allow exclusion of an overlap without the need to contact the study owner.

An example has been included at item 18 in the example dataset version 3.

2. The country where a study is planed.

We thought this may be evident from the research funder - but this would be simple

to add if it were useful.

3. The number of planned centres for recruitment.

We considered that the planning stage is probably too early to know this, and it

would not help in identifying planned similar studies. If occasionally important it

could be entered in the summary field - do you agree?

6. Other Information

Yes

nmlkj

No

nmlkj

Don't Know

nmlkj


55

66

Yes

nmlkj

No

nmlkj

Don't Know

nmlkj


55

66

Yes

nmlkj

No

nmlkj

Don't Know

nmlkj


55

66


In 2011 we intend to undertake some validation of this dataset. To validate efficacy we will invite participants to submit possible entries using this dataset in particular clinical areas. We will then explore how good the dataset is at identifying overlapping studies and excluding non overlapping studies. One clinical area we will look at is Breast Cancer and Herceptin (the issue which sparked the first thoughts about a registry). To address this next year, we will ask you to submit any studies you have about Breast Cancer; those involving Herceptin, and one or two (if you have any) which don’t include Herceptin.

1. We also aim to undertake similar validation in one or two other clinical areas. When

selecting we will try to pick areas where multiple organisations have suggested an

interest.

What clinical areas would you suggest?

2. Validation of effectiveness will need to wait until there is a reasonably well-

populated registry. Then we will investigate characteristics of the registry and its

search tools to try to identify diagnostic accuracy etc.

We believe that sensitivity should be favoured rather than specificity in searches.

This would mean that the registry should tend to suggest more overlaps than there

actually are-the excess would be trimmed by the user. We feel that a little more work

to exclude non-overlapping studies would be better than a more specific search

which would miss studies of interest.

Do you agree?

7. Validation

55

66

Yes

nmlkj

Don't Know

nmlkj

Other approach would be better

nmlkj

If you think other approach would be better - please specify;

55

66


For most of the respondents one project contains approximately 1 or 2 studies – eg 1 RCT, or two RCTs with different eligibility criteria. However, one respondent indicated that in their organisation one project can contain many different studies using different designs. We’re looking for a solution which allows ‘good enough’ matching of similar studies, to allow organisations to engage in discussion.

1. We suggest that the unit of registration is the PROJECT, which may consist of one

or more substudies (either in parallel or series). This may require multiple entries

against the PICO fields. The summary field can then contain information about

substudies (if any) to allow searchers to determine whether to make contact with the

study owner to investigate further. This would have little impact for most

organisations, but would help for organisations which fund programmes of research.

Would this work for your organisation?

8. Registration Unit

Yes

nmlkj

No

nmlkj

Don't Know

nmlkj


55

66


When we invited you to take part in this Delphi process, we suggested it would not take a large investment of your time. We’d like to check

that our estimates were correct.

1. How many person-hours has your organisation taken to respond to round one of

this Delphi process (the previous survey)?

2. How many person-hours has your organisation taken to respond to round two of

this Delphi process (this survey)?

3. Is there anything else you would like to add, either about the current version of the

dataset or the project as a whole?

Many thanks for your responses.

Now please click FINISH to submit your responses.

9. Information about this project

55

66


2011-02-28



APPENDIX WP7.9_A

Survey on Dataset for trials in development: second Delphi round (Dec 2010)

front page tech interim report 2010 wp7appendices · eunethta ja on hta – wp 7 1/1 2011-02-28...

Documents