eunethtaja2 wp6 deliverable report on information ... · wp6 deliverable report on information...

The EUnetHTA JA 2 (2012-2015) has received funding

from the European Union, in the framework of the Health

Programme

EUnetHTA JA2

WP6 DELIVERABLE

Report on InformationManagement Infrastructureand Services

EUnetHTA JA2 Report on Information Management Infrastructure and Services WP 6

2016-01-20 © EUnetHTA, 2015. Reproduction is authorised provided EUnetHTA is explicitly acknowledged 2/19

Joint Action on HTA 2012-2015

Report on InformationManagement Infrastructure

and Services

Date: Jan 2016

Was developed by Work Package 6 - Information Management Infrastructure and Services(IMIS)

WP 6 Lead Partner: Belgian Health Care Knowledge Centre (KCE)

WP 6 Co-Lead Partner: German Institute for Medical Documentation and Information (DIMDI)



Disclaimer: EUnetHTA Joint Action 2 is supported by a grant from the European Commission. The soleresponsibility for the content of this document lies with the authors and neither the European Commission norEUnetHTA are responsible for any use that may be made of the information contained therein.

Author: Patrice X. Chalon

Reviewers: Peter Kraemer, Carl Devos

Table of Contents

Introduction ....................................................................................................................................................... 4

Background .................................................................................................................................................... 4

Objective ........................................................................................................................................................ 4

Methods ............................................................................................................................................................. 4

Results ............................................................................................................................................................... 5

Objective ........................................................................................................................................................ 5

Indicators ....................................................................................................................................................... 5

Indicator 1: Input........................................................................................................................................ 5

Indicator 2: Technical limitations ................................................................................................................. 5

Indicator 3: POP DB .................................................................................................................................... 6

Milestones ...................................................................................................................................................... 7

Website, Intranet and Workrooms............................................................................................................... 7

Other deliverables......................................................................................................................................... 10

Information Management environment...................................................................................................... 10

Trainings activities..................................................................................................................................... 11

Helpdesk ................................................................................................................................................... 11

Discussion........................................................................................................................................................ 12

Conclusion........................................................................................................................................................ 12

References ....................................................................................................................................................... 13

Appendices....................................................................................................................................................... 14

Tools available at start of JA2 ....................................................................................................................... 14

Details evaluation input (indicator 1)............................................................................................................. 15

Methodology ............................................................................................................................................. 15

Results ...................................................................................................................................................... 15

Extracts from yearly survey results ............................................................................................................... 17

Specific trainings (details) ............................................................................................................................. 18

Estimated resources for the maintenance of the infrastructure and services.................................................. 19

Estimated budget for the next four years...................................................................................................... 19



Introduction

Background

Since its inception in 2006, EUnetHTA has developed several IT tools (see table). During joint action 1 (JA1),one work package (WP) was dedicated to this “information management system” and aimed at providinghomogeneity between tools by way of similar looking interfaces, common metadata standards, singleauthentication (Chalon & Kraemer, 2011).

With joint action 2 (JA2), this WP was pursued in order to keep a space for tools developers to exchange. WP6had also to provide new Website and Intranet (version of content management system used was end ofsupport). WP6 also integrated the content management of the Planned and Ongoing Project dataset (POP DB).

But not all tools were integrated in WP6: the web conferencing was provided by WP1, the EVIDENT databasewas part or WP7 activities, and the HTA Core Model® online tool was part of WP8 activities.

Objective

The objective of the Work package 6 (WP6) was expressed as the “provision of an informationmanagement infrastructure and the related services to support the piloting of collaborativeproduction of HTAs by partner agencies, and facilitate the tasks and team working of the otherWPs”.

MethodsA 3 year work plan has been created; it contains the description of the activities to be conducted.

Three indicators have been defined to be reached at the end of the 3 year period:

1. 90% of JA2 Participants use the IMIS and have provided at least one piece of information.2. Results of the surveys will show a reduction of technical limitations related to the Intranet and the “work

rooms” (at least 90% of respondents do not express such a limitation)3. The POP DB will, in average, contain the description of 1000 current projects from at least 40 partners

from 22 countries (in average).

Two milestones were defined:

M4: New website, intranet and workrooms

M36: Information Management environment

Questions related to the IT tools and related services were included in the yearly general survey conducted byWP3.

Trainings for tools were planned in coordination with WP2.



Results

Objective

WP6 delivered the following elements of the information management infrastructure and services (Chalon,2015):

A public website was provided to EUnetHTA Secretariat in charge of content and users management.

Above the standard information to the external world, an extranet functionality was added for theStakeholder forum and the Stakeholder Action groups (SAGs).

A newsletter system was provided to Secretariat in order to reach both partners (LinK newsletter) andgeneral public (Public newsletter).

A central authentication system was provided to all tools requiring user access management An intranet was provided to EUnetHTA partners. 90 groups were created for Secretariat (general group,

executive committee, Conference preparation, financial reporting of partners…), lead Partners (WPgeneral management) and teams (WP subgroups). The Intranet was also selected to serve as an e-learning platform (Chalon, 2014).

A video hosting platform was selected to host video material produced by partners under thecoordination of WP2

A database holding description of planned and ongoing projects was provided to EUnetHTA partnersparticipating to the provision of content.

A news aggregator was provided to EUnetHTA partners An online survey system was provided to EUnetHTA partners

WP6 also included representatives of WP7 and WP8 responsible for development, maintenance and provision oftraining material and activities of respectively the EVIDENT database and the HTA Core Model® online tool.

Indicators

Indicator 1: Input

More than 90% of partners have provided at least one piece of information in the information managementsystem

Table 1: Amount Partners having provided at least one input

Input #partners %

yes 63 91,30

no 6 8,70

TOTAL 69 100

Indicator 2: Technical limitations

In average, more than 90% of respondents did not encountered technical limitations to access the Intranet.However, score in year 2 is below 90%.

Table 2: Amount Partners having expressed technical issues when using the Intranet

Year Rate (%) #Respondent

Y1 1,31 199

Y2 11,585 164

Y3 4,74 173

AVERAGE 5,88 178,67



Indicator 3: POP DB

The POP database always contained more than 1000 current projects from more than 40 partners from more 22

countries.

Table 3: Data from the POP database alerts

Date #Projects #Partners #Countries #Alert topics

23/01/2013 1250 43 24 132

8/05/2013 1250 42 23 106

16/09/2013 1239 44 23 94

20/12/2013 1219 46 23 101

19/03/2014 1211 44 24 106

9/07/2014 1308 47 25 90

9/10/2014 1266 47 25 88

30/01/2015 1249 47 25 103

29/04/2015 1314 47 26 103

13/08/2015 1068 43 26 100

2/12/2015 1031 43 24 92



Milestones

Website, Intranet and Workrooms

The CMS used during Joint Action 1 having reached its end of support, a new website and a new Intranet havebeen provided. Because of different purpose and targeted public, there were separated websites.

The public website provides different types of content (news, static pages, activities, events, output, andorganization). The public site had its role extended to an extranet to host documents to be shared withStakeholder forum and SAGs. It is based on the open source CMS Drupal 6.

The intranet is organized by “groups” (general group for all members, specific groups for each WP, for specificgroups like Joint Assessment teams, webmasters…). Each group may provide the following functionalities:dashboard, news, document library, tasks, online chat, calendar, group members, polls, discussions. Each userget a personal dashboard, group list, contact database (with skills) and calendar. It is based on Open Atrium 1,a specific distribution of the open source CMS Drupal 6. Combined with a video hosting platform, the Intranet

also provides access to e-learning material in so-called “Virtual Classrooms”.

Figure 1: EUnetHTA Website homepage



Figure 2: Stakeholder Forum area on the Website (= extranet, restricted area)



Figure 3: EUnetHTA Intranet Dashboard

Figure 4: Virtual classroom (in the Intranet)



Other deliverables

Information Management environment

The Information management environment available for EUnetHTA contains three types of tools

Online services: micro blogging (Twitter®), Video hosting (YouTube® and Vimeo®), Slideshow hosting

(Slideshare®) Publicly available tools: Public Website, News aggregator, Adaptation glossary, Web survey Tools requiring authentication (with EUnetHTA ID): Intranet, POP DB, EVIDENT DB, HTA Core Model®

online

The cost of maintenance is estimated to be 746,25 person days, and 138.395,89 euros for subcontracting, eachyear.

Developments are excluded from this estimation, aside the already envisioned updates / upgrades, the Websiteand the intranet, both based on the Open source Content Management System Drupal version 6 will soon be outof support: their replacement is thus to foreseen.

Figure 4: Current information Management Environnement (cross WP)



Trainings activities

Tools developers provided training activities and material under coordination of WP2 or separately.

Table 5: Training activities or material

WP2 coordinated activities Tools specific activities

Partnertraining2014/01



One dayWebinar2015/09

e-learning Face toface

training

Webinars

Intranet Yes Yes Yes Yes Recording ofwebinarRecording of theone day Webinar

2015/09

None 14

POP DB Yes Yes Yes Yes One dedicatedVirtual Classroomwith 4 webcasts

None 3

HTA CoreModel online

Yes Yes Yes Yes Recording of theone day Webinar2015/09

3 None

EVIDENT DB Yes Yes No Yes Recording of the

one day Webinar2015/09

None None

Export done on the 06/12/2015

Helpdesk

WP6 LP provided both the Website and the Intranet, and also a Newsletter system; WP6 Co-LP provided theauthentication repository. Above the specific training, WP6 also provided support to Secretariat (as a second linesupport to end users). A ticketing system was set up in the Intranet in May 2013 so Secretariat could describeissues and WP6 provide a follow up. When necessary, the problem was escalated to the tool provider (Website,Intranet), and the result provided back to Secretariat.

Table 4: Amount tickets by tool

Tools Tickets @ EUnetHTA

Website 88

Intranet 107

Newsletter 16

EUnetHTA ID -user/org management 22

Others 21

TOTAL 254




DiscussionThe general objective of WP6 has been met. However, all tools required to “support the piloting of collaborativeproduction of HTAs by partner agencies, and facilitate the tasks and team working of the other WPs” were notunder the responsibility of WP6: WP6 has thus no direct influence on the results of those tools. The objectiverelated to the percentage of partner providing at least one information is thus difficult to manage since not alltools composing the Information management are located at WP6.

In JA2, no activity related to the integration of the different tools produced in JA1 was planned, even if some

options were evoked in JA1 and also during JA2 (Chalon, 2014). Each tool was further developed separately,and the opportunity to discuss the best option for new functionalities, or to further develop the integrationbetween tools (see examples below) using the WP6 “forum” was not taken. To address this kind of issues, anew organisation of EUnetHTA tasks could be considered, placing all tools in the same activity centre, under thecoordination of one entity. This would support the centralisation of needs evaluation and implementation, andprovide a more integrated approach to support the interaction and complementarity of the tools.

E.g.: Collecting and listing the re-use of EUnetHTA output (national uptake): this was coordinated by

WP1-Secretariat, but the inclusion of the different tools in this process was not envisioned, resulting induplication of efforts and several places to find similar information. Secretariat implemented a table inthe public site; WP6 added new fields to complete the POP DB; WP8 added new functionalities on theHTA Core Model® online tool, but limited to the re-use of its content. Also, communication about theinput process was not centralised. As a result, end users are probably not aware on where to find suchinformation, and contact persons at the agency probably do not know where to provide the information

and how. E.g.: Integration of tools was evoked already during JA1: basic links could be added in the different

tools to bring the user from one tool to another, or to provide information about the next step in theprocess (when a project changes status in POP DB, from planned to ongoing, a link could send to HTACore Model online + CRD HTA database; when changing status from ongoing to published, a link couldsend to EVIDENT DB and CRD HTA DB again)

Besides functionalities, usability and accessibility are essentials aspects of the provided tools. For example, whilemeeting the expectation in average, the technical barriers to the use of the Intranet were above the 10% in oneof the three yearly surveys (year 2).Those aspects should be regularly evaluated for the whole informationmanagement system, and the necessary enhancements should be implemented.

Considering the objective of a self-sustainable network, costs to maintain/develop tools was collected. Theresults shows that the current infrastructure will require a very high amount of resources (forecast is about 10%of the planned JA3 budget). In order to keep the infrastructure in the long run, each tool should be accuratelyevaluated: which are the tools that should be further developed; maintained as such, stopped? Prioritizingdevelopments will be necessary in order to contain the costs. Also, options allowing to reduce the costs ofinfrastructure should be considered. European Union could provide support based on existing resources1, furtheruse of online services should also be explored2.

ConclusionIn order to pursue the provision of a sustainable and integrated information infrastructure and the relatedservices, placing all tools under one entity should be considered in order to centralize needs evaluation andimplementation, enhance interaction and complementarity of the existing tools, and keep cost as low aspossible.

1 The European Union provides some IT tools for free (e.g. EU survey (DG DIGIT, 2015)). European Union alsofavours shared IT infrastructure (e.g. common web platform based on the Open Source CMS Drupal 7

(Offerman, 2015)).2 If reducing the functionalities requirements, the public Web site could make use of affordable online solutionslike Drupal Gardens (https://www.drupalgardens.com/pricing), WordPress.com(https://wordpress.com/start/delta-site/plans/compare), Wix.com (http://www.wix.com/upgrade/website ) andothers.



ReferencesChalon, P. X., 2014. e-learning platform – technical report, s.l.: EUnetHTA.

Chalon, P. X., 2014. Minutes from WP6 e-meeting all Partners M24, s.l.: EUnetHTA.

Chalon, P. X., 2015. EUnetHTA JA2 WP6 Information Management Infrastructure and Services - EUnetHTA tools,s.l.: EUnetHTA.

Chalon, P. X. & Kraemer, P., 2011. EUnetHTA JA WP6 Information and Management System, s.l.: EUnetHTA.

DG DIGIT, 2015. Welcome. [Online]

Available at: https://ec.europa.eu/eusurvey/home/publicsurveys

Offerman, A., 2015. EC consolidates hundreds of websites onto new Drupal 7 Europa platform. [Online]Available at: https://joinup.ec.europa.eu/community/osor/case/ec-consolidates-hundreds-websites-new-drupal-7-europa-platform



Appendices

Tools available at start of JA2

Tools Project Joint Action 1 Joint Action 2

Public Website WP2 WP2 WP6

Newsletter WP2 / WP6

Web conferencing system WP1 WP1 WP1

Central authentication system(LDAP)

/ WP6 WP6

Intranet WP2 (and clearinghousedemonstrator)

WP2 (infrastructure)/ WP6 (services)

WP6

Planned and Ongoing Projects

database (POP DB)

/ WP6 (infrastructure)

/ WP7 (services)

WP6

HTA Core Model® online / WP WP8

EVIDENT DATABASE WP7 (was EIFFEL DB) WP7 WP7

News aggregator / WP6 WP6

OAI aggregator / WP6 STOPPED

Adaptation glossary WP5 (PDF file) WP6 WP6

Browser toolbar / WP6 WP6

Micro blogging / WP2 WP1

Video Hosting / WP2 WP1

WP6

Slideshow hosting / WP2 WP1



Details evaluation input (indicator 1)

Methodology

Current members were identified by consulting the “Contact us” page of the EUnetHTA Website. Intranet users were identified by consulting the intranet members’ directory and filtering on “Agency”

name. Input in the POP DB was identified using the “Browse by agency” function (at least one project

described) Input in the output was identified by looking at authors, co-authors or reviewers affiliations on the

collections hosted at the core model online or in joint assessments published on the Website (at leastone publication)

Input in the EVIDENT DB was identified by using the “Search by agency” functionality Input in the INTRANET / WHOSWHO was identified by looking to employees’ profile: last activity

(intranet content) or completed fields (who’s-who) were considered.

Results6/11/2015 INPUT CURRENT

MEMBERINTRANETUSERS

POP db Output EVIDENT

INTRANET /WHOSWHO

A. Gemelli YES YES YES yes Prognostic tests for breast cancerrecurrence

NO N/A

AAZ YES YES YES yes Prognostic tests for breast cancerrecurrence

NO N/A

AETSA YES YES YES yes Fecal Immunochemical Test (FIT ) NO N/A

Agenas YES YES YES yes Abdominal Aorta Aneurysm Screening YES N/A

AIFA YES YES YES yes SORAFENIB YES N/A

AOTMiT YES YES YES yes Prognostic tests for breast cancerrecurrence

NO N/A

AQuAS YES YES NO NO Joint Assessment on Vorapaxar(ZONTIVITY™) for the reduction ofthrombotic cardiovascular events inpatients with a history of myocardialinfection (MI)

NO N/A

ASSR YES YES YES yes NO NO N/A

AVALIA-t YES YES YES yes Prognostic tests for breast cancerrecurrence

NO N/A

BIQG/GÖG YES YES YES yes Abdominal Aorta Aneurysm Screening NO N/A

CEM YES YES YES yes Fecal Immunochemical Test (FIT ) NO N/A

CFK YES YES YES yes Renal denervation systems for treatment-resistant hypertension

NO N/A

CHIF YES YES YES NO NO NO YES

CMSS YES YES YES NO NO NO YES

DanubeUniversityKrems

NO YES NO NO NO NO NO

DEDI YES YES YES NO NO NO YES

DGFPS MSPSI YES YES YES No Zostavax for the prevention herpes zoster NO N/A

DHMA YES YES YES NO NO NO YES

DIMDI YES YES YES yes NO NO N/A

FIMEA YES YES YES yes RAMUCIRUMAB IN COMBINATION WITHPACLITAXEL

NO N/A

GBA YES YES YES NO NO NO YES



6/11/2015 INPUT CURRENTMEMBER

INTRANETUSERS


INTRANET /WHOSWHO

HAS YES YES YES yes Fecal Immunochemical Test (FIT ) YES N/A

HIQA YES YES YES yes Balloon Eustachian Tuboplasty (BET) NO N/A

HIS YES YES YES yes Renal denervation systems for treatment-resistant hypertension

NO N/A

HVB YES YES YES yes Abdominal Aorta Aneurysm Screening NO N/A

IACS NO YES YES NO NO NO NO

IER YES YES YES yes Fecal Immunochemical Test (FIT ) NO N/A

INFARMED YES YES YES No Prognostic tests for breast cancerrecurrence

NO N/A

IQWIG YES YES YES yes Renal denervation systems for treatment-resistant hypertension

NO N/A

ISC III YES YES YES yes Abdominal Aorta Aneurysm Screening YES N/A

KCE YES YES YES yes Zostavax for the prevention herpes zoster YES N/A

KDTD NO YES YES NO NO NO NO

KORA NO YES YES NO NO NO NO

LBI-HTA YES YES YES yes Abdominal Aorta Aneurysm Screening NO N/A

MEDICALVALLEY

YES YES YES NO NO NO YES

MEH YES YES YES yes Zostavax for the prevention herpes zoster N/A

MoH Cyprus NO YES YES NO NO NO NO

MoH CzechRep

YES YES YES No canagliflozin for the treatment of type 2diabetes mellitus

NO N/A

MoH Slovakia YES YES YES NO NO NO YES

MSSSI YES YES NO NO The first pilot rapid assessment on‘Zostavax for the prevention herpes zoster'

NO N/A

MU Sofia YES YES YES No canagliflozin for the treatment of type 2diabetes mellitus

NO N/A

NCHTA YES YES YES NO NO NO YES

NCPE(NCPHP)

YES YES YES yes SORAFENIB NO N/A

NCPHP YES YES YES NO NO NO YES

NCPRMP YES YES YES NO NO NO YES

NETSCC YES YES YES yes NO NO N/A

NHS, LV YES YES YES yes NO N/A

NICE YES YES YES yes Prognostic tests for breast cancerrecurrence

NO N/A

NIJZ YES YES YES yes Prognostic tests for breast cancer recurrence N/A

NOKC YES YES YES yes Abdominal Aorta Aneurysm Screening NO N/A

NSPH YES YES YES No Fecal Immunochemical Test (FIT ) NO N/A

NSPH MPD YES YES YES yes Fecal Immunochemical Test (FIT ) NO N/A

OGYÈI YES YES YES yes RAMUCIRUMAB IN COMBINATION WITHPACLITAXEL

NO N/A

Osteba YES YES YES yes Fecal Immunochemical Test (FIT ) NO N/A

Regione delVeneto

YES YES YES yes Prognostic tests for breast cancerrecurrence

NO N/A

RIZIV-INAMI YES YES YES No SORAFENIB NO N/A



6/11/2015 INPUT CURRENTMEMBER

INTRANETUSERS


INTRANET /WHOSWHO

SAGEM YES YES YES yes Biodegradable stents for the treatment ofrefractory or recurrent benignoesophageal stenosis

NO N/A

SBU YES YES YES yes NO NO N/A

SESCS YES YES YES yes NO NO N/A

SMCA NO YES YES NO NO NO NO

SNHTA YES YES YES NO Prognostic tests for breast cancerrecurrence

NO N/A

THL YES YES YES YES Abdominal Aorta Aneurysm Screening NO N/A

TLV YES YES YES yes NO NO N/A

TUBITAK-TUSSIDE

YES YES YES NO NO NO YES

UETS YES YES YES NO HTA Core Model® for Rapid RelativeEffectiveness Assessment ofPharmaceuticals

NO N/A

UMIT YES YES YES yes Prognostic tests for breast cancerrecurrence

NO N/A

UTA YES YES YES yes Abdominal Aorta Aneurysm Screening NO N/A

VASPVT YES YES YES yes Biodegradable stents for the treatment ofrefractory or recurrent benignoesophageal stenosis

NO N/A

ZIN YES YES YES yes NO YES N/A


Extracts from yearly survey resultsTechnical limitation for tools Year #people Rate (%) total respondents Total invited

EUnetHTA contact database (in theIntranet)

Y1 2 1,74 199 422

EUnetHTA general group (in the Intranet,

with news, document library, discussion…)

Y1 1 0,88 199 422

EUnetHTA contact database (in theIntranet)

Y2 14 10,77 164 332

EUnetHTA general group (in the Intranet,with news, document library, discussion…)

Y2 16 12,4 164 332

EUnetHTA contact database (in the

Intranet)

Y3 6 4,76 173 375

EUnetHTA general group (in the Intranet,with news, document library, discussion…)

Y3 6 4,72 173 375



Specific trainings (details)

Face to face training Specific Webinars e-learning (with

support of WP2)

Intranet None 2015-06-19

2015-05-11

2015-04-22

2015-03-16

2015-02-25

2015-01-15

2014-12-04

2014-11-04

2014-10-08

2013-08-27

2013-08-07

2013-07-26

2013-07-09

2013-01-31

Recording of webinar

Recording of the one dayWebinar 2015/09

POP DB None 2015-04-22

2015-04-15

2014-11-10

One dedicated VirtualClassroom with 4webcasts

HTA Core Modelonline

2015-10-01/022014-06-05/062013-10-31/11-1

None Recording of the one dayWebinar 2015/09

EVIDENT DB None None Recording of the one dayWebinar 2015/09



Estimated resources for the maintenance of the infrastructure and services

In Oct 2015, WP6 LP asked all tools maintainers to provide an estimation of the resources (person days andsubcontracted budget) that would be necessary for the maintenance of their tools, each year.

EUnetHTA IMS(maintenance, without all initial development costs)

staff (day/year) (sub)contracting(euros / year)

Project management 159,60

Provision of training (face to face, webinars) anddocumentation

55,85

Support (problem resolution, tool updates) 232,20 19.779,50

Hosting and maintenance (infrastructure update / upgrades) 14,40 17.860,50

Domain names 618,39

Certificates (https) 358,00

Development (change request management, tool upgrades) 284,20 99.779,50

TOTAL 746,25 138.395,89

Estimated budget for the next four years

In Oct 2015, WP6 LP asked all tools maintainers to provide an estimation of the resources (person days and

subcontracted budget) that would be necessary for the maintenance and further development of their tools forthe next four years. Cost sharing of the subcontracted costs should be considered.

EUnetHTA IMSMaintenance and developments,

4 years

staff (days) Staf cost(evaluation*)

(sub)contracting andother costs (euros)

TOTAL(evaluation)

Project management 690 € 241.500,00 € 241.500,00

Provision of training (face toface, webinars) and

documentation

250 € 87.500,00 € 87.500,00

Support (problem resolution,tool updates)

960 € 336.000,00 € 99.200,00 € 435.200,00

Hosting and maintenance(infrastructure update /upgrades)

60 € 21.000,00 € 102.000,00 € 123.000,00

Domain names € 2.500,00 € 2.500,00

Certificates (https) € 2.700,00 € 2.700,00

Development (change requestmanagement, tool upgrades)

1330 € 465.500,00 € 500.200,00 € 965.700,00

Total 4 years 3290 € 1.151.500,00 € 706.600,00 € 1.858.100,00

* Daily rate is estimated to 350,00 euros

eunethtaja2 wp6 deliverable report on information ... · wp6 deliverable report on information...

Documents