consultation: science europe draft dmp evaluation rubric · consultation: science europe draft dmp...

Science Europe

Rue de la Science 14, 1040 Brussels, Belgium

Tel: +32 (0)2 226 03 00 Fax: +32 (0)2 226 03 01 Email: [email protected] www.scienceeurope.org

Consultation: Science Europe Draft DMP

Evaluation Rubric

Overview of Survey Questions

Introduction

After the publication of its Practical Guide to the International Alignment of Research Data

Management (hereafter referred to as ‘Science Europe RDM Guide’) in January 2019, Science Europe is

currently developing a rubric for the evaluation of Data Management Plans (DMPs).

This rubric is completely aligned with the Science Europe RDM Guide and will support evaluators of

DMPs. It sets out the expectations that a DMP has to meet by listing criteria for two performance

levels, “Sufficiently Addressed” and “Insufficiently Addressed”.

With the consultation, Science Europe would like to gather feedback from stakeholders involved in

research data management (RDM) activities and/or developing RDM policies on the definition of criteria

for the two performance levels. The consultation is structured as follows:

A. Information on the responding organisation and contact person

B. Evaluation criteria for the different aspects of DMPs

C. General comments

Please note: A number of organisations have already implemented the Science Europe RDM Guide on

institutional level or with partners on national level. For this reason, we only seek feedback on the criteria

for the performance levels. The Science Europe DMP guiding questions included in the rubric to facilitate

its use are a direct extract from the Science Europe RDM Guide, and therefore not subject of the

consultation.

The consultation will run until 16 September 2020. Please note that only responses submitted online on

behalf of an organisation or initiative by this deadline will be considered. Only one response per

organisation will be accepted.

If you have any questions, please do not hesitate to contact Marie Timmermann, Senior Policy Officer at

Science Europe: [email protected]

mailto:[email protected]

https://www.scienceeurope.org/our-resources/practical-guide-to-the-international-alignment-of-research-data-management/

mailto:[email protected]

CONSULTATION: SCIENCE EUROPE DRAFT DMP EVALUATION RUBRIC

2

A. Information on the Responding Organisation and

Contact Person

1. Name of organisation:

2. Stakeholder group

Multiple choice:

o Research Funding Organisation

o Research Performing Organisation

o University

o Research infrastructure

o eInfrastructure

o Repository / data curation

o Other:

3. Contact Person, replying on behalf of the organisation mentioned above

Name:

Function:

Email:

4. Do you agree to be contacted by one of our experts in case we need more information

regarding the feedback you gave on behalf of your organisation?

5. Has your organisation currently a DMP policy in place or is one being developed?

Multiple choice:

o policy in place

o policy under development

o using DMP template, but no fully fledged policy in place

o no policy in place

6. Did your organisation take part in the consultation rounds in 2018 in preparation of the Science

Europe Practical Guide?

7. Has your organisation used the Science Europe RDM Guide as basis to set up DMP policies?

7a. If yes, how has your organisation used the Guide?

7b. If no, why did your organisation not use the Guide?


3

B. Evaluation Criteria for the Different Aspects of

DMPs

The DMP core requirements detailed in the Science Europe Practical Guide are divided in six topics with

different subtopics each that are presented in the form of guiding questions.

The following consultation focuses on the criteria to assess whether the core requirements are

sufficiently addressed in a DMP. You will have the opportunity to provide your feedback to the criteria

for each subtopic.

1. Data Description and Collection or Re-use of Existing Data

1a. How will new data be collected or produced and/or how will existing data be re-used?

Performance levels

Requirement sufficiently addressed Requirement insufficiently addressed

Gives clear details of where the data come

from and how new data will be collected or

produced.

If data is reused, explains how existing data

will be accessed and any constraints to

reuse.

If applicable, explains clearly why new data

must be collected, rather than re-using

existing data.

Provides no explanation, or details

insufficient to get a clear understanding of

where the data come from and what data

will be collected or re-used.

If applicable, does not explain sufficient

rationale for generating new data.

Does your organisation agree with the evaluation criteria presented above?

o yes

o no, I would like to suggest changes to the criteria ‘sufficiently addressed’

o no, I would like to suggest changes to the criteria ‘insufficiently addressed’


4

1. Data Description and Collection or Re-use of Existing Data

1b. What data (for example the kind, formats, and volumes), will be collected or produced?

Performance levels


Clearly describes or lists what data types will

be generated (e.g. numeric, textual, audio,

video, etc.) and their associated data

formats, including, if needed, data

conversion strategies.

Explains why certain formats have been

chosen, and indicates if they are in open and

standard format.

Provides information about the data volume.

Alternatively, it clearly states that no new

data will be produced or generated by the

project.

NB. Information derived from previously existing

data sources, namely output, processed,

analysed data – are to be considered new data

under this question.

Provides no information, without a valid reason to do so (e.g. a statement that no data will be produced or generated)

Only lists/describes data types without specifying the data formats

Only lists formats, without specifying the data types.


o yes




5

2. Documentation and Data Quality

2a. What metadata and documentation (for example the methodology of data collection and way of

organising data) will accompany the data?

Performance levels


Clearly (even if brief) outlines the documentation needed to enable data re-use, stating where the information will be recorded (e.g. a database with links to each item, a ‘readme’ text file, file headers, code books, or lab notebooks).

Clearly outlines the metadata that will accompany the data, with reference to good practice in the community (e.g. metadata standards). May indicate that no metadata standards exist for the discipline.

Indicates how the data will be organised during the project.

Indicates how such information will be captured and archived with the data.

Provides no information, or only a very vague mention of documentation, without providing any detail or explanation.

Provides little or no detail on metadata that will accompany the data.


o yes




6

2. Documentation and Data Quality

2b. What data quality control measures will be used?

Performance levels


Clearly describes the approach taken to ensure quality control in the collection of data

Provides only a vague mention on how data quality is controlled and documented during the lifetime of the project.


o yes




7

3. Storage and Backup During the Research Process

3a. How will data and metadata be stored and backed up during the research?

Performance levels


Clearly (even if briefly) describes the location where the data and backups will be stored during the research activities.

Indicates that the data will be stored in at least two separate locations.

Clearly indicates how often backups will be performed.

Indicates the use of robust, managed storage with automatic backup, such as storage provided by the home institution.

If not, then it clearly explains why institutional storage cannot be used (and for what part of the data) and describes the (additional) locations, storage media and procedures that will be used for storing and backing up data during the project.

Provides no information or very vague mentions how data will be stored and backed up during the project.


o yes




8

3. Storage and Backup During the Research Process

3b. How will data security and protection of sensitive data be taken care of during the research?

Performance levels


Clearly explains:

› how the data will be recovered in the event

of an incident

› which institutional data protection policies

are in place

› who will have access to the data during the

research.

If sensitive data are involved (for example personal data, politically sensitive information or trade secrets), then it clearly describes the additional security measures (in terms of physical security, network security, and security of computer systems and files) that will be taken to ensure that stored and transferred data are safe.

Gives details of institutional data protection policies in place and provides a link to where they can be accessed.

Provides no information or very vague mentions how the data will be recovered in the event of an incident, which institutional data protection policies are in place and who will have access to the data during the research.

Provides no explanation about data protection and risk management or the explanation is too vague, even though sensitive data (for example personal data, politically sensitive information or trade secrets) are involved.


o yes




9

4. Legal and Ethical Requirements, Codes of Conduct

4a. If personal data are processed, how will compliance with legislation on personal data and

security be ensured?

Performance levels


Clearly indicates if personal data will be collected/used as part of the project, and how data will be lawfully processed.

Indicates what method will be used in order to process data, such as anonymisation

Describes the procedure to manage access to only authorised users.

Considers possible issues with sharing data with collaborators.

Provides information that is too vague.

Lacks assurance which enables reviewers to be satisfied that the project has considered the requirements linked to the processing of personal data, including access management and data sharing issues


o yes




10


4b. How will other legal issues, such as intellectual property rights and ownership, be managed?

What legislation is applicable?

Performance levels


If applicable, it clearly explains › who will control access to which part of the

data

› how intellectual property rights will be

managed.

For multi-partner projects and multiple data owners, it explains how these matters are addressed in the consortium agreement. It may state that this agreement is still being discussed and that the plan will be updated accordingly.

Alternatively, there is a clear statement that there are no such restrictions on the data.

Does not address this issue or only for a subset of the data, without good reason or explanation for doing so.

It is a multi-partner project and matters of rights to control access to the data are not addressed at all, without good reason or explanation for not doing so.


o yes




11


4c. What ethical issues and codes of conduct are there, and how will they be taken into account?

Performance levels


Provides evidence that an ethics committee approval has been obtained for the study (if applicable).

Gives details of what ethical issues have been considered and that adequate measures are in place to manage requirements.

Refers to relevant ethical guidelines.

Does not provide any or enough details to demonstrate that ethical implications have been considered.


o yes




12

5. Data Sharing and Long-term Preservation

5a. How and when will data be shared? Are there possible restrictions to data sharing or embargo

reasons?

Performance levels


Clearly indicates how the data and / or metadata will be discoverable and shared.

Specifies when data will be shared.

Includes information on how long the data will be retained and gives precision on the timely release.

If data sharing is limited or not possible clearly explains why, and under which conditions and who can access the data (certain communities or under sharing an agreement….).

Also explains what actions, where possible, will be taken to overcome or to minimise such restrictions.

There is no explanation, or the explanation is too vague.


o yes




13


5b. How will data for preservation be selected, and where data will be preserved long-term (for

example a data repository or archive)?

Performance levels


Provides details of what data collected or created in the project will be preserved and clearly indicates for how long, in alignment with funders, institutional, national policies or legislation, or community standards

Provides details of which (versions of) data and accompanying documentation will be retained or destroyed, and explains rationale (e.g. contractual, legal requirements or regulatory purposes).

Provides details of how the selection is made, and what possible interest the data would have for re-use (or not).

Provides details of how the data and accompanying documentation will be preserved.

Explains how data will be managed in a sustainable way beyond the lifetime of the grant.

Provides name of the repository - or the way to curate and preserve data - that will be used to make data available for re-use

Provides no further information or lacks adequate explanation.


o yes




14


5c. What methods or software tools are needed to access and use data?

Performance levels


Clearly indicates which specific tools or software (e.g. specific scripts, codes or algorithms developed during the project) potential users may need to access, interpret and (re-)use the data.

Outlines also a strategy for making specific tools and software available. Full details may not be yet available, but there is a clearly stated intention and plan to do so, or explanation of why this may not be possible.

It is stated that software (developed during the project) will be necessary to access and interpret the data, but no explanation of how it will be made available (or why that may not be possible) is offered.


o yes




15


5d. How will the application of a unique and persistent identifier (such as a Digital Object Identifier

(DOI)) to each data set be ensured?

Performance levels


Specifies how the data can be re-used in other contexts.

Clearly indicates if persistent identifiers are

provided for all data sets.

The approach is clearly presented and the choice of identifiers is justified and refers to international standards

Indicates whether a persistent identifier (for instance DOI, Handle, URN, ARK) will be assigned to the data (if not, why not).

Makes no mention of persistent identifiers without a valid reason for not providing them.


o yes




16

6. Data Management Responsibilities and Resources

6a. Who (for example role, position, and institution) will be responsible for data management (i.e.

the data steward)?

Performance levels


Clearly outlines the roles and responsibilities for data management / stewardship (e.g. data capture, metadata production, data quality, storage and backup, data archiving, and data sharing), naming responsible individual(s) where possible.

Clearly indicates who is responsible for reviewing and revising the DMP and for its day-to-day implementation.

For collaborative projects, explains the coordination of data management responsibilities across partners.

Does not discuss responsibility for data management / stewardship activities and/or does not indicate who is responsible for reviewing and revising the DMP and for its day-to-day implementation

It is a collaborative project but there is no description of how data management responsibilities will be coordinated across partners.


o yes




17

6. Data Management Responsibilities and Resources

6b. What resources (for example financial and time) will be dedicated to data management and

ensuring that data will be FAIR (Findable, Accessible, Interoperable, Re-usable)?

Performance levels


Provides clear estimates of the resources and costs (e.g. storage costs, hardware, staff time, costs of preparing data for deposit, and repository charges) that will be dedicated to data management and ensuring that data will be FAIR; describes how these costs will be covered.

Alternatively, there is a statement that no additional resources are needed.

Provides no answer, or is vague about the resources required for data management and ensuring that data will be FAIR (e.g. resources are not listed or costed inappropriately), and/or does not describe how the costs will be covered.


o yes




18

C. General Comments

Do you have additional comments or suggestions on the DMP evaluation criteria? [text field]

Do you have comments on the Science Europe RDM Guide in general and on the objective to

achieve a broad alignment of RDM policies across Europe?

consultation: science europe draft dmp evaluation rubric · consultation: science europe draft dmp...

Documents