OPENSCIENCEMONITOR

STUDYONOPENSCIENCE:MONITORINGTRENDSAND

DRIVERSReference:PP-05622-2017

D.2.4Finalreport

13thofDecember2019

Consortiumpartners:

TheLisbonCouncil

ESADEBusinessSchool

CentreforScienceandTechnologyStudies(CWTS)atLeidenUniversity

Subcontractor: Elsevier

2

TableofContents1 Introduction...........................................................................................................................................................5

1.1 Objectives....................................................................................................................................................5

1.2 Scope..............................................................................................................................................................7

1.3 Methodologicalapproach....................................................................................................................8

2 Overviewoftrendsandindicators............................................................................................................9

2.1 Openaccesstopublications...............................................................................................................9

2.2 Openresearchdata..............................................................................................................................13

2.2.1 Researchers’behaviour..........................................................................................................14

2.2.2 Howdataareshared................................................................................................................14

2.2.3 Dataownership..........................................................................................................................16

2.2.4 Attitudesacrossdisciplines.................................................................................................17

2.2.5 Datarepositories.......................................................................................................................19

2.3 Opencollaboration..............................................................................................................................20

3 Driversandbarriers.......................................................................................................................................23

3.1 Drivers........................................................................................................................................................23

3.2 Barriers......................................................................................................................................................30

4 Impact....................................................................................................................................................................34

4.1 Scientificimpact....................................................................................................................................35

4.1.1 Crossanalysisofcases............................................................................................................36

4.1.2 Qualitativeinsightfromcases.............................................................................................37

4.2 Industrialimpact...................................................................................................................................40

4.2.1 Crossanalysisofcases............................................................................................................42

4.2.2 Qualitativeinsights...................................................................................................................44

4.3 Societalimpact.......................................................................................................................................49

4.3.1 Crossanalysisofcases............................................................................................................50

5 Policyanalysis...................................................................................................................................................53

5.1 Statisticaloverview.............................................................................................................................53

5.1.1 Openaccesspolicies.................................................................................................................53

5.1.2 Opendatapolicies.....................................................................................................................55

5.1.3 Otherpolicies...............................................................................................................................57

5.2 Indepthcross-analysisofpolicycases......................................................................................58

5.2.1 Objectives......................................................................................................................................58

5.2.2 Provisions......................................................................................................................................59

5.2.3 Lessonslearnt.............................................................................................................................61

6 Policyconclusions...........................................................................................................................................62

3

1.1. Conclusionsonthefutureofopenscience..............................................................................62

1.2. ConclusionsonthefutureoftheOpenScienceMonitor..................................................63

7 References...........................................................................................................................................................67

ListofFiguresFigure1:Aconceptualmodel:aninterventionlogicapproach.............................................................6

Figure2:Percentageofopenaccesspublications........................................................................................9

Figure3:Percentageofopenaccesspublicationsbycountry.............................................................11

Figure4:PercentageofopenaccesspublicationsbyFieldofScienceandTechnology........13

Figure5:Haveyoupublishedtheresearchdatathatyouusedorcreatedaspartofyourlastresearchprojectinanyofthefollowingways?.......................................................................14

Figure6:Doyoutakestepstomanageyourresearchdataand/orarchiveitforpotentialre-usebyyourselfand/orothers?..........................................................................................................15

Figure7:Haveyoudoneanyofthefollowingwithanyoralloftheresearchdatathatyouusedorcreatedaspartofyourlastresearchproject?.................................................................15

Figure8:Categoriesofresearchdatausedinyourlastresearchproject.....................................15

Figure9:Researchdataformatsusedorcreatedaspartofyourlastresearchproject........16

Figure10:Whatwastheprimarysourceoffundingforyourlastresearchproject?.............16

Figure11:Whatwastheprimarysourceoffundingforyourlastresearchproject?.............17

Figure12:Attitudestodatamanagementbysubject(I/II)..................................................................17

Figure13:Attitudestodatamanagementbysubject(II/II)................................................................18

Figure14:Attitudestodatamanagementbyregion/age(I/II).........................................................18

Figure15:Attitudestodatamanagementbyregion/age(II/II)........................................................19

Figure16:Datarepositoriesbysubject..........................................................................................................19

Figure17:Datarepositoriesbytypeofaccess............................................................................................19

Figure18:Datarepositoriesbycountry,bluebarindicateEuropeancountries......................20

Figure19:NumberofscientificAPIs................................................................................................................21

Figure20&Figure21:Openhardwareprojects........................................................................................21

Figure22:ProjectsinScistarterbydiscipline.............................................................................................22

Figure23:projectsinZooniversebydiscipline..........................................................................................22

Figure24:Driversofsharingresearchdata.................................................................................................25

Figure25:Howimportantwerethefollowingfactorswhendecidingtouseother’sresearchdata?...................................................................................................................................................26

Figure26:Howwouldyoudescribetheefforttypicallyrequiredtomakeyourresearchdatare-useablebyothers?..........................................................................................................................26

Figure27:Barriersacrossthecasestudies..................................................................................................32

Figure28:Numberofbarriersacrosscasestudies...................................................................................32

Figure29:Barrieracrossopensciencecategories....................................................................................33

4

Figure30:Thinkingaboutthemostrecentresearchprojectonwhichyoushareddata,didindividualsoutsideoftheresearchteamcontactyouconcerningthedatathatyoushared?Iwascontactedbyresearchersfrom..................................................................................34

Figure31:Stillthinkingaboutthemostrecentresearchprojectonwhichyoushareddata,doyoubelieveanyofthefollowinghappenedasaconsequenceofsharingthedata?...................................................................................................................................................................................34

Figure32:Scientificimpactsaslistedinthecrossanalysisofpolicies..........................................36

Figure33:Topfivedriversfoundincasestudieswithhighindustryimpact.............................44

Figure34:Impactonsocietyoftheanalysedcasestudies...................................................................50

Figure35:Casestudieswithhighandmediumimpactonsociety,byresearchphases.......51

Figure36:Funder’spoliciesonopenacesspublishingbycountry..................................................53

Figure37:Funders’policiesonopenaccessarchiving...........................................................................54

Figure38:Archivingpolicesbytypeofmandate.......................................................................................55

Figure39:Opendatapoliciesbytypeofmandate....................................................................................56

Figure40:Opendatapoliciesbycountry......................................................................................................56

Figure41:Journalsbydatasharingmodality..............................................................................................57

Figure42:Journalsbycodesharingmodality.............................................................................................57

ListofTablesTable1:Articulationofthetrendstobemonitored....................................................................................7

Table2:overviewofthemethodology...............................................................................................................8

Table3:Openaccesspublications(gold,green,hybridandbronze)byyear.............................10

Table4:Openaccesspublications(gold,green,hybridandbronze)bycountry.....................12

Table5:Codingdriversanddescription.........................................................................................................24

Table6:Analyticalsummaryofmechanismsenablingthepositiveimpactofopensciencetoindustry...........................................................................................................................................................48

Table7:Casestudieswithhighandmediumimpactonsociety,byopensciencecategory...................................................................................................................................................................................51

Table8:Overviewofscientific,societalandindustrialdriversacrossNLandFIOpenScienceandUKOpenResearchPolicies..............................................................................................58

Table9:ProvisionsaddressedintheNLandFIOpenScienceandUKOpenResearchPolicies..................................................................................................................................................................60

Table10:Highlevelmethodologicalapproach,past(x)andfuture(greyshade)....................65

5

1 IntroductionOpen science has emerged as a powerful trend in research policy. To be clear,opennesshasalwaysbeenacorevalueofscience,butitmeantpublishingtheresultsor research in a journal article. Today, there is consensus that, by ensuring thewidestpossibleaccessandreusetopublications,data,codeandotherintermediateoutputs, scientific productivity grows, scientific misconduct becomes rarer,discoveriesareaccelerated.Yetitisalsoclearthatprogresstowardsopenscienceisslow,because ithas to fit ina system thatprovidesappropriate incentives toallparties.TheEuropeanCommissionhasrecognizedthischallengeandmovedforwardwithstronginitiativesfromtheinitial2012recommendationonscientificinformation(C(2012)4890), suchas theOpenSciencePolicyPlatformand theEuropeanOpenScienceCloud.1OpenaccessandopendataarenowthedefaultoptionforgranteesofH2020.The Open Science Monitor (OSM) aims to provide data and insight needed tosupporttheimplementationofthesepolicies.Itgathersthebestavailableevidenceon the evolution of open science, its drivers and impacts, drawing on multipleindicatorsaswellasonarichsetofcasestudies.2This monitoring exercise is challenging. Open science is a fast evolving,multidimensional phenomenon. According to the OECD (2015), “open scienceencompasses unhindered access to scientific articles, access to data frompublicresearch,andcollaborativeresearchenabledbyICTtoolsandincentives”.Thisverydefinition confirms the relative fuzzinessof the conceptand theneed fora cleardefinitionofthe"trends"thatcomposeopenscience.Preciselybecauseofthefastevolutionandnoveltyofthesetrends,inmanycasesitis not possible to find consolidated, widely recognized indicators. For moreestablished trends, such as open access to publications, robust indicators areavailable through bibliometric analysis. Formost others, such as open code andopenhardware,therearenostandardisedmetricsordatagatheringtechniquesandthereistheneedtoidentifythebestavailableindicatorthatallowsonetocapturetheevolutionandshowtheimportanceofthetrend.

Today, especially at European levelwhere competences on research are limited,data and indicator play a powerful role in executing policies. Conversely, theabsenceofrobustdatacanhindertheimplementationofthepolicy.

ThisispreciselytheobjectiveoftheOpenScienceMonitor.

1.1 ObjectivesTheOSMcoversfourtasks:

1. Toprovidemetricsontheopensciencetrendsandtheirdevelopment.2. Toassessthedrivers(andbarriers)toopenscienceadoption.

1https://ec.europa.eu/research/openscience/index.cfm?pg=open-science-policy-platform;https://ec.europa.eu/research/openscience/index.cfm?pg=open-science-cloud2TheOSMhasbeenpublishedin2017asapilotandre-launchedbytheEuropeanCommissionin2018throughacontractwithaconsortiumcomposedbytheLisbonCouncil,ESADEBusinessSchoolandCWTSofLeidenUniversity(plusElsevierassubcontractor).Seehttps://ec.europa.eu/research/openscience/index.cfm?pg=home&section=monitor

6

3. Toidentifytheimpacts(bothpositiveandnegative)ofopenscience.4. Tosupportevidence-basedpolicyactions.

The indicators presented here focusmainly on the first two tasks:mapping thetrends, and understanding the drivers (and barriers) for open scienceimplementation.Thechartbelowprovidesanoverviewoftheunderlyingconceptualmodel.

Figure1:Aconceptualmodel:aninterventionlogicapproach

Thecentralaspectofthemodelreferstotheanalysisoftheopensciencetrendsandis articulated alongside three dimensions: supply, uptake and reuse of scientificoutputs.In the OSM framework, supply refers to the emergence of services such as datarepositories.Thenumberofdatarepositories(oneof theexisting indicators) isasupplyindicatorofthedevelopmentofopenscience.Onthedemandside,indicatorsinclude,forexample,theamountofdatastoredintherepositories,thepercentageofscientistssharingdata.Finally,becauseofthenatureofopenscience,theanalysiswillgobeyondusage,sincethereusedimension isparticularlyimportant. In thiscase,relevantindicatorsincludethenumberofscientistsreusingdatapublishedbyotherscientists,orthenumberofpapersusingthesedata.On the left side of the chart, themodel identifies the key factors influencing thetrends,bothpositivelyandnegatively(i.e.driversandbarriers).Bothdriversandbarriers are particularly relevant for policy-makers as this is the areawhere anaction canmake greatest difference, and are therefore strongly related topolicyrecommendations.Theseinclude“policydrivers”,suchasfunders’mandates.Itisimportanttoassessnotonlypolicydriversdedicatedtoopenscience,butalsomoregeneralpolicydriversthatcouldhaveanimpactontheuptakeofopenscience.Forinstance,theincreasingrelianceonperformance-basedfundingortheemphasison

7

marketexploitationofresearcharegeneralpolicydriversthatcouldactuallyslowdowntheuptakeofopenscience.Therightsideof thechart in themodel illustrates the impactsofopensciencetoresearch or the scientific process itself; to industry or the capacity to translateresearchintomarketableproductsandservices;tosocietyorthecapacitytoaddresssocietalchallenges.

1.2 ScopeBydefinition,openscienceconcernstheentirecycleofthescientificprocess,notonly open access to publications. Hence the macro-trends covered by the studyinclude:openaccess topublications,open researchdata andopencollaboration.Whilethefirsttwoareself-explanatory,openscientificcollaborationisanumbrellaconcepttoincludeformsofcollaborationinthecourseofthescientificprocessthatdonotfitunderopendataandopenpublications.Table1:Articulationofthetrendstobemonitored

Categories TrendsOpenaccesstopublications

• Openaccesspolicies(fundersandjournals),• Greenandgoldopenaccessadoption(bibliometrics).3

Openresearchdata

• Opendatapolicies(fundersandjournals)• Opendatarepositories• Opendataadoptionandresearchers’attitudes.

Opencollaboration

• Opencode,• Altmetrics,• Openhardware,• Citizenscience.

Newtrendswithintheopenscienceframeworkwillbeidentifiedthroughinteractionwiththestakeholder’scommunitybymonitoringdiscussiongroups,associations(suchasResearchDataAlliance-RDA),mailinglists,andconferencessuchasthoseorganisedbyForce11(www.force11.org).

Thestudycoversallresearchdisciplines,andaimstoidentifythedifferencesinopenscienceadoptionanddynamicsbetweendiversedisciplines.Currentevidenceshowsdiversityinopensciencepracticesindifferentresearchfields,particularlyindata-intensive research domains (e.g. life sciences) compared to others (e.g.humanities)

Thegeographiccoverageofthestudyis28MemberStates(MS)andG8countries,includingthemaininternationalpartners,withdifferentdegreesofgranularityforthedifferentvariables.Asfaraspossible,datahastobepresentedatcountrylevel.

3AccordingtotheEC,“‘Goldopenaccess’meansthatopenaccessisprovidedimmediatelyviathepublisherwhenanarticleispublished,i.e.whereitispublishedinopenaccessjournalsorin‘hybrid’journalscombiningsubscriptionaccessandopenaccesstoindividualarticles.Ingoldopenaccess,thepaymentofpublicationcosts(‘articleprocessingcharges’)isshiftedfromreaders’subscriptionsto(generallyone-off)paymentsbytheauthor.[…]‘Green.openaccess’meansthatthepublishedarticleorthefinalpeer-reviewedmanuscriptisarchivedbytheresearcher(orarepresentative)inanonlinerepository.”(Source:H2020ModelGrantAgreement)

8

1.3 MethodologicalapproachTheOpenScienceMonitorgathersdatafromapluralityofsources.Whilethedetailsarefullyexplainedinthemethodologicalnote,itisworthsummarizingthemainaspectshere.Thestudyproduceskeyindicatorsandmetricsrelatedtotheabove-mentionedopensciencetrends.Theseindicatorsaregeneratedusingmultipledatasourceswhichcanbecategorizedinthreemaintypes.First,ananalysisofexistingdataandmetrics,forinstanceasprovidedbytheSherpaorRe3datadatabase.Indicatorshereareproducedbyanalysingtheunderlyingdataprovidedbytheseservices.Secondly,thestudygeneratesitsownmetrics,namelyonopenaccesstopublications,byanalysingpublicationsbasedontheScopusandUnpaywalldata.Thirdly,thestudyincludedasurveyofaboutathousandresearchersworldwide,performedbyElsevier.Thestudyalsoprovidesananalysisofpolicies,drivers,barriersandimpacts.Inthiscase,theevidenceisprovidedmostlyby28casestudiesandtherelatedmeta-analysis.

Table2:overviewofthemethodology

Existingmetrics

Ownmetrics

Survey Cases

Trends X X X Publication X X

Data X X Collaboration X Drivers X XBarriers X XPolicies X XImpact X X

9

2 OverviewoftrendsandindicatorsThissectionpresentsanoverviewofupdatedindicatorsofOpenScienceMonitor.Theinitialoverviewoftrendsisbasedontheinterventionlogicofthestructureofthemethodologyproposedintheinceptionreport:

• Openaccesstopublications,• Openresearchdata,• Opencollaboration.

AllthedataincludedinthissectionaretakenfromtheonlinedashboardoftheOpenScience Monitor, available at: https://ec.europa.eu/info/research-and-innovation/strategy/goals-research-and-innovation-policy/open-science/open-science-monitor_en.Fulldetailsonthemethodologyareavailableonthewebsiteandinannextothisreport.

2.1 OpenaccesstopublicationsThe indicators ofOpen access to publications aremeasuring towhat extent it ispossible to freelyaccess researchpublications.The indicators coverbibliometricdataonpublications,aswellasdataonfunders'andjournals'policies.Concerningtheavailabilityofopenaccesspublications,thetotalnumberofopenaccess publications (gold, green, hybrid and bronze) had been growing between2009and2018(theamountofopenaccesspublicationshasalmostdoubledinthisperiodfrom361thousandtoover684thousand),though,asignificantslowdowningrowth since 2016was registered (8% growth as compared to growth rangingbetween12%to16%inthepreviousyears).Datafor2017and2018showadecline,however inourconsiderationthis issimplyduetotheembargoperiod forgreenopenaccesssothattheywillbecomeavailableoverthenextmonths–asshownbythe fact that gold open access, where embargo does not apply, is continuouslygrowing.Figure2:Percentageofopenaccesspublications

10

Table3:Openaccesspublications(gold,green,hybridandbronze)byyear

Year 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018

NotOA 807672 824427 876706 917504 971428 1010275 1017187 1044540 1086564 1208464

NotOA%ofchange 2.1% 6.3% 4.7% 5.9% 4.0% 0.7% 2.7% 4.0% 11.2%

OpenAccess 361932 389100 445341 503875 569939 625503 677841 731812 736656 684636

OA%change 7.5% 14.5% 13.1% 13.1% 9.7% 8.4% 8.0% 0.7% -7.1%

GoldOA 54910 72608 101740 140147 175309 212999 242785 269360 309593 349589

%change 32.2% 40.1% 37.8% 25.1% 21.5% 14.0% 10.9% 14.9% 12.9%

GreenOA 239013 264567 303047 344491 395195 444121 478279 514907 490630 289461

%change 10.7% 14.5% 13.7% 14.7% 12.4% 7.7% 7.7% -4.7% -41.0%

HybridOA 42571 43850 57033 62363 70992 84624 102384 119348 117541 119019

%change 3.0% 30.1% 9.3% 13.8% 19.2% 21.0% 16.6% -1.5% 1.3%

BronzeOA 134029 134547 139945 147194 151145 144268 143379 138292 134500 93216

%change 0.4% 4.0% 5.2% 2.7% -4.5% -0.6% -3.5% -2.7% -30.7%

Thecloserlookintothedatarevealsthatthegrowthwasmainlydrivenbytherapidgrowthofthenumberandshareofopenaccessgoldpublications.In2018,theopenaccess gold publications outnumbered for the first-time open access greenpublications(green–289thousandtogold–349thousand).

At the country level, top five countries with the biggest share of open accesspublications(totalofgold,green,hybridandbronze)are:UnitedKingdom(52.3%),Switzerland (51.8%), Croatia (50.8%), Luxembourg (50%) and Netherlands(49.9%).Ontheoppositespectrumbottomfivecountrieswiththelowestshareofopen access publications are: Russian Federation (23.9%), China (27.8%), India(29.9%),Greece(34.7%),Canada(37.1%).

However, if we look at the absolute numbers of open access publications, thesituation changes significantly with US clearly leading the way to open access(27.6% of all open access publications), followed by China (10%), UK (9.9%),Germany(6.6%)andJapan(5.4%).Togetheropenaccesspublicationsofthesefivecountriesrepresentnearly60%ofallopenaccesspublications.

11

Figure3:Percentageofopenaccesspublicationsbycountry.

12

Table4:Openaccesspublications(gold,green,hybridandbronze)bycountry

Intermsofthescienceandtechnologyfield,Multidisciplinaryfieldhasthebiggestshareofopendatapublications–86.2%,followedbyOthermedicalsciences55.7%,Agriculture,forestry,fisheries(51.9%),andlifesciences:Biologicalsciences(49.2%),Basicmedicalresearch(48.5%)andClinicalmedicine(44%).

Themost closed fieldswith the lowest number of open access publications are:Mechanicalengineering(17.3%),Art(19%),Chemicalengineering(19%),Chemicalsciences(19.4%)andMaterialsengineering(19.8%).

13

Figure4:PercentageofopenaccesspublicationsbyFieldofScienceandTechnology

It isworthnoticing, however, that in termsof absolute numbers the situation isslightly different. Clinical medicine, Biological sciences, Basic medical research,PhysicalsciencesandastronomyandEarthandrelatedenvironmentalscienceshavethe highest aggregated number of open access publications, whereas Otheragriculturalsciences,Art,Mediaandcommunication,LawandPoliticalsciencearethefieldswiththelowestnumberofopenaccesspublications.

2.2 OpenresearchdataTheOpenresearchdataindicatorsmeasuretowhatextentdataunderpinningscientificresearchresultshasnorestrictionsonitsaccess.Themainsourcefor

14

theseindicatorsisthesurveyrunbyElsevierin2018,butsomeindicatorsaredevelopedthroughexistingmetrics.

2.2.1 Researchers’behaviourDataareafundamentalinputinallresearchactivities.Observationaldataarethemostcommondatacategory,usedby48%ofresearchers,followedbyexperimentaldata,usedby35%ofresearchers.Intermsofsize,thevastmajorityofresearchers(70%)generateslessthan10GBofdataintheirlastresearchproject.Researcherssharedataindifferentways.Themostcommonlyusedmethodisasanappendixtoanarticle(35%),whileonly14%ofresearchersshareitthroughdatarepositories.Overall,athirdofresearchersdeclarenottosharetheirdata.This ismostlygoodnews: thevastmajorityofresearchersdeclaretosharedata.However,acloserlookmakesthesedatalessgoodthantheylook:

1. Thisisdatasharingasself-reportedbyresearchers.Notonlyitispronetotheusuallimitationsofthemethodcomparedtoobservation,butalsoitdoesnotreflectthearticleasaunitofanalysis.Othersourcesshowthatthevastmajorityofarticlesdonotincluderesearchdata.

2. Repositoriesareconsideredthemostsuitablewaytomakedatagenuinelyaccessibleandreusableforotherresearchers,buttheyareyetusedonlybyasmallminorityofresearchers(roughlyoneinseven).

3. These results are practically identical to the 2016 survey, both in theaggregateand in the individualoption, showing that limitedprogresshasbeenachieved.

Figure5:Haveyoupublishedtheresearchdatathatyouusedorcreatedaspartofyourlastresearchprojectinanyofthefollowingways?

Ontheotherhand,researchersneeddatasharing:42%declarethattheyrelyondatafromotherresearchers,and74%declaretheywouldbenefitfromaccesstoothers’researchdata.

2.2.2 HowdataaresharedRegardingapproachestoresearchdatamanagement,themajorityofresearchersdotakestepstomanagetheirresearchdataforpotentialfuturereuse(89%),however,

15

this isnotalwayscarriedout inaconsistent fashionwith49%statingthat thesestepsweretaken“sometimes”.Again,thereisnoprogressvisiblefrom2016.Figure6:Doyoutakestepstomanageyourresearchdataand/orarchiveitforpotentialre-usebyyourselfand/orothers?

When focussing on data sharing practices, research data sharing is carried outmostlybetweencollaboratorsonthesameprojects(80%).Only38%ofthosewhoshare, do it with researchers outside of their own project. This suggests thatresearcher do not adopt fully open data approaches, but rather discriminatoryapproach,sharingwithselectedpartnersonacasebycasebasis.Figure7:Haveyoudoneanyofthefollowingwithanyoralloftheresearchdatathatyouusedorcreatedaspartofyourlastresearchproject?

Thenumberofresearcherssharingdataintheirlastprojecthasremainedstable,withnogrowthshownoverthepasttwoyears.Observationalandexperimentaldatacontinue to be the main type of data used by researchers (48% and 35%respectively).Figure8:Categoriesofresearchdatausedinyourlastresearchproject.

16

Whenanalysingtheresearchdataformatsusedinresearchprojects,numericalandtext formats are the most popular (60% and 42% respectively) amongstresearchers.Figure9:Researchdataformatsusedorcreatedaspartofyourlastresearchproject

2.2.3 DataownershipRegardingresearchfunding,acountry/subjectspecificfunderremainstheprimarysourceoffunding.Whenobservingdataownership,thedatawasdividedintotwosections:1)beforepublicationand2)afterpublication.Themajorityofresearcherssaytheyownthedatapre-publication(62%).Figure10:Whatwastheprimarysourceoffundingforyourlastresearchproject?

Afterpublication,theperceptionthattheresearcherownsthedatadecreasesandthereisastronggrowthintheperceptionthatthepublisherownsthedatawithanincreasefrom4%(priortopublication)to35%(afterpublication).

17

Figure11:Whatwastheprimarysourceoffundingforyourlastresearchproject?

2.2.4 AttitudesacrossdisciplinesRegardingresearcherattitudestodatamanagementandsharing,Chemists(87%)andComputerScientists(86%)aremostlikelytobenefitfromdatasharingandhavethestrongestrelianceonresearchdatafromoutsideoftheirresearchteam(54%and58%respectively).Mathematiciansstandoutasthegroupwhoaremostwillingtoallowotherstoaccesstheirresearchdata(82%)andwhohavesharedtheirdatawithothersinthepast(79%).Figure12:Attitudestodatamanagementbysubject(I/II)

ComputerScientistsaremostlikelytoberewardedforsharing(60%)andastronglinkbetweendatasharingandrewardcanalsobeseenamongstChemists(58%)andLifeScientists(52%).Meanwhileonly30%ofresearchersinthefieldofSocialSciences,ArtsandHumanitiesandEconomicsstatedthatsharingresearchdataisassociatedwithcreditandrewardintheirfield.Intermsoftraininginresearchdata,

18

Chemists believe they have received the most experience in this area (51%).However,theperceptionofagenerallackoftrainingcanbeseenacrosstheboardintheothersubjects.Chemistsalsostandoutasthegroupthatmakestheirresearchdataavailabletopublisherstomakeitaccessiblewiththeirresearcharticles(79%).Those researchers in Medicine & Allied Health most believe there is a role forresearch data management specialists in research data sharing. Finally, bothMathematicians and Computer Scientists are the groups that most reuse otherresearcher’sdata(76%and66%respectively).Figure13:Attitudestodatamanagementbysubject(II/II)

Whenanalysingtheattitudestodatamanagementbyregionandage,itcanbeseenthatthoseinAsiaPacific(APAC)areleastlikelytosharetheirdata(69%)andthoseinWesternEuropeare finding itmoredifficult toaccessother researcher’sdata(44%).Thoseresearchersundertheageof36arethegroupthatmostagreethataccesstootherresearcher’sdatawouldbenefittheirownresearch(82%).However,thestrongwillingnesstoallowaccesstoresearchdatabytheover55sshouldalsobehighlighted(76%).

Figure14:Attitudestodatamanagementbyregion/age(I/II)

19

In Asia Pacific there is a stronger link between sharing research data and theperceivedcreditandrewardthatitbringstoresearchers(60%).Africa(90%),theMiddleEast(71%)andLatinAmerica(69%)standoutasthoseregionswherethereisaneedforresearchdatamanagementspecialistsinresearchdatasharing.Figure15:Attitudestodatamanagementbyregion/age(II/II)

2.2.5 DatarepositoriesRegarding the availability of data repositories, there were no less than 3449repositorieslistedonRe3data.orginOctober2019,upfrom2986in2018,outofwhichrepositoriesoflifescienceconstituted36%,naturalscience33%,humanitiesand socialscience21%andengineeringsciences10%.Thevastmajorityofdatarepositoriesprovideopenaccesstothedata–over94%ofallrepositoriesforwhichthisinformationisavailable.Figure16:Datarepositoriesbysubject

Figure17:Datarepositoriesbytypeofaccess

ThevastmajorityofdatarepositoriesislocatedinUS–39,2%(1048repositories)–almostthreetimesthenumberofdatarepositoriesofGermany–asecondcountryon the list with 14% of data repositories and the EU country with the highestnumberofdatarepositories.

Lookingattop10countrieswiththehighestnumberofdatarepositoriesonlyfourEU countries are prominent: Germany (381 repositories), United Kingdom (282repositories),France(103repositories)andNetherlands(56repositories).

20

Fromnon-EUcountriesCanada(252repositories),Australia(90),Switzerland(68),Japan(56)andIndia(50)areplacedintop10countries.

RomaniaandCroatiaarethecountrieswiththelowestnumberofdatarepositories.Figure18:Datarepositoriesbycountry,bluebarindicateEuropeancountries

2.3 OpencollaborationOpenscientificcollaborationreferstotheformsofcollaborationinthecourseofthescientificprocessthatgobeyondopendataandopenpublications.Measuringopenscientificcollaborationincludesmeasuringofdifferenttypeofoutputssuchasopencode,openhardware,theuseofcollaborativeplatformsbetweenscientistsandthe"citizen-science"phenomenon.Regarding the availability of scientific Application Programming Interfaces(APIs),thenumberofscientificAPIsisgrowingoverthecourseoflasttenyears.In2012therewasasuddenincreaseinanumberofAPIs(it´snumberedquadrupled)andsincethenanumberofAPIsgrewsteadilyto625scientificAPIsin2019.WhileAPIsarenotnew,theyareshowingcontinuousgrowthandbecomingcommonplacealsointhescientificcontext.

21

Figure19:NumberofscientificAPIs

JustasthenumberAPIs,numberofprojectsonOpenHardwareRepositoryisgrowingsteadily,amountingto323projectsin2019.

Figure20&Figure21:Openhardwareprojects

Anothergrowingphenomenonintherealmofopencollaborationisopencitizen-science. Open Science Monitor is using data from to science crowdsourcingplatformsSciStarterandZooniverse.ByOctober2019theSciStarterrecorded4506projects,upfrom4095in2018,outofwhichthemajorityreferredtolifesciencesandenvironment.Similarly,outof217projectsrecordedbyZooniversetillOctober2019,morethanhalfofthemconcernedlifesciencesandenvironment.

22

Figure22:ProjectsinScistarterbydiscipline

Figure23:projectsinZooniversebydiscipline

23

3 Driversandbarriers3.1 DriversTherearemanydifferentdriversatplay,whichcanfacilitateorhindertheadoptionofopenscience.Their identification is crucial to thedevelopmentof appropriatepolicyrecommendations,whicharetypicallybuiltaroundthese“levers”:theyaimtoenhancethedriverstoaccelerateadoption.Whiletherearegeneraltrends,externaltoscience,thatareimportantbecausetheyreinforce or damage open science, in the framework of the 28 case studiesperformed,theresearchteamattemptedtoidentifythespecificdriversorenablingfactorsthatwerebehindtheopenscienceinitiativesselected.Someofthefactorsmostcommonlymentionedinpriorresearchandstrictlyrelatedtoopenscience(non-includinggeneraltrendsaffectingopenscienceadoption)aresummarisedinTable5.Acrosstheanalysisofcaseswehave identifiedninemaindrivers thathavebeenclusteredinthreeoverarchingcategories:

1. Supporter: Refers to factors that describe the support of certainstakeholdersintheecosystemtotheinitiative.Suchsupportfacilitatesandencourages the adoption of open science behaviours by the researchcommunitythroughthecases.Underthesecategories, the factors include:Institutional support (i.e. when an institution is behind the initiative);publishers support (i.e.when a publisher or publishers are supporting insomeformtheinitiative);funderssupport(i.e.theinitiativehasthesupportofa fundingagency); industrysupport(i.e.whenthe initiativehasprivatesponsors);andcommunitysupport(i.e.thecaseissupportedbythescientificcommunity,citizensorfinalusersingeneral).

2. Intention: Includes factors where different design and implementationmeasuresweretakenintoaccount,thatis:whethertheinitiativewaseasilyto plug in already existing measures, processes and practices (i.e.commodity);andwhetheritimpliesmajororminorchangesintheexistingworkflows(i.e.changes).

3. Focus:Thethirdcategoryreferstohowtheinitiativetargetedthedifferent

collectives and includes:whether the initiativewas designed arounduserneeds,understandingtheirpracticespatternsandcommonbehaviour(i.e.user-centricity); or whether the initiative involved suppliers capable ofprovidetheproductorserviceandsupportadoptionorwasdesignedtakingintoaccountorinvolvingfuturesuppliers(i.e.supplier)

The description of all these factors, most commonly mentioned and uncoveredthroughtheanalysisof24cases,aresummarizedinTable5below.Wehavealsoincludedhowtheyaffectopenscienceadoptionthroughthecases.

24

Table5:Codingdriversanddescription

Category Drivers Description EffectsinopenscienceadoptionSupporter

Institutionalsupport Institutionalsupportsdesignswhenaninstitutionisbehindtheinitiative

Supportfactorsenablethegestationofthecasesinthefirstplace,butalsotheadoptionoftheinitiativewhensuchsupportcomesfromoneofthetargetgroupsoraffectstargetgroupsbehaviour.

Publisherssupport Publisherssupportdescribeswhenapublisherorpublishersaresupportinginsomeformtheinitiative

Funderssupport Funderssupportdesignswhentheinitiativehasthesupportofafundingagency

Industrysupport Industrysupportdesignswhentheinitiativehasprivatesponsors

Communitysupport Communitysupportreferstothesituationwhentherewasasupportbythescientificcommunity,citizensorfinalusersingeneralbehindtheinitiative

Intention Commodity Commodityreferstowhethertheinitiativewaseasilytopluginalreadyexistingmeasures,processesandpractices

Intentionfactorsenhanceadoptionbyeasingtheprocessofincorporatingsuchopensciencepracticetowardalreadyexistingbehavioursandprocesses.

Changes Changereferstowhethertheinitiativeimpliedmajorchangesintheworkprocessesofthetargetgroups

Focus Usercentricity Usercentricityreferstowhethertheinitiativewasdesignedarounduserneeds,understandingtheirpracticespatternsandcommonbehaviour

Focusfactorsfacilitateadoptionbytakingintoaccountbeneficiariesandstakeholdersviewinthedesignandthedevelopmentprocessoftheinitiative.

Supplier Supplierreferstowhethertheinitiativeinvolvedsupplierscapableofprovidingtheproductorserviceandsupportadoptionorwasdesignedtakingintoaccountorinvolvingfuturesuppliers

25

OverviewThe coding of the 24 cases developed in the framework of the Open Science Monitoruncoverednotonlythefactorsdescribedabovebutalsoitrevealsthattheyareunequallyimportanttotheadoptionofopensciencepractice.Acrossthethematicanalysisofcases,andcodingexercisedevelopedacrossthedatasources,threemajorgroupsoffactors,accordingtotheirrelevance,wereidentified.First,communitysupportandasupplierapproachappeartobethemostimportantfactorsexplainingthecaseadoptionandtakeup.Secondly,usercentricity and institutional support appears almost as much as relevant to explain thesuccessoftheopenscienceinitiativesselected.Finally,funderssupportandtheremainingfactorswouldexplainthemainresidualvarianceforthecaseadoption.Astheanalysisshowallfactorsuncoveredhadasignificantroleinthecases.However,wewillzoominnowinthedifferentopensciencetrendstounderstandhowdriverschangeandimpactdifferentlydependingonthetrend.

AnalysisperopensciencetrendTheanalysisofcasesreflectedsimilarpatternsacrossthediverseopensciencetrends(i.e.open access,open data, open collaboration and citizen science).Whenwezoom into thedifferenttrends,weobservethefollowing:OpenscientificdataLooking at the factors driving research data sharing, those benefits related to advancingsciencethroughcollaboration(66%)orreproducibility(57%)areconsideredtobemoreimportant toresearchersthanmeeting journal/publisherrequirements(28%)or fundingbodymandates(21%).Thefactthatsharingresearchdataalsoencouragesotherresearcherstomaketheirdatapubliclyavailablewasalsodeemedtobeofsubstantialimportance(55%).Figure24:Driversofsharingresearchdata

26

Gooddocumentationandcompliancewithbestpracticearethemostimportantfactorsfortrustingandusinganotherresearcher’sdata.Citationsinapublicationinformalliteratureorarepositorywithagoodreputationwerealsoconsideredtobeimportantfactors.Ontheotherhand,personalconnectionsarelessimportant.44%ofresearchersuseddatathathadbeensharedbyothersontheirlastresearchproject.Figure25:Howimportantwerethefollowingfactorswhendecidingtouseother’sresearchdata?

Formorethanhalfofresearchers,“someeffort”isrequiredtomakeresearchreusablebyotherssuggestingthatamoderateamountofworkhastobedonetomakedatareusableforothers. 27% of the surveyed researchers stated that “a lot of effort” was required,highlightingthecomplexityofdataprocessingandformattingasapotentialbarriertodatasharing.Figure26:Howwouldyoudescribetheefforttypicallyrequiredtomakeyourresearchdatare-useablebyothers?

Themajordriverspresentinthecasesrelatedtoopenscientificdatashowtherelevanceofinstitutional,communityorfundersupport,user-centricdesignoftheinitiativeandasupplierapproachwherethefutureexploitationisclear.Thequalitative analysis and fine-grainedanalysis through the cases reflect the followingdriverswithinopenscientificdata:

27

Drivers1. Contextoflowresearchanddevelopment(R&D)productivity:Opensciencemodelis

astrategythatcontributestofightlengthy,costly,lowsuccessrate,highattritionrates,andcomplexityinR&Dprocesses,inparticularindrugdiscoveryprocesses.

2. Newcomputationalapproaches:Theincreasingvalueofaggregatingdatarelyonnewcomputationalapproachesthatrequiredifferentgovernancemodels

3. Responsiblesharingofscientificdataisessentialtoacceleratescientificdiscovery4. Socialormoralobligation:Transparencyinclinicalresearchdatacanimprovepublic

healthandsafety;itisalsoconsideredasocialobligationforthecitizens'thatvolunteerthemselvestothosetrials

5. Pressurefromshareholdersabouttheliabilityofunpublisheddata

1. About fighting lengthy, costly, low success rate: The main drivers seen in OpenTargets,PistoiaAllianceandYODAaretoeffectivelylowerthebarrierstoR&DinnovationandincreaseR&Dproductivityinbiomedicalresearch.AsGSKdeclares:"Webelievethatharnessingthepotentialof"bigdata"andgenomesequencingthroughthiscollaborationcould helpus dramatically improve our success rate for discovering newmedicines,"(GSK,2017).

2. New computational approaches and the increasing valueof aggregatingdata inbiomedical research: The explosion of data availability in biomedical research hasincreasingly rendered it a data-driven science. As a result, drug discovery effortsaccumulatetodayfarmoreinformationthanforpreviousdrugs.Therearevastvolumesofdataassociatedwithdrugdiscoverythatisbeingaccumulatedinpharmaceuticalfirms.Throughdataaggregation,anintegrationofalltheheterogeneousandcomplementaryevidenceavailableanda combinationof computational approaches, theOpenTargetscaseshowhowopenresearchdataiseffectivelyfacilitatingscientiststosystematicallyidentifyandassessalltheevidenceavailablethatassociatesadrugwithdiversediseases.

3. Aboutsocialandmoralobligation:Transparencyinclinicalresearchdatacanimprovepublichealthandsafety.AsthecaseofYODAreflectsresearchdatasharingisconsideredasocialobligationforthecitizens'thatvolunteerthemselvestoclinicaltrials.YODAisalsodrivenbytheviewthatpatients,healthcareproviders,andthegreaterlifesciencesindustrywillbenefitifacademiacanprovideindependentreviewsofdatarelevanttothepotential benefits and harms of industry products, and that these analyses allowphysiciansandpatientstoinformtheirdecisionsonthemostcomprehensiveandrecentevidenceavailable.

4. Aboutthepressurefromshareholdersabouttheliabilityofunpublisheddata:Onereason to believe that companies agreed to join YODA, but also other frameworks todisseminateclinicaltrialdata,isthattheirshareholdersareincreasinglyworriedaboutthe liability of unpublished data. As reported by different sources, owners of drug-companiessupportthebroaddisseminationofclinicaltrialsresultmakingsurethattheirresultsarefullyreported(TheEconomist,2015).Thereasonisthatlong-terminvestors

28

prefertoreducethelevelofriskintheirportfoliobyhavingalltheinformationabouttrialsfullydisclosed.

OpencollaborationOpencollaborationisanumbrellaconceptthatembraces:opensourcehardware,opencodeandreproducibilitycases.Themajordriverspresentinthecasesrelatedtoopencollaborationshowtherelevanceofauser-centricandsupplierapproachbehindtheinitiativeandthesupportfromcommunityandinstitutions.Regardingopensciencehardware,themore‘granular’driversthatwerefoundedacrosstheanalysisofWhiteRabbit(WR)casesandthecross-analysisofexistingopensourcehardwarelicencesare(includeheresourcestothecases):

Drivers1. To maximise the value of the technology for society by avoiding duplicative

investmentsonthesamehardware

2. Toallowtechnologyflexibility

3. Toallowextensivepeer-reviewofthetechnologydesigntoincreasetechnologyqualityandreliability

4. Toallowreusabilityofhardwaredesigns

5. Strongandhistoricalcommitmentwithopensciencebytheinitiativeleader

1. Tomaximise societal impact: When CERN decided to developWR as open source

hardware to solve a problem in their accelerators, they had the intention towidelydisseminate the technology tomaximise theirusebyotherorganisations inresearch.Opensourceproductsinsoftwarearerecognisedforofferingdecreaseddependencyonmonopoly suppliers (Bruns, 2000; Kogut and Metiu, 2001). This is analogous tohardware,anditisespeciallyvaluableforscientists.ByadoptingasimilarapproachforWR development, CERN intended to develop a platform to integrate a variety ofcontributionsfromdispersingstakeholders,whilemovingawayfromavendorlock-insituation,wherescientificinfrastructuresbuilddependencieswithtechnologyprovidersforhighlyspecifictechnologies.

2. To allow flexibility: The Open source hardware approach provides the requiredflexibility forscientificorganisations,whichisanessentialcharacteristic forscientistswhoneedcustomising,never-before-seenequipmentfortheirexperimentalendeavoursin uncertain andmodular environments. Such flexibility arguably leads to better andfaster progress of science (Pearce, 2014). By developingWR in the open, CERNwasseeking topave theway for thehigh customisationpotentialof the technology,whilefosteringeffectivepeer-reviewbyacommunityofexpertswhocanprovideusefulinput,testcases,andfeedbackinanopenspace.

29

3. Toallowextensivepeer-reviewof the technologydesign to increase technologyquality and reliability: By developing open hardware, the research infrastructure(CERN)was seeking to a dynamic peer-review of the technology by a community ofexpertswhocouldprovideusefulinput,testcases,andfeedbackinanopenspace.

4. To allow reusability: By adopting an open hardware approach, CERN was alsoexpandingthepossibilityofreusingthedesignsofotherelectronicsengineersworkingin experimental physics laboratories. The idea was to increase the efficacy of thetechnologybyreducingthenumberofdifferentteamsworkingindependentlyonsimilarproblemsandfocusdisperseyetcomplementaryknowledgeonacentralgoal.

5. Strongandhistoricalcommitmentwithopensciencebytheinitiativeleader:The

principleofopennessisstatedinCERN’sfoundingconvention,andtheorganisationhasbeenarelentlesspioneerinthisregardsincethereleaseoftheWorldWideWebunderanopensourcemodelbackin1994.CERNhascontinuouslyembracedtheprinciplesofopen science, such as open access with the Sponsoring Consortium for Open AccessPublishinginParticlePhysics-SAP3andallLHCpublicationshavebeenpublishedunderOpen Access conditions; open data, setting up the Open Data Portal for the LHCexperimentsorZenodo,a freeOpenDatarepository, launchedbytheorganisation foruse beyond the high-energy physics community; Invenio, an open source librarymanagementsoftwarepackageandtheiruseofopensourcelicences(NilsenandAnelli,2016; Murillo and Kauttu, 2018). Developing WR as an open source hardware wasalreadyinlinewiththephilosophicalmottoofCERNresearchactivity.

Reproducibility(includingpre-registration)

Drivers1. Awarenessabout‘reproducibilitycrisis'

2. Socialimpactmaximization

3. Strongcommitmentbyanorganisationworkingtofosteropenscience

1. Awareness about ‘reproducibility crisis’: Nature published in 2012 a study that

reviewed ten years of research, where scholars found that 47 out of 53 biomedicalresearchpapersoncancerresearchwasirreproducible(Prinzetal.,2011).Fouryearslater, in 2016, Nature surveyed 1576 scientists, finding that more than 70% ofresearchers failed to reproduce another scientist’s experiments andmore than 50%failed to reproduce the results of their experiments. Also, a famous study wasimplementedin2015inthefieldofPsychologywhenanopen,registeredempiricalstudyofreproducibilitywas launchedand270researchersaroundtheworldworktogetherandtrytoreplicate100empiricalstudiespublishedinthethreetopPsychologyjournals(theReproducibilityProject).Asa resultof thisgrowingawareness, reproducible researchhasbecomeapervasiveobjective in the research policy agenda at different governmental levels, includingfundersand journalpolicies. In thestruggle to improvethereproducibilityofscience,

30

there have been different initiatives from the scientific community to fight thereproducibility crisis, which include our case: REANA, a reusable and reproducibleresearchdataanalysisplatform.

2. Socialimpactmaximization:BydoingREANA,CERNknewnotonlytheparticle-physicscommunitywouldbenefit,butotherswouldalsodo,andthattheprojectwouldbenefitthevastscientificcommunity.

3. Strongcommitmentbyanorganisationworkingtofosteropenscience:REANAwasbornwiththegoaltofightthereproducibilitycrisisandasanaturalnextstepofCERN’sefforttomoveforwardtheiropenscienceeffortstothenextstep.

CitizenscienceFinally,themajordriverspresentinthecasesrelatedtocitizenscienceshowtherelevanceof community, institutional or publisher support and user centricity. The cases assessedshowed a successful adoption thanks to the support of the scientific community ororganization behind it and the user-centric design that facilitated the contributions by adistributedbaseofcitizensaroundtheglobe.

Drivers1. Increasedavailabilityofdatasetsinmanyresearchdisciplines

2. Motivationbyprojectownerstoproduceannotateddatasets

3. Usabilityofthetechnologyinterface

ThecasessuchasZooniverseshowthatthegrowthofcitizenscienceprojectsisbeingdrivenbytheincreasedavailabilityofdatasetsinmanyresearchdisciplinesaswellastheuseofweb-connectedcomputerandmobiletechnology.Ontheotherside,projectownersarealsomotivatedbytheneedtoproduceannotateddatasets forresearchpurposes.Finally, it isimportantthatthecitizenscienceprojectsmakeiteasytostartnewcitizenscienceprojectsandcontributetothemwithoutanysophisticatedtechnologicalexpertise.

3.2 BarriersThere aremany different drivers and barriers at play,which can facilitate orhinder theadoptionofopenscience.Theiridentificationiscrucialtothedevelopmentofappropriatepolicyrecommendations,whicharetypicallybuiltaroundthese“levers”:theyaimtoremovethebarriersandtoenhancethedrivers.Whiletherearegeneraltrends,externaltoscience,thatareimportantbecausetheyreinforceordamageopenscience,intheframeworkofthecasestudiesperformed,theresearchteamattempted to identify the specific drivers or enabling factors thatwere behind the openscienceinitiativesselectedandthebarriersorbottlenecksthatthedifferentprojectstudiedneededtoface.

31

Fromthecasestudies,anumberofrecurrentbarrierswereanticipated.Theyarelistedbelow:Category Barrier DescriptionMicro Lackofuserorientation Whenatool,method,approach,orcaseasawhole

wasdesignedwithouttakingintoaccountorinvolvingthefutureusergroups.

LackofAdoption Whenatool,method,approach,orcaseasawholeisonlyusedinaselectedsetoffields,usergroupsorinstitutes.

Technologicalbarrier Whenatool,method,approach,orcaseasawhole,showsthatthetechnologicalaspectsaretoocomplicatedforusers.

Lackofskills Whenatool,method,approach,orcaseasawholerequirestoospecificortoocomplicatedskillsofusers.

Cultural&behaviouralbarrier

Whenatool,method,approach,orcaseasawhole,doesnotfitinthewaythattheresearchisusuallycarriedout.

Lackofawareness Whentheexistenceofatool,method,approach,orcaseasawhole,isonlyknowninasmallusergroup.

Meso Lackofinstitutionalsupport

Whenatool,method,approach,orcaseasawholeisnotsupportedbyinstitutionalfunding.

Lackoffunderssupport Whenatool,method,approach,orcaseasawholeisnotsupportedbyfundingfromresearchfunders.

Lackofpublishers’support

Whenatool,method,approach,orcaseasawholeisnotsupportedbypublishers.

Macro Lackofbusinessmodels Whenatool,method,approach,orcaseasawholeisnotsupportedbyabusinessmodel.

Lackofstandards Whenitisnotclearwhetheratool,method,approach,orcaseasawholeisstandardized,andstandardsareeasilyavailable.

Difficultyinupscaling-fragmentation

Whenatool,method,approach,orcaseasawholeisnoteasytoupscaleinotherinstitutes/usergroupsandhencetheuseremainsfragmented.

Lackofdiversity Whenatool,method,approach,orcaseasawholeisnotsuitabletootherindividualswithdifferentskills,backgrounds,traditionsandbeliefs.

For each of the case studies, the research team has assessed whether the barrier wasapplicable.Ascorefrom1to3wasgiven,a1indicatingthatthisbarrierscoredhighinthecase,a2meaningthatthebarrierscoredmedium,anda3whenabarrierwasn’trelevantatall.Thescoreswereassignedinagroupdiscussion,basedonthecasestudystory.Inthefigurebelowisindicatedwhichofthebarrierswasgivena1(scoredhigh)mostoften.It isclear that ‘soft’barrierssuchasculturalandbehaviouralbarriersaremost frequent,occurring in 13 out 24 cases. The second highest barrier is difficulty in upscaling/

32

fragmentation(11/24),andthethirdoneisthelackofbusinessmodels(9/24),forwhichthereareahighnumberofscores2aswell.Thebarrierthathasthelowestscoreistechnologicalaspectsofatool,method,approachorcase;inonly2outof24casesthisbarrierisscoreda1.Also,thefinancialsupportbyfunders(2/24)orpublisher(5/24)israrelyscoredasastrongbarrier.Again,inthecasewheretherewasanequalnumberinscore1,thenumberofscores2wasdecisive.Thebarriersasscoredhereconfirmearlierstudiesonopendatapractice.Figure27:Barriersacrossthecasestudies

Thecrossanalysisofcasesalsoshowsthatthereisabroadrangeinthenumberofbarriersthatareexperiencedpercase.Inthefigurebelow,thecasestudiesarepresentedwiththetotalnumberofhighbarriers(score1).Ascanbeseenupto8outof13barriersapply,buttherearealsocaseswhichscorezerohighbarriers.Figure28:Numberofbarriersacrosscasestudies

33

The cases with the most barriers are Two use cases, As predicted, Mendeley and Openmetadata. They cover the research phases of data-integration, pre-registration andpublication.However,itseemsthatthebarriersinthecasesarenotnecessarilydependentontheresearchphase,becausethecaseswiththelowestnumberofhighbarriers(score1)arealso inpublication:YODAandORCIDwithzerobarriers,andZooniverseandGOFAIR(onemajorbarrier).Fromthenumberofbarriers,itseemsthatbroad‘technological’casessufferfromlessbarriersthan‘use’cases.Thisisconsistentwiththehighnumberofculturalandbehaviouralbarriers,andthefragmentation.Inordertofindpatternsinthebarriers,thescore1wasassessedacrossbarriercategoriesandopensciencetrends.Thescoresarenormalizedforthenumberofcasespertrend,andthenumberofsub-barrierswithinthemicro-,meso-ormacrobarriers.Theyarepresentedbelow:Figure29:Barrieracrossopensciencecategories

ThebarriersaremostprominentforOpencollaborations(micro-level)andCitizenscience(macro-level).ThisreflectsthefactthatthereisstrongpolicysupportforOpenaccess,alsoatmeso-level,butthattheactualbehaviouratthemicro-levelislaggingbehindinchange.Forcitizensciencethesituationisthereverse:thereislittlepolicysupportorinfrastructureforcitizenscience,whichshowstobetherelativelystrongestbarrier.The lowestbarrierscoresareforCitizenscienceandOpendata(meso-level),whichbothseemstobeodd,sincebothcitizenscienceandopendataareinaprematurestateinHEI’s.Otherwise,thebarriersaremoreorlessevenlyspread.

34

4 ImpactThissectionpresentsthekeyfindingsontheimpactofopenscienceasemergingfromthecasestudies.Inaddition,thesurveyprovidesadditionalinsightontheimpactofdatasharingpractices.Whenlookingattheconsequencesandimpactofdatasharing,overathirdofresearcherswerecontactedbyanotheruniversity/instituteaftersharingdata.10%werecontactedbyacompany,highlightingtheadditionalpositiveconsequencesofdatasharing.Figure30:Thinkingaboutthemostrecentresearchprojectonwhichyoushareddata,didindividualsoutside of the research teamcontact you concerning the data that you shared?Iwas contactedbyresearchersfrom

Furthermore, over a third of researchers believe that sharing data promoted newcollaborationswithresearchersintheirdiscipline.Figure31: Still thinking about themost recent researchproject onwhich you shareddata, do youbelieveanyofthefollowinghappenedasaconsequenceofsharingthedata?

Collaborationwith[academic]researchers…

35

4.1 ScientificimpactThereissubstantialevidencethatopenscienceiscorrelatedwithbetterscience,mainlybyremovingmistakesatanearlierstage,enhancingtheproductivityofsciencethroughdatareuseandreducingthecostsofsciencethroughcollaboration.Themaindifferentpositiveeffectsofimplementinganopenscienceapproachcapturedintheliteraturesofarinclude:

• Fasterdiscovery:Fasttracktodiscoverythankstoexternalcontributionsandcomments.Knowledge mining of open access literature can automatically classify, analyse andreason on existing literature,making newdiscoveries and connections that currentlyhappenveryseldomandunexpectedly.

• Increasedefficiency:Openscienceincreasestheefficiencyoftheresearchsystembecauseithelpstoreducetheduplicationofcosts,aswellascostsstemmingfromthecreation,transferandre-useofdata.Itallowsmoreresearchtobecreatedbythesamedataandthusboostsreturnonpubliclyfundedresearch(OECD,2015;Lyon,2009;Whyte&Pryor,2010).

• Greaterproductivity:Contributionsfromvolunteerstodatasets.

• Morerobustfindings:Willingnesstoshareresearchdataisrelatedtothestrengthoftheevidenceandthequalityofreportingofstatisticalresults(Wichertsetal.,2011).Intheabsence of open code, computational science as practiced today does not generatereliableknowledgebut“breezydemos”(Ioannidis2005).

• Increasedtransparencyandreplicationofstudies:Openscienceraisestransparencyandquality in the validation of research results and multiplies the opportunities forreplicabilityandvalidationofscientificfindings(Franceschet&Constantini,2010;OECD,2015;Fecheretal.2015).Bymakinginformationonmethods,protocols,anddataeasierto peer review, and by strengthening the scope of thematerial published (includingnegative results), open science offers greater scrutiny as it allows formore accurateverificationofresearchresults(Lyon,2009;Whyte&Pryor,2010).

• Increasedcollaborationandscientificinterdisciplinarity:Theimplementationofanopenapproachinscienceenhancescollaborationacrossinstitutional,nationalanddisciplinaryboundaries, and it fosters the exchange of information and expert knowledge (OECD,2015;Whyte&Pryor, 2010). Scientists are re-scaling the levelof contribution in thescientificprocessbycuttingitintosmallpiecesandenablingmicro-contributionsonamacro-scale.Hyper-specialisationisforeseenbyscientistsinmanycasesandtheylookfor micro-contributions from ‘citizen scientists’, which could lead to a furthergamificationofScience(VonAhn&Dabbish,2008;Szkuta&Osimo,2016).

• Minimisepublicationbiases:Minimisepublicationbiasesifstudiesthatreportnegativeornoeffectsarenotpublishedintraditionalvenues,butaremorelikelytobepublishedonnovelpublicationplatforms(Boultonetal.,2012).

• Newresearchfieldsandenhancedopportunities:BigDataishelpingtocreatebrandnewfieldsofscience:computationalchemistry,biology,economics,engineering,mechanics,neuroscience, and geophysics. Open science enables sharing of complex models andsimulationsbasedonlarge-scaleopendataanalysis(Lyon,2009;Boultonetal.,2012).

36

Modern computers can identify highly complex, unperceived relationships. Thus,technology is supporting traditional ways of doing science, and enables enquiry byconstructinghypothesesafteridentifyingrelationshipsinthedatasets.

• Newemerginginstrumentstoevaluatescience:Overthelasttenyears,withtheadoptionofweb 2.0 tools by scientists, new instruments are emerging to assess the quality ofscience outputs (Herman et al., 2015). Today, reputation (which is key to scholarlyresearch) is intrinsically linked to the model of journal publication. The impact ofresearch ismeasuredbycitationsandpatents.Recently, therehavebeenexperimentswith new evaluation and reputationmodels, such as data citation and data journals,altmetricsandpost-publicationreviews(SzkutaandOsimo,2016).

Regardingthepotentialnegativeimpacts,theliteraturesuggeststhatopensciencemayalsoposevariousthreats.Mostoftheconcernsarerelatedtoresearchquality:Withtheopennessoftheresearchprocesstherearesomeconcernsabouttheeffectivenessandpracticalityofqualitychecks.Thereisawidespreadacknowledgementthatassessingthequalityofsuchvastvolumesandrangesofscientificmaterialsischallenging(Whyte&Pryor,2010).ThemainquestionsthatpersistaftertherevisionofthedifferentstudiesthatOpenScienceMonitorsoughttoaddressthroughtheselectionandanalysisofasetofcasestudiesacrossopensciencetrendsare:

• Howdoesopenscienceimpactthequalityofresearch?

• Underwhichmechanismsdoesopenscienceincreasetheefficiency,effectivenessandproductivityofresearch?

• Hownewformsofevaluationofscienceaffectscientificperformance?

• Howopenscienceimpactsmultidisciplinaryresearch?

4.1.1 CrossanalysisofcasesAccordingtotheassessmentofthecases,in19out24casesthescientificbenefitofthetool,method,approachorcaseisconsideredhighbyascore1.Onlyfivecasesscorea2-PistoiaAlliance,Twousecases,DataCite,RDAandOpenmetadata.Thesecasescoverabroadrangeof research phases so again there is no specific phasewhere open science is critical forscientificimpact.Figure32:Scientificimpactsaslistedinthecrossanalysisofpolicies

Fasterdiscovery

Increased

efficiency

Greater

productivity

Morerobust

findings

Increased

transparency

andreplication

ofstudies

Increased

collaboration

andscientific

interdisciplinari

ty

Minimise

pub lication

biases

Newresearch

fieldsand

enhanced

opportunities

Newemerging

instrumentsto

evaluatescience

GBIF x x x x x Hypothes.is x x x x x Mendeley x x xWhiteRabbit x x x xOSHlicences

37

Fasterdiscovery

Increased

efficiency

Greater

productivity

Morerobust

findings

Increased

transparency

andreplication

ofstudies

Increased

collaboration

andscientific

interdisciplinari

ty

Minimise

publication

biases

Newresearch

fieldsand

enhanced

opportunities

Newemerging

instrumentsto

evaluatescience

Opentargets x x x Zooniverse x x x x Mosquitoalert x x x Pistoiaalliance x x x x xGlobalAlliance Twousescases UtrechtOSP x x x xRDA x xELN Zenodo GOFAIR x x Aspredicted RSE F1000 YODA x x x OpenMetadata x xREANA ORCID x x xDataCite

4.1.2 QualitativeinsightfromcasesTheopendatacasestudieswereallbottom-upinitiativesinthebroaderbiomedicalfieldinwhich PistoiaAlliance andOpenTargetswere industry-driven; andYODAwas led by anacademicorganisation.Therefore,itisnosurprisethatthescientificimpactofthesecasesrelates to faster discovery and increased efficiency. Thismay provide an answer to thesecondkeyquestion.Allcasestudiesshowthattheyaimtobridgepracticesthatarenotnecessarilytakenupin‘normal’scientificpractices. It takestime,effortandfundingtosetup infrastructuresordeveloptoolsthathaveprimarilyabridgingcapacity.Inallcases,industryistakingpartandisbringinginfunding.LargesustainableinfrastructuressuchasCERNprovideenvironmentsinwhichopenscienceinitiativesflourishbenefittingtheoverallscientificcommunity.In the current cases, except from YODA, there seems to be low impact on minimisingpublicationbiases.Thismayhavetodowiththefactthattheyareallbridgingactivitiesthatdonotoperateinthecoreofthescientificpublicationarena,wheremostscienceistakingplace.Openscientifichardwareisanemergingformulabeingexploredbyresearchinfrastructuresthat produce experimental technologies. The example ofWhite Rabbit (WR) providesrelevantinsightsinhowthepublicationofevenafirstversionoftheWhiteRabbitdesigns,

38

resulted in the uptake by the scientific industry, and other scientific infrastructures andfacilities.Atpresent,accordingtolastupdateintheWRrepository,30organisationshaveimplemented WR and 15 others are current evaluating if the technology can fit theirpurposes. In addition, nineR&Dprojects have appliedWhiteRabbit technology for theirscientificpurposes.ExamplesofthisscientificapplicationincludeGSIwhereWRbecamethetimingsystemoftheFacility forAntiprotonandIonResearch(FAIR),an internationalacceleratorbased inGermany of an estimated investment of $2 billion. Also, the Cubic Kilometre NeutrinoTelescope (KM3Net), a European research infrastructure located at the bottom of theMediterraneanSea,usesWRinordertosynchronizethedetectorunits.OpenTargetsisaninnovative,large-scale,public-privatecollaborationonpre-competitiveresearchthatprovidescomprehensiveanduptodatedatafordrugtargetidentificationandprioritisation.OpenTargetsintegratespubliclyavailableinformationanddatarelevanttotargetsanddiseasesintheOpenTargetsPlatform.Itperformshighthroughputexperimentalprojectsthatgeneratetarget-centreddatainphysiologicallyrelevantsystemstounderstandcausal relationships between targets and diseases in three therapeutic areas: Oncology,Immunology, and Neurodegeneration. A cornerstone of this public-private collaborationfrom thebeginning is anagreementamong theorganisations that alldataand resourcesgeneratedwithinOpenTargetsshouldbemadeavailablerapidlyinthepublicdomaintotheentire scientific community. The Open Targets Platform is an open access “Google”-typesearchenginethatextensivelysearchesassessandintegratesthehugequantityofgeneticandbiologicaldataavailable.Theplatformsupportstwomainworkflows:thefirstistarget-centric;thesecondisadisease-centricworkflow.Theexplosionofdataavailabilityinbiomedicalresearchhasincreasinglyrendereditadata-drivenscience.Thereareextensivevolumesofdataassociatedwithdrugdiscoverythatisbeing accumulated in pharmaceutical firms. Drug repositioning is one strategy in drugdevelopmentthatseekstoexpandtheindicationspaceforasuccessfuldrugorfindanewindication foradrugthatwasnotsuccessful in theclinical trials.OpenTargets facilitatesidentificationofpotentialrepositioningopportunities.ThroughtheOpenTargetPlatform,scientists are able to systematically identify and assess all the evidence available thatassociatesadrugwithdiversediseases.Basically,theaggregationandintegrationofalltheheterogeneousandcomplementaryevidenceavailable in theplatformhelpprioritiseandanalysepotentialdrugrepositioning.AsreportedbyKhaldakaretal.(2017),OpenTargetshas been able to uncover 2,540 potential new indications for 791 existing drug targets.Among these 2,540 new indications, 1,366 are for rare diseases where the target isassociatedwithmorecommondiseases.RegardingtheuseoftheOpenTargetsPlatform,900uniqueIPaddressesaccesstheOpenTargetsPlatformeveryweek. Somemetricsavailable aboutOpenTargetplatformusage,fromApril2016-March2017,revealthattheplatformisusedsubstantially.ThemetricsarealsoalignedwiththequalitativefeedbackreportedbytheOpenTargetsteamfromplatformusers.Thescientific impactof theevidenceavailable in theplatformissubstantial.Associationsbetweendrugtargetanddiseasearethemainfocusforbothnewdrugdevelopmentanddrug

39

repositioning.Scientists seekevidence supporting target-diseaseassociations that canbestored in structured databases and integrated to obtain a comprehensive assessment intargetvalidationstudies(Kafkasetal.,2017).ThefundamentalconclusionfromtheOpenTargetscasestudyisthattheinherenttensionbetweenthegoalsofscientificopennessandcommercialexploitationdoesnotnecessarilyimplyincompatibility,butaneedtoidentifysophisticatedsolutionsthatadequatelybalancethedivergentinterestsatdifferentphasesofscientificprocesses.TheYaleUniversityOpenDataAccessProject(YODA)was launchedbytheCenter forOutcomesResearch andEvaluation (CORE) fromYaleUniversity ofMedicine in2011, toaddresstheproblemofunpublished,andselectivelypublished,clinicaltrialresearchdata.YODA has the goal to increase access and availability of clinical research data whilepromotingthere-useofsuchdatatocreatenewknowledge.YODAhasbeenactingasanindependent,unbiasedbridgebetweenresearchersandclinicaldatafrompharmaceuticalcompaniesandconsumerbusinesses.Todoso,YODAhasdevelopedamodeltomakedataavailabletoscientists,whichismediatedthroughtheYODAteam.YODAfacilitatesaccesstoclinical trial data made available by third party Data Holders to promote research thatsupports scientific endeavours and public health. YODA partnered with Medtronic andJohnson&Johnsontoshareitsdata.Metricsonalldatarequests, includingresponsetimesshowsthatcurrently, thereare90clinicaltrialslistedforarequest.DifferenttypesofstakeholdershaverequestedYODAforaccessingclinicaltrialsdata, includinggovernmental institutions, industry,andacademia.Themajority of requests for accessing clinical data trial come from academia (~90%ofrequests),andusuallytherequestsareaccepted.Despite governments and regulatory agencies efforts towards transparency of clinicalresearchdata,agreatdealofclinicaltrialdataispartiallyorcompletelyundisclosed.Thefirstlessonisthatregulatoryenforcementcanplayacriticalroleinincreasingdisclosure.TheframeworkfromYODAcanalsobeappliedforotherfieldsthatexpandmedicalresearch,wheredataisconsideredsensitiveandwherestakeholderssharesignificantconcernsthatpreventthemfromdisclosingrelevantdatatoacceleratescientificdiscovery.Tobere-usedfor scientific goals, clinical research data has to be disclosed with complementary andcomprehensivesourcesofclinicalresearch,whichhasalevelofevidencegranularitythatisexceedsjournalpublications;annotatedcasereportforms;datasetspecifications;protocolswithanyamendmentandreportingandanalysisplan;amongstothers.YODAhascarriedoutactivedisseminationabouttheprojectandpotentialitiesofre-usingclinical data research. Sharing the data is insufficient to accelerate its re-use; activedisseminationandengagementoftheresearchcommunityareneededtofosteranalysisofalreadyexistingdata.Pistoia Alliance is an industry-driven partnership created by large pharmaceuticalcompanies (GlaxoSmithKline (GSK), AstraZeneca, Pfizer, and Novartis) in 2009 withoutgovernmentinterventiontocooperateandshareresourcesinthepre-competitivephaseofdrugdevelopment.TheAlliancehasbeenexpandingsincethenacceptingapplicationsfromboth private and public who share the ideology of sharing data and resources of drugdiscovery research pipeline. Pistoia started to generate mutual standards in industry,

40

ontologiesandwebservicesthataremadeavailableunderanopen-source frameworktoacademicinstitutions,vendorsandcompanies,tofacilitatedatasharing,datarepresentation,text-mining activities and improve the R&D efficiency of the drug discovery anddevelopment process.Withmany projects underway since its foundation, covering fromInternetofThings (IoT) to ‘ontologiesmapping,' thealliancepools resources frommanycompanies,andopenlysharestheoutputofitsworkwiththeoutsideresearchcommunity.ThealliancehasdesigneddifferenttypesofmembershipinordertoengageadiversesetofinterestinenhancinglifescienceR&D.WhenPistoiaAlliancestarteditsactivityin2009,thepartnershipsetouttoworkaroundfour major technology pilots: Sequence services; SESL (Scientifically Enriched ScientificLiterature);ElectronicLabNotebook(ELN);VSI(VocabularyStandardsInitiative).PistoiaAlliancehasaprojectportfolio,whichiscontinuouslyevolvingwithprojectsenteringthepipelineperiodically.Inthecasestudy,oneusecase(project)wasanalysedtoshowtheimpact of the collaboration:Hierarchical Editing for LargeMolecules (HELM), which is aglobalstandardnotationthatismachine-readableforlargemoleculesoriginatedatPistoiaAlliance4 . HELM is now a FAIR standard. The project team includedmembers from 23differentorganisationsinthelifesciencespaceintegratingalargeanddiverseecosystem.TheHELMtechnologywasofficiallyreleasedintotheopensourcecommunityconsistingofaGitHubrepositorycontainingthesourcecodeofthetoolkitandeditor,alongwithawebsite (www.openhelm.org) containing the notation language specification, a free appletversion of the editor, training videos and user guides. HELM notation became an openstandard,publishedopenlyandfor free.HELMis increasinglybeingadoptedthroughthecompanies and academic organisations working on biopharma, including also solutionprovidersandregulatoryagents(PistoiaAlliance,2018).The uptake ofHELMhas been rapid and includes today a vast rangeof organisations ascontentproviders, informaticsvendors, and life science companies.ThebenefitsofusingHELM include facilitating the work of scientists willing to employ computationalmanipulations and calculations with complexmolecules. HELMmakes easier researchertasks.Bydoingthis,HELMfacilitatestheintegrationofprivateandpublicdataeasierusingthestandard.

4.2 IndustrialimpactMovinginnovationsfromscientificdiscoverytothecommercialisationofnewproductsandservicesinvolvesnumerousstakeholdersandchallengestoovercome(e.g.,CarayannisandCampbell,2009).Atoneendofthespectrumthereisaheavyconcentrationofgovernmentinvestment inbasicresearch; andat theother, in thecommercialmarketplace, there is amuchhigherlevelofindustryinvestmentindirectproductdevelopment(NationalScienceFoundation,2017).Betweenbothofthem,thereisthepathforpotentialinnovationbutfirmsand industry players need to invest resources to develop them to a stage where theircommercialpotentialcanbeexploited.Successfullygoingthisprocessrequirescomplextiesand relationships through the innovation spectrum, which usually involves contractualrelationships with non-disclosure-agreements (NDAs), Intellectual Property issues and

4AboutHELM:www.openhelm.org

41

otherformalapproachesforcollaborationthatguaranteethattheinterestsofthedifferentstakeholdersareprotected.Thesociologiesofscientificcommunitiesareoftenuniqueandthepursuingofopennessandfreesharinginthescientificprocesscanbeatoddswiththeinstitutionallogicofthebusinessworld. On the one hand, the norms of Science, which were famously described by thesociologist Robert K. Merton (1973), highlight the cooperative character of inquiry,emphasising that knowledge growth stems from a collaborative process where fulldisclosureof findingsandmethodsare fundamental.Ontheotherside, for-profitseekingfirmsfollowtherationaleofprivateinvestmentmodelsthatseektoprotectR&Dinvestmentstoextractrentsfromtheresultingnewproductsandservicesinthemarket.Forinstance,thetraditionalbusinessmodelformostcommercialhardwarecompaniesisbasedonkeepingthedetailsofthedesignsecrettobeabletomaximisethemargins(Pearce,2017).In the current context, this historical ‘tension’ coming from the sociology of scientificcommunityandbusinessworld,whichhasbeenextensivelystudiedbyscholarsespeciallysince the ’80safter theBayh-DoleAct,needs tobe re-examined in the lightof the recenteventsboostingopenscience,wherepolicy-makersandthescientificcommunityitselfhavestarted placing greater emphasis on the transparency and public scrutiny of scientificprocesses(OECD,2015).Opensciencemandatesacrossthescientificcommunitystresstheefforts by scientists and governments to make the primary outputs of publicly fundedresearchresults(bothdataandpublications)publiclyaccessibleinadigitalformat,withoutany (or minimal) restriction in order to accelerate knowledge growth, while enhancingtransparencyandcollaboration.Theacademicandgreyliteratureassessingtheimpactofapplyinganopenscienceapproachhighlights as positive impacts the increased collaboration and innovation. Theimplementationofanopenapproachinscienceenhancescollaborationacrossinstitutional,nationalanddisciplinaryboundaries,and it fostersthesharingof informationandexpertknowledge(OECD,2015;NESTA,2010).Byfacilitatingaccesstoresearchdataandoutputs,opensciencefostersknowledgespilloversandinnovationacrossdifferenteconomicsectors.It facilitates collaboration among research organisations and businesses, enhances firms’absorptivecapabilities(CohenandLevinthal,1990;Fabrizio,2009;SorensonandFleming,2004)andmakesforaswifterpathfromresearchtoinnovationbycuttingdelaysinthere-useofscientificresults,includingdatasets(OECD,2015;NESTA,2010).Also,theincreaseaccess to results by improving public access to research outcomes and data has beenhighlightedaspositive impactofopenscience towardsbusiness,which fostersspilloversfrom science and research (OECD, 2015). As an example, the use of data fromPubMedCentral, theUSNational InstitutesofHealth's repository, shows that 25%of theunique daily users are fromuniversities,while 17%are from companies and 40% fromindividual citizens (UNESCO, 2012). A recent study on R&D-intensive SMEs in Denmarkshowsthat48%ofthefirmsconsiderresearchresultsessentialtotheirbusinesses,andover65%reportbarriersinaccessingresearchoutcomes(Houghtonetal.,2011).Ontheotherhand,otherscholarssuggestthattheforcesofsharingscientificknowledgeandprotection of commercial interests run in opposition. The commercialisation of scientificoutputsusuallyrequiresasignificantinvestmentthatcompaniesarewillingtobeariftheycanprotecttheinnovationfromimitationorunfaircompetition.Somescholarshighlightthe

42

potentialrisksforfirmsengaginginopenknowledgesystems,withrisksarisingfromtheopposed institutionalrationalesof theworldsofscienceandtechnology(Jong&Slavova,2014). For instance, Perkman & Schildt (2015) show that participation in Open Datapartnershipswithuniversitiesmayjeopardiseacompany’sattemptstocapturevaluefromresearch.

4.2.1 CrossanalysisofcasesThediversecasestudiescarriedoutintheframeworkoftheOpenScienceMonitorprovideanunderstandingoftheeffectsofopensciencetrendsoninnovation,exploresthedriversofsucheffectsandinparticularassessunderwhichconditionsopenscientificpracticesarecompatiblewiththemarketlogic.Wediscussthethreeoftheminthefollowingsections.First,regardingthemajoreffectsthattheanalysisofthethirtycasesstudiesreflectarethefollowing.Wedescribe theeffectsbyemploying theexamplesof the cases that score thehighestintermsofindustryimpact(i.e.WhiteRabbit;ComparativeanalysisofOpensourcehardwarelicences;OpenTargets;PistoiaAlliance;Re-useofpublicdata;Openmetadata;andGOFAIR).However,allcaseshaveinformedthepresentassessment,thusweinvitereaderstoreadthemforfurtherinformation.1. Re-useofscientificoutputsbyfirmsinR&DprocessesOpeningupscientificdata(openscientificdata),thedesignsofscientificexperimentaltoolsthatlaterfindapplicationsindifferentindustrysettings(opensourcehardware),codethatcanbere-usedindataanalyticsprocesses(opencode)andmakingavailabletheprocessesthroughwhichscientistsreachconclusions(opennotebooks),amongstothers,showinthecasespositiveknowledgespilloverstowardsbusinesses.Forexample,inopenscientificdata,theOpenTargetscasereflectedthatnotonlypartnerswithinopentargetsconsortiumbenefittedofthedatapublishedinopentargetsplatformforbetteridentifyingandvalidatingtargets,butalsoexternalstakeholders(seeTwouse-casesofRe-useofopendata)takeadvantagefromtheopendata.OpenTargetsreceive1000visitsaverageperweek;butalso,weseeanindustryemergingofintermediateplayerssupportingpharmaceutical, biotech and companies in biomedical research to extract value byintegratingopenscientificdataintheirworkflowsandinternalinformationsystems.PistoiaAlliancecasealsoreflectedthebenefitsfromopenscientificdataandtheneedforindustrytoimplementFAIR-ificationofdataandadoptdatastandardssothatdatainteroperabilityacrosssystemsisnotabarriertograspsuchbenefits.Additionally,thecaseofWhiteRabbitshowforopensourcehardware,thepositiveeffectsofopeningupdesignsandschematicsaboutscientifichardware.WhiteRabbit,whichwasfirstconceived as a solution for the synchronisation of CERN’s distributed network, findapplicationslateroninawiderangeofscientificbutalsoindustrysettingssuchasfinancialservices(Frankfurtstockexchange),telecommunicationoperators(e.g.Vodafone)butalsoSmartgrid,air-trafficcontrolorautomationvehicles.Ifthesponsororganization,CERNinsuchcase,wouldhavechosenthealternativesituation,thatistooutsourcethedevelopmentofWhiteRabbittechnologytoavendorandprotectthedesignwithIPinsteadofdevelopingasanopensourcehardware,alltheotherscientificinfrastructuresthatwereinneedandre-usedsuchdesign,wouldhavehadtoincurthesameinvestmentfordevelopingsomething

43

that may be not as good as this one that leverage collective skills and expertise. Thecomparativeanalysisofopensourcehardwarelicencesshedslightinhowbusinessesmaybenefit from open source hardware licences by setting governance rules that allowproprietarydevelopmentsandimprovementstoemerge.2. AccelerationofindustrialR&Dprocesses:EfficiencyandEfficacygainsThegrowingcomplexityinR&Dprocessesisincreasinglyrequiringcollaborativemethodsacross different stakeholders (including competitors) to try to leverage complementaryknowledgeandskillstoaccelerateR&Dprocesses.ThisispreciselywhatthecaseofOpenTargetsreflectedwherecompanieshavebeenabletogainefficiencyintheearlystagesofdrug discovery process by sharing and openly disclosing pre-competitive data andknowledge regarding targets.Theyhavealsobeen able,byemploying suchcollaborativeresources,toidentifybettertargetsandthustoimproveefficacy.Other cases such asGOFAIR seeks precisely to accelerateR&Dprocesses bymaking thescientificdataavailabletobere-usedbydifferentplayersstartingbythemselves.CasessuchasPistoiaAllianceorGA4Healthshowedustheimportanceofsettingupandwidelyagreeingonstandardsinordertomakepossiblesuchre-useofdataandknowledgewithinandacrossorganizations,whichturnsinefficienciesinR&Dprocessesforpharmaandbiotechindustry.3. NewproductsandservicesemergingaroundopenscienceOpen science can also lead to new products and services commercialised by a set ofcompanies.TheexampleoftwousecasesofcompaniesemployingpublicscientificdatatobuildsemanticplatformsandanalyticsthattheyselltopharmaceuticalindustryandotherplayersinR&Dbiomedicalresearch;consultanciesthatselltheirservicestootherplayersintheecosystemtoworkaroundplatformsofpublicdatasuchasopentargetswhichhelptointegratepublicdataintheirworkflows.Beyonddata,thecaseofWhiteRabbitalsoreflectedhowcompanieswereabletore-useandsellWhiteRabbitswitchesandnodesandtheirservicestodifferentorganizationsinmultipleindustrialsettings.CurrentlyfourvendorsofferWhiteRabbittechnology,accordingtotheinformationintheopensourcehardwarerepository.Beyondthecasesselected,therearedifferent examples that showhow open science is generating an industrywith differentproductsandservicesaroundpublicresourcestoacceleratere-useand integratewhat isavailableininternalprocesses.4. IncreasequalityofR&DprocessesThecasessuchasYODAorREANAwhohavereproducibilityas theirraisond’êtremovedR&Dprocessestowardsincreasequalityandoverallreliability.REANAwhostartedasCERNprojectearly findanapplication inthe lifesciencesdomainandYODAprovidesgoodusecasesabouthowthecompleteandfullavailabilityofclinical trialshelpedpharmaceuticalcompanies to scientifically crowd source a better understanding of their results. OpenTargetsalsodescribeshowcollaborativeefforts inpre-competitive research improve thequalityoftheearlystagesofdrugdiscoveryprocess;andWhiteRabbit,clearlyreflectedhowtheopensourceapproachallowedtopooltogetherdifferenttypesofexpertiseandtocrowdsourcetheidentificationofbugsinthetechnologywhichleadtobetterqualityofWRoverall.

44

5. NewcooperativestrategiesinindustryR&DprocessesThe cases in themonitor describe emerging cooperative strategies and new governancemodels that are being built around open scientific outputs. Some examples include: thecreativegovernancemodelsputinplaceinYODAtogoverntheaccessandcontroltowardsdatabetweencompanieswhoaredataholdersandthescientificcommunity;opentargetsalsosetupa setof governanceboardsand rules toallowboth thedisclosureof relevantinformation to cooperate in a pre-competitive space while allowing protecting relevantassetsfromcompetition;theevolutionofthelicencesinWhiteRabbitwhichgovernedthecollaborationacrosscontributorswasalso insightful inseeinghowCERNsetupa licencewhichgothroughasetofdebatesaroundthecommunitywhofinallyagreeuponsomebasicrulesofthegame.Second, going through the thematic analysis carried out across the thirty cases and thecodingexerciseimplementedbytheOpenScienceMonitorresearchteam,weuncoverthedriversofthecasesthatscorethehighestintermsofindustryimpact.Notsurprisinglywesee that the caseswith high industry impact also score the highest in terms of supplierapproach.Usercentricityapproachappearsrelevant,whichislogicwhenthinkingaboutthere-useofdataandknowledge,whichmeanstakingseriouslyusers’workflowssothatthebenefitsofpublicdisclosurearegrasped.Finally,allthosecasesreceivedthesupportfrominstitutions,communityandfundersFigure33:Topfivedriversfoundincasestudieswithhighindustryimpact

4.2.2 QualitativeinsightsWhile companies increasingly seek to cooperate with other organisations outside firmboundaries to tap into ideas for new products and services (e.g. Chesbrough, 2003),mechanisms that motivate innovators to reveal the processes and the outcome of theirinvestmentsopenlyarenotevident.Theresultsofthecasestudiesarecoherentwithwhatpreviousliteraturedescribed.Opensciencebenefits innovationbyenablingaswifterpath fromscience tonewproductsandservicesinthemarket.However,asthecasestudiesshow,thiscanonlyhappenifdifferent

45

mechanismsareputinplaceinordertomakecompatiblebothgoalsofwidelysharingthescientific outputs while letting companies protect their private investments in the R&Dprocessthattheyarebeingpartof.Thosemechanismsvarydependingontheopensciencetrend:OpensourcehardwareTheexampleofWRprovidesrelevantinsightsonhowitispossibletogoonestepfurtherfromthemodelofopensourcesoftwaretoother,morecapital-intensivetechnologiesthatrequire different incentives, dynamics, and conditions, combining open revelation andtechnologycommercialisation.Theopensourceapproachpossessesan“essentialtension”about appropriating the returns from an innovation versus gaining adoption of thatinnovation(West,2003).FromtheWhiteRabbitcase,welearnedthattheinitiativewouldnoteven“takeoff”unlesstherightincentiveswereinplacesothatsuppliersandotherstakeholderswouldcontributetheirknowledgetoanopensourceschemewherebothR&Dprocessesandoutcomeswereopentoeverybody.WRidentifiedthefollowingmechanismsthatneededtobeputinplaceinordertostimulateprivateR&Dinanopensourcehardwarescheme,thatbenefitedavastnumberofscientificinfrastructures (higher than 30 research organisations usingWhiteRabbit), but also thetechnologyfoundlateracommercialapplicationinthemarket(e.g.inthefinancialsector,telecommunications,smartgrid,IoT,airtrafficcontroletc.):Fundamentalmechanismsidentifiedinthecaseinclude:

• New legal framework: CERN Knowledge Transfer Group along with its engineersprovidedanewlegalframework,theOpenHardwareLicence,inspiredbythe“TucsonAmateurPacketRadio”licence(TAPR5),tofosterknowledgesharingamongthediverseorganisations(itsfirstversionwasreleasedonMarch2011).

• Compromisebetweenproprietaryandopenhardware:Thelicencemadecompatible,afteraspecialrequestbymultiplefirmsandagreementoftheorganisationsbeingpartoftheWRdevelopment,thatcontributorswerenotobligedtodiscloseimprovementstothetechnology.UndertheOpenHardwarelicence,organisationswerenotobligedtonotifythechangestoupstreamlicensors. Thus,whilecontributingto theopensourcecoretechnologiesofWR,firmswereabletodevelopinparalleltheirproprietaryhardwarebasedonWRtechnology.

• Standardisation:ItwascrucialforstimulatingbothR&DrevelationbycompaniestakingpartinWRdevelopmenttoprovidesomestabilityaroundthetechnology.Goingthroughastandardisationprocesswasenvisagedtoadditionallyincreasethestability,viability,andcredibilityofthetechnologybysolicitingthefeedbackofexpertstofine-tunethetechnologyfurther.FirmswerewillingtoputprivateinvestmentandhelpdevelopingWR if they could anticipate some stability over the technology, meaning that thetechnicalspecificationswouldnotchangeovertime.ThisiswhyCERN,incollaboration

5 Description of TAPR in Open Hardware Repository: https://www.oshwa.org/research/brief-history-of-open-source-hardware-organizations-and-definitions/

46

withotherresearchinfrastructuresandcompanies(partofWRdevelopment),decidedtostartastandardisationprocessofWRtechnologies.

• Arbitrageandorchestrationofexchangesbyanacademicpartner:CERNnotonlyhadtoengagenewcompaniescapableofprovidingWRtechnologyproactivelybutalsoithadtogoverntheexchanges,beinganarbiterofdifferenceswhenneeded(Serrano,2016).ThedifferentorganisationsinvolvedinWRdevelopmentacceptedthearbitrageofCERNinthedifferentexchangesanditsleadership.

• High engagement and close interactionswithacademic partners in order to providelearning and reputational gainsby companies developing the hardware.Firmswereenvisaging an upgrade of their technical skills by closely collaborating with highlyqualifiedengineersatCERN.Additionally, organisationsengaged in thedevelopmentexpectedtobedifferentiatedfromfuturemarketcompetitorsbybeingconsideredatier-1partnerofCERN.Thebrandingandreputationofsuchcollaborationwereexpectedtosignalthequalityoftheorganisationwhileextendingthecompanies’capacitytoattractclients.

OpenscientificdataOpen Targets, Yoda and Pistoia Alliance cases showed that open research data benefitsindustryandaccelerates innovationprocessesifdifferentmechanismsareput inplace inordertoprovidetherightincentivesandconditionsforcompaniestobothsharedata,andto re-use third party data. From the three cases the following mainmechanisms wereidentified:• Agreement among industry players ofwhat data and resources are considered pre-

competitiveandwhatiscompetitive.Pre-competitivedataissusceptibletobeingshared,whilecompetitiveinformationandknowledgearegoingtobeheldsecretsimultaneously.As an example, the collaboration within Open Targets required a clear delineationbetweenwhatwasagreedtobesharedamongstorganisationsandwhatremainedclosedfortheparticipants.The‘discontinuity’inthelinegoingfromcooperationtocompetitionneeds to be demarcated to generate a comfortable environment for corporations tocollaborate and share data to accelerate the early stages of drug discovery whilepreservingtheirassetstocompeteamongsteachotherinlaterstagesofthedevelopmentprocess.FirmsinOpenTargetsagreedtocollaborateanddisclosedataandknowledgerelatedtophases1(targetidentification)andphase2(targetvalidation)tosucceedinsystematicallyfindthebesttargetstosafelyandeffectivelytreatdisease.However,firmsdonotrevealtheknowledgeandproprietarydatathattheyaregoingtousetoidentifythemultiplemoleculesactiveagainstthepotentialtargets,norotherdatausefulinlaterstagesofthedrugdevelopmentprocess(i.e.phase3-leadoptimisation-andonwards).

• Thefundamentalneedfordatastandardstoeffectivelyshareresearchdata.AsGOFAIR,PistoiaAlliance,GA4Health,RDAandothercasesshow,withoutcommondatastandardswidelyadoptedbyacademicandindustryorganisations,thebenefitsofopenresearchdatacannotbefullygraspedbyfirms.Inordertoactuallyfacilitatedataexchanges,andthustaponthebenefitsofartificialintelligence,machinelearningandanycomputationalapproach relying upon the aggregation of high volumes of quality data, the industry

47

needstoagreeandwidelyadoptstandardscollectively.PistoiaAlliancecaseshowedthatstandardsarefundamentalinbiomedicalresearch(e.g.UnifiedDataModel)butarealsoresourceintensivebecausetheyrequirehoursofdistributedteamsworkingtogethertoagree on common tech specifications and solutions. Once the standard is out, if itsuccessfully implemented by a wide range of players, it can dramatically reduceinefficiencies and accelerate information sharing within and outside organisationalboundaries.

• Arbitragebyanacademicpartner.Forinstance,inthecaseofYoda,theinitiativeledby

the Centre for Outcomes Research and Evaluation (CORE) from Yale University ofMedicine, developed amodel tomake data available to scientists, which ismediatedthroughYODAteam.YODAfacilitatesaccesstoclinicaltrialdatamadeavailablebythirdpartyDataHolderstopromoteresearchthatsupportsscientificendeavoursandpublichealth.YODAdoesnot supportnor facilitateaccess todata for commercialpurposes,litigation, or any goal that is not purely scientific. Themodelworks as follows: DataholdersgiveaccesstotheirclinicalresearchdatatoYODA.TheYODAteamfollowsastrictprotocoltograntaccesstothesedatatoresearchers.AnotherexamplewasprovidedbyOpen Targets, which showed a model to pool together distributed capabilities andresources from the different organisations and companies through matchmakingexercises to match people from distributed teams working on a joint research idea;merging companies individual investments devoted to early drugdiscovery in a jointresearch agenda; and other additional approaches that helped orchestrate thedistributed resources (data, capabilities and expertise) with a prominent role of theacademicpartnersinthegovernanceofexchanges.

• Highengagementbyacademicpartnerstoincreaselearninggainsandpossibilitiesforresearchdatare-use.BothPistoiaAllianceandOpenTargetscasesdescribedhowface-to-faceexchangesandcloseinteractionsarestillcrucialandneedtobecombinedwithaninteractiveplatformthatfacilitatesdataandresourcesexchanges(e.g.IP3inthecaseofPistoiaAllianceandOpenTargetsPlatform).Technologicalplatformapproachesneedtobecombinedwithclosecooperationamongtheorganisationsthatgeneratethedataandthosewillingtore-usethedata.AsthecaseofOpenTargetsdescribed,sharingscientificdatainthepublicdomainisanecessarybutinsufficient requirement for being able to re-use such data for drug developmentpurposes. Companies taking part of Open Targets consortium agree to privateinvestments–bothcashandin-kind–inordertobeclosetothescientiststhatgeneratethedatatobeabletounderstandthedata,howitwasgenerated,andhowtointerpretit.

• Time lagand ‘premium’ features indataplatforms forcompaniessharingtheresearchdata.ThecaseofOpenTargetsdescribesamodelwherecompaniesgeneratingthedataagree to publicly share all scientific data produced in the framework ofopen targetscollaboration.However,whilepartners in theconsortiumhaveaccess to theresearchdatafromminuteone,thedataisopentothepublicthroughtheopentarget'splatformafterpublication.ThistimedifferencegivescompetitiveadvantagetowardsfirmsinsideOpen Targets consortium. On the other side, the functionality of the Open Targets

48

platformbythepublicandpartnersdiffer.Forinstance,partnerswithintheconsortiumcanaggregateprivatedataintheplatform(e.g.,compoundlibraries)toacceleratefurtherstagesofdrugdiscovery.

GeneralpatternsacrosstrendsTheanalysisoftheresultsofthecasestudiesimplementedagreeswiththeliteraturethatclaimsthatopensciencebenefitsinnovationbyenablingaswifterpathfromsciencetonewproductsandservicesinthemarket.However,thecasestudiesalsoshow,thatthiseffectcanonly happen if different mechanisms are put in place. Table 6 below summarises thosemechanismsalsodescribedabovebasicallyforthetwoopensciencetrendsthatreflectedhigherindustryimpact:openscientificdataandopensciencehardware.Thosetwotrendswhichconcentratedmostofthecasesinthisrespect.Table6:Analyticalsummaryofmechanismsenablingthepositiveimpactofopensciencetoindustry

Openscientifichardware OpenScientificData Governance Newlegalframework:See

comparativeanalysisfordifferentgovernancemodelsinOSH.Thereisaneedforlicencesthatenablethenon-disclosureofimprovementsorthecompromisebetweenproprietaryandopenhardware.

Thereisaneedforaprioragreementbycompaniesandacademicpartnersaboutwhatisgoingtobedisclosed(pre-competitive)andwhatfallsunderthecompetitivearena:whatdata/resourcesarenotsubjecttoIP;whataresubjecttoIP.

Standards Standardisationprocesstoincreasethestability,viability,andcredibilityofthetechnologydevelopedopenlytoallowperipheraldevelopmentstoemergeandthusnewproprietarytobecommercializedaroundtheopentechnology.

Datastandardsforindustryandacademicpartnerstoenabledatasharingandre-usebutalsointegrationwithininformationsystemsinsideorganizations.

Arbitragebynon-industryfirm

Arbitrageandorchestrationofexchangesbytheacademicpartner,asatrustedpartytogoverntheexchangesamongststakeholdersandguarantee,insomecases,re-useforthatdonotdamagecompetitiveadvantage.

Arbitrage by the academicpartnerbetweendataholdersanddata(re)usersputting inplace protocols that providetrust to commercialpartnersandthird-parties.

Learninggainsthroughinteraction

Highengagementandcloseinteractionsbytheacademicpartnerforcompaniestoextractlearninggainsfromtheinteraction.

Highengagementandcloseinteractionswiththeacademicpartnersforlearninggainsandtoincreasepossibilitiesofresearchdatare-use.

49

Openscientifichardware OpenScientificData Competitiveadvantagetowardscontributors

Favouringindustrialcontributorstowardsfree-ridersthroughbrandingandreputationalgains.

Timelagandpremiumservicesforcontributors.

Finally,thestudyteaminvitesreaderstogothroughthedifferentcasestudiesinordertogetmore insights about the different mechanisms, boundary conditions and contextualcharacteristicsthatmadethecasestudiesbeselectedinthefirstplaceasinsightfulexamplesofsuccessfulopenscienceimplementations.

4.3 SocietalimpactOpensciencecertainlyhasavisibleimpactonsocietyasanindirecteffectofthescientificandindustrialimpactsanalysedabove.Enhanced,fasterscientificdiscovery–suchasthoseidentifiedthroughOpenTargets–benefitallpartsofsociety.Earlydetectionofsideeffects–providedbyYODA–lowersthecostsofdevelopmentandacceleratesthedetectionofsideeffects.Secondly, industrial innovation is a determinant of growth and employment. Thedevelopmentofnewproducts,suchasWhiteRabbit,hascreatedanewmarketopportunity,andincreasedtheproductivityofthecompaniesadoptingtheseinnovations–fromfinancetotelecom.Besidethesewell-knownbenefitstowardssocietythroughscientificandindustrialimpact,thereareother relevantaspects tobe considered.Firstly, openscience cancontribute togreaterpublicunderstandingofscience.Opensciencehasthepotentialtoenhancepublicengagementandtheunderstandingofscienceprinciplesandpracticebyraisingawareness,pro-activeparticipationandcitizens’directcontributiontoresearch(Boultonetal.,2012;Kowalczyk & Shankar, 2010). The openness of the scientific process fosters new actors’access to the research process (Nowotny, 2001). The increasing importance of citizenscienceisaforcefordisruptivechange,engagingamateursandthegeneralpublicinscience.Thoseinclusiveandparticipatoryapproachestoboosthumancapacityandcapabilityfromprofessionals, amateurs, volunteers and citizens to assist in collecting, curating andpreservingthegrowingscientificrecord(Lyon,2009).Citizen science is also a method for formal and informal science education and publicunderstanding of science, which should not be underestimated (Christian et al., n/a.).Greatercitizenengagementcanleadtogreaterparticipationinscientificexperimentsanddatacollection(OECD,2015;Boultonetal.,2012;Whyte&Pryor,2010).Makingresearchdatapubliclyavailablecanadvancepeople’sunderstandingofscienceandcitizenscienceinitiatives (Kowalczyk & Shankar, 2010). TheGlobalMosquito Alert Project has broughttogetherkeycitizensciencemonitoringinitiativestoworktogethertohelpunderstandthegeographicspreadofmosquitoswhoarecapableofcarryingdisease,inturnhelpinghealthprofessionals and citizens to manage any potential risks. 6 Zooniverse, a world-leading 6JonSwitters,DavidOsimo,CitizenScienceintheSurveillanceandMonitoringofMosquito-BorneDiseases,OpenScienceMonitorCaseStudy(Brussels:EuropeanCommission,2019)

50

platform foronline crowdsourced research,hasmore than100activeprojects (in2019)wherevolunteersparticipateincrowdsourcedscientificresearch,classifyinglargedatasetsacross multiple knowledge domains (such astronomy, ecology, cell biology, humanities,climatescience,etc.,withsignificantoverlapbetweenareas).TheZooniverseresearchbringsforwardnewdiscoveries,datasetsusefultothewiderresearchcommunity,andpublications.Hypothes.isdevelopedatooltoenableanyonetoannotateanywhere,andopenedupandstandardisedtheannotation’stechnologiesandpractices.Itaimedtomakeiteasyforusersto leave annotations across the entireweb. And, the open-source nature of the platformallowsanyone(developers,publishers,academicinstitutions,researchers,andindividuals)who use it to create their own annotation reader orwriter. The nature of the tool alsofacilitates the collectionof annotations ina structuredway, allowing tobuild theoverallpictureoftheinformationknowledge.On theotherhand, increasedresearchdata sharing canconstitutea challenge to societalvalues, mainly by increasing the risk for infringement of data protection. There is afundamental tension between the scientific value of data built through correlation ofdisparatedatasetsvariablesandtheneedforpreservinganonymity.Severalstudieshaveshownthatpersonaldatasecuritycannotbesafeguardedbyanonymisation.Companiesthatrelease data openly for research purposes have encountered problems related to the re-identifiability of subjects. Datasets that contain information on individuals supportinferencesregardingtheprobabilityofotherinformationaboutpeople(Denning&Schlorer,1980; Lyon, 2009). Conversely, the recent emphasis around the implementation of theGeneral Data Protection Regulation is forcing scientists and research institutes to adoptmore stringent data management practices that can be consistent with increased datasharing:forinstance,bylimitingstorageonowncomputersandrequiringscientiststousededicatedlaw-compliantrepositories(Stark,2019).Thiscouldturnouttoforcetheadoptionofappropriatedatamanagementstrategies(rathertheinformalonesillustratedinsection2.2)thatcouldultimatelyleadtowiderreusabilityofdata.

4.3.1 CrossanalysisofcasesWhenperformingacross-analysisofthecasestudies,fewofthemshowhighimpactonsociety(seeFigure1).Figure34:Impactonsocietyoftheanalysedcasestudies

51

Accordingtotheassessmentofthecases,thereareonlyfourcasestudies(ofthetotalof24analysed)withhighimpactonsociety:OpenTargets,Yoda,GlobalBiodiversityInformationFacility(GBIF)andZooniverse.Forother10cases,theyshowamediumlevelofimpactonsociety:WhiteRabbit,PistoiaAlliance,REANA,OSHlicences,Aspredicted,Twousecases,Global Alliance, GOFAIR, Hypothes.is and Mosquito alert. The rest of 10 remaining casestudiesshowaratherlowimpactonsociety.From research lifecycle side, the caseswith high andmedium impact are found in datageneration(OpenTargets,Mosquitoalert),dataintegration(OpenTargets,YODA,Twousecases),datapublication(YODA),replication(YODA,REANA),discovery(GlobalBiodiversityInformation Facility, Hypothes.is), prototyping and developing tools (White rabbit, OpenHardwarelicencesandHypothes.is),datadescriptionstorageandmanagement(GOFAIR),pre-registration(Aspredicted)anddatastandards(PistoiaAllianceandGlobalAlliance).Figure35:Casestudieswithhighandmediumimpactonsociety,byresearchphases

Itisimportanttomentionthatthecasestudieswithhighandmediumimpactonsocietyarenotrestrictedtoonlyoneopensciencecategory(opendata,openaccess,opencollaborationorcitizenscience).Fouroutofthe14casestudieswithhighandmediumimpactonsocietyareclassifiedinmorethanoneopensciencecategory(seeTable8).Table7:Casestudieswithhighandmediumimpactonsociety,byopensciencecategory

OpensciencecategoryClassifiedinone

areaClassifiedinmorethanonearea*

Opencollaboration 2(14.3%) 5(27.8%)Opendata 7(50.0%) 9(50.0%)Openaccess - 1(5.6%)Citizenscience 1(7.1%) 3(16.7%)Totalcasestudies(considered) 14 18Note:*Thecasestudieswithhighandmediumimpactonsocietyclassifiedinmorethanoneopensciencecategoryarecountedmultipletimes.

52

Opendataismostfrequentcategorythecasestudiesareassociatedto,regardlessofsingleormultiplecoverage–halfofthecasestudiesincludetheopendataprinciple.Itisfollowedbyopencollaborationandcitizenscience.Whenitcomestoopenaccess,onlyonecasestudyincludesit,togetherwithopencollaboration(Aspredicted).

53

5 PolicyanalysisThissectionprovidestheintegratedpolicyanalysisofthestudy.ThefirstpartpresentstheoverviewprovidedbythedatacontainedintheSherpadatabase,aboutfundersandjournalspolicies on open access and open data (Sherpa-Juliet and Sherpa-Romeo database). Thesecondpartpresentstheindepthqualitativecrossanalysisofcasestudies.

5.1 StatisticaloverviewThestatisticaloverviewisdividedinopenaccesstopublicationsandopendata.Itincludesdataonfundersandjournalpolicies.

5.1.1 OpenaccesspoliciesOne of themain factors affecting open access to publication is themandates from theresearchfunders.Datashowsthatopenaccesspublicationisenforcedbyresearchfunderstoamuchlesserextentthanopenaccessarchiving-outof145institutionsidentified,33%hasnoopenaccesspolicyinplace,whereas35%encouragesitandonly31%requiresit.Figure36:Funder’spoliciesonopenacesspublishingbycountry

Concerningnumberofresearch fundersregardingmandateonopenaccessarchivingthevastmajority (71%) requiresopenaccessarchivingand further12%encourages it, only17%ofinstitutions(allfromUK)donothaveanypolicyregardingopenaccessarchiving.

54

Figure37:Funders’policiesonopenaccessarchiving

Policiesbyresearchjournalsarealsoimportantinsettingtheboundariesforresearchers’choices.Thevastmajoritysupportssomesortofarchiving(82%):42%supportsarchivingofpre-printandpost-print,33%canarchivepost-printandfurther7%canarchivepre-print.Onlyin18%ofjournalsarchivingisnotformallysupported.US research journals stand for the majority of those journals where archiving is notsupported,followedbyUnitedKingdomandPortugal.Thesearealsothecountrieswiththelargest amount of the research journals.However, from all top five countries (India andGermanyastop4andtop5)onlyinPortugalmoreresearchjournalssupportpost-printonlyratherthanpost-printandpre-printtogether,thelatterbeingandoverallpredominanttrendinmostofthecountries.

55

Figure38:Archivingpolicesbytypeofmandate

5.1.2 OpendatapoliciesYet,concerningpoliciesofresearchfunders,morethanahalf (56.6%)donothaveanyopendatapolicyinrelationstodataarchivingandonly27.6%requiresit.

56

Figure39:Opendatapoliciesbytypeofmandate

Asinthecaseofopenaccess,thesestatisticsarelargelyinfluencedbyUnitedKingdom,acountrywiththehighestnumberofresearchfundersmappedintheSherpa-Julietdatabase.Almost76%outof64researchfundinginstitutionsintheUKdoesnothaveopendatapolicy.Furthermore,inmanyothercountries,researchfundinginstitutionseitherdonothaveopendatapolicy(e.g.Spain,China,Luxembourg)ortherearemoreinstitutionsthatdonothavean open data policy than those that encourages it or requires it (e.g. France, Belgium,Denmark,Italy,Sweden).Figure40:Opendatapoliciesbycountry

57

Regardingjournalsanditsdatasharingmodality,accordingtothelatestavailabledata,only21%ofjournalsrequiresdatasharingand32%donotaddressesdatasharingatall.Thisindicatorisaone-offanalysiscarriedoutin2017.Figure41:Journalsbydatasharingmodality

5.1.3 OtherpoliciesThereislittledataonjournals’policiesconcerningopencode.Datafrom2013showthatjournals´opencodepolicyisalmostnon-existent.Outof170journalsanalysed79%ofthemdidnotaddresscodesharing.Infact,codessharingwasexplicitlyrequiredonlyincaseof12journals(seeFigurebelow).Figure42:Journalsbycodesharingmodality

58

5.2 Indepthcross-analysisofpolicycasesThiscross-analysispresentsthesystematicinvestigationofthenationalimplementationofopensciencepoliciesintheNetherlands(NL)andFinland(FI),andopenresearchpoliciesintheUnitedKingdom(UK).Firstly,thepurposeofthisreportistofacilitatediscussionsontheimplicationsand thepolicy recommendationsmade in theEuropeanopensciencepolicycontext.Secondly,thisstudysynthesisesthenationalpoliciesconductedbytheOpenScienceMonitor(OSM)thusfar,byanalysingthemainbottlenecksandthepolicymeasuresgatheredacrossthethreecountries.Finally,theresultofthisevaluationisthefirststepinabroaderanalytical task of monitoring open science development in Europe –what open sciencestrategy(atEuropeanlevel)isneeded,why,andtowhichextent?Fromthere,thisstudywillconcludewithpotentialinsightsthatcanguidefutureanalysisonopenscience.

5.2.1 ObjectivesAcrossthenationalpolicycasestudiesintheNL,FIandUK,twokeydriversareguidingthenarrativeofthepolicies:(1)themoralimperativethatpubliclyfundedresearchgeneratesknowledge that belongs to everyone, thusmust be shared to and accessed by the publicimmediately, and (2) the economic gains of easily andwidely accessible research,whichgenerates productivity and competitiveness in their respective national research andinnovationsystems.Sinceallthreecountriesarekeentobeknownastheleadersinopenscienceandresearch,Table8presentsacomparableoverviewoftheirscientific,societalandindustrialdriverswithineachofthenationalpolicycontext.

Table8:Overviewofscientific,societalandindustrialdriversacrossNLandFIOpenScienceandUKOpenResearchPolicies

Driver NL FI UK Science Openscienceimproves

thetransparency,verifiability,efficiency,reproducibility,andsustainabilityofresearchprocesses

Opennesswillpromotetransparency,maketheresearchprocessmoreeffectiveandallowresearcherstodealwithcomplex,multidisciplinaryquestions

Openresearchimprovesefficiencyintheresearchprocess,transparency,accountability,andpublicengagementwithresearch

Industry Openscienceincreasesinnovativecapacityasindividualsbenefitmorereadilyfrompubliclyavailableinformationanduseitincombinationwiththeirownknowledgeandexperiencesto

Openscienceleadstosurprisingdiscoveriesandcreativeinsights-fosteringtheresearchsysteminFinlandtowardsbettercompetitivenessandhighertransparencyandinnovation

Openresearchcreatescloserlinkagesbetweenresearchandinnovation,benefitsproductsandservices,encouragesthecommercialisationofresearchand

59

Driver NL FI UK developnovelproductsandprocesses

stimulateseconomicgrowth

Society Everyoneoutsidethescientificcommunitymaybenefitfromopensciencebecausetheycanreadilyaccessandusescientificinformation

Thecirculationandpromotionofresearchdataoutsideacademiacontributestothegreatersocietalimpactbyincreasingscientificliteracyamongcitizens.Transparencyincreasesthecredibilityofscienceforthecitizens

Openresearchpromotespublicunderstandingofscience,evidence-basedpractices,andcitizen-scienceinitiatives

Source:OSMcasestudies

Similardriverswere identified inall threecases,althoughtheopensciencepolicieswereintroducedindifferentyearsforeachofthecountries.FortheNetherlands,itwas2016asledbytheDutchMinistryofEducation,CultureandScience,andforFinlanditwas2014,asinitiated by the FinnishMinistry of Education and Culture.Meanwhile in the UK, it wastermed open research and declared by the Minister of State for Universities, Science,ResearchandInnovationin2013.Themoralandeconomicdriversforopenscienceandopenresearchinthesethreecountrieswereexpressedclearly,whichhardlyleaveanyroomforobjection.

5.2.2 ProvisionsPublicpolicyplaysanimportantroleininfluencingresearcherbehaviour.Inrecentyears,severalinstitutionaladjustmentstothegeneralgovernanceframeworkofsciencehavetakenplace:

• Theintroductionofcompetitivefundingmechanisms,asopposedtobulkfundingintheEuropean higher education sector. Publications output and impact factors becamecommon performance indicators for quality-related research in higher educationinstitutions. This has increased competition and reinforced the importance ofassessmentmetricsinthecontextoffundingdistribution.

• Theincreasedpolicyemphasisoninnovation,toboostthecompetitivenessofEuropeanindustry. Industrial impact, characterised by collaboration with private companies,generationofpatentandintellectualpropertyrights(IPR),wereimplementedintothebroader European Commission policies, to better convert scientific research into

60

improvedgoods,servicesandprocessesforthemarketforthewelfareandwellbeingofsociety.

These twosciencepolicy trends certainly createa tensionwith the specificopensciencepolicygoalsofopennessandcollaboration, increasinglyadoptedby fundingagenciesandnationalinstitutions.Table9presentstheobservedtensionthroughtheevidencegatheredfromthenationalopensciencepoliciesofNLandFI,aswellasopenresearchpoliciesintheUK in three key features: open access, open data, and research evaluation and rewardsystems.Table9:ProvisionsaddressedintheNLandFIOpenScienceandUKOpenResearchPolicies

Feature NL FI UK Openaccess

Hybridjournalswillnolongerbepaidfor.ChallengetoturnGreenOAroutetoanacceptedmeansofpublishing.

Alackofcommonproceduresonopenpublishing.LowpublicationinOAJournalsandthecostsforOApublishingremainsexpensiveformostoftheevaluatedpublishers.

70%ofOApublishingisgoingtohybridjournals.TheaverageAPCforhybridpaymentbyUKuniversitieswereincreasedby19%,between2013and2016.

Opendata Alluniversitieshadtheirownonlinerepository,whichislinkedtoanationalportal–NARCIS.Althoughthereisaslightincreaseofopendatasetsovertheyears(exceptfor2011and2013),mostdatasetsremainedclosedorrestricted.TheNationalCoordinationPointResearchDataManagement(LCRDM)willconductastudytoestablishthequalitativeandquantitativeneedfordatastewardsandresearchsoftwareengineers,thecontentoftheirrole,thetrainingrequired,andtherewardingoftheirwork.

Stronglydata-drivenwithafocusonthedevelopmentofresearchdataarchitecture:managementofownership,metadata,andIPR.Theinitiativewasplanneduntil2017,withnofurtherstrategyorplandevelopedforthefollowingyears.

Patchyopendataprovisions:researchcouncilsareeither(1)silentonthematter,(2)providedirectsupportfortheworkofdatacentresandservices,or(3)relyonuniversityorthird-partyrepositoriestopreserveandprovideaccesstoresearchdata.

Researchevaluationandrewardsystems

Negativeevidenceforopensciencerewardinperformanceagreementsbetweentheministryanduniversities,andtenuretrackpolicybetweenuniversitiesandindividualresearchers.

Theevaluationframeworkforopennesswasdevelopedtoincentiviseresearchorganisationswithrespecttotheirpoliciesonandimplementationofopensciencepractices.

OApublications,resourcesandinfrastructurethatsupportopenresearchareconsideredinthequalityofresearchoutputsandresearchenvironmentofthe

61

Feature NL FI UK ResearchExcellenceFramework.

Source:OSMcasestudies

Finally,theGeneralDataProtectionRegulation(GDPR)wasenforcedbytheEUonthe25thofMay2018.WhiletheGDPRdoesstrengthendataprotectionandprivacyofEUpersons,italsocomplicatestheproprietaryaspectofdataasinstitutionsandindividualsarestilltryingtounderstandhowchangesinprivacylawswillimpacttheirwork.GDPRthuscreatesfurthertensionbetweenopendataandprivacyandsecurity,asillustratedinsection4.3.

5.2.3 LessonslearntTheUKisleadinginthenumberoftotalOApublications,followedbyNLandFI.GiventhattheUKhasthehighestnumberofresearchfundermandatesconcerningOApublication,nottomentionthattheResearchExcellenceFramework(REF)andresearchfundingdependsonit, it is unsurprising that the UK is leading in Green OA publications. Contrary to thepreferenceforGoldOAasannouncedbytheUKScienceMinister,DavidWilletsin2013,theGold route is still far frombeing a desirable and acceptablemeans of publishing forUKresearchers.IntheNetherlands,astrongfocusinGreenOAwasobservedingeneral,sincealluniversitieshadtheirownonlinerepository,whichmakesiteasierforresearcherstoself-archivetheirpublications.The current policies in NL, FI and UK demonstrate converging policies, with minordifferencesinthedriversonopenscience.Thefocusofallthreecountriesaremainlyonopenaccess publishing from the top-down, where funders and governing bodies are leadingprogressinopenscienceinsteadofbottom-up,researchcommunitiesledinitiatives.It isworthnotingthat the fundingbodiesofall threecountries in thepresentreport,aremembers of ‘cOAlition S’, an international consortium of research funders to accelerateimmediateOpenAccesspublishing.Clearly,theirinvolvementinPlanSdemonstratedthecountry’scommitmentinrealisingthemoralimperativeandeconomicbenefitsasoutlinedinthe‘Drivers’section.TheUnitedKingdomisseenasthestrongestinopenscienceandisinfact,dominatedbyimpact-orientedpractices thataresanctionedand imposedbyresearch funders.Thishasresultedinadivergenceinopenaccessambitionsandactualpublicationpractices,withariseinpublicationsinhybridjournals,andanimmediatedecreaseingoldOApublications.Thecostofopenscienceopeningcontinuestotakecentrestageintheopensciencepolicyarena,whilethekeyissuesofopendata,researchevaluationandrewardsystemarehardlyaddressedbypolicymakersinallthreecountries.

62

6 PolicyconclusionsThe largeresearcheffort leadstomanypossiblepolicyreflections, indifferentareas.Forsimplicity,wedividetheminthoseconcerningopensciencepolicy,andthoseconcerningfuturemonitoringefforts.

1.1. Conclusions on the future of open science 1. Datashowsclearlythatopenscienceistheretostay.

All trends clearly indicate progress: from open access to publications (considering theimpactofembargoperiodonstatistics)toopendata(withlimitedprogress)toopenAPI,openhardwareandcitizenscience.Everyindicatorandeverytrendshowprogress,evenifslow.

2. Openscienceisawide-rangingsetoftrends:itismorethanopenaccesstopublicationandopenresearchdata.

Thisperhapsobviousstatementneedstoberepeated.Inthepolicydebate,itappearsthatopenaccessattractsalltheattentionandthe“expenditure”intermsofpolicycapital.Afterthat,thetopicofopenresearchdatainpolicytermsisemergingstrongly,ledmostlybythepolicydebateaboutthedataeconomy.Yetalltheothertrendsaregrowingstronglydespitethelackofpolicyattention.AtatimewhenMicrosoftacquiresGithub,thelargestrepositoryofopensourcesoftware,forabout7billioneuros,fewjournals,fundersandregulatorshaveyet introduced policies concerning open scientific software, and the developments aremostlybottomup.

3. Theprogressdetectedbytheindicatorsdoesnotjustifyanycomplacency.Data shows a slowprogress. For instance, data sharing among researchers show limitedprogressinthetwoyearsmeasured.Progressonopenaccesstopublicationsinunequal,withadeclineingreenopenaccessthatwecurrentlyattributetoembargoes–butthatneedstobemonitoredinthefuture.Overall,itisclearthatopenscienceisnotthedefaultoption–butratherchosenbyaminorityofscientistsbothin termsofopenaccessandopenresearchdata.Inotherwords,opensciencerequirescontinuouspolicyattention.

4. The policy attention needs to be focussed in particular on getting the incentivesystemsright.

Open science requires changes by all stakeholders involved and can’t be achieved onlythroughregulation.ItisclearthatweseeaproliferationofpolicyinitiativesatnationalandEuropean level,butwhenwelookatbehaviouralchangesbyscientists the importanceoffunders’policies is limited– it ismuchmore important toact through the communityofpeers. Ultimately, open science requires a substantial effort of governance, a continuoustiltingoftheplayingfieldtoensureoptimalresults.Moreconcretely,initiativessuchastheEuropeanOpenScienceCloudrequireacontinuouseffort ingovernanceevenAFTERthelaunch.Thelaunchisthebeginning,nottheendofthepolicyprocess.Allthesuccessful,largescaleopenscienceinitiativesdescribedinthecasestudiesshowthattheonlywaytoscaleup is through continuous adaption and experimentation of new modalities to involveddifferentstakeholders.Thereasonforthisisclear:

63

5. Open science does not appear as an ideological radical choice, but as a rationaldecisionbasedontheinherentadvantages.

Mostlarge-scaleprojectshavebalancedclosedandopenmethods,dependingonthespecificchallengesandopportunities.Somemethodsworkunderspecificconditionsanddependingonthestakeholders’interests.Thereisnoonesizefitsallapproachtoopenscience.Itisclearthatbothscientistsandcompaniesareusingdifferentdegreesofopennessdependingonthecircumstances. Researchers share data mostly within their research group or projectspartners, rather than openly. Datasets in repositories have different levels of openness.Research centres develop new licences to accommodate for the needs of differentstakeholders.

6. Scientificprogressistheultimategoal–andtheimpactisvisible.Whatisclearfromthecasesanalysedisthattheachievementofscientificprogressisthemain priority, and the openness varies in order to maximize that goal. Equally in thecorporatesector,gettingnewproductsfastertomarketisaprioritywithrespecttoopennessinitself.Inotherwords,whenopensciencescales,itisameansnotagoal.Someofthehighestprofilescientificandcommercialendeavourhaveincorporatedelementsofopenscience.Andit’snotonlybasicsciencewhereopenpracticesareconsolidated.Opensciencepracticesarenowusedincommercialendeavours,suchasthedevelopmentofhighprecisiontimemeasurementorthediscoveryofnewpharmaceuticalproducts.Despitetheattentiononopenaccesstopublications,itisinopenresearchdataandopencollaborationthatweseegreaterindustrialimpact.Openpracticesandcommercialgoalscangohandinhand,butonlyunderspecificconditions.Thereisatrade-off,notanincompatibility.

7. Opensciencepolicyshouldlookbeyondopenscience.Whatisclearisthatopenscienceispartofawidertrendtowardsdatadrivenandlarge-scalecollaboration, for instance through the adoption of APIs, the diffusion of data portabilitypractices,servitisationandmore.Policiessuchasthedataeconomy,thedigitalagendaandindustrialpolicyshouldbealignedwiththeopenscienceagenda.Ultimately,opensciencepolicyshouldnotbedrivenbyideologybutbyconcreteneeds.Onlyinthiswaycantheparticipationofallrelevantstakeholdersbeensured.Toachievethis,thecommunitybuildingworkthattheEChasimplementedsofarisanecessaryfoundation,butitisonlythebeginning.Opensciencepracticesrequiresystemicchangethattakesseveralyearstotakeplace.Andgoodmetricsareafundamentalpartofthiseffort.

1.2. Conclusions on the future of the Open Science Monitor 1. The existence of an Open Science Monitor is necessary to successful policy

implementationThetransitiontoopenscienceisacomplex,multi-stakeholderorganictransition,notatopdown decision-making process. In such a context, monitoring is even more important,becauseithelpsallstakeholdersbuildacommonassessmentofthesituationandtoaligneveryone’sefforts.Obviously,amonitordoesnotpersegenerateconsensusandcommonaction,butitsabsencemakesachievingthosegoalsimpossible.Nationalcasestudiessuchas

64

theNetherlandsshowedthattheabsenceofmonitoringmechanismsforspecificpolicyareassuchasopendatawasamajorbarriertoprogress.Monitoringopenscienceisnotasimplestraightforwardtask:itrequiresclearmethodologicalchoicesandrobustmethods.Assuch,itcannotrelypurelyonthirdpartymetrics,whichoftendifferalotintermsofcoverageanddefinitions, including between member states. Moreover, the sensitivity of the issue, asshown by the conflict about the existing Open Science Monitor, requires strong centralknowledgepresidiumtojustifyanddefendmethodologicaldecisions.Putsimply,EuropeanopensciencepolicyrequiresaEuropeanopensciencemonitor.

2. The next Open Science Monitor should reduce the robustness gap between openaccessandothertrends

While openscience is recognized as amultifaceted trends, some trends aremuch bettercoveredthanotherswhenitcomestoindicators.Therearemajordifferencesbetweentheavailability and robustness of data on open access to publications, open data and opencollaboration.Dataaboutopenaccesstopublicationareeasilyavailable,althoughthereareconstant discussions about the modalities of measurement. Indicators about open datasharinghavebeenprovidedthroughadhocsurvey,inviewofitsrelevance.Butfortheothertrendsweareatlossofreliableindicatorandwecanonlyuseproxies,mainlyrelatedtotheadoptionofexistingservicesorexistingcollectionsthatarenecessarilypartial.Theothertrendsarethe“Cinderella”ofopenscienceintermsofdataavailability.Forthefuture,weneedthereforetoincreasetheattentiontomeasuringreliablythedifferenttrends.Thiscouldbe done in the first instance by extending the survey fromopen research data to otherimportanttrends,startingfromopenscientificsoftware;secondly,throughadhocsamplebasedbibliometricanalysistoassesstheavailabilityofdataandcodeassociatedtoarticles.

3. MultiplesourcesandmethodsshouldcontinuetobeusedIndicatorsproducedbythemonitorshouldrelyonthewidestsetofsourcesandmethods.The only criterion for choice should remain reliability and validity of indicators. A goodexampleishowthepresentMonitorextendedthesourcesforopenaccessmetricstoincludeUnpaywall alongside Scopus. This integration of different sources allowed a more fine-grained,timelyandextendedcoverage.Inadditiontodataintegration,datatriangulationisalsonecessarytoreinforcethelegitimacyofthefindings.ThepresentMonitorcomparedtheresultswithothersources,suchasWebofScienceforopenaccessandthe“Stateofopendata”surveybyFigshare,recognizingexplicitlythesimilaritiesandcontradictions.Inotherwords,thefutureOpenScienceMonitorshouldbeasbroadaspossibleandprovideacriticalmeta-analysisofothersurveysandstudies.

4. ThekeyfocusshouldbeadoptionOpenscienceismaturing,andsoshoulddotheindicators.Supplysideindicatorssuchasthemereavailabilityoftools,thenumberofdatarepositories,ortheexistenceofpoliciesshouldbesecondarytotheotherpriority:measuringtheadoptionofopensciencepracticesbyallstakeholders.Adoptionshouldincludenotonlythenumberorpercentageofusers,buttheintensityorfrequencyofadoption.Inotherwords,theindicatorsshouldnotonlymeasurethepercentageofscientiststhatpublishinopenaccess,sharedataandcode,butwhether

65

they do it systematically on all papers or occasionally. When measuring adoption,bibliometricindicatorsappearmoreeffectivethansurveys,insofartheyallowforobjectivesmeasurementofbehaviourratherthanself-reported.

5. Bibliometricsfirst,surveysecondThereislittledoubtthatmeasuringadoptionisbetterthroughobservationratherthanself-reporting.Askingresearcherswhethertheyadoptopensciencepracticesismorepronetobiasedresults thangatheringbibliometricdataaboutwhethertheyeffectivepublishdataandcodealongsideanarticle.Indeed,itismoredifficulttomeasuredataandcodesharing,thanitistomeasurepublicationinopenaccess,butitisstilldoable.Forinstance,samplebased bibliometric have been used by many individual studies – while open access topublications is measured on the universe of publications. Finally, survey still have animportantroletoplay,inparticularwhenitcomestoassessingdriversandbarriers.Instead,adoptionmetricsfromspecificservicesareoflimitedvalue:indicatorssuchasthenumberofcitizenscienceoropenhardwareprojectscanprovideasuggestionofgrowthinusage,butdonotallowforassessingadoptiongrowthintheoverallscientificcommunityasawhole.Finally,casestudiesremainuniqueinprovidinginsightaboutabetterunderstandingofhowopenscienceworksandtheimpactitachieves.Theyareveryimportantinthisearlyandfaststageofevolution.Insummary,wesuggestmovingtowardsgreaterusageofbibliometricindicatorsforassessingadoption,andreducingmetricsprovidedbyspecificservices,whilededicatingsurveysandcasestudiesspecificallytodriversandimpacts,asshowninthetablebelow.Table10:Highlevelmethodologicalapproach,past(x)andfuture(greyshade)

Drivers Open access Open data Open collaboration

Impact

Bibliometric (universe)

x

Bibliometric (sample)

Survey x x x Service metrics

x x x

Cases x x

6. The next open sciencemonitor should be as open as possible – and as closed asnecessary

TherehavebeensomestrongcriticismsbypartoftheopensciencecommunitytowardsthepresentMonitorbecauseoftheusageofproprietarydata.Thishasledtoaseriousinternalreview of the methodology and significant improvement, including an expansion of theoriginalsources.Themethodologywasthendiscussed inahigh-profileexpertworkshop.Theresultsconfirmedthechoicetousesomeproprietarydata,becausetheyprovidehighquality additional information (namely about affiliation). The final choice should bepragmaticanddrivenbytheobjective,whichistoprovidereliablehigh-qualitydatapointsinsupporttopolicy.Asoftoday,thereisstillroomforproprietarydatainthiscontext.Using

66

onlyopendatawouldleadtolowerqualityoutput.Still,thereareimportantoptionstobeused when using proprietary data, such as allowing researchers to access the data ondemandforreplicationandresearchpurposes.Andthisisexactlywhathasbeendone.Onasimilarnote,theprocessshouldcontinuetoberadicallyopen,butthisdoesnotmeanthat decisions should be taken by the community. There is a need for strong centralleadershipinmakingtheappropriatemethodologicalchoices.Asinthecaseofthepresentmonitor, the open process included allowing everyone to post comments on themethodology (visible to everyone), and provided systematic feedback to each individualcomment about the choices made. The choices are made centrally, but after openconsultation,expertreviewandprovidingclearfeedbackontherationalebehindrefusingoracceptingsuggestions.Inotherwords,boththedataandtheprocessshouldbeasopenaspossibleandas closedasnecessary– butalways transparent.Theopensciencemonitorshouldembracethespiritofopenscience–butaswehaveseenintheanalysis,thisdoesnotmeanexcludingatallcostanyformofclosedprocessandoutput.Justasopenscience,theopensciencemonitorrequirescarefulgovernanceoftrade-offs.

7. Thescopeofthework,andthecommunityaddressed,shouldbescienceasawhole,notopenscience

The open science monitor has to cover the developments across science – not merelytrackingthosewhopracticeopenscience. Itshouldnotprovide indicatorsonlyabout theadoptersbutobservethetransitionintheoverallscientificprocess.Thisiswhyitshouldnolongerprioritizedataprovidedbyopenscienceservices,suchasthenumberofusersofaservices, or the number of projects in an open repository. It should instead focusmoreheavily on measuring the adoption of open science practices by the whole scientificcommunity–suchasthepercentageofarticlesorresearchersthatincludetheadoptionofopensciencetrends.Equally,theusersandthestakeholdersoftheMonitorarenottheopensciencecommunity,butthescientificcommunityasawhole.Thisisasdifficultasitisimportant,becausetheopensciencecommunityisveryactiveanddedicated–andvocalonsocialmedia.Ultimately,thelegitimacyoftheMonitorcomesfromtheattentionandimpactintheoverallscientificcommunity,not from theapprovalofopenscienceadvocates.TheOpenScienceMonitorshoulddesignitsservicesaroundtheneedsofthescientificcommunity–andproactivelyreach out through articles, presentations and communication as a priority, rather thanengagemainlywiththevocalopensciencecommunity.

67

7 ReferencesAndreoli-Versbach,P.andMueller-Langer,F.(2014)Openaccesstodata:Anidealprofessed

butnotpractised.ResearchPolicy,43(9),pp.1621-1633.Archambault,É.,Amyot,D.,Deschamps,P.,Nicol,A.,Provencher,F.,Rebout,L.,andRoberge,

G. Proportion of Open Access Papers Published in Peer-Reviewed Journals at theEuropeanandWorldLevels—1996–2013,ReporttotheEuropeanCommission,April2014

Arnold, E. (2005). What the Evaluation Record tells us about Framework ProgrammePerformance.Management,(January).

Auer, S., Bizer, C., Kobilarov,G., Lehmann, J., Cyganiak, R. and Ives, Z. (2007)Dbpedia: Anucleus for a web of open data. The semantic web (pp. 722-735). Springer BerlinHeidelberg.

Baker,M(2016).1,500scientistsliftthelidonreproducibility.Nature,533(7604):452–454.DOI:https://doi.org/10.1038/533452a

Bannister,F.(2007).Thecurseofthebenchmark:anassessmentofthevalidityandvalueofe-government comparisons. International Review of Administrative Sciences, 73(2),171–188.http://doi.org/10.1177/0020852307077959

Berghmans,S.,Cousijn,H.,Deakin,G.,Meijer,I.,Mulligan,A.,Plume,A.,…Waltman,L.(2017).Opendata :Theresearcherperspective,48p.http://doi.org/10.17632/bwrnfb4bvh.1

Blumenthal, D., E.G. Campbell,M. Gokhale, R.Yucel,B. Clarridge, S.Hilgartner, andN. A.Holtzman(2006).DataWithholdinginGeneticsandtheOtherLifeSciences:PrevalencesandPredictors.AcademicMedicine,81(2),137-145.

Boulton,G.,Campbell,P.,Collins,B.,Elias,P.,Hall,W.,Laurie,G., ...andWalport,M.(2012)Scienceasanopenenterprise.TheRoyalSociety.

Bücheler, T. and Sieg, J. H. (2011). Understanding science 2.0: crowdsourcing and openinnovationinthescientificmethod.ProcediaComputerScience,7,327-329.

Burgelman,J.C.,Osimo,D.andBogdanowicz,M.(2010)Science2.0(changewillhappen….).FirstMonday,15(7).

Burgelman,J.-C.,Osimo,D.,&Bogdanowicz,M.(2010).Science2.0(changewillhappen…).FirstMonday,15(7).

Campbell,E.G.,Clarridge,B.R.,Gokhale,M.,Birenbaum,L.,Hilgartner,S.,Holtzman,N.A.andBlumenthal,D.(2002)Datawithholdinginacademicgenetics:evidencefromanationalsurvey.Jama,287(4),473-480.

Carlson, S and Anderson, B 2007 What Are Data? The Many Kinds of Data and TheirImplications for Data Re-Use. Journal of Computer-Mediated Communication, 12(2):635–651.DOI:https://doi.org/10.1111/j.1083-6101.2007.00342.x

Carrara,W., Nieuwenhuis,M., & Vollers, H. (2016). Open DataMaturity in Europe 2016.Brussels.

Chokshi, D. A., Parker, M. and Kwiatkowski, D. P. (2006). Data sharing and intellectualproperty in a genomic epidemiology network: policies for large-scale researchcollaboration.BulletinoftheWorldHealthOrganization,84(5),382-387.

CODATA-ICSTITaskGrouponDataCitationStandardsPractices2013OutofCite,OutofMind:TheCurrentStateofPractice,Policy,andTechnologyfortheCitationofData.DataScienceJournal,12:CIDCR1–CIDCR75.DOI:https://doi.org/10.2481/dsj.OSOM13-043

Collison,P.,&Nielsen,M.(2018,November16).ScienceIsGettingLessBangforItsBuck.

68

Retrieved fromhttps://www.theatlantic.com/science/archive/2018/11/diminishing-returns-science/575665/

Costas,R.,Zahedi,Z.,Wouters,P.(2015a).Do“altmetrics”correlatewithcitations?Extensivecomparisonofaltmetricindicatorswithcitationsfromamultidisciplinaryperspective.JournaloftheAssociationforInformationScienceandTechnology,66(10),2003–2019.https://doi.org/10.1002/asi

Costas, R., Zahedi, Z., Wouters, P. (2015b). The thematic orientation of publicationsmentionedonsocialmedia:large-scaledisciplinarycomparisonofsocialmediametricswith citatios. Aslib Journal of Information Management, 67(3), 260–288.https://doi.org/10.1108/AJIM-12-2014-0173

Crouzier,T.(2015).ScienceEcosystem2.0:howwillchangeoccur.Actabiomaterialia,20,51-59.

Dasgupta,P.andDavid,P.A.(1994)Towardaneweconomicsofscience.ResearchPolicy,23,487–521.

David, P. A. (2004) Can "open science" be protected from the evolving regime of IPRprotections?.JournalofInstitutionalandTheoreticalEconomics(JITE)/ZeitschriftfürdiegesamteStaatswissenschaft,9-34.

David,P.A.(2004)Patronage,Reputation,andCommonAgencyContractingintheScientificRevolution:FromKeeping’Nature’sSecrets’totheInstitutionalizationof’OpenScience’.SIEPRPolicypaperNo.03-039.

Denning,D.E.andSchlörer,J.(1980).Afastprocedureforfindingatrackerinastatisticaldatabase.ACMTransactionsonDatabaseSystems(TODS),5(1),88-102.

Dosi,G.,Llerena,P.andLabini,M.S.(2006)Therelationshipsbetweenscience,technologiesandtheirindustrialexploitation:Anillustrationthroughthemythsandrealitiesoftheso-called‘EuropeanParadox’.Researchpolicy,35(10),1450-1464.

Drummond,C2009ReplicabilityisnotReproducibility:NorisitGoodScience.In:ICML’09Proceedingsof the26thAnnual InternationalConferenceonMachineLearning (Thefourth workshop on evaluation methods for machine learning). Montreal, Quebec,Canada.Retrievedfrom:http://cogprints.org/7691/.

Eisenhardt, K. M. (1989). Building Theories from Case Study Research. Academy ofManagementReview,14(4),532–550.http://doi.org/10.5465/AMR.1989.4308385

Eisenhardt, K. M., & Graebner, M. E. (2014). THEORY BUILDING FROM CASES :OPPORTUNITIESANDCHALLENGESdiverse.AcademyofManagementJournal,50(1),25–32.http://doi.org/Article

EuropeanCommission(2009)PreparingEuropeforanewRenaissance:AstrategicviewoftheEuropeanresearcharea.FirstreportoftheEuropeanResearchAreaBoard—2009.Brussels:EuropeanCommission.

EuropeanCommission(2010)Europe2020:AStrategyforSmart,SustainableandInclusiveGrowth:CommunicationfromtheCommission.Brussels:EuropeanCommission.

EuropeanCommission(2012)Recommendationonaccesstoandpreservationofscientificinformation.Brussels:EuropeanCommission

European Commission (2014) Innovation Union Scoreboards. Brussels: EuropeanCommissionDGEnterprise

European Commission Expert Group on FAIR Data (November, 2018). Turning Fair intoReality-FinalReportandActionPlanfromtheEuropeanCommissionExpertGrouponFAIRData.EuropeanCommissionDirectorateGeneralforResearchandInnovation.

69

EuropeanCommissionHighLevelExpertGrouponScientificData2010Ridingthewave:HowEuropecangainfromtherisingtideofscientificdata(FinalreportoftheHighLevelExpertGrouponScientificData.AsubmissiontotheEuropeanCommission).EuropeanUnion.Retrievedfrom:https://www.fosteropenscience.eu/content/riding-wave-how-europe-can-gain-rising-tide-scientific-data.

Fang,F.C.,&Casadevall,A. (2015).CompetitiveScience: IsCompetitionRuiningScience?InfectionandImmunity,83(4),1229–1233.http://doi.org/10.1128/IAI.02939-14

Fecher,B.,Friesike,S.andHebing,M.(2015)Whatdrivesacademicdatasharing?PLoSOne,10(2),e0118053.

Feller,J.andFitzgerald,B.(2002).Understandingopensourcesoftwaredevelopment(pp.143-159).London:Addison-Wesley.

Franceschet,M.andCostantini,A.(2010)Theeffectofscholarcollaborationonimpactandqualityofacademicpapers.Journalofinformetrics,4(4),540-553.

Friesike, S., Widenmayer, B., Gassmann, O. and Schildhauer, T. (2015) Opening science:towardsanagendaofopenscienceinacademiaandindustry.TheJournalofTechnologyTransfer,40(4),pp.581-601.

Furman, J. L., Porter,M. E. and Stern, S. (2002)The determinants of national innovativecapacity.Researchpolicy,31(6),899-933.

Gittelman, M. and Kogut, B. (2003) Does good science lead to valuable knowledge?Biotechnology firms and the evolutionary logic of citation patterns. ManagementScience49,366–382.

Goodman,AA,Pepe,A,Blocker,AW,Borgman,CL,Cranmer,K,Crosas,M,etal.2014TenSimpleRulesfortheCareandFeedingofScientificData.PLoSComputationalBiology,10(4):e1003542.DOI:https://doi.org/10.1371/journal.pcbi.1003542

Guston, D. H. (2000) Retiring the social contract for science. Issues in Science andTechnology,16(4),32-36

Haeussler, C. (2011) Information-sharing in academia and the industry: A comparativestudy.ResearchPolicy,40(1),pp.105-122.

Haeussler,C.,Jiang,L.,Thursby,J.andThursby,M.(2014)Specificandgeneralinformationsharingamongcompetingacademicresearchers.Researchpolicy,43(3),pp.465-475.

Haustein,S.,Costas,R.,Larivière,V.(2015).Characterizingsocialmediametricsofscholarlypapers: the effect of document properties and collaboration patterns. PloS ONE.https://doi.org/10.1371/journal.pone.0120495

Herman,E.,Jamali,H.R.,Nicholas,D.,Osimo,D.,Porcu,F.andPujol,L.(2015).AnalysisofEmergingReputationandFundingMechanismsintheContextofOpenScience2.0(No.JRC94952).InstituteforProspectiveTechnologicalStudies,JointResearchCentre.

Hey,A.J.andTrefethen,A.E.(2003).Thedatadeluge:Ane-scienceperspectiveHoughton,J.,Swan,A.andBrown,S.(2011)AccesstoResearchandTechnicalInformationin

Denmark:ReporttoTheDanishAgencyforScience,TechnologyandInnovation(FI)andDenmark'sElectronicResearchLibrary(DEFF.DanishAgencyforScience,TechnologyandInnovation.

Jong, S. and Slavova, K. (2014) When publications lead to products: The open scienceconundruminnewproductdevelopment.ResearchPolicy,43(4),pp.645-654.

Jonkers, K., & Zacharewicz, T. (2016). Research Performance Based Funding Systems: aComparativeAssessment.EuropeanCommission.http://doi.org/10.2791/659483

Jubb, M., Plume, A., Oeben, S., Brammer, L., Johnson, R., Bütün, C., & Pinfield, S. (2017).

70

Monitoringthetransitiontoopenaccess.Jubb, M., Plume, A., Oeben, S., Brammer, L., Johnson, R., Bütün, C., & Pinfield, S. (2017).

Monitoringthetransitiontoopenaccess.Kowalczyk, S. and Shankar, K. (2011) Data sharing in the sciences. Annual review of

informationscienceandtechnology,45(1),247-294.Lakhani,K.R.andPanetta,J.A.(2007)Theprinciplesofdistributedinnovation.Innovations,

2(3),97-112.Lawson, S (2017).Reportonoffset agreements: evaluating current JiscCollectionsdeals.

Year2–evaluating2016deals.Liu,C.C.andStuart,T.(2014).Positionsandrewards:Theallocationofresourceswithina

science-basedentrepreneurialfirm.ResearchPolicy,43(7),pp.1134-1143.Lundvall,B.A. (1992)National systemsof innovation:Ananalytical framework.London:

Pinter.Lyon, L. (2009). Open Science at Web-Scale : Optimising Participation and Predictive

PotentialConsultativeReport.InformationSystemsJournal,29(November),1–45.Margolis,R,Derr,L,Dunn,M,Huerta,M,Larkin, J, Sheehan, J, et al. (2014).TheNational

Institutes of Health’s Big Data to Knowledge (BD2K) initiative: capitalizing onbiomedicalbigdata. Journalof theAmericanMedical InformaticsAssociation,21(6):957–958.DOI:https://doi.org/10.1136/amiajnl-2014-002974

Maxwell,J.A.(2012).QualitativeResearchDesign:AnInteractiveApproach:AnInteractiveApproach.

Mayer-Schonberger,VandCukier,K(2013)BigData:ARevolutionThatWillTransformHowWeLive,Work,andThink.Boston:HoughtonMifflinHarcourt.

McKiernan,E.C.,Bourne,P.E.,Brown,C.T.,Buck,S.,Kenall,A.,Lin,J.,McDougall,D.,Nosek,B.A.,Ram,K.,Soderberg,C.K.andSpies,J.R.(2016)Howopensciencehelpsresearcherssucceed.Elife,5,p.e16800.

Murray,F.,andS.Stern(2007)."DoFormalIntellectualPropertyRightsHindertheFreeFlowofScientificKnowledge?AnEmpiricalTestoftheAnti-CommonsHypothesis,"JournalofEconomicBehavior&Organization,63(4),648–687.

Nagel,E.,&Kaplan,A.(1965).TheConductofInquiry:MethodologyforBehavioralScience.AmericanSociologicalReview(Vol.30).http://doi.org/10.2307/2091353

NationalInstitutesofHealth(2015)CommonsHomePage|DataScienceatNIH.Retrievedfrom:https://datascience.nih.gov/commons

National Institutes of Health (2016) NIH Sharing Policies and RelatedGuidance on NIH-Funded Research Resources. Retrieved from:https://grants.nih.gov/policy/sharing.htm

NationalScienceFoundation(2011)AwardandAdministrationGuide:DataSharingPolicy.Retrieved from:https://www.nsf.gov/pubs/policydocs/pappguide/nsf11001/aag_6.jsp#VID4

Nielsen,M. (2012). Reinventing discovery: the new era of networked science. PrincetonUniversityPress.

OECD(2015)MakingOpenScienceaReality.OECDScience,TechnologyandIndustryPolicyPapers,No.25,OECDPublishing,Paris.

Olensky, M., Schmidt, M., & Van Eck, N.J. (2016). Evaluation of the Citation MatchingAlgorithms of CWTS and iFQ in Comparison to the Web of Science. Journal of theAssociation for Information Science and Technology, 67(10), 2550-2564.

71

doi:10.1002/asi.23590.Olson,G.M.,Zimmerman,A.andBos,N.(2008)ScientificcollaborationontheInternet.The

MITPress.OpenKnowledge Foundation (2015)OpenDefinition:DefiningOpen inOpenData, Open

ContentandOpenKnowledge.Retrievedfrom:http://opendefinition.org/od/.Orelli, B (2016) The Business of Genomic Data. O’Reilly Media. Retrieved from:

http://www.oreilly.com/data/free/the-business-of-genomic-data.csp.Organisation for Economic Co-operation and Development (2007) OECD Principles and

Guidelines forAccess toResearchData fromPublic Funding. Paris: Organisation forEconomic Co-Operation and Development, p. 24. Retrieved from:http://www.oecd.org/dataoecd/9/61/38500813.pdf.

Palmer,CL,Weber,NMandCragin,MH(2011)TheAnalyticPotentialofScientificData:Understand¬ing Re-use Value. Proceedings of the American Society for InformationScienceandTechnology,48(1):

Pasquetto, I V, Sands, A E, Darch, P T and Borgman, C L (2016) Open Data in ScientificSettings:FromPolicytoPractice.In:Proceedingsofthe2016CHIConferenceonHumanFactorsinComputingSystems.Presentedatthe2016CHIConferenceonHumanFactorsin Computing Systems, New York, NY, USA: ACM, pp. 1585–1596. DOI:https://doi.org/10.1145/2858036.2858543

Pasquetto,I.,Randles,B.,&Borgman,C.(2017).OntheReuseofScientificData.DataScienceJournal,16.

Perkmann, M. and Schildt, H., (2015). Open data partnerships between firms anduniversities:Theroleofboundaryorganizations.ResearchPolicy,44(5),pp.1133-1143.

Pilat,D.andFukasaku,Y.(2007).OECDprinciplesandguidelinesforaccesstoresearchdatafrompublicfunding.DataScienceJournal,6,OD4-OD11.

Piwowar,H.A.,Vision,T.J.andWhitlock,M.C.(2011).Dataarchivingisagoodinvestment.Nature,473(7347),285-285.

Porter,M.E.(1990)Thecompetitiveadvantageofnations.HarvardBusinessReview,68(2),73-93.

Powell,A. (2012).Democratizingproductionthroughopensourceknowledge: fromopensoftwaretoopenhardware.Media,Culture&Society,34(6),691-708.

Powell,A.B.(2015).Opencultureandinnovation:integratingknowledgeacrossboundaries.Media,Culture&Society,

Procter, R.,Williams, R., Stewart, J., Poschen,M., Snee, H., Voss, A. and Asgari-Targhi,M.(2010)AdoptionanduseofWeb2.0 inscholarlycommunications.PhilosTransactAMathPhysEngSci,368(1926),4039–4056.

Raymond,E.(1999)Thecathedralandthebazaar.Knowledge,Technology&Policy,12(3),23-49.

Robinson-García, N., Torres-Salinas, D., Zahedi, Z., Costas, R. (2014). New data, newpossibilities:exploringtheinsidesofAltmetric.com.ElProfesionaldeLaInformacion,23(4),359–366.https://doi.org/10.3145/epi.2014.jul.03

Savage,C.J.andVickers,A.J.(2009)EmpiricalstudyofdatasharingbyauthorspublishinginPLoSjournals.PLoSONE,4(9),9–11.

Science,Digital;Hahnel,Mark;Fane,Briony;Treadway,Jon;Baynes,Grace;Wilkinson,Ross;et al. (2018): The State of Open Data Report 2018. figshare. Paper.https://figshare.com/articles/The_State_of_Open_Data_Report_2018/7195058

72

ShreeveJ(2005)TheGenomeWar:HowCraigVenterTriedtoCapturetheCodeofLifeandSavetheWorld.BallantineBooks:NewYork.

Sorenson,O.andFleming,L.(2004)Scienceandthediffusionofknowledge.Researchpolicy,33(10),pp.1615-1634.

Stodden,V.(2010).TheScientificMethodinPractice:ReproducibilityintheComputationalSciences.

Szkuta,K.andOsimo,D.(2016)Rebootingscience?Implicationsofscience2.0maintrendsforscientificmethodandresearchinstitutions.Foresight,18(3).

Szkuta,K.,&Osimo,D.(2016).Rebootingscience?Implicationsofscience2.0maintrendsfor scientific method and research institutions. Foresight, 18(3).http://doi.org/10.1108/FS-06-2014-0040

Taubenberger, J.K.,Reid,A.H.,Lourens,R.M.,Wang,R., Jin,G. andFanning,T.G. (2005)Characterizationofthe1918influenzaviruspolymerasegenes.Nature,437.

Tenopir,C,Allard,S,Douglass,K,Aydinoglu,AU,Wu,L,Read,E,etal.2011DataSharingbyScientists: Practices and Perceptions. PLoS ONE, 6(6): e21101. DOI:https://doi.org/10.1371/journal.pone.0021101

TheMoRRI indicators report,MonitoringReportD4.3,Pre-publicationversion (February2018): http://www.technopolis-group.com/report/evolution-responsible-research-inovation-europe-morri-indicators-report-d4-3/

UE. (2015). Better Regulation Guidelines. Guidelines European Commission.http://doi.org/10.1016/S0022-3913(12)00047-9

Van Leeuwen, T., Meijer, I., Yegros, A., Costas, R. (2017). Developing indicators on OpenAccessby combiningevidence fromdivesedata sources. STI2017Conference.Paris(France)

Von Hippel, E. (2009) Democratizing innovation: the evolving phenomenon of userinnovation.InternationalJournalofInnovationScience,1(1),29-40.

Walsh,J.P.andHuang,H.(2014)Localcontext,academicentrepreneurshipandopenscience:Publication secrecy and commercial activity among Japanese and US scientists.ResearchPolicy,43(2),pp.245-260.

Waltman,L.,Costas,R.(2014).F1000RecommendationsasaPotentialNewDataSourceforResearch Evaluation: A Comparison With Citations. Journal of the Association forInformationScienceandTechnology,65(3),433–445.https://doi.org/10.1002/asi

Whyte,A.,andPryor,G.(2010)Opentoall?CasestudiesofopennessinResearch.UnitedKingdom:Nesta

Wilkinson,MD,Dumontier,M,Aalbersberg,IjJ,Appleton,G,Axton,M,Baak,A,etal.(2016)TheFAIRGuidingPrinciplesforscientificdatamanagementandstewardship.ScientificData,3:160018.DOI:https://doi.org/10.1038/sdata.2016.18

Wilsdon, J. R., Bar-Ilan, J., Frodeman, R., Lex, E., Peters, I., & Wouters, P. (2017). Next-generationmetrics:responsiblemetricsandevaluationforopenscience.

Wood,J.,Andersson,T.,Bachem,A.,Best,C.,Genova,F.,Lopez,D.R.,Los,W.,Marinucci,M.,Romary,L.,VandeSompel,H.andVigen,J.,(2010)Ridingthewave:HowEuropecangainfromtherisingtideofscientificdata.EuropeanUnion.

Wouters, P., Costas, R. (2012). Users , narcissism and control – tracking the impact ofscholarlypublications in the21stcentury. (M.VanBerchumK.Russell,Eds.), ImageRochester NY. SURFfoundation. Retrieved fromhttp://www.surffoundation.nl/en/publicaties/Pages/Users_narcissism_control.aspx

73

Yin,R.K.(2009).Casestudyresearch :designandmethods.Appliedsocialresearchmethodsseries ;(Vol.5.).http://doi.org/10.1097/FCH.0b013e31822dda9e

Zahedi, Z., Costas, R., Wouters, P. (2014). How well developed are altmetrics? cross-disciplinaryanalysisofthepresenceof“alternativemetrics”inscientificpublications.Scientometrics.https://doi.org/10.1007/s11192-014-1264-0

Zahedi,Z.,Fenner,M.,Costas,R.(2014).Howconsistentarealtmetricsproviders?Studyof1000PLOSONEpublicationsusingthePLOSALM,MendeleyandAltmetric.comAPIS.Inaltmetrics14:expandingimpactsandmetrics(pp.5–6).Bloomintong,Indiana,USA:AnACM Web Science Conference 2014 Altmetrics Workshop.https://doi.org/10.6084/m9.figshare.1041821