Towards common methods for assessing open data:

workshop report & draft framework

May 8th and 9th 2014, New York University.

Executive Summary:

Introduction Towards a common framework: common assessment methods workshop The framework at a glance: Next steps:

Background: What do we mean by assessment? Why assess open data? The current landscape of open data assessment

A common framework: draft proposal 1) Context/environment

Existing sources 2) Data

Existing sources Methodological considerations

3) Use Methodological considerations

4) Impact Methodological challenges

Next steps: developing the framework further: Appendix 1: Workshop Participants Appendix 2: Background reading list

Published 10th June 2014.

Note taking, compilation and editing by:

Robyn Caplan, Tim Davies, Asiya Wadud, Stefaan Verhulst, Jose M. Alonso and Hania Farhan

Full list of workshop participants in Appendix 1.

www.opendataresearch.org | www.webfoundation.org | www.thegovlab.org | 1

Executive Summary:

Introduction Governments across the world are adopting open data policies and practices. From national portals, to municipal open data initiatives, and sector­specific efforts in transport, health and international aid to name just a few, open data has been adopted as an important governance innovation. In line with the growth of open data, a range of different efforts have emerged to measure various aspects of open data readiness, implementation, outcomes and impacts. This paper outlines a common framework, developed through a collaborative workshop, that offers opportunities to join­up these efforts through common and complementary assessment methods for open data activities.

Towards a common framework: common assessment methods workshop On May 8th and 9th 2014 The World Wide Web Foundation and The Governance Lab at NYU convened a two­day workshop in New York, bringing together a group of open data assessment experts to explore the development of common methods and frameworks for the study of open data . The 1

meeting examined opportunities for greater collaboration and coordination between projects and organisations working to understand the implementation and impacts of open data efforts in different settings around the world. The workshop was held over two days. The first part of the workshop consisted of reviewing existing research and projects measuring open data, determining use cases for open data measurement and identifying key questions concerning open data, before looking at conceptual frameworks that could connect questions to use cases. The second part of the workshop focussed on identifying common categories and indicators within an overarching framework. Meeting participants split into sub­groups to consider specific questions, and to build out the larger framework with which we can study open data.

1 Participants attended from the Web Foundation, the Open Data Institute, the GovLab, the International Data Research Centre, the Open Knowledge Foundation, the Ford Foundation, The Knight Foundation, UNDESA and OECD.

www.opendataresearch.org | www.webfoundation.org | www.thegovlab.org | 2

This paper builds on the outcomes of the workshop to present an initial framework for discussion. The outcomes do not necessarily represent the views of all the organisations that participated, and are offered in this draft form for open comment.

The framework at a glance: In assessing open data activities, projects may look at:

1. Context/Environment ­ The context within which open data is being provided. This might be the national context in the case of central Open Government Data, or might be the context in a particular sector. Important aspects of the environment to assess include the legal and regulatory environment; organisational context; political will & leadership; technical capacity; the wider social environment, in terms of civil society and political freedoms; and the commercial environment and capacity of firms to engage with open data.

2. Data ­ the nature and qualities of open datasets. Including the legal, technical, practical and social openness of data, and issues of data relevance and quality. The framework also looks to identify core categories of data which might be evaluated in assessments.

3. Use ­ the context of use of the open dataset. Includes the category of users accessing (or providing) the dataset, the purposes for which the data will be used, and the activities being undertaken. This part of the framework addresses the who, what and why of open data programmes.

4. Impact ­ the benefits to be gained from using the open dataset. Potential benefits can be studied according to social, environmental, political/governance, and economic/commercial dimensions.

Next steps: A summary draft of the framework can be found here. Over June and July we will be inviting feedback on this framework, and through the project mailing list inviting others to contribute to 2

the further development of the framework. The framework may give rise to particular methodological tools, harmonisation of research questions, creation of shared guidance for researchers, and other collaborative projects to use it as a means of deepening research and understanding of open data.

2 https://groups.google.com/forum/#!forum/common­assessment­methods­for­open­data

www.opendataresearch.org | www.webfoundation.org | www.thegovlab.org | 3

Background:

What do we mean by assessment? There are many different kinds of assessment. Assessing open data may involve:

Technical assessment of datasets; Assessment and ranking of Open Government Data (OGD) initiatives; Providing quantitative metrics of open data outcomes and impacts; Providing qualitative judgements on performance of an open data initiative; Developing qualitative case studies about open data use and impacts;

There are often overlaps between the questions and measurement methods employed in these different kinds of assessments, and in the areas of focus. A common framework helps different measurement and assessment projects to locate their overlaps, and to move towards shared definitions of key terms, concepts and categories.

Why assess open data? The appropriateness of any framework and assessment methods is ultimately determined by the use to which they will be put. During our workshop we identified a range of different motivations for measuring and assessing aspects of open data activities. Policy makers, activists and researchers may variously be interested in measurement in order to:

Benchmark and compare the use of open data between different countries;

Compare the use of open data in specific sectors within and between countries;

General learning from the use of open data in other countries/sectors;

Support the day­to­day management of open data initiatives;

Improve the quality, reliability and quantity of open datasets, and establishing standards and guidelines for the collection and availability of open data;

Understand the commercial or social impacts of open data, and assess the Return on Investment;

Prioritize the availability of certain open data;

Support critical research that can improve policy and practice;

Support advocacy for more and better open data;

www.opendataresearch.org | www.webfoundation.org | www.thegovlab.org | 4

The current landscape of open data assessment A range of existing projects are working on aspects of open data assessment. The table below summarises a number of these:

Study & description Methods Unit of analysis

Coverage

The Open Data Barometer www.opendatabarometer.org A multi­dimensional index of open data readiness, implementation and impact developed by the Web Foundation and ODI.

Expert survey and secondary data. Assessment based on quantitative data. Also gathers and reports on qualititative assessments

Countries (focussing on national governments)

77 Countries in 2013. Target of 80+ countries in 2014. Broad range from across the world. Includes all G8 countries, but only partial OGP, OECD etc.

Open Data Index & Open Data Census https://index.okfn.org/ and http://national.census.okfn.org/ A regular assessment of whether key datasets are available as open data.

Ongoing crowdsourcing with expert review to create an annual index. Checklist with qualitative justifications.

Countries. A municipal census is also taking place in many countries.

70 Countries in 2013

UN E­Government Survey http://unpan3.un.org/egovkb/ The 2014 E­Gov Survey includes questions on open data.

Desk research with structured survey. Quantitative data.

National web portals

193 countries.

Open Data Monitor http://opendatamonitor.eu/ New EU project to look at automated assessment of open data.

Automated analysis of data portals providing ‘real­time’ statistics. Methods under development.

National open data catalogues.

EU (no data yet)

Open Data Certificate https://certificates.theodi.org/ Platform for data publishers to assess and improve the quality of their own open data.

Self­assessment with guided questionaire. Reports ‘level’ of open data quality.

Datasets Four different countries so far. Strongest in UK

OECD Framework http://www.oecd­ilibrary.org/governance/open­government­data_5k46bj4f03s7­en

Government survey completed by officials.

Countries OECD Countries

Open Data 500 http://www.opendata500.com/ Assessment of the value of open data in the private sector

Desk research & survey of companies. Linking commercial activity with source datasets.

Companies USA. Expanding to other countries.

Health Sector Indicators http://thegovlab.org/?s=NHS

Various quantiative and qualitative indicators

Sector (health) UK. Applicable to other countries.

www.opendataresearch.org | www.webfoundation.org | www.thegovlab.org | 5

Conceptual framework for measuring open data impact in specific sector

European PSI Scoreboard 3http://www.epsiplatform.eu/content/european­psi­scoreboard Crowdsourced and expert survey of European Public Sector Information policies and practices

Quantatitive scores against 7 different dimensions. Total score out of 700.

Countries EU countries (subject to the Public Sector Information directive)

Open Data Compass3 http://compass.arachnys.com Focussed on the availability of open corporate, litigation and media information worldwide

Quantitative score based on researcher assessment of data sources.

Countries Global

3 Added after the workshop

www.opendataresearch.org | www.webfoundation.org | www.thegovlab.org | 6

A common framework: draft proposal By looking at existing indicators and identifying gaps to be filled in order to meet the various use­cases for open data assessment outlined above, we identified four key components of a comprehensive open data assessment framework.

Context/environment Data Use Impact

Each of these components is detailed and broken down more in the sections below.

1) Context/environment Open data initiatives are taking place in very different settings across the world. The rate at which data is made open, or the extent to which effective use can be made of data, may be shaped by a range of factors in the country or sector under assessment. Assessment of context can support:

judgements on open data readiness: measuring key prerequisite or complementary factors associated with successful implementation of open data initiatives;

research into what makes for a successful open data initiative; tailored design of interventions to support context­sensitive open data initiatives.

The context component can be broken down into a number of sub­components. For each of these sub­components we outline the core question(s) this sub­component addresses, and suggest a range of indicators or data points which might fall into this part of the framework. Many of these are drawn from the existing studies detailed above: although further work is needed to identify a complete set of indicators, and clear operational definitions for them.

SUBCOMPONENTS – CORE QUESTIONS POTENTIAL INDICATORS (EXAMPLES)

Legal and Regulatory: What is the legal framework affecting access to, and use of, data?

Organisational: What is the type, or

structure of organisations involved, and what roles do different units, teams or individuals play in the open data programme or policy? Is there an open data eco­system?

Political Will/Leadership: To what

extent is there political/corporate will

Legal: Existence and strength of

legislation or regulation including Right To Information (RTI), open data, privacy and data protection.

Organisational: Existence of financial,

communication, and educational/training resources, as well as intermediary organisations, to implement open data initiatives.

Leadership: Statements of support

and policies on open data. OGP action

www.opendataresearch.org | www.webfoundation.org | www.thegovlab.org | 7

and/or leadership at a range of levels to support open data implementation?

Technical: What are the technical

skills or resources available to support publishing and using open data?

Social: What is the wider social

context into which open data is being introduced? Do civil­society groups, small businesses and independent citizens have freedoms and capacity to make use of open data?

Economic: What is the wider

economic context in which open data is being introduced? Do established firms and entrepreneurs have capacity to make use of open data?

plans. Financial and human resources committed to open data.

Technical: Extent of digitization in

government and government /organizational ICT capacity. Extent of ICT access across country and in government.

Social: Levels of education and

training; political freedoms; independence and vibrancy of civil society. Level of empowerment of small businesses or micro­entrepreneurs.

Economic: Extent of data skills in the

private sector; access to capital; presence of start­up culture; existence of government support for innovation.

Existing sources A number of contextual variables may be gathered from existing governance surveys. Others are open data specific and require the design of harmonized questions. Some of the resources to draw upon include:

RTI Rankings provides normative assessment of RTI laws. OECD Survey 2013 UNPAN Proposed OGD Index Open Data Barometer Expert Survey 2013 eGovernment Openness Index – Dataset Assessment ODDC Contextual Assessment Framework

There may be opportunities to include key open data context indicators within other global surveys, and to harmonize definitions used in sectoral and country context assessments.

2) Data An effective open data initiative results in open data being available. The dataset component of the framework looks at ways of assessing data availability, openness and usability, and seeks to harmonise definitions of key datasets that may be assessed in measurement excercises. The Open Definition offers a clear set of criteria for what it means for a dataset to be open or not, and a number of principles exist for what constitutes high quality open government data,

www.opendataresearch.org | www.webfoundation.org | www.thegovlab.org | 8

including the Sebastopol Principles. Other aspects of data openness, such as ‘social openness’, or the extent to which there is community built to support the re­use of data may also matter for some assessments. This section of the framework divides into four sub­components: definitions, dimensions, classifications and quality assessment.

SUBCOMPONENTS – CORE QUESTIONS POTENTIAL INDICATORS (EXAMPLES)

Definitions: What should be considered as open data? The Open Definition provides a clear account of when a dataset can be labeled open data or not. However, alongside the binary definition of open/non­open data, there can be different dimensions of openness, and measuring moves towards a fully ‘open’ state is useful in research.

Dimensions: What are the technical,

legal, practical, and social dimensions of openness?

Classification/Sector of Datasets:

What kinds of datasets are available within a country or sector?

Quality: How complete, primary, timely,

usable and reliable are specific datasets, or the open data provided in general?

Definitions: Explicit license*,

machine­readability and accessibility. (*There may need to be sensitivity here to different legal traditions when carrying out global comparative assessments)

Dimensions: Technical: data format and structure; availability of APIs; quality of meta­data. Social: quality of documentation; existence of support and feedback channels. Legal: re­use permissions granted through license or other mechanisms. Practical: where it appears; how is referenced; how is it reachable.

Sector/Classification of Datasets: A core

set of datasets that all governments should have, and/or common list of datasets that govenrments might have. Some datasets are important as a foundation (e.g. maps) and others have important impacts for particular use­cases. Assessments should be based on thematic coverage and relevance, not just counting datasets. Assessments could look at which sectors are producing data, as well as who is placing data onto data portals.

Quality assessment: Standards being

used to assess data/information across countries and sectors (ex; standards for records’ auditing). Assessment against common use­cases for data.

www.opendataresearch.org | www.webfoundation.org | www.thegovlab.org | 9

Existing sources A number of existing lists of key datasets exist, including lists in:

The Annexe of the G8 Open Data Charter; The Open Data Census; The Open Data Barometer; The OECD Survey of Countries;

Methodological considerations In selecting a list of datasets to use in assessments it is important to be sensitive to:

The political concerns that may have shaped the creation of the list; Whether the datasets can be reasonably expected to be available in all countries

surveyed, and how missing data should be treated; Whether the datasets will be held by central government, or may exist only at a municipal

level, or with private firms, and how this should be handled in the assessment; Whether the list describes specific datasets, or describes a category of data which could

be filled by multiple different datasets; Harmonising the operational definitions of datasets or data categories in different surveys can help to avoid confusion when the results of independent assessments of dataset availability are presented. The weighting of different dataset features in dataset assessment is also an important consideration for discussion.

3) Use How is data being used and with what possible outcomes? Separating use and outcomes from impacts is conceptually helpful in developing a deeper understanding of how open data can bring about change. This section of the framework looks at the category of users accessing data, the purposes for which the data will be used, and the activities being undertaken. It addresses the ‘who, what and why’ of open data in use. Understanding the purpose to which data is being put, and the goals for it’s use, is important for many forms of impact assessment.

SUBCOMPONENTS – CORE QUESTIONS POTENTIAL INDICATORS (EXAMPLES)

www.opendataresearch.org | www.webfoundation.org | www.thegovlab.org | 10

Users: Who is using open data? Who should be, or could be using open data?

Purpose: For what purposes are

individuals/organisations using open data? To achieve what goals?

Activities: To what uses are

individuals/organisations putting open data? In which sectors?

Users: Categories of users drawn from case studies, site analytics, surveys, and other sources (ex; researchers, entrepreneurs, media organisations).

Purposes: The intended goal/result of

open data programmes as expressed in research/project objectives (ex; to reduce spending).

Activities: The particular forms of uses for

open data (ex; benchmarking, hotspotting). What business models are being used in commercial open data re­use?

Methodological considerations There are a number of taxonomies of typologies needed to support further haromisation in this part of the framework. In particular, taxonomies of:

Kinds of users ­ going deeper than just ‘private sector’ / ‘public sector’ etc. Purposes for open data use Activities of open data use Business models for open data

These lists may be sector specific, although it may be possible to move towards shared taxonomies in this area. This section of the framework is likely to benefit from in­depth analysis of existing case studies of open data in use. Further development of this section could result in a case study template or interview schedule to help gather comparable case study data to further illuminate how open data is being used.

4) Impact Impact measurement may look at the benefits gained from using specific open datasets, or the returns from open data initiatives in general. In this section of the framework we suggest that benefits can be studied according to social, environmental, political/governance, and economic/commercial dimensions. We split out environmental and social as distinct categories, although some existing frameworks group these together, or group political and governance benefits in with social to offer a ‘triple bottom line’ impact assessment. Collecting data against the four categories allows analysis to be carried out in a range of different ways.

www.opendataresearch.org | www.webfoundation.org | www.thegovlab.org | 11

SUBCOMPONENTS – CORE QUESTIONS POTENTIAL INDICATORS (EXAMPLES)

Social: What are the social benefits to

be gained from the use of open data? How can open data be used to increase equality, target resources to citizens, and improve public services?

Environment: What are the potential

benefits of open data for environmental sustainability?

Political/Governance: How does open

data help improve government efficiency and accountability?

Economic/Commercial: What are the

impacts of open data on economic growth and innovation?

Social: Rates of public participation in

government and policy­making; evidence of increased social inclusion; evidence of improved social policy.

Environment: Number and effectiveness

of sustainability programs implemented using/as a result of open data; citizen awareness of environmental issues.

Political/Governance: use of open data

to identify and challenge corruption; reductions in perception of corruption; use of data to deliver better public services.

Economic/Commercial: Number of jobs created out of programs/products/services using OGD; contributions to economic growth from open data producing businesses.

Existing work

The Open Data Barometer collects qualitative stories of impact across social, political and economic dimensions.

The Open Data 500 estimates quantiative impacts of open data on business

The Open Data Era in Health and Social Care (GovLab/NHS) framework puts forward indicators for measuring impact, related to the specific goals of open data use.

The Open Data in Developing Countries case study research asks about ‘unintended consequences’ of open data, to also get at negative impacts it may have had;

Methodological challenges Research designs to evidence impact generally need to be specific to targets and goals, making them tricky to implement given one of the potential advantages of open data is that it allows many possible benefits, including unanticipated benefits.

www.opendataresearch.org | www.webfoundation.org | www.thegovlab.org | 12

Next steps: developing the framework further: This document presents a high level overview of the framework resulting from our workshop. Over the coming months, workshop participants, and other partners who join the process, will be reviewing this outline framework and exploring ways to provide more detail, including through:

Piloting the use of the framework for categorising case studies;

Developing common taxonomies to fill in key areas of the framework;

Using the framework to design new data collection instruments, or refine existing data collection;

Collaborating on detailed indicator definitions and guidance materials for researchers;

This will be coordinated through the Common Assessment Methods for Open Data mailing list which is open for anyone to join at the URL below: https://groups.google.com/forum/#!forum/common­assessment­methods­for­open­data

www.opendataresearch.org | www.webfoundation.org | www.thegovlab.org | 13

Appendix 1: Workshop Participants

José M. Alonso, Web Foundation Ulrich Atz, ODI Robyn Caplan, GovLab Tim Davies, Web Foundation Hania Farhan, Web Foundation Joel Gurin, GovLab Beth Noveck, GovLab Fernando Perini, IDRC Rufus Pollock, Open Knowledge Fabrizio Scrollini, ILDA Raed Sharif, Syracuse University Matthew Smith, IRDC Jonathan Sotsky, Knight Foundation Richard Stirling, ODI Deniz Susar, UNDESA Barbara Ubaldi, OECD Stefaan Verhulst, GovLab Asiya Wadud, GovLab

Appendix 2: Background reading list Suggested Background Reading Materials (given to participants)

Barnett, A., Dembo, D., Verhulst, S.G. (April 18, 2013). Toward Metrics for Re(imagining) Governance: The Promise and Challenge of Evaluating Innovations in How We Govern. GovLab Working Paper. V. 1.

Bonina, C. New business models and the value of open data: definitions, challenges and

opportunities. (August 30, 2013). Department of Management. London School of Economics and Political Science.

Granickas, K. (August 2013). Understanding the Impact of Releasing and Re­using Open

Government Data. European Public Sector Information Platform. Topic Report No. 2013 / 08. Gurin, J. Open Data 500 methodology. (2014). The Governance Lab. (forthcoming) Open data barometer. (October 2013). Open Data Institute and World Wide Web Foundation. Open Data Index, OKF https://index.okfn.org/ Researching the Emerging Impacts of Open Data. (July 2013). International Development Research

Centre and World Wide Web Foundation. Working Paper #1.

www.opendataresearch.org | www.webfoundation.org | www.thegovlab.org | 14

Ubaldi, B. (2013), Open Government Data: Towards Empirical Analysis of Open Government Data Initiatives. OECD Working Papers on Public Governance, No. 22, OECD Publishing.

Veljković, N., Bogdanović­Dinić, S., & Stoimenov, L. (2014). Benchmarking open government: An open data perspective. Government Information Quarterly. doi:10.1016/j.giq.2013.10.011

Verhulst, S., Noveck, B., Caplan, R., Brown, K., and Paz, C. (February 2014). The Open Data Era in Health and Social Care: A blueprint for the National Health Service (NHS England) to develop a research and learning programme for the open data era in health and social care.

www.opendataresearch.org | www.webfoundation.org | www.thegovlab.org | 15