soton2013 opendata

Get Started With Open Data

Tony HirstDept of Communication and Systems,

The Open University

So what do we mean by

“OPEN DATA”

Open Public Data

- copy, publish, distribute and transmit the Information;- adapt the Information;- exploit the Information commercially for example, by combining it with other Information, or by including it in your own product or application

You are free to:

You must:- acknowledge the source of the Information by including any attribution statement specified by the Information Provider(s) and, where possible, provide a link to this licence;- ensure that you do not use the Information in a way that suggests any official status;- ensure that you do not mislead others or misrepresent the Information or its source;- ensure that your use of the Information does not breach the Data Protection Act 1998 or the Privacy and Electronic Communications (EC Directive) Regs 2003.

Exemptions:- personal data;- Information that has neither been published nor disclosed under information access legislation (FOI) by or with the consent of the Information Provider;- departmental or public sector organisation logos, crests etc;- third party rights the Information Provider is not authorised to license;- Information subject to other IPR

Availability and Access

Reuse and Redistribution

Universal Participation

The Open Knowledge Foundation

Availability and Access: the data must be available as a whole and at no more than a reasonable reproduction cost, preferably by downloading over the internet. The data must also be available in a convenient and modifiable form.

The Open Knowledge Foundation

Reuse and Redistribution: the data must be provided under terms that permit reuse and redistribution including the intermixing with other datasets.

The Open Knowledge Foundation

Universal Participation: everyone must be able to use, reuse and redistribute – there should be no discrimination against fields of endeavour or against persons or groups. For example, ‘non-commercial’ restrictions that would prevent ‘commercial’ use, or restrictions of use for certain purposes (e.g. only in education), are not allowed.

The Open Knowledge Foundation

/via http://antictrl.com/chapter-3-2-regulability-of-the-internet/

Lessig’s “

dot”

Licensing

DATAAuthentication

Closed standards

Messy Data

Crappy spreadsheets

Paywalls

“Privacy”

FOI exemptions

Data protection Act

PDFs

Right to access data

So where’s the data?

“First” generation:data catalogues

Breathing life into data…

=importData(“CSV_URL”)

Google Sheets

the spreadsheet becomes

A DATABASE

Google Charts

Visualisation API

Google Charts

Visualisation API

Google Charts

Visualisation API

“Second” generation:data management

systems

DMS – Data Management System

Digging for data…

BUT

There’s lots more data that’s locked up in web pages…

Scraping…

“grabbing web content in a machine readable

format and then processing it for your

own purposes”

Original HTML web

page

Accessible web page

Extract Information

-> data

Recreating the database that was used

to populate a (templated) page

“Creating” Data

[Disruptive Innovation?]

Company

DirectorDirector

DirectorDirector

CompanyCompany

CompanyCompany

Barriers to Use

OpenRefine

Also:- month overflows at week end- year overflows

- Character string dates- Erratic whitespace- Arbitrary separators- Excel Dates

Open is as open

does… DATA

@psychemedia

blog.ouseful.info