Open @ FAO

Open For DevelopmentEADI IMWG Conference 2012

Stephen.Katz@ fao.org (Twitter: @SteveK1958)Chief, Knowledge Management and Library ServicesFood and Agriculture Organization of the United Nations

Open @ FAOAgenda

Context and History of Open @ FAO

Issues, Challenges and Lessons Learned

Group Discussion

1

2

3

4

Ongoing Practical Initiatives• FAO Open Archive • Open Data (data.fao.org)• Data Governance and Standards

Open @ FAO : Food For Thought?

Food for Thought

• FAO is a specialized agency of the United Nations with its own independent governance

• 190+ Member Countries

• HQs in Rome, Offices in over 80 countries with over 5000 staff.

Food and Agriculture Organization of the United Nations (FAO)

Food and Agriculture Organization of the United Nations (FAO)

• Collects, analyses, interprets and disseminates information on nutrition, food and agriculture

• Policy Advice

• Furnishes Technical Assistance

• A Neutral Forum for International Cooperation

FAO has been in the “knowledge” business since 1946!

Our mandate....Ensure that the world’s knowledge of food and agriculture is available to those who need it when they need it and in a form which they can access and use.

1995 – Central Publishing Unit Abolished 1996 – SGML Repository Proposal; FAOSTAT on-line 1997 – Document Repository (XML Compatible) 2003 – Document Repository (PDF) 2007 – Open Archive Proposal (Fedora Commons) 2010 – Open Data Repository Proposal (data.fao.org) 2012 – OpenArchive.Fao.Org; Data.Fao.Org

Open @ FAO : A Bit of History

FAO Open ArchiveGoals/Objectives

To make FAO’s Global Public Goods openly accessible from a single access point

To be able to exchange data in an open and standardized way

To have a smooth/efficient workflow to manage FAO’s Institutional memory

To integrate e-publishing and library workflows

FAO Open ArchiveArchitecture

Based on Open Source tools (Fedora Commons and Java)

Based on modern standards for data management (MODS and FRBR)

Allowing for easier management and sharing of multilingual content

And this is what it looks like:

Open Archive Resources Available at Start-up Time

Resource Type Number of Records

Full Text Documents 40,100

Photos and Videos 17,100

Audio Files 1,200

To address fragmentation and duplication of information systems and data presently distributed across many organizational units

http://data.fao.org: one-stop shop that aggregates, integrates, and catalogues data from multiple sources across FAO. Topics are related to nutrition, food and agriculture and include statistics, maps, pictures, documents and more.

Open Data (data.fao.org)Goals/Objectives

Uniting FAO data with one brand : http://data.fao.org Engaging a Community : #FAOdata Mobile First Serve the data in the most convenient format Integrate, don't reimplement

Open Data (data.fao.org)Guiding Principles

data.fao.org - The Big Picture

Orchestration and Integration

Content

DocumentsPicturesVideoMultimediaPages

StatisticsStatistical Data Warehouse

Time SeriesIndicatorsObservations

Maps

Geospatial RasterVectorPoint

Catalogue

IdentityMetadataLinked Data...

Infrastructure

LoggingCachingSecurityAudit...

WebsiteSpecialised

application(s)consume/provide

Search

Full textStructured

Services and Widgets

DataSource

Data Flow Architecture

Ingest

Publish

HarmoniseIntegrate

Enrich

DataSource

DataSource

DataSource

DataSource

Landing Page

Some Numbers

356,000,000 Statistical values 2 Terabytes

1,500,000 Statistical Maps

734,000 Geo Layers 30 Terabytes

435 Documents

90 Pictures

25 Information Systems

Opscode –Chef, Red Hat- RHQ, Nagios- Enterprises Nagios, Jenkins-CI – Opensource – Jenkins, Apache Maven, Apache – SVN, Atlassian – JIRA, Apache – Jmeter, Sourceforge Opensource – Junit, Oracle- Java, Liferay - Liferay Portal, PostgreSQL – PostgreSQL PostGIS, Pgbouncer – pgbouncer, Pentaho - Pentaho BI, Analytical Labs – Saiku, Talend - Talend Open Studio, RedHat/Jboss - Application Server, Vmware - Server Virtualization,

Exceliance – HAProxy, RedHat- Enterprise Linux Server, GeoNetwork OpenSource – GeoNetwork, OpenGeo, GeoSolutions, Fedora Commons, Refractions Research – GeoServer, Ontotext - OWLIM, RedHat/Jboss - JBoss AS / JEE, Alfresco - Activiti BPM Platform, jasig - CAS Client, Batchwork Software - Doc2Doc, Google Code Opensource - ZXing ("Zebra Crossing"), Highsoft Solutions AS – highcharts, Twitter – Bootstrap, Liferay - Alloy-UI, JQuery - JQuery-ui, Geert de Deckere - Graph Up and the there are all the SaaS products ….

Simple and few technologies

Further questions on data.fao.org to the Project Manager: Karl.Morteo@fao.org

http://www.ciard.net

CIARD – a global movement

To make agricultural research information and knowledge truly acessible to all

• All organizations that create and possess public agricultural research information disseminate and share it more widely

• CIARD partners create coherence by a) coordinating their efforts, b) promoting common formats, c) adopting open systems and standards

• Create a global network of public collections of data and information

http://aims.fao.org

Distributed Data Sets

•stats

•gene banks

•gis data

•blogs,

•journals

•open archives

•raw data

•technologies

•learning objects

•………..

How to make value added services?How to infer new knowledge?How to organize collaboration?Maybe we really need this?...

…to

•stats•gene banks•gis data•blogs, • journals•open archives•raw data• technologies• learning objects•………..

Creating Linked Applications

OpenAgris

Aggregates different data sources to expand knowledge about a topic

Is a “linked-data” environment mashing-up interlinked datasets to create an integrated knowledge base

OpenAgris uses the Agrovoc thesaurus as backbone to interlink to other existing datasets (DBPedia, WorldBank, Geopolitical Ontology…)

Open Archive : Issues, Challenges, Lessons

Unclear Policy Framework Unclear collection selection policy Variable quality standards (content, legal, editorial,

accountability) Licensing policy/conditions for re-use Working with partners and scientific journals Freely available but need attribution Supply vs demand (personal interest vs impact) Tension with Sales and Marketing needs

May Lead To Negative Consequences such as: Low credibility/trust, reputational risk, legal exposure?

Open Data : Issues, Challenges, Lessons

Well the same stuff as before really Unclear Policy Framework Unclear collection selection policy Variable quality standards Licensing policy/conditions for re-use Working with partners Freely available but need attribution Supply vs demand (personal interest vs impact) Tension with Sales and Marketing needs

May Lead To Negative Consequences such as: Low credibility/trust, reputational risk, legal exposure?

Open Data : Issues, Challenges, Lessons

But also: Every data-type has it’s own standards (e.g. OGC for GIS,

SDMX for stats, MODS for documents, IPTC for Photos) Aggregate data quality set by lowest common denominator Poor data governance leads to:

Conflicting/contradictory data values from different sources Lack of agreement of definitions and concepts, and Insufficient metadata Comparing apples, pears and oranges (different units, different

assumptions, different contexts)

May Lead To Negative Consequences such as: Low credibility/trust, reputational risk, legal exposure?

Thank you!

Time for Discussion

and soon for Lunch!