discover the power of ddi metadata - carleton university ?· discover the power of ddi metadata...

Download Discover the Power of DDI Metadata - Carleton University ?· Discover the Power of DDI Metadata Wednesday…

Post on 15-Jul-2018




0 download

Embed Size (px)


  • Discover the Power of DDI Metadata

    Wednesday April 8, 2015Workshop

    North American DDI User Conference

    Jane FryCarleton University

  • Schedule

    Introductions Brief background of DDI Exercise 1 Getting started Examples Integration into a data lifecycle workflow Exercise 2 Wrap-up


  • Brief Background of DDI

    What is DDI? Data Documentation Initiative

    An international specification Not yet a formal ISO standard Goal

    Formats documentation for a social science data file More useful than a word or text file

    Supports the entire research data lifecycle


  • Relationship to Other Standards: Archival

    Dublin Core Basic bibliographic citation information Basic holdings and format information

    METS Upper level descriptive information for managing digital objects Provides specified structures for domain specific metadata

    OAIS Reference model for the archival lifecycle

    PREMIS Supports and documents the digital preservation process 4

  • Relationship to Other Standards: Non-Archival

    ISO 19115 Geography Metadata structure for describing geographic feature files such as

    shape, boundary, or map image files and their associated attributes

    ISO/IEC 11179 International standard for representing metadata in a Metadata

    Registry Consists of a hierarchy of concepts with associated properties

    for each concept

    ISO 17369 SDMX Exchange of statistical information (time series/indicators) Supports metadata capture as well as implementation of registries


  • Brief Background (contd)

    Creates a standard format Used to mark up codebooks Meaningful and consistent Metadata is both human and machine readable

    Gives codebook level details such as dataset contents, variable labels, summary Statistics and


    Also question text for each variable 6

  • Caveat

    DDI is powerful provided that the information is entered into the appropriate fields when marking up the document.


  • Brief Background (contd)

    Remember DDI facilitates the creation of metadata

    Expressed in XML XML Schema

    A way of tagging text for meaning, not appearance Older version of DDI used DTD (Document type definition) Defines

    Which tags are available The order the tags will appear in a document Whether the tags are required or optional Whether the tags are repeatable or not


  • Example of DDI Compliant Codebook

    Canadian Tobacco Use Monitoring Survey, 2004: Annual, Person File

    Annual, Person File

    CTUMS 2004: Annual, Person File



  • Brief Background (contd)

    Benefits of DDI Interoperability Rich content A single document with multiple purposes Increased search capability

    precision in searching More visibility for the data because it is web-based

    Research is no longer on just one computer


  • Brief Background (contd)

    Challenges Complexity Level of researcher buy-in Need for tools Changes in workflow Access to metadata


    Reference: Vardigan and Wackerow, 2013

  • Brief Background (contd)

    History Started in 1995 The director of ICPSR, Richard Rockwell, constituted a

    Committee on metadata

    Developed a draft list of codebook elements An international committee Quebec City


  • Brief Background (contd)

    Norwegian Social Science Data Services

    Harvard University American University Statistics Canada Health Canada Bureau of the Census University of Michigan Bureau of Labor Statistics ICPSR

    Yale University ESRC Data Archive (UK) University of California,

    Berkeley University of Southern

    Denmark The Roper Center Zentralarchiv fr

    Empirische Sozialforschung (GESIS)


  • Brief Background (contd)

    Bjorn Henrichsen Micah Altman Atle Alvheim Grant Blank Ernie Boyko Bill Bradley Cavan Capps Bill Connett Cathryn Dippo Pat Doyle Dan Gillman Peter Granda

    Ann Green Peter Joftis Ken Miller Tom Piazza Karsten Boye Rasmussen Richard Rockwell Jostein Ryssevik Merrill Shanks Peter Solenberger Rolf Uher Mary Vardigan


  • Brief Background (contd)

    1996: Prepared first DTD (document type definition) Set up markups/specifications

    1997: Received funding from the NSF Specifications translated to XML

    1999: Beta testing Reports submitted

    2000: DDI Version 1 published15

  • Brief Background (contd)

    2001: First DDI Training 2001: Received funding from Health Canada

    2002: DDI Alliance charter drafted 2003

    DDI 2 published DDI Alliance established

    First Steering Committee meeting

    Final meeting of original Committee


  • Brief Background (contd)

    2007: Public review of DDI 3 2008: DDI 3 published as XML Schemas 2009: DDI Lifecycle 3.1 published 2012: DDI C 2.5 published as XML Schemas 2013: DDI Sprints

    First DDI Executive Board meeting

    2014: DDI C 2.5 modified DDI 3.2 published 17

  • Brief Background (contd)

    DDI Alliance Self-sustaining Membership organization Members have a voice in the development of DDI Membership documents are all online

    Charter, bylaws, member forms, Publications are all one line

    Newsletters, annual reports,


  • Brief Background (contd)

    DDI now branched into 2 separate development lines or metadata standards

    DDI Codebook aka DDI C Formerly DDI 2 This was the original one Built to emulate a physical codebook


  • Brief Background (contd)

    DDI Lifecycle aka DDI L formerly DDI 3 Supports the research data lifecycle The one new users are learning


  • Brief Background (contd)

    DDI 1 and 2 Document Description Study Description Data Files Description Variable Description Other Study Related


    DDI 3 Study Concept Data Collection Data Processing Data Distribution Data Archiving Data Discovery Data Analysis Repurposing

    21Reference: Jim Jacobs, 2006

  • Brief Background (contd)

    How to determine which version of DDI to use if you are new to the game?

    DDI C Relatively straight forward If you want to catalog a dataset If you are describing a single study

    DDI L If you are focusing on a lifecycle model Broken down into different functions

    Are you documenting questionnaires? Are you documenting data? Are you doing both?


  • Brief Background (contd)

    Challenge How to go from DDI C to DDI L? Even though DDI L is fast becoming the more preferred

    one, if you are already using DDI C, it would be too much work to convert to DDI L

    Insufficient resources to do this conversion for most institutions (people, $)

    Solution Crosswalks, other resources


  • Brief Background (contd)

    Coming soon! DDI 4

    Has an integrative vision DDI C (DDI 2) and DDI L (DDI 3)

    Will be aimed at machine-actionable processing at the beginning


  • Brief Background (contd)

    Who is Using DDI? CESSDA Data Portal (European quantitative social science datasets) Australian Social Science Data Archive DAMES Project (UK) DataFirst at University of Cape Town Data Liberation Initiative (Statistics Canada) General Social Survey (US) ICPSR (Interuniversity Consortium for Political & Social Research) Israel Social Science Data Center Philippines National Statistics Office Statistics New Zealand Vision of Britain (historical view between 1801 and 2001) World Bank (International Household Survey Network)


  • Global DDI


    In 2013

  • Brief Background (contd)

    What Projects are using DDI? California Digital Librarys Counting California Project Centre for Comparative European Survey Data Canadian Research Data Centres DAMES Project (Universities of Stirling and Glasgow) ICPSR Data Catalog ODESI (Ontario Data Portal) Sri Lanka Department of Census and Surveys World Bank (International Household Survey Network)


  • Brief Background (contd)

    DDI website Excellent resource

    FAQ Tools Markup Examples Metadata Resources

    Directions: A newsletter of the Data Documentation Initiative 28

  • Time to think!


  • Exercise 1

    As a researcher, what metadata do you absolutelyneed?

    How do you want it to streamline your research?

    What metadata would you like to have, if it is available, but it is not integral to your research?


  • Getting Started with DDI

    Daunting at first Process is broken down into steps

    Lots of help available DDI Alliance Colleagues Other researchers

    DDI List-serv DDI Best Practices

    Work in progress Feedback always welcome


  • Getting Started with DDI (contd)

    Tools to help you get started

    Drop down menu Browse a list

    Choose the one that will suit your purposes License choices

    Commercial Freeware GNU Lesser General Public License (LGPL)

    For different versions of DDI


  • Getting Started with DDI (contd)

    One tool: Nesstar Publisher Norwegian Social Science Data Services Data management program Freeware Data and metadata conversion and editing tools

    Enhance datasets Combine catalogue and contextual information

    Merge DDI documents with markup for differen