Microdata Microdata Management ToolkitManagement Toolkit

Tools to facilitate archive and dissemination Tools to facilitate archive and dissemination of surveysof surveys

A PDF for Data?A PDF for Data?Metadata Editor / Nesstar Publisher 3.5Metadata Editor / Nesstar Publisher 3.5

CD builderCD builderGuidelines for Archiving & DisseminationGuidelines for Archiving & Dissemination

Session E2 - Thursday, 26 MayTools for Preservation: Integration and Assessment

Preserving and improving the access to large and complex household surveys

Mark DiggoryMark Diggory Olivier DupriezOlivier Dupriez Pascal HeusPascal Heus Jostein RyssevikJostein RyssevikHarvard / MIT Data Harvard / MIT Data

CenterCenterWorld BankWorld Bank World bankWorld bank Nesstar Ltd.Nesstar Ltd.

mdiggory@latte.harvard.mdiggory@latte.harvard.eduedu

odupriez@worldbank.orgodupriez@worldbank.org pascal.heus@gmail.compascal.heus@gmail.com Jostein.Ryssevik@nsd.uibJostein.Ryssevik@nsd.uib.no.no

BackgroundBackground• Sponsored by World Bank / International Household Survey Sponsored by World Bank / International Household Survey

NetworkNetwork– Presented earlier this week Presented earlier this week – Created in September 2004Created in September 2004– International organizations actively sponsoring household surveys International organizations actively sponsoring household surveys – Marrakech Action Plan for StatisticsMarrakech Action Plan for Statistics– http://www.surveynetwork.orghttp://www.surveynetwork.org

• Survey often under-used: limited access for users which leads Survey often under-used: limited access for users which leads to poor return on investment limited impact on the ground, to poor return on investment limited impact on the ground, difficulties in policy makingdifficulties in policy making

• Common obstacles: quality, technical capacity, legal/political Common obstacles: quality, technical capacity, legal/political issuesissues

• Common problems:Common problems:– Accessibility, Timeliness, CoherenceAccessibility, Timeliness, Coherence– Lack of metadata / documentation / dataLack of metadata / documentation / data– Poorly organized archivesPoorly organized archives

• To address technical issues: Need for new tools and To address technical issues: Need for new tools and guidelinesguidelines

Microdata Management Toolkit Microdata Management Toolkit

Toolkit RequirementsToolkit Requirements• User friendly software suite and guidelines to User friendly software suite and guidelines to

archive and disseminate microdataarchive and disseminate microdata• Facilitate metadata exchange: compliant with Facilitate metadata exchange: compliant with

common XML specifications (DDI, Dublin Core)common XML specifications (DDI, Dublin Core)• Facilitate archiving: put together metadata and Facilitate archiving: put together metadata and

data, address common quality control issuesdata, address common quality control issues• Facilitate dissemination: simple to redistribute on Facilitate dissemination: simple to redistribute on

cd/dvd and the web, answer producer/depositor cd/dvd and the web, answer producer/depositor needs (subset, anonymization, quality control)needs (subset, anonymization, quality control)

• Works with common data formats (spss, sas, stata, Works with common data formats (spss, sas, stata, statistica, cspro/imps/issa)statistica, cspro/imps/issa)

• Multilingual supportMultilingual support• Free or InexpensiveFree or Inexpensive• Availability of technical support and trainingAvailability of technical support and training• Accompanied with guidelines and training programAccompanied with guidelines and training program• Supported by national, international and research Supported by national, international and research

communitiescommunities

Core file format - A PDF for Core file format - A PDF for Data?Data?

• How can we carry around the information?How can we carry around the information?• Looking at documents Looking at documents PDF PDF• Can we do the same for data?Can we do the same for data?

– Yes, a Nesstar file holds data + metadata!Yes, a Nesstar file holds data + metadata!

• Partner with Nesstar Ltd to develop new toolsPartner with Nesstar Ltd to develop new tools– Why: strong tool for metadata management, available Why: strong tool for metadata management, available

today, community acceptance, technical support, past today, community acceptance, technical support, past experienceexperience

• Development agreementDevelopment agreement– Enhance existing publisher software and make available Enhance existing publisher software and make available

as a stand alone productas a stand alone product– Open binary file format (not a black box) and availability Open binary file format (not a black box) and availability

of APIof API– Free data reader (like pdf) that allows user to access at Free data reader (like pdf) that allows user to access at

the data and metadata and convert to their favorite formatthe data and metadata and convert to their favorite format– Special licensing agreement for developing countriesSpecial licensing agreement for developing countries

Toolkit ComponentsToolkit Components• Archiving: Metadata Editor (World Bank / Nesstar Archiving: Metadata Editor (World Bank / Nesstar

Ltd.)Ltd.)– To compile survey data, documentation and metadata in a To compile survey data, documentation and metadata in a

standard format (Nesstar/DDI). Free data reader for users.standard format (Nesstar/DDI). Free data reader for users.– Built on Nesstar PublisherBuilt on Nesstar Publisher

• Dissemination: CD Builder (World Bank / Mark Dissemination: CD Builder (World Bank / Mark Diggory)Diggory)– To facilitate the publication of survey data, documentation To facilitate the publication of survey data, documentation

and metadata on CD-ROM and on the web (transforms DDI and metadata on CD-ROM and on the web (transforms DDI into HTML based navigation)into HTML based navigation)

– Based on Eclipse Platform, open sourceBased on Eclipse Platform, open source

• Guidelines: Handbook (World Bank / ICPSR)Guidelines: Handbook (World Bank / ICPSR)– To provide data producer with information on policies and To provide data producer with information on policies and

legal aspect of data dissemination, guidelines to document legal aspect of data dissemination, guidelines to document datasets and recommendations in setting up a data archivedatasets and recommendations in setting up a data archive

Generate HTML based CD-ROM

Import metadata and prepare CD-ROM

Import data and compile metadata

The Toolkit ProcessThe Toolkit Process1

2

3

What is the Nesstar What is the Nesstar Publisher?Publisher?

• Advanced data managementAdvanced data management program program• DDI /DC Metadata authoring toolDDI /DC Metadata authoring tool• Import/Export to common data formatsImport/Export to common data formats• Standalone or w/Nesstar serverStandalone or w/Nesstar server• http://www.nesstar.comhttp://www.nesstar.com

•Easy editing/creation of DDI documented datasets. No need to know XML. •Full DDI import and export for single file/language studies. •Templates which lets your organization standardize the use of the DDI. •Default texts in templates. •Local controlled vocabularies. •Possible to share the documentation work between different persons. •A Category Repository which lets you share categories within a dataset and between datasets. •Variable groups. •Easy setting of weights. •Frequency and summary statistics output, with options for each variable. •Import and export to the most common statistical formats.

What is the Metadata What is the Metadata Editor?Editor?

• Nesstar Publisher 3.0:Nesstar Publisher 3.0:– A tool to prepare and publish A tool to prepare and publish

surveys to a Nesstar Serversurveys to a Nesstar Server– Sold as a component of the Sold as a component of the

Nesstar Software SuiteNesstar Software Suite– Multiple components Multiple components

(editor, hierarchy, cube, resources)(editor, hierarchy, cube, resources)

New Model for Version 3.5:New Model for Version 3.5:– All components integrated under one interfaceAll components integrated under one interface– A study is stored in a single Nesstar fileA study is stored in a single Nesstar file– Enhanced and new functionalitiesEnhanced and new functionalities

• Quality control, computed variables, recodes, Quality control, computed variables, recodes, anonymize, subsetanonymize, subset

– Availability of a free Nesstar Data ReaderAvailability of a free Nesstar Data Reader– Produce DDI / Dublin Core (DC) XML documentsProduce DDI / Dublin Core (DC) XML documents– Available as a stand-alone software packageAvailable as a stand-alone software package

Editor key features (1)Editor key features (1)

All components integrated under a single interface

All components integrated under a single interface

Import/Export support common

data formats

Import/Export support common

data formats

All surveys stored as projects in a single

tree hierarchy

All surveys stored as projects in a single

tree hierarchy

Template driven metadata editor

allows for users to decide which

DDI/DC elements to use.

Template driven metadata editor

allows for users to decide which

DDI/DC elements to use.

Editor key features (2)Editor key features (2)

Easy to use interface for document, survey,

file and variable metadata editing

Easy to use interface for document, survey,

file and variable metadata editing

Editor key features (3)Editor key features (3)Data import preserves

existing dictionary and generates

summary statistics

Data import preserves existing dictionary

and generates summary statistics

Manage variable groups

Manage variable groups

DDI and Dublin CoreMetadata

import/export

DDI and Dublin CoreMetadata

import/export

Editor key features (4)Editor key features (4)Support for survey documentation as

Dublin Core resources

Support for survey documentation as

Dublin Core resources

Description of a dataset primary keys

and hierarchy

Description of a dataset primary keys

and hierarchy

…and validation of dataset relationships

…and validation of dataset relationshipsAutomatic randomization of primary

key variables

Automatic randomization of primary key variables

AND MORE…AND MORE…

Data ReaderData Reader• Free softwareFree software• PDF philosophyPDF philosophy• Access to survey Access to survey

metadatametadata• Access to data (no need Access to data (no need

for specialized for specialized software)software)

• Export to common Export to common formatsformats

• Single file holds data Single file holds data and metadataand metadata

What is the CD Builder?What is the CD Builder?• Purpose is to publish survey Purpose is to publish survey

metadata, documents and data metadata, documents and data on a CD-Rom (or web site)on a CD-Rom (or web site)

• Transforms DDI into an HTML Transforms DDI into an HTML based interfacebased interface

• User can customize the layout User can customize the layout (branding) and content of the CD (single (branding) and content of the CD (single or multi-surveys)or multi-surveys)

• Open source applicationOpen source application• Build on the Eclipse FrameworkBuild on the Eclipse Framework• Based on DDI / Dublin CoreBased on DDI / Dublin Core• Integrates with Metadata EditorIntegrates with Metadata Editor• Easy to useEasy to use

CD Builder ProcessCD Builder ProcessCreate new CD-ROM

Project

Create new CD-ROM Project

Add a survey to the project and select its

type and branding

Add a survey to the project and select its

type and branding

1

2• Selecting a survey consist in opening the DDI-XML or Nesstar file• The survey “branding” determines the overall look and feel of the CD• The survey “type” determines the default metadata content

• Selecting a survey consist in opening the DDI-XML or Nesstar file• The survey “branding” determines the overall look and feel of the CD• The survey “type” determines the default metadata content

Click the “Save” button to generate the HTML

interface

Click the “Save” button to generate the HTML

interface

3

After a few minutes, your CD Project is ready

for publishing!

After a few minutes, your CD Project is ready

for publishing!

4

Key FeaturesKey FeaturesContent of CD pages is fully customizable

Content of CD pages is fully customizable A CD-ROM project

can hold several surveys

A CD-ROM project can hold several

surveys

• Branding customization• Can be published to web • Multilingual support• Automatic updates…and more…

• Branding customization• Can be published to web • Multilingual support• Automatic updates…and more…

Sample outputSample output

HandbookHandbook

• Handbook on the Documentation, Handbook on the Documentation, Dissemination, and Preservation of Dissemination, and Preservation of MicrodataMicrodata– Part I: Policy, legal and ethical issues and Part I: Policy, legal and ethical issues and

recommendations. Benefits and costs of recommendations. Benefits and costs of microdata disseminationmicrodata dissemination

– Part II: Technical guidelines: documenting, Part II: Technical guidelines: documenting, disseminating and preserving a datasetdisseminating and preserving a dataset

– Part III: Setting-up a central data archivePart III: Setting-up a central data archive

Benefits and Users (1)Benefits and Users (1)

• What will the toolkit improve?What will the toolkit improve?– Documentation (based on standards, Documentation (based on standards,

guidelines and validation)guidelines and validation)– Preservation: data and metadata stay Preservation: data and metadata stay

together, CD archivingtogether, CD archiving– Cataloguing: facilitate metadata Cataloguing: facilitate metadata

exchangeexchange– Dissemination: CD, DVD, WebDissemination: CD, DVD, Web– Quality: validation procedures, use of Quality: validation procedures, use of

common language, adoption of best common language, adoption of best practicespractices

Benefits and Users (2)Benefits and Users (2)

• Potential users?Potential users?– Survey producers at national level: Survey producers at national level:

preservation, dissemination, harmonize preservation, dissemination, harmonize frameworkframework

– International survey sponsorsInternational survey sponsors– Data archivesData archives

• Who will benefit?Who will benefit?– Data producersData producers– National & International survey sponsorsNational & International survey sponsors– Survey data repositoriesSurvey data repositories– Data analystsData analysts– Policy makers and populationPolicy makers and population– DDI CommunityDDI Community

Status & AvailabilityStatus & Availability• Publisher 3.5Publisher 3.5

– Beta version available Beta version available – Nesstar commercial release during the Nesstar commercial release during the

summersummer

• CD BuilderCD Builder– Beta version availableBeta version available– Public release expected in September Public release expected in September

(Open Source)(Open Source)

• GuidelinesGuidelines– Draft completedDraft completed– Review over the summerReview over the summer

Next?Next?• Distribution, training and adoption of the toolkitDistribution, training and adoption of the toolkit• User acceptance tests and pilot sitesUser acceptance tests and pilot sites• Release of open source components (Sourceforge, DDI)Release of open source components (Sourceforge, DDI)• Future developments:Future developments:

– Translations in other languagesTranslations in other languages– Plug-ins for Publisher and/or Reader (open source)Plug-ins for Publisher and/or Reader (open source)– Availability of API libraryAvailability of API library– Basic analytical functionalities (tabulation, graphs, etc.)Basic analytical functionalities (tabulation, graphs, etc.)– Evaluation of disclosure risks / anonymization proceduresEvaluation of disclosure risks / anonymization procedures– Embed document in archive file (?)Embed document in archive file (?)– Plan for DDI 3.0 supportPlan for DDI 3.0 support– Bug fixes / enhancements / new features (based on user Bug fixes / enhancements / new features (based on user

feedback)feedback)– And more based on feedback from users, DDI & open source And more based on feedback from users, DDI & open source

communitycommunity• Integration of other tools:Integration of other tools:

– Argus [confidentiality]Argus [confidentiality]– CSPro [production]CSPro [production]– Virtual Data Center (VDC) [web based dissemination]Virtual Data Center (VDC) [web based dissemination]

• Strong collaboration and participation of the communityStrong collaboration and participation of the community

Thank you!Thank you!

Mark DiggoryMark Diggory Olivier DupriezOlivier Dupriez Pascal HeusPascal Heus Jostein RyssevikJostein RyssevikHarvard / MIT Data Harvard / MIT Data

CenterCenterWorld BankWorld Bank World bankWorld bank Nesstar Ltd.Nesstar Ltd.

mdiggory@latte.harvard.mdiggory@latte.harvard.eduedu

odupriez@worldbank.orgodupriez@worldbank.org pascal.heus@gmail.compascal.heus@gmail.com Jostein.Ryssevik@nsd.uibJostein.Ryssevik@nsd.uib.no.no

QUESTION / ANSWERQUESTION / ANSWER