metadata architecture at statcan msis 2008 luxembourg, april 7-9, 2008 karen doherty director...
TRANSCRIPT
MMetadata Architecture etadata Architecture at StatCanat StatCan
MSIS 2008Luxembourg, April 7-9, 2008
Karen DohertyDirector General Informatics BranchStatistics Canada
22
Table of ContentsTable of Contents
• Building Blocks
• Viewing Metadata Via STCWiki
• Metadata Architecture
• Conclusion
33
BackgroundBackground
• Large collection of data offerings led to the need for metadata on the surveys and administrative data sources
• A lot of effort was spent initially on defining the surveys to support the dissemination function
• Focus has shifted to the descriptions of the variables and code sets used
44
Business DriversBusiness Drivers
• Many programs now load their data into warehouses to support analysis and data confrontation
• Users were asking for the supporting metadata to be available from within the warehouses
• Users wanted to be able to submit corrections to the metadata as they were working on the data
55
Building BlocksBuilding Blocks
• Metadata collection– Integrated Metadata Base (IMDB)
• Data manipulation and data display tools– Data Warehouse Framework– EzWeb– STCwiki
66
Integrated Metadata BaseIntegrated Metadata Base
• Contains information on our 590 active surveys• Benefits:
– Improves the interpretability of our surveys
– Assists in assuring the coherence of our data
– Promotes knowledge sharing within StatCan and with external users
– Preserves corporate memory
– Promotes the reuse and standardization of metadata assets including definitions and code sets
88
Integrated Metadata BaseIntegrated Metadata Base
• Architecture:– Oracle Database– Java
• Model based on ISO/IEC 11179 Metadata Registries and the Corporate Metadata Repository (CMR) from the US Census Bureau
99
Integrated Metadata BaseIntegrated Metadata Base
• Data dimension model :– Describes the data– Based on ISO/IEC 11179– Data elements (variables) specified by an
object class with properties and values (code sets, classifications)
– Can be used to describe any set of objects not just metadata
1010
Integrated Metadata BaseIntegrated Metadata Base
• ISO 11179 data element definition :– Object class: type of object being described (person,
establishment, household, etc.)
– Property: attribute that describes the object (sex, age, etc.)
– Conceptual Domain: list of possible settings (value meanings) for a property (property Sex has two possible meanings Male and Female)
– Value Domain: code set used to represent the value meanings (1 = Male, 2 = Female, etc.)
1111
Data Warehouse Data Warehouse FrameworkFramework
• Standard development framework for data warehouses at Statcan
• Based on Microsoft .net / SQLServer
• Users can access metadata from within the warehouse
1212
EZWeb & STCwikiEZWeb & STCwiki
• EzWeb– used to develop Intranet sites in a very standardized
way– incorporates MS Excel pivot tables to allow users to
view data in a warehouse via OLAP– can navigate easily from one OLAP report to another
• STCwiki– used to display metadata from the IMDB– allows users to collaborate and to submit proposed
changes to the IMDB– based on MediaWiki (used by Wikipedia)
1313
Access from Access from a Data Warehousea Data Warehouse
A user viewing data in a warehouse by age range and year can see the associate metadata by clicking on the Metadata icon (top menu).
1414
Metadata Browsing Metadata Browsing in STCwiki in STCwiki
The user can view any of the metadata associated with the table they were looking at – in this case the definitions of the age range variable.
1515
Metadata Browsing Metadata Browsing in STCwiki in STCwiki
The user can view any metadata stored in IMDB – all is accessible from STCwiki.
1616
Put it All TogetherPut it All Together
• So now we have the building blocks, how do we put it together into a coherent architecture?
1717
ObjectivesObjectives
• Leverage investment in IMDB to describe not only statistical metadata but also metadata on IT systems and the enterprise architecture
• Develop user-friendly interfaces to metadata from within a warehouse
• Improve support for coding activities and classifications systems
• Allow users to submit proposed changes to the metadata in the IMDB
1818
The IMDB is Expanding The IMDB is Expanding
Expansion plans for this year are highlighted in blue.
1919
Multiple Formats Multiple Formats
Warehouse data and IMDB metadata can be combined and rendered in any format via converter modules.
2020
Metadata Architecture Metadata Architecture
Target architecture
Take data and metadata and render/publish it in a variety of modes via software based format converters.
Most of this architecture exists today at StatCan.
2121
ConclusionConclusion
• IMDB has the design required to support a full range of metadata functionality at StatCan
• A significant amount of variable-related information already exists in the IMDB
• Work for 2008/09– Completing the process for reviewing and approving
metadata changes from wiki users – Completing the Classification Management System– Mapping out a proposal for funding to develop a format
converter (DDI or SDMX)