oracle enterprise metadata management (oemm) · pdf fileibm datastage ibm discovery ibm...
TRANSCRIPT
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Oracle Enterprise Metadata Management (OEMM)
Hüseyin Celebi
Cloud Integration Specialist - EMEA
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
2
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
OEMM Overview
Oracle Confidential – Internal/Restricted/Highly Restricted 3
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
What do we mean by Data Governance and why do we need to do it?
4
“Data governance encompasses the people, processes, and information technology required to create a
consistent and proper handling of an organization's data across the business enterprise.” [Wikipedia]
Where did these
numbers come from?Do we trust what this
graph is telling us?
How confident are we
in this data?
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
OEMM and Data Governance
Oracle Big Data Governance 5
Data Governance
Metadata Management
Business Glossary
Data Profiling
Data Cleansing
Data Archiving
Data Privacy
PEOPLE
PROCESS TECHNOLOGY
…people and process
“Poor data quality is the
primary reason for 40% of
all business initiatives
failing to achieve their
targeted benefits”
Gartner, Measuring
Business Value of Data
Quality, 2011
“More than 50 percent of
data warehouse projects
will have limited
acceptance, or will be
outright failures, as a
result of a lack of attention
to data quality issues”
Gartner PR 2005
(http://www.gartner.com/new
sroom/id/492112)
…supported by tools and
capabilities
…in partnership with the business
OEMM
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Value of Enterprise Metadata Management
6
ETL
BIDashboards
App
ETL
ETL
How was sales figure calculated?
How do I organize my DW and
Reports
What reports use the mainframe
data? Sys Admin
Executive
BI Developer
Where did this data
come from?
Application User
What will happen if I change this
table?
CDC
Data Reservoir
Data Steward
Can I trust the sources of this
customer data?
ETL
Developer
Solves significant pain points for wide variety of business consumers and technical staff
I want to design an experiment to measure the
success of a signup page. What data do I have?
Data Scientist
GG
Which reports use this
customer data?Enterprise
Architect
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
It’s all about…
7
…in your data
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Oracle Enterprise Metadata Management (OEMM)
8
• Metadata Management – horizontal and semantic data lineage for all data sources
• Business Glossary – simple tools to catalog, link and collaborate on business terms
✓ Business Data Catalog
✓ Report to Source Lineage
✓ Impact Analysis
✓ Audit, Versioning & Diff Reports
✓ Social/Collaboration Features
✓ Annotations and Tagging
✓ Comprehensive Harvesting
✓ 3rd Party BI Metadata
✓ 3rd Party ETL Metadata
✓ 3rd Party DB Metadata
✓ 3rd Party Modeling Tools
✓ Big Data Metadata
✓ Metadata Standards
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Business User Understanding
✓ Lineage Analysis for reports
✓ Semantic Analysis
✓ Browsing/Searching/organizing of metadata across the enterprise
✓ Business Term maintenance and understanding it’s usage
Simplifying Enterprise Complexities
✓ Understanding past implementations by various teams on various platforms
✓ Fills the knowledge gap due to lack of documentation & employee turn-around
✓ Liberates metadata from its platform silos and consolidates for centralized access
Metadata Management Aids In…
Addressing the Data Mess
✓ Identifies duplicate metadata/data flows
✓ Checks for semantic consistency in data flows
✓ Analyzes metadata for detecting errors /standards
Providing Structure to Change Management
✓ Assists in application migration efforts by understanding old application metadata
✓ Assists ETL & BI development by metadata discovery of source & target systems
✓ Provides metadata standardization across the enterprise
✓ Keeps track of metadata changes over time
9
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 10
Data Lineage in Three Crucial Varieties
Vertical Lineage▪ For the Data Stewards▪ Links a business friendly set of
terms to the IT metadata and operational assets
▪ Links Business Glossary, Taxonomy, Ontology, Conceptual Models to Technical Metadata (Columns, Files, Objects etc)
Horizontal Column Level ▪ For the Architects▪ Links the data fields from Business
Intelligence Dashboards or Reports back to the Source DB/App Columns
▪ Schemas, BI View Layers, ETL Transformations, Calculations, etc.
Business Lineage▪ For the Business Users▪ Links a business friendly set of
Glossary terms to the BI reporting models (eg; Semantic Models, Universe, etc.)
▪ Links Business Glossary, Taxonomy, Ontology, Conceptual Models to Reporting Fields
BIZTERM
BIZTERM
REPORTFIELD
DBCOLS REPORT
FIELDSOURCE
COLS
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Customer Initiatives & Project Drivers
Realtime BI and Data Warehousing
Cloud Integration
Big Data Reservoir
All Data Integration Use Cases
Oracle Enterprise Metadata Mgmnt
Oracle Enterprise Metadata MgmntOracle Enterprise Metadata Mgmnt
11
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
High Level OverviewMetadata sources , data flow and semantic links
•Business Glossary•Metadata harvested into models
•Databases•Data Modeling•ETL•Business Intelligence•Big Data•… many more
•Data flow•Semantic flow
Oracle Confidential, under Non-Disclosure 12
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Business GlossaryBusiness Glossary is a collection of business terms and relationship definitions across enterprise. It provides a common understanding of terms to employees and business partners.
The Business Glossary can be used to document• Business Terms• Relationship between Business Terms• Business Rules • Domains • Data Entities logical/physical • Attributes across IT collaboratively. • Relationship of Terms with physical data assets
Oracle Confidential, under Non-Disclosure 13
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Business Glossary
Business Term
•Description•Properties•Custom Attribute value (see “Line Of Business”)•Documentation•Relationship with other terms•Comments
Oracle Confidential, under Non-Disclosure 14
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Business Glossary
Business Rules and Policies
• Defined in Business Terms• Linked to supported document for further details• Work flow from draft to approval• Linked to physical assets
Oracle Confidential, under Non-Disclosure 15
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential, under Non-Disclosure 16
Glossary Workflow
•Drafting•Reviewing•Approving•Publishing
Business Glossary
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Harvesting Models
Oracle Confidential, under Non-Disclosure 17
Harvesting bridges and Metadata Stitching
• 150+ metadata sources• One click harvest• Fully automated metadata stitching
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Searching for Metadata
Easy to use search for finding metadata of interest
• Tables/Columns• Attributes• Data Models• Reports
•Search UI is customizable •Reporting and analysis after finding metadata
Oracle Confidential, under Non-Disclosure 18
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Browsing Metadata
Oracle Confidential, under Non-Disclosure 19
Browsing is another way for finding metadata of interest
• Tables/Columns• Attributes• Data Models• Reports
•Reporting and analysis after finding metadata
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Ad-hoc reporting of metadata
Excel like grid view for
•Listing•Sorting•Filtering•Updating•Sharing•Exporting
Oracle Confidential, under Non-Disclosure 20
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Ad-hoc reporting of metadata
Oracle Confidential, under Non-Disclosure 21
Filtering result set
Add/remove filters on•Metadata attributes•Custom attributes•Labels
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Ad-hoc reporting of metadata
Documenting metadata
• Any number of custom attributes• Update metadata for documentation• Share metadata
Oracle Confidential, under Non-Disclosure 22
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Ad-hoc reporting of metadata
Oracle Confidential, under Non-Disclosure 23
Documenting metadata
• Any number of custom attributes• Update metadata for documentation• Share metadata
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Browsing Data Modeling Diagram
Creating and Browsing diagrams
•Diagram harvested from Modeling tools•Diagrams can be created if it doesn’t exist. Full feature editor available.•Fully interactive diagram•Viewing tools for navigating complex diagrams•Lineage and Impact analysis can be initiated from the diagram
Oracle Confidential, under Non-Disclosure 24
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Social Collaboration for Stewardship Teams
Oracle Confidential, under Non-Disclosure 25
Collaboration
•Discussion Threads•Labeling•Sharing•Bookmarking•eMailing
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Lineage Analysis
Lineage analysis from everywhere
•Search results•Browsing list•Data Model diagrams
Lineage Results shown as•Textual overview•Graphical overview•Export for further analysis
Ability to trim lineage results by excluding interim details.
Oracle Confidential, under Non-Disclosure 26
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Lineage Analysis
Oracle Confidential, under Non-Disclosure 27
Lineage analysis from everywhere
•Search results•Browsing list•Data Model diagrams
Lineage Results shown as•Textual overview•Graphical overview•Export for further analysis
Ability to trim lineage results by excluding interim details.
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Lineage Analysis
Oracle Confidential, under Non-Disclosure 28
Drilling into Lineage Analysis
•Source/target attributes•Transformations•Intermediate attributes•Control flows•Drill through for more details
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Lineage Analysis
Oracle Confidential, under Non-Disclosure 29
Tracing Lineage Analysis
•Source to target data flow•Transformations overview•Drill into for more details
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Impact Analysis
Oracle Confidential, under Non-Disclosure 30
Impact analysis from everywhere
•Search results•Browsing list•Data Model diagrams
Impact Results shown as•Textual overview•Graphical overview•Export for further analysis
Ability to trim Impact results by excluding interim details.
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Semantic Analysis
Oracle Confidential, under Non-Disclosure 31
Semantic analysis from Glossary
• Find linked physical assets • Graphical view of semantic relationship• Semantic distance• Drill into for more details
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Semantic Analysis
Oracle Confidential, under Non-Disclosure 32
Semantic analysis from search results
• Find linked physical assets • Graphical view of semantic relationship• Semantic distance• Drill into for more details
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Semantic Analysis
Oracle Confidential, under Non-Disclosure 33
Semantic analysis from search results
• Find linked physical assets • Graphical view of semantic relationship• Semantic distance• Drill into for more details
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Versioning with Model Comparisons
Oracle Confidential, under Non-Disclosure 34
Easily group and compare data models from similar databases to determine crucial differences
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Open Source, Standards and Third Party (150+ sources)
Oracle Confidential, under Non-Disclosure 35
▪ Apache HDFS
▪ Apache Hive Metastore
▪ Apache Hive Server
▪ Borland
▪ CA Component Modeler
▪ CA COOL (previously Sterling)
▪ CA Erwin
▪ CA Gen
▪ Cloudera Enterprise
▪ Cloudera Impala
▪ Database (JDBC)
▪ DataStax
▪ Embarcadero
▪ EMC ProActivity
▪ GentleWare
▪ Google BigQuery
▪ Grandite
▪ Hadapt Hive
▪ Hortonworks Hive
▪ IBM Cognos
▪ IBM DB2
▪ IBM Data Architect
▪ IBM DataStage
▪ IBM Discovery
▪ IBM Federation Server
▪ IBM Information Server
▪ IBM Lotus Notes
▪ IBM Netezza
▪ IBM Rational Rose
▪ IBM Rational Architect
▪ IBM Telelogic
▪ Informatica Developer
▪ Informatica PowerCenter
▪ MapR Hadoop Hive
▪ MicroFocus
▪ Microsoft Office Excel
▪ Microsoft Visio
▪ Microsoft SQL Server
▪ Microsoft SSAS
▪ Microsoft SSIS
▪ Microsoft SSRS
▪ Microsoft Visual Studio
▪ Microstrategy
▪ Magic Draw
▪ OMG CWM Standard
▪ OMG UML Standard
▪ Oracle BI Answers
▪ Oracle BI Enterprise Edition
▪ Oracle BI Server
▪ Oracle DAC
▪ Oracle Data Integrator
▪ Oracle Data Modeler
▪ Oracle Database
▪ Oracle Designer
▪ Oracle Hyperion EIS
▪ Oracle Warehouse Builder
▪ Pivotal Greenplum
▪ PostgreSQL
▪ SAP BO Crystal Reports
▪ SAP BO Designer
▪ SAP BO Desktop Intelligence
▪ SAP BO Information Design Tool
▪ SAP BO Repository
▪ SAP Sybase PowerDesigner
▪ SAP Sybase ASE Database
▪ SAS OLAP Server (via ODBO)
▪ Select SE
▪ Sparx Architect
▪ Tableau
▪ Talend
▪ Teradata
▪ Tigris
▪ Visible
▪ W3C XSD Schema
Metadata Harvesting (Glossary, Lineage & Impact Analysis)Key Standards
Oracle Confidential 36