Motivation: Why data warehouse? What is a data warehouse? [JH] ·…

Download Motivation: Why data warehouse? What is a data warehouse? [JH] ·…

Post on 07-Nov-2018




0 download

Embed Size (px)


<ul><li><p>Data Warehousing and OLAPData Warehousing and OLAPLecture 2/DMBI/IKI83403T/MTI/UI</p><p>Yudho Giri Sucahyo, Ph.D, CISA ( of Computer Science, University of Indonesia</p><p>ObjectivesObjectivesMotivation: Why data warehouse?What is a data warehouse?Why separate DW?y pConceptual modeling of DWData MartData MartData Warehousing ArchitecturesData Warehouse DevelopmentData Warehouse DevelopmentData Warehouse VendorsR l DWReal-time DW</p><p>2</p><p>Motivation: Why data warehouse?Motivation: Why data warehouse?Construction of data warehouses (DW) involves data cleaning and data integration important preprocessing step for data mining (DM).DW provide OLAP for the interactive analysis of multidimensional data, which facilitates effective DM.,Data mining functions can be integrated with OLAP operations to enhance interactive mining of knowledge.operations to enhance interactive mining of knowledge.DW will provide an effective platform for DM.Whil DW t i t t d DM DW t While DWs are not requirements to do DM, DW store massive amounts of data that can be uses for DM. [DO]</p><p>3</p><p>What is a data warehouse? [JH]What is a data warehouse? [JH]Defined in many different ways, but not rigorously.</p><p>A decision support database that is maintained separately from the organizations ODB.Support information processing by providing a solid platform of consolidated, historical data for analysis.</p><p>A data warehouse is a subject-oriented, integrated, time-variant, and nonvolatile collection of data in support of managements decision-making process. W. H. InmonCase Study 2: Continental Airlines flies high with its real-time data warehouse</p><p>4</p></li><li><p>What is a data warehouse? [ET]What is a data warehouse? [ET]Data warehouseA physical repository where relational data are specially organized to provide enterprise-wide, cleansed data in a standardized format.Characteristics</p><p>Subject oriented, Integrated, Time Variant, Non-volatileWeb-based, Relational/multidimensional, Client/server, Real-timeInclude metadata</p><p>Data warehousingProcess of constructing and using data warehouses.Requires data integration, data cleaning, and data consolidation.</p><p>5</p><p>Subject OrientedSubject Oriented</p><p>Organized around major subjects, such as Organized around major subjects, such as customer, product, sales.P id i l d i i d Provide a simple and concise view around particular subject issues by excluding data that are not useful in the decision support process.Focusing on the modeling and analysis of data Focusing on the modeling and analysis of data for decision makers, not on daily operations or transaction processingtransaction processing.</p><p>6</p><p>IntegratedIntegratedIntegrate multiple, heterogeneous data sources</p><p>Relational databases, flat-files, on-line transaction records</p><p>Data cleaning and data integration techniques are g g qapplied</p><p>Ensure consistency in naming conventions, encoding Ensure consistency in naming conventions, encoding structures, attribute measures, etc. among different data sourcessources</p><p>E.g., Hotel price: currency, tax, breakfast covered, etc.</p><p>Wh d i d h h i i dWhen data is moved to the warehouse, it is converted.</p><p>7</p><p>Time VariantTime VariantThe time horizon for the data warehouse is significantly longer than that of operational systems.</p><p>Operational database: current value data.Operational database: current value data.</p><p>Data warehouse data: provide information from a historical perspective (e g past 5-10 years)perspective (e.g., past 5-10 years)</p><p>Every key structure in the data warehouse</p><p>Contains an element of time, explicitly or implicitly</p><p>But the key of operational data may or may not contain time element.</p><p>8</p></li><li><p>Non volatileNon-volatileA physically separate store of data transformed from the p y y p</p><p>operational environment.</p><p>O i l d f d d i h d Operational update of data does not occur in the data </p><p>warehouse environment.</p><p>Does not require transaction processing, recovery, and </p><p>concurrency control mechanismsy</p><p>Requires only two operations in data accessing: </p><p>i i i l l di f d d f dinitial loading of data and access of data.</p><p>9</p><p>Data Warehouse vs Heterogeneous DBMSData Warehouse vs. Heterogeneous DBMSTraditional heterogeneous DB integration: </p><p>Build wrappers/mediators on top of multiple, heterogeneous databases. Ex: IBM Data Joiner, Informix DataBlade</p><p>Q d i h Query driven approach: </p><p>When a query is posed to a client site, a metadata-dictionary is used to translate the query into queries appropriate for the individual to translate the query into queries appropriate for the individual heterogeneous sites involved. There queries are then mapped and sent to local query processors. The results returned from the different </p><p> d l b l sites are integrated into a global answer set.</p><p>Complex information filtering and integration processes, compete forresourcesresources.</p><p>Inefficient and potentially expensive for frequent queries, especially for </p><p>queries requireing aggregations.q q g gg g</p><p>10</p><p>Data Warehouse vs Heterogeneous DBMS (2)Data Warehouse vs. Heterogeneous DBMS (2)Using DW update-driven approach</p><p>Information from multiple, heterogeneous sources is integrated in advance and stored in a warehouse for direct querying and analysis. </p><p>Unlike OLTP DW do not contain the most current informationUnlike OLTP, DW do not contain the most current information.DW brings high performance to the integrated heterogeneous DB system since data are copied preprocessed integrated DB system since data are copied, preprocessed, integrated, annotated, summarized, and restructured into one data store.Query processing in DW does not interfere with the processing Query processing in DW does not interfere with the processing at local sourcesDW can store and integrate historical information and support g ppcomplex multidimensional queries.</p><p>11</p><p>DW vs ODBDW vs. ODBMajor task of ODB OLTP:</p><p>Day-to-day operations: purchasing, inventory, banking, manufacturing, payroll, registration, accounting, etc.</p><p>DW f d l i d d i i ki OLAPDW serve for data analysis and decision making OLAPDistinct Features (OLTP vs. OLAP)</p><p>U d i i kUser and system orientation: customer vs. marketData contents: current, detailed vs. historical, consolidatedDatabase design: ER + application vs star + subjectDatabase design: ER + application vs. star + subjectView: current, local vs. evolutionary, integratedAccess patterns: update vs. read-only but complex queriesAccess patterns: update vs. read only but complex queries</p><p>12</p></li><li><p>OLTP vs OLAPOLTP vs OLAPOLTP OLAP</p><p>users Clerk IT professional Knowledge workerusers Clerk, IT professional Knowledge workerfunction day to day operations decision supportDB design application-oriented subject-orienteddata current, up-to-date</p><p>detailed, flat relationalisolated</p><p>historical,summarized, multidimensionalintegrated, consolidated</p><p>usage repetitive ad-hocaccess read/write</p><p>index/hash on prim. keylots of scans</p><p>unit of work short, simple transaction complex query# records accessed tens millions#users thousands hundreds#users thousands hundredsDB size 100MB-GB 100GB-TBmetric transaction throughput query throughput, response</p><p>13</p><p>Why Separate DW?Why Separate DW?</p><p>High performance for both systems:g p yDBMS tuned for OLTP: access methods, indexing, concurrency control, recoveryWarehouse tuned for OLAP: complex OLAP queries,computation of large groups of data at summarized levels,multidimensional view, consolidation.,</p><p>Processing OLAP queries in operational databases would degrade the performance of operational tasks.In ODB, concurrency control and recovery mechanisms (locking, logging) are required to ensure the consistency </p><p>d b f i and robustness of transactions. OLAP read only access. No need for concurrency control and recoverycontrol and recovery.</p><p>14</p><p>Why Separate DW? (2)Why Separate DW? (2)Different functions and different data:</p><p>missing data: Decision support requires historical data which operational DBs do not typically maintain. So, data in ODB is usually far from complete for decision making. y p gdata consolidation: DS requires consolidation (aggregation, summarization) of data from heterogeneous sources. ODB </p><p>t i d t il d d t (t ti ) hi h d t b contain detailed raw data (transactions) which need to be consolidated before quality: different sources typically use inconsistent data q y yp yrepresentations, codes and formats which have to be reconciled.</p><p>15</p><p>Conceptual Modeling of DWConceptual Modeling of DWData Cube: </p><p>see TSBD Lecture Notes on Visualization of Data Cubes</p><p>M d li d t h di i &amp; tModeling data warehouses: dimensions &amp; measurementsStar schema: A single object (fact table) in the middle connected to a number of objects (dimension tables one for each to a number of objects (dimension tables, one for each dimension).Snowflake schema: A refinement of star schema where the dimensional hierarchy is represented explicitly by normalizing the dimension tables.Fact constellations: Multiple fact tables share dimension tables.</p><p>Also known as galaxy schema</p><p>16</p></li><li><p>Example of Star SchemaExample of Star SchemaDate</p><p>Product</p><p>DayMonthYear</p><p>Sales Fact Table</p><p>Date</p><p>ProductNoProdNameProdDescCDate</p><p>Product</p><p>Store</p><p>CategoryQOH</p><p>Store</p><p>CustIdC tN</p><p>CustStore </p><p>CustomerStoreIDCityState CustName</p><p>CustCityCustCountry</p><p>unit_sales</p><p>dollar_sales</p><p>StateCountryRegion</p><p>Yen_sales</p><p>MeasurementsPotensi Redundansi</p><p>Bandung, Bogor keduanya </p><p>17</p><p>ada di Jawa Barat</p><p>Snowflake SchemaSnowflake SchemaProductYear</p><p>Day</p><p>Date Sales Fact TableProductNoProdNameProdDesc</p><p>MonthYear</p><p>MonthYear</p><p>DayMonth Date</p><p>Product</p><p>ProdDescCategoryQOH</p><p>Year</p><p>CustId</p><p>CustStore </p><p>CustomerCity StoreIDCit</p><p>Store</p><p>CustIdCustNameCustCityCustCountry</p><p>unit_sales</p><p>dollar sales</p><p>CityState</p><p>State</p><p>State</p><p>City</p><p>CustCountry_</p><p>Yen_salesCountryRegion</p><p>CountryStateCountry</p><p>18</p><p>MeasurementsRegion</p><p>View of Warehouses and HierarchiesView of Warehouses and Hierarchies</p><p>Importing data</p><p>Table Browsing</p><p>Dimension creation</p><p>Dimension browsing</p><p>Cube buildingg</p><p>Cube browsing</p><p>19</p><p>Data CubeData Cube</p><p>Total annual salesD t Total annual salesof TV in U.S.A.</p><p>DatesumTV</p><p>PC</p><p>1Qtr 2Qtr 3Qtr 4QtrU S A</p><p>ry</p><p>sumVCR</p><p>PC U.S.A</p><p>Canada</p><p>Cou</p><p>nt</p><p>Ca ada</p><p>Mexico</p><p>sum</p><p>20</p></li><li><p>Data CubeData Cube</p><p>VisualizationOLAP capabilities</p><p>21</p><p>pInteractive manipulation</p><p>Typical OLAP OperationsTypical OLAP OperationsRoll up (drill-up): summarize data</p><p>by climbing up hierarchy or by dimension reductionby climbing up hierarchy or by dimension reductionDrill down (roll down): reverse of roll-up</p><p>from higher level summary to lower level summary or detailed data or from higher level summary to lower level summary or detailed data, or introducing new dimensions</p><p>Slice and dice: project and select</p><p>Pivot (rotate): reorient the cube, visualization, 3D to series of 2D planes.</p><p>Other operationsd ill i l i ( ) th f t t bldrill across: involving (across) more than one fact table.drill through: through the bottom level to its back-end relational tables.</p><p>More info: More info:,,</p><p>22</p><p>Data MartData MartDW collects information about subjects that span the entire organization, such as customers, products, sales, assets, and personnel. Its scope is enterprise-wide.For DW, fact constellation schema is commonly used since it can model multiple, interrelated subjects.Data Mart is a subset of a DW, focuses on a particular subject. Its scope is department-wide. Typically, a data mart </p><p> f l b ( k consisting of a single subject area (e.g. marketing, operations).For Data Mart, star or snowflake schema are commonly used since both are geared towards modeling single </p><p>bj t lth h th t h i lsubjects, although the star schema is more popular.23</p><p>Data MartData MartA data mart can be either dependent or independent.A dependent data mart is a subset that is created directly from the DW.</p><p>Consistent data modelProviding quality dataDW must be constructed firstEnsures that the user viewing the same version of the data that </p><p> d b ll h d h are accessed by all other data warehouse users</p><p>An independent data mart is a small warehouse designed f d d i i EDWfor a department, and its source is not an EDW.</p><p>24</p></li><li><p>Data Warehousing Process OverviewData Warehousing Process Overview</p><p>25</p><p>Data Warehousing Process OverviewData Warehousing Process OverviewThe major components of a data warehousing process </p><p>Data sources Legacy systems, external data providers (e.g. BPS), OLTP, ERP Systems</p><p>Data extraction Data loading Comprehensive database Metadata Middleware tools </p><p>26</p><p>Data Warehousing ArchitecturesData Warehousing Architectures</p><p>27</p><p>Data Warehousing ArchitecturesData Warehousing Architectures</p><p>28</p></li><li><p>Data Warehousing ArchitecturesData Warehousing Architectures</p><p>29</p><p>Data Warehousing ArchitecturesData Warehousing Architectures</p><p>30</p><p>Data Integration and the ETL ProcessData Integration and the ETL ProcessVarious integration technologies:</p><p>Enterprise Application Integration (EAI)A technology that provides a vehicle for pushing data from source </p><p>t i t d t h systems into a data warehouse Integrating application functionality and is focused on sharing functionality across systemsTraditionally, API. Nowadays, SOA (web services).</p><p>Enterprise Information Integration (EII)An evolving tool space that promises real-time data integration from </p><p>a variety of sources, such as relational databases, Web services, and multidimensional databases A mechanism for pulling data from source systems to satisfy a request for information. </p><p>31</p><p>Data Integration and the ETL ProcessData Integration and the ETL ProcessETL</p><p>60-70% of the time in a data-centric project.Extraction: Reading data from one or more databasesTransformationTransformation</p><p>Converting the extracted data from its previous form into the form in which it needs to be so that it can be placed into a DW</p><p>LoadPutting the </p><p>d data into the DW</p><p>32</p></li><li><p>Data Warehouse DevelopmentData Warehouse DevelopmentDirect benefits</p><p>Allowing end users to perform extensive analysis in numerous waysA f ( f A consolidated view of corporate data (i.e a single version of the truth)Better and more timely informationBetter and more timely informationEnhanced system performance. DW frees production processing because some operational system reporting processing because some operational system reporting requirements are moved to DSSSimplification of data access</p><p>33</p><p>Data Warehouse DevelopmentData Warehouse DevelopmentSome best practices for implementing a DW (Weir, 2002):</p><p>Project must fit with corporate strategy and business objectivesThere must be complete buy-in to the project by executives, managers and usersmanagers, and usersIt is important to manage user expectations about the completed projectThe data warehouse must be built incrementallyBuild in adaptabilityM d b b h IT d b i f i lManaged by both IT and business professionalsDevelop a business/supplier relationshipO l l d d t th t h b l d d f lit Only load data that have been cleansed and are of a quality understood by the organizationDo not overlook training requirementsDo not overlook training requirementsBe politically aware </p><p>34</p><p>Data Warehouse Ven...</p></li></ul>