advanced querying olap data warehousing. database applications transaction processing –online...

Advanced Querying

OLAP

Data Warehousing

Database Applications

• Transaction processing– Online setting– Supports day-to-day operation of business

• Decision support– Offline setting– Strategic planning (statistics)

Transaction Processing

Transaction processing

• Operational setting• Up-to-date = critical

• Simple data

• Simple queries

Flight reservations

• ticket sales• do not sell a seat

twice• reservation, date,

name• Give flight details of X

List flights to Y

Transaction Processing

• Database must support– simple data

• tables

– simple queries• select from where …

– consistency & integrity CRITICAL– concurrency

• Relational databases, Object-Oriented, Object-Relational

Decision Support

Decision support

• Off-line setting• « Historical » data• Summarized data• Different databases

• Statistical queries

Flight company

• Evaluate ROI flights• Flights of last year• # passengers on line L• Passengers, fuel costs,

maintenance info• Average % of seats

sold/month/destination

A decision support DB that is maintained separately from the organization’s operational databases.

Why Separate Data Warehouse?• High performance for both systems

– DBMS— tuned for OLTP • access methods, indexing, concurrency control, recovery

– Warehouse—tuned for OLAP • complex OLAP queries, multidimensional view, consolidation.

• Different functions and different data– Missing data: Decision support requires historical data which

operational DBs do not typically maintain– Data consolidation: DS requires consolidation (aggregation,

summarization) of data from heterogeneous sources– Data quality: different sources typically use inconsistent data

representations, codes and formats which have to be reconciled

Data Warehouse

Three-Tier Architecture

DataWarehouse

ExtractTransformLoadRefresh

OLAP Engine

Monitor&

IntegratorMetadata

Data Sources Front-End Tools

Serve

Data Marts

Operational DBs

other

sources

Data Storage

OLAP Server

Analysis

Query/Reporting

Data Mining

ROLAPServer

OLAP

• OLAP = OnLine Analytical Processing– Online = no waiting for answers

• OLAP system = system that supports analytical queries that are dimensional in nature.

This Lecture

• Examples of decision support queries

• Data Cubes– Conceptual data model– Typical operations

• Implementation– ROLAP vs MOLAP– Indexing structures

• SQL:1999 support for OLAP

Examples of Queries

• Flight company: evaluate ticket sales– give total, average, minimal, maximal amount– per date: week, month, year– by destination/source port/country/continent– by ticket type– by # of connections– …

Characteristics

• One special attribute: amount measure

• Other attributes: select relevant regions dimensions

Different levels of generality (month, year, …)

hierarchies

• Measure data is summarized: sum, min, max, average

aggregations

Supermarket example

• Evaluate the sales of products– Product cost in $– Customer: ID, city, state, country, – Store: chain, size, location, – Product: brand, type, …– …

• What are the measure and dimensional attributes, where are the hierarchies?

measure

Dim.hierarchies

Why dimensions?

customer

store

product

Cost in $

• Multidimensional view on the data

Cross Tabulation

• Cross-tabulations are highly useful– Sales of clothes JuneAugust ‘06

Blue Red Orange Total

June 51 25 158 234

July 58 20 120 198

August 65 22 51 138

Total 174 67 329 570

Product: color

Date:month,JuneAugust2006

Data cubes

• Extension of Cross-Tables to multiple dimensions

• Conceptual notion

Blue Red Orange Total

June 51 25 158 234

July 58 20 120 198

August 65 22 51 138

Total 174 67 329 570

Dimensions

Data Points/1st level of aggregation

Aggregatedw.r.t. X-dim

Aggregatedw.r.t. Y-dim

Aggregatedw.r.t. X and Y

Data Cubes

Date

Produ

ct

Cou

ntr

ysum

sum TV

VCRPC

1Qtr 2Qtr 3Qtr 4Qtr

Ireland

France

Germany

sum

Data Cubes

• Base cuboid = n-dimensional cube with n number of dimensions

• The top most 0-D cuboid, which holds the highest-level of summarization, is called the apex cuboid

• The lattice of cuboids forms a data cube

Lattice of Cuboidsall

product, date, country

product date country

product, date product, countrydate, country

Operations with Data Cubes

Scenario:

• Before starting the analysis task:– what data?

• select a few relevant dimensions• define hierarchy• aggregation functions of interest

– Pre-materialize• load data• compute counts/max, min, avg, … on beforehand

Operations with Data Cubes

• What operations can you think of an analyst might find useful? (e.g., store)

Operations with Data Cubes

• What operations can you think of that an analyst might find useful? (e.g., store)– only look at stores in the Netherlands– look at cities instead of individual stores– look at the cross-table for product-date– restrict analysis to 2006, product O1– go back to a finer granularity at the store level

Roll-Up

• Move in one dimension from a lower granularity to a higher one– store city– cities country– product product type

Drill-down

• Move in one dimension from a higher granularity to a lower one– city store– country cities– product type product

• Drill-through:– go back to the original, individual data records

Pivoting

• Change the dimensions that are “displayed”; select a cross-tab.– look at the cross-table for product-date– display cross-table for date-customer

Slice & dice

• Select a part of the cube by restricting one or more dimensions– restrict analysis to “city = Eindhoven”

Summary of Concepts

• Cube: Multidimensional view on data– dimensional attributes– measure attribute

• Operations:– roll-up/drill-down– pivoting– slice and dice

Implementation

• To make query answering more efficient: consolidate (materialize) aggregations

• Obvious implementation: multidimensional array.– Fast lookup: cell(prod. p, date d, prom. pr):

• look up index of p1, index of d, index of pr:

index = (p x D x PR) + (d x PR) + pr

Implementation

• Multidimensional array– obvious problem: sparse data

can easily be solved, though.

Example:

binary search tree, key on index

hash table.

Implementation

• However: very quickly people were confronted with the Data Explosion Problem

Consolidating the summaries blows up the data enormously !

Reasons are often misunderstood and confusing.

Data Explosion Problem

• Why?

Suppose:– n dimensions, every dimension has d values– dn possible tuples.

– Number of cells in the cube: (d+1)n

– So, this is not the problem

Data Explosion Problem

• Why?

Suppose– n dimensions, every dimension has d values– every dimension has a hierarchy

– most extreme case: binary tree

2d possibilities/dimension

Data Explosion Problem

• Why?

Suppose– n dimensions, every dimension has d values– every dimension has a hierarchy– most extreme case: binary tree

2d possibilities/dimension 2n x dn cells

Only partial explanation (factor 2n comes from an extremely pathological case)

Data Explosion Problem

• Why?– The problem is that most data is not dense,

but sparse.– Hence, not all dn combinations are possible.

Example: 10 dimensions with 10 values– 10 000 000 000 possibilities

Suppose « only » 1 000 000 are present

Data Explosion Problem

Example: 10 dimensions with 10 values– 10 000 000 000 possibilities

Suppose « only » 1 000 000 are present

Every tuple increases count of 210 cells !

With hierarchies: effect even worse!

If every hierarchy has 5 items:

510 = 9 765 625 cells!

View Selection Problem• Suffices to precompute some aggregates, and

compute others on demand.– aggregate on (item-name, color) from an aggregate

on (item-name, color, size) – For all but a few “non-decomposable” aggregates

such as median

• Several optimizations for computing multiple aggregates– Compute aggregate on (item-name, color) from an

aggregate on (item-name, color, size)

– Compute aggregates on (item-name, color, size), (item-name, color) and (item-name) in single DB sort

View Selection Problemall

product, date, country

product date country

product, date product, countrydate, country

View Selection Problemall

product, date, country

product date country

product, date product, countrydate, country

Which views to select:hard research problem !

Implementation

Nowadays systems can be divided in three categories:– ROLAP (Relational OLAP)

• OLAP supported on top of a relational database

– MOLAP (Multi-Dimensional OLAP)• Use of special multi-dimensional data structures

– HOLAP: (Hybrid)• combination of previous two

ROLAP

• Cubes can easily be represented in relational tables: special value “all”

Month Prod. Cust. PriceJan p1 c1 10Jan p2 c1 8Jan p1 c2 10Feb p1 c1 9

…all p1 c1 102Jan all c1 18Jan p1 all 1 230all all c1 4 235

…all all all 1 253 458

ROLAP

• Typical database scheme:– star schema

• fact table is central• links to dimensional tables

– Extensions:• snowflake schema

– dimensions have hierarchy/extra information attached

• Star constellation– multiple star schemas sharing dimensions

Example of a Star Schema

Order NoOrder No

Order DateOrder Date

Customer NoCustomer No

Customer Customer NameName

Customer Customer AddressAddress

CityCity

SalespersonIDSalespersonID

SalespersonNaSalespersonNameme

CityCity

QuotaQuota

OrderNOOrderNO

SalespersonIDSalespersonID

CustomerNOCustomerNO

ProdNoProdNo

DateKeyDateKey

CityNameCityName

QuantityQuantity

Total Price

ProductNOProductNO

ProdNameProdName

ProdDescrProdDescr

CategoryCategory

CategoryDescriptionCategoryDescription

UnitPriceUnitPrice

DateKeyDateKey

DateDate

CityNameCityName

StateState

CountryCountry

OrderOrder

CustomerCustomer

SalespersoSalespersonn

CityCity

DateDate

ProductProduct

Fact TableFact Table

Example of a Snowflake Schema

Order NoOrder No

Order DateOrder Date

Customer NoCustomer No

Customer Customer NameName

Customer Customer AddressAddress

CityCity

SalespersonIDSalespersonID

SalespersonNaSalespersonNameme

CityCity

QuotaQuota

OrderNOOrderNO

SalespersonIDSalespersonID

CustomerNOCustomerNO

ProdNoProdNo

DateKeyDateKey

CityNameCityName

QuantityQuantity

Total Price

ProductNOProductNO

ProdNameProdName

ProdDescrProdDescr

CategoryCategory

CategoryCategory

UnitPriceUnitPrice

DateKeyDateKey

DateDate

MonthMonth

CityNameCityName

StateState

CountryCountry

OrderOrder

CustomerCustomer

SalespersoSalespersonn

CityCity

DateDate

ProductProduct

Fact TableFact Table

CategoryNaCategoryNameme

CategoryDeCategoryDescrscr

MontMonthh

YearYear YearYear

StateNameStateName

CountryCountry

CategoryCategory

StateState

MonthMonth

YearYear

branch_keybranch_namebranch_type

time_keydayday_of_the_weekmonthquarteryear

Measures

Branch

Time

item_keyitem_namebrandtypesupplier_key

Item

location_keystreetcityProvince/streetcountry

Location

Sales Fact Table

Avg_sales

Euros_sold

Unit_sold

Location_key

Branch_key

Item_key

Time_key

shipper_keyshipper_namelocation_keyshipper_type

shipper

unit_shipped

Euros_sold

to_location

from_location

shipper_key

Item_key

Time_key

Shipping Fact TableMultiple fact tables share dimension tables

Example of Fact Constellation

SQL 1999 support for OLAP

• see other set of slides