Slowly Changing Dimension: Categories

By: Prof. Sunita Sahu Assistant Prof, VESIT,Mumbai

Slowly Changing Dimension: Categories Dimensions that change slowly over time,

rather than changing on regular schedule, time-base.

In Data Warehouse there is a need to track changes in dimension attributes in order to report historical data.

The usual changes to dimension tables are classified into three types Type 1 Type 2 Type 3

2

Example3

Order factProduct Key

Time KeyCustomer KeySalesperson KeyOrder DollarsCost Dollars

Margin DollarsSale Units

CustomerCustomer Key

Customer NameCustomer CodeMartial Status

AddressStateZip

SalespersonSalesperson KeySalesperson Name

Territory NameRegion Name

ProductProduct Key

Product NameProduct CodeProduct Line

Brand

TimeTime Key

DateMonth

QuarterYear

Type 1 Changes: Error Correction

Usually relate to corrections of errors in the source system.

For example, the customer dimension: change in name because of spelling mistake

4

Type 1 Changes, cont.

General Principles for Type 1 changes:

Usually, the changes relate to correction of errors in the source system

Sometimes the change in the source system has no significance

The old value in the source system needs to be discarded

The change in the source system need not be preserved in the DWH

5

Applying Type 1 changes

Overwrite the attribute value in the dimension table row with the new value

The old value of the attribute is not preserved No other changes are made in the dimension

table row. The key of this dimension table or any other

key values are not affected. Easiest to implement.

6

Before the change: Customer_ID Customer_Name Customer_Type 1 Cust_1

Corporate

After the change: Customer_ID Customer_Name Customer_Type 1 Cust_1

Retail

Type 2 Changes:

Let’s look at the martial status of customer. One the DWH’s requirements is to track orders

by martial status All changes before 11/10/2004 will be under

Martial Status = Single, and all changes after that date will be under Martial Status = Married

We need to aggregate the orders before and after the marriage separately

8

Type 2 Changes, cont.

General Principles for Type 2 changes: They usually relate to true changes in source

systems. There is a need to preserve history in the DWH. This type of change partitions the history in the

DWH. Every change for the same attributes must be

preserved.

9

Type 2 Implementation

The steps: Add a new dimension table row with the new

value of the changed attribute An effective date will be included in the

dimension table There are no changes to the original row in the

dimension table The key of the original row is not affected The new row is inserted with a new surrogate

key

10

Before the change: Customer_ID

Customer_Name

Customer_Type

Start_Date End_Date

1 Cust_1 Corporate 22-07-2010 31-12-9999

Customer_ID

Customer_Name

Customer_Type

Start_Date End_Date

1 Cust_1 Corporate 22-07-2010 31-12-9999

2 Cust_1 Retail 22-07-2010 31-12-9999

Type 2 Example

Type 3 Changes

Type 3 Slowly Changing Dimension, there will be two columns to indicate the particular attribute of interest, one indicating the original value, and one indicating the current value.

There will also be a column that indicates when the current value becomes active.

Not common at all Time-consuming We want to track history without lifting heavy

burden. There are many soft changes and we don’t care for

the “far” history

12

Type 3 Changes

General Principles: They usually relate to “soft” or tentative

changes in the source systems There is a need to keep track of history with old

and new values of the changes attribute They are used to compare performances across

the transition They provide the ability to track forward and

backward

13

Type 3

No new dimension row is needed The existing queries will seamlessly switch to

the current value. Any queries that need to use the old value

must be revised accordingly. The technique works best for one soft change

at a time. If there is a succession of changes, more

sophisticated techniques must be advised

14

Customer Key Name State 1001 Williams New York

After Williams moved from New York to Los Angeles, the original information gets updated, and we have the following table (assuming the effective date of change is February 20, 2010):

Customer Key Name Original State Current State Effective Date

1001 Williams New York Los Angeles 20-FEB-2010

Type 3

Advantages

This does not increase the size of the table, since new information is updated.

This allows us to keep some part of history.

Disadvantages

Type 3 will not be able to keep all history where an attribute is changed more than once. For example, if Williams later moves to Texas on December 15, 2003, the Los Angeles information will be lost.

Type 3

Large Dimension Table

Dimension table is large based on two factors. very deep: that is, the dimension has a very

large number of rows. Very wide: that is, the dimension may have a

large number of attributes or columns. In a data warehouse, typically the customer and

product dimensions are likely to be large. Such customer dimension tables may have as

many as 100 million rows. The product dimension of large retailers is also quite huge.

Junk Dimension

The junk dimension is simply a structure that provides a convenient place to store the junk attributes. It is just a collection of random transactional codes, flags and/or text attributes that are unrelated to any particular dimension.

In OLTP tables that are full of flag fields and yes/no attributes, many of which are used for operational support and have no documentation except for the column names and the memory banks of the person who created them. Not only do those types of attributes not integrate easily into conventional dimensions such as Customer, Vendor, Time, Location, and Product, but you also don’t want to carry bad design into the data warehouse.However, some of the miscellaneous attributes will contain data that has significant business value, so you have to do something with them.

Advantage of junk dimension: It provides a recognizable location for related codes,

indicators and their descriptors in a dimensional framework. This avoids the creation of multiple dimension tables. Provide a smaller, quicker point of entry for queries

compared to performance when these attributes are directly in the fact table. 

An interesting use for a junk dimension is to capture the context of a specific transaction.  While our common, conformed dimensions contain the key dimensional attributes of interest, there are likely attributes about the transaction that are not known until the transaction is processed.

Rapidly Changing Dimensions

If one or more of its attributes changes frequently.

when you deal with a type 2 change, you create an additional dimension table row with the new value of the changed attribute. By doing so, you are able to preserve the history.

consider customer dimension. Here the number of rows tends to be large, sometimes in the range of even a million or more rows. But significant attributes in a customer dimension may change many timesin a year. Rapidly changing large dimensions can be too problematic for the type 2 approach.

Rapidly Changing Dimensions

One effective approach is to break the large dimension table into one or more simpler dimension tables. How can you accomplish this?

Obviously, you need to break off the rapidly changing attributes into another dimension table, leaving the slowly changing attributes behind in the original table.

Solution to rapidly changing dimension

Large dimensions call for special considerations.

Because of the sheer size, many data warehouse functions involving large dimensions may be slow and inefficient.

You need to address the following issues by using effective design methods, by choosing proper

indexes, and by applying other optimizing techniques: