data profiling 13 th meeting course name: business intelligence year: 2009

Data Profiling13th Meeting

Course Name: Business IntelligenceYear: 2009

Bina Nusantara University

3

Source of this Material

(2). Loshin, David (2003). Business Intelligence: The Savvy Manager’s

Guide. Chapter 8

The Business CaseNo business intelligence program can be built without information,

and that information may be coming from many different sources and providers, each of which may have little or no stake in the success of the outcome of BI program.

When faced with an integration process incorporating disparate data sets of dubious quality, data profiling is the first step toward adding value to that data.

Data profiling automates the initial processes of what we might call inferred metadata resolution: discovering what the data items really look like and providing a characterization of that data for next steps of integration.

Bina Nusantara University 4

Data profiling is a hierarchical process that attempts to build an assessment of the metadata associated with a collection of data sets.

The bottom level of the hierarchy characterizes the values associated with individual attributes. At the next level, the assessment looks at relationships between multiple columns within a single table. At the highest level, the profile describes relationships that exist between data attributes across different tables.

One other item to keep in mind while profiling data is that the most significant value is derived from discovering business knowledge that has been embedded in the data itself.


Data Profiling Activities

When presented with a set of data tables of questionable origin, a data consumer may want to verify or discover the data model that is embedded within that set. This is a hierarchical process that first focuses on exposing information about the individual columns within a single table, and then resolves relationships between different tables to generate a proposed data model for a data set.

• Simple Type InferenceWhen a data set is first introduced into a data environment, even if the set is accompanied by a corresponding data definition, the analyst may choose to verify the corresponding types through the profiling process. This is done through simple type inference, which is a process of resolving the view of each column’s data type to its most closely matched system data type. Data type inference is an iterative analysis of a value set to refine the perceived data type. Simple type inference centers only on assigning system data types (such as integer, decimal, date) to columns.


Data Model Inference

• Table Model Inference and Relational Model InferenceThere are two approaches to resolving a relational model from a collection of data tables. The first is a brute-force approach that uses the results of overlap analysis to determine whether any pair of columns exhibits a key relationship. The second approach is more of a semantic approach that evaluates column names to see if there is any implied relation.


Data Model Inference (cont…)

Attribute analysis is a process of looking at all of the values populating a particular column as a way to characterize that set of values. Attribute analysis is the first step in profiling because it yields a significant amount of metadata relating to data set. The result of any of these analyses provides greater insight into the business logic that is applied to each column.

Typically this evaluation revolves around the following aspects of a data set.• Range Analysis

Relating a value set to a simple type already restricts the set of values that a column can take; most data types still allow for an infinite number of possible choices. Range analysis can be used in an intelligence application to explore minimum values of interest, perhaps related to customer activity and monthly sales or to prices customers are being charged for the same products at different retail location.

• SparsenessThe degree of sparseness may indicate some business meaning regarding the importance of that attribute. Depending on the value set, it probably means one of two things.


Attributes Analysis

• Format EvaluationIt is useful to look for the existence of patterns that might characterize the values assigned to a column. We can use the discovered definitions as a validation rule, which we would then add to a metadata database of domain pattern. Simple examples of rule-based data domains include telephone numbers, zip codes, and social security number.

• Cardinality and UniquenessThe cardinality of a value set is the number of distinct values that exists within a column. Cardinality is interesting because it relates to different aspects of the correctness of the value set and because how it relates exposes business knowledge. Cardinality can be used to find columns whose values are unique, from which candidate keys can be inferred.

• Frequency DistributionThe frequency distribution of values yields the number of times each of the distinct values appears in a value set. Frequency distribution is also useful when looking for variations from the norm that might indicate something suspicious.


Attributes Analysis (cont…)

• Value AbsenceThe are actually two problems associated with the absence of values that can be explored through data profiling. The first involves looking for values that are not there, and the second is to look for nonvalues that are there.

• Abstract Type RecognitionAn abstract type is a more semantically descriptive qualification of a type definition that conveys business meaning. Typically abstract data types are represented by some kind of semantic definition, including:

Constructive Assertion Value enumeration Pattern Conformance

The goal of abstract type analysis is to propose an abstract data type for a specific column based on a suggestive statistical conformance to one defined abstract type.

• OverloadingIt is possible that as an attribute’s data value are checked against know domains the profiling process will see significant matches against more than one domain. This


Attributes Analysis (cont…)

might indicate that two attributes’ worth of information is in one column, where the same column is being used to represent more than one actual attribute (Fig. 13-1).Alternatively, the use of more than one domain by a single attribute might indicate that more complex business rules are in effect, such as the existence of a split attribute, which is characterized by the use of different domains based on other data quality or business rules. Overloading can appear in other ways as well, such as the compaction of multiple pieces of data into a single character string.


Attributes Analysis (cont…)Figure 13-1

Relationship analysis is frequently referred to in the data profiling space as cross-column analysis focuses on establishing relationships between sets of data. The goal of these processing stages is to identify relationships between value sets and known reference data, to identify dependencies between column (either in the same table or across different tables), and to indentify key relationships between columns across multiple tables.

• Domain AnalysisDomain analysis covers two tasks: identifying data domains and identifying references to data domains. The brute-force method for identifying enumerated domains is to look at all possible value sets.

• Functional DependencyA functional dependency establishes a relationship between two sets of attributes. If the relationship is casual (i.e., the dependent attribute’s value is filled in as a function of the defining attributes), that is an interesting piece of business knowledge that can be added to growing knowledge base. If the relationship is not a casual, then that piece of knowledge can be used to infer information about normalization of the data. I


Relationship Analysis

If a pair of data attribute values is consistently bound together, the those two columns can be extracted from the targeted table and the instance pairs inserted uniquely into a new table and assigned a reference identifier.

• Key RelationshipsA table key is a set of attributes that can be used to uniquely indentify any individual record within the table. When one table’s key is used as a reference to another table, that key is called a foreign key. Modern relational databases enforce a constraint known as referential integrity, which states that if an attribute’s value is used in table A as a foreign key to table B, then that key value must exist in one record in table B. There are two aspect to profiling key relationship: indentifying that a key relationship exist, and indentifying what are called orphans in a violated referential integrity situation.


Relationship Analysis (cont…)

The most significant management issues involve the relatively steep costs of good data profiling tools and the performance of these tools. It is worth-while to explore the questionable performance of these tools. First, some of the algorithms used in data profiling are actually quite computationally intensive, and it is not unusual for some of the analysis to require both large amount of computational resources (memory, disk space) and time to successfully complete. Second, because the computations are summaries of frequency analysis and counts, the results presented tend to be almost endless, with long lists of values, each of which may have appeared only once in a column.


Management Issues

End of Slide


data profiling 13 th meeting course name: business intelligence year: 2009

Documents