DATA DATA

RESOURCERESOURCE

MANAGEMENTMANAGEMENT

Data Hierarchy in a Computer System

Entitities and Attributes

Data redundancy

Program-Data dependence

Lack of flexibility

Poor security

Lack of data-sharing and availability

Problems with the Traditional File Environment

Traditional File Processing

Figure 7-3

• Creates and maintains databasesCreates and maintains databases

• Eliminates requirement for data definition Eliminates requirement for data definition statementsstatements

• Acts as interface between application Acts as interface between application programs and physical data filesprograms and physical data files

• Separates logical and physical views of dataSeparates logical and physical views of data

Database Management System (DBMS)

The Contemporary Database Environment

Components of DBMS

• Data definition language: Data definition language: Specifies Specifies content and structure of database and content and structure of database and defines each data elementdefines each data element

• Data manipulation language:Data manipulation language:Manipulates data in a databaseManipulates data in a database

• Data dictionary:Data dictionary: Stores definitions of Stores definitions of data elements, and data characteristicsdata elements, and data characteristics

Sample Data Dictionary Report

Figure 7-6

Relational Data Model

Three Basic Operations in a Relational Database

• Select:Select: Creates subset of rows that meet Creates subset of rows that meet specific criteriaspecific criteria

• Join:Join: Combines relational tables to provide Combines relational tables to provide users with informationusers with information

• Project:Project: Enables users to create new tables Enables users to create new tables containing only relevant informationcontaining only relevant information

Figure 7-7

Three Basic Operations in a Relational Database

FLAT FILE – NOT NORMALIZED

A Normalized Relation of ORDER

Ensuring Database Integrity

Database integrity involves the maintenance of the logical and business rules of the database.

There are two kinds of “DB Integrity” that must be addressed: Entity Integrity Referential Integrity

Entity Integrity

Entity integrity deals with within-entity rules.

These rules deal with ranges and the permission of null values in attributes or possibly between records

Examples of Entity Integrity

Data Type Integrity: very common and most basic. Checks only for “data type” compatibility with DB Schema, such as: numeric, character, logical, date format, etc.

Commonly referred to in GIS manuals as: Range and List domains

Ranges - acceptable Numeric ranges for input List - acceptable text entries or drop-down lists.

Enforcing Integrity

Not a trivial task! Not all database management systems or

GIS software enable users to “enforce data integrity” during attribute entry or edit sessions.

Therefore, the programmer or the Database Administrator must enforce and/or check for “Integrity.”

Referential Integrity

Referential integrity concerns two or more tables that are related.

Example: IF table A contains a foreign key that matches the primary key of table B THEN values of this foreign key either match the value of the primary key for a row in table B or must be null.

Necessary to avoid: Update anomaly, Delete anomaly.

Basic SQL Commands

SELECT: Specifies columns

FROM: Identifies tables or views

WHERE: Specifies conditions

Querying Databases: Elements of SQL

Using SQL- Structured Query Language SQL is a standard database protocol,

adopted by most ‘relational’ databases Provides syntax for data:

Definition Retrieval Functions (COUNT, SUM, MIN, MAX, etc) Updates and Deletes

SQL Examples

CREATE TABLE SALESREP Item definition expression(s)

{item, type, (width)}

DELETE table WHERE expression

Data Retrieval

SELECT list FROM table WHERE condition list - a list of items or * for all items

WHERE - a logical expression limiting the number of records selected

can be combined with Boolean logic: AND, OR, NOT

ORDER may be used to format results

UPDATE tables

SET item = expression WHERE expression INSERT INTO table VALUES …..

Database Normalization

Normalization: The process of structuring data to minimize duplication and inconsistencies.

The process usually involves breaking down a single Table into two or more tables and defining relationships between those tables.

Normalization is usually done in stages, with each stage applying more rigorous rules to the types of information which can be stored in a table.

Normalization

Normalization: a process for analyzing the design of a relational database Database Design - Arrangement of attributes

into entities It permits the identification of potential

problems in your database design Concepts related to Normalization:

KEYS and FUNCTIONAL DEPENDENCE

Ex: Database Normalization (1)

Sample Student Activities DB Table

Poorly Designed Non-unique records

John Smith

Test the Design by developing sample reports and queries

Created a unique “ID” for each Record in the Activities Table

Required the creation of an “ID” look-up table for reporting (Students Table)

Converted the “Flat-File into a Relational Database

Ex: Database Normalization (2)

Ex: Database Normalization (3)

Wasted Space Redundant data entry What about taking a 3rd

Activity? Query Difficulties - trying

to find all swimmers Data Inconsistencies -

conflicting prices

Ex: Database Normalization (4)

Students table is fine Elimination of two

columns and an Activities Table restructuring, Simplifies the Table

BUT, we still have Redundant data (activity fees) and data insertion anomalies.

Problem: If student #219 transfers we lose all references to Golf and its price.

Ex: Database Normalization (5)

Modify the Design to ensure that “every non-key field is dependent on the whole key”

Creation of the Participants Table, corrects our problems and forms a union between 2 tables.

This is a Better Design!

The Normal Forms

A series of logical steps to take to normalize data tables

First Normal Form Second Third Boyce Codd There’s more, but beyond scope of this

First Normal Form (1NF)

All columns (fields) must be atomic Means : no repeating items in columns

OrderDate Customer Items11/30/1998 Joe Smith Hammer, Saw, Nails

OrderDate Customer Item1 Item2 Item311/30/1998 Joe Smith Hammer Saw Nails

Solution: make a separate table for each set of attributes with a primary key (parser, append query)

CustomersCustomerIDName

OrdersOrderIDItem CustomerIDOrderDate

Second Normal Form (2NF)

In 1NF and every non-key column is fully dependent on the (entire) primary key Means : Do(es) the key field(s) imply the rest of the fields? Do we

need to know both OrderID and Item to know the Customer and Date? Clue: repeating fields

Solution: Remove to a separate table (Make Table)

OrderID Item CustomerID OrderDate1 Hammer 1 11/30/19981 Saw 1 11/30/19981 Nails 1 11/30/1998

OrderDetailsOrderIDItem

OrdersOrderIDCustomerIDOrderDate

Third Normal Form (3NF)

In 2NF and every non-key column is mutually independent means : Calculations

•Solution: Put calculations in queries and forms

Item Quantity Price TotalHammer 2 $10 $20Saw 5 $40 $200Nails 8 $1 $8

OrderDetailsOrderIDItemQuantityPrice

Put expression in text control or in query:=Quantity * Price

Data warehouse

Supports reporting and query tools

Stores current and historical data

Consolidates data for management analysis and decision making

Data Warehousing and Datamining

What is a Data Warehouse?

"A warehouse is a subject-oriented, integrated, time-variant and non-volatile collection of data in support of management's decision making process". Bill Inmon (1990)

"A Data Warehouse is a repository of integrated information, available for queries and analysis. Data and information are extracted from heterogeneous sources as they are generated.…”Anonymous

Components of a Data Warehouse

Data Mining

ON-LINE ANALYTICAL PROCESSING (OLAP): ability to manipulate, analyze large volumes of data from multiple perspectives

MINING: Seeking relationships that are not known in advance. A function of the software and data organization.

DW Characteristics

Subject Oriented:Data that gives information about a particular subject instead of about a company's ongoing operations.

Integrated: Data that is gathered into the data warehouse from a variety of sources and merged into a coherent whole.

Time Variant: All data in the data warehouse is identified with a particular time period.

Data Acquisition

The process of moving company data from the source systems into the warehouse.

Often the most time-consuming and costly effort.

Performed with software products known as ETL (Extract/Transform/Load) tools.

Over 50 ETL tools on market.

Data Cleansing

Typically performed in conjunction with data acquisition.

A complicated process that validates and, if necessary, corrects the data before it is inserted.

AKA "data scrubbing" or "data quality assurance".