Lessons learned from American Tower Building a flexible and affordable enterprise data

warehouse with the expressor semantic data integration system

2

Who is ATC?

• Cellular and broadcast tower ownership and operation– Over 30,000 towers

• Leader in tower industry worldwide– Operations in US, Mexico, Brazil, India

• $1.6B revenue (worldwide), 1200 employees (US)

3

Business Challenges

• International operations– 4 markets at present, plus future expansions

• Business model– Real estate

• Outdated reporting environment– Single purpose data marts

– Lengthy, redundant data extracts

– No clear definition of contents

– Poor reporting

4

Enterprise Data Warehouse Program

• Started October 2008

• Improve user experience for reporting– Integrated data

– Improved reporting tool

– Faster data refreshes

• Three-part solution– Data Warehouse: Kimball methodology, SQL Server 2005

– BI reporting tool: Cognos 8

– Governance process: Business-led management of DW

5

Selecting the right tools

• Reporting tool (Cognos 8)– Lengthy vendor selection process, close business involvement

• Database for DW (SQL Server 2005)– Technical evaluation (data volumes and future capacity reqs)

– Experience of existing personnel

• ETL tool (initially SSIS)– Experience of existing personnel

– Budget (used existing SQL Server licenses)

– Technical evaluation (could live with shortcomings)

6

ETL Structure - High Level

• Three-step process

• Separate jobs for each process

Extract Transform Load

Raw data pulls Prepare data Load Facts & Dimensions (SCD)

Source System EDW

7

ETL Structure - Detail

• Three-layer process for each step

Extract

Scheduler

Metadata Wrapper

ETL Execution

Control Flow, timing

Data quality, logging, etc.

Actual data transfer / processing

8

SSIS Issues

• Functionality– Bulk updates

– Awkward scripting (two languages, not well integrated)

• Performance– Oracle extracts not performing optimally

9

Opportunity for expressor

• Became aware of expressor mid-2009

• Proof of concept to establish benefits– Performance: 8-24x faster for Oracle extracts (1-4

channels)

– Scripting: expressor datascript very powerful

– Functionality: bulk updates, general capabilities

• Acquisition made much easier by low cost of adoption

10

ETL Structure - expressor

• Three-step process

Extract Transform Load

Raw data pulls Prepare data Load Facts & Dimensions (SCD)

Improved performanceSemantic rationalization

ScriptingETL functionality

Bulk updates

expressor benefits

11

ETL Structure

• Three-layer process for each step

Extract

Scheduler

Metadata Wrapper

ETL Execution

Control Flow, timing

Data quality, logging, etc.

Actual data transfer / processing

Change execution method via metadata

Replace with expressor

12

expressor Downstream Benefits

• Semantic dictionary– Clarify confusing business terms

• Multiple formats for Tower Number

• Differing business terms for same concept (milestone / date)

– Direct input from BAs into data modeling / ETL

• Growth potential– Add channels as needed

• Development / Maintenance– Simpler ETL

– Fewer stored procedures or “inventive” solutions

13

expressor Challenges

• expressor datascript is different from MS / .Net world– Very powerful scripting language

• Single library– All semantic terms share a namespace

• Process / Flow control

• Involving BAs in semantic rationalization– Requires process change outside development team

14

Lessons Learned

• SSIS is “free”, but you get what you pay for– Functionality limitations; we didn’t know what we were missing

– Performance

• Transition requires effort– Small learning curve for expressor datascript

• Semantic Rationalization process impacts– Work with all affected groups

• ETL Architecture preparation pays off– Plan for scalable hardware and flexible software

15