implementing tivoli data warehouse v 1.2 sg247100

ibm.com/redbooks

Implementing Tivoli Data Warehouse 1.21.2

Edson ManoelCristiano Colantuono

Hans-Georg KöhneDevi Raju

Ghufran ShahSergio Henrique Soares Monteiro

A primer for deployments of any size and proof of concepts

Latest Version 1.2 features including Crystal Enterprise

Warehouse enablement pack case studies

Front cover

http://www.redbooks.ibm.com/


Implementing Tivoli Data Warehouse 1.2

June 2004

International Technical Support Organization

SG24-7100-00

© Copyright International Business Machines Corporation 2004. All rights reserved.Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADPSchedule Contract with IBM Corp.

First Edition (June 2004)

This edition applies to Version 1.2 of the Tivoli Data Warehouse product.

Note: Before using this information and the product it supports, read the information in “Notices” on page xix.

Contents

Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix

Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv

Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvii

Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xixTrademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xx

Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiThe team that wrote this redbook. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiBecome a published author . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiiiComments welcome. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiii

Part 1. Fundamentals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

Chapter 1. Introducing Tivoli Data Warehouse 1.2. . . . . . . . . . . . . . . . . . . . 31.1 Data warehousing basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

1.1.1 Data warehouse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.1.2 Data mart . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51.1.3 Business intelligence. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51.1.4 Data mining . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

1.2 Tivoli Data Warehouse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81.3 What is new in Tivoli Data Warehouse 1.2 . . . . . . . . . . . . . . . . . . . . . . . . 10

1.3.1 Crystal Enterprise™ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101.3.2 IBM DB2 UDB for OS/390 and z/OS support . . . . . . . . . . . . . . . . . . 131.3.3 Flexible and extended configuration support . . . . . . . . . . . . . . . . . . 171.3.4 Installation enhancements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181.3.5 Serviceability and scalability improvements . . . . . . . . . . . . . . . . . . . 19

1.4 Tivoli Data Warehouse architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201.4.1 Tivoli Data Warehouse control center server . . . . . . . . . . . . . . . . . . 211.4.2 Source databases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211.4.3 Central data warehouse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211.4.4 Data marts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211.4.5 Warehouse agents and agent sites. . . . . . . . . . . . . . . . . . . . . . . . . . 221.4.6 Crystal Enterprise Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

1.5 Benefits of using Tivoli Data Warehouse . . . . . . . . . . . . . . . . . . . . . . . . . 23

Chapter 2. Planning for Tivoli Data Warehouse 1.2 . . . . . . . . . . . . . . . . . . 27

© Copyright IBM Corp. 2004. All rights reserved. iii

2.1 Hardware and software requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282.1.1 Hardware requirements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292.1.2 Software requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302.1.3 Database requirements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 322.1.4 Crystal Enterprise requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

2.2 Physical and logical design considerations . . . . . . . . . . . . . . . . . . . . . . . . 362.2.1 Source databases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 372.2.2 Control server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 372.2.3 Central data warehouse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 382.2.4 Data marts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 402.2.5 Single machine installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 422.2.6 Distributed deployment on UNIX and Windows servers . . . . . . . . . . 432.2.7 Distributed deployment on z/OS, UNIX, and Windows servers . . . . 452.2.8 Warehouse agents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 492.2.9 Considerations about warehouse databases on z/OS . . . . . . . . . . . 542.2.10 Coexistence with other products . . . . . . . . . . . . . . . . . . . . . . . . . . . 552.2.11 Selecting port numbers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

2.3 Database sizing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 562.4 Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

2.4.1 Authority required to install and maintain IBM DB2 UDB . . . . . . . . . 572.4.2 Authority required to install Tivoli Data Warehouse . . . . . . . . . . . . . 572.4.3 Firewalls . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 582.4.4 Controlling access to data in the warehouse . . . . . . . . . . . . . . . . . . 592.4.5 Protecting information in Crystal Enterprise Professional for Tivoli . 592.4.6 Multicustomer and multicenter support . . . . . . . . . . . . . . . . . . . . . . . 60

2.5 Network traffic considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 612.5.1 Architectural choices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 622.5.2 Scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

2.6 Integration with other business intelligence tools . . . . . . . . . . . . . . . . . . . 642.7 ETL development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 652.8 Skills required for a Tivoli Data Warehouse project . . . . . . . . . . . . . . . . . 67

2.8.1 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 672.8.2 Data collection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 672.8.3 Data manipulation (ETL1 and ETL2). . . . . . . . . . . . . . . . . . . . . . . . . 682.8.4 Reporting. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

Chapter 3. Getting Tivoli Data Warehouse 1.2 up and running. . . . . . . . . 713.1 Preparing for the installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

3.1.1 Ensuring fully qualified host names. . . . . . . . . . . . . . . . . . . . . . . . . . 743.1.2 Installing and configuring IBM DB2 client and server . . . . . . . . . . . . 763.1.3 Crystal Enterprise installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86

3.2 Tivoli Data Warehouse 1.2 installation . . . . . . . . . . . . . . . . . . . . . . . . . . . 923.3 Quick start deployment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93

iv Implementing Tivoli Data Warehouse 1.2

3.3.1 Quick start deployment: installation and configuration . . . . . . . . . . . 943.3.2 Configuring the control database . . . . . . . . . . . . . . . . . . . . . . . . . . . 993.3.3 Creating ODBC connections to the data mart databases . . . . . . . . 101

3.4 Distributed deployment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1033.4.1 Distributed deployment installation: Windows and UNIX . . . . . . . . 1043.4.2 Distributed deployment installation: z/OS . . . . . . . . . . . . . . . . . . . . 1153.4.3 Creating ODBC connections to the data mart databases . . . . . . . . 123

3.5 Installing warehouse agents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1263.5.1 Installing IBM DB2 Warehouse Manager . . . . . . . . . . . . . . . . . . . . 1283.5.2 Creating the remote agent sites . . . . . . . . . . . . . . . . . . . . . . . . . . . 131

3.6 Verification of the installation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1353.6.1 Verifying the remote agent install . . . . . . . . . . . . . . . . . . . . . . . . . . 141

3.7 Installing warehouse enablement packs . . . . . . . . . . . . . . . . . . . . . . . . . 142

Chapter 4. Performance maximization techniques . . . . . . . . . . . . . . . . . 1454.1 DB2 performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1464.2 Operating system performance tuning . . . . . . . . . . . . . . . . . . . . . . . . . . 150

4.2.1 Windows environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1504.2.2 Primary Windows performance factors . . . . . . . . . . . . . . . . . . . . . . 1514.2.3 AIX environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155

4.3 Tivoli Data Warehouse performance. . . . . . . . . . . . . . . . . . . . . . . . . . . . 155

Part 2. Case study scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159

Chapter 5. IBM Tivoli NetView Warehouse Enablement Pack . . . . . . . . . 1615.1 Case study overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1625.2 IBM Tivoli NetView WEP overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1635.3 Prerequisites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165

5.3.1 Verifying prerequisites. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1655.3.2 Gathering installation information . . . . . . . . . . . . . . . . . . . . . . . . . . 166

5.4 Preparing NetView for data collection . . . . . . . . . . . . . . . . . . . . . . . . . . . 1675.4.1 Enabling NetView to export data for Tivoli Data Warehouse . . . . . 1675.4.2 NetView SmartSets configuration . . . . . . . . . . . . . . . . . . . . . . . . . . 1705.4.3 Configuring NetView Data Warehouse daemon (tdwdaemon) . . . . 1765.4.4 Verifying NetView data collection enablement . . . . . . . . . . . . . . . . 178

5.5 Installation of the NetView WEPs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1815.5.1 Backing up the TDW environment . . . . . . . . . . . . . . . . . . . . . . . . . 1815.5.2 Establishing ODBC connections . . . . . . . . . . . . . . . . . . . . . . . . . . . 1825.5.3 Installing NetView Enablement Pack Software . . . . . . . . . . . . . . . . 1855.5.4 Defining the authority to the warehouse sources and targets . . . . . 188

5.6 Testing, scheduling, and promoting the ETLs . . . . . . . . . . . . . . . . . . . . . 1915.6.1 Promoting the ETLs to TEST mode . . . . . . . . . . . . . . . . . . . . . . . . 1925.6.2 Testing the ETLs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1935.6.3 Scheduling the ETLs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195

Contents v

5.6.4 Promoting the ETLs to Production status . . . . . . . . . . . . . . . . . . . . 1975.7 Running NetView ETLs on remote agent sites . . . . . . . . . . . . . . . . . . . . 1985.8 Reporting. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206

5.8.1 Accessing the Crystal ePortfolio feature . . . . . . . . . . . . . . . . . . . . . 206

Chapter 6. IBM Tivoli Monitoring Warehouse Enablement Pack. . . . . . . 2256.1 Case study overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2266.2 IBM Tivoli Monitoring WEP overview . . . . . . . . . . . . . . . . . . . . . . . . . . . 2276.3 Prerequisites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2316.4 Installing the ITM WEP data collector component. . . . . . . . . . . . . . . . . . 232

6.4.1 Activate data collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2376.5 Installing and configuring ITM Generic WEP. . . . . . . . . . . . . . . . . . . . . . 241

6.5.1 Backing up the TWH databases . . . . . . . . . . . . . . . . . . . . . . . . . . . 2416.5.2 Establishing an ODBC connection on the Control Center. . . . . . . . 2426.5.3 Installing the ITM 5.1.1 AMX ETL processes . . . . . . . . . . . . . . . . . 2476.5.4 Installing AMX Fix Packs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2536.5.5 Defining the authority to the warehouse sources and targets . . . . . 2546.5.6 Modifying the ETL for the source table name to the RIM user . . . . 257

6.6 Installing and configuring ITM for OS WEP. . . . . . . . . . . . . . . . . . . . . . . 2626.6.1 Backing up the TWH databases . . . . . . . . . . . . . . . . . . . . . . . . . . . 2626.6.2 Installing the ITM 5.1.1 AMY ETL processes . . . . . . . . . . . . . . . . . 2626.6.3 Installing AMY Fix Packs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2646.6.4 Defining the authority to the warehouse sources and targets . . . . . 265

6.7 Testing, scheduling, and promoting the ETLs . . . . . . . . . . . . . . . . . . . . . 2676.7.1 Testing the ETLs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2676.7.2 Checking that data has been collected . . . . . . . . . . . . . . . . . . . . . . 2706.7.3 Scheduling the ETLs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2726.7.4 Promoting the ETL status to Production mode . . . . . . . . . . . . . . . . 274

6.8 Reporting. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2756.8.1 Available reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2756.8.2 Accessing the Crystal ePortfolio . . . . . . . . . . . . . . . . . . . . . . . . . . . 275

6.9 Troubleshooting of ITM data collection . . . . . . . . . . . . . . . . . . . . . . . . . . 2866.9.1 Using itmchk.sh script . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2876.9.2 Manual checking of ITM data collection . . . . . . . . . . . . . . . . . . . . . 290

Chapter 7. IBM Tivoli Storage Manager Warehouse Enablement Pack . 2977.1 Case study overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2987.2 IBM Tivoli Storage Manager WEP overview . . . . . . . . . . . . . . . . . . . . . . 2997.3 Prerequisites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3007.4 Installing and configuring ITSM WEP 5.2 . . . . . . . . . . . . . . . . . . . . . . . . 301

7.4.1 Changes required on the IBM Tivoli Storage Manager servers . . . 3017.4.2 Installing the IBM Tivoli Storage Manager ODBC . . . . . . . . . . . . . . 3017.4.3 Backing up the TWH databases . . . . . . . . . . . . . . . . . . . . . . . . . . . 304

vi Implementing Tivoli Data Warehouse 1.2

7.4.4 IBM Tivoli Storage Manager WEP installation . . . . . . . . . . . . . . . . 3057.4.5 Defining the authority to the warehouse sources and targets . . . . . 313

7.5 IBM Tivoli Storage Manager ETL processes. . . . . . . . . . . . . . . . . . . . . . 3147.5.1 ANR_C05_ETL1_Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3157.5.2 ANR_C10_EXPServer_Process . . . . . . . . . . . . . . . . . . . . . . . . . . . 3197.5.3 ANR_M05_ETL2_Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 319

7.6 Testing, scheduling, and promoting the ETLs . . . . . . . . . . . . . . . . . . . . . 3207.6.1 ETL data collection verification . . . . . . . . . . . . . . . . . . . . . . . . . . . . 320

7.7 Reporting. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3227.7.1 Available reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3227.7.2 Accessing the Crystal ePortfolio . . . . . . . . . . . . . . . . . . . . . . . . . . . 322

Part 3. Appendixes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 337

Appendix A. IBM DB2 UDB administration for other relational DBAs . . 339Common DBA tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 340Creating databases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 340

Creating databases in IBM DB2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 341Creating databases in Oracle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 341Creating databases in Sybase . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 342

Managing space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343DB2 space management. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343Oracle space management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 345Sybase space management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 346

Creating objects in the database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 346Creating tables in DB2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 346Creating tables in Oracle. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 347Creating tables in Sybase . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 347

Additional table control parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 347

Appendix B. Tivoli Data Warehouse 1.2 reference . . . . . . . . . . . . . . . . . . 349Report listing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 350Measurement sources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 357

Appendix C. Warehouse Enablement Packs properties file . . . . . . . . . . 361The twh_install_props.cfg properties file . . . . . . . . . . . . . . . . . . . . . . . . . . . . 362

Related publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 365IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 365Other publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 365Online resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 366How to get IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 367Help from IBM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 367

Contents vii

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 369

viii Implementing Tivoli Data Warehouse 1.2

Figures

1-1 IBM DB2 Data Warehouse Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91-2 Crystal Enterprise multi-tier architecture . . . . . . . . . . . . . . . . . . . . . . . . 121-3 TDS OS/390 and TDW 1.2 Data flow . . . . . . . . . . . . . . . . . . . . . . . . . . 151-4 Distributed and OS/390 Data feeds into Tivoli Data Warehouse . . . . . . 161-5 Multiple source applications loading into a central data warehouse . . . 171-6 Tivoli Data Warehouse — the big picture . . . . . . . . . . . . . . . . . . . . . . . 201-7 Detail Component view of Tivoli Data Warehouse 1.2. . . . . . . . . . . . . . 231-8 Integrated Systems Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252-1 Single machine installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 432-2 Distributed deployment on Windows and UNIX systems. . . . . . . . . . . . 442-3 Operational data sources and the CDW databases on the same server452-4 Data sources, CDW, and data mart databases on a z/OS system . . . . 462-5 Operational data sources both on z/OS and on distributed systems . . . 472-6 Separate data mart databases on z/OS system and distributed system 482-7 Two CDWs on a Windows or UNIX system and on a z/OS system. . . . 492-8 Warehouse agent on control server. . . . . . . . . . . . . . . . . . . . . . . . . . . . 502-9 Warehouse agents on data targets . . . . . . . . . . . . . . . . . . . . . . . . . . . . 512-10 Configuration with a warehouse agent on the source . . . . . . . . . . . . . . 522-11 Tivoli Data Warehouse and firewalls . . . . . . . . . . . . . . . . . . . . . . . . . . . 582-12 Business intelligence integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 653-1 Installation process overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 733-2 Install DB2 V7 components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 783-3 Create DB2 Services - DB2 Instance db2inst1 . . . . . . . . . . . . . . . . . . . 793-4 Create the DB2 fenced user . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 803-5 Administration Server window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 813-6 Select DB2 Enterprise Edition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 833-7 Installation Type. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 883-8 MSDE security configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 893-9 Installation window. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 893-10 Completion window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 903-11 Crystal Enterprise Launchpad . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 913-12 Crystal Administration Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 923-13 Quick start deployment configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . 943-14 InstallShield Wizard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 953-15 Tivoli common logging directory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 953-16 Setup window. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 963-17 DB2 connection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 973-18 Crystal connection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98

© Copyright IBM Corp. 2004. All rights reserved. ix

3-19 Summary window. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 983-20 Completion window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 993-21 Configuring the IBM DB2 data warehouse center . . . . . . . . . . . . . . . . 1003-22 Configuring the Warehouse Control Database Management . . . . . . . 1013-23 Distributed deployment scenario . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1043-24 Install Shield Wizard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1053-25 Tivoli Common Logging Directory . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1063-26 Setup Window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1073-27 Before proceeding with TDW 1.2 distributed installation . . . . . . . . . . . 1083-28 DB2 connection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1083-29 Central data warehouse on remote host . . . . . . . . . . . . . . . . . . . . . . . 1093-30 Central data warehouse database server list. . . . . . . . . . . . . . . . . . . . 1103-31 Data mart on remote host . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1103-32 Data mart database server list . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1113-33 Crystal connection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1123-34 Summary window. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1123-35 Completion window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1133-36 Configuring the IBM DB2 Data Warehouse Center . . . . . . . . . . . . . . . 1143-37 Configuring the Warehouse Control Database Management . . . . . . . 1153-38 Adding central data warehouses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1163-39 z/OS IBM DB2 Server information . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1173-40 z/OS central data warehouse database configuration . . . . . . . . . . . . . 1173-41 Central data warehouse server on z/OS . . . . . . . . . . . . . . . . . . . . . . . 1183-42 Central data warehouse summary window . . . . . . . . . . . . . . . . . . . . . 1183-43 Central data warehouse on z/OS install. . . . . . . . . . . . . . . . . . . . . . . . 1193-44 Adding data marts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1203-45 z/OS IBM DB2 Server information . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1213-46 z/OS data mart database configuration . . . . . . . . . . . . . . . . . . . . . . . . 1213-47 Data mart server on z/OS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1223-48 Data mart creation summary window. . . . . . . . . . . . . . . . . . . . . . . . . . 1223-49 Data mart on z/OS install . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1233-50 Distributed environment with agent . . . . . . . . . . . . . . . . . . . . . . . . . . . 1273-51 Select the DB2 Warehouse Manager components . . . . . . . . . . . . . . . 1293-52 Install DB2 V7 menu on AIX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1303-53 Create DB2 Service Menu on AIX . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1313-54 Setup Window - create warehouse agents . . . . . . . . . . . . . . . . . . . . . 1323-55 Before proceeding with remote agent sites creation . . . . . . . . . . . . . . 1333-56 Warehouse agents - specify the TDW control server . . . . . . . . . . . . . 1343-57 Successful remote agent creation window. . . . . . . . . . . . . . . . . . . . . . 1353-58 DB2 Data Warehouse services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1363-59 Remote Agent Sites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1393-60 Verify Remote Agents on Tivoli Data Warehouse Control Center . . . . 1415-1 Distributed deployment scenario . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163

x Implementing Tivoli Data Warehouse 1.2

5-2 IBM Tivoli NetView Warehouse Enablement Pack data flow. . . . . . . . 1645-3 NetView Configure data export to DB2 - Parameters . . . . . . . . . . . . . 1685-4 NetView Configure data export to DB2 - create database . . . . . . . . . . 1685-5 NetView Configure data export to DB2 - register and start tdwdaemon1695-6 NetView SmartSet desktop . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1715-7 Microsoft SmartSet Advanced attributes . . . . . . . . . . . . . . . . . . . . . . . 1725-8 Create Microsoft SmartSet. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1735-9 NetView SmartSets - Attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1745-10 NetView SmartSets - Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1755-11 SmartSet Microsoft contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1765-12 Create an ODBC data source for NETVIEW . . . . . . . . . . . . . . . . . . . . 1835-13 Add an ODBC data source for NETVIEW . . . . . . . . . . . . . . . . . . . . . . 1835-14 Configure NetView Source database connectivity . . . . . . . . . . . . . . . . 1845-15 NetView WEP installation - List of WEPs to install . . . . . . . . . . . . . . . 1855-16 NetView WEP installation - Properties file . . . . . . . . . . . . . . . . . . . . . . 1865-17 NetView WEP installation - List of WEPs to install NetView . . . . . . . . 1875-18 NetView WEP installation - successful installation . . . . . . . . . . . . . . . 1875-19 Data Warehouse Control Center - check control database . . . . . . . . . 1895-20 Configure NetView data warehouse sources. . . . . . . . . . . . . . . . . . . . 1905-21 Configure NetView data warehouse targets . . . . . . . . . . . . . . . . . . . . 1915-22 Promote ETLs to test mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1925-23 Test ETL process steps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1935-24 Work in Progress - Log file. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1945-25 Sample contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1955-26 Schedule ANM_c05_ETL1_Process . . . . . . . . . . . . . . . . . . . . . . . . . . 1965-27 Schedule configuration for ANM_C05_ETL1_Process . . . . . . . . . . . . 1975-28 Promote ANM_c05_ETL1_Process . . . . . . . . . . . . . . . . . . . . . . . . . . . 1985-29 Select remote agents properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2005-30 Change remote agents properties - sources and targets. . . . . . . . . . . 2015-31 Select ETL process properties. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2025-32 Demote ETL processes to development mode . . . . . . . . . . . . . . . . . . 2025-33 Change the ETL processes agent site . . . . . . . . . . . . . . . . . . . . . . . . . 2035-34 Work in Progress - Run ETL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2045-35 Work in progress - Check ETL. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2055-36 Log Details menu . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2055-37 Crystal Enterprise - Launchpad . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2075-38 Crystal Enterprise 9 - ePortfolio . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2085-39 Crystal Enterprise 9 - Log in . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2095-40 Crystal Enterprise 9 - Folders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2105-41 Crystal Enterprise 9 - Tivoli Reports: IBM Tivoli NetView . . . . . . . . . . 2115-42 Crystal Enterprise 9 - Daily Status Summary by SmartSet . . . . . . . . . 2125-43 Crystal Enterprise 9 - Schedule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2125-44 Crystal Enterprise 9 - Parameters for Schedule Option. . . . . . . . . . . . 213

Figures xi

5-45 Crystal Enterprise 9 - Schedule Parameters . . . . . . . . . . . . . . . . . . . . 2145-46 Crystal Enterprise 9 - Schedule Parameter Selection . . . . . . . . . . . . . 2155-47 Crystal Enterprise 9 - Parameters: Specific Time Frame. . . . . . . . . . . 2165-48 Crystal Enterprise 9 - Report History . . . . . . . . . . . . . . . . . . . . . . . . . . 2175-49 Failed report generation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2185-50 Crystal Enterprise 9 - Report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2195-51 Crystal Enterprise 9 - Report (count) . . . . . . . . . . . . . . . . . . . . . . . . . . 2205-52 Summary of total status changes by SmartSet example . . . . . . . . . . . 2215-53 Nodes with longest outage times example . . . . . . . . . . . . . . . . . . . . . 2225-54 Total daily status changes in monitored network example . . . . . . . . . 2236-1 Environment for our case study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2276-2 Overview of ITM integration with Tivoli Data Warehouse . . . . . . . . . . 2286-3 IBM Tivoli Monitoring data flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2296-4 Resource Model Data Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2306-5 Aggregation time line . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2316-6 Installing warehouse support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2336-7 RIM setup options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2346-8 Logging option . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2386-9 Client Configuration Assistant opening dialog . . . . . . . . . . . . . . . . . . . 2436-10 Add Database Wizard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2446-11 Add System dialog window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2456-12 Select ITM_DB in the dialog window . . . . . . . . . . . . . . . . . . . . . . . . . . 2466-13 Confirmation dialog window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2466-14 User ID and Password dialog window . . . . . . . . . . . . . . . . . . . . . . . . . 2476-15 ODBC connection successful . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2476-16 Install a Warehouse Pack window . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2486-17 Tivoli Common Logging Directory window. . . . . . . . . . . . . . . . . . . . . . 2496-18 Add Warehouse Pack window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2496-19 Location of installation properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2506-20 Installation menu window. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2516-21 Installation summary window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2526-22 AMX installation completion window.. . . . . . . . . . . . . . . . . . . . . . . . . . 2536-23 Installation of AMX Fix Pack 6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2546-24 IBM Tivoli Monitoring, Version 5.1.1 Generic ETL1 Sources. . . . . . . . 2556-25 AMX_ITM_RIM_Source user ID information . . . . . . . . . . . . . . . . . . . . 2556-26 AMX_TWH_CDW_Source user ID information . . . . . . . . . . . . . . . . . . 2566-27 IBM Tivoli Monitoring, Version 5.1.1 Generic ETL1 Target . . . . . . . . . 2566-28 AMX_TWH_CDW_Target user ID information. . . . . . . . . . . . . . . . . . . 2576-29 Tables and views of AMX_ITM_TIM_Source. . . . . . . . . . . . . . . . . . . . 2586-30 Table name filter specification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2586-31 Endpoint tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2596-32 AMX_c05_ETL1 process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2606-33 Selecting new table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261

xii Implementing Tivoli Data Warehouse 1.2

6-34 Installation menu window with the AMY pack . . . . . . . . . . . . . . . . . . . 2636-35 AMY installation completion window . . . . . . . . . . . . . . . . . . . . . . . . . . 2646-36 Installation of AMY Fix Pack 6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2656-37 AMY_TWH_CDW_Source user ID information . . . . . . . . . . . . . . . . . . 2666-38 AMY_TWH_MART_Target user ID information . . . . . . . . . . . . . . . . . . 2676-39 Change ETL mode to Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2686-40 Manually test the ETLs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2696-41 Work in progress window. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2706-42 Sample Content of table F_OS_HOUR . . . . . . . . . . . . . . . . . . . . . . . . 2716-43 Schedule AMX_c05_ETL1_Process . . . . . . . . . . . . . . . . . . . . . . . . . . 2726-44 Schedule configuration for AMX_c05_ETL1_Process . . . . . . . . . . . . . 2736-45 Promoting ETLs to Production . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2746-46 Crystal Enterprise - Launchpad . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2766-47 Crystal Enterprise 9 - ePortfolio . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2776-48 Crystal Enterprise 9 - Log in . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2786-49 Crystal Enterprise 9 - Folders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2796-50 Crystal Enterprise 9 - available reports for ITM . . . . . . . . . . . . . . . . . . 2806-51 Scheduling Operating System Busiest System report . . . . . . . . . . . . . 2816-52 Crystal Enterprise 9 - parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2826-53 Crystal Enterprise 9 - Parameters for the report . . . . . . . . . . . . . . . . . 2826-54 Crystal Enterprise 9 - Report History . . . . . . . . . . . . . . . . . . . . . . . . . . 2836-55 Operating System Busiest Systems report . . . . . . . . . . . . . . . . . . . . . 2846-56 Operating System Paging File Utilization. . . . . . . . . . . . . . . . . . . . . . . 2856-57 Operating System Operating System UNIX CPU Statistics . . . . . . . . . 2867-1 TDW 1.2 - distributed deployment scenario. . . . . . . . . . . . . . . . . . . . . 2997-2 ITSM ODBC Installation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3027-3 ITSM ODBC data source configuration panel . . . . . . . . . . . . . . . . . . . 3037-4 Install a Warehouse Pack window . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3057-5 Tivoli Common Logging Directory window. . . . . . . . . . . . . . . . . . . . . . 3067-6 Add Warehouse Pack window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3067-7 Location of installation properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3077-8 Data mart and remote agent site settings . . . . . . . . . . . . . . . . . . . . . . 3087-9 Central data warehouse and remote agent site settings . . . . . . . . . . . 3097-10 Editing IBM Tivoli Storage Manager ODBC settings . . . . . . . . . . . . . . 3107-11 ITSM ODBC Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3117-12 Installation menu window. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3117-13 Installation summary window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3127-14 Installation Progress and Completion window . . . . . . . . . . . . . . . . . . . 3137-15 Sample of Process Model ANR_C05_ETL1_Process . . . . . . . . . . . . . 3167-16 Sample Content of Table D_NODE . . . . . . . . . . . . . . . . . . . . . . . . . . . 3217-17 Crystal Enterprise - Launchpad . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3237-18 Crystal Enterprise 9 - ePortfolio . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3247-19 Crystal Enterprise 9 - Log in . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325

Figures xiii

7-20 Crystal Enterprise 9 - Folders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3267-21 Crystal Enterprise 9 - available reports for ITSM . . . . . . . . . . . . . . . . . 3277-22 Scheduling Operating System Busiest System report . . . . . . . . . . . . . 3287-23 Crystal Enterprise 9 - parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3297-24 Crystal Enterprise 9 - Parameters for the report . . . . . . . . . . . . . . . . . 3307-25 How Has Clients use of Server Storage Changed Over Time? . . . . . . 3317-26 How Has Clients Use of Server Storage Changed Over Time? . . . . . 3327-27 How Has Clients Use of Server Storage Changed by Platform? . . . . . 3337-28 How Has My Server Storage Space Utilization Changed Over Time? 3347-29 Which Clients are Using the Most Server Storage?. . . . . . . . . . . . . . . 335C-1 Location of the twh_install_props.cfg file . . . . . . . . . . . . . . . . . . . . . . . 362

xiv Implementing Tivoli Data Warehouse 1.2

Tables

2-1 Hardware recommendations for Tivoli Data Warehouse components . 292-2 Additional hard disk space requirements . . . . . . . . . . . . . . . . . . . . . . . . 302-3 Software requirements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312-4 Web servers and OS supported by Crystal Web Connector . . . . . . . . . 352-5 Requirements for Tivoli Data Warehouse components . . . . . . . . . . . . . 362-6 Agent sites placement for data transfers to a central data warehouse . 522-7 Where to place agent sites for data transfers to data marts . . . . . . . . . 532-8 Default port used in Tivoli Data Warehouse environments . . . . . . . . . . 565-1 Environment for NetView integration . . . . . . . . . . . . . . . . . . . . . . . . . . 1625-2 NetView WEP Prerequisite Check - NetView server platform . . . . . . . 1655-3 Netview Enablement Pack installation information . . . . . . . . . . . . . . . 1665-4 Case Study SmartSets attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1735-5 Add database wizard - register TCP/IP . . . . . . . . . . . . . . . . . . . . . . . . 1845-6 NetView sources and targets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1886-1 Hardware and operating systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2277-1 Environment for NetView integration . . . . . . . . . . . . . . . . . . . . . . . . . . 2987-2 ITSM WEP Warehouse Object Names . . . . . . . . . . . . . . . . . . . . . . . . 314C-1 WEP installation properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363

© Copyright IBM Corp. 2004. All rights reserved. xv

xvi Implementing Tivoli Data Warehouse 1.2

Examples

3-1 twh_create_datasource script . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1023-2 Verification of central data warehouse database on z/OS . . . . . . . . . . 1193-3 Verification of data mart database on z/OS . . . . . . . . . . . . . . . . . . . . . 1233-4 twh_create_datasource script . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1243-5 Verify control server (twh_list_cs) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1363-6 Verify central data warehouse (twh_list_cdws) . . . . . . . . . . . . . . . . . . 1373-7 Verify data mart databases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1373-8 Verify remote agent site (twh_list_agentsites) . . . . . . . . . . . . . . . . . . . 1383-9 Verify Crystal Enterprise Professional for Tivoli installation . . . . . . . . . 1393-10 Verify data user . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1403-11 twh_configwep command output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1435-1 Verify NetView source database updates . . . . . . . . . . . . . . . . . . . . . . 1695-2 NetView tdwdaemon configuration file tdwdaemon.properties . . . . . . 1785-3 Restart the NetView data warehouse daemon tdwdaemon. . . . . . . . . 1785-4 Status of NetView data warehouse daemon (tdwdaemon) . . . . . . . . . 1795-5 Status of the NetView SNMP collector daemon (snmpcollect) . . . . . . 1795-6 Check the NetView source database. . . . . . . . . . . . . . . . . . . . . . . . . . 1806-1 Testing the RIM object . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2366-2 Datacollector configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2366-3 wdmlseng command output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2396-4 wdmcollect command output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2396-5 Sample SQL that check the collection . . . . . . . . . . . . . . . . . . . . . . . . . 2406-6 Running itmchk.sh tool. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2876-7 itmchk.sh tool report. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2886-8 Retrieving the date of last data upload into ITM database. . . . . . . . . . 2906-9 Names of the endpoints collecting data . . . . . . . . . . . . . . . . . . . . . . . . 2906-10 wrimtest command output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2916-11 Status of resource models distributed on an endpoint . . . . . . . . . . . . . 2926-12 msg_DataCollector.log. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2936-13 trace_tmnt_rimh_eng1.log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2946-14 trace_dmxengine.log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 295

© Copyright IBM Corp. 2004. All rights reserved. xvii

xviii Implementing Tivoli Data Warehouse 1.2

Notices

This information was developed for products and services offered in the U.S.A.

IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to evaluate and verify the operation of any non-IBM product, program, or service.

IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing, IBM Corporation, North Castle Drive Armonk, NY 10504-1785 U.S.A.

The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you.

This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice.

Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk.

IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you.

Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products.

This information contains examples of data and reports used in daily business operations. To illustrate them as completely as possible, the examples include the names of individuals, companies, brands, and products. All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental.

COPYRIGHT LICENSE: This information contains sample application programs in source language, which illustrates programming techniques on various operating platforms. You may copy, modify, and distribute these sample programs in any form without payment to IBM, for the purposes of developing, using, marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore, cannot guarantee or imply reliability, serviceability, or function of these programs. You may copy, modify, and distribute these sample programs in any form without payment to IBM for the purposes of developing, using, marketing, or distributing application programs conforming to IBM's application programming interfaces.

© Copyright IBM Corp. 2004. All rights reserved. xix

TrademarksThe following terms are trademarks of the International Business Machines Corporation in the United States, other countries, or both:

AIX®CICS®DataJoiner®DB2 Universal Database™DB2®Domino®DRDA®^™

™Everyplace®ibm.com®

IBM®IMS™Informix®Lotus®MQSeries®MVS™NetView®NetVista™OS/390®RACF®Redbooks (logo) ™

Redbooks™RMF™S/390®SP2®Tivoli Enterprise Console®Tivoli Enterprise™Tivoli®WebSphere®z/OS®

The following terms are trademarks of other companies:

Crystal and Crystal Enterprise are trademarks of Business Objects.

Intel and Intel Inside (logos) are trademarks of Intel Corporation in the United States, other countries, or both.

Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.

UNIX is a registered trademark of The Open Group in the United States and other countries.

Other company, product, and service names may be trademarks or service marks of others.

xx Implementing Tivoli Data Warehouse 1.2

Preface

With Tivoli® Data Warehouse, you can analyze historical trends from various Tivoli and customer applications. The Tivoli Data Warehouse infrastructure enables a set of extract, transform, and load (ETL) utilities to extract and move data from Tivoli application data stores to a central repository. The open architecture of Tivoli Data Warehouse also enables data from non-Tivoli applications to be integrated into its central repository. Data from the central repository can be extracted into data marts that pertain to the reporting needs of selected groups. These data marts can also be used to produce cross application reports.

This IBM Redbook focuses on planning, installation, customization, use, maintenance, and troubleshooting topics related to the new features of the Tivoli Data Warehouse version 1.2. This is done using a number of case study scenarios and several warehouse enablement packs.

The instructions given in this book are very detailed and explicit. These instructions are not the only way to install the products and related prerequisites. They are meant to be followed by anyone to successfully install, configure, and set up Tivoli Data Warehouse environments of any size.

The team that wrote this redbookThis redbook was produced by a team of specialists from around the world working at the International Technical Support Organization, Austin Center.

Edson Manoel is a Software Engineer at IBM Corporation - International Technical Support Organization, Austin Center, working as an IT Specialist in the Systems Management area. Prior to joining the ITSO, Edson worked in the IBM Software Group as a Tivoli Technology Ambassador and in IBM Brasil Professional Services Organization as a Certified IT Specialist. He was involved in numerous projects, designing and implementing systems management solutions for IBM customers and Business Partners. Edson holds a BSc degree in Applied Mathematics from Universidade de Sao Paulo, Brazil.

Cristiano Colantuono is an IT Specialist at IBM Tivoli Laboratory in Rome. He joined IBM in 1999, working in the distributed systems management area. He projected and implemented several Tivoli solutions for the IT infrastructures in Rome as well as in other European IBM development laboratories. Before joining IBM, Cristiano had also some experience as a Web developer and a Web

© Copyright IBM Corp. 2004. All rights reserved. xxi

administrator. He graduated in Physics at the University of Rome and collaborated with the Italian National Institute of Nuclear Physics developing simulation programs for high energy physics experiments.

Dr. Hans-Georg Köhne is a software architect for SerCon in Germany. He graduated in physics at the University of Muenster developing simulation programs for high energy physics experiments. He joined SerCon in 1996, working in the distributed systems management area. He planned and implemented several systems management solutions in the areas software distribution, availability management, and business automation.

Devi Raju is a Tivoli Implementation Specialist for IBM India. She started her career with IBM and has been with IBM for 8 years now. Devi has 4 years of experience in Enterprise System Management. She has worked in various large Tivoli customer projects. She is also a Tivoli Certified Consultant on PACO products.

Ghufran Shah is an IBM Certified Deployment Professional and an IBM Certified Instructor based in the UK with os-security.com. He holds a degree in Computer Science, and has over 8 years of experience in Systems Development and Enterprise Systems Management. As well as teaching Tivoli courses worldwide, his areas of expertise include Tivoli Systems Management Architecture, Implementation, and Training together with Provisioning and Orchestration. His focus in now on leveraging IBM solutions to provide customers with the vision and reality of an OnDemand environment.

Sergio Henrique Soares Monteiro is an IT Specialist in Brazil. He has over 10 years of experience in database administration and development fields. He has worked with Oracle, DB2, Informix and SQL Server on UNIX and Windows, including clustered servers. He currently works as a Database administrator in the CTI’s IBM in Hortolandia, Brazil. His areas of expertise include sizing, performance tuning, and internals of RDBMS.

Thanks to the following people for their contributions to this project:

Budi DarmawanInternational Technical Support Organization, Austin Center

David StephensonIBM Global Services, Australia

Diana MarcattiliIBM Global Services, Italy

Georg HolzknechtSenior Systems Consultant, T-Systems CDS GmbH, Germany

xxii Implementing Tivoli Data Warehouse 1.2

Jonathan Cook, Brian Jeffrey, Mike MalloTivoli Data Warehouse development team, IBM Software Group, Austin

Ken HanniganIBM Tivoli Storage Manager development team, IBM Software Group, Tucson

Yvonne Lyon, editorInternational Technical Support Organization, San Jose Center

Become a published authorJoin us for a two- to six-week residency program! Help write an IBM Redbook dealing with specific products or solutions, while getting hands-on experience with leading-edge technologies. You'll team with IBM technical professionals, Business Partners and/or customers.

Your efforts will help increase product acceptance and customer satisfaction. As a bonus, you'll develop a network of contacts in IBM development labs, and increase your productivity and marketability.

Find out more about the residency program, browse the residency index, and apply online at:

ibm.com/redbooks/residencies.html

Comments welcomeYour comments are important to us!

We want our Redbooks™ to be as helpful as possible. Send us your comments about this or other Redbooks in one of the following ways:

� Use the online Contact us review redbook form found at:

ibm.com/redbooks

� Send your comments in an Internet note to:

[email protected]

� Mail your comments to:

IBM® Corporation, International Technical Support OrganizationDept. JN9B Building 003 Internal Zip 283411400 Burnet RoadAustin, Texas 78758-3493

Preface xxiii

http://www.redbooks.ibm.com/residencies.html

http://www.redbooks.ibm.com/residencies.html


http://www.ibm.com/redbooks/


http://www.redbooks.ibm.com/contacts.html

xxiv Implementing Tivoli Data Warehouse 1.2

Part 1 Fundamentals

Part 1

© Copyright IBM Corp. 2004. All rights reserved. 1

2 Implementing Tivoli Data Warehouse 1.2

Chapter 1. Introducing Tivoli Data Warehouse 1.2

This chapter provides a brief introduction to the concepts, technologies, and products behind the Tivoli Data Warehouse, and the new features that can be found in Version 1.2. We cover the following topics:

� “Data warehousing basics” on page 4� “Tivoli Data Warehouse” on page 8� “What is new in Tivoli Data Warehouse 1.2” on page 10� “Tivoli Data Warehouse architecture” on page 20� “Benefits of using Tivoli Data Warehouse” on page 23

1


1.1 Data warehousing basicsData warehousing is the process of managing a data warehouse and its components, called data marts. This management process includes all the ongoing support needs of the refresh cycle, database maintenance, and continual refinements to the underlying data model. In addition to that, data warehousing can be thought of as a tool to enable and support business intelligence.

The concept of data warehousing carries several other important terms mentioned in the above paragraph. Such terms will be explained in the sections to follow. They are:

� Data warehouse� Data mart� Business intelligence� Data mining

1.1.1 Data warehouse A data warehouse is the cohesive data model that defines the central data repository for an organization. An important point is that we don't define a warehouse in terms of the number of databases. Instead, we consider it a complete, integrated data model of the enterprise, regardless of how or where the data is stored.

A data warehouse is a collection of databases where data is collected for the purpose of being analyzed. This collection of databases can be formed by one or more databases. The defining characteristic of a data warehouse is its purpose. Most data is collected to handle a company's on-going business. This type of data can be called operational data. The systems used to collect operational data are referred to as OLTP.

A data warehouse collects, organizes, and makes data available for the purpose of analysis in order to give management the ability to access and analyze information about its business. This type of data can be called informational data. The systems used to work with informational data are referred to as online analytical processing (OLAP).

Bill Inmon coined the term data warehouse in 1990. His definition is as follows:

“A (data) warehouse is a subject-oriented, integrated, time-variant, and non-volatile collection of data in support of management's decision-making process.”


These are the main types of data:

� Subject-oriented: Data that gives information about a particular subject instead of about a company's on-going operations

� Integrated: Data that is gathered into the data warehouse from a variety of sources and merged into a coherent whole

� Time-variant: All data in the data warehouse that is identified with a particular time period

1.1.2 Data martA data mart is a repository containing data specific to a particular business group in an enterprise. All data in a data mart derives from the data warehouse, and all data relates directly to the enterprise wide data model. Often, data marts contain summarized or aggregated data that the user community can easily consume.

Another way to differentiate a data warehouse from a data mart is to look at the data's consumers and format. IT analysts and canned reporting utilities consume warehouse data, whose storage is usually coded and cryptic. The user community consumes data mart data, whose storage is usually in a more readable format. For example, to reduce the need for complex queries and assist business users who might be uncomfortable with the SQL language, data tables could contain the de-normalized code table values.

A data mart contains a subset of corporate data that is of value to a specific business unit, department, or set of users. This subset consists of historical, summarized, and possibly detailed data captured from transaction processing systems, or from an enterprise data warehouse. It is important to realize that a data mart is defined by the functional scope of its users, and not by the size of the data mart database. In parallel to increasing data mart usage, the underlying databases will rapidly increase in size.

1.1.3 Business intelligenceBusiness intelligence (BI) is not business as usual. It is about making better decisions more quickly and easily.

Businesses collect enormous amounts of data every day: Information about orders, inventory, accounts payable, point-of-sale transactions, and, of course, customers. Businesses also acquire data, such as demographics and mailing lists, from outside sources. Unfortunately, based on a recent survey, over 93 percent of corporate data is not usable in the business decision-making process today. This applies also to systems management, where data tends to be of more technical nature.

Chapter 1. Introducing Tivoli Data Warehouse 1.2 5

Consolidating and organizing data for better business decisions can lead to a competitive advantage, and learning to uncover and leverage those advantages is what business intelligence is all about.

The amount of business data is increasing exponentially. In fact, it doubles every two to three years. More information means more competition. In the age of the information explosion, executives, managers, professionals, and workers all need to be able to make better decisions faster. Because now, more than ever, time is money.

Much more than a combination of data and technology, BI helps you to create knowledge from a world of information. Get the right data, discover its power, and share the value, BI transforms information into knowledge. Business intelligence is the application of putting the right information into the hands of the right user at the right time to support the decision-making process.

Business driving forcesIt can be noted that there are some business driving forces behind business intelligence, one being the need to improve ease-of-use and reduce the resources required to implement and use new information technologies. Other driving forces behind business intelligence include these:

� The need to increase revenues, reduce costs, and compete more effectively.

Gone are the days when end users could manage and plan business operations using monthly batch reports, and IT organizations had months to implement new applications. Today companies need to deploy informational applications rapidly, and provide business users with easy and fast access to business information that reflects the rapidly changing business environment. Business intelligence systems are focused towards end user information access and delivery, and provide packaged business solutions in addition to supporting the sophisticated information technologies required for the processing of today’s business information.

� The need to manage and model the complexity of today’s business environment.

Corporate mergers and deregulation means that companies today are providing and supporting a wider range of products and services to a broader and more diverse audience than ever before. Understanding and managing such a complex business environment and maximizing business investment is becoming increasingly more difficult. Business intelligence systems provide more than just basic query and reporting mechanisms, they also offer sophisticated information analysis and information discovery tools that are designed to handle and process the complex business information associated with today’s business environment.


� The need to reduce IT costs and leverage existing corporate business information.

The investment in IT systems today is usually a significant percentage of corporate expenses, and there is a need not only to reduce this overhead, but also to gain the maximum business benefits from the information managed by IT systems. New information technologies like corporate intranets, thin-client computing, and subscription-driven information delivery help reduce the cost of deploying business intelligence systems to a wider user audience, especially information consumers like executives and business managers. Business intelligence systems also broaden the scope of the information that can be processed to include not only operational and warehouse data, but also information managed by office systems and corporate Web servers.

1.1.4 Data miningData mining is the process of extracting valid, useful, previously unknown, and comprehensible information from data and using it to make business decisions.

The organizations of today are under tremendous pressure to compete in an environment of tight deadlines and reduced profits. Legacy and lengthy business processes that require data to be extracted and manipulated prior to use will no longer be acceptable. Instead, enterprises need rapid decision support based on the analysis and forecasting of predictive behavior. Data warehousing and data mining techniques provide this capability.

Data mining can be defined as the extraction of hidden predictive information from large databases, and is a powerful technology with great potential to help companies focus on the most important information in their data warehouses. Once a Tivoli Data Warehouse has been established, data mining tools can then be used to predict future trends and behaviors, allowing businesses to make proactive, knowledge-driven decisions.

Data mining tools can answer business questions that traditionally were too time consuming to resolve. These tools hunt databases for hidden patterns, finding predictive information that experts may miss because it lies outside their expectations.

The art of data mining is not trivial, and it can be similar to “finding the needle in the haystack”. In this case, the needle is that single piece of intelligence your business needs, and the haystack is the large data warehouse you've built up over a period of time within your business.


Most companies already collect and analyze massive quantities of data. Data mining techniques can be implemented rapidly on existing software and hardware platforms to enhance the value of existing information resources, and can be integrated with new products and systems as they are brought on-line.

Given databases of sufficient size and quality, data mining technology can generate new business opportunities by providing these capabilities:

� Automated prediction of trends and behaviors: Data mining automates the process of finding predictive information in large databases. Questions that traditionally required extensive hands-on analysis can now be answered directly from the data and quickly. A typical example of a predictive problem is targeted server performance. Data mining uses data on past critical events to identify the servers most likely to cause future critical problems. Other predictive problems include forecasting server outage and other forms of performance degradation that is likely to occur, given certain events.

� Automated discovery of previously unknown patterns: Data mining tools sweep through databases and identify previously hidden patterns in one step. An example of pattern discovery is the analysis of IBM Tivoli Monitoring data to identify seemingly unrelated events that are often received together.

1.2 Tivoli Data WarehouseThe Tivoli Data Warehouse 1.2 is built on an IBM DB2® Data Warehouse. It offers all IBM DB2 Data Warehouse functionality with additional Tivoli specific extensions.

The IBM Data Warehouse Management uses the IBM DB2 Universal Database Enterprise Edition and the IBM DB2 Data Warehouse Manager feature. It provides an integrated, distributed, heterogeneous warehouse management infrastructure for designing, building, maintaining, governing, and accessing highly scalable, robust data warehouses, operational data stores, and data marts stored in IBM DB2 databases.

IBM DB2 Data Warehouse Manager helps warehouse administrators:

� To manage data volumes, to move data directly from source to target (also allowing packaged and simplified access to popular partner products such as SAP R/3), and to control the servers on which transformations take place with distributed warehouse agents

� To speed warehouse and data mart deployment with commonly used, pre-built data cleansing and statistical transformations

� To build and manage from a central point of control, integrated in IBM DB2, utilizing the Data Warehouse Center graphical user interface


DB2 warehouse management consists of:

� An administrative client to define and manage data warehousing tasks and objects, and warehouse or data mart operations: the Data Warehouse Center

� A manager to manage and control the flow of data: the warehouse server

� Agents residing on IBM DB2 Universal Database Enterprise Edition server platforms to perform requests from the manager or warehouse server: the local or remote warehouse agent

� A warehouse control database storing the warehouse management metadata on a IBM DB2 database server

� A metadata administrative and publishing tool with its own administration graphical user interface (GUI): Information Catalog Manager to manage and present both technical and business metadata

The different components of the IBM DB2 Data Warehouse Manager are shown in Figure 1-1.

Figure 1-1 IBM DB2 Data Warehouse Manager

Clients Warehouse Server

Warehouse Agents

Databases

RelationalSource

Non- RelationalSource

yy

Data

Message

Message

Message

DB2 Target

End Users

Data

Data

Data

Data

Data

Non-DB2 Target

LogEditionsConfiguration

ControlDatabase

DB2

Metadata

Metadata

Flat Files, Web or SAP R/3

Data

Included with IBM DB2

Data Warehouse

Center

Message


1.3 What is new in Tivoli Data Warehouse 1.2Tivoli Data Warehouse 1.2 provides a number of enhancements and new features over Version 1.1, such as these:

� Improved interfaced and Web-based reporting using Crystal Enterprise™� DB2 UDB for OS/390 and z/OS support� Flexible and extended configuration support� Installation enhancements� Serviceability and scalability improvements

We now discuss each of these areas.

1.3.1 Crystal Enterprise™

Among the enhancements that Tivoli Data Warehouse 1.2 provides, an important change is the new mechanism for producing reports and the user interface. Version 1.1 of Tivoli Data Warehouse used Tivoli Presentation Services and the IBM Console. Tivoli Data Warehouse 1.2 does not use the IBM Console nor Tivoli Presentation Services. The reporting technology is now provided by Crystal Enterprise™, by Business Objects, which is a world standard for high-quality and high-performance reporting.

Crystal Enterprise™ provides:

� Out-of-the-box Web-based reporting and information delivery for all your Tivoli products

� An extendable, scalable reporting solution to meet the information delivery needs of your IT organization

� A report scheduling capability

� An export feature to export reports to variety of formats (Excel, Word, PDF)

� The capability to change the look and feel of the reports

The Tivoli Data Warehouse 1.2 comes supplied with Crystal Enterprise Professional Version 9 for Tivoli (limited use version) to analyze and deliver out-of-the-box Reports from the Tivoli Data Warehouse into the hands of decision-makers using a Web browser.

This will allow for a rapid return on investment you have made in your Tivoli solution, by providing out-of-the-box Web based reporting, including scheduling and report export capability. All of this is achieved using a customizable platform for organizing, categorizing, and delivering information.


If we define a report as an entity that visualizes the output of SQL clauses, or an “SQL Pull”, then the Crystal Enterprise Professional Version 9 for Tivoli, which is shipped with the Tivoli Data Warehouse 1.2 product, is supplied with a number of standard reports provided by the Tivoli Data Warehouse Enablement Packs (WEPs). When a report is made available by the WEP to Crystal Enterprise, the layout, legends, colors, and the look-and-feel of the report can all be customized.

However, to create a new report (using the definition above), or to modify the SQL pull criteria of an existing report, Crystal Reports and a different version of Crystal Enterprise™ is required: Crystal Enterprise Version 9 Special Edition. A license for Crystal Enterprise Version 9 Special Edition must be purchased separately. The Crystal Enterprise Version 9 Special Edition will allow you to:

� Extend your reporting capabilities to develop, deliver, and analyze new reports created from your Tivoli Systems Management Data using Crystal Reports version 9

� Provide support for approximately 75 concurrent online users

� Add, modify, and design new reports from your Tivoli Systems Management Data using Crystal Reports version 9

For the tasks listed above, Crystal Reports Version 9 Special Edition is required and must be purchased separately.

Next we present a brief introduction to the Crystal Enterprise architecture. As shown in Figure 1-2, by using the Crystal Enterprise™ multi-tier architecture, the IBM Tivoli product portfolio has a key partnership developed to ensure the deepest level of integration and ongoing support for this solution. Please note that some of the functions may not be available on the Crystal Enterprise Professional for Tivoli product.


Figure 1-2 Crystal Enterprise multi-tier architecture

In Crystal Enterprise, there are four tiers, each of which can be installed on one machine, or with the Crystal Enterprise Version 9 Special Edition, spread across many. The Crystal Enterprise architecture tiers are as follows:

� Client tier: Administrators and end users interact with this component directly, which is made up of the applications that enable people to administer, publish, and view reports.

� Intelligence tier: These components manage the Crystal Enterprise administration system, which consists of maintaining all aspects of the security information, storing report instances, and controlling the flow of requests to the appropriate servers.

Client TierCrystal Management ConsoleeProtfolioCrystal Configuration ManagerPublishing WizardImport Wizard

Browser or Crystal applications

Intelligence TierWeb Component Server

Web Server / Web Connector

File Repository Server

Cache Server

Automated Process Scheduler

Event Server

Processing TierJob Server

Page Server Report Application Server

Data TierOLAP

Relational ODBC XML, ERP, CRM, COM

Cry

stal

En

terp

rise

Fra

mew

ork


� Processing tier: These components access the data and generate the reports. This is the only tier that communicates directly with the databases that contain the report data.

� Data tier: The databases that contain the data used in the reports fall into this tier. These databases are referred as Data Sources in Crystal Enterprise, and a wide range of databases are supported. These databases could contain historic data and/or operational data.

This redbook does not go into the details of Crystal Enterprise Professional Version 9 for Tivoli administration and configuration. Refer to the following documentation shipped with the product:

� Crystal Enterprise 9 Installation Guide� Crystal Enterprise 9 Administrator’s Guide� Crystal Enterprise 9 Getting Started Guide� Crystal Enterprise 9 ePortfolio User’s Guide

1.3.2 IBM DB2 UDB for OS/390 and z/OS supportOn z/OS® systems, operational data is extracted from a Tivoli Decision Support for OS/390® database.

As the next generation of historical data reporting and analysis solutions, Tivoli Data Warehouse is a successor to Tivoli Decision Support (TDS) on distributed platforms (Wintel/UNIX) and a companion product for Tivoli Decision Support for OS/390. The following sections explain how Tivoli Data Warehouse relates to and works with Tivoli Decision Support and Tivoli Decision Support for OS/390.

Tivoli Decision Support for OS/390 and Tivoli Data WarehouseThe following items compare and contrast Tivoli Data Warehouse and Tivoli Decision Support:

� Both Tivoli Data Warehouse and Tivoli Decision Support collect and analyze data gathered by the system management products in your enterprise.

� Both provide an infrastructure for reporting and analysis, but do not themselves extract data or provide reports. Each relies on other applications to use the infrastructure to extract and analyze data and to provide reports that satisfy a specific reporting or analysis need.

� In Tivoli Decision Support, an application that provides a solution to a specific reporting need is called a Tivoli Decision Support Guide. In Tivoli Data Warehouse, the corresponding application is called a Warehouse Enablement Pack.


� Some Tivoli Decision Support Guides require direct access to the data in your operational data stores, which can decrease the performance of the products creating and using those data stores. Tivoli Data Warehouse ensures that your operational data stores are not impacted by users running reports. It also ensures that users can run reports efficiently by accessing databases that are optimized for interactive reporting.

� By saving historical data in a central location and in a common format, Tivoli Data Warehouse makes it easier to create reports that draw on data collected by more than one product.

� Tivoli Decision Support stores and accesses data using Cognos Powerplay and Crystal Reports. In contrast, Tivoli Data Warehouse publishes the format of its data, as well as the format of the data in the products that feed the warehouse, allowing the use of various reporting tools.

� This enables you to use the business intelligence solutions you already know. In addition, Tivoli software uses Crystal Enterprise, which is provided with Tivoli Data Warehouse, as a common reporting solution.

� Tivoli Data Warehouse provides support for multiple languages. Tivoli Decision Support is available only in English. Tivoli Decision Support for OS/390 is available in English and Japanese.

TDS for OS/390 and Tivoli Data Warehouse 1.2 InteractionThis section describes how Tivoli Decision Support for OS/390 works with Tivoli Data Warehouse 1.2 to store and aggregate data.

Tivoli Decision Support for OS/390 collects system management data from System Management Facility (SMF), Information Management System (IMS™), and other logs. It aggregates and summarizes data on a hourly, daily, and monthly basis and places the data into its own database.

The Tivoli Decision Support for OS/390 database contains data primarily from z/OS systems. Tivoli Data Warehouse 1.2 can use the Tivoli Decision Support for OS/390 databases as an operational data source for the z/OS applications.

The flow of data between Tivoli Decision Support for OS/390 and Tivoli Data Warehouse 1.2 can be seen in Figure 1-3.


Figure 1-3 TDS OS/390 and TDW 1.2 Data flow

Data from z/OS systemsData from z/OS systems can only be placed in a central data warehouse on z/OS. A set of Extract Transform and Load (ETL) programs from a warehouse enablement pack will extract data from the TDS for OS/390 database, transform, and place the data into the central data warehouse on the z/OS system.

Another set of ETLs will then extract the data from the central data warehouse database on the z/OS system and load the data into data mart databases. There are no restrictions on the location of the data mart databases. These databases can be located on z/OS systems or any other supported platforms. You can then analyze the data and generate reports from the data contained in the data mart databases.

Central DataWarehouseCentral DataWarehouse

Data sourceIMS

Web Reports

Web Server

Crystal Server

Web Server

Crystal Server

Data sourceSMF

Data sourceLogs

Reports

Source Data Consolidated by TDS for OS/390

TDS for OS/390

TDW 1.2 CDW on OS/390

Data MartData Mart

TDW 1.2 Control Center


Figure 1-4 shows how data from the distributed environment and the OS/390 or z/OS environment can be encapsulated to produce real, end-to-end enterprise reporting.

Figure 1-4 Distributed and OS/390 Data feeds into Tivoli Data Warehouse

For additional details on Tivoli Data Warehouse 1.2 components placement, refer to Chapter 2, “Planning for Tivoli Data Warehouse 1.2” on page 27.

IMS

SMF

Logs

Reports

Source Data Consolidated by TDS for OS/390

TDS for OS/390

ITM

SLA

TEC

Reports

IT Infrastructure Management Data


Central Data Warehouse on OS/390 ETL

Central DataWarehouseCentral DataWarehouseCentral Data

WarehouseCentral DataWarehouse

Central Data Warehouse distributed

Data Mart for reporting functionality

ETL

ETL

Other Business Intelligence Tools

Other Reporting Tools

Crystal Enterprise

Other Data Analysis Tools

Gain the competitive edge

Data MartData Mart


1.3.3 Flexible and extended configuration supportYou can have up to four central data warehouse databases and up to four data mart databases on Windows®, UNIX®, or z/OS systems, and multiple source application support controlled by Tivoli Data Warehouse 1.2, as shown in Figure 1-5.

Figure 1-5 Multiple source applications loading into a central data warehouse


Data sourceApplication 1


Data sourceApplication N

....




....




....




....




Data MartData Mart



Data MartData Mart

Data MartData Mart

Data MartData Mart


However, you could also keep data about multiple customers and data centers in one central data warehouse database, while restricting access so that customers can see and work with data and reports based only on their data and not any other customer’s data. For example, support for multiple customer environments enables a service provider to keep historical data about all of its customers in one deployment of Tivoli Data Warehouse. Multiple data center support provides a way to partition data physically by customizable criteria such as location, application, or business purpose.

1.3.4 Installation enhancementsThe following enhancements have been made to the installation process of Tivoli Data Warehouse 1.2:

� The Tivoli Data Warehouse product and Warehouse Enablement Pack (WEP) installation processes have been separated:

– In version 1.1 of the Tivoli Data Warehouse, to install a warehouse enablement pack, it was necessary to launch the installation program of the Tivoli Data Warehouse. In Tivoli Data Warehouse 1.2, the installation processes for the base product and WEPs have been separated. Once the Tivoli Data Warehouse base product has been installed, a shortcut is created, which launches the WEP Installation Wizard.

� The Warehouse Enablement Pack Installer:

– Allows installation and un-installation as well as patch installs from the same GUI

– Leaves the warehouse enablement pack in a scheduled state

� Many manual steps have been automated, such as:

– DB2 Warehouse Manager Remote Agent discovery and configuration

– Client/Server connections for IBM DB2 or Oracle

– Creation of ODBC data sources on Windows, AIX®, and Solaris

– Auto-configuration of Data Warehouse Center Metadata, including passwords and schedules


1.3.5 Serviceability and scalability improvementsThe following serviceability improvements have been made to Tivoli Data Warehouse 1.2:

� Centralized logging in XML and log viewer tool:

Tivoli applications support a common XML format, in which they log messages and traces. This common format is called Log XML. The LogXML Viewer processes logs in that format, enabling you to view the logs in any Web browser in an easy-to-read format. The viewer is installed with Tivoli Data Warehouse and is located in the Tivoli common logging directory/cdw/tools directory. The LogXML Viewer converts the logged messages into ASCII or HTML for presentation. Visual cues are associated with error and warning messages.

� HTML viewer that enables you to view your logs online.

� Flexible tracing capability.

� First failure data capture:

First Failure Data Capture (FFDC) is the ability to capture useful data after a problem has been logged. The purpose of FFDC features is to collect enough data so that a problem can be diagnosed without having to reproduce it a second time. Since it is likely that a failure will happen while the system is in an unattended mode, it is important to capture this data in such a way that it is not overwritten, before it can be gathered and sent to a support center or help desk. Data that is captured as part of FFDC can include trace logs, message logs, dumps of in-memory data structures, and so forth.

For scalability, Tivoli Data Warehouse 1.2 now provides:

� Support for multiple instances of operational data source. A warehouse enablement pack can extract operational data from multiple instances of the same data source.

� Capability to grow the environment overtime by adding central data warehouses and data marts after the initial installation.


1.4 Tivoli Data Warehouse architectureTivoli Data Warehouse 1.2 is made available by Tivoli to consolidate historical data from different management applications into one or more central data warehouse databases, and to make it easy to analyze comprehensible information from data stored into its data mart databases, thus facilitating the business decision making process. Figure 1-6 shows how the Tivoli Data Warehouse technology together with Crystal Enterprise can be used to provide an end-to-end business intelligence enabler.

Figure 1-6 Tivoli Data Warehouse — the big picture

A Tivoli Data Warehouse 1.2 architecture can be composed of the following elements:

� Tivoli Data Warehouse control center server� One or more central data warehouse databases� One or more data mart databases� IBM DB2 warehouse agents and agents sites� Crystal Enterprise server

Tivoli Data Warehouse 1.2 Technology


Data MartData Mart

Central DataWarehouseCentral DataWarehouse Central Data

WarehouseCentral DataWarehouseCentral Data


Web Server

CrystalEnterprise

Server

OLAP, Business Intelligence

& Analysis Tools

Crystal Web InterfaceWeb Reports

Data sourceITM

Data sourceSLA

Data sourceTEC

IT Infrastructure Management Data

Data sourceAppl2

Data sourceAppl1Operational Data


The following sections describe these elements in detail.

1.4.1 Tivoli Data Warehouse control center serverThe control center server is the system that contains the control database for Tivoli Data Warehouse and is the system from which you manage your data. The control database contains metadata for both Tivoli Data Warehouse and for the warehouse management functions of IBM DB2 Universal Database™ Enterprise Edition. There can only be one control server in a Tivoli Data Warehouse 1.2 deployment.

1.4.2 Source databasesA source databases holds operational data worth to be loaded into the Tivoli Data Warehouse environment. Typically, the source databases are application specific and their number is likely to increase for a data warehouse installation.

Most Tivoli products provide a warehouse enablement pack which makes application specific data available in a source database. This can be a dedicated warehouse source database as it is coming with IBM Tivoli Monitoring, or it can be an interface to the applications built in database as provided for IBM Tivoli Storage Manager or IBM Tivoli NetView®. WEP for Tivoli products also include a means to upload data from the source database to the central data warehouse, thus minimizing the efforts for data collection.

1.4.3 Central data warehouseThe central data warehouse is a set of IBM DB2 databases that contains the historical data for your enterprise. You can have up to four central data warehouse databases in a Tivoli Data Warehouse 1.2 deployment.

1.4.4 Data martsA separate set of IBM DB2 databases contains the data marts for your enterprise. Each data mart contains a subset of the historical data from the central data warehouse that satisfies the analysis and reporting needs of a specific department, team, customer, or application. You can have up to four data mart databases in a Tivoli Data Warehouse 1.2 deployment. Each data mart database can contain the data for multiple central data warehouse databases.

A WEP for a Tivoli application provides all the necessary means to fill data marts with their specific data.


1.4.5 Warehouse agents and agent sitesThe warehouse agent is the component of IBM DB2 Warehouse Manager that manages the flow of data between data sources and targets that are on different computers. By default, the control center server uses a local warehouse agent to manage the data flow between operational data sources, central data warehouse databases, and data mart databases. You can optionally install the warehouse agent component of IBM DB2 Warehouse Manager on a computer other than the control center server.

Typically, you place an agent on the computer that is the target of a data transfer. That computer becomes a remote agent site, which the Data Warehouse Center uses to manage the transfer of Tivoli Data Warehouse data. This can speed up the data transfer as well as reduce the workload on the control server.

1.4.6 Crystal Enterprise ServerCrystal Enterprise Professional for Tivoli completely replaces the Reports Interface of TEDW 1.1 version. It gives us a new mechanism for obtaining the Reports provided by the Warehouse Enablement Packs. The installation and configuration of a Crystal Enterprise environment is mandatory before you begin installing Tivoli Data Warehouse 1.2. Tivoli Data Warehouse 1.2 supports only the full stand-alone installation of Crystal Enterprise. In the full stand-alone installation, Crystal Enterprise is installed on a single computer that is already running as a Web Server.

Crystal Enterprise is dependent of a number of software components that must be up and running prior to its installation:

� Operating Systems: Windows NT®, Windows 2000, or Windows 2003

� Internet Browser: Internet Explorer or Netscape Navigator

� Web Servers: IBM HTTP Server, Microsoft® IIS, iPlanet Enterprise Server, or Lotus® Domino®


Figure 1-7 shows an overview of the Tivoli Data Warehouse 1.2 architecture and supported software components.

Figure 1-7 Detail Component view of Tivoli Data Warehouse 1.2

1.5 Benefits of using Tivoli Data Warehouse By using Tivoli Data Warehouse, you can access the underlying data about your IT infrastructure, including network devices and connections, desktops, hardware, software, events, and other information. With this information in a data warehouse, you can look at your IT costs, performance, and other trends across your enterprise. Tivoli Data Warehouse can be used to show the value and return on investment of Tivoli and IBM software, and it can be used to identify areas where you can be more effective.

Tivoli Data Warehouse provides the following capabilities:

� It has an open architecture for storing, aggregating, and correlating historical data.

� Most Tivoli products are shipped with warehouse enablement packs to provide an easy and fast integration of the Tivoli products with the Tivoli Data Warehouse.

Central DataWarehouse Data MartData MartData MartData MartStar Schema

Data MartData MartData MartData MartStar Schema

DB2 UDB EE & DB2/390

Applications’Data Store

WM Agent

ETL1ETL2

Web-based Reports

AIX,Sun Solaris, NT/2K, MVS

AIX,Sun Solaris, HP-UX,NT/2K, OS/390, Turbo,

RedHat and SuSE Linux

IE 5.5 SP2 & 6.0Netscape 6.2.3


Web Server

CrystalEnterprise

Server

Win NT/2000/2003

Win NT/2000

IBM HTTP ServerIIS v4 & v5

iPlanetLotus Domino

Crystal ePortfolio

Data Mart


� In addition to the data collected by diverse IBM Tivoli software, Tivoli Data Warehouse has the flexibility and extensibility to enable you to integrate your own application data.

� It offers database optimizations both for the efficient storage of large amounts of historical data and for fast access to data for analysis and report generation.

� It provides the infrastructure and tools necessary for maintaining the data:

– These tools include the Tivoli Data Warehouse application, IBM DB2 Universal Database Enterprise Edition, the Data Warehouse Center, the DB2 Warehouse Manager, and Crystal Enterprise.

– It includes the ability to use your choice of data analysis tools to examine your historical data.

– In addition to the Crystal Enterprise reporting solution that is shipped with Tivoli Data Warehouse, you can analyze your data using any product that performs online analytical processing (OLAP), planning, trending, analysis, accounting, or data mining.

� It offers multi-customer and multicenter support:

– You can keep data about multiple customers and multiple data centers in one warehouse, but restrict access so that customers can see and work with data and reports based only on their own data and not any other customer’s data.

– You can also restrict an individual user’s ability to access data.

� It includes internationalization support:

– Reports can be displayed in the language of the user’s choice. Crystal Enterprise only comes in English, French, German, and Japanese; however, the reports generated by Crystal are translated into Brazilian Portuguese, French, German, Italian, Spanish, Japanese, Korean, Simplified Chinese, and Traditional Chinese.

– This means that even if you are running the Crystal Enterprise server in English, you could view a report in Italian through the Crystal Report Viewer Web interface if that language is set as your locale preference.


As shown in Figure 1-8, the Tivoli Data Warehouse 1.2 could be used as a single integration point for all Systems Management data, and it could also be used as both tool and technology to drive business intelligence within your enterprise.

Figure 1-8 Integrated Systems Management

Tivoli Data WarehouseThird Party Systems Management Applications

Provisioning

Tivoli Service Level Advisor

Reporting and B

usiness Intelligence Vendors

Business Impact Management

Event Correlation & Automation

Monitoring

Storage

Security

Tivoli Data WarehouseThird Party Systems Management Applications

ProvisioningProvisioning

Tivoli Service Level Advisor

Reporting and B

usiness Intelligence Vendors

Business Impact Management

Event Correlation & Automation

Monitoring

Storage

SecuritySecurity


Chapter 2. Planning for Tivoli Data Warehouse 1.2

This chapter provides planning considerations for Tivoli Data Warehouse. We cover the following topics:

� “Hardware and software requirements” on page 28� “Physical and logical design considerations” on page 36� “Database sizing” on page 56� “Security” on page 57� “Network traffic considerations” on page 61� “Integration with other business intelligence tools” on page 64� “ETL development” on page 65� “Skills required for a Tivoli Data Warehouse project” on page 67

2


2.1 Hardware and software requirementsTivoli Data Warehouse 1.2 can be deployed in two different configurations:

� Quick start installation, also known as stand-alone installation, with all the components installed on a single Microsoft Windows NT, Windows 2000, or Windows 2003 system. This is convenient for demonstrations, as an educational or test platform, and for companies that do not plan to have many users concurrently accessing the data stored in the Tivoli Data Warehouse databases and/or those that do not need to capture and analyze large amounts of data.

� Distributed installation, with the components installed on multiple systems in your enterprise, including UNIX and z/OS servers. See “Software requirements” on page 30 to determine the operating systems supporting each component of Tivoli Data Warehouse.

The historical reporting for Tivoli Data Warehouse 1.2 is provided by Crystal Enterprise, which can be installed in three different configurations, depending on the version used:

� Full stand-alone installation on a single Windows system that is already running as a Web server. This provides the quickest way to install Crystal Enterprise.

� Server-side installation connected to a Web server, which allows you to maintain separation between Crystal Enterprise and the Web server by running them on separate machines. In this scenario the Web server can also be on a UNIX system.

� Expanded installation, which allows you to install Crystal Enterprise server components on more machines in order to create an Automated Process Scheduler (APS) cluster, to increase available resources and to distribute the processing workload.

In the following sections we provide the current hardware and software requirements for a Tivoli Data Warehouse environment, but you should also check the Tivoli Data Warehouse Release Notes, SC32-1399, for possible updates about these requirements.

Note: IBM ships a limited license of Crystal Enterprise Professional Version 9 for Tivoli with Tivoli Data Warehouse 1.2 that only allows the full stand-alone installation option.


2.1.1 Hardware requirementsThis section provides information about the hardware requirements for installing Tivoli Data Warehouse 1.2 components. Please be aware that as the warehouse enablement packs are added to the Tivoli Data Warehouse 1.2 installation, additional hard disk space is required. See the documentation provided by the Warehouse Enablement Pack for application planning information and additional hard disk space requirements.

Table 2-1 lists the recommended hardware requirements for Tivoli Data Warehouse 1.2 for both stand-alone and distributed configurations.

Table 2-1 Hardware recommendations for Tivoli Data Warehouse components

Disk space requirements for central data warehouse and data marts can vary greatly according to the amount of actual collected data. See 2.3, “Database sizing” on page 56 to help you in evaluating the storage required by the different metrics you wish to collect in your own environment.

Note: The values in Table 2-1 differ from the information provided in the Tivoli Data Warehouse Release Notes, SC32-1399. They represent the hardware used at the time of writing this book and serve as our recommended starting configuration for both stand-alone and distributed configurations.

Installation configuration

Tivoli Data Warehouse components

Recommended size

Stand-alone installation All 1.0 GB RAM2.4 GHz processor 30 GB disk

Distributed installation Control server 512 MB RAM1.3 MHz processor10 GB disk

Central data warehouse 512 MB RAM1.3 MHz processor (400 MHz CPU or equivalent for UNIX)30 GB disk

Data mart 512 MB RAM1.3 MHz processor (400 MHz CPU or equivalent for UNIX)30 GB disk

Chapter 2. Planning for Tivoli Data Warehouse 1.2 29

Table 2-2 lists the hardware requirements for additional components of a Tivoli Data Warehouse 1.2 solution.

Table 2-2 Additional hard disk space requirements

The storage required by the installation of Crystal Enterprise Professional Version 9 for Tivoli shown in Table 2-2 assumes a full stand-alone installation of Crystal Enterprise Professional Version 9 for Tivoli. Please note that on the Crystal Enterprise server additional storage is required for Web server, database client software and all the reports installed by each warehouse enablement pack.

2.1.2 Software requirementsTable 2-3 provides information about the software requirements for the Tivoli Data Warehouse 1.2 components.

Component Recommended hardware

Crystal Enterprise Professional Version 9 for Tivoli

1.0 GB RAM2.4 GHz processor 30 GB disk (1 GB for installation alone)

Warehouse agent 50 MB disk

Note: You might receive confusing error messages if your systems do not meet the software requirements listed in this section.


Table 2-3 Software requirements

Tivoli Data Warehouse core components

Operating system

Data source

Warehouse agents

Control server

Central data warehouse

Data mart database

Crystal Back End

Crystal Weba

a. The Crystal Enterprise limited edition provided with Tivoli Data Warehouse requires that the Webserver is on the same system of Crystal Enterprise server

Microsoft® Windows NT®, service pack 6 or higher and Microsoft Data Access Components (MDAC) 2.7 service pack 1

Windows 2000 Server SP2®+

Windows 2000 Advanced Server SP2+

Windows 2003 Server

Yes Yes Yes Yes Yes Yes,also NT4 Server SP6a

Yes

IBM AIX® 4.3.3, 5.1, and 5.2

Yes Yes No Yes Yes No 4.3.3, 5.1 only

Sun Solaris Versions 2.8 and 2.9

Yes Yes No Yes Yes No 2.8 only

RedHat Linux Version 7.1, 7.2, 7.3 and Advanced Server 2.1

Yes No No No No No Red Hat 7.2, 7.3 only

SuSE Linux Version 7.2

Yes No No No No No SuSe 7.3, 8.0

Turbo Linux 7 Yes No No No No No No

zOS 1.2, 1.3, 1.4

Yes No No Yes Yes No No


2.1.3 Database requirementsTivoli Data Warehouse 1.2 requires the minimum level of IBM DB2 Universal Database Enterprise Edition 7.2 to be at Fix Pack 8e (that is, Fix Pack 8 plus the e-fix for Fix Pack 8). If you are currently running an IBM DB2 version prior to 7.2, you must upgrade to version 7.2 and apply the Fix Pack 8 and e-fix for Fix Pack 8, at minimum.

Even though IBM DB2 Universal Database Enterprise Edition 7.2 Fix Pack 8e is the minimum level of IBM DB2 required, Tivoli Data Warehouse 1.2 is bundled with IBM DB2 Universal Database Enterprise Edition 7.2 at the Fix Pack 10a level (FP10a). It is recommended to use the IBM DB2 Universal Database Enterprise Edition media provided with Tivoli Data Warehouse 1.2 to ensure the use of the correct levels of software and Fix Pack.

For warehouse databases on z/OS systems, the supported database version is DB2 Universal Database for OS/390 and z/OS V7.1 with the DB2 Management Clients Package installed (FMID JDB771D).

The recommended hard disk space in Table 2-1 on page 29 may not be large enough to accommodate some data growth as transactions are added to the database. However, when you plan for your database requirements, you must consider the following possibilities:

� Future data growth� Addition of warehouse enablement packs

It is recommended that you install your central data warehouse on an expandable system with a minimum of 30 GB of data space.

The data source support is determined by Open Database Connectivity (ODBC) support in IBM DB2 releases and specified by the warehouse enablement pack product.

Note: IBM DB2 Universal Enterprise Edition Version 8 and above are not supported.

Note: IBM DB2 Universal Database Enterprise Edition 7.2 must be AT LEAST at the Fix Pack 8e level in order to be able to install TDW 1.2, otherwise the TDW 1.2 installation will detect an inadequate level of IBM DB2 and fail.

Note: The actual RDBMSs supported as data sources for a given warehouse enablement pack are documented in the README file for that warehouse enablement pack.


2.1.4 Crystal Enterprise requirementsThe server components of Crystal Enterprise Professional Version 9 for Tivoli can be installed on the following operating systems:

� Microsoft Windows NT4 Server SP6a� Microsoft Windows 2000 SP3, Server version� Microsoft Windows 2000 SP3, Advanced Server version� Microsoft Windows 2000 SP3, Data Center version

The Crystal Enterprise Automated Process Scheduler (APS) component requires a database to store information about the system and its users. By default, the Setup program installs and configures its own Microsoft Data Engine (MSDE) database if necessary. The MSDE is a client/server data engine that provides local data storage and is compatible with Microsoft SQL Server. Crystal Enterprise Professional for Tivoli APS clustering is automatically supported by the default MSDE database.

If you already have the Microsoft Data Engine or Microsoft SQL Server installed, the installation program creates the APS database using your existing database engine.

You can migrate this initial APS database to another database server later. Supported servers for APS database migration are:

� Microsoft SQL Server 7 SP4 (ODBC)� Microsoft SQL Server 2000 SP2 (ODBC)� Microsoft MSDE (ODBC)� Oracle 9i (native)� Oracle 8i (8.1.7) (native)� IBM DB2 UDB 7.2 (native)� Sybase Adaptive Server 12.5 (ODBC and native)� Informix® Dynamic Server 2000 v 9.21 (ODBC)


For details about APS database migration, see “Configuring the intelligence tier” in the Crystal Enterprise 9 Administrator’s Guide manual, which is shipped with the product.

In case you have acquired a Crystal Enterprise 9 Special Edition license and choose to deploy the server-side installation method using the Crystal server connected to an external Web server, you also need to install and configure the appropriate Web Connector on your Web server machine. The supported Web servers are listed in Table 2-4.

Note: if the Microsoft Data Engine (MSDE) or Microsoft SQL Server is already installed on the local machine, you must set up a user account for the Crystal Enterprise Professional for Tivoli APS before installing Crystal Enterprise Professional for Tivoli, as follows:

1. Determine whether the Crystal Enterprise Professional for Tivoli APS should use Windows NT or Microsoft SQL Server authentication when connecting to your local database installation.

2. Using your usual administrative tools, create or select a user account that provides Crystal Enterprise with the appropriate privileges to your database server:

– If you want the APS to connect to its database using Windows NT authentication, ensure that the Windows NT user account that you assign to the APS has System Administrator’s role in your SQL Server installation.

In this scenario, the Windows NT user account that you assign to the APS is not actually used to create the system database during the installation process. Instead, your own Windows NT administrative account is used to create the database, so verify that your Windows NT account also has the System Administrator’s role in your SQL Server installation.

– If you want the APS to connect to its database using SQL Server authentication, the login that you assign to the APS must belong to the Database Creators role in your SQL Server installation. In this scenario, the SQL Server credentials that you assign to the APS are also used to create the database and its tables.

3. Verify that you can log on to SQL Server and carry out administrative tasks using the account you set up for use by the APS.

Note: For a detailed list of environments tested with Crystal Enterprise, please consult the Platforms.txt file included with your product distribution.


Table 2-4 Web servers and OS supported by Crystal Web Connector

Web browser requirementsCrystal Enterprise Professional for Tivoli supports the following Web browsers:

� Microsoft Internet Explorer 6.0 (Windows)� Microsoft Internet Explorer 5.5 SP2 (Windows)� Netscape 6.2.3 (Windows)� Microsoft IE 5.0 on OS9 or OS X’s Classic mode (Macintosh)

Web server Operating systems

Microsoft IIS5 / ISAPIYMicrosoft IIS5 / CGIY

Windows 2000 Server

Microsoft IIS4 / ISAPI-YMicrosoft IIS4 / CGI-Y

Windows NT 4.0 Server

iPlanet 6.0 SP3 / NSAPIiPlanet 6.0 SP3 / CGIiPlanet 4.1 SP10 / NSAPIiPlanet 4.1 SP10 / CGI

Windows 2000 ServerWindows NT 4.0 ServerSun Solaris 2.7, 2.8

Domino 5.0.8 / DSAPI Windows 2000 ServerWindows NT 4.0 Server

Domino 5.0.8 / CGI Windows 2000 ServerWindows NT 4.0 ServerSun Solaris 2.7, 2.8IBM AIX 4.3.3, 5.1

Apache 1.3.26 / ASAPIApache 1.3.26 / CGI

Sun Solaris 2.7, 2.8RedHat 6.2/7.3 (x86)SuSe 7.3/8.0 (x86)IBM AIX 4.3.3, 5.1

IBM HTTP 1.3.19.2 / ASAPI IBM AIX 4.3.3, 5.1

IBM HTTP 1.3.19.2 / CGI Windows 2000 ServerIBM AIX 4.3.3, 5.1

Note: Crystal Enterprise delivers reports and analytic content to any supported browser using pure DHTML.


2.2 Physical and logical design considerationsIn the previous section we have briefly introduced the basic components of the Tivoli Data Warehouse 1.2 application. Now we discuss how those components can be installed on different servers and what are the advantages or the possible drawbacks of the different configurations.

All the possible architectures for a Tivoli Data Warehouse implementation are determined by the location of the following components:

� Control server� Central data warehouse� Data marts� Crystal Enterprise server� Warehouse agents� Source databases

All these components can coexist on the same server or they can be spread out on different servers. See Table 2-5 to determine which operating systems are the platforms required by each Tivoli Data Warehouse component.

Table 2-5 Requirements for Tivoli Data Warehouse components

Component Supported operating systems platforma

DB2 components required

Control server Windows IBM DB2 Universal Database Enterprise Edition

Central data warehouse databases

Windows, AIX, Solaris, z/OS

IBM DB2 Universal Database Enterprise EditionIBM DB2 Warehouse Manager (optional)

Data mart databases Windows, AIX, Solaris, z/OS

IBM DB2 Universal Database Enterprise EditionIBM DB2 Warehouse Manager (optional)

Crystal Enterprise server Windows IBM DB2 Universal Database Enterprise Edition or IBM DB2 database client


2.2.1 Source databasesThe source databases are the fundaments of a data warehouse implementation. Their health is of crucial impact to the whole Tivoli Data Warehouse environment.

From an administrative point of view, source databases can be divided into two types:

� The first type of source database is part of the product from which data should be collected. An example is the IBM Tivoli Storage Manager (ITSM), where the source database is the internal ITSM database. In this case, the application administrator should be responsible for this source database.

� The second type, such as the IBM Tivoli Monitoring (ITM) source database, is built for the purpose of a source database exclusively. In this sense, you may consider these as part of the data warehouse infrastructure.

2.2.2 Control serverThe control server is the system that contains the control database for Tivoli Data Warehouse and from which you manage your data warehouse environment. Only one control server can be installed in a Tivoli Data Warehouse environment and it must be installed on a Windows platform.

Before installing the control server, you must install IBM DB2 Universal Database Enterprise Edition 7.2 locally and have the Crystal Enterprise environment up and running on the same or on a different server. You must also install IBM DB2 Universal Database Enterprise Edition 7.2 or DB2 Universal Database for OS/390 and z/OS V7 on each computer that will contain a central data warehouse database or a data mart database.

Warehouse agents Windows, AIX, Solaris IBM DB2 Universal Database Enterprise EditionIBM DB2 Warehouse Manager

a. See the Tivoli Data Warehouse Release Notes, SC32-1399 for complete detailsabout supported operating systems.

Component Supported operating systems platforma

DB2 components required


The control server creates a control database (TWH_MD) which contains descriptions of the stored data (known as metadata) for both Tivoli Data Warehouse and for the warehouse management functions. The control servers uses the DB2 Data Warehouse Center to automate the data warehouse processing, to define the ETL processes that move and transform data into the central data warehouse and the data marts.

The control server runs also a warehouse agent, the component of IBM DB2 Warehouse Manager that manages the data flow between warehouse sources and targets. In advanced Tivoli Data Warehouse scenarios, you can move the warehouse agent to other locations (see “Warehouse agents” on page 49).

When using the configuration with the warehouse agent on the control server, the computer on which you install the control server must also connect to the operational data stores of your enterprise, which potentially reside on other systems and in relational databases other than IBM DB2. To enable the control server to access these data sources, you must install the appropriate database client for each data source on the control server system.

2.2.3 Central data warehouseThe central data warehouse is a set of IBM DB2 databases containing the historical data for your enterprise.

The central data warehouse does not require any Tivoli Data Warehouse software or DB2 Warehouse components. You may choose to install a warehouse agent on the same server containing the central data warehouse to improve the performance of data transfer between warehouse databases (refer to “Warehouse agents” on page 49).

With Tivoli Data Warehouse 1.2, it will support up to four central data warehouse databases. You can have one or more central data warehouse databases in a Tivoli Data Warehouse deployment on Windows and UNIX. On z/OS you can have only one central data warehouse database (for example, see Figure 2-7 on page 49). The central databases can be installed in the following configurations:

� Only 1 central data warehouse database on Windows and UNIX platforms or 1 central data warehouse database on a z/OS system

� Up to 4 central data warehouse databases on Windows and UNIX platforms

� Up to 3 central data warehouse databases on Windows and UNIX platforms and 1 central data warehouse database on a z/OS system


On Windows and UNIX platforms, the first central data warehouse database created by Tivoli Data Warehouse 1.2 is named TWH_CDW. Subsequent central data warehouse database will be named TCDW1, TCDW2, and TCDW3.

On z/OS systems, there are no hard specifications on the name used for the central data warehouse database. However, it is a good practice to follow the naming convention adopted by Tivoli Data Warehouse 1.2.

Multiple central data warehouse databases might be useful in the following situations:

� If your Tivoli Data Warehouse deployment contains systems in widely separated time zones or geographies: A central data warehouse ETL typically runs during off-peak hours to avoid impacting the performance of your operational data stores: Having central data warehouse databases located on servers in different time zones enables you to schedule ETLs for each system at an appropriate off-peak time.

� If your deployment includes z/OS systems. Warehouse packs that use data extracted from z/OS data sources must load their data into a data mart database on a z/OS system. In contrast, warehouse packs that use operational data stores from Windows or UNIX systems can load that data into a data mart database on any supported operating system. Therefore, when you have sources on z/OS and on distributed systems, you must have at least one central data warehouse database on a z/OS system (see Figure 2-5 on page 47 and Figure 2-6 on page 48).

You may optionally choose to have also a second central data warehouse database on a distributed system in order to keep distributed applications data completely separate from z/OS applications data (see Figure 2-7 on page 49).

Note: If you plan to install warehouse packs that were created to run on Tivoli Enterprise Data Warehouse 1.1, you need to create at least one central data warehouse database on a Windows or UNIX system. These warehouse packs use only the first central data warehouse database that is created on a Windows or UNIX system. This central data warehouse database must be named TWH_CDW.

A central data warehouse database on a z/OS system can be populated only by warehouse packs developed for Tivoli Data Warehouse 1.2.


� If you want to distribute the central data warehouse workload. When using different warehouse packs that do not provide cross-application reports, you can have each warehouse pack load its data into separate central data warehouse databases. This allows you to schedule the central data warehouse ETLs for both warehouse packs to run at the same off-peak time without causing database performance problems.

When planning to use multiple central data warehouse databases, consider the following information:

� If you use a set of warehouse packs that collect historical data intended for the same reporting purpose, all of the warehouse packs must write their data into the same central data warehouse database.

Note that if a warehouse pack supports extracting data from multiple central data warehouse databases, its documentation contains information about the placement of the central data warehouse databases.

� Distributed application data may flow through a central data warehouse database either on a z/OS or on a distributed system into a data mart database either on a z/OS or on a distributed system.

� z/OS application data can flow only through a central data warehouse database on a z/OS system into a data mart database on a z/OS system.

2.2.4 Data martsThe data marts for your enterprise are contained in a separate set of IBM DB2 databases. Each data mart contains a subset of the historical data from the central data warehouse that satisfies the analysis and reporting needs of a specific department, team, customer, or application.

With Tivoli Data Warehouse 1.2, there are up to four data mart databases supported. You can have one or more central data warehouse databases in a Tivoli Data Warehouse deployment on Windows and UNIX. You can have one or more data mart databases for each central data warehouse deployed on Windows, UNIX, and z/OS systems.

Important: Although it is possible for a data analysis program to read data directly from central data warehouse databases without using data marts, this is highly not recommended and not supported.

Analyzing historical data directly from the central data warehouse database can cause performance problems for all applications using the central data warehouse.


On z/OS systems, Tivoli Data Warehouse 1.2 only supports one data mart database per IBM DB2 subsystem. In addition to that, the central data warehouse and data marts must be in the same IBM DB2 subsystem, and the IBM DB2 subsystem must have an unique location name.

On Windows and UNIX platforms, the first data mart database created by Tivoli Data Warehouse 1.2 is named TWH_MART. Subsequent central data warehouse database will be named TMART1, TMART2, and TMART3.

On z/OS systems, there are no hard specifications on the name for the data mart database. However, it is a good practice to follow the naming convention adopted by Tivoli Data Warehouse 1.2.

Each data mart database can contain the data from multiple central data warehouse databases.

The data mart databases do not require any Tivoli Data Warehouse software or DB2 Warehouse components, but you may choose to install a warehouse agent on the servers containing the data mart databases to improve the performance of data transfer from central data warehouse databases (refer to “Warehouse agents” on page 49)

Multiple data mart databases might be useful in the following situations:

� If your deployment includes z/OS systems. Warehouse packs that use data extracted from z/OS data sources must load their data into a data mart database on a z/OS system. In contrast, warehouse packs that use operational data stores from Windows or UNIX systems can load that data into a data mart database on any supported operating system. You might optionally place a data mart database on a Windows or UNIX system to keep data from those systems separate from data from z/OS applications (see Figure 2-6 on page 48).

Note: If you plan to install warehouse packs that were created to run on Tivoli Enterprise Data Warehouse 1.1, you need to create at least one data mart database on a Windows or UNIX system. These warehouse packs use only the first data mart database that is created on a Windows or UNIX system (TWH_MART).

A data mart database on a z/OS system can be populated only by warehouse packs for Tivoli Data Warehouse 1.2.


� If you want to store your enterprise data in different database for security reasons. You can allow each user to access only to the data mart database containing the information which that user is authorized to examine.

� If you plan to access data marts using different reporting or data analysis programs. You can format the data and you can tune each data mart database according to the programs that is used to analyze it and the expected workload.

For an effective planning of data mart databases locations, you should consider these requirements:

� Each warehouse enablement pack provides its own data structure called star schema.

� A single data mart database can contain many star schemas.

� Data from different warehouse enablement packs can be stored in the same data mart database. Each using its separate star schema.

� Each warehouse pack can write to only one data mart database, and it must pull all of the data for the data mart from a single central data warehouse database.

� Different star schemas in one data mart database can pull their data from different central data warehouse databases.

� Data mart databases on a Windows or UNIX system cannot pull z/OS applications data while a data mart database on z/OS can receive data coming all supported platforms (z/OS, UNIX and Windows).

2.2.5 Single machine installationYou can install all components of Tivoli Data Warehouse on one single machine (see Figure 2-1). This configuration uses the quick start installation method. It is easy to set up and to maintain but is recommended only for test or demonstration environments. It is also useful for those who do not plan to have many users concurrently accessing the data stored in the Tivoli Data Warehouse database and/or those who do not need to capture and analyze large amounts of data.

The control server must be on Windows NT, Windows 2000 Server, Windows 2000 Advanced Server, or Windows Server 2003. Thus the single machine configuration cannot be available on UNIX or z/OS systems.


Figure 2-1 Single machine installation

2.2.6 Distributed deployment on UNIX and Windows serversMost production environments usually require that the Tivoli Data Warehouse components be installed on different servers, in order to distribute workload or leverage preexisting infrastructures.

Next we consider some typical scenarios for UNIX and Windows environments:

� All Tivoli Data Warehouse components on different Windows or UNIX servers:

The control server is on a Microsoft Windows server, a central data warehouse database and data mart database on separate large server-class computers (Windows or UNIX), and Crystal Enterprise on a Microsoft Windows server, as seen in Figure 2-2.

Data source

Crystal Server

WebServer

Web Reports

TDW environment(Windows)

Central DataWarehouse

Control ServerMetadata

Data Mart

Crystal Enterprise(Windows)

Data sourceData source

Crystal Server

WebServer

Crystal Server

WebServer

Web Reports

TDW environment(Windows)



Data Mart

Crystal Enterprise(Windows)


Figure 2-2 Distributed deployment on Windows and UNIX systems

� Central data warehouse and source databases on the same server:

The central data warehouse database is on the same computer as the database containing the operational data sources. The control server and Crystal Enterprise are on two different Microsoft Windows servers, as seen in Figure 2-3.

Because operational data sources usually have a high rate of transactions per hour, it is not recommended to share the same IBM DB2 server for data source and data marts: This configuration may increase the time needed to obtain reports from data marts.


Windowssystem


Web Reports

Data source

Data Mart

Data source

Data source

Web Server

Crystal Server

Windowssystem

Windows or UNIXsystem



Windowssystem


Web Reports

Data source

Data Mart

Data source

Data source

Web Server

Crystal Server

Web Server

Crystal Server

Windowssystem




On the other hand, it is possible to have a common IBM DB2 server for data sources and central data warehouse without affecting the performances whenever the ETL1 and ETL2 can be scheduled in off-peak times.

Figure 2-3 Operational data sources and the CDW databases on the same server

2.2.7 Distributed deployment on z/OS, UNIX, and Windows serversNext we describe some examples of Tivoli Data Warehouse 1.2 architecture design scenarios that also include z/OS system management data sources:

Data source


Windowssystem

Web Reports

Web Server

Crystal Server

Windowssystem

Windows or UNIXsystem Windows or UNIX

system


Data source Data Mart



Windowssystem

Web Reports

Web Server

Crystal Server

Web Server

Crystal Server

Windowssystem

Windows or UNIXsystem Windows or UNIX

system


Data sourceData source Data Mart


� Data sources, central data warehouse and data mart database on a z/OS system:

The central data warehouse and the data mart database are in a IBM DB2 UDB for OS/390 and z/OS. The control server and Crystal Enterprise are on two different Microsoft Windows servers, as seen in Figure 2-4.

This kind of configuration is typically used when all management data source comes from z/OS applications. The reason is that all warehouse enablement packs extracting data from z/OS data sources must load their data into a central data database located at the Z/OS system.

Figure 2-4 Data sources, CDW, and data mart databases on a z/OS system

� Operational data sources both on z/OS and on distributed systems:

You can transfer data to a central data warehouse database on a z/OS system even if your operational data sources are distributed among z/OS and distributed systems, as seen in Figure 2-5. In this configuration you can use only warehouse enablement packs for Tivoli Data Warehouse 1.2, because those warehouse enablement packs for version 1.1 do not allow any data transfer to a central data warehouse located on a z/OS system.

Windowssystem

Web Reports

Web Server

Crystal Server

Control Server(Windows)

z/OS system SourceSource

Data Mart Central DataWarehouse

SourceSource

TDW environment

Windowssystem

Web Reports

Web Server

Crystal Server



z/OS system SourceSource

Data Mart Central DataWarehouse

SourceSource

z/OS system SourceSourceSourceSource

Data Mart Data Mart Central DataWarehouse


SourceSourceSourceSource

TDW environment


The common data mart database on z/OS provides the reports for both data sources on z/OS and distributed systems.

You may choose to have the common data mart database on a distributed system instead of z/OS, but in that case you cannot have any reports from operational data sources on z/OS, as seen in Figure 2-6.

Figure 2-5 Operational data sources both on z/OS and on distributed systems

� Separate data mart databases on a z/OS and on a distributed system:

This configuration is typically used to segment the reporting functions into two logical areas, one for the z/OS and the other for the distributed environment.

All z/OS application data flows through the central data warehouse database to the data mart database on z/OS, while the distributed applications data is transferred to a data mart on a distributed system.


Windowssystem

Web Reports

Web Server

Crystal Server

Windowssystem

Z/OS system


Data source

Data source


Data source

Data Mart


Windowssystem

Web Reports

Web Server

Crystal Server

Web Server

Crystal Server

Windowssystem

Z/OS system


Data source

Data source





Data Mart


Figure 2-6 Separate data mart databases on z/OS system and distributed system

� Two central data warehouse servers, one on z/OS, and one on a UNIX or Windows system:

This is a more complex deployment with one central data warehouse and one data mart database in a DB2 UDB for OS/390 and z/OS, a second central data warehouse database on a UNIX or Windows server, and the control server and Crystal Enterprise server on separate Windows systems, as seen in Figure 2-7.

This configuration may be chosen by customers who want to keep completely separate z/OS applications data from distributed applications data.


Data MartData MartData Mart 2

Windowssystem

Web Reports

Web Server

Crystal Server

Windowssystem

Z/OS system


Data source

Data source


Data source

Data source




Data MartData MartData Mart 2Data MartData MartData Mart 2

Windowssystem

Web Reports

Web Server

Crystal Server

Web Server

Crystal Server

Windowssystem

Z/OS system


Data source

Data source




Data source

Data source






Figure 2-7 Two CDWs on a Windows or UNIX system and on a z/OS system

2.2.8 Warehouse agentsWarehouse agents are IBM DB2 Warehouse Manager components that manage the flow of data between warehouse sources and targets using DB2 CLI or Open Database Connectivity (ODBC) drivers to communicate with different databases.

Every system having a warehouse agent installed is called an agent site or a remote agent. Tivoli Data Warehouse supports warehouse agents on Windows and UNIX operating systems, but not on z/OS.


Windowssystem

Web Reports

Web Server

Crystal Server

Windowssystem

Z/OS system

Central DataWarehouse 2


Data source

Data source


Data source

Data source


Central DataWarehouse 1 Data MartData MartData Mart 2



Windowssystem

Web Reports

Web Server

Crystal Server

Web Server

Crystal Server

Windowssystem

Z/OS system

Central DataWarehouse 2Central DataWarehouse 2


Data source

Data source




Data source

Data source




Central DataWarehouse 1Central DataWarehouse 1 Data MartData MartData Mart 2Data MartData MartData Mart 2



When you install the Tivoli Data Warehouse control server, a warehouse agent is automatically installed on the Control server machine. In the basic configuration, shown in Figure 2-8, the control server uses its local warehouse agent to manage data flow from the operational data sources to the central data warehouse (ETL1 process) and from that to the data marts (ETL2 process). In case the Tivoli Data Warehouse databases are located on the same system as the control server, the warehouse agent is not used.

Figure 2-8 Warehouse agent on control server

In a distributed scenario, as shown in Figure 2-9, you might improve the performance of Tivoli Data Warehouse by placing a warehouse agent on each central data warehouse server and data mart server. These remote warehouse agents allow a straight data flow from target to source without passing through the control server, reducing the workload on this server and increasing the speed of data transfer.

Warehouse Server(Control Server)


ETL1 ETL2

ETL2

Data Mart Data Mart


SourceSourceWarehouse

AgentWarehouse

Agent

ETL1

ETL1ETL2




ETL1 ETL2

ETL2

Data Mart Data Mart


SourceSourceSourceSourceWarehouse

AgentWarehouse

AgentWarehouse

AgentWarehouse

Agent

ETL1

ETL1ETL2


Figure 2-9 Warehouse agents on data targets

Typically the warehouse agent is placed on the target of a data transfer. In this configuration, the warehouse agent performs the following tasks:

� Passes SQL statements that extract data from the remote source tables� Transforms the data if required� Writes the data to the target table on the local database

This configuration offers the best performance in a distributed environment by optimizing the DB2 data flow and using block fetching for the extraction. This is the recommended configuration to use with Tivoli Data Warehouse when source and target are on physically separate machines.

The warehouse agent may be installed on the system that contains the source database. In this configuration, the agent:

� Passes SQL statements that extract data from the local source tables.� Transforms the data if required.� Writes the data to the target table on the remote database.

This alternative configuration does not optimize the DB2 data flow (Distributed Relational Database Architecture DRDA® or DB2 private protocols) and should be used only if justified by specific architecture requirements. For example, if your data source and data mart databases are on distributed systems while the central data warehouse database is on a Z/OS system, you are forced to place warehouse agents on the distributed systems, one of which is the source of data transfer, as seen in Figure 2-10.


WarehouseAgent


Data Mart Source

WarehouseAgent

ETL1

ETL2

Agent site

Agent site


Warehouse Server

Data

Data


WarehouseAgent



Data Mart Source

WarehouseAgent

ETL1

ETL2

Agent site

Agent site


Warehouse Server

Data

Data


Figure 2-10 Configuration with a warehouse agent on the source

To improve the performance of your ETL process, you should carefully plan where to place agent sites in your environment and which site to associate with each ETL. Table 2-6 suggests where to place the warehouse agents when transferring data from data sources to the central data warehouse database in different scenarios.

Table 2-6 Agent sites placement for data transfers to a central data warehouse


WarehouseAgent


Data Mart Source

WarehouseAgent

ETL1

ETL2

Agent site Agent site

z/OS system

UNIX or Windows system






WarehouseAgent



Data Mart Source

WarehouseAgent

ETL1

ETL2

Agent site Agent site

z/OS system






Operational data source location

Central data warehouse database location

Warehouse agentlocation

A Windows or UNIX system

A different Windows or UNIX system

Central data warehouse system

The same system No agent required

A z/OS system Operational data source system


Table 2-7 suggests where to place the warehouse agents when transferring data from central data warehouse databases to data mart databases in different scenarios.

Table 2-7 Where to place agent sites for data transfers to data marts

Note that Tivoli Data Warehouse automatically recognizes when the source and target data are on the same computer, and in that case it transfers the data without using the warehouse agent.

Here are some common situations in which data transfer does not use the warehouse agent:

� When using operational data sources, central data warehouse, and data mart in the same IBM DB2 location on a z/OS system.

� When the operational data and the central data warehouse database are on the same computer running Windows or UNIX.

� When transferring data between a central data warehouse database and data mart database on the same computer running Windows or UNIX.

z/OS system The same z/OS location No agent required

A different z/OS location Control server


Deployment not supported. Data sources on z/OS can load data only in a central data warehouse on z/OS.


Data mart database location


a Windows or UNIX system

A different Windows or UNIX system

Data mart system

The same system No agent required

A z/OS system Central data warehouse system

z/OS system The same z/OS location No agent required

A different z/OS location Control server


Data mart

Operational data source location




To install warehouse agents, you must install IBM DB2 Warehouse Manager and Tivoli Data Warehouse on each machine that will be an agent site. If you are using operational data stored in databases other than IBM DB2 (Oracle, Informix etc.) you are also required to install on that computer a database client for each type of remote database that the agent needs to access. For example, if the operational data source for a warehouse pack is an Oracle database on another computer, you must install also an Oracle database client on the agent site.

For more information about warehouse agents, refer to the IBM DB2 Warehouse Manager Installation Guide, and the redbook, DB2 Warehouse Management: High Availability and Problem Determination Guide, SG24-6544-00.

2.2.9 Considerations about warehouse databases on z/OSAs already explained in the previous sections, there are some limitations when placing Tivoli Data Warehouse databases on z/OS. Here is a summary:

� Only one central data warehouse database can be placed on a z/OS system.

� Data from applications on z/OS or OS/390 systems can be placed only in central data warehouse and data mart databases in DB2 UDB for OS/390 and z/OS.

� To extract operational data from a Tivoli Decision Support for OS/390 database, the central data warehouse database and the Tivoli Decision Support for OS/390 database must be in the same DB2 subsystem.

� The central data warehouse database and data mart database can be a storage group that is shared with other applications.

� Only one data mart can be installed per single IBM DB2/390 location (same host and port).

� Tivoli Data Warehouse 1.2 do not support warehouse agents on z/OS systems.

Before installing a Tivoli Data Warehouse database on a z/OS system, you should know the following parameters about the existing IBM DB2 installation:

� Host name of the z/OS system � TCP/IP port configured for IBM DB2� Database location� Storage volume, storage group, and storage VCAT� Buffer pool� Tablespace name� UTF8 tablespace name� Tablespace size (primary and secondary)


2.2.10 Coexistence with other productsThis section describes coexistence issues between Tivoli Data Warehouse and other applications such as Tivoli Decision Support (distributed), Tivoli Decision Support for OS/390, and other IBM DB2 databases and data warehouses:

� Coexistence with Tivoli Decision Support:

You can continue to use Tivoli Decision Support at the same time and on the same systems as Tivoli Data Warehouse. For the best performance, do not schedule Tivoli Data Warehouse ETLs and Tivoli Decision Support cube builds concurrently, if they access the same databases.

� Coexistence with Tivoli Decision Support for OS/390:

You can continue to use Tivoli Decision Support for OS/390 at the same time and on the same systems as Tivoli Data Warehouse. For the best performance, schedule Tivoli Data Warehouse ETLs after Tivoli Decision Support for OS/390 data collection has occurred.

� Coexistence with other data warehouses:

Do not mix components from different deployments of Tivoli Data Warehouse on any computer. For example, if you can have both a production deployment and a test deployment, you must not place any warehouse components of the test deployment on the same computer as any components of the production deployment.

You can use the IBM DB2 Data Warehouse Center to manage multiple data warehouses. However, only one data warehouse can be active (running scheduled jobs) at a time. Scheduled jobs run only for the warehouse whose control database (TWH_MD in case of Tivoli Data Warehouse) is selected in the IBM DB2 Data Warehouse Center. Other warehouses, each identified by a different control database, retain their information but do not run scheduled jobs until their control database is selected in the IBM DB2 Data Warehouse Center.

� Coexistence with other IBM DB2 database applications:

You can have other applications using the same IBM DB2 instances used by Tivoli Data Warehouse, provided that they do not use the same database names required by warehouse components (TWH_MD, TWH_CDW, TCDW1, TCDW2, TCDW3, TWH_MART, TMART1, TMART2, TMART3, and any other database names defined on z/OS systems).

If you install Tivoli Data Warehouse using an existing IBM DB2 instance, all other clients that are connected to that instance lose their IBM DB2 connections during the installation process. Therefore, remember to stop all existing connection before starting Tivoli Data Warehouse installation.


2.2.11 Selecting port numbersYou must allocate port numbers for Tivoli Data Warehouse for communication between the Tivoli Data Warehouse control server and the remote IBM DB2 databases (data sources, central data warehouse, data marts) and for communications with Crystal Enterprise.

Table 2-8 lists the default port numbers that are used by Tivoli Data Warehouse. You can use the same or different port numbers for the control server, data mart database computers, and central data warehouse database computers.

Table 2-8 Default port used in Tivoli Data Warehouse environments

2.3 Database sizingA correct database sizing is one of the most important issues you have to face when planning a Tivoli Data Warehouse deployment.

The growth rate of your central data warehouse and data mart databases is related to the number of measured items in your environment (hosts, database instances, events and so on), the measurements you require for each item and the sampling frequency.

You may find detailed instructions for estimating the amount of required storage in the implementation guide of each warehouse pack you install in your environment. If the implementation guides do not provide this kind of information, refer to the planning chapter of Installing and Configuring Tivoli Data Warehouse, GC32-0744-02. In this publication, some calculations are illustrated for estimating the central data warehouse database storage requirements for a warehouse pack.

Component Default port

DB2 databases on UNIX and Windows 50000

DB2 databases on z/OS 5000

Crystal Enterprise server 6400

Internal communication between the Crystal APS and other Crystal components, such as the Crystal Web Connector running on the HTTP Server

6401

HTTP server for CrystalEnterprise

80


You can also find useful information about database sizing in the redbook Planning a Tivoli Enterprise Data Warehouse Project, SG24-6608, even if the estimates indicated in that redbook are for warehouse enablement packs running on Tivoli Enterprise™ Data Warehouse version 1.1.

2.4 SecurityThe following sections describe security considerations for installing and using Tivoli Data Warehouse.

2.4.1 Authority required to install and maintain IBM DB2 UDB For UNIX machines, the user must be logged in with root authority. User and group accounts will be created for the instance owner, Administration Server, fenced user defined functions, and fenced stored procedures. It is recommended that you use different user IDs for UDFs and the instance owner.

For Windows NT and 2000 machines, the user must be part of the administrators group. Specifically, the user must have these advanced user rights:

� Act as part of the operating system� Create token object� Increase quotas� Replace a process level token

More detailed user account information can be obtained from the DB2 UDB Quick Beginnings EEE for NT V7, GC09-2963 or DB2 UDB Quick Beginnings EEE for UNIX V7, GC09-2964.

2.4.2 Authority required to install Tivoli Data WarehouseOn Windows systems, to install Tivoli Data Warehouse or to install, start, or stop the Crystal Enterprise Professional for Tivoli server, you must be logged on as a locally defined administrator.

On UNIX-based systems, to install Tivoli Data Warehouse you must be logged as the root user.

On z/OS systems, to add or remove a central data warehouse database or data mart SYSADM authority is required, while DBADM authority on central data warehouse and data mart databases is required to install warehouse packs.


2.4.3 FirewallsYou cannot have a firewall between any Tivoli Data Warehouse components. This includes the control server and the Data Warehouse Agent sites.

However, it is possible to have configurations with a firewall between source databases and a central data warehouse, or a firewall between a central data warehouse and data marts, if the source database vendor supports communication through firewalls, and the Data Warehouse Agent resides on the central data warehouse. Figure 2-11 shows both a valid configuration and a non-functional configuration. Only the ODBC communication can be transported through the firewall.

Figure 2-11 Tivoli Data Warehouse and firewalls

Crystal Enterprise can deliver a broad range of reporting and analytic content to any browser using pure DHTML. Unlike plug-in based technologies, DHTML requires no software downloads and no special configuration to enable viewing, making it ideal for deployments through a firewall without compromising security.

ODBC Data

Data warehouse Control Data

ODBC Data

Data warehouse Control Data

SourceSource

WarehouseControl Server

Firewall

WarehouseAgent


Firewall

WarehouseAgent


SourceSource Central DataWarehouse

SourceSourceSourceSource



Firewall

WarehouseAgent

WarehouseAgent



Firewall

WarehouseAgent

WarehouseAgent



SourceSourceSourceSource Central DataWarehouseCentral DataWarehouse


2.4.4 Controlling access to data in the warehouseThe following techniques can be used to control access to data in Tivoli Enterprise Data Warehouse:

� Restricting which DB2 users have access to specific views in the central data warehouse and in the data mart database. This is done using DB2 administrative tools.

� Using multicustomer or multicenter support, as described in “Multicustomer and multicenter support”

2.4.5 Protecting information in Crystal Enterprise Professional for Tivoli

Crystal Enterprise Professional for Tivoli allows you to choose one of three types of user authentications:

� Enterprise� Windows NT� LDAP

The Enterprise authentication is based on Crystal proprietary internally defined accounts and groups used only for Crystal Enterprise.

The Windows NT authentication uses already existing NT and 2000 user accounts and groups for authentication in Crystal Enterprise Professional for Tivoli, eliminating the need to create new additional accounts within the application.

The LDAP authentication can be chosen if you have already an LDAP directory server in your environment. In this case, you can use existing LDAP user accounts and groups also for Crystal Enterprise Professional for Tivoli.

Crystal Enterprise Professional for Tivoli not only authenticates its users, but also allows an access control to all the objects within the application using the Crystal Management Console (CMC). The base unit for controlling consists of specific object rights that provide a user or a group with permission to perform a particular action on an object. Each object right can be Explicitly Granted, Explicitly Denied, or Not Specified.

The Crystal Enterprise Professional for Tivoli object security model is designed according a “denial base” in order to guarantee that users and groups do not automatically acquire rights that are not explicitly granted. All rights that are not specified are denied by default. Additionally, in case of contradictory settings that grant and deny the same right to the same user or group, the right is denied by default.


Crystal Enterprise includes a set of predefined access levels that allow you to set common security levels quickly and facilitate administration and maintenance. Each access level grants a set of rights that combine to allow users to accomplish common tasks such as viewing reports, scheduling reports, etc.

Users can inherit rights as the result of group membership, subgroups can inherit rights from parent groups, and both users and groups can inherit rights from parent folders. However, you can always disable inheritance or customize security levels for particular objects, users, or groups.

Refer to the manual, Crystal Enterprise Professional Version 9 for Tivoli Administrator’s Guide, which is provided with Crystal Enterprise Professional Version 9 for Tivoli, for further information about security aspects of Crystal Enterprise Professional for Tivoli.

2.4.6 Multicustomer and multicenter supportTivoli Data Warehouse allows you to keep data about multiple customers and centers in one central data warehouse, while restricting access so that customers can see and work with data and reports based only on their data and not any other customer’s data.

In order to support a multicustomer environment, you must be sure that your application data correctly discriminates between different customers. Various applications can use different data fields to distinguish between customers, but within applications, you should make the data fields consistent.

You must determine which field in a data source is applicable for identifying multiple customer data. An application might use a single data field to determine customer uniqueness, while other applications might use two or more data fields to distinguish customer data. Either case is permitted as long as the data fields used are consistent within a single application.

Important: If you intend to have multicustomer and multicenter support in your Tivoli Data Warehouse, you must plan and configure for this before importing any data into your central data warehouse.

If you decide to enable multicustomer and multicenter support after you have already imported some data, you must set up a new central data warehouse for the new multicustomer environment and, if necessary, migrate your old data into the new environment.


When the central data warehouse ETL is run, data from applications is assigned valid customer account codes by matching certain data fields in the incoming data with pre-identified values in a matching database table. Because each application can use different fields and different numbers of fields to identify customers, each application has its own matching table that it uses during the central data warehouse ETL process.

To configure the central data warehouse for multicustomer functionality, you must manually define your customers into the TWG.CUST and Product_Code.CUST_LOOKUP tables, where Product_Code is the code univocally associated to the warehouse pack you intend to use. In the same way, you can configure the multicenter support simply defining your centers into the TWG.CENTR and Product_Code.CENTR_LOOKUP tables on the central data warehouse database.

See the multicustomer and multicenter support information in the manual, Enabling an Application for Tivoli Data Warehouse, GC32-0745-02, for details on how to configure the central data warehouse for multicustomer or multicenter support.

2.5 Network traffic considerationsAn accurate planning of a Tivoli Data Warehouse environment should also consider the possible impact on the network.

When the ETLs run, large amounts of data are transferred, and this can greatly affect the network performance. However, a correct scheduling of the ETLs and a proper design of Tivoli Data Warehouse implementation can reduce the impact on the network.

Every customer environment will have unique network requirements. Therefore, Tivoli Data Warehouse must be tailored to fit each customer’s specific needs. In this section, we discuss some of the decisions that a customer will face when planning for Tivoli Data Warehouse.

Note: The ETLs are based on Structured Query Language (SQL). The SQL processing exhibits a typical request/response pattern, meaning that requests send small amounts of data, whereas responses return large amounts of data. This can have an adverse effect on the network.


2.5.1 Architectural choicesThe single machine installation of Tivoli Data Warehouse (Figure 2-1 on page 43) affects the network far less than any other distributed installation (for example, see Figure 2-2 on page 44). Simply put, there is less data transferred across the network in a stand-alone environment than on a distributed environment.

On the other hand, a single server cannot handle the entire workload of a Tivoli Data Warehouse for most production environments and then a distributed installation is very often required. In this case the network performance can be improved by using remote warehouse agents and carefully placing the data marts and the Crystal Enterprise Professional for Tivoli server.

Remote warehouse agentsDuring a typical distributed installation, a local warehouse agent is installed on the system with the control server. Utilizing the local agent is the easiest and fastest in terms of installation, but this configuration is the least efficient in terms of performance and network traffic.

In an advanced distributed Tivoli Data Warehouse, remote warehouse agents are usually located on the target databases. In this setup, communication between the source or target and the control server is minimized.

Please refer to 2.2.8, “Warehouse agents” on page 49 for more details on remote warehouse agents and their possible locations.

Data marts and Crystal Enterprise serverCrystal Enterprise Professional for Tivoli server uses an IBM DB2 client connection to retrieve data from data mart databases. You may reduce the network traffic by simply installing the data mart databases on the same machine where the Crystal Enterprise Professional for Tivoli server is located.

On the other hand, this configuration requires that the data mart databases be on a Windows server, while you may prefer to have them on a UNIX or z/OS system.


2.5.2 SchedulingAs stated earlier, ETLs transfer potentially large amounts of data over the network. In a production environment, it is advised not to run these during normal business hours. Instead, ETLs should be scheduled to run when network traffic is low.

Production versus maintenance windowsThe production and maintenance windows should be the first parameters defined when determining when to schedule ETLs. Generally, the production window consists of normal business hours. This usually means the hours representing the highest traffic volume on the network. This is also the time during the day that end users view and run reports against the data marts.

ETLs and database backups should not be run during the production window, as they can both negatively impact report performance and other users across the network as well. In contrast, the maintenance window consists of non-business hours, which represent the off-peak hours on the network. ETLs and backups should run during this time.

If your environment is spread over wide geographical areas, you may choose to have different central data warehouse and databases for different time zones. This configuration allows you to schedule ETLs separately during each maintenance window in the different time zones (see “Physical and logical design considerations” on page 36).

ETLsAfter establishing the production window, the ETLs may now be scheduled. Once again, different strategies may be implemented to fit the needs of the customer. There are a number of different variables that come into play.

Two important variables to consider are the number of warehouse enablement packs (WEPs) and the types of WEPs installed in the Tivoli Data Warehouse. One or two WEPs that extract large amounts of data can take much longer than the combination of many WEPs that extract small amounts of data.

If it seems that the amount of time needed to run the ETLs will be longer than the time allocated, a couple of strategies could be implemented.

You can refer to the redbook Planning a Tivoli Enterprise Data Warehouse Project, SG24-6608 for estimates about the time required for each ETL to run. This redbook provides information about ETLs times running on a Tivoli Enterprise Data Warehouse version 1.1 environment.


ETL groupingAnother consideration when planning the scheduling of ETLs should be grouping. When you have several warehouse packs installed in your environment, you can use two basically different approaches for ETLs schedule:

� First run all the ETL1s (from operational data sources to central data warehouse), then all the ETL2s (from central data warehouse to data mart).

� Run sequentially the ETL1 and ETL2 for each warehouse pack.

The first approach will place most of the network load at the beginning of your maintenance window when ETL1s are run. This is because the ETL1s query the application databases and must transfer data across the network when doing so. ETL2s extract data from the central data warehouse and load it into the data marts. If these two databases reside on the same machine, network traffic is minimal. Therefore if the total time of all ETLs runs into the production window, there is relatively little impact on the network. The drawback is that if the ETL2s carry over into the production window, the reports will be unavailable during normal business until the ETLs complete.

The second approach guarantees that at least some data marts would be already updated at the start of business even if the total time of all ETLs runs into the production window. The drawback is that an ETL1 might still be running at the start of business, which might put a heavy load on the network at a high peak time.

2.6 Integration with other business intelligence toolsTivoli Data Warehouse 1.2 natively provides Web reports through Crystal Enterprise Professional Version 9 for Tivoli. However, if you have existing business intelligence infrastructures already in place or if you prefer using other tools to analyze your data, you can easily integrate them with Tivoli Data Warehouse. However, the installation of Crystal Enterprise Professional Version 9 for Tivoli is a prerequisite for Tivoli Data Warehouse 1.2 and is mandatory.

You can connect your preferred applications to the data marts and extract your own reports without using Crystal Enterprise Professional for Tivoli. Technically, these applications may report directly from central data warehouse, but this is a non-supported configuration, provides very poor performance, and may interfere with the ETL processes (see Figure 2-12).

Tip: ETL execution times are largely dependent upon the amount data they query. Always try to schedule ETLs that handle large amounts of data before your ETLs that handle less data in the maintenance window when considering either strategy.


Figure 2-12 Business intelligence integration

By having only a subset of the data in the data mart, the database system can manage the data faster and easier. Also, since the filtering has already been applied at the ETL2 run time, the reporting queries become much smaller. Smaller queries performing on a small subset of data are, of course, easier to tune. Therefore, the end customer will experience better reporting performance.

Refer to the redbook, Tivoli Data Warehouse Report Interfacing, SG24-6084 for details on how to integrate with other business intelligence and reporting tools.

2.7 ETL developmentA typical organization generally relies on very different tools to generate data for its business and stores historical information on separate repositories and in a variety of formats, including relational databases, spreadsheet data, log files, etc. These repositories have different data schemas, since they were designed by different vendors, and they may have metrics specific to that platform. While platform-specific reports are usually required, it is often useful to create enterprise level reports, which mix information from the various platforms. An example of this is to report performance utilization statistics of servers in a single report, regardless of the platform architecture.

CrystalEnterprise

Central Data Warehouse

Data Mart 1 Data

Mart 2

Data Mart 3

Other BI Applications

Do not connect directly to CDW

CrystalEnterprise


Data Mart 1 Data

Mart 2

Data Mart 3

Other BI Applications

Do not connect directly to CDW


Tivoli Data Warehouse allows mapping of the existing data sources into one or more central data repository the repository, so that reports having a common look and feel can be easily generated, thus improving the understanding of different platforms.

In addition to having different data repositories caused by different platforms, different tools are frequently deployed on the same platforms, but on different servers. This is usually the case when these servers are supported by different personnel and then consolidated at some future time (for example, companies that have grown through acquisition, where the servers have come from different companies, or where different support organizations have been allowed to select their own system management tools). By mapping these tools to the central data repository, common reports can be generated, masking the different tools used to create the data.

Lastly, one may want to convert existing infrastructure to Tivoli based products. By using Tivoli Data Warehouse, historical data from existing servers can be loaded into the central data repository without any data loss. This will allow one to convert over to a Tivoli product that uses the central data repository and not lose any data. While servers are in the process of converting, both the new and old data sources can be used to generate reports in the new common format.

The ETL1 programs take the data from these sources and places it in the central data warehouse, while the ETL2 programs extract from the central data warehouse a subset of historical data that contains data tailored to and optimized for a specific reporting or analysis task. This subset of data is used to create one or more data marts, a subset of the historical data that satisfies the needs of a specific department, team, or customer.

A data mart is optimized for interactive reporting and data analysis. The format of a data mart is specific to the reporting or analysis tool you plan to use. Customers can then use Crystal Enterprise Professional for Tivoli or other analysis program to analyze a specific aspect of their enterprise using the data in one or more data marts.

Whenever you need to extract data from sources not supported by existing Tivoli warehouse packs or if you decide to use customized data marts, you have to develop your own ETLs. The guide, Enabling an Application for Tivoli Data Warehouse, GC32-0745-02, provides all the information needed to develop customized ETLs.


2.8 Skills required for a Tivoli Data Warehouse projectImplementation of a complete Tivoli Data Warehouse solution requires different skills and usually involves different departments inside a company.

A Tivoli Data Warehouse project can be schematically divided into four phases:

1. Implementation2. Data collection3. Data manipulation (ETL1 and ETL2)4. Reporting

2.8.1 ImplementationThe setup of a Tivoli Data Warehouse environment is highly automated and therefore does not require a very specific training.

The Tivoli Data Warehouse administrator should have DB2 administrative skills on distributed platform in order to manage and optimize all the database used in the data warehouse. If the managed environment includes also z/OS systems, the administrator should have administrative skills also in DB2 UDB for OS/390 and z/OS or he should be supported by a person with these skills at least during the databases setup on z/OS.

2.8.2 Data collectionThe most important phase of a warehouse project consist of defining all the data sources involved and which is the really relevant data in our business.

Generally IT departments are responsible for the collection of metrics concerning availability, performances and utilization of IT resources (systems, applications, networks). Other departments could provide additional data, such as financial statistics or inventories.

Skills required to select and actually perform data collection vary according the type of data under scope. Generally a system administrator or an application administrator is directly responsible for the implementation of IT resources monitors or he can provide the information required to implement them.

Tivoli software products already offer a wide range of solutions for collecting data about IT infrastructure, but if a company has already implemented different data collection processes before Tivoli Data Warehouse installation, there is no need to modify them. The data manipulation phase is in charge of retrieving data from legacy sources and to convert it into a common format.


Let us consider the following scenario to explain this further. Suppose that a company plans to store in a Tivoli Data Warehouse all data retrieved by Tivoli monitoring applications as well as by some preexisting and highly customized applications.

Tivoli Data Warehouse allows that company to correlate data coming indifferently from Tivoli and non Tivoli sources without affecting the preexisting processes: that can be obtained simply customizing the SQL code necessary to transfer data from the old application database to the central data warehouse (source ETL) and that required to populate the data marts (target ETL) that will be used to generate reports.

Please note that the ETLs perform not only a plain data transfer between different databases, but they are also in charge of all the transformation tasks required to have a common format for all data independently from the sources.

A typical example of data transformation may be the time stamp of a measurement: each monitoring application generally uses the local time and that could generate confusion whenever we examine data produced in different areas of the world. Therefore the ETLs are also required to convert the times referring to a common standard, such as the Greenwich Mean Time.

Another example of data transformation concerns the standardization in the denomination of the measured components: different applications may use different names to indicate the same object and the ETLs must correct any possible mismatch in order to produce always coherent reports also when comparing data from different sources.

2.8.3 Data manipulation (ETL1 and ETL2)Once the data sources are established, the next steps are as follows:

1. Loading legacy sources data into central data warehouse and transforming data into a standard format (ETL1 or Source ETL process)

2. Selecting data subsets from a central data warehouse according different business areas, and loading them into data marts (ETL2 process or Target ETL process)

3. Scheduling convenient time frames to run ETL processes

Tivoli already provides free ETLs for most Tivoli products, These ETLs can be immediately utilized to manipulate data from Tivoli applications or they can be used as models to create internally developed ETLs to manage other products.


Customized ETLs can be packaged and shipped to customers or colleagues, who can install them using the Tivoli Enterprise Data Warehouse installation wizard. You can find out how to integrate your own components into the Tivoli Data Warehouse in the manual Enabling an Application for Tivoli Data Warehouse, GC32-0745-02.

Source and target ETLs are developed with the same method and slight variations in design considerations. Skills required to develop ETLs are:

� Standard SQL� Some experience in Data Warehousing and fair amount of DB2 skills� Knowledge of source databases involved

ETL developers should totally understand their end users’ requirements in order to project proper data marts, while each data source administrator is requested to provide ETL developers with the structure of their databases. No interface is provided to allow users unfamiliar with the data and with SQL to simply move data from any application’s data store to the data warehouse.

In a Tivoli-only environment with Tivoli WEPs, the following skills are recommended:

� Knowledge of the source application

� Knowledge of the used Tivoli product to collect the date (IBM Tivoli Monitoring or IBM Tivoli Storage Manager)

� Basic Knowledge of Tivoli Data Warehouse

To set up data collection for a new application, the following skills are needed:

� Knowledge of the source application� Knowledge of Standard SQL� Knowledge of Tivoli Data Warehouse and its underlying data model

One of the most critical aspects of the data manipulation phase in a large environment is the scheduling of all the different ETL processes running in a Tivoli Enterprise Data Warehouse. The person responsible for scheduling ETLs should have a thorough knowledge about:

� The timing of data source updates� The requirements of end users for all reports � The workload for each server in the Tivoli Data Warehouse environment� The impact of ETLs on the network

You can find a discussion about the last two items in “Network traffic considerations” on page 61.


2.8.4 ReportingThe final step of a Tivoli Data Warehouse process is producing timely updated reports according different end users’ specifications.

Tivoli warehouse enablement packs already provide out-of-the-box reports for the Crystal Enterprise Professional for Tivoli Limited Edition bundled with Tivoli Data Warehouse, but if there is a need to define additional customized reports, then either Crystal Enterprise Professional for Tivoli Professional Edition or another Business Intelligence tool is required.

These are the skills required to implement customized reports for Tivoli Data Warehouse:

� Basic knowledge of how to connect the data mart databases via ODBC� Basic knowledge of standard SQL� Knowledge of the data mart structure and data� Experience with Crystal Enterprise products or other Business Intelligence

tools

Report designers usually interact with the ETL2 developers, who are providing all of their requirements about relevant metrics, details on information, aggregation times, preservation of old data, and so on, in order to optimize the star schemas according to their reporting needs.


Chapter 3. Getting Tivoli Data Warehouse 1.2 up and running

This chapter describes the installation steps for Tivoli Data Warehouse. You can install the application with all components on a single system or with the components distributed throughout your network.

Always check the Tivoli Data Warehouse Release Notes, SC32-1399 for details about required Fix Packs, patches, or operating system-specific configurations. Look for late-breaking information about the required service at the TDW support Web site at the following address:

http://www-3.ibm.com/software/sysmgmt/products/support/TivoliDataWarehouse.html

Topics covered include:

� “Installing and configuring IBM DB2 client and server” on page 76� “Crystal Enterprise installation” on page 86� “Quick start deployment” on page 93� “Distributed deployment” on page 103� “Installing warehouse agents” on page 126� “Verification of the installation” on page 135� “Installing warehouse enablement packs” on page 142

3


http://www-3.ibm.com/software/sysmgmt/products/support/TivoliDataWarehouse.html

3.1 Preparing for the installationTivoli Data Warehouse 1.2 requires additional supporting applications, such as IBM DB2 Universal Database Enterprise Edition and Crystal Enterprise Professional Version 9 for Tivoli. The entire installation process is depicted in Figure 3-1. Even though this diagram refers to an installation process for a distributed Tivoli Data Warehouse 1.2 environment, the same applies also for a single machine environment.

The remainder of this section contains tasks that need to be performed prior to installing the Tivoli Data Warehouse 1.2 environment. Depending on the architecture to be used, you may need to perform some or all of these tasks. Figure 3-1 also provides a guide to which tasks should be performed and the machines they should be performed on.


Figure 3-1 Installation process overview

Install a Web Server Define whether Crystal Enterprise will use Microsoft SQL Server or Windows NT authenticationInstall IBM DB2 Client for access to the Data Mart databasesInstall Crystal Enterprise Professional version 9 for Tivoli

Phase 1

Crystal Enterprise Server

Install IBM DB2 Universal Database Enterprise Edition 7.2 and MINIMUM Fixpack 8 or upgrade existing IBM DB2 Servers to version 7.2 at least Fixpack 8 level on the TDW Control Server, and all Servers that will host Central Data Warehouse and Data Mart databases on Windows / UNIX platforms

Install or upgrade IBM DB2 Universal Database for OS/390 and z/OS V7.1 on the mainframe that will host Central Data Warehouse and Data Mart databases

Phase 2

Start the installation process from the TDW Control Server machine. During the install process, provide user IDs, password and Port number so that the Control Server can access and create all the Central Data Warehouse and Data Mart databases on the remote systems

On the TDW Control Server, using the DB2 Warehouse Manager, configure IBM DB2 to use the TDW Control Database (TWH_MD) From the TDW Control Server, install and configure all the Central Data Warehouse and Data Mart databases on the z/OS systems

Phase 3

Install the Crystal Enterprise Server

Prepare the IBM DB2 database Servers

Central Data WarehouseData Mart


TDW Control Server

Data Mart

Perform TDW installation from the Control Server

Central Data WarehouseData Mart


TDW Control Server

Data Mart

On the systems that will act as Warehouse Agent Sites:

Install the IBM DB2 Warehouse Manager

Install the Warehouse Agent component

Phase 4 Create Warehouse agent sites (optional)

Warehouse AgentSite 3

TDW Control Server



Chapter 3. Getting Tivoli Data Warehouse 1.2 up and running 73

3.1.1 Ensuring fully qualified host namesYour operating system must be configured to provide Tivoli Data Warehouse with a fully qualified computer name rather than a short name. This is especially important in environments with many different operating systems. To ensure that your system is configured to provide a fully qualified computer name, follow the instructions in the next sections.

On AIX systemsThe default domain name search order is as follows:

1. Domain Name Server (DNS) Server2. Network Information Service (NIS)3. Local /etc/hosts file

If the /etc/resolv.conf does not exist, the /etc/hosts file is used. If only the /etc/hosts file is used, the fully qualified computer name must be first one that is listed after the IP address.

Verify that the /etc/resolv.conf file exists and contains the appropriate information, such as:

domain mydivision.mycompany.comnameserver 123.123.123.123

If NIS is installed, the /etc/irs.conf file overrides the system default. It contains the following information:

hosts = bind, local

The /etc/netsvc.conf file, if it exists, overrides the /etc/irs.conf and the system default. It contains the following information:

hosts = bind,local

On Linux systemsVerify that the /etc/resolv.conf file exists and contains the appropriate information, such as:

domain mydivision.mycompany.comnameserver 123.123.123.123

A short name is used if the /etc/nsswitch.conf file contains a line that begins as follows and if the /etc/hosts file contains the short name for the computer:

hosts: files


To correct this, follow these steps:

1. Change the line in the /etc/nsswitch.conf file to:

hosts:dns nis files

2. Stop the network services.

3. Restart the network service.

On Solaris systemsVerify that the /etc/resolv.conf file exists and contains the appropriate information, such as:

domain mydivison.mycompany.comnameserver 123.123.123.123

A short name is used if the /etc/nsswitch.conf file contains a line that begins as follows and if the /etc/hosts file contains the short name for the computer:

hosts: files

To correct this, follow these steps:

1. Change the line in the /etc/nsswitch.conf file to:

hosts: dns nis files

2. Enter the following command to stop the inet service:

/etc/init.d/inetsvc stop

3. Enter the following command to restart the inet service:

/etc/init.d/inetsvc start

On Microsoft Windows NT systemsTo verify that a primary domain system (DNS) suffix is set, follow these steps:

1. From the Windows task bar, click Start -> Settings -> Control Panel.

2. In the Control Panel windows, double-click Network.

3. Click the Protocols tab.

4. Select the TCP/IP Protocol and then click Properties.

5. Click the DNS tab.

6. Ensure that the field Domain contains a domain suffix. If it does not, type the suffix, click OK, and restart the computer when prompted.


On Windows 2000 systemsTo verify that a primary domain name system (DNS) suffix is set, follow these steps:

1. On the desktop, right-click My computer -> Properties.

2. Click the Network Identification tab.

3. Ensure that the field Full Computer Name contains a fully qualified domain name. If it does not, follow these steps.

a. Click Properties.

b. Click More.

c. In the field Primary DNS suffix for this computer, type the primary DNS suffix, and restart the computer when prompted.

3.1.2 Installing and configuring IBM DB2 client and serverAs described in 2.1.3, “Database requirements” on page 32, Tivoli Data Warehouse 1.2 is dependent of IBM DB2 Universal Database Enterprise Edition server for a single system installation, or for the control server, central data warehouse, and data mart components of a distributed installation. For the Crystal Enterprise Professional for Tivoli in a distributed Tivoli Data Warehouse installation, you can use IBM DB2 client at the proper level to access the data mart databases.

This section provides information about the following topics:

� “Using an existing installation of DB2 client or server” on page 76� “IBM DB2 Server installation on AIX” on page 78� “IBM DB2 Fix Pack 8 installation on AIX” on page 82� “IBM DB2 Server installation on Windows 2000” on page 82� “IBM DB2 Fix Pack 8 installation on Windows 2000” on page 84� “Installing IBM DB2 Client” on page 85

Using an existing installation of DB2 client or server To use an existing installation of IBM DB2 client or server, make sure it is at the version and patch level specified by the Tivoli Data Warehouse 1.2 software requirements. Refer to Chapter 2, “Planning for Tivoli Data Warehouse 1.2” on page 27 and to the Tivoli Data Warehouse Release Notes, SC32-1399. Use the db2level command to determine the Fix Pack Level of the existing IBM DB2 server environment.


On Windows systems, at DB2 7.2 Fix Pack 8, db2level command returns a message similar to the following:

DB21085I Instance "db2admin"uses DB2 code release "SQL07026"with level identifier "03070105"and informational tokens "DB2 v7.1.0.75","n021110" and "WR21314".

The string "DB2 v7.1.0.75" indicates that the system is at IBM DB2 V7.2 Fix Pack 8 level. The last two information tokens ("n021110" and "WR21314") vary by operating system type and the specific patches that are installed.

If the IBM DB2 client or server is installed but does not have Fix Pack 8 or higher, it is required to upgrade to Version 7.2 Fix Pack 8 (at minimum) before running the Tivoli Data Warehouse installation wizard. You cannot install Tivoli Data Warehouse with a lower Fix Pack level.

In addition to checking the IBM DB2 proper version and level, consider the following possibilities:

� If you have Lightweight Directory Access Protocol (LDAP), make sure that it is disabled for this IBM DB2 instance.

In a IBM DB2 command window, run the following command:

db2set -all | more

Examine the value of DB2_ENABLE_LDAP settings. If this value is listed and is set to YES, disable LDAP and restart the IBM DB2 server by running the following commands in a IBM DB2 command windows:

db2set db2_enable_ldap=NOdb2stop forcedb2start

If the value is not listed or is set to NO, no action is required.

� On UNIX systems, make sure that the IBM DB2 administration client is installed. If you can start the DB2 Control Center, the client is installed.

� On Windows Systems, if you did not perform a typical installation of IBM DB2 Universal Database, make sure you have the following components:

– Administration and configuration Tool– Application Development Interfaces– Data Warehousing Tools

� If you selected the Typical installation when installing IBM DB2 Universal Database on a Windows system, all of the necessary components are available.

� Make sure that other applications using the IBM DB2 instance do not have database names that duplicate those of Tivoli Data Warehouse.


IBM DB2 Server installation on AIXThis section describes the IBM DB2 Universal Database Enterprise Edition Server Version 7.2 installation process on AIX.

1. Log in as a user with root authority, move to the directory where the DB2 7.2 Server for AIX CDROM is mounted, and start the DB2 setup utility, as follows:

# ./db2setup

2. The Install DB2 V7.2 window, shown in Figure 3-2, appears. Select DB2 Administration Client and DB2 UDB Enterprise Edition.

Figure 3-2 Install DB2 V7 components


3. A new DB2 instance should be created for the Administration Server database. We specified the DB2 instance name db2inst1, as shown in Figure 3-3. You should also specify /home/db2inst1 as the instance owner directory.

Figure 3-3 Create DB2 Services - DB2 Instance db2inst1


4. The installation process creates the DB2 fenced user. We specified the DB2 instance name db2fenc1, as shown in Figure 3-4.

Figure 3-4 Create the DB2 fenced user


5. Next, Figure 3-5 shows the values we used to create the user ID for the DB2 Administration Server.

Figure 3-5 Administration Server window

6. The installation process creates and sets the values of several environment variables, for example, DB2SYSTEM.

7. At the end of the installation process, you may check the installation log file created at /tmp/db2setup.log.

8. The installed JDBC code level needs to be upgraded to Version 2.0. You should log on to the system with a valid DB2 user ID, and issue the following commands:

– For Bash, Bourne, or Korn shell:

# . INSTHOME/sqllib/db2profile# cd /INSTHOME/sqllib/java12/# . ./usejdbc2

Where INSTHOME is the home directory of the instance.

– Verify that the JDBC level is correct by entering the following command:

# echo $CLASSPATH

The output must include the following path:

INSTHOME/sqllib/java12/db2java.zip


IBM DB2 Fix Pack 8 installation on AIXThis session describes the installation of DB2 Fix Pack 8 on AIX. Here are the steps for installing IBM DB2 Fix Pack 8:

1. Stop all database activity before applying this Fix Pack. To stop all database activity, issue the commands:

# db2stop# db2admin stop

2. Unzip the Fix Pack using the following command to get a tar file:

# gzip FP8_U484610.tar.Z

3. Un-tar the Fix Pack using the following command to extract the Fix Pack files.

# tar -xvf FP8_U484610.tar

4. Run the following command to install the Fix Pack from the location where you un-tar the Fix Pack files.

# ./installFixpack

5. Provide the DB2 instance password if prompted.

6. The installation wizard copies the files and finishes the installation of the Fix Pack.

7. Un-tar the efix for Fix Pack 8

tar -xvf special_U484610.tar

8. Make a backup of files db2bp and db2level, in directory /usr/lpp/db2_07_01/bin directory

9. Copy the files db2bp and db2level to the /usr/lpp/db2_07_01/bin directory.

IBM DB2 Server installation on Windows 2000This section describes the IBM DB2 Universal Database Enterprise Edition Server Version 7.2 installation process on Windows.

1. Load the DB2 installation media.

Note: If you are using a 32-bit IBM DB2 Server, make sure to install the 32-bit Fix Pack 8. Or if you are using a 64-bit IBM DB2 Server, make sure to install the 64-bit Fix Pack 8.

Note: Use the installation media provided with the TDW product. This ensures that you install the correct version and Fix Pack of DB2.


2. Select Start -> Run. Type in D:\setup.exe and click OK to start the installation. From the Installation window, select Install.

3. The Select Products window opens. From this window you can select the component(s) of DB2 for Windows you would like to install. Select DB2 Enterprise Edition as shown in Figure 3-6. Click Next.

Figure 3-6 Select DB2 Enterprise Edition

4. The Select Installation Type window opens. Select the installation type you prefer. We selected Typical.

5. Select the installation directory. In our environment, we used C:\DB2\SQLLIB.

6. The installation prompts for the DB2 administrative user ID. We selected and set the password for the db2admin user ID.

7. After the installation wizard copies the DB2 files onto the machine, the Install OLAP Starter Kit window opens. Select Do not install the OLAP Starter Kit and then click Continue.

8. When the setup complete the installation process, click Finish.

9. Update Java™. The installed JDBC code level needs to be upgraded to Version 2.0. You should open a DOS-command prompt window and issue the following commands, where DB2_DIR is the DB2 installation directory:

cd DB2_DIR\java12usejdbc2

The usejdbc2 command will copy the appropriate version of db2java.zip into the DB2_DIR\java12 directory.

10.Reboot the machine.


IBM DB2 Fix Pack 8 installation on Windows 2000This section describes the installation of IBM DB2 Fix Pack 8 on Windows.

If you are installing the Fix Pack by using the Administrator account of Windows 2000 Advanced Server, please make sure you complete the following steps:

1. Click Start -> Programs -> Administrative Tools -> Local Security Policy In the popup window, click Local Security Settings -> User Rights Assignment.

2. In the window, you will see lists of user rights. Make sure the Administrator account has the following rights:

– Act as part of operating system– Create a token object– Increase quotas– Replace a process level token

3. Stop all database activity before applying this Fix Pack. To stop all database activity, on a DB2 command window, run:

c:\db2\sqllib\bin:\>db2stop

or

c:\db2\sqllib\bin:\>db2stop force

and

c:\db2\sqllib\bin:\>db2admin stop

4. Unzip and extract the Fix Pack files to a temporary directory.

5. Run the following command to install Fix Pack from the Fix Pack directory:

c:\fp8_wr21314\setup.exe

6. Key in the DB2 instance owner password if the setup prompts for it and click Next.

7. The wizard shows the selection window. Click Next to continue.

8. Extract the efix for Fix Pack 8 into a temporary directory. Take a backup of the files db2bp.exe and db2level.exe

Note: IBM DB2 Universal Database Enterprise Edition 7.2 Fix Pack 8 is the minimum requirement to install Tivoli Data Warehouse 1.2 on top. This section describes how to install Fix Pack 8 on an IBM DB2 Universal Database Enterprise Edition. If you use the IBM DB2 version 7.10 Fix Pack 10 supplied on Tivoli Data Warehouse 1.2 installation media, you must skip this section!

Note: Once you have installed a Fix Pack, you won’t be able to un-install it.


9. Copy the files db2bp.exe and db2level.exe from the temporary directory to the <DB2DIR>\bin directory, where <DB2DIR> is the IDM DB2 installation directory:

c:\special_wr21314\copy db2bp.exe c:\db2\sqllib\bin\c:\special_wr21314\copy db2level.exe c:\db2\qllib\bin\

10.Reboot the machine.

Installing IBM DB2 ClientThis section describes the installation of IBM DB2 Client on Windows.

If using an existing installation of the IBM DB2 client, ensure that it is at V7.2 Fix Pack 8 level or higher. Otherwise, upgrade it to at least V7.2 Fix Pack 8.

The IBM DB2 client software can be obtained by installing the IBM DB2 Administration client package, that comes with the IBM DB2 installation media provided by Tivoli Data Warehouse 1.2. Use the IBM DB2 installation media provided with the Tivoli Data Warehouse. This ensures that you get the correct version.

The IBM DB2 Administration client package can be installed by performing the following actions:

1. Insert the IBM DB2 installation media into the CD-ROM drive. If the program does not start automatically when you insert the DB2 installation CD, run the setup.exe program in the root directory of the CD. Click Install, provide the following information when prompted:

2. Select DB2 Administration Client. Make sure that the DB2 Enterprise Edition product is not selected.

3. Select the Typical Installation.

4. You can change the default destination folder (optional). However, we used c:\db2\sqllib in our installation. Accept the default values for the remaining items in this panel.

5. For the Control center server user name, specify a user that does not already exist on your system. When the DB2 installation wizard creates the user for you, it ensures that the user has the correct roles and privileges. If the user already exists, consider deleting it and letting the IBM DB2 installation recreate it. For more information about DB2 naming rules, refer to IBM DB2 Universal Database for Windows Quick Beginnings, GC09-2971.

Note: The Server of IBM DB2 Universal Database Enterprise Edition has a client included.


In a safe place, record the IBM DB2 user name and password that was specified for the IBM DB2 client.

6. On the Start Copying Files panel, select Next. and Finish to complete the setup.

3.1.3 Crystal Enterprise installationCrystal Enterprise Professional Version 9 for Tivoli, a reporting solution provided with Tivoli Data Warehouse 1.2. Crystal Enterprise, typically resides on a system that does not have other Tivoli Data Warehouse components installed on it. Users access Crystal Enterprise reports using a Web browser from any system.

Crystal Enterprise is integrated with Tivoli Data Warehouse 1.2 to provide historical reporting. Warehouse enablement packs provide reports designed for use with Crystal Enterprise. You access the reports with a Web browser that connects to the Crystal Enterprise server.

Crystal Enterprise Automated Process SchedulerThe Crystal Enterprise Automated Process Scheduler (APS) requires a database to store information about the system and its users. By default, the Setup program installs and configures its own Microsoft Data Engine (MSDE) database if necessary. APS clustering is automatically supported by the default MSDE database. The MSDE is a client/server data engine that provides local data storage and is compatible with Microsoft SQL Server. If you already have the MSDE or SQL Server installed, the installation program creates the APS database using your existing database engine.

If the Microsoft Data Engine (MSDE) or Microsoft SQL Server is already installed on the local machine, you must first set up an account for the APS, as follows:

1. Determine whether the APS should use Windows NT or SQL Server authentication when connecting to your local database installation.

2. Using your usual administrative tools, create or select a user account that provides Crystal Enterprise with the appropriate privileges to your database server:

– If you want the APS to connect to its database using Windows NT authentication, ensure that the Windows NT user account that you assign to the APS belongs to the System Administrator’s role in your SQL Server installation.

In this scenario, the Windows NT user account that you assign to the APS is not actually used to create the system database during the installation process. Instead, your own Windows NT administrative account is used to create the database, so verify that your Windows NT account also belongs to the System Administrator’s role in your SQL Server installation.


– If you want the APS to connect to its database using SQL Server authentication, the login that you assign to the APS must belong to the Database Creators role in your SQL Server installation. In this scenario, the SQL Server credentials that you assign to the APS are also used to create the database and its tables.

3. Verify that you can log on to SQL Server and carry out administrative tasks using the account you set up for use by the APS.

Installing and configuring Crystal Enterprise ServerYou must install and configure Crystal Enterprise Professional Version 9 for Tivoli prior to installing Tivoli Data Warehouse 1.2. Tivoli Data Warehouse supports the full stand-alone installation of Crystal Enterprise Professional Version 9 for Tivoli. Crystal Enterprise requires that a compatible Web server be running on the same machine for a full stand-alone installation.

Ensure that the following components are installed and configured correctly before you install Crystal Enterprise Professional for Tivoli. Refer also to 2.1.4, “Crystal Enterprise requirements” on page 33 for additional information.

� Windows NT or Windows 2000 SP3 at minimum.

� Web Server software, such as Microsoft IIS, iPlanet Enterprise Server, or IBM HTTP Server.

� Internet Explorer or Netscape Navigator Web browser.

� IBM DB2 Client, in order to have access to the Data Mart for reporting. Refer to 3.1.2, “Installing and configuring IBM DB2 client and server” on page 76 for details.

In order to install Crystal Enterprise Professional Version 9 for Tivoli, complete the following steps:

1. Run setup.exe, from the win32 directory of your product distribution. On the Welcome window, click Next to proceed.

2. Accept the license agreement.

Note: Use the installation media provided with the Tivoli Data Warehouse 1.2 product. This ensures that you install the correct version of Crystal Enterprise.


3. As shown in Figure 3-7, change the destination folder (optional) and choose the installation type New - Install a new Crystal Enterprise System. Click Next.

Figure 3-7 Installation Type

4. The setup program checks to see whether or not the Microsoft Data Engine (MSDE) or Microsoft SQL server is installed on the local machine. If the setup program detects a database, use the Microsoft SQL Server Authentication window to provide the credentials that correspond to the database account you setup for the APS. The default user ID for the database account is named sa. For more information refer to “Crystal Enterprise Automated Process Scheduler” on page 86.

If the setup program does not detect an existing database, the installation process will install Microsoft Data Engine and will create the credentials for the default SQL administrator account (sa) user ID. The installation wizards prompts you for the password to be used by the sa user ID. The setup program later configures the APS to connect to its system database using the sa account and the password you create here. Click Next.


This is shown in Figure 3-8.

Figure 3-8 MSDE security configuration

5. Figure 3-9 shows the Crystal Enterprise Professional Version 9 for Tivoli installation in progress.

Figure 3-9 Installation window


6. Click Finish on the completion window, as shown in Figure 3-10.

Figure 3-10 Completion window

7. If the Web Server installed on the local machine is a supported version, then the setup program installs and configures the appropriate Crystal Enterprise Web Connector. Therefore, when the installation is complete, you can access the Crystal Enterprise Professional for Tivoli server by opening your Web browser and pointing it to:

http://<CRYSTALSERVER>/crystal/enterprise9/

In this URL, <CRYSTALSERVER> is the host name of the Crystal Enterprise Professional for Tivoli server machine.


8. Crystal Enterprise Launchpad is launched, as shown in Figure 3-11. Click Administrative Tool Console.

Figure 3-11 Crystal Enterprise Launchpad


9. Logon as Administrator, as shown in Figure 3-12 to verify that the setup program installed and configured the appropriate Crystal Enterprise Web connector. The default password for the Administrator account is set to blank (no password).

Figure 3-12 Crystal Administration Tools

For details on how to use the Crystal Enterprise Professional for Tivoli Web interface, refer to the Crystal Enterprise Professional Version 9 for Tivoli Administrator’s Guide manual, which is provided with Crystal Enterprise Professional Version 9 for Tivoli.

3.2 Tivoli Data Warehouse 1.2 installationThe basic prerequisite components of the Tivoli Data Warehouse 1.2 application have been introduced in the previous sections. We now cover the different architectures of a Tivoli Data Warehouse 1.2 installation, for example, how the components can be reasonably placed on one or more machines, and how they work together.


As described in 2.2, “Physical and logical design considerations” on page 36, there are several ways to deploy a Tivoli Data Warehouse 1.2 environment. Next, we go into detail on the installation steps and initial configuration of the environment, as follows:

� In 3.3, “Quick start deployment” on page 93 we provide installation steps for a stand-alone system installation with all the components installed on a single Windows 2000 Server system.

� In 3.4, “Distributed deployment” on page 103 we describe an example scenario of a Tivoli Data Warehouse 1.2 distributed environment and provides the installation steps for a TDW control server, one or more central data warehouse, and data mart databases on Windows, UNIX, and z/OS systems. Such deployment is best suited for large enterprises, enterprises distributed across widely-separated time zones, and enterprises containing z/OS systems collecting systems management data.

3.3 Quick start deploymentThis type of installation quickly deploys Tivoli Data Warehouse 1.2, with the TDW control server and its database (TWH_MD), one central data warehouse database (TWH_CDW), and one data mart database (TWH_MART) on a single Windows system. Since the TDW control server must be on a Windows 2000, Windows 2003, or Windows NT machine, thus the single machine configuration cannot be installed on a UNIX system.

Before proceeding with the quick start installation of Tivoli Data Warehouse, ensure that the following is setup and installed:

1. Check all the prerequisites. Refer to 2.1, “Hardware and software requirements” on page 28.

2. Ensure that the system’s DNS settings resolve fully qualified host names. Refer to 3.1.1, “Ensuring fully qualified host names” on page 74.

3. Make sure the Crystal Enterprise Professional Version 9 for Tivoli Server is functional. Refer to 3.1.3, “Crystal Enterprise installation” on page 86.

4. Install and/or upgrade IBM DB2 Universal Database Enterprise Edition to version 7.2 and minimum of Fix Pack 8 level. Refer to 3.1.2, “Installing and configuring IBM DB2 client and server” on page 76.


Figure 3-13 shows a typical quick start configuration, mapped to an existing Tivoli environment. All the Tivoli Data Warehouse 1.2 components (TDW control server, central data warehouse, and data mart databases) are installed on a single Windows 2000 machine.

Figure 3-13 Quick start deployment configuration

3.3.1 Quick start deployment: installation and configurationThis section provides instructions for installing Tivoli Data Warehouse 1.2 on a single system. This will install and configure the following components:

� TDW control server� Control center, central data warehouse, and data mart databases.� Settings on how to connect to the Crystal Enterprise server

In order to perform the Tivoli Data Warehouse 1.2 installation, use the following steps:

1. Insert the Tivoli Data Warehouse 1.2, installation CD into the CD-ROM drive. If the installation wizard does not start up, run the setup.exe program, which is located in the root directory of the CD.

Crystal Enterprise Server - Windows 2000 Server - IIS Web Server - Crystal Enterprise Professional version 9 for Tivoli

Stand alone architecture

- Windows 2000 Server - TDW Control Server - Central data warehouse - Data Mart

Tivoli EnvironmentTWH_MART

TWH_MD

TWH_CDW

Attention: Make sure you have completed all the prerequisite tasks before proceeding with the Tivoli Data Warehouse 1.2 installation.


2. Figure 3-14, the InstallShield Wizard, is displayed. Click Next to proceed.

Figure 3-14 InstallShield Wizard

3. Figure 3-15 displays the directory of the Tivoli common logging. The default location is %ProgramFiles%\IBM\Tivoli\common\. Each product stores logging information in a separate subdirectory within the Tivoli Common Logging directory. Click Next.

Figure 3-15 Tivoli common logging directory


4. Figure 3-16 displays the Setup Window. Select Quick start. Specify the installation directory (Optional); the default is %ProgramFiles%\TWH\; and click Next.

Figure 3-16 Setup window


5. Figure 3-17, the information window, specifies the existing IBM DB2 instance owner user ID and password to be used to create and connect to the Tivoli Data Warehouse databases. Click Next.

Figure 3-17 DB2 connection

6. Figure 3-18 specifies the following connection information for the Crystal Enterprise server, and then click Next.

– Host name: The fully qualified host name of the machine name where Crystal Enterprise Professional Version 9 for Tivoli is installed.

– User name: The user credentials Tivoli Data Warehouse will use on the connection to the Crystal Enterprise server. Defaults to Administrator.

– Password: Password for the Administrator user ID. Defaults to blank (no password).

The default values above are based on a new Crystal Enterprise Professional Version 9 for Tivoli Server limited edition shipped with Tivoli Data Warehouse 1.2.


Figure 3-18 Crystal connection

7. Figure 3-19 indicates the components to be installed and their location. In our case, one TDW control server, one central data warehouse database, and one data mart database on the local computer. Click Install to start the installation.

Figure 3-19 Summary window


8. After the installation is over, a completion window is displayed, as shown in Figure 3-20. It has a successful completion notice or messages describing problems. Make sure the window does not list any warnings or errors. Click Next. If you are prompted to restart, click Yes, restart my system.


3.3.2 Configuring the control databaseThis section contains instructions for configuring the control database. This database contains metadata for Tivoli Data Warehouse and the IBM DB2 Data Warehouse Center. Therefore we need to specify the Tivoli Data Warehouse control database (TWH_MD) to be the main control database for both the IBM DB2 Data Warehouse Center and the IBM DB2 Warehouse Control Database Management components.

In order to configure the IBM DB2 Data Warehouse Center, complete the following steps:

1. On the Windows task bar, click Start -> Programs -> IBM DB2 -> Control Center.

2. From the IBM DB2 Control Center, start the IBM DB2 Data Warehouse Center by choosing Tools -> Data Warehouse Center.

3. In the Data Warehouse Center Logon window, click Advanced. There is no need to provide login information.


4. In the Advanced window, shown in Figure 3-21, type TWH_MD in the control database field. Click OK to return to the logon window.

Figure 3-21 Configuring the IBM DB2 data warehouse center

5. Click Cancel to exit from the Data Warehouse Center login screen.

In order to configure the IBM DB2 Warehouse Control Database Management, complete the following steps:

1. Open the Data Warehouse Center - Control Database Management window by selecting Start -> Programs -> IBM DB2 -> Warehouse Control Database Management.

2. Type the TWH_MD in new control database.

3. Do not change the schema name.

4. Type the IBM DB2 instance owner user ID and password for the control database, and then click OK. In our scenario, we used the db2admin user ID.

5. When the message, Processing has completed, appears as shown in Figure 3-22, click Cancel.


Figure 3-22 Configuring the Warehouse Control Database Management

3.3.3 Creating ODBC connections to the data mart databasesThe Crystal Enterprise server needs to have access to the information stored in the data mart database for reporting. After the quick start install of Tivoli Data Warehouse 1.2 finished, there is only one data mart database created (TWH_MART), and an ODBC connection for this database must be set up. You may further expand this environment by either creating additional data mart databases, or installing warehouse enablement packs that use different data marts. In any case, there must be an ODBC connection to these data marts defined on the Crystal Enterprise server.

Tivoli Data Warehouse 1.2 provides the twh_create_datasource script that sets up ODBC connections to the data mart databases. You may use this script or create the ODBC connections manually. In order to create an ODBC connection to the TWH_MART database using the twh_create_datasources script, perform the following tasks:

1. On the Crystal Enterprise server machine, open an IBM DB2 command window: Start-> Programs-> IBM DB2-> Command Window.

2. Copy both the twh_create_datasource.bat and ODBCcfg.exe files from the disk1\tools directory of the Tivoli Data Warehouse 1.2 installation media to a temporary directory on the Crystal Enterprise server machine.


3. Run the twh_create_datasource script using the following syntax as shown in Example 3-1:

twh_create_datasource <DBtype> <ID> <odbcname> <DBname> <SRVname> <port>

In this script:

<DBtype> Can be set to DB2UDB or DB390 depending on the location of the database.

<ID> An unique identifier for the local node name. The script creates node names following the naming convention: TDWCS%ID%. In our example, we used ID=10 resulting in node name TDWCS10.

<odbcname> The ODBC data source name.

<DBname> The data mart database name.

<SRVname> The IBM DB2 server in which the data mart database resides. This must be the fully qualified host name.

<port> The port number to connect to the IBM DB2 server.

Example 3-1 twh_create_datasource script

C:\Temp>twh_create_datasource.bat DB2UDB 10 TWH_MART TWH_MART tdw001.itsc.austin.ibm.com 50000

Creating DB2/UDB datasource TWH_MART

C:\Temp>db2cmd /w /c /i db2 catalog tcpip node TDWCS10 remote tdw001.itsc.austin.ibm.com server 50000DB20000I The CATALOG TCPIP NODE command completed successfully.DB21056W Directory changes may not be effective until the directory cache is refreshed.

C:\Temp>db2cmd /w /c /i db2 catalog database TWH_MART at node TDWCS10 authentication serverDB20000I The CATALOG DATABASE command completed successfully.DB21056W Directory changes may not be effective until the directory cache is refreshed.

C:\Temp>C:\Temp\ODBCcfg.exe DB2 TWH_MART TWH_MART

No Username was provided. Skipping connection test.

C:\Temp>


3.4 Distributed deploymentA distributed deployment, has a central data warehouse or data mart database that is not on the control server, or has more than one central data warehouse or data mart databases.

Before proceeding with the installation of Tivoli Data Warehouse on a distributed environment, ensure that the following software is set up and installed:

1. Check all the prerequisites. Refer to 2.1, “Hardware and software requirements” on page 28.

2. Ensure that the DNS settings on ALL the systems resolve fully qualified host names. Refer to 3.1.1, “Ensuring fully qualified host names” on page 74.

3. Make sure the Crystal Enterprise Professional Version 9 for Tivoli Server is functional. Refer to 3.1.3, “Crystal Enterprise installation” on page 86.

4. Install and/or upgrade IBM DB2 Universal Database Enterprise Edition to version 7.2 and minimum Fix Pack 8 on the Windows and UNIX systems that serve as:

– TDW control server– Central data warehouse database servers– Data mart database servers

Refer to 3.1.2, “Installing and configuring IBM DB2 client and server” on page 76.

5. Install and/or upgrade DB2 UDB for OS/390 and z/OS to version 7.1 on the z/OS and OS/390 systems that host a central data warehouse database and Data Mart databases.

Figure 3-23 shows a distributed deployment scenario that will illustrate the installation steps presented in this section. This distributed environment is mapped to an existing Tivoli environment and consists of z/OS and distributed systems. In this scenario we have all the Tivoli Data Warehouse components distributed across different systems, that is, the TDW control server is on a Windows 2000 machine, two central data warehouse database servers are on Windows 2000 and z/OS, and 2 data mart databases on AIX and z/OS.

Such an environment can be installed using a two-step approach. The first step installs all Tivoli Data Warehouse components on distributed systems. The second step installs the Tivoli Data Warehouse components on the z/OS system.

The installation steps based on this scenario are provided next in 3.4.1, “Distributed deployment installation: Windows and UNIX” on page 104 and 3.4.2, “Distributed deployment installation: z/OS” on page 115.


Figure 3-23 Distributed deployment scenario

3.4.1 Distributed deployment installation: Windows and UNIXThis section provides instructions for installing the following components of the Tivoli Data Warehouse environment shown in Figure 3-23 as follows:

� TDW control server on machine TDW003� One central data warehouse database on machine TDW004� One data mart database on machine TDW009

The installation steps described here must be performed on the TDW control server machine.

TDW Control Server

Windows 2000 ServerHostname: TDW003

TWH_MD


Windows 2000 ServerIIS Web ServerCrystal Enterprise Professional 9 for TivoliHostname: TDW002

Tivoli Environment



TWH_CDW

Data Mart

AIX 5Hostname: TDW009

Data MartTWH_MART

z/OS environment

Data SourcesCentral Data WarehouseData MartHostname: wtsc66oe

Data MartCentral DataWarehouse

Data Source

Data Source

Data Source


We assume that the Crystal Enterprise Professional for Tivoli Server has already been deployed. In this case, the Tivoli Data Warehouse installation process will connect to the Crystal Enterprise Professional for Tivoli Server and install only the Crystal Publishing Wizard on the TDW control server system.

In order to install the distributed environment portion of the scenario presented in Figure 3-23, perform the following steps:

1. Insert the Tivoli Data Warehouse 1.2, installation CD into the CD-ROM drive. If the installation wizard does not start up, run the setup.exe program, which is located in the root directory of the CD.

2. The panel in shown in Figure 3-24, the InstallShield Wizard, is displayed. Click Next to proceed.

Figure 3-24 Install Shield Wizard


3. Figure 3-25 shows the directory of the Tivoli common logging. The default location is %ProgramFiles%\IBM\Tivoli\common\. Each product stores logging information in a separate subdirectory within the Tivoli Common Logging directory. Click Next.

Figure 3-25 Tivoli Common Logging Directory


4. Select Custom or Distributed, for distributed deployment. You can use the default installation directory %ProgramFiles%\TWH or change it to your needs. Then click Next. However, we changed the installation directory name to C:\TWH as shown in Figure 3-26.

Figure 3-26 Setup Window

5. A warning message is displayed to emphasize the need to fulfill all of the prerequisite tasks required for installing Tivoli Data Warehouse 1.2 in a distributed architecture. This is shown in Figure 3-27.


Figure 3-27 Before proceeding with TDW 1.2 distributed installation

6. Specify the existing IBM DB2 instance owner user ID and password to be used to create and connect to the TDW control server database (TWH_MD), as shown in Figure 3-28. Click Next.

Figure 3-28 DB2 connection


7. You are prompted to provide the list of servers that will host the central data warehouse database in your environment. Click Add and type the IBM DB2 Server information on the system that will host the remote central data warehouse database, as shown in Figure 3-29. Click Next.

Figure 3-29 Central data warehouse on remote host

8. Make sure that all of the central data warehouse database servers are included in your list. The list for our case study environment is shown in Figure 3-30. Click Next.


Figure 3-30 Central data warehouse database server list

9. You are prompted to provide the list of servers that will host the data mart database in your environment. Click Add and type the IBM DB2 Server information on the system that will host the remote data mart database, as shown in Figure 3-31. Click Next.

Figure 3-31 Data mart on remote host


10.Make sure that all of the data mart database servers are included in your list. The list for our case study environment is shown in Figure 3-32. Click Next.

Figure 3-32 Data mart database server list

11.Figure 3-33 specifies the following connection information for the Crystal Enterprise server:

– Host name: The fully qualified host name of the machine name where Crystal Enterprise Professional Version 9 for Tivoli is installed.

– User name: User credentials Tivoli Data Warehouse will use on the connection to the Crystal Enterprise server. Default to Administrator.

– Password: Password for the Administrator user ID. Default to blank (no password).

The default values above are based on a new Crystal Enterprise Professional Version 9 for Tivoli Server limited edition shipped with Tivoli Data Warehouse 1.2. Click Next.


Figure 3-33 Crystal connection

12.An installation summary is then displayed showing which Tivoli Data Warehouse component is going to be installed on which server, as shown in Figure 3-34. Click Install, or click Back to make changes.

Figure 3-34 Summary window

Components location


13.A completion window is displayed as shown in Figure 3-35. It has a successful completion notice or messages describing problems. Make sure the window does not list any warnings or errors. Click Next. If you are prompted to restart, click Yes, restart my system.


Configuring the control databaseThis section contains instructions for configuring the control database. This instructions should be performed on the TDW Control Center machine.

It is necessary to specify the Tivoli Data Warehouse control database (TWH_MD) to be the main control database for both the IBM DB2 Data Warehouse Center and the IBM DB2 Warehouse Control Database Management components.

In order to configure the IBM DB2 Data Warehouse Center, complete the following steps:

1. On the Windows task bar, click Start -> Programs -> IBM DB2 -> Control Center.

2. From the IBM DB2 Control Center, start the IBM DB2 Data Warehouse Center by choosing Tools -> Data Warehouse Center.

3. In the Data Warehouse Center Logon window, click Advanced. There is no need to provide login information.


4. In the Advanced window, shown in Figure 3-36, type TWH_MD in the control database field. Click OK to return to the logon window.

Figure 3-36 Configuring the IBM DB2 Data Warehouse Center

5. Click Cancel to exit from the Data Warehouse Center login screen.

In order to configure the IBM DB2 Warehouse Control Database Management, complete the following steps:

1. Open the Data Warehouse Center - Control Database Management window by selecting Start -> Programs -> IBM DB2 -> Warehouse Control Database Management.

2. Type TWH_MD in the new control database.

3. Do not change the schema name.

4. Type the IBM DB2 instance owner user ID and password for the control database, and then click OK.

5. When the Processing has completed message appears, as shown in Figure 3-37, click Cancel.


Figure 3-37 Configuring the Warehouse Control Database Management

3.4.2 Distributed deployment installation: z/OSThis section contains instructions for configuring completing the installation of the distributed deployment scenario that was depicted in Figure 3-23 on page 104. Before proceeding with the installation, refer to 2.1, “Hardware and software requirements” on page 28 to ensure that the z/OS environment is at the proper levels. Here we show the steps used to define the central data warehouse and the data mart on the z/OS system. This configuration is particularly needed if you plan to use Tivoli Decision Support for 390, which requires that the central data warehouse and data marts databases be created on the same system.

Before installing central data warehouse or data mart databases on z/OS, ensure that your DB2 on z/OS is correctly configured. Make sure you have completed the following tasks:

1. Ensure that the DB2 Universal Database for OS/390 and z/OS V7.1 has the DB2 Management Clients Package installed (FMID JDB771D).

2. Consult your DB2 UDB for OS/390 and z/OS administrator for the following information:

– User ID and password with SYSADM authority– Database location– Storage volume, storage group, and storage VCAT– Buffer pool– Database names– Tablespace names– UTF8 tablespace names– Tablespace size (primary and secondary)


Creating the central data warehouse on z/OSThe following steps should be performed on the TDW Control Center machine. The installation process will trigger scripts on the z/OS system that will create and configure the database:

1. Insert the Tivoli Data Warehouse 1.2, installation CD into the CD-ROM drive of the control server. If the installation wizard does not start up, run the setup.exe program, which is located in the root directory of the CD.

2. The install process displays the welcome panel. Click Next. It also provides the location of the common log files. Click Next.

3. The install proceeds by verifying the current Tivoli Data Warehouse environment and configuration. You will be prompted with two options, as shown in Figure 3-38: You can create one or more central data warehouse databases, or you can create one or more data mart databases. In this case, select Add central data warehouses and click Next.

Figure 3-38 Adding central data warehouses

4. The installation wizard searches for the existing Tivoli Data Warehouse configuration and displays a list of central data warehouse locations found. It also allows you to add to the existing list of systems in which central data warehouse databases will be created. Click Add on the install window.

5. With the help of your z/OS system administrator, specify the IBM DB2 configuration information on the z/OS system, as shown in Figure 3-39. Specify database type DB2 for z/OS and S/390®, a fully qualified host name, port number, and a user ID with SYSADM authority. Click Next.


Figure 3-39 z/OS IBM DB2 Server information

6. With the help of your z/OS system administrator, specify the central data warehouse database configuration information as shown in Figure 3-40. Click Next.

Refer to 2.2.9, “Considerations about warehouse databases on z/OS” on page 54 for details.

Figure 3-40 z/OS central data warehouse database configuration


7. Make sure that the new central data warehouse database server is included in your list. The list for our case study environment is shown in Figure 3-41.

Figure 3-41 Central data warehouse server on z/OS

8. As displayed in Figure 3-42, the Summary window shows the Tivoli Data Warehouse components that will be installed and configured. Click Install or click Back to make changes.

Figure 3-42 Central data warehouse summary window


9. The completion window is displayed in Figure 3-43. It has a successful completion notice, or messages describing problems. Make sure the window does not list any warnings or errors and that the installation of the central data warehouse databases was successful. Click Finish.

Figure 3-43 Central data warehouse on z/OS install

10.You can verify your installation by issuing the commands listed in Example 3-2 on the z/OS system. These commands will confirm the creation of the central data warehouse database on the z/OS system.

Example 3-2 Verification of central data warehouse database on z/OS

select * from sysibm.sysdatabase;select * from sysibm.systablespace;select * from sysibm.sysstogroup;select * from sysibm.systables where dbname = 'TCDW1';

-- where TCDW1 is the new database name specified during the instalation

Creating the data mart on z/OSThe following steps should be performed on the TDW Control Center machine. The installation process will trigger scripts on the z/OS system that will create and configure the data mart database.


These are the steps for creating the data mart:

1. Insert the Tivoli Data Warehouse 1.2, installation CD into the CD-ROM drive of the control server. If the installation wizard does not start up, run the setup.exe program, which is located in the root directory of the CD.

2. The install process display the welcome panel. Click Next. It also provides the location of the common log files. Click Next.

3. The install proceeds by verifying the current Tivoli Data Warehouse environment and configuration. You will be prompted with two options, as shown in Figure 3-44. You can to create one or more central data warehouse databases, or you can create one or more data mart databases. In this case, select Add data marts and click Next.

Figure 3-44 Adding data marts

4. The installation wizard searches for the existing Tivoli Data Warehouse configuration and display a list of found data mart locations. It also allows you to add to the existing list of systems in which data mart databases will be created. Click Add on the install window.

5. With the help of your z/OS system administrator, specify the IBM DB2 configuration information on the z/OS system, as shown in Figure 3-45. Specify database type DB2 for z/OS and S/390, a fully qualified host name, port number, and a user ID with SYSADM authority. Click Next.


Figure 3-45 z/OS IBM DB2 Server information

6. With the help of your z/OS system administrator, specify the data mart database configuration information as shown in Figure 3-46. Click Next.

Refer to 2.2.9, “Considerations about warehouse databases on z/OS” on page 54 for details.

Figure 3-46 z/OS data mart database configuration


7. Make sure that the new data mart database server is included in your list. The list for our case study environment is shown in Figure 3-47.

Figure 3-47 Data mart server on z/OS

8. The Summary window, as displayed in Figure 3-48, shows the Tivoli Data Warehouse components that will be installed and configured. Click Install or click Back to make changes.

Figure 3-48 Data mart creation summary window


9. The completion window is displayed in Figure 3-49. It has a successful completion notice or messages describing problems. Make sure the window does not list any warnings or errors and that the installation of the data mart database was successful. Click Finish.

Figure 3-49 Data mart on z/OS install

10.You can verify your installation by issuing the commands listed in Example 3-3 on the z/OS system. These commands will confirm the creation of the data mart database on the z/OS system.

Example 3-3 Verification of data mart database on z/OS

select * from sysibm.sysdatabase;select * from sysibm.systablespace;select * from sysibm.sysstogroup;select * from sysibm.systables where dbname = 'TCMART1'

-- where TMART1 is the new database name specified during the instalation

3.4.3 Creating ODBC connections to the data mart databasesThe Crystal Enterprise server needs to have access to the information stored in the data mart databases for reporting. After completing the distributed installation of Tivoli Data Warehouse 1.2, you must set up ODBC connections from the Crystal Enterprise server machine to the data mart databases. You may further expand your environment by either creating additional data mart databases, or installing warehouse enablement packs that use different data marts.


In any case, there must be ODBC connections to these data marts defined on the Crystal Enterprise server.

Tivoli Data Warehouse 1.2 provides the twh_create_datasource script that sets up ODBC connections to the data mart databases. You may use this script or create the ODBC connections manually. In our case study scenario, we need to create ODBC connections to the TWH_MART data mart database located in the AIX machine TDW009 and to the TMART1 data mart database running on the z/OS system WTSC66. In order to create an ODBC connection to the TWH_MART database using the twh_create_datasources script, perform the following tasks:

1. On the Crystal Enterprise server machine, open an IBM DB2 command window: Start-> Programs-> IBM DB2-> Command Window.

2. Copy both the twh_create_datasource.bat and ODBCcfg.exe files from the disk1\tools directory of the Tivoli Data Warehouse 1.2 installation media to a temporary directory on the Crystal Enterprise server machine.

3. Run the twh_create_datasource script using the following syntax and shown in Example 3-1.

twh_create_datasource <DBtype> <ID> <odbcname> <DBname> <SRVname> <390LocalDBName> <port>

Where:

<DBtype> Can by set to DB2UDB or DB390 depending on the location of the database.

<ID> An unique identifier for the local node name. The script creates node names following the naming convention: TDWCS%ID%

<odbcname> The ODBC data source name

<DBname> The data mart database name

<SRVname> The IBM DB2 server in which the data mart database resides. This must be the fully qualified host name.

<390LocalDBName> For databases on z/OS only. Specifies the local database name.

<port> The port number to connect to the IBM DB2 server

Example 3-4 twh_create_datasource script

C:\Temp>twh_create_datasource.bat DB2UDB 09 TWH_MART TWH_MART tdw009.itsc.austin.ibm.com 50000

Creating DB2/UDB datasource TWH_MART


C:\Temp>db2cmd /w /c /i db2 catalog tcpip node TDWCS09 remote tdw009.itsc.austin.ibm.com server 50000DB20000I The CATALOG TCPIP NODE command completed successfully.DB21056W Directory changes may not be effective until the directory cache is refreshed.

C:\Temp>db2cmd /w /c /i db2 catalog database TWH_MART at node TDWCS09 authentication serverDB20000I The CATALOG DATABASE command completed successfully.DB21056W Directory changes may not be effective until the directory cache is refreshed.

C:\Temp>C:\Temp\ODBCcfg.exe DB2 TWH_MART TWH_MART


C:\Temp>C:\Temp>twh_create_datasource.bat DB2390 66 TMART1 TMART1 wtsc66oe.itso.ibm.com TMART1 33768

Creating DB2/390 datasource TMART1

C:\Temp>db2cmd /w /c /i db2 catalog tcpip node TDWCS66 remote wtsc66oe.itso.ibm.com server TMART1DB20000I The CATALOG TCPIP NODE command completed successfully.DB21056W Directory changes may not be effective until the directory cache isrefreshed.

C:\Temp>db2cmd /w /c /i db2 catalog database TMART1 at node TDWCS66 authentication dcsDB20000I The CATALOG DATABASE command completed successfully.DB21056W Directory changes may not be effective until the directory cache isrefreshed.

C:\Temp>db2cmd /w /c /i db2 catalog dcs database TMART1 as 33768 parms ',,INTERRUPT_ENABLED,,,,,'DB20000I The CATALOG DCS DATABASE command completed successfully.DB21056W Directory changes may not be effective until the directory cache isrefreshed.

C:\Temp>C:\Temp\ODBCcfg.exe DB2 TMART1 TMART1


C:\Temp


3.5 Installing warehouse agents The warehouse agent is the component of IBM DB2 Warehouse Manager that manages the flow of data between data sources and targets databases that are on different computers. By default, the TDW control server uses a local warehouse agent to manage the data flow between operational data sources, central data warehouse, and data mart databases.

You can optionally install the warehouse agent component of IBM DB2 Warehouse Manager on a computer other than the TDW control server. Typically, you place an agent on the computer that is the target of a data transfer. From the Tivoli Data Warehouse perspective, that computer will become a remote warehouse agent site or, simply an agent site, after registering and enabling the warehouse agent to run ETLs.

IBM DB2 Data Warehouse Center will then use the remote warehouse agent site machine to manage the transfer of Tivoli Data Warehouse data. This can speed up the data transfer as well as reduce the workload on the control server. The computer on which you install a warehouse agent is called an agent site.

In our case study installation scenario, we decided to have agent sites on both the central data warehouse and data mart servers running on distributed platforms, as shown in Figure 3-50.

Note: Make sure the warehouse agent site machines that will run the ETL1 can connect to the operational data source databases.

Make sure the warehouse agent site machines that will run the ETL2 can connect to the correspondent central data warehouse databases and data mart databases.


Figure 3-50 Distributed environment with agent

There are two steps to be performed in order to create remote warehouse agent sites:

1. Install IBM DB2 Warehouse Manager on every server that will become warehouse agent sites. In our case study scenario, IBM DB2 Warehouse Manager must be installed on the central data warehouse server (tdw004) and on the data mart server (tdw009).

2. On the machine to become the warehouse agent site, use the Tivoli Data Warehouse installation wizard to register and enable the warehouse agent to run ETLs.

After the warehouse agents have been registered with the control server, the following steps should be performed:

1. On the remote agents site: Catalog the databases that the remote agent is supposed to use.

TDW Control Server


TWH_MD



Tivoli Environment



TWH_CDW

Data Mart


Data MartTWH_MART

z/OS environment

Data SourcesCentral Data WarehouseData MartHostname: wtsc66oe

Data MartCentral DataWarehouse

Data Source

Data Source

Data Source

Agent Site

Agent Site

Operational data flow

Operational data flow


2. On the remote agents site: Make available the warehouse enablement pack files.

3. On the data warehouse control servers site: Configure the ETL processes to use the remote agent.

Chapter 5, “IBM Tivoli NetView Warehouse Enablement Pack” on page 161 provides details on the steps needed to enable the WEP to use the remote agent site, described above.

Warehouse agents are supported on Windows and UNIX systems only. If you are using IBM DB2 databases on a z/OS system, you must use the warehouse agent on another computer in your deployment.

3.5.1 Installing IBM DB2 Warehouse ManagerIn this section we provide the steps required for installing IBM DB2 Warehouse Manager on Windows. The installation process is described in the manual, Installing and Configuring Tivoli Data Warehouse, GC32-0744-02.

Installing IBM DB2 Warehouse Manager on Windows platform

1. Stop ALL IBM DB2 processes before proceeding with the installation. Open a IBM DB2 command window and issue the following commands:

db2stop forcedb2admin stop

Also, on the Windows Services panel, stop the following services:

– DB2-DB2CTLSV– DB2 JDBC Applet Server– DB2 License Server– DB2 Security Server– Warehouse server– Warehouse logger

2. Load the IBM DB2 Warehouse Manager installation media.

3. Select Start -> Run. Type in D:\setup.exe and click OK to start the installation. From the Installation window, select Install.

Attention: See the IBM DB2 Warehouse Manager installation media provided with Tivoli Data Warehouse 1.2. This ensures that you install the correct version and Fix Pack level.

Make sure you stop ALL IBM DB2 processes before proceeding with the installation.


4. The Select Products window opens. Make sure that DB2 Warehouse Manager is selected. Click Next.

5. The Select Installation Type window opens. Select Custom.

6. On the Select Components window, make sure that only Warehouse Agent and Documentation are selected, as shown in Figure 3-51. Click Next.

Figure 3-51 Select the DB2 Warehouse Manager components

7. At the Start Copying Files window, you can review the installation options. Click Next.

8. When the setup complete the installation process, click Finish.

9. Reboot the machine.

Installing IBM DB2 Warehouse Manager on AIX platform1. Stop ALL IBM DB2 processes before proceeding with the installation. Log in

as DB2 administrator (in our case the DB2 administrator is named db2inst1) and issue the following commands:

db2stop forcedb2admin stop

2. Log in as user with root authority.

3. Mount the IBM DB2 Warehouse Manager installation media.

4. Change to the directory where the installation media is mounted.

5. Enter the ./db2setup command.


6. Within the Install DB2 V7 Menu select DB2 Data Warehouse Agent as shown in Figure 3-52.

Figure 3-52 Install DB2 V7 menu on AIX

7. In the Configuration - DB2 Data Warehouse Agent Services menu, accept the default service names and port numbers.

8. In the Create DB2 Services menu, select: Do not create a DB2 Instance.andDo not create the Administration Server.as shown in Figure 3-53. Then click OK.


Figure 3-53 Create DB2 Service Menu on AIX

9. In the DB2 Setup Utility window your settings are summarized. Click Continue and the installation process starts.

10.After the installation process check the DB2 Setup Utility window for error messages. Click OK to finish the installation of the DB2 warehouse agent.

3.5.2 Creating the remote agent sitesPerform the following steps on each computer that will be a remote agent site. In order to create a remote agent site, IBM DB2 Warehouse Manager must have been installed on each computer. This procedure will register existing IBM DB2 warehouse agents on the TDW control server and will enable the warehouse agent to run ETL processes.

Note: Do not perform this step on the TDW control server. It should be performed on each computer that will become a remote agent site.


Creating agent sites on a Windows system and a UNIX systemTo create a remote agent site, perform the following steps:

1. Insert the Tivoli Data Warehouse 1.2 CD into your CD-ROM drive.

2. Start the Tivoli Data Warehouse installation wizard using the command for your operating system:

On Windows If the TDW installation wizard does not start automatically, run the setup.exe program, which is located in the root directory of the CD.

On UNIX run the setup_unix.sh program.

3. In the Welcome window, click Next.

4. The Tivoli Common Logging Directory window displays the name of the Tivoli common logging directory. Click Next.

5. Select Create a remote agent site. Change the installation directory (Optional); the default directory is %ProgramFiles%\TWH. We have changed it to C:\TWH as shown in Figure 3-54. Click Next.

Figure 3-54 Setup Window - create warehouse agents


6. A warning message is displayed to emphasize the need to fulfill all of the prerequisite tasks required for creating remote agent sites. This is shown in Figure 3-55.

Figure 3-55 Before proceeding with remote agent sites creation

7. Specify the existing IBM DB2 instance owner user ID and password to connect to the local IBM DB2 database. Click Next.

8. In the Connection to Remote control server window, specify the following information, as shown in Figure 3-56:

– The fully qualified host name of the TDW control server of the Tivoli Data Warehouse 1.2 environment.

– The port number for IBM DB2 on the TDW control server

– The IBM DB2 instance owner user ID and password to connect to IBM DB2 on the TDW control server

Then click Next.


Figure 3-56 Warehouse agents - specify the TDW control server

9. The Summary window indicates that you are creating an agent site. Click Install to start the installation.

10.In the Progress Window, review the progress of the program. When the program completes, the Installation Results windows, as shown in Figure 3-57, contains either a successful completion notice or messages describing problems. Make sure the window does not list any warnings or errors and then click Next.


Figure 3-57 Successful remote agent creation window

11.Click Finish to exit. You must restart the system before the agent site can be used by any warehouse packs.

3.6 Verification of the installationIn this section we describe some means to verify the installation of various Tivoli Data Warehouse products.

� Verify Tivoli Data Warehouse control server.

In order to verify the Tivoli Data Warehouse control server, issue the command twh_list_cs.bat. If the control server was installed successfully, this command gives you the host name where the control server was installed and the name of the control server database. However, the control server database must be on the same host as the control server.

Note: We assume a successful IBM DB2 Universal Database Enterprise Edition installation as base for the Tivoli Data Warehouse installation. Therefore we do not explain how to verify a IBM DB2 Universal Database Enterprise Edition installation.


For our case study installation Example 3-5 gives the output of the twh_list_cs.bat command. You will find host name and database name in Figure 3-50, which gives an overview of our case study scenario.

Example 3-5 Verify control server (twh_list_cs)

C:\TWH\tools\bin>twh_list_cs.batListing the control server information in the Tivoli Data Warehouse registry.

Control Server: Control Server Database Server Information: Host name: tdw003.itsc.austin.ibm.com Vendor: DB2 UDB Port: 50000 Database name: TWH_MD Control Server Database Client Information: Node name: Not applicable. Database alias: TWH_MD ODBC connection name: TWH_MD Tivoli Data Warehouse component version: 1.2.0.0

The Tivoli Data Warehouse is built on top of the IBM DB2 Universal Database Enterprise Edition Data Warehouse. To check whether the DB2 warehouse services are running enter Start -> Programs -> Administrative Tools -> Services and look for the services named Warehouse logger and Warehouse server. Both must be up and running. Figure 3-58 shows the services windows focusing on the DB2 data warehouse services.

Figure 3-58 DB2 Data Warehouse services

� Verify Tivoli Data Warehouse central data warehouse databases

Use the batch twh_list_cdws.bat to display information about the central data warehouse databases. Example 3-6 shows the output for the case study installation. Both central data warehouses are displayed. The second installed central data warehouse on Z/OS has TCDW1 as database alias and TWH_CDW1 as ODBC connection name assigned.


Example 3-6 Verify central data warehouse (twh_list_cdws)

C:\TWH\tools\bin>twh_list_cdws.batListing the central data warehouse information in the Tivoli Data Warehouse registry.

Central Data Warehouse: Central Data Warehouse Database Server Information: Host name: tdw004.itsc.austin.ibm.com Vendor: DB2 UDB Port: 50000 Database name: TWH_CDW Control Server Database Client Information: Node name: TDW1 Database alias: TWH_CDW ODBC connection name: TWH_CDW Tivoli Data Warehouse component version: 1.2.0.0 Enabled/Disabled: E

Central Data Warehouse: Central Data Warehouse Database Server Information: Host name: wtsc66oe.itso.ibm.com Vendor: DB2 390 Port: 33768 Database name: DB2E Control Server Database Client Information: Node name: TDW3 Database alias: TCDW1 ODBC connection name: TWH_CDW1 Tivoli Data Warehouse component version: 1.2.0.0 Enabled/Disabled: E

� Verify Tivoli Data Warehouse Data Mart databases.

To check on the data warehouse data mart databases use the twh_list_marts.bat command. Example 3-7 shows the output for our case study scenario. You notice the difference between DB2 UDB for OS/390 and z/OS databases and IBM DB2 Universal Database Enterprise Edition DB2 databases. However, you see no differences between Windows and UNIX based databases. In our case study the TWH_MART database resides on an AIX box (refer to Figure 3-50 on page 127).

Example 3-7 Verify data mart databases

C:\TWH\tools\bin>twh_list_marts.batListing the data mart information in the Tivoli Data Warehouse registry.

Data Mart: Data Mart Database Server Information:


Host name: tdw009.itsc.austin.ibm.com Vendor: DB2 UDB Port: 50000 Database name: TWH_MART Control Server Database Client Information: Node name: TDW2 Database alias: TWH_MART ODBC connection name: TWH_MART Tivoli Data Warehouse component version: 1.2.0.0 Enabled/Disabled: E

Data Mart: Data Mart Database Server Information: Host name: wtsc66oe.itso.ibm.com Vendor: DB2 390 Port: 33768 Database name: DB2E Control Server Database Client Information: Node name: TDW3 Database alias: TMART1 ODBC connection name: TWH_MART1 Tivoli Data Warehouse component version: 1.2.0.0 Enabled/Disabled: E

� Verify remote agents sites.

To verify the remote agent sites enter the command twh_list_agentsites.bat. Example 3-8 shows the output for our case study installation. The command does not return the remote agents for internal use!

Example 3-8 Verify remote agent site (twh_list_agentsites)

C:\TWH\tools\bin>twh_list_agentsites.batListing the agent site information in the Tivoli Data Warehouse registry.

Local Agent Site: Host name: tdw003.itsc.austin.ibm.com Version: 1.2.0.0 Enabled/Disabled: Enabled Warehouse Pack Usage:

Local Agent Site: Host name: tdw004.itsc.austin.ibm.com Version: 1.2.0.0 Enabled/Disabled: Enabled Warehouse Pack Usage:

Local Agent Site: Host name: tdw009.itsc.austin.ibm.com


Version: 1.2.0.0 Enabled/Disabled: Enabled Warehouse Pack Usage:

To view the data warehouse remote agent sites you may also select from the windows desktop Start -> Programs -> IBM DB2 -> Control Center to open the DB2 control center. From the DB2 Control Center select Tools -> Data Warehouse Center to open the Data Warehouse Center where you select Administration -> Agent Sites.

Figure 3-59 shows the result for the case study installation. All remote agents are listed in this view. However, in addition to the agents listed by the twh_list_agentsites in Figure 3-59, the agents for internal use are also listed.

Figure 3-59 Remote Agent Sites

� Verify the installation of the Crystal Enterprise Professional for Tivoli Server (twh_update_rptsrv)

To verify the installation of the Crystal Enterprise Professional for Tivoli Server and its registration with the Tivoli Data Warehouse control server enter the command twh_update_rptsvr -l from DOS-shell. Figure 3-9 shows the output of this command for the case study installation.

Example 3-9 Verify Crystal Enterprise Professional for Tivoli installation

C:\TWH\tools\bin>twh_update_rptsrv -lReport Server Host Name: tdw002.itsc.austin.ibm.comReport Server User ID: Administrator


� Verify users, sources, and targets.

Enter the command twh_update_userinfo -l to get an overview of the used user names and the available sources and targets.

Example 3-10 Verify data user

C:\TWH\tools\bin>twh_update_userinfo -l

tdw003.itsc.austin.ibm.com: 50000: TWH_MD: db2admin: ANM_TWH_MD_Target CDW_TWH_MD_Source

tdw004.itsc.austin.ibm.com: 50000: TWH_CDW: db2admin: AN1_TWH_CDW_Target ANM_TWH_CDW_Source ANM_TWH_CDW_Target CDW_TWH_CDW_Source CDW_TWH_CDW_Target

tdw009.itsc.austin.ibm.com: 50000: TWH_MART: db2inst1: ANM_TWH_MART_Target CDW_TWH_MART_Source CDW_TWH_MART_Target

wtsc66oe.itso.ibm.com: 33768: DB2E: tivo01: CDW_TWH_CDW1_Source CDW_TWH_CDW1_Target CDW_TWH_MART1_Source CDW_TWH_MART1_Target

CDWTD0002I The command that changes user IDs and passwords (twh_update_userinfo) completed.


3.6.1 Verifying the remote agent installRemote agents register with the central data warehouse control server. In order to verify the installation and thus the successful registration with the central data warehouse control server, perform the following steps:

1. Login to the computer running the central data warehouse control server.

2. Select Start -> IBM DB2 -> Control Center.

3. Within the DB2 Control Center window, select the Data Warehouse Center button from the toolbox.

4. Within the Data Warehouse Center window select Warehouse -> Administration -> Agent Sites in the browser tree. The all registered data warehouse agents are displayed. Figure 3-60 shows this view for our test environment. Here the agent sites list includes tdw004.itsc.austin.ibm.com and tdw009.itsc.austin.ibm.com, installations that were described previously in 3.5.1, “Installing IBM DB2 Warehouse Manager” on page 128 and 3.5.2, “Creating the remote agent sites” on page 131.

Figure 3-60 Verify Remote Agents on Tivoli Data Warehouse Control Center


3.7 Installing warehouse enablement packsA warehouse enablement pack is the part of a Tivoli software product that provides warehouse functionality. It can be provided on the installation media for the product, on a separate CD, or in a collection of warehouse enablement packs. When provided on the installation media for the product, a warehouse enablement pack is located in a subdirectory named tedw_apps_etl.

Part 2, “Case study scenarios” on page 159 provides instructions on how to install and configure a warehouse enablement pack in great detail. However, before installing any warehouse enablement pack, we recommend that you complete the following tasks:

1. Review the implementation guide manual for each warehouse enablement pack you plan to install. The implementation guide manual is located on the installation media of the warehouse enablement pack. For warehouse enablement packs designed for Tivoli Enterprise Data Warehouse 1.1, it is usually in the tedw_apps_etl/<productcode>/pkg/version/doc subdirectory, and for warehouse enablement packs designed for Tivoli Data Warehouse 1.2, it is typically in the <productcode>/doc subdirectory of the warehouse enablement pack installation media, where <productcode> is the AVA code of the product.

2. Define and create warehouse agent sites. See 3.5, “Installing warehouse agents” on page 126.

3. If the warehouse enablement pack provides an ETL1, which reads data from operation data sources, make sure the warehouse agent site that will run the ELT1 can connect to the operational data source databases.

4. Back up your TDW deployment to ensure that you can return to a known valid state if you encounter an unrecoverable error.

5. Check if any other ETL process steps for any other warehouse enablement packs that are already installed are scheduled to run during the warehouse enablement pack installation. Change the mode of the existing ETL steps to Test to prevent them from running.

6. Verify the WEP installation by issuing the following command on the Tivoli Data Warehouse control server:

twh_configwep -u db2admin -p <your password> -f list

Example 3-11 shows the output for the IBM Tivoli NetView Warehouse Enablement Pack.


Example 3-11 twh_configwep command output

C:\TWH\tools\bin>twh_configwep -u db2admin -p <your password> -f listCDWCW0002I The twh_config_wep.pl program started.

Installed warehouse enablement packs:CODE VERSION ROLE NAME---- ------------------ ---- ----------------------------------------------AN1 Version 1.1.0 V11 IBM Tivoli NetviewANM Version 1.1.0 V11 IBM Tivoli Netview---- ------------------ ---- ----------------------------------------------

Source datasources used by warehouse enablement packs:CODE DATASOURCE CLIENT_HOSTNAME---- ---------------- ------------------------------------------------------AN1 ANM_SOURCE localhostANM ANM_SOURCE localhost---- ---------------- ------------------------------------------------------

Central data warehouse datasources used by warehouse enablement packs:

CODE DATASOURCE CLIENT_HOSTNAME---- ---------------- ------------------------------------------------------AN1 TWH_CDW localhostANM TWH_CDW localhost---- ---------------- ------------------------------------------------------

Data mart datasources used by warehouse enablement packs:CODE DATASOURCE CLIENT_HOSTNAME---- ---------------- ------------------------------------------------------ANM TWH_MART localhost---- ---------------- ------------------------------------------------------

CDWCW0001I The twh_config_wep.pl program completed successfully.


Chapter 4. Performance maximization techniques

This chapter provides an overview, tips, and techniques you can use to improve the performance of your Tivoli Data Warehouse 1.2. We cover the following topics:

� “DB2 performance” on page 146� “Operating system performance tuning” on page 150� “Tivoli Data Warehouse performance” on page 155

The following sections provide performance tuning tips for small, medium, and large configurations. Because these may not be the best settings for your environment, a suggested approach would be to use these values as a starting point and then to monitor system performance, making changes where necessary.

4


4.1 DB2 performanceDepending on the size of your environment, there are many options available when tuning the performance of your IBM DB2 Universal Database Enterprise Edition servers.

The following topics introduce some of the factors affecting DB2 performance. This is based on the information provided by the redbook Database Performance Tuning on AIX, SG24-5511.

Buffer poolsA buffer pool is an area of memory. In this area, database pages of user table data, index data, and catalog data are temporarily moved from disk storage. DB2 agents read and modify data pages in the buffer pool. The buffer pool is a key influence of overall database performance, because data can be accessed much faster from memory than from a disk. If most of the data required by applications is present in the buffer pool, then less time is needed to access this data. This improves performance. Buffer pools can be defined with varying page sizes including 4K, 8K, 16K and 32K.

Prefetchers Prefetchers are present to retrieve data from disk and move it into the buffer pool before applications need the data. For example, applications needing to scan through large volumes of data would have to wait for data to be moved from disk into the buffer pool if there were no data prefetchers. Prefetchers are designed to improve the read performance of applications as well as utilities such as backup and restore, since they prefetch index and move data pages into the buffer pool, thereby reducing the time spent waiting for I/O to complete. The number of prefetchers may be controlled by the database configuration parameter NUM_IOSERVERS.

Page cleanersPage cleaners are present to make room in the buffer pool, before agents and prefetchers read pages from disk storage and move them into the buffer pool. For example, if an application has updated a large amount of data in a table, many of the updated data pages in the buffer pool may not yet have been written on to disk storage. Such pages are called dirty pages.

Since prefetchers cannot place fetched data pages onto the dirty pages in the buffer pool, these dirty pages must first be flushed to disk storage and become clean pages, so that prefetchers can find room to place fetched data pages from disk storage. The number of page cleaners may be controlled by the database configuration parameter NUM_IOCLEANERS.


The NUM_IOCLEANERS parameter allows you to specify the number of asynchronous page cleaners for a database. These page cleaners write changed pages from the buffer pool to disk before the space in the buffer pool is required by a database agent. As a result, database agents should not have to wait for changed pages to be written out so that they may use the space in the buffer pool. This improves overall performance of the database applications.

If you set the parameter to zero (0), no page cleaners are started and as a result, the database agents will perform all of the page writes from the buffer pool to disk. This parameter can have a significant performance impact on a database stored across many physical storage devices, since in this case there is a greater chance that one of the devices will be idle. If no page cleaners are configured, your applications may encounter periodic log full conditions.

LogsChanges to data pages in the buffer pool are logged. Agent processes updating a data record in the database update the associated page in the buffer pool, and write a log record into a log buffer. To optimize performance, neither the updated data pages in the buffer pool, nor the log records in the log buffer are written to disk immediately. They are asynchronously written to disk by page cleaners, and the logger, respectively. The logger and the buffer pool manager cooperate and ensure that an updated data page is not written to disk storage before its associated log record is written to the log. This ensures database recovery to a consistent state from the log in the event of a crash such as a power failure. A number parameters can be used here:

� logfilsiz: This parameter represents the size of the log files.

� logprimary: The primary log files establish a fixed amount of storage allocated to the recovery log files. This parameter allows you to specify the number of primary log files to be pre-allocated.

� logsecond: This parameter specifies the number of secondary log files that are created and used for recovery log files (only as needed).

� logbufsz: This parameter specifies the amount of database shared memory to use a buffer log before writing these records to disk.

Java heap parametersIn general, increasing the size of the Java heap improves throughput until the heap no longer resides in physical memory. After the heap begins swapping to disk, Java performance deteriorates drastically.

Chapter 4. Performance maximization techniques 147

Therefore, the maximum heap size needs to be low enough to contain the heap within physical memory. MON_HEAP_SZ indicates the amount of memory (in 4K pages) which is allocated for database monitor data (at db2start). The amount of memory needed will depend on the number of snapshot switches which are turned on and active Event Monitors. If the memory heap is insufficient, an error will be returned when trying to activate a monitor and it will be logged to the db2diag.log file.

Database heapThere is one database heap per database, and the database manager uses it on behalf of all applications connected to the database. It contains control block information for tables, indexes, table spaces, and buffer pools. It also contains space for event monitor buffers, the log buffer (logbufsz), and temporary memory used by utilities. Therefore, the size of the heap will be dependent on a large number of variables. The control block information is kept in the heap until all applications disconnect from the database.

Sort heap size (sortheap)If the rows to be sorted occupy more than the space available in the sort heap, several sort passes are performed, where each pass sorts a subset of the entire set of rows. Each sort pass is stored in a temporary table in the buffer pool, which might be written to disk. When all the sort passes are complete, these sorted subsets are merged into a single sorted set of rows. A sort is considered to be piped if it does not require a temporary table to store the final, sorted list of data. That is, the results of the sort can be read in a single, sequential access. Piped sorts result in better performance than non-piped sorts and will be used if possible.

I/O serversConfiguring enough I/O servers with the num_ioservers configuration parameter can greatly enhance the performance of queries for which prefetching of data can be used. To maximize the opportunity for parallel I/O, set num_ioservers to at least the number of physical disks in the database.

It is better to overestimate the number of I/O servers than to underestimate. If you specify extra I/O servers, these servers are not used, and their pages are paged out. As a result, performance does not suffer.

Next, we describe some further performance features that can affect the SQL performance with indexes.


RUNSTATSThis command updates statistics about the physical characteristics of a table and the associated indexes. The optimizer uses these statistics when determining access paths to the data.

This utility should be called when a table has had many updates, or after reorganizing a table.

This command should be run on all tables, and a typical usage would be:

runstats on table <table name> with distribution and detailed indexes all

During our testing, we developed a script which will perform runstats on every table, in every database, in the instance pointed to by the environment variable DB2INSTANCE.

REORGCHKThis command line utility calculates statistics on the database to determine if tables or indexes, or both, need to be reorganized or cleaned up. It can either use existing statistics or perform runstats to create up-to-date statistics.

REORGThis command reorganizes an index or a table. The index option reorganizes all indexes defined on a table by rebuilding the index data into un-fragmented, physically contiguous pages. The table option reorganizes a table by reconstructing the rows to eliminate fragmented data, and by compacting information. If you specify an index as part of the table reorganization, then the table will be physically ordered by that index. With Version 7 of DB2, both table and index reorganization cannot take place online.

Data distributionIf you have a system that has multiple physical disk drives available to DB2, you would probably want to split out the DB2 data across these multiple disks to reduce I/O contention as much as possible. In an ideal environment this could mean separate disks for:

� Database logs

� Mirrored database logs (if this option has been set up in the database configuration file)

� Temporary space (used for sorting data and storing intermediate result sets)

� Table data

� Index data


Due to cost and resource constraints, this ideal scenario may not be completely possible. However, you should try to maximize the utilization of the available disks in your system. As an example, if you only have two physical drives, keep the logs on one drive and create the database and the physical data on the other.

Tablespace typeOnce we have created a DB2 system and chosen our disk distribution, then we can decide how we want to store the data. DB2 has two types of tablespaces to store data, System Managed Space (SMS) and Database Managed Space (DMS).

SMS is the default tablespace type, and three tablespaces of this type are created after the installation of DB2. These tablespaces allow the operating system to allocate and manage the space where the table data resides.

DMS tablespaces give DB2 the ability to control storage space. The amount of space allocated to a DMS tablespace must exist upon creation, since these files are allocated upon creation, as opposed to SMS tablespaces, which grow within the specified file system.

Both types of tablespaces have their own advantages and disadvantages, but in general SMS is easier to manage, whereas DMS can be faster. DMS is also more flexible. The regular data, large data, and indexes can be split between different DMS tablespaces. DMS can be created on raw unformatted disks.

Recent enhancements to DMS tablespaces mean that it is now possible to add containers to a tablespace, remove containers, and reduce the size of containers.

The Create Tablespace Wizard in the Control Center guides you through creating a tablespace.

4.2 Operating system performance tuningAny application running in your environment will be subject to the performance and optimization settings of your operating system. It is important for these settings to be tuned before deployment to the Tivoli Data Warehouse and be monitored on a regular basis, possibly using IBM Tivoli Monitoring.

4.2.1 Windows environmentsIf you are using a Windows NT or Windows 2000 server for any Tivoli Data Warehouse components, you can improve performance by optimizing your system traffic. Perform the following steps to make these changes:


1. Navigate through the path Start -> Settings -> Control Panel -> Network and Dial-up Connections -> Local Area Connection.

2. Click Properties on the Local Area Connection window, and select File and Printer Sharing for Microsoft Networks.

3. Select the option Maximize data throughput for networking applications.

4.2.2 Primary Windows performance factorsWe begin our discussion on system performance by taking into consideration the primary Windows performance factors. As with any system, performance begins by laying a good foundation of well balanced hardware resources that can be exploited by the operating system, and eventually, application specific software such as DB2 UDB.

In this section we focus on two primary Windows performance factors: system hardware and system software. Specifically, we cover system hardware, memory, processor, and storage.

� System hardware:

The performance goals of system hardware selection and configuration are to produce a well balanced system that can sustain high rates of overall system utilization. To achieve a well balanced system, we must adequately size individual hardware resources so that no single resource results in a system bottleneck. In this section we look at primary Windows performance factors from a system hardware perspective.

You should take into consideration how you plan to scale your system, when planning for additional capacity requirements. Scaling-out is achieved by adding more resources, such as memory, processors, and storage across multiple systems.

In general, scaling-out is much easier in terms of hardware planning, as you scale-out by simply adding additional hardware resources (processors, memory, etc.) by adding additional servers. A scale-up approach typically requires a little more planning, as you may reach hardware system limitations in terms of the amount of resources that can be added to the server.

� Memory:

Since accessing data in memory is faster than accessing data on hard drives, the primary factor in terms of memory is quantity. Although memory speeds are also factors, this is seldom an option when configuring a system, unless you are willing to select and configure another system altogether. The amount of physical memory is a critical system hardware resource that can have a huge impact on overall performance. In general, the cost of memory on commodity servers is usually an insignificant factor when compared to the cost of other hardware resources.


Today most systems based on the 32-bit Intel® Architecture (IA-32) support the Physical Address Extensions (PAE) capabilities of the IA-32. PAE provides operating system software with an instruction set to address physical memory above 4 GB. Operating systems that take advantage of the PAE can address up to 64 GB physical memory.

Given the memory extensions of the IA-32, the primary factor in memory selection will most likely not be the cost of the memory itself, but rather the incremental cost moving from one edition of the Windows operating system to the next in order to address more physical memory.

� Processor:

Most systems are limited by the total number of central processing units (CPU) they can support. Typically a 4-way system cannot be upgraded to an 8-way system unless it is indeed a true 8-way that was populated with only 4 processors. There are a few systems on the market today, such as the IBM x440 and the Unisys ES7000, that can be expanded beyond the total number of original processors by adding additional processor expansion modules.

Besides quantity and speed, another important consideration in terms of processor selection is the size of the internal L2 cache. Slower processors with larger internal caches have shown significant throughput advantages for database applications over faster processors with smaller internal caches. Another factor to consider when selecting the number of processors is the operating system software costs. There are incremental licensing costs associated with each Windows 2000 or Windows Server 2003 edition to support more processors.

� Storage:

The disk subsystem has been an area of much debate over the last several years. Most disk subsystems will implement some form of redundancy that has always favored recoverability over performance. In recent years improvements in technology have been able to overcome many of the performance limitations imposed by implementing redundant disk arrays.

Performance characteristics of disk controllers include speed, throughput, channels, and cache. Care should be taken in the placement of disk controllers in the system.

Although most disk adapters are backwards compatible, you want to match the disk controller speed with that of the systems PCI bus. You should avoid placing faster 64-bit 66 Mhz disk controllers in slower 32-bit 33 Mhz PCI slots. You should also consider the number of disk controllers in your system as well. Attaching a single disk controller with several I/O channels might be capable of driving your subsystem, but can quickly saturate a single PCI bus, not to mention introduce a single point of failure into your system. If possible, you should also avoid placing disk controllers on PCI buses populated with other I/O intensive resources.


Performance characteristics of disk subsystems include disk speed, size, cache, and the number of physical disks in the subsystem. You should favor a subsystem with a large number of small drives over a small number of large drives. If this is impractical, plan for growth by choosing a large number of large drives. Best performance will be achieved for database applications with a large number of physical disks (5-10) per processor.

For example, a large 32-way SMP server should be attached to a disk subsystem with a minimum of 160 physical disks, not including parity disks or hot spares. With an average 18-GB disk you would have almost 3 TB of total storage. At first this might seem impractical for a 1 TB database, but you need to consider space requirements for loading files and storing the most recent backup image before copying to tape.

Hardware implementations of disk arrays are now commonplace on Intel based servers. Modern disk controllers support RAID levels 0, 1, 5, and 10, sometimes referred to as 0+1. As with most performance decisions, there is always a give (cost) and take (performance) associated with choosing which RAID level to implement.

Operating system softwareIn this section we look at primary Windows performance factors from a system software perspective. We explore the Windows 2000 and Windows Server 2003 operating system offerings.

Windows 2000 serversThe Windows 2000 family of servers consists of Windows 2000 Server, Windows 2000 Advanced Server, and Windows 2000 Datacenter Server. All Windows 2000 server products build upon the already rich features of Windows NT 4.0 Server and Windows NT 4.0 Server Enterprise Edition.

Windows 2000 Server EditionWindows 2000 Server is the entry level edition of the operating system. It provides the same basic support functionality of its predecessor Windows NT 4.0 Server.

The Windows 2000 Server edition is licensed to support 32-bit Intel based server from one (1) to four (4) processors and can address up to 4 GB of physical memory. Although limited to 4 GB of addressable physical memory, Windows 2000 server as well as all other versions of the operating system support the Physical Address Extensions (PAE) provided by the IA32 architecture.

The Microsoft specific implementation provides a set of APIs called Address Windowing Extensions (AWE). These APIs allows 32-bit applications to address real physical memory above the 4 GB line using a windowing approach that we discuss in detail later in this chapter.


Windows 2000 Advanced Server EditionWindows 2000 Advanced Server provides the same basic server functionality of its predecessor, Windows NT 4.0 Server Enterprise Edition. It includes all of the existing features of the base Windows 2000 Server, is licensed to support 32-bit Intel servers with one (1) to eight (8) processors. It can address up to 8 GB of physical memory using the Physical Address Extensions (PAE) provided by the IA32 architecture.

Windows 2000 Advanced Server also includes 4 GB Tuning support. 4 GB Tuning, introduced by Microsoft in Windows NT 4.0 Enterprise Edition Service Pack 3, allows the operating system to make an additional 1 GB of memory available to applications.

Windows 2000 Advanced Server also include the Microsoft Clustering Service (MSCS). This optionally installed service provides failover clustering support for up to two nodes in the cluster, as did Windows NT 4.0 Server Enterprise Edition.

Windows 2000 Datacenter Server EditionWindows 2000 Datacenter Server includes all of the features of Windows 2000 Advanced Server. This edition of the Windows 2000 operating system can be licensed to support 32-bit Intel servers with 1-8, 1-16 or 1-32 processors. It can address up to 64 GB of physical memory using the Physical Address Extension provided by the IA32 architecture. Windows 2000 Datacenter Server extends the MSCS support to include up to 4 nodes in a single cluster.

Windows 2000 Datacenter Server also includes a feature called Winsock Direct. It enables unmodified Windows Sockets (Winsock) applications that use TCP/IP to exploit the performance benefits of system area networks (SANs). Winsock Direct enables efficient high-bandwidth, low-latency messaging that conserves processor time for application use. High-bandwidth and low-latency inter-process communication (IPC) and network system I/O allow more users on the system and provide faster response times and higher transaction rates.

Windows 2000 Server or Windows 2000 Advanced Server cannot be upgraded to Windows 2000 Datacenter Server. This is a rather new and unique approach that Microsoft has taken with Windows 2000 Datacenter Server, as the operating system cannot be purchased directly from Microsoft, but instead must be acquired as a complete solution including hardware, software, and services, from a Windows 2000 Datacenter Server OEM provider.

Windows Server 2003Windows Server 2003 is the follow-on to the Windows 2000 family of operating systems. There are three editions of Windows Server 2003.


For entry level systems, Windows Server 2003 Standard Edition will provide support for 32-bit Intel servers with up to 2 CPUs and 4 GB memory. Windows Server 2003 Enterprise Edition will provide support for both 32-bit and 64-bit Intel servers with up to 8 CPUs. The 32-bit version will support 32 GB of memory and the 64-bit version will support up to 64 GB of memory.

The Microsoft Cluster Service will be included and provide support for up to 8 nodes in a single cluster. Windows Server 2003 Datacenter Edition will provide support for both 32-bit and 64-bit Intel servers with up to 32 CPUs. The 32-bit version will support 64 GB of memory and the 64-bit version will support up to 128 GB of memory. The Microsoft Cluster Service will be included and provide support for up to 8 nodes in a single cluster.

4.2.3 AIX environmentsTo enable large file support, ensure that the Soft FILE size value is set to -1 for the IBM DB2 Universal Database Enterprise Edition administrative user. This parameter defines the largest soft file size, in 512-byte blocks, that this user can create or extend when invoking a process.

Perform the following steps to make this change: From the SMIT menu, navigate through the path Security & Users -> Users -> Change/Show Characteristics of a User. Type in the name of the IBM DB2 administrative user in the entry field. Scroll down to the Soft FILE size line and change the value listed to -1.

4.3 Tivoli Data Warehouse performanceThe overall performance of the Tivoli Data Warehouse will be very much dependent on how well the hardware, operating system, and DB2 environment has been configured. A number of Tivoli Data Warehouse components which can be tuned are discussed next.

Distributed installThe Tivoli Data Warehouse components can be, but do not need to be, installed on the same systems as other Tivoli software or on the systems where the operational data stores reside.

The operational data stores are on a system that is not part of the Tivoli Data Warehouse deployment. As an example, the TEC database will have operational data.


Crystal Enterprise should reside on a system that does not have other Tivoli Data Warehouse or Tivoli software products on it. Users access Crystal Enterprise reports using a Web browser from any system

Other types of data analysis tools, should be located on systems outside your Tivoli Data Warehouse deployment.

Control server and warehouse agentThe DB2 warehouse agent is the component of IBM DB2 Warehouse Manager that manages the flow of data between data sources and targets that are on different computers. By default, the control server uses a local warehouse agent to manage the data flow between operational data sources, central data warehouse databases, and data mart databases.

In some environments, this configuration is sufficient. Optionally, you can create agent sites on other Windows and UNIX systems in your environment. The control server can use the agents on these computers to manage data flow. In a distributed deployment, you can improve the performance of Tivoli Data Warehouse by creating an agent site on the computer that is the target of the data transfer for a central data warehouse or data mart ETL routine.

That computer becomes a remote agent site, which the Data Warehouse Center uses to manage the transfer of Tivoli Data Warehouse data. This can speed up the data transfer as well as reduce the workload on the control server.

The control server is the system that contains the control database for Tivoli Data Warehouse and is the system from which you manage your data. The control database contains metadata for both Tivoli Data Warehouse and for the warehouse management functions of IBM DB2 Universal Database Enterprise Edition. You can have only one control server in a Tivoli Data Warehouse deployment.

On Windows and UNIX systems, using a warehouse agent on the computer that contains the central data warehouse database or a data mart database can potentially improve performance.

ETL routinesThe scheduling of data warehouse ETLs should be done during off-peak hours to avoid impacting the performance of your operational data stores. For distributed environments across geographic locations, you may consider putting central data warehouse databases at each location, as each location may have different off-peak hours.


For example, if your operational data is on systems in the United Kingdom and the United States, you might put a central data warehouse database on a system in each location. This enables you to schedule the central data warehouse ETL for each system at an appropriate off-peak time.

The time taken for ETLs to complete depends on many factors, including the amount of data they have to process, the speed of the database in which the source and target data reside, the performance of the network, and so forth.

Ensure that the default scheduling interval is changed to an appropriate interval for your environment and data level.

The ETL processes that update tables in the central data warehouse should not all be scheduled at the same time. There might be unknown dependencies in the data, and updates to the same tables might cause performance problems, depending on your environment.

Data Analysis programs can read directly from central data warehouse databases without using data marts; but this use is not supported. Analyzing historical data directly from the central data warehouse database can cause performance problems for all applications using the central data warehouse.

Tivoli Decision Support and Tivoli Data WarehouseYou can continue to use Tivoli Decision Support at the same time and on the same systems as Tivoli Data Warehouse. For the best performance, do not schedule Tivoli Data Warehouse ETLs and Tivoli Decision Support cube builds concurrently, if they access the same databases.

You can continue to use Tivoli Decision Support for OS/390 at the same time and on the same systems as Tivoli Data Warehouse. For the best performance, schedule Tivoli Data Warehouse ETLs after Tivoli Decision Support for OS/390 data collection has occurred.

Sample tuningIn a recent deployment of Tivoli Data Warehouse, approximately 85% of the current execution time of the ETL processes was consumed by the triggers and corresponding sequences.

With optimal tuning (without changes of SQL-statements), performance was increased by 82%, and with changes to SQL statements, performance can be increased by 90%.


Each environment is different, and these performance improvements cannot be guaranteed, but our recommendations are as follows:

� Distribute the database to the three available hard disks (the test server in this scenario had three hard disks).

� Use DMS tablespaces instead of SMS tablespaces.

� Increase caches of the sequences (approximately 5000 may be enough, but this has to be tested).

� Execute runstats after loading data to the staging tables.

� Distribute the temporary tablespace also to the three available hard disks.


Part 2 Case study scenarios

Part 2


Chapter 5. IBM Tivoli NetView Warehouse Enablement Pack

In this chapter we look at what is required to provide network availability reporting in an IBM Tivoli Netview 7.1.4 and Tivoli Data Warehouse 1.2 environment using the IBM Tivoli NetView Warehouse Enablement Pack (WEP). We cover the following topics:

� “Case study overview” on page 162� “IBM Tivoli NetView WEP overview” on page 163� “Prerequisites” on page 165� “Preparing NetView for data collection” on page 167� “Installation of the NetView WEPs” on page 181� “Testing, scheduling, and promoting the ETLs” on page 191� “Running NetView ETLs on remote agent sites” on page 198� “Reporting” on page 206

5


5.1 Case study overviewTable 5-1 gives a brief summary of the distributed Tivoli Data Warehouse environment we started with to install the IBM Tivoli NetView Warehouse Enablement Pack.

Table 5-1 Environment for NetView integration

Figure 5-1 summarizes the distributed Tivoli Data Warehouse environment used in this chapter.

Host name Component Database Operating system

TDW003 TDW Control Server 1.2

DB2 Server 7.2FP10a

W2000 FP4

TDW004 TDW Central Warehouse 1.2(remote agent site)

DB2 Server 7.2FP10a

W2000 FP4

TDW009 TDW Data Mart 1.2 DB2 Server 7.2FP10a

AIX 5.2

TDW002 Crystal Enterprise Server 9

DB2 Client 7.2FP10a

W2000 FP4

TDW008 NetView 7.1.4 - W2000 FP4


Figure 5-1 Distributed deployment scenario

5.2 IBM Tivoli NetView WEP overviewThe IBM Tivoli Netview warehouse enablement pack enables Tivoli Data Warehouse reporting of network availability and performance information. The latter is provided by the IBM Tivoli Netview snmpcol daemon. Network node, SmartSet, layer 2, and performance information is recorded to an availability database by IBM Tivoli Netview (hereafter referred to as NetView), which is used as input to Tivoli Data Warehouse.

Tivoli NetView


TDW Control Server


TWH_MD





TWH_CDW

Data Mart


Data MartTWH_MART

Agent site

Note: Layer 2 information is only collected when the IBM Tivoli Switch Analyzer product is installed.

Chapter 5. IBM Tivoli NetView Warehouse Enablement Pack 163

The IBM Tivoli NetView Warehouse Enablement Pack code is provided with the IBM Tivoli Netview 7.1.4 software.

In Figure 5-2 the data flow of an integration of NetView in a Tivoli Data Warehouse environment is illustrated. We start with a brief description of the processes and their control.

Node availability information is stored by the NetView process tdwdaemon into the NetView source database. The NetView snmpcol daemon writes performance information gathered by SNMP polling (CPU load, number of processes, etc.) into the NetView source database. The data upload to the NetView source database is controlled by the NetView server that is illustrated in Figure 5-2 by broken lines. The data flow within the Tivoli Data Warehouse is controlled by the Tivoli Data Warehouse control server. The generation and publishing of the NetView specific reports is controlled by the Crystal Enterprise server.

Figure 5-2 IBM Tivoli NetView Warehouse Enablement Pack data flow

Web Reports

NetViewServer

NetView SourceDatabase


Data Mart

Web Server

DatawarehouseControl Server

CrystalServer

IP NetworkIP Network

DataControl

ETL1

ETL2

snmpcol

tdwdaemon

Web Reports

NetViewServer

NetView SourceDatabase


Data Mart

Web Server

DatawarehouseControl ServerDatawarehouseControl Server

CrystalServerCrystalServer

IP NetworkIP Network

DataControlDataControl

ETL1

ETL2

snmpcol

tdwdaemon


5.3 PrerequisitesThe stated prerequisites, as per the IBM Tivoli NetView Warehouse Enablement Pack Implementation Guide - Version 1.1.0, SC32-1237-00, which can be found in the \tedw_apps_etl\anm\pkg\v110\doc directory of the enablement pack software, are:

� IBM Tivoli NetView Version 7.1.4 � IBM DB2 Universal Database Enterprise Edition Version 7.2� IBM DB2 Universal Database Enterprise Edition Version 7.2 Fix Pack 6� Tivoli Enterprise Data Warehouse required e-fixes to IBM DB2 UDE v7

Fix Pack 6 (1.1-TDW-0002)� Tivoli Enterprise Data Warehouse Version 1.1� Tivoli Enterprise Data Warehouse 1.1 Fix Pack 2 (1.1-TDW-FP02)

In this case study scenario chapter we will be using Tivoli Data Warehouse 1.2 in a previously built distributed environment, as described in Chapter 3, “Getting Tivoli Data Warehouse 1.2 up and running” on page 71.

Here is a list of the products used and their releases in the case study scenario:

� IBM Tivoli NetView 7.1.4� IBM Tivoli Netview 7.1.4 early fix PJ29597� IBM DB2 Universal Database Enterprise Edition Version 7.2 Fix Pack 10a

(shipped with Tivoli Data Warehouse 1.2)� IBM Tivoli NetView Warehouse Enablement Pack version 1.1.0 � Tivoli Data Warehouse Version 1.2

5.3.1 Verifying prerequisitesFrom the stated prerequisites, we determine what action (if any) is required on our NetView Server platform, in our case study environment. These actions are described in Table 5-2.

Table 5-2 NetView WEP Prerequisite Check - NetView server platform

Prerequisite Comment Action required

IBM Tivoli Netview 7.1.4 Install IBM Tivoli Netview 7.1.4

IBM DB2 Universal Database Enterprise Edition Version 7.2 Fix Pack6

We will be using a DB2 UDB 7.2 Fix Pack 10a level on a separate server from the NetView server.

Install the DB2 Client version 7.2 fix pack 10a on the NetView server machine. We used the installation media shipped with Tivoli Data Warehouse 1.2


5.3.2 Gathering installation informationTable 5-3 provides the information to be used during the installation of the NetView warehouse enablement pack.

Table 5-3 Netview Enablement Pack installation information

Information required

Description Default value Used value

NetView availability DB name

Name of the database that will be created during installation to store the NetView availability data.

NETVIEW NETVIEW

DB2 user ID A DB2 user ID with create authority.

db2admin db2inst1

DB2 user ID password

The DB2 user ID password

As required

Data purge days The number of days the availability data will be retained before being purged.

90 Leave as default

SmartSet membership retrieval time

The time of day to load SmartSet information.

23 Leave as default

List of SmartSets The name of the SmartSets, for which availability data will be collected.Note: The Router SmartSet is required.

Routers Change to All in order to produce more data.

Warehouse pack install directory

The directory name where the enablement pack will be installed.

\usr\ov\dwpack Leave as default

NetView availability DB name

Name of the database that will be created during installation to store the NetView availability data.

NETVIEW NETVIEW

DB2 user ID A DB2 user ID with create authority.

db2admin db2inst1

DB2 database password

The DB2 password.


5.4 Preparing NetView for data collectionNetView uses a DB2 database as source database for its availability and performance data. In this section we discuss how to set up NetView to fill the source database with data.

Before performing any configuration on the NetView server, we need to perform the following steps:

� Ensure that the DB2 Server to host the Netview source database is at proper level. The minimum required by the NetView warehouse enablement pack is IBM DB2 Universal Database Enterprise Edition Version 7.2 Fix Pack 6.

� Install and configure the DB2 Client on the NetView Server machine in case the NetView source database will be placed in a separate machine. Ensure the DB2 client version and level is compatible with the DB2 server.

5.4.1 Enabling NetView to export data for Tivoli Data WarehouseIn this section we describe how to enable NetView to fill out its source database for use with the Tivoli Data Warehouse 1.2. NetView includes an easy-to-use administration tool that:

� Creates and populates the netView source database

� Configures the ODBC connection to the DB2 source database

� Modifies the SNMP Collection daemon to store performance data on the source database

� Registers and starts the tdwdaemon which stores network availability data to the source database

Perform following steps configure enable NetView to collect data:

1. Log in to your NetView server with administrator authority.

2. Start the installer for data export from NetView to the DB2 source database by selecting Start -> Programs -> Tivoli NetView -> Administration -> Configure data Export to DB2 for use in Tivoli Enterprise Data Warehouse from the windows desktop.

3. A configuration menu, as shown in Figure 5-3 opens. Fill in the appropriate data as collected in 5.3.2, “Gathering installation information” on page 166. Figure 5-3 shows this menu filled with data according to our case study scenario.

Note: We recommend to use the IBM DB2 version and level shipped with Tivoli Data Warehouse 1.2.


4. Select OK to proceed. In our case study installation, we used a remote DB2 server residing on AIX host tdw009.

Figure 5-3 NetView Configure data export to DB2 - Parameters

5. Select Create Database in the popup shown in Figure 5-4.The NETVIEW source database and its tables are created.

Figure 5-4 NetView Configure data export to DB2 - create database

6. Select Yes in the popup window shown in Figure 5-5 to register and start the Data Warehouse daemon (tdwdaemon).


Figure 5-5 NetView Configure data export to DB2 - register and start tdwdaemon

During the configuration a number of logs may be written, depending on whether any problems are encountered during installation. The log files we discovered were:

� \usr\ov\dwpack\DWP_Install_stdout.log� \usr\ov\log\TDWError_mmddhh.log� \usr\ov\log\tdwdaemon.log

Check these logs for unusual messages.

Verifying db updatesWe verified that data was actually being written to the availability database by issuing the following command:

db2 select count(*) from netview.netview_nodes

The greater than zero count returned (as shown in Example 5-1) indicated that data was in fact being written.

Example 5-1 Verify NetView source database updates

C:\DB2\SQLLIB\BIN>db2 connect to NETVIEW user db2inst1 using <DB password>

Database Connection Information

Database server = DB2/6000 7.2.8 SQL authorization ID = DB2INST1 Local database alias = NETVIEW

C:\DB2\SQLLIB\BIN>db2 select count(*) from netview.netview_nodes

1----------- 349

1 record(s) selected.


If the count had been zero, we could have tried the following sequence of actions:

1. Stop the tdwdaemon daemon:

ovstop tdwdaemon

2. Stop the netmon daemon:

ovstop netmon

3. Start the tdwdaemon daemon:

ovstart tdwdaemon

4. Start the netmon daemon:

ovstart netmon

5. Verify that both daemons are now active:

ovstatus tdwdaemonovstatus netmon

6. Preload the current NetView status to the availability db:

netmonaction.bat 500

7. Verify that data is being written by checking the count on the netview.netview_nodes table again:


5.4.2 NetView SmartSets configuration Some of the reports make use of NetView SmartSets. In this section we give a short introduction to the SmartSets concept and show how to create and configure them.

SmartSets are containers that are populated with NetView objects. Dynamic or static filters define which objects are mapped to a SmartSet. You can filter by the objects attributes.

A fresh NetView installation brings already a number of predefined SmartSets like CriticalNodes, which contain all network nodes that are unavailable, or like Routers, which contain all router nodes.


Figure 5-6 shows the SmartSet desktop of NetView. Some of the network nodes are not available. Therefore, the CriticalNodes SmartSet is displayed as red.

Figure 5-6 NetView SmartSet desktop

We now explain how to create new NetView SmartSets. However, you may need to vary the steps to meet your requirements.


From the NetView desktop, select Submap -> New Smartset ... from the toolbar to open the SmartSets configuration window shown in Figure 5-7. There, select the folder Advanced.

Figure 5-7 Microsoft SmartSet Advanced attributes

In the Combine Find Condition field you may insert conditions which meet your needs. We insert the phrase (refer to Figure 5-7):

(“SNMP sysObjectID = “1.3.6.1.4.1.311.1.1.3.1.2”)

Thus the SmartSet is populated by all network nodes, whose SNMP enterprise ID is equal to the specified value. In this case all Microsoft Windows 2000 servers with SNMP option populate this SmartSet.


Table 5-4 shows the configuration for all three SmartSets we created for our case study. The SmartSet IBM contains all Nodes running on an IBM operating system. (However, in our case study, NetView found only AIX hosts.) The SmartSet TDW contains all nodes whose host names start with the letters tdw.

All private enterprise object IDs are listed at the following Web site:

http://www.iana.org/assignments/enterprise-numbers

Table 5-4 Case Study SmartSets attributes

After filling the Combine Find Condition field shown in Figure 5-7, select Create SmartSet ... . A popup appears as shown in Figure 5-8. Insert the name of your SmartSet and optionally a descriptive text. We used the name Microsoft for the SmartSet in our case study. Then click OK to actually create the SmartSet.

Figure 5-8 Create Microsoft SmartSet

SmartSet Name Combine Find Condition field

Microsoft (“SNMP sysObjectID = “1.3.6.1.4.1.311.1.1.3.1.2”)

IBM “SNMP sysObjectID ~ “1.3.6.1.4.1.2.*”)

TDW “SNMP sysObjectID ~ “^tdw.*”)


http://www.iana.org/assignments/enterprise-numbers

If you open the folder SmartSets in the SmartSet configuration window again, the newly created SmartSets are now listed as shown in Figure 5-9. The highlighted line shows the new Microsoft SmartSet.

Figure 5-9 NetView SmartSets - Attributes


The new SmartSets are also displayed on the NetView SmartSet desktop as shown in Figure 5-10.

Figure 5-10 NetView SmartSets - Overview

Double-click the Microsoft SmartSet icon to open the container as shown in Figure 5-11. NetView polls the enterprise object ID as specified in Table 5-4 on page 173 via SNMP. Therefore, only SNMP managed nodes populate the Microsoft SmartSet. Windows 2000 server without SNMP option or with different SNMP community names are not included.


Figure 5-11 SmartSet Microsoft contents

5.4.3 Configuring NetView Data Warehouse daemon (tdwdaemon)The NetView tdwdaemon fills the NetView source database with availability data from the network nodes. It also controls the expiration of data within the NetView data warehouse source database. The behavior of the tdwdaemon can be changed by editing its configuration file, which is located by default under the name \usr\ov\conf\tdwdaemon.properties.

Most of the parameters are set by the configuration process explained in the previous 5.4.1, “Enabling NetView to export data for Tivoli Data Warehouse” on page 167 and seldom need to be changed.

The configuration parameters defined in the tdwdaemon.properties file are:

� SMARTSETSHere you can specify a comma-separated list of the NetView SmartSets you want to report. Here are some notes on this issue:

– The SmartSet names are separated by commas and no white characters.

– SmartSet names are case sensitive.


– The default value is Routers.

– The Routers SmartSet is required. If you change the SMARTSETS settings, make sure that Routers is included in your list.

– If you specify All, availability data of all network nodes will be stored in the source database.

– In our case study installation, we have created new NetView SmartSets called TDW, IBM and Microsoft. We selected our self-made SmartSets along with the required SmartSet Routers as shown in Example 5-2.

� SMARTSET_LOAD_TIMEGives the hour when the NetView SmartSet population is loaded to the NetView data warehouse source database. The default value 23 means, the data is loaded every day at 11 pm.

� OUTAGE_STORAGE_TIMEThis parameter specifies the number of days before availability data expires in the NetView data warehouse source database. The default value is 90 days.

� DBPASSWORDThe encrypted db2 password

� DB2USERThe db2 user ID (db2 administrator or any user ID with create authority).

� DBNAMEThe name of the NetView source database. Default is NETVIEW.

Note: It is recommended to schedule the ETL1 ANM_c05_ETL1_Process within the Tivoli data warehouse at least 1 hour after the SmartSet load time

Note: The SmartSet data is loaded once a day. Therefore the NetView source database contains a snapshot of this particular point in time. SmartSets which population changes rapidly like for instance CriticalNodes may not be suitable for reporting purposes

Note: To change the DB2 password, execute the updateDBPassword.bat command in the /usr/ov/bin directory. An example of this follows:

updateDBPassword.bat c:\usr\ov\conf\tdwdaemon.properties newpwd


� PortDB2 database IP-port of the NetView source database. Default is 50000.

� HOSTNAMEHostname of the server which hosts the NetView source database.

� NODENAMEDB2 database node of the NetView source database. Default is TDWNODE.

Example 5-2 NetView tdwdaemon configuration file tdwdaemon.properties

#Thu Mar 18 14:36:46 CST 2004PORT=50000HOSTNAME=tdw009.itsc.austin.ibm.comDBUSER=db2inst1SMARTSET_LOAD_TIME=23NODENAME=TDWNOTESMARTSETS=Routers,TDW,Microsoft,IBMDBPASSWORD=c3dqNDNyOUTAGE_STORAGE_TIME=90DBNAME=netview

After changing the tdwdaemon.properties file, you have to restart the tdwdaemon to apply the changes. Use the NetView commands ovstop and ovstart for this purpose. Example 5-3 shows the command shell dialog.

Example 5-3 Restart the NetView data warehouse daemon tdwdaemon

C:\usr\ov\bin>ovstop tdwdaemonDone

C:\usr\ov\bin>ovstart tdwdaemonDone

5.4.4 Verifying NetView data collection enablementIn order to verify that the data collection is taking place, perform these tasks:

� Check the tdwdaemon.� Check the snmpcollect daemon.� Check the existence of data in the NetView source database.

Verifying NetView Data Warehouse daemon (tdwdaemon)The NetView daemon named tdwdaemon is used to perform availability data recording. We verify that this daemon started successfully by using the following command:

ovstatus tdwdaemon


Example 5-4 shows the command shell output executing this command.

Example 5-4 Status of NetView data warehouse daemon (tdwdaemon)

C:\usr\ov\bin>ovstatus tdwdaemon object manager name: tdwdaemon behavior: OVs_WELL_BEHAVED state: RUNNING PID: 2092 last message: Initialization complete. exit status: -

Done

The log file for the tdwdaemon is named tdwdaemon.log and can be found in the \usr\ov\log directory. Verify that no apparent errors were being reported on the tdwdaemon.log file. You will also find DB2 errors in this log file that the tdwdaemon has encountered while communicating with the NetView data warehouse source database.

Verifying NetView SNMP collector daemon (snmpcollect)Performance data is stored to the NetView data warehouse source database via the NetView snmp collector daemon named snmpcollect. We verify that this daemon started successfully by using the following command:

ovstatus snmpcollect

Example 5-5 shows the command shell output executing this command.

Example 5-5 Status of the NetView SNMP collector daemon (snmpcollect)

C:\usr\ov\bin>ovstatus snmpcollect object manager name: snmpcollect behavior: OVs_WELL_BEHAVED state: RUNNING PID: 1616 last message: Initialization complete. exit status: -

Done

Note: If no tdwdaemon.log exists for tdwdaemon to write to, it will create a new one. Deleting it prior to restarting the tdwdaemon makes it easier to review, because all the old entries are removed.


The log file for the tdwdaemon is named snmpCol.trace and can be found in the \usr\ov\log directory. Verify that no apparent errors were being reported on the snmpCol.trace log file in the directory \usr\ov\log. You will also find DB2 errors in this log file that the snmpcollect daemon has encountered while communicating with the NetView data warehouse source database.

Verifying data import to the Netview source databaseTo verify whether data is imported to the NetView source database, you can use the DB2 control center to view the contents of the source databases tables. However, we give an example using the DB2 command line. Log in to the Netview database using the command:

db2 connect to NETVIEW user db2inst1 using <password>

Where <password> is your password for the db2inst1 database user.

Here is a list of the items to check and the commands to use to check them:

� Availability data:


� Performance data:

db2 select count (*) from netview.snmpcollection

� NetView Smartsets:

db2 select count (*) from netview.smartsets

In all three cases the count must be greater than 0. Example 5-6 shows the results of these commands for our test study environment.

Example 5-6 Check the NetView source database

C:\DB2\SQLLIB\BIN>db2 connect to NETVIEW user db2inst1 using <password>


Database server = DB2/6000 7.2.8 SQL authorization ID = DB2INST1 Local database alias = NETVIEWC:\DB2\SQLLIB\BIN>db2 select count(*) from netview.netview_nodes

1-----------

Note: You may have to wait until the SmartSet data upload has taken place. The time for this upload is specified in the configuration file tdwdaemon.properties.


349


C:\DB2\SQLLIB\BIN>db2 select count(*) from netview.snmpcollection

1----------- 40787


C:\DB2\SQLLIB\BIN>db2 select count(*) from netview.smartsets

1----------- 5


C:\DB2\SQLLIB\BIN>

5.5 Installation of the NetView WEPsAs described in 5.2, “IBM Tivoli NetView WEP overview” on page 163, NetView can collect availability and performance information into the NetView data source database. In order to gather this data into the Tivoli Data Warehouse 1.2 environment, the following two warehouse enablement packs must be installed:

� NetView availability warehouse enablement pack (ANM) � NetView SNMP performance warehouse enablement pack (AN1)

In this section we explain how to install and configure both of them. Furthermore, we run the SNMP performance WEP on a data warehouse remote agent site.

5.5.1 Backing up the TDW environmentWe strongly advise that you back up your data warehouse environment prior to any installation activity.

In order to back up the Tivoli Data Warehouse 1.2 environment, stop the DB2 Warehouse logger service (vwlogger) and back up the control server database (TWH_MD), all central data warehouse databases, and all data mart databases.


For example, in order to back up the TWH_MD, TWH_CDW, and TWH_MART databases, perform the following commands:

db2 backup database TWH_MD to C:\\backupdb2 backup database TWH_CDW to C:\\backupdb2 backup database TWH_MART to C:\\backup

5.5.2 Establishing ODBC connectionsIn this section we show how to set up database connections to the NetView source database and how to configure the data warehouse sources and targets to make use of the ODBC data sources.

Creating a NETVIEW ODBC data sourceBy default, the data warehouse agent on the data warehouse control server is used to execute the NetView ETLs. So you have to create the ODBC data source on the data warehouse control server. However, if you plan to execute the ETLs or parts of them on a remote data warehouse agent, you must also create an ODBC data source for the NetView source database on the remote agent site.

In order to create a NetView ODBC data source for the NetView source database Log in to the data warehouse control server or the remote agent site server with administrator authority and select Start -> Control Panel-> Administrative Tools -> Data Sources (ODBC).

Then Select System DSN to view ODBC data source administration window as shown on the left side in Figure 5-12. Select Add... to open the Create New Data Source window as shown on the right side in Figure 5-12. Here you select the IBM DB2 ODBC DRIVER and proceed with selecting Finish.

Note: In difference to the IBM Tivoli NetView Warehouse Enablement Pack manual, we have not used ANM_SOURCE but NETVIEW as ODBC Data Source name. AIX DB2 instances can only use a maximum of 8 characters in names for database aliases. ANM_SOURCE has 10 characters and therefore cannot be used on AIX DB2 instances.


Figure 5-12 Create an ODBC data source for NETVIEW

Figure 5-13 shows the ODBC IBM DB2 Driver - Add window. You can add an optional description. If your NETVIEW source database is local or already cataloged within the DB2 client, you need to select NETVIEW from the Database alias dialog.

If you use a remote database, select Add.... This will open the Add Database Wizard window.

Figure 5-13 Add an ODBC data source for NETVIEW

Here are the different steps we have to perform in the Add Database Wizard (these steps are also shown in Figure 5-14):

1. Source We selected Manually configure connection to database.

2. ProtocolWe selected TCP/IP.


3. TCP/IPWe inserted data as shown in Table 5-5.

Table 5-5 Add database wizard - register TCP/IP

4. DatabaseWe inserted NETVIEW as the database name. Select Finish to start the registration.

Figure 5-14 Configure NetView Source database connectivity

Selection inserted values remark

Host name tdw009.itsc.austin.ibm.com Hostname of the database server

Port number 50000 default DB2 service port

Service name - optional left blank

We wanted to maually configure our database connection

The protocol standard used in our environment is TCP/IP

The hostname of the DB2 UDB Server i.e. where our NetView availability data is kept, along with the default DB2 port number.


5.5.3 Installing NetView Enablement Pack SoftwareIn this section we describe how we installed the NetView availability warehouse enablement pack (ANM) in our case study scenario environment. The installation of the NetView SNMP performance warehouse enablement pack (AN1) can be performed in a similar way.

Warehouse enablement pack installations must be performed on the control server machine.

In order to install the NetView availability warehouse enablement pack (ANM), perform the following steps:

1. While logged on to the control server as a user ID with administrator authority, select Start -> Programs -> Tivoli Data Warehouse -> Install a Warehouse Pack. The InstallShield wizard is started.

2. On the welcome window, select Next to continue. In the next window, the path to your Common Logging Directory is displayed. In the case study installation, the path is C:\Program Files\ibm\tivoli\common.

3. The InstallShield wizard checks the Tivoli data warehouse environment. This might take a few minutes. The next window allows you to select the warehouse packs to install. This list is empty, as shown in Figure 5-15.

Figure 5-15 NetView WEP installation - List of WEPs to install


4. Select Add and insert the location of the NetView warehouse pack installations properties file. This file is named twh_install_props.cfg. The properties file for NetView availability WEP with code ANM can be found on the installation media under directory \tedw_apps_etl\anm\pkg\, as shown in Figure 5-16. The properties file for NetView SNMP performance WEP with code AN1 can be found on the installation media under directory \snmp_etl\an1\pkg\.

Figure 5-16 NetView WEP installation - Properties file

5. Select OK and/or Next to get back to List of Warehouse Packs to install. In contrast to Figure 5-15, the list is now populated with the NetView WEP as shown in Figure 5-17.


Figure 5-17 NetView WEP installation - List of WEPs to install NetView

6. Select Next to proceed with the installation. The installation takes a few minutes. Figure 5-18 shows the window which is displayed after successful installation.

Figure 5-18 NetView WEP installation - successful installation

To install both NetView WEPs, you have to perform the installation steps twice using the different properties files of the two WEPs.


5.5.4 Defining the authority to the warehouse sources and targetsThe IBM Tivoli NetView Warehouse Enablement Pack adds NetView specific data warehouse sources and targets to the Tivoli Data Warehouse environment. After the initial installation of the NetView WEPs, these sources and targets need to be configured. (Refer to the IBM Tivoli NetView Warehouse Pack Implementation Guide, SG32-1237-00). The installation manual suggests using ANM_SOURCE as the name for the data source. In our case study we preferred NETVIEW for this purpose, as we explain in 5.5.2, “Establishing ODBC connections” on page 182.

The IBM Tivoli NetView Warehouse Enablement Pack includes two WEPs: the availability WEP with code ANM and the SNMP performance WEP with code AN1. The latter does not include ETL2, data mart structure, or reports. Therefore neither data mart target nor central data warehouse source exists for this WEP.

Table 5-6 shows an overview of all NetView based sources and targets. In the last column, the appropriate user IDs for our case study installation are shown. The databases TWH_MART and NETVIEW are running on the AIX platform and their default user is db2inst1. All other databases are running on Windows2000 platform with DB2 default user db2admin.

Table 5-6 NetView sources and targets

Name in data warehouse Description Database Used by User

ANM_AVAIL_Source ODBC data source for the NetView source database

NETVIEW ETL1 db2inst1

ANM_TWH_CDW_Source ODBC data source for the central data warehouse

TWH_CDW ETL2 db2admin

ANM_TWH_CDW_Target ODBC data target for the central data warehouse


ANM_TWH_MART_Target ODBC data target for the data mart

TWH_MART ETL2 db2admin

ANM_TWH_MD_Target ODBC data target control server database

TWH_MD ETL2 db2admin

AN1_SNMP_Source ODBC data source for the NetView source database

NETVIEW ETL1 db2inst1

AN1_TWH_CDW_Target ODBC data source for the Tivoli NetView warehouse source database



To configure the Tivoli data warehouse sources and targets for Netview, perform the following steps:

1. Open the DB2 control center by selecting Start -> Programs -> IBM DB2 -> Control Center.

2. From the DB2 control center, open the data warehouse center by selecting Tools-> Data Warehouse Center from the toolbar.

3. In the data warehouse logon window, type the user ID of the data warehouse administrator (default to db2admin) and the appropriate password. Select Advanced to ensure that the control database is set to TWH_MD as shown in Figure 5-19.

Figure 5-19 Data Warehouse Control Center - check control database

If you have opened the data warehouse control center, you will see a browser tree as shown in Figure 5-20. There are among others the leaves Warehouse Sources and Warehouse targets. In this section we first discuss the configuration of the data warehouse authorities for the data warehouse sources and then the data warehouse targets for use with the NetView WEPs.

Authority for data warehouse sources1. Open the Warehouse sources tree and select Properties in the context

menu of a source. In Figure 5-20 we selected ANM_AVAIL_Source for example.

2. In the Properties window of the selected data warehouse source, select Data Source, also shown for ANM_AVAIL_Source in Figure 5-20.


3. As Data source name, select the specific data source for your environment. For our case study installation the data sources for the NetView data warehouse sources are listed in Table 5-6 on page 188. Therefore we inserted NETVIEW as data source for our ANM_AVAIL_Source example in Figure 5-20.

4. Insert the User ID and appropriate password to the NetView source database. In our case study, we used a DB2 database on AIX for which the default user ID is db2inst1, as shown in Figure 5-20.

5. Select OK to finish the data warehouse source configuration.

Repeat these steps for all NetView data warehouse sources as listed in Table 5-6.

Figure 5-20 Configure NetView data warehouse sources

Authority for data warehouse targetsFor the NetView data warehouse target configuration, open the data warehouse control center. Starting with the control center, perform the following steps:

1. Open the Warehouse targets tree and select Properties in the context menu of a target. In Figure 5-21, for example, we choose ANM_TWH_CDW_Target.

2. In the properties window of the selected data warehouse target select Database, as also shown for ANM_TWH_CDW_Target in Figure 5-21.


3. As Database name, select the specific database for your environment. For our case study installation, the databases for the NetView data warehouse targets are listed in Table 5-6 on page 188. TWH_CDW was already configured as needed, so we can leave it as is for our case study (refer to Figure 5-21).

4. Insert the User ID and appropriate password to the target database. Because in our case study the TWH_CDW database is on a Windows machine, we used db2admin, as shown in Figure 5-21.

5. Select OK to finish the data warehouse source configuration.

Repeat these steps for all NetView data warehouse targets as listed in Table 5-6 on page 188.

Figure 5-21 Configure NetView data warehouse targets

5.6 Testing, scheduling, and promoting the ETLsAfter successfully installing and configuring the Netview warehouse enablement packs, all the ETL1 and ETL2 processes can now be tested, scheduled, and promoted to production.


There are three known modes for ETL processes in Tivoli Data Warehouse:

� Development: In this mode, process steps can be changed and their schedule can be configured. However, they do not run on their scheduled times and they cannot be tested.

� Test: In this mode, process steps are not scheduled, but they can be tested and their schedule can be changed.

� Production: In this mode, the processes run as scheduled. Neither the process steps nor their schedules can be changed.

5.6.1 Promoting the ETLs to TEST modeIn order to test ETLs we must promote them to test or production mode first. For the case study environment, we choose the test mode.

On the control server machine, open the Data Warehouse Control Center and select Subject Areas -> ANM_IBM_Tivoli_Netview_v1.1.0_Subject_Area -> Processes -> ANM_c05_ETL1_Process to open the list with the process steps. Right-click the all selected ANM_c05_ETL1_Process processes and select Mode -> Test in its context menu, as shown in Figure 5-22.

Figure 5-22 Promote ETLs to test mode

You have to promote all process steps you want to test into Test mode.


5.6.2 Testing the ETLsYou can test the ETLs manually first before scheduling and promoting to production. After you have promoted the process steps to Test mode, right-click the ANM_c05_s010_extractNodeInfo process and select Test, as shown in Figure 5-23. The process step is executed immediately.

Figure 5-23 Test ETL process steps

To view the results, select Warehouse -> Work in progress from the Data Warehouse Control Center. The Work in Progress window opens displaying a line for each executed process step as shown in Figure 5-24. You can right-click the process and select Show Log from the context menu to open the log window. There you can see additional information regarding the process step execution. In case of failure, that’s where you will find the error messages.


Figure 5-24 Work in Progress - Log file

ETLs: data collection verificationOnce all processes are tested, you need to verify data collection on the Tivoli Data Warehouse databases: central data warehouse and data mart.

There are several ways to accomplish this task. Here we show one of them:

1. On the control server machine, open the DB2 Control Center:Click Start -> Programs -> DB2 -> Control Center.

2. Double-click the desired database, for example, the TWH_MART database.

3. Double-click Tables; on the right hand pane, all tables are displayed. In the case of the TWH_MART database, select the D_L3NODES table. Right-click and select Sample Contents.

You will see a window as shown in Figure 5-25, displaying the sample content. This verifies that your ETLs executions are successful.

Important: Do not change the sequence of the processing steps. You might cause great damage to the data within the Tivoli Data Warehouse databases.


Figure 5-25 Sample contents

5.6.3 Scheduling the ETLsThere are two processes that need to be scheduled for the Netview WEP:

� ANM_c05_ETL1_Process� AN1_c05_SNMP_ETL1_Process

The following steps are similar for both processes, and we will use ANM_c05_ETL1_Process as an example.

On the Tivoli Enterprise Data Warehouse Control Center server using expand Subject Areas, select ANM_IBM_Tivoli_Netview_v1.1.0_Subject_Area -> Processes and right-click AMN_c05_ETL1_Process. Choose Schedule, as shown in Figure 5-26.

Note: Do not schedule the ANM_m05_ETL2_Process to run. This process is automatically started by the ANM_c05_ETL1_Process.


Figure 5-26 Schedule ANM_c05_ETL1_Process

Selecting Schedule will open up a dialog as shown Figure 5-27. Here you have to define the date and time, for this process to run.

If you use NetView SmartSets as described in 5.4, “Preparing NetView for data collection” on page 167, you have to synchronize the hour when the SmartSet data is loaded to the NetView source database specified in the tdwdaemon.properties file and the schedule of the NetView WEP ETLs.

We recommend that the ETLs be scheduled at least one hour later then the SmartSets loading time. In our case study installation, the SmartSet data load is done at 11pm as shown in Example 5-2 on page 178; one hour later at 0am, the ANM_c05_ETL1_Process is scheduled as shown in Figure 5-27.

Note: Changes apply only when the process is in development mode.


Figure 5-27 Schedule configuration for ANM_C05_ETL1_Process

5.6.4 Promoting the ETLs to Production statusAll of the processes are composed by components that have Development or Test status set as default. In order for them to run, their status needs to be changed from Development to Production status. They are:

� ANM_c05_ETL1_Process:

– ANM_c05_s010_extractNodeInfo– ANM_c05_s020_transformNodeInfo– ANM_c05_s030_loadNodeInfo– ANM_m05_s010_metric (This step is a link and cannot be promoted to

Test, Development or Production)

� ANM_m05_ETL2_Process:

– ANM_m05_s010_metric– ANM_m05_s020_fact– ANM_m05_s030_outage_rollUp– ANM_m05_s040_transition_rollup– ANM_m05_s050_ss_trans_rollup– ANM_m05_s060_out_rollup– ANM_m05_s070_total


� AN1_c05_SNMP_ETL1_Process:

– AN1_c05_s010_extractsnmpdata– AN1_c05_s020_transformsnmpdata– AN1_c05_s030_loadsnmpdata

The following steps must be performed for all processes described above. Here we use ANM_c05_ETL1_Process to describe it.

On the control server, using Data Warehouse Center tool, select the above processes and right-click them. Choose Mode -> Production, as shown in Figure 5-28.

Figure 5-28 Promote ANM_c05_ETL1_Process

After promoting the processes to production mode, they are scheduled for the configured times and they are visible in the Work in progress list.

5.7 Running NetView ETLs on remote agent sitesBy default, the NetView ETLs are scheduled on the default data warehouse agent named Default DWC AgentSite running on the control server. In Chapter 3, “Getting Tivoli Data Warehouse 1.2 up and running” on page 71, we explain how to install a data warehouse remote agent and create agent sites.


In this section we explain how to make use of such a remote agent site. For our case study installation, we move the execution of the NetView SNMP performance warehouse enablement pack (AN1) ETLs from the control server to a remote agent site.

Here is an overview of the steps required to move the execution of the ETLs to a remote agent site:

1. Make the ETLs code available on the remote agent site.

2. Make all data warehouse data sources and targets, which are used by the ETLs, available at the remote agent site.

3. Assign the remote agent the necessary data warehouse sources and targets.

4. Demote the moving ETL processes to mode development.

5. Configure the ETL processes to use the remote agent.

6. Promote the moved ETL processes to mode production.

7. Verify ETLs running on remote agent.

Making the ETL code available on the remote agent siteThe installation of the NetView SNMP performance WEP creates a new directory with path %InstallDir%\TWH\apps\an1. (The NetView availability WEP creates the directory %InstallDir%\TWH\apps\anm). Under this directory are all files stored used by the NetView ETLs.

However, the installation of the NetView SNMP performance WEP does not install this directory on remote agent sites. We also found no installation mechanism to install the warehouse enablement pack on the remote agent sites.

Therefore we mapped the warehouse folder on the data warehouse control server (tdw003.itsc.austin.ibm.com) to the remote agent site on the central data warehouse server (tdw004.itsc.austin.ibm.com) and copied the an1\ subdirectory to the %InstallDir%\TWH\apps\ directory on the remote agent site.

Making data sources and targets available for remote agentOn the agent site all necessary databases must be available or in DB2 terms cataloged. Table 5-6 on page 188 shows that the AN1 ETLs need access to the databases TDH_CDW and NETVIEW. In our case study installation the remote agent is on the central data warehouse server where the TWH_CDW ODBC database source is available already. We created an ODBC data source named NETVIEW on the remote agent site machine to the NETVIEW database.


Assigning remote agent sources and targetsTable 5-6 on page 188 shows that the AN1 ETLs make use of both:

� The source AN1_SNMP_Source� The target AN1_TWH_CDW_Target

On the control server machine, from the Data Warehouse Center, we assign both objects to the remote agent site, which in our case study is named tdw004.itsc.austin.ibm.com. Select Administration -> Agent Sites -> <remote agent name>, as shown in Figure 5-29.

Figure 5-29 Select remote agents properties

In the properties window of the remote agent site, available sources and targets are displayed on the left pane. Assigned sources and targets are displayed on the right pane. Move the proper warehouse sources and targets related to the warehouse enablement pack in question. In our case, shown in Figure 5-30, we choose only the SNMP performance ETL relevant AN1_SNMP_Source and AN1_TWH_CDW_Target to assign to the remote agent site.

Click OK to finish the assignment.


Figure 5-30 Change remote agents properties - sources and targets

Demoting the moving ETL processes to Development modeThe ETLs are still configured to run on the default agent on the control server. In order to change the processes configuration, they must be demoted to the Development mode.

On the control server machine, open the Data Warehouse Center and demote the related processes. in our case study scenario, select Subject Areas -> AN1_IBM_Tivoli_NetviewSNMP_v1.1.0_Subject_Area -> Processes -> AN1_c05_SNMP_ETL1_Process. On the right pane of the window, a list with process steps and their data warehouse sources and targets is displayed as shown in Figure 5-31.


Figure 5-31 Select ETL process properties

Right-click all AN1 processes in the list:

� AN1_c05_s010_extractsnmpdata� AN1_c05_s020_transformsnmpdata� AN1_c05_s030_loadsnmpdata

Then select Mode -> Development from the processes context menu as shown in Figure 5-32.

Figure 5-32 Demote ETL processes to development mode


Configuring the ETL processes to use the remote agentAfter demoting the ETL processes to Development, their properties can be changed. To configure the processes to run on a different warehouse agent site, right-click a process and select Properties from the context menu to open the processes Properties window as shown for the AN1_c05_s010_extractsnmpdata process in Figure 5-33. In the Agent Site drop-down menu, all agent sites are available that have the required data warehouse sources and targets assigned. In our case study installation, we see only the default agent and the tdw004.itsc.austin.ibm.com remote agent. We select the latter one, then select OK to finish the configuration.

Perform this step for each process in the list:


Figure 5-33 Change the ETL processes agent site

Promoting the moving ETL processes to Production modeAfter applying changes to the process configuration, the processes must be promoted to Production mode in order to schedule them again.

As explained in 5.6.4, “Promoting the ETLs to Production status” on page 197, right-click a process and select Mode -> Production from the processes context menu as shown in Figure 5-32. Perform this step for each process in the list:


The AN1_c05_s010_extractsnmpdata is scheduled now.


Verifying ETLs running on remote agentTo verify the ETLs running on the remote agent, we run the AN1 ETL at once from the data warehouse Work in Progress window.

On the control server machine, open the Data Warehouse Center and select Warehouse -> Work in Progress. The Work in Progress window opens as shown in Figure 5-34. In this presentation, scheduled processes are marked with a clock in their icon (as the highlighted AN1_c05_s010_extractsnmpdata process), failed processes with a red cross, and successful processes with a green check mark.

Right-click the scheduled process AN1_c05_s010_extractsnmpdata to open its context menu. There, select Run now as shown in Figure 5-34.

Figure 5-34 Work in Progress - Run ETL

The AN1_c05_s010_extractsnmpdata process is executed on the remote agent now and its subprocesses are started there automatically. If all AN1 processes execute successfully, the Work in Progress window looks like that shown in Figure 5-35.

You find new lines for each AN1 subprocess, AN1_c05_s010_extractsnmpdata, AN1_c05_s020_transformsnmpdata, and AN1_c05_s030_loadsnmpdata, which are all check marked after their successful completion. Additionally, the process AN1_c05_s010_extractsnmpdata is scheduled again marked with a clock. This line is highlighted in Figure 5-35.


Figure 5-35 Work in progress - Check ETL

Double-click one of the lines in Figure 5-35 to open the related Log Details window in Figure 5-36 for the ANM_c05_s010_extractNodeInfo process step.

Figure 5-36 Log Details menu


In our example, the process is successful and no error messages are displayed in the detailed view. With the buttons Previous and Next you can navigate the log details for all process steps that are displayed in the Log window shown in Figure 5-35.

5.8 ReportingIn this section we show how to set up, configure, and use some of the reports provided by the NetView availability warehouse enablement pack (ANM).

Here is a list of the predefined reports provided by the IBM Tivoli NetView Warehouse Enablement Pack:

� Daily Status Summary By SmartSet� Nodes With Longest Outage Time In Routers SmartSet� Nodes With Most Status Changes In Routers SmartSet� Nodes With The Longest Outage Times� Nodes With The Most Daily Status Changes� Summary Of Daily Network Status� Summary Of Total Outage Time By SmartSet� Summary Of Total Status Changes By SmartSet� Total Daily Status Changes In Monitored Network� Total Daily Status Changes In Routers SmartSet

Crystal Enterprise Professional version 9 for Tivoli has in comparison to a full Crystal license reduced configuration options. If the reports shipped with IBM Tivoli NetView Warehouse Enablement Pack do not match your needs and you want to develop additional reports, you have to upgrade your Crystal Enterprise installation.

5.8.1 Accessing the Crystal ePortfolio featureAll reports provided by Tivoli Data Warehouse 1.2 must be accessed using the Crystal Enterprise ePortfolio feature. The ePortfolio can be accessed from the Crystal Enterprise launchpad. To access the Crystal Enterprise launchpad, start an Internet browser and open the following URL:

http://<hostname>/crystal/enterprise9

Note: As described in Chapter 3, “Getting Tivoli Data Warehouse 1.2 up and running” on page 71, an ODBC connection to the data mart database needs to be defined on the Crystal Enterprise server before we can work with the reports. Please refer to that chapter for details.


Here, <hostname> represents the hostname of the Crystal Enterprise report server, as shown in Figure 5-37.

Figure 5-37 Crystal Enterprise - Launchpad

In this section, we concentrate on viewing NetView reports, and we do not explain the configuration of Crystal Enterprise to its full extent. For details on configuration and administration tasks, refer to the following manuals shipped with the product:

� Crystal Enterprise 9 Installation Guide � Crystal Enterprise 9 Administrator’s Guide � Crystal Enterprise 9 Getting Started Guide � Crystal Enterprise 9 ePortfolio User’s Guide


From the Crystal Enterprise Launchpad, proceed by selecting the ePortfolio link, which will bring you to the window shown in Figure 5-38. In the top bar, you can see that we are authorized as user guest. By default, the guest user has no access to the NetView reports, as indicated by the words No Folders on the left side of the window.

Figure 5-38 Crystal Enterprise 9 - ePortfolio

The installation process of the first warehouse enablement pack on the Tivoli Data Warehouse environment creates a user ID on the Crystal Enterprise environment named Tivoli. This user ID is to be used to access the reports provided by any IBM Tivoli software.


To log in as the Tivoli user ID, select the Log On button in the upper right corner of the ePortfolio window in Figure 5-38 on page 208. The Log On window, as shown in Figure 5-39, is presented. The Tivoli user ID has no password by default. We use the Enterprise authentication method as we have specified during the Crystal Enterprise installation.

Figure 5-39 Crystal Enterprise 9 - Log in


After entering the required data, select Log On to proceed. Now we are back at the ePortfolio window in Figure 5-40, but now with user Tivoli authority. Instead of No Folders in the guest users ePortfolio window in Figure 5-38, there is now a link visible with the name IBM Tivoli NetView in the Tivoli user ePortfolio window shown in Figure 5-40.

Figure 5-40 Crystal Enterprise 9 - Folders


We follow this link by selecting IBM Tivoli NetView and proceed to the IBM Tivoli NetView reports as shown in Figure 5-41. All reports provided by the IBM Tivoli NetView Warehouse Enablement Pack are listed there. As already mentioned, there are only reports on availability and no reports on performance.

Figure 5-41 Crystal Enterprise 9 - Tivoli Reports: IBM Tivoli NetView

We open the reports context menu by left-clicking on the desired report name, as shown in Figure 5-42. We are presented with a menu which contains the items:

� View: Generate report instantaneously.

� View latest instance: View last report.

� Schedule: Change or create a new schedule for report generation.

� History: View already generated reports.


Figure 5-42 Crystal Enterprise 9 - Daily Status Summary by SmartSet

We continue by selecting Schedule from the Daily Status Summary by SmartSet report, for example. The Schedule window, as shown in Figure 5-43, is opened.

Figure 5-43 Crystal Enterprise 9 - Schedule


The Customize your options toolbar offers three selection buttons:

� Schedule: Pressing this button starts a new schedule with the current options and parameters.

� Cancel: Pressing this button closes the Schedule window and you get back to the reports window without adding a new schedule for the report.

� Help: Opens the help window.

Figure 5-44 shows the selection of parameters for the schedule option. Here you can select the frequency the reports should be generated.

Figure 5-44 Crystal Enterprise 9 - Parameters for Schedule Option

We want to schedule the report to run now.

Next, it is necessary to provide the required parameters of the report. From the Customize your options pull-down menu, select Parameters, as shown in Figure 5-45. We left the other options settings to the default values.


Figure 5-45 Crystal Enterprise 9 - Schedule Parameters

The Schedule windows changes to Figure 5-46 and we are presented with three selection fields:

� Time Filter� General Time Frame� Specific Time Frame

Note: Schedule requirements may differ for each report. The schedule selections presented here are for the Daily Status Summary by SmartSet report.


Figure 5-46 Crystal Enterprise 9 - Schedule Parameter Selection

For the items Time Filter and General Time Frame we select the default values None by pressing the Add button at each selection.

Thus we specify a lower and an upper bound for the specific time frame by selecting the Start of range and End of range parameters. Select Add Range to accept the settings. Figure 5-47 shows the parameters window after the selections for our case study report.


Figure 5-47 Crystal Enterprise 9 - Parameters: Specific Time Frame

Now all required parameters are specified. Start the report generation by pressing the Schedule button from the toolbar.


As we have left the Schedule parameter set to Now, as shown in Figure 5-43, the report is scheduled immediately, and the reports history window is opened as shown in Figure 5-48.

Figure 5-48 Crystal Enterprise 9 - Report History

The report just scheduled is still running, and therefore it is in status Pending.

Figure 5-48 shows four different status, as follows:

� Pending: Report generation is still running.

� Success: Report is generated successfully. Click the link Instance Time in the left column of the table to view the report.

� Recurring: Report is scheduled to be generated certain times. Refer back to Figure 5-43 on page 212.

Note: The History window is not updated automatically. Press the Refresh button to view the current state.


� Failed: Report generation failed. Click the link Failed to open the log window as shown in Figure 5-49. The error message, Information is needed before this report can be processed, means that your parameter settings are not valid. Go back to the window shown in Figure 5-46 on page 215 and reenter your parameter settings.

Figure 5-49 Failed report generation

To view successfully generated reports from the history window, as shown before in Figure 5-48 on page 217, click the link Instance Time in the left column of the table to view the associated report. On the following pages, Figure 5-50 and Figure 5-51 show the report Daily Status Summary by SmartSet for our case study scenario.

The report is text based and is split into two pages. At the top of the report window are buttons to navigate through multi-page reports.

In the NetView data warehouse daemon properties file, tdwdaemon.properties, all specified SmartSets are populated and included in this report (refer back to Example 5-2 on page 178).

Some nodes are members of one or more SmartSets, and therefore the availability of these nodes is used multiple times to calculate the total availability.


Figure 5-50 Crystal Enterprise 9 - Report


Figure 5-51 Crystal Enterprise 9 - Report (count)


Figure 5-52, Figure 5-53, and Figure 5-54 show more examples of reports provided by the IBM Tivoli NetView Warehouse Enablement Pack.

Figure 5-52 Summary of total status changes by SmartSet example


Figure 5-53 Nodes with longest outage times example


Figure 5-54 Total daily status changes in monitored network example


Chapter 6. IBM Tivoli Monitoring Warehouse Enablement Pack

This chapter describes a case study for integrating data collected from IBM Tivoli Monitoring, Version 5.1.1 with Tivoli Data Warehouse 1.2 using the IBM Tivoli Monitoring Generic Warehouse Enablement Pack and the IBM Tivoli Monitoring for Operating Systems Warehouse Enablement Pack (WEP). We cover the following topics:

� “Case study overview” on page 226� “IBM Tivoli Monitoring WEP overview” on page 227� “Prerequisites” on page 231� “Installing the ITM WEP data collector component” on page 232� “Installing and configuring ITM Generic WEP” on page 241� “Installing and configuring ITM for OS WEP” on page 262� “Testing, scheduling, and promoting the ETLs” on page 267� “Reporting” on page 275� “Troubleshooting of ITM data collection” on page 286

6


6.1 Case study overviewThis case study shows all the steps required to configure IBM Tivoli Monitoring, Version 5.1.1 in order to gather historical data, process the ITM data into a Tivoli Data Warehouse 1.2, and produce end user reports. The tasks performed in this case study can be summarized as follows:

� Installing and configuring IBM Tivoli Monitoring Enablement Pack, Version 5.1 on the Tivoli server

� Activating ITM data collection

� Installing and configuring ITM Generic Warehouse Enablement Pack (AMX) on the Tivoli Data Warehouse server

� Installing and configuring ITM Warehouse Enablement Pack for OS (AMY) on the Tivoli Data Warehouse server

� Managing the AMX and AMY ETLs

� Reporting

The environment used for our case study is composed of the following servers, as shown in Figure 6-1:

TDW001 Tivoli Data Warehouse single server installation

TDW015 Crystal Enterprise Server

TDW009 TMR server, Tivoli gateway, and ITM data source

TDW0xx Tivoli endpoints (for xx=01,02,...08 Windows endpoints, for xx=09,10,11 AIX endpoints)


Figure 6-1 Environment for our case study

Hardware and operating systems used in our case study environment are listed in Table 6-1.

Table 6-1 Hardware and operating systems

6.2 IBM Tivoli Monitoring WEP overviewIBM Tivoli Monitoring can generate operational data and store it into a central repository called IBM Tivoli Monitoring Middle Layer Database (the default database name is ITM_DB). A program called ETL (Extract, Transform, and Load) takes the operational data, elaborates it, and places the output in the central data warehouse, which can contain historical data from diverse sources

Hostname OS Model Memory Disk size

TDW001 Window 2000 Server SP4

NetVista™ 8305

512 MB 40 GB

TDW009 AIX 5.1.0 7028 6E1 2048 MB 18 GB

TDW015 Window 2000 Server SP4

NetVista 8305 512 MB 40 GB

Web ServerCrystal Server

Web Reports


Data MartData MartData Mart

ITM_DBData source


TDW001

Monitored Endpoints

TDW0XX

TDW009Tivoli Management

Server

Tivoli Data Warehouse

TDW015



Web Reports

WarehouseCentral DataWarehouseWarehouseCentral DataWarehouse

Data MartData MartData MartData MartData MartData MartData MartData MartData Mart

ITM_DBData source

ITM_DBData source



TDW001

Monitored Endpoints

TDW0XX

TDW009Tivoli Management

Server

Tivoli Data Warehouse

TDW015

Chapter 6. IBM Tivoli Monitoring Warehouse Enablement Pack 227

(not only IBM Tivoli Monitoring). A second ETL program extracts a subset of data from the CDW and transfers it into a data mart database specifically designed for reporting (see Figure 6-2).

Figure 6-2 Overview of ITM integration with Tivoli Data Warehouse

The IBM Tivoli Monitoring 5.1.1 is shipped with two Warehouse Enablement Packs (WEP) that provide the ETLs required by the Tivoli Data Warehouse integration:

� IBM Tivoli Monitoring Warehouse Enablement Pack Version 1.1 (also called AMX pack)

� IBM Tivoli Monitoring for Operating Systems Warehouse Enablement Pack Version 1.1 (also called AMY pack)

The AMX warehouse enablement pack provides warehouse functionality for all the applications based on IBM Tivoli Monitoring 5.1.1. It is intended as a generic tool, driven by the metadata provided by each monitoring application. It can be used to extract the data stored into the ITM Middle Layer Repository (ITM_DB) and to load it into the Tivoli Data Warehouse central data warehouse database (TWH_CDW). Due to this “generic” nature, the IBM Tivoli Monitoring WEP does not provide either data mart ETL process, or star schemas. These functions are provided by the WEP of the specific monitoring application.

Data Mart


Web Reports

Web Server

Crystal Server

Central DataWarehouse Data MartData Mart



Web Reports

Web Server

Crystal Server

Web Server

Crystal Server



The ETL process provided by the IBM Tivoli Monitoring WEP (also called generic ETL1) is able to load the correct data into the central data warehouse common schema looking at the rules defined by the specific monitoring application metadata. It is also able to detect and trace any exceptions in the operational data (any data which is not properly described by the application metadata).

Instead, the IBM Tivoli Monitoring for Operating Systems warehouse pack (AMY WEP) provides a set of metadata to drive the IBM Tivoli Monitoring, Version 5.1.1 warehouse pack (AMX or Generic ETL1) to retrieve data collected by the IBM Tivoli Monitoring 5.1.1 Operating Systems Resource Models. It also provides a sample star schema definition used to build some data marts and general-purpose reports.

Figure 6-3 depicts the role of the two WEPs in IBM Tivoli Monitoring 5.1.1 integration with Tivoli Data Warehouse.

Figure 6-3 IBM Tivoli Monitoring data flow

The primary link between IBM Tivoli Monitoring and Tivoli Data Warehouse is the resource model data logging. When you select the Tivoli Enterprise Data Warehouse logging option for a given Resource Model, the Aggregation option is grayed out. You can still select the Raw data logging option (Figure 6-4) for the same Resource Model. If you select both options (TEDW Data and Raw Data), the data will be aggregated at the top of the hour, and both raw data and aggregated data will be visible in the Web health console.

Tivoli AgentRIM

ITM_DB

WarehouseCentral DataWarehouse Data MartData MartData Mart


ITM collect

Tivoli Agent

Tivoli Agent

ETL 2 (AMY)ITM WEP(AMX)

ITM for OS WEP(AMY)

ETL 1 (AMX)

Tivoli AgentRIM

ITM_DB

WarehouseCentral DataWarehouseWarehouseCentral DataWarehouse Data MartData MartData MartData Mart



ITM collect

Tivoli Agent

Tivoli Agent

ETL 2 (AMY)ITM WEP(AMX)

ITM for OS WEP(AMY)

ETL 1 (AMX)


Aggregation data is processed at each cycle time. The IBM Tivoli Monitoring engine aggregates the current data with the “previous” data (that is already a result of previous aggregation), so that, only the data resulting from the aggregation is kept in memory. This data is logged in the endpoint database when the aggregation period expires.

The Tivoli Data Warehouse data aggregation algorithm is exactly the same, but the aggregation period is fixed to 1 hour.

The IBM Tivoli Monitoring engine calculates the Min, Max, Avg, and Total values. However, the Total value is not applicable for all metrics. For example, the Web Health console will only show the Min, Max, Avg data. The total is the sum of the metrics value for each cycle time in the aggregation period.

A thread is spawned every hour and the endpoint database is queried for all the aggregated data saved in the previous hour. Data are then saved in XML files. The thread is spawned with a fixed delay time (presently not possible to change, that is, 20 minutes) with respect to the full hour.

Figure 6-4 shows the data logging options dialog in an IBM Tivoli Monitoring profile.

Figure 6-4 Resource Model Data Logging


If Raw Data logging is enabled, the IBM Tivoli Monitoring engine aggregates data collected during each cycle time and stores the resulting value in the endpoint database when the aggregation period expires. For TEDW Data logging option, the algorithm is exactly the same, but the aggregation period is fixed to 1 hour.

In both cases, data logging will start at the next full hour after the profile distribution time. Figure 6-5 shows an example of an aggregation time line.

Figure 6-5 Aggregation time line

The hourly aggregated data is exported from the endpoint database into a well-formed XML file, 20 minutes after the expiration of the aggregation period.

For more information about IBM Tivoli Monitoring, you can refer to the redbook: IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring, SG24-5519.

6.3 PrerequisitesBefore installing the warehouse enablement packs for IBM Tivoli Monitoring, Version 5.1.1, the following software must be installed:

� IBM Tivoli Monitoring Version, 5.1.1

� IBM Tivoli Monitoring Version, 5.1.1 Fix Pack 6 (5.1.1-ITM-FP06)

� IBM DB2 Universal Database Enterprise Edition 7.2

� IBM DB2 Universal Database Enterprise Edition 7.2 Fix Pack 8 with its eFix (This is the minimum level of IBM DB2 supported by Tivoli Data Warehouse 1.2. Fix Pack 10a is the recommended level and it is shipped with Tivoli Data Warehouse 1.2).


� Tivoli Data Warehouse 1.2

� Crystal Enterprise Professional Version 9 for Tivoli

� IBM DB2 Warehouse Manager, Version 7.2 Fix Pack 8 with its eFix, on the remote agent sites in case of a distributed deployment. (This is the minimum level of IBM DB2 Warehouse Manager supported by Tivoli Data Warehouse 1.2. Fix Pack 10a is the recommended level and it is shipped with Tivoli Data Warehouse 1.2).

6.4 Installing the ITM WEP data collector componentIn this section we describe the steps to install and configure the IBM Tivoli Monitoring Enablement Pack, Version 5.1.1, and to start collecting data from the monitored endpoints.

This component should be installed on the TMR server and all gateways in your Tivoli environment where you want to collect data from endpoints and make the data available for Tivoli Data Warehouse. We have only one gateway in our environment and it is installed on the TMR server, therefore we need to install the component exclusively on the TMR server TDW009.

As a requirement, the IBM Tivoli Monitoring 5.1.1 product must be already installed and a RIM (RDBMS Interface Module) host should be defined. You can choose to install the ITM database on a Tivoli managed node defined as RIM host or on a server outside the Tivoli environment. In our case study we adopted the first configuration, with the ITM data repository (ITM_DB) on the TMR server (TDW009) that is also a RIM host.

The installation method described here uses the Tivoli Desktop, as follows:

1. From the Tivoli Desktop, select Desktop -> Install -> Install Product. The Install Product dialog shows the products that are available to install as shown in Figure 6-6.

Note: You can also install the Tivoli Enterprise Data Warehouse Support 5.1.1 from the command line. Use the following to install it from the command line:

winstall –c path /cdrom -i DM511TED managed_node

Here, DM511ED is the index file for Tivoli Enterprise Data Warehouse Support 5.1.1 and managed_node is the name of the managed node to be installed on.


Figure 6-6 Installing warehouse support

2. Select IBM Tivoli Monitoring - Tivoli Enterprise Data Warehouse Support, Version 5.1.1, then select your Tivoli Management Region server and the gateways that you want to have it installed on.

3. RIM configuration is required to proceed with the installation, as shown in Figure 6-7.


Figure 6-7 RIM setup options

The installation process will create a RIM object named itm_rim_<nodename>, where <nodename> is the RIM host in your Tivoli environment (tdw009 for ours). The RIM object can also be created at a later time using the following command, for instance, assuming that you have a DB2 database server:

wcrtrim -v DB2 -h tdw009 -d itm_db -u db2inst1 -H /usr/lpp/db2_07_01 -s TCPIP -I /home/db2inst1 itm_rim_tdw009

This RIM object by default has itmitm as a password, which must be changed to match the password of your database instance owner. Use the wsetrimpw command as follows:

wsetrimpw itm_rim_<nodename> itmitm <newpw>

Here, <newpw> is the database instance owner password.

4. Click Set and then select Install and follow the normal installation dialogs.


5. The physical database for the Warehouse Support component name is ITM_DB and needs now to be created. This process can be accomplished by either using a provided shell script or using SQL scripts. If you intend to use the provided shell script, make sure you grant the RDBMS administrator (or database instance owner) user ID with Administrator (root) and Tivoli_Admin_Privileges and run the script logged in as your user ID.

The reason is that the shell script collects information from the previously created RIM object in order to create both the database and its structure. The shell script name is cr_itm_db.sh, and it is located in the $BINDIR/TME/Tmw2k/Warehousecfg directory.

As an alternative method, you can use the SQL scripts. These scripts are also located in the $BINDIR/TME/Tmw2k/Warehousecfg directory and have the following name standard:

cr_db.<DBext> and cr_tbl.<DBext>, where <DBext> is the database vendor designator (db2 in our case).

The following sequence describes the creation process for DB2 using the SQL scripts:

a. On the RIM host machine, login as your instance owner (in our case, db2inst1).

b. Only perform this step if the RIM host machine does not have the Warehouse Support component installed. Copy the cr_db.db2 and cr_tbl.db2 files from the $BINDIR/TME/Tmw2k/Warehousecfg directory from your TMR server to the RIM host machine.

c. Move to the directory where the SQL scripts are located and rename cr_db.db2 to cr_db_db2.sql and rename cr_tbl.db2 to cr_tbl_db2.sql. Edit cr_db_db2.sql and replace “CREATE DATABASE _xz_db” with “CREATE DATABASE itm_db”.

d. Run the following command to create the itm_db database:

db2 -td$ -vf cr_db_db2.sql

e. In order to have the itm_db database structure created, run the following commands, where <db2inst1pw> is the database instance owner password.

db2 connect to itm_db user db2inst1 using <db2inst1pw>

db2 -td$ -vf cr_tbl_db2.sql

6. On the TMR server, test the RIM object connection:

wrimtest -l itm_rim_<rimhost>

The output should be similar to Example 6-1.


Example 6-1 Testing the RIM object

tdw009:/>wrimtest -l itm_rim_tdw009Resource Type : RIMResource Label : itm_rim_tdw009Hostname : tdw009User Name : db2inst1Vendor : DB2Database : itm_dbDatabase Home : /usr/lpp/db2_07_01Server ID : tcpipInstance Home : /home/db2inst1Opening Regular Session...Session OpenedRIM : Enter Option >

Type x and press Enter to release the session.

7. The data collection process of the warehouse support component needs to be configured. The configuration file is named .config and it is located in the $DBDIR/dmml directory. The warehouse support entries in the.config file have the prefix datacollector. Such entries should be added/modified using the wdmconfig command, and it is important to notice that this file must not be modified manually. For details on the wdmconfig command, refer to the IBM Tivoli Monitoring User’s Guide Version 5.1.1, SH19-4569 manual. To set the collection parameters, issue the following command:

wdmconfig -m <nodename> -D datacollector.rim_name=itm_rim_<rimhost> \-D datacollector.db_purge_interval=30 \-D datacollector.db_purge_time=0 \-D datacollector.delay=30 \-D datacollector.sleep_time=1 \-D datacollector.max_retry_time=6

Here, <node name> is the gateway specified for the monitored endpoints. You can check if the entries were correctly set by issuing:

wdmconfig -m <nodename> -G datacollector*

The output should be similar to Example 6-2.

Example 6-2 Datacollector configuration

# wdmconfig -m tdw009 -G datacollector*Managed Node: 'tdw009':======================= datacollector.db_purge_interval = 30 datacollector.db_purge_time = 0 datacollector.delay = 30 datacollector.sleep_time = 1 datacollector.max_retry_time = 6 datacollector.rim_name = itm_rim_tdw009


datacollector.rim_name

Specifies the name of the RIM object that the data collection process will use to load data to the database. The default is itm_rim_<RIM host name>.

datacollector.db_purge_interval

Specifies the number of days the data is kept on the database: older data is automatically removed from the database. The value can range from 10 to 60. The default is 30 days.

datacollector.db_purge_time

Specifies the time during the day for the data removal operation. The value can range from 0 (midnight) to 23. The default is 0 (midnight).

datacollector.delay

Specifies the time delay (in minutes, compared to the hour) after which the data collector process uploads data from the endpoints. The value can range from 1 to 60 minutes. The default is 30 minutes.

datacollector.sleep_time

Specifies the time interval (in minutes) between two consecutive requests of data uploading generated by the data collector processor. The value can range from 1 to 60 minutes. The default is 1 minute.

datacollector.max_retry_time

Specifies the maximum number of times an XML data file must be processed before being archived when an error occurs. The default is 6 times.

6.4.1 Activate data collectionThe following steps are required to start monitors and data collection into ITM_DB database:

1. Open the Tivoli desktop and create a profile manager for storing the data warehouse collection profiles. The policy region containing the profiles must have Tmw2kProfile as Managed Resource.

2. Create the Tmw2kProfile objects in the profile manager and add the appropriate resource models. For details about resource models, please refer to the manual IBM Tivoli Monitoring - Resource Model Reference, SH19-4570.

3. For each resource models, open it by clicking Edit. In the Edit resource model window, click Logging. The logging window will be shown as in Figure 6-8. You need to check the Enable Data Logging and TEDW Data check boxes. Click Apply Changes and Close when done. Click Modify & Close from the Edit resource model window.


Figure 6-8 Logging option

4. Now, you can specify the subscribers. Each resource model can only have specific subscriber types. You may want to create multiple profiles which each profile contains resource models for specific object types. For example, in our ITSO environment we created two different profiles, one for Windows 2000 endpoints and the other for AIX endpoints.

Note: By using the Logging window, you can specify how to log data collected by a resource model.

� If you select the Raw Data option, data will be written exactly as it is collected by the resource model. All the monitored values are collected and copied in the database. This data later can be used by the Health Console.

� Selecting the TEDW Data option allows data to be collected and copied in the database for later use by Tivoli Enterprise Data Warehouse.

� If you choose the Aggregated Data option, data will be collected and aggregated at fixed intervals you define (Aggregation Period). Then only the aggregated values are written in the database.

� The Historical Period option is used to specify the period for which data is to be stored in the database.

� For the Tivoli Data Warehouse you need to check TEDW Data option. In addition, you can also check the Raw Data option (this does not affect data collection for Tivoli Data Warehouse).


5. Distribute the profile to your subscribers. You can either do it from the Tivoli desktop, or using the wdmdistrib command.

6. Check the distribution of the profile and the execution of it using the wdmlseng command. Ensure that all the profiles are having the state of Running, as shown in Example 6-3.

Example 6-3 wdmlseng command output

# wdmlseng -e tdw002

Forwarding the request to the engine...

The following profiles are running:

tmw2kDefProfile#tdw009-region TMW_EventLog :Running TMW_PhysicalDiskModel :Running TMW_TCPIP :Running TMW_LogicalDisk :Running TMW_MemoryModel :Running TMW_Process :Running TMW_Processor :Running

7. You need to tell the gateways, to which the endpoint reports to, to start collecting historical information to be put into the RIM object. Use the wdmcollect command. The syntax of the command is:

wdmcollect -e <endpoint> -s <time> -m <managed_node>

Here, <endpoint> is the endpoint name, <time> is the collection interval in hours, and <managed_node> is the managed node to be defined as the data collector node. After you run this command for all the endpoints, you can check the result using the command wdmcollect -q. The managed node will pull the data from the endpoint every interval and store it under $DBDIR/dmml/tedw. Example 6-4 shows the command output in our case study environment. It means that data is being collected from nine endpoints (tdw001, tdw002, tdw003, tdw004, tdw005, tdw006, tdw008, tdw010, tdw011) via the managed node tdw009 every hour (3600 seconds).

Example 6-4 wdmcollect command output

# wdmcollect -qProcessing ManagedNode tdw009...<Requests> <Request Id="235d1afd" Endpoint="tdw011" RefreshTime="3600" LastCall="Fri Aug 1 14:32:00 CDT 2003"/> <Request Id="545a2a6b" Endpoint="tdw010" RefreshTime="3600" LastCall="Fri Aug 1 14:32:00 CDT 2003"/>


<Request Id="52f58445" Endpoint="tdw004" RefreshTime="3600" LastCall="Fri Aug 1 14:32:00 CDT 2003"/> <Request Id="cc9111e6" Endpoint="tdw003" RefreshTime="3600" LastCall="Fri Aug 1 14:32:00 CDT 2003"/> <Request Id="bb962170" Endpoint="tdw002" RefreshTime="3600" LastCall="Fri Aug 1 14:32:00 CDT 2003"/> <Request Id="229f70ca" Endpoint="tdw001" RefreshTime="3600" LastCall="Fri Aug 1 14:32:01 CDT 2003"/> <Request Id="25f2b4d3" Endpoint="tdw005" RefreshTime="3600" LastCall="Fri Aug 1 14:39:04 CDT 2003"/> <Request Id="bcfbe569" Endpoint="tdw006" RefreshTime="3600" LastCall="Fri Aug 1 14:39:12 CDT 2003"/> <Request Id="5b43c86e" Endpoint="tdw008" RefreshTime="3600" LastCall="Fri Aug 1 14:39:23 CDT 2003"/></Requests>

8. Before you can start checking on the RIM database to see whether the data has been collected, you have to wait for a time that depends on the selected collection interval and the data collector delay. We check the collection by verifying that the ENDPOINTS table has been populated and checking the timekey_dttm in metricsdata table as shown in Example 6-5. Please note that the time reported in timekey_dttm is a GMT time.

Example 6-5 Sample SQL that check the collection

db2 => connect to ITM_DB


Database server = DB2/6000 7.2.6 SQL authorization ID = DB2INST1 Local database alias = ITM_DB

db2 => select host_name from endpoints

HOST_NAME

-------------------------------------------------------------------------------tdw001tdw002tdw003tdw004tdw005tdw006tdw007tdw008tdw009tdw010tdw011



db2 => select max(timekey_dttm) from metricsdata

1--------------------------2003-07-31-11.00.56.000000


6.5 Installing and configuring ITM Generic WEP This section describes how we installed and configured the IBM Tivoli Monitoring V 5.1.1 Generic Warehouse Enablement Pack (AMX) in our lab environment.

The step required for AMX installation and configuration are:

1. Backup TWH databases2. Configure DB2 connectivity to data sources3. Installing the ITM 5.1.1 AMX ETL processes4. Installing AMX Fix Packs5. Define the authority to the warehouse sources and targets6. Modify the ETL for source table name to the RIM user

6.5.1 Backing up the TWH databasesBefore installing any warehouse enablement pack (WEP), it is highly recommended to back up all the warehouse databases to ensure that you can return to a known valid state if you encounter an unrecoverable error during installation.

In our case study we used a single server installation of Tivoli Data Warehouse, so that we have to back up the three databases TWH_CDW, TWH_MART, and TWH_MD (if you use different configurations, refer to Chapter 2, “Planning for Tivoli Data Warehouse 1.2” on page 27 for warehouse database naming conventions).

In order to do so, we created a local directory structure on our DB2 server to store our backups by creating the following three directories:

� C:\tdwbackups\pre-amx\twh_cdw� C:\tdwbackups\pre-amx\twh_mart� C:\tdwbackups\pre-amx\twh_md


After creating these directories, we completed the following steps on the DB2 server from a command window:

1. Stop and restart DB2:

db2stop forcedb2start

2. Back up the TWH_CDW database:

db2 backup db TWH_CDW to C:\tedwbackups\pre-amx\twh_cdw

3. Back up the TWH_MART database:

db2 backup db TWH_MART to C:\tedwbackups\pre-amx\twh_mart

4. Back up the TWH_MD database:

db2 backup db TWH_MD to C:\tedwbackups\pre-amx\twh_md

6.5.2 Establishing an ODBC connection on the Control CenterBecause the IBM Tivoli Monitoring Version 5.1.1 Generic ETL1 retrieves its data from the ITM database (ITM_DB), we have to create an ODBC connection to that database in our control server.

To create the ODBC connection to ITM_DB database you can use the command line or the DB2 Client Configuration Assistant.

If you are using a DB2 command line window on the control server, issue the following commands:

db2 catalog tcpip node CAT_ODBC remote <REMOTE_SVR> server <DB2_PORT>db2 catalog database ITM_DB at node CAT_ODBCdb2 catalog system odbc data source ITM_DB

Where <REMOTE_SVR> is the hostname of the DB2 server that contains the ITM database (TDW009 in our case) and <DB2_PORT> is the TCP/IP port used by DB2 (the default is 50000).

If you prefer using DB2 Client Configuration Assistant, execute the following steps:

1. On the Windows task bar, click Start -> Programs -> IBM DB2 -> Client Configuration Assistant. The Client Configuration Assistant is displayed (Figure 6-9).


Figure 6-9 Client Configuration Assistant opening dialog

2. Click Add and the Add Database Wizard shows (Figure 6-10). Select Search the network and then click the Database name tab.


Figure 6-10 Add Database Wizard

3. In the Database name tab, click Add System. An Add System dialog appears (Figure 6-11). Selected TCPIP and complete the Host name information. Click OK.


Figure 6-11 Add System dialog window

4. A list of DB2 databases should expand from the DB2 server you added (Figure 6-12). Select ITM_DB and click Finish.


Figure 6-12 Select ITM_DB in the dialog window

5. A Confirmation window appears (Figure 6-13). It is recommended to test the ODBC connection to verify a healthy link. Click Test Connection.

Figure 6-13 Confirmation dialog window

6. Fill out the User ID and Password information (Figure 6-14). Click OK.


Figure 6-14 User ID and Password dialog window

7. The connection should be successful (Figure 6-15). If not, review your settings and change them accordingly.

Figure 6-15 ODBC connection successful

6.5.3 Installing the ITM 5.1.1 AMX ETL processesTo install the AMX warehouse pack, we completed the following steps on the control server:

1. Click Start -> Programs -> TDW -> Install a Warehouse Pack, this starts the warehouse pack installation wizard (see Figure 6-16 on page 248). In the Welcome windows, click Next.


Figure 6-16 Install a Warehouse Pack window

2. The following window shows the location of the Tivoli common logging directory (Figure 6-17 on page 249) which will contain all TDW log files. In our installation we use the default location C:\Program Files\ibm\Tivoli\common. Click Next.


Figure 6-17 Tivoli Common Logging Directory window

3. In the windows as shown in Figure 6-18 on page 249, click Add to add the AMX warehouse pack.

Figure 6-18 Add Warehouse Pack window


4. In the Location of installation properties file window, as shown Figure 6-19 on page 250, specify the location of the AMX warehouse pack installation properties file, twh_install_props.cfg: you can find this file in the IBM Tivoli Monitoring version 5.1 media, under the tedw_apps_etl\AMX\pkg directory. Click Next.

Figure 6-19 Location of installation properties

5. The installation menu window (Figure 6-20 on page 251) now lists the IBM Tivoli Monitoring warehouse enablement pack. Select it and click Next to continue.


Figure 6-20 Installation menu window

6. Click Install in the summary window (Figure 6-21 on page 252) to start the warehouse pack installation.


Figure 6-21 Installation summary window

7. View the progress of installation through the messages that are shown in windows until its completion. The final installation window (Figure 6-22 on page 253) contains either a successful completion notice or messages describing problems. Make sure the window does not list any warnings or errors, and then click Next. If warnings are listed, check the logs to ensure that the warnings can safely be ignored. Click Finish to exit the wizard.


Figure 6-22 AMX installation completion window.

6.5.4 Installing AMX Fix PacksIn Tivoli Data Warehouse 1.2 the installation process for a WEP patch is exactly the same used for the WEP installation. We installed Fix Pack 6 for AMX following the steps described in “Installing the ITM 5.1.1 AMX ETL processes” on page 247 and specifying the location of file twh_install_props.cfg under of the AMX subdirectory in the Fix Pack media (see Figure 6-23 on page 254).

Important:

� Verify that all the ETL processes belonging to the enablement pack for which you are installing a Fix Pack are in 'development' mode. This is required for preventing that any ETL process may run during the Fix Pack installation.

� If you have already configured your ETLs with DB accounts information before installing the Fix Pack, you have to configure them again after the Fix Pack installation (see “Defining the authority to the warehouse sources and targets” on page 254).


Figure 6-23 Installation of AMX Fix Pack 6

6.5.5 Defining the authority to the warehouse sources and targetsYou should inform the Tivoli Data Warehouse Control Center server of user access information for every Source and Target ETL process installed by the IBM Tivoli Monitoring, Version 5.1.1 Generic ETL1. Follow these steps:

1. Start the IBM DB2 Control Center utility by selecting Start -> Programs -> IBM DB2 -> Control Center.

2. On the IBM DB2 Control Center utility, start the IBM DB2 Data Warehouse Center utility by selecting Tools -> Data Warehouse Center. The Data Warehouse Center logon windows appears.

3. Log into the IBM DB2 Data Warehouse Center utility using the local DB2 administrator user ID. In our case, db2admin.

4. In the Data Warehouse Center window, expand the Warehouse Sources folder. As shown in Figure 6-24, there are two entries for the IBM Tivoli Monitoring, Version 5.1.1 Generic ETL1 programs that need to be configured, as follows:

– AMX_ITM_RIM_Source– AMX_TWH_CDW_Source


Figure 6-24 IBM Tivoli Monitoring, Version 5.1.1 Generic ETL1 Sources

You should edit the properties of each one of the above entries. In order to do that, right-click it and select Properties and then select the Data Source tab. Fill in the database instance owner user ID information (Figure 6-25 and Figure 6-26).

Figure 6-25 AMX_ITM_RIM_Source user ID information


Figure 6-26 AMX_TWH_CDW_Source user ID information

5. For the IBM Tivoli Monitoring, Version 5.1.1 Generic ETL1 Target ETL, shown in Figure 6-27, expand the Warehouse Target folder, right-click the AMX_TWH_CDW_Target, select Properties and then select the Database tab. Fill in the user ID information.

Figure 6-27 IBM Tivoli Monitoring, Version 5.1.1 Generic ETL1 Target


For our environment the values are shown in Figure 6-28.

Figure 6-28 AMX_TWH_CDW_Target user ID information

6.5.6 Modifying the ETL for the source table name to the RIM userThe AMX ETL assumes that the schema name of ITM_DB RIM is set to db2admin. This is the default name for databases created on a Windows machine, but if you are using DB2 on other platforms or you have a different instance name, you must modify the source table name into the DB2 Warehouse Manager.

In our case, we have the ITM_DB on a UNIX server and we use db2inst1 as the user ID and instance name. Therefore we have to modify the source table name, executing the following steps:

1. From the Data Warehouse Center, from the Warehouse sources list, select the AMX_ITM_RIM_Source and open the Property page.

2. Go to the Tables and views tab as shown in Figure 6-29.


Figure 6-29 Tables and views of AMX_ITM_TIM_Source

3. Expand the Tables folder and you will get a dialog asking for the name filter, such as shown in Figure 6-30. We only need to get the table called ENDPOINTS. The schema name is the RIM user ID.

Figure 6-30 Table name filter specification


4. When the DB2.ENDPOINTS has been found, move it from the available tables and views to the selected tables and views box by clicking the > button. You will have two ENDPOINTS tables as shown in Figure 6-31. Click OK.

Figure 6-31 Endpoint tables

5. Now from the Data Warehouse Center, expand the Subject Area and find the process called AMX_c05_ETL1_Process. Right-click it and select Open. The Process Modeller window shown is similar to Figure 6-32.


Figure 6-32 AMX_c05_ETL1 process


6. Click the tables icon and click the work area; a dialog box will be presented, as shown in Figure 6-33. Select the DB2INST1.ENDPOINTS table and click the > button. Then click OK.

Figure 6-33 Selecting new table

7. The new table is now shown in the process modeller window; now we need to connect the tables to the first step. Use the link icon and select data links. Drag the cursor from the ENDPOINTS table to the AMX_c05_s010_RIM_Extract step, and a new link is created.

8. Remove the old link by selecting the link, right-click and select Remove. Remove also the old DB2ADMIN.ENDPOINTS table by selecting it, right-click and select Remove.

9. Save the process model using the menu Process -> Save and close the window.

Attention: At this time, we bypass scheduling the AMX ETL and changing the status of the ETL to production. We do this because we are not ready to run the AMX ETL until an ETL2 is installed (AMY). Running the AMX ETL prematurely will result in errors and prevent you from gathering data in future collections.


6.6 Installing and configuring ITM for OS WEP This section describes how we installed and configured the IBM Tivoli Monitoring, Version 5.1.1 Warehouse Enablement Pack for Operating Systems (AMY) in our lab environment. The steps required for AMY installation are the same required by AMX, as described in “Installing and configuring ITM Generic WEP” on page 241, but in this case we are not required to configure a new DB2 connectivity to the data source, because AMY uses the central data warehouse database as data source.

6.6.1 Backing up the TWH databasesOnce again, before installing any warehouse enablement pack (WEP), it is highly recommended to back up all the Tivoli Data Warehouse databases. We created different directories to store these backups. This enables us to restore Tivoli Data Warehouse at the last known workable level should any unforeseen problems occur during installation or afterwards.

� C:\tedwbackups\pre-amy\twh_cdw� C:\tedwbackups\pre-amy\twh_mart� C:\tedwbackups\pre-amy\twh_md



db2 force applications alldb2stop forcedb2start


db2 backup db TWH_CDW to C:\tedwbackups\pre-amy\twh_cdw


db2 backup db TWH_MART to C:\tedwbackups\pre-amy\twh_mart


db2 backup db TWH_MD to C:\tedwbackups\pre-amy\twh_md

6.6.2 Installing the ITM 5.1.1 AMY ETL processesTo install the IBM Tivoli Monitoring for Operating Systems v. 1.1 warehouse pack (AMY), we followed the same steps we executed on the control server for the AMX pack:


1. Click Start -> Programs -> TDW -> Install a Warehouse Pack, this starts the warehouse pack installation wizard (see Figure 6-16 on page 248). In the Welcome windows, click Next.

2. The following window shows the location of the Tivoli common logging directory (Figure 6-17 on page 249) which will contain all TDW log files. In our installation we use the default location C:\Program Files\ibm\Tivoli\common. Click Next.

3. In the windows as shown in Figure 6-18 on page 249, click Add to add the AMY warehouse pack.

4. In the Location of installation properties file window, as shown Figure 6-19 on page 250, specify the location of the AMY warehouse pack installation properties file, twh_install_props.cfg: you can find this file in the IBM Tivoli Monitoring version 5.1 media, under the tedw_apps_etl\AMY\pkg directory. Click Next.

5. The installation menu window now lists the IBM Tivoli Monitoring warehouse enablement pack (Figure 6-34). Select it and click Next to continue.

Figure 6-34 Installation menu window with the AMY pack

6. Click Install in the summary window (Figure 6-21 on page 252) to start the warehouse pack installation.


7. View the progress of installation through the messages that are shown in windows until its completion. The final installation window contains either a successful completion notice or messages describing problems. Make sure the window does not list any warnings or errors, and then click Next. If warnings are listed, check the logs to ensure that the warnings can safely be ignored. Click Finish to exit the wizard.

Figure 6-35 AMY installation completion window

6.6.3 Installing AMY Fix PacksSimilarly to the installation procedure of the Fix Pack 6 for the AMX ETL, we have to install Fix Pack 6 for AMY by following the steps described in 6.6.2, “Installing the ITM 5.1.1 AMY ETL processes” on page 262 and specifying the location of file twh_install_props.cfg under of the AMY subdirectory in the Fix Pack media (see Figure 6-36).


Figure 6-36 Installation of AMY Fix Pack 6

6.6.4 Defining the authority to the warehouse sources and targetsWe now have to provide Tivoli Data Warehouse Control Center with user access information for every Source and Target ETL process installed by the IBM Tivoli Monitoring, Version 5.1.1 AMY ETL. To do so, we performed the following steps:


Important:

� Verify that all the ETL processes belonging to the warehouse enablement pack for which you are installing a Fix Pack are in “development” mode. This is required for preventing the possibility that any ETL process may run during the Fix Pack installation.

� If you have already configured your ETLs with DB accounts information before installing the Fix Pack, you have to configure them again after the Fix Pack installation (see “Defining the authority to the warehouse sources and targets” on page 254).



3. Log into the IBM DB2 Data Warehouse Center utility using the local DB2 administrator user ID. In our case, db2admin.

4. In the Data Warehouse Center window, expand the Warehouse Sources folder. Update the database sources that relates to the application that you want to configure. In this example, this is AMY_TWH_CDW_Source.

You should edit the properties of this warehouse source. In order to do that, right-click and select Properties and then select the Data Source tab. Fill in the database user ID and password information. For our environment the values are shown in Figure 6-37.

Figure 6-37 AMY_TWH_CDW_Source user ID information

5. Again, for the AMY warehouse targets, you need to modify the user ID information from the property pages. You should edit the properties of each one of those warehouse targets. In order to do that, right-click it and select Properties and then select the Database tab. Fill in the database user ID and password information. For our environment the values are shown in Figure 6-38, using the AMY_TWH_MART_Target as example.


Figure 6-38 AMY_TWH_MART_Target user ID information

6.7 Testing, scheduling, and promoting the ETLsAfter successfully installing and configuring the IBM Tivoli Monitoring, Version 5.1.1 Generic Warehouse Enablement Pack (ETL1) and the IBM Tivoli Monitoring, Version 5.1.1. Warehouse Pack for Operating Systems (ETL2), we can now test all the processes from both ETLs. If they run correctly, we can schedule and then promote them into production mode.

6.7.1 Testing the ETLsTo test the ETLs manually, the following steps are required:

1. Select the ETL processes in the Data Warehouse Center windows, right-click them and choose Mode -> Test as shown in Figure 6-39.


Figure 6-39 Change ETL mode to Test

2. Right-click the process you want to test and choose Test (see Figure 6-40 on page 269). The process must be run sequentially according the dependency described in the ETL process model (see Figure 6-32 on page 260). The sequence for all AMX and AMY processes is as follows:

a. AMX_c05_ETL1_Process

i. AMX_c05_s05_Pre_Extract ii. AMX_c05_s010_Rim_Extractiii. AMX_c05_s020_Parsingiv. AMX_c05_s030_Exceptionv. AMX_c05_s040_Comp_Msmt

b. AMX_c10_Rim_Prune_Process

i. AMX_c10_s010_Rim_Prune

c. AMY_c05_ETL1_Data_Update_Process

i. AMY_c05_s010_Update

d. AMY_m05_ETL2_Process

i. AMY_m05_s05_Mart_Prepare_Stageii. AMY_m05_s010_Mart_Pre_Extractiii. AMY_m05_s020_Mart_Extractiv. AMY_m05_s030_Mart_Loadv. AMY_m05_s040_Mart_Rollupvi. AMY_m05_s050_Mart_Prune


The following processes should only be tested and executed in case you need to reset the OS data mart. All the data will be cleaned out of the data mart database.

e. AMY_m10_Reset_ETL2_Process

i. AMY_m10_s010_Reset_OS_Data_Mart

Figure 6-40 Manually test the ETLs

3. To verify the status of processes (Successful, Failed, In Progress, or Scheduled), select in the Data Warehouse Center Warehouse -> Work in Progress. The Work in Progress window (see Figure 6-41 on page 270) not only shows the status of all processes for all ETLs with status set to Successful, Failed, In Progress, or, if promoted to production, Scheduled, but also allows you to run a process again by simply right-clicking on it and selecting Run now.


Figure 6-41 Work in progress window

6.7.2 Checking that data has been collectedOnce all the processes are tested, we checked that the data mart database has been populated with ITM data. To do this, we used the DB2 Control Center to connect to TWH_MART and show a sample content of one of the tables, for example, the TABLE F_OS_HOUR, as shown in Figure 6-42.

If you have some data in the table, you have a proof that all the ETL processes are running correctly. If that table is empty, you should verify that your data source is note empty and then run all the ETL processes again.


Figure 6-42 Sample Content of table F_OS_HOUR


6.7.3 Scheduling the ETLsThe following four processes need to be scheduled for the IBM Tivoli Monitoring, Version 5.1.1, Generic ETL1 and IBM Tivoli Monitoring, Version 5.1.1. Warehouse Pack for Operating Systems:

� AMX_c05_ETL1_Process� AMX_c10_Rim_Prune_Process� AMY_c05_ETL1_Data_Update_Process� AMY_m05_ETL2_Process

The following steps are similar for both processes and we will use AMX_c05_ETL1_Process to describe them.

On the DB2 Control Center server, using the Data Warehouse Center window, expand Subject Areas.

Select AMX_IBM_TIVOLI_Monitoring_v5.1.1_Subject_Area -> Processes and right-click AMX_c05_ETL1_Process. Choose Schedule, as shown in Figure 6-43.

Figure 6-43 Schedule AMX_c05_ETL1_Process


Selecting Schedule will open up a dialog as shown in Figure 6-44.

Figure 6-44 Schedule configuration for AMX_c05_ETL1_Process


6.7.4 Promoting the ETL status to Production modeAll of the processes are composed of components that have Development status set as default. In order for them to run, their status need to be changed from Development (or Test) to Production. To do this, we used the Data Warehouse Center window again, selecting all the AMX and AMY processes listed in “Scheduling the ETLs” on page 272, right-clicking them and choosing Mode -> Production, as shown in Figure 6-45 on page 274 for the AMX_c05_ETL1_Process.

Figure 6-45 Promoting ETLs to Production


6.8 ReportingNext we show how to set up, configure, and use some of the reports provided by the IBM Tivoli Monitoring, Version 5.1.1 Warehouse Enablement Pack.

6.8.1 Available reportsHere is a list of predefined reports shipped with the IBM Tivoli Monitoring, Version 5.1.1 Warehouse Enablement Pack:

� Operating System Busiest Systems � Operating System Health of a Backup Server� Operating System Memory Utilization� Operating System Network Statistics� Operating System Paging File Utilization� Operating System UNIX CPU Statistics� Operating System Usage of Domain Controller� Operating System Windows CPU Statistics

Crystal Enterprise Professional Version 9 for Tivoli has reduced configuration options in comparison to a full Crystal license. If the reports shipped with IBM Tivoli Monitoring, Version 5.1.1 Warehouse Enablement Pack do not match your needs and you want to develop additional reports, you have to upgrade your Crystal Enterprise installation.

6.8.2 Accessing the Crystal ePortfolioAll reports provided by Tivoli Data Warehouse 1.2 must be accessed using the ePortfolio feature of the Crystal Enterprise. The ePortfolio can be accessed from the Crystal Enterprise launchpad. In order to access the Crystal Enterprise launchpad, start an Internet browser and open the following URL:






In this section, we concentrate on viewing IBM Tivoli Monitoring, Version 5.1.1 reports and we do not explain the configuration of Crystal Enterprise to its full extent. For details on configuration and administration tasks, refer to the following manuals shipped with the product:



From the Crystal Enterprise Launchpad, proceed by selecting the ePortfolio link, which will bring you to the window shown in Figure 6-47. In the top bar, you can see that we are authorized as user guest. By default, the guest user has no access to the NetView reports as indicated by the words No Folders on the left side of the window.




To log in as the Tivoli user ID, select the Log On button in the upper right corner of the ePortfolio window in Figure 6-47 on page 277. The Log On window as shown in Figure 6-48 is presented. The Tivoli user ID has no password by default. We use the Enterprise authentication method as we have specified during the Crystal Enterprise installation.



After entering the required data, select Log On to proceed. Now we are back at the ePortfolio window in Figure 6-49, but now with user Tivoli authority. Instead of No Folders in the guest users ePortfolio window in Figure 6-47, there is now a link visible with the name IBM Tivoli Monitoring for Operating Systems in the Tivoli user ePortfolio window in Figure 6-49 .



We follow this link by selecting IBM Tivoli Monitoring for Operating Systems and proceed to the IBM Tivoli Monitoring, Version 5.1.1 reports as shown in Figure 6-50. All reports provided by the IBM Tivoli Monitoring Warehouse Enablement Pack are listed there.

Figure 6-50 Crystal Enterprise 9 - available reports for ITM


In order to obtain the reports, select the desired report, for example, Operating System Busiest Systems, and select Schedule, as shown in Figure 6-51.

Figure 6-51 Scheduling Operating System Busiest System report

The schedule report panel starts. In order to run the reports at this time, select Now under the Run Report option. As this report requires additional parameters, such as time frame, select Parameters under the Customize your Options option, as shown in Figure 6-52.


Figure 6-52 Crystal Enterprise 9 - parameters

Figure 6-53 shows the selection of parameters for the report. Select Schedule when ready to run the report.

Figure 6-53 Crystal Enterprise 9 - Parameters for the report


Because we selected to run this report now, the report is scheduled immediately and the reports history window is opened. The just-scheduled report will run and its initial status is set to Pending.

Figure 6-54 Crystal Enterprise 9 - Report History

To view successful generated reports from the history window as shown in Figure 6-54, click the link Instance Time in the left column of the table to view the associated report. The report is shown in Figure 6-54.

Note: The History window is not updated automatically. Click the Refresh button to view the current state.


Figure 6-55 Operating System Busiest Systems report


Next we present some more examples of reports provided by the IBM Tivoli Monitoring, Version 5.1.1 Warehouse Enablement Pack.

Figure 6-56 shows the Operating System Paging File Utilization report.

Figure 6-56 Operating System Paging File Utilization


Figure 6-57 shows the Operating System UNIX CPU Statistics report.

Figure 6-57 Operating System Operating System UNIX CPU Statistics

6.9 Troubleshooting of ITM data collectionIn this section we discuss the commands and techniques that can be used whenever the expected data is not collected into the IBM Tivoli Monitoring database used as a data sources for the Tivoli Data Warehouse. First we show how to use the itmchk.sh tool provided by Tivoli Support to check the ITM environment automatically, then we discuss all the steps required for doing a manual check of the ITM data collection instead.


6.9.1 Using itmchk.sh scriptTivoli Software Support provides a script (itmchk.sh) to check automatically most of the configuration setting required to have data successfully uploaded into the ITM database using a RIM Object. This script can be downloaded from the IBM Software Support Web site at the following link:

http://www-1.ibm.com/support/docview.wss?uid=swg24003854

To access that Web site and download the script, you must provide an IBM ID: If you do not have it, you can apply on the same Web link to receive one.

The itmchk.sh script is bundled in a package called ITM - TEDW Analysis Tools together with another script (twhchk.sh), which can be used to check the installed components and their installation order in a Tivoli Data Warehouse 1.2 environment and to provide a summary of meaningful data from both the ITM and TEDW databases.

To run the itmchk.sh script, follow these steps:

1. Download the ITM-TEDW-Health.tar in the Tivoli Manage Node where your ITM RIM object is available (in our case study scenario, the TMR server TDW009 is also the RIM host for ITM database).

2. Extract all the files and directories contained in the archive file.

3. Go into the ITM-TEDW-Health directory.

4. Setup the Tivoli environment (. /etc/Tivoli/setup_env.sh for a UNIX system).

5. Run the command ./itmchk.sh6. The program prompts for the ITM RIM name as shown in Example 6-6.

Choose the number corresponding to the RIM name used for ITM_DB database and press ENTER.

Example 6-6 Running itmchk.sh tool

# ./itmchk.sh

======================================================== IBM Tivoli Monitoring - Tivoli Enterprise Data Warehouse Configuration and Status Snapshot (c) IBM - Tivoli ========================================================

==> Analysis started at: Aug 14 2003 11:46:10==> Setting Debug Mode... (I) Debug Disabled==> Including Help Messages Routines... (I) Done Including Help Messages Routines


http://www-1.ibm.com/support/docview.wss?uid=swg24003854

==> Looking up TME 10 System Information... (I) Valid User Found==> Getting Environmental Configuration... (I) Environment Successfully Imported==> Looking for RIM Objects definition... (I) Discovered the following RIM Objects: 1) itm_rim_tdw009 2) mdist2 3) spr_rim-n (?) Select the One used by Warehouse Component:

7. The tool checks ITM configuration and then prints a report as shown in Example 6-7.

Example 6-7 itmchk.sh tool report

(I) Using RIM Object (itm_rim_tdw009)==> Retrieving RIM Object Information... (I) Valid RIM Object Type Found (I) RIM Object Configuration is: <> Database Vendor: DB2 <> Database Home: /usr/lpp/db2_07_01 <> Database Name: itm_db <> Database User: db2inst1 <> DB2 Instance Home: /home/db2inst1 (I) The RIM Host Trace is turned OFF==> Testing RIM Object (itm_rim_tdw009) Connectivity... (I) Attempting to Connect... (I) RIM Object (itm_rim_tdw009) is Working Properly==> Retrieving Data Collector Configuration... (I) RIM Object (itm_rim_tdw009) Matches the Data Collector Configuration (I) The XML files will be processed 6 (default) times before being archived. (I) The time between two consecutive requests of data upload is 1 (default) minutes. (I) Data will be uploaded 30 minutes past the full hour. (I) Old data will be removed from the ITM database at 0 (default) o'clock. (I) The ITM database will not retain data older then 30 (default) days.==> Retrieving Middle Layer Trace Configuration... (I) The profile distribution trace is at level 1 (default) and has a size of 1 (default) bytes.==> Retrieving the Status of the EP cache... (I) Cache status is: Alive for Endpoint: tdw001 (I) Cache status is: DMEngineOff for Endpoint: tdw002


(I) Cache status is: Alive for Endpoint: tdw003 (I) Cache status is: DMEngineOff for Endpoint: tdw004 (I) Cache status is: Alive for Endpoint: tdw005 (I) Cache status is: Alive for Endpoint: tdw006 (I) Cache status is: Alive for Endpoint: tdw007 (I) Cache status is: Alive for Endpoint: tdw008 (I) Cache status is: Alive for Endpoint: tdw009 (I) Cache status is: Alive for Endpoint: tdw010 (I) Cache status is: DMEngineOff for Endpoint: tdw011

==> Retrieving Info about the Queued Requests... (I) Endpoint="tdw001" LastCall="Wed Aug 13 14:39:21 CDT 2003" (I) Endpoint="tdw002" LastCall="Wed Aug 13 14:39:21 CDT 2003" (I) Endpoint="tdw003" LastCall="Wed Aug 13 14:39:21 CDT 2003" (I) Endpoint="tdw004" LastCall="Wed Aug 13 14:39:21 CDT 2003" (I) Endpoint="tdw005" LastCall="Wed Aug 13 14:39:21 CDT 2003" (I) Endpoint="tdw006" LastCall="Wed Aug 13 14:39:21 CDT 2003" (I) Endpoint="tdw007" LastCall="Wed Aug 13 14:39:21 CDT 2003" (I) Endpoint="tdw008" LastCall="Wed Aug 13 14:39:21 CDT 2003" (I) Endpoint="tdw009" LastCall="Wed Aug 13 14:39:21 CDT 2003" (I) Endpoint="tdw010" LastCall="Wed Aug 13 14:39:21 CDT 2003" (I) Endpoint="tdw011" LastCall="Wed Aug 13 14:39:21 CDT 2003"==> Done.

======================================================== ================== Analysis Completed ================== ========================================================

The itmchk.sh report shows the following information:

� RIM Object configuration� RIM connection status� Data Collector configuration parameters (see “Installing the ITM WEP data

collector component” on page 232)� Middle Layer Trace Configuration (used for debugging)� Status of the endpoint cache� The last data upload request to each endpoint


6.9.2 Manual checking of ITM data collectionIn this section we show how to check manually every step of ITM data collection. We suggest the following approach:

1. Retrieve the date of last data upload into ITM database2. Test the RIM connection3. Check the status of distributed ITM resource models4. Review data collection parameters5. Analyze trace files

Retrieving date of last data upload into ITM databaseConnect to ITM_DB, as shown in Example 6-8.

Example 6-8 Retrieving the date of last data upload into ITM database

db2 => connect to ITM_DB user db2inst1 using <password>


Database server = DB2/NT 7.2.0 SQL authorization ID = DB2INST1 Local database alias = ITM_DB

Then select the date of the last data upload:

db2 => select max(timekey_dttm) from metricsdata

1--------------------------2003-08-12-05.00.21.000000


The date of the last data upload allows an estimation of the time frame in which the data collection process stopped. This estimate can be very useful when examining the monitoring trace files to help in understanding the reasons for data upload failure.

You can also check the hostnames of all endpoints that provided data, as shown in Example 6-9.

Example 6-9 Names of the endpoints collecting data

db2 => select host_name from endpoints

HOST_NAME----------------------------------------------------------


tdw001tdw002tdw003tdw004tdw005tdw006tdw007tdw008tdw009tdw010tdw011


This information can be used to track which monitored endpoints are not providing data.

Testing the connection between RIM host and ITM databaseOnce you have ascertained that ITM_DB does not receive data from monitored endpoints, you should check the connection between the RIM host and the database. From any managed node of your Tivoli region having Tivoli environment configured, run the wrimtest -l <rim_object> command as in Example 6-10. If you do not remember the name of your RIM object, use the command wlookup -ar RIM to show all the RIM objects defined in your Tivoli environment.

If the wrimtest command detects a connection failure, you should review all the parameters specified during RIM object creation.

Example 6-10 wrimtest command output

# wrimtest -l itm_rim_tdw009Resource Type : RIMResource Label : itm_rim_tdw009Host Name : tdw009User Name : db2inst1Vendor : DB2Database : itm_dbDatabase Home : /usr/lpp/db2_07_01Server ID : tcpipInstance Home : /home/db2inst1Instance Name :Opening Regular Session...Session OpenedRIM : Enter Option >


Often a connection failure is simply caused by a change in the database password without the needed update of the RIM object. To change the RIM password use the command:

wsetrimpw rim_name <old_password> <new_password>

Checking the status of distributed resource modelsThe command wdmlseng -e <endpoint_name> can be used to list the status of all resource models distributed to the specified endpoint. Only the resource models with status Running are collecting data on the endpoint.

Example 6-11 Status of resource models distributed on an endpoint

# wdmlseng -e tdw006

Forwarding the request to the engine...

The following profiles are running:

tmw2kDefProfile#tdw009-region TMW_EventLog :Running TMW_PhysicalDiskModel :Running TMW_TCPIP :Running TMW_LogicalDisk :Running TMW_MemoryModel :Running TMW_Process :Running TMW_Processor :Running

To change the status of resource models, modify the original monitoring profile using the Tivoli desktop and redistribute it to the endpoint.

If the monitoring engine is not running correctly on the endpoint, you can try to restart it using the command:

wdmcmd -restart -e <endpoint_name>

Reviewing data collection parametersThe data collector process populates the ITM_DB database with the monitoring value metric from the endpoints at every interval you specified in the wdmcollect command.

The collected metrics are then temporarily cached in the managed node in the $DBDIR/dmml/tedw/<endpoint> directory. The file is a zip file that contains the metrics information in XML format. Once you issue the wdmcollect command, after the interval is expired, you can check whether the file is created.


If your ITM database does not receive data even if your resource models are correctly running on the monitored endpoints and the connection between RIM host and database is working, you should verify that:

1. Enable Data Logging and TEDW Data in the Logging option of your monitoring profiles are checked (see “Activate data collection” on page 237).

2. The data collection parameters are correct (use wdmconfig -m <nodename> -G datacollector* command).

3. The gateway data collection process was started with the wdmcollect command (use wdmcollect -m all -q to list the active collection processes for all managed nodes).

Checking trace filesIf you went through all the previous steps but your ITM database still does not receive data, you can examine the log files. There are several trace and log files related to the data collector process under $DBDIR/AMW/logs directory on the managed nodes:

� msg_DataCollector.log� trace_tmnt_datacollector_eng1.log� trace_tmnt_hb_eng1.log� trace_tmnt_profile_core1.log� trace_tmnt_rimh_eng1.log� trace_tmnt_rm_eng1.log� trace_tmnt_task_eng1.log

The message file contains operational messages, while the trace files contain error messages. Each function of the data collector has its own trace file. These functions are the main data collector engine, heartbeat engine, RIM interface, resource model engine, and task execution engine.

For TEDW interface, the important files are msg_DataCollector.log and trace_tmnt_rimh_eng1.log. In Example 6-12 we show the output of msg_DataCollector.log for a successful data upload from endpoint tedw1: you can easily follow all the steps from the distribution to the upload to the database.

Example 6-12 msg_DataCollector.log

<F>1035246605000<F>Tue Oct 22 00:30:05 2002 GMT<F>AMW<F>DataCollector<F>eastham<F>12488<F>AMW<F> - AMW0181I -The distribution with ID '19431921771035246602' succeeded on the endpoint 'tedw1'.

<F>1035247476000<F>Tue Oct 22 00:44:36 2002 GMT<F>AMW<F>DataCollector<F>eastham<F>12488<F>AMW<F> - AMW0189I -


The data related to the request '520207df' has been successfully received<F>1035247480000<F>Tue Oct 22 00:44:40 2002 GMT<F>AMW<F>DataCollector<F>eastham<F>19074<F>AMW<F> - AMW0202I -The file '/var/spool/Tivoli/eastham.db/dmml/tedw/tedw1/1035246605.zip' is going to be processed for upload.

<F>1035247481000<F>Tue Oct 22 00:44:41 2002 GMT<F>AMW<F>DataCollector<F>eastham<F>19074<F>AMW<F> - AMW0198I -The data related to file: '/var/spool/Tivoli/eastham.db/dmml/tedw/tedw1/1035246605.zip.dir/ITM_WH@2002#10#22#0#20.xml' have been successfully loaded into the DataBase

In Example 6-13, instead, we show the occurrence of a failure in the data collection process. In this case the problem was caused by a stop of Tivoli Framework processes on the RIM host.

Example 6-13 trace_tmnt_rimh_eng1.log

<F>1037723936000<F>Tue Nov 19 16:38:56 2002 GMT<F>AMW<F>datacollector<F>eastham<F>18332<F>MIN<F>../../../../../src/objects/DataCollector/platform/StoreData/RIMConnectionHandler.cxx<F>RIMConnectionHandler::connect()<F>537026664<F>'FRWSL0005E A communications failure occurred: FRWOG0014E destination dispatcher unavailablePlease refer to the TME 10 Framework Planning and Installation Guide, "TME Maintenance and Troubleshooting" for details on diagnosing communication errors or contact your Tivoli support provider.

The IBM Tivoli Monitoring has important log files also at the endpoints:

� For Windows platform, most information is stored in Tmw2k.log file under the $LCF_DATDIR/LCFNEW/Tmw2k directory.

� For the UNIX platform, the logs are under the $LCF_DATDIR/LCFNEW/AMW/logs directory. These log files are:

– msg_dmxengine.log– trace_dmxengine.log– trace_dmxeu.log– trace_dmxntv.log

As for the managed nodes logs, the message file contains operational messages, while the trace files contain error messages.

In Example 6-14 we show how the trace_dmxengine.log reports a problem with a distributed resource model (in this case, the Network Interface resource model in ITM_Unix_monitor profile).


Example 6-14 trace_dmxengine.log

<F>1036710042642<F>Thu Nov 07 17:00:42 CST 2002<F>AMW<F>Engine<F>yarmouth<F>12240<F>MIN<F>ReferenceModel profile=ITM_Unix_monitor#eastham-region model=DMXNetworkInterface<F><F>Thread[TmrSrvAction_RMTimer,5,main]<F>No NICs found!<F>None

For further information about IBM Tivoli Monitoring troubleshooting, refer to Appendix C of IBM Tivoli Monitoring User’s Guide Version 5.1.1, SH19-4569-01.


Chapter 7. IBM Tivoli Storage Manager Warehouse Enablement Pack

This chapter provides an overview of the tasks to be performed to install IBM Tivoli Storage Manager, Version 5.2 Enablement Pack for Tivoli Data Warehouse 1.2. We cover the following topics:

� “Case study overview” on page 298� “IBM Tivoli Storage Manager WEP overview” on page 299� “Prerequisites” on page 300� “Installing and configuring ITSM WEP 5.2” on page 301� “IBM Tivoli Storage Manager ETL processes” on page 314� “Testing, scheduling, and promoting the ETLs” on page 320� “Reporting” on page 322

7


7.1 Case study overviewTable 7-1 gives a brief summary of the distributed Tivoli Data Warehouse 1.2 environment we started with in order to install the IBM Tivoli Storage Manager Warehouse Enablement Pack Version 1.1.0.

Table 7-1 Environment for NetView integration

Hostname Component Database Operating system

TDW003 TDW Control Server 1.2

DB2 Server 7.2FP10a

W2000 FP4

TDW004 TDW Central Warehouse 1.2(remote agent site)

DB2 Server 7.2FP10a

W2000 FP4

TDW009 TDW Data Mart 1.2 DB2 Server 7.2FP10a

AIX 5.2

TDW002 Crystal Enterprise Server 9

DB2 Client 7.2FP10a

W2000 FP4

TSMSRV01 IBM Tivoli Storage Manager 5.2

- W2000 FP4


Figure 7-1 gives a brief summary of the distributed Tivoli Data Warehouse environment used in this chapter.

Figure 7-1 TDW 1.2 - distributed deployment scenario

7.2 IBM Tivoli Storage Manager WEP overviewThe ITSM WEP extracts historical information from one or more IBM Tivoli Storage Manager servers and loads it into the Tivoli Data Warehouse 1.2 environment. Subsets of the historical information stored in the central data warehouse database may be loaded into data mart databases to enable reporting in specific areas of interest.

IBM Tivoli Storage Manager 5.2

Windows 2000 ServerHostname: TSMSVR01

TDW Control Server


TWH_MD





TWH_CDW

Data Mart


Data MartTWH_MART

Agent site

Chapter 7. IBM Tivoli Storage Manager Warehouse Enablement Pack 299

The IBM Tivoli Storage Manager Warehouse Enablement Pack extracts information related to the following components:

� IBM Tivoli Storage Manager servers� Nodes that are managed by the servers� Filespaces belonging to the nodes� Amount of data stored on the servers on behalf of nodes and their filespaces� Storage pools in which the data is stored.

IBM Tivoli Storage Manager itself does not record historical information about these entities. As a result, the IBM Tivoli Storage Manager Warehouse Enablement Pack records the current state of each entity as it extracts information from the IBM Tivoli Storage Manager servers. The current state, collected regularly over a period of time, becomes the historical record of the IBM Tivoli Storage Manager activity.

7.3 PrerequisitesBefore installing the IBM Tivoli Storage Manager Warehouse Enablement Pack, you must install the following software:

� IBM Tivoli Storage Manager 5.2.

� IBM DB2 Universal Database Enterprise Edition, Version 7.2.

� IBM DB2 Universal Database Enterprise Edition, Version 7.2 Fix Pack 8e, 9, 10, or 10a.

� IBM DB2 Universal Enterprise for z/OS and OS/390, Version 7.

� Tivoli Data Warehouse 1.2.

� Crystal Enterprise Professional Version 9 for Tivoli.

� IBM DB2 Warehouse Manager, Version 7.2 Fix Pack 8 with its eFix, on the remote agent sites in case of a distributed deployment. (This is the minimum level of IBM DB2 Warehouse Manager supported by Tivoli Data Warehouse 1.2. Note that Fix Pack 10a is the recommended level and it is shipped with Tivoli Data Warehouse 1.2).

� IBM Tivoli Storage Manager 5.2 ODBC Driver.

The IBM Tivoli Storage Manager Warehouse Enablement Pack supports central data warehouses and data mart databases on DB2 UDB for z\OS and for OS\390, or on DB2 UDB for Windows and UNIX systems. Regardless of the platform on which data warehouses or data marts reside, the warehouse enablement pack doesn’t support multiple central data warehouse or data mart databases.


7.4 Installing and configuring ITSM WEP 5.2In this section we discuss in detail the installation and configuration of the ITSM WEP.

The steps required for the IBM Tivoli Storage Manager warehouse enablement pack installation are as follows:

� Ensure that the IBM Tivoli Storage Manager servers are properly configured.

� Install and configure the IBM Tivoli Storage Manager 5.2 ODBC on the control server or on the remote agent sites that will access the IBM Tivoli Storage Manager databases (Windows only).

� Back up the Tivoli Data Warehouse 1.2 environment.

� Install the IBM Tivoli Storage Manager Warehouse Enablement Pack processes.

7.4.1 Changes required on the IBM Tivoli Storage Manager serversBefore installing the IBM Tivoli Storage Manager Warehouse Enablement Pack, it is necessary to configure the IBM Tivoli Storage Manager servers from which information will be extracted. The following items must be configured on the servers for the warehouse enablement pack to operate correctly.

� The servers’ SERVERNAME must be changed from the default value to a value that is distinct from other servers. The SERVERNAME value is used to distinguish between the different servers; otherwise, multiple servers with the same name will result in corruption of the information stored in the central data warehouse.

� The servers’ SERVERHLADDRESS must be set to one of the following:

– The fully qualified TCP/IP hostname of the computer on which the server is running

– The IP address of the computer on which the server is running

7.4.2 Installing the IBM Tivoli Storage Manager ODBCIBM Tivoli Storage Manager uses a proprietary database. However, it provides a Windows ODBC client and an SQL ‘97 compatible interface by which information about the server may be extracted. As it extracts information from one or more ITSM servers, the central data warehouse ETL (ETL1) analyzes and then stores the information in the central data warehouse.


The data mart ETL (ETL2) extracts a subset of the information stored in the central data warehouse database and stores it in a data mart database. The data mart is optimized for reporting on specific areas of interest, and may be used by a number of different reporting tools.

In this section we will be installing the IBM Tivoli Storage Manager ODBC on the control center server, which in our case study environment is the TDW003 server.

Perform the following tasks to install the IBM Tivoli Storage Manager ODBC on the control server:

1. Locate the TSM520C_GA_ODBC_.exe file in the IBM Tivoli Storage Manager 5.2 installation Media.

2. Double-click the file, and on the Welcome screen, click Next.

3. Select a temporary directory to save installation files and click Next twice to confirm the installation.

4. Select the installation directory for ODBC and click Next.

5. Select Complete setup type and click Next, as shown in Figure 7-2.

Figure 7-2 ITSM ODBC Installation

6. Click Install to begin the installation and Finish to complete.

Restriction: Since the IBM Tivoli Storage Manager ODBC client runs only on Microsoft Windows platform, it is important that you don’t configure Tivoli Data Warehouse remote agent sites to run on non-Windows platform.


The Tivoli Data Warehouse warehouse pack installer cannot configure IBM Tivoli Storage Manager data sources. It will recognize an existing IBM Tivoli Storage Manager ODBC data source named ITSMSRC, and allow it to be used with the warehouse enablement pack.

To create the IBM Tivoli Storage Manager ODBC data source on the control server. perform the following steps:

1. Click Start ->Control Panel -> Administrative Tools -> Data Source (ODBC).

2. Go to the System DSN tab and click Add. Select TSM ODBC Driver and Click Finish: The configuration Panel is opened.

Figure 7-3 ITSM ODBC data source configuration panel

Fill the Data source name with ITSMSRC, fill the Administrator name with the IBM Tivoli Storage Manager administrator name and the TCP/IP with the full qualified hostname of IBM Tivoli Storage Manager server (the name must match with the SERVERHLADDRESS value configured on the IBM Tivoli Storage Manager), leave other parameters as default, and then click OK.

If you plan on using the IBM Tivoli Storage Manager Warehouse Enablement Pack with multiple data sources, then you may need to follow this process to configure the ODBC data sources:

� Create a dummy DB2 database on the control server and create as many ODBC connections to the dummy database as you have IBM Tivoli Storage Manager servers. Create the ODBC connections as if you were creating the IBM Tivoli Storage Manager ODBC connections (same name).


� Do not configure any IBM Tivoli Storage Manager ODBC connections to the real IBM Tivoli Storage Manager servers’ databases.

� Proceed with the installation of the IBM Tivoli Storage Manager warehouse enablement pack as described in the next section. When the installer prompts for the IBM Tivoli Storage Manager data sources, configure the data sources to point to the ODBC connections to the dummy DB2 database.

� After the IBM Tivoli Storage Manager warehouse enablement pack installation has completed, replace the DB2 data sources with IBM Storage Manager ODBC data sources with the same name.

� Manually update the user name and passwords associated with each source in the DB2 Data Warehouse Center.

7.4.3 Backing up the TWH databasesBefore installing any warehouse enablement pack (WEP), it is highly recommended to back up all the warehouse databases to ensure that you can return to a known valid state if you encounter an unrecoverable error during installation.

In our case study, we have to back up the three databases, TWH_CDW, TWH_MART, and TWH_MD (if you use different configurations, refer to Chapter 2, “Planning for Tivoli Data Warehouse 1.2” on page 27 for warehouse database naming conventions).

In order to do so, we created a local directory structure on our DB2 server to store our backups by creating the following three directories:

� C:\tdwbackups\pre-itsm\twh_cdw� C:\tdwbackups\pre-itsm\twh_mart� C:\tdwbackups\pre-itsm\twh_md



db2stop forcedb2start


db2 backup db TWH_CDW to C:\tedwbackups\pre-itsm\twh_cdw


db2 backup db TWH_MART to C:\tedwbackups\pre-itsm\twh_mart



db2 backup db TWH_MD to C:\tedwbackups\pre-itsm\twh_md

5. Restart Warehouse logger and Warehouse server services on the control server.

7.4.4 IBM Tivoli Storage Manager WEP installationTo install the AMX warehouse pack, we completed the following steps on the control server:

1. Click Start -> Programs -> TDW -> Install a Warehouse Pack, this starts the warehouse pack installation wizard (see Figure 7-4 on page 305). In the Welcome window, click Next.

Figure 7-4 Install a Warehouse Pack window

2. The following window (Figure 7-5) shows the location of the Tivoli common logging directory which will contain all TDW log files. In our installation we use the default location C:\Program Files\ibm\Tivoli\common. Click Next.


Figure 7-5 Tivoli Common Logging Directory window

3. In the window shown in Figure 7-6, click Add to add the IBM Tivoli Storage Manager Warehouse Enablement Pack.

Figure 7-6 Add Warehouse Pack window


4. In the Location of installation properties file window, as shown Figure 7-7, specify the location of the IBM Tivoli Storage Manager warehouse enablement pack installation properties file, twh_install_props.cfg. You can find this file in the IBM Tivoli Storage Manager 5.2 Warehouse Enablement Pack media, under the tedw_apps_etl\AMN directory. Click Next.

Figure 7-7 Location of installation properties


5. The installer will prompt for the data mart database to be used by the processes of the IBM Tivoli Storage Manager Warehouse Enablement Pack. It also prompts for the remote agent site that will run the ETL2 processes, as shown in Figure 7-8. On our case study scenario, the data mart is on TDW009 and we will be using the default agent site on the control server on TDW003.

Figure 7-8 Data mart and remote agent site settings


6. The installer will prompt for the central data warehouse database to be used by the processes of the IBM Tivoli Storage Manager Warehouse Enablement Pack. It also prompts for the remote agent site that will run the ETL1 processes, as shown in Figure 7-9. In our case study scenario, the central data warehouse is on TDW004 and we will be using the default agent site on the control server on TDW003.

At this time you can opt to perform the scheduling settings for the ETL1 processes. In case you opt to do so, the installation process will schedule the processes to run at the specified time and promote the processes to Production status. You can also opt not to schedule the ETLs at installation time and perform the foregoing tasks manually.

The installer will also define the user authority for each one of the warehouse sources and targets processes.

Figure 7-9 Central data warehouse and remote agent site settings


7. The installer will investigate the existence of an IBM Tivoli Storage Manager ODBC connection named ITSMSRC. Click Edit to specify its settings, as shown in Figure 7-10.

Figure 7-10 Editing IBM Tivoli Storage Manager ODBC settings

8. As shown in Figure 7-11, set the User ID of IBM Tivoli Storage Manager Administrator and Password. The Administrator name should be the same as the IBM Tivoli Storage Manager ODBC system data source you created prior to the installation. Click Next. The installer will test the ODBC connection and return to ODBC Data Source Properties Panel. Click Next again.


Figure 7-11 ITSM ODBC Settings

9. The installation menu window (Figure 7-12) now lists the IBM Tivoli Storage Manager Warehouse Enablement Pack. Select it and click Next to continue.

Figure 7-12 Installation menu window


10.Click Install in the summary window (Figure 7-13) to start the IBM Tivoli Storage Manager Warehouse Enablement Pack installation.

Figure 7-13 Installation summary window

11.View the progress of installation through the messages that are shown in windows until its completion. The final installation window (Figure 7-14) contains either a successful completion notice or messages describing problems. Make sure the window does not list any warnings or errors, and then click Next. If warnings are listed, check the logs to ensure that the warnings can safely be ignored. Click Finish to exit the wizard.


Figure 7-14 Installation Progress and Completion window

7.4.5 Defining the authority to the warehouse sources and targetsTivoli Data Warehouse 1.2 has simplified the installation process of warehouse enablement packs to the extent that there is no need to define the user authority to the warehouse sources and targets of warehouse enablement packs’ processes developed for Version 1.2. The installer sets the correct user ID and password information required for each source and target.

However, we recommend for you to certify that the processes have proper authorization by following the steps defined in the previous chapters of this redbook describing warehouse enablement packs developed for Version 1.1.


Table 7-2 provides a list of the warehouse sources and targets whose authority should be checked.

Table 7-2 ITSM WEP Warehouse Object Names

7.5 IBM Tivoli Storage Manager ETL processes Figure 7-2 shows that the IBM Tivoli Storage Manager Warehouse Enablement Pack has the following processes:

� ANR_C05_ETL1_Process� ANR_C10_EXPServer_Process� ANR_M05_ETLS2_Process

Warehouse targets ANR_TWH_CDW_TargetANR_TWH_MART_Target

Warehouse sources ANR_ITSMRC_SourceANR_TWH_CDW_SourceANR_TWH_MART_Source

Subject area ANR_IBM_Tivoli_Storage_Manager_v1.1.0_Subject_Area

Processes ANR_C05_ETL1_ProcessANR_C10_EXPServer_ProcessANR_M05_ETLS2_Process

Steps ANR_C05_S010_PreextractANR_C05_S020_ExtractANR_C05_S030_TransformANR_C05_S040_SRVR_LOADANR_C05_S050_STGP_LOADANR_C05_S060_NODE_LOADANR_C05_S070_FILESP_LOADANR_C05_S080_OCCUP_LOAD

ANR_C10_S010_EXPServer

ANR_m05_s010_spbuildmartANR_m05_s020_sprollupANR_m05_s030_spupdatestatsANR_m05_s040_fspuildmartANR_m05_s050_fsrollupANR_m05_s060_fsupdatestats


7.5.1 ANR_C05_ETL1_ProcessThis process is the main central data warehouse ETL. It extracts information from an IBM Tivoli Storage Manager server and loads it into the central data warehouse database.

Run this process once each 24 hour period to collect information about the previous day’s processing. You should choose a time of low activity on the ITSM server in which to run this process. For example, you might choose to schedule the process in the early morning, after your nightly backups have completed. but before the daily server maintenance processes begin.

Figure 7-15 illustrates the Process Model of ANR_C05_ETL1_Process. In order to obtain the content of Figure 7-15 on page 316, perform the following steps:



3. Log into the IBM DB2 Data Warehouse Center utility using the local DB2 administrator user ID, in our case, db2admin.

4. In the Data Warehouse Center Window, expand Subject Areas -> Processes and double-click ANR_ITSM_C05_ETL1_Process.

Similar procedures can be followed for the other processes, ANR_C10_EXPServer_Process and ANR_M05_ETLS2_Process.

Important: Do not modify or make any changes on the Process Model. If you are prompted to Save, click No.


Figure 7-15 Sample of Process Model ANR_C05_ETL1_Process

The process ANR_C05_ETL1_Process has the following steps.

� ANR_c05_s010_preextract

This step prepares for the subsequent extraction step by creating staging tables in the central data warehouse database. It is a separate step so that ANR_C05_S020_Extract may be run multiple times to extract information from multiple IBM Tivoli Storage Manager servers.


� ANR_c05_s020_extract

This step extracts information from an IBM Tivoli Storage Manager server and stores it in staging tables in the central data warehouse. Typically, there is a one-to-one correspondence between the table from which data is extracted and the staging table the central data warehouse. For example, information extracted from the IBM Tivoli Storage Manager server’s STATUS table is stored in a table called ANR.STG_STATUS in the central data warehouse database.

� ANR_c05_s030_transform

This step transforms some of the data in the staging tables created by ANR_c05_s020_extract for use by subsequent steps. In particular, this step analyzes the information stored in the server’s SERVER_HLA field in the STATUS table to determine if it is a fully qualified TCP/IP host name, a TCPI/IP address, or some other value. A new staging table, ANR.STG_SERVER is created by this step.

� ANR_c05_s040_srvr_load

This step loads information about the IBM Tivoli Storage Manager server and the computer on which is running into the central data warehouse. Information about the server and its host computer is obtained from two staging tables: ANR.STG_STATUS and ANR.STG_SERVER.

This step inserts or updates, as necessary, component entries in the central data warehouse database. It defines a parent-child relationship between the host computer and the server. Finally, it inserts or updates any attributes that are associated with the components.

� ANR_c05_s050_stgp_load

This step loads information about the server’s storage pools into the central data warehouse. Information primarily comes from one staging table, ANR.STG_STGP.

This step inserts or updates, as necessary, component entries in the central data warehouse database. It defines a parent-child relationship between the server and storage pools. Storage pools that were removed from the server are detected and expired from the central data warehouse database. This step inserts or updates any attributes that are associated with the components and loads new measurements collected since the last time the process ran.

� ANR_c05_s060_node_load

This step loads information into the central data warehouse about the client nodes that are registered to the IBM Tivoli Storage Manager server. Information primarily comes from one staging table, ANR.STG_NODE.


This step inserts or updates, as necessary, component entries in the central data warehouse database. It defines a parent-child relationship between the server and the nodes. Nodes that were removed from the server are detected and expired from the central data warehouse database. This step inserts or updates any attributes that are associated with the components and loads new measurements collected since the last time the process ran.

� ANR_c05_s070_filesp_load

This step loads information about the client nodes’ file spaces into the central data warehouse. Information primarily comes from one staging table, ANR.STG_FILESP.

This step inserts or updates, as necessary, component entries in the central data warehouse database. It defines a parent-child relationship between the file space and its owning node. File spaces that were removed from the server are detected and expired from the central data warehouse database. This step inserts or updates any attributes that are associated with the components and loads new measurements collected since the last time the process ran.

� ANR_c05_s080_occup_load

This step loads client occupancy information into the central data warehouse. Occupancy information describes the amount of server storage that is being used for a given node’s file spaces. It is broken down by node, file space, storage pool and data type (backup, archive, or space management). Information primarily comes from one staging table, ANR.STG_OCCUP.

The IBM Tivoli Storage Manager warehouse enablement pack uses an abstract component type to represent occupancy information. The component type, ANR_FS_OCCUPANCY, represents the server storage being used by a file space in a particular storage pool for a given type of file. It is a child in a parent-child relationship with the file space. ANR_FS_OCCUPANCY components are named after the storage pool that owns the storage and the type of storage being described. Names are generated by concatenating the storage pool name with the storage type and separating the two with a colon.

This step inserts or updates, as necessary, component entries in the central data warehouse database. It defines a parent-child relationship between a file space and the ANR_FS_OCCUPANCY component. Occupancy components that were removed from the server (due to storage pool migration, for example) are detected and expired from the central data warehouse database. This step inserts or updates any attributes that are associated with the components and loads new measurements collected since the last time the process ran.

If invalid data is detected, the ETL process creates the exception table ANR.EXCEPT_SRVR.


7.5.2 ANR_C10_EXPServer_ProcessThis process is used to expire information about IBM Tivoli Storage Manager servers and their subcomponents from the central data warehouse.

Run this process manually whenever information about a specific IBM Tivoli Storage Manager server must be expired from the central data warehouse. Do not schedule this process to run automatically, as it will expire data from the central data warehouse that you may wish to keep.

The ANR_C10_EXPServer_Process process expires servers based on the contents of table ANR.EXPServer_List. ANR.EXPServer_List contains a single column, SERVER_NAME, which names the server to be expired. Multiple servers may be expired by inserting a row for each server into the ANR.EXPServer_List table. If ANR.EXPServer_List contains no rows, then ANR_C10_EXPServer_Process will not expire any servers.

This process has the following step:

� ANR_c10_s010_expserver

This step expires each of the servers listed in the ANR.EXPServer_List table. The servers and their attributes are marked as expired. Then all of the servers’ subcomponents and their attributes are expired, as are relationships between the components. After all components related to a given server have been expired, the entry for that server is deleted from the ANR.EXPServer_List table.

7.5.3 ANR_M05_ETL2_ProcessThis process loads data from the central data warehouse into the Storage Pool Occupancy Data Mart and the Filespace Occupancy Data Mart.

Run this process once very 24 hours to load new data into the Storage Pool Occupancy and Filespace Occupancy data marts. This process should be run only after the central data warehouse process ANR_C05_ETL1_Process has completed. This process has the following steps:

� ANR_m05_s010_spbuildmart

This step loads data from the central data warehouse into the Hourly Storage Pool star schema in the Storage Pool Occupancy Data Mart.

� ANR_m05_s020_sprollup

This step aggregates the hourly data loaded by the ANR_m05_s010_spbuildmart step and loads it into Daily, Weekly, Monthly, Quarterly and Yearly star schemas.


� ANR_m05_s030_spupdatestats

This step updates statistics about fact tables loaded in the previous steps. The statistics are used by reports to determine which set of facts are the most recent for any given server.

� ANR_m05_s040_fsbuildmart

This step loads data from the central data warehouse into the Hourly Filespace star schema in the Filespace Occupancy Data Mart.

� ANR_m05_s050_fsrollup

This step aggregates the hourly data loaded by the ANR_m05_s040_fsbuildmart step and loads it into Daily, Weekly, Monthly, Quarterly and Yearly star schemas.

� ANR_m05_s060_fsupdatestats

This step updates statistics about fact tables loaded in the previous steps. The statistics are used by reports to determine which set of facts are the most recent for any given server.

7.6 Testing, scheduling, and promoting the ETLsTivoli Data Warehouse 1.2 has simplified the installation process of warehouse enablement packs to the extent that the scheduling and promoting of ETLs developed for Version 1.2 are set during the warehouse enablement pack installation time. You can still demote the ETLs to Test mode and test the execution of the processes in order to verify accuracy. The process of promoting and demoting of ETLs is similar to the steps defined in previous chapters of this redbook describing warehouse enablement packs developed for Version 1.1.

7.6.1 ETL data collection verificationOnce the processes run at scheduled time, or run manually (in test mode), you can verify data collection on the data mart database, as follows:

1. On the control server, start the DB2 Control Center tool: Click Start -> Programs -> DB2 -> Control Center. See the example in Figure 7-16.

2. Type the User ID and password, when prompted, in our case it was db2inst1.

3. Double-click Tables, on the right hand pane, all tables are displayed. Select Table D_NODE, right-click, and select Sample Contents.

You will see a window displaying the content. This verifies your ETL execution as successful.


Figure 7-16 Sample Content of Table D_NODE


7.7 ReportingIn this section we show how to set up, configure, and use some of the reports provided by the IBM Tivoli Storage Manager 5.2 Warehouse Enablement Pack.

7.7.1 Available reportsHere is a list of predefined reports shipped with the IBM Tivoli Storage Manager 5.2 Warehouse Enablement Pack:

� How Has Clients Use of Server Storage Changed Over Time?� How Has Clients Use of Server Storage Changed Over Time by Platform?� How Has My Server Storage Space Utilization Changed Over Time?� Which Clients are Using the Most Server Storage/

Crystal Enterprise Professional Version 9 for Tivoli has reduced configuration options in comparison to a full Crystal license. If the reports shipped with IBM Tivoli Storage Manager 5.2 Warehouse Enablement Pack do not match your needs and you want to develop additional reports, you have to upgrade your Crystal Enterprise installation.

7.7.2 Accessing the Crystal ePortfolioAll reports provided by Tivoli Data Warehouse 1.2 must be accessed using the ePortfolio feature of the Crystal Enterprise. The ePortfolio can be accessed from the Crystal Enterprise launchpad. In order to access the Crystal Enterprise launchpad, start an Internet browser and open the following URL:






In this section, we concentrate on viewing IBM Tivoli Storage Manager 5.2 reports and we do not explain the configuration of Crystal Enterprise to its full extent. For details on configuration and administration tasks, refer to the following manuals shipped with the product:



From the Crystal Enterprise Launchpad, proceed by selecting the ePortfolio link, which will bring you to the window shown in Figure 7-18. In the top bar, you can see that we are authorized as user guest. By default, the guest user has no access to the NetView reports, as indicated by the words No Folders on the left side of the window.




To log in as the Tivoli user ID, select the Log On button in the upper right corner of the ePortfolio window in Figure 7-18 on page 324. The Log On window as shown in Figure 7-19 is presented. The Tivoli user ID has no password by default. We use the Enterprise authentication method as we have specified during the Crystal Enterprise installation.



After entering the required data, select Log On to proceed. Now we are back at the ePortfolio window in Figure 7-20, but now with user Tivoli authority. Instead of No Folders in the guest users ePortfolio window in Figure 7-18, there is now a link visible with the name IBM Tivoli Storage Manager, Warehouse Enablement Pack in the Tivoli user ePortfolio window in Figure 7-20.



We follow this link by selecting IBM Tivoli Storage Manager, Warehouse Enablement Pack and proceed to the IBM Tivoli Storage Manager 5.2 reports as shown in Figure 7-21. All reports provided by the IBM Tivoli Storage Manager 5.2 warehouse enablement pack are listed there.

Figure 7-21 Crystal Enterprise 9 - available reports for ITSM


In order to obtain the reports, select the desired report, for example, How Has Clients use of Server Storage Changed Over Time?, and select Schedule, as shown in Figure 7-22.

Figure 7-22 Scheduling Operating System Busiest System report


The schedule report panel starts. In order to run the reports at this time, select Now under the Run Report option. As this report requires additional parameters, such as time frame, select Parameters under the Customize your Options option, as shown in Figure 7-23.

Figure 7-23 Crystal Enterprise 9 - parameters


Figure 7-24 shows the selection of parameters for the report. Select Schedule when ready to run the report.

Figure 7-24 Crystal Enterprise 9 - Parameters for the report

Because we selected to run this report now, the report is scheduled immediately and the reports history window is opened. The just-scheduled report will run and its initial status is set to Pending.

Note: The History window is not updated automatically. Press the Refresh button to view the current state.


To view successful generated reports from the history window, click the link Instance Time in the left column of the table to view the associated report. This report is shown in Figure 7-25.

Figure 7-25 How Has Clients use of Server Storage Changed Over Time?

Next we show some more examples of reports provided by the IBM Tivoli Storage Manager 5.2 Warehouse Enablement Pack.


Figure 7-26 displays a sample report of a single server called AMORRIS on How Has Clients Use of Server Storage Changed Over Time? It shows how to drill down to a client.

Figure 7-26 How Has Clients Use of Server Storage Changed Over Time?


Figure 7-27 displays a sample report of all servers on How Has Clients Use of Server Storage Changed Over Time by Platform?

Figure 7-27 How Has Clients Use of Server Storage Changed by Platform?


Figure 7-28 displays a sample report of all servers on How Has My Server Storage Space Utilization Changed Over Time?

Figure 7-28 How Has My Server Storage Space Utilization Changed Over Time?


Figure 7-29 displays a sample report of all servers on Which Clients Are Using the Most Server Storage?.

Figure 7-29 Which Clients are Using the Most Server Storage?


Part 3 Appendixes

Part 3


Appendix A. IBM DB2 UDB administration for other relational DBAs

DB2 is the main RDBMS product from IBM. It is a top level RDBMS competing with Oracle, IBM Informix, and Microsoft SQL Server in the market. Each of these products has its advantages and disadvantages; MS SQL Server®, for example, has a huge advantage in integrating enterprise running heterogeneous applications and Data Sources.

However, MS SQL Server has the disadvantage of executing only on Windows, while all the competitors are cross platform capable. Oracle® has the advantage of supporting parallel servers using its RAC, but the disadvantage of needing highly specialized professionals at a high cost and demanding considerable time to accomplish even trivial tasks. Informix® is easy to use and to maintain, but still cannot perform online reorganization.

IBM DB2 can provide all of these features and even more. It can connect to heterogeneous data sources through the IBM DB2 DataJoiner®; High Availability features using Partitioned Databases or Parallel Features using Sysplex Technology in z/OS, without the overhead presented by Oracle RAC, and can perform data reorganization online with little or no downtime.

Also, DB2 is highly scalable, providing data management from hand-held computers with the Everyplace® edition up to multi-terabyte Enterprise Servers with UDB ESE (old EEE), with the same easy-to-use framework.

A


Common DBA tasks

No matter what RDBMS you use, some tasks are common:

� Installing RDBMS� Creating databases� Setting up security� Allocating space� Optimizing performance

Installing RDBMS can be accomplished using setup instructions found in the manuals, and usually does not change in general from RDBMS to RDBMS, except in regard to parameters specific to that RDBMS. The differences between the DB2 and other RDBMS methods for accomplishing the tasks listed above are presented in the following sections of this appendix. Our discussion assumes that we are working on a UNIX platform.

Creating databasesAfter installing the RDBMS software, the next step is to create an instance or/and database to hold the data structure. The concept of database may vary among RDBMSs. For example, Oracle uses the term DATABASE to refer to the logical grouping of all objects storing the necessary data for an application as well as its internal catalog, and INSTANCE to refers to the memory, processes, and software pieces necessary to manage and access this data.

The same naming convention is used by DB2, while Informix, MSSQL, and Sybase ASE use a slightly different convention. Informix, MSSQL, and Sybase all use the term database to refer to the logical grouping of objects that can be accessed by an application. This database is separated from its catalog, which is a database itself. The grouping of architectural objects, such as memory, disk, processes, etc., in each of these RDBMSs is called a server.

Note: This appendix does not aim to compare the features of the different RDBMSs. It was written to help DBAs working with these RDBMS to perform the same tasks when working with IBM DB2.

Because some configurations use pure UNIX, without graphical extensions, we do not show the graphic tools that you can use to perform tasks in an easier way, except when we think this is necessary for clarity.


In general, an instance/server is configured during the install process, leaving to you the tasks of creating the database to hold data and setting up the protocols for providing access to the instance/server. The process to create a database, without using the graphical tools, is the same for each RDBMS.

1. Connect to the instance

2. Issue the CREATE DATABASE command, providing the location for datafiles, database size, etc.

Creating databases in IBM DB2Connect to the local machine as superuser (root), then set the DB2INSTANCE to the instance created by db2icrt at install time, which was named by you or the default db2inst1. In order to create a database named TIVDW, for example, Issue the following command:

db2 CREATE DATABASE TIVDW ON /DB2/tivdwda1/tivdw USING CODESET 1252 TERRITORY US COLLATE USING SYSTEM USER TABLESPACE MANAGED BY SYSTEM USING ('/DB2/tivdwda1/tivdw/d1sms') EXTENTSIZE 16 PREFETCHSIZE 16 CATALOG TABLESPACE MANAGED BY SYSTEM USING ('/DB2/tivdwda1/tivdw/SYSCATSPACE') EXTENTSIZE 8 PREFETCHSIZE 8 TEMPORARY TABLESPACE MANAGED BY SYSTEM USING ('/DB2/tivdwda1/tivdw/TEMPSPACE') EXTENTSIZE 32 PREFETCHSIZE 32;

The foregoing command shows many more parameters and attributes than the minimum required to create a database in DB2; this is done for illustration purposes only. The command will create our TIVDW database. After that command is completed, you can start to create the needed tables, Storage Polls, etc.

How we could create the same database on a heterogeneous RDBMS? Let's start with Oracle:

Creating databases in OracleSome of the options available to create a database in DB2 are not available in Oracle, and vice versa. The Oracle CREATE DATABASE command is far more complex than in IBM DB2: With Oracle you must specify a large number of parameters to get your database created. To create the same TIVDW Database in Oracle, perform the following steps:

1. Connect to sqlplus, using the SYS or INTERNAL account.

2. Issue the command:

CREATE DATABASE tivdwCONTROLFILE REUSE LOGFILE

Appendix A. IBM DB2 UDB administration for other relational DBAs 341

GROUP 1 ('/oracle/tivdw/logs/log1.log', /oracle/tivdw/logsmirror/log1.log') SIZE 50M, GROUP 2 ('/oracle/tivdw/logs/log2.log', '/oracle/tivdw/logsmirror/log2.log') SIZE 50M

MAXLOGFILES 50 MAXLOGHISTORY 100 MAXDATAFILES 100 MAXINSTANCES 2 ARCHIVELOG CHARACTER SET UTF8NATIONAL CHARACTER SET AL16UTF16DATAFILE '/oracle/tivdw/data/df1.dbf' SIZE 50M AUTOEXTEND ON,'/oracle/tivdw/logs/df2.dbf' SIZE 50M AUTOEXTEND ON NEXT 10M MAXSIZE UNLIMITEDDEFAULT TEMPORARY TABLESPACE temp_ts EXTENT MANAGEMENT LOCAL UNIFORM 32K SIZE 32KUNDO TABLESPACE undo_ts;

Notice the following differences between DB2 and Oracle:

1. DB2 does not specify LOG file destination, because its default location is the SQLOGDIR under the path you specified in the create database <dbname> on clause.

2. You can initialize the CATALOG, TEMP, AND USER tablespaces for DB2 by specifying their definitions inside the Create Database command.

3. There are two kinds of tablespaces in DB2, just as in Oracle: System Managed Tablespace and Database Managed Tablespace. See the following sections for a brief description of the differences. For the PREFETCHSIZE and EXTENTSIZE, refer to the section on Creating Tablespaces.

Creating databases in SybaseLet us create the same database on Sybase SQL Server or Sybase ASE (Adaptive Server), as it is now called.

Every Sybase installation has at least four databases: master, model, tempdb, and sybsystemprocs. Each of these databases has a role in keeping Sybase ASE up and running. The master database is the main database in a Sybase ASE installation, because it contains all the information for databases you create, logins you add to Sybase, SQL Server configuration and options, etc. Mapping to DB2, this is the System Catalog.


To create a database, you must connect to the server, using the isql program, and issue the following tasks:

1. Initialize disk device to be used by your database. This is done by mapping a disk or file to a logical device in the Sybase subsystem, by using the DISK INIT command:

DISK INIT NAME = "tvdw_data", PHYSNAME = "/dev/tvdw_data", VDEVNO = 2, SIZE = 25600

DISK INIT NAME = "tvdw_log", PHYSNAME = "/dev/tvdw_log", VDEVNO = 2, SIZE = 25600

These commands would initialize a device called tvdw_data, with 50MB to be used for data storage by our Tivoli Database and a device named tvdw_log, with the same size for transaction log. Information for this device would be inserted in sysdevices table in the master database.

2. After have initialized the device, you could issue the CREATE DATABASE command to create the database for use by Tivoli Data Warehouse, as in the following example:

CREATE DATABASE tivdwdbON tvdw_data = 50, LOG ON tvdw_log=50

Comparing the Sybase syntax to DB2 syntax is fruitless. All of the options in the DB2 syntax are missing in the Sybase syntax, except by database name. Also in DB2, you do not need to pre-initialize a database device. Sybase collation, for example, is server-wide, while DB2 collation is specified at database creation time.

Managing spaceDB2 organizes datafiles in objects called tablespaces, just as Oracle does. But Oracle datafiles are called containers in DB2. Sybase and MSSQL do not have a tablespace concept; Informix uses DBSpace for organizing physical objects.

DB2 space managementTo create a tablespace in DB2, you use the CREATE TABLESPACE command. It has a number of parameters that allows you to control the behavior of this tablespace, how space will be allocated, managed, and accessed. We've used some of these parameters while creating the initial database.


Consider the syntax:

db2 CREATE TABLESPACE TIVTS MANAGED BY DATABASE USING(FILE '/DB2/tivdwda1/tivdw/d2dms' 1G) EXTENTSIZE 16 PREFETCHSIZE 16

We are creating a tablespace with one container of 1GB (specified as 1000M). One of the options you can use, and which we left out, is PAGESIZE. This parameter can be used when you want to use a non-default pagesize, just in the same way Oracle 9i does with non-default block size caches. To use this non-default PAGESIZE in DB2, you have to create a new BUFFERPOOL configured to use the pagesize you want to, as in the following sample:

db2 CREATE BUFFERPOOL TIVBP SIZE 1000 PAGESIZE 16K.

Then you would modify our CREATE TABLESPACE example to use this bufferpool and the custom pagesize, in the following way:

db2 CREATE TABLESPACE TIVTS PAGESIZE 16K MANAGED BY DATABASE USING(FILE '/DB2/tivdwda1/tivdw/d2dms' 1G) EXTENTSIZE 16 PREFETCHSIZE 16 BUFFERPOOL TIVBP

When creating a tablespace in DB2, here are some considerations:

1. You should specify the space management system for that specific tablespace, except when you want to use SYSTEM MANAGED SPACE, the default. We are using DATABASE MANAGED SPACE because we want to have full control over space allocation. The key difference between these two types of tablespaces is that SMS has its space allocation managed by operating system, while DMS has its space allocation managed by the database manager.

When deciding what kind of tablespace to use, keep in mind that SMT cannot have containers added to it after it is created, and that you need to pay attention to OS limitations and allow enough disk space for the tablespace to grow automatically. For example, if your OS has a limit of 2 GB for files, and you need a tablespace of 128 GB, you have to create the tablespace with 64 containers. This is true for the initial CREATE DATABASE command when you specify one of the tablespace creation clauses and for the CREATE TABLESPACE command.

2. EXTENTSIZE is used to set the number of pages DB2 will write before skipping to the next container. This parameter is important for performance because of the way that DB2 stores data, balancing the writing of data between all containers in the tablespace in a cycling manner.

3. PREFETCHSIZE specifies the number of pages that will be read in read-ahead operations. You can use this parameter to reduce I/O in queries.

4. To be able to recover dropped tables, use the DROPPED TABLE RECOVERY option in the CREATE TABLESPACE or in the ALTER TABLESPACE clauses.


After creating the tablespace, you can use the ALTER TABLESPACE clause to modify tablespace options. You can change every option except PAGESIZE and EXTENTSIZE, so it is a good idea to think very carefully about these values. Use ALTER TABLESPACE to add, resize or extend a DMS tablespace's container.

For example, the following command will add a container with 1GB:

db2 ALTER TABLESPACE TIVTS ADD(FILE '/DB2/tivdwda1/tivdw/d3dms' 1G)

The following command will expand that container by 200 MB:

db2 ALTER TABLESPACE TIVTS EXPAND(FILE '/DB2/tivdwda1/tivdw/d3dms' 200M)

Oracle space managementTo create the same tablespace as discussed above, in Oracle, we could use a variation of the following command:

CREATE TABLESPACE TIVTSDATAFILE '/oracle/tivdw/data/df1.dbf' SIZE 200MMINIMUM EXTENT 500KDEFAULT STORAGE (INITIAL 128K NEXT 128K)LOGGING;

Or we could use a locally managed tablespace:

CREATE TABLESPACE TIVTSDATAFILE '/oracle/tivdw/data/df1.dbf' SIZE 200MEXTENT MANAGEMENT LOCAL UNIFORM SIZE 128K;

To add a new datafile to an Oracle tablespace, we use the ALTER TABLESPACE command:

ALTER TABLESPACE TIVTS ADD DATAFILE '/oracle/tivdw/data/df2.dbf' SIZE 200M AUTOEXTEND ON NEXT 1M MAXSIZE 1G;

And to resize it, we use the ALTER DATABASE command:

ALTER DATABASE DATAFILE '/oracle/tivdw/data/df2.dbf' RESIZE 300M;


Sybase space managementSybase has no tablespace concept. Space is allocated directly to databases, using DISK INIT and ALTER DATABASE commands. For example:

You use DISK INIT to map a disk area to a new database device, just as at database create time:

DISK INIT NAME = "tvdw_data1", PHYSNAME = "/dev/tvdw_data1", VDEVNO = 2, SIZE = 25600

Then you use the ALTER DATABASE command to increase the database size, using the new device:

ALTER DATABASE tivdwdb ON tvdw_data1 = 50

Creating objects in the databaseAfter you create the database, and allocate space for its objects, it is time to create the objects themselves. Objects include tables, indexes, views and stored procedures.

You issue SQL commands against the DB to create all of these objects. The SQL standard establishes these SQL commands.

Creating tables in DB2In its simplest form, you create tables in DB2 using the following command (this will create a table named DEPTO):

CREATE TABLE DEPTO (DEPTNO CHAR(3) NOT NULL, DEPTNAME VARCHAR(36) NOT NULL, MGRNO CHAR(6), ADMRDEPT CHAR(3) NOT NULL, PRIMARY KEY(DEPTNO)) IN DEPARTX INDEX IN DEPARTIDX


Creating tables in OracleTo create a table in an Oracle database, issue the following command (this is also going to create a table named DEPTO):

CREATE TABLE DEPTO (DEPTNO CHAR(3) NOT NULL, DEPTNAME VARCHAR(36) NOT NULL, MGRNO CHAR(6), ADMRDEPT CHAR(3) NOT NULL, PRIMARY KEY(DEPTNO))PCTFREE 10 PCTUSED 40 TABLESPACE DEPARTXSTORAGE (INITIAL 50K NEXT 50K MINEXTENTS 1 MAXEXTENTS 100FREELISTS 1)

Creating tables in SybaseTo create a table in a Sybase database, issue the following statements (this is also going to create a table named DEPTO):

CREATE TABLE DEPTO (DEPTNO CHAR(3) NOT NULL, DEPTNAME VARCHAR(36) NOT NULL, MGRNO CHAR(6), ADMRDEPT CHAR(3) NOT NULL, PRIMARY KEY(DEPTNO))ON DPTO_SEG

Additional table control parametersDB2, Oracle, and Sybase have a number of specialized parameters that control the behavior, storage, and memory management for tables. As an example, in DB2 you can define a SUMMARY table to hold a definition from full select using the following clause:

CREATE TABLE tbsummary AS(SELECT DEPTO.*, CURRENT TIMESTAMP AS TIMESTAMP FROM DEPTO)DEFINITION ONLY

This will create the table, but will not populate it. This is similar to the CREATE TABLE AS Oracle syntax or SELECT INTO clause in Sybase. However, DB2 also implements a single copy syntax that can be used to directly copy table structure:

CREATE TABLE tbLikeSamp LIKE DEPTO


This will create a table called tbLikeSamp with the same structure as the DEPTO Table. As with Oracle syntax, the created table will not have referential attributes, constraints, or indexes.

Stored procedures are created using CREATE PROCEDURE command, like this:

CREATE PROCEDURE PROC_NAME (IN PRM1 INT, OUT prm2 DOUBLE) RESULT SETS 1 LANGUAGE SQL BEGIN

<SQL COMMANDS HERE>DECLARE EXIT HANDLER FOR NOT FOUND;

END

DB2 procedures can be written in external languages like C, Java, COBOL, or any language that allows writing an ActiveX (on Windows) DLL.

Note that you can use an exception handler in the procedure, one feature that is not available in Sybase.

The Oracle syntax to create a procedure is again similar:

CREATE PROCEDURE PROC_NAME (PRM1 IN INT, prm2 OUT DOUBLE) AS BEGIN

<SQL COMMANDS HERE>EXCEPTIONWHEN NOT FOUND THEN… ;

END

Note that Oracle also allows a stored procedure to be written in Java or C languages, giving some flexibility.

In Sybase, you issue this command:

CREATE PROC PROC_NAME @PRM1 INT = "D%", @PRM2 DOUBLE OUTPUT AS

Unfortunately, Sybase has no error handling for procedures, and has no flexibility in choosing the language to write the procedure. If you need more power than those offered by Transact-SQL, then you will have to use an extended procedure, which is generally written in languages like C or Pascal, using the Sybase Open Service API. (This can be really painful!)


Appendix B. Tivoli Data Warehouse 1.2 reference

This appendix lists the reports provided by each warehouse enablement pack and their measurement sources.

B


Report listing

AMY : IBM Tivoli Monitoring for Operating Systems, Version 5.1.1, Warehouse Enablement Pack, Version 1.1.0.3

Name AVA Code

Frequency Type

Health of a backup server amy Hourly HealthCheck

Usage of a Domain Controller amy Hourly HealthCheck

Memory Utilization amy Hourly HealthCheck

Paging File Utilization amy Hourly HealthCheck

GWA : IBM Tivoli Monitoring for Web Infrastructure, Version 5.1.0: Apache HTTP Server, Warehouse Enablement Pack, Version 1.1.0

Name AVA Code

Frequency Type

Apache Health Check Error Report gwa Hourly HealthCheck

Apache Health Check Performance Report gwa Hourly HealthCheck

INV : Tivoli Configuration Manager, Version 4.2.0, Warehouse Enablement Pack, Version 1.1.0

Name AVA Code

Frequency Type

Distribution Failures related to package size inv Daily WorstCase

Distribution Status by distribution id inv Daily Summary

Distribution results by file pack and operation inv Daily Summary

Distribution results by file pack, host, operation inv Daily Summary

Success rate of distributions by distribution id inv Daily WorstCase

Success rate of distributions by file pack name inv Daily WorstCase

Success Rate of distributions by time inv Daily HealthCheck

Elapsed distribution time to target inv Daily WorstCase

Receiving distribution time by target inv Daily WorstCase

Distribution transfer rate by file package in kb/s inv Daily WorstCase

File pack distributions that have the most failure inv Daily WorstCase

Distributions that have the most failures inv Daily WorstCase

OS that have the most distribution failures inv Daily WorstCase

Networks that have the most distribution failures inv Daily WorstCase


Operations in verify failure state inv Daily WorstCase

Hosts that have the most distribution failures inv Daily WorstCase

CTD : IBM Tivoli Monitoring for Databases, Version 5.1.0: IBM DB2, Warehouse Enablement Pack, Version 1.1.0

Name AVA Code

Frequency Type

CTD Hourly Percent Connections Used ctd Hourly HealthCheck

CTD Hourly Deadlocks Delta Health Report ctd Hourly HealthCheck

CTD Hourly Percent Catalog Cache Hits ctd Hourly HealthCheck

CTD Hourly Minimum Buffer Pool Hit Ratio ctd Hourly HealthCheck

CTD Hourly Maximum Percentage Used of Primary Log ctd Hourly HealthCheck

ABA : IBM Tivoli Monitoring for Messaging and Collaboration, Version 5.1.0: Lotus Domino, Warehouse Enablement Pack, Version 1.1.0

Name AVA Code

Frequency Type

Bottom Servers By Availability ABA Daily Summary

Calendar Entry ABA Daily WorstCase

Database Access ABA Daily WorstCase

Domino Server Usage: Sessions Dropped ABA Daily WorstCase

Mail Average Statistics ABA Daily Summary

Mail Dead ABA Daily WorstCase

Mail Waiting ABA Daily WorstCase

NAB Search ABA Daily WorstCase

Net Echo ABA Daily WorstCase

Replicate Local ABA Daily WorstCase

Round Trip Mail ABA Daily WorstCase

Web Access ABA Daily WorstCase

GWI : IBM Tivoli Monitoring for Web Infrastructure, Version 5.1.0: Internet Information Server, Warehouse Enablement Pack, Version 1.1.0

Name AVA Code

Frequency Type

Appendix B. Tivoli Data Warehouse 1.2 reference 351

Internet Information Server Performance Report gwi Hourly HealthCheck

CTR : IBM Tivoli Monitoring for Databases, Version 5.1.0: Informix, Warehouse Enablement Pack, Version 1.1.0

Name AVA Code

Frequency Type

Informix Health Check - 7 Days ctr Daily HealthCheck

Informix Thread Activity - 7 Days ctr Daily HealthCheck

Informix Disk Utilization - 7 Days ctr Daily HealthCheck

Informix Logical Log - 7 Days ctr Daily HealthCheck

GWP : IBM Tivoli Monitoring for Web Infrastructure, Version 5.1.0: iPlanet Web Server, Warehouse Enablement Pack, Version 1.1.0

Name AVA Code

Frequency Type

iPlanet Web Server Performance Report gwp Hourly HealthCheck

AMW : IBM Tivoli Monitoring, Version 5.1.0, Warehouse Enablement Pack, Version 1.1.0

Name AVA Code

Frequency Type

WIN_CPU_STATS amw Daily HealthCheck

UNIX_CPU_STATS amw Daily HealthCheck

NET_STATS amw Daily HealthCheck

BUSIEST_SYS amw Daily WorstCase

AMY : IBM Tivoli Monitoring for Operating Systems, Version 5.1.1, Warehouse Enablement Pack, Version 1.1.0

Name AVA Code

Frequency Type

NET_STATS amy Hourly HealthCheck

UNIX_CPU_STATS amy Hourly HealthCheck

WIN_CPU_STATS amy Hourly HealthCheck

BUSIEST_SYS amy Hourly WorstCase


HMI : IBM Tivoli Monitoring for Business Integration 5.1.0 : WebSphere® MQ Integrator, Warehouse Enablement Pack, Version 1.1.0

Name AVA Code

Frequency Type

HMI Metrics for WebSphere MQI Monitor Nodes hmi Daily Summary

HMI Status Down for WebSphere MQI Brokers hmi Daily WorstCase

HMI Status for WebSphere MQI Brokers hmi Daily Summary

HMI Status for WebSphere MQI Config Managers hmi Daily Summary

HMI Status for WebSphere MQI User Name Servers hmi Daily Summary

CTQ : IBM Tivoli Monitoring for Business Integration, Version 5.1.0 : WebSphere MQ, Warehouse Enablement Pack, Version 1.1.0

Name AVA Code

Frequency Type

CTQ Maximum Down Status for Queue Managers Daily ctq Daily WorstCase

CTQ Maximum Outstanding Messages for Queues Daily ctq Daily WorstCase

CTQ Maximum Running Status for Channels Daily ctq Daily WorstCase

CTQ Availability Status for Queue Managers Daily ctq Daily Summary

CTQ Message and Handle Summary for Queues Daily ctq Daily Summary

CTQ Availability Status for Channels Daily ctq Daily Summary

CTW : IBM Tivoli Monitoring for Databases, Version 5.1.1: Microsoft SQL Server, Warehouse Enablement Pack, Version 1.1.0

Name AVA Code

Frequency Type

Daily Filegroup Space Usage Health Check ctw Daily HealthCheck

Daily Database Space Used (Filegroup) Summary ctw Daily Summary

Daily Server Availability Extreme Case ctw Daily WorstCase

Daily Replication Agent Latency Health Check ctw Daily HealthCheck

Daily Server CPU Usage Extreme Case ctw Daily WorstCase

Daily Server Error Message Count Summary ctw Daily Summary

Daily Database Usage Health Check ctw Daily HealthCheck


ANM : IBM Tivoli Netview, Version 7.1.3, Warehouse Enablement Pack, Version 1.1.0

Name AVA Code

Frequency Type

Daily Status Summary By SmartSet anm Daily Summary

Nodes With Longest Outage Time In Routers SmartSet anm Hourly WorstCase

Nodes With Most Status Changes In Routers SmartSet anm Daily WorstCase

Nodes With The Longest Outage Times anm Hourly WorstCase

Nodes With The Most Daily Status Changes anm Daily WorstCase

Summary Of Daily Network Status anm Daily HealthCheck

Summary Of Total Outage Time By SmartSet anm Daily Summary

Summary Of Total Status Changes By SmartSet anm Daily Summary

Total Daily Status Changes In Monitored Network anm Daily HealthCheck

Total Daily Status Changes In Routers SmartSet anm Daily HealthCheck

CTO : IBM Tivoli Monitoring for Databases, Version 5.1.0: Oracle, Warehouse Enablement Pack, Version 1.1.0

Name AVA Code

Frequency Type

BufferCache Hit Ratio (Daily) - Extreme Case cto Daily WorstCase

Deadlocks (Daily) - Health Check cto Daily HealthCheck

Dispatcher contention (Daily) - Summary cto Daily Summary

Oracle RDBMS Availability (Daily) - Extreme Case cto Daily WorstCase

Tablespace Usage (Daily) - Extreme Case cto Daily WorstCase

HRM : IBM Tivoli Risk Manager, Version 4.1.0, Warehouse Enablement Pack, Version 1.1.0

Name AVA Code

Frequency Type

Events by Destination Host - Last 30 Days hrm Daily WorstCase

Service Compromise Events - Last 30 Days hrm Daily HealthCheck

Events by Destination Subnetwork - Last 30 Days hrm Daily Summary

Events by Destination and Category - Last 30 Days hrm Daily WorstCase

Access/Authentication Events - Last 30 Days hrm Daily HealthCheck

Infection Events - Last 30 Days hrm Daily HealthCheck


Policy/Configuration Events - Last 30 Days hrm Daily HealthCheck

ABH : IBM Tivoli Monitoring for Applications, Version 5.1.0: mySAP.com, Warehouse Enablement Pack, Version 1.1.0

Name AVA Code

Frequency Type

mySAP server CPU ranking by rolling 30 days abh Hourly WorstCase

mySAP application area by rolling 30 days abh Hourly WorstCase

mySAP application area dialogs by rolling 30 days abh Hourly WorstCase

mySAP SLA conformance by rolling 30 days abh Hourly WorstCase

mySAP application server logins by rolling 30 days abh Hourly WorstCase

mySAP program count by rolling 30 days abh Hourly WorstCase

mySAP dialog count by rolling 30 days abh Hourly WorstCase

mySAP task type CPU ranking by rolling 30 days abh Hourly WorstCase

mySAP application server uptime by rolling 30 days abh Hourly WorstCase

GMS : IBM Tivoli Monitoring for Applications, Version 5.1.0 : Siebel, Warehouse Enablement Pack, Version 1.1.0

Name AVA Code

Frequency Type

Daily CPU Usage for Siebel Servers gms Daily WorstCase

Weekly CPU Usage for Siebel Servers gms Weekly WorstCase

Hourly CPU Usage for Siebel Servers gms Hourly WorstCase

Monthly CPU Usage for Siebel Servers gms Monthly WorstCase

Daily Memory Usage for Siebel Servers gms Daily WorstCase

Daily Memory Usage Health Check for Siebel Servers gms Daily HealthCheck

Daily CPU Usage Health Check for Siebel Servers gms Daily HealthCheck

Daily Memory Usage Health Check for Siebel Tasks gms Daily HealthCheck

Daily CPU Usage Health Check for Siebel Tasks gms Daily HealthCheck

Summary of Daily Status for Connection Broker gms Daily Summary

Summary of Daily Status for Siebel Gateways gms Daily Summary

Summary of Daily Status for Siebel Servers gms Daily Summary

Summary of Daily Memory Usage for Siebel Tasks gms Daily Summary

Summary of Daily Memory Usage for Siebel Servers gms Daily Summary

Summary of Daily CPU Usage for Siebel Tasks gms Daily Summary


Summary of Daily CPU Usage for Siebel Servers gms Daily Summary

COD : Tivoli License Manager v1.1.0, Warehouse Enablement Pack, Version 1.1.0

Name AVA Code

Frequency Type

Products Installed by Division Summary Report COD Daily Summary

Products Installed by Division Extreme Report COD Daily WorstCase

Agents 1.1 Installed by Division Summary Report COD Daily Summary

Agents 1.1 Installed by Division Extreme Report COD Daily WorstCase

Agents 1.0 Installed by Division Summary Report COD Daily Summary

Agents 1.0 Installed by Division Extreme Report COD Daily WorstCase

Agents 1.1.1 Installed by Division Summary Report COD Daily Summary

Agents 1.1.1 Installed by Division Extreme Report COD Daily WorstCase

AWS : IBM Tivoli Workload Scheduler, Version 8.2.0, Warehouse Enablement Pack, Version 1.1.0

Name AVA Code

Frequency Type

Jobs with the highest average duration time aws Daily WorstCase

Workstation with highest CPU utilization aws Daily WorstCase

Run time statistics for all jobs aws Daily HealthCheck

Run states statistics for all jobs aws Daily HealthCheck

Jobs with highest number of unsuccessful runs aws Daily WorstCase

Unsuccessful runs for a workstation aws Daily WorstCase

IZY : IBM Tivoli Monitoring for Web Infrastructure, Version 5.1.0: WebSphere Application Server, Warehouse Enablement Pack, Version 1.1.0

Name AVA Code

Frequency Type

IZY EJB Performance Health izy Weekly HealthCheck

IZY EJB Resource Model Summary izy Weekly Summary

IZY EJBs with the Most Hits izy Weekly WorstCase

IZY JVM Runtime Resource Model Summary izy Weekly Summary

IZY Servlet Performance Health izy Hourly HealthCheck


Measurement sources

IZY Servlet Session Resource Model Summary izy Weekly Summary

IZY Servlets with the Highest Response Time izy Monthly WorstCase

IZY Servlets with the Most Hits izy Weekly WorstCase

IZY Transaction Manager Resource Model Summary izy Weekly Summary

IZY Web Application Resource Model Summary izy Weekly Summary

GWL : IBM Tivoli Monitoring for Web Infrastructure: WebLogic, Version 5.1.0, Warehouse Enablement Pack, Version 1.1.0

Name AVA Code

Frequency Type

WebLogic Server Availability (Daily) - EC gwl Daily WorstCase

WebLogic JDBC Connection Pool Statistics (Daily) gwl Daily Summary

WebLogic EJB Transactions (Daily) - HC gwl Daily HealthCheck

WebLogic Servlet Performance (Daily) - EC gwl Daily WorstCase

WebLogic JMS Load (Daily) - EC gwl Daily WorstCase

App Name Msmt Source Code

Name Parent Code

390 CICS® D04 Tivoli Decision Support for OS/390 (CICS component) DRL

390 CICS DRL Tivoli Decision Support for OS/390 Tivoli

390 DB2 D05 Tivoli Decision Support for OS/390 (DB2 component) DRL

390 DB2 DRL Tivoli Decision Support for OS/390 Tivoli

390 DFSMS D08 Tivoli Decision Support for OS/390 (DFSMS component) DRL

390 DFSMS DRL Tivoli Decision Support for OS/390 Tivoli

390 IMS D03 Tivoli Decision Support for OS/390 (IMS component) DRL

390 IMS DRL Tivoli Decision Support for OS/390 Tivoli

390 MQS D06 Tivoli Decision Support for OS/390(MQS component) DRL

390 MQS DRL Tivoli Decision Support for OS/390 Tivoli

390 MVS™ D01 Tivoli Decision Support for OS/390 (MVS component) DRL

390 MVS DRL Tivoli Decision Support for OS/390 Tivoli

390 NPM D10 Tivoli Decision Support for OS/390 (NPM component) DRL

390 NPM DRL Tivoli Decision Support for OS/390 Tivoli


390 NPMIP D11 Tivoli Decision Support for OS/390 (NPMIP component) DRL

390 NPMIP DRL Tivoli Decision Support for OS/390 Tivoli

390 OPC D07 Tivoli Decision Support for OS/390 (OPC component) DRL

390 OPC DRL Tivoli Decision Support for OS/390 Tivoli

390 RACF® D09 Tivoli Decision Support for OS/390 (RACF component) DRL

390 RACF DRL Tivoli Decision Support for OS/390 Tivoli

390 RMF™ D02 Tivoli Decision Support for OS/390(RMF component) DRL

390 RMF DRL Tivoli Decision Support for OS/390 Tivoli

390 WAS D12 Tivoli Decision Support for OS/390 (WebSphere component) DRL

390 WAS DRL Tivoli Decision Support for OS/390 Tivoli

Apache PAC AMX IBM Tivoli Monitoring Tivoli

Apache PAC GWA IBM Tivoli Monitoring for Web Infrastructure, Version 5.1.0: Apache HTTP Server

AMX

BMC Patrol BP6 BMC PATROL null

CM for ATMs CMA IBM Tivoli Configuration Manager for Automated Teller Machines

Tivoli

CM for ATMs Tivoli Tivoli Application null

DB2 PAC AMX IBM Tivoli Monitoring Tivoli

DB2 PAC CTD IBM Tivoli Monitoring for Databases: DB2 AMX

DM 3.7 DMN Distributed Monitoring Classic Edition Tivoli

Domino PAC ABA IBM Tivoli Messaging and Collaboration, Version 5.1.0: Lotus Domino

AMX

IIS PAC AMX IBM Tivoli Monitoring Tivoli

IIS PAC GWI IBM Tivoli Monitoring for Web Infrastructure, Version 5.1.0: Internet Information Server

AMX

ITM 4.1 AMW Distributed Monitoring Advanced Edition Tivoli

ITM 5.1.0 AMW IBM - Tivoli Monitoring 5.1 Tivoli

ITM for OS 5.1.1 AMY IBM Tivoli Monitoring for Operating Systems AMX

Informix PAC AMX IBM Tivoli Monitoring Tivoli

Informix PAC CTR IBM Tivoli Monitoring for Databases, Version 5.1.0: Informix AMX

Inventory-Software Distribution

DIS IBM Tivoli Software Distribution Tivoli

Inventory-Software Distribution

INV IBM Tivoli Inventory Tivoli

License Manager COD Tivoli License Manager 1.1 Tivoli

MQ PAC CTQ IBM Tivoli Monitoring for Business Integration, Version 5.1.0 : WebSphere MQ

AMX


MQI PAC HMI IBM Tivoli Monitoring for Business Integration 5.1.0 : WebSphere MQ Integrator

AMX

MQWorkflow PAC BIW IBM Tivoli Monitoring for Business Integration - MQSeries® Workflow 5.1.0

AMX

MS SQL CTW IBM Tivoli Monitoring for Databases, Version 5.1.0: Microsoft SQL Server

AMX

Netview ANM IBM Tivoli Netview Tivoli

Netview MODEL1 Tivoli Common Data Model v 1 null

Oracle PAC CTO IBM Tivoli Monitoring for Databases, Version 5.1.0: Oracle AMX

Risk Manager HRM IBM Tivoli Risk Manager Tivoli

Siebel PAC AMX Tivoli Monitoring Tivoli

Siebel PAC GMS IBM Tivoli Monitoring for Applications 5.1.0 : Siebel eBusiness Applications

AMX

Siebel PAC Tivoli Tivoli Application null

TAPM 2.1 APF Tivoli Application Performance Management Tivoli

TBSM 1.5 GTM Tivoli Business System Manager Tivoli

TEC 3.7.1 ECO Tivoli Enterprise Console® Tivoli

TEC 3.8 ECO Tivoli Enterprise Console Tivoli

TEDW 1.2 Tivoli Tivoli Application null

TEDW 1.2 MODEL1 Tivoli Common Data Model V1 null

TMTP 5.1 BWM IBM Tivoli Monitoring for Transaction Performance - Web Transaction Performance 5.1

Tivoli

TSRM BTM IBM Tivoli Storage Resource Manager Tivoli

TSRM Tivoli Tivoli Application null

TWSA AWT Tivoli Web Site Analyzer 4.2 Tivoli

TWSM 1.7 BWM Tivoli Web Services Manager 1.7 Tivoli

Talking Blocks TS2 Talking Blocks null

Talking Blocks SNMP Simple Network Management Protocol null

Talking Blocks SHARED Shared null

Talking Blocks SDESK1 Service Desk null

WS Interchange PAC BIX IBM Tivoli Monitoring for Business Integration - WebSphere InterChange Server 5.2.0

AMX

WebLogic PAC AMX IBM Tivoli Monitoring Tivoli

WebLogic PAC GWL IBM Tivoli Monitoring for Web Infrastructure : WebLogic AMX

WebSphere PAC IZY IBM Tivoli Monitoring for Web Infrastructure, Version 5.1.0: WebSphere Application Server

AMX

iPlanet PAC AMX IBM Tivoli Monitoring Tivoli


iPlanet PAC GWP IBM Tivoli Monitoring for Web Infrastructure, Version 5.1.0: iPlanet Web Server

AMX

mySAP PAC ABH IBM Tivoli Monitoring for Applications, Version 5.1.0: mySAP.com

Tivoli

mySAP PAC Tivoli Tivoli application null


Appendix C. Warehouse Enablement Packs properties file

This appendix provides a description of the various attributes of the properties file of a warehouse enablement pack.

C


The twh_install_props.cfg properties fileEvery warehouse enablement pack must have defined a properties file that indicates how Tivoli Data Warehouse will perform the warehouse pack installation. The properties file is named twh_install_props.cfg and its location in the warehouse enablement pack installation media depends on the version of the Tivoli Data Warehouse product for which the warehouse enablement pack was created.

The twh_install_props.cfg file of warehouse enablement packs created for the Tivoli Data Warehouse 1.2 resides in the top level directory of the installation media, specified by the AVA code. For warehouse enablement packs created for the Tivoli Enterprise Data Warehouse 1.1, the twh_install_props.cfg is located in the TDW 1.2 WEPs located in the pkg directory, as shown in Figure C-1.

Figure C-1 Location of the twh_install_props.cfg file

Here we list some of the characteristics of the properties defined in the twh_install_props.cfg file:

� Statements take the form PROPERTY=VALUE.� The property name must be uppercase.� Each property in the file must have a value.� The properties file is used by the installer.� The installer will request the location of this file when installing the WEP.� Ensure that a carriage return terminates each line — including the last line in

the file.� A property cannot contain white space. That is, do not put white space on the

left side of the equal sign (=).

twh_install_props.cfg for TDW 1.2

twh_install_props.cfg for TEDW 1.1


Table C-1 shows the attributes and parameters of the warehouse enablement pack properties file: twh_install_props.cfg.

Table C-1 WEP installation properties

Property Description

AFTER_SCRIPT Optional Perl script that runs at the end of the warehouse pack installation. The script name must be misc/<ava_code>_after.pl, and must be lowercase..

ALLOW_ETL1_MULTI_SOURCE Specifies if the WEP was coded to support multiple data sources.

APP_DISPLAY NAME User display name for the WEP.

APP_DISPLAY_VERSION User displayed version.

APP_REL_VERSION The internal version of the warehouse pack.

AVA_CODE Three-character identifier unique to the warehouse pack.

CDW_DB_TYPE Vendor support type for CDW. Allowed values are:- DB2UDB (for IBM DB2 UDB Enterprise Edition)- DB2390 (for DB2 UDB for z/OS and OS/390)- DB2UDB,DB2390 (for both)

Must be in uppercase.

DWC_INIT_STEP_NAME One-time initialization step that must be run manually after installation, and before the first job is scheduled.

MART_DB_TYPE Data mart vendor types supported by this warehouse pack. Same options as CDW_DB_TYPE.

NLS_LANG_LIST Languages supported by the WEP. Supported values of pt_BR,en,fr,de,it,ja,ko,zh_CN,es,zh_TW or if no internationalization TDW_NO_BUNDLES_REQUIRED.

Appendix C. Warehouse Enablement Packs properties file 363

PRE_ETL_SCRIPT Optional Perl script that runs before the TAG file is loaded into the DB2 Data Warehouse Center.

PRE_SCRIPT Optional Perl script that runs after the environment has been validated and the files have been copied to the $TWH_TOPDIR directory.

REPORTS_INCLUDED If reports are included in the WEP. TRUE or FALSE.

SCHEDULABLE_ETL1_STEP_NAME First step in the central data warehouse ETL process.

SCHEDULABLE_ETL2_STEP_NAME First step in a data mart ETL process.

SOURCE_APP_REL_VERSION The version number of your source application that is required by this warehouse pack. For documentation purposes only.

TWH_CORE_PREREQ_VERSION The required version of TDW for this WEP.

WEP_ROLE The type of ETL processes implemented by the WEP.ETL1 – only ETL1 processes.ETL2 – only ETL2 processes.E1E2 – both ETL1 and ETL2.

WPACK_PREREQ_AVA_NAME Minimum prerequisite WEP names required for this WEP to run.

WPACK_PREREQ_VERSION_n Minimum prerequisite WEP version required for this WEP to run.

Property Description


Related publications

The publications listed in this section are considered particularly suitable for a more detailed discussion of the topics covered in this redbook.

IBM RedbooksFor information on ordering these publications, see “How to get IBM Redbooks” on page 367. Note that some of the documents referenced here may be available in softcopy only.

� Introduction to Tivoli Enterprise Data Warehouse, SG24-6607

� IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring, SG24-5519

� Tivoli Data Warehouse Report Interfacing, SG24-6084

Other publicationsThese publications are also relevant as further information sources:

� IBM DB2 Universal Database Command Reference Version 7, SC09-2951

� IBM Tivoli Monitoring User’s Guide Version 5.1.1, SH19-4569

� IBM Tivoli Monitoring - Resource Model Reference, SH19-4570

� Tivoli Enterprise Data Warehouse Release Notes, GI11-0857

� IBM DB2 Universal Database for Windows Quick Beginnings, GC09-2971

� Crystal Enterprise 9 Installation Guide

� Crystal Enterprise 9 Administrator’s Guide

� Crystal Enterprise 9 Getting Started Guide

� Crystal Enterprise 9 ePortfolio User’s Guide


Online resourcesThese Web sites and URLs are also relevant as further information sources:

� IBM Software Support Web site

http://www.ibm.com/software/sysmgmt/products/support

� Crystal Enterprise Professional version 9 for Tivoli product site

http://www.businessobjects.com/products/reporting/default.asp

� J2SE download site Crystal Enterprise Professional version 9 for Tivoli

http://java.sun.com/j2se/1.3/install-solaris-patches.html

� SunSolve Web site

http://sunsolve.sun.com

� Tivoli Enterprise Data Warehouse

http://www.ibm.com/software/sysmgmt/products/support/TivoliEnterpriseDataWarehouse.html

� Java Tutorial provided by Sun Microsystems

http://java.sun.com/docs/books/tutorial/jdbc

� World Wide Web Consortium

http://www.w3.org

� IBM DB2 Universal Database Enterprise Edition support Web site

http://www-3.ibm.com/cgi-bin/db2www/data/db2/udb/winos2unix/support/download.d2

� Microsoft Support Web site

http://support.microsoft.com/default.aspx?scid=kb

� Object management group Web site

http://www.omg.org

� DB2 magazine Web site

http://www.db2mag.com/db_area/archives/2001/q1/hayes.shtml

� Index of /html/Something_about_XSLM/About_XSLM

http://www.xslm.org/html/Something_about_XSLM/About_XSLM/Original_Requirements

� IBM HTTP Server

http://www-3.ibm.com/software/webservers/httpservers


http://www-3.ibm.com/cgi-bin/db2www/data/db2/udb/winos2unix/support/download.d2

http://www.ibm.com/software/sysmgmt/products/support

http://java.sun.com/j2se/1.3/install-solaris-patches.html

http://sunsolve.sun.com

http://www.ibm.com/software/sysmgmt/products/support/TivoliEnterpriseDataWarehouse.html

http://java.sun.com/docs/books/tutorial/jdbc

http://www.w3.org

http://www.businessobjects.com/products/reporting/default.asp

http://support.microsoft.com/default.aspx?scid=kb

http://www.omg.org

http://www.db2mag.com/db_area/archives/2001/q1/hayes.shtml

http://www.xslm.org/html/Something_about_XSLM/About_XSLM/Original_Requirements

http://www-3.ibm.com/software/webservers/httpservers

� HTTP Server Plug-in

http://www7.software.ibm.com/vadd-bin/ftpdl?1/vadc/wsdd/pdf/presents/WS40ST08.pdf

� DB2 Technical Support

– http://www-3.ibm.com/cgi-bin/db2www/data/db2/udb/winos2unix/support/v7pubs.d2w/en_main

– http://www-3.ibm.com/cgi-bin/db2www/data/db2/udb/winos2unix/support/v7fphist.d2w/report

How to get IBM RedbooksYou can search for, view, or download Redbooks, Redpapers, Hints and Tips, draft publications and Additional materials, as well as order hardcopy Redbooks or CD-ROMs, at this Web site:

ibm.com/redbooks

Help from IBMIBM Support and downloads

ibm.com/support

IBM Global Services

ibm.com/services

Related publications 367




http://www.ibm.com/support/

http://www.ibm.com/support/

http://www.ibm.com/services/

http://www.ibm.com/services/

http://www7.software.ibm.com/vadd-bin/ftpdl?1/vadc/wsdd/pdf/presents/WS40ST08.pdf

http://www-3.ibm.com/cgi-bin/db2www/data/db2/udb/winos2unix/support/v7pubs.d2w/en_main

http://www-3.ibm.com/cgi-bin/db2www/data/db2/udb/winos2unix/support/v7fphist.d2w/report

Index

Symbols.config 236

Numerics1.1 WEPs 46390 systems 45

AActivating data collection 237adding CDWs 116adding MARTs 119Addition of warehouse packs 32admin server user 81adverse effects 61agent 8–9agent configuration 18agent discovery 18agent site 126Agent Sites 22agents installation 126Aggregation time line 231analytical processing 24Apache 35APS 28, 33Architectural choices 62Automated Process Schedule 28Automated Process Scheduler 33

Bbacking up the TWH_CDW database 242, 304backing up the TWH_MART database 242, 304backing up the TWH_MD database 242, 305batch report 6BI 5Buffer Pool 54, 115buffer pool 146Business Intelligence 5business intelligence 4, 64Business Intelligence tools 70

© Copyright IBM Corp. 2004. All rights reserved.

CCENTR_LOOKUP 61Central data warehouse

naming convention 39central data warehouse 4, 36, 38Centralized logging 19client

administrative 9Client tier 12CMC 59Cognos Powerplay 14collecting data 232compete effectively 6components requirements 36compromising security 58configuring Crystal Enterprise 87configuring the control database 99control center server 21control database 38control database configuration 99, 113control server 36, 38create the itm_db database 235creating agent sites 126, 132creating CDWs 116creating MARTs 119Crystal Enterprise

architecture 11Limited Edition 10Professional Edition 10user authority 59

Crystal Enterprise 9 33Crystal Enterprise configuration 87Crystal Enterprise installation 86Crystal Enterprise port numbers 56Crystal Enterprise Professional version 9 for Tivoli 11Crystal Enterprise requirements 33Crystal Enterprise Server 22Crystal Enterprise server 36Crystal Enterprise Version 9 Special Edition 11Crystal Management Console 59Crystal object rights 59Crystal object security model 59Crystal Publishing Wizard 105

369

Crystal Reports 11CUST_LOOKUP 61

Ddata

cleansing 8Data collection 67data collection process 236data manipulation 67–68data mart 5, 21, 36Data mart database

naming convention 41data mining 7data model 5data sources 32Data tier 13data warehouse 4

accessing 8building 8designing 8governing 8integrated 5maintaining 8subject-oriented 5time-variant 5

Data Warehouse Center 8–9database heap 148Database Managed Space 150datacollector delay 240datacollector prefix 236datacollector.db_purge_interval 237datacollector.db_purge_time 237datacollector.delay 237datacollector.max_retry_time 237datacollector.rim_name 237datacollector.sleep_time 237datamart 8DB2 78DB2 admin 81DB2 data flow 51DB2 fenced 80DB2 Fix Pack 32DB2 Fix Pack 8 installation 82, 84DB2 instance 79DB2 JDBC Applet Server 128DB2 logs 147DB2 on z/OS 32DB2 performance 146

DB2 private protocols 51DB2 Server installation 82DB2 server installation 76DB2 setup 78DB2 Warehouse Manager

administrative client 9agent 9components 8metadata 9

DB2 Warehouse Manager installation 128DB2_ENABLE_LDAP settings 77DB2-DB2CTLSV 128DB2SYSTEM 81demographic data 5Development status 274DHTML 58distributed deployment install 103Distributed installation 29DMS 150DMS tablespaces 150DNS 74Domain Name Server 74Domino 35driving forces 6

Eeffect on network 61endpoint 239ETL development 65ETL grouping 64ETLs 63

Ffenced procedures 57fenced user 80fetched data 146FFDC 19firewall considerations 58First Failure Data Capture 19Fix Pack for DB2 32FMID JDB771D 32, 115fp8_wr21314 84Future data growth 32

Ggateway 233, 239generic ETL1 229


GMT time 240graphical user interface 8–9

II/O servers 148IBM Console 10IBM DB2 admin 81IBM DB2 fenced 80IBM DB2 instance 79IBM DB2 subsystem 41IBM Tivoli Monitoring 37IBM Tivoli Monitoring 5.1.1 232IBM Tivoli Storage Manager 37IMS 14increase revenues 6information access 6Information Catalog Manager 9information delivery 7Information Management System 14information tokens 77Informix 33Install methods

Distributed installation 28, 43Quick start installation 28, 42

InstallationIBM Tivoli Monitoring 5.1.1 232single machine 42

installation enhancements 18installation preparation 72installation process overview 73installation steps 232installing CDWs on z/OS 116installing DB2 Warehouse Manager 128installing MARTs on z/OS 119installing remote agent sites 126installing TDW 93, 103installing WEPs 142instance 79instance owner 57, 235INSTHOME 81integrated 5integrated warehouse 5Intelligence tier 12intranet 7iPlanet 35iPlanet Enterprise Server 87irs.conf file 74ITM 37

ITM Middle Layer Repository 228ITM V 5.1.1 Generic Warehouse Enablement Pack 241ITSM 37

JJava heap 147JavaScript 39, 41JDB771D 32, 115JDBC code level 81

Llarge amounts of data 28, 42LDAP 59, 77LDAP authentication 59level of DB2 on z/OS 32level token 57local warehouse agent 50location name 41logbufsz 147–148logfilsiz 147logical design 36logprimary 147logsecond 147LogXML Viewer 19

Mmaintenance window 63Managed Resource 237manager 9manipulate data 68mapping data sources 66MDAC 31metadata 9metricsdata 240metricsdata table 240Microsoft Data Engine 33, 86Microsoft SQL Server 33MON_HEAP_SZ 148monitoring application metadata 229MSDE 86MSDE database 33MSDE user account 88multicenter support 59–60Multicustomer support 60multicustomer support 59multiple customers support 60

Index 371

multiple machines installation 43

Nnaming convention 39, 41netsvc.conf file 74Network Information Service 74NIS 74nsswitch.conf file 74NUM_IOCLEANERS 146NUM_IOSERVERS 146

Oobject rights 59object security 59ODBC 32ODBC connections 101, 123ODBC data sources 18OLAP 4, 24OLTP 4online analytical processing 4operational data 4Operational Data Stores 8Oracle 33organization 67OS/390 45

PPAE 152page cleaners 146PCI bus 152performance DB2 146Physical Address Extensions 152physical database 235physical design 36point of control 8policy region 237port numbers 56prefetchers 146Process Modeller window 259Processing tier 13Product_Code 61Production status 274production window 63profile 237–238profile manager 237Promote the ETL status 272

Qquery 6quick start install 93Quick start installation 29

RRDBMS administrator 235RDBMS Interface Module

See RIMRedbooks Web site 367

Contact us xxiiireduce costs 6remote agents creation 132remote agents installation 126remote warehouse agent 50remote warehouse agent site 126Remote warehouse agents 62REORG 149REORGCHK 149Report Interface requirements 33Reporting 70reporting 6requirements for Crystal Enterprise 33resolv.conf file 74resource model 237–238restart DB2 242, 304RIM 232RIM configuration 233RIM host 232RIM host machine 235RIM object 234, 239RIM object connection 235RUNSTATS 149

Ssa user ID 88SAP R/3 8scalability 8security considerations 57security model 59Service Pack 6 33shell script 235Single machine installation 42single machine installation 42skills required 67SMF 14SMS 150sortheap 148


source databases 36specific object types 238SQL 61SQL admin account 88SQL administrator account 88SQL scripts 235stand alone installation 29, 42star schema 42Storage Group 54, 115Structured Query Language 61

See SQLSubject Area 259subject-oriented 5subject-oriented warehouse 5supported Web browsers 35Sybase 33System Managed Space 150System Management Facility 14

TTablespace size 54, 115TCDWn databases 39TCPIP ports 56TDS for OS/390 14TDW architecture 20TDW installation

distributed deployment 103quick start 93

TDW performance 155tedw_apps_etl 142Test the ETLs 267thin-client 7timekey_dttm 240time-variant 5time-variant warehouse 5Tivoli Data Warehouse

advanced configuration 50basic components 36components requirements 36database requirements 32logical design 36ODBC 32physical design 36security considerations 57supported Web browsers 35warehouse agent 50

Tivoli Decision Support 13Tivoli desktop 232

Tivoli Enterprise 28Tivoli Enterprise Data Warehouse 28

deploying 28hardware requirements 29software requirements 30supported Web browsers 33

Tivoli Enterprise Data Warehouse server 226Tivoli Enterprise Data Warehouse Support, Version 5.1.1

configuration 232Installation 232

Tivoli environment 232Tivoli gateway 226Tivoli managed node 232Tivoli Management Region server 233Tivoli Presentation Services 10Tivoli_Admin_Privileges 235TMARTn databases 41TMR server 226Tmw2kProfile 237token object 57transformations

statistical 8TWG.CUST 61TWH_CDW 39, 93twh_configwep 142twh_create_datasource 102, 124twh_list_agentsites.bat 138twh_list_cdws.bat 136twh_list_cs.bat 135twh_list_marts.bat 137TWH_MART 41, 93TWH_MD 38, 93twh_update_rptsrv 139twh_update_rptsvr 139twh_update_userinfo 140

UUDF 57update JDBC level for DB2 81user authentication for Crystal 59user rights 84UTF8 54, 115

VVCAT 54, 115verify remote agent install 141

Index 373

Wwarehouse

management infrastructure 8warehouse agent 18, 22, 36, 50

distributed 8local 9remote 9

warehouse agent site 126warehouse databases on z/OS 54Warehouse Enablement Pack (WEP) 18warehouse enablement pack install 142warehouse enablement packs

SeeWEP

warehouse server 9warehousing 4wcrtrim 234wdmcollect 239, 292wdmconfig 236wdmdistrib 239wdmlseng 239WEP 11, 63, 262WEP installation 142WEP Installation Wizard 18Windows 2000 Advanced Serve 33work area 261work flow 67wrimtest 235wsetrimpw 234

Zz/OS 36, 45z/OS support 13


(0.5” spine)0.475”<->

0.875”250 <->

459 pages

Implem

enting Tivoli Data Warehouse 1.2

®

SG24-7100-00 ISBN 0738497894

INTERNATIONAL TECHNICALSUPPORTORGANIZATION

BUILDING TECHNICALINFORMATION BASED ONPRACTICAL EXPERIENCE

IBM Redbooks are developed by the IBM International Technical Support Organization. Experts from IBM, Customers and Partners from around the world create timely technical information based on realistic scenarios. Specific recommendations are provided to help you implement IT solutions more effectively in your environment.

For more information:ibm.com/redbooks

Implementing Tivoli Data Warehouse 1.2

A primer for deployments of any size and proof of concepts

Latest Version 1.2 features including Crystal Enterprise

Warehouse enablement pack case studies

With Tivoli Data Warehouse, you can analyze historical trends from various Tivoli and customer applications. The Tivoli Data Warehouse infrastructure enables a set of extract, transform, and load (ETL) utilities to extract and move data from Tivoli application data stores to a central repository.

The open architecture of Tivoli Data Warehouse also enables data from non-Tivoli applications to be integrated into its central repository. Data from the central repository can be extracted into data marts that pertain to the reporting needs of selected groups. These data marts can also be used to produce cross application reports.

This IBM Redbook focuses on planning, installation, customization, use, maintenance, and troubleshooting topics related to the new features of the Tivoli Data Warehouse Version 1.2. This is done using a number of case study scenarios and several warehouse enablement packs.

The instructions given in this book are very detailed and explicit. These instructions are not the only way to install the products and related prerequisites. They are meant to be followed by anyone to successfully install, configure, and set up Tivoli Data Warehouse environments of any size.

Back cover




implementing tivoli data warehouse v 1.2 sg247100

Technology

tivoli data warehouse

tivoli data warehouse

data marts

data collection

data mart5

data mining

warehouse databases

warehouse agents492