timesten - beyond the summary advisor (odtug kscope'14)

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] W : www.rittmanmead.com

TimesTen as your Analytic Sandbox or - “Going beyond the Summary Advisor” Mark Rittman, CTO, Rittman Mead ODTUG KScope’14, Seattle, June 2014





About Rittman Mead

•Oracle BI and DW Gold partner •Winner of five UKOUG Partner of the Year awards in 2013 - including BI •World leading specialist partner for technical excellence, solutions delivery and innovation in Oracle BI

•Approximately 80 consultants worldwide •All expert in Oracle BI and DW •Offices in US (Atlanta), Europe, Australia and India •Skills in broad range of supporting Oracle tools: ‣OBIEE, OBIA ‣ODIEE ‣Essbase, Oracle OLAP ‣GoldenGate ‣Endeca



Oracle Exalytics In-Memory Machine

• In-Memory analytics - lightening-fast response, free-form analysis and aggregation •Rich, immersive interactive dashboards powered-by high-spec hardware

•Extra OBIEE + other features only available on this platform

•Enables fast development controlled by the business

•Faster planning and budgeting



TimesTen as an In-Memory Aggregate Cache

•Uses an in-memory aggregate cache co-located with OBIEE •Based around Oracle TimesTen for Exalytics, Oracle’s In-Memory Database •Aggregates are recommended based on query patterns, and automatically created in TimesTen

•Summary Advisor makes recommendations, that adapt as queries change

•Meant to be “plug-and-play” - no need for expensive data warehouse tuning

TimesTen BI Server

Exal

ytic

s

Aggregates

Data WarehouseDetail-level

Data



Summary Advisor for Aggregate Recommendation & Creation

•Utility within Oracle BI Administrator tool that recommends aggregates •Bases recommendations on usage tracking and summary statistics data •Captured based on past activity •Runs an iterative algorithm that searches,each iteration, for the best aggregate



… But TimesTen for Exalytic Can Do Much, Much More!

•TimesTen is a fully-featured in-memory analytic database •You can use it for much more than just storing automatically created aggregates •For example… ‣Also store aggregates that you specifically request based on your knowledge of queries ‣Hold “hot data” - transaction-level data replicated into memory formost recent periods - i.e. last six months ‣Add additional data that you’d like to put into memory,e.g. other reference data, additional attributes etc

•And you can be a lot more efficient in how it’s updated ‣The Summary Advisor drops + recreates aggs each load ‣But you can modify this to do incremental, trickle-feed updates



What is Oracle TimesTen?

•An In-Memory Database acquired by Oracle in 2005 •Uses standard (and largely Oracle-compatible) SQL •ACID transactions, multi-user concurrency ‣Persists data to disk in-background ‣Commits and Rollbacks; HA and DR via replication

•Standard SQL APIs; Java, C++, C, C# •Available for use standalone, or as part of Oracle Exalytics •Extremely fast!

•Entire database in RAM •Microsecond (and consistent) response time •Very high throughput •Cost-based in-memory optimizer



TimesTen’s Origins - In-Memory OLTP Database

•TimesTen originated as a super-fast, low-latency OLTP database •Trading applications, real-time capture and storage of data •Can be used in conjunction with Oracle RDBMS, as data cache

•Lightweight indexing, less impacton DML than Oracle RDBMS indexes

•Highly-scalable, linear throughputas processes / CPUs added to server

•Full ACID compliance, commit/rollback,checkpointing of data to disk



More Recently - TimesTen as an Analytic Database

•Packaged as part of Oracle Exalytics, to provide the in-memory aggregate cache •Two key features added to enable for BI workloads:

•Columnar-compression •Extensions to SQL for grouping, aggregation etc •But - only available on Exalytics hardware + license

•Accessed through OBIEE’s Summary Advisor •Heuristic, automatic population of aggregate cache •Adapt to changing workloads •“Fire-and-forget” - automatically improved query times

1 TB RAM

TimesTen In-Memory

Database for Exalytics

Heuristic In-Memory

Cache



TimesTen Key Architectural Features

•Database entirely in-memory - loaded-up from disk at start •All operations in-memory, with commits check-pointed to disk •Has usual ACID-compliant internals - locks, latches etc

•But without the overhead of assuming disk-based storage •At present, DB size limited to RAM on server

•And 25% - 50% of RAM usually required for OS, temp space •Connections are by OCI / JDBC / ODBC etc

•Compatible with SQL*Developer, as dev tool •Supported data source for OBIEE



TimesTen BI Use-Cases

•As the in-memory cache for OBIEE aggregates in Exalytics •As an in-memory cache for the Oracle Database •Real-time high-volume departmental analytics •General purpose BI data mart •Rapidly developing and changing departmental analytic server



A Mid-Tier In-Memory Analytic Database Under Your Control

•Major benefit of using a standalone analytic database - it’s under your control •Most data warehouses are enterprise resources - under change control etc

•TimesTen can sit in the department, be treated as a business / mid-tier server •Can work with it in an agile way - rapidly iterating the design away from DW constraints •Use regular development tools - SQL*Developer, ODI, OBIEE •Many tools exist to make development faster ‣Summary Advisor ‣Aggregate Persistence Wizard ‣ttImportFromOracle utility ‣GoldenGate / ODI etc



Taking TimesTen for Exalytics Beyond the Summary Advisor

•An example TimesTen for Exalytics Deployment, going beyond the Summary Advisor !

1. Use the Summary Advisor to create initial TT aggregate tables 2. Supplement these with other custom aggregates, using Aggregate Persistance 3. Also use TimesTen to store the last six months’ transactional activity in-memory 4. Look at what’s involved in adding additional data into TimesTen 5. Enable the TimesTen dataset for incremental load



Source Dataset - Oracle Sales History (SH) Schema

•One of the standard sample datasets in Oracle Database

•Single fact table used, with six dimensions •Hierarchies within dimension used for aggregation

•900k rows in fact table



Memory and System Configuration for TimesTen

•Hugepages allows large amounts of memory to be utilised with a reduced overhead in page management.

•We also need to allow TimesTen to lock RAM. 1. Set the memlock option in limits.conf 2. Set the the MemoryLock=4 option in the TimesTen database configuration. 3. Set the ttendaemon.options - linuxLargePageAlignment 2

•Validate your memory configuration:- ‣kernel.shmall (pages) = shared memory on the system (shmax) / page_size ‣Start you DB and use the Dssize command to review memory high water marks. ‣Remember TT total memory = PermSize + TempSize + LogFileSize + LogBufMB

• Amend the default timeouts for your BI workload. TTC_TIMEOUT and SQLQueryTimeout • Set your Log Buffers and and Recovery Threads to your core count - 2. •Monitor buffer waits for any I/O bottlenecks.



What About the Oracle Database In-Memory Option?

•Announced at Oracle Openworld 2013 - GA later in 2014? •Adds in-memory analytic capability to 12c EE database ‣Copy of tables etc placed into RAM, in SGA ‣Stored in column-store format ‣Automatic process, with tables etc selectable

•Obvious overlap with TimesTen ‣and in some respects, better (column-store)

•But there are still reasons to use TimesTen 1. TimesTen is purely in-memory - s/b faster 2. Standalone DB outside DW change control 3. In-Memory OLTP capabilities 4. Cheaper to license than DB EE + Options



Step 1 : Auto-Generation of Aggregates using Summary Adv.

•Workload generated using OBIEE dashboard, recorded into Usage Tracking table •Exalytics-enabled OBIEE 11g instances record additional statistics and query data ‣S_NQ_ACCT now supplemented by S_NQ_DB_ACCT ‣Extra usage tracking information, includes physical SQL ‣Exalytics Summary Advisor uses S_NQ_SUMMARY_ADVISOR

-Contains summary statistics, execution time etc -Gathered at same time as usage tracking whenExalytics is enabed

-Contents can be derived from usage tracking if needed



Initial Aggregate Recommendations

•Recommended a set of aggregates based on historic query patterns & stats •Recommendation is at the dimension level basis, removes individual query predicates ‣city = “Seattle” for example, leaving just “city” as recommendation



Aggregate Generation Results

• Initial aggregate build failed •Check nquery.log file fordetail of build failure •Usually not that clear •But some clues can be found

•Key dimension failed to build,cascaded down to fact table build

c:\Middleware\Oracle_BI1\bifoundation\server\bin>nqcmd -d coreapplication_OH799 36239 -u weblogic -p welcome1 -s c:\ttscripts\summ_adv_SH_full_blog.sql ------------------------------------------------------------------------------- Oracle BI ODBC Client Copyright (c) 1997-2013 Oracle Corporation, All rights reserved ------------------------------------------------------------------------------- create aggregates "ag_561184940" for "Sales History (Oracle + TT)"."SALES"("AMOUNT_SOLD","QUANTITY_SOLD") at levels ("Sales History (Oracle + TT)"."CustomerDim"."City", "Sales History Oracle + TT)"."TimesDim"."Month", "Sales History (Oracle + TT)"."ProductsDim"."Category") using connection pool "tt_aggr_store"."Connection Pool" in "tt_aggr_store".."EXALYTICS", … "ag_1162984574" for "Sales History (Oracle + TT)"."SALES"("AMOUNT_SOLD","QUANTITY_SOLD") at levels ("Sales History (Oracle + TT)"."TimesDim"."Year", "Sales History (Or cle + TT)"."ProductsDim"."Category", "Sales History (Oracle + TT)"."PromoDim"."Promo Subcategory") using connection pool "tt_aggr_store"."Connection Pool" in "tt_aggr_store".."EXALYTICS" [10058][State: S1000] [NQODBC] [SQL_STATE: S1000] [nQSError: 10058] A general error has occurred. Statement preparation failed Processed: 1 queries Encountered 1 errors



Model Checker, and Model Validity (for Aggregation)

•Always run the model checker before running Summary Advisor •Mandatory to clear errors (led to dimension table build failure in previous step)

•Advised to clear warnings •Usually down to data errors

•non-unique values within a column defined as unique in RPD

•dimension entries for same member rolling up to different parents

•Must address these before continuing with agg build



Final Set of Auto-Generated Aggregates

•Two aggregates created (fact table + conformed dimensions) •Automatically added to RPD physical layer, and mapped into business model

•TimesTen tables created and populated via BI Server ETL process



Step 2 : Add Custom Aggregates, via Aggregate Persistence

•Summary Advisor creates aggregate recommendations based on history query patterns •But you might want to create more aggregates based on queries you’ll have in-future

• Include additional new dimensions or levels in the aggregate •Aggregates you know will be useful for users

•Aggregate Persistance Wizard uses same underlying build mechanism as Summary Advisor, but allows youto directly specify aggregates you want ‣Same BI Server ETL ‣Same RPD as source ‣Same mapping of aggregates back into BMM layer



Running the Aggregate Persistence Wizard

•Specify measures to aggregate (uses default aggregation method for each measure) •Specify set (or subset) of dimension levels to aggregate at •Generate aggregate persistence script •Run script in same manner (nqcmd) as Summary Advisor (and same model validity rules)



Final set of BI Server-Generated Aggregates

•Three aggregated fact tables •Set of conformed short dimension tables • Initially populated with full data load •Mapped into RPD



Step 3 : Adding “Hot Data” to TimesTen

•As well as holding aggregates, we’ll copy the last six monthsof transactional data into TimesTen ‣More chance that TT will be able to satisfy query ‣More chance query will run within TT when usersdrill-down a hierarchy to detail

•Six months is arbitrary time-span, could be smalleror larger depending on RAM available on Exalytics

•OBIEE will take care of directing query to correct source (or both)



Loading Data Into TimesTen

•Various techniques for loading data into TimesTen ‣Regular database INSERT, UPDATE via OCI, ODBC, JDBC etc ‣ttimportfromOracle utility

-Recommends optimal datatypes and compression clauses -Parallel data load of regular and compressed tables -Note - version shipped with TT “frozen”, morerecent version available for d/l from OTN

‣Custom scripts / application code ‣Dedicated ETL tools (extra license cost)

-Oracle GoldenGate ‣Oracle Data Integrator ‣Oracle DAC(as part of BI Apps data load)



ttImportFromOracle

•Create a TimesTen user matching the schema name in Oracle ‣(or use -ttOwner parameter to specify TT schema for data to go into)

•Specify tables to be imported, and degree of aggressiveness in optimising datatypes ‣typeMap parameters

•Specify degree of aggressiveness in compressing data ‣Compression parameter

•Generates DDL scripts, and data loadingscript for importing data from Oracle

cd c:\tt1122_64_3\support>ttImportFromOracle.exe -oraconn sh/password@orcl -tables CUSTOMERS PRODUCTS SALES CHANNELS PROMOTIONS TIMES -typeMap 2,1 -compression 1 Beginning processing Resolving any tablename wildcards Eliminating any duplicate tables Getting metadata from source Generating database user list Assigning TimesTen datatypes Analyzing source tables (this may take some time) Analyzing table 'SH.CUSTOMERS' ... Analyzing table 'SH.PRODUCTS' ... Analyzing table 'SH.SALES' ... Analyzing table 'SH.CHANNELS' ... Analyzing table 'SH.PROMOTIONS' ... Analyzing table 'SH.TIMES' ... Optimizing TimesTen datatypes Estimating compression ratios Generating output files Finished processing



Running the ttImportFromOracle Scripts

•Edit the auto-generated DataLoad.sql script to import just last six months data ‣Leave the dimension load scripts as default, full load always !!!!

•Run the DDL and DML scripts to initially populate the hot data set !!!!

•Re-run the DataLoad.sql DML script to refresh hot data

timing 1; call ttLoadFromOracle('SH', 'SALES', 'SELECT S.PROD_ID, S.CUST_ID, S.TIME_ID, S.CHANNEL_ID, S.PROMO_ID, S.QUANTITY_SOLD, S.AMOUNT_SOLD FROM SH.SALES S, SH.TIMES T WHERE S.TIME_ID = T.TIME_ID AND T.CALENDAR_MONTH_DESC > ''2001-06''');

Command> connect "DSN=tt_exalytics_3;uid=sh;password=welcome1;oraclepwd=password"; Command> run c:\ttimportscripts\CreateTables.sql Command> run c:\ttimportscripts\LoadData_dims.sql Command> run c:\ttimportscripts\LoadData_fact.sql Command> run c:\ttimportscripts\CreateIndexes.sql Command> run c:\ttimportscripts\UpdateStats.sql



Reviewing the Created Hot Data Tables

•TT tables loaded with data, using compression and optimised datatypes •Need to manually import into RPD physical layer



Adding to RPD BMM Layer as Additional LTSs

•New TT logical table sources (LTSs) mapped into existing BMM layer •Add as new LTSs into BMM layer ‣In addition to aggregate LTSscreated by Summary Advisor /Aggregate Persistence Wizard



Adding Fragmentation Settings to Fact Table LTSs

•Add fragmentation clause to both logical fact table sources ‣Existing Oracle source covers > 6mths ago ‣New TT source covers last 6 mths ‣Use repository variable instead of literal predicate values



BI Server Query Resolution : Detail, Aggs and Hot Data

•BI Server will use original Oracle source for detail-levelqueries using data >6 mths old

•Will use TT hot data tables for queries <6 mths old ‣Or combination of two if query spans both periods

•Will use TT aggregate tables if aggregate exists in TT •Maximise chance of TT being used •Still have means to query larger data set than can fit into TT / Exalytics RAM



Step 4 : Adding New Columns, New Data Sources

•As a stand-alone full database, TT table structures can be added to •Use SQL*Developer, ODI or other regular development tools - but a few “gotchas” •Adding of additional columns into TT tables can cause issues > 1 column addition ‣New columns go into separate TT partition ‣Can slow down table reads as additional read required for each partition ‣ttImportFromOracle doesn’t work with TT tables with > 1 partition

-Needs table to be recreated, or use ttMigrate create -c -relaxedUpgrade •Direct replacement datatypes may not be as efficient as TT-specific ones ‣Especially with Numbers/Integers e.g. use TT_TINYINT/TT_SMALLINT/TT_INTEGER/TT_BIGINT instead of NUMBER



TimesTen Indexing

•Just like regular Oracle RDBMS tables, TimesTen tables benefit from indexing ‣OBIEE Summary Advisor, and ttImportFromOracle, both create indexes on the tables they create ‣ttImportFromOracle creates indexes based on the existing ones on the Oracle source

•Summary Advisor creates indexes based on hierarchy keys and attributes • In most cases, you’ll need to optimize these indexes to reflect real-world query patterns ‣Can improve query performance significantly ‣Avoids on-the-fly index creation

-TmpHashScan and TmpRangeScan operations in query planshttp://www.rittmanmead.com/2013/08/optimizing-timesten-for-exalytics-queries-using-the-timestens-index-advisor/



TimesTen Indexing Best Practices

•Built-in utility for index optimization: TimesTen Index Advisor ‣Start with the default created indexes ‣Use Index Advisor to capture a representative workload ‣Generate new index recommendations, and implement ‣Update optimizer statistics ‣Repeat until no new recommendations are generated

•Consider creating HASH indexes rather than defaultRANGE index for columns used in equality (=) joins ‣Alter DDL to CREATE HASH INDEX... ‣Can be 30%-40% faster than RANGE indexes for equi-joins and equality lookups

Command> call ttindexAdviceCaptureStart(1,0);

Command> call ttindexAdviceCaptureEnd(1);

Command> call ttindexAdviceCaptureOutput(1); < 33, create hash index SALES_i6 on SH.SALES(CHANNEL_ID); > < 2, create hash index SALES_i10 on SH.SALES(PROD_ID); > < 1, create unique hash index SALES_i12 on SH.SALES(TIME_ID); > < 1, create hash index SALES_i12 on SH.SALES(TIME_ID); > < 16, create hash index PRODUCTS_i7 on SH.PRODUCTS(PROD_ID,PROD_CATEGORY); > < 1, create hash index PRODUCTS_i13 on SH.PRODUCTS(PROD_ID,PROD_SUBCATEGORY); > < 35, create hash index TIMES_i8 on SH.TIMES(TIME_ID,CALENDAR_YEAR); > < 1, create index TIMES_i11 on SH.TIMES(CALENDAR_YEAR); > < 8, create unique hash index CUSTOMERS_i9 on SH.CUSTOMERS(CUST_ID); > < 1, create hash index CUSTOMERS_i9 on SH.CUSTOMERS(CUST_ID); >



Step 5 : Adding Incremental Table Refresh to Aggs

•By default, aggregate persistance (and Summary Advisor) refreshes are inefficient •Drops aggregate table, removes from RPD, then recreates and re-adds •Full refresh of aggregate tables, not just new data •How can we make this more efficient?



Use BI Server POPULATE Command For Refresh, Instead

•BI Server log file (nqquery.log) shows POPULATE command after each agg table creation • “Undocumented” command to just refresh the table data, not drop and recreate it

1

populate "SA_Promo_C0000B8E4" mode ( append table connection pool "tt_aggr_store"."Connection Pool") as select_business_model "Sales History (Oracle + TT)"."PROMOTIONS"."PROMO_CATEGORY" as "PROMO_CATE0000B8C6", "Sales History (Oracle + TT)"."PROMOTIONS"."PROMO_CATEGORY_ID" as "PROMO_CATE0000B8C7" , RCOUNT (1) as "Promo_C_0000B8E4SK" from "Sales History (Oracle + TT)"; [[



Add Stale Row Deletion, and WHERE Clause to Script

execute physical connection pool "tt_aggr_store"."Connection Pool" truncate table SA_Categor0000AFE8; SET VARIABLE INACTIVE_SCHEMAS='"tt_aggr_store".."EXALYTICS_AGG_WIZ"': populate "SA_Categor0000AFE8" mode ( append table connection pool "tt_aggr_store"."Connection Pool") as select_business_model "SH (Oracle)"."PRODUCTS"."PROD_CATEGORY_DESC" as "PROD_CATEG0000AFB4", RCOUNT (1) as "Categor_0000AFE8SK" from "SH (Oracle)"; … execute physical connection pool "tt_aggr_store"."Connection Pool" delete from ag_SALES where CALENDAR_M0000AC0A = '2001-12'; SET VARIABLE INACTIVE_SCHEMAS='"tt_aggr_store".."EXALYTICS"': populate "ag_SALES" mode ( append table connection pool "tt_aggr_store"."Connection Pool") as select_business_model "SH (Oracle)"."CUSTOMERS"."CUST_CITY" as "CUST_CITY0000ABC5", "SH (Oracle)"."SALES"."AMOUNT_SOLD" as "AMOUNT_SOL0000ABF6", "SH (Oracle)"."SALES"."QUANTITY_SOLD" as "QUANTITY_S0000ABFB", "SH (Oracle)"."TIMES"."CALENDAR_MONTH_DESC" as "CALENDAR_M0000AC0A", "SH (Oracle)"."PRODUCTS"."PROD_CATEGORY_DESC" as "PROD_CATEG0000AFB4" from "SH (Oracle)" where "SH (Oracle)"."TIMES"."CALENDAR_MONTH_DESC" = '2001-12';



Conclusions

•Whilst both Oracle Database and Oracle TimeTen will have in-memory, column-store /compression abilities, there are distinct benefits in using TimesTen as your “analytic sandbox”

•Database that sits in the application / mid-tier, closer to the BI tools •Database that’s under the control of the BI Team / the business = more ability to control it •Less need for formal, complex change control as it doesn’t affect other users •Can be used to store regular aggregate tables from Summary Advisor / Agg Persistence •Can be used to hold “hot data” - last X months / years of transaction-level data •Add custom tables into TimesTen using SQL*Developer, ODI etc •With some basic knowledge around indexing, memory usage etc, very easy to use •Simple to add incremental refresh to Summary Advisor / Agg Persistence summary tables •Much less expensive per-CPU than Oracle Database Enterprise Edition





Thank You for Attending!

•Thank you for attending this presentation, and more information can be found at http://www.rittmanmead.com

•Contact us at [email protected] or [email protected] •Look out for our book, “Oracle Business Intelligence Developers Guide” out now! •Follow-us on Twitter (@rittmanmead) or Facebook (facebook.com/rittmanmead)

http://www.rittmanmead.com

mailto:[email protected]

mailto:[email protected]



TimesTen as your Analytic Sandbox or - “Going beyond the Summary Advisor” Mark Rittman, CTO, Rittman Mead ODTUG KScope’14, Seattle, June 2014

timesten - beyond the summary advisor (odtug kscope'14)

Data & Analytics

oracle timesten

india e

australia new zealand

oracle exalytics

memory database aggregates

memory oltp database

memory analytic database

level data