kpi run your qa cycles more effectively · run your obi qa cycles more effectively jeff mcquigg sr....
TRANSCRIPT
Contact Us510.818.9480 | www.kpipartners.com
© KPI Partners Inc.
Start Here
Run Your OBI QA Cycles More Effectively
Jeff McQuigg
Sr. Architect
KPI Partners, Inc.
BIWA Summit
Jan 10, 2013
� Senior Architect at KPI Partners
� 10 years OBIEE consulting experience, 20+ years overall
� Personally been involved with over 45+ OBI projects in every capacity (BI
Architect, Data Modeling, RPD Metadata, Business Analyst, Report
Developer, ETL Architect/Developer, Project Manager, Pre-Sales)
� Oracle Ace thought leader for BI & OBI:
• Blogging on OBI best practices since 2007 at
GreatOBI.WordPress.com
• Frequent Oracle Open World speaker
� Personal: My 3,000+ beer bottles of beer are on display at Brewpalace.com
2
www.kpipartners.com
The Leader In Oracle BI & EPM 3
Strategic Consulting | Systems Implementation | Training
Depot Repair Analytics
Fixed Asset Analytics
Manufacturing Analytics
Salesforce.com Analytics
Student Info Analytics
Subledger (SLA) Analytics
and moreD
Transform Data Into InsightTransform Data Into Insight
� Staff built from Oracle/Siebel/Hyperion engineering teams
� On-site, off-shore and blended shore delivery models
� Exclusive pre-built solutions for Oracle BI & E-Business Suite
Oracle BI
Hyperion
Endeca
Exalytics
Oracle BI
Hyperion
Endeca
Exalytics
1. Why QA plans are always different
2. The importance of Builds
3. What resources will you need
4. The benefits of automation
5. How to test OBI Meta Data
Goal: Build better QA plans!
4
� Break the problem down to its layers
� Each layer can be tested independently for the most part• Some assumptions needed
� End-to-End and End-to-Mid testing ensures proper handoff between layers
Extracts
Loads
OBI Model (Ad-hoc)
Security Model
Reports
UI Alerts
5
Test the OBI stack for the following:1. Data Validation – Is the data accurate?
2. User Functionality – Does the UI work properly?
3. Security – Are the appropriate visibility rules applied?
4. Performance – Do the loads and reports run fast enough?
5. Infrastructure – Is the infrastructure reliable and robust?
6
1. Resources always differ• Consultants, IT BI &/or DW team, source system SMEs, internal QA teams, power users
2. Project execution always differs• Agile or iterative vs. traditional waterfall• Document robustness• QA team members’ involvement during project• Different corporate toll gates/methodology
3. Legacy reports / Re-platforming• May or may not have something to compare to
4. QA source application environments differ• Dedicated vs. shared, controlled vs. no control
5. Technical stack differences (less so on BI Apps)• E.g.: Real-time layer, ODS, DW, DM, UI and security integrations, large user volumes
No two QA Plans are ever the same!
“QA Plans are like a box of chocolates”
- Forrest McQuigg
7
� A BI system is different than compiling code modules for a new build
� It’s about the Data Transformation:Applying code to data to make different data
� ETL loads drive the QA plan as they take time (12-48 hours typically)
� ETL has a high degree of integration• Facts depend on Dimensions• Extract � Load � Post Load Process � Aggregate
� Several iterations of ETL Builds are needed to get it right• Full load for base logic• Incremental loads for subsequent loads
� In-line ETL testing during Development can lower QA risk, but Builds are still needed
The ETL Machine
8
Putting the Build cycles together yields a staggered plan
Focus on the FullLoad first
then on Incremental Loads
DMore than 2 Full
loads may be neededQA Begins
Staggering can help if two environments
9
Consider two levels of QA test cases:
1. Code-to-Spec• Identify transformational logic and develop test cases based on
the design spec
2. Spec-to-Business Objective• Ensure the spec was written correctly. Examples:
� All widgets should be assigned to a customer
� If an owner doesn’t exist in System A, then it should use the record from system B
• Perhaps 2-5 goals per spec
• More applicable for custom coded solutions and less so for OOB BI Apps extensions
10
QA Script Development should occur alongside ETL development using the same spec
• Keep QA and Dev resources separated
Design
Spec
Object Code Dev
Object QA Script Dev
QA
Execution
11
� Pipeline as much as your team and environment can handle• Offshore capability helps tremendously• Weekend builds are important• Baby sitting loads takes time & effort
� Multiple DW environments are a must• QA Pass #1 in Environment #1 while Load #2 ongoing in Environment #2• Flexibility is key; use Pre-Production server if available• Greater complexity when SIT and UAT are used in parallel• Post Release 1 Deployments are even more complicated
� Plan on at least 2 QA iterations for Full Load and at least 3 or more for Incremental Load
� Keep in mind any special loads like weekly or monthly jobs
� OBI RPD and Reports can be layered easily on any of these environments
Fix
+
Build
Test
+
Solve
12
� Now is not the time to be optimistic!
� Add sufficient buffers in the schedule• Problems are 100% guaranteed
� Dependencies: A bug in a dimension may require a full reload to retest
� QA Cycles typically can run for several months on a complex system
� Plan on your source system support approach for Incremental Loads early
• Multiple Prod snapshots to manage – or –• Manual creation of test cases in a QA system
13
There are a variety of tests to run for ETL• Table row counts
• Allocations & Summations (Total $ by month)
• Attribute ranges (Min and Max values)
• Specific Transformation logic (If-Then-Else)
• Slowly Changing Dimensions, Snapshot Facts
• Metadata columns (minimal on BI Apps)
• Aggregates sync with base tables
• 10 Random Records tracing
• Engineered Records (great for incrementals and special test cases)
14
� Use SQL script files to automateSelect ApplLogic(Status_CD), Count(*)
from Source.Table
Group by Status_CD order by Status_CD
Minus
Select Status, Count(*)
from Target.Table
Group by Status order by Status;
� For Extracts, consider external file comparison tool
� Source system technical SMEs write source scripts
• Challenging to replicate business transformation logic – get your top experts
� Compare source SQL with Metric totals in OBI Answers (Anchor Metrics)
� Perform both single-hop (Extracts vs. Loads) and multi-hop tests (Source vs. DW Target)
� Leverage database constraints (NOT NULL, FK) during DEV and QA to assist
Status_CD Count(*)
Open 4,129
Closed 65,536
Rejected 80,085
UnSpecified 1,024
Status Count(*)
Open 4,129
Closed 65,536
Rejected 80,085
UnSpecified 1,024
MINUS
15
� Ad-hoc testing of the OBI Model is skipped too frequently
� Even if reports are accurate, what about ad-hoc queries?
� Reports are built on top of the ad-hoc subject area
� OBI thinks and generates SQL – does it do so correctly?
� There are tradeoffs between QA effort vs. confidence
• 100% confidence is not possible Confidence
# Tests
Database
OBI Model
Reports
16
� Ensure proper SQL generation and consistent results
� Can be done on a buggy DW (within reason)• ETL QA Team ensures raw numbers in tables are accurate
• OBI Tests are relative to those numbers even if incorrect
� Automation of OBI Testing: Build test reports in OBI alongside reports in DEV
• Place in separate IT only dashboard
• Can be run in any environment at any time
• Excellent Automation technique
• Great for fast diagnosis of problems – catching unintended consequences
17
1. Skeleton accuracy: (Tables & Joins)• Does OBI generate the proper SQL? (BI Architect)• Do the metric values remain constant for each dimension?
(Tests join paths and aggregates)
2. Derived metric accuracy:• Check OBI Derived metrics based on an Anchor metrics
3. Dimensional attribute accuracy:• Is descriptive data coming over from the source properly?
4. Data security model accuracy:• Are filters applied properly to existing queries?• Is the SQL generated still correct & results consistent?
5. Drill Paths:• Confirm that dimensional drill paths are correct
18
“Tree Top” Tests• Break an Anchor metric out by the tops of each dimension• Make sure correct SQL & correct tables are used (Architect)• Demonstrates Unit Test
• Also try multiple dimensions at the same time if possible
19
� Raw OBI mappings to base Fact table fields are “Base” or “Anchor” metrics
• E.g.: Count(Headcount_Ind) or Sum(Total_Amt)• There can be filtering logic in the RPD• ETL QA verifies that these are correct
� Especially for BI Apps projects
� Derived metrics are those calculated in OBI:• Filtered metrics: Headcount vs. Employee Headcount • Time Series: Prior Year Employee Headcount• Rates & Ratios: # Cases per Employee• Complex Metrics: Rolling 12 Avg. Headcount
� Incorrect Anchor metrics due to ETL do not matter• Prior Year $ should match even if the TOTAL_AMT_USD field is
wrong in the Database• Data fields are variables, just like algebra: Prior Year(x)
M3 M4
M5 M6
M7
M1 M2
Database Table
20
Hint: Capture definitions leveraging other defined fields
� Reuse BI definitions as opposed to always mapping to raw tables
1. Order.Status: IfNull(ORDER_TABLE.STATUS_CD, ‘Unspecified’)
2. # Orders: Count(ORDER_TABLE.ORDER_NUM)
3. # Open Orders: # Orders where Order.Status = ‘Open’
Reuse business terminology as much as possible
Three benefits:1) Makes creating test cases much easier2) Communicates the definitions better to business users3) Helps developers reuse logic when building in RPD
21
� Make reports that confirm the Derived metrics using their Anchor metrics
� Use color coding to assist� Use report calculations to demonstrate� Creative solutions are a must!
Report Calc
22
� Try to avoid downloads to Excel if possible• Hinders automation
� Use two reports if needed – Be Creative!� Provide some instructions;
• These reports should be used for a very long time
RSUM() RCOUNT()
RSUM()
RCOUNT()
23
� Again be creative!� This test verifies Prior Year Ship $ is accurate� Solution: Run the report for 2012 and 2011
• Compare 2011.CY Ship $ to 2012.PY Ship $
2011 2012
24
Extensive OBI ad-hoc testing may take too long
Leverage your reports as surrogates for much of the OBI ad-hoc tests
• They will include multiple dimensions (Skeleton test)
• Various versions of the metrics within the many report structures
• “Hit it from multiple angles”
� More reports per topic � greater confidence
25
� Can be done by an internal QA group not familiar with all the details
� Must have a decent report spec to use• Difficult if an iterative report design approach is taken – minimal specs to use
� Displaying the proper data set - filters (Report shows Open Orders only)
� All columns relatively match (% Deviation = 100 * (Plan – Actual)/ Actual)• Can be done without OBI & ETL testing (Algebra again!)
� Drill downs and navigations work• For navigations, key test is making sure #s remain the same!
� Interactions with prompts
� UI Items: Labeling, formatting, colors, conditional formats, UI Standards, Help links, etc.
26
� 3 Aspects:• Visibility – see the correct dashboards & subject areas & folders• Capability – Answers access, create iBots, etc.• Data Access – Correct dataset - data filtering is happening
� Some security testing should be done during OBI tests• Are the basics working?• Will it work for ad-hoc?
� Creation of test accounts flows easily from security model design
� Final layer is to run reports as test users and verify data set accuracy
• For this user, for this report, are the numbers what they are supposed to be?
Database
OBI Model
Reports
Security
27
ETL QA Roles
Strong SQL, source system knowledge, source
system access for entering records
Systems Analysts, Developers,
Source SMEs
OBI Model
OBI Answers, knowledge of data model, metric
definitions guide
Systems Analysts, QA Team, OBI
Developers, OBI Architect
Reports
Ability to independently confirm data from the
source, general analyst, business user
Systems Analysts, Business
Analysts, End Users, QA Team
Security
General Analyst or power user skillset, Answers Systems Analysts, Business
Analysts, End Users, QA Team
Infrastructure
Deep technical skills, typically those who set up
the infrastructure
Infrastructure Admins
28
1. Perform QA as early in the process as possible
2. Design a QA plan with your team’s skill sets in mind
3. Plan QA Cycles around the Build
4. Automate as much as possible
5. Don’t ignore OBI Ad-hoc tests
29
Q&A
30
Contact Us
The Leader In Oracle BI & EPM
facebook.com/kpipartners
linkedin.com/company/kpipartners
twitter.com/kpipartners
youtube.com/kpipartners
31
Email: [email protected]
Web: kpipartners.com/contact
KPI World Headquarters
39899 Balentine Drive
Suite #375
Newark, CA 94560
Phone: (510) 818-9480
Contact Us
The Leader In Oracle BI & EPM
New York, NY
Chicago, IL
Boston, MA
Minneapolis, MN
San Diego, CA
Greensboro, NC
North America Offices
Bangalore, India Hyderabad, India
Global Offices
32
www.kpipartners.com
33