© 2013 wellesley information services. all rights reserved. testing and go-live best practices for...

© 2013 Wellesley Information Services. All rights reserved.

Testing and Go-Live Best Practices for a Successful Dashboard Rollout

Dr. Bjarne BergComerit

© SAP AG 2009 / 2

Deployment,Testing & Change

Management

Best-in-Class Dashboards

Performance & Security

Options & Prototyping

• Scoping vs. requirements gathering • KPI definitions• Required skills and resources• Data connectivity deep dive• Key criteria to retrieve data sets

• Dashboards vs. reports• Answers to dashboard FAQs• SAP BusinessObjects Dashboards

4.0 overview• Product updates and

implementation criteria• Recent changes to dashboard

terms

• Key dashboard roll-out decisions• Mobilizing your dashboard• Support organization• Volume, stress, and UAT• Training and change management

Day 1

Day

3

Seminar Roadmap

Landscape, Connectivity &

Sizing• Hands-on lab: Build a dashboard

with BOBJ Dashboards 4.0• Sizing and scaling recommendations• User management and access

control• SAP NetWeaver® BW

Accelerator and SAP HANA

• Hands-on lab: Advanced techniques• Web service integration and Adobe

Flex Builder• Panel discussion: Dashboard Projects

•Ownership and branding•Post-production changes

4

678

9

10

2

Customization, Branding & Governance

Day 2

• Common causes of poor dashboard performance

• Effective performance testing• Performance-enhancing design

techniques• Preventing unauthorized access

to dashboards• Password protection

and SSO

14

13

11

5

3

We are here

12

1

What We’ll Cover …

• Planning for the SAP BusinessObjects Dashboards initiative• Understanding the JAD, RAD, Agile (XP), or ASAP methodology• Getting the right dashboard requirements• Upgrading and migrating tools• Defining the SAP BusinessObjects Dashboards testing approach• Rolling out dashboards: Big-bang vs. gradual go-lives• Designing the BI support organization• Wrap-up

3

Go-Live Planning for the SAP BusinessObjects Dashboards (Formerly Xcelsius) Initiative• During the planning phase, you should also start planning your

go-live strategy• This includes answering the following questions:

Where are my users located? What is the network capacity? Do I need support people for extended time periods — Different

time zones? Do I need multi-currency support on my dashboards? Do I need multi-language support? What type of users do I have in

each region?

4

Create a user map as part of your project planning. This will help you understand your user base.

User Training Options

• There are four core options for the training strategy• Classroom training

Best when users are similar and centrally located

• Online training Best when users are dispersed, dashboards

are simple, or go-live is over a long time period• Train the trainer

Best when users reside in many locations, multiple languages are involved, and when there is a very high number of users

• One-on-one training Best for executives and senior management Should be done at each user’s office

Communicate and schedule training early in the project so that everyone will be available

Source: Dashboard Insights, 2009

User Training for Complex Dashboards

• If users need training, then you have often failed to create good dashboards

• However, there are times when training is needed; this includes: Interactive dashboards and dashboards with complex graphing

6

In this dashboard, users can budget for travel categories for each month, and also save scenarios.

Therefore, some training is required and should be planned early.

Plan for an Online Help System for Your Dashboards Go-Live• Online help should be created for each dashboard• The online help system should

explain: How numbers are calculated How to read graphs What functionality is

embedded

7

8

How to Create Online Help Systems

• Some ways to create a low-cost online help system include: Flash files

Create a simple help dashboard and embed this into the overall dashboard

Word Create a word document with

screenshots Save it as HTML and store the Web

pages on a Web server You can then link the URL on your

dashboard Custom Application

Use a tool-like front page or any Web authoring tool and create a complex online help system with menus, search functionality, movies that show demonstrations, etc.

Online help centers can include contact information and training schedules. They can also be used to communicate information on future projects.



9

The JAD, RAD, Agile (XP), or ASAP Methodologies

• There are several options for the SAP BusinessObjects Dashboards project Joint Application Design (JAD) Rapid Application Development (RAD) Agile or Extreme Programming (XP) Accelerated SAP Methodology/System Development Lifecycle

(SDLC)• Many of the methodologies are not appropriate for the dashboard

development effort

10

Pick your SAP BusinessObjects Dashboards methodology carefully. Do not use ASAP unless your project is part of a budgeting, consolidation, or planning effort.

The “Waterfall Methodologies” Are Not Good for Dashboards• The System Development Lifecycle (SDLC) methodologies, such

as ASAP, are known collectively as “waterfall methodologies”• They give a false sense of clear-cut stages, and do not address

substantial functionality changes during development It is hard to fix missing functionality during integration testing

11

The waterfall

• Project Plan, Estimating

• Design Strategies, Scope Definition

• Documentation, Issues Db

• Workshop Agenda

• Questionnaires

• End-User Procedures

• Test Plans

• Technical Procedures

• Made Easy guidebooks (printout, data transfer, system administration…)

Fill in the BlankVersus

Start from Scratch

Fill in the BlankVersus

Start from Scratch

Examples for Accelerators:

The challenge with ASAP is that users don’t know what they want until they see it …

Source: SAP

The ASAP Methodology Overview

12

Create Functional

specs

Peer Review

Complete?

Complete?

Peer Review

Complete?

Complete?Structured

walkthrough

Approved?

Configuration

Unit Testing

Integration

Testing

System Testing

Structured

walkthrough

Approved?

No

No

No

No

No

Yes

Yes

Yes

Yes

Yes

Yes

No

Create Technical

specs

Joint Application Design (JAD) — Who Participates?

1. Facilitator — Facilitates discussions, enforces rules 2. End Users — Three to five, attend all sessions 3. Developers — Two or three, question for clarity 4. Tie Breaker — Senior manager; breaks end-user ties, doesn’t

attend5. Observers — Two or three, do not speak 6. SMEs — A few subject matter experts (SMEs) for understanding

business and technology

• Keep it very focused and explore the interfaces. How do the users want to see the screen layouts and functionality?

13

A study of 60 development projects found that,without JAD, 35% of the functionality was missed

(Source: Caper Jones, Software Quality, Reliability, and Error Prediction )

14

Rapid Application Development (RAD)

• RAD has an abbreviated blueprinting phase where meetings are executed in short succession to get the requirements Most of the blueprinting and realization

phases of the project are combined• The first meeting: A one or two-day work session

with uninterrupted time • Who: Power users, casual users, people who today interact with

the current system, and managers who have a stake in the outcome of the dashboards

• How many: A rapid pace is kept in these meetings and the number of attendees is kept to no more than seven people in attendance

• The coordinators should focus on shared information needs and conduct multiple sessions (typically once a week)

Why RAD? Increase involvement, less business disruption, less opinions, more consensus, information sharing, and an education event.

Why RAD? Increase involvement, less business disruption, less opinions, more consensus, information sharing, and an education event.

Agile and Extreme Programming (XP) for SAP BusinessObjects Dashboards• XP was started by programmers who decided

that the traditional requirements gathering sessions took too much time, and often just verified what they already knew

• The argument for XP is that other methodologies were developed to build software for low levels of change and reasonably predictable outcomes

But the business world is no longer very predictable, and software requirements change at extremely high rates

Development can be completed faster with collaborative efforts of paired programmers with small “sprint” timelines and many go-lives

15

The core premise of XP is that you can only pick three out of these four dimensions: cost, quality, scope, and time

Framework for Picking Your Dashboard Methodology

Joint Application Design(JAD)

Rapid Application Development(RAD)

Extreme Programming(EP)

System development Life-Cycle based methodologies

(SDLC)

Impact of FailureLow High

Low

High

Time to Delivery

When to Select Different Methodologies

16

Source: Dr. Berg, DM Review, 2006



17

The SAP BusinessObjects Dashboards Business Requirements• Business requirements can be collected in a variety of ways

based on the methodology that the company employs• It is a complex process and involves the periods:

Discovery and education Formal communication Prototypes and reviews Final approvals

A dashboard implementation does not simply involve a series of black-and-white technical decisions; just because something is technically feasible does not mean it is wise or desirable from a business perspective

A dashboard implementation does not simply involve a series of black-and-white technical decisions; just because something is technically feasible does not mean it is wise or desirable from a business perspective

Source: Gooy_GUI, 2007

You can start with a blank template and fill in the capabilities

Focus on graphs, layout, measures, and navigation

One method is to write storyboards from a user perspective and add needed functionality to support this

Where Do You Start? — First Alternative

19

Get a group of five to seven people for a brainstorming session

Draw the solution, knowing that it may look somewhat different once developed

Focus on the use of space, graphs, navigation, available data, and the purpose of the dashboards

Do not design fixed-format “reports”

Where Do You Start? — Second Alternative

20

Building a Mockup in Excel

• If you can make a “mockup” in Excel, users can see what it may look like very quickly You do not need to have any SAP BusinessObjects tools installed

21This can be done in 30-60 minutes

Prototyping the Dashboard Requirements

• Once the first day of brainstorming is completed, you can create data in Excel and prototype the solution in SAP BusinessObjects Dashboards

• Focus on layout, space management, colors, and basic formatting

22

Plan for multiple weekly prototypes before you get the solution that everyone can agree on

Flexible Options to Meet Many Requirements

• There are often disagreements on how to present numbers and graphs

23

Make your dashboard flexible and present data in many interactive ways

Amount vs. Percentages

Different graphs

Users can select what they want graphed



24

Tool Upgrades and Deployment of New SAP BusinessObjects Dashboards Functionality• When you upgrade your SAP BusinessObjects Dashboards,

consider a multi-step approach Conduct a developer training or workshop session to learn the

new functionality and agree on the upgrade timeline Do a technical upgrade over the weekend Copy all of the dashboards Implement new functionality on the copied dashboards Have a formal feedback session with user involvement to see

how they like the changes Implement the new changes two to six weeks after the technical

upgrade

25

Stability is the key to success in all end-user systems. Even if the new functionality is better, users do not like a system that changes look and feel frequently.

Tool Migration Strategy

• If you are migrating users from a legacy reporting tool to your new dashboards, you need a formal migration strategy. This could include: Maintaining two systems (not recommended) Running two systems in parallel for a short time to reconcile

results Removing a legacy reporting tool as part of go-live

(recommended)• When Vikings settled new lands, they always burned the boats so

that the settlers were 100% committed to the new situation While it caused some anxiety, this is a great migration strategy

that can be used for system migration efforts, as well

26

A “burn-the-boats” reporting migration strategy assures high commitment to the new solution and possibly a high number of new requirements after go-live. Be prepared to support this.



27

The Dashboards Test Methodology

• Dashboards testing should follow a formal methodology

28

Test Strategy

Test Plan

Test Execution

Problem Resolution

• The test strategy is written at the beginning of the project and should include: What will be tested? Who will test it and approve it? Where will the test occur? When will the test be

scheduled?

The test plan is more detailed and should be written once the first prototype is built

The SAP BusinessObjects Dashboards User Acceptance Testing (UAT) Form• By requiring each of the

five to seven UAT members to complete a form for each dashboard, you get solid feedback that you can use in the next RAD development cycle

• During each UAT test cycle, you should solicit detailed feedback on layout, graphs, theme, tables, and navigation

29

Area Approved Change Wanted

Description

Background color & ImageButton layout & locationsLink names & locationsChart and Table names & location

Table location

Table and font size

Colum order

Sort ability

Summary row layout & colors

Graph type

Graph location

Graph labels & size

Graph colors

Ability to change graph typeAbility to select fieldsUsefulness of fields displayedDrilldown to more detail

On-line help

Print

Save

Single sign-on

Other

Approvers Name:

Approvers Signature:

Dashboard Name:

Them

eTa

bles

Gra

phs

Nav

igati

on

Comments:

30

Dashboard Name:

User Type Number FactorConcurrent

UsersCasual Users (less then 2-5 times monthly) 20%Power Users (daily) 40%Executive Users (2-10 times monthly) 25%

Test lab setup Number of PCs required (concurrent users divided by 5)

Batch script created (i.e. Java script)

5 copied of script installed on each PC

Execution Pass IssueStart scripts on each PC (continuous loop)

Monitor BOBJ BI Server Load

Monitor BW App sever load

Monitor BW Server load

PC load time (average of 10 dashboard loads)

Comments

Load Testing

Estimated number of named users

Total:

Findings

The SAP BusinessObjects Load Testing Form

• The load testing is intended to show any performance issues prior to the go-live

• While all areas cannot be tested, this will give you an idea on bottlenecks

• For large scale go-lives, you should consider having test PCs in multiple locations

31

Dashboard Name:

User Type Number FactorConcurrent

UsersCasual Users (less then 2-5 times monthly) 40%Power Users (daily) 80%Executive Users (2-10 times monthly) 50%

Test lab setup Number of PCs required (concurrent users divided by 5)

Batch script created (i.e. Java script)

5 copied of script installed on each PC

Execution Pass IssueStart scripts on each PC (continuous loop)

Monitor BOBJ BI Server Load

Monitor BW App sever load

Monitor BW Server load

PC load time (average of 10 dashboard loads)

Comments

Estimated number of named users

Total:

Stress Testing

Findings

The SAP BusinessObjects Stress Testing Form

• Stress testing is very similar to load testing

• The difference is that the number of concurrent users is expected to be doubled

• Not all companies will use a stress test, nor pass it This is due to the

unrealistic concurrent load and the high hardware costs of meeting them

• However, it provides very useful information

SAP BusinessObjects Dashboards Test Planning

Key Activities

32

1 Create test script 6 Identify key contacts 2 Identify roles to be used 7 Communicate about transports3 Documentation on using test tools 8 Arrange time for progress control4 Procedure for documenting test results 9 Schedule facilities5 Training sessions for using test scripts

Tasks

The business analysts are responsible for planning, coordinating, and executing the testing of dashboards

Large-Scale SAP BusinessObjects Dashboards Test Scheduling — Example

• Each UAT team could have dedicated time in the test room • Provide food and snacks • At least two testers (preferably more) should be assigned to test

each functionality

33

3/1

3/2

3/3

3/4

3/5

3/6

3/7

3/8

3/9

3/1

0

3/1

1

3/1

2

3/1

3

3/1

4

3/1

5

3/1

6

3/1

7

3/1

8

3/1

9

3/2

0

3/2

1

3/2

2

3/2

3

3/2

4

3/2

5

3/2

6

3/2

7

3/2

8

3/2

9

3/3

0

3/3

1

4/1

4/2

DeliverCost and ProfitabilityOrderManufacturingPlan and schedulingDemand planningSource

Resolving outstanding

issues and re-testing

= Morning session 8:30 - noon= Evening session 12:30 - 5:00

Environment preparation

All test results should be logged so that fixes do not impact other dashboards and consistency is maintained

3333

http://images.google.com/imgres?imgurl=www.geocities.com/whunt171/stress.jpg&imgrefurl=http://www.geocities.com/whunt171/HypnoticSolutionsstress.html&h=411&w=490&prev=/images?q=stress+image&start=120&svnum=10&hl=en&lr=&ie=UTF-8&oe=UTF-8&sa=N



34

35

Big-Bang vs. Gradual Go-Lives

• There are many ways to do a gradual rollout of the dashboards• The simplest way is:

Bring all content into the production box Set up the roles in the production

environment Gradually release roles to the end users

• The benefit of doing this is that you can see how the system performs And you can solicit feedback from an increasingly large user

group before you release the dashboards to many users• A gradual go-live can reduce the potentially negative impact on

the organization

Unless the dashboards are for a single department, or very few users, you should always plan for a gradual go-live

Gradual Go-Lives by Region

• A gradual go-live by region makes sense if: Training is needed Multi-language support is required A high number of users are expected Dashboards are tailored to local needs

• When you use a regional go-live strategy, you can also roll out the dashboards to power users first (i.e., one month before go-live) to see how the system works in a real production setting

36

Global dashboards require serious attention to language support, user support, 24/7 availability, and to non-native English speakers

SAP BusinessObjects Dashboards Go-Live by Organization Units• A gradual go-live by Organization Units makes sense if:

Organization is very large Departments have very different needs Dashboards are tailored to department needs Data must be secured between organizations

• When you use an organizational go-live strategy, you need to focus on the branding of each dashboard (e.g., logos of subsidiaries) and on integration of support functions in their organizations (e.g., help desk)

37

Departmental dashboards should have first-level support in their respective business units



38

BI Support Organization — Big Picture

• You need to separate the operations of BI systems from the project work

• If there is no support organization, the BI system quickly becomes an orphan when the project ends

Without a support org there is a risk that future BI projects will be delayed, since the project team has to support previous projects

39

An Example of a Large BI Support Team

This large team can support complex applications, cockpits, BI portals, and broadcasting, while providing training and help desk support, as well as on-going data warehousing production support

Support leaderFull-time

Help desk,user support

Full-time

SAP BI BasisFull-time

Data loads & fixesFull-time

Dashboards(i.e., SAP

BusinessObjects Dashboards)Full-time

Data loads & fixesFull-time

Training,user support

Full-time

Portal, collaboration,KM, security

Full-time

BI Query (i.e., Web Intelligence, BEx)

Full-time

Data quality & data resource mgmt.

Full-time

Note: Job areas are meant for illustration and will vary depending on the BI applications supported

40

What to Include in a BI Dashboard Service-Level Agreement (SLA)1. When must data stores be loaded by (time)?

What will happened if a persistent problem occurs (“swat” teams)? Who is responsible for fixing process chains, and who pays? Do you get a discount for each data store that is not loaded in time?

2. How should Service Packs and Fix Packs be applied? When will SP, FP, and SAP Notes be applied? Who pays for it? Who is responsible for testing them?

3. When will the SAP BusinessObjects system be upgraded? When will upgrades occur, how is the pricing determined? Who pays for it, and who is responsible for testing? How long can the system be offline?

4. Minimum uptime and target uptime What is uptime defined as (data store loaded vs. queries available vs.

security fixes applied vs. portal uptime vs. third-party reporting tool uptime vs. network uptime vs. SAP BusinessObjects Dashboards issues, etc.)?

What are the penalties (money) for missing the dashboard uptime requirements?

41

What to Include in a BI Dashboard Service-Level Agreement (SLA) (cont.)5. Issues log

What issues must be logged? Who owns the log? Do you have access? Can entries be updated, or must an audit trail be preserved?

6. Backup and disaster recovery What is included in the backup and when is it taken? When will restore abilities be tested? How fast must restore occur, and what data stores and users will first have access

(priority list)?7. Who owns the data?

If you switch vendors, who owns the data? How will you get access to the data? Do you get full insights to all? Who, of the vendor’s employees, gets access to your data? Can they share it with

your competitor?8. Service tickets

When will service tickets be monitored? What are the categories and who will resolve them?

What are the resolution processes and timelines? How are customer and support satisfaction measured?

42

43

What to Include in a BI Dashboard Service-Level Agreement (SLA) (cont.)

1. Escalation process What will happened if an

issue cannot be resolved by the internal IT department/vendor and your business SLA manager?

What are the steps needed to terminate the SLA contract, and are there any payments/fault payments or a budget recourse (i.e., move money from cost centers)?

The more details you put into the dashboards SLA upfront, the easier it will be to measure, and the more likely you are to have a successful relationship

44

Selecting Objective Measures for the SLA

• Measures drive behavior, so be careful when selecting them. They should be: Simple to understand

and easy to calculate Meaningful, and drive

the behavior you want to encourage

Controllable and immune to manipulations

• Instruments that collect measures must be consistent over time and as automated as possible

Sites such as the free online KPI Library have over 6,000 standard measure definitions based on the SEC, API, ISO, FASB, GAAP, IEEE, and other organizations

45

Standardized SLA Performance Measures

• Standard measures exist for SLAs. These include: First-Level Call Resolution (FLCR) Average Call Answer Time (ACAT) Percentage Calls Re-Opened Within Two Weeks (PCRT) Percentage of Training Type Calls (PTTC) Number of Tickets Escalated to Level-2 Support (NTE2) Percent of Tickets Escalated to Level-2 Support (PTE2) Number of Tickets per Service Employees (NTSE) Number of Service Employees per User (NSEU) Average Service Employees Training Level (ASET) Percent Service Employees Certified (PSEC) Turnover Rate of Service Employees (TRSE) End-User Satisfaction Score (EUSS) Number of System Failures (NOSF) Number of Critical System Failures (NOCF)

Standardized SLA Performance Measures — More to Consider• Additional measures

Mean Time Between Failures (MTBF) Mean Time Between Critical Failures (MTCF) Mean Time to Provision (MTTP) Mean Time to Repair (MTTR) Percent Up-Time Per System (PUPS) Percent Down-Time Per System (PDPS) Percent Call-Back to Customers (PCBC) Cost per Service Ticket (CPST) Cost per Service Employee (CPSE) Cost per Serviced System (CPSS) SLA Operating Efficiency (SLAOE) SLA Operating Effectiveness

(SLAOF) Employee Turnover Rate (EMTR)

Consultants can drive the measures and create balanced scorecards by assigning weights.BEST PRACTICE: Pick 10-12 initially, and add more as the relationship matures.

Reasonable SAP BusinessObjects Dashboards SLA Performance• Examples of reasonable performance targets

90% of all dashboards run under 20 seconds

System is available 98% of the time Data loads are available at 8 am — 99% of the time User support tickets are answered within 30 minutes (first

response) User support tickets are closed within 48 hours — 95% of

the time System is never unavailable for more than 72 hours —

including upgrades, fix packs, service packs, and disaster recovery

Delta backups are done each 24-hour cycle and system backups are done every weekend

47



48

Where to Find More Information

• Harold Kerzner, Project Management Metrics, KPIs, and Dashboards: A Guide to Measuring and Monitoring Project Performance (Hoboken, NJ: John Wiley & Sons, 2011).

• Leopoldo Simini, “Agile Project Dashboards – Bringing Value to Stakeholders and Top Management (What Can You Do for Your PO Today?)” (July 2011).

• David Parmenter, Key Performance Indicators: Developing, Implementing, and Using Winning KPIs (Hoboken, NJ: John Wiley & Sons, 2010).

49

7 Key Points to Take Home

• Use a RAD, JAD, or XP approach for your dashboards• Have multiple meetings with the user groups• Build interactive prototypes and expect requirements changes• Plan for a formal load testing of the dashboards• Have a roll-out plan and a long-term vision of how to get there• Requirements gathering is interactive, and users are discovering

what they want• Spend serious time planning for support and on-going

enhancements of your dashboards, or they will become useless in a very short time …

50

Your Turn!

51

How to contact me:Dr. Bjarne Berg

[email protected]

Disclaimer

SAP, R/3, mySAP, mySAP.com, SAP NetWeaver®, Duet®, PartnerEdge, and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and in several other countries all over the world. All other product and service names mentioned are the trademarks of their respective companies. Wellesley Information Services is neither owned nor controlled by SAP.

52

© 2013 wellesley information services. all rights reserved. testing and go-live best practices for...

Documents