SAP Testing The anatomy of an ERP/SAP Test Manager

Smart Testing with SAP

|

The Seven Habits of Highly Effective (SAP/ERP) Test Management

Pepijn Paap Senior SAP Test Manager Business Line SAP Solutions

Utrecht, October 2016

2

|

|

SAP HANA TEST Acceptance Criteria

# Exit Criteria Responsible Results (Example)

1 For all functionality, at least, the happy flow (Positive Tests) is tested and approved. Test Manager OK

2 For all critical and high risks specified in the test strategy also negative flows are created, tested and approved. Test Manager OK

3

When User Acceptance Test is finished and test cases are completely executed, no more defects are available with impact:

• High (Blocking, Critical);

• Maximum of 5 Medium;

• Maximum of 25 Low (Cosmetic).

Test Manager

Project Manager

NOK

High = 2

Medium =8

4

For User Acceptance Test an overall percentage of executed test cases is 98%.

• Completeness of test scripts in risk class critical and high is 100%

• Completeness of test scripts in risk class medium is at least 80%

• Completeness of test scripts in risk class low is at least 60%

Test Manager

OK

100%

5 For User Acceptance Test a percentage of positive executed test cases is 95%. Test Manager

OK 97,5% without postponed

97,8% with postponed

6

To finalize UAT phase, the following indication for open defects has to be taken into account:

• Postponed to next release (Wish/ Change)

• Accept (Known-Error) by user organization

• Don’t accept included with reason and remarks (Reject)

• Housed in a different manner to enable the system is able to continue to the next stage (in Software Development Life Cycle).

Test Manager

Project Manager

NOK

Not for all remaining issues a

follow up action is defined

7 A documented work-around is available for Known-Errors. Test Manager N.A.

|

|

Integral international Test Process

6 Sogeti PowerPoint Referentie 2014

Blueprint/ Funct Specs

Process Flows

Requirement

Knowledge transfer

Replicate KT Test

Case Design

Review

Video Capture Training

Test Automation

Capture

RQ coverage

Test Automation

Test Execution Manual

Test Execution

Automation

Defect Management

Test Closure Test End

Report (Go/ NoGo)

Preparation Specification

Execution Completion

Management & Control & Infrastructure

Management & Control & Infrastructure

Client/ India India India Client India NL

India/ NL India/ Client/ NL India/ NL

Ap

pro

ved

Peer/TA Review

India/NL

Input

Input

Training Team

Client

India/ Client/ NL

Client

Resource cost reduce 25% - 35%

Productivity and reusability increase 25%

|

|

|

|

11 Sogeti PowerPoint Referentie 2014

SAP Process Risk Analyses -Rating Guide Process Complexity

1 Very simple process, main flow + max 2 alternate flows visible to the user (less then 3

Tcodes)

2 Simple process, max 6 alternate flows, max 10 data tables involved (4 to 6 Tcodes)

3 Medium complex algorithm, max 6 alternate flows, max 10 data tables involved (7 to 10

Tcodes)

4 Complex algorithms: Multiple decisions or many dependencies or complex

computations (11 to 15 Tcodes)

5 Very complex (combination of complexity factors above) (more then 16 Tcodes)

Technical Impact

1 Standard SAP, simple parameterizations only

2 Standard SAP, extensive parameterizations

3 Enhancement, simple ABAP, simple

parameterizations and no interfaces

Or data setup that has significant impact

on processing (e.g. basic reference

model setup)

4 Enhancement, simple ABAP, average

parameterizations and a few interfaces

Or processing significant data setup

5 Enhancement, complex ABAP, A lot of

parameterizations and a lot of interfaces

Or high impact (master ) data setup

Business Criticality

1 Limited impact (may be caught up or corrected without impacting related business

processes)

2 Errors will induce significant hours to be

spent on internal corrections or rework

Primary business process not affected

3 Errors are likely to affect primary

business processes (e.g. standstill or

under performance) but impact will be

local and have no external visibility

Or are likely to have internal financial

impact, but financial impact limited to

single transactions

4 Errors may lead to significant financial

losses (normally on local scale)

Or may have external visibility

5 Errors may have structural negative impact on profitability or company image

Frequency of use

1 Used incidentally / on a monthly basis or less

2 Used on Weekly basis Or monthly basis and over 10% of all

users involved

3 Normal use is on a daily basis

4 Used continually (during office hours or normal productive hours)

5 Heavy duty: continual usage by a significant number of users

|

What is PRACTICES

SAP specific characteristics classified to points of intrest

12

• Processes

• Reports

• Authorizations

• Customizations

• Transports

• Interfaces

• Conversions

• Enhancements

• Screens

Relation with: • Quality Attributes • Test base • Test phase (and environment) • Test specification technique

P

R

E

C

C

A

T

I

S

PRACTICES “Successful SAP test

Approach”

|

PRACTICES “Successful SAP test approach”

SAP Best Test PRACTICES 13

Motivation Preventive Measures

(Static Testing) Quality

Attribute Test Base Where

Coverage Based (Formal)

Experienced Based (Informal)

Processes Business processes are leading! Review Requirements,

Process flows, Use cases

Functionality

Continuity

Business Blue Print

Process Models

Functional Design

FAT/ UAT / RT Process Cycle Test

Data Comb. Test

Error guessing

Exploratory testing

Reports Everything that is being sent of the

customer has to be 100% OK Review Requirements

Functionality

Performance Functional Design

UT/ SIT / FAT/

UAT/ PAT Data Comb. Test

Error guessing

Exploratory testing

Check list

Authorizations Authorizations have a important role in

SAP process handling

Review Business Blue Print,

Authorization matrix

Security

Suitability

Business Blue Print

Authorization matrix FAT/ UAT / PAT Process Cycle Test

Error guessing

Exploratory testing

Check list

Customizing Understand the intended behavior that is

achieved through customizing

Review Business Blue Print,

Installation doc Functionality

BBP & BPM Installation doc

UT/ SIT / FAT/

UAT / PAT

Process Cycle Test

Data Comb. Test

Error guessing

Exploratory testing

Transports A mistake in a transport can cause

serious issues

Review naming convention

for transports Functionality Functional Design

UT/ SIT / FAT/

UAT/ PAT

Error guessing

Exploratory testing

Check list

Interfaces Testing interfaces is a multi discipline

activity

Review func. & technical

design, Interface doc

Functionality

Connectivity

Functional Design

Interface doc UT/ SIT / FAT/

UAT/ PAT

Process Cycle Test

Data Comb. Test

Error guessing

Exploratory testing

Check list

Conversions Verify (early) if the processes can handle

the converted data

Review func. & technical

design Functionality

Functional Design

Technical Design

SIT / FAT/ UAT /

PAT Data Comb. Test Check list

Enhancements Enhancements require extra test

attention and effort

Review func. & technical

design

Functionality

Suitability

Functional Design

Technical Design

UT/ SIT / FAT/

UAT/ PAT

Process Cycle Test

Data Comb. Test

Error guessing

Exploratory testing

Check list

Screens SAP Screens can be heavily modified Review func. & technical

design User-Friendliness

Functional Design

Technical Design SIT / FAT/ UAT Check list

| 14

Energy & Utility Automotive Manufacturing Consumer

Product Retail Life Science Telecom Distribution Insurance Media &

Entertainment Aviation

Banking

5 6 33 33 4091 100

141 158 183 198 198

330373 377

471

633711

807866

905954 964

1005 1006 1010 1026 1037 1042 1048

0 11 1 1

2 22 2

2 2 2

22 2

2

2

2

2

23

3 33 3 3 3 3 4 4

0 00 0 0

0 00 0

0 0 0

00 0

0

0

0

0

00

0 00 0 0 0 0 0 0

0 00 0 0

0 00 0

0 0 0

00 0

0

0

0

0

00

0 00 0 0 0 0 0 0

273 273309 309 309

392433

486519

542581

620

695739 748

869

1005

1140

1226

1287

1340

1404 1404 1404 1404 1419 1419 1419 1419 1419

0

100

200

300

400

500

600

700

800

900

1000

1100

1200

1300

1400

15006

-ju

n

7-j

un

8-j

un

9-j

un

10

-ju

n

13

-ju

n

14

-ju

n

15

-ju

n

16

-ju

n

17

-ju

n

20

-ju

n

21

-ju

n

22

-ju

n

23

-ju

n

24

-ju

n

27

-ju

n

28

-ju

n

29

-ju

n

30

-ju

n

1-j

ul

4-j

ul

5-j

ul

6-j

ul

7-j

ul

8-j

ul

11

-ju

l

12

-ju

l

13

-ju

l

14

-ju

l

15

-ju

l

Test

Sce

nar

ios

Test Case Execution Plan vs Actuals

POSTPONED

OBSOLETE

BLOCKED

STARTED

FAILED

PASSED

Planned Test Cases

Executed Test Cases

|

|

Tips for tomorrow

16

Summary 1. Raise the bar

2. Continuous learning

3. Find talent

4. Always see challenges

When you master the hard skills in combination with the soft skill, you master the testing

game. And as soon as you master the game, the stakeholders are changing the rules

Looking Ahead Implement a SAP Product Risk Analysis (PRA)

Use the right expertise and methodology (PRACTICES)

HANA migration – Custom code and integration

Cloud migration – Integration

SAP Stakeholder management

Choice the battles you want to win

Reduce resource costs and increase productivity using different

testing models

5. Be fast

6. Be efficient

7. Be critical

Professional SAP testing….

….our expertise !

Thank You