Produced by Wellesley Information Services, LLC, publisher of SAPinsider. © 2014 Wellesley

Information Services. All rights reserved.

Case Study: How Siemens Energy Saved Time and Money by Managing and Automating Testing for Its Recent Upgrade Project

Nathan SharpSiemens Energy

1

Case Study Project

• Primary goal of the project

Technical upgrade of SAP release 4.6C to ECC6.0

Support for SAP and Oracle versions was ending

Required for new functionality after the project

• Secondary goal of the project

Implement and utilize automated testing software

Aim was automation of 70% of manual business tests

Upgrade was the pilot project

2

Introduced a Governance Model

• Attended StarEast Conference (2012) and heard the strong

recommendation for a governance model

• We wanted to avoid a “Wild West” scenario in automation tool

• Governance model covers

Granting access to the tool – but only after training

Defining the development process

Documenting and enforcing naming conventions

Setting up and maintaining security in the tool

3

Started Development of a “Product Bible”

• The Product Bible includes

Roles and responsibilities

New user information

Folder structure

Training material

Issue tracking

Test design

Best practices

4

Started Development of a “Product Roadmap”

• The Product Roadmap includes

When we install and evaluate the next version

Completion date for the Product Bible

Which automation projects will be supported

When additional modules will be acquired and/or implemented

5

Appointed Automation Product Lead

• Responsible internally for the service

• Conduit for the vendor

• Organize training

• Prepare and manage budget

• Build and maintain the Product Roadmap

6

Automated Testing Tool Training

• Diverse group of attendees – both business and technical

• 3 classes, 40+ attendees trained in total

• Each class 4.5 days in duration

• Used SurveyMonkey to get feedback from attendees

7

Building an Example to Impress

• We chose a large end-to-end order-to-cash process

• Collaborative effort across multiple groups

• 30 reusable sub-processes and 391 steps

• No manual intervention

• Would take 2 or 3 days to accomplish manually

• Runs in 12 minutes

• Example has been used in various meetings and demos to help

understanding of test automation and to get buy-in

8

Built a SharePoint System to Support Automation

4 SharePoint linked lists make up the

SharePoint testing database

9

Documented Test Script Execution and Issues

SharePoint used to track testing progress

and allows for instant status reporting

10

Progress at End of Round 1 Testing

• Manual or Automation tests

Executed 96% of tests

Passed 94% of tests

• However, less than 10% automation achieved

11

Progress at End of Round 1 Testing (cont.)

• Some root causes for low amount of automation

Core team was not trained early and first

Not enough time between tool purchase and use

The business group initially driving for the tool developed new

priorities

The automation tool was mastered by only a few due to the

functionality restriction caused by both the automation tool

version and SAP version

No powerful and interested sponsor

12

Second Approach

• Round 2 of upgrade testing completed with 100% executed and

successful

• Three business analysts trained on process capture module (now

usable, as an SAP ECC 6.0 environment was available)

• Those business analysts used the period following completion of

upgrade testing and the end of project to work on further

automation

13

Second Approach (cont.)

• Planned script automation fell below expectations

Automation tasks more complicated and took longer than the

individuals themselves estimated

Business analysts not fully dedicated to the task

In one case, it was just not interesting work for the individual

14

Third Approach

• Tests captured locally by business analysts (and later extended to

end users)

• Fully dedicated off-shore developer completed automation

• 69% of planned automation achieved with this approach

15

Third Approach (cont.)

• Renewed management commitment due to the capture/off-shore

approach which works for us

• Developed a product roadmap – including new versions and

modules

• Automation extended to other projects

• “Lights-out” testing was developed. This is the scheduling of test

scripts to execute while unattended

16

Lights-Out Test Example

• The script is a total of 2,347 steps and runs in approximately 16

minutes

• The script tests the following 7 closing programs:

Reclass PC equity

Reclass 482/483

Z85/Z86 entries

Income statement closeout

Class 6 to 7 allocation

Personnel cost reclass

Calculate and post third-party inventories

17

Lights-Out Test Example (cont.)

• Additionally, the script:

Creates 4 unique multi-line journal entries in two different

company codes using both actual and statistical accounts (for

unique data creation)

Changes status on WBS elements and internal orders

Changes sets and custom CL+ tables

Runs the profit center balancing program

Automatically determines current date/period/fiscal year

Validates the results of the 7 closing programs against

expected results from the JEs created

18

Lights-Out Test Example (cont.)

• Benefits:

This script detected an issue before the changes planned by a

new project were put live

Prevented a major failure in production saving both IT and

business time

19

Expectations from Automation

• Efficiency and labor savings

Hundreds of manual testers freed up

• Shorter project timelines

• With 70% test automation, just one upgrade project on one of our

SAP boxes would eliminate testing costs of $270,000

20

Expectations from Automation (cont.)

• Improved quality

Fewer production defects

Catch defects before they impact the business

• Supports our direction to move from Waterfall to Agile

development frameworks

Testing is required in each development iteration and an

automation tool can allow this to happen more efficiently

21

Internal Conference Held

• Two days of sessions related specifically to the automation tool

• Attended by key players from various Siemens companies

• Shared plans, issues, hints, and tips

22

7 Key Points to Take Home

• Sponsorship is essential for ensuring success of any project

• Test automation is a project to itself – don’t try to combine with

another initiative

• Appointing a Product Lead helps ensure there is someone driving

progress

• Creating a Product Roadmap will give direction to an automation

initiative

• Avoid the “Wild West” in the automation tool with governance

• Ensure there is a mechanism for tracking manual and automatic

test development and execution

• A great way to generate excitement is to create an “Example to

Impress” automated test