1

Discussion at the TMF UK - London

Test Automation Framework using Jenkins CI to drive the execution and

present the results

April 2015

Presented at the TMF UKSee http://uktmf.com/

2

About

• This presentation details the thinking behind how I integrated Test Automation with Continuous Integration.

April 2015

Feel free to interrupt at any time!

3

Today’s aim

Sell the vision and principles of continuous test automation to help you make the change on your project.

April 2015

4

Today’s agenda

1st Half – presentation• Walk through the

concepts and case study.

April 2015

2nd Half – Discussion• How to get started?• What needs

automating?• What skills are

needed?• Advantages and

Disadvantages

5

Take away message

“Test automation alone is useful but test automation hooked up with a Continuous

Integration Platform is damn powerful”Viresh Doshi – April 2015

April 2015

6April 2015

Ultimate Aim

“Continuous Deployment” – Sit back and Enjoy the Ride

7

Alternative view point

April 2015

“Continuous Deployment” – Sit back and Relax

And get sacked!

8

About Viresh Doshi

• Test Specialist with over 15 years industry experience.

• Started out testing Air Traffic Management Systems for Lockheed Martin.

• Gained Test Consultancy working for one of the first pure play test consultancies.

April 2015

9

CASE STUDYReal life example

April 2015

10

The problems

• Manually testing with some adhoc automated unit testing.

• The team was transforming to Agile.• Team was using Jenkins CI for building software.• The test environments were not controlled.• Multiple teams from the UK, USA and Canada

updating the main code base and difficult to pinpoint regressions.

April 2015

11

The biggest problem

April 2015

• The software gets to the client and…– Software does not install.– The software is missing a database change.– The client can not login– Basic functionality is broken

12

Pressure to Deliver

April 2015

13

Hold on!

• Jenkins is a build tool and not a testing tool.

April 2015

14

The system

Made up of five key componentsTest

automation tool

Jenkins CI

Jenkins Plugins

GIT – source code Control

April 2015

Glue Tools

15

Test automation tool

• This is any test automation tool that best compliments the system under test e.g.– Test Complete– Selenium– jMeter– Quick Test Professional– Home Grown Code and Tools

April 2015

16

GIT source control

• GIT is open source• Easy to use and manage• Client tools allow for easy usage• Integrates with Jenkins CI• Web management system• Ideal choice for management of all test

configurable items and test code.

April 2015

17

Build Tools and Glue

• ANT• PHING• NANT• Bash Shell Scripts• PowerShell Scripts• Other

April 2015

18

What does it look like?

April 2015

19

In more detail?

The radiator view is what we stare at daily!

Each block is a labeled “job” that does something.

Green indicates success and Red/Yellow indicates failure or issues.

Running progress is shown. Job run time and version is

shown. Red blocks need investigating

immediately.

April 2015

20

What “things” does it do?

• Smoke tests – functional test coverage of the key functions of the system

• End to End business scenarios• New features • Software installation• Configuration• Monitoring of servers , database server and

log files

April 2015

21

Test Starting point

• A new production development build is the initial trigger point

• That build is copied across to the test automation system and then installed and then the suite of automation tests are run and reported.

April 2015

22

System Vision and Values

• Continuous installation• Continuous testing • Continuous reporting• Easy to manage• Source Controlled • Open Source Technology• Scalable

April 2015

23

JENKINSContinuous Integration Platform

April 2015

24

About

• This section contains information on the Jenkins plugins used that achieve Continuous Test Automation.

April 2015

25

Jenkins

What is Jenkins?• A CI Build Platform that allows the user to

create build process jobs that do “things”.• Jenkins Plugins allow the integration and

ability to enrich the “things” being done.• When those “things” are not successful then a

red fail is reported and it’s possible to drill down further and investigate problems.

April 2015

26

Concepts

• Automate Repetitive tasks• Report Pass and Fails• Provide easy access to Log files• Allow for Parameters• Integration to existing systems• Establish a Workspace• Reduce School Boy Errrors.

April 2015

27

Plugin: Build Monitor

What does it do?• Provides a visual “radiator” view of configured

jobs.Why?• The view shows the jobs key features like: how

long it took to execute, running state, pass/fail and build number.

April 2015

28

Plugin: Build Name Setter

What does it do?• Allows for a user set a build nameWhy?• The standard build name is just an

incremented number.

April 2015

29

Plugin: Build Pipeline

What does it do?• Provides a visual view of upstream and

downstream connected jobs.Why?• Builds and jobs can get complicated, so it is

useful to have a visual view of connected jobs.

April 2015

30

Plugin: Config slicing

What does it do?• Allows multiple projects settings to be

adjusted conveniently.Why?• A great way to manage configuration settings

when you have 100s of similar Jenkins jobs.

April 2015

31

Plugin: Copy Artifact

What does it do?• Allows artifacts to be copied from other build

projectsWhy?• Useful for taking successful executables from

other projects for usage.

April 2015

32

Plugin: Green Balls

What does it do?• Simply shows a green for Pass and Red for FailWhy?• Jenkins automatically uses a colour blind

scheme – orange and yellow.

April 2015

33

Plugin: GIT Plugin

What does it do?• Integrates into GIT source code control.Why?• Needed to pull the latest code changes from

source control.

April 2015

34

Plugin: HTML Publisher

What does it do?• A HTML report is published per runWhy?• It was used to publish the HTML execution

results as created by Test Complete Automation Tool

April 2015

35

Plugin: PowerShell

What does it do?• Integrates the ability to run PowerShell scripts.Why?• PowerShell is a very powerful scripting

language that adds endless possibilities for windows based servers.

April 2015

36

Plugin: jUnit

What does it do?• A nice graph with a pass/fail report in jUnit

style is presented per run.Why?• A test is not a test without a pass/fail result.

Any failures can also display stack traces or error screenshots.

April 2015

37

Plugin: MSBuild

What does it do?• Allows MSBuild to build .NET projectsWhy?• The executable isbuilt consistently using

Jenkins instead of adhoc on a developer’s machine.

April 2015

38

Plugin: Sidebar Link

What does it do?• Simply adds a HTML anchor link to the Jenkins

SidebarWhy?• To simplify our life!

April 2015

39

Plugin: Workspace Cleanup

What does it do?• Cleans up the workspace before a job is

executed. Why?• Only that run’s current artifacts are stored.

April 2015

40

CONCLUSIONLessons learned and tips

April 2015

41

Tips

• Try and limit the full suite of tests to 2 hours execution time.

• Too many builds provided by the developers – Limit the execution to twice a day.

• Failures could be with problems with the tests and not the code.

• The system needs to be tested. Watch the test execution and double check that it is doing what you expect.

April 2015

42

Next Steps

• Introduce Maven or equivalent to manage dependencies.

• Introduce a first time install test on virtualized servers.

• Introduce Continuous Delivery

April 2015

43

DISCUSSIONExperiences and Feedback

April 2015

44

Test Automation Challenge

• Manual testing only• Manual Builds• Manual deployments• Manual Release notes• Manual dependencies• Semi Automated Development Integration• Some Automated Test Data Generation• Monthly release to productionApril 2015

45April 2015

• The company is doubled every year for the past three years

• One dedicated QA Body• Developers are the kings• Not everybody is sold• More clients• More staff• More Room for…

My Current Challenge

46

Let’s talk

April 2015

Wake up!

47

Your challenges

• Tell me about your manual challenges?• What is stopping you from automating?• What is the starting point?• Do you have the skills?• Who is responsible?• What can go wrong?• What are the advantages and disadvantages?• What is the overall goal?April 2015

48

Thank You

April 2015