Download - TMF UK - Test automation with CI Disussion
![Page 1: TMF UK - Test automation with CI Disussion](https://reader035.vdocuments.site/reader035/viewer/2022070523/58ecf4351a28ab95198b46a3/html5/thumbnails/1.jpg)
1
Discussion at the TMF UK - London
Test Automation Framework using Jenkins CI to drive the execution and
present the results
April 2015
Presented at the TMF UKSee http://uktmf.com/
![Page 2: TMF UK - Test automation with CI Disussion](https://reader035.vdocuments.site/reader035/viewer/2022070523/58ecf4351a28ab95198b46a3/html5/thumbnails/2.jpg)
2
About
• This presentation details the thinking behind how I integrated Test Automation with Continuous Integration.
April 2015
Feel free to interrupt at any time!
![Page 3: TMF UK - Test automation with CI Disussion](https://reader035.vdocuments.site/reader035/viewer/2022070523/58ecf4351a28ab95198b46a3/html5/thumbnails/3.jpg)
3
Today’s aim
Sell the vision and principles of continuous test automation to help you make the change on your project.
April 2015
![Page 4: TMF UK - Test automation with CI Disussion](https://reader035.vdocuments.site/reader035/viewer/2022070523/58ecf4351a28ab95198b46a3/html5/thumbnails/4.jpg)
4
Today’s agenda
1st Half – presentation• Walk through the
concepts and case study.
April 2015
2nd Half – Discussion• How to get started?• What needs
automating?• What skills are
needed?• Advantages and
Disadvantages
![Page 5: TMF UK - Test automation with CI Disussion](https://reader035.vdocuments.site/reader035/viewer/2022070523/58ecf4351a28ab95198b46a3/html5/thumbnails/5.jpg)
5
Take away message
“Test automation alone is useful but test automation hooked up with a Continuous
Integration Platform is damn powerful”Viresh Doshi – April 2015
April 2015
![Page 6: TMF UK - Test automation with CI Disussion](https://reader035.vdocuments.site/reader035/viewer/2022070523/58ecf4351a28ab95198b46a3/html5/thumbnails/6.jpg)
6April 2015
Ultimate Aim
“Continuous Deployment” – Sit back and Enjoy the Ride
![Page 7: TMF UK - Test automation with CI Disussion](https://reader035.vdocuments.site/reader035/viewer/2022070523/58ecf4351a28ab95198b46a3/html5/thumbnails/7.jpg)
7
Alternative view point
April 2015
“Continuous Deployment” – Sit back and Relax
And get sacked!
![Page 8: TMF UK - Test automation with CI Disussion](https://reader035.vdocuments.site/reader035/viewer/2022070523/58ecf4351a28ab95198b46a3/html5/thumbnails/8.jpg)
8
About Viresh Doshi
• Test Specialist with over 15 years industry experience.
• Started out testing Air Traffic Management Systems for Lockheed Martin.
• Gained Test Consultancy working for one of the first pure play test consultancies.
April 2015
![Page 9: TMF UK - Test automation with CI Disussion](https://reader035.vdocuments.site/reader035/viewer/2022070523/58ecf4351a28ab95198b46a3/html5/thumbnails/9.jpg)
9
CASE STUDYReal life example
April 2015
![Page 10: TMF UK - Test automation with CI Disussion](https://reader035.vdocuments.site/reader035/viewer/2022070523/58ecf4351a28ab95198b46a3/html5/thumbnails/10.jpg)
10
The problems
• Manually testing with some adhoc automated unit testing.
• The team was transforming to Agile.• Team was using Jenkins CI for building software.• The test environments were not controlled.• Multiple teams from the UK, USA and Canada
updating the main code base and difficult to pinpoint regressions.
April 2015
![Page 11: TMF UK - Test automation with CI Disussion](https://reader035.vdocuments.site/reader035/viewer/2022070523/58ecf4351a28ab95198b46a3/html5/thumbnails/11.jpg)
11
The biggest problem
April 2015
• The software gets to the client and…– Software does not install.– The software is missing a database change.– The client can not login– Basic functionality is broken
![Page 12: TMF UK - Test automation with CI Disussion](https://reader035.vdocuments.site/reader035/viewer/2022070523/58ecf4351a28ab95198b46a3/html5/thumbnails/12.jpg)
12
Pressure to Deliver
April 2015
![Page 13: TMF UK - Test automation with CI Disussion](https://reader035.vdocuments.site/reader035/viewer/2022070523/58ecf4351a28ab95198b46a3/html5/thumbnails/13.jpg)
13
Hold on!
• Jenkins is a build tool and not a testing tool.
April 2015
![Page 14: TMF UK - Test automation with CI Disussion](https://reader035.vdocuments.site/reader035/viewer/2022070523/58ecf4351a28ab95198b46a3/html5/thumbnails/14.jpg)
14
The system
Made up of five key componentsTest
automation tool
Jenkins CI
Jenkins Plugins
GIT – source code Control
April 2015
Glue Tools
![Page 15: TMF UK - Test automation with CI Disussion](https://reader035.vdocuments.site/reader035/viewer/2022070523/58ecf4351a28ab95198b46a3/html5/thumbnails/15.jpg)
15
Test automation tool
• This is any test automation tool that best compliments the system under test e.g.– Test Complete– Selenium– jMeter– Quick Test Professional– Home Grown Code and Tools
April 2015
![Page 16: TMF UK - Test automation with CI Disussion](https://reader035.vdocuments.site/reader035/viewer/2022070523/58ecf4351a28ab95198b46a3/html5/thumbnails/16.jpg)
16
GIT source control
• GIT is open source• Easy to use and manage• Client tools allow for easy usage• Integrates with Jenkins CI• Web management system• Ideal choice for management of all test
configurable items and test code.
April 2015
![Page 17: TMF UK - Test automation with CI Disussion](https://reader035.vdocuments.site/reader035/viewer/2022070523/58ecf4351a28ab95198b46a3/html5/thumbnails/17.jpg)
17
Build Tools and Glue
• ANT• PHING• NANT• Bash Shell Scripts• PowerShell Scripts• Other
April 2015
![Page 18: TMF UK - Test automation with CI Disussion](https://reader035.vdocuments.site/reader035/viewer/2022070523/58ecf4351a28ab95198b46a3/html5/thumbnails/18.jpg)
18
What does it look like?
April 2015
![Page 19: TMF UK - Test automation with CI Disussion](https://reader035.vdocuments.site/reader035/viewer/2022070523/58ecf4351a28ab95198b46a3/html5/thumbnails/19.jpg)
19
In more detail?
The radiator view is what we stare at daily!
Each block is a labeled “job” that does something.
Green indicates success and Red/Yellow indicates failure or issues.
Running progress is shown. Job run time and version is
shown. Red blocks need investigating
immediately.
April 2015
![Page 20: TMF UK - Test automation with CI Disussion](https://reader035.vdocuments.site/reader035/viewer/2022070523/58ecf4351a28ab95198b46a3/html5/thumbnails/20.jpg)
20
What “things” does it do?
• Smoke tests – functional test coverage of the key functions of the system
• End to End business scenarios• New features • Software installation• Configuration• Monitoring of servers , database server and
log files
April 2015
![Page 21: TMF UK - Test automation with CI Disussion](https://reader035.vdocuments.site/reader035/viewer/2022070523/58ecf4351a28ab95198b46a3/html5/thumbnails/21.jpg)
21
Test Starting point
• A new production development build is the initial trigger point
• That build is copied across to the test automation system and then installed and then the suite of automation tests are run and reported.
April 2015
![Page 22: TMF UK - Test automation with CI Disussion](https://reader035.vdocuments.site/reader035/viewer/2022070523/58ecf4351a28ab95198b46a3/html5/thumbnails/22.jpg)
22
System Vision and Values
• Continuous installation• Continuous testing • Continuous reporting• Easy to manage• Source Controlled • Open Source Technology• Scalable
April 2015
![Page 23: TMF UK - Test automation with CI Disussion](https://reader035.vdocuments.site/reader035/viewer/2022070523/58ecf4351a28ab95198b46a3/html5/thumbnails/23.jpg)
23
JENKINSContinuous Integration Platform
April 2015
![Page 24: TMF UK - Test automation with CI Disussion](https://reader035.vdocuments.site/reader035/viewer/2022070523/58ecf4351a28ab95198b46a3/html5/thumbnails/24.jpg)
24
About
• This section contains information on the Jenkins plugins used that achieve Continuous Test Automation.
April 2015
![Page 25: TMF UK - Test automation with CI Disussion](https://reader035.vdocuments.site/reader035/viewer/2022070523/58ecf4351a28ab95198b46a3/html5/thumbnails/25.jpg)
25
Jenkins
What is Jenkins?• A CI Build Platform that allows the user to
create build process jobs that do “things”.• Jenkins Plugins allow the integration and
ability to enrich the “things” being done.• When those “things” are not successful then a
red fail is reported and it’s possible to drill down further and investigate problems.
April 2015
![Page 26: TMF UK - Test automation with CI Disussion](https://reader035.vdocuments.site/reader035/viewer/2022070523/58ecf4351a28ab95198b46a3/html5/thumbnails/26.jpg)
26
Concepts
• Automate Repetitive tasks• Report Pass and Fails• Provide easy access to Log files• Allow for Parameters• Integration to existing systems• Establish a Workspace• Reduce School Boy Errrors.
April 2015
![Page 27: TMF UK - Test automation with CI Disussion](https://reader035.vdocuments.site/reader035/viewer/2022070523/58ecf4351a28ab95198b46a3/html5/thumbnails/27.jpg)
27
Plugin: Build Monitor
What does it do?• Provides a visual “radiator” view of configured
jobs.Why?• The view shows the jobs key features like: how
long it took to execute, running state, pass/fail and build number.
April 2015
![Page 28: TMF UK - Test automation with CI Disussion](https://reader035.vdocuments.site/reader035/viewer/2022070523/58ecf4351a28ab95198b46a3/html5/thumbnails/28.jpg)
28
Plugin: Build Name Setter
What does it do?• Allows for a user set a build nameWhy?• The standard build name is just an
incremented number.
April 2015
![Page 29: TMF UK - Test automation with CI Disussion](https://reader035.vdocuments.site/reader035/viewer/2022070523/58ecf4351a28ab95198b46a3/html5/thumbnails/29.jpg)
29
Plugin: Build Pipeline
What does it do?• Provides a visual view of upstream and
downstream connected jobs.Why?• Builds and jobs can get complicated, so it is
useful to have a visual view of connected jobs.
April 2015
![Page 30: TMF UK - Test automation with CI Disussion](https://reader035.vdocuments.site/reader035/viewer/2022070523/58ecf4351a28ab95198b46a3/html5/thumbnails/30.jpg)
30
Plugin: Config slicing
What does it do?• Allows multiple projects settings to be
adjusted conveniently.Why?• A great way to manage configuration settings
when you have 100s of similar Jenkins jobs.
April 2015
![Page 31: TMF UK - Test automation with CI Disussion](https://reader035.vdocuments.site/reader035/viewer/2022070523/58ecf4351a28ab95198b46a3/html5/thumbnails/31.jpg)
31
Plugin: Copy Artifact
What does it do?• Allows artifacts to be copied from other build
projectsWhy?• Useful for taking successful executables from
other projects for usage.
April 2015
![Page 32: TMF UK - Test automation with CI Disussion](https://reader035.vdocuments.site/reader035/viewer/2022070523/58ecf4351a28ab95198b46a3/html5/thumbnails/32.jpg)
32
Plugin: Green Balls
What does it do?• Simply shows a green for Pass and Red for FailWhy?• Jenkins automatically uses a colour blind
scheme – orange and yellow.
April 2015
![Page 33: TMF UK - Test automation with CI Disussion](https://reader035.vdocuments.site/reader035/viewer/2022070523/58ecf4351a28ab95198b46a3/html5/thumbnails/33.jpg)
33
Plugin: GIT Plugin
What does it do?• Integrates into GIT source code control.Why?• Needed to pull the latest code changes from
source control.
April 2015
![Page 34: TMF UK - Test automation with CI Disussion](https://reader035.vdocuments.site/reader035/viewer/2022070523/58ecf4351a28ab95198b46a3/html5/thumbnails/34.jpg)
34
Plugin: HTML Publisher
What does it do?• A HTML report is published per runWhy?• It was used to publish the HTML execution
results as created by Test Complete Automation Tool
April 2015
![Page 35: TMF UK - Test automation with CI Disussion](https://reader035.vdocuments.site/reader035/viewer/2022070523/58ecf4351a28ab95198b46a3/html5/thumbnails/35.jpg)
35
Plugin: PowerShell
What does it do?• Integrates the ability to run PowerShell scripts.Why?• PowerShell is a very powerful scripting
language that adds endless possibilities for windows based servers.
April 2015
![Page 36: TMF UK - Test automation with CI Disussion](https://reader035.vdocuments.site/reader035/viewer/2022070523/58ecf4351a28ab95198b46a3/html5/thumbnails/36.jpg)
36
Plugin: jUnit
What does it do?• A nice graph with a pass/fail report in jUnit
style is presented per run.Why?• A test is not a test without a pass/fail result.
Any failures can also display stack traces or error screenshots.
April 2015
![Page 37: TMF UK - Test automation with CI Disussion](https://reader035.vdocuments.site/reader035/viewer/2022070523/58ecf4351a28ab95198b46a3/html5/thumbnails/37.jpg)
37
Plugin: MSBuild
What does it do?• Allows MSBuild to build .NET projectsWhy?• The executable isbuilt consistently using
Jenkins instead of adhoc on a developer’s machine.
April 2015
![Page 38: TMF UK - Test automation with CI Disussion](https://reader035.vdocuments.site/reader035/viewer/2022070523/58ecf4351a28ab95198b46a3/html5/thumbnails/38.jpg)
38
Plugin: Sidebar Link
What does it do?• Simply adds a HTML anchor link to the Jenkins
SidebarWhy?• To simplify our life!
April 2015
![Page 39: TMF UK - Test automation with CI Disussion](https://reader035.vdocuments.site/reader035/viewer/2022070523/58ecf4351a28ab95198b46a3/html5/thumbnails/39.jpg)
39
Plugin: Workspace Cleanup
What does it do?• Cleans up the workspace before a job is
executed. Why?• Only that run’s current artifacts are stored.
April 2015
![Page 40: TMF UK - Test automation with CI Disussion](https://reader035.vdocuments.site/reader035/viewer/2022070523/58ecf4351a28ab95198b46a3/html5/thumbnails/40.jpg)
40
CONCLUSIONLessons learned and tips
April 2015
![Page 41: TMF UK - Test automation with CI Disussion](https://reader035.vdocuments.site/reader035/viewer/2022070523/58ecf4351a28ab95198b46a3/html5/thumbnails/41.jpg)
41
Tips
• Try and limit the full suite of tests to 2 hours execution time.
• Too many builds provided by the developers – Limit the execution to twice a day.
• Failures could be with problems with the tests and not the code.
• The system needs to be tested. Watch the test execution and double check that it is doing what you expect.
April 2015
![Page 42: TMF UK - Test automation with CI Disussion](https://reader035.vdocuments.site/reader035/viewer/2022070523/58ecf4351a28ab95198b46a3/html5/thumbnails/42.jpg)
42
Next Steps
• Introduce Maven or equivalent to manage dependencies.
• Introduce a first time install test on virtualized servers.
• Introduce Continuous Delivery
April 2015
![Page 43: TMF UK - Test automation with CI Disussion](https://reader035.vdocuments.site/reader035/viewer/2022070523/58ecf4351a28ab95198b46a3/html5/thumbnails/43.jpg)
43
DISCUSSIONExperiences and Feedback
April 2015
![Page 44: TMF UK - Test automation with CI Disussion](https://reader035.vdocuments.site/reader035/viewer/2022070523/58ecf4351a28ab95198b46a3/html5/thumbnails/44.jpg)
44
Test Automation Challenge
• Manual testing only• Manual Builds• Manual deployments• Manual Release notes• Manual dependencies• Semi Automated Development Integration• Some Automated Test Data Generation• Monthly release to productionApril 2015
![Page 45: TMF UK - Test automation with CI Disussion](https://reader035.vdocuments.site/reader035/viewer/2022070523/58ecf4351a28ab95198b46a3/html5/thumbnails/45.jpg)
45April 2015
• The company is doubled every year for the past three years
• One dedicated QA Body• Developers are the kings• Not everybody is sold• More clients• More staff• More Room for…
My Current Challenge
![Page 46: TMF UK - Test automation with CI Disussion](https://reader035.vdocuments.site/reader035/viewer/2022070523/58ecf4351a28ab95198b46a3/html5/thumbnails/46.jpg)
46
Let’s talk
April 2015
Wake up!
![Page 47: TMF UK - Test automation with CI Disussion](https://reader035.vdocuments.site/reader035/viewer/2022070523/58ecf4351a28ab95198b46a3/html5/thumbnails/47.jpg)
47
Your challenges
• Tell me about your manual challenges?• What is stopping you from automating?• What is the starting point?• Do you have the skills?• Who is responsible?• What can go wrong?• What are the advantages and disadvantages?• What is the overall goal?April 2015
![Page 48: TMF UK - Test automation with CI Disussion](https://reader035.vdocuments.site/reader035/viewer/2022070523/58ecf4351a28ab95198b46a3/html5/thumbnails/48.jpg)
48
Thank You
April 2015