journey to 1000 tests a day

Journey to 1000 Tests a DayBruce McLeod

Tuesday, 15 November 11

DISCLAIMER

The views and content presented in this presentationare my own and do not represent the views of any of my employers, past or present.

All content is copyright 2011 Bruce McLeod

Creative commons attribution for images is given at the end of this presentation


Every journey has a first stepMy introduction to Test Automation


Record and playback


Huge gains compared to manual execution

• Win32 desktop application

• 5 different clients (95, 98 Beta, NT4, NT 351 and Citrix)

• 5 days work for 3 people could be done in 4 hours

• 97% efficiency improvement


But there were problems

• Tests were not maintainable

• Spent days in maintenance and development

• They were brittle and fragile

• And when I left for my next job ...


KindergartenThe early manual coding years


Manual coding beginsTuesday, 15 November 11

A 2001 test automation recipe

Ingredients1 eager client 1 test application14 test automators


A 2001 test automation recipe

MethodStep 1 : Write an automation “framework”Step 2 : Create a gui mapStep 3 : Write a testStep 4 : Repeat step 4 another 200 times


Lessons learnt

• A Successful project• 200 tests automated in 6 months

• Team was a real dev. team with all the same issues• Versioning• Merging and branching• Coding standards• Individual differences in style and design


ChildhoodThe innocent years of keyword testing


What if we remove the need to code?


Spreadsheets !!!Tuesday, 15 November 11

Keyword driven testing

Object Action DataTextfield (domain) Enter text <domain>

Textfield (username) Enter text <username>

Textfield (password) Enter text <password>Button (login) Click One left click

lObject - Action - data syntax

lAutomation libraries with excel front end


Keyword driven testing• Advantages• Reduction in coding effort• “Anyone” can write tests!!

• Challenges• “Anyone” can write tests!!• Each implementation is custom• Maintenance effort moves from code into ...


… Lots of Spreadsheets !!!Tuesday, 15 November 11

AdolescenceEmbracing open source


Manual coding returnsTuesday, 15 November 11

Until …


Unexpected benefits

• We could now use 1st generation developer tools• Visual Studio 2005• Version control• C# and .net instead of VBA


Unexpected benefits

• Testers were now true peers to developers• Same tools• Same development language• Same recruitment process• Object oriented design


A discovery & epiphany!The “braidy tester” blog posts


The “sparkle” stack• Used to test Microsoft Expression• Key principles• Test cases written in customer view• Verification and data separated from tests• Application internals re-created in test code


How can I execute 1000 tests a day ?


Design goals

• Maintainability• Robustness• Performance• Maintainability• Rapid test development


Maintainability


Devs change stuffIt’s their job !


Each action should occur onceand once only


Each control should be mapped onceand once only


Abstract the engineYou shouldn’t know what’s “under the bonnet”


Robustness


Tests should fail, not your tools.


Embrace regex


Leverage your vendor, and the community


Performance


CPU should be your bottleneck


Never wait !Wait, don’t sleep


Scale-out


Rapid test development


K.I.S.S


Cheat !• Leverage an IDE

•Refactoring• Use code generation•Write less code• Use design patterns


What about the future?


Open source frameworks will consolidateDriven by adoption of webdriver


More focus on mobile automationiOS, Android, Windows Phone


Behaviour driven developmentWill become mainstream


Given I am a logged in userWhen I have a valid credit cardAnd I have 1 click purchasing enabledThen I should see the message “your purchase was successful”And I should be re-directed to the home page.


Automation is development.Hire, plan, develop and execute with that in mind.


Image attribution• Creative commons attribution credit is required for the

following users on flickr• seattlemunicipalarchives• mathias-erhart• jaffathecake• 28096801• loty• brilliantmichael


journey to 1000 tests a day

Technology

test step

test application

test codetuesday

test automatorstuesday

test automationtuesday

problems tests

automation framework

tests application internals