…or proving einstein was right about insanity presented by jim hazen

or proving Einstein was right about insanity Presented by Jim Hazen

Introduction Jim Hazen Veteran of the Software Testing Trenches Experience in software testing, both commercial and consulting work. Working with automation and tools since 1989. Implemented Automation Tools on DOS, OS/2, Windows and Web platforms. Experience with tools; including HP/Mercury, IBM/Rational, and other scripting languages.

Sage Words to Remember Insanity: doing the same thing over and over again and expecting different results. Albert Einstein

Agenda What is Test Automation A Brief History Different types of test automation, tools and uses Automation Myths How to correct them Resources & Skills needed Setting expectations for Management & Others Summary Q&A

What is Test Automation?

Its a Big Topic to Discuss

Bigger Than We Think!

Is It This?

Or More Like This?

Who Builds the Automation?

Some Are Seen This Way

Most Look Like This Instead

What Test Automation Really Is Automation definition Websters the technique of making an apparatus, a process, or a system operate automatically In Software Testing we use tools to drive our processes People only think of tools that execute a test as Automation. Other tools can be used for Test Design, Test Data Generation, Defect Tracking, Test Management, etc. Automation is using a tool during any part of the process of testing.

Who Test Automators Really Are

A Brief History Test Execution tools have been around a long time! Mainframes (pre Mid-80s) PC DOS (Mid 80s 90s) PC GUI & Web (1990s present) Mobile (2008 present) Mainframe Home grown macro tools 1, Hyperion PC DOS AutoTester, SuperKey (macro tool) PC GUI Automated Test Facility (ATF), Rational Robot, QA Run, Winrunner, etc. PC Web QTP, SilkTest, Selenium, TestComplete, etc. Mobile MonkeyTalk, Robotium, etc.

Other Types of Test Automation Tools Other Test Execution Tools xUnit (JUnit & NUnit), SOAPUI, FitNesse, etc. Test Design & Test Data Generation Hexawise, AllPairs, Tosca Test Suite, etc. Test Execution Management HP ALM, MS Team Foundation Server, TestLink, etc. Test Environment Management VMWare vCloud Director, etc. Defect Tracking & Reporting Bugzilla, JIRA, etc.

Automation Myths & Misconceptions

Old Myths You can automate 100% of your tests Automation will cure all problems with testing Automation tools are usable by anyone Record/Playback is all you need Automation allows Management to cut test staff One tool is all you need Automation will speed up testing process * Oldies, but goodies!!

New Myths You dont have to know how to program Opensource tools are free Commercial tools are expensive GUI level automation is a waste of time Codeless and Scriptless tools Commercial tools cant support agile Developers only need to focus on Unit Tests

How to Correct Them 100% Automation Certain tests shouldnt or cant be automated. 80/20 Rule; automate 20% of functionality used 80% of time. Automation solves all testing problems it helps, but it creates new problems too. Automation can be done by anyone Not everyone has the mindset, it is both learned and inherent (logic). Record/Playback is all you need Proven time & time again to be false, and to fail.

How to Correct Them cont. One tool is all you need There isnt a one-size fits all. Automation speeds up testing process The Illusion of Speed problem. Both execution and creation. You dont need to know programming Anyone can Automate problem. Opensource tools are free Costs due to ramp up time (money) and skilled staff to do it (more expensive). 2 Commercial = Expensive Not really in grand scheme of things. 2

How to Correct Them cont. GUI level automation is a waste of time Not really

How to Correct Them cont. Codeless/Scriptless tools Just a different UI. Youre still writing a test which is a set of instructions (code). Commercial tools dont do Agile Yes they do, its how you use the tool that counts. Developers only create Unit Tests Other tools require someone to build the fixtures/code to talk to the SUT.

Why Failures Happen Source: IDT Survey 3 85% Failure Rate! Too High!!

Einstein was right! We keep repeating history, why? Always a new group of management and testers. Companies always looking to do testing faster, cheaper and with less resources. Vendors and others who sell Automagic. Instant gratification mindset. Automation takes time. Not learning from past mistakes. Software not designed with testability in mind. Testers not involved early with decisions, not included by management.

Resources for Automation Dedicated Staff Skilled & experienced staff Dedicated Equipment/Environments Reduce risk of competition of resource or teams stepping on each other Dedicated Budget Automation isnt cheap, plan appropriately Dedicated Management Support Cant have any of the other 3 without it

Skills for Test Automation Its a different view and purpose for the script code. Creating software to test software; i.e., PROGRAMMING! Need basic understanding of programming methods. Understand different types of technology layers GUI, API, System, Web, Services, Database, etc. You need to know how to program in different languages when needed. C#, Java, VBScript, Jscript, XML, SQL, etc.

Test Automation Skills Cont. Need to understand how to hook the software under test and interact with it, and at what level. API, or Object, or Service, or Protocol, etc. Understand how tool interacts with the software under test; what functions/methods are to use. GUI Object methods (functions) Ex: SwfWindow(LoginForm").SwfEdit(ID).SET Jim Need to understand how to pick apart the software and drive it programmatically.

Axioms to remember SMART Standalone Test/Script able to run by itself, minimize dependencies Modular/Maintainable Scripts and code need to be modular in design and easy to maintain Appropriate Build according to need for project, dont try to automate 100% Reusable Build for reuse; keep script code compact and focused. Testable & Traceable Make it easy to debug your code, and be able to link together.

Axioms to remember cont. SEARCH 4,5 Setup Test/Script needs to check for pre-test conditions and flag if needed. Execute The logic and data needed to run the test itself Analyze Validations, Error Handling, etc. Report Progress, Error, Summation of run information (Logfile) Cleanup Test/Script needs to cleanup (temp files, reset data, etc.) Help (or Home)

Setting Realistic Expectations Management & Other Groups 100% automation is an unreachable goal. 6,7 Automation is its own form of software development. Ramp-up time will be required, longer than you think. Benefits realized over time, no immediate payback for investment.

Expectations cont. Management & Other Groups No tool supports all systems & technologies out of the box. Not all testers can write scripts. Specialized resources are a must and need to be dedicated to the work. Nothing is Free, there are always costs involved. Dont believe the marketing hype! Dont buy the snake oil. 8 There is no Magic!

Otherwise, the result

Summary What is Test Automation A lot more than you think! History Lots of tools over the last 30 plus years. Different types of test automation tools Not just for execution of a test only. Automation Myths Old and New; lots of failures due to them. How to fix the problems Be realistic, dispel the myths. Resources and Skills needed for test automation Again, a lot more than you think. It is Programming! Setting expectations for Management & Others Without proper expectations you are painted into a corner.

Remember Its automation, Not Automagic!

Contact Info Jim Hazen Company: Freelance Contractor Home email: [email protected] [email protected] LinkedIn profile: http://www.linkedin.com/profile/view?id=2835000&trk=tab_pro http://www.linkedin.com/profile/view?id=2835000&trk=tab_pro

References 1) Experiences of Test Automation: Case Studies of Software Test Automation, Chapter 5, Graham & Fewster, Addison-Wesley, 2012 2) Test Automation - Let's Talk Business, Igor Gershovich, Connected Testing, 2006. 3) Implementing Automated Software Testing, IDT Survey, Elfriede Dustin et al., Addison-Wesley, 2009. 4) How We Test Software At Microsoft, page 187, Alan Page, Microsoft Press, 2009. 5) How to Automate Testing: The Big Picture, Keith Stobie & Mark Bergman, 1992. 6) Seven Steps to Test Automation Success, Bret Pettichord, http://www.io.com/~wazmo/papers/seven_steps.html, 2001. http://www.io.com/~wazmo/papers/seven_steps.html 7) Success with Test Automation, Bret Pettichord, http://www.io.com/~wazmo/succpap.htm, 2001. http://www.io.com/~wazmo/succpap.htm 8) Test Automation Snake Oil, James Bach, 1996

Thank you for attending this session. Please fill out an evaluation form.

…or proving einstein was right about insanity presented by jim hazen

Documents

testing automation tools

types of test automation

test management

tests automation

test design

way slide

test staff

macro tools