fostering long-term test automation success
TRANSCRIPT
BT9 ConcurrentSession11/12/151:30pm
“Fostering Long-Term Test Automation Success”
Presented by:
Carl Nagle
SAS Institute, Inc.
Broughttoyouby:
340CorporateWay,Suite300,OrangePark,FL32073888-268-8770·904-278-0524·[email protected]·www.techwell.com
Carl Nagle SAS Institute, Inc.
At SAS Institute Carl Nagle is a principal software developer, focused for the past seventeen years on the development, integration, and evolution of enterprise software test automation tools and the open source SAFSDEV framework. For more than thirty years, Carl has been responsible for engineering both hardware and software testing systems including the design of advanced electronics for analog and digital subsystem testing with embedded software for high-end power generation and communication systems. This naturally evolved into the development of advanced software testing systems and frameworks. Carl and his team thrive on solving “impossible” test automation scenarios across multiple processes, platforms, and technologies.
Fostering*Long,term*Test*Automa2on*Success
!�Sustainable�!Test!Automa0on!
!with!Carl%Nagle,!!SAS!Ins0tute!
• Why%are%YOU%here?%
• Why%am%I%here?%(Has%it%really%been%9%years?)%
• How%did%we%achieve%ini@al%success?%
• Where%are%we%now?%
• Q%&%A%%!
In%A%Nutshell%
If!these!apply!to!YOU:!!• NEED!or!WANT!easy!test!automa0on!NOW,!• For!MY!current!team!or!project,!• Have!NO!concern!about!others,!or!even!future!projects,!• Hope!to!�fix�!a!failing!automa0on!strategy…!
Then,!you!might!be!looking!at!the!wrong!presenta0on!!!(!But!stay!!!We!might!be!able!to!change!your!mind!;Q)!
Why%Are%YOU%Here?%
If!these!apply!to!YOU:!!• NEED!test!automa0on!processes!that!last!• To!be!shared!and!leveraged!across!projects!• That!don�t!change!with!technology!or!tools!• That!might!�replace�!a!�struggling�!automa0on!strategy…!
Then,!you!might!be!looking!at!the!right!presenta0on!!!!
Why%Are%YOU%Here?%
!!!!!!!!!!!!!Over!18!years!ago,!SAS!Ins0tute!tasked!me!to:!!• Provide!a!common!test!automa0on!frameworkQQeliminate!silos!• Make!test!automa0on!accessible!to!developers!*and*!testers!• Handle!all!product!and!everQchanging!technology!domains!• Sustain!test!automa0on!success!over!years,!not!cycles!• Develop!it!in!an!openQsource!environment!
!!!!!!Today!I!report!that!this!prospers!s0llQQeven!aWer!18!years!!
Why%am%I%Here?%
1997% 2002% 2004% 2006% Today%
1. We!listened!to!the!industry!experts!�We#must#minimize#the#impact#caused#by#changes#in#the#applica4ons#we#are#tes4ng,#*and*#changes#in#the#tools#we#use#to#test#them.�!
2. We!listened!to!poten0al!users!�We#need#to#leverage#and#share#our#test#exper4se#across#current#and#future#projects#regardless#of#the#technologies#or#tools#in#use.�!
3. We!collaborated!with!professional!peers!�Focus#the#test#automa4on#framework#to#deal#with#the#common#components#that#make#up#all#our#unique#applica4ons.�!!
4. We!implemented!it,!!shared!it,!!and!it!con0nues!to!evolve.!
How%did%we%achieve%ini@al%success?%
The!headlines!read:!!
!“In%today’s%environment%of%plumme@ng%cycle%@mes,%test%automa@on%becomes%an%increasingly%cri@cal%and%strategic%necessity.%%%Assuming%the%level%of%tes@ng%in%the%past%was%sufficient%(which%is%rarely%the%case),%how%do%we%possibly%keep%up%with%this%new%explosive%pace%of%webUenabled%deployment%while%retaining%sa@sfactory%test%coverage%and%reducing%risk?%%%The%answer%is%either%more%people%for%manual%tes@ng,%or%a%greater%level%of%test%automa@on.%AVer%all,%a%reduc@on%in%project%cycle%@mes%generally%correlates%to%a%reduc@on%of%@me%for%test.”%
The!headlines!read:!!!
%“With%the%onset%and%demand%for%rapidly%developed%and%deployed%web%clients,%test%automa@on%is%even%more%crucial.%%%Add%to%this%the%cold,%hard%reality%that%we%are%oVen%facing%more%than%one%ac@ve%project%at%a%@me.%%%Perhaps%the%team%is%finishing%up%Version%1.0,%adding%missing%features%to%Version%1.1,%and%prototyping%some%new%technologies%for%Version%2.0!”%
IBM!Ra0onal!Robot,!Func0onal!Tester!
HP!WinRunner,!QuickTest!Pro!
AutoIt!
“We#must#minimize#the#impact#caused#by#changes#in#the#applica4ons#we#are#tes4ng,#*and*#changes#in#the#tools#we#use#to#test#them.”%
DIFFERENT!Technologies!
HTML!3.2,!4.0,!5,!SAP,!DOJO,!GWT!
Flex!
Java!
iOS,!Android!
VB,!.NET,!WPF,!Office!
DIFFERENT!Tes0ng!Tools!
SmartBear!TestComplete,!TestExecute!
EggPlant,!Sikuli,!ImageQBased!Tes0ng!
Mobile!SDKs,!Robo0um!
Selenium!1.0,!WebDriver!
Require%
C/C++!
• Separate!the!test!design!process!and!assets:!!• Completely!independent!of!the!test!execu0on!“engines”!• DataQdriven,!ac0onQbased,!keywordQdriven,!behaviorQdriven!• Support!testQfirst!principles!where!desirable!
!
�We#must#minimize#the#impact#caused#by#changes#in#the#applica4ons#we#are#tes4ng,#*and*#changes#in#the#tools#we#use#to#test#them.�%
PseudoQTypical!Example!Test!Records:!!
Google%%%%SignIn%%%%%%%%%%%Click%Explorer%%FolderTree%%Click%%"Coords=50,200�%%%%(tsk,%tsk,%tsk)%Explorer%%FolderTree%%Click%%My%Documents%%%%%%%%(beher)%
• Separate!the!test!design!process!and!assets:!!• Keep!vola0le!test!data!in!Maps!for!singleQpoint!of!maintenance!
�We#must#minimize#the#impact#caused#by#changes#in#the#applica4ons#we#are#tes4ng,#*and*#changes#in#the#tools#we#use#to#test#them.�%
[Google]%SignIn=�id=gb_70�%%[Explorer]%FolderTree=�Type=Treeview;Index=1�%%[FolderTree]%My%Documents=�Coords=50,200�%
[Google]%SignIn=�id=gb_70�%%[Explorer]%FolderTree=�Type=Treeview;Index=1�%%[FolderTree]%My%Documents=�Coords=50,200�%
The!Map:!
The!Test:!
Keep!vola0le!test!data!in!Maps!for!singleQpoint!of!maintenance.!
• Separate!the!test!design!process!and!assets:!!• Common%API!despite!years!of!evolving!technologies!and!tools!
�We#must#minimize#the#impact#caused#by#changes#in#the#applica4ons#we#are#tes4ng,#*and*#changes#in#the#tools#we#use#to#test#them.�%
• Doesn’t!maler!what!the!run0me!execu0on!engine!is:!!SQA%Robot,%IBM%Ra@onal%Robot,%XDE%Tester,%Func@onal%Tester%HP%WinRunner,%QuickTest%Pro,%SmartBear%Test%Complete,%AutoIt,%EggPlant,%Sikuli,%IBT,%Selenium,%Mobile%SDKs,%Robo@um%…%
Google%%%%SignIn%%%%%%%%%%Click%Explorer%%FolderTree%%Click%%My%Documents%
�We#must#minimize#the#impact#caused#by#changes#in#the#applica4ons#we#are#tes4ng,#*and*#changes#in#the#tools#we#use#to#test#them.�%
hlp://safsdev.sourceforge.net/sqabasic2000/SAFSReference.php!
• Separate!the!test!design!process!and!assets:!!• Test!IDE!can!be!different,!depending!on!skills!and!preferences!• CSV%Text%Files,!Spreadsheets,!Test!Composer!(CSV!Exporters)!• Java!IDE,!Eclipse,!SeleniumPlus!(Java!API!Drivers)!• Custom!IDE,!or!programs!(COM,!Ruby,!Perl…)!invoking!the!API!
�We#must#minimize#the#impact#caused#by#changes#in#the#applica4ons#we#are#tes4ng,#*and*#changes#in#the#tools#we#use#to#test#them.�%
Java!API!Examples:!!
Click(Map.Google.SignIn);%Click(Map.Explorer.FolderTree,%�My%Documents�);%
Click(Map.Google.SignIn);%Click(Map.Explorer.FolderTree,%�My%Documents�);%
Java!API!!Test:!
CSV!Test:!
The!Map:!
[Google]%SignIn=�id=gb_70�%%[Explorer]%FolderTree=�Type=Treeview;Index=1�%%[FolderTree]%My%Documents=�Coords=50,200�%
Then%do%safs%ac@on%Click%on%SignIn%in%Google%Then%do%safs%ac@on%Click%on%FolderTree%in%Explorer%using%�My%Documents�%
Cucumber!Test:!
• We!need!to!be!able!to!move!our!people!around!easily.!• A!person!tes0ng!.NET!can!easily!switch!to!tes0ng!HTML.!
!
• They!can’t!be!bogged!down!constantly!learning!new!tools.!• We!just!want!to!focus!on!test!design!using!the!Common%API.!
!
• They!want!to!support!each!other!across!divisions.!• Experience!applies!to!all!current!and!future!projects.!
“We#need#to#leverage#and#share#our#test#exper4se#across#current#and#future#projects#regardless#of#the#technologies#or#tools#in#use.”%
Applica@on%Map(s)%%(Vola0le!SPOM)!
SAFS%Services%and%Engine(s)!!
Java,!Web,!Flex,!Android!
Applica@on%Under%Test%%Java,!Web,!Flex,!Android!
Logs%
Func@onal%Tester%TestComplete%SeleniumPlus%Robo@um%ImageUBased%Tes@ng%
Executable%%Test%Designs%
%(CSV,!Cucumber,!
!Java,!SeleniumPlus)!
What%this%looks%like%in%a%complex%graph?%
• Implement!independent!Engines!for!test!execu0on:!!• Completely!independent!of!the!applica0on!being!tested!• Engines!interpret!and!implement!the!Common%API%
• Whether!the!domain!is!Java,!.NET,!Flex,!HTML,!SAP,!DOJO…!• Whether!the!tool!is!Adobe,!IBM,!Selenium,!SmartBear…!
• Engines!can!be!services!available!to!any!machine!process!• Engines!don�t!know!and!don�t!care!what!is!driving!them!• Mul0ple!engines!can!run!concurrently,!on!the!same!machine!• Support!engines!centrally!for!ALL!users,!and!ALL!applica0ons!
“Focus#the#test#automa4on#framework#to#deal#with#the#common#components#that#make#up#all#our#unique#applica4ons.”%
What%are%those%internal%Services?%
Logging!
Maps!
Variables!
Drivers!
Engines!
(But%you%really%don’t%have%to%care%about%this.)%
Where%are%we%now?%• We!have!grown!to!4!fullQ0me!inQhouse!framework!developers.!• Internally,!our!userQbase!grows!in!the!US,!Canada,!India,!and!China.!• We!just!completed!development!migra0on!to!SAFSDEV!on!GitHub.!• New!discussion!forums!at!safsdev.freeforums.net.!• Older!forum!archives!at!sourceforge.net/p/safsdev/mailman/!
!• We!have!doc!and!updated!binaries:!SAFSDEV!on!SourceForge.!
!• We!have!always!welcomed!contributors!for!new!engines!or!
features.!• Frankly,!we!have!been!VERY!BUSY!since!our!last!public!talk!in!2006.!
Fostering*Long,term*Test*Automa2on*Success
!�Sustainable�!Test!Automa0on!
!with!Carl%Nagle,!!SAS!Ins0tute!Q*&*A