the what, why and how of web analytics testing
DESCRIPTION
Slides from my talk in UNICOM's Next Generation Testing Conference on 13th December in Bangalore on "The What, Why and How of Web Analytics Testing". This is based on my open-source tool - WAAT. More information about the talk is available here: http://goo.gl/FxISG Information about WAAT is available here: http://goo.gl/oUNHUTRANSCRIPT
The What, Why & How of
Web Analy3cs Tes3ng
Anand Bagmar So#ware Quality Evangelist
Principal Consultant, ThoughtWorks
Why do we do Tes3ng?
What is Web Analy3cs?
Why do we need it?
Web Analy*cs Solu*on Providers
Adobe Marke*ng Cloud / Omniture
SiteCatalyst
WebTrends
AWStats
Google Analy*cs
StatCounter
…
How to test Web Analy3cs?
The end report
Pros
• Ensure report is setup correctly
Cons • Licensing • Reports not yet setup • Validate all requests are sent / captured
Tes3ng at the report level
At the origin!
The problem
Manual
The challenge
Repea3ng over-‐and-‐over again
The solu3on
What does WAAT do for me?
• Minimal changes to exisFng test framework • Does not depend on junit / testng / … • Web AnalyFc tool independent • UI Driver framework independent • Browser independent
Flavors
Architecture
• Original flavor • Source code + jar • Supports 3 plugins • Omniture Debugger
• HUpSniffer • JsSniffer
• Available on github
WAAT -‐ Java
• Ruby gem implemented over WAAT-‐Java
• Uses RJB – Ruby-‐Java-‐Bridge • Supports 2 plugins • HUpSniffer • JsSniffer
• Available on github and rubygems.org
WAAT -‐ Ruby
Omniture Debugger
Omniture Debugger How does it work? Pros • OS independent • Run using the regular test-‐user Cons • Browser dependent – need to implement
ScriptRunner for the UI-‐driver in use • Web-‐AnalyFc soluFon dependent – Adobe MarkeFng
Cloud / Omniture SiteCatalyst
HHpSniffer How does it work? Pros • Web-‐analyFc soluFon independent • Browser independent • UI-‐driver independent Cons • 3rd party libraries are OS Dependent • HTTPs not supported out-‐of-‐the-‐box • Run tests as ‘root’
JsSniffer How does it work? Pros • Web-‐analyFc soluFon independent • Browser independent • HTTPs supported out-‐of-‐the-‐box • No dependency on any 3rd party library Cons • Need to write javascript to get the URL from the
browser • UI-‐driver dependent
• Download zip from github / rubygems.org
• Setup
• jpcap / ScriptRunner setup
• Update exis*ng tests
How to use WAAT?
Update exis3ng Tests • Import relevant packages / gems
• Define & provide parameter values
• Ini*alize Engine
• Enable Web Analy*cs Tes*ng
• Define Test Data
• Verify Web Analy*cs repor*ng using WAAT
• Valida*ons
• Disable Web Analy*cs Tes*ng
<Section !!!actionName="OpenUpcomingPage_HttpSniffer"!!numberOfEventsTriggered="1"!
!tagList="pageName=upcoming:index||!! ! !cc=USD||!! ! !ch=upcoming||!! ! !events=event2||"/>!
Define Test Data
Ini3alize Engine
!engine = getInstance(!! ! !webAnalyticTool, !! ! !inputFileType, !! ! !keepLoadedFileInMemory, !!! ! !log4jPropertiesAbsoluteFilePath!! ! !);!
Enable Web Analy3cs Tes3ng
engine.enableWebAnalyticsTesting();!
Verify Web Analy3cs Repor3ng
!Result verificationResult = !!! !engine.verifyWebAnalyticsData (!! ! !inputDataFileName, !! ! !actionName, !! ! !urlPatterns, !! ! !minimumNumberOfPackets);!
Cart.checkout();!
Sample Valida3ons assertNotNull(verificationResult.getVerificationStatus(), "Verification status should NOT be NULL"); !!assertNotNull(verificationResult.getListOfErrors(), "Failure details should NOT be NULL"); logVerificationErrors(verificationResult); !!assertEquals(verificationResult.getVerificationStatus(), Status.PASS, "Verification status should be PASS"); !!assertEquals(verificationResult.getListOfErrors().size(), 0, "Failure details should be empty");!
Disable Web Analy3cs Tes3ng
engine.disableWebAnalyticsTesting();!
hUp://essenceo#esFng.blogspot.com/search/label/waat
hUps://github.com/anandbagmar/
hUp://rubygems.org/gems/WAAT
[email protected] [email protected]
@BagmarAnand