Script Recipes for TestersAdam Goucher

Manager of Quality, Zerofootprint Software

adam@goucher.cahttp://adam.goucher.ca

About Me

• Testing for 10 years

• Tool-smithing for most of them

• Scripting has been and important part of both my career and testing success

About You

• Want to be a better tester

• Who to find better bugs

• More efficiently

About This Tutorial

• Reusable scripts explained

• Tips for your own scripts

So Why Script?

• Provide new insight into your product

• Generate test data

• Automate repetitive / common / complex tasks

• Replace expensive tools with ones tailored to your environment

Language Choice

• Is far too often a religious choice

• Use what is right for the task

Language Choice

• Do you know the language?

• Could you learn it?

• Is it supported on all your (current) platforms?

• Are those platforms all considered first-class citizens?

• Can it interact with your database?

Language Choice• Does it control your other tools? (Selenium, Watir,

etc.)

• Can you recycle code (Java jars or .NET assemblies) from elsewhere in your application?

• Does it support Unicode as a core data type?

• Is there seamless interaction with the operating system?

• Is there an active member community (to whom you can turn to for help)

Language Choice

Developer Notes

• hack, broken, kludge are not things I like seeing in code I am testing

• fixme and todo in comments imply hidden knowledge

• style

A better grep

• grep is a unix tool for searching for things inside files

• The ‘developer notes’ script is like a cross-platform grep

How to run

• Set the source_base to point to the root of your project source code

• Set the interesting value to the types of file you want to check

• Add items you want to check to the notes list; all lowercase

Future Changes

• Faster

• Use regex

Tip: Complexity

• If you don’t have to do something in a complicated way, don’t.

Tip: Common Case

• Converting both sides of a comparison to a common case means you don’t have to worry about case

Tip: Performance

• Unless performance really matters, don’t worry about it

• Optimizations add complexity

LOUD

• i18n and l10n are hard things to get right

• But is pretty easy to test

• Since we (the testers) are not responsible for content; the translators are

LOUD

• The ‘loud’ screen alters the text which is shown on the screen - ^LOUD$

• Everything that should be translated should be UPPERCASED

• Everything that shouldn’t be is not

• Blocks of text should not be split into different LOUDed strings

How to run

• This script needs to be customized for your organization’s method of creating resource bundles

Tip: Use fake ‘files’

• Rather than using your real resource bundle to build the script, use a variable that looks like a file

• It is faster, and you cannot accidently delete your master copy while on the train without an internet connection

Audible Log Monitoring

• Log files provide a wealth of information

• If your eyes are on a log while doing a test they cannot be somewhere else

• Use your ears instead

How to run

• in watchlist.txt, create a list of strings or regexes which it needs to watch for

• and the wav file it will play when it finds it

Tip: Package for Convenience

• Some people are scared of anything that looks like ‘code’

• More often than not these are ones who are not ‘unix people’

• A tool is only useful if it is used

• All scripting languages have the ability to compile themselves into .exe

Tail’ing the Event Log

• Another tool I miss on windows is tail

• When on remote unix machines I am constantly tailing logs

• ‘Proper’ windows applications however log to the Event Viewer which cannot be tailed (even if you have a windows version installed)

How to run

• Once you have installed the Python Win32 Extensions, just run the script

Future Changes

• Merge with Audible Log Monitoring

Tip: Platform neutrality

• Platform neutrality is a noble goal

• But the goal should be abandoned if the situation makes sense

Browser Percents

• When testing web applications, one of the key decisions to make is how much to test with what browser

• Aggregate 3rd party information, or

• Information from your actual webserver?

How to run

• I only run this script every so often so the filename is in the script

• The regex in the script is written for the Apache access log format

• If you are using a different webserver you just have to change the regex

Tip: Reinvent the wheel

• There are a tonne of libraries and tools which parse log files

• Reinvent the wheel if

• you can learn something

• you want specific information and don’t need anything else

Tip: Filter the data

• Your access log is going to be littered with your own office; filter it out to get a real information

Algorithmic Oracles

• Oracles can be people, specifications (like an RFC) or a comparable product

• Or if something is an algorithm it can be an alternative implication

• In something like a script

How to Use

• Implement your algorithm as a verification task

• Run what your developers produce through the script

• Warning: If you implement it wrong, then you will could be ignoring bugs or creating red herring ones

Test Case Generation

• This is the opposite of the Oracle script

• One of the main reasons I create test data ahead of time is to not have to think under pressure

• Script that thinking

How to Use

• Just implement the algorithm

• Warning: If you implement it wrong and you will be using bad data

Tip: Random

• Real world data is random

• Create your test data the same way

• The infamous Pesticide Paradox

Tip: Dual Purpose

• I put the oracle and generation scripts for the same algorithm in the same script

Excel manipulation

• Excel is a really convenient tool for data creation

• It is also really powerful and comes with its own scripting language

Tip: Layers

• If something has a native scripting layer, use it

GUI Automation

• Commercial GUI automation tools are budget killers

• For a number of application types the open-source alternatives are just as effective

• You just need to fall back on manual scripting sooner

GUI Automation

• Here are the 4 steps I use when automating a UI

1. Record the test

2. Add the testing conditions

3. Traditional data driving

4. Give the script smarts

• 2 - 4 tend to require scripting

Metaframeworks

• Metaframeworks are frameworks which run different test frameworks and aggregates the results

Meta why?

• Imagine from one command getting the results of your JUnit, Selenium, homebrew and Watir at once?

• This is a big bit of scripting...

Tip: Language Variants

• A lot of the major scripting languages have been ported to other interpreters

• Java: Jython, JRuby

• .NET: IronPython, IronRuby

Go Forth and Script!