Selenium Web Testing

Demonstration of a Custom Framework

2012-05-17Aaron Shaver

● Application under test: electronic discovery (pre-trial phase in a lawsuit)

● High level view of the framework and test suite● All test logic and assertions in the test_ directories● Categorized, e.g. can run "nosetests tests_smoke"

● Subfolders contain individual tests● Test dirs are Python packages via the init files, can be imported● nose (test runner) discovers tests based on "test" regular expression

● setup.cfg: nose, WebDriver, and application-specific settings● Change timeouts for WebDriverWait, application task waits● Adjust server URLs, usernames, passwords

● util.py -- utilities module● Product-neutral code: generic functions and classes● Could be reused to test other web apps

● shift.py (our product is called Shift)● Module for all application-specific code● Pulls configuration settings from setup.cfg

● data.py -- another application-specific module● Keeps document count assertions outside of the _test.py files● Easier to maintain: doc counts change sometimes as the app

improves or degrades, e.g. file formats become supported that weren't earlier

● log.txt -- optional test results file● Version-controlled via TortoiseHg (Mercurial) + Bitbucket, so I can

see past test runs● Useful if results exceed cmd.exe buffer size● nose needs special syntax to pipe output of console to the log

● Example of test run outside of log.txt (outputs to cmd.exe window)● Tests slow by usual unit testing standards? App performs many

operations on uploads (recursively extracts, indexes metadata properties, deduplicates, generates reports, etc.)

● Entire test suite takes ~6 hours, easily fitting into an overnight run, so I haven't bothered to setup Selenium Grid yet

● 90% of tests in suite use standard setup routine● App under test needs to ingest data before doing anything useful● All it needs: project name, data source, total docs assertion

● Example of a simple test which does not use the "standard intake workflow"

● Note the product "action" functions: user_login, add_proj● clean_up deletes the project and closes the browser, but only if the

test was successful● Failed tests leave the browser window open, in case it's a new bug

● Built my framework before hearing about PageObjects● Plan was to make "atomic" app actions ("save a search", etc.)● Not as fine-grained as PageObjects● Still a robust layer of abstraction for when UI changes● Ended up clumping together app actions; even less fine-grained than

my own plan

● Developers were nice to me: many UI elements generic across app● No AJAX yet (fingers crossed) -- timing issues from Webdriver itself

● UI is very good / simple / elegant... but not always● Example: reuse of list_item_detail class for both <a> and <div>● Have to ensure what we're clicking is a link● Could do a CSS selector, but would be more fragile

● Potential difficulty: can't automate Java or Silverlight uploaders● Thankfully, administrator backend allows data intake● Uses standard HTML form

● Questions about my automation?● Suggestions for improvement?