Selenium Web Testing
Demonstration of a Custom Framework
2012-05-17Aaron Shaver
● Application under test: electronic discovery (pre-trial phase in a lawsuit)
● High level view of the framework and test suite● All test logic and assertions in the test_ directories● Categorized, e.g. can run "nosetests tests_smoke"
● Subfolders contain individual tests● Test dirs are Python packages via the init files, can be imported● nose (test runner) discovers tests based on "test" regular expression
● setup.cfg: nose, WebDriver, and application-specific settings● Change timeouts for WebDriverWait, application task waits● Adjust server URLs, usernames, passwords
● util.py -- utilities module● Product-neutral code: generic functions and classes● Could be reused to test other web apps
● shift.py (our product is called Shift)● Module for all application-specific code● Pulls configuration settings from setup.cfg
● data.py -- another application-specific module● Keeps document count assertions outside of the _test.py files● Easier to maintain: doc counts change sometimes as the app
improves or degrades, e.g. file formats become supported that weren't earlier
● log.txt -- optional test results file● Version-controlled via TortoiseHg (Mercurial) + Bitbucket, so I can
see past test runs● Useful if results exceed cmd.exe buffer size● nose needs special syntax to pipe output of console to the log
● Example of test run outside of log.txt (outputs to cmd.exe window)● Tests slow by usual unit testing standards? App performs many
operations on uploads (recursively extracts, indexes metadata properties, deduplicates, generates reports, etc.)
● Entire test suite takes ~6 hours, easily fitting into an overnight run, so I haven't bothered to setup Selenium Grid yet
● 90% of tests in suite use standard setup routine● App under test needs to ingest data before doing anything useful● All it needs: project name, data source, total docs assertion
● Example of a simple test which does not use the "standard intake workflow"
● Note the product "action" functions: user_login, add_proj● clean_up deletes the project and closes the browser, but only if the
test was successful● Failed tests leave the browser window open, in case it's a new bug
● Built my framework before hearing about PageObjects● Plan was to make "atomic" app actions ("save a search", etc.)● Not as fine-grained as PageObjects● Still a robust layer of abstraction for when UI changes● Ended up clumping together app actions; even less fine-grained than
my own plan
● Developers were nice to me: many UI elements generic across app● No AJAX yet (fingers crossed) -- timing issues from Webdriver itself
● UI is very good / simple / elegant... but not always● Example: reuse of list_item_detail class for both <a> and <div>● Have to ensure what we're clicking is a link● Could do a CSS selector, but would be more fragile
● Potential difficulty: can't automate Java or Silverlight uploaders● Thankfully, administrator backend allows data intake● Uses standard HTML form
● Questions about my automation?● Suggestions for improvement?