make automated testing agile and scalablefiles.meetup.com/18247475/omaha qa event 2015 02 19 hans...
TRANSCRIPT
Make automated testing
agile and scalable
Hans Buwalda
Omaha Software Testing and QA user group
February 21st 2015
©2015 LogiGear Corporation
WELCOME
• Introduction/Welcome
• Cadence
• Announcements– Speakers
– Hosts
– Leaders
• Logistics
• Sponsors– Deliveron
• Feature Speaker Introduction
SPONSOR - DELIVERONDeliveron is a consulting organization that delivers business-driven technology
solutions. Our high success rate is the result of our:
• Project leadership and guidance
• Agile delivery framework and ALM best practices
• Team approach with client engagements
• Integrated client partnership and collaboration
• Deep expertise in core Microsoft technologies
• Strategic offerings
Our solutions span several areas:
• Custom Application Development
• Microsoft Cloud & Azure
• Agile, ALM, & DevOps
• Collaboration & Analytics
• Application Integration
©2015 LogiGear Corporation
Who is your speaker
• Partner of Deliveron
• Software testing company, around since 1994
• Testing and test automation services:– consultancy, training
– test development and automation services
– focus on big and complex testing projects
• Product line: TestArchitect™, TestArchitect for Visual Studio™
– integrating test development with test management and automation
– based on modularized keyword-driven testing
www.logigear.comwww.testarchitect.comwww.deliveron.com
• Dutch guy, in California since 2001
• Background in math, computer science, management
• Since 1994 focusing on automated testing– keyword testing, agile testing, big testing
Hans Buwalda
LogiGear Corporation
hans @ logigear.comwww.happytester.com
©2015 LogiGear Corporation
• No more than 5% of all test cases should be executed
manually
• No more than 5% of all efforts around testing should
involve automating the tests
The 5% Rules of Test Automation
©2015 LogiGear Corporation
Business Approach to Testing
• Business goals for testing and automation– Time-To-Market
– Quality-To-Market
– Control
• Agree on a method– integrated method for testing and automation
– testers in the driver seat
– maximum flexibility, best technology
– minimal technical efforts
– maximum maintainability
• Then select the tooling to support it– providing a framework for the testing and the automation
– encouraging testers to follow the method
– not standing in the way of the engineers
©2015 LogiGear Corporation
Don't Automate Manual Testing
©2015 LogiGear Corporation
Don't just automate manual testing
©2015 LogiGear Corporation
Don't just automate manual testing
Good automated testing is not the same as automating good manual testing. . .
Record and Playback
select window "Logon"
enter text "username", "administrator"
enter text "password", "testonly"
push button "Ok"
select window "Main"
push button "New Customer"
expect window "Customer Information"
select field "First Name"
type "Paul"
select field "Last Name"
type "Jones"
select field "Address"
type "54321 Space Drive"
.
.
.
Scripting
Test CaseDesign
Test CaseAutomation
Test CaseExecution
TEST DESIGNER
AUTOMATION ENGINEER
PLAYBACK
©2015 LogiGear Corporation
Example scripting
/// <summary>/// AddItems - Use 'AddItemsParams' to pass parameters into this method./// </summary>public void AddItems(){
#region Variable DeclarationsWinControl uICalculatorDialog =
this.UICalculatorWindow.UICalculatorDialog;WinEdit uIItemEdit =
this.UICalculatorWindow.UIItemWindow.UIItemEdit;#endregion
Keyboard.SendKeys(uICalculatorDialog,this.AddItemsParams.UICalculatorDialogSendKeys,ModifierKeys.None);
Keyboard.SendKeys(uIItemEdit,this.AddItemsParams.UIItemEditSendKeys,ModifierKeys.None);
}
• State of the art, but stuff for coders . . .
©2015 LogiGear Corporation
Action Based Testing
• Modularized keyword-driven method to describe
behaviors as a series of keyword-based actions
• Particularly suited for large and complex projects
• High degree of agility, even when used in traditional
projects
• High level of automation, 95% or more
• Puts the focus on test design, less on technology
©2015 LogiGear Corporation
Actions
4 actions, each with an action keyword and arguments
read from top to bottom
fragment from a test with actions
acc nr first last
open account 123123 John Doe
acc nr amount
deposit 123123 10.11
deposit 123123 20.22
acc nr expected
check balance 123123 30.33
• The test developer creates tests using actions with keywords and arguments
• Checks are, as much as possible, explicit (specified expected values)
• The automation task focuses on automating the keywords, each keyword is automated only once
• note: instead of keywords one can also use "gherkin" scenarios (BDD), in particular if the tests are smaller
Test Module Plan
Tests
Objectives
Test Module 1
Actions
. . .
AUTOMATION
user password
log in jdoe StarEast
first last brand model
rent car John Renter Ford Escape
rent car John Renter Chevrolet Volt
last total
check bill Renter 140.42
interaction test business test
window control value
enter log in user name jdoe
enter log in password StarEast
window control property expected
check prop log in ok button enabled true
car guycar guy
Action Based Testing
Tests
Objectives
Test Module 2
Tests
Objectives
Test Module n
©2015 LogiGear Corporation
Why Better Test Design?
• Quality and manageability of test – many tests are often quite "mechanical" now
– one to one related to specifications, user stories or requirements, which often is ok, but lacks aggression
– no combinations, no unexpected situations, lame and boring
– such tests have a hard time finding (interesting) bugs
• Better automation– when unneeded details are left out of tests, they don't have to be
maintained
– avoiding "over checking": creating checks that are not in the scope of a test, but may fail after system changes
– limit the impact of system changes on tests, making such impact more manageable
I have become to believe that successful automation is usually less of a technical challenge as it is a test design challenge.
The 3 "Holy Grails" of Test Design
• Effective breakdown of tests into test modules– each with a focused and differentiated scope
• Right approach for each test module– which test design technique
– who to involve, etc
• Proper level of detail in the test lines– what actions
– what checks
©2015 LogiGear Corporation
What's the trick...
©2015 LogiGear Corporation
What's the trick...
• Have or acquire facilities to store and organize you
content
• Edit your stuff
• Decide where to put what
– assign and label the shelves
• Put it there
• If the organization is not sufficient anymore, add to it or
change it
©2015 LogiGear Corporation
Breakdown Criteria
• Straightforward Criteria– Functionality (customers, finances, management information, UI, ...)
– Architecture of the system under test (client, server, protocol, sub systems, components, modules, ...)
– Kind of test (navigation flow, negative tests, response time, ...)
• Additional Criteria– Stakeholders (like "Accounting", "Compliance", "HR", ...)
– Complexity of the test (put complex tests in separate modules)
– Technical aspects of execution (picture checks, special hardware, multi-station, ...)
– Overall project planning (availability of information, timelines, sprints, ...)
– Risks involved (extra test modules for high risk areas)
– Ambition level (smoke test, regression, aggressive, …)
©2015 LogiGear Corporation
• Business objects and business flows– objects are like cars, invoices, locations, etc
– flows are like place, fulfill, pay and close an order
• Other tests– functions and features, like premium calculation or PDF output
– administration, users, security, authorizations
– graphics
– technologies, protocols, ...
– customization, extensibility
– . . .
• Business versus interaction– differentiate within business objects and other categories
– interaction can be further differentiated into: values, UI, keyboard,
etc
Example of considerations
©2015 LogiGear Corporation
Example Top Level Structure
Project create, update, delete/closecopy, movecategorize, enumerate, identifyconvert, serialize, export/import, ...
UI, dialogs, forms, pagesinput (validation, defaulting, dependencies)flows (primary paths, alternate paths)keyboard shortcuts, keyboard controls, ...
. . .
<business object 1>
Lifecycles, data operations
Interaction
Functions and Features
Technologies, protocols, controls
Data (handling, quality, ETL, ...)
Security, authorization, admin
Graphics, multi-media, charts, ...
Load, performance
Integration
Customizing, extensibility
Business Flows
Concurrency, race conditions, ...
Business Objects
processes, transactions, end-to-end, day in the life,combinations of flows, ...
calculations, analyses, PDF output, ...
©2015 LogiGear Corporation
Example of business level test module
• Consists of an (1) initial part, (2) test cases and (3) a final part
• Focus is on business functionality, with a clear business scope
• Navigation details are avoided
TEST MODULE Car Rental Payments
user
start system john
TEST CASE TC 01 Rent some cars
first name last name car
rent car John Doe Ford Escape
rent car John Doe Chevvy Volt
last name amount
check payment Doe 140.4
FINAL
close application
©2015 LogiGear Corporation
Example of an interaction level test
module
• Interaction details that are the target of the test are not hidden
• Focus is not on business ("is the payment amount correct"), but on
interaction ("can I retrieve the payment amount")
TEST MODULE Screen Flow
user
start system john
TEST CASE TC 01 Order button
window button
click main create order
window
check window exists new order
FINAL
close application
©2015 LogiGear Corporation
Example Script for an Action
# get table object, column number and column count
windowName = LIBRARY.NamedArgument("window")
tableName = LIBRARY.NamedArgument("table")
columnName = LIBRARY.NamedArgument("column")
table = ABT.OpenElement(windowName, tableName)
column = table.GetColumnIndex(columnName)
rowCount = table.GetRowCount()
# check the sort order, row by row
previous = table.GetCellText(0, column)
for i in range(1, rowCount):
current = table.GetCellText(i, column)
if current < previous :
LIBRARY.AdministerCheck("order", "sorted", "fails " + str(i+1), 0)
return
previous = current
LIBRARY.AdministerCheck("order", "sorted", "all rows in order", 1)
The following action script will verify whether the rows in a table are sorted:
find the table in the UI
if a value is smaller than before, fail the test
if all rows are ascending, pass the test
get arguments from the test line
def action_checkSortOrder():
©2015 LogiGear Corporation
Using the new action
• By keeping an action generic it can be applied for a
variety of situations
• Some examples of using "check sort order":
window table column
check sort order view orders orders table ID
window table column
check sort order annual results regions revenue
window table column
check sort order inventory cars price
window table column
check sort order registration students last name
Example application
A Test Module for the application
We click a tree node, and then do a check
The actions here are built-in in the framework
Making a new "action"
This action definition uses existing actions to create a new action called "check bitrate"
Argument names can be used in cell expressions, that start with "#", and support the usual string and numeric operators
create a node path from the first two argumentsthe expected value is given
by the 3rd argument
the arguments of the new action
name of the new action
Using the action in a test
These test lines don't care about the navigation in the UI of the application, the focus is functional: verify data
Such functional tests are easier to read with high level actions, and the reduced dependency on navigation makes them (much) easier to maintain in the long term
©2015 LogiGear Corporation
Low-level, high-level, mid-level actions
• Low-level: detailed interaction with the UI (or API)– generic, do not show any functional or business logic
– examples: "click", "expand tree node", "select menu"
• High-level: represent a business function specific to the scope of the test– hide the interaction
– examples: "enter customer", "rent car", "check balance"
• Mid-level: auxiliary actions that represent common sequences of low level actions– usually to wrap a form or dialog
– greatly enhance maintainability
– example: "enter address fields"
enter customer
enter address fields
enter select set . . .. . .
Identifying controls
Identify windows and controls, and assign names to them
These names encapsulate the properties that the tool can use to identify the windows and controls when executing the tests
©2015 LogiGear Corporation
Mapping an interface
• An interface mapping (common in test tools) will map windows and controls to names
• When the interface of an application changes, you only have to update this in one place
• The interface mapping is a key step in your automation success, allocate time to design it well, in particular naming and choosing identifying properties
INTERFACE ENTITY library
interface entity setting title {.*Music Library}
name class label
interface element title text Title:
interface element artist text Artist:
interface element file size text File size (Kb):
name class position
interface element playing time text textbox 4
interface element file type text textbox 5
interface element bitrate text textbox 6
name class position
interface element music treeview treeview 1
©2015 LogiGear Corporation
Hidden Identifying Property
©2015 LogiGear Corporation
Mapping the interface
• An interface mapping will map windows and controls to names
• When the interface of an application changes, you only have to update this in one place
• The interface mapping is a key step in your automation success, allocate time to design it well
©2015 LogiGear Corporation
Stable Automation
• Test your automation– develop separate tests for the actions themselves
• Let developers supply values for hidden identifying properties of UI element, to ease automated identification
– like “accessible name” in .Net and Java, or “id” in Web controls
– a lot of time is spent on identifying and re-identifying UI elements
• Use "active" timing– wait based on conditions in the system under test, avoid fixed “sleeps”
– if needed request developers to provide hooks
• Use automation to identify differences between versions of the system under test
– test in a loop if all control can still be found
• Keep an eye on the test design– unsuccessful automation virtually always can be traced back to poor test design
©2015 LogiGear Corporation
• Identifying properties
• Hooks for timing
• White-box access to anything relevant:– input data (ability to emulate)
– output data (what is underlying data being displayed)
– random generators (can I set a seed?)
– states (like in a game)
– objects displayed (like monsters in a game)
• Testability should be a "must have" requirement– first question in a development project: "how do we test this"?
Testability, some key items
©2015 LogiGear Corporation
Variables and expressions with keywords
• This test does not need an absolute number for the available cars,
just wants to see if a stock is updated
• As a convention we denote an assignment with ">>"
• The "#" indicates an expression
TEST CASE TC 02 Rent some more cars
car available
get quantity Chevvy Volt >> volts
first name last name car
rent car John Doe Chevvy Volt
rent car John Doe Chevvy Volt
car expected
check quantity Chevvy Volt # volts - 2
©2015 LogiGear Corporation
Data driven testing with keywords
• The test lines will be repeated for each row in the data set
• The values represented by "car", "first" and "last" come from the
selected row of the data set
DATA SET cars
car first last value
Chevvy Volt John Doe 40000
Ford Escape Mary Kane 22500
Chrysler 300 Jane Collins 29000
Buick Verano Tom Anderson 23000
BMW 750 Henry Smyth 87000
Toyota Corolla Vivian Major 16000
TEST CASE TC 03 Check stocks
data set
use data set /cars
car available
get quantity # car >> quantity
first name last name car
rent car # first # last # car
car expected
check quantity # car # quantity - 1
repeat for data set
©2015 LogiGear Corporation
"Lead Deputy" Testing
• For "multi station" testing, when multiple machines have to
participate in a test in real-time
• For example if a supervisor needs to approve a withdrawal in a bank
teller system
• Can be "sync" and "parallel" (with a rendezvous point)
acc nr amount tx id
withdraw 123123 10,000 >> tx
name
use deputy supervisor
tx id amount
approve # tx 10,000
use lead
©2015 LogiGear Corporation
Non-UI Testing
• Examples– application programming interfaces (API’s)
– embedded software
– protocols
– files, batches
– databases
– command line interfaces (CLI’s)
– multi-media
– mobile devices
• Impact is mainly on the automation– test design should in most cases be transparent towards the
specific interfaces
• Often non-UI automation can speed up functional tests
that do not address the UI
testing devices
Multiple System Access
SystemUnder Test
Action Automation
APIaccess
protocolaccess
UIaccess
databaseaccess
Test Modules
Device Testing
Software Under Test
AgentABTAutomation
InterfaceInfo
Testing HostDevice
Android
©2015 LogiGear Corporation
Life Cycles
• Product life cycles, rather than task life cycles
• The project planning and execution largely determines when the
products are created
systemdevelopment
testdevelopment
testautomation
©2015 LogiGear Corporation
Typical Time Allocation
TEST DEVELOPMENT
AUTOMATION
time
effo
rts
©2015 LogiGear Corporation
ABT in Agile
Test ModuleDefinition(optional)
Test Module Development
Interface Definition
Action Automation
Test Execution
Sprint ProductsProduct Backlog
Test re-use
Automation re-use
product owner
teamprod owner
& team
User storiesDocumentation
Domain understanding
Acceptance CriteriaPO Questions
SituationsRelations
Agile life cycle
Test development
Main Level Test Modules
Interaction Test Modules
Cross over Test Modules
©2015 LogiGear Corporation
Using ABT in Sprints (1)
• Aim for "sprint + zero", meaning: try to get test
development and automation "done" in the same sprint,
not the next one
– next one means work clutters up, part of team is not working on the
same sprint, work is done double (manually and automated), ...
• Agree on the approach:
– questions like does "done" include tests developed and automated?
– do we see testing and automation as distinguishable tasks and
skillsets?
– is testability a requirement for the software?
©2015 LogiGear Corporation
Using ABT in Sprints (2)
• Just like for development, use discussions with the team and
product owners
– deepen understanding, for the whole team
– help identify items like negative, alternate and unexpected situations
• Start with the main test modules, that address the user stories
and acceptance criteria
– try to keep the main test modules at a similar level as those stories and
criteria
– test modules can double as modeling device for the sprint
• Plan for additional test modules:
– low-level testing of the interaction with the system under test (like UI's)
– crossing over to other parts of the system under test
©2015 LogiGear Corporation
Using ABT in Sprints (3)
• Create good starting conditions for a sprint:
– automation technology available (like hooks, calling functions, etc)
– how to deal with data and environments
– understanding of subject matter, testing, automation, etc
• Do interface mapping by hand, using developer provided
identifications
– saves time by not having to use the viewer or other spy tools
– recording of actions (not tests) will go better
• Make testing and automation part of the evaluations and
hardening sprints
©2015 LogiGear Corporation
Outsourcing and Agile
• Model 1: Full team outsourcing– development, testing and automation
– automated tests can be positioned as part of the delivery
• Model 2: Integrated team:– needs online tool like Jira or Rally
– you must have shared meetings
– advantage: more project time
• Model 3: "2nd unit"– off-shore team works under control of one or more sprint team members
• Model 4: Test Production and management– off-shore team takes the deliveries of the primary team, creates/automates more tests,
and executes and maintains them
©2015 LogiGear Corporation
Summary
• Keywords is one of the techniques for automated testing, in addition
to record & playback and scripting
• In itself keywords are not a silver bullet. It needs a good approach,
careful planning and good organization to be successful, maintainable
and scalable