model-based testing with keywords
DESCRIPTION
Model-based testing can be a powerful alternative to just writing test cases. However, modeling tools are specialized and not suitable for everyone. On the other hand, keyword-driven test automation has gained wide acceptance as a powerful way to create maintainable automated tests, and, unlike models, keywords are simple to use. Hans Buwalda demonstrates different ways that keyword testing and models can be combined to make model-based testing more readily accessible. Learn how you can use keywords to create the models directly. The results of this "poor man's approach" to model-based testing are clean, concise test cases that are interpreted dynamically. In other words, the model executes the tests rather than generating the tests for execution by another tool. This allows the model to actively respond to changing conditions in the application under test. See this demonstrated with a simple state-transition model, written with keywords, that plays a game until all relevant situations have been visited.TRANSCRIPT
W9 Concurrent Class
10/2/2013 1:45:00 PM
"Model-Based Testing with
Keywords"
Presented by:
Hans Buwalda
LogiGear Corporation
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com
Hans Buwalda
LogiGear
An internationally recognized expert in testing, Hans Buwalda is a pioneer of keyword-driven
test automation, an approach now widely adopted throughout the testing industry. Originally
from the Netherlands, Hans is the CTO of LogiGear, directing the development of the successful
Action Based Testing™ methodology for keyword-driven test automation and its supporting
TestArchitect™ toolset. Prior to joining LogiGear, Hans served as project director at CMG (now
CFI).
9/20/2013
1
© 2013 LogiGear Corporation. All Rights Reserved
Hans Buwalda
LogiGear
STARWEST 2013, Presentation W9
Anaheim, Wednesday Oct 2nd, 2013
Model-Based Testing
with Keywords
© 2013 LogiGear Corporation. All Rights Reserved
Topics for this presentation
Action Words (Action Based Testing)
Testing with models
Using actions to describe models
Non-deterministic applications (like games)
Examples
9/20/2013
2
© 2013 LogiGear Corporation. All Rights Reserved
Action Words / Action Based Testing
Around since 1994 presented for the first time at Eurostar in 1996
Offers testers the ability to create a "language" to
express tests and expected results benefit: flexibility
Automation concentrates on implementing the
language, not the tests benefit: maintainability
The language concept works for straightforward
cases, but also for expressing and controlling
models
© 2013 LogiGear Corporation. All Rights Reserved
The test developer creates tests in "test modules", using actions
Each action consists of a keyword and arguments
The automation task focuses on automating the actions
Each action is automated only once
Action Based Testing
number name quantity
new product P-9009 Sledge Hammer 5
number quantity
add quantity P-9009 20
add quantity P-9009 3
add quantity P-9009 6
number quantity
check quantity P-9009 34
actions, each with a
keyword and
arguments
"34" is the
expected value
here
read from top
to bottom
fragment from a test with actions
9/20/2013
3
© 2013 LogiGear Corporation. All Rights Reserved
Example of business level test module
Consists of an (1) initial part, (2) test cases and (3) a final part
Focus is on business functionality, with a clear business scope
Navigation details are avoided
TEST MODULE Car Rental Payments
user
start system john
TEST CASE TC 01 Rent some cars
first name last name car
rent car John Doe Ford Escape
rent car John Doe Chevvy Volt
last name amount
check payment Doe 140.4
FINAL
close application
© 2013 LogiGear Corporation. All Rights Reserved
Example of an interaction level test module
Lay-out the same, with an initial part, test cases and a final part
Interaction details that are the target of the test are not hidden
Focus is not on business ("is the payment amount correct"), but on
interaction ("do I see the payment amount")
TEST MODULE Screen Flow
user
start system john
TEST CASE TC 01 Order button
window button
click main create order
window
check window exists new order
FINAL
close application
9/20/2013
4
© 2013 LogiGear Corporation. All Rights Reserved
Variables and expressions with keywords
This test does not need an absolute number for the
available cars, just wants to see if a stock is updated
As a convention we denote an assignment with ">>"
The "#" indicates an expression
TEST CASE TC 02 Rent some more cars
car available
get quantity Chevvy Volt >> volts
first name last name car
rent car John Doe Chevvy Volt
rent car John Doe Chevvy Volt
car expected
check quantity Chevvy Volt # volts - 2
© 2013 LogiGear Corporation. All Rights Reserved
Data driven testing with actions
The test lines will be repeated for each row in the data set
The values represented by "car", "first" and "last" come
from the selected row of the data set
DATA SET cars
car first last value
Chevvy Volt John Doe 40000
Ford Escape Mary Kane 22500
Chrysler 300 Jane Collins 29000
Buick Verano Tom Anderson 23000
BMW 750 Henry Smyth 87000
Toyota Corolla Vivian Major 16000
TEST CASE TC 03 Check stocks
data set
use data set /cars
car available
get quantity # car >> quantity
first name last name car
rent car # first # last # car
car expected
check quantity # car # quantity - 1
repeat for data set
9/20/2013
5
© 2013 LogiGear Corporation. All Rights Reserved
Identify modules
Test Development Plan
Test Cases
Test
Objectives
Test Module 1
Test Cases
Test
Objectives
Test Module 2
Test Cases
Test
Objectives
Test Module N
Actions
. . .
ACTION AUTOMATION
Break down
Automate
for each module
decide what approach
to use, for example
models
© 2013 LogiGear Corporation. All Rights Reserved
Models
You can make models for just about everything automated systems, business processes, organizations,
the universe, conference presentations, . . .
A model is an abstraction it represents certain aspects of a subject matter in a more
accessable form, it hides others
when using a model beware of what it is that is hidden
Examples of formats: picture, set of mathematical
formulas, a simulation program, toy, game
9/20/2013
6
© 2013 LogiGear Corporation. All Rights Reserved
Models for testing
For a system under test, a model can usually
describe: structure
behavior
data
rules, calculations
Scope of the modeling: subject matter under test
system under test
the test
Models and testing: models can be used to generate test cases
however, actions can be used to describe models
© 2013 LogiGear Corporation. All Rights Reserved
Model: Implicit Specification of a Test
Using a "model" to get some variation in an automated
test
Models can be simpler than you think you might have used them already
Examples: business rules, like "entitled for social security when 67"
formulas, like "sales tax is 0.09 times price"
state-transition diagrams
monkey tests: random or semi-random moves through a GUI to spot
issues
Models can have an "oracle" that can help verify if a test
result is ok example for a mapping application: driving times both ways are
normally not far apart*
*source: Harry Robinson, Microsoft Bing Maps, this model-based test was executed with LogiGear's TestArchitect
9/20/2013
7
© 2013 LogiGear Corporation. All Rights Reserved
State Models
Among the most commonly used models
Particular good to describe behavior of a system
A system is describes as a collection of "states", plus
input events that cause "transitions" between the states a transition typically has outputs that can be tested
states can be fixed or derived from equivalence classes, for example
including parameters to further details them out
example of a state: "monster attacks player", which monster can then
be a parameter
To apply a state model: your test system must be able to detect which state the system is in
be able to generate the events that migrate you through the states
© 2013 LogiGear Corporation. All Rights Reserved
State Transition Diagram
doors closed
first check
second check
alert state
self destruct doors open
initial
signal
correct
correct
incorrect
incorrect
signal
tampering
time-out
9/20/2013
8
© 2013 LogiGear Corporation. All Rights Reserved
State Transition Table
NR STATE EVENT OUTPUT NEXT
1 doors closed signal beep first check
2 doors closed tampering hi-lo alert state
3 first check correct code green doors open
4 first check incorrect code hi-lo second check
5 doors open close signal beep doors closed
6 second check correct code green doors open
7 second check incorrect code hi tone self destruct
8 self destruct anything boom oblivion
9 alert state time out hi tone self destruct
10 alert state signal beep first check
Car Safety System, Model 007, Luxury Version*
*certain insurance restrictions might apply, please check with your agent
© 2013 LogiGear Corporation. All Rights Reserved
When to use Models?
Functionality under test is complex like an airplane control system
The system under test has non-deterministic
(unpredictable) behavior like a game
Functionality is specified in a way suitable for models like business rules, formulas or tables/matrices
be aware whether the test model is independent enough from
anything used in the application under test
Look for clear benefits if you consider using a model worth the efforts and the relative high complexity
don't make it a hobby
9/20/2013
9
© 2013 LogiGear Corporation. All Rights Reserved
Using actions to create models
Actions can be a way to describe your intentions
quickly and clearly
One can also use models to generate action
based tests, however: such models will then be more static
tests will be longer and harder to read
Using actions for modeling can: be used as a quick ad-hoc technique for specific cases
don't require investment and buying and learning model-
based tooling
create models that dynamically respond to behavior of an
application under test
© 2013 LogiGear Corporation. All Rights Reserved
Example: An Insurance System
Optimism Insurances Inc. "Let us do your damage!"
Car Insurance Guidelines and Conditions
…
Article 11 – Policy Acceptance
11a Applicants with no more than one accident in the past three years are
accepted.
11b When two accidents are recorded in which the applicant was the guilty
party, the application is only accepted if the applicant is female.
11c An applicant with three or more accidents in the past two years is not
accepted.
11d Applications from testers are rejected (a car is not a system under test).
…
9/20/2013
10
© 2013 LogiGear Corporation. All Rights Reserved
Example: An Insurance System
A table to hold the business rules, mapping inputs to states.
start state accidents gender end state
application 0 --- accept
application 2 f accept
application 2 m refuse
application 3 --- refuse
© 2013 LogiGear Corporation. All Rights Reserved
Example: An Insurance System
Handle states. The check in the first state is an "oracle", it may not
be able to calculate the premium exactly, but rather verify that a
premium is within a certain tolerance:
name argument argument
state accept accidents car value
accidents car value
check premium # accidents # car value
end state
name
state refuse
text
check message Not eligible for insurance.
end state
9/20/2013
11
© 2013 LogiGear Corporation. All Rights Reserved
Example: An Insurance System
File with some input data (you could also consider monkey style random
data generation):
data set customers
name accidents gender car value
Owen 0 m 19500
Mary 3 f 32010
Kim 1 f 15322
Bob 2 m 21399
Niyati 0 f 27666
© 2013 LogiGear Corporation. All Rights Reserved
Example: An Insurance System
This scenario uses the data from the data set to try car insurance
applications. Based on the table "try state" will either jump to the
state or do nothing:
name accidents gender
enter customer # name # accidents # gender
owner value
enter car value # name # car value
state argument argument
try state accept # accidents # car value
state
try state refuse
9/20/2013
12
© 2013 LogiGear Corporation. All Rights Reserved
Example: An Insurance System
To start the test:
table scenario
run table action acceptance table test insurance
© 2013 LogiGear Corporation. All Rights Reserved
Non Deterministic Systems
Non-deterministic: difficult to determine/predict outcomes
Typical reasons for non-determinism: system depends on external events, like the weather
system has deliberate random behavior (games, security)
subject matter of the system is too complex, like rocket science
Non-determinism poses in particular a challenge for
automation manual testing usually is easier than automation to test non-
deterministic behavior
however, automation can exhaust more variations if alternative paths
are involved
9/20/2013
13
© 2013 LogiGear Corporation. All Rights Reserved
Handling Non Determinism
If possible still make your test predictable set the environment (or basic data) yourself
shut off the random generator (test it separately)
Select out the parts that are not non-deterministic and test
them in separately
Define what it is that you want to test and focus on that
(what is it that you do know): "when 10 monsters are shot you move on to the next level"
"the company logo should be in the upper left corner of the page:
within 10% of the page size"
Use models
© 2013 LogiGear Corporation. All Rights Reserved
Handling Multi-media, some strategies
Consider manual testing
Find criteria and objects that you CAN test colors, volume levels, text recognition, etc
Investigate: try to find handles that represent
what is going on file names, numeric handles, api calls
Use a "play list" approach to reduce human
testing
9/20/2013
14
© 2013 LogiGear Corporation. All Rights Reserved
The "Play List" Approach
The testers focus on a play list items are played/shown while the machine tells what it is
the tester should see/hear
the tester acknowledges/falsifies
the system stores information on the items that were ok
Next time only the changed items go on the play
list again in most systems this will only be a limited amount of items
Using this technique can free up the model, to
just work with handles
© 2013 LogiGear Corporation. All Rights Reserved
Example: a Quiz Game
The game asks questions and poses challenges a "challenge" for this example is a calculation (like "what is
32*3")
After each successful reply from the player it will
randomly either ask another question or pose
another challenge
The questions are randomly taken from a set of
available ones
The calculation questions are also non-
deterministic
9/20/2013
15
© 2013 LogiGear Corporation. All Rights Reserved
Example: a Quiz Game
Prompt: Here is a question:
Prompt: Name a leading West Coast event on testing.
Answer: StarWest
Prompt: Very good
Prompt: Let's calculate
Prompt: What is 23*2?
Answer: 46
Prompt: Very good
…
© 2013 LogiGear Corporation. All Rights Reserved
Example: a Quiz Game, a Table
Use of a table for questions and answers (in this example we simplify bye
presuming answers are given verbatim):
table questions
question answer
Name a leading West Coast conference
on testing.
Star West
What approach separates testing and test
automation.
Action Based Testing
Which city hosts the next Star conference Orlando
Which kind of models are mostly used for
model-based testing.
State Models
9/20/2013
16
© 2013 LogiGear Corporation. All Rights Reserved
Example: a Quiz Game, Making a Model
the question state:
state question
row column
get question >> q1 1
row column
answer question # q1 2
text
check prompt Very good
variable
get prompt >> prompt
prompt match state
move # prompt Let's calculate question
move # prompt Here is a question formula
© 2013 LogiGear Corporation. All Rights Reserved
Example: a Quiz Game, Making a Model
the challenge state (we also assume we have a parser to extract and resolve
the formula):
state challenge
challenge
get challenge >> chall
challenge result
resolve challenge # chall >> res
text
answer challenge # res
variable
get prompt >> prompt
prompt match state
move # prompt Let's calculate question
move # prompt Here is a question formula
9/20/2013
17
© 2013 LogiGear Corporation. All Rights Reserved
Dealing with non-determination
In this example we saw three ways to deal with
non-deterministic behavior a table for the questions
a parser to extract and calculate the formula challenges
a state machine to distinguish between question and
challenge states
You could typically let the test play until: all transitions between states have been traversed at
least once (including wrong answers etc)
all questions have been played
all possible kinds of formulas have been played • you have to define what that means, typically with equivalence
classes
© 2013 LogiGear Corporation. All Rights Reserved
Summarizing
Models can be used to: as a more efficient alternative to straightforward test cases
handle complex situations like non-deterministic system
behavior
Many models are based on state models
Models are not a panacea use them if there are clear benefits
Using actions can facilitate model-based and
multi-media testing providing means to describe and automate in an
accessible way