exploring no mans land with keyword-driven testing
DESCRIPTION
Free Test 2009 conference slidesTRANSCRIPT
![Page 1: Exploring No Mans Land with Keyword-Driven Testing](https://reader033.vdocuments.site/reader033/viewer/2022052505/5556f0a4d8b42ac60c8b500a/html5/thumbnails/1.jpg)
Exploring no man's landwith keyword driven testing
Martin Gijsen, [email protected]
Free Test ConferenceMarch 23, 2009
![Page 2: Exploring No Mans Land with Keyword-Driven Testing](https://reader033.vdocuments.site/reader033/viewer/2022052505/5556f0a4d8b42ac60c8b500a/html5/thumbnails/2.jpg)
Presentation overview● Automated testing● When (not) to use Record & Playback● Automated testing as no man's land● Keyword driven testing● The process● Where to get the automator
![Page 3: Exploring No Mans Land with Keyword-Driven Testing](https://reader033.vdocuments.site/reader033/viewer/2022052505/5556f0a4d8b42ac60c8b500a/html5/thumbnails/3.jpg)
Automated (functional) testing● A computer performs the test● Fast and consistent● Potential benefits:
● Time (to market) / cost / quality● More effective use of (test) resources● More challenged (= happier) testers● Accurate and up2date test status
● Continuity● Business goals are drivers
![Page 4: Exploring No Mans Land with Keyword-Driven Testing](https://reader033.vdocuments.site/reader033/viewer/2022052505/5556f0a4d8b42ac60c8b500a/html5/thumbnails/4.jpg)
The Record & Playback approach● Record test actions, replay as needed● Checks are added to scripts● Fundamental problem is maintenance
● Software evolves until it is decommissioned● Maintainability requires structure and abstractions● Recorded test cases are long, unstructured and low
level programs● Maintenance sensitivity is high● Maintainability is low
![Page 5: Exploring No Mans Land with Keyword-Driven Testing](https://reader033.vdocuments.site/reader033/viewer/2022052505/5556f0a4d8b42ac60c8b500a/html5/thumbnails/5.jpg)
So why is it still used?● Automated testing is often the job of a tester● Most testers are:
● willing to record, but● unwilling or unable to create good automation
solutions● Vendor sales pitch is not always clear on this
![Page 6: Exploring No Mans Land with Keyword-Driven Testing](https://reader033.vdocuments.site/reader033/viewer/2022052505/5556f0a4d8b42ac60c8b500a/html5/thumbnails/6.jpg)
Result and conclusion
It is often initially unclear whateffective automated testing requires
2 out of every 3 TA projectsfails sooner or later
– Brian LaSuer
Use R&P only in case oflittle manual maintenance
![Page 7: Exploring No Mans Land with Keyword-Driven Testing](https://reader033.vdocuments.site/reader033/viewer/2022052505/5556f0a4d8b42ac60c8b500a/html5/thumbnails/7.jpg)
What about the developers?● Test automation is software engineering● Developers have the skills to create a good
automation solution● Many developers are unwilling or unable to
create good test sets● … or automate test cases that testers wrote
![Page 8: Exploring No Mans Land with Keyword-Driven Testing](https://reader033.vdocuments.site/reader033/viewer/2022052505/5556f0a4d8b42ac60c8b500a/html5/thumbnails/8.jpg)
Welcome to no man's land● Most testers cannot or will not automate well● Most developers cannot or will not test well● Automated testing is like a no man's land in
between testing and development
● Is automated testing doomed?
![Page 9: Exploring No Mans Land with Keyword-Driven Testing](https://reader033.vdocuments.site/reader033/viewer/2022052505/5556f0a4d8b42ac60c8b500a/html5/thumbnails/9.jpg)
Exploring no man's land
Testing
Automated testing
Development
![Page 10: Exploring No Mans Land with Keyword-Driven Testing](https://reader033.vdocuments.site/reader033/viewer/2022052505/5556f0a4d8b42ac60c8b500a/html5/thumbnails/10.jpg)
Testing
Automated testing= analysis (what to test)
+ automation (how to test)
Development
Exploring no man's land
![Page 11: Exploring No Mans Land with Keyword-Driven Testing](https://reader033.vdocuments.site/reader033/viewer/2022052505/5556f0a4d8b42ac60c8b500a/html5/thumbnails/11.jpg)
What is in between?
Testing
?
Development
Test analysis
Test automation
![Page 12: Exploring No Mans Land with Keyword-Driven Testing](https://reader033.vdocuments.site/reader033/viewer/2022052505/5556f0a4d8b42ac60c8b500a/html5/thumbnails/12.jpg)
No more no man's land?
Testing
Development
Test analysis
Test automation
![Page 13: Exploring No Mans Land with Keyword-Driven Testing](https://reader033.vdocuments.site/reader033/viewer/2022052505/5556f0a4d8b42ac60c8b500a/html5/thumbnails/13.jpg)
Keywords – The DS(T)L● Domain Specific (Test) Language
● Used to express test cases● Defined by (or with) the testers● Specific to the application under test● No programming● High level
![Page 14: Exploring No Mans Land with Keyword-Driven Testing](https://reader033.vdocuments.site/reader033/viewer/2022052505/5556f0a4d8b42ac60c8b500a/html5/thumbnails/14.jpg)
Bad example
select (“a box”);type (“some text”);click (“OK”);...
![Page 15: Exploring No Mans Land with Keyword-Driven Testing](https://reader033.vdocuments.site/reader033/viewer/2022052505/5556f0a4d8b42ac60c8b500a/html5/thumbnails/15.jpg)
Better example
create account;1234567890;John;Doecreate account;2345678901;Jane;Doe
deposit;1234567890;12345,67
transfer;1234,56;1234567890;2345678901
check balance;1234567890;11111,11check balance;2345678901;1234,56
![Page 16: Exploring No Mans Land with Keyword-Driven Testing](https://reader033.vdocuments.site/reader033/viewer/2022052505/5556f0a4d8b42ac60c8b500a/html5/thumbnails/16.jpg)
Good exampleAccount First name Last name
create account 1234567890 John Doe
create account 2345678901 Jane Doe
Account Amount
deposit John 12345,67
Amount From To
transfer 1234,56 John Jane
Account Amount
check balance John 11111,11
check balance Jane 1234,56
![Page 17: Exploring No Mans Land with Keyword-Driven Testing](https://reader033.vdocuments.site/reader033/viewer/2022052505/5556f0a4d8b42ac60c8b500a/html5/thumbnails/17.jpg)
Keywords / Instructions● Often have arguments
● No (irrelevant) test execution details● No (irrelevant) interfacing details● No tooling details
● Only the essence of a test case remains● Easy to read, write and maintain test cases● Low maintenance sensitivity
![Page 18: Exploring No Mans Land with Keyword-Driven Testing](https://reader033.vdocuments.site/reader033/viewer/2022052505/5556f0a4d8b42ac60c8b500a/html5/thumbnails/18.jpg)
Sample instruction documentationName and aliases 'begin test case' and 'begin testcase'
Description Indicates the beginning of a (new) test case.
Parameters Two parameters, both optional
1 The identification of the test case, printable characters, preferably unique within the test. Optional.
2 The test case description, printable characters. Optional.
Validity Anywhere except inside a procedure definition.
Pre-condition None.
Post-condition If the previous input line was part of a test case, that test case is closed. A new test case is opened and assigned the next sequence number for the report(s), starting at one.
Error condition If in a procedure definition.
![Page 19: Exploring No Mans Land with Keyword-Driven Testing](https://reader033.vdocuments.site/reader033/viewer/2022052505/5556f0a4d8b42ac60c8b500a/html5/thumbnails/19.jpg)
Automating (basic)
AutomationTest Testreport
Application under test
![Page 20: Exploring No Mans Land with Keyword-Driven Testing](https://reader033.vdocuments.site/reader033/viewer/2022052505/5556f0a4d8b42ac60c8b500a/html5/thumbnails/20.jpg)
Automating (more detailed)
EngineTest Testreport
Application under test
Instructions
Interfacing
![Page 21: Exploring No Mans Land with Keyword-Driven Testing](https://reader033.vdocuments.site/reader033/viewer/2022052505/5556f0a4d8b42ac60c8b500a/html5/thumbnails/21.jpg)
The ETA Framework 'engine'● Can read from and report to a spreadsheet● Supports test cases, variables and procedures● Implement keywords in Java● Use any Java library (Abbot, WebDriver, MQ,
generated web service client, ...)● Documented, actively used and developed● Freeware
![Page 22: Exploring No Mans Land with Keyword-Driven Testing](https://reader033.vdocuments.site/reader033/viewer/2022052505/5556f0a4d8b42ac60c8b500a/html5/thumbnails/22.jpg)
The process (waterfall)
Requirements
Keyword defs
Product development
Test cases
Automate Test
![Page 23: Exploring No Mans Land with Keyword-Driven Testing](https://reader033.vdocuments.site/reader033/viewer/2022052505/5556f0a4d8b42ac60c8b500a/html5/thumbnails/23.jpg)
The process (agile)
Requirements
Keyword defs
Product development
Test cases
Automate TestRequirements
Keyword defs
Product development
Test cases
Automate Test
Requirements
Keyword defs
Product development
Test cases
Automate TestRequirements
Keyword defs
Product development
Test cases
Automate Test
Requirements
Keyword defs
Product development
Test cases
Automate TestRequirements
Keyword defs
Product development
Test cases
Automate Test
![Page 24: Exploring No Mans Land with Keyword-Driven Testing](https://reader033.vdocuments.site/reader033/viewer/2022052505/5556f0a4d8b42ac60c8b500a/html5/thumbnails/24.jpg)
Where to get the automator● Main criteria (for continuity):
● Skills● Availability● Cost
● Educate someone from test team● Borrow from development team● Hire or reserve developer for test team● Hire external consultant
![Page 25: Exploring No Mans Land with Keyword-Driven Testing](https://reader033.vdocuments.site/reader033/viewer/2022052505/5556f0a4d8b42ac60c8b500a/html5/thumbnails/25.jpg)
Summary● Automated testing can help achieve your goals● Avoid Record & Playback● Good keywords help:
● The tester can focus on the testing● The developer can focus on developing● Easy to write test cases and little maintenance
● A good test engine also helps● Use skilled automators● The automators must be available when needed
![Page 26: Exploring No Mans Land with Keyword-Driven Testing](https://reader033.vdocuments.site/reader033/viewer/2022052505/5556f0a4d8b42ac60c8b500a/html5/thumbnails/26.jpg)
Questions
?
![Page 27: Exploring No Mans Land with Keyword-Driven Testing](https://reader033.vdocuments.site/reader033/viewer/2022052505/5556f0a4d8b42ac60c8b500a/html5/thumbnails/27.jpg)
Automation design
EngineTest Testreport
Application under test
Instructions
Interfacing
![Page 28: Exploring No Mans Land with Keyword-Driven Testing](https://reader033.vdocuments.site/reader033/viewer/2022052505/5556f0a4d8b42ac60c8b500a/html5/thumbnails/28.jpg)
Design for a telephony switch
Engine
Telephony switch
Instructions (C++)
Existing library (iTCL)
![Page 29: Exploring No Mans Land with Keyword-Driven Testing](https://reader033.vdocuments.site/reader033/viewer/2022052505/5556f0a4d8b42ac60c8b500a/html5/thumbnails/29.jpg)
Design for a voice portalEngine
Voice portal
Instructions (C++)
Tool
Tool library
![Page 30: Exploring No Mans Land with Keyword-Driven Testing](https://reader033.vdocuments.site/reader033/viewer/2022052505/5556f0a4d8b42ac60c8b500a/html5/thumbnails/30.jpg)
Design for web services (1)
Engine
Web services
Generic web serviceinstructions (Java)
Generatedweb service client (Java)
![Page 31: Exploring No Mans Land with Keyword-Driven Testing](https://reader033.vdocuments.site/reader033/viewer/2022052505/5556f0a4d8b42ac60c8b500a/html5/thumbnails/31.jpg)
Design for web services (2)
Engine
Web services
Specific web serviceinstructions (Java)
Generatedweb service client (Java)
![Page 32: Exploring No Mans Land with Keyword-Driven Testing](https://reader033.vdocuments.site/reader033/viewer/2022052505/5556f0a4d8b42ac60c8b500a/html5/thumbnails/32.jpg)
Design for a payments system
Engine
SEPA banking system
Iso20022 protocolinstructions Java)
SWIFT MT protocolinstructions (Java)
Web serviceinstructions (Java)
Web instructions(Java)
Messaging library (Java) Generated web serviceclient (Java)
Web driverlibrary (Java)