Tips for Success whenSelecting Testing Tools

Marie Ackley

ackley @ promail.com

SASQAG panel discussionOctober 18, 2001

2

Different Categories of Testing Tools Requirements Management Configuration Management Test Planning and Test Case Management Test Evaluation - Coverage Analysis Automated Testing Systems - Front End testing Load Simulators and Performance Measurement Tools Defect Tracking

others: Cyclomatic Complexity; Bounds Checkers; Comparators - Files, Screens, Reports; Data Generators - Combinatoric, Cause / Effect Node Graphing

3

Leading Cause of SHELF WARE in Testing Tools

is Putting the CART before the HORSE

Don't invest in Tools without having good QA Methods and Practices.

A good tool will compliment and augment good processes but not

even the greatest tool can overcome process chaos.

4

Requirements Management

Supports multi levels of linking and tracing: Requirements to Specifications to Test Cases, including hierarchies within those groups

Provides traceability - both forward and backward

Provides coverage analysis

Provides multi relationships: one-to-many, many-to-one and coverage is tracked for partials when relationship is one-to-many

Provides Change Management of requirements along with audit history and maybe security privileges for modifications

Import and export features so stuff came go and come between the tool and word processors or spreadsheets

5

Requirements Management

Links to tool comparisons

the "Volere" Requirements Method - by Atlantic Systems Guildhttp://www.systemsguild.com/GuildSite/Robs/

Template.html

Automating Requirements Management - by Karl E. WiegersDetailed feature comparisons of many requirements

management tools and guidance on selecting a requirements management tool

http://www.processimpact.com/articles/rm_tools.html

6

Configuration Management

"To change and to change for the better are two different things" - German Proverb

Provides Identifications (indexing of artifacts)

Controls not only software code but documentation, hardware, firmware, and system environment configuration

Provides audit trail and history tracking

Provides customizable reports

High ease of use for View Only functions

Sufficient security levels for the Promote functions

Accounting and reporting for different ”Status Levels" - WIP, Waiting Room, Release/Production

7

Configuration Management

Links to tool comparisons

Configuration Management Vendor Links - by UCM Centralhttp://www.snuffybear.com/ucmcentral_new_vendorlinks.htm

config-mgmt Newsgroup http://www.faqs.org/faqs/by-newsgroup/comp/comp.software.config-

mgmt.html

8

Test Planning and Test Case Management

Import test cases from other file formats such as word processing or spreadsheet

Customizable reporting

Tracks history and status of test cases

Allows file attachments

Integration - can tool link to other tools such as your Requirements tool and your Automated Testing tool

Provide a Test Execution Results repository

Maintains Master Set of test cases and then can build “subsets” for project specific needs

9

Test Planning and Test Case Management

Links for tool comparisons

Testing and Test Management Tools

http://www.methods-tools.com/tools/frames_testing.html

Information About Testing Tools - http://www.cs.queensu.ca/home/shepard/testing.dir/under.construction/

tool_list.html

10

Test Evaluation - Coverage Analysis

Identifies what "unique" data combinations are needed

Uses an appropriate measurement techniqueDecision; Condition/decision; Path; Predicate

Types of coverage included in the reportingBranch; Loop; Multi; Operator; Operand

Integration - is it compatible with your other tools such as Automated Test Execution tools

Implementation - is the tool "invasive" (must be compiled along with the source code)

11

Test Evaluation - Coverage Analysis

Links to tool comparisons

Experience With the Cost of Different Coverage Goals for Testing - by Brian Marick

http://www.testing.com/writings/experience.pdf

Coverage Analysis - by Steve Cornetthttp://www.bullseye.com/webCoverage.html

12

Automated Testing Systems - Front End testing

Maintenance - object recognition when the application changes

Ease Of Use - complexity matches the programming talent of QA persons

Tool language and syntax matches what company uses: Visual Basic or C

Extensible Language - can use external subroutines or DLLs

Integration with other tools - Code Coverage, Test Case Management, Defect Tracking

Compatibilty - platforms (Client/Server, Web, Mainframe) Operating Systems(Windows, Mac, Unix, OS/2, Mainframe) and GUI Development (PowerBuilder, C, Visual Basic, Galaxy

13

Automated Testing Systems - Front End testing

Supports Data Driven inputs from flat files, spreadsheet files, and database connections

For Evaluation/Proof of Concept - include automating a Level One test

Links to tool comparisons

Making the Right Choice - by Elisabeth Hendrickson http://www.qualitytree.com/feature/mtrc.pdf

Short Sharp Advice about how to choose a testing tool - Mark Fewster and Dorothy Graham

http://www.grove.co.uk/Tool_Information/Choosing_Tools.html

Tool Comparison between Rational, Mercury, Segue, and Compuware - by Ray Robinson

http://www.qacity.com/testtype/AUTOMATION_TOOLS.doc

14

Automated Testing SystemsLinks to articles

Lessons in Test Automation - by Elfriede Dustinhttp://www.sqa-test.com/articles.html

The Truth About Automated Test Tools - by Linda Hayeshttp://www.worksoft.com/resources/4-97.htmhttp://www.worksoft.com/resources/5-97.htm

Test Automation - by Bret Pettichordhttp://www.io.com/~wazmo/qa/#test_automation

Testers' Network - VeriTest websitehttp://www.veritest.com/testers'network

Test Automation Frameworks - by Carl J. Naglehttp://members.aol.com/sascanagl/

FRAMESDataDrivenTestAutomationFrameworks.htm

15

Load Simulators and Performance Measurement

Reporting includes graphical charts

Debugging features for script creation

Reporting includes graphical charts and logs of performance for server speed and responsiveness

Quantity of simulated users

Variations in session-specific items such as cookies, usernames, passwords, proxies and ASP-session IDs

Support for your middleware - SQLnet, DBLib or CBLib, SQL Server, ODBC, Telnet

Challenges include SSL sessions, cookies, forms, proxies, password authentication, dynamic HTML, full HTTP, SSL, XML, Javascript, ActiveX, WAP, Java

Platforms and protocols

16

Load Simulators and Performance Measurement

Link to tool comparisons

Load and Performance Test Tools - by Rick Hower http://www.softwareqatest.com/qatweb1.html

Software Research - TestWorks

http://www.methods-tools.com/tools/testing.html

17

Defect Tracking

Determine how many current users you have and consider and scaling for future users

Web based or client-server

Features: Customizable Fields; Customizable Reports; File Attachments; Audit Trail; Email linking; Security Levels

Provides customizable rules for the workflows, security privileges and notifications

Integration - does it link to your other tools

18

Defect Tracking

Links to tool comparisons

Tracking Down a Defect Management Tool - by Hung Quoc Nguyenhttp://www.qacity.com/testgen/

DefectTrackingNguyenSTQE701.pdf

The Problem with Problem Tracking - by Linda Hayeshttp://www.worksoft.com/resources/3-99.htm

19

Websites for Tool Information

Vendor contact informationhttp://www.testingfaqs.org/tools.htmhttp://www.ondaweb.com/sti/stivend.htm#Lhttp://www.dbmsmag.com/pctest.html#P0890

Bulletin board forums and articleshttp://www.qaforums.comhttp://community.sqatester.comhttp://www.qualitytree.com/links/links.htmhttp://www.stqemagazine.com