rational quality manager eval guide

8/12/2019 Rational Quality Manager Eval Guide

1/22

Rational Quality ManagerEvaluation Guide

Created: 29thDecember 2010

This guide is based on: RQM 2.0.1.1


2/22


Page 1

Contents

Introduction ..................................................................................................................2Installation....................................................................................................................2Setup ...........................................................................................................................2

Option 1 ...................................................................................................................2Option 2 ...................................................................................................................2

Typical Evaluation Criteria.............................................................................................3Requirements Management.......................................................................................3Test Management.....................................................................................................3Defect Management..................................................................................................4Lifecycle Integrations.................................................................................................5Non Functional.........................................................................................................5

Additional Evaluation Criteria.........................................................................................6Planning the Evaluation .................................................................................................7

Evaluation Milestones....................................................................................................8Rational Quality Manager Overview ................................................................................8Rational Quality Manager Capabilities.............................................................................9

Dashboards..............................................................................................................9Requirements.........................................................................................................10Test Planning.........................................................................................................11Test Construction....................................................................................................12Test Implementation................................................................................................13Keywords...............................................................................................................14Test Environments..................................................................................................15Test Execution........................................................................................................16

Test Lab Management.............................................................................................17Defect Tracking.......................................................................................................18Test Reporting........................................................................................................19Customisation.........................................................................................................20

Next Steps..................................................................................................................20Additional Resources...................................................................................................21


3/22


Page 2

Introduction

The purpose of this guide is to highlight key features and functionality of Rational QualityManager and to enable quality professionals to successfully explore and evaluate it'sfunctionality.

Installation

Rational Quality Manager is available for downloadfrom both the IBM and Jazz.net websites and can beevaluated for 60 days.

Installation documentation is available on both sitesbut installing Rational Quality Manager is a simpleaffair and can be initiated by extracting the files,

running the setup wizard and accepting the defaults

Alternatively, your IBM representative may have suppliedan automated installation including both sample and evaluation projects, these containexamples and guidance relating to the evaluation. In order to install this version, simply run thesupplied batch file.

The Jazz website can be found at www.jazz.net and contains vast quantities of information onboth Rational Quality Manager and the Jazz initiative including wikis, forums and a window intothe development organisation, allowing customers unprecedented access directly to productmanagement and the development teams.

Setup

Once Rational Quality Manager is installed, simply start the Rational Quality Manager server ifrequired, open IBM Rational Quality Manager and login.

Option 1: For those customers who downloaded the directly from either the IBM or Jazzwebsites, the default user ID and password are both ADMIN (notice the uppercase).

On initial login, the ADMIN user will be redirected to the RationalQuality Manager set up screen, for evaluations, selecting theFast Pass Setup is recommended. Accept all the default settings.

As part of the setup process, a new user will be created and the ADMIN user disabled. Thesetup process will also create a sample project area named 'Quality Manager'.

Option 2: For customers using the pre-packaged automated installation, the username and

password are both rqmeval(notice the lowercase).


4/22


Page 3

Typical Evaluation Criteria

When evaluating a test management solution, there are a number of questions that aretypically asked. The following evaluation criteria has been consolidated from questions posedto IBM from our customers. These questions have been taken from numerous RFI's andsanitised for use in this document.

This shouldn't be considered a definitive list, the purpose of these sample questions isn't to askonly those questions that Rational Quality Manager is known to address, but to provide anunbiased view of the types of things that should be considered when choosing a tool.

Additional space has been included in these tables as every customer will have individualneeds specific to their organisation.

Requirements Management

ID Criteria Findings

RM1The system shall provide the ability to definerequirements identified by a unique reference Yes No Partial

RM2Requirements can be weighted based on riskand probability Yes No Partial

RM3Requirements can be associated to testspecifications (many to many) Yes No Partial

RM4Requirement coverage can be monitored andreported against Yes No Partial

RM5

The system shall be able to import structured

text from Microsoft Excel and Word file formats Yes No Partial

RM6Yes No Partial

RM7Yes No Partial

RM8Yes No Partial

RM9Yes No Partial

RM10Yes No Partial

Test Management

ID Criteria Findings

TM1Ability to define test specifications for a projectand sub-project Yes No Partial

TM2Ability to define test scripts for a project andsub-project Yes No Partial

TM3It should be possible to import test assets fromMS Excel Yes No Partial

TM4 Ability to define the environment the tests will beexecuted Yes No Partial


5/22


6/22


7/22


8/22


Page 7

Planning the Evaluation

The time taken to fully explore and evaluate a tool against a set of agreed criteria will vary as itwill typically run parallel with existing work commitments. For that reason dates and durations

have been omitted from the following plan. Its purpose is purely to document evaluationactivities and includes optional milestones where your IBM representative can visit to addressany questions arising from the evaluation.


9/22


Page 8

Evaluation Milestones

As was mentioned in the planning section above, your IBM representative will be more thanhappy to visit throughout the evaluation process and address any questions or concerns thathave arisen as well as demonstrating and/or discussing some of the more advanced

capabilities offered by Rational Quality Manager.

Any milestone dates can be agreed with your IBM representative and captured below

Milestone Date and Time

Evaluation set-up and Overview Today's date

Optional Milestone...

Evaluation Progress Review

Optional Milestone...

Evaluation wrap-up and next steps

Rational Quality Manager Overview

Rational Quality Manager is a web-based centralized test management environment that allowsquality professionals to effectively collaborate on all aspects of testing including test planning,construction, implementation, lab management and execution.

Rational Quality Manager includes the ability to create a meaningful test plan directly within thetool based on an agreed set of requirements. Users can create these requirements directly inRational Quality Manager, or requirements can be tracked through a dynamic link to

requirements management tools such as IBM Rational RequisitePro or IBM Rational DOORs.


10/22


Page 9

Test Plans can be customised to conform to a customers test processes but would normallyinclude the requirements that are in scope and the test cases that will verify their functionality.

Rational Quality Manager includes a rich manual testing capability which provides qualityprofessionals with a rich text editor that can capture textual test steps alongside embedded

images. Using this editor, testers can record the test steps that must be executed in order tovalidate a given function of their application. These test scripts can then be associated withtest cases and executed.

For those looking to automate testing, Rational Quality Manager can also integrate with testautomation tools such as Rational Functional Tester, Rational Services Tester and RationalPerformance Tester.

To enable test reuse, Rational Quality Manager uses keywords, which allow testers to sharecommon steps between tests. These can be manual steps or automated test scripts that arepowered, for example, by IBM Rational Functional Tester (RFT) software. Creating hybrid

manual tests with automated keywords is often a very simple way for manually focused testteams to leverage the power of test automation, without making a full commitment to a testautomation framework.

Rational Quality Manager provides each user with an individual dashboard that is continuallyupdated, allowing each user to view the most current information that pertains to his or herwork. In addition to this, a wide variety of formal and customizable reports are also availablethat can capture live and historic information at varying levels of granularity to help ensure thatall team members have access to the information they need.

Rational Quality Manager Capabilities

The remainder of this Evaluation Guide is designed to support the evaluation activities definedabove and will highlight key areas of Rational Quality Manager, enabling quality professionalsto explore and evaluate it's functionality

Dashboards

Rational Quality Manager offers comprehensive dashboardcapabilities, allowing users to see in real time, information thatis relevant to the role they play in the software development

lifecycle. For a Tester this may only extend to their task listand the status of any issues they have raised. Whereas forthe Test Manager, it may include requirement and testcoverage, team loading or reporting.

Additional viewlets and tabs can be added to the users personal dashboard by clicking theand buttons respectively.


11/22


Page 10

Once the button is clicked, the user is presented with a choice of viewlet types that can beadded to their dashboard.

Viewlet types include the users task list, project and team activities, requirements and testcoverage, work item statistics as well as news feeds (ie. RSS) for monitoring user forums,

blogs as well as commercial news sites.

Each viewlet can the be individually configured toshow different subsets of information such asrequirements coverage by test plan.

Viewlets can also be duplicated and moved betweentabs, for example if a user wanted to view defects bypriority and severity. It is also worth noting that mostviewlets are interactive allowing users to drill thoughto the data used in those charts

Requirements

Rational Quality Manager allows quality professionalsto capture those requirements relevant to testing inorder to build traceability links to both test plans andcases, this ensures requirements coverage andallows for impact analysis should one of thoserequirements change.


12/22


Page 11

As has already been mentioned in the overview, Rational Quality Manager integrates with boththe industry leading requirements management tools, IBM Rational RequisitePro and IBMRational DOORs (formally Telelogic DOORs).This allows quality professionals visibility of the

requirements as they are synchronised betweenthe external tools and Rational Quality Managersinternal ones.

In addition to this, requirements can also be entered directly into Rational Quality Managerallowing quality professionals to capture their testing requirements and test conditions andtrace these to test plans and test cases.

Rational Quality Manager provides a comprehensivequery engine that allows users to query work itemdata such as requirements and defects in order to get

the necessary data in the required format.

Where appropriate, these queries can also be usedto form the basis of a number of viewlet types thatcan be added to the users dashboard.

Test Planning

Rational Quality Manager includes the ability to create a meaningful test plan directly within thetool and removes the need for externally managed documentation. While traditional test plansare text based documents containing headings representing the information that needs to becaptured, the test plans in Rational Quality Manager are accessible online enablingcollaboration and sharing of information across the whole project team.

These plans still include sections that shouldbe completed prior to the plan being reviewedand approved, however, sections of the plancan be assigned to an individual in order todistributed the work load evenly across the test

team. These tasks will then appear in theindividuals task list awaiting completion.

Because the Rational Quality Manager test plan is designed to be an alternative to the test plandocument, Rational Quality Manager offers full rich text editor capabilities allowing the user

include tables, images, various font sizes, styles and colours.


13/22


Page 12

Each test plan can be based on a user-defined template allowing users to pick and choose thesections to be completed based on the size, complexity and formality of the project, rangingfrom a lightweight agile approach to a full traditional waterfall version.

A typical test plan would normally include the

requirements in scope for current phase oftesting as well as document the types oftesting and test cases that will be performedin order to validate the functionality. Theapproach to test planning may varydepending on the software developmentmethodology being used.

For example an iterative approach may resultin a test plan per iteration where eachiteration delivers a subset of the overall

systems requirements.

Regardless on the approach taken, RationalQuality Manager will provide completetraceability across the software developmentlifecycle ensuring that all requirements arefully validated.

Test Construction

Test Cases within Rational Quality Manager are reusable assets that can be included inmultiple test plans throughout a software development project. The primary function of a testcase is to document the test that is to be performed and capture the appropriate level of detail

in order to allow any member of the quality assurance team to be able to perform it.

Typically, this detail will consist of:

Who : The person responsible forcreating and maintaining the test.

What : The test to be performed When : The iteration or phase in

which the test should be performed

Where : The environment in whichthe test is to be executed

Why : The reason why the test isbeing performed


14/22


Page 13

Rational Quality Managers test cases and plans include the ability to create user-definedcategories in order to tag these assets with meaningful identifiers such as product, version,function and test phase.

Test cases also include the concept of

weighting. The weight is used to representthe complexity of the test.

The level on granularity may vary (1-10, 1-100, etc) but thepurpose is to differentiate between test cases in order to produce more accurate progressreporting. For example, registering a user on a website is a simple scenario and may take lessthan 30 minutes to create the test case whereas the scenario of applying for a mortgage may

take a number of hours. A report showing completed test cases may indicate 50% of the workis complete when in reality, the easy test case (ie. 20%) is complete and more time consumingtest case (ie. 80%) is still outstanding.

Similar to test plans, Rational Quality Manager test cases include sections that should becompleted prior to the test being reviewed and approved.

While sections of the test casecan be assigned to an individual,typically the entire test case would begiven to a member of the qualityteam to complete.

The test case will then appear on theindividuals task list similar to those

already mentioned. The quality teammember can then mark this completeonce the test case is ready.

Test cases can also be based on a user-defined template allowing a user to pick and choosethe sections to be completed based on the size, complexity and formality of the project.

Test Implementation

The test case needs to address the who, what, when, whereand why.The missing question that needs to be answered is how. Rational Quality

Manager includes a rich manual testing capability, allowing testers tocreate manual test scripts and use these to implement their test cases.


15/22


Page 14

The manual test editor allows testers tocapture textual test steps and embeddedimages that describe the scenario to beexecuted.

In addition to this, Rational Quality Manageralso allows testers to parameterise values inthe test scripts allowing them to be executedwith different datasets.

While the manual testing capability is built-in to Rational Quality Manager, there are a numberof integrations with test automation tools such as Rational Functional Tester, Rational ServicesTester and Rational Performance Tester.

Your IBM representative will be happy to address any questions around these or otherautomation tools.

Keywords

To enable test reuse, Rational Quality Manager uses keywords, which allow testers to sharecommon steps between tests. This is effectively a reusable library of common businessoperations. For example, with a car insurance system, adding drivers details, accident history,endorsements are all ideally suited for reuse as they will be performed multiple times withvarying data as part of testing this type of application.

Keywords can simply be dragged from the keyword view intointo the manual test editor and dropped at a suitable positionin the manual test script.

While a keyword looks to be part of the manual test, it isn't being copied into the test itself, themanual test script is calling out to the keyword. This substantially reduces maintenance timeas a single keyword can be called many times, updating the keyword results in every script thatcalls it being updated instantly. The use of keywords also speeds up script creation due to the


16/22


Page 15

level of reuse that can take place. Keywords are executed as part of the test script executionallowing the quality professional to verify the entire usage scenario.

Keywords could be considered the building blocksused to construct a test script, because Rational

Quality Manager has integrations with testautomation tools. Some keywords may also besuitable for automation.

This may be part of a gradual move towards test automation where business testers aresupported by a member of the quality team using an automated testing tool or just amechanism for making efficiency savings, automating manual activities such as a rolling backthe database prior to performinga series of test runs.

Rational Quality Manager includes

a command line adapter that allowsRational Quality Manager to issuecommands to the machine onwhich the adapter is running, suchas executing batch, shell and SQLscripts and executables.

Test Environments

In order to verify that the software being developed will function correctly when it is deployed orinstalled in the end users environment, these test environments need to be defined based onthe non-functional requirements of the software being developed and could include operatingsystem support, service packs, web browsers andversions. Rational Quality Manager allows qualityprofessionals to capture this test environmentinformation and associate it with both the test planand test cases to ensure coverage and verify thesoftware being developed functions correctly ineach environment.

Testing every possible combination typically isn't

feasible as the number of environments will growexponentially with each new addition.

Rational Quality Manager includesthe ability to substantially reduce thenumber of permutations usingtechniques such as pair-wise testingto provide maximum breadth butlimited depth. This ensures that eachattribute is verified at least once.Attributes can be weighted to ensure

there is a bias towards importantattributes.


17/22


Page 16

Test Execution

Something that is sometimes overlooked when testing is the environment in which the test isbeing executed. This is especially common when the software being developed is an internalsystem to be deployed to an employees desktop. As this is a known entity, environmental

attributes such as operating system, service pack and browser are often not consideredrelevant.

However when a test is executed, it is always executed somewhere and this information needsto be recorded for accuracy of results. When executing a test case with Rational QualityManager, it needs to be associated with a test environment, this combination of test case andenvironment is known as a Test Execution Record. Every test case will have at least one ormore of these.

The test execution record captures the result ofexecuting the test case in that environment and isused in reporting to show that the software beingdeveloped functions correctly in each environmentdocumented in the non-functional requirements.

While the test execution record will show the latest result, the full history of previous results isalso available.

When executing a test execution record, the quality professional will select the record to run

and click the button, this will then trigger the tests implementation to execute. Typically thiswill be a manual script but as has already been mentioned, could be an automated equivalent.

Each step is a verification point with anexpected result and can be passed orfailed using the appropriate icon, either

for pass or for fail.

Manual test execution can also be paused

and resumed to allow for executionsthat may be required to run across multipledays. For example, testing a batch process.

Rational Quality Manager provides comprehensivetest logs to allow both the tester and developer tore-examine any failures that occurred during executionand follow those steps that exposed the issue.

Similar to the weighting idea used within the Rational Quality Manager test case, test resultsare broken down by the result type captured during the tests execution.


18/22


Page 17

This allows Rational Quality Manager to report on quality at a more granular level so while thetest execution record may have failed, we cansee that 2/3of the functionality exercised is stillworking. Rational Quality Manager is able toreport on results based on both TER count orweighting to provide a more accurate view ofthe quality of the software being developed.

Rational Quality Manager provides the ability to group test cases into test suites and executethem in a specific sequence.

Similar to test cases, the test suite should be associated with the test environment to ensureenvironmental coverage and verify the software being developed functions correctly in eachenvironment.

Test Lab Management

Rational Quality Manager includes the ability to managetest lab assets such as desktops and servers. Part of thismanagement includes a scheduling capability allowingmembers of the quality team to search and reserve a testmachine in order to execute a series of tests. This allowsthe quality team to make maximise use the hardwareavailable for their testing.


19/22


Page 18

Part of any booking may well consist of having to set aside the time required to tailor themachine to fit in with your test environment needs. This could involve software installation,service pack updates or browser upgrades.

An optional component of Rational Quality Managerprovides the ability to execute a series of automation

jobs on these test lab machines.

These automations allow Rational Quality Managerto setup the test machines by automatically installingand removing any piece of software includingapplications, service packs or testing tools.

Automations can be combined with test execution records or suites and called from a testschedule. Rational Quality Manager allows the quality professional to schedule a test run for a

specified date and time, This is typically used in conjunction with automated test tools in orderto perform out of office testing. By combining automations and automated tests in the sameschedule, Rational Quality Manager is able to automatically build the test environment, install

and configure the test automation tool, deploythe application under test and run a series ofautomated scripts. Once they are complete,Rational Quality Manager can remove thesoftware leaving the machine in a clean state.

Defect Tracking

Rational Quality Manager has built-in defect trackingcapabilities as well as integrations with the industryleading change management solutions such as RationalClearQuest and Rational Team Concert as well as opensource solutions.

This allows quality professionals to raise defects either directly or as the result of a test failure,raising issues from the test log allows Rational Quality Manager to automatically establishtraceability from the defect to the test log which then traces to the test case and requirement.

Once a defect has been submitted it will follow an

agreed workflow and will progress though a numberof states until it can be marked as closed.


20/22


Page 19

As the defect gets moved to each state, certain fieldswill become mandatory if information is required in orderfor the defect to progress. For example, to mark a defectas fixed a user may need to specify the resolution.

Rational Quality Manager allows quality professionals to create queries against work itemssuch as defects, requirements or tasks. This allows the user tofilter the data and return the subset required. These queriescan be personal to each user or shared across multiple usersor teams.

The results of these queries can then be added to the quality professionals dashboard todisplay the latest status of these items.

Test Reporting

The reporting engine supplied as part of Rational Quality Manager is known as RationalCommon Reporting and is based on IBM Cognos technology.

Rational Common Reporting is a separate downloadand is available from Jazz.net. It includes a largenumber of out of the box reports that can be customisedif required to fit your individual needs.

Your IBM representative will be more than happy to visitand demonstrate or discuss this functionality.


21/22


Page 20

Customisation

Rational Quality Manager offers customisation capabilities, depending on the asset, the level ofcustomisation will vary. Test Plans and cases allow users to add user defined categories andsections whereas work items such as defects allow additional fields, tabs and work flows.

Your IBM representative will be more thanhappy to visit and demonstrate or discussthis functionality.

Next Steps

Thank you for taking the time to evaluate IBM Rational QualityManager, any questions resulting from this evaluation or any of theproducts mentioned, please ask your IBM representative who will bemore than happy to help.

In the hope that your Rational Quality Manager evaluation has proved successful, your IBMrepresentative will be able to discuss the deployment of the proposed solution and provide

guidance around any infrastructure requirements.

A typical deployment of Rational Quality Manager may look similar to the below


22/22


As part of any proposed solution, your IBM representative will work in conjunction with you andour services team to develop an Implementation Plan, documenting the rollout and adoption ofthe chosen toolset.

The services team will be led by an experienced engagement manager whose responsibility is

to ensure the successful adoption of the IBM solution. The goal of any engagement is to makeour customers self sufficient through the gradual implementation of tools, process, training andconsultancy.

Rational Quality Manager training will include:

Essentials of Test Management with IBM Rational Quality ManagerThis course demonstrates the power of test management that is integrated withrequirements, defect tracking, and test execution capabilities. You will learn howRational Quality Manager provides a collaborative and customizable solution for test

planning and workflow control. When this solution is combined with the ready-to-usetracking and metrics reporting, you can quantify how project decisions and deliverablesimpact and align with business objectives.

Essentials of Manual Testing with IBM Rational Quality ManagerThe course helps you to create manual tests more quickly, streamline the execution ofmanual tests, and monitor manual test results. Additionally, you will learn how toincrease the reuse of test assets, and how to use external test data with test scripts.

Your IBM team is dedicated to ensuring your success using our software as the long term aimis the hope that you will be willing to highlight your success by becoming a reference.

Additional Resources

Additional articles and resources relating to IBM Rational Quality Manager can be found below

IBM Rational Quality Manager in 3 minuteshttp://www.youtube.com/watch?v=kao1K30LXEk

IBM Rational Quality Manager Demonstrations

http://www-01.ibm.com/software/awdtools/rqm/standard/

IBM Rational Quality Management ROI Calculator:http://www-01.ibm.com/software/rational/offerings/testing/roi/

IBM developerWorks Quality Manager pages for hints, tools, tips and tricks:http://www.ibm.com/developerworks/rational/roadmaps/rqmroadmap.html

IBM Rational Quality Manager Online Helphttp://publib.boulder.ibm.com/infocenter/rqmhelp/v2r0/index.jsp

rational quality manager eval guide

Documents