vanq - january 25, 2007© josip pajk 2007page 1 sw quality assurance in cmmi ® vanq meeting january...

VanQ - January 25, 2007 © Josip Pajk 2007

SW Quality AssuranceSW Quality Assurancein CMMIin CMMI® ®

VanQ meetingVanQ meetingJanuary 25, 2007January 25, 2007

Josip Pajk Josip Pajk


AgendaAgenda

● Brief overview of the CMMI®

● Testing & Quality Assurance Process Areas● Some Considerations and Examples● Q&A ● Discussion

CMMI, CMM, and Capability Maturity Model are registered in the U.S. Patent and Trademark Office.CMM Integration, SCAMPI, and IDEAL are service marks of Carnegie Mellon University.

http://www.sei.cmu.edu/cmmi/cmmi.html

http://www.sei.cmu.edu/cmmi/cmmi.html


What is CMMIWhat is CMMI®® ? ?


1 Process Framework1 Process Framework


22 Process Areas in 4 Categories22 Process Areas in 4 Categories


6 Process Capability Levels6 Process Capability Levels

0. Incomplete0. Incomplete - a process that is either not performed or is partially performed.

1. Performed1. Performed - a process that supports and enables the work needed to produce work products.

2. Managed2. Managed - a performed (capability level 1) process that is planned, executed and tracked in accordance with a (documenteddocumented) plan, policy or procedure.

SG 1 SG 1 Objectively Evaluate Processes and Objectively Evaluate Processes and Work ProductsWork Products

SP 1.1 Objectively Evaluate ProcessesSP 1.2 Objectively Evaluate Work Products and

ServicesSG 2 SG 2 Provide Objective InsightProvide Objective Insight

SP 2.1 Communicate and Ensure Resolution of Noncompliance Issues

SP 2.2 Establish Records


Capability Levels (cont)Capability Levels (cont)

3. Defined3. Defined - is a managed (capability level 2) process that is tailoredtailored from the organization’s set of standard standard processesprocesses and contributes work products, measures, and other process improvement information to the organizational process assetsorganizational process assets.

4. Quantitatively Managed4. Quantitatively Managed - a defined (capability level 3) process that is controlled by establishing quantitative objectives for quality and process performance which is understood in statistical terms and managed by eliminating special causes of variationspecial causes of variation.

5. Optimizing5. Optimizing - a quantitatively managed (capability level 4) process that is improved based on an understanding of the common causes of variationcommon causes of variation inherent in the process.


5 Organizational Maturity Levels5 Organizational Maturity Levels

A Maturity Level is an “Evolutionary PlateauEvolutionary Plateau” for the organizationorganization on its way of process improvement.

=O=

1. Initial 1. Initial - Success in these organizations depends on the competence and heroics of the peoplecompetence and heroics of the people in the organization. Organizations are characterized by a tendency to over commit, abandonment of processes in a time of crisis, and an inability to repeat their successesinability to repeat their successes.

2. Managed2. Managed - Processes are planned and executed monitored, controlled, and reviewed in accordance with documented plansdocumented plans; involve relevant stakeholders; and are evaluated for adherence to their process descriptions. Work products and services are appropriately controlled and satisfy their specified process descriptions, standards, and process descriptions, standards, and proceduresprocedures.


Maturity Levels (Cont.)Maturity Levels (Cont.)3. Defined3. Defined - The organization’s set of standard processesorganization’s set of standard processes

are described in standards, procedures, tools, and methods. Standard processes are used to establish consistency across the organization and different projects. Projects establish their defined processesdefined processes by tailoringtailoring the organization’s set of standard processes.

4. Quantitatively Managed4. Quantitatively Managed - At maturity level 3, processes are typically only qualitatively predictable while at maturity level 4, the performance of processes is controlled using statistical and other quantitative techniques, and are quantitatively quantitatively predictablepredictable.

5. Optimizing5. Optimizing – The organization continually improves its processes based on a quantitative understanding of the understanding of the common causes of variationcommon causes of variation inherent in processes.


Generic Goals & PracticesGeneric Goals & Practices3 Institutionalize a Defined 3 Institutionalize a Defined

ProcessProcess3.1 Establish a Defined Process

3.2 Collect Improvement Information

4 Institutionalize a Quantitatively 4 Institutionalize a Quantitatively Managed ProcessManaged Process

4.1 Establish Quantitative Objectives for the Process

4.2 Stabilize Subprocess Performance

5 Institutionalize an Optimizing 5 Institutionalize an Optimizing ProcessProcess

5.1 Ensure Continuous Process Improvement

5.2 Correct Root Causes of Problems

1 Achieve Specific Goals1 Achieve Specific Goals1.1 Perform Specific Practices

2 Institutionalize a Managed 2 Institutionalize a Managed ProcessProcess

2.1 Establish an Organizational Policy

2.2 Plan the Process

2.3 Provide Resources

2.4 Assign Responsibility

2.5 Train People

2.6 Manage Configurations

2.7 Identify and Involve Relevant Stakeholders

2.8 Monitor and Control the Process

2.9 Objectively Evaluate Adherence

2.10 Review Status with Higher Level Management

Are inclusive and define both the Are inclusive and define both the Capability level of a PA as well as the Capability level of a PA as well as the

Maturity Level of the organizationMaturity Level of the organization


Organizational AssessmentsOrganizational Assessments

The MLsMLs are defined by a # ofGeneric GoalsGeneric Goals and Generic Practices Practices

SG 1 Objectively Evaluate Processes and Work Products

SP 1.1 Objectively Evaluate ProcessesSP 1.2 Objectively Evaluate Work Products

and ServicesSG 2 Provide Objective Insight

SP 2.1 Communicate and Ensure Resolution of Noncompliance Issues

SP 2.2 Establish Records

Each Process AreaProcess Area is described by a # ofSpecific GoalsSpecific Goals and Specific PracticesSpecific Practices

22

33


QA & Testing QA & Testing the CMMIthe CMMI® ® WayWay


Verification and Validation (ENG)Verification and Validation (ENG)

iterativeiterative

requirement (1)(1) A condition or capability needed by a user to solve a problem or achieve an objective. (2)(2) A condition or capability that must be met or possessed by a product or product component to satisfy a contract, standard, specification, or other formally imposed documents. (3)(3) A documented representation of a condition or capability as in (1) or (2).

The Engineering process areas apply to the development of any product or serviceany product or service in the development domain (e.g., software products, hardware products, services, or processes).

ML3ML3

ML2ML2


VERIFICATIONVERIFICATION

The purpose of Verification (VER)Verification (VER) is to ensure that selected work productswork products meet their specified requirements. Ensures “you built it right.”

work productwork product - any useful result of a processany useful result of a process. Can include files, documents, products, parts of a product, services, process descriptions, specifications, and invoices. A key distinction between a work product and a product component is that a work product is not is not necessarily part of the deliverable productnecessarily part of the deliverable product.


VALIDATIONVALIDATION

The purpose of Validation (VAL)Validation (VAL) is to demonstrate that a product or product componentproduct or product component fulfills its intended use when placed in its intended in its intended environmentenvironment.. Ensures “you built the right thing.”

productproduct - a work product that is intended for intended for deliverydelivery to a customer or end user.

product componentproduct component - a work product that is a lower level component of the product. Product components are integrated to produce the integrated to produce the productproduct. There may be multiple levels of product components.


Process & Product QA (SUP)Process & Product QA (SUP)

qualityquality The ability of a set of inherent inherent characteristics of a characteristics of a productproduct, product component, or process to fulfill requirements (and expectations) of customers.

quality assurancequality assurance A planned and systematic means for assuring management (and the customer) that the defined (product requirements and) standards, practices, procedures, and methods of the (development) process are applied.

(additions are mine)

Note: quality standards Note: quality standards and procedures are and procedures are NOT defined by QANOT defined by QA

AdvancedAdvancedPAs ML3&5PAs ML3&5

BasicBasicPAs ML2PAs ML2


Process & Product QA (2)Process & Product QA (2)The purpose of Process and Product Quality

Assurance (PPQA) is to provide staff and management with objective insightobjective insight into processes and associated work productsprocesses and associated work products.

PPQA involves the following:– Objectively evaluatingObjectively evaluating performed processes, work

products, and services against the applicable process descriptions, standards, and procedures

– Identifying and documenting noncompliancenoncompliance issues– Providing feedbackfeedback to project staff and managers on

the results of quality assurance activities– Ensuring that noncompliance issues are addressed


Relevant Specific PracticesRelevant Specific Practices

PPQAPPQA

VERVER

VER and PPQA are applied VER and PPQA are applied on on work productswork products while VAL while VAL is performed on is performed on products.products.

VALVAL


Objective EvaluationObjective Evaluation

objectively evaluateobjectively evaluate - To review activities and work products against criteria which minimize minimize subjectivity and biassubjectivity and bias by the reviewer. An example of an objective evaluation is an audit against requirements, standards, or procedures by an independent quality assurance functionindependent quality assurance function.

It may be appropriate in some organizations, however, to implement the process and product quality assurance role without that kind of independence. For example, in an organization with an open, in an organization with an open, quality-oriented culture, the process and product quality quality-oriented culture, the process and product quality assurance role may be performed, partially or completely, by assurance role may be performed, partially or completely, by peerspeers; and the quality assurance function may be embedded in the (development? management?) process.

It may be appropriate in some organizations, however, to implement the process and product quality assurance role without that kind of independence. For example, in an organization with an open, in an organization with an open, quality-oriented culture, the process and product quality quality-oriented culture, the process and product quality assurance role may be performed, partially or completely, by assurance role may be performed, partially or completely, by peerspeers; and the quality assurance function may be embedded in the (development? management?) process.

Peer review Peer review ??


Objective Evaluation (2)Objective Evaluation (2)

Issues that must be addressed to ensure objectivity if quality assurance is embedded in the processif quality assurance is embedded in the process:●Everyone performing quality assurance activities should be trained in quality assurancetrained in quality assurance.●Those performing quality assurance activities for a work product should be separateseparate from those directly involved in developing or maintaining the work product.●An independent reporting channelindependent reporting channel to the appropriate level of organizational management must be available so that noncompliance issues can be escalated as necessary.

Peer ReviewsPeer Reviews??


ExamplesExamplesand Some Finaland Some FinalConsiderationsConsiderations


Example 1- All in OneExample 1- All in One

The quality assurance engineer performs various product-testing dutiesproduct-testing duties to assure specified guidelines are being followedassure specified guidelines are being followed. The engineer begins corrective action for procedural or processing deficienciesprocedural or processing deficiencies by making sure programs conform to documentation specifications programs conform to documentation specifications; certifies that products are bug free and stable and works to develop, apply and develop, apply and maintain quality requirementsmaintain quality requirements that include the creation and execution of methods and procedures for testing and debugging programs.

Understand and follow agile methodologies and software engineering practices / Develop and execute test plans and test cases according to requirements documents / Perform black box, white box, regression, and load testing on test units / Work with developers for clarification in resolving issues / Participate in design and code reviews / Write user procedures and participate in system testing and training / Notify group leader of any problems or potential problemspotential problems as they may arise / Participate in SWATSWAT Team activities as assigned.


Example 2 – Mainly TestingExample 2 – Mainly TestingA SIT Tester is the individual responsible for contributing to a test plancontributing to a test plan, and creating and maintaining test scenarioscreating and maintaining test scenarios and test cases for a given for a given test typetest type. As required, a SIT Tester is also expected to contribute to the ongoing improvement of the testing process and to maintain current knowledge of testing tools and techniques.

Analyze requirements, providing feedback to developers relating to the providing feedback to developers relating to the testability of requirementstestability of requirements / Prepare SIT test estimates for assigned change requests as deemed appropriate / Determine test environment requirements including test data, physical location, user access, change control and any training needs / Obtain project team approval on all project team approval on all assigned testing deliverablesassigned testing deliverables (e.g., scenarios, cases, data, and where applicable, scripts) / Execute test cases for the assigned test type according to the test plan (if a test plan is available and signed off) / Turn over testing deliverables to specified testers / Record defects when actual Record defects when actual SIT results do not match SIT expected resultsSIT results do not match SIT expected results / Identify and report all SIT testing-related defects, issues and riskstesting-related defects, issues and risks / Record required process and product measurements


Example 3 – Mainly QA?Example 3 – Mainly QA?Quality Engineer / Quality Assurance Specialist / Define and establish the necessary systems, practices, methods and tooling to objectively objectively implement the elements of the Quality Programimplement the elements of the Quality Program / Prepare product quality and control plans / Assist in training of plant personnel in the training of plant personnel in the application of the quality programapplication of the quality program/new procedures/instructions / Assist the Quality Assurance Manager in the implementation of continuous process improvement methodology / Analyze statistical data used to monitor quality performance and improvement, to ensure ensure internal and external customer requirements are metinternal and external customer requirements are met / Investigate product/process quality issues and develop and initiate preventative develop and initiate preventative corrective actionscorrective actions / Conduct external supplier audits when required / Maintain good relationships with all functional groups and locations within the company by participating in functional and multi-functional teams / Attend various multi-disciplinary meetings as member and/or QA Manager back up / Maintain good relationships and liaison with external customers and suppliers on quality related matters / Adhere to all health and safety rules and procedures / Any direction from Manager to be followed / Maintenance and housekeeping of work area.


Testing Testing ≡≡ QA ? QA ?

● Who is then responsible for objectively evaluating testobjectively evaluating test work products?

● Customers?● Peers?

– Other testers?– Developers, SE, PM?

● They are subjective stakeholders subjective stakeholders (evaluation criteria?)● Is multi-subjectivity = objectivitymulti-subjectivity = objectivity?● Also, who is responsible for the evaluation (audit) of

development (including testing) processesprocesses?


QA QA ≡≡ Testing ? Testing ?● Evaluation (testing?) of work productswork products and

processes processes (peer review enough?)

– Pair programming (and testing) ?– Test Driven Development ?

● What about creativity?● Development (including testing) isis creative

while QA mustmust be bureaucratic!

– Do we really want moving quality criteria (not only requirements)?

● Again, who would you choose to evaluate your work?


The Cost View The Cost View (P. Crosby)(P. Crosby)● Cost of Performance (CoP)Cost of Performance (CoP) – when everything is

done properly the first time (no failures whatsoever)● Cost of Quality (CoQ) Cost of Quality (CoQ) - extra cost because we are

not sure it was done right the first time

– Cost of Conformance (CoC)Cost of Conformance (CoC) ● Cost of Apraisal (CoA)Cost of Apraisal (CoA) – Product Reviews,

Testing (only the first time, not re-testing)● Cost of Prevention (CoP)Cost of Prevention (CoP) – Process reviews,

Training, Process Improvement.– Cost of Failure or Non-conformance (CoN)Cost of Failure or Non-conformance (CoN) –

Any rework (whenever we find a bug)

CoQ=CoC+CoNCoQ=CoC+CoN


Cost of Quality Cost of Quality

From: Schiffauerova, A. and Thomson, V., “A review of research on cost of quality models and best practices”,International Journal of Quality and Reliability Management, Vol.23, No.4, 2006http://www.mcgill.ca/files/mmm/CoQModels-BestPractices.pdf

Traditional viewTraditional view Modern viewModern view

http://www.mcgill.ca/files/mmm/CoQModels-BestPractices.pdf


Questions ?Questions ?


DiscussionDiscussion● Should QA be

independent from Engineering and PM?

● Is quality just in the “eye of the beholder” (the user)?

● Can we assure quality by following some particular process?

● Is QA a management tool?● How much QA is enough?


Thank YouThank YouI would love to hear from you

[email protected]

vanq - january 25, 2007© josip pajk 2007page 1 sw quality assurance in cmmi ® vanq meeting january...

Documents

process performance

process capability levels

josip pajk slide

process framework slide

managed capability level

way of process improvement

performed capability

processes sp