by
Tom Gilchrist, CSQA, CSQE,
Software Process Reviews/AuditsSoftware Process Reviews/Audits
Process OverviewProcess Overview
SASQAG 10/17/2002 [email protected] 2
Before we start…
• SQA Context
• Overview of SW Audit Process
• SW Audit Examples
Information in this presentation are my opinions and not necessary those of my employer.
SASQAG 10/17/2002 [email protected] 3
Some Terms/Ideas
• Process
• Deterministic vs. Non Deterministic
• Quality vs. Value
SASQAG 10/17/2002 [email protected] 4
Software Quality Assurance
• Check software products and processes to verify that they comply with the applicable procedures and standards. (Process Reviews or Audits)
• Review and measure the quality of software products and processes throughout development. (Dynamic & Static Testing)
• Provide software project management (and other appropriate parties) with the results of reviews and process checks.
• Work with the software project during early stages to establish plans, standards, and procedures to keep errors from occurring in the first place.
SASQAG 10/17/2002 [email protected] 5
Formal Definition
Audits provide an independent evaluation of software products or processes to ascertain compliance to standards, specifications, and procedures based on objective criteria that included documents that specify:
– The form or content of the product to be produced
– The process by which the products shall be produced
– How compliance to standards or guidelines shall be measured.
IEEE STD 1028, (1988)
SASQAG 10/17/2002 [email protected] 6
Audit Types
• First Party AuditFirst Party Audit– Within you company or organization
• Second Party AuditSecond Party Audit– Sometimes called “external audits”– By a Customer on his Supplier– By a Supplier on you.
• Third Party AuditThird Party Audit– Outside third party is contracted to do
the audit.
SASQAG 10/17/2002 [email protected] 7
Audit/Process Review Principles
• Conducted by individuals who are organizationally independent of the developers.
• Begin early in the requirements phase and continue throughout the development process.
• Professionally planned, conducted and documented.
• Follow-up on corrective action.• Project Management is involved in the Audit
process and is responsible for rework and process improvements.
SASQAG 10/17/2002 [email protected] 8
What Software Audit Should Do
• Determine:• Compliance to requirements• Conformance to plans, policies, procedures, and
standards• Drive process improvement based on:
• Adequacy of plans, policies, procedures, and standards
• Effectiveness and efficiency of plans, policies, procedures, and standards
• Assess personnel familiarity to requirements and documentation
• Assure availability, use and adherence to software standards
SASQAG 10/17/2002 [email protected] 9
What Triggers an Audit?
• Quality Assurance Plan• Event• Date
• Requests from management• Requests from developers• Requests from customers• Integration with process improvement activities• Outside requirements — regulatory• Gut feel
SASQAG 10/17/2002 [email protected] 10
Scope: Requirements, Time, and Target
Audit
Target
External
Standards
Organizational
Procedures and
Methods
• Spread around organization
• Cover all functions and activities
• Try to hit things early
• Move towards process audits
SASQAG 10/17/2002 [email protected] 11
Process Review/Audit Process
OK
PrepareAudit
Developers Project ManagerAuditor
ConductAudit
Write-upReport &Findings
Follow-upAudit
Re-Work
Findings?
NO
YES
CloseoutAudit &
File END
Reviewwith
Manager
Plan(Requirements,
Scope, & Checklist)Start
CorrectiveActions
SASQAG 10/17/2002 [email protected] 12
Identify Requirements
• Policies/Standards Corporate, Group, IEEE• Processes/Plans SCMP, SQAP, SDP, Project Plan• Procedures Change Management, Design
Reviews, Document Standards,
Testing • Task Instructions Library updates, unit testing, peer
reviews
• Success of an audit is directly proportional to preparation, research and analysis conducted before the audit is performed.
SASQAG 10/17/2002 [email protected] 13
Requirement Types
• Functional (ascertainably true or false)• Quality (range of acceptable values)
SASQAG 10/17/2002 [email protected] 14
Types of Audits (Internal)
• Quality System Audits
• Product Audit
• Process Audit
• Project Audit
• CM Audit
SASQAG 10/17/2002 [email protected] 15
Evidence Collection
• Collect Factual Information• Analyze and Evaluate the Evidence• Draw Conclusions• Generate Findings
SASQAG 10/17/2002 [email protected] 16
Corrective Action of Findings
• Determine Action– Immediate Remedial Action– Process Improvement/Fix– Acceptable Risk
• Identify Root Cause• Corrective Actions Plan • Manage CA Plan to completion• Analyze Effects of CA
SASQAG 10/17/2002 [email protected] 17
Develop Audit Checklist
• Focus on clear requirements (or unclear to fix)
• Select subset of requirements• Focus on important steps/products• Write clear concise questions• Canned checklist vs. straw horse
SASQAG 10/17/2002 [email protected] 18
Checklist Sample
Requirement Checklist Item Details Observations Results (P/F)
Company Standard ABC-234, page 7
Does project QA plan will have a list of deliverables subject to Peer Reviews?
Check SQA document for a list of approved peer reviews and which documents are to be reviewed. (if no documents are found, then fail. If no peer review procedures are referenced, then fail)
Project SQA Plan
Were the number of audits completed equal to the number planned?
Check to see which audits were planned for the last 60 days. Check for evidence that the audit was completed and if there were findings, that a CA plan was signed.
Project SQA Plan
Were the number of peer reviews completed equal to the number planned?
For each peer review type, check the CM records for the past 60 days to see if the document type specified in the QA plan was checked into CM for the first time. If so, check for records of the peer review being completed as per peer review process cited in SQA plan.
SASQAG 10/17/2002 [email protected] 19
Interviewing
• Ask open-ended questions• Know the types of answers expected• Focus on Process and not People• Seek Corroboration and Evidence
SASQAG 10/17/2002 [email protected] 20
Sample Interview Questions
• How do you track your progress?• Do you have a CM Plan?• Tracing
– What are you working on?– Is it a configured item?– Do you have an approved CR or PR?– Is the version you are working on
checked out of CM?
SASQAG 10/17/2002 [email protected] 21
Desirable Auditor Characteristics
• EmotionalEmotional• Interviews• Group
dynamics• Oral reports• Empathy• Don’t take
things personally
• MechanicalMechanical• Sampling• Root Cause
Analysis• IntellectualIntellectual
• Writing• Planning• Speaking• Detail
Oriented• Concise
SASQAG 10/17/2002 [email protected] 22
Desirable Auditor Characteristics(Cont.)
• Knowledge of Audit process• Knowledge of target (SW) processes• Knowledge of techniques • Professional attitude• Good listener• Inquisitive/analytical• Communicates at all levels• Detailed Notes and Observations• Diplomatic