*AS9101 Rev D Fundamentals9101 defines requirements for the preparation and execution of the audit process It defines the content and composition for the audit reporting of conformity and process effectiveness, based on9100-series standards requirementsthe organizations quality management system documentation, andcustomer/regulatory requirementsIt builds on and mandates requirements and guidelines in ISO/IEC 17021 and ISO 19011 Reminder

*AS9101 Rev D:

CBs have now been using AS9101 Rev D for some time and for all 91XX:2009 transition auditsWe have learned that auditing practices and use of 9101 is inconsistentWe need to improve the effectiveness of CBs auditing and audit reporting toDetect and stop nonconforming practices before they become normalisedEnsure that CB clients and their clients gain benefit from the audits and the audit reportsBuild and retain confidence airworthiness and other authorities confidence in the ICOP scheme

*Review of CB Audit Reports:

The IAQG OPMT carried out a review of sample reports in October 2011ABs have also been reporting findings made as a result of witnessed assessments and review of CB audit reportsFollowing slides provide information on those findings and will help prepare you for assessments you will carry outThere will be IAQG OPMT material in future that focuses on positive models for improvement

*General 9101 Report Findings:

Established audit objectives are not always in line with the example in 9101 instructions e.g. Upgrading of certification to 9100:2009Audit Plans and OERs do not always demonstrate that an interview with top management has taken placeAnnex E audit conclusions are weak in relation to overall performance, effectiveness and compliance of the QMS of the client organisation or in respect of the audit objectives often providing stories of the client history.

*General 9101 Report Findings:

Forms are not being filled accurately from 9101 instructionsStage 1 form is incorrectly populated with regard to Aviation, Space and Defence business and associated employeesForms other than the OER are being modifiedPrevious version 9101 forms are being used on occasions e.g. for NCRsAnnex E not being completed until after all of the NCRs raised were closed out i.e. completed at final situation not situation at closing meeting

*General 9101 Report Findings:

Recommendations being made incorrectly:Recommendation for certification made even though NCRs are raised (i.e. wrong recommendation box ticked)Virtually no use of Annex G for complex clients with more than one siteLack of understanding that a second Annex E is required after a visit to close out NCRsDecision-making is not finding and resolving issues in audit reports No specific requirement for decision-makers to attend AATT or similar to understand 91XX:2009 and AS9101 Rev D requirements on auditors

*9101 Report Findings QMS Matrix:

The QMS Matrix does not always contain all processes of the clientQMS Matrix does not contain sub-processes or valued-added activitiesQMS Matrix completed with X rather than the required C or N

*9101 Report Findings - PEARs:

Processes that contain clause 7 of 91XX:2009 requirements do not have an associated PEAR in all casesPEAR forms not always contain a description of the process and interactions that is sufficient to understand the clients process (or meet requirements)Clauses recorded on the PEAR are not always consistent with QMS matrix or listed as see QMS MatrixPEAR process measures do not always contain targets

*9101 Report Findings PEARs:

PEAR process measures accepted are not always consistent or applicable to the control of the processActual process performance is not always reportedImprovement actions against performance below target do not in all cases result in process achieving targets but process described as Level 3Improvement actions are not described just recorded as in place

*9101 Report Findings PEARs:

Several processes effectiveness measures described but actual performance data not reported against all themProcess effectiveness not always recorded on the PEARAudit conclusions and process effectiveness determination contain information that appears to be nonconformities but no conformities raised.PEARs do not always contain the detail of the audit trail to confirm the process is deployed, working and effective as described by the client

*9101 Report Findings NCRs:

Incorrect classification of nonconformities against the 9101 definitions (previous pre-2009 version definitions are still being used)Containment, root cause and corrective actions being accepted by auditors even if they dont address or fully address the nonconformity raisedDetails of the verification activity are not always described on the NCR form

*General Report Findings - OERs:

OER forms are not being fully populated many boxes have no objective evidence (or reference to a PEAR)OER does not have columns correctly completed with required C or NCRPEAR forms reference the OER however no objective evidence recorded in the OER at the reference point

*Additional Feedback:

Because a stage 1 was not mandated for transition insufficient planning is taking place ahead of the on-site transition audit to fully inform the audit teamPerformance measures set by the company are not traced back to the customer or other requirements to confirm validity of targetAuditors are guiding or prompting the client as to the correct number of processesAuditors are seeking to minimise the number of PEARs to be raised by compressing processes together at a higher level e.g. Manufacturing

*Oversight and 9101 During oversight you will see 9101 in use:As part of the audit process requirements during witness assessmentsAs part of the overall set of records held by the CB to demonstrate the audit process In both cases you will be looking to see that 9101 requirements have been followed but also that CB audit teamsAudit processes and process performance Follow and demonstrate the required focus elements Conduct a competent audit that demonstrates the overall QMS performance and effectivenessAudit conclusions reflect the audit trail and findings

*Oversight and 9101 Think about how the OEM will use the audit report, to understand:How their supplier worksHow effectively the client is turning customer requirements into deliverablesIf the supplier will deliver conforming products on-timeWhere the weaknesses in the suppliers management system existsHow effectively the supplier is fixing problems and improving to deliver conforming products on timeIf further assessment of the supplier is required

*Approach to oversight Use the QMS Matrix to quickly understand the established processesUse the PEARs to gain an understanding ofWhat each process does inputs to outputsHow the process works - activitiesHow the process is measured and monitoredIf the measures are appropriate to the processHow the process is performing against targetsIf not performing effectively what the supplier is doing to improve and if that improvement is workingIf you cant determine these items then the PEAR is not complete or effective

*Approach to oversight Where the PEAR identifies nonconformityFollow the nonconformityNonconformities:What were they?Is the statement of nonconformity actually a statement of nonconformity?Is the nonconformity consistent with the issue identified?Was it correctly graded?What caused it?How was it contained and corrected?What was used to verify the correction?Is the process now conforming and effective?

*Approach to oversight What are the other nonconformities?Should they have been inside the PEAR?Are they correctly graded?Review the OER if you need detail on a specific item or from a PEAR promptReview the audit reportDoes it reflect the audit results?Does it draw appropriate conclusions?Are the conclusions in line with the results and the audit objectives?Is the recommendation appropriate given everything else?

*Approach to oversight The certification decisionWho took the decision?Were they competent?Were they effective?Was the complete report uploaded to OASIS?Correctly?Everything uploaded that was supposed to be?Uploaded before previous certificate expired?Uploaded within correct timescales?Last day of surveillance + 90 daysCertification decision + 30 days

*Forms Focus QMS Matrix (Annex D) QMS Matrix:To be completed by the audit team for each visited site to demonstrate which processes and quality management system clauses have been auditedRelates the Suppliers processes to clauses within the 91XX:2009 standardWhy useful?Identifies which processes contain clauses that fall within clause 7 of 91XX:2009Each product realisation process is to be recorded on a PEAR (see 9101 clause 4.2.2.5)Quick cross check to determine if the CB has raised sufficient PEAR (see 9101 clause 3.7) forms.

*Forms Focus PEAR (Annex C) Process Effectiveness Assessment Report:Defined as A document stating results and providing evidence of determination on the effectiveness of a processEach product realisation process is to be recordedContent is to reflect 9101 clauses 4.1.2.4 and 4.1.2.5 regarding process management and process performance and effectiveness 9101 Clause 4.2.2.5 NOTE 2: If clause 7 of the 9100-series standards and its sub-clauses are associated with processes that are addressed by PEARs, the objective evidence is only required to be documented on the PEAR. In such case, the PEAR number should be referenced in the OER.

*Forms Focus PEAR (Annex C) Process Effectiveness Assessment Report:9101 Clause 4.2.2.5 NOTE 4: At the discretion of the auditing organization, other processes can be recorded on a PEARThe results of effectiveness shall be recorded on the PEAR (see Annex C) for each audited product realisation process. The level of effectiveness for each recorded process shall be recorded on the PEAR (statement of effectiveness level). The level of effectiveness for each recorded process has been classified as a 2 or a 1, this shall result in a nonconformity being issued against 9100-series standards clauses 4.1.c and f (see clause 4.2.4).

*Forms Focus PEAR (Annex C) Statement of Effectiveness:

Instructions:

*Forms Focus PEAR (Annex C) Turn to PEAR form and instructions pageReview form and associated instructions and note the followingBox 4 Requires the OASIS OIN numberBox 9 - Process details, including associated process interfacesSummarise the process activities, inputs, and outputs; including the identification of associated process interfacesBox 10 Applicable AQMS Clause(s)Identify the applicable primary 9100/9110/9120 clause(s) for this process.

*Forms Focus PEAR (Annex C) Review form and associated instructions and note the followingBox 11 - Organizations method for determining process effectivenessDescribe the method used by the organization to determine process effectiveness[e.g., identification of Key Performance Indicators (KPIs) and associated targets, process capability data].Box 12 - Auditor observations and comments supporting process effectiveness determinationAnnotate relevant objective evidence, observed conditions, data, information, comments, etc. to support the auditor's statement of effectiveness or ineffectiveness, as indicated in Box #13.This evidence can replace the evidence in the OER

*Forms Focus PEAR (Annex C) Example PEAR: Purchasing ProcessExample has three parts; bad example, good example and the form instructions as below:

Discuss the examples

*Forms Focus OER (Annex A) Objective Evidence Record:A document recording objective evidence of the audit findings, including reference to the reviewed or observed procedures, records, products, processes, and associated NCRs and opportunities for improvement.9101 Clause 4.2.2.5The audit team shall record detailed objective evidence (e.g., reviewed procedures, shop orders, training records, products, verification records). The objective evidence shall be on a standardised form [i.e., the OER (see Annex A)], or on the CBs own documentation. In this case, the CB document shall meet the intent of the OER. The completed forms shall be included in the audit records maintained by the CB.

*Forms Focus OER (Annex A) Objective Evidence Record:9101 Clause 4.2.2.5 continued NOTE 1 Population of the OER may start during the Stage 1 audit to record the documents reviewed.NOTE 2 If clause 7 of the 9100-series standards and its sub-clauses are associated with processes that are addressed by PEARs, the objective evidence is only required to be documented on the PEAR. In such case, the PEAR number should be referenced in the OER.No other form can have an equivalent or alternative developed by the CB for its own useDiscussion Point:What would you expect a CBs own documentation to look like that meets the INTENT of the OER?

*Forms Focus NCR (Annex B) Nonconformity Report:The NCR (see Annex B) shall be used to record nonconformities; Each NCR shall contain only one nonconformity. When nonconformities are identified, the audit team shall categorize the nonconformity as major or minor, according to the definitions provided in section 3 of 9101. Please read 3.2 Major NonconformityPlease read 3.3 Minor NonconformityDiscussion Point:What has changed in these definitions from the previous 9101 document?

*Forms Focus NCR (Annex B) Nonconformity Report:The need for containment in accordance with the organizations corrective action process shall be reviewed by the audit team.Please read 9101 clause 3.1 ContainmentRecurrence of the same or similar nonconformity found during consecutive audits at a particular site/location shall be considered as a failure of the corrective action process (see 9100-series standards clause 8.5.2) and shall result in a major nonconformity being issued.

*Forms Focus NCR (Annex B) Nonconformity Management (9101 clause 4.2.4):Turn to 9101 clause 4.2.4 and read the clauseNote the timescales set out in this section: 7 calendar days after the audit when the nature of the nonconformity needs immediate containment next 14 calendar days to reach agreement with the audit for the effectiveness of the action takena maximum of 30 days from the end of the on-site audit to agree corrective action(s) and corrective action plansDiscussion Point:According to ISO 17021 or 9101, what is the maximum time a nonconformity report can remain open before verification activities are required to be completed?Consider Initial audit, Surveillance, Recertification

*AS9101 Rev D FAQsAnswers to Frequently Asked Questions have been developedAvailable on the IAQG Website at: http://www.sae.org/iaqg/organization/requirements.htm

*91XX:2009 Questions? Refer to Deployment Support Material i.e. FAQs, Clarifications, Changes and rational and Auditor Guidance MaterialAvailable on the IAQG Website at: http://www.sae.org/iaqg/organization/requirements.htm

Questions?*