2010 report on oasis data 2007-2010
DESCRIPTION
2010 Report on Oasis Data 2007-2010. Long Beach, CA September 13 -14, 2011 R. Darrell Taylor Raytheon. Executive Summary. Baselines have been established for 2007 – 2010 No Feedback data, changing metric Review Proposed revision to the CB Report Card. Agenda. OASIS Entries – Audits - PowerPoint PPT PresentationTRANSCRIPT
Company Confidential 1
Registration Management Committee (RMC)
2010 Report on Oasis Data 2007-2010
Long Beach, CASeptember 13 -14, 2011
R. Darrell TaylorRaytheon
Long Beach, CA September 13-14, 2011
Registration Management Committee (RMC)
2
Executive Summary• Baselines have been established for 2007 –
2010• No Feedback data, changing metric• Review Proposed revision to the CB Report
Card
Registration Management Committee (RMC)
3
Agenda• OASIS Entries – Audits• OASIS Nonconformities• AS9100 Elements• Feedback • Monitoring Model
Company Confidential 4
Registration Management Committee (RMC)
Entries – Audits
Long Beach, CA September 13-14, 2011
Registration Management Committee (RMC)
5
Comparison of Entries 2007 -2010
• Entries:– 2007 – 5114– 2008 – 5972 ^16%– 2009 – 6969 ^17%– 2010 – 7502 ^7.6%– -2011 – 32538/1/2011
2007 2008 2009 20100
1000
2000
3000
4000
5000
6000
7000
8000
5114
5972
6969
7502
Total Entries
Increase slowing down
Registration Management Committee (RMC)
6
Comparison of Types2007 -2010
SUR REC INT
SUR=SurveillanceREC = RecertificationINT = Initial
2011 – 72%, 21 %, 7%
SUR REC INT2007 0.66 0.17 0.172008 0.67 0.15 0.182009 0.63 0.21 0.152010 0.68 0.20 0.12
Registration Management Committee (RMC)
7
Standards Entered Into Database
2007 2008 2009 20100.0000.0050.0100.0150.0200.0250.0300.0350.0400.0450.050 AS9120
AS9110NBR15100:2004EN9100:2003/S1EN9120:P2/S1EN9100:P2/S1AS9100C
AS9100B:
2007- 95%2008 - 95%2009 - 94%2010 -92%
38 AS9100C – 2010; 225 in 2011 as 05 8/1/2011
Registration Management Committee (RMC)
8
CRB Comments - 2010• 2007 – 2010 Data baselined
Going to Rolling Format
Company Confidential 9
Registration Management Committee (RMC)
Nonconformities
Long Beach, CA September 13-14, 2011
Registration Management Committee (RMC)
10
Pareto of Total Nonconformities
Totals
2010 – 20,0092009 -20,980
2008 -18,803
2007 – 14,957
2011 – 65% 2 or less
Registration Management Committee (RMC)
11
OASIS Ratio of Total Findings
Violates Normal DistributionStrongest fit 3 Parameter Lognormal
Identify CB’s in the bottom 5% writing nonconformities
157.5135.0112.590.067.545.022.50.0
Median
Mean
2.82.62.42.22.0
1st Quartile 0.000Median 2.0003rd Quartile 4.000Maximum 157.000
2.594 2.765
2.000 2.000
3.701 3.822
A-Squared 498.36P-Value < 0.005Mean 2.680StDev 3.761Variance 14.143Skewness 11.681Kurtosis 399.522N 7467Minimum 0.000
Anderson-Darling Normality Test
95% Confidence I nterval for Mean
95% Confidence Interval for Median
95% Confidence I nterval for StDev95% Confidence I ntervals
Summary for TOTAL
1751501251007550250
99.99
999580502051
0.01
TOTAL
Perc
ent
Mean 2.680StDev 3.761N 7467AD 498.357P-Value <0.005
Probability Plot of TOTALNormal - 95% CI
Registration Management Committee (RMC)
12
Total - Comments
• Field “Family of Standards” states ISO9001:2000 s/b 2008?
• 35 (<.5%)Entries with N/A in Numbers, what does that mean?– 2 Initials, 1 surveillance, 32 recertification
Registration Management Committee (RMC)
13
Ratio Majors/Entries
On average, one Major is written every fifth audit – NOTE: Average down to .18 per audit for 2010 (1/5 audits)
No Change in 2011 (.19) .22 for AS9100C audits
Count 6738 431 298Percent 90.2 5.8 4.0Cum % 90.2 96.0 100.0
RESP_TOTAL_MA Other10
8000
7000
6000
5000
4000
3000
2000
1000
0
100
80
60
40
20
0
Coun
t
Perc
ent
Pareto Chart of RESP_TOTAL_MA
Totals
2010 – 13872009 -1135
2008 – 976
2007 - 870
Registration Management Committee (RMC)
14
OASIS Ratio of Majors
NOT Normal Distribution strongest 3 parameter loglogistic
Identify CB’s in the bottom 5% writing majors
1086420
Median
Mean
0.200.150.100.050.00
1st Quartile 0.0000Median 0.00003rd Quartile 0.0000Maximum 11.0000
0.1689 0.2026
0.0000 0.0000
0.7309 0.7547
A-Squared 2197.91P-Value < 0.005Mean 0.1858StDev 0.7426Variance 0.5515Skewness 6.2414Kurtosis 50.7385N 7467Minimum 0.0000
Anderson-Darling Normality Test
95% Confidence Interval for Mean
95% Confidence Interval for Median
95% Confidence Interval for StDev95% Confidence I ntervals
Summary for RESP_TOTAL_MA
12.510.07.55.02.50.0-2.5-5.0
99.99
999580502051
0.01
RESP_TOTAL_MAPerc
ent
Mean 0.1858StDev 0.7426N 7467AD 2197.911P-Value <0.005
Probability Plot of RESP_TOTAL_MANormal - 95% CI
Registration Management Committee (RMC)
15
Ratio Minors/Entries
On average, 2.5 Minors are written every audit2011 – down to 2.12
AS9100C – 3.18
Count 149 2932191 1307 1190 906 553 399 263 217Percent 2.0 3.929.3 17.5 15.9 12.1 7.4 5.3 3.5 2.9Cum % 96.1 100.029.3 46.8 62.8 74.9 82.3 87.7 91.2 94.1
RESP_TOTAL_MI Other876543210
8000
7000
6000
5000
4000
3000
2000
1000
0
100
80
60
40
20
0
Coun
t
Perce
nt
Pareto Chart of RESP_TOTAL_MI
Totals
2010 – 18,6272009 -19,414
2008 -17,393
2007 –14,087
Registration Management Committee (RMC)
16
OASIS Ratio of Minors
NOT Normal DistributionLognormal Distribution
Identify CB’s in the bottom 5% writing minors
135.0112.590.067.545.022.50.0
Median
Mean
2.62.52.42.32.22.12.0
1st Quartile 0.000Median 2.0003rd Quartile 4.000Maximum 151.000
2.415 2.573
2.000 2.000
3.424 3.535
A-Squared 481.38P-Value < 0.005Mean 2.494StDev 3.479Variance 12.102Skewness 12.705Kurtosis 464.332N 7468Minimum 0.000
Anderson-Darling Normality Test
95% Confidence Interval for Mean
95% Confidence Interval for Median
95% Confidence Interval for StDev95% Confidence I ntervals
Summary for RESP_TOTAL_MI
1501251007550250
99.99
999580502051
0.01
RESP_TOTAL_MIPerc
ent
Mean 2.494StDev 3.479N 7468AD 481.379P-Value <0.005
Probability Plot of RESP_TOTAL_MINormal - 95% CI
Registration Management Committee (RMC)
17
Evaluation of Audit Type INT, REC, SUR• Chi Squared Test returned a P-Value = 0.069
Major AVG SD Count MIN MED MAXINT .2514 .7991 217 0 0 6REC .2609 .9707 384 0 0 11SUR .1532 .6492 786 0 0 10MinorINT 3.907 4.023 3372 0 3 29REC 3.369 3.980 4966 0 2 66SUR 2.005 3.091 10284 0 1 151
Registration Management Committee (RMC)
18
AS9100 Nonconformities Top 10
Top Eight 2010 same as 2009 & 2008
NO CHANGE
Registration Management Committee (RMC)
19
Feedback Analysis
2010 average 103 per month2009 average 112 per month2008 average 110 per month
2007 2008 2009 20100
500
1000
1500
2000
2500
As of June 6% of 2010 Feedback Responses still open
ON HOLD
Registration Management Committee (RMC)
20
Certification Body Statistical Assessment Model
ANAB Audits - 6 Criteria
OASIS Results - 3 CriteriaFeedback - 3 CriteriaGeneral - 1 Criteria
ANAB Performance -4
ON HOLD
Registration Management Committee (RMC)
21
Threshold Violations
Violations0
2
4
6
8
10
12
14
16 • RMC & ANAB need to research– >6
• ANAB to report to RMC – 4-5
• ANAB to investigate – 2-3
• No Action – <2
Company Confidential
Registration Management Committee (RMC)
Oversight Results2011 Major & Minor Numbers2 WA – 1 Minor / 0 Majors & 2 Minors / 0 Majors2 OA – 3 Minors / 0 Majors & 10 Minors / 3 Majors
2009 Major & Minor Numbers 2 WA –1 Minor /4 Majors 2 OA –7 Minors /0 Major
NCR trends – Not verifying OASIS admin. on site, OASIS accuracy, depth of audit, not documenting all NCRs, various checklist issues
CB Activity & AEA’sStandard Sites AEA’s9100 100 149110 Applicant 09120 8 8
OASIS Data Supplemental ActivityAB Actions: 2010 – March WA was a 2009 re-auditCB has received 2 OA’s per year for two years.1 major NCR written outside of an assessment
RMC Member: (Supplemental oversight data) 2 events, 4 minor NCR’s
CB Name: XXX -2011
AUDITS 74TOT MAJORS 75MAJOR AVERAGE 1.01TOT MINORS 275MINOR AVERAGE 3.72FEEDBACK 22AVERAGE DAYS OPEN 9.31NUMBER STILL OPEN 0
Company Confidential
Registration Management Committee (RMC)
CB Name: XXX -2011
Certification Body Performance ReviewCertification Body: XXXXXXXXXXXX
METRICS RULEICOP AUDIT RESULTS 2008 2009 2010 2011Number of Audits ### ### ### ###Minors >1SDMajors >1SDRatio of Minors to Audits >1SDRatio of Majors to Audits >1SDRatio of Total to Audits >1SDMinors issued outside audit >1Majors issued outside audit >0Increased audits >2
PERFOMANCE METRICSNegative Shift in Audit Totals >-1SDEscalated N/Cs >0Customer Complaints >0Suspensions (2 years) >0Negative Shift in Violations > - 1SDGENERAL (RISK) OPEN
VIOLATIONS
OASIS METRICSMinors ### ### ### ###Majors ### ### ### ###Total ### ### ### ###Ratio of Minors to Audits <1SDRatio of Majors to Audits <1SDRatio of Total to Audits <1SD
FEEDBACK MANAGEMENTClosed Average Closure >30 OPEN Average open days >30OPEN > 1 Year >365
ANAB Additional Comments
Registration Management Committee (RMC)
24
Questions