certification: assisting the ir measurement community randy j. jost utah state university roger w....

Certification: Assisting the IR Measurement Community

Randy J. JostUtah State University

Roger W. DavisEG&G Technical Services, Inc.

6th Annual RCS Measurement Facilities Certification Conference9-11 July 2002

IR Certification, Jost & Davis @ 6th Annual RCS Measurement Facilities Certification Conference - 9 July 02

2

"Measurement science today is a race between engineers and physicists trying to develop better, and more detailed, idiot-proof calibration and characterization procedures, and the universe trying to build bigger and better idiots.

So far, the universe is winning."

R. Jost

Motivation #1


3

"Nothing is fool-proof

to a talented fool”- Unknown

“Good documentation helps find

the proper fool to counsel”- R. Jost

Motivation #2


4

This briefing will cover:

• What is ANSI Z540 and what are its origins?

• Why and how has ANSI Z540 been applied in signature measurements?

• Issues for the IR measurement communities to address in order to employ ANSI Z540.

• How do we get there from here?


5

ANSI Z540 and its origins?

• ANSI/NCSL Z540-1-1994 (Formal name)

• American National Standard for Calibration – Calibration Laboratories and Measuring and Test Equipment – General Requirements (Formal title)

– Approved 27 July 1994

“The intent of this Standard is that laboratories meeting its requirements comply, for calibration activities, with the relevant requirements of the ISO 9000 series (ANSI/ASQC Q 90 series) of standards, … when they are acting as suppliers producing calibration results.”– From the Foreword, emphasis added

• American implementation of ISO Guide 25


6

Some Background

• Radar Cross Section (RCS) measurement community joined forces to collectively raise the quality bar

• Primarily DoD and its contractors, with important contributions made by NIST, NASA & academia

• National Institute of Standards & Technology (NIST) surveyed existing standards

– Found ISO Guide 25 met many of the needs of the RCS community

• ANSI/NCSL Z540-1-1994 US implementation


7

RCS Community Adopted ANSI Z540

• ISO 9000-like approach to improve quality of RCS measurements– Primarily a documentation standard

– Provides focus on key issues common to RCS measurement facilities of all kinds

– Enables uniform concepts & language for addressing & discussing quality issues

– Can be used in a certification program, but not necessarily so

• the term certification caused some friction within the RCS community

• Lessons learned can be applied elsewhere


8

History of RCS Certification Program• Initiated 1995 in DoD by USAF & Navy

– Major proponents: Air Force Research Laboratory & Naval Air Warfare Center Aircraft Division

– Other DoD participants: 46th Test Wing (Holloman & Eglin), Army Research Lab & Aberdeen Test Center, Air Force Information Warfare Center

• NIST playing key role– Objectivity and degree of rigor not commonly found in DoD

• Industry on board– Lockheed Martin, Boeing, Northrup-Grumman, Raytheon, Pratt &

Whitney, GE Aircraft Engines, BAE

• First two DoD facilities certified June 2000 as compliant with published evaluation criteria based on Z540– First Industry facility certified in June 2001; Six facilities now

certified


9

Certification: What it is & is not• Certification, in context of National RCS

Certification Program, means:– Reviewed by independent reviewers using published

evaluation criteria and found to be substantially compliant with the documentation provisions of Z-540

• Certification does NOT mean:– A facility can make measurements to some degree of

accuracy or other quantitative metric

• Why the distinction?– Quantitative assessments readily and often abused,

especially by political hacks


10

Possible Benefits to Other Communities

• Rigorous & methodical approach helps focus on common issues impacting quality of data

• Document what you do, who does what, how it is done, why, what training is required, what processes & procedures are followed, & when

• Establishes regular reviews (internal / external)

• Provides excellent visibility into your operations by customers and potential customers

– Assures customers they are getting more than lip service when it comes to quality data products

Value to training new personnel, alone, is worth the effort!


11

Possible Benefits to Other Communities

• Note that these benefits are available to any entity, group or community that is required to perform calibrated measurements as part of their operations.

• For the remainder of this briefing, we will focus on the various IR measurement communities.

• The IR community is a much more segmented group than the RCS community, and thus faces additional challenges and advantages.


12

IR community-specific issues

• Challenges

– More types of measurements => more types of calibration approaches.

• reflection, emission, transmission and absorption processes.

• Incoherent, partially coherent, coherent radiation

• Transmission path a major player

– Spectrum of interest is much more “diverse” than that of the RCS community.


13


• Advantages

– NIST is already involved, especially the Gaithersburg office.

– “Basic science” of calibration has been examined for a longer period of time, and is potentially better understood.

– More people are looking at the problem => more approaches to consider (note that this could be construed by some people to be a challenge).


14


• Benefits

– Many of the same benefits that the RCS community has seen

– Will help develop a priority list of “best defined” characterization/calibration methods

– Facilitate filling in the calibration and methodology “gaps” that may be identified.


15


• How do we get there?

– Realize that as a community it won’t happen overnight, but as an individual entity, benefits are available in the short run

– Need to identify 1-2 prime movers with an ownership mentality

– Find a means to achieve a clearly defined and early success

– Take advantage of the RCS community’s expertise, but tailor to IR community’s needs


16


• How do we get there?

– Establish contacts in participating organizations.

• 2001 RCS Conference: ~ 34 organizations

• 2002 SDL IR Conference: ~ 40 organizations

• 12 of these organizations were common to both conferences, Government, Industry and Academia

– NIST, AF Metrology, NASA-Langley, MIT/LL, AEDC

– Ball Aerospace, Lockheed-Martin, Northrop-Grumman, Raytheon, SAIC, Sverdrup/AEDC

– USU


17

Range Books = One Approach1 Introduction and Endorsement2 References3 Glossary4 Organization and Management5 Quality System, Audit, and Review6 Personnel7 Accommodation and Environment8 Equipment and Reference Materials9 Measurement Traceability and Calibration10 Calibration Methods

11 Handling of Calibration Items12 Records13 Certificates and Reports14 Subcontracting of Calibration15 Outside Support Services and Suppliers16 Complaints17 Inter-range Comparison Programs18 Data Processing Procedures19 Range-Specific Uncertainty Analysis20 Ongoing Research, Planned Improvements

Sections 9, 10, 17 & 19 deserve and receive considerable attention and yield tremendous benefit through rigorous examination; however, the holistic approach is necessary to ensure consistent quality across the board.

If applied in a certification program, sections such as these best applied with a priority and weighting scheme.


18

ANSI Z540 / ISO Guide 25 / ISO 17025?

• At the outset, the application of ISO Guide 25 & ANSI Z540 ran into some difficulties relating to ISO 9000

• Re-write to address those issues yielded ISO 17025– published in late 1999

• 5-year migration plan in US

• If starting a completely new quality or certification program in a signature measurement discipline, consider 17025


19

Summary• Across-the-board data quality improvement was

the original impetus; actual, measurable benefits to date far exceed that goal

• What makes sense for your facility is key element of National RCS Certification Program– flexibility, ingenuity, applicability

• A well-structured, phased approach built on the approach that the RCS community used can be also be used by the IR community– Identify a prime mover and a champion (with

resources)

– Locate and use all (experienced) company resources.


20

Summary• We all face tremendous personnel issues

– training = critical component to ensuring adequate skill sets

– documentation is key to any training program related to complex systems

• Z-540 offers an approach if you need to “eat the elephant”: – do it one orderly bite at a time

– documenting how you hold the fork and knife as well as the number of times you chew and swallow.

• This approach is based on a process, thus, develop a long term vision.


21

Concluding Thought

"If you can't describe what you

are doing as a process, you

don't know what you're doing."

- W. Edwards Deming

“If you can’t document your process,

no one else will believe you

know what you are doing.”

- R. Jost

Questions / Comments ?Randy J. Jost

Assistant Professor of Electrical & Computer Engineering, Utah State University(435) 797-0789

[email protected]

Roger W. DavisConsultant

EG&G Technical Services, Inc.(301) 863-0443

[email protected]

For a copy of this presentation, go to:

http://www.engineering.usu.edu/ece/faculty/rjost/Research/Research.html

certification: assisting the ir measurement community randy j. jost utah state university roger w....

Documents

ir certification

term certification

jost davis

ir measurement communities

rcs community ansincsl

measurement science

rcs community lessons

ir measurement community