Audits Overview: NPAP, PEP, Annual PE, Flow Rate Audits

Guest Speaker:

Jeremy HoweLittle River Band Of Ottawa Indians

The Measuring Sticks of Air Monitoring

2

April 7: QA Systems, EPA definitions, PQAOs and common sense – Mike Papp

April 14: Routine Quality Control and Data Management (1-pt QC, flow rate, and instrument stability checks) – Travis Maki

April 21: Audits Overview (NPAP, PEP, Annual PE, Flow Rate Audits) – Jeremy Howe

April 28: Calculating Bias and Precision and AQS reports – Bill Frietsche

May 5: 40 CFR 58 App. A- Gaseous Pollutants – Glenn Gehring

May 12: 40 CFR 58 App. A- Ozone – Brenda Jarrell May 19: 40 CFR 58 App. A- PM filter and continuous

methods –Brandy Toft

Overall Course Overview:

3

Audit – “a systematic and independent examination to determine whether quality activities and related results comply with planned arrangements and whether these arrangements are implemented effectively and are suitable to achieve objectives”

Independent means using a device other than the one you use routinely at a minimum, and in some cases a completely independent system of instrument and operator

Audit definition:

4

Basically compare◦What is actually being done in the field and the office

◦Against what is stated in the QAPP and SOPs

◦Are done by you or someone outside your office, depending on the type of audit

All Audits (in some cases called “assessments”):

5

Number and types of assessments People or organizations doing the assessments

Schedule Criteria for assessments (we will

compare what we do against the SOP or QAPP)

Reporting and responsibility for follow-up (e.g., we will revise the SOP in pen for now, and retype it with changes within a month)

Describe in your QAPP

6

Program: CLOSE THE LOOP (fix problems and take action to make sure

they do not happen again)

And make sure it is documented

Responsibility for Follow-Up and Verification of Corrective Action

7

1. Six-month flow rate audits you make with non-routine standards (PM)

2. Annual Performance Evaluations (gas) 3. NPAP and PEP audits conducted by

certified auditors 4. Technical Systems Audits conducted by

EPA regions (every 3 years) To see how many/often of the 1st three types

you need for NAAQS-comparable data, see Calculations for Tribes Interested in Joining a Tribal PQAO - 2011-02-08A.xls

4 categories of audits:

QC Checks (and audits):Method Pollutants Frequency MQO

One-Point QC SO2, NO2, O3, CO Every 2 Weeks

O3 :

Precision = 7% Bias = 7%

SO2, NO2, CO :

Precision = 10%Bias = 10%

Annual Performance Evaluation (Audit)

SO2, NO2, O3, CO Once per Year <= 15% for each audit concentration

Flow Rate Verification (QC check)

PM10 (lo-Vol), PM2.5

Once every 4 weeks

<= 4% of Standard & 5% of Design Value

Flow Rate Verification (QC check)

PM10 (high-Vol), TSP

Once per quarter

<= 7% of Standard & 10% of Design Value

Semi-Annual Flow Rate Audit

PM10, PM2.5 Every 6 Months

<= 4% of Standard & 5% of Design Value

Semi-Annual Flow Rate Audit

PM10 (high-Vol), TSP

Every 6 Months

<= 10% of Standard & 10% of Design Value

Collocated Sampling PM10, TSP, PM2.5 15% of Network Every 12 Days

PM2.5, TSP, PM10 :

Precision = 10%

PM2.5 PEP Program

NPAP

PM2.5

SO2, NO2, O3, CO

Quarter *!Year (see Calculations…spreadsheet)

Bias = 10%

8

Who can do these, and with what?Method Pollutants Frequency Who? With What?

One-Point QC SO2, NO2, O3, CO

Every 2 Weeks Site operator Annual calibration standard, owned by tribe

Annual Performance Evaluation

SO2, NO2, O3, CO

Once per Year Preferably not site operator, no national certification required, not the same standard that is used for routine QC checks

Flow Rate Verification (not required to be reported to AQS)

PM10, PM2.5 Every 4 weeks Can be site operator, no national certification required, not the same standards that are used to calibrate your analyzer

Semi-Annual Flow Rate Audit

PM10, PM2.5 Every 6 Months Preferably not site operator, no national certification required, not the same standards that are used for monthly QC checks

PM PEP, NPAP PM2.5,

All gaseous

See Calculations…spreadsheet

OAQPS/TAMS certified auditor

OAQPS/TAMS equipment

9

10

First type: PM flow rate audits (counts as an audit, for bias, because it

is not the stnd used for precision FR checks):

Flow Rate Verification

PM10, PM2.5

automated and

manual

Every 4 weeks Can be site operator, no national certification required, the flow rate standard used must be certified annually

Flow Rate Verification

PM10 (high-Vol), TSP

manual

Once per quarter

Semi-Annual Flow Rate Audit

PM10, PM2.5

automated and

manual

Every 6 Months Preferably not site operator, no national certification required, not the same standards that are used to calibrate the analyzer, but can be calibrated to the same primary stnd as your cal stnd

11

A PM2.5 example, using the spreadsheet:

12

This yields, in the QC Checks tab:

13

And in the Audits tab:

14

Discussed in more detail in PM webinar Rely on owner’s manuals EPA guidance available: 2.12 "Monitoring PM2.5 in Ambient Air Using

Designated Reference or Class I Equivalent Methods" 1998 http://www.epa.gov/ttn/amtic/files/ambient/pm25/qa/m212covd.pdf

See docs in course webpage and AMTIC and CARB SOPs

1st type, cont.: PM Flow Rate Audits:

15

check the operational flow rate of the analyzer, before any flow rate adjustment

Randomize the time and day Use a flow rate transfer standard certified to

be traceable to an authoritative volume or other applicable standard (within 2% of NIST –Traceable Standard Part 50, App L Sec 9.2)

Reported to AQS as RA (bias) because it is made with a non-routine standard (the ones with your routine stnd are precision (RP)

% diff =[ (meas-audit)/audit ]*100

1st type, cont.: Semi-Annual Flow Rate Audits (PM):

16

3.2.2  Annual performance evaluation for SO2, NO2, O3, or CO

…The gas standards and equipment used for evaluations must not be the same as the standards and equipment used for calibration or calibration span adjustments…the auditor should not be the operator or analyst who conducts the routine monitoring, calibration, and analysis.

2nd type of audits: Annual Performance Evaluations (gas)

17

40 CFR 58 App A sec 3.2.2: If there are fewer than four analyzers for a

pollutant within a PQAO, P.E. annually (preferably in different quarters)

The evaluation should be conducted by a trained experienced technician other than the routine site operator

Current CFR guidance will change soon

2nd type of audits: Annual Performance Evaluations, gas, cont.:

18

A gas analyzers example, 3 at 1 site:

19

This yields, in the Checks tab:

20

And in the Audits tab, this yields 1 NPAP and 3 Annual PEs:

21

use…audit gas from at least three consecutive audit levels. The audit levels selected should represent or bracket 80 percent of ambient concentrations measured by the analyzer being evaluated:

2nd type of audits: Annual Performance Evaluations, gas, cont.:

22

New guidance for audit levels: Ten levels, the lowest as low as 4-6 ppb for

O3 Consecutive audit levels no longer required See memo: http://itep68.itep.nau.edu/itep_downloads/Q

A101_Resources/EPA%20guidance%20and%20other%20reference%20material/TechMemoforPEAuditLevels.pdf

2nd type of audits: Annual Performance Evaluations, gas, cont.:

23

3.2.2…Report both the audit and measured concentration

The % differences (di ) between these concentrations are used to assess the quality of the monitoring data, and to confirm your biweekly QC check results:

2nd type of audits: Annual Performance Evaluations, gas, cont.:

24

Allow…the audit gas to pass through all filters, scrubbers, conditioners, and other sample inlet components used during normal ambient sampling and as much of the ambient air inlet system as is practicable

< 15 % for each audit concentration, and EPA uses these (RA) values you report to

AQS to generate probability limits for your bias

2nd type of audits: Annual Performance Evaluations, gas, cont.:

25

2.4  National Performance Evaluation Programs. ….the consent of the monitoring organization for EPA to apply an appropriate portion of the grant funds, which EPA would otherwise award to the monitoring organization for monitoring activities, will be deemed by EPA to meet this requirement. For clarification and to participate, monitoring organizations should contact either the appropriate EPA Regional Quality Assurance (QA) Coordinator…

3rd type of audits: NPAP and PEP audits conducted by

certified auditors

26

PEP- $2000/audit◦ 5 audits/yr for PQAO with < 5 sites = 10K/year◦ All sites audited at least once every 6 years

NPAP- $2200/audit◦ Each site audited at least once every 5 yrs◦ At least 1 NPAP audit per PQAO network that

includes even 1 gas analyzer◦ Would need 10 sites for 2 audits a year◦ Each NPAP audit covers all gas analyzers at a site

These (rough) costs are for EPA contractors doing the audits

3rd type, cont.: NPAP and PEP audits conducted by certified auditors

(cont.)

27

Auditors (including tribal folks) can take training and exam

Equipment must be approved/provided as audit-specific equipment

SOPs available on AMTIC, http://www.epa.gov/ttn/amtic/npepqa.html

3rd type, cont.: PEP and NPAP audits

28

NPAP Audits (3rd type, for gas)• Separate from Cradle to Grave• For Gaseous Pollutants

• O3 + CO + SO2 + NOX

29

PEP Audits (3rd type, for PM2.5)

• Separate from Cradle to Grave• For Filter Based Pollutants

• PM 2.5 + Lead

PEP

CollocationPEP Audit

30

2.5  Technical Systems Audit Program. Technical systems audits of each ambient air monitoring organization shall be conducted at least every 3 years by your EPA regional office

Review of siting, documentation, procedures’ implementation

Assessment of whether the “QA function is independent”

Checklists that the region will use are available as QA Handbook Volume II, Appendix H

4th type of Audit: Technical Systems Audits conducted by EPA

regions (every 3 years)

31

At least 3 months notice, with written audit plan

Checklist sent ahead of time, see NAAM Technical Systems Audit Form in: http://itep68.itep.nau.edu/itep_downloads/QA101_Resources/EPA%20guidance%20and%20other%20reference%20material/

4th type, cont.: EPA Technical Systems Audits

32

All audits are “friendly” and intended to improve the quality of the data

Findings will always include insufficient documentation

Site visits are included Some AQS data are pulled, then records of

all the QC and original data are looked for in office

Important for small organizations: the “independency of the QA function”

How are audit results used?

33

CFR App. A: 2.2  …a quality assurance management function…includes strategic planning, allocation of resources and …assessing and reporting…

must have sufficient technical expertise and management authority to conduct independent oversight…

should be organizationally independent of environmental data generation activities (e.g., NOT the site operator or their manager)

Ideal QA manager is described at: http://www.epa.gov/ttn/amtic/files/ambient/monitorstrat/qamroles2.pdf

“Independent” QA Function:

34

Adds necessary quantitative and qualitative (TSAs) yardsticks for data defensibility

Adds value by exchanging information Course website:

http://itep68.itep.nau.edu/itep_downloads/QA101_Resources/

Consider joining the TAMS Steering Committee to help develop national tribal training to address these complicated issues (email us)◦ Jeremy Howe: jhowe@lrboi.com◦Melinda.ronca-battista@nau.edu◦Christopher.lee@nau.edu

Audits Conclusions: