standard 4 breakout session inspection audit program

Standard 4 Breakout SessionInspection Audit Program

Russell LillyManager, Manufactured Food Program

Missouri Department of Health and Senior Services

Wednesday, March 12, 20148:00 – 10:00

Elm Fork I

Patrick KennellyChief, Food Safety Section

California Department of Public Health

Peter Haase Director, Bureau of Food Safety and Inspection Wisconsin Department of Agriculture, Trade and

Consumer Protection

Skya MurphyProgram and Policy Analyst

Wisconsin Department of Agriculture, Trade and Consumer Protection

Angela KohlsStandards Implementation StaffFDA ORA, Office of Partnerships

AgendaStrengthening Compliance and

Enforcement • Welcome and Introductions• Current Status of Standard 4 Implementation• Presentations on Implementing Standard 4• Q & A/Group Discussion• Summary and Challenge

Housekeeping• Honor time schedule• Stay on topic • Get input from everyone• Be willing to share• Respect different program cultures• Do not reference specific audits• Be creative, ask questions – parking lot• Facilitators – responsible for process• Participants – responsible for content

Session Objectives

Generate a list of resources, best practices, lessons learned and challenges/recommendations for developing, implementing and monitoring an inspection protocol.

(Group Exercise Instructions)1. Listen to presentations2. Discuss and list suggestions on flip charts 3. Share your group’s suggestions with room

Standard Four

Full Conformance

(100%)80-99% 60-79% 40-59% <39%

PE Conformance Rating

4 4 2 2 2710 10 5 5 69

Total StatesPercentage (%)

Standard Four

1.a 1.b 2.a 2.b 2.c 3.a 3.b 3.c 4.a 4.b 4.c 5.a 5.b 6.a 6.b

Field Inspection Audits Inspection Report Audits Sample Report AuditQuality Assurance Program

Corrective Action Plan Documentation

No. of YES 8 10 11 18 16 8 12 12 8 9 8 11 8 15 12No. of NO 30 28 28 21 23 31 27 27 30 27 27 28 31 24 27No. of NA 0 0 0 0 0 0 0 0 1 3 3 0 0 0 0

Breakout Session Focus

• Elements 4.2b &c, 4.3b & c, and 4.4b & c Implementation of the 3 Types of Audits– Field Inspection– Inspection Report– Sampling

Standard 4: Field Inspection Audits

Patrick Kennelly, ChiefFood Safety Section


March 12, 2014

Field Inspection Audits

• Quality assurance review to assess the effectiveness of its inspection program.– 2 audits of each inspector every 36 months– Conducted by a qualified auditor– Documented on Audit form that is equivalent to

the FDA Contract Audit form– Need an 80% audit rating to be considered a

successful audit


• Then and Now– Better Defined Process– Uniform Documentation– Ability to Compare Ratings Across the Program


DEPARTMENT OF HEALTH AND HUMAN SERVICES FOOD AND DRUG ADMINISTRATION

CONTRACT AUDIT ☐ FDA CONTRACT AUDIT ☐ FDB INSPECTION AUDIT

AUDITOR STATE INSPECTOR

FIRM CFN / FEI NUMBER OR FIRM ID NUMBER

FIRM ADDRESS

PRODUCT(S) COVERED DATE

TIME IN TIME OUT OVERALL RATING

☐ ACCEPTABLE ☐ NEEDS IMPROVEMENT

I. PREINSPECTION ASSESSMENT

1. DID THE INSPECTOR REVIEW THE STATE’S ESTABLISHMENT FILE FOR THE PREVIOUS INSPECTION REPORT AND POSSIBLE COMPLAINTS OR ACCESS OTHER AVAILABLE RESOURCES IN PREPARATION FOR THE INSPECTION?

☐ Acceptable ☐ Needs Improvement

COMMENTS (required for Needs Improvement)

2. DID THE INSPECTOR HAVE THE APPROPRIATE EQUIPMENT AND FORMS TO PROPERLY CONDUCT THE INSPECTION? ☐ Acceptable ☐ Needs Improvement


Field Inspection AuditsII. INSPECTION OBSERVATIONS AND PERFORMANCE (Continued)

6. DID THE INSPECTOR DEMONSTRATE THE ABILITY TO DISTINGUISH BETWEEN SIGNIFICANT VERSUS INSIGNIFICANT OBSERVATIONS AND ISOLATED INCIDENTS VERSUS TRENDS?



7. DID THE INSPECTOR REVIEW AND EVALUATE THE APPROPRIATE RECORDS AND PROCEDURES FOR THIS ESTABLISHMENT’S OPERATION AND EFFECTIVELY APPLY THE INFORMATION OBTAINED FROM THIS REVIEW?



8. DID THE INSPECTOR COLLECT ADEQUATE EVIDENCE AND DOCUMENTATION IN ACCORDANCE WITH STATE PROCEDURES GIVEN THE NATURE OF THE INSPECTIONAL FINDINGS?




• Things to Consider– How do you select facilities for staff audits?– What constitutes a “qualified auditor”?– Do you want to vary or alternate auditors for

each staff?– How do you address individual “Needs

Improvement” Ratings?

Worksheet 4-BPerformance rating for the field inspection audits

State program: Performance period: Field inspection audit perfomance rating (4): Name/title of reviewer: Office: Date:

Auditor's initials, firm identification number, and date of inspection (1)

Performance Factors (5)

At

(3) NIt (3)

Performance Ratings (2)I.1 0 0I.2 0 0II.1 0 0II.2 0 0II.3 0 0II.4 0 0II.5 0 0II.6 0 0II.7 0 0II.8 0 0II.9 0 0

II.10 0 0IIA.1 0 0IIA.2 0 0IIA.3 0 0IIA.4 0 0III.1 0 0III.2 0 0III.3 0 0III.4 0 0III.5 0 0III.6 0 0

Subtotal Enter the sum of the totals from all continuation sheets. 0 0Total Enter the final sums (subtotal + sums of (3) on this form). 0 0

(5) USE THIS SPACE TO IDENTIFY AND MAKE NOTES ABOUT SINGLE PERFORMANCE FACTORS RATED AS “NEEDS IMPROVEMENT” IN MULTIPLE AUDITS.

0

Field Inspection Audits• Scoring across the spreadsheet identifies areas of

“Needs Improvement” ratings within a rated category.

– Need 80% “Acceptable” ratings within each rated category across the program to be in conformance.

– How do you address Non-Conformance ratings?


• Benefits– More Knowledgeable Workforce– Improved Inspection Procedures– Identifying Training Gaps– Findings Drive Quality Improvement

Contact Information

Patrick Kennelly, ChiefFood Safety Section


(916) 650-6598

[email protected]

mailto:[email protected]


Inspection Report Audits

Russell LillyManufactured food program manager

Missouri Department of Health

March 12, 20148:00 AM

Because…

• I had three inspectors and reviewed every inspection report

• It is wildly prescriptive – At least half of the widgets don’t apply

• If the shoe doesn’t fit it is uncomfortable

Our situation changed

• 18% of 22 inspectors

• A quality assurance program becomes necessary and valuable

We took the money…

• We owe an honest effort to comply with the standard

We did it…

• My best advice– Hire someone smart and assign it to them

Was it worth it?

• I think so

– We have sent a number of “corrective emails”• Mark this here not there• Be sure to put your “time out” on the form• Turn in your reports on time

I would still like

• To have a quality system designed from the ground up to fit our program

Contact Information

Russell Lilly

Manufactured food program manager


Email: [email protected]

Steps to Implementing Standard 4 Sample Collection Record Audit

Peter Haase Director, Bureau of Food Safety and Inspection



Sample Collection Record Audit Outline

• Introduction and Challenges • Steps to customization • Conclusion

• Documents on flash drive

Intro: Wisconsin Department of AgTrade & Consumer Protection

• Enrolled in MFRPS in 2007 • Challenges: Full Standard 4

Implementation Planned for 2015 – Phase III Contract Audit since 2011

• Currently Piloting Non-Contract Field Inspection Audit

• Inspection Report Audit 2013• SCR Audit 2014


Current Surveillance Sampling Program for Manufactured Food Establishments>Sampling at 2,400 establishments annually:

– 2 product samplesand

– 5-10 environmental swabs at each routine inspection


• Standard 10 Full Conformance– ISO Certified since 2007

• Sampling Policies and Procedures • Two Sample Collection Record Forms,

Food and Dairy

Challenges of Auditing our Sample Collection Records: Step 1 Test drive of Appendix 4.7.

Challenges of Auditing our Sample Collection Record (SCR): Step 1 Results of Test drive of Appendix 4.7.

– Findings: • 4.7 is more easily

managed in table format

• FDA Criteria:– Some not applicable or

redundant for our reports

– Very general: guidance specific to our SCRs required

Audited against our Policy

Added guidance specific to our Sample Collection Record formand procedures

Audited against our SCRs

Customized 4.7 Form:

Added guidance specific to our Sample Collection Record formand procedures

Piloted new form and guidance by auditing 71 SCR’s from Calendar 2012

• Results: – Opportunities for improvement: Use of hash

marks and down arrows, form completeness• Redundant criteria:

– SCR and sample always together

Challenges of Auditing our Sample Collection Records (SCR): Problem detected but not an audit criterion: Use of Correct Form

Substituted alternate criteria: Use of correct form

Challenges of Auditing our Sample Collection Records (SCR): For redundant criteria, combined and substituted

alternate criteria

Sample Selection (included in SOP for Standard 4)

• 75 SCR’s chosen per calendar year from MF establishments for auditing: – at least 1 per sampler, – random with regard to date, establishment

SOP for Standard 4 Cont.

• Entering and tabulating results: Excel Spreadsheet

• 1 for acceptable, • 0 for needs improvement

SOP for Standard 4 Cont.

• SCR Audit completed in March for previous calendar year

• Needs Improvement Factors (<80%) become training topics at staff meetings

• Other improvements can be made immediately upon discovery:– Already identified from SCR 2013 audit:

“Foodborne Outbreak” reason for sampling on new Dairy SCR

Conclusion

– Adapting 4.7 to our Sample Collection Record

• Meaningful results for quality improvement– Documents on AFDO-MFRPA website:

• Lab SOP #2060-Receipt and Integrity of Samples• SCR Policy and Procedures• SCR Forms (Food and Dairy)• Our Audit Form 4.7 and Data Entry/Calculations Sheet• Description of sample selection and procedures for

Standard 4 including Sample Collection Record Audits

Contact Information

Skya Murphy

Program and Policy Analyst




Contact InformationPatrick Kennelly, Chief

Food Safety SectionCalifornia Department of Public Health


Russell LillyManufactured Food Program Manager






Peter Haase, Director Bureau of Food Safety and Inspection Wisconsin Department of Agriculture,

Trade and Consumer Protection


standard 4 breakout session inspection audit program

Documents

inspection program

inspection observations

inspection protocol

audit rating

inspector review

b c implementation

list suggestions

list of resources