quality assurance & post award performance measurement
TRANSCRIPT
NATIONAL CONTRACTMANAGEMENT ASSOCIATION
BETHESDA NORTH MARRIOTT HOTEL & CONFERENCE CENTER BETHESDA, MD
NATIONAL CONTRACT MANAGEMENT ASSOCIATION 2
QUALITY ASSURANCE & POST-AWARD PERFORMANCE MEASUREMENT
Marc Berson, Director, Grant Thornton LLPEric Heffernan, Senior Manager, Grant Thornton LLP
Thursday, November 202:10 – 3:10 pm
NATIONAL CONTRACT MANAGEMENT ASSOCIATION 3
QUALITY ASSURANCE AND POST-AWARD PERFORMANCE TOPICS
Policy & Guidance
Quality & Performance throughout an Acquisition
Lessons Learned
Questions & Answers
NATIONAL CONTRACT MANAGEMENT ASSOCIATION 4
IN BRIEF
"It's much more difficult to measure nonperformance than performance.“
- Harold S. Geneen, CEO ITT
"What gets measured gets done, what gets measured and fed back gets done well, what gets rewarded” gets repeated
- John E. Jones, Federal Judge
"You get what you measure. Measure the wrong thing and you get the wrong behaviors."
- John H. Lingle , Business Author
NATIONAL CONTRACT MANAGEMENT ASSOCIATION 5
POLICY & GUIDANCE
• Performance Based Service Acquisitions defined– Public Law 106-398, section 821– FAR Section Subpart 37.6– FAR Section Subpart 46.4
• OFPP pamphlet No. 4: "A Guide to Writing and Administering Performance Statements of Work for Service Contracts (1980)
• SARA Acquisition Advisory Panel Report Chapter 2: Improving Implementation of PBSA in the Federal Government
• 2006 Survey of Acquisition Executives finds few know how to do it right!
NATIONAL CONTRACT MANAGEMENT ASSOCIATION 6
"Could use performance incentives more effectively"
"Has relied on invalidated contractor data to monitor and assess contractors performance"
SAMPLE GOVERNMENT ACCOUNTABILITY OFFICE REPORT
NATIONAL CONTRACT MANAGEMENT ASSOCIATION 7
WHY IS QUALITY ASSURANCE AND PERFORMANCE MONITORING IMPORTANT?
• Continuity and consistency in the delivery of
products or services
• Means to hold vendor accountable and provide a
basis for incentives/disincentives
• Provides a structured process for evaluation
• Means to hold the vendor to an expected level of
service
NATIONAL CONTRACT MANAGEMENT ASSOCIATION 8
ADDRESSING QUALITY AND PERFORMANCE THROUGHOUT THE ACQUISITION LIFE-CYCLE
• Critical to a performance-based acquisition, yet often times the most overlooked
• Guided far less by law, regulation, and policy
• Is in the best interest of all parties for a successful contract
• Leads to improved performance over time
• Transparency of post award monitoring with other federal agencies
– Agency IGs– GAO– OMB
Contract T
ype
Performance
Requirements Summary
Cost Impact
Evaluation
Criteria
Qua
lity
Man
agem
ent
Pla
nC
on
trac
t C
lau
ses
&
Ince
nti
ves
Quality A
ssurance
Surveillance P
lan
Perfo
rman
ce R
epo
rting
Performance MeasureDevelopment
Emphasis onPerformance
Planning
Man
age &
Measu
re
Pro
cure
men
t
NATIONAL CONTRACT MANAGEMENT ASSOCIATION 9
PLANNING
NATIONAL CONTRACT MANAGEMENT ASSOCIATION 10
THREE CONCEPTS TO CONSIDER IN DESIGNING PERFORMANCE MEASURES
AlignmentAlignment Measure Measure TypesTypes
FrameworkFramework
Measures should directly support or be
in alignment with objectives and requirements
Measures should be defined in the context of a performance management framework
Various types of measures should be
considered
NATIONAL CONTRACT MANAGEMENT ASSOCIATION 11
BALANCED SCORECARD FRAMEWORK
What measures within each perspective can help indicate whether an objective was achieved?
Program Success
Fina
ncia
l Custom
er
Lear
ning
& G
row
th
Process
EVMCustomer
Complaints
CertifiedEmployees
Error Rate
Cause & Effect Relationship
Integration of External &
Internal Measures
NATIONAL CONTRACT MANAGEMENT ASSOCIATION 12
MEASURE SELECTION CRITERIA
MEANINGFUL – Related significantly and directly to organizations
mission and goal
VALUABLE – Designed to measure the most important activities of
the organization
BALANCED – Inclusive of several types of measures
PRACTICAL – Affordable price to retrieve and/or capture data
COMPARABLE – Used to make comparisons with other data over
time
CREDIBLE – Based on accurate and reliable data
TIMELY – Designed to use and report data in a usable timeframe
SIMPLE – Easy to calculate and understand
NATIONAL CONTRACT MANAGEMENT ASSOCIATION 13
PROCUREMENT ACTIVITIES
NATIONAL CONTRACT MANAGEMENT ASSOCIATION 14
INTEGRATING PERFORMANCE METRICS
Statement of Objectives
QASPMonitoring Procedures
PRSPerformance
Metrics
Performance measures must be identified for
requirements defined in the
SOO
Monitoring tools and techniques are outlined in
detail
NATIONAL CONTRACT MANAGEMENT ASSOCIATION 15
IMPORTANT QUALITY AND PERFORMANCE SOLICITATION REQUIREMENTS
Performance Oversight Board
Source Selection Evaluation
Quality Management Requirements
Ø FAR 46.4: Government Contract Quality Assurance
Ø ISO 9000:2001: Requirements for Quality Management Systems (other standards)
Ø ASQ Certifications: Six Sigma, Certified Quality Engineer(s)
Ø Program Management Institute: PMP/PgMP, Earned-Value Management
Ø Incorporate Perf. Mgmt and QA as part of Section M: Evaluation Factors for Award
Ø Require submission of the vendors Quality Management Plan (QMP) as part of Section L: Instructions to Offerors
Ø Include Quality SMEs as part of evaluation team
Ø Incorporate Perf. Mgmt and QA as part of Section M: Evaluation Factors for Award
Ø Require submission of the vendors Quality Management Plan (QMP) as part of Section L: Instructions to Offerors
Ø Include Quality SMEs as part of evaluation team
Ø Encourages discussion and coordination with service provider
Ø Establishes partnership between service provider and government
Ø Provides an objective perspective on performance
Ø Involves key stakeholders to accomplish agency objectives
Ø Encourages discussion and coordination with service provider
Ø Establishes partnership between service provider and government
Ø Provides an objective perspective on performance
Ø Involves key stakeholders to accomplish agency objectives
NATIONAL CONTRACT MANAGEMENT ASSOCIATION 16
WHAT IS A QUALITY MANAGEMENT PLAN?
• Documents how an organization structures its quality system that describes their:– Policies and procedures– Criteria for application– Roles and responsibilities– Tools used for measuring quality– Processes and corrective action
• Provides a "blueprint" on how vendors will implement specific quality assurance and quality control activities for a specific activity
• QMP level of detail should mirror the scope of the program
NATIONAL CONTRACT MANAGEMENT ASSOCIATION 17
EXECUTING AN INCENTIVE PROGRAM
• Clearly articulate incentive clauses in both the solicitation and resulting contact, which may include non-monetary alternatives– Financial impact– Frequency of assessment
• Develop auditable and verifiable data collection trail
• Determine clear payout (or reduction) timelines and prepare finances accordingly
• Be diligent!
NATIONAL CONTRACT MANAGEMENT ASSOCIATION 18
MANAGE & MEASURE
NATIONAL CONTRACT MANAGEMENT ASSOCIATION 19
WHAT IS THE PERFORMANCE REQUIREMENTS SUMMARY (PRS)
• Typically part of both the solicitation and the QASP that identifies the performance outcomes expected by the Government
• Can be transformed into a "scorecard" for overall vendor performance
• Determines the how well a vendor is performing
NATIONAL CONTRACT MANAGEMENT ASSOCIATION 20
SAMPLE PRS
Definition Unit of MeasureAPL
Sources of DataMeasure
Detailed description of
measure
Unit in which to express measure
Maximum deviation from
standard
Sources to seek for data collection
Wait time
Error RateReorders as a percent of total orders fulfilled
Percentage• 90% order
accuracy
• Order log in workload management system
Average time customers wait in line before cashier
greets them – weekly average
Minutes• 80% of customers
greeted in 5 minutes
• Wait time log
NATIONAL CONTRACT MANAGEMENT ASSOCIATION 21
WHAT IS THE QUALITY ASSURANCE SURVEILLANCE PLAN (QASP)
• Defines process for performance monitoring & evaluation by
the government
• Designed to implement and enforce the PRS
• Identifies the surveillance methods that will be used to apprise
the Government of the vendor performance
• The Government retains the right to modify the QASP
– Reasons may include: surveillance method, increase/decrease
degree of effort, and change procedures
• Identifies procedures to correct deficiencies observed in vendor
performance
NATIONAL CONTRACT MANAGEMENT ASSOCIATION 22
QASP IN ACTION
• The QASP provides the bridge between the work requirements in the SOO and actual performance by the vendor.
SOOSOO QASPQASP
NATIONAL CONTRACT MANAGEMENT ASSOCIATION 23
CORE QASP COMPONENTS
Obtain Data
Mo
nito
r Ve n
do
r Acti o
n P
la ns
Do
cum
e nt an
d A
naly ze D
a ta
Report Performance Information
QASP
Ensure data is:– Obtained– Recorded– Analyzed– Reported
NATIONAL CONTRACT MANAGEMENT ASSOCIATION 24
REPORTING PERFORMANCECONSIDER ENABLING TECHNOLOGIES
– Enabling technologies provide accountable, intuitive, automated processes to collect data, measure it, and evaluate vendor performance while:
• Reducing levels of effort
• Increasing standardization
• Enhancing the quality of work products
• Providing and storing data that can be tracked and audited
NATIONAL CONTRACT MANAGEMENT ASSOCIATION 25
REPORT PERFORMANCECOST AND PERFORMANCE MANAGEMENT TOOLS
NATIONAL CONTRACT MANAGEMENT ASSOCIATION 26
LESSONS LEARNED• Planning
– Incorporate quality and performance throughout the acquisition life-cycle– Assess the cost impacts of any quality and performance initiatives– Contract type may help determine the type and degree of required
oversight• Procurement
– Include quality and performance management frameworks as part of your solicitation documents (B, C, L, M)
– Government should determine performance measures to support program goals
– Consider the use of a performance oversight board• Manage and Measure
– The QASP should be developed on how the program will organize itself to monitor the selected vendor
– Modify the procedures as necessary, should be a living document– Provide quality and performance monitoring training to all stakeholders – Prioritize limited resources to accurately document vendor performance– Create mechanism to recommend changes to the PRS, may require a
contract modification– Automate Quality & Performance Management System(s)
NATIONAL CONTRACT MANAGEMENT ASSOCIATION 27
QUESTIONS AND ANSWERS
NATIONAL CONTRACT MANAGEMENT ASSOCIATION 28
nMarc BersonDirector T 703.637.2716C 202.236.1182E [email protected]
Eric HeffernanSenior Manager T 703.637.2769E [email protected]
Jennifer PalazzoloManager T 703.637.2846E [email protected]
GRANT THORNTON CONTACTS
Global Public Sector333 John Carlyle Street
Suite 400Alexandria, Virginia 22314
703.837.4400