internal audit report contract and work authorizations txdot office of internal audit – full scope...
TRANSCRIPT
Internal Audit Report
Professional Engineering Procurement Services (PEPS) Contract and Work Authorizations TxDOT Office of Internal Audit
PEPS Contract and Work Authorizations TxDOT Office of Internal Audit – Full Scope
February 2015 2
Objective To determine the effectiveness and efficiency of the Professional Engineering Procurement
Services (PEPS) consultant selection and work authorization process.
Opinion Based on the audit scope areas reviewed, control mechanisms are effective and
substantially address risk factors and exposures considered significant relative to impacting
operational execution and compliance. The organization's system of internal controls
provides reasonable assurance that most key goals and objectives will be achieved despite
significant control gap corrections and improvement opportunities identified. Control gap
corrections and improvement opportunities identified are likely to impact the achievement
of the organization's business/control objectives, but management has agreed to corrective
action plans to address the relevant risks within 6 months.
Overall Engagement Assessment Satisfactory
Findings
Title Control Design
Operating
Effectiveness Rating
Finding 1 Provider Evaluation and Selection
Process x x Needs Improvement
Finding 2 Statement of Qualification
Screening Process x x Needs Improvement
Finding 3 Work Authorization Assignment x x Satisfactory
Management concurs with the above findings and prepared management action plans to
address deficiencies.
Control Environment Both PEPS management and staff recognize the division’s importance and the overall
impact it has on the department’s core business functions. They are also aware of the risks
inherent in procuring TxDOT’s professional engineering services contracts and work
authorizations. Management strives to reduce risks and implement controls necessary to
ensure the success of the division and TxDOT. However, concerns over limited rationale and
documentation to support decisions in the screening and scoring process lead to an
increased need for additional management monitoring and oversight controls. While
exceptions were noted in the screening and scoring process, PEPS management has
designed a robust process that provides assurance that program objectives are being met
across the state. Exceptions identified in this report did not have an impact in PEPS
division’s selection of the most qualified providers, but likelihood of a negative impact could
increase if not corrected.
PEPS Contract and Work Authorizations TxDOT Office of Internal Audit – Full Scope
February 2015 3
Summary Results
Finding Scope Area Evidence
1 Selection
Process
Scoring
Testing included reviewing 18 score sheets from 10 Request
for Qualifications (RFQs) to evaluate if the evaluation
component of the score sheet was properly calculated. The
score sheets include an evaluation component, which generally
makes up 5% of the total score.
3 of 18 (17%) provider score sheets tested did not include
both the firm and project manager scores within the
Evaluation Score Average (ESA) calculation, as required.
These exceptions did not result in erroneous selection of
provider and were isolated to one service center.
Annual Prime Provider Evaluations
Prime provider evaluations are required to be performed at
least annually in the Consultant Certification Information
System (CCIS). These evaluations are then used to calculate a
portion (generally 5%) of the prime provider’s score. If an
evaluation is not completed in CCIS, a provider is given
maximum points in this category.
19 of 30 (63%) completed contracts tested from Fiscal
Year (FY) 2011 through FY 2013 did not have an annual
firm evaluation or evaluation performed at the completion
of the contract in CCIS, as required. Without an evaluation
in CCIS, PEPS is unable to determine the adequacy of the
provider’s past performance and by default, would be given
maximum points in this category if the provider applied for
a new contract.
2 Selection
Process
Statement of Qualifications (SOQ) Screening
Testing included reviewing 10 Request for Qualifications
(RFQs) and 367 screening checklists used to determine pre-
qualification of each provider to be a consultant for a PEPS
contract.
25 of 367 (7%) SOQ screening checklists did not include
final determinations (i.e. “meets” or “does not meet”
decision).
o 13 of 342 (4%) SOQ screening checklists with a final
determination had no supporting evidence for that
decision and the checklist was left blank.
Other exceptions identified, which related to
inconsistencies in the completion of the SOQ screening
checklist, primarily stemmed from two of seven PEPS
service centers.
PEPS Contract and Work Authorizations TxDOT Office of Internal Audit – Full Scope
February 2015 4
3 Work
Authorizations
Multiple Work Authorization Assignments
Individual work authorizations under $1 million (M) had
characteristics that in aggregate would require approval higher
than a Division Director. Work authorizations under $1M
completed by the districts/divisions (without additional
signature authorization) made up 40% of the entire amount of
work authorizations.
13 of 826 (2%) work authorizations reviewed were issued
by four divisions to their same vendors, under the same
contracts, either on the same day or within one week of
each other, that totaled over $1M. Work authorizations
over $1M require approval from the Chief Procurement
Officer or TxDOT Administration. The work authorizations, in
aggregate, issued by each of these divisions totaled over
$1M which would exceed the delegation of authority limit.
This was referred to the Office of Compliance, Ethics, and
Investigations (CEI) for further review, and no inappropriate
activities were identified.
Audit Scope The scope of the audit work focused on the organizational environment, screening and
selection process for contractors, and work authorization controls. These activities were
reviewed to assess the effectiveness and efficiency of the contracting and work
authorization processes.
The scope for audit fieldwork conducted included Request for Qualifications (RFQs) and
resulting contracts executed from October 1, 2013 through July 7, 2014; tests were
performed on sample periods and dollar amounts of work authorizations initiated.
Evaluations were tested from Fiscal Year (FY) 2011 to FY 2013.
The audit was performed by Alma Alvarez, Jessica Esqueda, David Kossa, and Karen Henry
(Engagement Lead). The audit was conducted during the period from June 11, 2014 to
October 8, 2014.
Methodology The methodologies used to complete the objectives of this audit included:
Reviewing TxDOT internal documents, including policy and procedure manuals,
organization charts, process maps, management reports, and training records
Reviewing state codes and manuals
Inquiring and interviewing of key personnel
Reviewing prior audit reports from TxDOT’s Office of Internal Audit and the State
Auditor’s Office
Evaluating PEPS selection process and work authorizations process
Conducting data analysis, obtaining supporting documentation, and sample testing
Reviewing communication and management philosophy of overall organizational tone
Performing an overall risk assessment of the professional engineering procurement
services and work authorization process functions
PEPS Contract and Work Authorizations TxDOT Office of Internal Audit – Full Scope
February 2015 5
Reviewing screening determination documents and if available, selection rationale to
ensure appropriateness
These procedures were applied as necessary to perform the audit fieldwork.
Background This report is prepared for the Texas Transportation Commission and for the administration
and management of TxDOT. The report presents the results of the Professional Engineering
Procurement Services (PEPS) Contract and Work Authorizations audit which was conducted
as part of the Fiscal Year 2014 Audit Plan.
PEPS was formed in October 2013 as a redesigned program for procuring engineering,
architectural, and surveying services under consultant contracts. PEPS was redesigned to
reduce the time to process contracts and provide quality selection outcomes. Prior to the
PEPS formation, the procurement for professional engineering services was under the
consultant contracting program.
PEPS has six divisional service centers in the TxDOT metro and urban districts (Austin,
Dallas, El Paso, Houston, Fort Worth, and San Antonio) and a central office that supports the
additional 19 TxDOT districts and the divisions. PEPS oversees the selection process for
engineering, surveying, and architectural contracts to ensure providers are selected based
solely on their qualifications. PEPS executed 137 contracts totaling over $343 million from
inception (October 2013) through August 2014.
There are several processes that encompass the selection process. Once it is determined
that there is a need for a PEPS contract, a Consultant Selection Team (CST) is assembled.
The team initiates the provider selection process by assembling and submitting “Intent to
Contract”. After developing and posting a Request for Qualifications (RFQ) to advertise the
contract, responses from interested providers called Statements of Qualifications (SOQs) are
collected and evaluated by the CST. Depending on the selection process, providers are
either selected or advanced to a short list for an interview, final evaluation, and selection.
PEPS is also responsible for providing general oversight, guidance, and support to districts
and divisions for work authorizations.
We conducted this performance audit in accordance with Generally Accepted Government
Auditing Standards and in conformance with the International Standards for the
Professional Practice of Internal Auditing. Those standards require that we plan and perform
the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objectives. Recommendations to mitigate risks
identified were provided to management during the engagement to assist in the formulation
of the management action plans included in this report. We believe that the evidence
obtained provides a reasonable basis for our findings and conclusions based on our audit
objectives. The Office of Internal Audit transitioned to Committee of Sponsoring
Organizations of the Treadway Commission (COSO) Internal Control – Integrated Framework
version 2013 in December 2013.
PEPS Contract and Work Authorizations TxDOT Office of Internal Audit – Full Scope
February 2015 6
A defined set of control objectives was utilized to focus on operational and compliance goals
for the identified scope areas. Our audit opinion is an assessment of the health of the
overall control environment based on (1) the effectiveness of the enterprise risk
management activities throughout the audit period, and (2) the degree to which the defined
control objectives were being met. Our audit opinion is not a guarantee against operational
sub-optimization or non-compliance, particularly in areas not included in the scope of this
audit.
Best Practices PEPS has established contracting performance measures that evaluate the time it takes for
a contract to be signed starting with the time the solicitation occurs (e. g., this includes the
review of the SOQs, RFQs, and scoring of the provider). PEPS personnel review the
performance measures on a frequent basis and monitor their performance. When internal
performance measures are not being met, PEPS management, including the service center
managers, Administration and Finance Manager, Center of Excellence Manager, and the
PEPS Director review and discuss what has caused the performance variance. The
frequency and type of reviews performed are detailed and include planning, coordination,
strategic considerations, and problem solving. The reviews are performed to help ensure
efficiency in the processes that have been established.
PEPS Contract and Work Authorizations TxDOT Office of Internal Audit – Full Scope
February 2015 7
Detailed Findings and Management Action Plans (MAP)
Finding No. 1: Provider Evaluation and Selection Process
Condition
The processes in place for provider evaluations and selections are not consistently applied.
The Evaluation Score Average (ESA) used to calculate a portion (generally 5%) of the prime
providers’ score was not being calculated correctly by one of the service centers. In addition,
annual evaluations of providers were not always completed.
Rationale to support scoring and score refinement on deliberations were not included on
score sheets.
Effect/Potential Impact
Inconsistent application of evaluations and limited or missing rationale to support decisions
in the scoring process can raise concerns over the objectivity, consistency, and accuracy of
TxDOT’s selection process.
Criteria
Title 43, Texas Administrative Code, Part 1, Section 9.41(d)(1), states that evaluations are to
be conducted during the ongoing activity and at the completion of the contract. In addition,
the TxDOT Provider Evaluation Guide and CTR615 (formerly DES615)–Consultant
Management and Administration Module 5 of the Consultant Selection Team (CST), whose
member’s required training, state that the minimum time frame for performing provider
evaluations is annually.
The PEPS Contracting: Selection Process Manual also requires Consultant Selection Team
(CST) members to take TxDOT’s CTR615 (formerly DES615)–Consultant Management and
Administration training course to understand their roles and responsibilities.
The Professional Engineering Procurement Services (PEPS) Contracting: Selection Process
Manual includes policies and procedures for application of ESA in the scoring process which
requires the use of project manager score and firm score in the calculation.
The State of Texas Contract Management Guide Appendix 5 – Evaluation Team Guidelines
states that evaluators should write comments during the de-briefing process, especially if
the score is unusually low or high.
Cause
There is no automated process in the Consultant Certification Information System (CCIS) to
track evaluation status or to ensure evaluation completion. There is no review process or
monitoring controls to ensure rationale supports the provider selection or that selection files
are complete and include all key components to ensure compliance requirements are met
and that ESA scores are calculated correctly. In addition, not everyone involved in the
selection process is attending required training that provides expectations and guidance on
how to conduct the selection process.
PEPS Contract and Work Authorizations TxDOT Office of Internal Audit – Full Scope
February 2015 8
Evidence
The evidence obtained in the review included:
Scoring
Testing included reviewing the score sheets for 10 Request for Qualifications (RFQs). These
score sheets provide the support for why a provider is selected for a PEPS contract:
10 of 10 (100%) RFQs tested did not have rationale included with score sheets to
support scoring and score refinement on deliberations
o Auditors could not determine if the objectivity and accuracy of selections were
impaired
18 score sheets were selected from the 10 RFQs to determine if the evaluation
component of the scoring sheet was properly calculated:
o 3 of 18 (17%) provider score sheets tested did not include both the firm and
project manager scores within the ESA calculation as required;
These exceptions did not result in erroneous selection of providers
Annual Prime Provider Evaluations
Prime provider evaluations are required to be performed at least annually in CCIS. These
evaluations are used to calculate a portion (generally 5%) of the prime provider’s score. If an
evaluation is not completed in CCIS, a provider is given maximum points (generally 5%) in
this category.
A sample of 30 completed contracts was selected and tested to review if these contracts
had the required annual prime provider evaluations.
19 of 30 (63%) completed contracts tested from Fiscal Year (FY) 2011 through FY
2013 did not have an annual firm evaluation or evaluation performed at the
completion of the contract in CCIS, as required. Without an evaluation in CCIS, PEPS
is unable to determine adequacy of a provider’s past performance and the provider is
given maximum points in this category when applying for a new contract
Training
Training was not taken by all those that participated in the selection process:
10 of 36 (28%) Consultant Selection Team (CST) members reviewed did not attend
required CTR615 (formerly DES615)–Consultant Management and Administration
training in a 5 year period
Management Action Plan (MAP):
MAP Owners:
Lorena Echeverria de Misi, PEPS Division Director
Lucio Vasquez, PEPS Deputy Division Director
Camille Thomason, PEPS Center of Excellence Manager
MAP 1.1:
Management agrees with the findings and is committed to ensuring the provider evaluation
process used in the selection process is performed in accordance with the Professional
Engineering Procurement Services (PEPS) procedures. The following tasks will be performed
during this period:
PEPS Contract and Work Authorizations TxDOT Office of Internal Audit – Full Scope
February 2015 9
A central server is currently under development to provide the division with one main
location to store and retrieve documents. At this time, PEPS Service Centers are
working on individual servers as per the previous Region model.
PEPS will revisit the procedures for the scoring process in the PEPS Contracting:
Selection Process Manual to ensure it fits within best-practices in consultant
selection, making revisions, if needed.
PEPS will work with NTT Data to create an automated mechanism to alert appropriate
individuals on actions required for performing provider evaluations.
On verifying whether the Consultant Selection Team (CST) members have taken
required courses: a new procedure will be included in the PEPS Contracting:
Selection Process Manual to have contract specialists perform verification of CST
member’s training through the Professional Services Contract Administration
Management System (PS-CAMS).
MAP 1.2:
PEPS will finalize the PEPS Contracting: Contract Management and Administration Manual.
Language concerning the provider evaluation procedure will be included.
Completion Date:
May 15, 2015
PEPS Contract and Work Authorizations TxDOT Office of Internal Audit – Full Scope
February 2015 10
Finding No. 2: Statement of Qualification (SOQ) Screening Process
Condition
Statement of Qualification (SOQ) screening process checklists, which are used in
determining if consultants are qualified, were incomplete and completed by more than one
person. In addition, signatures indicating review and approval of the SOQ screening
checklists were not consistently retained, documented, or supported.
Effect/Potential Impact
Inconsistencies in the completion of SOQ screening checklists and limited or missing
rationale to support decisions in the screening process can raise concerns over the
objectivity of TxDOT’s selection process. This can also affect the overall outcome and the
most qualified consultant may not be selected.
Criteria
The PEPS Contracting: Selection Process Manual Chapter 8 Section 4 states that one person
should conduct all screenings to help ensure consistency, SOQs are to be screened using
the SOQ screening checklist, and no checklist items may be deleted.
In addition, the TxDOT Solicitation Package (SP) Screening Guidance and Instructions
Checklist for final determination requires:
Each item in the checklist to be marked whether or not the SP meets the
requirement
If an item is marked as “Does Not Meet,” the situation is to be described in the
comments section
If an item is not applicable to the solicitation, mark “N/A”
The PEPS representative must review, sign, and date each completed checklist
The PEPS Contracting: Selection Process Manual also requires Consultant Selection Team
(CST) members to take TxDOT’s CTR615 (formerly DES615)–Consultant Management and
Administration training course to understand their roles and responsibilities.
Cause
There is no review process or monitoring controls to ensure supporting evidence supports
decisions and finalization of screening determinations is complete and includes all key
components to ensure compliance requirements are met. In addition, not everyone involved
in the screening selection process is attending required training.
Evidence
The evidence obtained in the review included:
Screening
Testing included reviewing 10 Request for Qualifications (RFQs) and 367 screening
checklists from those RFQs to determine pre-qualification of each provider to be a
consultant for a PEPS contract:
7 of 10 (70%) RFQs had more than one person conducting the SOQ screenings
PEPS Contract and Work Authorizations TxDOT Office of Internal Audit – Full Scope
February 2015 11
132 of 367 (36%) SOQ screening checklists did not have signature approval
documented
106 of 367 (29%) SOQ screening checklists did not indicate if the provider met the
indicated requirement even though comments were provided
25 of 367 (7%) SOQ screening checklists had missing final determinations (“meets”
or “does not meet” decision)
o 13 of 342 (4%) SOQ screening checklists, with a final determination, had no
supporting evidence for that decision and the checklist was left blank. All 13 of
these checklists were performed by the same individual
From the 276 total SOQ screening checklist exceptions noted in the bullet points above, 111
(40%) exceptions were from one service center, and 98 (36%) exceptions were from another
service center.
Auditors did not identify any providers being selected that were not qualified.
Training
Training was not taken by all those that participated in the screening process:
10 of 36 (28%) Consultant Selection Team (CST) members reviewed did not attend
required CTR615 (formerly DES615)–Consultant Management and Administration
training in a 5 year period
Management Action Plan (MAP):
MAP Owners:
Lorena Echeverria de Misi, PEPS Division Director
Lucio Vasquez, PEPS Deputy Division Director
Camille Thomason, PEPS Center of Excellence Manager
MAP 2.1:
Management agrees with the findings and is committed to ensuring the SOQ screening
process is performed in accordance with the PEPS Contracting: Selection Process Manual.
The following tasks will be performed during this period:
PEPS will revisit the procedures for the SOQ screening in the PEPS Contracting:
Selection Process Manual to ensure it fits within best-practices in consultant
selection, making revisions, if needed.
PEPS Management will provide guidance to division to promote adherence to the
division’s manuals.
On verifying whether CST members have taken required courses: a new procedure
will be included in the PEPS Contracting: Selection Process Manual to have contract
specialists perform verification of CST member’s training through the Professional
Services Contract Administration Management System (PS-CAMS).
Completion Date:
May 15, 2015
PEPS Contract and Work Authorizations TxDOT Office of Internal Audit – Full Scope
February 2015 12
Finding No. 3: Work Authorization Assignment
Condition
Individual work authorizations under $1 million (M) have characteristics that, in aggregate,
would require approval higher than a Division Director. Four divisions individually issued
work authorizations to the same vendors, under the same contract, on the same day or
within one week of each other that totaled over $1M which would exceed the delegation of
authority limit and require additional administration approval.
Effect/Potential Impact
Work authorizations may not have the necessary review and oversight to ensure work scope
and amounts are appropriate and in accordance with signature authority roles.
Criteria
Based on the Delegations of Signature Authority, Division Directors and District Engineers
can execute work authorizations of $1M or less. Work authorizations over $1M require
additional approval by the Chief Procurement Officer or TxDOT Administration.
Cause
Monitoring controls do not exist to detect whether Districts/Divisions/Offices (D/D/O)
are issuing multiple work authorizations that are under the $1M threshold to
circumvent current rules.
Training on how to properly utilize work authorizations is not provided to D/D/O staff
that issue work authorizations under $1M.
The Professional Engineering Procurement Service (PEPS) Division Contracting:
Contract Management and Administration Manual, which includes work
authorizations, is still in development.
Evidence
The evidence obtained in the review included:
Multiple Work Authorization Assignments
13 of 826 (2%) work authorizations analyzed were issued by four divisions to their same
vendors, under the same contracts, either on the same day or within one week of each other
that totaled over $1M. This was referred to the Office of Compliance, Ethics, and
Investigations (CEI) for further review, and no inappropriate activities were identified:
On 3/25/2014, 6 work authorizations were executed by one division to one provider
under the same contract totaling over $4.2M
On 3/28/2014 and 3/31/2014, 3 work authorizations were executed by one
division to one provider under the same contract totaling over $1.7M
On 2/14/2014, 2 work authorizations were executed by one division to one provider
under the same contract totaling over $1.1M
On 10/1/2013, 2 work authorizations were executed by one division to one provider
under the same contract totaling over $1.2M
PEPS Contract and Work Authorizations TxDOT Office of Internal Audit – Full Scope
February 2015 13
Management Action Plan (MAP):
MAP Owners:
Lorena Echeverria de Misi, PEPS Division Director
Lucio Vasquez, PEPS Deputy Division Director
Camille Thomason, PEPS Center of Excellence Manager
MAP 3.1:
PEPS will communicate the procedure for work authorization prioritization and signature
authority compliance rules to the Districts/Divisions/Offices (D/D/O).
MAP 3.2:
PEPS will include signature authority reviews by service center managers as part of
the current review process that is performed over work authorizations.
PEPS will finalize the PEPS Contracting: Contract Management and Administration
Manual to provide Districts/Divisions/Offices (D/D/O) guidance on best practices for
work authorization assignment and development.
Completion Date:
March 15, 2015
PEPS Contract and Work Authorizations TxDOT Office of Internal Audit – Full Scope
February 2015 14
Observations and Recommendations
Audit Observation (a): Records Management
Missing records and documentation were noted throughout the audit testing (highlighting
areas for improvement over records management) including the following:
10 of 10 (100%) Request for Qualifications (RFQs) reviewed did not have deliberation
meeting minutes/notes available for review
6 of 9 (67%) Consultant Selection Team (CST) members interviewed indicated that
the records retention requirements were not communicated to them
4 of 27 (15%) CST members indicated having a second job (outside TxDOT); however,
there were no Outside Employment/Professional Activity Request Forms (Form
2490s) on file for any of these employees for Professional Engineering Procurement
Services (PEPS) Division to assess if a potential conflict of interest is present with
their TxDOT evaluation responsibilities.
1 of 10 (10%) work authorizations tested was entered into the wrong contract
number in the Professional Services Contract Administration Management System
(PS-CAMS). In addition, the system does not retain an audit trail to track movement
or deletion of work authorizations
Effect/Potential Impact
Not communicating record retention requirements to all staff involved in the screening and
scoring process, inability to provide documents subject to open records request, and not
obtaining CST member conflict of interest information could leave TxDOT susceptible to
compliance issues and can raise concerns over the objectivity of TxDOT’s selection process.
Work authorizations entered under the wrong contract can cause encumbrances against
other Divisions/Districts/Office (D/D/O) contracts preventing them from issuing work
authorizations needed on the same contract. In addition, allowing movement of work
authorizations between contracts or deletion from PS-CAMS without an audit trail to track
movement creates a risk for potential fraud, waste, or abuse.
Audit Recommendation
Professional Engineering Procurement Services (PEPS) Division should:
Implement a time-frame requirement to have selection files completed and closed
with all files uploaded to a central controlled location
Establish a file close-out checklist to be completed by the PEPS representative (lead)
Periodic management review should be implemented and completed (with a required
completed review checklist) to ensure all required documents are being retained and
available upon request
As part of the CST team kickoff meeting, Form 2490 should be obtained/retained for
all employees having a second job (outside TxDOT), and records retention
requirements should be communicated to all staff serving on the CST team. These
should be included as part of the planning checklist the PEPS representative (lead)
should complete in the planning phase prior to the screening process
Implement system programming edit checks or additional monitoring and review to
prevent work authorizations being entered under the wrong contract or
moved/deleted without a tracking mechanism
PEPS Contract and Work Authorizations TxDOT Office of Internal Audit – Full Scope
February 2015 15
Summary Results Based on Enterprise Risk Management Framework
Closing Comments The results of this audit were discussed with the Professional Engineering Procurement
Services (PEPS) Division Director and Deputy Director on October 27, 2014. We appreciate
the assistance and cooperation received from the PEPS, Human Resources, and Contract
Services Divisions; Office of Compliance, Ethics, and Investigations; and personnel in the
districts contacted during this audit.