AIR FORCE

AUDIT AGENCY

AUDIT REPORT

F2009-0005-FD3000

4 May 2009

AIR FORCE NUCLEAR

ROADMAP ASSESSMENT

Executive Summary

i

INTRODUCTION Two incidents along with other impairments caused the

Air Force to put high level management emphasis on the

nuclear mission. In 2006, critical nuclear related inter-

continental ballistic missile parts, labeled as helicopter

batteries, were mistakenly sent to Taiwan. In 2007, a

B-52 crew mistakenly flew six nuclear weapons from

Minot AFB, North Dakota to Barksdale AFB, Louisiana.

As a result of these incidents and subsequent investigations,

the Air Force established a Nuclear Task Force to develop a

nuclear enterprise roadmap. The Task Force published the

roadmap titled “Reinvigorating the Air Force Nuclear

Enterprise” (hereafter referred to as the Roadmap) on

24 October 2008. The Roadmap requires an assessment

method capable of measuring progress made to improve

the nuclear enterprise through action plans. This method

uses measures of performance and effectiveness (hereafter

referred to as measures) to determine how well the

Air Force is implementing the Roadmap’s objectives and to

determine if the Air Force is accomplishing a safe, secure,

effective, and reliable nuclear mission.

OBJECTIVES We performed this audit because the Secretary of the

Air Force made improving the nuclear enterprise the

Service’s highest priority and to provide immediate input

and recommendations for management actions during the

Roadmap development process. Our objective was to

evaluate whether the Air Force established an assessment

method to adequately evaluate the nuclear enterprise.

Specifically, we determined whether Air Force personnel

established (a) adequate measures to assess Roadmap

implementation status and facilitate accurate reporting to

nuclear oversight bodies, and (b) adequate data collection

internal controls. We plan to perform a follow-on audit to

validate the accuracy of data collected and reported

through the established Roadmap measures.

CONCLUSIONS Air Force nuclear enterprise assessment methodology,

while generally adequate, required strengthening.

Specifically, Air Force nuclear personnel:

Developed measures adequately addressing root

causes identified in the Roadmap and the measures

should facilitate reporting to responsible oversight

bodies. However, 11 (14 percent) of 79 measures

did not fully address Roadmap objectives.

Executive Summary

ii

Incomplete measures could lead to inaccurate

nuclear program assessments. (Tab A, page 1)

Did not develop adequate internal controls over

data collection to guide Major Commands

(MAJCOMs) and wings in their data collection

efforts. Without these controls, nuclear

assessment data is more susceptible to errors and

inconsistencies. (Tab B, page 5)

RECOMMENDATIONS From November 2008 through January 2009, Air Force

personnel corrected the deficient evaluation measures

identified by audit. Further, in February 2009, Assistant

Chief of Staff, Strategic Deterrence & Nuclear Integration

(AF/A10) officials established a data collection internal

control plan in response to audit concerns. These com-

pleted management actions properly address identified

deficiencies; therefore, this report contains no

recommendations requiring corrective action.

MANAGEMENT’S RESPONSE

Management officials agreed with the audit results and

corrective actions taken during audit were responsive to the

issues and addressed identified deficiencies.

CHARLES E. ATKINSON, JR.

Associate Director

Air and Space Operations Division

TONY M. AMES

Deputy Assistant Auditor General

Support and Personnel Audits

Table of Contents

Page

EXECUTIVE SUMMARY i

TAB

A Nuclear Roadmap Measures 1

B Internal Control Plan 5

APPENDIX

I Audit Scope and Prior Audit Coverage 9

II Locations Audited/Reports Issued 11

III Points of Contact 13

IV Final Report Distribution 15

Tab A Nuclear Roadmap Measures

1

BACKGROUND

The nuclear roadmap assessment process was designed to identify and measure progress

made toward reinvigorating the Air Force nuclear enterprise and meet strategic objectives

described in the Roadmap. Subject matter experts (SMEs) evaluate and validate

measures such as those created to assess how well the Air Force performs critical self-

assessments and root cause analysis (RCA). Then, Air Force senior leadership assign a

weight based on importance for each level of command hierarchy. The weighted

measures produce a score for each sub-objective and an overall performance score for

each unit which indicate progress achieved on Roadmap objectives. When implemented,

it should provide the Air Force the ability to communicate its nuclear enterprise status

and trends to DoD, national leaders, and nuclear oversight bodies.

Following validation and leadership approval of the measures, appropriate offices will

collect the required data and construct an initial or baseline assessment. Many planned

measures have never been collected and evaluated across the Air Force nuclear enter-

prise. The first assessment will serve as the baseline comparison for future assessments

to evaluate the progress or regression of any particular objective. Air Force leadership

will direct periodic assessments to maintain visibility on current performance and to track

trends.

AUDIT RESULTS 1 – NUCLEAR ROADMAP MEASURES

Condition. Overall, AF/A10 officials developed measures adequately addressing root

causes identified in the Roadmap and the measures should facilitate reporting to respon-

sible oversight bodies. However, we determined 11 (14 percent) of 79 measures did not

fully address Roadmap objectives (Exhibit 1). For example:

Inspection Findings. The inspection findings Roadmap measure did not identify

who was authorized to certify that RCA had been accomplished. Although SMEs

developed a measure requiring units to report the number of MAJCOM inspection

findings for which RCA was accomplished compared to the number of MAJCOM

inspection findings, the measure did not clearly state who was authorized to

certify that RCA had been accomplished. This created the potential for nuclear

units to respond differently to this measure because authority to certify RCA

completion was not assigned to a specific individual involved in the inspection

process. During the audit, SMEs adjusted this measure to specify commanders as

the sole authority to certify RCA completion. This change should provide senior

leaders with consistent measurement across all units and provide better assurance

that units are implementing Roadmap objectives.

Nuclear Infrastructure. Officials did not effectively plan to collect data needed to

compare the dollar value of Program Objective Memorandum (POM) disconnects

Tab A Nuclear Roadmap Measures

2

against the total Air Force budget. Although Air Force SMEs developed a

measure requiring units to report the “Number of POM disconnects (unfunded

requirements) in support of nuclear infrastructure,” they did not fully consider the

detailed data needed to resolve ambiguities and to give senior leaders a better

assessment that the projects programmed and planned for a given execution year.

During the audit, management changed this measure to “Number of projects

awarded in the execution year/Number of projects programmed in the execution

year.” With this added context within the measure, senior leadership will have

information needed to make necessary budget adjustments.

Cause. This condition occurred because AF/A10 personnel did not establish a process

to systemically analyze measures and associated data requirements against Roadmap

objectives to determine if the measures were adequate. Instead, AF/A10 officials invited

broad participation by SMEs from Air Staff and MAJCOM organizations with nuclear

missions to develop Roadmap measures and relied upon the expertise of personnel

participating in the process. A systemic process to evaluate the measures helps reduce

the risk that developed measures do not adequately address Roadmap objectives.

Impact. Evaluating measures and data used to report Roadmap progress is critical to

providing an assessment of the health of the nuclear enterprise to the Secretary of the

Air Force, Chief of Staff of the Air Force, and MAJCOM commanders who make up the

Nuclear Oversight Board. These assessments will be the basis for senior leadership

decisions on additional changes to the Air Force nuclear program.

Management Corrective Action. From November 2008 to January 2009, management

corrected the deficiencies noted for all 11 measures identified by audit. Further, AF/A10

officials developed a process for SMEs to analyze data requirements as the measures

were developed. Therefore, this report contains no recommendations addressing these

issues.

Management Comments. AF/A10 concurred with the findings in the report and stated

action taken during the course of the audit should correct the issues identified in the

report.

Evaluation of Management Comments. Management comments addressed the issues

raised in the finding and corrective actions taken during audit were responsive to the

issues and addressed identified deficiencies.

Exh

ibit 1

. Mea

sures R

equ

iring A

dju

stmen

t.

3

Original Measure Deficiency Identified Corrective Action* T

ab

A

Nu

cle

ar R

oa

dm

ap

Me

as

ure

s

Strategic Objective: The Air Force will implement effective processes for uncovering, analyzing, addressing and reviewing systemic weaknesses.

Number of MAJCOM inspection findings for which RCA

was accomplished/Number of MAJCOM inspection

findings.

Measure did not identify who was

authorized to report completed RCA.

Wing commanders were designated

as sole authority to certify RCA

completion.

Number of unit-identified discrepancies that include

RCA/Number of unit-identified discrepancies.

Measure did not specify who was

authorized to report RCA inclusion.

Wing commanders were specifically

identified as reporting authority.

Number root causes mitigated/Number root causes

identified.

Measure did not specify who was

authorized to report mitigated root cause.

Same corrective action as above.

Number corrective actions closed/Number corrective

actions identified.

Measure did not specify who was

authorized to close corrective action.

Same corrective action as above.

Number nuclear corrective actions and issues viewed in

shared database such as Joint Lessons Learned Integration

System (JLLIS)/Number nuclear corrective actions and

issues loaded into shared database such as JLLIS.

Measure did not evaluate whether units

were addressing systemic weaknesses.

Intent of measure was clarified to

users; to begin process of uncovering

systemic weaknesses.

Strategic Objective: The Air Force will sufficiently invest in the nuclear deterrence mission arena.

Number of POM disconnects (unfunded requirements) in

support of nuclear infrastructure.

Measure did not provide a clear assessment

of investment in the nuclear mission arena.

Changed measure to compare dollar

value of POM disconnects to

Air Force budget.

Number of execution year nuclear systems and equipment

funding adjustments.

Measure did not provide context/dollar

amount for adjustments.

Changed measure to report dollar

value of adjustments.

Strategic Objective: The Air Force will implement a comprehensive process for ensuring sustained advocacy, focus, and commitment.

Number of strategic communications affirming importance

of the nuclear mission delivered by Air Force General

Officers.

Measure did not identify what could be

reported as a strategic communication.

Clarified measure to only include

communication focused on nuclear

mission approved through public

affairs.

Number of Air Force General Officers who conduct visits

to nuclear capable units.

Measure did not specify the value of visits

or the office responsible for reporting the

visits.

Clarified intent measure will

establish baseline for Senior Leader

focus on units with nuclear missions;

reported by Protocol offices.

Strategic Objective: The Air Force will implement an end-to-end inter-related, systems, life-cycle nuclear enterprise methodology and discipline.

Number of Nuclear Weapon Related Material^ the

Air Force properly packed/Number of Nuclear Weapon

Related Material the Air Force packed.

Individual units did not have the visibility

of both ends of a transaction to accurately

report data.

Air Force Materiel Command

developed processes to report data

through Air Force-wide systems.

Number of Nuclear Weapon Related Material shipments

correctly delivered/Number of Nuclear Weapon Related

Material shipments.

Same deficiency as above. Same corrective action as above.

*AF/A10 officials are staffing the revised measures for approval because the coordination process identified areas of additional concern.

^This includes Nuclear Weapon Related Material as defined by the Office of the Secretary of Defense and the Air Force. The Air Force definition is broader and

includes more items.

4

Th

is Page In

tentio

na

lly L

eft Bla

nk

Tab B Internal Control Plan

5

BACKGROUND

Internal controls provide reasonable assurance that program objectives will be met. An

internal control plan (ICP) should provide systematic guidance on organization and

individual responsibilities, documentation requirements, data validity certification, key

word definitions, and standard data sources.

In an effort to streamline data collection, review, and reporting processes, the Air Force

planned to use a web-based continuous process improvement management tool called

the Nuclear Enterprise Management Tool to gather required data. A Roadmap ICP is

necessary to provide a consistent set of management controls designed to provide an

unbroken chain of accountability for the web-based nuclear enterprise’s assessment

Roadmap process. The goal is to ensure accuracy, completeness, and integrity of all

information capable of measuring progress made to improve the Air Force nuclear

enterprise through Roadmap action plans. Because Air Force nuclear operations are a

high visibility area, data accuracy, completeness, and integrity are imperative to meet

Roadmap objectives.

AUDIT RESULTS 2 – INTERNAL CONTROL PLAN

Condition. AF/A10 personnel did not develop internal controls necessary to guide

MAJCOMs and wings in consistently collecting data to support the Roadmap assessment

process. Specifically, AF/A10 did not develop an ICP outlining responsibilities for data

validity certification, key word definitions, standard data sources, and documentation

requirements. For example:

Data Validity Certification. The Air Force’s planned use of a web-based data

collection tool for data input posed a risk that unauthorized users could access

and manipulate nuclear measures data.1 An ICP would counter this risk by

limiting access to the Nuclear Enterprise Management Tool to only appoint

MAJCOM points of contact (or their alternate).2 Further, an ICP would require

these points of contact to certify that data collection is complete and was reviewed

for accuracy and completeness. This control provides a reasonable level of

confidence in the accuracy of the data collection process that would not have

existed prior to the ICP development.

1 Although the Nuclear Enterprise Management Tool is hosted on a common access card-enabled server,

the ICP provides guidance on who should be allowed a user account for this tool.

2 This control would also limit data entry capability to appointed points of contact.

Tab B Internal Control Plan

6

Key Word Definitions. Although AF/A10 personnel attempted to use the most

specific wording possible in developing the Roadmap measures, they had not

defined key terminology. An ICP provides Air Force management a tool to

provide in-depth definitions to units to facilitate consistent data gathering efforts.

To illustrate, one Roadmap measure required units to identify the number of

squadron, group, or wing commanders with nuclear experience. The SMEs

formulating the measures intended nuclear experience to be an individual who

had worked a minimum of 2 of the past 8 years in a nuclear unit. Without an ICP,

each nuclear unit could define the experience requirement differently. This would

create major inconsistencies in data reported by units and could misinform

Air Force senior leadership on the experience level of nuclear squadron, group,

and wing commanders. This control facilitates fact-based decisions related to

future training required to strengthen the nuclear enterprise.

Cause. This condition occurred because AF/A10 personnel had not considered following

proven data collection processes, like those followed for Base Realignment and Closure

data calls.3 Further, they did not consider that an ICP would be a useful management tool

during data collection.

Impact. Sufficiently detailing measures, required data, and the measures’ intent faci-

litates a consistent and accurate picture of the health of each Air Force nuclear unit

reporting to the Secretary of the Air Force, Chief of Staff of the Air Force, and MAJCOM

commanders who make up the Nuclear Oversight Board. Without an ICP, senior leader-

ship could direct unnecessary changes to the nuclear enterprise or, conversely, inaccu-

rately conclude the nuclear enterprise or segments of the nuclear enterprise are effective

based on inaccuracies and inconsistencies in the underlying data.

Management Corrective Action. During January 2009, AF/A10 developed and

coordinated a data collection ICP, based on the Air Force 2005 Base Realignment and

Closure ICP; outlining responsibilities, standard data sources, documentation require-

ments, data validity certification, and key word definitions. The ICP is currently in two-

digit coordination. Therefore, this report contains no recommendations addressing these

issues.

Management Comments. AF/A10 concurred with the findings in the report and stated

action taken during the course of the audit should correct the issues identified in the

report.

3 During the Base Realignment and Closure Process, Air Force managers developed evaluation criteria

and made data calls to obtain information necessary to evaluate which bases should be closed. A critical

element of this process was developing an internal control plan providing guidance to Air Force units on

issues such as key word definitions, standard data sources, and documentation requirements.

Tab B Internal Control Plan

7

Evaluation of Management Comments. Management comments addressed the issues

raised in the finding and corrective actions taken during audit were responsive to the

issues and addressed identified deficiencies.

8

This Page Intentionally Left Blank

Audit Scope and Prior Audit Coverage

9 Appendix I

AUDIT SCOPE

Audit Coverage. We performed this audit at HQ Air Force, AF/A10 (Appendix II). We

performed the review from October 2008 to January 2009 and issued a draft report to

management in March 2009.

Nuclear Roadmap Measures. To determine whether Air Force officials estab-

lished adequate measures to assess Roadmap implementation status and facilitate

accurate reporting to nuclear oversight bodies, we participated in the Roadmap

measures development meetings in November 2008 and January 2009, held

discussions with AF/A10 officials, analyzed the measures against Roadmap

objectives, and provided input on development of nuclear measures.

Internal Control Plan. To determine whether Air Force officials established

adequate controls over data collection, we reviewed control plans and audit

reports from the Air Force Base Realignment and Closure 2005 data collection

process and applied relevant lessons learned.

Sampling Methodology. We did not use sampling methods or Computer Assisted

Auditing Tools and Techniques to accomplish the objectives of this audit.

Data Reliability. We did not rely on computer-generated data to support conclusions in

this audit.

Auditing Standards. We accomplished this audit work in accordance with generally

accepted government auditing standards and, accordingly, included tests of key internal

controls as considered necessary. Specifically, we reviewed the internal controls

established for the data collection process used to assess Roadmap implementation.

PRIOR AUDIT COVERAGE

We did not identify any Air Force Audit Agency, Inspector General, DoD, or

Government Accountability Office reports issued within the past 5 years that addressed

the same or similar objectives as this audit.

10

This Page Intentionally Left Blank

Locations Audited/ Reports Issued

Installation-Level

Organization/Location Reports Issued

11 Appendix II

HQ United States Air Force

Assistant Chief of Staff, Strategic Deterrence and Nuclear Integration NONE

(AF/A10)

Pentagon Washington DC

12

This Page Intentionally Left Blank

Points of Contact

13 Appendix III

Air and Space Operations Division (AFAA/SPO)

Support and Personnel Audits

2509 Kennedy Circle

Brooks City-Base TX 78235-5116

Charles E. Atkinson, Jr., Associate Director

DSN 240-5080

Commercial (210) 536-5080

Terri L. Dilly, Program Manager

Bryce L. Barton, Audit Manager

We accomplished this audit under project number F2009-FD3000-0038.000.

14

This Page Intentionally Left Blank

Final Report Distribution

FREEDOM OF INFORMATION ACT

The disclosure/denial authority prescribed in AFPD 65-3 will make all decisions relative

to the release of this report to the public.

15 Appendix IV

SAF/OS

SAF/US

SAF/FM

SAF/IG

SAF/LL

SAF/PA

SAF/XC, AF/A6

AF/CC

AF/CV

AF/CVA

AF/A8

AF/A10

AF/RE

AU Library

DoD Comptroller

OMB

ACC

AETC

AFISRA

AFMA

AFMC

AFOSI

AFRC

AFSOC

AFSPC

AMC

ANG

PACAF

USAFA

USAFE

Units/Orgs Audited

16

This Page Intentionally Left Blank

To request copies of this report or to suggest audit topics

for future audits, contact the Operations Directorate at

(703) 696-7913 (DSN 426-7913) or E-mail to

reports@pentagon.af.mil. Certain government users may

download copies of audit reports from our home page at

www.afaa.hq.af.mil/. Finally, you may mail requests to:

Air Force Audit Agency

Operations Directorate

1126 Air Force Pentagon

Washington DC 20330-1126