1

DoD Systems and Software Engineering

Taking it to the Next Level

25 Oct 2006

Mark D. Schaeffer

Director, Systems and Software Engineering

Office of the Deputy Under Secretary of Defense (A&T)

2

Office of the Under Secretary of DefenseAcquisition, Technology and Logistics

USD, AcquisitionTechnology & Logistics

DUSD, Acquisition &Technology

Technical Advisor,Interoperability

Dir, Systems ofSystems Management

Dir, Systems andSoftware Engineering

Dir, PortfolioSystems Acquisition

IndustrialPrograms

Defense Procurementand Acquisition Policy

Small BusinessPrograms

Defense ContractManagement Agency

Defense AcquisitionUniversity

Flatter, Leaner, Empowered!

3

Systems and Software EngineeringMission Statement

Shape acquisition solutions and promote early technical planning

Promote the application of sound systems and software

engineering, developmental test and evaluation, and related

technical disciplines across the Department's acquisition community

and programs

Raise awareness of the importance of effective systems engineering

and drive the state-of-the-practice into program planning and

execution

Establish policy, guidance, education and training in collaboration

with academia, industry, and government communities

Provide technical insight to program managers and leadership to

support decision making

Evolving System Engineering Challenges

4

Systems Engineering Revitalization Framework

Guidance E&T

Policy

Program

Support

Acquisition

Community

SE and T&E

CommunitiesAcademic

Community

Industry

Associations

Driving Technical Excellence into Programs!

5

Systems EngineeringRevitalization Effort

Issued Department-wide Systems Engineering (SE) policy

Issued guidance on SE, T&E, and SE Plans (SEPs)

Instituted system-level assessments in support of DAB, OIPT, DAES, and in support of programs

Established SE Forum to ensure senior-level focus within DoD

Integrating DT&E with SE policy and assessment functions--focused on effective, early engagement of both

Instituting a renewed emphasis on modeling and simulation in acquisition

Working with Defense Acquisition University to revise curricula (SPRDE, T&E, PQM, LOG, PM, ACQ, FM, CONT)

Leveraging close working relationships with industry and academia

Necessary but not sufficient!

6

Vision for Systems Engineering and Software

Competencies Improved Delivered Product Suite

• Courseware• Policy/Guidance• Program Support

methods Elevated Stature Raised Awareness Positive Influence

World class leadership Broaden to Software Engineering,

System Assurance, Complex Systems-of- Systems

Responsive and agile, proactive to changing customer needs

Focused technical assistance, guidance, and workforce education and training

Systems and Software Engineering

Centers of Excellence

. . . the Technical Foundation that Enables Acquisition Excellence

Systems Engineering

Revitalization

7

Why the Focus on Software…

Research investment has been static or declining• DARPA computer science R&D funding 50% ↓ (’01 -

’04, universities) Requirements growth 10X (% functionality) ’60s -’00s Need vs. skilled/clearable workforce - gaps increasing President’s Information Technology Advisory Committee

Report, February 2005• Identifies SW as “major vulnerability”• Recommends priority attention: “Secure Software

Engineering and Software Assurance” and “Metrics, Benchmarks, and Best Practices”

Cost, schedule and performance issues

Software is an increasingly, important factor

8

Systems and Software Engineering inPrograms Reduces Costly Mistakes

$222.8B RDT&E FYDP**

**SAR data for MAIS and MDAP programs under OSD Systems Engineering Oversight

33% historical RDT&E Cost Growth

-5%

0%

5%

10%

15%

20%

25%

30%

35% 137 Systems

Per

cen

t C

ost

Gro

wth Other Mistakes

ILS Factors; Spares & Support

Schedule Slips/Management Factors

Engineering/Test/Development

Production Assumptions & Estimation

RDT&E MistakesUnder estimating engineering effort is Major source of

error

% of RDT&E Total

(CG

)

Yields aPotential

Applied to

$73.52B RDT&E Cost Growth FYDP

Source: DoD Cost Avoidance Study (CAIG) 10 year ongoing

* SSE positive impact on just 1/3 of RDT&E mistakes (11%)

$24.51B RDT&E Cost Avoidance FYDP

Yields aPotential

1/3 *SSEimpact

9

Director, Systems &Software Engineering

Mark Schaeffer SES

Deputy DirectorEnterprise Development

Bob Skalamera SES

Deputy DirectorDevelopmental Test

& Evaluation

Chris DiPetto SES

Deputy DirectorSoftware Engineering &

System Assurance

Mark Schaeffer (Acting) SES

Deputy DirectorAssessments & Support

Dave Castellano SES

CORE COMPETENCIES

• SE Policy• SE Guidance

• SE in Defense Acquisition Guidebook

• Technical Planning• Risk Management• Reliability &

Maintainability• Contracting for SE• SoS SE Guide

• SE Education and Training• DAU SE Curriculum• SPRDE Certification Rqmt

• Corrosion• R-TOC• Value Engineering

CORE COMPETENCIES

• DT&E Policy• DT&E Guidance

• T&E in Defense Acquisition Guidebook

• TEMP Development Process

• DT&E Education and Training

• DAU DT&E Curriculum• DT&E Certification Rqmt

• Joint Testing, Capabilities & Infrastructure

• Targets Oversight• Acq Modeling & Simulation• Energy• DSOC/Acq Tech Task Force

CORE COMPETENCIES

• SWE and SA Policy• SWE and SA Guidance

• SoS, SA Guides• SWE and SA Education and

Training• DAU SW Acq Curriculum• Continuous Learning

Modules for SWE, SoS, SA• Software Engineering

• Acquisition Support• Software Engineering

Institute (SEI)• Process Improvement

• CMMI Sponsor• DoD/National Software

Investment Strategy

CORE COMPETENCIES

• Support of ACAT I and Other Special Interest Programs (MDAP, MAIS)

• Assessment Methodology (Program Support Reviews - PSRs)

• T&E Oversight and Assessment of Operational Test Readiness (AOTR)

• Systems Engineering and Developmental Test Planning and Support

• Lean/6-Sigma Training/Cert

State of Systems EngineeringDirector, Systems & Software Engineering

Acquisition program excellence through sound systems and software engineering

Est. Aug 06

10

DoD Software Center of ExcellenceOrganizational Tenets

Support Acquisition Success

• Ensure effective and efficient software solutions across the acquisition spectrum of systems, SoS and capability portfolios

Improve the State-of-the-Practice of Software Engineering

• Advocate and lead software initiatives to improve the state-of-the-practices through transition of tools, techniques, etc.

Provide Leadership, Outreach and Advocacy

• Implement at Department and National levels, a strategic plan for meeting Defense software requirements

Foster Software Resources to meet DoD needs

• Enable the US and global capability to meet Department software needs, in an assured and responsive manner

Promote World-Class Leadership for Defense Software Engineering

Promote World-Class Leadership for Defense Software Engineering

11

What We’re Finding

and

What We’re Hearing

12

Top 10 Emerging Systemic Issues

Major contributors to poor program performance

1. Management • IPT roles, responsibilities, authority, poor communication• Inexperienced staff, lack of technical expertise

2. Requirements • Creep/stability• Tangible, measurable, testable

3. Systems Engineering • Lack of a rigorous approach, technical expertise• Process compliance

4. Staffing • Inadequate Government program office staff

5. Reliability • Ambitious growth curves, unrealistic requirements• Inadequate “test time” for statistical calculations

6. Acquisition Strategy • Competing budget priorities, schedule-driven• Contracting issues, poor technical assumptions

7. Schedule • Realism, compression

8. Test Planning • Breadth, depth, resources

9. Software • Architecture, design/development discipline• Staffing/skill levels, organizational competency (process)

10. Maintainability/Logistics • Sustainment costs not fully considered (short-sighted)• Supportability considerations traded

13

Top 5 SE IssuesNDIA Workshop – July 2006

1. Key systems engineering practices known to be effective are not consistently applied across all phases of the program life cycle.

2. Insufficient systems engineering is applied early in the program life cycle, compromising the foundation for initial requirements and architecture development.

3. Requirements are not always well-managed, including the effective translation from capabilities statements into executable requirements to achieve successful acquisition programs.

4. The quantity and quality of systems engineering expertise is insufficient to meet the demands of the government and the defense industry.

5. Collaborative environments, including SE tools, are inadequate to effectively execute SE at the joint capability, system of systems (SoS), and system levels.

NDIA Systems Engineering Division - Top 5 SE IssuesJuly 26-27, 2006

14

Top 7 Software IssuesNDIA/DoD Workshop – August 2006

1. The impact of requirements upon software is not consistently quantified and managed in development or sustainment.

2. Fundamental system engineering decisions are made without full participation of software engineering.

3. Software life-cycle planning and management by acquirers and suppliers is ineffective.

4. The quantity and quality of software engineering expertise is insufficient to meet the demands of government and the defense industry.

5. Traditional software verification techniques are costly and ineffective for dealing with the scale and complexity of modern systems.

6. There is a failure to assure correct, predictable, safe, secure execution of complex software in distributed environments.

7. Inadequate attention is given to total lifecycle issues for COTS/NDI impacts on lifecycle cost and risk.

NDIA Systems Engineering Division - Top SW IssuesAugust 24-25, 2006

15

Defense Software Strategy Summit

OSD led Summit co-hosted by NDIA Oct 18-19, 2006• Build on the Top 7 SW Issue Workshop Findings

Nearly 100 participants from across DoD, industry and academia

Identified numerous issues, barriers and actionable recommendations across:• Software acquisition and sustainment• Software policy• Human capital• Software engineering practices

Next steps:• Final report forthcoming• Prioritize work, develop action plans, and move out

16

Challenges Remain

Implementing a DoD vision and strategy for software

Component and Industry adoption and effective implementation of sound SE practices as early as possible in the system life cycle

SE Working Integrated Product Teams (SE WIPTs)

Retention and development of technical acumen in an aging and shrinking acquisition workforce

Meeting all requests for technical support to programs

SE support to Acquisition Initiatives stemming from the QDR

Continue to evolve “high visibility” initiatives:

- Energy - System-of-Systems

- CMMI - Modeling & Simulation

- Defense Safety - System Assurance Oversight Council

17

DoDCenters ofExcellence

How do we get there?

19

CMMI: New Release and Next Steps

Issues: Integrity of CMMI appraisals Misperception and misuse of the CMMI by acquirersActions: Implemented changes to the CMMI v1.2 product suite to ensure:

• Integrity of appraisals• Quality of the product suite• Education of acquirers• Opportunities for streamlining where appropriate

Developing a CMMI model for Acquirer process improvement• Partnership with General Motors• Stakeholders cross DoD, Govt Agencies and Industry

Writing a CMMI guidebook• Help acquirers understand what CMMI is and is not

Conducting study of actual process implementation post-Level 5

20

System Assurance

Issues:

Vulnerability of our systems to malicious tampering or access

Numerous assurance, protection and safety initiatives that are not aligned

Actions:

Developing a comprehensive System Assurance strategy

Promoting nationwide collaboration

Identifying standards activities to address system vulnerabilities

Developing a Handbook for Engineering System Assurance

• Guidance for PMs and Engineers on how Systems Engineering practice can be applied to mitigate system vulnerability to malicious control/tampering

21

System-of-Systems Engineering

DUSD (A&T) directed OSD-led effort to develop and publish System-of-Systems (SoS) Systems Engineering guide

• 6-month effort addressing areas of agreement across community

• Initial focus on SoS with stated requirements and organizations responsible for execution

• Addresses DAG technical process and considerations for technical management across system life cycle

- Focused on systems engineering challenges characteristics of SoS and suggested approaches

• Audience: Program Managers and Lead/Chief Engineers for SoS acquisition programs, legacy systems, and constituent programs

Draft of initial version of guide is out for review