university of southern california center for software engineering cse usc engineering systems at...

University of Southern CaliforniaCenter for Software EngineeringC S E

USC

Engineering Systems at USC:Center for Systems and Software Engineering

Barry Boehm

CESUN University Roundtable

April 2, [email protected]; http://csse.usc.edu


USC

04-02-2008 ©USC-CSE 2

Outline

• CSE, SAE, and CSSE

• CSSE scope, objectives, and strategy

• Research and education examples

• Recent major developments– Integrating systems and software engineering– Value-based systems and software engineering


USC

CSE, SAE, and CSSE• Center for Software Engineering (CSE)

– Large research program, Affiliates’ program– MSCS-SwE and PhD program

• Systems Architecting and Engineering (SAE) Program– Large MS program– Plans for PhD, Affiliates’ program

• Proposal to Affiliates– Set up SAE Affiliates’ program– Membership: $25K/year for either; $40K/year for both

• Affiliates’ response– We’re integrating SysE and SwE; you should do this too

• Result: Center for Systems and Software Engineering– Membership: $35K/year ($5K/year for small organizations)

04-02-2008 ©USC-CSE 3


USC

10-08-2007 ©USC-CSE 4

Research, Transition, and Education Strategies

• Stakeholder Win-Win– Principals, affiliates, project clients, practitioners,

students• Opportunistic collaboration

– Focus on high-impact current, future problem areas– Principals’ criterion: shared interest in SysE/SwE

• Commercial or open-source versions of tools• Research/education integration via real-client projects

– Primarily USC campus, neighborhood e-services– Validate new methods and tools via project usage– Partial basis of 11 PhD dissertations

– Rqts. negotiation, formalization (2), COTS integration (2), Value-based methods (2), Agile methods (1), Quality tradeoffs (1), Risk analysis (1), Cost estimation (1), Testbeds (1)


USC

10-08-2007 ©USC-CSE 5

Why Software Projects Fail


USC

10-08-2007 ©USC-CSE 6

“Software Engineering:” The disciplines which distinguish the coding of a computer program from the development of a software system product

Requirements,

Architecture

Design,

Code

Implement,

Maintain

Computer Science CS Focus

User Applications

Economics

People

• Accommodate new tools and techniques•Web apps, Architecting tools, WikiWinWin, Spiral processes

• Integrate all these considerations–Via Incremental Commitment Model (ICM)–Necessarily emphasizes systems architecting, engineering

Stages

Issues

- Involves 9 times as much effort [Brooks, 1975]


USC

2007-08 Software Engineering ProjectsBox Office Database System Jay McAdams 24th Street Theatre

Online Peer Review System Stephen Bucher USC, Engineering Writing Program

Thai CDC Software Tool Pheel Wang Thai Community Development Center

USC COINCOMO Ed Colbert, Vu Nguyen USC CSSE

Crystal Stairs Dashboard Rick Capella Crystal Stairs, Inc.

Family & Homeless Well-Being Project Jill Remelski St. Francis Center (SFC)

BTI Appraisal Projects Megan Tunnell O'Rourke BTI Appraisal

LAMAS Customer Service Application Celia Castellanos Los Angeles Music and Art School

Web-based service for TPC Foundation Pat Means Turning Point URBAN Magazine

REEO Database Scott Stimpfel Resources for Educational and Employment Opportunities

BID review System Keith Culling Background Investigation Division, LA city

THSA Website Project Poonchana Thitametakul USC THSA

EEO Section Database Art Irigoyen LA city

Conference Room Reservation System Linda Lanier LA city

Applicant-processing systems for CLAPD Dee Dee Coultas LA city

E-Mentoring program Carlene Davis Los Angeles Urban League

Social Networking Tools for Librarians Julie Kwan Social Networking Tools for Librarians

EZSIM II Ray Madachy USC-CSSE

Los Angeles County Generation Web Initiative Frank Cheng, Jeramy Gray LA County

Development Of Hunter-gatherer Database Chris Boehm USC

10-08-2007 ©USC-CSE 7


USC

4/18/06 ©USC-CSE 8

First Six Weeks of Project Course

Domain Model

WinWin Taxonomy

Basic Conceptof Operation

Frequent

RisksStakeholders,

Primary win conditions

WinWin Negotiation

Model

IKIWISI Model,Prototypes,

Properties Models

EnvironmentModels

WinWin Agreements, Shared Vision

ViableArchitecture

Options

Updated Conceptof Operation

Life Cycle Planelements

Outstanding LCO risks

RequirementsDescription

LCO Rationale

Life Cycle Objectives (LCO) Package

Anchor PointModel

determinesidentifiesidentifiesdetermines

situates exercise exercise focususe of

focus use of determines

guidesdetermination of validate

inputs for

provides

initialize adopt identify identify

update update

achieveiterate to feasibility, consistency determines exit

criteria for validates readiness of

initializes


USC

CSSE Principals (37)

• Core in CS, SAE, ISE (20)• Engineering: Aerospace, Civil, Electrical,

Mechanical (8)• Schools of Business, Public Policy (4) • Information Sciences Institute, Institute for

Creative Technology, CREATE Center (5)• 7 NAE members; 4 INCOSE Fellows; 9 IEEE

Fellows

10-08-2007 ©USC-CSE 9


USC

Major Research Projects• Integrating systems and software engineering (OSD-SSE)

– 2007: Inhibitors and enablers study– 2008: IS&SE via Incremental Commitment Model guidebook

• Systems of systems engineering (Army FCS, OSD-SSE)– 2001- : Future Combat Systems S&SE support– 2007- : OSD-SSE Systems of Systems Guidebook support– 2007: DACS system of systems cost estimation framework

• Next-generation space systems (DARPA, NASA)– 2003- : Software reliability; processes; risk assessment

• Scaling up agile methods (Affiliates)– 2003- : Architecture-based agile methods

• Value-based science of design (NSF)– 2003- : Value-based theories of software, systems engineering

04-02-2008 ©USC-CSE 10


USC

08/31/06 ©USC-CSE 11

FCS WinWin Spiral Model1b. StakeholdersIdentify SystemObjectives, Constraints,& Priorities (OC&Ps);Alternative SolutionElements

1a. IdentifySuccess-CriticalStakeholders

2a. EvaluateAlternativeswith respect toOC&Ps

2b.Assess,AddressRisks

3. ElaborateProduct andProcessDefinition4. Verify and Validate

Product and ProcessDefinitions

Stakeholders’

Commitment4

5

6

8

2

1

Stakeholders’Review

7

3

L COL CA

Build 2

Build 3

Progress Through Steps

Bl1

Driven By:

Success-critical

stakeholders’ win

conditions

Risk Management

Spiral anchor point

milestones

Feasibility Rationale


USC

10-08-2007 ©USC-CSE 12

How much Architecting is Enough?- A COCOMOII Analysis

0

10

20

30

40

50

60

70

80

90

100

0 10 20 30 40 50 60

Percent of Time Added for Architecture and Risk Resolution

Per

cen

t of T

ime

Ad

ded

to O

vera

ll S

ched

ule

Percent of Project Schedule Devoted to Initial Architecture and Risk Resolution

Added Schedule Devoted to Rework(COCOMO II RESL factor)

Total % Added Schedule

10000KSLOC

100 KSLOC

10 KSLOC

Sweet Spot

Sweet Spot Drivers:

Rapid Change: leftward

High Assurance: rightward


USC

10-08-2007 ©USC-CSE 13

Tradeoffs Among Cost, Schedule, and Reliability:Want $5.5M, 20 months, 10K hours MTBF

0

1

2

3

4

5

6

7

8

9

0 10 20 30 40 50

Development Time (Months)

Co

st

($M

)

(VL, 1)

(L, 10)

(N, 300)

(H, 10K)

(VH, 300K)

-- Cost/Schedule/RELY:

“pick any two” points

(RELY, MTBF (hours))

•Cost, schedule, quality: pick any two?•For 100-KSLOC set of features•Can “pick all three” with 77-KSLOC set of features


USC

08/31/06 ©USC-CSE 14

COSYSMO

SizeDrivers

EffortMultipliers

Effort

Calibration to 42 projects

# Requirements# Interfaces# Scenarios# Algorithms

+3 adjustment factors

- Application factors-8 factors

- Team factors-6 factors

COSYSMO Operational Concept


USC

04-02-2008 ©USC-CSE 15

Outline




• Recent major developments– Integrating systems and software engineering– Value-based systems and software engineering


USC

03/19/2008 ©USC-CSSE 16

2007 OSD-SSE Study Scope and Context• Statement of Work

– Evaluate current DoD SysE, SwE processes and standards• Primarily SEP Guide, DAG Ch. 4 (SysE), DoDI 5000.2• Identify deltas, issue areas, recommendations

– Evaluate ICM for integration applicability– Evaluate relevant DoD guidance, education, and training

• Provide recommendations based on evaluations• Context: current and future DoD S&SE challenges

– Multi-owner, multi-mission systems of systems– Rapid pace of change– Always-on, never-fail systems– Need to turn within adversaries’ OODA loop

• Observe, orient, decide, act– Within asymmetric adversary constraints


USC

• An omniscient authority pre-establishes the requirements, preferred system concept, etc. (A4.1, 4.2, 4.4)

• Technical solutions are mapped and verified with respect to these (A4.1, 4.2, 4.4)

• They and technology do not change very often (A4.2, 4.5, 6.2)– Emphasis on rigor vs. adaptability

• The program has stable and controllable external interfaces (A4.5)• Project organization is function-hierarchical and WBS-oriented

(A3.1, 3.2)• All requirements are equally important (A4.2)• Reviews event/product-based vs. evidence-based (A5.2)• The program’s critical path can be defined up front (A6.1)• Can achieve optimum readiness at minimum life-cycle cost (A6.5)

– No slack for contingencies03/19/2008 ©USC-CSSE 17

Current SysE Guidance: Outdated Assumptions

Example: SysE Plan Preparation GuideSometimes OK for hardware; generally not for software


USC

03/19/2008 ©USC-CSSE 18

System/Software Architecture Mismatches- Maier, 2006

• System Hierarchy

– Part-of relationships; no shared parts

– Function-centric; single data dictionary

– Interface dataflows

– Static functional-physical allocation

• Software Hierarchy

– Uses relationships; layered multi-access

– Data-centric; class-object data relations

– Interface protocols; concurrency challenges

– Dynamic functional-physical migration


USC

03/19/2008 ©USC-CSSE 19

• Fractionated, incompatible sensor data management

• “Touch Football” interface definition earned value– Full earned value taken for defining interface dataflow

– No earned value left for defining interface dynamics• Joining/leaving network, publish-subscribe, interrupt handling,

security protocols, exception handling, mode transitions

– Result: all green EVMS turns red in integration

Examples of Architecture Mismatches

…

Sensor 1

SDMS1

Sensor 2

SDMS2

Sensor 3

SDMS3

Sensor n

SDMSn

……


USC

03/19/2008 ©USC-CSSE 20

The Incremental Commitment Life Cycle Process: OverviewStage I: Definition Stage II: Development and Operations

Anchor Point Milestones


Synchronize, stabilize concurrency via FEDsSynchronize, stabilize concurrency via FEDs

Risk patterns determine life cycle process

Risk patterns determine life cycle process


USC

03/19/2008 ©USC-CSSE 21

Anchor Point Feasibility Evidence Description

• Evidence provided by developer and validated by independent experts that:

If the system is built to the specified architecture, it will– Satisfy the requirements: capability, interfaces, level of

service, and evolution

– Support the operational concept

– Be buildable within the budgets and schedules in the plan

– Generate a viable return on investment

– Generate satisfactory outcomes for all of the success-critical stakeholders

• All major risks resolved or covered by risk management plans

• Serves as basis for stakeholders’ commitment to proceed

Can be used to strengthen current schedule- or event-based reviews


USC

03/19/2008 ©USC-CSSE 22

Incremental Commitment in Gambling

• Total Commitment: Roulette– Put your chips on a number

• E.g., a value of a key performance parameter– Wait and see if you win or lose

• Incremental Commitment: Poker, Blackjack– Put some chips in– See your cards, some of others’ cards– Decide whether, how much to commit to proceed


USC

03/19/2008 ©USC-CSSE 23

Scalable remotely controlled operations


USC

03/19/2008 ©USC-CSSE 24

Total vs. Incremental Commitment – 4:1 RPV

• Total Commitment– Agent technology demo and PR: Can do 4:1 for $1B

– Winning bidder: $800M; PDR in 120 days; 4:1 capability in 40 months

– PDR: many outstanding risks, undefined interfaces

– $800M, 40 months: “halfway” through integration and test

– 1:1 IOC after $3B, 80 months

• Incremental Commitment [number of competing teams]– $25M, 6 mo. to VCR [4]: may beat 1:2 with agent technology, but not

4:1

– $75M, 8 mo. to ACR [3]: agent technology may do 1:1; some risks

– $225M, 10 mo. to DCR [2]: validated architecture, high-risk elements

– $675M, 18 mo. to IOC [1]: viable 1:1 capability

– 1:1 IOC after $1B, 42 months


USC

03/19/2008 ©USC-CSSE 25

The Incremental Commitment Life Cycle Process: OverviewStage I: Definition Stage II: Development and Operations



Concurrently engr. OpCon, rqts, arch, plans, prototypes

Concurrently engr. OpCon, rqts, arch, plans, prototypes

Concurrently engr. Incr.N (ops), N+1 (devel), N+2 (arch)

Concurrently engr. Incr.N (ops), N+1 (devel), N+2 (arch)


USC

03/19/2008 ©USC-CSSE 26

ICM HSI Levels of Activity for Complex Systems


USC

03/19/2008 ©USC-CSSE 27

Different Risk Patterns Yield Different Processes


USC

03/19/2008

Common Risk-Driven Special Cases of the ICMSpecial Case Example Size,

ComplexityChange Rate %/Month

Criticality NDI Support Org, Personnel Capability

Key Stage I Activities : Incremental Definition Key Stage II Activities: Incremental Development, Operations

Time per Build; per Increment

1. Use NDI Small Accounting Complete Acquire NDI Use NDI

2. Agile E-services Low 1 – 30 Low-Med Good; in place

Agile-readyMed-high

Skip Valuation , Architecting phases Scrum plus agile methods of choice <= 1 day; 2-6 weeks

3. Architected Agile

Business data processing

Med 1 – 10 Med-High Good; most in place

Agile-readyMed-high

Combine Valuation, Architecting phases. Complete NDI preparation

Architecture-based Scrum of Scrums 2-4 weeks; 2-6 months

4. Formal Methods

Security kernel; Safety-critical LSI chip

Low 0.3 Extra High None Strong formal methods experience

Precise formal specification Formally-based programming language; formal verification

1-5 days;1-4 weeks

5. HW component with embedded SW

Multi-sensor control device

Low 0.3 – 1 Med-Very High

Good; In place

Experienced; med-high

Concurrent HW/SW engineering. CDR-level ICM DCR

IOC Development, LRIP, FRP. Concurrent Version N+1 engineering

SW: 1-5 days; Market-driven

6. Indivisible IOC Complete vehicle platform

Med – High

0.3 – 1 High-Very High

Some in place Experienced; med-high

Determine minimum-IOC likely, conservative cost. Add deferrable SW features as risk reserve

Drop deferrable features to meet conservative cost. Strong award fee for features not dropped

SW: 2-6 weeks;Platform: 6-18 months

7. NDI- Intensive Supply Chain Management

Med – High

0.3 – 3 Med- Very High

NDI-driven architecture

NDI-experienced; Med-high

Thorough NDI-suite life cycle cost-benefit analysis, selection, concurrent requirements/ architecture definition

Pro-active NDI evolution influencing, NDI upgrade synchronization

SW: 1-4 weeks; System: 6-18 months

9. Hybrid agile / plan-driven system

C4ISR Med – Very High

Mixed parts: 1 – 10

Mixed parts; Med-Very High

Mixed parts Mixed parts Full ICM; encapsulated agile in high change, low-medium criticality parts (Often HMI, external interfaces)

Full ICM ,three-team incremental development, concurrent V&V, next-increment rebaselining

1-2 months; 9-18 months

9. Multi-owner system of systems

Net-centric military operations

Very High Mixed parts: 1 – 10

Very High Many NDIs; some in place

Related experience, med-high

Full ICM; extensive multi-owner team building, negotiation

Full ICM; large ongoing system/software engineering effort


10. Family of systems

Medical Device Product Line

Med – Very High

1 – 3 Med – Very High

Some in place Related experience, med – high

Full ICM; Full stakeholder participation in product line scoping. Strong business case

Full ICM. Extra resources for first system, version control, multi-stakeholder support


C4ISR: Command, Control, Computing, Communications, Intelligence, Surveillance, Reconnaissance. CDR: Critical Design Review. DCR: Development Commitment Review. FRP: Full-Rate Production. HMI: Human-Machine Interface. HW: Hard ware. IOC: Initial Operational Capability. LRIP: Low-Rate Initial Production. NDI: Non-Development Item. SW: Software


USC

03/19/2008 ©USC-CSSE 29

Example ICM Commercial Application:Symbiq Medical Infusion Pump

Winner of 2006 HFES Best New Design AwardDescribed in NRC HSI Report, Chapter 5


USC

04-02-2008 ©USC-CSE 30

Outline




• Recent major developments– Integrating systems and software engineering– Value-based systems and software engineering (VBSEE)


USC

06/25/07 © USC-CSE 31

VBSEE Motivation and Context

• INCOSE Definition of “Systems Engineering”– An interdisciplinary approach and means to enable the

realization of successful systems– A systems engineering theory should provide criteria and

guidance for realizing successful systems

• Addressing the Intellectual Content of Systems Engineering– How distinguished from other engineering disciplines– How related to other theories with intellectual content


USC

06/25/07 © USC-CSE 32

Intellectual Content of Systems Engineering: Distinguishing Features

• The intellectual content of most engineering disciplines is component-oriented and value-neutral– Ohm’s Law, Hooke’s Law, Newton’s Laws

• The intellectual content of systems engineering is focused on:– How people and components can be combined into a cost-

effective system• System definition, architecting, analysis, planning,

evaluation, improvement• Cost-effectiveness in terms of value to stakeholders

• The ability to address value considerations is an essential qualification for a theory of systems engineering


USC

06/25/07 © USC-CSE 33

Theory W: Enterprise Success Theorem– And informal proof

Theorem: Your enterprise will succeed if and only if

it makes winners of your success-critical stakeholders

• Proof of “if”:Everyone that counts is a winner.Nobody significant is left to complain.

• Proof of “only if”:

Nobody wants to lose.Prospective losers will refuse to participate, or will counterattack.The usual result is lose-lose.


USC

06/25/07 © USC-CSE 34

Theory W: WinWin Achievement Theorem

Making winners of your success-critical stakeholders requires:

i. Identifying all of the success-critical stakeholders (SCSs).

ii. Understanding how the SCSs want to win.

iii. Having the SCSs negotiate a win-win set of product and process plans.

iv. Controlling progress toward SCS win-win realization, including adaptation to change.


USC

06/25/07 © USC-CSE 35

VBSET Component Theories• Theory W (Stakeholder win-win)

– Enterprise Success Theorem, Win-Win Achievement Theorem

• Dependency Theory (Product, process, people interdependencies)– Systems architecture/performance theory, costing and

scheduling theory; organization theory

• Utility Theory– Utility functions, bounded rationality, Maslow need

hierarchy, multi-attribute utility theory

• Decision Theory– Statistical decision theory, game theory, negotiation theory,

Theory of Justice

• Control Theory– Observability, predictability, controllability, stability theory


USC

08/2007 ©USC-CSSE 36

Initial VBSE Theory: 4+1 Process– With a great deal of concurrency and backtracking

Utility Theory

Theory W:SCS Win-Win

Decision Theory

Dependency Theory

Control Theory

6a, 7c. State measurement, prediction, correction; Milestone synchronization

5a. Investment analysis, Risk analysis

1. Protagonist goals3a. Solution exploration7. Risk, opportunity, change management

5a, 7b. Prototyping

2a. Results Chains3b, 5a, 7b. Cost/schedule/performance tradeoffs

2. Identify SCSs

3b, 7a. Solution Analysis

5a, 7b. Option, solution development & analysis

4. SCS expectations management

3. SCS Value Propositions(Win conditions)

SCS: Success-Critical Stakeholder

6, 7c. Refine, Execute, Monitor & Control Plans

5. SCS Win-Win Negotiation

Provides process for executing ICM Exploration phase


USC

06/25/07 © USC-CSE 37

Value-Based Testing: Empirical Data and ROI— LiGuo Huang, ISESE 2005

-1.5

-1

-0.5

0

0.5

1

1.5

2

0 10 20 30 40 50 60 70 80 90 100

% Tests Run

Re

turn

On

In

ve

stm

en

t (R

OI)

Value-Neutral ATG Testing Value-Based Pareto Testing

% of Valuefor

CorrectCustomer

Billing

Customer Type

100

80

60

40

20

5 10 15

Automated test generation (ATG) tool

- all tests have equal value

Bullock data– Pareto distribution% of

Valuefor

CorrectCustomer

Billing

Customer Type

100

80

60

40

20

5 10 15



% of Valuefor

CorrectCustomer

Billing

Customer Type

100

80

60

40

20

5 10 15



Bullock data– Pareto distribution

(a)

(b)


USC

04/19/07 38

By Number P-value % Gr A higher

By Impact P-value % Gr A higher

Average of Concerns

0.202 34 Average Impact of Concerns

0.049 65

Average of Problems

0.056 51 Average Impact of Problems

0.012 89

Average of Concerns per hour

0.026 55 Average Cost Effectiveness of Concerns

0.004 105

Average of Problems per hour

0.023 61 Average Cost Effectiveness of Problems

0.007 108

• Group A: 15 IV&V personnel using VBR procedures and checklists

• Group B 13 IV&V personnel using previous value-neutral checklists– Significantly higher numbers of trivial typo and grammar faults

Experiment: Reviews of Prioritized Ops Concept, Requirements, and Architecture DescriptionsExperiment: Reviews of Prioritized Ops Concept, Requirements, and Architecture Descriptions

Value-Based Reading (VBR) Experiment— Keun Lee, ISESE 2005


USC

03/19/2008

Conclusions: IS&SE with ICM

• Current DoD (and other) SysE guidance seriously inhibits SwE– Largely sequential definition, design, development, integration, and test– Slows agility, ability to turn inside adversaries’ OODA loop– Functional “part-of” vs. layered “served by” product hierarchy– Static vs. dynamic interfaces: messages vs. protocols – One-size-fits-all process guidance inhibits balancing agility and assurance

• Incremental Commitment Model (ICM) enables better IS&SE– Integrates hardware, software, human factors aspects of SysE – Concurrent exploration of needs, opportunities, solutions

• Enabled by value-based systems engineering theory and process– Successfully addresses issues above in commercial practice– Only partially proven in DoD practice (examples: CrossTalk Top-5 projects)– Key practices applied to help major programs (example: Future Combat Systems)– Being adopted by major programs (example: Missile Defense Agency)– Effort underway to work with adopters to develop DoD ICM Guidebook, in

coordination with other USD(AT&L)SSE initiatives

©USC-CSSE 39


USC

2008 SOW: DoD ICM Guidebook

• Develop a draft Guidebook for next-generation DoD IS&SE based on the ICM– In collaboration with DoD and industry participants

• Collaborate with selected DoD and industry parties in experimental tailoring of draft Guidebook elements

• Hold a series of Guidebook workshops– Mar 19-20 (USC), July 15-17 (DC area), Oct 29-30 (USC)

• Coordinate Guidebook with other IS&SE initiatives– Systems of systems engineering, systemic analysis,

acquisition, education, assessment, research and technology

03/19/2008 ©USC-CSSE 40


USC

03/19/2008 ©USC-CSSE 41

References - IBeck, K., Extreme Programming Explained, Addison Wesley, 1999.Boehm, B., “Some Future Trends and Implications for Systems and Software Engineering Processes”,

Systems Engineering 9(1), pp. 1-19, 2006. Boehm, B., Brown, W., Basili, V., and Turner, R., “Spiral Acquisition of Software-Intensive Systems of

Systems, CrossTalk, Vol. 17, No. 5, pp. 4-9, 2004.Boehm, B. and Lane J., "21st Century Processes for Acquiring 21st Century Software-Intensive Systems

of Systems." CrossTalk: Vol. 19, No. 5, pp.4-9, 2006. Boehm, B., and Lane, J., “Using the ICM to Integrate System Acquisition, Systems Engineering, and

Software Engineering,” CrossTalk, October 2007, pp. 4-9.

Boehm, B., and Lane, J., “A Process Decision Table for Integrated Systems and Software Engineering,” Proceedings, CSER 2008, April 2008.

Boehm, B., “Future Challenges and Rewards for Software Engineers,” DoD Software Tech News, October 2007, pp. 6-12.

Boehm, B. et al., Software Cost Estimation with COCOMO II, Prentice Hall, 2000.Boehm, B., Software Engineering Economics, Prentice Hall, 2000.Carlock, P. and Fenton, R., "System of Systems (SoS) Enterprise Systems for Information-Intensive

Organizations," Systems Engineering, Vol. 4, No. 4, pp. 242-26, 2001.Carlock, P., and J. Lane, “System of Systems Enterprise Systems Engineering, the Enterprise

Architecture Management Framework, and System of Systems Cost Estimation”, 21st International Forum on COCOMO and Systems/Software Cost Modeling, 2006.

Checkland, P., Systems Thinking, Systems Practice, Wiley, 1980 (2nd ed., 1999).

Department of Defense (DoD), Defense Acquisition Guidebook, version 1.6, http://akss.dau.mil/dag/, 2006.

Department of Defense (DoD), Instruction 5000.2, Operation of the Defense Acquisition System, May 2003.

Department of Defense (DoD), Systems Engineering Plan Preparation Guide, USD(AT&L), 2004.

Electronic Industries Alliance (1999); EIA Standard 632: Processes for Engineering a System

Galorath, D., and Evans, M., Software Sizing, Estimation, and Risk Management, Auerbach, 2006.

Hall, E.T., Beyond Culture, Anchor Books/Doubleday, 1976.


USC

03/19/2008 ©USC-CSSE 42

References -IIHighsmith, J., Adaptive Software Development, Dorset House, 2000.

International Standards Organization, Information Technology Software Life Cycle Processes, ISO/IEC 12207, 1995

ISO, Systems Engineering – System Life Cycle Processes, ISO/IEC 15288, 2002.

Jensen, R. “An Improved Macrolevel Software Development Resource Estimation Model,” Proceedings, ISPA 5, April 1983, pp. 88-92.

Krygiel, A., Behind the Wizard’s Curtain; CCRP Publication Series, July, 1999, p. 33Lane, J. and Boehm, B., "System of Systems Cost Estimation: Analysis of Lead System Integrator

Engineering Activities", Information Resources Management Journal, Vol. 20, No. 2, pp. 23-32, 2007.

Lane, J. and Valerdi, R., “Synthesizing SoS Concepts for Use in Cost Estimation”, Proceedings of IEEE Systems, Man, and Cybernetics Conference, 2005.

Lientz, B., and Swanson, E.B., Software Maintenance Management, Addison Wesley, 1980. Madachy, R., Boehm, B., Lane, J., "Assessing Hybrid Incremental Processes for SISOS Development",

USC CSSE Technical Report USC-CSSE-2006-623, 2006. Maier, M., “Architecting Principles for Systems-of-Systems”; Systems Engineering, Vol. 1, No. 4 (pp 267-

284).Maier, M., “System and Software Architecture Reconciliation,” Systems Engineering 9 (2), 2006, pp. 146-

159.Northrop, L., et al., Ultra-Large-Scale Systems: The Software Challenge of the Future, Software

Engineering Institute, 2006. Pew, R. W., and Mavor, A. S., Human-System Integration in the System Development Process: A New

Look, National Academy Press, 2007.Putnam, L., “A General Empirical Solution to the Macro Software Sizing and Estimating Problem,” IEEE

Trans SW Engr., July 1978, pp. 345-361.Rechtin, E. Systems Architecting, Prentice Hall, 1991.Schroeder, T., “Integrating Systems and Software Engineering: Observations in Practice,” OSD/USC

Integrating Systems and Software Engineering Workshop, http://csse.usc.edu/events/2007/CIIForum/pages/program.html, October 2007.


USC

03/19/2008©USC-CSSE

List of AcronymsB/L Baselined

C4ISR Command, Control, Computing, Communications, Intelligence, Surveillance, Reconnaissance

CD Concept Development

CDR Critical Design Review

COTS Commercial Off-the-Shelf

DCR Development Commitment Review

DI Development Increment

DoD Department of Defense

ECR Exploration Commitment Review

EVMS Earned Value Management System

FCR Foundations Commitment Review

FDN Foundations, as in FDN Package

FED Feasibility Evidence Description

FMEA Failure Modes and Effects Analysis

FRP Full-Rate Production

GAO Government Accountability Office

GUI Graphical User Interface


USC

03/19/2008©USC-CSSE

List of Acronyms (continued)

HMI Human-Machine Interface

HSI Human-System Interface

HW Hardware

ICM Incremental Commitment Model

IOC Initial Operational Capability

IRR Inception Readiness Review

IS&SE Integrating Systems and Software Engineering

LCA Life Cycle Architecture

LCO Life Cycle Objectives

LRIP Low-Rate Initial Production

MBASE Model-Based Architecting and Software Engineering

NDI Non-Developmental Item

NRC National Research Council

OC Operational Capability

OCR Operations Commitment Review

OO&D Observe, Orient and Decide

OODA Observe, Orient, Decide, Act

O&M Operations and Maintenance


USC

03/19/2008©USC-CSSE

List of Acronyms (continued)

PDR Preliminary Design Review

PM Program Manager

PR Public Relations

PRR Product Release Review

RUP Rational Unified Process

SoS System of Systems

SoSE System of Systems Engineering

SSE Systems and Software Engineering

SW Software

SwE Software Engineering

SysE Systems Engineering

Sys Engr Systems Engineer

S&SE Systems and Software Engineering

USD (AT&L) Under Secretary of Defense for Acquisition, Technology, and Logistics

VCR Validation Commitment Review

V&V Verification and Validation

WBS Work Breakdown Structure

WMI Warfighter-Machine Interface

university of southern california center for software engineering cse usc engineering systems at...

Documents