Report of Architecture Report of Architecture and Product Working and Product Working

GroupGroup

ICM WorkshopWashington, DC

July 17, 2008

Working Group Members

• J. D. Baker, BAE Systems • A. Winsor Brown, USC-CSSE• Karl Brunson, Lockheed Martin• Paul Croll, CSC• Thomas Knott, OSD • Art Pyster, Stevens• Paul Russell, Aerospace • Robert Schwenk, Army ASA(ALT) • J. Bruce Walker, SAF/AQRE• Lee Zhou, Boeing

2

Working Group Charter• Identify and prioritize the most important issues

associated with Architecture and Products (engineering artifacts) for ICM and Competitive Prototyping (CP)

• Suggest OSD initiatives and other actions to address those issues

3

Definition of Architecture• IEEE 1471: fundamental organization of a system

embodied in its components, their relationships to each other, and to the environment, and the principles guiding its design and evolution.

• Don Firesmith (from the SEI): The set of all the most important, pervasive, higher-level strategic decisions, inventions, engineering trade-offs, and assumptions (DIETAs), and their associated rationales concerning how the system meets its allocated and derived product and process requirements.

4

The Firesmith definition is the more useful for CP and ICM

Focus•Because CP is conducted to reduce risk, and the ICM is

a risk-driven life cycle model, we focused on how to use Architecture and Product to understand, manage, and reduce risk.

•As defined by Firesmith, the architecture includes many DIETAs and their rationale, not just the risky ones.

•For CP and anchor points in the ICM, we will focus on risky DIETAs; i.e., DIETAs with weak rationale which, if wrong, could have a significant negative impact on program cost, schedule, or performance.

•Strong rationale is based on objective evidence. Weak rationale is based on assertion and opinion.

5

System Architecting Paradigm

Three activities should happen concurrently and iteratively:

1. Systems and software engineers establish the most critical requirements/objectives – including those for “ilities”

2. Systems and software architects develop a system and software architecture that the architects believe will simultaneously support all critical requirements/objectives

3. Engineers evaluate the architecture for how well it really supports critical requirements/objectives, creating substantiating evidence for the architecture or identifying weaknesses in it

Today, it is common for any of these activities to be shortchanged, especially the third.

6

Types of Evidence

1. Analytic models

2. Scenario-based execution of prototypes

3. Scenario-based execution of simulations

4. Benchmarking

5. Appeal to historical analogy (we did something similar several times before)

6. Architecture Quality Cases (analogous to safety cases) with claims, arguments, and evidence

7. Process execution results, such as test results from early software builds

7

CP/ICM Issues and Actions (unordered)1. Architectures expressed using DoDAF typically do not

include all of the DIETAs in sufficient detail to support rigorous evaluation.

Action: Develop architectural representation guidance requiring DIETAs to be developed in sufficient detail to support rigorous evaluation. For example, DoDAF architectures typically don’t contain enough information to perform safety case analyses or to understand the security properties of the system.

2. The “ilities” are often understated in the requirements/objectives, yet are often a key source of problems later in system development. An architectural view for each of the relevant quality characteristics is required.

Action: Develop guidance requiring “ilities” to be sufficiently documented and articulating what sufficient means.

Action: Research how to present sufficient information in the views to support adequate evaluation.

8

Examples of Quality Characteristics1. Efficiency

2. Completeness

3. Correctness

4. Security

5. Compatibility

6. Interoperability

7. Maintainability

8. Expandability

9. Testability

10. Portability

11. Hardware Independence

12. Software Independence

9

13. Installability

14. Reusability

15. Reliability

16. Error Tolerance

17. Availability

18. Usability

19. Understandability

20. Ease of Learning

21. Operability

22. Communicativeness

23. Survivability

24. Flexibility

CP/ICM Issues and Actions (unordered)3. Architectures often do not state the rationale (evidence)

for their DIETAs in sufficient detail to understand which ones are particularly risky.

Action: Develop guidance requiring the rationale for DIETAs to be stated in sufficient detail and articulating what sufficient means.

4. There is no guidance for what evidence is adequate for any given situation or how that evidence should be presented (analogous to the problem of knowing when you have tested enough). How much prototyping is “enough”? How much evidence is “enough”?

Action: Conduct research on how much prototyping and evidence is enough and then document the research results in guidance.

Action: Engage Chris Powell on his dissertation research based upon his assessment of ACAT 1D program architectures since July 2004.

10

CP/ICM Issues and Actions (unordered)5. Government program offices are probably not staffed with

enough people with the skills to request the correct evidence from the supplier and to evaluate that evidence when the supplier provides it. Government offices should not request evidence unless they are able to evaluate it.

Action: Consider forming an architecture assessment team (and other types of assessment teams) at the OSD level that would be a resource available to interested programs.

6. Since competing suppliers will have different architectures, the architectures will have different risk profiles and therefore require different evidence. Who decides what evidence will be provided? The government? The supplier? How will the government fairly evaluate competing prototypes when presented with different types of evidence?

Action: Investigate legal and contractual implications of specific evidence requirements.

11

CP/ICM Issues and Actions (unordered)7. A competition should involve regular submission of

evidence – not just once at the end of the competition. Can suppliers “fix” problems along the way and resubmit stronger evidence? It would seem to be in the government’s best interest to allow this, but could be construed as “unfair” by some competitors.

Action: Investigate legal and contractual implications of requesting regular submission of evidence and propose ways to enable regular submission of evidence.

8. Creating evidence is often dependent on exercising scenarios, which are extremely difficult to generate in sufficient number and sufficient diversity to uncover weak DIETAs, especially for SoS.

Action: Research how to generate an adequate and diverse set of scenarios, especially for SoS or investigate alternative approaches to developing scenarios.

12

ICM Issues and Actions (Unordered)

9. Providing evidence for an SoS at regular milestones is especially challenging because the evidence provided by the individual system elements may not be available when originally expected. Understanding impact analysis across elements when something changes is challenging.

Action: Research how to perform impact analysis across elements and how to respond to “breakage” in synchronization across elements.

10.As development progresses from milestone to milestone, new evidence reconfirming key DIETAs is needed. There is no guidance as to what that evidence should be and how often it should be collected.

Action: Research what evidence is required to reconfirm key DIETAs and then document the approaches in guidance.

13

ICM Issues and Actions (Unordered)

11.Program offices are inherently biased when it comes to evaluating evidence that a supplier is making sufficient progress to pass a milestone. Having independent non-advocate reviews of evidence eliminates that problem, but can be expensive and difficult to staff.

Action: Investigate the cost and feasibility of independent non-advocate reviews vs. the cost of inadequate review by failing to use independent reviewers.

14

Value and Ease of Implementing Actions

Value and Ease of Implementing Actions

16