aew&c software metrics program - practical … wedgetail slide 1 aew&c software metrics...

Project Wedgetail

AEW&C Software Metrics Program

Stuart GarrettAEW&C RPT Software Engineering Manager

PSM Users’ ConferenceKeystone, Colorado

14 - 18 July 2003

Project Wedgetail

Acknowledgment

Data sourced from the Prime (Boeing)Boeing obtains the data from Software Development Team and SubcontractorsAcquisition Project Office is a user - not generator - of the data

Project Wedgetail

AEW&C Software Metrics Program -Introduction

Metrics used on AEW&C ProjectBrief history of selection processSource of DataHow metrics are used by Contractor

not “just another CDRL”How metrics are used within the CoA

Project Wedgetail

Project Elements

737 AEW&C737 AEW&CAircraftAircraft

PersonnelPersonnel and Trainingand Training

AEW&C Support Facility

Mission Mission SupportSupport

Flight CrewFlight CrewSimulatorSimulator

Mission CrewMission CrewSimulatorSimulator

Integrated Logistics SupportIntegrated Logistics Support

Project Wedgetail

Airborne Mission Segment

Project Wedgetail

Schedule - The Big Picture

Jul 00 Jul 07

Jan 01

Jul 01

Jan 02

Jul 02

Jan 03

Jul 03

Jan 04

Jul 04

Jan 05

Jul 05

ContractSignature

DT&E OT&EAT&E

Jan 06

Jul 06

Jan 07

#3 #4AMS# 1&2

Design ReviewsDesign Reviews

FacilitiesFacilities

IOCIOC

Jan 08

OFT/OMSOFT/OMS ASFASF

AMS #1First Flight

System Employment KnowledgeSystem Employment Knowledge

AMS #1Delivery

PresentPresent timetime

Project Wedgetail

Software Build Integration

Subcontract Syste m Desig n

Subcontract Co ntrac t D ates

ESM/EWSP

Tr ac ker /Infa structur e M o ds

AEW&C S/WDev elopment

HMI Infrastructure& Tracker ID Software

Radar

OM S B ld 1

SupportSegment Subcontract Syste m Desig n

Subcontract Co ntrac t D eliv erie s

DT &E TA T&E

IntegInteg

Integ

V er ifP1 T A T &E

B uil d 8 D ev el o pme ntB uil d 7 D ev el o pme nt

B uil d 6 D ev el o pme ntIntegB uil d 5 D ev el o pme nt

IntegB uil d 4 D ev el o pme ntInteg

SDLPL/ WT SDL

PL/WT SIL

Subcontract Syste m Desig n

B uil d 3 D ev el o pme nt

R2 Simulation

Project Wedgetail

Testing

DesktopSoftware Development LaboratorySystem Integration LaboratoryAircraft on groundAircraft in air

Project Wedgetail

Software Metrics Program -What Type of Measurements Were Sought

Required measurements that could assist with:

Early identification of problems

Evaluating the maturity of the software

Measuring progress of the development and test efforts

Evaluating software risk

Determining mitigation actions when required

Project Wedgetail

Metrics - Selection Process

Workshop with FacilitatorAttended by Project Office, Prime (Boeing) and Major Subcontractors

NG, BAE Systems

ObjectivesIdentify appropriate set of metricsscope and define the metricsfacilitate buy-in and ownership with all parties

Project Wedgetail

Software Metrics Program Measures to monitor Maturity

Percent Activity Complete Measures the progress of software development activities

Software Requirements Volatility

Measures requirements growth and maturity by tracking the number of requirements changes over time

Build ContentMeasures the planned and actual progress of the incorporation offunctionality for each software build release

Work Products Measures the progress of discrete software products during development

Test CoverageMeasures the planned and actual progress of the results of testing for each software build release

Problem ReportsMeasures the status, priority, effectivity, and resolution time of software change requests

Project Wedgetail

Software Metrics Program Measures for Schedule Performance and Risk

Staffing ProfileMeasures the headcount, experience, and training for the software organizations

Software Product Size Measures the planned and actual size of the software products

Data Deliverable ItemsMeasures the planned and actual, delivery and approval of contractually-required software data items

Computer Resource UtilizationMeasures the planned and actual utilization of computing resources (CPU, storage, input/output)

Project Wedgetail

Management of Resources

Percentage Activity Complete

Build Progress - Team 2

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Mar-03 Apr-03 May-03 Jun-03 Jul-03 Aug-03 Sep-03 Oct-03 Nov-03 Dec-03 Jan-04 Feb-04 Mar-04

Planned % Complete Actual % Complete Detailed Design Complete

Code/Unit Test Complete Integration Complete

Build Progress - Team 1

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

1-Oct-02

1-Nov-02

1-Dec-02

1-Jan-03

1-Feb-03

1-Mar-03

1-Apr-03

1-May-03

1-Jun-03

1-Jul-03

1-Aug-03

1-Sep-03

1-Oct-03

Planned % Complete Actual % CompleteDetailed Design Complete

Code/Unit Test Complete Integration Complete

Project Wedgetail

Risk Identification

Project Wedgetail

Status Monitoring

Test Coverage

Project Wedgetail

Status Reporting

Software Metrics database provided each month to CoA Project OfficeOn-line access to Boeing staff and ManagementMetrics summaries generated for weekly management reportsUsed as integral element of Risk Management SystemFeeds into technical and design reviews

Project Wedgetail

Percentage Activity Complete -Software Work Packages and Earned Value

MajorMilestone Apr May Jun Jul Aug Sep

2002

Build 3CSC (1...n)

Detailed Design

Detailed Design complete

78%

• Software work packages are established at WBS level 4• Schedule, plan for completion, and status (work complete)

tracked at lower (WBS 5) levels

What’s behind this earned value technique for software development?

Project Wedgetail

Detailed Software Development Status Reporting

• WBS level 4 status reported represents composite work package summary of software development activities

Plan Actual % CompleteCSC 1 12-Jul-02 30-Jul-02 100%CSC 2 12-Jul-02 30-Jul-02 100%CSC 3 24-Jul-02 78%CSC 4 22-Jul-02 0%CSC 5 19-Jul-02 31-Jul-02 100%CSC 6 19-Jul-02 23-Jul-02 100%CSC 7 28-Jun-02 11-Jul-02 100%CSC 8 19-Jul-02 97%CSC 9 19-Jul-02 31-Jul-02 100%CSC 10 18-Jul-02 42%CSC 11 19-Jul-02 97%CSC 12 25-Jul-02 40%CSC 13 22-Jul-02 22-Jul-02 100%CSC 14 25-Jul-02 50%CSC 15 12-Jul-02 22-Jul-02 100%CSC 16 1-Aug-02 81%CSC 17 2-Aug-02 84%CSC 18 9-Aug-02 9%CSC 19 20-Jul-02 2-Aug-01 100%Work Package Summary 78%

Architecture Team CSC Detailed Design Complete

• In this example, WBS level 5 status reported is based on detailed design development for Software CSCs (Computer Software Components)• Plan for individual

components established at start of build cycle.

• Status reflects actual completion dates and/or % completion.

Reported as summary Tier IV Work Package status

Project Wedgetail

Individual Developer Progress Status and Reporting

• WBS level 5 plans generated based on detailed design activities for each Software Component• Plans and Status maintained by

individual developers• Development Status updated

weekly• CSC detailed design complete

status is weighted summary of individual activities

CSC: CSC n Detailed Design

Detailed Design Status Completion date % Activity Complete

Interfaces (25%) 25%objects defined * 11-Jul-02

types defined * 11-Jul-02exceptions defined * 11-Jul-02

methods defined * 11-Jul-02impl classes (25%) 25%

implemented interface defined * 11-Jul-02other impl classes inherited * 11-Jul-02relationships between impls * 11-Jul-02

Component design (40%) 40%lead classes identified * 25-Jun-02

periodic jobs named * 25-Jun-02sequence diagrams * 15-Jul-02

Internal design defined 11-Jul-02MSDL data defined 31-Jul-02

Checkpoint data defined 31-Jul-02Recorded data defined 31-Jul-02

HMI defined n/aalgorithms defined n/a

Peer Review (10%) 10%review package available 10-Jul-02

Conduct peer review 11-Jul-02action items closed 11-Jul-02

Detailed Design Complete (Plan) 28-Jun-02Detailed Design % Complete 11-Jul-02 100%



Project Wedgetail

Summarized Team Metrics



Architecture Team Build 3 Plan & Status

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

3-May

17-M

ay31

-May

14-Ju

n28

-Jun

12-Ju

l26

-Jul

9-Aug

23-A

ug6-S

ep20

-Sep

4-Oct

18-O

ct1-N

ov15

-Nov

29-N

ov13

-Dec

27-D

ec10

-Jan

24-Ja

n7-F

eb21

-Feb7-M

ar21

-Mar

4-Apr

Perc

ent C

ompl

ete

Planned % Complete Actual % Complete

• Work package summary plans & status combined and reported weekly in Build metric as % of build complete

Detailed design complete status is weighted portion of overall build 3 development status for this team

Project Wedgetail

Build Summarized MetricsArchitecture Team Build 3 Plan & Status

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

3-May

17-M

ay31

-May

14-Ju

n28

-Jun

12-Ju

l26

-Jul

9-Aug

23-A

ug6-S

ep20

-Sep

4-Oct

18-O

ct1-N

ov15

-Nov

29-N

ov13

-Dec

27-D

ec10

-Jan

24-Ja

n7-F

eb21

-Feb7-M

ar21

-Mar

4-Apr

Perc

ent C

ompl

ete

Planned % Complete Actual % Complete

MCS S&C Build 3 Aggregate Progress

0.00%

20.00%

40.00%

60.00%

80.00%

100.00%

120.00%

6/14

/02

6/28

/02

7/12

/02

7/26

/02

8/9/

02

8/23

/02

9/6/

02

9/20

/02

10/4

/02

10/1

8/02

11/1

/02

11/1

5/02

11/2

9/02

12/1

3/02

12/2

7/02

Com

plet

e (%

)

Planned % CompleteActual % Complete

C&D Composite Status

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

07-Jun-02 26-Jul-02 13-Sep-02 01-Nov-02 20-Dec-02 07-Feb-03 28-Mar-03 16-May-03

Date

Perc

ent C

ompl

ete

% Plan Now % Actual Now

Mission Computing Build 3 Progress

0%

20%

40%

60%

80%

100%5/3

/026/3

/027/3

/028/3

/029/3

/0210

/3/02

11/3/

0212

/3/02

1/3/03

2/3/03

3/3/03

4/3/03

Planned % Complete

Actual % Complete

Detailed DesignComplete

Code & Unit TestComplete

CSC IntegrationComplete

Test DescriptonsComplete

DT&E Complete

• Work Package build status from each team is combined as a composite Mission Computing Build Status Metric

Project Wedgetail


0%

20%

40%

60%

80%

100%

5/3/02

6/3/02

7/3/02

8/3/02

9/3/02

10/3/

0211

/3/02

12/3/

021/3

/032/3

/033/3

/034/3

/03

Planned % Complete

Actual % Complete





DT&E Complete

Project Wedgetail


0%

20%

40%

60%

80%

100%

5/3/20

026/3

/2002

7/3/20

028/3

/2002

9/3/20

0210

/3/20

0211

/3/20

0212

/3/20

021/3

/2003

2/3/20

033/3

/2003

4/3/20

03

Planned % Complete

Actual % Complete





DT&E Complete

Actual Progress

Project Wedgetail


0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1-Oct-02 1-Nov-02 1-Dec-02 1-Jan-03 1-Feb-03 1-Mar-03 1-Apr-03 1-May-03 1-Jun-03 1-Jul-03 1-Aug-03 1-Sep-03 1-Oct-03

Planned % Complete Actual % CompleteDetailed Design Complete Code & Unit Test CompleteCSCI Integration Complete Subsystem Integration CompleteInterfaces Baselined

Project Wedgetail


0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Mar-03 Apr-03 May-03 Jun-03 Jul-03 Aug-03 Sep-03 Oct-03 Nov-03 Dec-03 Jan-04 Feb-04 Mar-04

Planned % Complete Actual % CompleteDetailed Design Complete Code & Unit Test CompleteCSCI Integration Complete Subsystem Integration CompleteInterfaces Baselined

Project Wedgetail

Brisbane - Australia

Project Wedgetail

Typical evening in Brisbane

Project Wedgetail

Day at the office - Australian style

Project Wedgetail

Use of Metrics within CoA

Provides insight into software development processConfirmation of understandings from meetings, documentation, reviewsInsight into hardware performanceInsight into maturity of designUnderstanding of effectiveness and status of recovery plans

Project Wedgetail

Use of Metrics within CoAMetrics reviewed in toto each monthAnalysing for unexpected trends, or unexplained spikes (degradation)Seeking confirmation of information obtained from design, management and technical reviewsFindings are consolidated into a monthly report to the Program Governance Board

Project Wedgetail

Value of Metrics Program

Unprecedented insight into software development activitiesInvaluable tool for management to make informed decisions re resourcesDifficult, if not impossible, to successfully manage a large complex software intensive project without an effective metrics program

Project Wedgetail

SummaryMetrics provide specific insight into software development activitiesAllows managers to assess progressIdentify problems - before they become unmanageablePlan (timely) solutionsObtain additional resources, as requiredMonitor the effectiveness of remedial actionsReduce overall risk to the program - excellent Risk Management / Reduction Strategy

Project Wedgetail

Questions?

aew&c software metrics program - practical … wedgetail slide 1 aew&c software metrics...

Documents