Copyright © 2004 Computer Sciences Corporation. All rights reserved. 3/16/2007 9:17:48 AM 1

NDIA National Test and Evaluation ConferenceMarch 12-15, 2007

Early Application of Computer Program Systems Integration, Test and Performance MeasurementTom SobieralskiComputer Sciences Corporation

3/16/2007 9:17:48 AM 2

Agenda

• Background

• Systems Integration, Test and Performance Measurement

• Summary

3/16/2007 9:17:48 AM 3

Background

3/16/2007 9:17:48 AM 4

What is Computer Program Systems Integration, Test and Performance Measurement?

• The ability to verify the interfaces, functions and measure performance of two or more computer programs on the target hardware suite and operating environment.

3/16/2007 9:17:48 AM 5

Catalyst for Early Computer Program Systems Integration, Test and Performance Measurement

• Introduction of COTS hardware• Conversion of legacy software to new languages• System complexity with multiple hardware and operating

environments

3/16/2007 9:17:48 AM 6

PB

System IntegrationTesting

Computer ProgramDefinition & Design

Computer ProgramImplementation

System Definition& Design

Computer ProgramTesting

Code

Unit Test

Element CPIntegration & Test

OperationalTests

System Demo &System Qual Test

System T&E,MEIT

EngineeringTest & Evaluation

Comp ProgramDetailed Design

ComputerProgram Design

CP PerformanceReqts Definition

OperationalReqts Definition

SystemReqts Definition

ElementReqts Definition

FB AB Development PhasesSystem Requirements Analysis/DesignSoftware Requirements AnalysisPreliminary DesignDetailed DesignCoding & CSU TestingCSC Integration & TestingCSCI TestingSystem Integration & Testing

DDR

IPR

CR

SDR

PDR

CDR

TPR

MRP

CPAP

PRIMEResponsibility

CSCResponsibility

HDR

Software Development Process

3/16/2007 9:17:48 AM 7

Multi-Processor Environments

• Legacy– AN/UYKs

• COTS Processors– Single Board Computers (SBCs)– Symetric Multi-Processors (SMPs)

AN/UYK-43

COTS ProcessorCabinet

OperatorConsole

UserDisplayConsole

3/16/2007 9:17:48 AM 8

Multi-Operating Environments• Legacy

– Aegis Tactical Executive System (ATES)• COTS

– Concurrent Powermax– Sun Solaris– Red Hawk Linux– LynxOS

3/16/2007 9:17:48 AM 9

Multi-Computer Programming Languages

• Legacy– Compiler Monitor System -2 (CMS-2)

• New– Ada– C/C++– Java

C/C++

3/16/2007 9:17:48 AM 10

Systems Integration, Test and Performance Measurement

3/16/2007 9:17:48 AM 11

Process Improvement

• Incremental Systems Integration, Test and Performance Measurement during the software development process

– Improve System Stability– Early Identification and Resolution of issues and defects– Significantly reduce Engineering Test & Evaluation failure rates– Processes and Procedures QA reviewed and approved

CSC DMEI DE CMMI Level 5

3/16/2007 9:17:48 AM 12

PB

System IntegrationTesting

Computer ProgramDefinition & Design

Computer ProgramImplementation

System Definition& Design

Computer ProgramTesting

Code

Unit Test

Element CPIntegration & Test

OperationalTests

System Demo &System Qual Test

System T&E,MEIT

EngineeringTest & Evaluation

Comp ProgramDetailed Design

ComputerProgram Design

CP PerformanceReqts Definition

OperationalReqts Definition

SystemReqts Definition

ElementReqts Definition

FB AB Development PhasesSystem Requirements Analysis/DesignSoftware Requirements AnalysisPreliminary DesignDetailed DesignCoding & CSU TestingCSC Integration & TestingCSCI TestingSystem Integration & Testing

DDR

IPR

CR

SDR

PDR

CDR

TPR

MRP

CPAP

PRIMEResponsibility

CSCResponsibility

Software Development Process

HDR

Systems Integration,Test and Performance Measurement

3/16/2007 9:17:48 AM 13

Systems Integration

• Validate and maintain operability of system hardware and operating environments

• Verify Computer Program Interfaces

Hardware, OE, Computer Program Interface issues resolved during the computer program development phase

3/16/2007 9:17:48 AM 14

Systems Integration continued• Integration issues tracking and reporting

– Integration Issue resolution prior to computer program delivery

0195271Totals

0163753Build 5

031518Build 4

BUILDS

ACTIVE ISSUES BEING

ADDRESSED

DEFECTS WHICH

BECAME TORs

ISSUES CLOSED

DURING BUILD

BUILD ISSUES FOUND

SYSTEMS INTEGRATION & TEST

3/16/2007 9:17:48 AM 15

Systems Test

• Development and Regression Functional Testing– Documentation

• Plans• Procedures

– Execution• Utilizing two or more computer programs

– Multiple hardware and operating environments

5% to 10% Improvement of Engineering Test & Evaluation

3/16/2007 9:17:48 AM 16

Systems Test continued• Test defect reporting and tracking

– Test defect resolution prior to computer program delivery

S y s t e m I n t e g r a t i o n & T e s t

0

2

4

6

8

10

12

14

C a p a b i l i t y

L O W

M E D I U M

H I G H

3/16/2007 9:17:48 AM 17

Systems Performance Measurement

• CPU and Memory Utilization• Thread and Response Timing• Measurement Tools

– Legacy• ATES Data Recording

– COTS• UNIX TOP and Kernel Trace• Concurrent Nightview• LynxOS Spyker

• Standard scenario– Function and information loading– Repeatable

3/16/2007 9:17:48 AM 18

Systems Performance Measurement continued• Performance Measurement Abnormalities

– CPU and Memory Utilization Increase– CPU Utilization Spikes– Memory Leaks– Slow Thread or Response

3/16/2007 9:17:48 AM 19

Systems Performance Measurement – CPU UtilizationU s e r / S y s e m C P U U t i l i z a t i o n

0 . 0 0

0 . 5 0

1 . 0 0

1 . 5 0

2 . 0 0

2 . 5 0

3 . 0 0

3 . 5 0

T i m e ( A u g - 2 9 - R u n )

S y s

U s e r

U s e r + S y s

3/16/2007 9:17:48 AM 20

Systems Performance Measurement – Memory UtilizationU s e r P r o c e s s M e m o r y

0

1 0

2 0

3 0

4 0

5 0

6 0

7 0

8 0

9 0

1 0 0

9 :4 6 : 3 2

9 :4 8 : 3 7

9 :5 0 : 4 2

9 :5 2 : 4 8

9 :5 4 : 5 3

9 :5 6 : 5 8

9 :5 9 : 0 31 0 : 0 1 :0 91 0 : 0 3 :1 4

1 0 : 0 5 :1 91 0 : 0 7 :2 4

1 0 : 0 9 :3 01 0 : 1 1 :3 5

T im e ( A u g - 2 9 - R u n )

Mem

ory

(Mem

) j a v a

t i m e s e r v i c e s

s y s te m c o n t r o l

3/16/2007 9:17:48 AM 21

Systems Performance Measurement – Memory LeakF r e e M e m o r y

1 7 0

1 8 0

1 9 0

2 0 0

2 1 0

2 2 0

2 3 0

2 4 0

2 5 0

2 6 0

2 7 0

9 :4 6 : 3 2

9 :4 8 : 3 7

9 :5 0 : 4 2

9 :5 2 : 4 8

9 :5 4 : 5 3

9 :5 6 : 5 8

9 :5 9 : 0 3

1 0 : 0 1 :0 9

1 0 : 0 3 :1 4

1 0 : 0 5 :1 9

1 0 : 0 7 :2 4

1 0 : 0 9 :3 0

1 0 : 1 1 :3 5

T im e ( A u g - 2 9 - R u n )

Mem

ory

(Meg

)

P 1

3/16/2007 9:17:48 AM 22

Systems Performance Measurement continued

• Performance Measurement issues resolved before computer program delivery

10% to 20% CPU Utilization Improvement

3/16/2007 9:17:48 AM 23

Summary

• Incremental Systems Integration, Test and Performance Measurement

• Integration and Test issue and defect resolution during the computer program development phase

• Improved Computer Systems Stability and Performance

Cost Effective by identifying and resolving systems issues and defects during the computer program development phase

3/16/2007 9:17:48 AM 24

Tom SobieralskiProject ManagerComputer Sciences Corporation304 West Route 38Moorestown, New Jersey 08057Voice: (856) 252-5052Email: tsobiera@csc.com