presentation from 2007 dinner meeting

8/9/2019 Presentation from 2007 Dinner Meeting

1/34

Metrics in Risk Determinationfor Large-Scale Distributed

Systems MaintenanceMaureen Ann Raley Advisor: Letha Hughes Etzkorn

University of Alabama in Huntsville Computer Science Department

Telephone: 703-325-3510 University of Alabama in Huntsville

[email protected] Telephone: 256-842-6291

[email protected]


2/34

Nationwide Computer System

Upgrade

Upgrade networked computers, distributed throughoutthe continental US

Systems consisted of a mix ofmainframe computersand client/server systems used in a financialenvironment

Lessons-learned survey concentrated on the client

computers or workstations (no mainframes or servers) Data entry centers

Computer centers (in-house software developed and maintained)

Field offices (operations)


3/34


UpgradeCondition: Firm deadline

Non-compliant systems at deadline would be either: Removed from operational use and transferred/disposed

Disconnected from the network and accorded strictlystand-alone status

No transfer of data other than hardcopy printout

Prohibited Sneaker-net transfer by floppy drive, modem, orother electronic means


4/34


UpgradePurpose of the upgrade

Standardize the software and hardware throughout theagency

Modernize the computers and software components

Dispose of obsolete components

Introduce a layered operating system with administrativeand user privileges


5/34


Upgrade

Expected benefits

More conducive to configuration management controls

Easier to maintain and upgrade

Barriers to prevent unauthorized software (games, favorite

programs, instant messaging, peer-peer networking) More secure and less susceptible to external or internal

hacking


6/34


UpgradeSoftware

Predominately COTS Business automation application suites

Operating systems Network

Individual workstations

But also . . . programs developed in-house data entry and data analysis

And . . . in-house customized COTS-based systems


7/34


Upgrade

Hardware

Commercial grade desktop computers

No consistency as to

Age

Manufacturer RAM, HD, I/O, capabilities


8/34


Upgrade

System Priorities

Availability

Data integrity

Performance

Security


9/34


UpgradeLessons Learned

94 personnel interviewed at 12 different locations

Data entry clerks

Inventory control specialists

Information technology personnel

Middle and upper management

Secretaries and administrative assistants.


10/34


UpgradeLessons Learned Questionnaire

1. What were the biggest obstacles you found in

achieving nationwide compliance?2. Based on what you have learned during the

nationwide compliance effort, what would youwant to see done differently if we were starting

the effort now?3. How is training being handled in your

organization? Are there adequate resources? Istime allocated for training?


11/34


UpgradeThe responses note:

1. Management change in direction

2. Lack of understanding in the side effects of managementdirectives

3. Need for bottoms-up input into management decisions

4. Status assessment based on an inaccurate inventory database

5. Understaffing

6. Mismatched arrival of hardware and software7. Unrealistic compatibility testing

8. Loss of functionality in the replacement software

9. Inadequate training


12/34


Upgrade1. & 2.Change in management direction and

inadequate side effect analysis caused rework

Upper management would direct paths to the goals thatwere always not the most reasonable or efficient.

Management failed to identify all the side effects oftheir direction and the implications to the end users.

Management would also direct action, then recall thatdirective, and direct a different approach. This causedfrustration and rework.


13/34


Upgrade3.Lack of bottoms-up input from users

Project office held weekly conference calls on nationwide

status and twice weekly conference calls on issues, as wellas proactive, engaged working groups consisting of thenational office and field coordinators, but . . . Some of these issues were due to management issuing directives

without field input or vetting.

Sometimes field-level management made the decisions for theirend-users without soliciting input.

Ensuring a bottoms up comment process would have helped to

mitigate some of the rework.


14/34


Upgrade1., 2. & 3. A process to vet upper management

decisions could help to reduce the amount of

rework by the end users.Choose a set of end users at diverse locales to beta

test the management directives.

Distribute proposed directives for comment and ensure

both headquarters personnel and field users, reviewedthese proposals.

Direct field management to allot time to end users toensure a meaningful review.


15/34


Upgrade4. Inaccurate inventory database

The IS inventory database was queried weekly to provide reports on

the HW and SW compliance status. Inaccuracies largely due to the small, mobile nature of the

computers and the large numbers of them to track.

Effort to correct IS database was being worked by another office,but did not coincide with the project office milestones.

As the inventory database became more accurate, a realisticestimation of the work done and yet to be done became clearer asthe project deadline approached.

Modern inventory control methods, such as radio frequencyID(RFID) tagging could be used for more accurate tracking of small,easily movable components.


16/34


Upgrade5. Excessive workload, staffing shortages

F

ield coordinators worked the upgrade effort in addition totheir normal workload. Because of this, some field coordinators were not as dedicated to

the upgrade effort as others

Additional staff (skilled support) was used at HQ

Management should also augment staffing levels in thefield.


17/34


Upgrade6. Mismatched arrival of replacementHW/SW

Delays were as long as several months, due to vendor supply shortages

Inadequate storage space for new HW platforms arriving first

Old HW platforms could not be removed without completereplacements

Incompatible or upgrade violation: new SW and old HW; new HW andold SW

Installation of new SW on old incompatible platforms caused systemsfailures and rework for reformatting and reinstallation of old SW.


18/34


Upgrade7.Lack of realistic compatibility testing

Compatibility testing was not independent and had no oversight.

The division overseeing the upgrade did the compatibility testing.

Testing was done on idealized machines with the new COTSsoftware suite at HQ, not in the field, not by field end-users, notwith in-house field application, and not under field workloadconditions.

Most of the software was compatible, BUT some criticalincompatibilities existed with the field applications not used at HQ.

Testing should have followed software engineering bestpractices independent tests, oversight, and under operationalconditions.


19/34


Upgrade8.Loss of functionality in the new software

Transition from one office automation software suite (i.e. Word

Perfect, Lotus, Oracle, email) to another (MS Word, Excel, Access,Outlook).

Data-entry transition: line-entry to graphical, mouse-driven system.

In-house developed replacement programs incurred the "loss of

functionality" complaint less often than COTS software.

In-house programming team came closer to maintaining the basicfunctions of the old in-house software.

"Not all the right features" and "too many unneeded features" issues arecommon to COTS based systems (not tailored to a specific task, butinstead are developed for mass market sales).

A more rigorous COTS selection process could mitigate theinadequacies.


20/34


Upgrade9. Inadequate training

Usually did not address the new skill sets needed. Not always tailored for the different skill levels.

Frequently expected to be self-taught.

Expected to occur in addition to the normal workload.


21/34


Upgrade9. Inadequate training

The most successful training: combination of introductory overview,

followed by hands-on classroom training with a knowledgeable andmotivated instructor.

Unsuccessful training:

Sometimes none at all

Unmotivated, poorly trained instructors

Self-study -- a CD ROM at one's desktop, (ok if computer literate,overwhelming if not)

Information technology and computer specialists adapted with ease.

Non-computer oriented people, such as secretaries, administrativeassistants, and data entry clerks, had much more trouble.


22/34


UpgradeBenefits

Uniform nationwide software and hardware

More effective configuration management Easier maintenance and upgrades installation

Cyclical replacement or upgrades hardware andsoftware easier to plan and effect

Layered operating systems (administrative user

privileges) inhibit installation of ad hoc end usersoftware

End Game Assessment has applications for recoveryfrom hostile information system attacks


23/34

Observations Not all risks are known and can be planned for in

advance

This project mitigated many of the problemsencountered during the transition by:

Consistent monitoring of the hardware and softwarecomponents compliance status using inventory data

An issues database, addressed weekly A risks database for issues with a probability of

occurrence with negative consequences, addressed everytwo weeks


24/34

Possibilities for

Future Research

Quantification of the loss due to rework and theeffect of double or treble responsibilities on the

lower level staff (data not available)

Further investigation on the consequences ofeffective and ineffective management decisions

Development of metrics for risk analysisDevelopment of metrics for risk analysis


25/34


UpgradeActual Compliance Growth from Inventory Data

Compliance Growth -- Field Division

0

20

40

60

80

100

120

1 3 5 7 9 11 1 3 1 5 17 1 9 21 23 2 5 27 2 9 3 1 33 3 5 37 39 4 1 4 3 45 4 7 4 9 51 53 5 5 57 5 9 6 1

Week

Compliance Growth -- IS Division

0.00

20.00

40.00

60.00

80.00

100.00

120.00

1 3 5 7 9 11 1 3 15 1 7 1 9 21 2 3 25 2 7 2 9 31 3 3 35 3 7 3 9 41 4 3 4 5 47 4 9 51 5 3 5 5 57 5 9 61

Week

Fi gur e 1 . P e r c e nt a ge Fi e ld Co m p li ance Gro w th Fi gu r e 2. P e r c e nt a ge IS C o m pl ia nce G row th


26/34


27/34

Metrics Considerations

The determination of an appropriate set of metrics to analyze risk during themaintenance phase of a distributed system upgrade

Standard (actual data), as well as normalized metrics Normalizing would deal with the varying number of devices (with this sample data, the total number of units in

the system changes as new units are added, old ones are disposed of, and as the inventory accuracy grows). Bynormalizing the metric suite, we can compare distributions of different size.

An adaptive sizing model that deals not with the total number of systemdevices (units), but only with the units modified during a certain period

A period of time, i.e., a week

A time-independent period, or threshold, determined by number of devices, i.e. 100. Using this model, insteadof dividing by the actual number of devices in the system, as in the standardized model, would be divided by thenumber of recently modified devices.

A sliding window might used as well.

A history complexity metric for each location (component) to assess the effectof the complexity of the period.

This could help determine if risk increases during bursty or chaotic periods and during periods of high activity.

Validation: Statistical analysis of the results of the actual compliance growth(from the inventory data) vs. the results of the alternative metrics.


28/34

Ta le 1. Soft are etrics Sets Com ariso#

o

t areT& Panel(

T P)

o

t are ngineeringInstitute(

I)

overnment

-P-70-14

-9-70-14

b

ective

1 ost ---------- ----- ost eviations

Trackdevelopmentexpenditures

2

chedule

cheduleProgress

chedule

eviationsTrack

progress vs.schedule

eadiness to

proceed tonext phase

3

omputer

esource

tilization

omputer

esource

tilization

omputer

esource

tilization

Trackplanned v s.actualmemoryutilization,I/

channels,throughput

4

o

t are

ngineering nvironment(

level)

---------------

o

t ar e

evelopmentTools

ssessmento

contractorsdevelopmentenvironment

5

equirementsTraceability

--------------- ---------- ----- Tracesrequirementsto design,code, andtest

6

equirements

tability

o

t areVolatility, nit evelopmentProgress

equirements e

inition &

tability

easureschanges inrequirementsand thee

ect ondevelopment

e

ort7 esign

tability

o

t areVolatility, esign omplexity

esign

tructureTrack designchanges andtheir e

ecto

developmentandcon

igura tion8 omplexity

o

t are

ize esign omplexity

--------------- Provides

indication o

potentialproblemareas heretest e

ort

should be

ocused.

9 Breadth o

Testing

TestingProgress

Test

overage

easures theextent andsuccess o

testcoverage.Providesindication o

su

iciencyo

testing.

10

epth o

Testing

TestingProgress

Test u

iciency

easures thedegree to

hich therequired

unctionalityhas beensuccess

ullydemonstrated

11 FaultPro

iles---------------

e

ect (Fault)

ensity

easures thenumber o

aults vs.time. Tracksopen, closedtroublereports by

priority

12

eliability --------------- ---------------

ttempts to

predictsystemdo ntime bytrackingtroublereports usingmath models

13

anpo er(optional)

Personnel

evelopment

anpo erTracks

personnelloading andturnover

14

evelopmentprogress(optional)

--

esignProgress

!

nitdevelopment

progress

TestingProgress

evelopmentProgress(

ompleteness)

Providespercentageo

developmentcompletion

across allphases and

orkelements

15 Incremental

elease

ontent

--------------- Tracksschedule andunits perrelease totrack

unctionalpreservation

16upportability Indicates

ho easy/di

icultthe so

t ar e

ill be tomaintain


29/34

OneP

ossibility - Entropy Metrics Essentials of mathematical theory of information - A

Mathematical Theory of Communication, Claude Shannon(1949)

Established fundamental bounds on the performance ofcommunication systems in the presence of noise

Exerted an enormous influence on many disciplines --communications, biology, mathematics, physics.

Entropy has been defined in terms of the information content of

software and used to measure code complexity Also has been used effectively as an indicator for reusability


30/34

Entropy Metrics

Have been used successfully to:

Measure SW quality during SW development

Used directed graphs to model a SW system and measure coupling, cohesion,size, length, complexity at module level (Allen, Khoshgoftaar, et al, 1996 -present)

Measured complexity in object-oriented design (Davis, Etzkorn, Bansiya,Gholston, et al, 1999 - present)

Measure SW complexity during SW maintenance

Based on message flowbetween modules, extended to COTS (Chapin, 1988)

Measure Cost Growth during large-scale system development

Queried experts to develop a cost model validated to 3% accuracy vice 300%predicted (Martin, Lenz, Glover, et al, 1981)

Measure SW complexity using process entropy (not code)

Theorized the number of times a module was modified adversely affected codecomplexity, validated 13%-45% improvement (Hassan Holt, 2003)


31/34


32/34

B

ackup Slides


33/34


34/34

Shannons EquationC.E. Shannon, in A Mathematical Theory ofCommunication, (1948) proposed to measure the amount

of uncertainty, or entropy, in a distribution by the followingequation:

n

Hn(P) = -7 (pk* log2pk)

k=1

n

where pku 0, k 1, 2, . . . n and 7pk= 1

k=1

presentation from 2007 dinner meeting

Documents