presentation

BAE SYSTEMS IN CONFIDENCE

Failure to Transition

Continued Airworthiness in the context ofSoftware Maintenance and Support

of AP-3C Weapon System


Topics

• Introduction• Revision of previous analysis on DMO Category of Support requirements• Revision on Technical Frameworks for SM&S• DT&E and IT&E across the Transitional boundary• Supportability analysis of Software, SAE Supportability Concepts et al• Program v Project Risk Management – who pays, who is left holding the bag


Introduction


Introduction

• The Contractor Support Facility (CSF) supplying Software Maintenance and Support (SM&S) services for the AP-3C Weapon System is housed in the Integration, Test and Training Facility (ITTF) at RAAF Edinburgh.

• The 3rd Party Non-Associative Software Support Agency (SSA) working within the ITTF is utilising capability delivered into the CSF under Project AIR 5276 to provide Indigenous Support of the AP-3C Weapon System.

• The Support Concept out of Project AIR 5276 included requirements for DMO Category Support Level 3 which is described by the AEO for the SSA as:

– Fault Rectifications, and– Minor Enhancements

• Project AIR 5276 was pre-TAMM which now pushes Continued Airworthiness requirements onto SM&S activities. Some latitude is taken to interpret this also as Mission Worthiness and Business Worthiness as the SSA has to deal with Mission Criticality and Business Criticality in addition to Technical Airworthiness (X-Worthiness).


Revision of previous analysis on DMO Category of Support requirements


DMO Category of Support requirements overview

• The Category of Support (CATSUP) is a DMO vehicle for specifying, at a high level, the degree of capability and the extent of the capacity required to provide software support.

• Four categories [1..4].

• Main Capability themes of the Category of Support model are:– Control and maintain configuration;– Validate installation;– Provide data;– Provide capability to modify software;– Provide capability to analyse, integrate and test modifications to software;

and– Monitor or predict performance.


Themes allocated per Category of Support level

• The inauguration of capability maps to the CATSUP levels in the following manner:

– Category 1• A) Control and maintain configuration• B) Validate Installation• C) Provide data

– Category 2• D) Provide capability to modify software• E) Provide capability to analyse, integrate and test modifications to

software– Category 3

• F) Monitor Performance– Category 4

• G) Predict Performance


Example aggregations of themes under CATSUP levels

• For example Category 2 will provide the following:– A) Control and maintain configuration (from CATSUP 1)– B) Validate Installation (from CATSUP 1)– C) Provide data (from CATSUP 1)– D) Provide capability to modify software– E) Provide capability to analyse, integrate and test modifications to

software

• While Category 3 would provide the following– Category 3

• A) Control and maintain configuration (from CATSUP 1)• B) Validate Installation (from CATSUP 1)• C) Provide data (from CATSUP 1)• D) Provide capability to modify software (from CATSUP 2)• E) Provide capability to analyse, integrate and test modifications to

software (from CATSUP 2)• F) Monitor Performance


CATSUP provided in CSF by CoA

• The table above captures the exec summary of coarse audit of SM&S capability owned by the CoA within the CSF.

• The most notable omission is Column F or Monitor Performance (Column G is not relevant as the CSF was supposedly targeting CATSUP 3). This is a relatively large omission given it is the core capability that distinguishes CATSUP 3 from CATSUP 2.

• Note the impact of the lack of a Performance Monitoring framework may be a Capability Capping of the CSF to CATSUP 2 which is colloquially Fault Corrections rather than Minor Enhancements.

Mission System CATSUP

1 2 3 4

A B C D E F G

Intercommunications Subsystem See later slides

Navigation Subsystem

Radar Subsystem

Acoustic Processing Subsystem

Acoustic Trainer Subsystem

Data Management Subsystem


Capability verses Ability

• To tackle the confusion that reigns between Capability and Ability:

– The SSA has the Ability to do, say, performance monitoring because engineers can (through best endeavours) exercise their engineering prowess. They have the Ability to use the equipment to carry out performance monitoring because it can run software, they can instrument software, they can collate data, they can analyse data.

– The Performance Monitoring Capability would have been infrastructure, plans, tools, data, issue management etc; delivered into the CSF as part of the technical frameworks.

• The scope of Technical Framework artefacts will include the following artefacts:– Planning artefacts.– Specification artefacts.– Construction artefacts.– Execution artefacts.– Analysis artefacts.– Reporting artefacts.– Problem resolution artefacts.

• No Technical Framework resembling system or sub-system level Performance Monitoring exists with the SM&S capability owned by the CoA nor has the SSA been tasked to develop such a Capability. Some tools exist and there are reporting artefacts out of Project AIR 5276 coffers but no substantive framework to support impact assessment to the system or software architecture of modifications.


Circular Warrantee problem

• The Circular Warrantee problem arises when CoA requires the SSA to warrant delivery of CATSUP 3 services utilising Capability supplied CoA owned CSF.

– The problem for the SSA arises when the CoA warrantee does not stand up to close scrutiny, or the CoA do not warrant the GFM.

– Note, GFI is not literally warranted but is bundled under the implied warrantee that the CSF is fit for purpose.

• Using the effects of Capability Capping (and the missing Performance Monitoring framework as an example) there are risks to Continued Airworthiness/Mission Worthiness/Business Worthiness (X-worthiness) if modifications to the Weapon System affect software or system architecture.

• Continued X-worthiness may be defended by capping modifications to Minor Enhancements or below (untenable); or tasks may be required to redeem Capability (requires non-Operational funding stream); or terminate Indigenous Support and pass back to OEM (NAV, ACO, RADAR).


TAMM Category of Support requirements in comparison

• Rectification may be interpreted as a deliberate or an accidental capping of Development through capping of Capability.

• Note absences of “Full design disclosure and IPR” clause against Rectification.

Rectification. Capability to analyse and rectify faults. The necessary tools to make a software change shall be included, however, testing and other related activities may require the direct use of aircraft.

Development. A full development and test capability similar in scope to the initial development environment, that can produce significant software changes to the system. The capability to do off-aircraft DT&E and provide simulated stimulus for testing purposes shall be provided. Full design disclosure and IPR shall also be provided.

Enhancement. A full development environment with additional simulation, modelling, and analysis tools that would allow significant enhancement of system operation. The development and test environment shall allow analysis of actual system performance using appropriate tools and techniques. The testing facilities shall enable environment testing of real time performance where appropriate to the systems. Full design disclosure and IPR shall also be provided.


Revision on Technical Frameworks for SM&S


Scope covered by Technical Frameworks

• The scope of AEO for the AP-3C Weapon System SSA currently is the following (with Resultant Change Capability and CATSUP mapping):

– Fault Correction ( ≡ Rectification or CATSUP 2)– Minor Enhancement ( ≡ Development or CATSUP 3)– Integration Support ( ≡ Rectification or CATSUP 2)

• Note no Enhancement or CATSUP 4

• The technical frameworks required to support this scope are the following:– System Software Qualification Framework– Software Integration Test Framework– CSCI Test Framework– Unit Test Framework– Software Construction Framework


Goals of Technical Frameworks

• Recapping: the scope of technical framework artefacts to support these goals will include the following artefacts at each level:– Planning artefacts.– Specification artefacts.– Construction artefacts.– Execution artefacts.– Analysis artefacts.– Reporting artefacts.– Problem resolution artefacts.

Technical Framework Components Rectification (Integration Support and Fault Correction) Goals

Development (Minor Enhancement) Goals

System Software Qualification Framework

REPRODUCTIONProblem reproduced at System Interface?

QUALIFICATIONDid we build the right thing?

Software Integration Test Framework LOCALISATIONOffending CSCI discovered?

ASSEMBLYDid the system components go together

correctly?

CSCI Test Framework CHARACTERISATIONBehaviour of offending CSCI determined?

CHECKOUTIs the component sound?

Unit Test Framework INSTRUMENTATIONWhat does it look like under the microscope?

VERIFICATIONDid we build it right?

Software Construction Framework RESOLUTIONConstruct solution!

RESOLUTIONConstruct solution!


Exec Summary of coarse audit of Technical Frameworks at CSF

AP3C Weapon System - Mission System

A B C D E

Intercommunications Subsystem SEL

Navigation Subsystem SEL [1]

Radar Subsystem SEL RSDL [2]

Acoustic Processing Subsystem SEL [3]

Acoustic Trainer Subsystem SEL [4]

Data Management Subsystem SEL [5]

KEY:

A) System Software Qualification FrameworkB) Software Integration Test FrameworkC) CSCI Test FrameworkD) Unit Test FrameworkE) Software Construction FrameworkSEL) Provides Ability, dearth of Technical Framework artefacts related to Software Integration Testing from PA5276

NOTES:

[1] SDE with CSF has less capability than full SDE within OEM.[2] RSDL can be utilised for CSCI and UNIT testing support.[3] OEM utilised separate hardware rigs for CSCI and UNIT testing that were not delivered. The mission box can be utilised for these activities but with restricted access because of SEL resource utilisation clashes and also with reduced throughput due to restrictions in interfaces with box internals. Hardware obsolescence has also broken the capability to connect the mission system to a high speed LAN.[4] OEM utilised separate hardware rigs for CSCI and UNIT testing that were not delivered. The mission box can be utilised for these activities but with restricted access because of SEL resource utilisation clashes and also with reduced throughput due to restrictions in interfaces with box internals. Hardware obsolescence has also broken the capability to connect the mission system to a high speed LAN.


High level Technical Framework discussion

System Software Qualification Framework

Using the AP-3C as the example, the experience is that system software qualification frameworks are part of sell off to customer so are generally available.

The delivered soft copy test procedures are often out of sync with the as-run test masters but the data is generally available – though not as ‘re-useable’ an asset as some would proffer.

Regardless, the high level qualification data is available.

Software Integration Test Framework

The ITTF SEL is the software integration test facility; however, very few of the other technical framework artefacts related to a software integration test framework has been delivered. The SEL was delivered “assembled” and so much of the data reflects the destination rather than the journey.

CSCI Test Framework, Unit Test Framework

In most cases, the OEM will deliver the mission boxes with at least a serial port and claimed this covers CSCI and UNIT testing.

However, none of the other technical framework artefacts related to CSCI or UNIT testing are generally delivered. So the Ability to run software (test or otherwise) exists as a consequence of having mission equipment, rather than a full

testing framework as a capability. In terms introduced in the reference material, the CSCU or UNIT testing delivered is generally:

• Un-managed• Decentralised• Un-metered

Software Construction Framework

Some issues exist with capacity of software construction frameworks falling short of requirements of the relevant CATSUP. Expansion of software construction frameworks to meet CATSUP capacity requirements is problematic where quality goals related to Portability or Maintainability of the SDE are not met, namely:

• CHANGEABILITY• ADAPTABILITY• INSTALLABILITY• CONFORMANCE• REPLACEABILITYMany of the SDE were delivered turn key without substantive software installation and configuration planning material

that would allow building the capability from vendor media. This coupled with hardware obsolescence (usually realised prior to delivery to the CSF) means that “hardware failures ≡ lost Capability”. This generally makes the process of replicating or expanding the SDE more costly; especially where there may be certification issues.


Impact of fractured Technical Frameworks

• With the shortfall in the technical framework artefacts delivered that address the CSCI and Unit Testing aspects (columns C and D of table), issues for continued certification arise out of discontinuous DT&E and IT&E across the transitional boundary from Developing agency to Maintenance agency (see next slide section below).

• Problems arise from fractured history and missing data (information) related to the journey that the software travelled on its way to delivery.

• This is compounded by being expected to work at a higher, or at least different, certification level without the framework artefacts delivered or the redemption of any gap in the frameworks sponsored by customer.


DT&E and IT&E across the Transitional boundary


Traditional V&V model


V&V Approaches mapped to Technical Frameworks

• The exec summary, continued airworthiness requires DT&E and IT&E, across the transitional boundaries from Developing agency into Maintaining agency and the 3rd Party SSA will generally be disadvantaged unless particular care is place on the supportability requirements on projects to ensure transition of solid DT&E and IT&E frameworks.

Technical Framework Components Development & Maintenance Operation

A B C D E F G H I J K L M

System Software Qualification Framework 6 8 9 9 9

Software Integration Test Framework 1 1 4 5 7 8

CSCI Test Framework 1 1 4 5 7 8

Unit Test Framework 1 5

Software Construction Framework 1 1 1 2 3 4

Key:A) Checklist-based inspectionB) Perspective-based inspectionC) Fagan-based inspectionD) Complexity MeasuresE) Language CompilersF) Design MeasuresG) Path TestingH) Scenario-based TestingI) Module Interface TestingJ) User Interface TestingK) User DiscoveredL) System AdministrationM) Environmental


TAMM Category of Support requirements recapped

• Rectification may be interpreted as a deliberate or an accidental capping of Development through capping of Capability.

• In terms of Technical Frameworks delivered it can be interpreted to mean incomplete or missing Unit, CSCI and Software Integration testing frameworks.

Rectification. Capability to analyse and rectify faults. The necessary tools to make a software change shall be included, however, testing and other related activities may require the direct use of aircraft.

Development. A full development and test capability similar in scope to the initial development environment, that can produce significant software changes to the system. The capability to do off-aircraft DT&E and provide simulated stimulus for testing purposes shall be provided. Full design disclosure and IPR shall also be provided.

Enhancement. A full development environment with additional simulation, modelling, and analysis tools that would allow significant enhancement of system operation. The development and test environment shall allow analysis of actual system performance using appropriate tools and techniques. The testing facilities shall enable environment testing of real time performance where appropriate to the systems. Full design disclosure and IPR shall also be provided.


Supportability analysis of Software, SAE Supportability Concepts et al


Support Analysis for Software (SAS) Tailoring

• Support Analysis for Software (SAS) has been devised as a consistent methodology that seeks the achievement of system and software supportability throughout requirements, specification and design, in order to define the most cost effective support concept that meets the operational requirements, and to ensure that the necessary support infrastructure is in place before the system enters into service.

• The goals of Support Analysis for Software (SAS) are the following:

– Evaluate sustainment capability

– Demonstrate expansion and growth capacity

– Verify logistic management of software

– Address operational support of software

– Support Concept planning

• Such a process (support concept planning and execution) has therefore to:

– determine those (support concept) requirements

– influence (product) design so that supportability is built into the product

– establish the Support Concept and ensure it is implemented be itself validated


SAS – Specification, Assessment, Qualification

• It is convenient, in order to ensure software supportability, to frame the management of software supportability around two key components:

– Software Supportability Plan: As part of the System Supportability Plan, it describes the activities to be undertaken in order to achieve the software supportability objectives. It also describes activities to be undertaken to demonstrate achievement of those objectives

– Software Supportability Case: A written documentation about how product supportability was verified/developed at each stage of software development as per the SW Supportability Plan

• These two elements are described in detail the SAE suite of Support Concept Standards:

– JA 1004 Software Supportability Program Standard– JA 1005 Software Supportability Program Implementation Guide– JA 1006 Software Support Concept


SAS – Full scope of Specification, Assessment, Qualification

• Overall Functional Architecture• Functional architecture at LRU Level• Functional architecture of related ground systems• Expansion and growth capacity information for each LRU• Expansion of data buses• Product Information• Identification Data• Management Data• Process Information• Software Loadable Units • Problem investigation capability• Operational support of Mission, Engineering and Diagnostic Data• Maintenance support indicators• CSCI inherent characteristics• CSCI Maintenance Support Resources• System-Level maintenance support• Proposed maintenance support options• Life-Cycle costs• Documentation Available


CATSUP Tailoring of SAS


Technical Framework Tailoring of SAS


SAS4SSA – Vehicle for assessing Supportability

• Thus, the modified SAS, coupled with the audit tool for technical frameworks, and the CATSUP tailoring, attempts to provide both a depth and a breadth of support concept expression (covering Specification, Assessment and Qualification) for elements such as the data requirements and the quality of enabling sets (combined as Capability) of utility to the SSA (colloquially SAS4SSA).

• Certainly, in the light of changing AP-3C Support Concept requirements, the SSA needs a basis for:

– providing guidance to customer and projects in and around support concept specification and sustainment goals; and

– vetting (assessment of) delivered products for sustainability (including continuing certification).

• In the hands of the SSA, SAS4SSA is a Assessment tool for developing Sustainment risk models based on a Supportability taxonomy.

• In the hands of the Acquirer it provides an approach to for Specification and Qualification of a Support Concept.


Program v Project Risk Management – who pays, who is left holding the bag


History repeats - tomorrow and tomorrow and tomorrow …

• The root cause of sustainment problems is as much right sizing the support concept, but it is also specifying the support concept deep enough to ensure goals such as maintenance of the certification continuum are assured.

• The CATSUP approach to support concept specification has tended to be too high a level of specification and has allowed quite wide ranging interpretations by OEM and Prime Contractors.

• The previous paper by the author was a working document which was part of a SSA driven reaction to the support concept fracturing seen in the AP-3C.

• Various mechanisms are culpable; the one that the author is trying to address is the lack of software sustainment/maintenance framework understanding – by stakeholders on both sides of the contract.


… and tomorrow

• Issues in an around the AP-3C enabling sets, and the quality of the technical frameworks, create a tension between the requirements of the regulatory body of the SGTA and the users of 92WG – with the SSA caught in the middle.

• The aging capability is under tension to “grow” from a small software maintenance and support organisation into a new sustainment organisation – but appears to be capped by legacy issues.

• Much of the discussion focuses on models gleaned from standards and best practice -all the models being developed by industry as part of the failure mode analysis of software intensive system sustainment.

• Continued Airworthiness (X-Worthiness) is, like many of the other goals of the AP-3C Weapon System sustainment organisation, impacted by risks injected from outside the Sustainment organisation.


Program Level Risk Management?

• Sustainment of a weapon system requires a program level risk management model.

• Someone Else’s Problem (SEP): Development Project view of Sustainment Issues.

• The techniques to manage the risk fall into one or more of these four major categories:

– Avoidance (eliminate) – Reduction (mitigate) – Transference (outsource or insure) – Retention (accept and budget)


Someone Else’s Problem

• Def: SEP– Support concepts defined at too high a level. – Possibly too small a support facility established. – Too little in the way of significant work passing through the doors to develop

and maintain system comprehension or program understanding in a changing resource pool.

– Fracturing of AIC by CoA going directly to OEM. – OEM dis-engagement, particularly where OEM is being driven by poor

market to product line approaches, rather than managing independent requirements of un-coordinated acquirers.

– Un-warranted GFM provided to SSA. – Un-financed changes in support concept. Obsolete enabling toolsets

supporting obsolete technology. Un-financed expectations to back fit regulatory frameworks requirements.


DoD speaks …

• DoD touts 80% of a system’s cost life is in Operation and Support.

• No stats appear to exist to account for the contribution of SEP to that cost.

• Best practice approaches offer intuitive notions of cost savings if certain practices are avoided.

• As mentioned in the preceding discussion, none of this is new. The standards embodied industry best practice yet lesson learned are un-learned.

• The concern is that 80% of the sustainment budget needs to include the fat in schedules for additional V&V effort to cover unwarranted GFM in the CSF; and additional fat in schedules for system comprehension tasks due to shortfalls in the quality of the GFM and lack of tool support to recover information about the system; aimed to back fit the regulatory frameworks.

• This is the risk avoided by projects realised.


Finale

• The imperative however, as always, is to avoid injecting the issues into Sustainment to allow the SSA to provide Development services up to CATSUP 3, where required, and to maintain Continued Airworthiness in that context.

• Extant systems and their sustainment issues aside, continued Airworthiness amongst them, with the expectation of future programs within the ADF, can we ever learn to better spend that 80%?


Summary

• Revision of previous analysis on DMO Category of Support requirements• Revision on Technical Frameworks for SM&S• DT&E and IT&E across the Transitional boundary• Supportability analysis of Software, SAE Supportability Concepts et al• Program v Project Risk Management – who pays, who is left holding the bag


Questions

presentation

Documents

software category

category of support

data category

category of support

software e

catsup levels

category of support

dmo category support