tmde and fleet system risk

1MSDJ

TMDE ANDFLEET SYSTEM RISK

TMDE ANDFLEET SYSTEM RISK

Dennis JacksonNSWC Corona Division (MS-20)

7 May 2009

2MSDJ

Overview

• System Specifications

• System Testing and Risk

• System Adjustment and Specifications

• System Specifications

• System Testing and Risk

• System Adjustment and Specifications

3MSDJ

Prime System Traceability

• Program / acquisition managers will provide measurement requirements that are in support of prime systems to the TMDE EA via MIL-STD-1839C CMRS data submitted. -- NAVSEAINST 4734.1B

• Recommended TMDE shall be capable of measuring or generating to a higher accuracy than the measurement parameters being supported. Unless otherwise specified, a minimum Test Uncertainty Ratio (TUR) of 4 to 1 is desired. The actual TUR shall be documented. -- MIL-STD-1839C

• Development procedures shall result in MRCs that ensure system or equipment operation is within performance standards and established readiness criteria. -- MIL-P-24534A

• Program / acquisition managers will provide measurement requirements that are in support of prime systems to the TMDE EA via MIL-STD-1839C CMRS data submitted. -- NAVSEAINST 4734.1B

• Recommended TMDE shall be capable of measuring or generating to a higher accuracy than the measurement parameters being supported. Unless otherwise specified, a minimum Test Uncertainty Ratio (TUR) of 4 to 1 is desired. The actual TUR shall be documented. -- MIL-STD-1839C

• Development procedures shall result in MRCs that ensure system or equipment operation is within performance standards and established readiness criteria. -- MIL-P-24534A

4MSDJ

Test Accuracy Ratio (TAR)

• TAR = The ratio of the system specification to the TMDE tolerance

• TAR is a key driver for test decision risk and cost

• TAR = The ratio of the system specification to the TMDE tolerance

• TAR is a key driver for test decision risk and cost

1001.0

1.0TAR (Expressed as 10:1)

Example:

5MSDJ

Test Uncertainty Ratio (TUR)

• TUR = The ratio of the system specification to the TMDE uncertainty

• Since the TMDE tolerance is often assumed to be the uncertainty, TAR and TUR are often used as equivalent.

• The new ANSI/NCSLI Z540.3 standard defines TUR as the ratio of the TMDE tolerance to the 95% Calibration Process uncertainty.

• The Z540.3 provides a less ambiguous definition.

• The Z540.3 definition applies to calibration rather than system testing.

• TUR = The ratio of the system specification to the TMDE uncertainty

• Since the TMDE tolerance is often assumed to be the uncertainty, TAR and TUR are often used as equivalent.

• The new ANSI/NCSLI Z540.3 standard defines TUR as the ratio of the TMDE tolerance to the 95% Calibration Process uncertainty.

• The Z540.3 provides a less ambiguous definition.

• The Z540.3 definition applies to calibration rather than system testing.

6MSDJ

Example System

• Controls the loading and arming of the selected weapon

• Launches the weapon

• Provides terminal guidance for AAW (Anti-Air Warfare) missiles

• Controls the target illumination for the terminal guidance of SM-2

Mk 99 Missile Fire Control System

9MSDJ

Specification Closeup

10

MSDJ

Specifications

• An MRC directs testing with a TMDE to ensure a system parameter is within a specification.

• For example:

– Ensure voltage measured is 27.0 V (± 2 V)

– Ensure voltage measured is 28.0 (26.6 to 29.4) VDC

• An MRC directs testing with a TMDE to ensure a system parameter is within a specification.

• For example:

– Ensure voltage measured is 27.0 V (± 2 V)

– Ensure voltage measured is 28.0 (26.6 to 29.4) VDC

11

MSDJ

Specification Choice

• What does ± 2.0 V mean?

• At 29.0 V, is performance degraded?


• Specifications should not be determined solely on TMDE accuracy

• What does ± 2.0 V mean?



• Specifications should not be determined solely on TMDE accuracy

12

MSDJ

Utility Curve

Degraded performance

Loss of Utility

Acceptable performance

13

MSDJ

Testing

• Navy systems are tested using test equipment (TMDE)• Navy systems are tested using test equipment (TMDE)

Test Equipment

Navy SystemMeasures

14

MSDJ

SCAT 4245

15

MSDJ

TMDE Index

How do TMDE tolerances relate to testing systems?

16

MSDJ

TMDE Measurement Error

• All Measurements have some error

• TMDE errors need to be small

• Large TMDE errors cause bad test decisions

• All Measurements have some error

• TMDE errors need to be small

• Large TMDE errors cause bad test decisions

Test Equipment

Navy SystemMeasures

• If, for example the test equipment measurement above was off by 1 volt, the system would fail incorrectly

• This would cause unnecessary maintenance

• If, for example the test equipment measurement above was off by 1 volt, the system would fail incorrectly

• This would cause unnecessary maintenance

17

MSDJ

Utility Curve

Uncertainty for TMDE

Measurements tested inside the specification could actually be in the degraded performance region


Loss of Utility


18

MSDJ

MeasurementRisk Events

• False Accept occurs when a parameter is observed through testing by a TMDE to be acceptable, but is actually outside specifications

• False Reject occurs when a parameter is rejected through testing by a TMDE , but is actually inside specifications

• False Accept occurs when a parameter is observed through testing by a TMDE to be acceptable, but is actually outside specifications

• False Reject occurs when a parameter is rejected through testing by a TMDE , but is actually inside specifications

19

MSDJ

Good Test Decisions

20

MSDJ

Risk Consequences

• False Accepts directly harm the Fleet

– Degradation of mission capability

– Failure of mission

– Injury

– Loss of life

• False Rejects indirectly harm the Fleet

– Increased maintenance cost

– Decreased availability

• False Accepts directly harm the Fleet


– Failure of mission

– Injury

– Loss of life

• False Rejects indirectly harm the Fleet

– Increased maintenance cost

– Decreased availability

21

MSDJ

TMDE Measurement Model

TMDESYSTEM

PARAMETER

Measures

TMDE Measurement = Parameter Value + Test Error

• The parameter value is the true output from the system

• The TMDE measurement estimates the parameter value

• The test error is due to inaccuracy in the TMDE as well as in the test setup

22

MSDJ

True In Tolerance

A parameter is truly in tolerance if:

Lower Spec < Parameter Value < Upper Spec

NominalLower Spec (-L) Upper Spec (L)

ParameterValue

23

MSDJ

Observed In Tolerance(Acceptance)

A parameter is observed in tolerance if:

Lower Spec < TMDE Measurement < Upper Spec

0Lower Spec (-L) Upper Spec (L)

TMDEMeasurement

24

MSDJ

False Accepts

TMDE Measurement

Nominal- L L

Parameter Value

TestError

False Accept (FA):

• The TMDE Meas is observed in tolerance [ -L < Meas < L ]

• The Par Value is out of tolerance [ Value > L or Value < -L ]

• The decision to accept the Parameter is incorrect

25

MSDJ

Probability of False Accept

Probability of False Accept (PFA):

PFA = Pr( [Observed In Tolerance] and [True Out Of Tolerance] )

= Pr( [-L < Meas < L] and [Value > L or Value < -L] )PFA is the probability of making an incorrect acceptance decision

TMDE Measurement

Nominal- L L

Parameter Value

TestError

26

MSDJ

True Out Of Tolerance

A parameter is truly out of tolerance if:

Parameter Value < Lower Spec or Parameter Value > Upper Spec


Parameter Value

27

MSDJ

Observed Out Of Tolerance(Rejection)

A parameter is observed out of tolerance if:

TMDE Measurement < Lower Spec or TMDE Measurement > Upper Spec


TMDE Measurement

28

MSDJ

False Rejects

TMDE Measurement

Nominal- L L

Parameter Value Test

Error

False Reject (FR):

• The Par Value is in tolerance [ -L < Value < L ]• The TMDE Meas is observed out of tolerance

[ Meas > L or Meas < -L ]• The decision to reject the Parameter is incorrect

29

MSDJ

Probability of False Reject

Probability of False Reject (PFR):

PFR = Pr( [Observed Out Of Tolerance] and [True In Tolerance] )

= Pr( [Meas > L or Meas < -L] and [-L < Value < L] )PFR is the probability of making an incorrect reject decision

TMDE Measurement

Nominal- L L

Parameter Value Test

Error

30

MSDJ

Inputs Needed to Calculate Risk Probability

• The Specification Limits (-L, L)• The MRC Card

• The Measurement Uncertainty for the Test Process• Generally, this can be considered the TMDE

uncertainty• Should also include test set-up uncertainty

(cables, etc.)

• The Observed Test Parameter Reliability• 3M Data

31

MSDJ

Risk Tool

32

MSDJ

Risk Examples

33

MSDJ

Not Calibrating Costs Money

• Inaccurate test equipment cause bad test decisions

• Wrong test decisions mean unnecessary maintenance

• Maintenance cost is driven by TMDE uncertainty, system spec’s, cal periodicity, and maintenance periodicity

• Inaccurate test equipment cause bad test decisions

• Wrong test decisions mean unnecessary maintenance

• Maintenance cost is driven by TMDE uncertainty, system spec’s, cal periodicity, and maintenance periodicity

34

MSDJ

The Key Is to Minimize Cost

• More calibration increases calibration budget

• Less calibration increases maintenance budget, but allows extended deployment and fewer personnel

• Minimize the total budget (calibration + maintenance)

• More calibration increases calibration budget

• Less calibration increases maintenance budget, but allows extended deployment and fewer personnel

• Minimize the total budget (calibration + maintenance)

35

MSDJ

Testing vs Adjusting

• For some parameters, the TMDE is used to adjust the parameter value rather than testing it

• Often, no specification is given for this situation

• With no specification, there is no basis for choosing the TMDE since TAR is unknown

• The following shows an example.

• For some parameters, the TMDE is used to adjust the parameter value rather than testing it

• Often, no specification is given for this situation

• With no specification, there is no basis for choosing the TMDE since TAR is unknown

• The following shows an example.

36

MSDJ

37

MSDJ

38

MSDJ

Utility Curve

Uncertainty for TMDE

TMDE adjustment accuracy should be 4 times better than the system requirement


Loss of Utility


39

MSDJ

Adjustment and TUR

• Assume TUR is the ratio of the 95% TMDE uncertainty to the system specification

• With a 4:1 TUR, an adjusted parameter has very close to a 0% chance of being out of tolerance after adjustment.

– Assuming a stable system parameter (good repeatability)

• With a 1:1 TUR, an adjusted parameter has about a 5% chance of being out of tolerance after adjustment.

– A 1:1 TUR occurs when the system specification is set to the TMDE tolerance

• Assume TUR is the ratio of the 95% TMDE uncertainty to the system specification

• With a 4:1 TUR, an adjusted parameter has very close to a 0% chance of being out of tolerance after adjustment.

– Assuming a stable system parameter (good repeatability)

• With a 1:1 TUR, an adjusted parameter has about a 5% chance of being out of tolerance after adjustment.

– A 1:1 TUR occurs when the system specification is set to the TMDE tolerance

40

MSDJ

Conclusions

• Specifications are needed for every measurement event requiring a TMDE

• Specifications should be directly related to system performance

• Specifications should answer the question: “Does system performance degrade when the parameter is outside the specification?”

• Specifications are needed for every measurement event requiring a TMDE

• Specifications should be directly related to system performance

• Specifications should answer the question: “Does system performance degrade when the parameter is outside the specification?”

41

MSDJ

Conclusions

• TMDE tests are used to make decisions about Fleet systems.

• Wrong decisions cause consequences to the Fleet:


– Loss of mission

– Injury

– Loss of life

– Unnecessary maintenance cost

– System unavailability

• The probability of wrong decisions can be calculated using risk methods

• TMDE tests are used to make decisions about Fleet systems.

• Wrong decisions cause consequences to the Fleet:


– Loss of mission

– Injury

– Loss of life

– Unnecessary maintenance cost

– System unavailability

• The probability of wrong decisions can be calculated using risk methods

42

MSDJ

Conclusions

• To assess the probability of wrong decisions (PFA and PFR), you need:

– System specifications

– TMDE tolerances/uncertainty

– System reliabilities

• If the system specifications are related to system performance, the risk measures directly relate to system reliability

– Allows assessment of impact of TMDE testing on system performance

– Allows assessment of impact of calibration of TMDE on system performance

• To assess the probability of wrong decisions (PFA and PFR), you need:

– System specifications

– TMDE tolerances/uncertainty

– System reliabilities

• If the system specifications are related to system performance, the risk measures directly relate to system reliability

– Allows assessment of impact of TMDE testing on system performance

– Allows assessment of impact of calibration of TMDE on system performance

tmde and fleet system risk

Documents