annual systems engineering conference october 26, 2016 ... · ndia 19th annual systems engineering...
TRANSCRIPT
NDIA 19th Annual Systems Engineering
Conference
October 26, 2016
Raymond D. O’Toole, Jr.
Dissertation Topic
[Department of Engineering
Management and Systems
Engineering - School of
Engineering and Applied Science]
Doctoral Research Advisors:
Dr. Michael Grenn & Dr. Blake Roberts
My research will answer the following question: When will
the cumulative effects of Departures from Specifications
result in an increase to the already defined Likelihood or
Consequence or both using the Risk Management
Framework?
My hypothesis is that when the analysis of impact from
multiple (i.e. cumulative effects) Departures from
Specifications is completed, the Likelihood and/or
Consequence to the system will be increased resulting in a
higher level of overall risk being identified.
Research Topic
Deviation/Waiver
Major Departure from Specification is one that affects:
performance;
durability;
reliability or maintainability;
interchangeability; effective use or operation;
weight or appearance (where a factor);
health or safety;
system design parameters such as schematics, flow, pressures, or
temperatures; or
compartment arrangements or assigned function.
Minor Departure from Specification. Not a major DFS.
Departure from Specifications
Systems Engineering Life Cycle
Systems engineering is concerned with the overall process of defining, developing, operating, maintaining, and ultimately
replacing quality systems. http://www.incose.org/educationcareers/careersinsystemseng.aspx
From the INCOSE Systems Engineering Handbook Version 3.2.2 and 4
From the Defense Acquisition Guide Book Figure 3.1.2.F1. Illustrative Program Life Cycle
Simple Definition (Merriam-Webster's Dictionary): The possibility that something bad or unpleasant (such as an injury or a loss) will
happen
Someone or something that may cause something bad or unpleasant to happen
A person or thing that someone judges to be a good or bad choice for insurance, a
loan, etc.
Risk Analysis: Support decision analysis
Enables us to take quantities (certain & uncertain) into account and determine to what
extent specific events or scenarios can be expected to occur in the future
Provides a basis for comparing alternative concepts, actions or system configurations
under uncertainty
International Organization for Standardization (ISO) 3100:2009
offers no guidance on aggregation, although it says you could
consider more than one risk at a time.
Risk
Risk Management Frameworks
Acceptable Levels of Risks • Personally • Socially • Nationally • Locally • Economically
New Construction
In S
erv
ice
Deg
rad
atio
n
Aggregate Effect Zone -
events that degrade the risk
margin are not constant.
In Service -
Maintenance Standards
Operational Time
Margin held by Hdqtrs
Local Tech Authority
Major Non-Conformance
Minor Non-Conformance
Maintenance Criteria Limits
Original Design Criteria
Significant Catastrophe!
Effect of Deviation on Margin of Risk
Deviation – must understand the aggregate effect of potential events to
protect against “Normalization of Deviation”
Reviewed over 5600 journal article abstracts/titles on the following terms:
“Departures from Specifications”
“Engineering Departures from Specifications”
“Risk and Standards”
“Specifications and Standards”, with and without ‘risk’
“Specifications and Departures”
“Operations and Maintenance” (GAO only)
Preliminary Results
Risk is still an inexact science
Risk based decisions are as much based on experience as the data available
Risk-informed decision making has been used in various industries
Economics play an important part as one must weigh the reduction of risk in terms of
monetary needs to achieve the reduced risk
.
Literature Background
Various analysis approaches:
System – deep dive
Ship – Ship – Model
Ship Class – Ship Class
Test for Normality of data – various methods are under
consideration – ANOVA, Non-parametrics
Hypothesis: Ha: Risk > CumDFSs; Ho: Risk = CumDFSs
Performance Factors under consideration (e.g.
safety/number of injuries or breakdowns, time out of service,
etc.)
Research Approach
Expert Judgment
Qualitative - vs. - Quantitative
Elicit methodologies – panel discussions, interviews,
group settings, etc.
Data Base
eDFS
Other data sources
Experts from U.S. Navy and non - U.S. Navy – looking
for volunteers
Potential Data Collection
[1] Aven T. On the new ISO guide on risk management terminology. Reliab Eng Syst Saf 2011;96:719–26. doi:10.1016/j.ress.2010.12.020.
[2] Aven T. Selective critique of risk assessments with recommendations for improving methodology and practise. Reliab Eng Syst Saf 2011;96:509–
14. doi:10.1016/j.ress.2010.12.021.
[3] Aven T. The risk concept-historical and recent development trends. Reliab Eng Syst Saf 2012;99:33–44. doi:10.1016/j.ress.2011.11.006.
[4] Aven T. Practical implications of the new risk perspectives. Reliab Eng Syst Saf 2013;115:136–45. doi:10.1016/j.ress.2013.02.020.
[5] Cooke RM, Goossens LLHJ. TU Delft expert judgment data base. Reliab Eng Syst Saf 2008;93:657–74. doi:10.1016/j.ress.2007.03.005.
[6] General specifications for machinery for vessels of the United States Navy covering work under cognizance of Bureau of Steam Engineering. Govt
Print Off, 1920.
https://books.google.com/books?id=goJQAAAAYAAJ&pg=PA326&lpg=PA326&dq=departures+from+specifications&source=bl&ots=GUBsxsxbBH
&sig=QXe9Qxtz0G4e9MgJrLN52GPPwGE&hl=en&sa=X&sqi=2&ved=0ahUKEwiPxb2f1r7PAhVSET4KHYEfAMUQ6AEIVDAJ#v=onepage&q=dep
artures from specifications&f=false (accessed October 1, 2016).
[7] Goossens LHJ, Cooke RM. Expert judgement – Calibration and combination. n.d.
[8] JFMM Vol 5 (COMUSFLTFORCOMINST 4790.3 REV C CH-3) para 8.2.4 n.d.
[9] Keeney RL, Von Winterfeldt D. On the Uses of Expert Judgment on Complex Technical Problems. IEEE Trans Eng Manag 1989;36:83–6.
doi:10.1109/17.18821.
[10] Knight KW. Developing a Risk Management Standard - the Australian experience. Saf Sci 2002;40:69–74. doi:10.1016/j.jallcom.2006.05.115.
[11] Leitch M. ISO 31000:2009 - The new international standard on risk management: Perspective. Risk Anal 2010;30:887–92. doi:10.1111/j.1539-
6924.2010.01397.x.
[12] Merrick JRW, Van Dorp JR, Harrald J, Mazzuchi T, SPAHN, John E GRABOWSKI M. A systems approach to managing oil transportation risk in
Prince William Sound. Syst Eng 2000;3:128–42. doi:10.1002/1520-6858(200033)3:3<128::AID-SYS2>3.3.CO;2-I.
[13] Moore R. Standards in the 21st Century - The Communication of Risk. Med Device Technol 2002:29.
[14] Murphy PH. Acceptable risk as a basis for regulation. Radiographics 1991;11:889–97. doi:10.1016/S0951-8320(97)00135-X.
[15] NASA. Rep Columbia Accid Investig Board, Vol I n.d. https://www.nasa.gov/columbia/home/CAIB_Vol1.html (accessed October 1, 2016).
[16] National Geographic n.d. http://www.nationalgeographic.com/k19/disasters_detail2.html (accessed October 1, 2016).
[17] Nilsen T, Aven T. Models and model uncertainty in the context of risk analysis. Reliab Eng Syst Saf 2003;79:309–17. doi:10.1016/S0951-
8320(02)00239-9.
[18] Pluess D, Groso A, Meyer T. Expert Judfgement in Risk Analysis: A Strategy to Overcome Uncertainities. Chem Eng Trans 2013;31:307–12.
[19] Ryan JJCH, Mazzuchi TA, Ryan DJ, Lopez De La Cruz J, Cooke R. Quantifying information security risks using expert judgment elicitation.
Comput Oper Res 2012;39:774–84. doi:10.1016/j.cor.2010.11.013.
[20] Skjong R, Wentworth B. Expert Judgement and Risk Perception. Det Nor Verit 2000:1–8.
[21] Van Bossuyt DL, Dong A, Tumer IY, Carvalho L. On measuring engineering risk attitudes. J Mech Des 2013;135:1–13. doi:10.1115/1.4025118.
[22] Vatn J. A discussion of the acceptable risk problem. Reliab Eng Syst Saf 1998;61:11–9. doi:10.1016/S0951-8320(97)00061-6.
References
Results of my research may cause a reanalysis of
how risk is characterized for operating systems
Results of research may recommend a new
approach for the Department of the Navy to process
Departure from Specifications (DFSs) once the
system enters the Operating and Support phase of
the system lifecycle
Summary
Mr. Raymond D. O’Toole, Jr. is a native of Hicksville, New York, located on Long Island. He graduated in 1982 from the
State University of New York - Maritime College earning a Bachelor of Engineering in Marine Engineering. He continued
his formal education receiving a Master of Engineering Degree in Systems Engineering in 1989 from Virginia Polytechnic
Institute and State University (VA TECH) and a Master of Science Degree in National Resource Strategy in 2009 from the
Industrial College of the Armed Forces. He is currently pursuing a Ph. D. in Systems Engineering from The George
Washington University.
Mr. O’Toole is an employee of the Naval Sea Systems Command as the Deputy Group Director of Aircraft Carrier Design
and Systems Engineering. Prior to his current assignment he was the Director of Systems Engineering Division
(Submarines and Undersea Systems) where he led a diverse team of engineers who supported all Submarine Program
Managers. His other assignments include being a Ship Design Manager/Navy’s Technical Authority for both the USS
VIRGINIA Class submarine and 54 ships of 14 ship classes with the major ones being all of the Amphibious Ships,
Auxiliary Ships, and Command & Control Ships.
Mr. O’Toole has also held other positions within the Department of Defense such as Deputy Program Executive Officer
(Maritime and Rotary Wing) at the United States Special Operations Acquisition Command, Staff to the Deputy Assistant
Secretary of the Navy for Research, Development & Acquisition (Ship Programs), and Deputy Director of Regional
Maintenance for COMPACFLT (N43).
In addition, Mr. O’Toole has over 30 years of experience as a Naval Officer (Active and Reserve) retiring at the rank of
CAPTAIN. His significant tours include 5 Commanding Officer tours.
Biography
Contact information: [email protected] Phone: (443) 624-1233
Departure from Specifications (DFS):
Department of the Navy definition of a DFS:
Deviation/Waiver
Major Departure from Specification is one that affects (a) performance; (b) durability; (c) reliability
or maintainability; (d) interchangeability; (e) effective use or operation; (f) weight or appearance
(where a factor); (g) health or safety; (h) system design parameters such as schematics, flow,
pressures, or temperatures; or (i) compartment arrangements or assigned function.
Minor Departure from Specification. Not a major DFS.
Simple Definition for my Research: (1) material system(s) that is/are not restored to
design standards and/or (2) software systems that have been degraded by cyber-
attacks by instituting a Boundary Layer Defense (system isolation).
Nonconformance:
Simple Definition from Merriam-Webster's Dictionary: Failure to conform
ISO 9000 2015: Nonconformity is a nonfulfillment or failure to meet a requirement. [A
requirement is a need, expectation, or obligation. It can be stated or implied by an
organization or interested parties.]
Departure from Specifications
- vs –
Nonconformance