ContractNo.HQ0034-13-D-0004 TaskOrder:48,118,141,157ReportNo.SERC-2017-TR-101

TransformingSystemsEngineeringthroughModel-CentricEngineering

TechnicalReportSERC-2017-TR-101January18,2017

PrincipalInvestigator

Dr.MarkBlackburn,StevensInstituteofTechnology

ResearchTeam

Mr.RogerBlake,StevensInstituteofTechnology

Dr.MaryBone,StevensInstituteofTechnology

Dr.PaulGrogan,StevensInstituteofTechnology

Dr.DevaHenry,StevensInstituteofTechnology

Dr.StevenHoffenson,StevensInstituteofTechnology

Dr.RussellPeak,GeorgiaTech

Mr.StephenEdwards,GeorgiaTech

Dr.MarkAustin,UniversityofMaryland

Dr.LeonardPetnga,UniversityofMaryland

Sponsor

NAVAIR,DASD(SE)

ii

Copyright©2017StevensInstituteofTechnology,SystemsEngineeringResearchCenterThismaterial is based uponwork supported, inwhole or in part, by theU.S.Department ofDefensethroughtheSystemsEngineeringResearchCenter(SERC)underContractH98230-08-D-0171(TaskOrder041, RT 157). SERC is a federally funded University Affiliated Research Center managed by StevensInstituteofTechnologyAnyopinions,findingsandconclusionsorrecommendationsexpressedinthismaterialarethoseoftheauthor(s)anddonotnecessarilyreflecttheviewsoftheUnitedStatesDepartmentofDefense.NOWARRANTYTHISSTEVENSINSTITUTEOFTECHNOLOGYANDSYSTEMSENGINEERINGRESEARCHCENTERMATERIALISFURNISHEDONAN“AS-IS”BASIS.STEVENSINSTITUTEOFTECHNOLOGYMAKESNOWARRANTIESOFANYKIND,EITHEREXPRESSEDORIMPLIED,ASTOANYMATTERINCLUDING,BUTNOTLIMITEDTO,WARRANTYOFFITNESSFORPURPOSEORMERCHANTABILITY,EXCLUSIVITY,ORRESULTSOBTAINEDFROMUSEOFTHEMATERIAL.STEVENSINSTITUTEOFTECHNOLOGYDOESNOTMAKEANYWARRANTYOFANYKINDWITHRESPECTTOFREEDOMFROMPATENT,TRADEMARK,ORCOPYRIGHTINFRINGEMENT.Thismaterialhasbeenapprovedforpublicreleaseandunlimiteddistribution.

iii

TableofContents1 Introduction.................................................................................................................................4

1.1 Objectives............................................................................................................................81.2 Scope...................................................................................................................................81.3 OrganizationofDocument.................................................................................................11

2 ResearchSummary.....................................................................................................................122.1 Background:CharacterizingProblemandVision................................................................122.2 SystemEngineeringTransformation(SET)–PerspectivesonClarifyingtheFocus...............182.3 Goal-DrivenPlan................................................................................................................192.4 WorkingSessionsandSponsor-SupportingEvents.............................................................212.5 OtherDeliverables.............................................................................................................232.6 ExpandedScopeUnderRT-170...........................................................................................24

2.6.1 RT-170Task1:MissionEngineeringandAnalysisusingMDAOMethods............................252.6.2 RT-170Task2:DecisionFrameworkrelatedtoCross-DomainIntegration..........................262.6.3 RT-170Task3:MethodsforIntegratedDigital/CollaborationEnvironment........................262.6.4 RT-170Task4-UpdateSystemEngineeringTransformationRoadmap–Task4................26

3 Task1–ModelCross-DomainIntegrationwithunderlyingSingleSourceofTruth(SST).............273.1 InformationModelforaSingleSourceofTechnicalTruth..................................................273.2 RequirementOntologyStatus............................................................................................29

4 Task2–ModelIntegrity–developingandaccessingtrustinmodelandsimulationpredictions.30

5 Task3–ModelingMethodologies..............................................................................................325.1 ModelingExamplesOverview............................................................................................325.2 Multi-disciplinaryDesignAnalysis&Optimization.............................................................33

5.2.1 MDAOMethods....................................................................................................................345.2.2 IntegrationswithRelatedTasks............................................................................................355.2.3 MDAOUAVExample.............................................................................................................36

5.3 ModelingExamplesUsingSysML........................................................................................375.3.1 TableofContents.................................................................................................................385.3.2 Process/Methods..................................................................................................................385.3.3 PackageHierarchyforStructuringandOrganizingModelInformation...............................415.3.4 MissionLevelModels...........................................................................................................425.3.5 Systemlevelmodels.............................................................................................................465.3.6 ActivityDiagramofDaveCohen’sFrameworkProcess........................................................49

5.4 ViewsandViewpoints........................................................................................................505.5 CapabilityandOperational-LevelModelingGuidelines......................................................515.6 NAVAIRStudyViews..........................................................................................................515.7 ModelingandMethodsforUncertaintyQuantification......................................................52

5.7.1 DakotaSensitivityAnalysisandUncertaintyQuantification(UQ)........................................535.7.2 AnOverviewofQuantificationofMarginsandUncertainty................................................55

5.8 ModelingMethodsforRisk................................................................................................575.8.1 PredictiveModelsforRisk....................................................................................................57

5.9 ControlledNaturalLanguageRequirementsinformation...................................................585.10 Cross-DomainIntegrationandNaturalLanguageProcessofRequirementsusingOntologies 59

6 Task4–DefineSystemEngineeringTransformationRoadmap...................................................61

iv

7 SERCResearchSynergies............................................................................................................627.1 RT-141IntegratedFrameworkforRiskIdentificationandManagement.............................627.2 RT-168DecisionFramework...............................................................................................637.3 RT-176VerificationandValidation(V&V)ofSystemBehaviorSpecifications.....................647.4 AerospaceIndustryAssociationCONOPSforMBSECollaboration......................................64

8 PartIISummary.........................................................................................................................65

9 AcronymsandAbbreviation.......................................................................................................66

10 Trademarks................................................................................................................................69

11 References.................................................................................................................................71

v

FiguresFigure1.SETransformation“Rollout”Strategy..........................................................................................3

Figure2.SETransformationPhaseII............................................................................................................4

Figure3.SETransformationResearchAreas(SERC)....................................................................................5

Figure4.ProposedFrameworkforNewOperationalParadigmforAcquisitionandDesign.......................7

Figure5.CharacterizestheBoundaryofModelsbetweenGovernmentandIndustry................................9

Figure6.SETActivityViews........................................................................................................................11

Figure7.IntegratedEnvironmentforIterativeTradespaceAnalysisofProblemandDesignSpace.........13

Figure8.DynamicCONOPSIntegratedwithMissionSimulations.............................................................14

Figure 9. Multidisciplinary Design, Analysis and Optimization Supports Tradespace Analysis AcrossDisciplines..........................................................................................................................................15

Figure10.IntegrateMultipleLevelsofSystemModelswithDiscipline-SpecificDesigns..........................16

Figure11.AppropriateMethodsNeededAcrossDomains........................................................................17

Figure12.NeedforObtainingDigitalInformationAcrosstheDomains....................................................18

Figure13.ConceptualPOAMRelatedtoISEE,SST,andMDAO.................................................................21

Figure14.TraceabilityandScopeofDataCollectionofMCERelevantTopics..........................................25

Figure15.IntegratedDataObjectsPartialEntityRelationalDiagram.......................................................28

Figure16.AssociationtoRequirements....................................................................................................29

Figure17.OntologyandRequirementManagerEnginePrototype...........................................................30

Figure18.MDAOExampleWorkflow.........................................................................................................36

Figure19.Paretofrontier(Paretooptimalset)ShowsTrade-offBetweenRangeandPropulsion...........37

Figure20.SensitivityofObjectivestoDesignVariables.............................................................................37

Figure21.TableofContentstoModelsandDiagrams..............................................................................38

Figure22.Pre-modelingGuidelines...........................................................................................................39

Figure23.ContainmentStructure..............................................................................................................40

Figure24.SimpleMBSEActivityDiagramwithLinktoMDAO...................................................................41

Figure25.ModelOrganization...................................................................................................................42

Figure26.High-LevelMissionUseCase.....................................................................................................43

Figure27.TextualElementoftheUseCase...............................................................................................44

Figure28.SurveillanceSystemDomainDiagram.......................................................................................45

Figure29.Mission-levelActivityDiagramwithSwimLanePartitions.......................................................46

Figure30.GenericUAVUseCaseDiagramwithActors.............................................................................47

Figure31.StateMachineDiagramofTop-LevelUAVOperationalStates.................................................47

vi

Figure32.Fixed-WingRefuelingUAVExtensiontoUAVPortfolio.............................................................48

Figure33.ParametricDiagramofFuelSystem..........................................................................................49

Figure34.CameoSimulationToolkitVerifiesConstraintsRepresentingNumericRequirements.............49

Figure35.DraftActivityDiagramofSETransformationFramework.........................................................50

Figure36.Viewpoint..................................................................................................................................51

Figure37.MissionContextforSystemCapability......................................................................................52

Figure38.DakotaFrameworkIntegrationWrapsUserApplication..........................................................54

Figure39.ExampleforUnderstandingMarginsandUncertainty..............................................................54

Figure40.PullingTogetherConceptAssociatedwithQMU......................................................................57

Figure41.BayesianModelDerivedfromAirworthinessFactors...............................................................58

Figure42.DecisionSupportModelConstruct...........................................................................................64

vii

TablesTable1.2016RT-157PlanObjectives,ActionsandMilestones.................................................................20

viii

AcknowledgmentsWewish toacknowledge thegreatsupportof theNAVAIRandSERCsponsors, includingstakeholdersfrom other industry partners that have been very helpful and open about the challenges andopportunitiesofthispromisingapproachtotransformsystemsengineering.

WewanttospecificallythankDaveCohenwhoestablishedthevisionforthisproject,andourimmediateNAVAIRteam,JaimeGuerrero,DavidMeiser,ChrisOwen,JeffSmallwood,MichaelGaydar,JamesLightandSandyNevillwhohasworkedcloselyonaweeklybasis inhelping tocollaboratively researchthiseffort.

We also want to thank all, currently more than 220 stakeholders that participated in over 30organizational discussion and 27 working session, and many follow-up sessions supporting the newSystemEngineeringTransformation.Therearesomanycontributor,supportersanddirectstakeholdersthat supported this effort, we wish to recognize them all. Please see our prior report for earliercontributors.Wesincerelyapologizeifwehavemissedanyoneelsethathassupportedourefforts.

BillBrickner Eric(Tre´)Johnsen JasonThomas PhilomenaZimmermanBrandiGertsner FranChamberlain JohnMcKeown RichardYatesBrianNolan GaryStrauss JohnQuartuccio RonCarlsonBrentGordon GaryWitus KeithCarter ScottLuceroDavidFields GeetheshKukkala LanceHernandez ShahramBavaniDennisReed JaePfeffer MeganClifford StuYoungDineshVerma JamesCarrol NathanielBarkley TomBlakely

1

ResearchTeamThefollowingisalistoftheresearchersandcontributor,andtheiraffiliations.SomeoftheseresearchersaresupportingRT-157and/orRT-170.We includedall researcherscontributingtothesetworesearchtasksonthisreport.

Affiliation ResearcherandAuthors

StevensInstituteofTechnology MarkBlackburn(PI)

RogerBlake

MaryBone

PaulGrogan

DevaHenry

StevenHoffenson

Studentresearchers

GeorgiaTechUniversity RussellPeak

StevenEdwards

DimitriMavris

MarlinBallard

UniversityofMaryland MarkAustin

LeonardPetnga

MariaCoelho

TedCarney

2

ExecutiveSummaryThisisthefinaltechnicalreportoftheSystemsEngineeringResearchCenter(SERC)researchtaskRT-157.This research task (RT) addresses research needs extending prior efforts under RT-48/118/141 thatinformed us thatmodel-centric engineering (MCE) is in use and adoption seems to be accelerating.Model-centric engineering1 can be characterized as an overarching digital engineering approach thatintegratesdifferentmodeltypeswithsimulations,surrogates,systemsandcomponentsatdifferentlevelsofabstractionandfidelityacrossdisciplinesthroughoutthelifecycle.Industryistrendingtowardsmoreintegrationof computational capabilities,models, software,hardware,platforms,andhumans-in-the-loop.The integratedperspectivesprovidecross-domainviews for rapidsystem levelanalysisallowingengineersfromvariousdisciplinesusingdynamicmodelsandsurrogatestosupportcontinuousandoftenvirtualverificationandvalidationfortradespacedecisionsinthefaceofchangingmissionneeds.

NAVAIRseniorleadershipconfirmedinlate2015thattheresearchfindingsandanalysisvalidatedtheirvision hypothesis stated at the System Engineering Transformation kickoff meeting of RT-48. TheyconcludedthatNAVAIRmustmovequicklytokeeppacewiththeotherorganizationsthathaveadoptedMCEandwhocontinuetoevolveatanacceleratingpaceenabledbytheadvancesincomputationalandmodelingtechnologies,andimprovedmethods.

InMarch of 2016, therewas a Change of Command at AIR 4.0 (Research and Engineering). NAVAIRdecidedtoacceleratetheSystemsEngineeringTransformation(SET).The“rollout”strategyisalayeredapproachwhere evolving research needs are provided by SERC research, as shown in Figure 1. Thisresearch provides analyses intoNAVAIR enterprise capability, and builds on efforts for cross-domainmodelintegrationandmodelintegrity(perRT-157).NAVAIRalsoextendedtheRT-157researchunderRT-170toaddresstheevolvingSETneedsandpriorities.

Thepathforwardhaschallengesbutalsomanyopportunities,bothtechnicalandsociotechnical.Itmustincludeamodelingframeworkwithhighperformancecomputing(HPC)thatenablessinglesourceoftruth(SST), integration of multi-domain and multi-physics models, and provides for a method for modelintegrity.ThemodelingandinfrastructureforadigitalengineeringenvironmentisacriticalsteptoenableaSST.Whilethereareliterallythousandsoftools2,theyareoftenfederatedandthereisnoonesinglesolutionthatcanbepurchased.EveryorganizationprovidinginputstothisresearchhashadtoarchitectandengineertheirMCEenvironment.Mostorganizationusecommercialtools,butalsohavedevelopedtheintegratingfabricbetweenthedifferenttools,models,simulationsanddata.Someorganizationshaveencodedhistoricalknowledgeinreferencemodels,modelpatternstoembedmethodologicalguidanceto support continuous orchestration of analysis through new modeling metrics, and automatedworkflows.NAVAIRismakingstridestodevelopanIntegratedModelingEnvironment(IME)thatcapturesrequirementstolinkartifactsandevidenceinsupportofdecision-makingaddressingallrequiredchecks

1DASDhasincreasedtheemphasisonusingthetermDigitalEngineering.AdraftdefinitionprovidedbytheDefenseAcquisitionUniversity(DAU)forDEis:Anintegrateddigitalapproachthatusesauthoritativesourcesofsystems'dataandmodelsasacontinuumacrossdisciplinestosupportlifecycleactivitiesfromconceptthroughdisposal.ThisdefinitionissimilartoworkingdefinitionusedthroughoutourpriorresearchtaskRT-48/118/141forModelCentricEngineering(MCE).2Certaincommercialsoftwareproductsareidentifiedinthismaterial.Theseproductswereusedonlyfordemonstrationpurposes.ThisusedoesnotimplyapprovalorendorsementbyStevens,SERC,orNAVIAR,nordoesitimplytheseproductsarenecessarilythebestavailableforthepurpose.Otherproductnames,companynames,images,ornamesofplatformsreferencedhereinmaybetrademarksorregisteredtrademarksoftheirrespectivecompanies,andtheyareusedforidentificationpurposesonly.

3

and risks. Key questions remain as to how to do that in the context of a new operational paradigmbetweengovernmentandindustryusinganewframeworkdescribedherein.

Figure1.SETransformation“Rollout”Strategy

ThekickoffofRT-157inJanuary2016definedaresearchplantoinvestigatechallengeareasincluding,butnotlimitedto:

§ Cross-domainintegrationofmodelstoaddresstheheterogeneityofthevarioustoolsandenvironments

§ Modelintegritytoensuretrustinthemodelpredictionsbyunderstandingandquantifyingmarginsanduncertainty

§ Modelingmethodologiesthatcanembeddemonstratedbestpracticesandprovidecomputationaltechnologiesforreal-timetrainingwithindigitalengineeringenvironments

§ MultidisciplinarySystemEngineeringtransformationroadmapthatlooksacross:o Technologiesandtheirevolutiono Howpeopleinteractthroughdigitallyenabledtechnologiesandnewneededcompetencieso Howmethodologiesenabledbytechnologieschangeandsubsumeprocesseso Howacquisitionorganizationsandindustryoperateinadigitalengineeringenvironment

throughoutthephasesofthelifecycle(includingoperationsandsustainment)o Governancewithinthisnewdigitalandcontinuallyadaptingenvironment

ThestrategicplansofSETandoverarchinggoalsofthisresearchhavebeenexpandedthroughRT-170.RT-170hassupportfromnewresearchcollaboratorsfromGeorgiaTechandUniversityofMaryland.ThisreportdoesblendRT-170-relatedaccomplishmentsintothisreporttodocumenttheongoingprogressinsupportoftheNAVAIRSET.Finally,wearealsoworkingcollaborativelywithUSArmyRDECOM-ARDECinPicatinny,NJunderRT-168,andsomeoftheplansforsynergiesbetweentheseeffortsaredocumentedinthisreport.

4

1 INTRODUCTION

In2013,NavalAirSystemsCommand(NAVAIR)attheNavalAirStation,PatuxentRiver,Marylandinitiatedresearch into a Vision held by NAVAIR’s leadership to assess the technical feasibility of a radicaltransformation through a more holistic model-centric system engineering (MCSE) approach. Theexpected capability of such an approach would enable mission-based analysis and engineering thatreducesthetypical timebyat least25percentfromwhat isachievedtodayfor large-scaleairvehiclesystems.Theresearchneedincludedtheevaluationofemergingsystemdesignthroughcomputer(i.e.,digital)models.

Through Systems Engineering Research Center (SERC) research tasks (RT-48, 118, 141) there wasconsiderable emphasis on understanding the state-of-the-art through discussions with industry,government and academia [21] [31]. The team comprised of both NAVAIR and SERC researchersconductedover30discussions,including21onsite,aswellasseveralfollow-updiscussionsonsomeoftheidentifiedchallengeareasandapproachesforanewoperationalparadigmbetweengovernmentandindustry.

In 2015, theNAVAIR leadership concluded that theymustmovequickly to keeppacewith theotherorganizations that have adopted MCE as the pace of evolution is accelerating by the enablingtechnologies.NAVAIRmadethedecisiontopress forwardwithaSystemsEngineeringTransformation(SET).ThateffortwasstartedinJanuaryof2016underRT-157andhadfourtasksasshowninFigure2:

§ Task1–ModelCross-DomainIntegrationwithunderlyingSingleSourceofTechnicalTruth(SST)§ Task2–ModelIntegrity–developingandaccessingtrustinmodelandsimulationpredictions§ Task3–ModelingMethodologiesaligningwiththerolloutoftechnologiesdefinedunderTask4§ Task4–DefineSystemEngineeringTransformationRoadmap

Figure2.SETransformationPhaseII

InMarch of 2016, therewas a Change of Command at AIR 4.0 (Research and Engineering). NAVAIRdecidedtoacceleratetheSET.NotionallyasshowninFigure1,theSEThasalayeredapproachwherethe

5

needed researchprovides analyses intoNAVAIR enterprise capability, but builds on efforts for cross-domainmodel integrationandmodel integrity (perRT-157).While theSERCresearchwasdirectedtofocusontheProgramofRecord(POR)/systemslevel,anewNAVAIRstrategyforacceleratingcapabilitydeliverytothewarfighterislookingtobetterassessthevalueandrisksofsystemandsystemofsystems(SoS) capabilities, potentially distributed across platforms tomission and campaign needs in amoredynamicallychangingenvironment.Therefore,NAVAIRbelievesthatthefollowingareasarecandidatesforSERCresearchascharacterizedinRT-170,whicharea layerontopoftheotherdimensionsoftheresearch,asshowninFigure3:

§ PrioritizationandTrade-offAnalysis§ ConceptEngineering§ Architecture&DesignAnalysis§ Design&TestReuseandSynthesis§ ActiveSystemCharacterization§ Human-SystemIntegration

Figure3.SETransformationResearchAreas(SERC)

DuringtheexecutionofRT-157,oursponsorDaveCohenproposedanewoperationalframework,whichisshowninFigure4.Thisevolvingframeworkisbeingassessedandrefinedinordertosupportanewoperationalparadigmtomissionengineering,analysisandacquisition,whichwouldbe ledbyNAVAIRwithacollaborativedesigneffort ledby industry.Wearealso involved in industrymeetingswithoursponsortoassistinunderstandingconceptsforanewtypeofcollaboration,andtoassesstheimpactsontheNAVAIRenterprise,frombothatechnicalandsocio-technicalperspective.

Agile Process Engineering

Prio

ritiz

atio

n an

d

Tr

ade-

off A

naly

sis

Con

cept

Eng

inee

ring

Arc

hite

ctur

e &

Des

ign

Ana

lysi

s

Des

ign

& T

est R

euse

and

Sy

nthe

sis

Act

ive

Syst

em

Cha

ract

eriz

atio

n

Hum

an-S

yste

m

Inte

grat

ion

Modeling Environment Infrastructure

Modeling Methodology at NAVAIR

SET Roadmap

Cross-Domain Model Integration

Model Integrity

6

BrieflytheconceptofthenewSETframeworkfortransformingfromadocument-centricprocesswithmonolithicreviewstoanevent-drivenmodel-centricapproachinvolves,butisnotlimitedto:

§ AconceptforcollaborativeinvolvementbetweenGovernmentandIndustrytoassessmissionandSystemofSystems(SoS)capabilityanalyses,whereNAVAIRhastheleadto:o InvolveindustryinSoScapabilitiesassessmentsduringmission-levelanalysis(tothedegree

possible)o Iterativelyperformtradespaceanalysesofthemissioncapabilitiesusingapproachessuchas

MultidisciplinaryDesign,AnalysisandOptimization(MDAO)asameanstodevelopandverifyamodel-basedspecification

o Synthesizeanengineeringconceptsystemmodelcharacterizedasamodel-centricspecificationandassociatedcontractualmechanismbasedonmodelsorassociatedformalism

§ Atthecontractualboundaries,industrywillleadaprocesstosatisfytheconceptualmodeladdressingtheKeySystemAttributes(KSAs)3,withparticularfocusonPerformance,Availability,Affordability,andAirworthinesstocreateanInitialBalancedDesigno IndustrytooappliesMDAOatthesystemandsubsystemlevelo Thereisapotentialneedtoiteratebacktore-balancetheneedsifthetradespaceanalyses

ofthesolution/systemfortheprogramofrecord(POR)cannotachievemission-levelobjectives

o Allrequirementsaretradeableiftheydon’taddvaluetothemission-levelKSAso TheseareasynchronousactivitiesincreatinganInitialBalancedDesigno GovernmentandIndustrymustworktogethertoassess“digitalevidence”and“production

feasibility”

3WehavebeeninformedthatKeySystemAttributesarebeingsubstitutedforKeyPerformanceParametersintheJointCapabilitiesIntegrationandDevelopmentSystem(JCIDS).

7

Figure4.ProposedFrameworkforNewOperationalParadigmforAcquisitionandDesign

Anotherobjectiveunderconsiderationinthecontextoftheoperationalmodelistoreplacelarge-scaledocument-centricreviewssuchasSystemsRequirementsReview(SRR),SystemFunctionalReview(SFR),PreliminaryDesign Review (PDR), etc.with continual event-driven reviews using objective evaluationbasedonmodel-centricinformation.NAVAIRneedssometypeofobjectivedecisionframeworktoassessevolvingdesignmaturitywithconsiderationsofvaluetotheKSAs,riskanduncertainty.Thisframework’soperationalconcepthasproducedsomespecificresearchquestionssuchas:

§ Howdoes/cangovernmentparticipateinthesecontinualevent-drivenandobjectiveevaluationsteps?

§ Howtojudgeevolvingmaturityofdesign?§ Howcangovernmentinteracttoprovidevaluewithoutimpedance?

TheseeffortsforimprovingthecollaborationbetweengovernmentandindustryarealsounderwaytodevelopanewCONOPSofoperationswithorganizationsinvolvedintheAerospaceIndustryAssociation(AIA)workinggroup[3],andtheNationalDefenseIndustryAssociation(NDIA)ModelingandSimulationgroupwhichislookingatapproachesforusingdigitalengineeringforcompetitivedownselect.Weareinvolvedinalloftheseeffortstofurthertheobjectivesofoursponsor.

ThisreportcoversboththeresearchperformedagainsttheRT-157objectives,andtherequestofoursponsorthathasexpandedtoaddresstheneedsoftheSETunderRT-170,specificallyinthecontextofthenewframework(Figure4).

8

1.1 OBJECTIVES

Asshown inFigure3, thescopeof theseresearch taskareashasexpandedandarecontinuallybeingrealigned to the prioritizes of the SET. Demonstrations and NAVAIR-relevant example models areimportant to supportworkforceneeds.Therefore,weare supporting the researchusinga case studybasedonaconceptualUnmannedAirVehicle(UAV).Thiscasestudyservesasasurrogateforthepilotproject identified inFigure1.Thisexampledirectly supports someof theneeds forTask3 (ModelingMethods),Task4andaddressessomequestionsrelatedtothenewframework.

Weareusingpublicallyavailableinformation[85]toconstructexamplesandreferencemodels,includingSysML,MDAOsystemexamples,andmission-levelCONOPSscenariossuchas:

§ Surveillance§ Refueling§ On-boardUAVrefueling

Thisapproachsupportsaresearchobjectivetoinformcompetencies(Task4)withreferencemodelsandmodelingmethods.Theexamplescanbe“referencemodels”ofmodelingpatternsorsurrogate.Thiscasestudysupportsthethreecross-cuttingcriticalitemsfromRT-157,butextendsittothemission/SoSlevel:

§ Cross-domainandmulti-physicsmodelintegration,andtheassociatedmethodologies§ Technologiestoestablishandquantifymodelintegrity§ HighPerformanceComputing(HPC),whichenablestheprevioustwobulletpoints

AsreflectedinFigure4therearefourelementsofresearchthatrelatetothecriticalenablersthatextendtheRT-157taskasdefinedintheRT-170thatinclude:

§ MissionEngineeringandAnalysisusingMDAOmethodsandapplicableoperational,capability,andsystemmodels

§ Decisionframeworkrelatedtocross-domainintegrationthroughthesinglesourceoftechnicaltruth(SST)o Providesabasisforanobjectiveapproachtoassessdesignmaturitybasedonanontological

representationofthesystemusingopensemanticwebtechnologies§ Integrateddigital/collaborationenvironmentcapabilitiesandoperationalmodelsbothwithin

NAVAIRandindustryo Thisincludesthedevelopmentofmethodsandreferencemodelstoenrichworkforce

understandingofMCEmethods,modelsandtools§ UpdatetheSETRoadmaptoaddresstheprioritizedSETneeds,suchas:

o Newworkforceskillso Integrationofre-engineeredprocesses,competencymanagementandculturalalignmento Communicationswithindustry,andstakeholdermanagemento Environmentevolution,moreanalytic-baseddecisionmakingandplanningo Leadership,governanceandprioritizationo Needstoaddressiterations(currentlythreeplanned)asreflectedinFigure1

1.2 SCOPE

Giventheobjectives,wealignedtheRT-157effortswithsponsorprioritiesrelevanttoSET,whichincludedresearchto:

9

§ Assessthenewframeworkandidentifychallengesandfocusareastodevelopaprioritizationfortheresearcho Identifykeyresearchgivengapsandchallengesofnewconcepto SupportmeetingwithNAVAIRsponsorsandindustryonnewapproachestocollaboration

§ Understandhowrequirementscanberepresentedasmodelso Usethecasestudytohelpshow,inthevariousways,howmodelscanbeusedtosupport

requirements,constraints,validation,andverificationplanningo Wehavedevelopedexamplemodelsforaddressingthisquestionandarelookingtousein

upcomingpilots§ Understandapproachestocompletenessandconsistencyofformalrequirements§ Supportdevelopmentofpilotprojects

Webelievewehaveprovidedinsightsintotheseobjectives,whicharediscussedinmoredetail inthisreport.SomeoftheresultsoftheresearchandcontributionsarereflectedintheDecember2016sponsorbriefing. The latestbriefing articulates theneed for formalizing theuseofmodels, including: level ofmodels,typesofmodels,andtheconceptualboundarybetweengovernmentmodelsasshowninFigure5.Thisconceptreflectsonadraft“SystemModel”thatispartof“requirement”forarequestforproposalthatwouldbeelaboratedbycontractorsduringsourceselectionintoa“FinalSystemModel.”Wewanttosimulatethisconceptduringthepilotsifatallpossible.WebelievewecanuseourevolvingUAVmodelasthe“DraftSystemModel”fortheupcomingsurrogatepilot.

Figure5.CharacterizestheBoundaryofModelsbetweenGovernmentandIndustry

TheongoingSERCresearchshouldsupporttheNAVAIRstrategiesthatincludeplansfor:

10

§ Formaluseofmodelsasastandardpracticeforspecifying,analyzing,designing,andverifyingsystems

§ Systemmodelsadaptedtotheapplicationdomainthatincludeabroadspectrumofdigitalmodelsforrepresentingallaspectsofsystems

§ Theuseofinternet-drivenknowledgerepresentationandimmersivetechnologiesenablehighlyefficientandsharedhumanunderstandingofsystemsinavirtualenvironmentthatspanthefulllifecyclefromconceptthroughdevelopment,manufacturing,operations,andsupport

Figure6providesanotherperspectivereflectingininformationprovidedunderRT-157ontheSSTthatmustincludevarioustypesofcross-disciplinemodels(e.g.,mechanical,electrical,software,testing,etc.)forthedifferentaspectsofthesystemmodel.Itmusttracetomission-levelmodels.MDAOwillplayaroleatthemission,systemandsubsystemlevel.Itmustsupportcross-domainanalysis.AkeyquestioniswhatiscapturedintheSSTthatprovidesinsightintotheevolving/maturingdesigninordertoprovideeffectiveoversight.ThiswillbeworkedunderRT-170,andmaybesupportedby:

§ Pilot/DemonstrationCoordinationwithProgramExecutiveOfficer(PEO)/ProgramManagerAir(PMA)onpilotcandidateso Determinationofidealpilotprogramso SETPilotexecutionplanning

§ CollaborationwithCompetenciestoconsider:o MovingfromContractDataRequirementsLists(CDRLs)toModelso Appropriate“new”ContractLanguageo SystemModelingo Pilots/Demonstration

§ CollaborationwithIndustry(potentialcollaboratorsremoved)§ WorkforceDevelopment

o ModelingMethodologyDefinition–SERC/NavalPostgraduateSchool(NPS)o ModelingTraining

§ SERCModelingResearchpresentedthroughworkingsessionsandreports§ OfficeofSecretaryofDefense(OSD)DigitalEngineeringcoordination§ SERCresearchsynergiesthroughRT-168withUSArmyandRT-176withNPS

11

Figure6.SETActivityViews

1.3 ORGANIZATIONOFDOCUMENT

Section1providesanoverviewofthecontextfortheneededresearch,objectives,expandedscopeandorganizationofthisreport.

Section2providesthesummaryofourefforts, findings,analysesandrecommendations includingkeyaspects fromRT-48/118/141.This sectionalsobriefly summarizes theexpandedscopeofourcurrentresearchunderthenewlyawardedRT-170.

Section3describesapproachandresultsofdevelopinginformationmodelsunderlyingthesinglesourceoftruth,andarequirementontologyandrequirementmanagerprototype.

Section4describestheneedforandapproachestoModelIntegrity–developingandaccessingtrustinmodelandsimulationpredictions.

Section 5 describes themodelingmethodologies, including examples and demonstrations created toillustrate mission, system, enterprise and reference models, including example and methods forMultidisciplinaryDesign,AnalysisandOptimization.

Section6discussestheSETroadmap.

Section7discussessomesynergiestotheongoingNAVAIRresearchtasksthatarebrieflymentionedinthisreporttoinformreadersoftherelationshipstotheseotheractivities.

Section8providesconclusionswithabriefsummaryoftheplannednextsteps.

12

2 RESEARCHSUMMARY

ThissectionprovidessomecontextintotheconcernsoftheNAVAIRleadershipinmovingforwardwithaSET.SimilartotheapproachusedinRT-48/118/141,thissectionprovidesahigh-levelsummaryoftheresearch,plans,resultsanddeliverables.PartIIofthisreportprovidesadditionaldetailsoneachtask.

2.1 BACKGROUND:CHARACTERIZINGPROBLEMANDVISION

The RT-141 final report [21] generalized the types of capabilities many organizations use inMCE[8][9][10][38][51][71][77][111].Thereportcharacterizesacanonicalreferencearchitectureofan IntegratedMCEEnvironment,asshown inFigure7.The followingprovidessomeperspectivesandcapabilitiesofthisvision:

§ Providesappropriateviewsforthevariousstakeholders§ StakeholdershaveviewsintotheSingleSourceofTruth(SST)

§ Usingrichmodelinginterfacesforthosewithexpertiseinmodeling§ Usingrich“web”interfaces,whichtodayprovidessupportforgraphics,integratedwith

structureinputs,generatedtextualviewsand3Dmodelviewing[115]§ MDAOlayerprovidesforproblemanddesignspaceexplorationof

o Physics-basedmodelso Integrity-basedmodelso Costandschedulingmodelso Riskmodelso Various“illities”models

§ Includessurrogatesandcomponents§ EnabledbyHighPerformanceComputing(HPC)§ SemanticallyrichlinkagesbetweendataandinformationintheSSTprovidesforcontinuous

workfloworchestration–enabledbyHPC§ Documentgenerationisenabledby

o SemanticallyrichlinkstoinformationintheSSTo Templatesthatformalizepatternsforrequirements,contracts,etc.

§ Enablingtechnologiessuchasmachinelearningprovidesavirtualknowledgelibrarianthatassistusersguidedbyembeddingknowledgeandtraining

§ Contractorandcollaboratorshaveasecuremeanstoplugintovieworsharedigitalinformation,asanewparadigmforinteractions

§ ThisviewoftheDesigningSystemprovideslinksdownstreamtofullylinkProductLifecycleManagement(PLM)

13

Figure7.IntegratedEnvironmentforIterativeTradespaceAnalysisofProblemandDesignSpace

Throughour27workingsessions,assummarizedinSection2.4,wecontinuetoinitiatediscussionsaboutwhattheSETisanditsimplicationsmovingforward,alongwithbenefitsandchallenges.WehaveusedthefollowingscenariostosupplementourcharacterizationofafuturevisionstateshowninFigure7.Ithas helped provide additional perspectives moving from a mission-perspective to several systems-perspectives down to specific design disciplines. It also provides a means to identify some of thechallengesassociatedwiththefourtasksofRT-157,butalsootherneedsdefinedinRT-170.

AsshowninFigure8,wehaveseentheuseoftheNet-CentricEvaluationCapabilityModule(NECM)thatusesmodelingandsimulationandaStudyViewsmethodtostructurethedevelopmentofneededmissioncapabilities. These capabilities support some aspects of the Joint Capabilities Integration andDevelopmentSystem(JCIDS)toanalyzejointmissionthreadsinnearreal-timeandautomatenet-readyKeyPerformanceParameter(KPP)analysis.Currentlythisinformation(andothers)areusedasinputtodevelopDepartmentofDefenseArchitecturalFramework(DoDAF)models,whicharefocusedprimarilyonthenet-readycapabilities.However,therearemoreconcernsindevelopingaweaponsystemthanjustthenet-readycapabilities.Therefore,ifwewanttosupportthevisionoftheNAVAIRsponsortohaveadeepersystemandcomponent-levelanalysisasitrelatestothevaluetokeyKSA/KPPsrelativetothemission,weneedbetterintegrationtohigher-levelfidelitymodelsatthesystemlevelsasdescribedinthescenariosbelow.WebelievethereareopportunitiesandbenefitstobetterlinktheNECMcapabilitiesintodynamicsimulationsofbothmissionandsystemcapabilitiestocreatemoredynamicoperationalrepresentationsoftheconceptsofoperation(CONOPS).

Multidiscipline Design,Analysis andOptimization(MDAO)

Single Source of TruthToolAgnostic,SemanticallyPreciseCrossDomainIntegrationandInteroperability

Performance Integrity

SecurePlugin

Cost&Schedule

Systems&Surrogates

Doc.Generation

Appropriate Views for Stakeholders

Knowledge …

WorkflowOrchestration

ComputerAugmentation

PLM

Rich ModelingViews and Visualization “Web” Interface Views

14

Figure8.DynamicCONOPSIntegratedwithMissionSimulations4

ThosedynamiccapabilitiesreflectedinFigure8thatarelinkedintothestudyviewsmaptohigher-levelfidelityviewsrelatedtospecificdesigndisciplinesasshowninFigure9.Atthislevel,wewanttoimprovethetradespacebyusingMDAO.MostorganizationsthatdevelopaircraftsystemshavebeenusingMDAOforover10years,morefocusedatthesystemlevel.Suchcapabilitiesallowfor1000xthenumberofdesignexcursions [59] as has been done with traditional approaches in the past. These types of MDAOapproaches provide for some amount of cross-domain analysis. However, wewant to developmorecomprehensiveapproachesandbesystematicaboutcoveringthetradespaceatthemissionandsystemlevels,atleastforthecriticalKSAs.AllofthemaincontractorstoNAVAIRusethesecapabilities,andweshared publically known informationwith attendees about such usage at ourworking sessions [110].MDAOcanalsobeusedatthemissionlevelasreflectedinFigure4.ThereareopportunitiesinresearchfordevelopingandimprovingMDAOmethods[86].

4Imagecredit:PhoenixIntegration

15

Figure9.MultidisciplinaryDesign,AnalysisandOptimizationSupportsTradespaceAnalysisAcrossDisciplines

Anotherareaofopportunityistoimprovetheintegrationofarchitectural,systemandcomponentmodelsacrossthedomains,andbetterlinkwithothermodelingandsimulationcapabilitiestargetedtospecificdisciplines. These “architectural” models may be developed using DoDAF to characterize missioncapabilitiesandoperationalviews.AtthesystemleveltheymaybedevelopedusingModelBasedSystemEngineering(MBSE)methodsandberepresentedinstandardmodelinglanguagessuchasSysML[104].The linkages between theMBSE anddesign disciplines is often not precisely represented,with a fewexceptions.Whenitisdoneusingtool-to-toolintegration,suchintegrationscanberathersusceptibletoupdatestothetools[34].Webelievethereareopportunitiestoaddressthisneedinmoretoolagnosticwaysusingsemanticwebtechnologies[27][126];thisisoneareaofresearchsupportingcross-domainmodelintegration(Task1).

VehicleDesign

Geometry&Packaging Airframe&

EngineAero

Sensors

Propulsion

Structures

“illites”

Comm/Radar

Store/Payloads

Detailed Design from AssociatedDisciplines and Competencies

MDAO Implements Workflowwith Solvers to Evaluate

Trades Systematically Driven by Design of Experiment

16

Figure10.IntegrateMultipleLevelsofSystemModelswithDiscipline-SpecificDesigns

Thekeyreasonfortheneedforcross-domainmodel integration(Task1) istheunderlyingcomplexityneededtoaccomplishthescenariosassociatedwithFigure9andFigure10.Inaddition,ourresearchasillustratedbytheDARPAMETAproject[9]hasshownthatmethodsareneededtoensurethatthetoolsprovidetheexpectedautomation,efficiencies,andproducethedesiredinformation.Thispointstotheneedforbothmethods(Task3)andbecausemanyofthemodelingandsimulationcapabilitiesthatmaybeintegratedintoanMDAOworkflowcanbemodelingandsimulationcapabilities,theyrequiresometypeofassessmenttoensuretheintegrityofthepredictions(Task2).

Webelievethereareresearchchallengestobetterquantifydesignmargins,parameteruncertainties,andsystemperformancesensitivitiesassociatedwithphysics-baseddigitalmodels.Thereareopportunitiesandchallenges in integrationofrelevantmulti-physicsmodelingandsimulation,needforearlierhigh-fidelitymodels,andmeanstoassessreduced-ordermodels.Inaddition,thereareneedsfordeterminingoptimal risk/cost tradeoff for continual Verification, Validation and Accreditation (VV&A) [55] oralternativemeansforassessingtrustinmodelandsimulationpredictions[120].

AsshowninFigure11[45],therecanbeaverylargesetoftoolsthatcanbeusedtodeveloptheneededdataandinformationacrossallofthedomains5.Therefore,itisimportantthatappropriatemethodsareappliedtotheselectedtoolsthatareassembledforuseonaprojectorprogram.AsasecondaryobjectivethatisbeingdemonstratedasleadingedgeapproachbyNASA/JPListoensuremodelsarecreatedthatcomplywithestablishedmodelingpatterns;theirapproachtransformsthemodelinformationintoatool-neutralSSTbasedonontologies,andthenusesstandardsemanticwebtechnologiestoapplycheckstoensurecompletenessandconsistency[77].Someofourdeliverablesareprovidingthebuildingblocksto

5Forexample,inaninventoryanalysisofmodelingandsimulationtoolsusedatNAVAIR,therewasmorethan300,andweweretoldthatthelistwasincomplete.

17

accomplishthesetypesofchecksandareplannedtobeintegratedintotheNAVAIRIntegratedSystemsEngineeringEnvironment(ISEE)asdiscussedbelow.

Figure11.AppropriateMethodsNeededAcrossDomains

Anotherchallengeunderlyingthereasonfortheneedforcross-domainmodel integration(Task1),asrenderednotionallyinFigure12,isthatthecurrentcompetenciesmaysupportoneormoredomainsandmayoften“buyallthedata,”oftenreferredtoasContractDataRequirementsList(CDRLs);thispracticeisaimedatreducingthepotential futurerisks.Currently,thereare limitedwaysofunderstandingthevalue of that data to themission or the implication of design decisions across the domains.Models,modeling,andcomputationallyenabledconceptssuchastheuseofprecisemodelsthatarelinkedacrossdomains in theSSTshouldprovideameanstobegin tounderstandthevalue, riskanduncertaintyofinformationrelativetothemission.Thisisaspecificobjectivearticulatedbythesponsor.However,thisalsoleadstoanothernewneedinhowdigitalengineeringimpactsthelanguageandinformationthatisput on contract in a statementofwork (SOW)or requests for proposal (RFP). Therefore,we too arelooking at potential approaches this need as that relates to the information that flows across thecontractual boundary (Figure 5). We have an effort to look at approaches to formalize contractuallanguage, and may need to define how a proposal response will be evaluated using more dynamicmodelingandsimulations,anddigitalengineeringinformation.

18

Figure12.NeedforObtainingDigitalInformationAcrosstheDomains

2.2 SYSTEMENGINEERINGTRANSFORMATION(SET)–PERSPECTIVESONCLARIFYINGTHEFOCUS

WewerealsorequestedtocontributetoanefforttocreateanexecutivelevelmessagethatcanbeusedbyseniorleadershiptodescribetheSET.Likemostcomplexenterprisetherearemanystakeholdersandtheyallhavedifferentconcernsandviewsonthesolutionand/orvision.WesharesomeperspectivesfromsomeoftheleadersoftheSEtransformation:

§ SeniorExecutiveforResearchandEngineering-EmphasizesDigitalizationandVirtualizationo AlsodiscussedintermsofBetter,Faster,Cheaper

§ Missionmodelingdirectoremphasizestheabilityformodelstorepresentahigherlevelofabstractionofthesystemincludingthemulti-physicsaspectsinordertogettoanIntegratedTestVehicle(perSETFramework)o Wealsoknowbasedontheframeworkisconcernedwithcontinuouscollaborationbetween

GovernmentandIndustryandevent-drivendecisionmaking§ Systemsengineeringdirectoremphasizestheunderlying"data"(InformationModel)ofthe

integratedinformationthatrepresents"all"aspectsofthesystem,mission,usersandenvironmentintheSSTo ThisperspectiveisaboutSystemsEngineering,includingprinciplesandmethodscarriedout

intermsofmoreprecisemodelsandstandardlanguageso Integratedviewsacrossallofthedomainsincludingrisk,uncertainty,andnewmetricsfor

understanding“designmaturity”(recognizingthatsomeandmaybemostcompetenciesstillliketothinkintheirstovepipes)

o Leveragingcomputationalcapabilitiestoletthecomputerdoworkthatinadocument-centricworldisdoneprimarilybyhumans

19

Thedigitalizationintermsofprecisemodelsallowscomputerstohelpfinddefectstomakethesystem“Better.”Computersalsoprovide themeans toanalyze1000xnumberof trades [59],againbasedonprecisemodelsandtheassociatedsimulationsleadingtoa“Better”design,bothintermsofmulti-physics,butalsointermsofothercostand“ilities”models.

Thedigitalizationintermsofprecisemodelsalsoallowscomputerstodoworkthathasinthepastbeendonebyhumans,allowingworktobedone“Faster.”

The precise digitalization including the increased emphasis on representing the cross-domainrelationships and dependencies associatedwith the competencies and disciplines provides for a SSTwhereworkisevent-driven(“Faster”),allowingforthecontinuousVirtualizationofmeetings(“Faster”)toeliminatethemonolithicandcostlyreviews(“Cheaper”).

As a community, perhaps we should create a Systems Engineering Transformation Manifesto (e.g.,ManifestoforAgileSoftwareDevelopmentisquitewellknown).Amanifestoisawrittenstatementwhereone(oragroup)publiclydeclarestheir:

§ Intentions(whatyou/weintendtodo)o Changetheoperationalparadigmforacquisitionofsystemandsystemofsystems

§ Opinions(whatyou/webelieve;stanceonaparticulartopic)o Wecanoperateinamorecollaborativeparadigmo ComputationallyenabledSystemsEngineerallowsustodealwithcomplexitiesnotpossible

throughtraditionaldocument-centricprocesses(“Better”)o ProcessofmodelsprovidesforearlyValidationofRequirements(“Better”)o ResultingmodelsproducesVerificationthreadstosupportVerificationplanningthatcan

serveasabasisofestimateearlier(“Better”)o Modelsarereusablefromprogram-to-program(“Faster,Cheaper”)

§ Vision(thetypeofworldthatyoudreamaboutandwishtocreate)o SeeFigure7andassociatedcharacterization

TherearemanyorganizationsinDoDthatmightbenefitfromcomingtogetherwithaunifiedvisiononSystemsEngineeringTransformations.

2.3 GOAL-DRIVENPLAN

RT-157 started in February using a goal-oriented method to develop a Plan Objectives, Action andMilestones(POA&M)forthe2016research,withamappingtotheroadmapcategories:(T)Technologies,(M)Methods,(C)Competencies(seeTable1).ThisplanwasdevelopedbeforeSETaccelerationplanwasstarted,asdiscussedinSection1.Thisplanwasbasedonthegoaltoestablisharigorousfoundationfora semantically precise and tool agnostic framework for the SST (Task 1). These detailed tasks arecontributingtotheextensionoftheIntegratedSystemEngineeringEnvironment(ISEE)asshowninFigure13.TherewasalsotheassumptionthattherewouldbeanevolutionaryadoptionofmodelingtoolsandmethodsatNAVAIR,andthefocuswouldstartwithimprovedformalizationofrequirementsthatwouldtracetomodels,risks,andevidence.Notealsothatthereisaroadmaptagmappingtotechnology(T),methods (M), competencies (C).We had no specific tasks that address interactionswith contractingorganizationsoranewapproachtogovernance.However,thenewframeworkdoesbringinplansforanewoperationalmodelbetweengovernmentandindustry.Inaddition,thechangeofcommandalteredthisplanimpliessomechangesingovernanceasreflectedbytheframework(Figure4).

20

Table1.2016RT-157PlanObjectives,ActionsandMilestones

ID What Why RoadmapTag

0 SeeGoalsTab Listgeneralgoals,prioritiesanddependencies T,M,C1 Requirements

EngineeringMethodThere are multiple capabilities involved in requirementsengineering, including ontology-driven requirement engineeringtoensureconsistencyandcompletenessofinformationcaptured;anothergoalthatmighttakelongertoimplementistoembedthemethodological guidance in the Requirement managerimplementation

M

1a RequirementsOntology

Characterizesinatoolagnosticwaytheinformationthatmustbecaptured,inadditiontorepresentingtheattributesofconsistencyandcompleteness

T

1b RequirementsManagerToolTrade

Determine the tools that can perform multiple activities ofrequirements engineering from elicitation, traceability, throughversioncontrol,role-basedaccess,security

T,M

1c RequirementManager

ImplementationandintegrationintotheSSEandsinglesourceoftechnicaltruth

T,M

2 RiskMethod The risk method will leverage the formality defined in a riskontologythatintegratesbothwithrequirementsandevidencetosupportarigorriskapproachthat isbasedonevidenceofwork,includingtheincorporationofevidenceprovidedasmodels

M

2a RiskOntology Thekeyclassesfromtheexistingprocessarealreadydefined,butwillbeformalizedintoanontologythatlinkstobothrequirementsandevidencethatwillbestoredintheSST

T

2b EvidenceOntology This is information that is related to the information from thechecklist;thiswillbeformalizedtolinktobothrequirementsandrisk.Thiswillformalizetheriskprocess(e.g.,lowevidenceimplieshighrisk)

T

2c Requirements to RiskMapping

RelaterisktorequirementsT

2d RiskManager ImplementationandintegrationofriskviewpointsandfunctionsintotheSST

T,M

2e Requirements toEvidenceMapping

RelateevidencetorequirementsT

2f Risk to EvidenceMapping

RelaterisktoevidenceT

3 Knowledge This involves the integration of embedded training, examples,modeling patterns, reference models, tradespace analyses;engineersworkwellwithexamples.Thereiscurrentlynotmuchavailable,andimplementationofthisconceptneedstoattempttoembedthemethodologiesintodifferenttools,asisthecurrentlythecasewithSETRmanager

T.M,C

4 MultidisciplinaryDesign, Analysis andOptimization

MDAOisasystematicapproachtotradespaceanalysis.Thistaskin 2016 involves informing people about the concept,methodsandtoolsoptions.

C,M

21

Figure13.ConceptualPOAMRelatedtoISEE,SST,andMDAO

2.4 WORKINGSESSIONSANDSPONSOR-SUPPORTINGEVENTS

AcomponentoftheresearchandrequireddeliverablesareconductingworkingsessionsthatinformtheNAVAIRteamaboutprogressagainsttheplan.TheseworkingsessionalsoinformtheteamaboutrelevantinformationandfeedbacktoscopethedeliverablesinthecontextappropriateforNAVAIR;thishasbeenespecially important given the recent changesunder SET.Wealsousebi-weekly drumbeats to sharestatusandupdates.Eachworkingsessionhasadefinedagenda,anddetailedmeetingnotesareprovidedtooursponsor.Thefollowingprovidesasummaryoftheworkingsessionsandotherevents,andabriefdescriptionofthecontributionstothetasksanddeliverables.

§ Workingsession#18:2/4/2016o Developedthegoal-drivenprioritizationplanforRT-157andconfirmedtheprioritieswith

theNAVAIRteamandsponsorso Discussedtheconceptfordevelopingtheontologyunderlyingtherequirementmanager

(top-levelpriority)§ Workingsession#19:3/3/2016

o PresentedasessiononmethodsforMultidisciplinaryDesign,AnalysisandOptimizationmethodsandtools

o PresentedevidenceandexampleforaconceptbasedonControlledNaturalLanguageRequirementsthatwilllikelybenecessaryasasupplementformodels

§ Workingsession#20:4/7/2016o DiscussedthenewplanoftheSETransformationaccelerationo DiscussedPOA&MfortheRT-157-specificdiscussedinSection2.3o Discussedtheimportanceofmodelingmethodsbeforetoolselection

22

o Discusseduseofreferencemodelsascuratedknowledgethatmayalterthewaytrainingisconducted,andmustbeconsideredinthecreationoftheISEE

o DiscussedtheapproachfordevelopingtheunderlyingInformationModelfortheSSTo Presentedaworkingdemonstrationoftherequirementontologyandrequirementmanager

prototypeandprovidedsoftwareprototypeandontologytoNAVAIR(abonusdeliverable)§ Workingsession#21:5/4/2016

o DiscussedconceptsandimplicationsassociatedwiththeSEtransformationandframework(seeFigure4)

§ Workingsession#22:6/7/2016o DiscussedfirstversionofthenewframeworkandimplicationsofdevelopingaPOA&Mfor

planningnextfewyearsoftheSETthatincludedallofNAVAIRandpilotprojectsasshowninFigure1

o Initiatedthediscussionforfollow-onresearchtasknowinRT-170,whichwasawardedon1-Sep-2016

§ Workingsession#23:7/14/2016o DiscussedunderlyingmeaningoftheSET(seeSection2.2)o Discussedofframeworkneedsandchallenges

• Implicationsonspecificationandcontractingo PresentedconceptualUAVasexamplecasestudywithmodelsforpresentingmodeling

methodso Changeinneedssuchasmodelingexampleso ChangeincollaboratorfromWayneStatetoUniversityofUniversityofMarylandtobetter

supportthenewprioritiesoftheSET§ Workingsession#24:8/8/2016

o Presentedmodelingmethodexamplesanddemonstrations• SysMLatmission,system,andenterprise• Referencemodels• MDAOexampleofaUAVperformanceworkflow• MDAOwebinarsdiscussingusagesbyindustryconfirminguseofMDAObyorganization

thatcontracttoNAVAIR§ Workingsession#25:9/15/2016

o Col.TimWest(PhDStudentofMarkBlackburn)presented–“ADigitalThreadFrameworkforDynamicallyIntegratingExperimentalandComputationalResultswithQuantifiedMarginsandUncertainties”• Topic is highly relevant to the NAVAIR System Engineering Transformation, and

specificallytothechallengeofModelIntegrityo ProvideexamplesandupdatesonUAVmissionandsystemmodelingexamplesand

guidelinesusingSysMLo PresentedMDAOexamples,webinarsanddemonstrationo Discussedconceptson“SynthesisofContract”RequestforProposal(RFP)andstatementof

work(SOW)asnewMCEapproachtosourceselectioninanewparadigm§ Workingsession#26:11/9/2016

o FirstsessionwithGeorgiaTech(Dr.RussellPeak)ResearchModelingUAVo FirstsessionwithUniversityofMaryland(Drs.MarkAustinandLeonardPetnga)Research–

Ontologieso InformationModel/Artifacts/OntologyeffortforIntegratedSystemsEngineering

Environment

23

o DiscussedotherrelatedSERCUpdates,RT-176NavalPostgraduateSchool(NPS),RT-168ArmamentResearch,DevelopmentandEngineeringCenter(ARDEC)

§ Workingsession#27:12/15/2016o UpdateonSysMLModelingofUAVo UseofNaturalLanguageProcessingexperiment(byNPS)andontologiesonStandardWork

Packages(SWP)o CapabilityBasedTestBaseandEvaluationo AirForceSourceSelectionusingModelingandSimulationo MBSE101:Three1-hourmodulesfromNASAAcademyOnlineo InformationModel/Artifacts/OntologyeffortforIntegratedSystemsEngineering

Environment(ISEE)§ IndustryandGovernmentForumonModelCentricEngineering:5/26/2016

o DaveCohenpresentedinformationthatwaspre-frameworko Seewhitepaper[40]onSERCwebsite(http://www.sercuarc.org/wp-

content/uploads/2014/05/MCE-Forum-Final-Report.pdf)§ SERCExecutiveAdvisoryBoard:6/29/2016

o PresentedresultsofNAVAIRresearchresultsandimpactso DaveCohenwasabletopresentearlyversionofhisframework

§ SpecialsessionwithNAVAIRsponsorsatAltairEngineeringinDetroitMI:9/29/2016o ThismeetingwasrequestedspecificallybyDaveCoheno Keytopicsincluded:

• UnderstandingAltair’sviewsontheadvancesincomputationalcapabilitiesforincreasingthespeedofdesign,analysisandoptimizationthroughmodelingandsimulation

• Understanding of the accuracy of multi-physics modeling and simulation predictions(Davereferstothisa‘ModelIntegrity’),basedontheircommercialofferingsandexpertiseinusageofthoseofferings(e.g.,tools)

• Altair’sviewsonadvancementoftheirofferingespeciallyinthepastfouryears,becauseDepartmentofDefense(DoD)organizationsareusingtheDoDComputationalResearchandEngineeringAcquisitionToolsandEnvironments(CREATE)AirVehicle(AV)familyofcomputationaltoolsandhavediscussedsignificantadvances

• Identifyareaswheretherearechallenges,andtodiscussstrategiesthatcanovercomesuchchallengeareas

§ PresentedNAVAIRresearchatSERCSponsorReviewtoOfficeoftheDeputyAssistantSecretaryofDefenseforSystemsEngineering:11/17/2016

2.5 OTHERDELIVERABLES

Wehavebeenproducingmodelsandexamples.ThefollowingprovidesalistofmodelsthathavebeenproducedandsuppliedtoNAVAIR:

§ RequirementontologyandassociatedRequirementEditorthatincludesintegrationswiththerequirementontology

§ SystemEngineeringTechnicalReport(SETR)ontologyderivedfromSETRProcessHandbookVersion1.0Dated06February2015

§ SysMLone-daycoursethatisprovidedbyStevensInstituteofTechnologyfortheSYS-671,672,673,674CyberPhysicalSystemscourse

§ DemonstrationofMDAOworkflowforUAVperformancescenario

24

§ StevenssuppliedUAVSysMLmodelstoGeorgiaTechandNavalPostgraduateSchoolresearchers

§ DemonstrationofSysMLmodelsforsimulationandrequirementverification§ PlanforusingNaturalLanguageProcessingandontologiestore-structureStandardWork

Packages

2.6 EXPANDEDSCOPEUNDERRT-170

Itisacknowledgedthattherearemanypossiblehurdlesbeyondtechnicalfeasibility(e.g.,organizationaladoption,training,usability,etc.),andwehavehadtoadjustourplansandworktoalignwiththenewprioritiestoalignwiththeacceleratedSET.Thepathforwardincludesadjustmentstotheroadmaptofactor in someof theseotherconsiderations.Theconceptproposedby the framework (Figure4)haschangedsomeofourassumptionsabouttheSST.

The actual statement ofwork identified research needs relating to the cross-cutting concerns of thelifecycleandmodelingenvironmentandinfrastructuresuchas:

§ Prioritization&TradeoffAnalysis§ ConceptEngineering§ Architecture&DesignAnalysis§ Design&TestReuse&Synthesis§ ActiveSystemCharacterization§ Human-SystemIntegration§ AgileProcessEngineering(new)

As shown in Figure 14 (columns to the right), these lifecycle perspectives were in the RT-141 finaltechnicalreport,andasshownbythetraceability,theseneedscrossmanyMCErelevanttopics.Allofthesemaybereasonableresearchneeds,butwealignedtheproposedtaskswiththeavailableleveloffundingtothekeyneedsdefinedintheframework(Figure4).Thesealigntheproposedtaskswithourunderstandingofthesponsorspriorities.WebrieflysummarizethenewproposedRT-170tasks.

25

Figure14.TraceabilityandScopeofDataCollectionofMCERelevantTopics

2.6.1 RT-170TASK1:MISSIONENGINEERINGANDANALYSISUSINGMDAOMETHODS

This task investigates factorsrelatingtotherelativevalueandpriorityofhigh-levelcapabilitiesof thesystem under development (some of whichmight be assessed under RT 170 Task 2). It investigatesdynamicrepresentationsofmissionandcampaignanalysisanddefinesmethodsformappingtoMCE-relevant capability representations in contrast to the traditional Capability Development Document(CDD).Thisshouldincludemodelingdifferentviewpointsforcapabilityviews,operationalviewsthatmaptosystemviews.

InthecontextoftheproposedframeworkshowninFigure4andFigure6,theconceptofMDAOarebeingconsiderfromatleastthreepointsofview,suchas:

§ Supportvalidationofthemodel-basedspecificationo Ensurecompletenessbytracingfromtheworkflowstothecapabilitythreadso EnsureanunderstandingoftheboundaryconditionsagainsttheKSAstobound,quantify

riskandsurfacepotentialanomalieso Provideameanstolookattheconstraintsimposedacrossdomainso Providesmeansforsimulatingdynamicbehaviors,spanningmultipleengineeringdomains

tobeabletobalancethetradeoffsofperformance,availability,affordabilityandairworthinessacrossdomains

o Performsensitivityanalysestoassessthevalueorriskofdifferentcapabilitiesacrossdomain/disciplines

o Supportdata-drivendecisionswithengineeringtechnicaldataandinformationthatisproduced,notjustdocuments(supportingRT-170Task2)

§ Captureandorganizeprioranalysisoftradespaceo Supportreuseofdataandanalysistoperformregressionsfortheinevitablesituationsof

evolvingspecificationso Tracetocapabilitythreadsassociatedwithspecification

Discussion(Topics(not(exhaustive) N

ASA

/JPL

A B C Alta

ir

GE

Sand

ia

DARP

A5M

ETA5(V

B)

DARP

A5M

ETA5(B

AE)

Mod

el5Cen

ter

Autom

otive

CREA

TE

Performance

Integrity

Affordability

Risk

Metho

dology

Single5Sou

rce5of5Tech5Truth

Prioritization5&

Tradeo

ff5Analysis

Concep

t5Engineering

Architecture5&

Design5Analysis

Design5&5Test

Reuse5&5Synthesis

Active5System

Characterizatio

n

Hum

anMSystem

Integration

Modeling5CONOPS x x x x x x xModeling5Patterns x x x x x x x x xMultiMPhysics5Modeling5and5Simulation x x x x x x x x x x x x x x xMultiMDiscpline/Domain5Analysis5and5Optimization x x x x x x x x x x x x x x x x x x xMissionMtoMSystemMlevel5Simulation5Integration x x x x x x x x x x xAffordability5Analysis x x x x x x x x x xQuantification5of5Margins x x x x x x x x x x xRequirement5Generation5(from5Models) x x x x x x x xTool5agnostic5digital5representation x x x x x x x x x x xModel5measures5(thru5formal5checks) x x x x x x x x x xModeling5and5Sim5for5Manufacturability x x x x x x x x x x x x x xProcess5Automation5(workflows) x x x x x x xIterative/Agile5use5of5MCE x x x x x x xHigh5Performance5Computing x x x x x x x x x x x x x x xPlatformMbased5and5Surrogates x x x x x x x x3D5Environments5and5Visualization x x x x x x x x x x x x x x x xImmersive5Environments x x x x x xDomainMspecific5modeling5languages5 x x x x x x x x x x x x x x x xSetMbased5design5 x x x x x x x x xModel5validation/qualification/trust x x x x x x x xModeling5Environment5and5Infrastructure x x x x x x x x x x x x x x x x x x x x x x x x

Instances5where5discussed5(not5exhaustive) From5Kickoff5BriefingCharacteristics

26

§ Providemeanstointegratedifferentresources(e.g.,simulations,surrogates)fromdifferentsources(governmentand/orcontractor)

2.6.2 RT-170TASK2:DECISIONFRAMEWORKRELATEDTOCROSS-DOMAININTEGRATION

TheSSTprovidesabasis foranobjectiveapproachtoassessdesignmaturitybasedonanontologicalrepresentationof the systemusing standards-based semanticweb technologies.Thiswillprovide themeansforassessingcompletenessandconsistencyacrossdifferentmodels,developedusingdifferentlanguagesandmethodologies(asreflectedinTask3).Thistaskwillleveragesemanticwebtechnologiesforcreatinganinformationmodeltodemonstrateconceptsforreasoningaboutconceptualmodelsanddesignmodelmaturity,whichistoolneutral.

ThistaskwillextendtheworkwiththerequirementsontologyandinformationmodelderivedfromtheCOREmodelofTier3artifactsdevelopedunderRT-141/157.

2.6.3 RT-170TASK3:METHODSFORINTEGRATEDDIGITAL/COLLABORATIONENVIRONMENT

This task focuses on the methodology transition from document-centric to model-centric in part toenhanceofourunderstanding/analysiscapabilityofthe increasingcomplexity intacticalsystems.Thisspecificallyrelatesto,butisnotlimitedtothemethodsusedtosupportRT-170Task1andRT-170Task2.Wearealsointerestedinmodel-basedalternativestospecificationrepresentationandtheabilityto“generaterequirements”thatwouldleadtoadigitalrepresentationofcontractorinputleadingintoStep5oftheframework.Thisincludesbutisnotlimitedto:

§ MDAOworkflows§ ModelBasedSystemEngineering(MBSE):Operational,Capability,Systemsviews§ ModelBasedEngineering(MBE):Discipline-specific,mechanical,electrical,controls,etc.§ ModelBased“illities”(MBX):Fault-trees,Bayesian,etc.§ Risk/Costmodels

Finally,wealsowanttoplanfortheuseanddevelopmentofmodelpatterns,modelreferenceswithintheir environment to embedded knowledge, and methodological guidance to support continuousorchestration of analysis through modeling metrics, and automated workflow into the integratedenvironment.Thecasestudyshouldproduceexamplemodels,methods,andreferencemodelstoenrichworkforceunderstandingofMCEmethods,modelsandtools.TheseeffortsshouldsupporttheevolutionandexperimentationwiththeIntegratedSystemEngineeringEnvironment(ISEE),anddefinegoalsandrequirementsfortheISEE.

2.6.4 RT-170TASK4-UPDATESYSTEMENGINEERINGTRANSFORMATIONROADMAP–TASK4

TheRT-157roadmaptaskwillbecontinuedunderRT-170.

27

PartII:TaskDetailSummaryThematerialinPartIIprovidesasummaryofsomeofthetaskdetails,includinginformationsharedduringsomeof theworking sessions.Anextensiveamountofmaterial covered inPart IIof theRT-141 finalreport[21]stillprovidesrelevantinformationtothisresearch,buthasnotbeenincludedinthisreport.

3 TASK1–MODELCROSS-DOMAININTEGRATIONWITHUNDERLYINGSINGLESOURCEOFTRUTH(SST)

AsdiscussedinSection2,understandingtheimpactsrelatedtocross-domainintegrationisakeyneedandchallenge;theseconcernscanimpactdecisionsmadebydifferentdisciplinesandcompetencies,andcan be a critical risk, especially as systems increase in complexity. Traditionally the cross-domainimplicationssurfaceduringintegrationandtest,whichistypicallylateinthelifecycle,andwherechangescanbecostly.Today,thisisanacknowledgedproblemandchallenge.Thesolutionsareoftenbelievedtobe better standards for tool integration, but as discussed earlier tools continually change and theintegrationsbecomebrittle[34].Newerapproachesbasedondatainteroperabilityasameansofsharinginformationusingstandardsandtoolneutralapproachesareemergingasbeingabetterapproach,andthisistheapproachwearepursuing[27][126].

3.1 INFORMATIONMODELFORASINGLESOURCEOFTECHNICALTRUTH

TheconceptfordevelopingaSSTisdirectlyrelatedtoidentifyingtheNAVAIRrelevantinformationwithinthedomainsof thecompetenciesandtheir relationshipswithinandacrossthedomains.Weselectedinformation from several sources in RT-141 final report to explain the evolving approach used byNASA/JPL.Crain[44]providedaprocesstoexplainhowtoapproachtheproblemofunderstandingtheunderlying“data” for theproducing systems.Startby identifying theobjects (classes inanontology),object properties, and object relationships. Figure 15 provides a perspective on some of the “dataobjects”andtheirassociatedrelationshipsthatarerelevanttotheenterpriseatNASA/JSC;similarobjectsarerelevanttoNAVAIRtoo.Wehaveidentifiedabout300“objects”thatareneededtodefineaNAVAIR-relevant InformationModel that underlies the SST. This information was collected by analyzing theartifactscollectedaspartoftheTier3productsfromthe“AsIs”effortofRT-48/118/141,andcreatedinaCORE(Vitech)model.

28

Figure15.IntegratedDataObjectsPartialEntityRelationalDiagram

Wehavediscussedontologiesasameansofcreatingatool-neutralinformationmodel.AnontologyisaconceptualizationforadomainwiththeassociatedrelationshipsasshowninFigure15.Whatisatleastas important is that an ontology can be represented in the standard language OWL (Web OntologyLanguage,actuallyOWL2) [143]whereopenandstandardSemanticWebTechnologies (tools) canbeusedtostore,update,delete,query,andreasonaboutconsistencyandcompleteness;suchinformationisstoredinatypeofdatabasecalledatriplestore.Anontologycanbethoughtaboutasaschemainthedatabase for thedata related toanontology. Inaddition,wecan relatedifferentdomainontologies,whichreflectonthecross-domaindependencies.Thisapproachiswhatwecalltoolagnostic(i.e.,toolneutral),butcanmaptoanytoolthatstoresmodels/data[27].

ArecommendedapproachforustoobtainthisinformationacrossthecompetenciesofNAVAIRisto:

1. Identifythe“objects”foreachcompetencyand/oraircraftdomain2. Definetheassociations,asshownforrequirementsinFigure163. Definetheintegrateddatamodelintheformofanontology,sothatwecanleverageSemantic

WebTechnologies,whichprovidethetoolagnosticapproachforanalysisofdependencies,assessingmeasuresofconsistencyandcompleteness

Withthenewproposedframework,thereneedstobesometypeofdecisionframeworkforassessingmaturity. We plan to address this using the Information Model (as a means of collecting relevantinformation,andusingcompletenessandconsistencychecksmuchlikeNASA/JPL[77]).

29

Figure16.AssociationtoRequirements

3.2 REQUIREMENTONTOLOGYSTATUS

Figure16providesaperspectiveonsomeofour2016deliverables.Wehavearequirementontology,anddeveloped and associated requirementmanager prototype,which uses semanticweb technology forcheckingrequirementconsistencyandcompleteness,asshowninFigure17.ThiscapabilitywasahighpriorityaspartofthedraftPOA&M,buthasmovedlowerinpriorityasdescribedearlier.OntheleftsideofFigure17istherequirementontologythatwe’reevolving,andontherightsideisasimpleGUIthatwehaveusedtoenterrequirements,whichareassociatedwithaTelepresenceRobotsystemthatweuseinacourseatStevens.Itissimple,butithasmanyfacetsofinterest:

§ Hardware:mechanical,electricalcomponents§ Software§ Systemofsystem(distributed)§ Sensors§ Multipleprocessors§ Communication,usesWi-Fiandinternet§ Humans-in-the-loop§ Mobile§ Semi-autonomous§ Needstoaddressavailability,dataintegrity,safetyandsecurity

30

Figure17.OntologyandRequirementManagerEnginePrototype

Thenextstepsaretodevelopcompletenessandconsistencychecksforrequirements[127],andevolvethattosupportriskassessmentpertheplanshowninTable1.

4 TASK2–MODELINTEGRITY–DEVELOPINGANDACCESSINGTRUSTINMODELANDSIMULATIONPREDICTIONS

Model integrity,fromoursponsor’sperspective, isameanstounderstandmarginsanduncertainty inwhatmodelsandassociatedsimulations“predict”orinotherwordswhen/howdowetrustthemodels.Theobjectivescharacterizedbythesponsoraretoensurethattheresearchcoveredthekeyobjectives;thoseobjectivesincluded:

§ Includebothmodelstoassess“performance”andmodelsforassessing“integrity”suchas:o Performance:aero,propulsion,sensors,etc.o Integrity:FiniteElementAnalysis(FEA),ComputationalFluidDynamics(CFD),reliability,etc.

–canwebuildit,canwetrustito Astatedchallengewas:howcan“integrity”beaccomplishedwhenthecurrentsituation

involvesfederationsofmodelsthatarenotintegrated?§ Continuoushierarchicalandverticalflowenabledbymodelsanditerativerefinementthrough

tradespaceanalysis,conceptengineering,andarchitectureanddesignanalysis

Sandia National Laboratory discussed advanced approaches for supporting uncertainty quantification(UQ)toenablerisk-informeddecision-making[98].Theirmethodsandtoolingaddressthesubjectsofmargins,sensitivities,anduncertainties.Theinformationtheyprovidedreflectsontheadvancednatureoftheireffortsandcontinuousevolutionthroughmodelingandsimulationscapabilitiesthatoperateon

31

someofthemostpowerfulhighperformancecomputing(HPC)resourcesintheworld.WeheardabouttheirHPCcapabilities,methodologiesonQuantificationofMarginsandUncertainty(QMU),anenablingframeworkcalledDesignAnalysisKitforOptimizationandTerascaleApplications(DAKOTA)Toolkit[123],andtheneedandchallengeofModelValidationandSimulationQualification[120].Theyalsodiscussedthemovement towards Common Engineering Environment thatmakes these capabilities pervasivelyavailabletotheirentireengineeringteam(i.e.,thedesigningsysteminourterminology).Wethinktheircapabilities provide substantial evidence for the types of capabilities that should be part of the riskframework.Thissectionprovidesadditionaldetails.

Alloftheseapproachesremaininscopeofourresearch,buttheyhavebeenpushedoutintimetoaddresstheprioritiesofSET.However,researchadvisedunderMarkBlackburnbyPhDCol.TimothyWest[139]atStevensInstituteofTechnologiesinvolvesaproposedmethodologytousetheSandiaNationalLaboratory(SNL) DAKOTA Toolkit [123] with the Department of Defense (DoD) Computational Research andEngineeringAcquisitionToolsandEnvironments(CREATE)AirVehicle(AV)[111]familyofcomputationaltools(e.g.,CFD),inordertodevelopanoptimizedwindtunnelcampaignfortwodifferentaerodynamicshapestoassesstheprocess.

Col.TimothyWestwasaguestspeakerattheworkingsession.HegaveabriefingonhisPhDresearchrelated to the Model Integrity Task, which involves the assessment of modeling and simulation forreducing or guiding the efforts inwind tunnel testing conducted atArnold EngineeringDevelopmentComplex(AEDC).Col.WestrunstheTestOperationsDivisionatAEDC,whichinvolvesthemanagementofallwindtunneltesting.HistalkbrieflydiscussedthehistoricalperspectivessettingthecontextofAEDCandthechallengeinrunningthesewindtunnels.

Col.West’sresearchistitled(“ADigitalThreadFrameworkforDynamicallyIntegratingExperimentalandComputationalResultswithQuantifiedMarginsandUncertainties”)involvesaproposedmethodologytousetheDAKOTAToolkitwithCREATEAirVehicle(AV)familyofcomputationaltools(e.g.,CFD,FEA),inordertodevelopanoptimizedwindtunnelcampaignfortwodifferentaerodynamicshapestoassesstheprocess. This topic is highly relevant to the NAVAIR SET, and specifically to the challenge of ModelIntegrity(howtotrustthepredictivecapabilitiesofmulti-physicsmodelingandsimulations).

TraditionalapproachesreferredtoasVerification,ValidationandAccreditation(VV&A)ofmodelingandsimulationcapabilitiesarestill relevantandusedbyorganizations.VV&A, inprinciple, isaprocessforreducing risk; in that senseVV&Aprovides away for establishingwhether aparticularmodeling andsimulation and its input data are suitable and credible for a particular use [55]. The word toolqualification[56]andsimulationqualification[120]havealsobeenusedbyorganizationsregardingthetrustinmodelsandsimulationscapabilities.

SeealsoSection5.7formoredetailsonModelingandMethodsforUncertaintyQuantification.

32

5 TASK3–MODELINGMETHODOLOGIES

Task1andtheSSTisastudyaboutwhatinformation/dataisneededtounderstandboththeproblemanddesignspace.Task2isaboutunderstandingthe“trust”intheinformation/datathatisproducedbyvarious types of models and tools. Task 3 is about best methods to systematically produce thatinformation.Oftenthereisasymbioticapproachtoassessthemethodsandselectsupportingtools.Therehave been extensive discussions about a broad spectrum of tools documented in the RT-141 finalreport [21]. However, there have been numerousmeetings, research papers, even presentations byrepresentativesofcompaniesthatsellmodelingtoolsthatalldescribethatitiscriticaltodoseveralthingsbefore“buyingtools.”Forexample,MatthewHausewhoworksforacompanythatsellsMBSEmodelingtoolsandotherrelatedproductsprovidesalistofthingsnottodowhenadoptingMBSE.Akeypointfromthelistinvolvestheneedfororganizationsto:1)understandwhattheyneedtoproduce(withmodels),and 2) the method for using the tools. Therefore, this section discusses the needs for methods asdiscussedinSection2.1.

Ourteamisalignedwithmanyofthese“betterpractices,”towardsMCEadoption:

§ Identifyinginformationthatisneeded(byNAVAIR)thatisproducedoranalyzedthroughmodelstosupportdecisionmaking,seeSection3

§ Methodsthatneedtobeusedtoenablethemodelingtoolstoworkinamoreefficientmannero OnesuchfailingofutilizingapropermethodwaspointedoutbytheDARPAMETAproject

programmanager[9]§ Weneedtoincreasefocusoncross-domainmethodologiestoensuretoolusageproduces

completeandconsistentinformationcompliantwithinformationcapturedintheSST§ Wealsowanttoembedmethodologicalguidanceinthenewtooling,suchas:designpatterns,

referencemodels,etc.o MethodscouldberepresentedasanSysMLActivityDiagram(typeofflowchart)thatshows

theprocessflow,anddataflow(bothinandout)totheSSTo NASA/JPLprovidesagoodexample[10]

5.1 MODELINGEXAMPLESOVERVIEW

Oursponsorrequestedthatwedevelopsomeexamplereferencemodels,MDAOmodels,andmissionandsystemlevelmodelstoimprovetheknowledgeoftheNAVAIRteam.PertheirrecommendationwestartedwithSysMLcharacterizingcapability,operational,structural,andbehavioralviews.Theneedisforpeopletobeabletoreadandhavediscussionsaboutthemodels(notnecessarilybeabletocreatethemodels–atleastatthispoint).WeselectedsomeUAVscenariosandcreatedmodelsshowingafewdifferenttypesofmodelingviews.Thefollowingisanoverviewofsomeoftheinformationshared;moredetailsareprovidedinSection5.1:

§ Activitydiagramstodescribedifferentprocessmodelso Pre-modelingguidelineso ExampleCONOPSo SimpleMBSEprocess,includingMDAOrelationshipo FunctionalRequirementsDecomposition

§ Packagehierarchyforstructuringandorganizingmodelinformationo Forexample,weorganizedthismodeltoinclude:

• Enterprisemodels• Referencemodels• Missionmodels

33

• Systemmodels• Aircraftsystemhierarchy

§ Missionlevelmodelso Createdthesemodelviewstosetthecontextforthesystem(bestpracticeguideline)o UseCasediagramformissionusing(Observe,Orient,Decide,Act)incontextofFind,Fix,

Finisho ActivitydiagramofMissionActivityrelatingaSensorPlatform(UAV)anditsinteractions

withCommunicationPlatform(s)§ Systemlevelmodels

o IllustratethesystemcontextusingaBlockDefinitionDiagram(BDD),whichshowstheelement(systems,actors,environment)intheMissionDomain

o Top-levelUseCaseforaUAV(fly,surveillance,refuel,on-shiprefueling)o StatemachinediagramofthestatesofaUAV,fromoff,taxi,takeoff,cruise,loiter,descend,

hold,land,etc.§ ActivitydiagramofDaveCohen’sframeworkprocess

o Weexpectthatmodelingtheframeworkwillhelpinsupportinganalysisofthechallengesandgapsmatrix

o ThisforceduseintotheneedtomodelKSAasaBDD§ IllustratedhowSysMLmodelscanrelatetoothermodelsusingMDAOparametersand

constraintstomodelaworkflowtoreflectsomecross-domainanalysisrelatedtoWeight,Aerodynamics,Propulsion,andPerformance(e.g.,vehiclerange)asshowninFigure18.ThisexampleallowedustodiscussthenotionalstepsinMDAO(ForadditionaldetailsseeSection8.7):o AMDAOanalysisisdefinedasasequenceofworkflows(scenarios)o Afteridentifyingasetofinputsandoutputs(parameters)o DefineaDesignofExperiments(DoE)anduseanalysessuchassensitivityanalysisand

visualizationstounderstandthekeyparameter(thisscopestheproblem)o UseOptimizationusingsolverswiththekeyparametersanddefinedifferent(keyobjective

functions–onoutputs)todeterminesetofsolutions(resultsoftenprovidedasatableofpossiblesolutions)

o Usevisualizationstounderstandrelationshipsofdifferentsolutionso NOTE:AnynodeonanArchitectural(SysML)modelcouldmaptosomePhysics-basedmodelo SomeofthesearchitecturalviewsfromSysMLmodelscanbeworkflowsofanalysisthrough

MDAO(usingMagicDraw,andRhapsody)§ WeshowedsomeMDAOwebinarsduringalunchofaworkingsessionanddiscussed:

o “PartIII:MDAOforConceptualAircraftDesignatNorthrupGrumman”o ProvidedlinkstootherrelevantWebinarsfromdifferentcontractingorganizationtoNAVAIR

• SystemTradeStudies&DesignOptimization,presentedbyLockheedMartin• PhoenixIntegration(ModelCenter)&theSkunkWorkspresentedbyBoeing• TheRoleofMulti-DomainDynamicModelsforFunctionalVerificationinMBSE

o Linkisheretomorewebinars[110]:http://www.phoenix-int.com/resources/webinars/on-demand-webinars.php

5.2 MULTI-DISCIPLINARYDESIGNANALYSIS&OPTIMIZATION

Multi-disciplinaryDesignAnalysis&Optimization(MDAO)isanapproachforcalculatingoptimaldesignsand understanding design trade-offs in an environment that simultaneously considersmany types of

34

simulations,evaluations,andobjectives.Forexample,whendesigningavehicle,thereistypicallyatrade-off between maximizing performance and maximizing efficiency, where calculating either of theseobjectives require multiple disciplinary models (geometry, weight, aerodynamics, propulsion)MDAOprescribeswaystointegratethesemodelsandexplorethenecessarytrade-offsamongtheobjectivestomakeadesigndecision.WhilethetheoreticalfoundationsofMDAOarewell-establishedbyacademics,anumberofbarrierstopracticalimplementationexist.Chiefamongtheseisthelackofmodelintegration,whichpreventsdesignersofonesubsystemfromeasilyassessinghowchangingadesignvariableaffectstheresultsofothersubsystems’modelsorsimulations.

Morespecificobjectivesinclude:

§ Assessingtheimpactsofindividualdesignchangesonsystemcapabilities§ Supportingearly-phase(conceptualdesign),system-leveltrade-offanalysisusingprevious

evaluationresultsfromexistingmodels§ Developstrategiestotransformthecontractingprocesssothatrequestsforproposals(RFPs)

canbedesignedmoreflexiblytowardvalue-based(ratherthantarget-based)design

Inpursuitoftheseobjectives,theresearchactivitiesentail:

§ Developgenericmulti-disciplinarymodelsofaUAV,includinganalysesofthegeometry,structure,aerodynamics,propulsion,andperformancecapabilities,tobeusedasanexamplecase

§ Exploreusingsystemsrepresentations(e.g.,SysML)tomapinputs(parametersandvariables)andoutputs(objectives,constraints,intermediateparameters)amongtheindividualmodels

§ ConducttradestudiesontheUAVdesignusingestablishedapproachesandtoolsforMDAO,exploringdifferentapproaches,tools,andvisualizationtechniquestomosteffectivelydisplayinformationanduncertaintyfordecision-makers

§ Explorewaysthatprevioustradestudyresultsondetail-phaseproductdesigncanbeusefultowardnewconceptualdesignofproductswithvaryingmissioncapabilityrequirements

§ WorkwithNAVAIRprojectleadstounderstandthebarrierstoimplementingthistypeofMDAO,culturallyandpractically/theoretically

§ Exploremoregeneralwaystomapandcoordinatesubjectmatterexperts(SMEs)anddata,models,andmeta-modelsforimproved(1)requirementssettingforRFPorCONOPS,and(2)value-drivendesign

Further,whiletheinitialresearchexamstheuseofMDAOatthesystemslevel,itisapplicabletouseatthemissionandsubsystemlevels.

5.2.1 MDAOMETHODS

Oneoftheobjectivesofthisprojectistoleveragethemostpowerfultoolsthatareoftenusedbyindustryaswellasgovernmentorganization.ThereanumberoftoolsthatsupportMDAO,includingbothopensourceandcommercial(non-exhaustive):

§ DAKOTA[123],OpenMDAO,iSight,ModelCenter[110],modeFRONTIER,FIDO,IMAGE,CONSOL-OPTCAD

§ SometimesreferredtoasProcessIntegrationandDesignOptimization(PIDO)§ Modelingandsimulationinventoryanalysisidentified348,mostofthesewereformission-level

simulation,butitwasbelievedthattherearemanymoreusedbythecompetencies;thesetoolsmayalsobewrappedwithinandMDAOworkflow

35

We have secured academic licenses to Phoenix Integration’s ModelCenter [110]. We then want toinvestigate themethods forapplysuch tools,andalso identify the relevant researchquestions in thecontextofthoseadvancedtools.Forexample,thestepsforanMDAOmethodmaybecharacterizedas:

§ Describeaworkflow(scenarios)foraKPP(e.g.,range,notionallysimilartosurveillancetime)§ Determinerelevantsetofinputsandoutputs(parameters)§ IllustratehowtouseaDesignofExperiments(DoE)anduseanalysessuchassensitivityanalysis

andvisualizationstounderstandthekeyparametertoscopethatwillbeusedinset1.d§ IllustrateOptimizationusingsolverswithkeyparametersanddefinedifferent(keyobjective

functions–onoutputs)todeterminesetofsolutions(resultsoftenprovidedasatableofpossiblesolutions)

§ Usevisualizationstounderstandrelationshipsofdifferentsolutions

A number ofmethods can be applied to formulatemulti-disciplinary optimization problems, developusefulsurrogatemodels,andcalculateoptimalandPareto-optimalsolutions.Optimizationproblemscanbe formulated with a number of different objectives by converting some objectives to targets orconstraints, summing the objectives with value-based and unit-consistent weighting schemes, ormultiplyinganddividingobjectivesbyoneanother.Surrogatemodelsareoftenusedtoquicklysimulatethebehaviorofamorecomputationally-intensivesimulationmodel,andsomecommonmethodsincludeinterpolation,responsesurfaceusingregressionmodels,artificialneuralnetworks,kriging,andsupportvector machines. Finally, numerical optimization can be performed using a number of differentalgorithmsandtechniques,includinggradient-basedmethods,patternsearchmethods,andpopulation-basedmethods.Foreachofthese,differenttechniqueshavebeenfoundtobemoresuitabletodifferentapplications,andpartofthisresearchdirectivewillbetoidentifyanddemonstratethebesttoolsforthisMCEarchitecture.

5.2.2 INTEGRATIONSWITHRELATEDTASKS

WhilethetheoreticalfoundationsofMDAOarewell-establishedbyacademics,anumberofbarrierstopractical implementation exist. Chief among these is the lack of model integration, which preventsdesigners from easily assessing how changing one design variable affects the outputs from differentmodelsorsimulations.Throughthisproject,andthecreationofanMCEarchitecturethatfollowsaSSTandaconsistentontology,wewillbeabletoleverageMDAOtechniquesinthedesigndecision-makingprocess.Fromanacademicperspective,themajorcontributionswillbetobuildaroadmapforintegratingMDAOpracticesintocomplexexistingandneworganizationalstructures.

AsolidframeworkforMDAOcanenablemulti-objectiveoptimization,showingproductdevelopershowdifferentdesignobjectivescompetewithoneanother.Forexample,weknowthatimprovinganobjectivelike“minimizeweight”typicallyrequiresasacrificeintheobjectiveto“maximizepower.”Themagnitudeof that improvement-sacrifice relationship, which often involves different units and requires humanjudgementtomakeamission-appropriatedecision,canberevealedbycombiningdifferentsimulationmodels,surrogatemodels,andoptimizationroutines.Asthismayinvolvebalancingalargenumberofobjectives,oneofthekeychallengesisinvisualizationoftheresultstoenableinformeddecision-making.Thisfitsintoallfivetasksoftheproject,astheentireinformationarchitecturemustbebuilttosupportcross-disciplinaryanalysis,andspecific toolsandtechniquescanbe integratedandtestedatdifferentstagesofthetransformation.

36

5.2.3 MDAOUAVEXAMPLE

NAVAIRsponsorswerepresentatademonstrationofaMDAOworkflowshowninFigure18thatwasdevelopedusingModelCenter.Thedemonstrationcoveredseveralaspectsoftheobjectivesdiscussedinthissection,including:

§ DescribeandexecuteaworkflowanalysisofUAVcapabilities(e.g.,range,velocity,andfuelconsumption)

§ Maprelationshipsamongparameters(inputs/outputs)indisciplinarymodels§ IllustrateuseofDesignofExperiments(DoE),sensitivityanalysis,andvisualizationsto

understandcapabilityrelationships/trade-offs§ OptimizeusingdifferentsolverstofindsetsofPareto-optimalsolutions§ Takeadvantageofpreviousmodelanalysesforuseinearly-phasedesignwithnewmission

capabilityrequirements

Figure18.MDAOExampleWorkflow

AsshowninFigure19,theParetofrontier(Paretooptimalset)showsthetrade-offbetweenrangeandpropulsion.ThebluepointsshowtheParetofrontier/non-dominatedsolutions.TheParetofrontierwascalculatedusingabi-objectiveoptimizationusingNSGA-IIalgorithmto:

§ Maximizerange§ Maximizepropulsion§ Given5designvariables

o Wingarea(ft2)o Wingspan(ft)o Altitude(ft)o Speed(knots)o Efficiencyfactor

Theseresultsreflectonhowmuchrangeonewouldhavetogiveupinordertoincreasethepropulsionbysomeamount.Basedonthecurrentsetofequationscharacterized intheworkflow,thesensitivityanalysisshowninFigure20indicatesthatthewingareaisthevariablethatexhibitstheclearesttrade-off. Thewing span has the largest effect on range, but does not present a trade-off between theseobjectives.

37

Figure19.Paretofrontier(Paretooptimalset)ShowsTrade-offBetweenRangeandPropulsion

Figure20.SensitivityofObjectivestoDesignVariables

5.3 MODELINGEXAMPLESUSINGSYSML

This section provides some examples of SysMLmodels shown during a number of differentworkingsessions.SysMLmodelscanbeusedtodescribebothmission,systemandprocessmodels.

38

5.3.1 TABLEOFCONTENTS

WecreatedaContentDiagramasatypeoftableofcontentswhereweputhyperlinkstothedifferentdiagrams.Thissectionshowssomeofthediagramsineachofthesegroups.

Figure21.TableofContentstoModelsandDiagrams

5.3.2 PROCESS/METHODS

WecreatedafewSysMLactivitydiagramstodescribedifferentprocess/methodsmodels.Ideally,thesetypesofmodelswouldbereferencemodelsthatestablishabestpracticeapproachforstartingdifferenttypesofmodels.Theseguidelinesareshowninanactivitydiagramcalledpre-modelingguidelines.Suchguidelines as shown in Figure 22, include defining the structure of the overall project, namingconventions,colors.ThispicturealsoshowsthedifferenttypesofSysMLdiagrams.Intheremainderofthissectionwewill showa fewdiagramtypes.Thepre-modelingguidelinesaredefined inanactivitydiagram,whichshowstheactions,controlflows,andcanshowdataflows.Thedarkbarcanrepresentafork(todistributeasynchronousprocesses)orajointosynchronizedistributedprocesses.Oneofthefirstthingsateamneedstodecideisonthestructureoftheoverarchingmodel.AnexampleisshownintheContainmentviewasshowninFigure23.WeusedtheMagicDrawtool,butSysMLisastandardmodelinglanguagesupportedbymanydifferenttools.

39

Figure22.Pre-modelingGuidelines

40

Figure23.ContainmentStructure

Figure24showsanactivitydiagramthatwasdevelopedundertheStevensInstituteofTechnologySYS-750AdvancedArchitectureCourse.Thisactivitydiagramillustratesanoverarching,butsimplifiedMBSEprocess.Thisactivitydiagramshowsthecontrolflow(withoutfeedback),butalsothedataflowtotheobjectscoloredinblue.Theseobjectshighlightthetypesofdiagramsthatmightbeusedtocapturetheinformationinthevariousstepsoftheprocess.Theseactivitiescanbefurtherdecomposedtodescribeadditionaldetailsoftheprocessstepsandotherinformationthatisusedorproduced.

Note also that there is a PerformMDAO activity, colored orange Figure 24. This type of analysis isperformedbyothertypesofmodelingtoolssuchasModelCenterasdiscussedinSection5.2.Thesetypesofactivitydiagramsthatrepresentprocessesormethodsarenotrepresentingthetargetsystem,butcanbecharacterizedasreferencemodelsusedaspartoftheEnterprisethatincludeswhatwehavereferredtoinourpriortechnicalreportsasthe“DesigningSystem”[21].

41

Figure24.SimpleMBSEActivityDiagramwithLinktoMDAO

5.3.3 PACKAGEHIERARCHYFORSTRUCTURINGANDORGANIZINGMODELINFORMATION

We showed an example of a package structure for organizing the different perspective (enterprise,mission,system),butwecanalsouseasimilarconcept toorganizetheactualsystemstructure, fromeitheralogicalorphysicalperspective,seeFigure25.Thisisagooddecisiontomakebytheteamasearlyaspossible.

42

Figure25.ModelOrganization

5.3.4 MISSIONLEVELMODELS

Mission-levelmodelssetthecontextforthe“systemofinterest,”whichisatypicalbestpractice.Fromamilitaryperspective,weincludedaUseCasediagramformissionusing(Observe,Orient,Decide,Act)incontextofFind,Fix,Finish.Thiscouldbea reusableUseCase,or referencemodelwhere thespecifictextualelementsoftheusecasecouldbetailoredtoaspecificmissionasshowninFigure27.

43

Figure26.High-LevelMissionUseCase

44

Figure27.TextualElementoftheUseCase

The systemdomain shows the various elements associatedwith surveillanceusing aBlockDefinitionDiagram(BDD),asshowninFigure28,.Thisshowsthecontextofthehigher-levelsystemofsystem,ofwhichtheUAVisonesystem.

45

Figure28.SurveillanceSystemDomainDiagram

WealsoprovidedanexampleofActivitydiagramofMissionActivityrelatingaSensorPlatform(UAV)anditsinteractionswithCommunicationPlatform(s)asshowninFigure29[134].Thisconceptispresentedfromalogicalperspectiveandshowsbothcontrolflow(dashlines),anddataflow(solidlines);thisactivitydiagramalsoshowsswimlanesthatillustratethedifferentpartitioningoftheactivities.

46

Figure29.Mission-levelActivityDiagramwithSwimLanePartitions

5.3.5 SYSTEMLEVELMODELS

UseCasesarealsoacommonstartingpointforsystemleveloperationalscenarios.Anexampletop-levelUseCaseforaUAV(fly,surveillance,refuel,on-shiprefueling)isshowninFigure30.EachoftheusecaseswouldtypicallyhaveastructurednarrativecreatedusingatemplatesimilartoFigure27.Therefore,whenwediscussusingmodels,wedonotimplythatthereisnotextualnarrative,ratherthenarrativeisoftenstructuredandembeddedwithinsometypeofmodelingelement.

47

Figure30.GenericUAVUseCaseDiagramwithActors

Toillustrateanothertypeofbehavioralperspective,weshowedanincompletestatemachinediagramofthestatesofaUAV,fromoff,parked,taxi,takeoff,cruise,loiter,descend,hold,land,etc.asshowninFigure31.

Figure31.StateMachineDiagramofTop-LevelUAVOperationalStates.

Wealsohaveexamples that arebasedonaproduct familyofUAVbeingdevelopedbyour researchcollaboratorDr.RussellPeakunderRT-170thatinclude:

§ RotorUAV2.1portfolioeffectivelycompletedo IncludesopticalcameraoptiontooriginalpackagedeliveryUAVsquadron

48

o IncludesphysicscalculationsviaSysMLparametrics(par)o IncludesbehaviorsimulationviaSysMLstatemachine(stm)/activity(act)/parametrics

(par)§ Fixed-wingUAV0.1portfolioinitiated(WIP)

o Inspiredbyfixedwingsurveillance.o Applying~sameapproachasforrotorUAVportfolio

SomeoftheWork inProgress(WIP)elements includethesystemmodel fortheFixed-WingRefuelingUAV.TheseareshownbelowinaSysMLBDD,whichincludessomeofthesubsystemsoftheUAVsuchas:propulsion,fuel,andrefuelingsubsystems.

Figure32.Fixed-WingRefuelingUAVExtensiontoUAVPortfolio

ThereareelaborationonsomeparametersofthefuelsystemasshowninFigure33todosomeanalysisontheFirst-OrderPhysicsusingSysMLParametrics.Aparametricsdiagramprovidesawaytodescribeconstraintsbetweenparameters.Add-onanalysistoolscanthenbeusedtoverifythattheconstraintsaresatisfiable(i.e.,notcontradictory).ThismodelisdevelopedinMagicDrawandusessomeautomationprovided by aMagicDraw plugin called the Cameo Simulation Toolkit for requirement verification asshowninFigure34.Forexample,theresultofpass/failonaconstraintcanbetraceddirectlybacktospecificrequirementobjectinthemodel.

49

Figure33.ParametricDiagramofFuelSystem

Figure34.CameoSimulationToolkitVerifiesConstraintsRepresentingNumericRequirements

TherewereothermodelspresentedinthevariousworkingsessionandthebriefingmaterialwasprovidedtoourNAVAIRsponsor.

5.3.6 ACTIVITYDIAGRAMOFDAVECOHEN’SFRAMEWORKPROCESS

We illustrate the generality of the SysMLmodeling approachby creating amodel for the framework(shown in Figure 4). We expect that modeling the framework, as shown in Figure 35, will help insupportinganalysisofthechallengesandgaps.ThisforcedusintotheneedtomodelKPP/KSAasaSysMLBlockDefinitionDiagram(BDD).

50

Figure35.DraftActivityDiagramofSETransformationFramework

5.4 VIEWSANDVIEWPOINTS

Theworkingsessionhadasectiononexplainingvariousdifferentmethodsfordoingsystemengineeringmodeling.ThisalsorelatestotheconceptofViewandViewpoints intheSST[51].EachViewpoint,asshown in Figure 36, is a specification of conventions and rules for constructing and using aView forpurposesofaddressingasetofstakeholderconcerns,whichisbasedonastandardthatrelatesto:

§ Purposeoftheviewpoint§ Stakeholderthatarelikelytousetheviewpoint§ Concernofthestakeholder§ MethodtodevelopaModelusingaModelingLanguage§ Analysisthatcanbeperformedwiththemodels

51

Figure36.Viewpoint

5.5 CAPABILITYANDOPERATIONAL-LEVELMODELINGGUIDELINES

NAVAIR has an architecture group that constructsmodels associatedwith the Capability DescriptionDocument.TheyhavedefinedguidelinetoprovidesomemethodologicalguidanceinconstructingDoDAFmodels.Notionally,theguidelinescover:capabilitiesview,operationalviewsandsystemviews.Thesearegeneralizedasarchitecturemodels.

§ The“architecturalmodelers”whocreateDoDAFbasedontheseguidelinesusetheUMLIntegratedArchitecture(UPIA)profile;theyhavesomenotional“ontologies”o IntegratedOperationalModelOntology

• CapabilityOntology(WP1010-0000-01)• OperationalOntology(WP1020-0000-01)

o IntegratedSystemInterfaceModelOntologyo SystemInterfaceOntology(WP1030-0000-02)o DiagramOntology(WP1060-0000-01)o OperationalDiagramOntology

• OV-5aUseCase• OV-5bActivityDiagram• OV-6cEventTrace

o SystemDiagramOntology• SV-4aUseCase• SV-4bActivityDiagram• SV-10cEventTrace

Currently, these efforts are fundamentally limited to the net-ready aspects of capabilities, governingcommunications and interoperability. It is acknowledged that extending this to logical and functionalviewsisdesirablefortheSET.

5.6 NAVAIRSTUDYVIEWS

StudyviewswerecreatedtoaddressanumberofchallengesattheProgramofRecord(POR)levelandincreatingDoDAFrequirements,howevertheredoesnotappeartobeextensiveknowledgeabouttheiruse.ThestudyviewconceptbuildsonlessonslearnedfromcreatingearlyDoDAFmodels;analyseshaveuncovered that interoperating at the lowest (data) levels is insufficient for scenarios, and scenariosrequire behaviors, which ismissing at the data level. DoDAF does not accommodate other scenario

52

requirements (e.g., conditions, assumptions) very well, and is insufficient to fully characterize thedynamicsneededforanalysis.

Amission-levelSoSanalysisbeginswithformalizationusingStudyViews,asreflectedinFigure37[133],whichhasmodelingandsimulationdynamicviewsandvisualization.StudyviewsprovidestructureandacommoncontextthatactsasabasisforframingandboundingthefunctionaldecompositionofDoDAFproducts.Studyviewsformalizetheneedandintent,provideasituationalcontextandinfluencingfactorstoframeandboundthefunctionsandactivitiesofthemissionandscenariosthatultimately leadintocorrespondingrepresentationsoftheMissionandSystemCapabilities(i.e.,thecapabilitiesforthePOR).Thesecapabilityrepresentationsarefurtheranalyzedusingmodelingandsimulationandcorrespondinganalysiscapabilities.TheoutputsofwhicharethenformalizedintermsofDoDAFartifactsbytheNAVAIRArchitecture group (see Section 5.5). This information forms the analysis boundaries for the SystemCapabilitiesinformationneededtodefinerequirementsforthePOR.

Figure37.MissionContextforSystemCapability

5.7 MODELINGANDMETHODSFORUNCERTAINTYQUANTIFICATION

AsdiscussedinSection4,SandiaNationalLaboratorydiscussedsomeadvancedmethodsforsupportinguncertaintyquantification(UQ)toenablerisk-informeddecision-making[98].Theirmethodsandtoolingaddressthesubjectsofmargins,sensitivities,anduncertainties.Theinformationtheyprovidedreflectson the advanced nature of their efforts and continuous evolution throughmodeling and simulationscapabilitiesthatoperateonsomeofthemostpowerfulhighperformancecomputing(HPC)resourcesin

53

the world. We heard about their HPC capabilities, methodologies on Quantification of Margins andUncertainty (QMU), an enabling framework called Dakota, and the need and challenge of ModelValidation and Simulation Qualification [120]. They also discussed the movement towards CommonEngineeringEnvironmentthatmakesthesecapabilitiespervasivelyavailabletotheirentireengineeringteam (i.e., the designing system in our terminology). We think their capabilities provide substantialevidenceforthetypesofcapabilitiesthatshouldbepartof therisk framework.Thissectionprovidesadditionaldetails.

5.7.1 DAKOTASENSITIVITYANALYSISANDUNCERTAINTYQUANTIFICATION(UQ)

TheDakotaframeworksupportsoptimizationanduncertaintyanalysis[123].ThereissignificantdemandatSandiaforrisk-informeddecision-makingusingcrediblemodelingandsimulation:

§ Predictivesimulations:verified,validatedforapplicationdomainofinterest§ Quantifiedmarginsanduncertainties:randomvariabilityeffectisunderstood,bestestimate

withuncertaintypredictionfordecision-making§ Especiallyimportanttorespondtoshiftfromtest-basedtomodelingandsimulation-based

designandcertificationo Thisgetstoanimportantpointabouthowtousemodelsasopposedtotesting,whichis

criticalforNAVAIR’sobjectivetorapidlyandcontinuously“crossthevirtualV”

TheHPCcapabilitiescomesintoplayastheyarebuilttotakeadvantageoftheHPCenvironmentandcanbecombinedwithpredictivecomputationalmodels,enabledbyenvironmentandculturethatfocusesontheoryandexperimentationtohelp:

§ Predict,analyzescenarios,includinginuntestableregimes§ Assessriskandsuitability§ Designthroughvirtualprototyping§ Generateortesttheories§ Guidephysicalexperiments

Dakotaisreferredtoasaframework,becauseitisacollectionofalgorithmssupportingvarioustypesofintegrationthroughprogrammatic(scripting)interfaces;thisisrepresentativeoftheconceptofmodel-centricengineering,seeFigure38.Itautomatestypical“parametervariation”studiestosupportvariousadvanced methods and a generic interface to simulations/code, enabling QMU and design withsimulationsinamanneranalogoustoexperiment-basedphysicaldesign/testcyclesto:

§ Enhancesunderstandingofriskbyquantifyingmarginsanduncertainties§ Improvesproductsthroughsimulation-baseddesign§ Assessessimulationcredibilitythroughverificationandvalidation§ Answerquestions:

o Whicharecrucialfactors/parameters,howdotheyaffectkeymetrics?(sensitivity)o Howsafe,reliable,robust,orvariableismysystem?

(quantificationofmarginsanduncertainty:QMU,UQ)o Whatisthebestperformingdesignorcontrol?(optimization)o Whatmodelsandparametersbestmatchexperimentaldata?(calibration)

54

Figure38.DakotaFrameworkIntegrationWrapsUserApplication

Toputmarginsanduncertaintyintocontext,assumethatthereisadevicethatissubjecttoheat,andweneedassesssometypeofthermaluncertaintyquantification.GivensomeresultsfromsomeDesignofExperiment(DoE)(alsosupportedbyDakota)resultsthatgiveaprobabilitydistributionasshowninFigure39 [2]. The Mean of the temperature: T, to the lower bound of the threshold (e.g., 72 degrees)characterizestheMargin,andtheStandardDeviation(T)characterizestheuncertainty.

Figure39.ExampleforUnderstandingMarginsandUncertainty

ThisapproachandDakotaframeworksupportsabroadsetofdomains,andthereforewethinkitcanbegenerallyappliedacrossdomainforNAVAIR,forexample:

§ Supportssimulationareassuchas:mechanics,structures,shock,fluids,electrical,radiation,bio,chemistry,climate,infrastructure

§ Isbestusedwithagoal-orientedstrategy:o Findbestperformingdesign,scenario,ormodelagreemento Identifysystemdesignswithmaximalperformanceo Determineoperationalsettingstoachievegoalso Minimizecostoversystemdesigns/operationalsettingso Identifybest/worstcasescenarioso Calibration:determineparametervaluesthatmaximizeagreementbetweensimulationand

experiment

00.51

1.52

2.53

3.54

4.55

30 36 42 48 54 60 66 72 78 84

% in

Bin

Temperature [deg C]

Final Temperature Valuesmargin

uncertainty

55

§ Handlesparallelism,whichisoftennotfeasiblewithcommercialtools,andwhyHPCcanplayanimportantrole

§ Providessensitivityanalysis–findthemostinfluentialvariables§ UncertaintyQuantification

o Modelsinherentlyhaveuncertaintyo Assesseffectofinputparameteruncertaintyonmodeloutputs

• Determinemeanormedianperformanceofasystem• Assessvariabilityinmodelresponse• Findprobabilityofreachingfailure/successcriteria(reliability)• Assessrange/intervalsofpossibleoutcomes

5.7.2 ANOVERVIEWOFQUANTIFICATIONOFMARGINSANDUNCERTAINTY

DakotaisatoolframeworkthatcansupportthemethodofQuantificationofMarginsandUncertainty(QMU).SomeofthematerialfromSandiaiscategorized“OfficialUseOnly[OUO].”Weprovideasummaryextractedfrompublicallyavailableinformation[98].

QMU pre-dates Dakota and is not unique to Sandia as it was used at Lawrence Livermore NationalLaboratoryandLosAlamosNationalLaboratory,withtheoriginalfocusofthemethodologytosupportnuclearstockpiledecision-making6.QMUisaphysicspackagecertificationmethodologyandalthoughithasbeenaroundandusedatSandiadatingbackto2003,andbothQMUtheoryandimplementationarestillbeingdeveloped/evolved[98].Webelievethemethodologyhasmoregeneralusethanjustphysicspackagecertification.

QMUappliestothelifecycleoftheentireweapon,withfocuson:

§ Specificationofperformancecharacteristicsandtheirthresholdso Performanceistheabilityofsystem/componenttoprovidetheproperfunction(e.g.,timing,

output,responsetodifferentenvironments)whenexposedtothesequenceofdesignenvironmentsandinputs

§ Identificationandquantificationofperformancemarginso Aperformancemarginisthedifferencebetweentherequiredperformanceofasystemand

thedemonstratedperformanceofasystem,withapositivemarginindicatingthattheexpectedperformanceexceedstherequiredperformance

§ Quantificationofuncertaintyintheperformancethresholdsandtheperformancemarginsaswellasinthelargerframeworkofthedecisionsbeingcontemplated

Therearetwotypesofuncertaintythataregenerallydiscussedthataccountfor,quantify,andaggregatewithinQMU:

§ Aleatoryuncertainty(variability)o Variabilityinmanufacturingprocesses,materialcomposition,testconditions,and

environmentalfactors,whichleadtovariabilityincomponentorsystemperformance§ Epistemicuncertainty(lackofknowledge)

o Modelsformuncertainty,bothknownandunknownunknownsinscenarios,andlimitedorpoor-qualityphysicaltestdata

6TheComprehensiveNuclearTestBanTreatyendsfull-scalenuclearweaponstestingintheU.S.PresidentBillClintonattheUnitedNations,September24,1996

56

The statistical tolerance interval methodology is an approach to quantification of margins anduncertainties forphysical simulationdata.There isalsoprobabilityof frequencyapproach,commonlyusedincomputationalsimulationQMUapplications[98],which:

§ Extendsthe“k-factor”QMUmethodologyforphysicalsimulationdatao k-factor,ingeneral,isdefinedasmargindividedbyuncertainty(M/U)

• Margin(M):differencebetweenthebestestimateandthethresholdforagivenmetric• Uncertainty(U):therangeofpotentialvaluesaroundabestestimateofaparticular

metricorthresholdo Providesessentialengineeringanalysistoensurethecollecteddatasampleincludes

measurementsthatmaybeusedtoinferperformanceinactualuseo Itisimportanttounderstandtheperformancerequirementtounderstandtheperformance

thresholdandassociateduncertainty• Threshold:aminimumormaximumallowablevalueofagivenmetricsetbythe

responsibleLaboratory§ Thenewmethodaddressesthesituationwhereperformancecharacteristichasshownthe

potentialforlowmarginoramarginischanging(likelygettingsmallerorthereisgreateruncertainty)withage[98]o Notionallythemarginshiftsfromthemeanoftheperformancecharacteristic(PC)andits

performancerequirement(PR)tothedifferencebetweenameaningfulpercentileofthedistributionoftheperformancecharacteristicanditsperformancerequirement

o Needtoquantifyuncertaintythroughthecomputationofastatisticalconfidenceboundonthebestestimateofthechosenpercentileratherthanbyasamplestandarddeviation(asreflectedinFigure39),whichdoesnotaccountforsamplingvariability

o Thisisaccomplishedbycomputingastatisticaltoleranceinterval

Wecreatedagraphic fromseveralpublicallyavailablesources,as shownFigure40 inorder tobetterexplain a few aspects about QMU, Dakota, epistemic and aleatory uncertainty. Typically, within theDakota framework there is an outer loop: epistemic (interval) variables and inner loop: uncertaintyquantification over aleatory (probability) variables (e.g., the probability distribution). The outer loopdeterminesintervalonstatistics,(e.g.,mean,variance).Theinnerloopusessamplingtodeterminetheresponses with respect to the aleatory variables. This information can be used to understand theepistemicandaleatoryuncertainties,relativetotheLowerPerformanceRequirement(LPR).

57

Figure40.PullingTogetherConceptAssociatedwithQMU

Theinformationisrelevanttotheriskframeworkasitprovidesevidenceaboutmethodologiesandtoolsto deal with several of the topics. QMU and Dakota are still evolving, and there are a number ofchallenges:

§ Howdoweensurethatweusetheright“data”asinputs?§ Howtorolluptothesystemlevel?§ Modelvalidationandsimulationqualification

5.8 MODELINGMETHODSFORRISK

Theriskmodelingandanalysismethodsaddressespotentialerrorsanduncertaintiesintheoveruseoflimiteddata.These typesofmodelscaptureandembedknowledgeassociatedwithexpert judgment,historicalevidenceandrulesofthumbsthatareusedinthedecision-makingprocess.Alternativemethodshelpdealwiththesetypeofissues.ThisparticularexampleusesaBayesianmodel[117].

5.8.1 PREDICTIVEMODELSFORRISK

Therearesituationswherewedonothavegoodhistoricalquantitativedataandweoftenuseexpertjudgment. This section discusses a predictive modeling approach when risk involves subjectiveinformation,smalldatasets,and“dirty”data[82].

58

The SERC teamhas developed andusedmodels in the prediction of risk, and plans to use predictiveanalyticmodels tosupport risk identificationandmanagement.Moregenerallywecanusemodels toprovideriskquantificationforalmostalltypesofdecisionsthataremadebystakeholders(e.g.,model-basedreviews)[116][117].Asanexample,wecreatedaBayesianmodelusingfactorsderivedfromtheAirworthiness standardMIL-HDBK-516B [53]as shown inFigure41.This is conceptually similar to theapproach we are using on an FAA NextGen research task for collaborative risk-informed decision-making [17] [18] [19]. The key characteristics of the approach are they ensure that all factors areconsidered in the decision-making process, and that all classes of stakeholders are adequatelyrepresentedinthedecision-makingprocess.Asystematicandcomprehensivetreatmentofallrelevantfactorsprovidesbetterriskidentification.

Weused thismodelandanexample froma truestory related toaC130WeaponDelivery systemtoillustratetheconcept.Whilethismodelisnotionalatthistime,thisexamplestartedadiscussionwiththeteam about how stochastic (probabilistic) models can play an important part of the Vision as theyformalizemanyaspectsof thehumandecisionmakingprocess thatwill be important atmanygates,reviews,anddecisionpointsoftheVisionconcept.Eachfactorcoversaspecificaspectofairworthiness,toensurethatallpossibleuncertaintiesandriskareconsideredinthequantificationofrisk.Theriskindexisaprobabilitydistribution,whereforexample,themeancanmaptoquantitiesinariskmatrix.

Figure41.BayesianModelDerivedfromAirworthinessFactors

5.9 CONTROLLEDNATURALLANGUAGEREQUIREMENTSINFORMATION

Weacknowledgethattheremaynotbevalueinmodelingeverything,andbelievearisk-drivenapproachtomodelingshouldbeconsidered.Inaddition,iftheconceptsassociatedwiththevisionarevalid,subjectmatterexpertsusingarichwebview(seeinFigure10)maywanttosupplementmodelswithothertypes

59

of constraints. We discussed in working session research into Controlled Natural Language (CNL)Requirementsandontology-drivenNaturalLanguageProcessingofrequirements[5].Thefundamentalpremiseisthatwecanstructuretextual-basedrequirementsthatwecanthenuseautomatedmeanstoformalizetherequirementsforanalysisofconsistencyandcompletenessinthecontextofanontology;there have been a number of different research efforts that have demonstrated the successfultransformationsofCNLrequirementsintoananalyzableform.Inaddition,wealsobelievethatwhilewewilltransitiontotheuseofmodels,therewillalwaysbesubjectmatterexpertsthatwillaugmenttherepresentationfrommodelswithconstraintsusingsomeformoftextual-basedspecification.Notealso,thatinourindustryvisitsthereweretwoorganizationsthatexplicitlydiscussedrequirementgenerationfrommodels,discussedaspartoftheDARPAMETAproject,andhasbeendiscussedbytheEngineeredResilientSystemresearch.

§ PurposeforCNLo Constrainsthewayrequirementstatementsareconstructedo Supportstool-basedanalysiso Improvesconsistencyo Allowsfortemplate-basedgenerationofformalizedandanalyzedrequirementso CanintegratewithRichModelinginSST

§ Approachexample–usedbybothLockheedMartinC5(andpresentedinopenforum)o Goal:specifythebehavioroftheoutputsintermsoftheinputso Uselimitedsetofactionverbscombinedwithstructured,repeatablephrasing(syntax)for

requirements,andimproveunderstandingbetweenthespecifieranddeveloper/reviewer• Eliminatesconfusioncausedbymultipletermsusedforthesamepurpose• Examples:derivevs.computevs.calculatevs.determinevs.process…• Alloftheseessentiallymean“executethelogical/algorithmicstepstosettheoutput

basedontheinput(s).Ӥ Thisprovidesapatternforanalysisanddevelopmentofrequirements

o Providerequirementsdefinetheoutputso Deriverequirementsspecifythealgorithmsforproducingthetermswhichareoutputo Acquire&Validaterequirementsidentifytheinputsignalsneededtoderivetheterms

• Definitionofactionverbshelpsensureallissuesgetaddressed• ValidationofInput• ErrorHandling• Source/DestinationSpecification

§ Exampleo DATAACQUISITION:

• MissionProcessingSystemshallacquire<alias>.o DATAVALIDATION:

• MissionProcessingSystemshallperformdatavalidationon<alias>perTable<table-id>.• MissionProcessingSystemshallset<validity_alias>to<enumeration>when<all|any>

respectivedatavalidationchecksinTable<table-id><pass|fail>.§ Hasbeendevelopedusingaspreadsheettocontrolstructure,verbs,etc.

5.10 CROSS-DOMAININTEGRATIONANDNATURALLANGUAGEPROCESSOFREQUIREMENTSUSINGONTOLOGIES

Dr.MarkAustinandDr.LeonardPetngafromUniversityofMaryland(UMD)joinedtheSERCresearchteam. Mark discussed various applications for using “Semantic-driven Modeling and Reasoning forSystemsEngineeringTransformation.”

60

Wehavediscussedtheuseofinformationmodelsandontologies,andmorespecificallytheWebOntologyLanguage(OWL),whichisakeystandardlanguageunderlyingsemanticwebtechnologies.

While it may not be completely apparent how semantic web technologies contributes to MCE, theInternational Council on Systems Engineering (INCOSE) Model Based System Engineering (MBSE)Roadmapidentifiesbothontologiesasabuildingblockfordistributedrepositoriesforcrossingmultipledomain.WeverymuchbelievethesearecriticalandunderlietheSTTconcept.Simplistically,anontologyinOWLisatypeofevolvableschema,andwewanttodefinethevariousdomainsrelevanttoNAVAIRandensurewecanrelateinformationcrossthosedomains.

Markdescribedanumberofapplicationsthatusethisunderlyingtechnology,buttworesonatedwiththeworkingsessionaudience:

§ AnapproachtoNaturalLanguageProcessing(NLP)ofRequirementontologieso ThisisbeingadvancedbyaUMDmastersstudentTedCarneyo MarkAustinplanstocontinuethisresearchbeyondthatofTed’smastersprojecto Theapproachandworkingprototype:

• Restructurerequirementsstatementbasedontemplatesofwell-formedrequirements• Identifyinconsistenciesandincompleteness

§ Applicationfordistributedandcross-domainintegrationo ThisbuildsonLeonard’sPhDdissertationandassociatedprototype[109]

OurNAVAIRsponsordiscussedthepotentialdesiretoleveragetheNPLandsemanticwebcapabilitiestoautomatethe“update”plannedfortheStandardWorkPackages.NAVAIRprovidedabout20SWPandweanalyzedsomeexamplesandcameupwithsomerecommendationstoresearchimprovingthewaythatSWParecreated,managedandused.Theinitialanalysisincluded:

§ DriversforModel-centricApproachestoAuthoringStandardWorkPackages(SWP)§ RepresentationandissueswiththesampleSWPsprovidedtous

61

§ Strategicapproachesto(Re)generationofDefenseAcquisitionProcessdocumentsandSWPinthecontextofincreasinglycomplex,diverse&changingregulatory,technologicalprogramenvironments

§ Implementationstrategiesandtechnologies,andoperationalscenarios§ Reviewofresearchthatdemonstratedabilitytoautomaticallygenerateactivitydiagrams

(processflows)fromunderlyingmodel(thisisaconfirmatorystory)

Finally, we believe that it would be interesting to apply the NPL processing to some of the NAVAIRrequirements.WedidhearatapriormeetingwithDavidFieldsthatrequirements(forMQ-25)arestoredinadatabase,andingeneralarewellstructured.Ifthoserequirementswerenonclassified,wecouldusethemforanalyzingandperformingNPLontextualrequirementstatementsinfutureresearch.

6 TASK4–DEFINESYSTEMENGINEERINGTRANSFORMATIONROADMAP

OurplansfortheroadmapatthestartofRT-157alignedwiththeprioritizedsetofgoalshowninTable1,whichalsohighlightstraceabilitytoTechnology,Methods,andCompetencies.TheexpecteddevelopmentofaroadmapfocusedontransitioningfromthetraditionalSETRapproach,toarequirement,risk,andevidence-basedapproachusinganevolvingunderlyingSST.WeplannedtofocusonMDAOtobemoresystematicintradespaceoftheproblemspace.ThiswasalsofocusedatthePORlevel.

However,theaccelerationoftheSEtransformationasalteredthatplan.Wearenowworkingourgapsandchallengesassociatedwiththenewframework,whichinclude,butarenotlimitedto:

§ UsinganinteractiveapproachtoMDAOinacollaborativeefforttodevelopanewtypeofspecification,RFP&SOW

§ DevelopingstrategiestotrackandassessvalueofrequirementstoKSA§ Investigatingacollaborativeoperationalparadigmbetweengovernmentandindustry§ Acceleratingtheawarenessofmodelingmethodsforthecompetencies(seeSection5)§ Consideringalternativedigitalengineeringstrategiesforevaluatingaproposalduringsource

selection

Insupportofthenewoperationalparadigm,wewererequestedbyoursponsortoattendaworkshoptodiscuss strategies for how digital models could be used in the DoD acquisition process to supportcompetitivedown-selectionfor:

§ CompetitiveprototypingintheTechnologyMaturationandRiskReductionphase§ EngineeringandManufacturingDevelopment

Theobjectivesoftheworkshopwere:

§ Obtaincommunityinputonthetypesofdigitalmodels(design,cost,performance,mission,etc.)thatcouldbeusedtosupportcompetitivedown-selection

§ Identifytheexistinggapsthatmustbeclosedtoenabletheuseofdigitalmodelstosupportcompetitivedown-selection

§ RecommendchangesintheDoDacquisitionprocessandGovernment-Industryinteractionstoenabletheuseofdigitalmodelstosupportcompetitivedown-selection

§ ObtaincommunityinputonthepolicyandlegalissuesassociatedwiththeuseofdigitalmodelsinRequestsforProposals(RFPs),proposals,andduringproposalevaluationtosupportcompetitivedown-selection

62

Therefore,aspartofthenewlyawardedRT-170,wewillupdatetheroadmaptoincludethenewtaskasdiscussedinSection2.5,andconsidertopicsdiscussedinworkingsessions:

§ Virtualizationandevent-drivenreviews§ ImplicationsofaComputationallyEnabledSystemEngineeringthroughtheformalizationofthe

DecisionFrameworko ConcepttoembedSystemEngineeringthroughcross-domainlinkagesandrelationshipsin

theInformationModelthatunderliesISEEo Canwemeasurerequirementstabilization?

§ SignificantdiscussionabouttheimplicationsofSoftwareIntensiveCyberPhysicalSystem(CPS)o InsightsgainedfromworkingwithJSFo RevisitRT-142onRiskLeadingIndicatoronSW-intensiveCPS

§ Modelsenablenewpossibilitiesforverificationatproposalevaluation§ Collaborativeapproachtodecisions-in-the-loop

§ Measureofdesignmaturing§ IntegratedSystemEngineeringEnvironment(ISEE)

o Variantmanagement,whichrelatescapturingtradespaceanalysiso Template-basedapproachtorequirementgeneration(andpossiblycontractgeneration)

§ Formalizingcontractingo Examplesincludecontractsspecificationlanguage(GCSL)developedbytheDesigningfor

AdaptabilityandevolutioNinSystemofsystemsEngineering(DANSE)projecto MakeContractDataRequirementsList(CDRLs)aboutVerificationandValidation(V&V)and

ModelIntegrity§ Earlyplanningforsustainment

o DigitalTwin–“Anintegrated,multiphysics,multiscale,probabilisticsimulationofanas-builtsystem,enabledbyDigitalThread,thatusesthebestavailablemodels,sensorinformation,andinputdatatomirrorandpredictactivities/performanceoverthelifeofitscorrespondingphysicaltwin.”

§ Riskso AirworthinessandSafety(mostcriticalinTechnicalFeasibilityassessment)o Programexecution(cost,scheduleandperformance)o Competenciesmaynotbereadyforthefirstpilot(seeFigure1)

§ UseofasurrogatepilotprojecttoevaluatethisapproachtoMCE-basedcontractingandcollaboration

7 SERCRESEARCHSYNERGIES

ThissectiondiscussessomesynergiestotheongoingNAVAIRresearchtasksthatarebrieflymentionedinthisreporttoinformreadersoftherelationshipstotheseotheractivities.

7.1 RT-141INTEGRATEDFRAMEWORKFORRISKIDENTIFICATIONANDMANAGEMENT

Many of the topics from the RT-141 final report [21] describing model integrity and modelingmethodologiesandtoolsforarisk-basedframeworkarestillrelevant,buthavenotbeenincludedinthisreport. We believe that many of these topics are still potential challenges in the new operationalparadigmcharacterizedbytheSETframework.

63

Model-centricdevelopmentintroducesadifferentrisk–theriskofuncriticalreviewofthemodelingandanalysismethodsandresults.Indocument-centricdevelopmentthereisusuallyhealthskepticism,andrelianceonexperiencedsubjectmatterexpertstoreviewthedevelopmentdocuments.Inmodel-centricdevelopment,DoDwill need todevelopa cadreof expertswithexpertiseboth in thedomainand inmodelingandanalysismethods.

7.2 RT-168DECISIONFRAMEWORK

Our NAVAIR sponsor was also informed about the relationships between Systems Engineering (SE)activities and the decision framework (related to Dr.Matt Cilli’s dissertation [39]) under the RT-168research task forU.S.ArmyArmamentResearch,DevelopmentandEngineeringCenter (ARDEC). Thisframeworkrelatestothedigitalthreadconceptthatcanshowhowtoleverageanalysisineachoftheareastodevelopadigitalthreadtosupportrepeatableanalysis,wherea“fully”integratedoperationalanalysisismissingcurrently.

WeareresearchingtoassessiftheDecisionFrameworkcandemonstratehowdatafromtheunderlyinginformation model (SST) can be used to populate the Decision Framework as implemented in theArmament Analytics Multiple Objectives Decision Analysis Tool (AAMODAT) tool with potentialrefinementsandextensions.Webelievethiscapabilityservesmanypurposes:

§ ProvideseniormanagementandprogrammanagerswithvisualrepresentationsofkeytradeoffdefinedintermsofPerformance,Cost,TimeandRiskasshowninFigure42

§ Scatterplotshowsinasinglecharthowallsystemlevelalternativesrespondinmultipledimensionsofstakeholdervalue

§ AssessmentFlowDiagrams(AFDs)tracetherelationshipsbetweenphysicalmeans,intermediatemeasures,andfundamentalobjectives

§ ProvidesmethodologicalguidanceforidentifyingKeyPerformanceParameters(KPPs)§ Canbeusedwithuncertaintyanalysisasameasureforunderstandingmaturingdesign§ Enablesbi-directionalanalysisthroughoutlifecycle

64

Figure42.DecisionSupportModelConstruct

7.3 RT-176VERIFICATIONANDVALIDATION(V&V)OFSYSTEMBEHAVIORSPECIFICATIONS

Our NAVAIR sponsor had requested that the SERC RT-176 research task being led by Dr. KristinGiammarcobealignedwiththeongoingresearchfromRT-157andRT-170.RT-176aimstoleverageandextend existing research in the area of methods, processes and tools (MPT) for performing earlyVerification & Validation (V&V) of requirements and architecture models managed within itsorganization, and to educate its workforce in the use of automated tools for conducting early andcontinuousV&Vacrosstheentirelifecycle.WehavesharedourUAVsystemmodeldiscussedinSection5with Kristin. We hope that this model will be developed as a surrogate to actual systems underdevelopmentatNAVAIRforuseasacasestudytotestneworimprovedMPTsthataredevelopedbasedonthosesummarizedinthebackgroundandasaresultofthistask,whichareexpectedtoapplytoothersystemsinmanydomainsthroughoutDoD.

7.4 AEROSPACEINDUSTRYASSOCIATIONCONOPSFORMBSECOLLABORATION

This isa follow-upto theeffortcompleted lastyearwhichdevelopedawhitepaperontheLifeCycleBenefitsofCollaborativeMBSEUseforEarlyRequirementsDevelopment[3].ThiswhitepaperdiscussesthecurrentstateandbenefitsofMBSEacrosstheentirelifecycleandprovidesproposalsforaddressingsuchissuesasMBSECollaborativeFramework,GovernmentDataRights,IntellectualProperty,andLifeCycleEffectivenesswithMBSE.

TheeffortforthisyearinvolvesmanyoftheindustrycontractorstoNAVAIRandDoD.TheresultsshouldproduceawhitepaperdescribingaCONOPSforhowindustryandgovernmentcancollaboratethroughMCE.

65

8 PARTIISUMMARY

Ourresearchsuggeststhatmodel-centricengineeringisinuseandadoptionseemstobeaccelerating.Asdescribedherein,oursponsorrecognizedtheneedtomakearadicaltransformationandaredevelopingastrategicplanbasedonanewoperationalparadigmforacquisitionanddesigntoaccelerateSET.Weareadaptingourresearchstrategyandfocustoalignwiththeirevolvingplan.Thismessagehasbeensharedmore broadlywith SERC sponsors, government sponsors of SERC research, and industry boththroughSERCandNDIAevents.

InarecentGovernment-IndustryModelCentricEngineeringforumconductedbytheSystemsEngineeringResearchCenter(SERC)andtheOfficeoftheUndersecretaryofDefense,thefollowingfourperceivedareasofbenefitswerefoundtobethekeythemestoimplementing[40]:

1. Improved Acquisition – accepting digital deliverables could improve the governmentsunderstandingofaprojectsstatusandriskalongwithallowingthemto“validate”thecontractor’sdeliverables.

2. ImprovedEfficiencyandEffectiveness– reduce timeandeffort in theperformanceofexistingtasksusingadigital“twin”ofthesystem.

3. ImprovedCommunication;BetterTrade-SpaceExploration;ReducedRisk–usingontology-basedinformationmodelstotranslateandextractusefulinformationbetweenavarietyofmodelsandmodeltypescouldallowforimprovedcommunicationamongspecialists.

4. ImprovedDesignsandresultingSystemsandSolutions–beingabletounderstandtheimpactofrequirement and/or design decisions early could help improve the overall system design andidentifyadverseconsequencesofthedesignbeforecommittingtoadesignchoice.

ThefutureresearchofMCEwillneedtotakeintoaccountthesefourperceivedareasofbenefitandhelpmakeprogresstowardthesedimensions.ThepathforwardtotransitioningtoMCEhasbothchallengesandmany opportunities, both technical and sociotechnical. Themodeling infrastructure for a digitalengineeringenvironmentisacriticalsteptoenableaSST,whichwebelievecanbetterlinkinformationacrossdomainsforbetterandearlierdecisionmaking.WhiletherearethousandsoftoolstheyareoftenfederatedandthereiscurrentlynotonesinglesolutionthatcanbepurchasedtospantheMCElifecycle.Everyorganizationprovidinginputstothisresearchhashadtoarchitectandengineertheirownmodel-centric engineering environment. Most have selected commercial tools and then developed theintegratingfabricbetweenthedifferenttools,model,anddata.Thisoftenuniquelypositionsthemwithsome advantages among others in theirs industry. Some organizations have encoded historicalknowledge in reference models, model patterns to embed methodological guidance to supportcontinuousorchestrationofanalysisthroughnewmodelingmetrics,automatedworkflow,andmore.OurimmediatechallengeisprovideresearchtotheSET“rollout”strategyasreflectedinFigure1.

66

9 ACRONYMSANDABBREVIATION

Thissectionprovidesalistofsomeofthetermsusedthroughoutthepaper.Themodellexiconshouldhaveallofthesetermsandmanyothers.

AADL ArchitectureAnalysis&DesignLanguageACAT AcquisitionCategoryAFT ArchitectureFrameworkToolofNASA/JPLAGI AnalyticalGraphics,Inc.AGM AcquisitionGuidanceModelANSI AmericanNationalStandardsInstituteAP233 ApplicationProtocol233ATL ATLASTransformationLanguageASR AlternativeSystemReviewAVSI AerospaceVehicleSystemsInstituteBDD SysMLBlockDefinitionDiagramBN BayesianNetworkBNF BackusNaurFormBOM BillofMaterialBPML BusinessProcessModelingLanguageCAD Computer-AidedDesignCASE Computer-AidedSoftwareEngineeringCDR CriticalDesignReviewCEO ChiefExecutiveOfficerCESUN InternationalEngineeringSystemsSymposiumCMM CapabilityMaturityModelCMMI CapabilityMaturityModelIntegrationCORBA CommonObjectRequestingBrokerArchitectureCREATE ComputationalResearchandEngineeringforAcquisitionToolsandEnvironmentsCWM CommonWarehouseMetamodeldB DecibelDBMS DatabaseManagementSystemDAG DefenseAcquisitionGuidebookDARPA DefenseAdvancedResearchProjectAgencyDAU DefenseAcquisitionUniversityDCDR DigitaldesignfromCriticalDesignReview(CDR)DL DescriptiveLogicDoD DepartmentofDefenseDoDAF DepartmentofDefenseArchitecturalFrameworkDoE DesignofExperimentsDSL DomainSpecificLanguagesDSM DomainSpecificModelingDSML DomainSpecificModelingLanguageE/DRAP EngineeringDataRequirementsAgreementPlanERS EngineeredResilientSystemsFAA FederalAviationAdministrationFMEA FailureModesandEffectsAnalysisFMI FunctionalMockupInterfaceFMU FunctionalMockupUnit

67

GAO GovernmentAccountingOfficeHPC HighPerformanceComputingHPCM HighPerformanceComputingModernizationHW HardwareI&I IntegrationandInteroperabilityIBM InternationalBusinessMachinesIBD SysMLInternalBlockDiagramICD InterfaceControlDocumentICTB IntegratedCapabilityTechnicalBaselineIDEF0 IcamDEFinitionforFunctionModelingIEEE InstituteofElectricalandElectronicsEngineersINCOSE InternationalCouncilonSystemsEngineeringIPR IntegrationProblemReportIRL IntegrationReadinessLevelISEF IntegratedSystemEngineeringFrameworkdevelopedbyArmy’sTARDECISO InternationalOrganizationforStandardizationIT InformationTechnologyIWC IntegratedWarfighterCapabilityJCIDS JointCapabilitiesIntegrationandDevelopmentSystemJEO JupiterEuropaOrbiterprojectatNASA/JPLJSF JointStrikeFighterJPL JetPropulsionLaboratoryofNASAKPP KeyPerformanceParameterKSA KeySystemAttributesLinux AnoperatingsystemcreatedbyLinusTorvaldsLOC LinesofCodeM&S ModelingandSimulationMARTE ModelingandAnalysisofRealTimeEmbeddedsystemsMATRIXx Productfamilyformodel-basedcontrolsystemdesignproducedbyNational

Instruments;SimilartoSimulinkMBEE Model-basedEngineeringEnvironmentMBSE Model-basedSystemEngineeringMBT ModelBasedTestingMC/DC ModifiedCondition/DecisionMCE Model-centricengineeringMDA® ModelDrivenArchitecture®MDD™ ModelDrivenDevelopmentMDE ModelDrivenEngineeringMDSD ModelDrivenSoftwareDevelopmentMDSE ModelDrivenSoftwareEngineeringMIC ModelIntegratedComputingMMM ModelingMaturityModelMoDAF UnitedKingdomMinistryofDefenceArchitecturalFrameworkMOE MeasureofEffectivenessMOF MetaObjectFacilityMOP MeasureofPerformanceMVS MultipleVirtualStorageNASA NationalAeronauticsandSpaceAdministrationNAVAIR U.S.NavyNavalAirSystemsCommand

68

NAVSEA U.S.NavalSeaSystemsCommandNDA Non-disclosureAgreementNDIA NationalDefenseIndustrialAssociationNEAR NavalEnterpriseArchitectureRepositoryNPS NavalPostgraduateSchoolOCL ObjectConstraintLanguageOMG ObjectManagementGroupOO ObjectorientedOSD OfficeoftheSecretaryofDefenseOSLC OpenServicesforLifecycleCollaborationOV1 OperationalView1–typeofDoDAFdiagramOWL WebOntologyLanguagePDM ProductDataManagementPDR PreliminaryDesignReviewPES PhysicalExchangeSpecificationPIA ProprietaryInformationAgreementPIM PlatformIndependentModelPLM ProductLifecycleManagementPOR ProgramofRecordPRR ProductionReadinessReviewPSM PlatformSpecificModelQMU QuantificationofMarginsandUncertaintyRT ResearchTaskRFP RequestforProposalROI ReturnOnInvestmentSAVI SystemArchitectureVirtualIntegrationSE SystemEngineeringSERC SystemsEngineeringResearchCenterSETR SystemEngineeringTechnicalReviewSimulink/Stateflow Productfamilyformodel-basedcontrolsystemproducedbyTheMathworksSCR SoftwareCostReductionSDD SoftwareDesignDocumentSE SystemEngineeringSFR SystemFunctionalReviewSLOC SoftwareLinesofCodeSME SubjectMatterExpertSOAP AprotocolforexchangingXML-basedmessages–originallystoodforSimpleObject

AccessProtocolSoS SystemofSystemsSoftwareFactory TermusedbyMicrosoftSRR SystemRequirementsReviewSRS SoftwareRequirementSpecificationSTOVL ShorttakeoffandverticallandingSVR SystemVerificationReviewSW SoftwareSysML SystemModelingLanguageTARDEC USArmyTankAutomotiveResearchTBD ToBeDeterminedTRL TechnologyReadinessLevel

69

TRR TestReadinessReviewUML UnifiedModelingLanguageXMI XMLMetadataInterchangeXML eXtensibleMarkupLanguageUS UnitedStatesXSLT eXtensibleStylesheetLanguagefamily(XSL)TransformationxUML ExecutableUMLUnix AnoperatingsystemwithtrademarkheldbytheOpenGroupUQ UncertaintyQuantificationVHDL VerilogHardwareDescriptionLanguageV&V VerificationandValidationVxWorks OperatingsystemdesignedforembeddedsystemsandownedbyWindRiver10 TRADEMARKS

AnalysisServerisaregisteredtrademarkofPhoenixIntegration,Inc.AstahSysMLisCopyrightofChangeVision,Inc.BridgePointisaregisteredtrademarkofMentorGraphics.CameoSimulationToolkitisaregisteredtrademarkofNoMagic,Inc.COREisaregisteredtrademarkofVitechCorporation.IBM™isatrademarkoftheIBMCorporationiGrafxisaregisteredtrademarkofiGrafx,LCC.Java™andJ2EE™aretrademarkofSUNMicrosystemsJavaistrademarkedbySunMicrosystems,Inc.LDRAisaregisteredtrademarkofTrademarkofLDRALtd.andSubsidiaries.LinuxisaregisteredtrademarkofLinuxMarkInstitute.Mathworks,Simulink,andStateflowareregisteredtrademarksofTheMathworks,Inc.MagicDrawisatrademarkofNoMagic,Inc.MATRIXxisaregisteredtrademarkofNationalInstruments.Microsoft®,Windows®,Windows NT®,Windows Server® andWindows VistaTM are either registeredtrademarks or trademarks of Microsoft Corporation in the United States and/or other countries.ModelCenter,isaregisteredtrademarkofPhoenixIntegration,Inc.Modelica®isaregisteredtrademarkoftheModelicaAssociation.Object Management Group (OMG): OMG's Registered Trademarks include: MDA®, Model DrivenArchitecture®,UML®,CORBA®,CORBAAcademy®,XMI®OMG's Trademarks include, CWM™, Model Based Application Development™, MDD™, Model BasedDevelopment™,Model BasedManagement™,Model BasedProgramming™,ModelDrivenApplicationDevelopment™,ModelDrivenDevelopment™Model Driven Programming™, Model Driven Systems™, OMG Interface Definition Language (IDL)™,UnifiedModelingLanguage™,<<UML>>™OMG®,MDA®,UML®,MOF®, XMI®, SysML™, BPML™ are registered trademarks or trademarks of theObjectManagementGroup.OracleandJavaareregisteredtrademarksofOracle,Inc.and/oritsaffiliates.ParaMagicisaregisteredtrademarkofInterCAX,Inc.PHXModelCenterisaregisteredtrademarkofPhoenixIntegration,Inc.PowerPointisaregisteredtrademarkofMicrosoft,Inc.Real-timeStudioProfessionalisaregisteredtrademarkofARTiSANSoftwareTools,Inc.

70

RhapsodyisaregisteredtrademarkofTelelogic/IBM.RoseXDEisaregisteredtrademarkofIBM.SCADEiscopyrightedtoEsterelTechnologies.SimulinkisaregisteredtrademarkofTheMathWorks.StateflowisaregisteredtrademarkofTheMathWorks.StatemateisaregisteredtrademarkofTelelogic/IBM.STKisaregisteredtrademarkofAnalyticalGraphics,Incorporated(AGI),Inc.UNIXisaregisteredtrademarkofTheOpenGroup.VAPSisregisteredateNGENUITYTechnologies.VectorCASTisaregisteredtrademarkofVectorSoftware,Inc.VisioisaregisteredtrademarkofMicrosoft,Inc.VxWorksisaregisteredtrademarkofWindRiverSystems,Inc.WindowsisaregisteredtrademarkofMicrosoftCorporationintheUnitedStatesandothercountries.XML™isatrademarkofW3CAllothertrademarksbelongtotheirrespectiveorganizations.

71

11 REFERENCES

[1] Ackoff,R.,L,andSheldonRodin.RedesigningSociety.Stanford:StanfordUniversityPress,2003.[2] Adams,B.,AdamStephens,DakotaSensitivityAnalysisandUncertaintyQuantification,withExamples,SNL

6230CourseonUQ/SA,April23,2014.[3] Aerospace Industry Association, Life Cycle Benefits of Collaborative MBSE Use for Early Requirements

Development,April2016,http://www.aia-aerospace.org/report/life-cycle-benefits-of-collaborative-mbse-use-for-early-requirements-development/.

[4] Allen,G.,F.Hartman,F.Mullen,DynamicMulti-levelModelingFramework,ResultsoftheFeasibilityStudy,NDIA,October2013.

[5] Arellano A., Zontek-Carney E., Austin M.A. Frameworks for Natural Language Processing of TextualRequirements. International Journal On Advances in Systems and Measurements, 8(3-4):230–240,December2015.

[6] ARTEMIS-GB-2012-D.46–Annex2,2013. https://ec.europa.eu/research/participants/portal/desktop/en/opportunities/fp7/calls/artemis-2013-1.html

[7] Baitch, L., Randall C. Smith, Physiological Correlates of Spatial Perceptual Discordance in a VirtualEnvironment,GeneralMotorsResearch&DevelopmentCenterVirtualEnvironmentsLaboratory.

[8] Bankes,S.,D.Challou,D.Cooper,T.Haynes,H.Holloway,P.Pukite,J.Tierno,C.Wentland,METAAdaptive,Reflective,RobustWorkflow(ARRoW),Phase1bFinalReport,TR-2742,October,2011.

[9] Bapty, T., S. Neema, J. Scott,Overviewof theMETA Toolchain in theAdaptive VehicleMake Program,Vanderbilt,ISIS-15-103,2015.

[10] Bayer, Todd J., Matthew Bennett, Christopher L. Delp, Daniel Dvorak, J. Steven Jenkins, and SandaMandutianu.“Update-ConceptofOperationsforIntegratedModel-CentricEngineeringatJPL,”1–15.IEEE,2011.doi:10.1109/AERO.2011.5747538.

[11] Bayer,Todd,SeungChung,BjornCole,BrianCooke,FrankDekens,ChrisDelp,I.Gontijo,etal.“11.5.1EarlyFormulationModel-CentricEngineeringonNASA’sEuropaMissionConceptStudy.”INCOSEInternationalSymposium22,no.1(July2012):1695–1710.doi:10.1002/j.2334-5837.2012.tb01431.x.

[12] Bergenthal,J.,FinalReportontheIdentificationofModelingandSimulationCapabilitiesbyAcquisitionLifeCycle Phases, Johns Hopkins University/Applied Physics Laboratory, 16th Annual Systems EngineeringConference,October,2013.

[13] Bergenthal,J., J.Coolahan,FinalReportontheIdentificationofModelingandSimulationCapabilitiesbyAcquisitionLifeCyclePhases,NDIASystemsEngineeringDivisionMeeting,February2014.

[14] Bhatt,D.,K. Schloegel,G.Madl,D.Oglesby. QuantifyingErrorPropagation inDataFlowModels. 20thAnnual IEEE International Conference andWorkshops on the Engineering of Computer Based Systems.2013.

[15] Blackburn,M.R.,What’sModelDrivenEngineering(MDE)andHowCanitImpactProcess,People,ToolsandProductivity, Systems and Software Consortium, Technical Report SSCI-2008002-MC, September, 2008http://www.knowledgebytes.net/downloads/Whats_MDE_and_How_Can_it_Impact_me.pdf.

[16] Blackburn,M.R.,Model-DrivenVerificationandValidation,Safe&SecureSystems&SoftwareSymposium,June,15-172010.ModifiedfromPaulEremenko,METANovelMethodsforDesign&VerificationofComplexSystems,December22,2009.

[17] Blackburn,M.,A.Pyster,R.Dillon-Merrill,T.Zigh,R.Turner,ResultsfromApplyingaModelingandAnalysisFrameworktoanFAANextGenSystemofSystemsProgram,NDIA,October,2013.

[18] Blackburn,M.,A.Pyster,R.Dillon-Merrill,T.Zigh,R.Turner,ModelingandAnalysisFrameworkforRisk-InformedDecisionMakingforFAANextGen,INCOSE,June2013.

[19] Blackburn,M.,A. Pyster, R.Dillon-Merrill, T. Zigh, R. Turner,UsingBayesianNetworks forModeling anAcquisitionDecision-MakingProcessfortheFAANextGenSystemsofSystems,NDIA,October,2012.

[20] Blackburn,M.,R.Busser,A.Nauman,andT.Morgan.“LifeCycleIntegrationUseofModel-BasedTesting

72

Tools,”2:10.D.4–1–10.D.4–13.IEEE,2005.doi:10.1109/DASC.2005.1563402.[21] Blackburn, M. R., M. Bone, and G. Witus, “Transforming System Engineering through Model-Centric

Engineering,”StevensInstituteofTechnology,SERC-2015-TR-109,Nov.2015.[22] Blackburn, M., R. Busser, H. Graves, Guidelines for Automated Analysis of System Models, Software

ProdutivityConsortiumTechnicalReport,December,2000.[23] Blackburn, M., Cloutier, R., Hole, E., Witus, G., 2014. Introducing Model-Based Systems Engineering

Transforming System Engineering throughModel-Based Systems Engineering (Technical ReportNo. TR-044).SystemsEngineeringResearchCenter.

[24] Blackburn,Mark,RobertCloutier,EirikHole,andGaryWitus.IntroducingModel-BasedSystemsEngineeringTransformingSystemEngineeringthroughModel-BasedSystemsEngineering.TechnicalReport.SystemsEngineeringResearchCenter,March31,2014.http://www.sercuarc.org/library/view/58.

[25] Blackburn,M.,R.Cloutier,G.Witus,E.Hole,M.Bone,TransformingSystemEngineeringthroughModel-CentricEngineering,SERC-2014-TR-044-2,January,2015.

[26] Blackburn,M.,P.Denno,VirtualDesignandVerificationofCyber-physicalSystems:IndustrialProcessPlantDesign, Conference on Systems Engineering Research, March, 2014;http://dx.doi.org/10.1016/j.procs.2014.03.006.

[27] Blackburn,M.,P.Denno,UsingSemanticWebTechnologiesforIntegratingDomainSpecificModelingandAnalyticalTools,ComplexAdaptiveSystemsConference,Nov.2015.

[28] Blackburn,M.,S.Kumar,EvolvingSystemsEngineering throughModelDrivenFunctionalAnalysis,NDIASystemEngineeringConference,October2009.

[29] Bleakley,G.,A.Lapping,A.Whitfield,DeterminingtheRightSolutionUsingSysMLandModelBasedSystemsEngineering,(MBSE)forTradeStudies,INCOSEInternationalSymposium,June,2011.

[30] Boehm,B.,SoftwareCostEstimationwithCocomoII,PrenticeHall,2000.[31] Bone,M.A.,M.Blackburn,G.Witus,H.Eirik,andR.Cloutier,“Model-CentricEngineering,”presentedat

the2016ConferenceonSystemsEngineeringResearch,Huntsville,Alabama,2016.[32] Box, George E. P. Empirical Model-Building and Response Surfaces. Wiley Series in Probability and

MathematicalStatistics.NewYork:Wiley,1987.[33] Brat, Guillaume, V & V of Flight-Critical Systems, NASA ARCS5 - Safe & Secure Systems & Software

Symposium,June2010.[34] Broy, M., M. Feilkas, M. Herrmannsdoerfer, S. Merenda, and D. Ratiu. “Seamless Model-Based

Development:FromIsolatedToolstoIntegratedModelEngineeringEnvironments.”ProceedingsoftheIEEE98,no.4(April2010):526–45.doi:10.1109/JPROC.2009.2037771.

[35] Business Process Modeling Notation. Retrieved March 2010, from Wikipedia, The Free Encyclopedia:http://en.wikipedia.org/wiki/Business_Process_Modeling_Notation.

[36] Browne,D.,R.Kempf,A.Hansena,M.O'Neal,W.Yates,EnablingSystemsModelingLanguageAuthoringina CollaborativeWeb-basedDecision Support Tool, Conference on System Engineering Research (CSER),March,2013.

[37] Castet,Jean-Francois,MatthewL.Rozek,MichelD.Ingham,NicolasF.Rouquette,SeungH.Chung,J.StevenJenkins,DavidA.Wagner,andDanielL.Dvorak.“OntologyandModelingPatternsforState-BasedBehaviorRepresentation.”AmericanInstituteofAeronauticsandAstronautics,2015.doi:10.2514/6.2015-1115.

[38] Chilenski,J.,SAVIPrincipalInvestigator,DonWard,TEESSAVIProgramManager,NDIAM&SSubcommittee–Arlington,Virginia8April2014.

[39] Cilli,M.SeekingImprovedDefenseProductDevelopmentSuccessRatesThroughInnovationstoTrade-OffAnalysisMethods,Dissertation,StevensInstituteofTechnology,Nov.2015.

[40] Clifford, M., M. Blackburn, D. Verma, and P. Zimmerman, Model-Centric Engineering - Insights andChallenges:PrimaryTakeawaysfromaGovernment-IndustryForum,StevensInstituteofTechnology,Jul.2016.

[41] Cooke, B., MBSE on Europa Clipper, NASA/JPL Symposium and Workshop on Model-Based SystemsEngineering,January2015.

[42] Coolahan,J.AVisionformodelingandsimulationatAPL,JohnshopkinsAPLTechnicalDigest,Volume26,number4(2005).

[43] Cloutier,Robert&MaryBone.2015.MBSESurvey.INCOSEIW2015.LosAngeles,CA.[44] Crain, Robert K. 2014. “MBSE without a Process-Based Data Architecture Is Just a Random Set of

73

Characters.”In,1–10.IEEE.doi:10.1109/AERO.2014.6836221.[45] CRitical SYSTemEngineeringAcceLeration, Interoperability Specification (IOS) –V1D601.021,ARTEMIS-

2012-1-332830,2014.[46] Dahmann, J., BA. Aumber, M, Kelley, Importance of Systems Engineering in Early Acquisition, MITRE

Corporation.ApprovedforPublicRelease;DistributionUnlimitedCase#09-0345.[47] DARPA, Producible Adaptive Model-based Software (PAMS) technology to the development of safety

criticalflightcontrolsoftware.PAMShasbeendevelopedundertheDefenseAdvancedResearchProjectsAgency (DARPA) Disruptive Manufacturing Technologies program. Contract # N00178-07-C-2011,http://www.isis.vanderbilt.edu/projects/PAMS.

[48] Darwiche,A.,ModelingandReasoningwithBayesianNetworks,CambridgeUniversityPress,2009.[49] Davidoff,S.,VisualizationofModelContentandEngineeringProcess,NASA/JPLSymposiumandWorkshop

onModel-BasedSystemsEngineering,January2015.[50] Defense Acquisition University, Defense Acquisition Guidebook Chapter 4 – Systems Engineering,May

2013;https://acc.dau.mil/dag4.[51] Delp,C.,D.Lam,E.Fosse,andCin-YoungLee.“ModelBasedDocumentandReportGenerationforSystems

Engineering,”1–11.IEEE,2013.doi:10.1109/AERO.2013.6496926.[52] DepartmentofDefense,INSTRUCTION–INTERIM,NUMBER5000.02November26,2013.[53] DepartmentofDefense,MIL-HDBK-516B,DepartmentOfDefenseHandbook:AirworthinessCertification

Criteria, Feb, 2008; http://www.everyspec.com/MIL-HDBK/MIL-HDBK-0500-0599/MIL-HDBK-516B_CHANGE-1_10217.

[54] DepartmentofDefense,RiskManagementGuideForDodAcquisition,SixthEdition,August,2006.[55] Elele,J.N.,AssessingRiskLevelsofVerification,Validation,andAccreditationofModelsandSimulations,

InternationalTestandEvaluationAssociation(ITEA)Journal2008.[56] DO-178B/ED-12B - Software Considerations in Airborne Systems and Equipment Certification, Radio

Technical Corporation for Aeronautics Special Committee 167 (RTCA)December,1992.

[57] Evans,B.,ModelingandSimulationAppliedintheF-35Program,BarryEvansLockheedMartinAeronautics,2011.

[58] Firesmith,D.,AreYourRequirementsComplete?,JournalofObjectTechnology,Volume4,no.1(January2005),pp.27-43,doi:10.5381/jot.2005.4.1.c3.

[59] Flager,F.,JohnHaymaker,AComparisonofMultidisciplinaryDesign,AnalysisandOptimizationProcessesintheBuildingConstructionandAerospace,Stanford,December2009.

[60] Graf, L., Transitioning Systems Engineering Research into Programs and Practice, NDIA 17th SE AnnualConference,October2014.

[61] GAO,ProblemsCompleting Software TestingMayHinderDeliveryof ExpectedWarfightingCapabilities,GAO-14-322:Published:Mar24,2014.PubliclyReleased:Mar24,2014.

[62] Graignic,Pascal,ThomasVosgien,MarijaJankovic,VincentTuloup,JenniferBerquet,andNadègeTroussier,ComplexSystemSimulation:PropositionofaMBSEFrameworkforDesign-Analysis Integration,ProcediaComputerScience16(January2013):59–68.doi:10.1016/j.procs.2013.01.007.

[63] Gill,Helen.“FromVisiontoReality:Cyber-PhysicalSystems”,HCSSNationalWorkshoponNewResearchDirectionsforHighConfidenceTransportationCPS:Automotive,Aviation,andRail,November18-20,2008.

[64] Gonzales,M.,C.Gogu,N.Binaud,C.Espinoza, J.Morlier, andS.Quoniam. Uncertaintyquantication inaircraftloadcalibration.10thWorldCongressonStructuralandMultidisciplinaryOptimization.2013.

[65] Hammen,D.,G. Turner, JSC EngineeringOrbitalDynamics IntegrationModel,NationalAeronautics andSpaceAdministration,December2014.

[66] Hannapel, Shari, Nickolas Vlahopoulos, and David Singer. “Including Principles of Set-Based Design inMultidisciplinary Design Optimization.” American Institute of Aeronautics and Astronautics, 2012.doi:10.2514/6.2012-5444.

[67] Hayhurst, Kelly J., Dan S. Veerhusen, John J. Chilenski, and Leanna K. Rierson. A Practical Tutorial onModified Condition/Decision Coverage, NASA/TM-2001-210876.http://techreports.larc.nasa.gov/ltrs/PDF/2001/tm/NASA-2001-tm210876.pdf

[68] HensonGraves,H.,S.Guest,J.Vermette,Y.Bijan,H.Banks,G.Whitehead,B.Ison,AirVehicleModel-BasedDesignandSimulationPilot,LockheedMartin,2009;availablehttp://www.omgwiki.org/MBSE.

74

[69] Herring, M., D. Owens, N. Leveson, M. Ingham, and K. Weiss. Safety-Driven Model-Based SystemEngineeringMethodology.2007.

[70] Herron, J. Model-Centric Design CAD Design in Aerospace. Retrieved fromhttp://www.findarticles.com/p/articles/mi_hb078/is_199801/aihibm1g16938479,2006.

[71] Holland,J.,EngineeredResilientSystems(ERS)Overview,December2013.[72] Hutchinson, J., J. Whittle, M. Rouncefield, S. Kristoffersen, Empirical Assessment of MDE in Industry,

Proceedingsofthe33rdInternationalConferenceonSoftwareEngineering,2011.[73] IDEFØ,ComputerSystemsLaboratoryoftheNationalInstituteofStandardsandTechnology(NIST),1993.[74] International Council on Systems Engineering (INCOSE), “MBSE initiative,” January 2007;

https://connect.incose.org/tb/MnT/mbseworkshop/.[75] ISO/IEC42010:2007,SystemsandSoftwareEngineering--ArchitectureDescription,2007.[76] Jackson,Ethan,andJanosSztipanovits.“FormalizingtheStructuralSemanticsofDomain-SpecificModeling

Languages.”Software&SystemsModeling8,no.4(September2009):451–78.doi:10.1007/s10270-008-0105-0.

[77] Jenkins,J.S.,N.Rouquette,Semantically-RigoroussystemsengineeringmodelingusingSysMLandOWL,5thInternationalWorkshoponSystems&ConcurrentEngineeringforSpaceApplications,Lisbon,Portugal,October17-19,2012.

[78] Joshi,A.,M.P.E.Heimdahl.Model-BasedSafetyAnalysisofSimulinkModelsUsingSCADEDesignVerifier.Proc.24thDigitalAvionicsSystemsConference.2005.

[79] Khan,O.,G.Dubos, J. Tirona, S. Standley,Model-BasedVerification andValidationof the SMAPUplinkProcesses,IEEEAerospaceConference,2013.

[80] Kim, H., Fried, D., Menegay, P., Connecting SysML Models with Engineering Analyses to SupportMultidisciplinarySystemDevelopment,AmericanInstituteofAeronauticsandAstronautics,2012.

[81] Kim,H.,Fried,D.,Menegay,P.,G.Soremekun,C.Oster,ApplicationofIntegratedModelingandAnalysistoDevelopment of Complex Systems, Conference on Systems Engineering Research, 2013;http://dx.doi.org/10.1016/j.procs.2013.01.011.

[82] Knudsen,K.T.,M.R.Blackburn,AKnowledgeandAnalytics-BasedFrameworkandModel forForecastingProgramSchedulePerformance,ComplexAdaptiveSystemsConferenceNovember2-4,2016.

[83] Kortelainen, J., Semantic Data Model for Multibody System Modelling, Dissertation, LappeenrantaUniversityofTechnology,2011.

[84] Leveson,N.,ANewAccidentModelforEngineeringSaferSystems,SafetyScience,Vol.42,No.4,April2004.[85] Liersch,C.M.,K.C.HuberConceptualDesignandAerodynamicAnalysesofaGenericUCAVConfiguration,

32ndAIAAAppliedAerodynamicsConference,16-20June2014.[86] Martins, Joaquim R. R. A., Andrew B. Lambe. "Multidisciplinary Design Optimization: A Survey of

Architectures",AIAAJournal,Vol.51,No.9(2013),pp.2049-2075.[87] Matei,I.,C.Bock,SysMLExtensionforDynamicalSystemSimulationTools,NationalInstituteofStandards

and Technology, NISTIR 7888, http://dx.doi.org/10.6028/NIST.IR.7888, October 2012,http://nvlpubs.nist.gov/nistpubs/ir/2012/NIST.IR.7888.pdf.

[88] McFarland, J., Uncertainty Analysis For Computer Simulations Through Validation And Calibration,Dissertation,VanderbiltUniversity,May2008.

[89] McFarland,J.,SankaranMahadevan,VicenteRomero,LauraSwiler,CalibrationandUncertaintyAnalysisforComputerSimulationswithMultivariateOutput,AIAA,October,2007.

[90] McKelvin, Jr., Mark, and Alejandro Jimenez. “Specification and Design of Electrical Flight SystemArchitectureswithSysML.”AmericanInstituteofAeronauticsandAstronautics,2012.doi:10.2514/6.2012-2534.

[91] MIL-HDBK-516C, Department Of Defense Handbook: Airworthiness Certification Criteria, December 12,2014.

[92] ModelBasedEnterprise,http://model-based-enterprise.org/.[93] Murray, Brian T., Alessandro Pinto, Randy Skelding, Olivier L. deWeck, Haifeng Zhu, Sujit Nair, Narek

Shougarian,KaushikSinha,ShaunakBodardikar,andLarryZeidner.METAIIComplexSystemsDesignandAnalysis(CODA),2011.

[94] NAOMI Project, Lockheed Martin Advanced Technology Laboratories;http://www.atl.external.lmco.com/programs/STI/programs/program1.php#experimentalinfrastructure,

75

2013.[95] National Institute of Standards and Technology, Foundations for Innovation in Cyber-Physical Systems,

WorkshopReport,2013.[96] NAVAIRINST13034.1C,Navair Instruction: FlightClearancePolicy ForAirVehiclesAndAircraft Systems,

September,28,2004.[97] Navy Integration and Interoperability (I&I) Integrated Capability Framework (ICF), Operational Concept

Document,Version2.0,30September2013.[98] Newcomer, J. T., SANDIA REPORT, SAND2012-7912Unlimited Release Printed September 2012, ANew

Approach to Quantification of Margins and Uncertainties for Physical Simulation Data.(http://prod.sandia.gov/techlib/access-control.cgi/2012/127912.pdf).

[99] Nixon,D.W.,FlightControlLawDevelopmentfortheF-35JointStrikeFighter,October5,2004.[100] Oberkampf, William Louis, Timothy Guy Trucano, andMartin M. Pilch. “Predictive Capability Maturity

Model for Computational Modeling and Simulation.,” October 1, 2007.http://www.osti.gov/servlets/purl/976951-meC28s/.

[101] Object Management Group, MBSE Wiki, Ontology Action Team,http://www.omgwiki.org/MBSE/doku.php?id=mbse:ontology,2014.

[102] Object Management Group, XML Metadata Interchange (XMI), Version, 2.4.2, April 2014,http://www.omg.org/spec/XMI/2.4.2.

[103] Object Management Group. OMG Unified Modeling LanguageTM (OMG UML), Superstructure. 2011.Version2.4.1.Availablefrom:http://www.omg.org/spec/UML/2.4.1/Superstructure/PDF.

[104] Object Management Group. OMG Systems Modeling Language (OMG SysMLTM). 2012. Version1.3.Availablefrom:http://www.omg.org/spec/SysML/1.3/PDF.

[105] Papadopoulos,Y.,D.Parker,C.Grant.AMethodandToolSupportforModel-basedSemi-automatedFailureModes and Effects Analysis of Engineering Designs. Proc. 9th AustralianWorkshop on Safety RelatedProgrammableSystems.2004.

[106] Paredis,C.,Y.Bernard,R.Burkhart,D.Koning,S.Friedenthal,P.Fritzson,N.Rouquette,W.Schamai,AnOverviewoftheSysML-ModelicaTransformationSpecification,INCOSEInternationalSymposium,Chicago,IL,July,2010.

[107] Peak,R.,S.Cimtalay,A.Scott,M.Wilson,B.Aikens,D.Martin,Verification,Validation,andAccreditationShortfallsforModelingandSimulation,FinalTechnicalReportSERC-2011-TR-018,SystemsEngineeringResearchCenter,2011.

[108] Pearl,J.(1985)."BayesianNetworks:AModelofSelf-ActivatedMemoryforEvidentialReasoning"(UCLATechnical Report CSD-850017). Proceedings of the 7th Conference of the Cognitive Science Society,UniversityofCalifornia,Irvine,CA.pp.329–334.Retrieved2009-05-01.

[109] Petnga,L.,M.Austin,“Anontologicalframeworkforknowledgemodelinganddecisionsupportincyber-physicalsystems,”AdvancedEngineeringInformatics,vol.30,no.1,pp.77–94,Jan.2016.

[110] PhoenixIntegration,ModelCenterhttp://www.phoenix-int.com.[111] Post,D.,ComputationalResearchEngineeringAcquisitionToolsandEnvironments,ADoDProgramtoAid

AcquisitionEngineering,NDIA,October2014.[112] Ray,S.,G.Karsai,K.McNeil,Model-BasedAdaptationofFlight-CriticalSystems,DigitalAvionicsSystems

Conference,2009.[113] Rasumussen,R.,R.Shishko,JupiterEuropaOrbiterArchitectureDefinitionProcess,INCOSEConferenceon

SystemsEngineeringResearch,RedondoBeach,California,April14-16,2011.[114] Rhodes,D.H.,A.M.Ross,P.Grogan,O.deWeck,InteractiveModel-CentricSystemsEngineering(IMCSE),

PhaseOneTechnicalReport SERC-2014-TR-048-1, SystemsEngineeringResearchCenter, September30,2014.

[115] Ressler, S., What's That 3D Model Doing in my Web Browser, Model-Based Enterprise Summit 2014,http://math.nist.gov/~SRessler/x3dom/revealjs14/mbeNISTtalk.html#/

[116] Rizzo,D.B.,M.R.Blackburn,UseofBayesianNetworksforQualificationPlanning:EarlyResultsofFactorAnalysis,ComplexAdaptiveSystemsConferenceNovember2-4,2016.

[117] Rizzo, D., M. R. Blackburn, Use of Bayesian networks for qualification planning: a predictive analysisframeworkforatechnicallycomplexsystemsengineeringproblem,ComplexAdaptiveSystemsConference,November,2015.

76

[118] Rodano,M.,K.Giammarco,AFormalMethodforEvaluationofaModeledSystemArchitectureMatthewStevensInstituteofTechnology,ComplexAdaptiveSystemsConference,2013.

[119] Romero,V.,ElementsofaPragmaticApproachfordealingwithBiasandUncertaintyinExperimentsthroughPredictions:ExperimentDesignandDataConditioning,“RealSpace”ModelValidationandConditioning,HierarchicalModelingandExtrapolativePrediction,SAND2011-7342UnlimitedReleasePrintedNovember2011.

[120] Romero, V., Uncertainty Quantification and Sensitivity Analysis—Some Fundamental Concepts,Terminology,Definitions, andRelationships,UQ/SA sectionof invitedpaper forAIAASciTech2015Non-DeterministicApproachesConference,Jan5-9,2015,Orlando,FL.

[121] Rothenberg, J. L.E.Widman,K.A.Loparo,N.R.Nielsen,TheNatureofModeling,Artificial Intelligence,SimulationandModeling,1989.

[122] SAEARP4761.GuidelinesandMethods forConducting theSafetyAssessmentProcessonCivilAirborneSystemsandEquipment.SAEInternational,December1996.

[123] SandiaNationalLaboratory,Dakota,https://dakota.sandia.gov/.[124] Schindel, W. D., Failure Analysis: Insights from Model-Based Systems Engineering. Proc. INCOSE Int’l

Symposium.2010.[125] Schroeder, C. A., A Study of HowModel-Centric Engineering Relates to Time-To-Market and Agility to

AccommodateCustomer-RequiredChanges,Dissertation,IndianaStateUniversity,2011.[126] Shani,U.,EngagingOntologiesinMBSE,ConferenceonSystemEngineeringResearch,March2016.[127] Siegemund, K., E. J Thomas, Y. Zhao, J. Pan, and U. Assmann. Towards ontology-driven requirements

engineering.InWorkshopSemanticWebEnabledSoftwareEngineeringat10thInternationalSemanticWebConference(ISWC),Bonn,2011.

[128] Simko,Gabor,TihamerLevendovszky,SandeepNeema,EthanJackson,TedBapty,JosephPorter,andJanosSztipanovits.“FoundationforModelIntegration:SemanticBackplane,”2012.

[129] Singer,DavidJ.,NorbertDoerry,andMichaelE.Buckley.“WhatIsSet-BasedDesign?:WhatIsSet-BasedDesign?”NavalEngineersJournal121,no.4(October2009):31–43.doi:10.1111/j.1559-3584.2009.00226.x.

[130] Snooke,N.,Model-BasedFailureModesandEffectsAnalysisofSoftware.ProceedingsDX04.2004.[131] Spangelo,S.D.Kaslow,C.Delp,L.Anderson,B.Cole,E.Foyse,L.Cheng,R.Yntema,M.Bajaj,G.Soremekum,

J.Cutler,MBSEChallengeTeam,ModelBasedSystemsEngineering(MBSE)AppliedtoRadioAuroraExplorer(RAX)CubeSatMissionOperationalScenarios,IEEEACPaper#2170,Version1,Updated29/01/2013.

[132] System Engineering Research Center, INCOSE, Stevens, Report Of The Workshop On The RelationshipBetweenSystemsEngineeringAndSoftwareEngineering,WorkshopsponsoredbyStevens,INCOSE,SERC,June2014.

[133] Thompson,T.,EnablingArchitectureInteroperabilityInitiative,B210-001D-0051Unclassified.[134] Topper,S.,ModelBasedSystemsEngineering(MBSE),NDIA,19-April-2016.[135] Umpfenbach,E.,IntegratedSystemEngineeringFramework(ISEF),NDIASystemsEngineeringConference,

October2014.[136] http://www.vitechcorp.com/products/core.shtml[137] Wagner,D.A.,M.Bennett,R.Karban,N.Rouquette,S.Jenkins,M.Ingham,AnOntologyforStateAnalysis:

FormalizingtheMappingtoSysML,IEEEAerospaceConference,2012.[138] Wade, J., R. Cohen, M. Blackburn, E. Hole, N. Bowen, Systems Engineering of Cyber-Physical Systems

EducationProgram,WorldInnovationSummitforEducation,Nov.2015.[139] West,T.,A.Pyster,UntanglingtheDigitalThread:TheChallengeandPromiseofModel-BasedEngineering

inDefenseAcquisition,INCOSEINSIGHT,Volume18,Issue2,pages45–55,August2015.[140] Wikipedia,Ontology,http://en.wikipedia.org/wiki/Ontology_(information_science),2014.[141] Witus, G., W. Bryzik, Trust under Uncertainty - Quantitative Risk, SERC RT-107, Systems Engineering

ResearchReview,December,2014.[142] Witherell, Paul, Boonserm Kulvatunyou, and Sudarsan Rachuri. “Towards the Synthesis of Product

KnowledgeAcrosstheLifecycle,”V012T13A071.ASME,2013.doi:10.1115/IMECE2013-65220.[143] WorldWideWebConsortium.OWL2WebOntologyLanguageDocumentOverview.2009.Availablefrom:

http://www.w3.org/TR/2009/REC-owl2-overview-20091027/.[144] Xie,H.Li,X.,C.Liu.,TheModel-BasedandBidirectionalSoftwareFailureModeandEffectAnalysisMethod.

IEEEIntlConfonReliability,MaintainabilityandSafety(ICRMS).2014.

77

[145] Zentner,J.,Ender,T.,Ballestrini-Robinso,S.,OnModelingandSimulationMethodsforCapturingEmergentBehaviorsforSystems-of-Systems,12thAnnualSystemsEngineeringConference,October,2009.

[146] Zimmerman, P.,Model-Based Systems Engineering (MBSE) inGovernment: Leveraging the ‘M’ for DoDAcquisition,2014INCOSEMBSEWorkshopJanuary25,2014.

[147] zurMuehlen,M.,D.Hamilton,R.Peak,IntegrationofM&S(ModelingandSimulation),SoftwareDesignandDoDAF,SERC-2012-TR-024,2012.