rt-173: helix 2017 helix technical report · participation in the helix interviews provided us data...

Report No. SERC-2018-TR-101 January 16, 2018

RT-173:Helix

2017HelixTechnicalReport

TechnicalReportSERC-2018-TR-101January16,2018

PrincipalInvestigator:Dr.NicoleA.C.Hutchison,StevensInstituteofTechnology

Co-PrincipalInvestigator:Dr.DineshVerma,StevensInstituteofTechnology

ResearchTeam:

StevensInstituteofTechnology:Dr.PamelaBurke,Ms.MeganClifford,Mr.Ralph

Giffin,Mr.SergioLuna,Mr.MatthewPartacz

Sponsor:OfficeoftheDeputyAssistantSecretaryofDefenseforSystemsEngineering

DASD(SE)


ii

Copyright©2018StevensInstituteofTechnology,SystemsEngineeringResearchCenterThe Systems Engineering Research Center (SERC) is a federally funded University Affiliated ResearchCentermanagedbyStevensInstituteofTechnology.Thismaterial is based uponwork supported, inwhole or in part, by theU.S.Department ofDefensethroughtheOfficeoftheAssistantSecretaryofDefenseforResearchandEngineering(ASD(R&E))underContractH98230-08-D-0171andHQ0034-13-D-0004.Anyviews,opinions,findingsandconclusionsorrecommendationsexpressedinthismaterialarethoseoftheauthor(s)anddonotnecessarilyreflecttheviewsoftheUnitedStatesDepartmentofDefensenorASD(R&E).NoWarranty.ThisStevensInstituteofTechnologyandSystemsEngineeringResearchCenterMaterialisfurnishedonan“as-is”basis. StevensInstituteofTechnologymakesnowarrantiesofanykind,eitherexpressedorimplied, as to any matter including, but not limited to, warranty of fitness for purpose ormerchantability, exclusivity, or results obtained from use of the material. Stevens Institute ofTechnologydoesnotmakeanywarrantyofanykindwithrespecttofreedomfrompatent,trademark,orcopyrightinfringement.Thismaterialhasbeenapprovedforpublicreleaseandunlimiteddistribution.


iii

TABLEOFCONTENTS

TableofContents..................................................................................................................iii

ListofFigures.........................................................................................................................v

ListofTables...........................................................................................................................v

Acknowledgements................................................................................................................1

ExecutiveSummary................................................................................................................2

1BackgroundandIntroduction...............................................................................................31.1TheHelixProject........................................................................................................................31.2HowisAtlasDifferentfromHelix?.............................................................................................41.3IncrementalDevelopmentofAtlas.............................................................................................51.4AboutThisDocument.................................................................................................................6

2Methodology.......................................................................................................................72.1ResearchPhilosophy..................................................................................................................72.2OverarchingResearchMethodology...........................................................................................82.3HelixResearchProcess...............................................................................................................9

2.3.1PreparationforDataCollection(A)..........................................................................................102.3.2DataCollection(B)...................................................................................................................102.3.3DataAnalysis(C)......................................................................................................................112.3.4MethodologyReview(D).........................................................................................................112.3.5TheoryDevelopment(E)..........................................................................................................122.3.6Publishing(F)............................................................................................................................122.3.7Validation,Feedback&Deployment(G)..................................................................................12

2.4QualitativeAnalysis.................................................................................................................142.4.1Coding......................................................................................................................................142.4.2ToolSupportandCombiningQualitativeAnalysiswithDemographics...................................16

2.5CareerPathMethodology........................................................................................................172.5.1CharacterizingaSystemsEngineer’sExperiences....................................................................172.5.2CharacterizingaSystemsEngineer’sEducation.......................................................................192.5.3IdentifyingKeyPositions..........................................................................................................202.5.4AssessingProficiency...............................................................................................................212.5.5MappingaCareerTimeline......................................................................................................222.5.6StepstoValidateFindingsfromCareerPaths..........................................................................24

2.6CapstoneonProficiencyandCareerPaths................................................................................242.7HelixData................................................................................................................................26

2.7.1DataSources............................................................................................................................262.7.2Demographics..........................................................................................................................28

2.8InterpretationandGeneralizationUsingtheDataset...............................................................31

3HelixActivitiesin2017.......................................................................................................333.1WorkFocusedonEffectiveSystemsEngineers.........................................................................33

3.1.1AdditionalAnalysisofProcessData.........................................................................................353.1.2AdditionalAnalysisofOrganizationalData..............................................................................363.1.3AdditionalAnalysisofMethodsandToolsData......................................................................38


iv

3.1.4AdditionalDataCollection.......................................................................................................403.2UpdatestotheResearchQuestions..........................................................................................413.3RelatedWork...........................................................................................................................43

4FutureDirections:Helixin2018andBeyond......................................................................454.1EnvisionedEndStateforHelix..................................................................................................46

5Conclusion.........................................................................................................................48

WorksCited..........................................................................................................................49

AppendixA:ListofHelixPublications..................................................................................51

AppendixB:UpdatedInterviewQuestionsforHelix.............................................................53

AppendixC:CulturalCharacteristics–DefinitionsandRelatedQuestions.............................55


v

LISTOFFIGURES

Figure1.RelationshipbetweenHelixandAtlas.............................................................................5

Figure2.HelixResearchProcess....................................................................................................9

Figure3.ExampleofCodingRelationships...................................................................................16

Figure4.GenericCareerPathMapping.......................................................................................23

Figure5.DistributionofsenioritylevelsacrossHelixdataset......................................................28

Figure6.DistributionofcareerpathclassificationacrossHelixdataset......................................29

Figure7.DistributionofgendersacrossHelixdataset.................................................................29

Figure8.DistributionofIndividualsbyOrganizationDomain......................................................30

Figure9.IndividualsbyOrganizationType...................................................................................30

Figure10.Relationshipbetween“Helix”and“Atlas”..................................................................34

Figure11.Comparisonofseniorityleveldistribution..................................................................40

Figure12.Comparisonoforganizationtypedistribution.............................................................41

Figure13.ExpectedVariablestoImpactSystemsEngineeringCapability...................................42

LISTOFTABLES

Table1:GeneralCareerPathClassificationDefinition................................................................25

Table2:AtlasProficiencyAreaDefinitions..................................................................................25

Table3.GeographicalDistributionofINCOSESEPApplicants.....................................................31

Table4.CitedToolsandMethodsfromtheHelixDataset...........................................................38


1

ACKNOWLEDGEMENTS

TheHelixteamwouldliketothankalltheorganizationsandindividualsthatparticipatedintheproject,offeringtheirresources,time,andeffort.Thiswascriticaltoourresearch.TheiractiveparticipationintheHelixinterviewsprovidedusdatathatwasrichinbothqualityandquantity,whichmakesthisresearchmorevaluableandusefultotheparticipatingorganizationsandthesystems engineering community at large. To the organizations that have publicly sharedinformation about their experiences with using Atlas, we owe tremendous thanks; withoutthem,thisdocumentwouldnotbepossible.

Wearemostgrateful to theOfficeof theDeputyAssistantSecretaryofDefense forSystemsEngineering (DASD(SE)), especially Kristen Baldwin and Scott Lucero, for their continuedsupport, without which this research would not be possible. The International Council onSystems Engineering (INCOSE) and the National Defense Industrial Association SystemsEngineeringDivision(NDIA-SED)arebothvaluedpartnersinthisresearchandwethankthem,especiallyCourtneyWright,DavidLong,BillMiller,andDonGelosh.

WethankallformermembersoftheHelixresearchteamwhosecontributionshaveshapedourresearchovertheyears.Inparticular,wewouldliketothankDr.ArtPyster,whoseleadershipanddedicationsincethebeginningoftheprojectwerecriticalandinstrumentaltoitssuccessandwhostillchampionsandsupportsHelix.

IwouldalsoliketopersonallythankthecurrentHelixteam,whohavehelpedtremendouslytomaturenotonlytheproject,butmyownthinkingandviewsonit.Pam,Ralph,Sergio,Megan,Matthew,andDinesh,pleaseacceptmygratitudeforyourhardworkanddedication.

NicoleACHutchison HelixPrincipalInvestigator


2

EXECUTIVESUMMARY

ThisdocumentsummarizestheworkdonebytheHelixteamin2017andhighlightstheworkplannedin2018.Inparticular,itprovidesdetailson:

• AdditionalanalysesconductedbytheHelixteam

• AdditionaldatacollectedbytheHelixteam

• Thechangeinresearchquestions,whicharenow:

o Howcanorganizationsimprovetheeffectivenessoftheirsystemsengineeringworkforce?

o Howdoestheeffectivenessof thesystemsengineeringworkforce impact theoverallsystemsengineeringcapabilityofanorganization?

o Whatcritical factors, inadditionaltoworkforceeffectiveness,arerequiredtoenablesystemsengineeringcapability?

• ThethreecompaniondocumentstothisreportdevelopedbytheHelixteam:

o Atlas 1.1: The Theory of Effective Systems Engineers – (SERC-2018-TR-101-A)This is an incremental evolution of Atlas that reflects feedback from thecommunity,additionalanalysis,andmaturationof theteam’sthinking in2017.In particular, Atlas includes minor updates on the values systems engineersprovide, the roles systems engineers play, the proficiency model for systemsengineers, and the personal characteristics of systems engineers. Henceforth,thiswillbereferredtoas“Atlas1.1”.

o AtlasCareer PathGuidebook – (SERC-2018-TR-101-B) This document providesanalysesoftheHelixdataset,providingcommonpatternsinsystemsengineers’careers. The Guidebook also provides some insights on questions commonlyaskedoftheHelixteamaroundcareerpathsandtheteam’sresponses.Finally,additionalworkonlinkingproficienciestocareerpathshasbeencompletedandisreflectedintheguide.Henceforth,thiswillbereferredtoasthe“Guidebook”.

o Atlas 1.1. Implementation Guide:Moving from Theory Into Practice – (SERC-2018-TR-101-C)WheneverAtlas is presented, there aremany questions abouthow to take the theory and apply it in practice. TheGuide provides examplesfromorganizationsthathaveimplementedpartsofAtlas,andguidancecreatedby the Helix team based on many interactions with organizations aroundimplementationaswell as theextensiveHelixdataset.Henceforth, thiswill bereferredtoasthe“ImplementationGuide”.

ThefuturevisionforAtlasincludesdevelopingatheoryofsystemsengineeringcapabilitywhichis predicated on the hypothesis that an appropriately skilled systems engineeringworkforcesupported by culture, governance, and infrastructure will deliver an effective systemsengineeringcapability.


3

1BACKGROUNDANDINTRODUCTION

The Systems Engineering Research Center (SERC), a University Affiliated Research Center(UARC),setupbytheU.S.DepartmentofDefense(DoD),respondedtothesystemsengineeringworkforce challenges by initiating the Helix Project to investigate the “DNA” of systemsengineers,beginningwiththosewhowork indefenseandthenmorebroadly.TheUSDeputyAssistantSecretaryofDefenseforSystemsEngineering(DASD(SE)),theInternationalCouncilonSystems Engineering (INCOSE) and the Systems EngineeringDivision of theNational DefenseIndustrial Association (NDIA-SED) jointly sponsor Helix. To ensure Helix delivers the greatestvalueand tohelpHelixobtain access to thenecessarydata,Helix formed theHelixAdvisoryPanel (HAP)with representativesprimarily from those three sponsororganizations.Helixhasheld three annual workshops with a broad set of representatives from across government,academia,andindustry.

Helix is a multi-year longitudinal research project, which has gathered data from manyorganizations with DoD and the Defense Industrial Base (DIB) through a combination oftechniques, including interviewswithhundredsofsystemsengineers. In2014,Helixbegantoreach beyond DoD and the DIB, to gather data from other types of organizations as well,includingnon-defenseorganizationsintheUSandnon-USorganizations.Version0.25ofAtlaswasalsopublishedin2014.Atlas identifiesthekeyvariablesthatimpactasystemsengineer’seffectiveness – positively or negatively – and provides, asmuch as possible, details on howthesevariablesimpacteffectiveness.

During 2015, Helix expanded its data collection by conducting interviews with non-DoDorganizationsaswell;maturedAtlasintothenextversions,Atlas0.6;definedandanalyzedthecareerpathsofsystemsengineers;anddidimplementationtrialsofAtlas.

During 2016, the team generated Atlas 0.6 and Atlas 1.0. Atlas 1.0 reflects the results ofanalysis of in-depth interviewswith 287 individuals.Most of these individualswere systemsengineers, though approximately 10%of the samplewas comprised of individualswhoworkwith systems engineers – organizational leaders, classic engineers (electrical, mechanical,software,etc.),andprogrammanagers.In2016theHelixteamalsoworkedonimplementationofAtlaswithanumberoforganizationsandlessonslearnedfromthoseactivitiesarecapturedhere.

1.1THEHELIXPROJECT

TheUSDepartmentofDefense(DoD)andtheDefenseIndustrialBase(DIB)–contractorsthatdevelop and deliver systems to the DoD – have been facing major systems engineeringchallenges in recent years (e.g. GAO 2008, 2011, 2012, 2013). Mission requirements areevolving and they demand evermore sophisticated and complex systems (e.g. Boehm et al.2010; INCOSETechnicalOperations2007;Davidz2006;DavidzandNightingale2007;Franketal. 2007; INCOSE 2014); the tools, processes, and technologies that systems engineersmustmaster keep changingmore rapidly (e.g. Frank 2006); and budgets and schedules are being


4

compresseddramatically.Anadditionalconcernisthatthousandsofsystemsengineersinthedefenseworkforcearenearingretirement;theywilltakewiththemhundredsofthousandsofstaff-yearsofexperience(DoD2013).

Organizations have responded to these challenges in a variety of ways, such as offeringextendedtrainingandeducationtotheircurrentworkforceorsystematicallyseekingtoselectspecialty engineerswith the potential to become systems engineers and incorporating theminto the ranks of systems engineers. Unknown is whether these actions are producing thedesired resultsbecause there isno commonunderstandingof thediverse roles that systemsengineersplay,howtheyareselectedandevaluated,whatcompetenciesaremost importantfor different roles, how to evaluate effectiveness, or how experiences impact effectiveness.These and many other insights will be critical to maintaining and growing the systemsengineeringworkforceintheUSDoDandDIB.

1.2HOWISATLASDIFFERENTFROMHELIX?

Helix is the name of the overarching SERC project. Helix has been examining what makessystemsengineerseffectiveforoverfouryears.Asaproject,Helixhascreatedmanydifferentdeliverablesorproducts.TheprimaryproductofHelixisAtlas:TheTheoryofEffectiveSystemsEngineers.ThisdocumentrepresentsAtlas1.0–expectedtobematureenoughforindividualsororganizationstousewithoutdirecthelpfromtheHelixteam.ItisastandalonedocumenttodetailthecontentsofAtlas.

Thisdocumentdoesnotcontainalloftheresearchthat ledtothedevelopmentofAtlas1.0.Instead, the detailed research results and how they led to Atlas 1.0 are contained in thecompanionHelixTechnicalReport(SERC-2016-TR-118). Individualsororganizationsthatwantnot just to use Atlas but to also understand the rationale and methodology behind itsdevelopmentshouldreferencetheTechnicalReport.SeveralearlierpublishedHelixpapersandtechnical reports are also referred to throughout this report. The reader is not expected toread the earlier technical reports or any of the other Helix papers or reports, in order tounderstandAtlas1.0.

In addition, there are tools that an individual or organization can use to support self-assessmentusingAtlas.Thepaper-basedtoolsarecontainedintheAppendicesofthisreport.TheteamhasalsodevelopedmoreeasilytailoredExcel-basedtools,whichcanbefoundontheHelixpageoftheSERCwebsite(http://www.sercuarc.org/projects/helix/).

The relationship between Helix, Atlas, the Technical Reports, and the tools is illustrated inFigure 1.


5

Figure1.RelationshipbetweenHelixandAtlas

1.3INCREMENTALDEVELOPMENTOFATLAS

TheHelixprojectusedanincrementalapproachtodevelopAtlas.ThisapproachwasdesignedtoenablepublicationanduseofaspectsofAtlasastheybecameappropriatelymature,whilemaintainingtheexpectationthatAtlaswouldbecomemorematureovertime.Theincrementswere:

• Atlas0.25:ThefirstdraftofAtlasbasedonworkdonein2014waspublishedasAtlas0.25 in November 2014. It included key elements that explain the effectiveness ofsystems engineers, and a preliminary explanation of the relationships between thoseelements. The structure and variables of the proficiency model were also included,alongwithsomeinitialanalysisofcareerpaths.

• Atlas 0.5: Based on subsequent work done in 2015, Atlas 0.5 was published inDecember2015. It reflected furtherunderstandingof theelementsofAtlas and theirinter-relationships.Significantnewworkwasdone in theareaofcareerpathsand0.5incorporated initial efforts to use Atlas to assess the level of proficiency of systems

Project

Products

Atlas1.1

SupportingTechnicalReports

SupportingTools

DetailsofDevelopment

EnableAssessment

DescriptionofTheory

ImplementationGuide

TailorandUse

CareerPathGuidebook

Basedon


6

engineers.Atlas0.5wasmatureenoughforanindividualoranorganizationtouseandgainvaluableinsightswithsomeguidancefromtheHelixteam.

• Atlas0.6:Wasanincremental improvementtoAtlas0.5. Itcontainedadditionaldetailand analysis for areas that were less mature in 0.5, namely: mentoring, personalinitiatives, and organizational initiatives. Atlas 0.6 was not created as a stand-alonedocument,butratherasasupplementto0.5.

• Atlas 1.0: Atlas 1.0 included amore complete description of the elements of Atlas and theirinter-relationships.Atlas1.0isbelievedtobematureenoughforindependentdeploymentandassessmentby individualsandorganizationswith littleornoguidance fromtheHelix team. Inaddition, the frameworks presented inAtlas 1.0 havebeen validatedusingdata fromoutsidethe US DoD, and therefore is believed to be applicable to systems engineers in a variety ofdomains.Thisisintentional.ThoughtheinitialimpetusfortheworkwasbasedontheneedsoftheUSDoD,theHelixteambelievesthatamoregenericframeworkwhichbenefitsallsystemsengineers, regardless of domain, is both more beneficial to the community at large and,ultimately will benefit the US DoD by setting consistent expectations for practitioners acrossdomains.

• Atlas1.1:ThisisanincrementalupdatetoAtlasthatreflectstheteams’learningin2017.

1.4ABOUTTHISDOCUMENT

Thistechnicalreportiswrittenasastandalonedocument,presentingversion1.0ofAtlas:TheTheory of Effective Systems Engineers. Several earlier published Helix papers and technicalreportsarereferredtothroughoutthisreport.However,thereaderisnotrequiredtoreadtheearlier technical reports or any of the other Helix papers or reports, in order to understandAtlas1.0.

Readersshouldnotethefollowingaboutthereport:

• Throughoutthereport,theterm‘Helix’isusedtodenoteeithertheprojectortheteamthatperformedtheworkindevelopingAtlas.

• TheGuide to the Systems Engineering Body of Knowledge (SEBoK) is used across thereport as the primary source of consistent terminology and definitions relevant tosystemsengineering.(BKCASEEditorialBoard2016)

• Allinsightsandobservationsarepresentedonlyinananonymous,aggregatedmanner.IndividualsororganizationsthatparticipatedintheHelixprojectareneithernamednoraretheyidentifiablefromthisreport.

Thereportisorganizedasfollows:


7

2METHODOLOGY

This sectionprovidesdetailonhow theHelixprojecthasbeen structured since its inception.This includes overarchingmethodology, details of analytic approaches, and detail about thedataset.TheorganizationofSection2isasfollows:

• Theoverarchingphilosophicalapproachestotheresearch(Sections2.1-2.3),

• Thedetailedanalysisapproachesfortheresearch(Sections2.4and2.5),

• Asummaryofthemethodologyusedforamaster’scapstoneprojectonHelix(2.6),

• Anoverviewofthedataset(Section2.7),and

• Guidanceonhowtointerpretthedata,includinglimitationsofthedataset(Section2.8).

2.1RESEARCHPHILOSOPHY

Helixisprimarilyaqualitativestudy,withtheprimarymeansofdatacollectionbeinginterviewswith systems engineers. From 2012-2013, the Helix team focused on a mixed-methodsapproach (CreswellandPlano2011), combining thedevelopmentofbasic researchquestionswithagroundedtheoryapproach.Groundedtheorywasdevelopedinthesocialsciencesasamethod for developing theory that is grounded in data that is systematically gathered andanalyzed.(Goulding2002)Thisapproachallowsthedataitselftodrivepointsoffurtherinquiry,guide categorization,etc.; rather than startinganalysiswithanexisting framework, all of thedataisreviewedholisticallyandanypotentialareasof interestarecoded.Overtimepatternsemerge and these guide further data collection and analysis. The development of drivingresearch questions makes the Helix project mixed method as opposed to pure groundedtheory.

Whenperforming initial data coding, theHelix teamcodedall data,notmaking suppositionsaboutwhichdatawouldprove“important”.Theteamalsocompareddatacollectedagainsttheexistingliteraturewherepossible.Forexample,assystemsengineersdefinedtheactivitiesthatthey perform, the team collected and organized the raw data but also compared it to the“TwelveRolesofSystemsEngineers”definedin(Sheard1996).

ThisapproachstillreflectsthephilosophyofHelix:Atlas1.0is largelyareflectionofthedata,usingthegroundedtheoryprinciplesto“letthedataspeak”.


8

2.2OVERARCHINGRESEARCHMETHODOLOGY

The research methodology adopted for Helix research may be considered to be a modifiedgroundedtheorybasedapproach,employingqualitativeandquantitativeresearchmethods.

During 2013 and 2014, Helix primarily focused on data collection from DoD and DIBorganizations through semi-structured in-person interviews with individuals or small groups,continuallyrefiningtheinterviewquestionsandprocess.Follow-upinterviewswereconductedbytelephonewithmostoftheparticipants.AnalysisofthedatatoaddresstheHelixresearchquestions offered insights into the effectiveness of systems engineers and led to thedevelopmentofanearlyversionofAtlasthatwaspublishedinNovember2014.During2015,data collection was expanded to organizations outside of DoD and DIB, andAtlas 0.25 wasvalidated and improved upon, leading to the next version,Atlas 0.5, published in December2015.

The Helix project adopted a grounded theory approach because it did not presuppose anyspecific theory or propose any hypotheses at the start of the project. Grounded theorywasdeveloped in the social sciences as amethod fordeveloping theory that is grounded indatathat is systematically gathered and analyzed (Goulding 2002). Rather than beginning with ahypothesis,thefirststepwasdatacollection.Thisapproachisunusualinengineeringresearch,wherearesearchertraditionallybeginswithatheoreticalframeworkthatheorsheappliestothephenomenon tobestudied. In theHelixproject, thedatacollected fromthemanysemi-structuredinterviewsweremarkedupwithcodesthatweregroupedintoconcepts,thatledtothe identification of constructs and categories that formed the building blocks ofAtlas. Thisapproachminimizedanybiasthatmightbeintroducedbytheresearchers,insteadallowingthelarge data set collected through the Helix project to drive theory development. Havingestablished a preliminary theory of effective system engineers and proficiency model ofsystemsengineers,datacollectionandinterviewsconductedduring2015focusedonvalidatingAtlas,andrefiningthetheorytowardsdevelopingAtlas1.0in2016.

Qualitativeresearchaimstocreateordiscoverwhatthingsaremadeof,andwhatiscreatedordiscovered are called constructs. Qualitative research is useful for obtaining insight intosituationsandproblemsonwhichonehaslittleknowledgeapriori.Thismethodiscommonlyused for providing in-depth descriptions of procedures, beliefs and knowledge, including theopinionsofrespondentsaboutparticularissues;detaileddataisgatheredthroughopen-endedquestions. Data collection for the Helix project and subsequent analysis of the data wasprimarilydoneemployingqualitativeresearchmethods;appropriatesoftwaretoolswereusedtosupportcodingandidentificationofconstructs.

Quantitativeresearchbeginsonceinitialconstructsareinhand.Itattemptstogatherdatabyobjectivemethodstoprovideinformationaboutrelations,comparisons,andpredictions.Inthecontext of the Helix project, quantitative researchwas performed once initial constructs fordemographics of systems engineers, their organizations, and their career paths wereestablished. Data was collected from their resumes, as well as through pointed questions


9

duringinterviews.QuantitativeanalysiscontinuestobeperformedonvariouselementsofAtlasthatweredevelopedbasedonqualitativeresearch,particularlyontheproficiencymodel.

2.3HELIXRESEARCHPROCESS

The Helix researchmethodology discussed in the preceding section was deployed using theresearch process illustrated in Figure 2 below. The Helix research process consists of sevenmajorsteps:

A. PreparationforDataCollectionB. DataCollectionC. DataAnalysisD. MethodologyReviewE. TheoryDevelopmentF. PublishingG. Validation,Feedback&Deployment

Figure2.HelixResearchProcess

A.PREPARATIONFORDATACOLLECTION

A1.Iden%fyOrganiza%onsand

Individuals

A2.ProvideProjectMaterialstoPar%cipants

F.PUBLISHING

F1.PublishHelixPublicReportsandPapers

F2.PublishOrganiza%onalProfiles

(Private)

B.DATACOLLECTION

B1.CollectConsentForms&Resumes

B3.ConductIni%alInterviews

B4.ConductFollow-upInterviews

B2.CollectIns%tu%onalData

D.METHODOLOGYREVIEW

D1.Iden%fyUpdatesforInterviews

D2.Iden%fyUpdatestoIns%tu%onalDataRequests

D3.Iden%fyUpdatesforDataCollec%onApproach

C.DATAANALYSIS

C1.PrepareInterviewSummaries

C2.PerformPreliminaryAnalysis

C4.PerformDetailedQualita%veand

Quan%ta%veAnalysis

C3.PerformContextualAnalysis(Individual/

Organiza%onal)

E.THEORYDEVELOPMENT

E1.AnswerHelixResearchQues%ons

E2.DevelopTheoryofSystemsEngineers

(Atlas)

G.VALIDATION,FEEDBACK&DEPLOYMENT

G1.ValidateTheorywithInterviewPar%cipants

G2.DeployAtlaswithIndividualsandOrganiza%ons

G3.ConductHelixWorkshops


10

ThefocusofHelix in2013wasonexecutingthe loopA-B-C-D-Amultipletimeswithdifferentorganizations. The loop B-C-B was executed a few times when follow-up interviews wereconductedwithsomeorganizations.During2014,inadditiontoperformingtheA-B-C-D-Aloopwithneworganizations,stepsE-F-GwereexecutedthatledtoinitialthedevelopmentofAtlas0.25.In2015,mucheffortwasconcentratedinexecutingstepG,aswellasexecutingtheloopA-B-C-D-Awithcommercialnon-DoDorganizationsaswellaswithmanyparticipantswhodidnotconsiderthemselvestobesystemsengineers.In2016,theprimaryeffortswerefocusedonexecuting step G, which included supporting organizations as they determined how toimplementAtlasandconductingextensiveoutreachwiththesystemsengineeringcommunity.ThishasledtofurtherrefinementofAtlasinstepE,leadingtostepF-thepublishingofAtlas1.0.

In2017,theHelixteamexecutedloopA-B-C-D-Awiththreeneworganizations.TheloopB-C-Bwasexecutedbyexaminingtheexistingdatasetandcomparingittonewdatacollected.StepsE-F-Gwereexecuted leading to theupdated toAtlas 1.1and thedevelopmentof this reportandthetwocompaniondocuments.

2.3.1PREPARATIONFORDATACOLLECTION(A)

SinceHelix research isbasedonagroundedtheoryapproach,preparation fordatacollectionwasthefirststepexecuted intheproject. Initially,organizationsfromwithintheUSDoDandotherorganizationsfromtheDIBwereidentifiedfordatacollection;also,theprimaryfocuswason systems engineers in these organizations (A1). As Helix progressed, in 2015 othercommercialorganizationsfromnon-DoDsectorssuchashealthcareandinformationtechnologywere identified for data collection. The latest reports and papers published from the Helixprojectwereprovidedtopotentialinterviewees(A2).Basedontheirwillingnesstoparticipatein Helix interviews, the organization makes final decisions on who participates in Helixinterviews.In2016,theprimaryfocushasbeenonadditionalpublicationandonassistingwiththe implementation of Atlas at several organizations. These efforts have helped the teamunderstandwherethetheoryneededadjustmentstomakeiteasiertoutilize.

In2017,therewasashiftinresearchquestionsfrompreviousyears(seesection3.1.1).Withadifferent focus to the research, the team invested time in developing new questions andguidelinesfordatacollection(seeAppendixBandAppendixC).TheupdatedresearchquestionswerecriticaltoenabletheHelixteamtocollectnewdatathatwillbridgethepreviousresultswithnewareasofinquiry.

2.3.2DATACOLLECTION(B)

The first round of data collectionwith an organization is typically through a site visit to theorganization,where in-person interviews are conducted. Typically, therewill be 2 or 3 Helix


11

interviewers and anywhere from1 to 6 interviewees in a single 90-minute interview session(B3). Following approved research protocols, a signed consent form is collected from theparticipants before conducting interviews; resumes are also requested from all participants(B1).Anyavailableorganizationaldatathatwillprovide insights intothesystemsengineers intheorganizationandhowtheyarestructuredwithintheorganizationaregatheredbeforeandduring the site visit (B2). In 2015, as the project expanded to include non-DoD participants,initial interviews were conducted over telephone when the number of participants from anorganization was very low. All follow-up interviews were conducted over telephone (B4). In2016, therehavebeennoadditional interviews. Instead, theteamhas focusedonanalysisofthe data collected and on assisting organizations with the implementation of Atlas. Datagathered in 2016 was focused, therefore, on an issues identified with implementation anddetermination of whether those issues reflected a weakness in Atlas to be addressed orwhethertheywereareflectionoftheuniqueenvironmentofagivenorganization.

Data collection in 2017 was focused on gathering new data to help fill gaps in the existingdataset, focusing on organizational culture, governance, and infrastructure, and how theseimpactsystemsengineers.

2.3.3DATAANALYSIS(C)

The first step in data analysis is to prepare summaries of all interview sessions (C1).Whereintervieweespermitaudiorecording,transcriptsarefirstcreatedthencleanedandpreparedforfurther analysis. If recording is not permitted, summaries are created from the notes takenduring the interviews.Preliminaryanalysis, typicallynotemploying significanteffortonusinganalysis tools, is performed toquickly identify additional questions tobeaskedor additionaldatatobecollectedduringfollow-upinterviews(C2).Since2014,significantresearchefforthasbeenput intoperformingcontextualanalysisonan individual,particularlyonhercareerpath(C3). Detailed qualitative and quantitative analyses, using software tools as necessary, havebeen performed on the large amounts of data that have been collected through Helixinterviews (C4). These analysesmake significant contributions to theory development efforts(F).

2.3.4METHODOLOGYREVIEW(D)

Data collectionandanalysis is beingperformed iteratively, asHelix continues to identify andvisitorganizations.Afteranysitevisitandbeforethenextone,areviewisconductedtoidentifyany updates to the interview questions or process (D1). While much organizational data isdesiredforHelixanalysis,notallinformationisbeingmadeavailablewithinanorganizationinaform that may be readily shared with Helix researchers. Based on experiences withorganizations, the nature and content of organizational data requested has been regularlyupdated(D2).Basedonsignificantdataanalysisandtheorydevelopmentthatwasperformedin


12

2014,thedatacollectionapproachwasrevisedfrombeingasemi-structuredinterviewtobeingadiscussiononassessingtheproficienciesofindividualsandanalyzingtheircareerpaths(D3).FeedbackreceivedfromindividualsandorganizationsontheHelixreports(G)also influencedtheupdatesperformedinstepD.In2015,workonbroadimplementationhasidentifiedareashas been the focus. However, if additional in-depth interviews are conducted in future, theapproach of reviewing findings and conducting interviews around existing frameworks isexpectedtoremainconsistent.

2.3.5THEORYDEVELOPMENT(E)

Analysis performed on data collection during 2013 focused primarily on answering theHelixresearchquestionsatabroadlevel(E1).Since2014,thefocusofanalysishasbeentodevelopAtlas (E2).Version0.25ofAtlaswaspublished inNovember2014.Atlas0.5waspublished inDecember 2015 and an incremental improvement, Atlas 0.6 was published in April 2016.Refining this theory, and packaging it for independent assessment and deployment byindividuals andorganizations has been the focusof research efforts in 2016.Additional datacollected and continued work with organizations implementing Atlas culminated in thedevelopmentofAtlas1.1.

2.3.6PUBLISHING(F)

PublishingreportsandpapersforpublicconsumptionisakeyobjectiveforHelixresearch(F1).AllresultsandobservationsreportedinHelixpublishingaredoneinananonymousaggregatedmanner. Nothing published by Helix is traceable to any particular individual or to anorganization.Organizationsmay choose to reveal their participation in theHelix project, butthey are not listed in any Helix report. In addition, peer-reviewed conference and journalpaperscontinuetobepublishedforwidedisseminationofHelixresults.WhilesomeformofanorganizationalprofileiscreatedaspartofinternalHelixanalysis,insomerarecases,aprivatereport is provided to participating organizations upon request to support their systemsengineeringworkforcedevelopmentefforts(F2).

AcompletelistofHelix-relatedpublicationscanbefoundinAppendixA.

2.3.7VALIDATION,FEEDBACK&DEPLOYMENT(G)

Since publishing Atlas 0.5, step G has become the primary focus of the Helix process. Inimplementationeffortsconductedin2016,theAtlas theoryandproficiencymodelhavebeenvalidated in a number of ways. During 2015, Helix began deploying Atlas with specificorganizationsinanattempttouseAtlastoestablishtheproficiencylevelsandcareerpathsof


13

participants and to be able to discuss ways to develop their careers in the future, towardsachieving targeted levels of proficiencies required for particular senior positions within theorganization (G1). In 2016, Helix helped a few organizations think critically about howAtlascouldbeimplemented–includinganymodificationstofittheorganizationalcontext.(G2)Theresult of this work has helped the Helix team to clearly identify where tailoring ofAtlas isexpected versus where Atlas is expected to remain very consistent regardless of theorganization, domain, etc. The primary expectations for tailoring are highlighted in thediscussionofimplicationsforuseoftheproficiencies.

ThefirstHelixworkshopwasheldinJuly2014,withparticipationofrepresentativesfromDoD,academia,andindustry,includingrepresentativesfromorganizationsthatparticipatedinHelixinterviews. Feedback from the workshop significantly shaped Atlas 0.25. The second HelixworkshopwasheldinAugust2015andreinforcedtherelevanceandpotentialvalueofAtlastoa variety of systems engineering organizations. The third Helixworkshop, an early adopter’sworkshop,washeldinSeptember2016andprovidedparticipantstheopportunitytoexamineAtlasindetailandevenallowedparticipantstheopportunitytousethetools.ThefourthHelixworkshopwasheldinOctober2017andfocusedonthemethodsthathadbeenutilizedtodateinemployingAtlasaswellasresearchupdatesin2017(G3).


14

2.4QUALITATIVEANALYSIS

TheHelix researchmethodology discussed in the preceding sectionswas deployedusing theanalysisprocessesdescribedbelow.

2.4.1CODING

The interview dataset comprises nearly 6,000 pages of transcripts and summaries from 287individuals. In order to make sense of such a large quantity of data, the Helix team usesqualitativedataanalysis,primarilythroughdatacoding.Codingis“asystematicwayinwhichtocondenseextensivedatasets intosmalleranalyzableunits throughthecreationofcategoriesand concepts derived from thedata.” (Lockyer 2004) Codes canbe layered, andevolveovertime,asexplainedbelow.Whendevelopingatheory,asinthedevelopmentofAtlas,categoriesandcodesaregeneratedafterexaminingthecollecteddata,aligningwiththegroundedtheoryapproach.(Bourque2004andLockyer2004)Themaintypeofcodingdonebytheteamsofariscalled “open coding”, the purpose ofwhich is to break down, compare, and categorize data(StraussandCorbin2014).

Theteamhasusedtwotechniquesforcoding:autocodingandmanualcoding."Autocoding"isonly the first stage in parsing the information contained in transcripts and is not fullyautomateddespiteitsname.Instead,astheteamreviewsandcleanseachtranscript,headingsareaddedtothesourcedocumentstoblockoutalargeareaoftextasaddressingaparticulartopic,suchaspersonalcharacteristics,mentoring,experiences,etc.Whenthedocumentsareimported into the qualitative analysis tool,NVIVO, the tool then automatically codes all textunderthatheadingforthegivensubject.Theteam,thencanpullupallautocodedtextrelatedtopersonalcharacteristics,forexample,fromacrosstheentiredatasetandexamineitatonce.This allows a more consistent look at the related data that can then evolve more quickly,allowingtheteamtoidentifypatternsthatoccuracrossdata.

Auto coding is a useful approach, but has its drawbacks.One of the strengths of the codingapproach is that codes can overlap - individuals may discuss several issues together andresearchers can layer multiple codes together. Not only does this help to give a truecharacterizationofthedata,butcommonpatternsinoverlapsmayprovideusefulinsights.Forexample, the proficiency of big picture thinking was often discussed simultaneously withseveral of the values that systems engineers provide. This helped explain, for example, therelationshipbetweenbigpicturethinkingasacriticalskillandhowthatapproachcanprovidevalue on diverse teams. Butwhen using auto coding, layered codes are not possible; in theexample of big picture thinking and value, the text would be tagged either as "PersonalCharacteristics"or "Proficiency" -notboth.Sinceautocoding isonly the first step, thereareadditionalopportunities to create the layeringandcomplexity that reflects thenatureof thedata. However, auto coding does limit the researcher tomake a choice aboutwhat ismostimportant ormost prevalent in a section at the outset,which raises the risk that important


15

relationshipscouldbemissedlater.Theotherdrawbacktoautocodingisthatcategorieshadtobedevelopedandappliedtoalldataand,therefore,couldnothappenearlyintheproject.Theteamagreedtoalimitedsetofcategories,largelyaligningwithelementsofAtlas0.5publishedin2015.

Ifautocodingwasnotused–forexample,iftherewereanewareaofinquiry,meaningthatnoheadingshadpreviouslybeen identifiedandapplied–thentheteamhadtomanuallyreviewandcodeall~6,000pagesofdata.Thoughkeywordsearchescouldbeused, therewasa riskthatdatacouldbemissedifonlykeywordsearcheswereused.Forthisreason,theteamusedavarietyofkeywordsearchesrelatedtoagiventopicaswellasascanningreadofatranscriptwhendoingthe initialpass formanualcoding.Forexample,when looking for informationontraining, keywords included, "train," "course," "class," "learn", and "study". Once the initialcodingwascomplete,thisisessentiallyequivalenttoautocodingintermsoflevelofdepth.

Additionalcodeswerethenaddedtothissubsetofthedatatofurtherclarifythepatterns.Forexample,atotalof30individualpersonalcharacteristicswereidentifiedbyparticipants.Someof these, in the discussion, were directly linked to the values that they helped to provide –thesesectionsweredoublecodedforboththecharacteristicandthevalue.Onceallofthedatahad been analyzed, the team identified a reportable threshold – for example, for personalcharacteristicstherewere141excerptsand30characteristics.

Individualcharacteristicswerementionedanywherefrom15timestoonlyasingletimeacrossthe excerpts. While none of the characteristics is “wrong”, it was also not useful to simplyprovidealaundrylistofitems,particularlythosethatwereonlymentionedonceacrosssuchalargedataset.Itwasmoreusefultofirstidentifywhethertherewereanyrelationshipsbetweenitems that might help identify areas of importance. This was done by comparing overlapsbetweencodes.Inotherwords,asingleexcerptmightbecodedformultiplecharacteristicsthatwerediscussedtogether.Byexamininghowoftencharacteristicswerecross-coded,ithelpedtoidentify relationships that participants believed are important across organizations. Forexample, in terms of personal characteristics, ambition and internal motivation often werediscussedsimultaneously,whichiswhytheyaregroupedtogetherinAtlas.Figure3providesanexample of the coding comparisons conducted by the Helix teams. The higher the bars, thehigher the overlap in coding between characteristics. This provides Helix with insight intorelationships between and patterns around characteristics based on how intervieweesdiscussedthem.


16

Figure3.ExampleofCodingRelationships

When reporting, there was also a natural cutoff – again, in the example of PersonalCharacteristics, any characteristic mentioned in more than 6% of the excerpts. This was anatural thresholdbecausethenextclusterofcharacteristicsoccurred inapproximately1%orless. Items above the 6% threshold were reported in Atlas 0.5; items below it were not.However,becausethecodingstructureremains inplace,asadditionaldata isadded, ifothercharacteristicsbecomemoreprevalent,thentheymaybeaddedtothenextiterationofAtlas.Personal Characteristics were also presented in the descending order by the number ofexcerpts in the dataset; e.g., “self awareness” is listed first because it was discussed in thehighestnumberofexcerpts.

2.4.2TOOLSUPPORTANDCOMBININGQUALITATIVEANALYSISWITHDEMOGRAPHICS

Tosupporttheitsanalysis,theteamhasimportedallofthedataintoNVIVO(QSRInternational2016),apowerfulandpopularcommercialqualitativeanalysistool.NVIVOallowstheteamtocode text as well as overlay additional information and identifiers about the sources. Thisreplacestoolsusedearlyintheproject,namelyacombinationofDedoose(Dedoose2016)andMicrosoftExcel,whichprovedinsufficienttohandlethevolumeanddiversityofdatarequired.In NVIVO, the sources are tagged with a code for the organization in which the individualworkedatthetimeoftheHelixinterview,andtheneachorganizationislinkedtocharacteristicssuchaswhetheritisgovernmentorcommercial;whetherdefense,healthcare,transportation,etc.; and how systems engineer are organized, e.g. embedded, matrixed, etc. As much aspossible,eachcommentisalsotaggedfortheindividualwhomadeit,thoughinthesummariesfrom the earliest interviews, there are a few exceptions. Each individual, again, has severalcharacteristics, including gender, whether they are a systems engineer or a peer, and someresultsoftheHelixcareerpathanalysis.


17

2.5CAREERPATHMETHODOLOGY

Inadditiontotheanalysisofinterviewdata,theHelixteamdevelopedamethodforanalyzingandvisualizingthecareerpathsofsystemsengineers.Thecareerpathmethodpresentedheresupplementsthequalitativedataanalysisdescribedearlierwithmorequantitativeinformationabout an individual’s career. This analysiswas conducted for 181 systems engineers1 from adozenorganizations.Theinitialdatacollectionforcareeranalysiswasconductedby:

• Reviewing the resumes submitted by each individual, including chronology,organizations,positiontitles,andalldescriptivetextprovidedwithintheresumes;

• Reviewinginterviewtranscriptsandnotestoadddetailtotheresumedata;

• Reviewing the preliminary results during follow up interviews to clarify analysis.Individuals self-selected whether or not they would like to participate in follow-upinterviews; roughly half of the individuals in the career analysis sample haveparticipatedinfollow-upinterviews;and

• ComparingthecareerpathswithexistingHelixresearchontheproficienciesofsystemsengineersandhowcareerpathelementsmayrelatetotheseproficiencies.(Pysteretal.2014b)

Using this approach, the Helix team developed a method to examine experiences and acommonframeworktocapture,analyze,andvisualizecareerpaths.Theself-assessmenttool(s)provided to individualsparticipating inHelix to create theirowncareerpaths, including theirproficienciesovertime,areavailableintheAtlas1.1.

2.5.1CHARACTERIZINGASYSTEMSENGINEER’SEXPERIENCES

Experimental literature on experiences has primarily focused on twometrics for experience:time(e.g.Fordetal.1993;Schmidtetal.1986;Firth1979;Davidz2006)andthefrequencyoftimes a specific task or activity of interest was performed (e.g. Stuart and Abetti 2002).Additional literature classifies human subjects based on their experiences – which is subtlydifferentthanclassifyingtheexperiencesthemselves–oftenusingacombinationoftimeandthe frequencyof tasksperformed.Thisapproachmayalso includeconsiderations for specificrolesplayed(e.g.StuartandAbetti2002,Kor2003,Kirschenbaum1992).Additional literatureinthefieldofsystemsengineering,suchasSheard’s“TwelveSystemsEngineeringRoles”(1996)or the Graduate Reference Curriculum for Systems Engineering (GRCSE) (Pyster et al. 2012)1The interviews for all systems engineers in the Helix study are included here. However, the resume data was not provided for some of these individuals, and not all resume data was sufficient to complete each type of analysis. In general, the career analysis sample is N=157. Where the analysis looks at a subset of the sample or where individuals were eliminated from analysis for insufficient data, the sample size (N=x) is provided in the text.


18

indicate, though, that the characterization of experiences is critically important tounderstandinghowexperiencesenablegrowth.

Thefirstchallengewastodetermineacommon“unitofmeasure”forexperience.Thoughtimeiscommon, itwasnoteasilyused in thedataavailable.Forexample, if someonedescribedapositiontheyheldoverafive-yearperiod,theydidnotexplaintheportionoftimetakenupbytheactivities theyperformedover those fiveyears. Inaddition, several individuals submittedinformationontheircareersthatincludeddetaileddescriptions,butdidnotincludemarkersforchronologicaltime.Becauseofthesedatalimitations,theHelixteamchosetouseapositionastheunitofmeasureforexperience.

BasedonboththeliteratureandtheHelixdataitself,eachpositionhasseveralcharacteristics:

• Relevance:A‘relevant’positionisonethatenablesasystemsengineertodeveloptheproficienciescriticaltosystemsengineering.

• Position:Everysystemsengineerwhoisemployedatanorganizationfillsapositionthatis established by the organization; that organization also defines the roles andresponsibilities to be performed. Helix considers position as a ‘unit of measure’ forexperience, sincemost of the characteristics of experience are in the context of theposition that is held. A ‘systems engineering’ position is one where the individual’sprimaryfocuswasonsystemsengineeringactivities.

• Chronological Time: The amount of time spent in any particular position or inperformingarole.

• NumberofOrganizations:Thenumberofdifferentorganizationsthatanindividualhasworkedat,notcountinginternalmovementwithinanorganizationacrossdepartmentsordivisions,reflectsthevarietyoftypesofexperiencesthatonemaypossess.

• OrganizationalSectors:Therearemanydifferencesinthegeneralcharacteristicsofanorganization based on its sector. InAtlas, three organizational sectors are identified:government,industry,andacademia.

• Roles: A role is a collection of related systems engineering activities. Roles wereidentified based on the activities consistently performedby systems engineers. Thereare16rolesidentifiedinAtlas,asdescribedinSection3.5,below.

• LifecyclePhases:Generic systemsengineering lifecyclephasesconsidered inAtlas arebased on the lifecycle phases in the Guide to the Systems Engineering Body ofKnowledge(SEBoK).(BKCASEAuthors2015)

• Systems:Therearemanyaspectstothetypesofsystemsonwhichasystemsengineercouldwork.Workingacrossthesedifferentcategoriesprovidesvaluableexperiencetoanindividualsystemsengineer.


19

o Domain:Thisistheprimaryareaofapplicationforthesystemsbeingworkedon.However, therearemanydomaincategorizations;somedomainsalsorelatetoindustrysectors.

o Type:Productsystems,servicesystems,andenterprisesystemsarethreemajortypes of systems, depending on the nature and composition of the system ofinterest. System of systems is another paradigm in systems engineering, andcouldbeacombinationofoneormoretypesofsystems.

o Level: A systems engineer could work on various levels of a system:component/element,subsystem,system,andplatformorsystemofsystems.

Byusingthedataavailableforeachindividual,thecharacteristicsofeachpositionplayedandtheorder that theyplayedthemcanbe identified.Looking forpatternsacross theHelixdataset,thisinformationcanbeusedtodevelopapreliminaryunderstandingofhowcareerpathsshapeproficiency.

Theways inwhich positionswere categorizedwere pulled from existing literaturewhereverpossible.Forexample,asystemsengineerworkinginthecommercialsectorofacompanymaydefine life cycle in different terms than those used by aUSDepartment ofDefense systemsengineer.Tonormalizethediscussion,thedefinitionoflifecyclestagesfromtheGuidetotheSystemsEngineeringBodyofKnowledge (SEBoK)wasused; the interviewee’sownwordsandphrasingwerecomparedwiththedescriptionsoflifecyclestagesintheSEBoKandcategorizedappropriately. (BKCASE Editorial Board, 2014) Likewise, the roles played by the intervieweeswere based on Sarah Sheard’s “Twelve Roles of Systems Engineers” (Sheard 1996), althoughroleshavebeenaddedtoreflectwhatwasseeninthedata.Whereexistingliteraturewasnotavailable,categorieswerecreatedthatreflectthecharacterofthedata.

2.5.2CHARACTERIZINGASYSTEMSENGINEER’SEDUCATION

Educationplays twokeyroles in thedevelopmentofsystemsengineers.First, itprovides thefoundation knowledge to support engineering-relatedwork. Typically, this takes the form ofundergraduate education in an engineering discipline, technical field, or physical science.Second,graduateleveleducationisanavenuetodevelopmoreadvancedskills,exploremorein-depthknowledge,andhelpsystemsengineersgrowastheymovethroughtheircareers.

Thecharacterizationofeducationwasmuchmorestraightforwardthanthecharacterizationofexperiences.Foreachsystemsengineerinthesample,theteamrecorded:

• ChronologicalTime:Thedateofthecompletionofthedegreeprogram.

• Type of Degree: This is the level of education an individual achieved. The categoriesusedwere:bachelor’s,master’s,anddoctorofphilosophy(PhD).Forthisanalysis,only


20

education that resulted in a degree was recorded. Individuals did receive graduatecertificates or took individual courses, but there was not enough data to draw anymeaningfulconclusions.Also,ifadegreewasinprogressbutnotcompleted,itwasnotrecorded.

• FieldofStudy:Theprimarydisciplineonwhichtheindividual’seducationwasfocused.These were initially recorded as reported. Over time, categories of related fields ofstudywerecreated.

AllsystemsengineersintheHelixsampleheldatleastabachelor’sdegreeandthemajority–58%–heldatleastamaster’sdegree.

2.5.3IDENTIFYINGKEYPOSITIONS

Athirdaspectofcareerpathsarethekeymilestonesforasystemsengineer’scareer.TheHelixteam focused on major steps or changes in a systems engineer’s positions. A position isequivalent to the rolesand responsibilitiesassociatedwithan individual’s title.Organizationswilldefinewhatrolesandresponsibilitieseachpositioncontainsandpositiondescriptionsmaynottranslateacrossorganizations.Thekeypositionsidentifiedforsystemsengineerincluded:

• First systemsengineeringposition. Thiswas self-identifiedbyparticipants as the firstposition in which systems engineering responsibilities were the primary focus of aposition, though theymayhavenon-systems engineering responsibilities aswell. Thiswas often difficult to identify, because participants indicated that their roles oftentransitionedgraduallyand itwashardto identifywhentheyofficiallybecamesystemsengineers, especially because so many never had that specific title. The Helix teamrecorded this information in whatever way it was provided by participants. In a feworganizations,thehierarchyandstructureforbecomingasystemsengineerwasmuchmorewell-defined,andforindividualsinthoseorganizations,thetransitiontosystemsengineerwasmoreeasilyidentified.

• Chief systemsengineeringpositions.A chief systems engineer (CSE) is someonewhohasformalresponsibilitytooverseeandshepherdthetechnicalcorrectnessofasystem,often coordinating with many other systems engineers who have smaller scopes ofresponsibility.ThesemilestonesareanypositionsinwhichanindividualactedasaCSE,regardlessoftheirtitlewithintheirorganization.

• Projectmanagerpositions.Aprojectmanagerissomeonewhohasformalresponsibilityto oversee the programmatic aspects of a system, generally focused on budget andschedule. Project management responsibilities sometimes overlap with SEresponsibilities,particularlythosearoundplanningandmanagement;insomeinstances,aCSEmayalsofunctionasaPM.


21

Thesemilestonesareimportantforunderstandinghowthenatureofasystemsengineer’sworkhaschangedovertime.Italsogivesinsightsintohowquicklyanindividualprogressesthroughdifferentstagesofhercareer.Bycomparingthesepatternsacrossindividuals,commonrangesofprogressioncanbeidentified,ascanoutliers.Forexample,amongtheCSEsdiscussedinthispaper,one individualbecameaCSEonly8yearsafter completinghisundergraduatedegree.However, only 12%of CSEs gained their first CSE positionwithin 10 years after entering theworkforce;therefore,thisisanoutlierratherthantypicalforCSEs.

2.5.4ASSESSINGPROFICIENCY

Intervieweeswereaskedaboutnotonly their commonactivitiesbutwhat theybelievewerethe critical knowledge, skills, abilities, behaviors, and patterns of thought (cognitions) thatenablethemtobeeffectiveinperformingthoseactivities.Helixcallstheseproficiencies.

By coding all of these responses individually and then aggregating like responses, the Helixteamhasidentifiedthekeyproficienciesofsystemsengineers.Theseareelaboratedin(Pysteret al. 2015). In brief, there are six proficiency areas, each of which contains several relatedgroupsofskills,orcategories,asdescribedbelow:

1. Math/Science/General Engineering: Foundational concepts from mathematics,physical sciences, and general engineering. Categories include: Natural ScienceFoundations; Engineering Fundamentals; Probability and Statistics; Calculus &AnalyticalGeometry;andComputingFundamentals.

2. System’s Domain & Operational Context: Relevant domains, disciplines, andtechnologies for a given system and its operation. Categories include: relevantdomains,relevanttechnologiesandsystems;relevantdisciplines;familiaritywiththesystem’sconceptofoperations.

3. Systems Engineering Discipline: Foundation of systems science and systemsengineering knowledge. Categories include: lifecycle; systems engineeringmanagement; systems engineering methods, processes, and tools; and systemcomplexity.

4. SystemsEngineeringMindset:Skills,behaviors,andcognitionassociatedwithbeinga systems engineer. Categories include: big-picture thinking; paradoxical mindset;flexiblecomfortzone;abstraction;andforesightandvision.

5. Interpersonal Skills: Skills and behaviors associated with the ability to workeffectivelyinateamenvironmentandtocoordinateacrosstheproblemdomainandsolutiondomain.Categoriesinclude:communication; listeningandcomprehension;working in a team; influence, persuasion, and negotiation; and building a socialnetwork.


22

6. Technical Leadership: Skills and behaviors associated with the ability to guide adiverseteamofexpertstowardaspecifictechnicalgoal.Categoriesinclude:buildingandorchestratingadiverseteam;balanceddecisionmakingandrationalrisktaking;managingstakeholdersandtheirneeds;conflictresolutionandbarrierbreaking;andbusinessandprojectmanagementskills.

In addition, the non-systems engineers in the sample – projectmanagers, classic engineers,executive leadership, and human resources personnel (HR) –were askedwhich proficienciestheyconsideredcriticalinthemosteffectivesystemsengineerswithwhomtheyworked.Thesewere also coded and aggregated with the systems engineers’ responses; they validated theexistingcategories.

In2015,theHelixteamprovidedtheandreviewedthedraftproficiencymodeltoparticipantsand had them react to the categories and structure directly. The existing structure wasvalidated,withnoadditionalskillsbeingcitedthatdidnotfitwithinexistingcategories;thisdid,however,helptheteaminre-allocatingsomeproficienciestoothercategoriestomakethemmoreeasilyunderstoodbyawideraudience.

Finally,systemsengineerswereaskedtoperformself-assessmentsoftheirownproficienciesatdifferentpointsintheircareers,whichcouldthenbeoverlaidwiththeircareerpaths.Earlyon,the Helix teamwould perform its own assessments during these discussions andmap themagainst the self-assessments to ensure alignment between the team’s approach and theparticipants’. They were also asked to cite what they believed were the most criticalproficiencies for their current positions. In addition, somewere asked to identify what theybelievedweretheminimumproficienciestobeeffectiveintheircurrentpositions.Non-systemsengineers also did self-assessments, to help identify where these proficiencies overlap withother disciplines. In addition, they were asked what they believed were the most criticalproficienciesortheminimumproficiencyleveltheywoulddesireinthesystemsengineersthatthey work with. All of this work helped to validate the proficiency set as a useful andcomprehensivemodel.

TheforcesidentifiedinFigure1–experiences,mentoring,educationandtraining–arelinkedtothegrowthofproficiencybyinterviewdata.Whenanindividualwouldciteacriticalskill,theHelix teamwould askhow that individual haddeveloped that skill over time. These typesofdiscussionswerecross-codedforboththerelevantforce(s)andtherelatedproficiency(ies).

2.5.5MAPPINGACAREERTIMELINE

As described above, chronological time is an attribute of all positions. The final step indevelopingacareerpathmapforanindividualwastocreateavisualizationovertimeofalloftheelements listedabove.Thisvisualization laysoutallofan individual’spositions,andtheircharacteristics over time, with their education, the career milestones, and their proficiencyassessments.Figure4,below,showsagenericexampleofthis.


23

In Figure 4, only the timing, roles and the lifecycle stages characteristics of positions areillustrated.Thisisfortworeasons:oneiseaseofvisualizationinasinglegraphic–thoughanycombinationofattributesispossibleinthisformat–andthesecondisthatsystemsengineerswereabletoprovidetheclearestdiscussionsonhowrolesandlifecycleexposurecontributetoproficiency.Forotherattributes,theserelationshipsweremoresporadicallyrepresentedinthedata;inaddition,notallsystemsengineersprovidedbasicdataonallattributes,buttheHelixteamwasabletocompleterolesandlifecycledatafornearlytheentiredataset(93%forroles;91%forlifecycles).

By creating these individuals “maps” for each career path, it is possible to start identifyingpatterns–notonlyinproficienciesbutinthecommonattributesthatleadtosimilarproficiencyprofiles.Additionalanalysisofthecareerpathsofindividualsinsimilarroleswasalsoinsightful;even though there is some individuality to each systems engineer’s career, the commonpatterns indicate ways that systems engineers may typically grow – or areas where certaintypesofsystemsengineersdifferfromothers.Theanalysishighlightedinthispaper isthatofchief systems engineers – not only of their career paths overall, but in a few critical cases,highlightingtheirdifferencesfromotherseniorsystemsengineers.

Figure4.GenericCareerPathMapping

X X Role1X X X Role2

X X Role3X X X Role4

X X Role…

TimeEduca+onalMilestones

CareerMilestones

CareerPath

Organiza7on1 Organiza7on2 Organiza7on…

Posi7on1 Posi7on…Posi7on2 Posi7on3

LifecyclePhase1LifecyclePhase2LifecyclePhase3LifecyclePhase…

1

2

3

4

5

6

1

2

3

4

5

6

1

2

3

4

5

6

ProficiencyProfiles

StartofCareer

Now


24

2.5.6STEPSTOVALIDATEFINDINGSFROMCAREERPATHS

Therewereseveralways inwhichthefindingsfromcareerpathswereverifiedandvalidated.Themoststraightforwardwasverificationthroughfollow-upinterviews,whereindividualswerepresentedwiththecurrentanalysisoftheircareerpathsandaskedtoprovideanycorrectionsorfillanygaps.Thoughonly48%ofintervieweesalsoparticipatedinfollow-upinterviews,thechangesandupdateswereminimal,andoftenreflectedadditionswherecertainaspectsofanindividual’s career had not been discussed in the initial interviews. Because the reviews byindividuals of their own career paths revealed nomajor issues with themethods, the Helixteamconsidered that themethod isavalidapproach tounderstanding thecausesof changeandgrowthovertime.

Intermsofadditionalvalidation,theHelixteamacknowledgesthat, foranumberofreasons,thewayanindividualprogressedthroughhercareermaynothavebeenanoptimalapproach.Participants were asked to identify areas where what they would have approached theircareersdifferentlybasedontheircurrentlevelsof insight.Theywerealsoaskedtheirgeneralsatisfactionwith their career progression and to identify areaswhere colleaguesmight havehad a different, preferred approach. And because – as explained in the dataset discussionabove–theintervieweesidentifiedthemselvesandtheirorganizationsidentifiedthemasallof“average” to “excellent” effectiveness, it is reasonable to draw conclusions about the careerpathsoftheseindividuals.

Finally,in2015,theHelixteamworkedwithoneorganizationtopilotthecareerpathapproach.IndividualsworkedthroughanexamplecareerpathwiththeHelixteamandthenmappedtheirowncareersinrealtime.Thefeedbackwasthatthisapproachwasmuchbetterstructuredandfocusedthananycareerguidancetheyhadreceived. Insomecases,thisreinforcedthattheircurrentplanningwasappropriate,andotherindividualsreportedthatwithinsightsfromtheircareerpaths,theyrealizedthattheyneededtoseeknewopportunities.Alloftheparticipants(n=34)agreedthatthisapproachwasusefulandwouldbeavaluabletoolforthemasindividualsystemsengineersandfortheorganization.

2.6CAPSTONEONPROFICIENCYANDCAREERPATHS

In2017,theHelixteamworkedwithastudentattheStevensInstituteofTechnology,MatthewPartacz. Mr. Partacz joined the Helix team as a student and his capstone project utilizedhistoricalHelixdataandHelixmethodologies,alongwithadditionalapproaches,toexaminetherelationship between career paths, proficiency, and project or program performance. Onehypothesis of Mr. Partacz’s study was that career path has a quantifiable impact on anindividual’ssystemsengineering(SE)proficiency.ThefollowingisanexcerptfromMr.Partacz’sCapstoneReport(2017),whichcapturesthestrongcorrelationhefoundbetweenexperiencesandproficiency.


25

There aremany differentways to look at career path, but for this study careerpathwas looked at very broadly. Career pathwas assessed by classifying eachindividualintheHELIXdatasetintooneofthreecategoriesasshowninTable1.

Table1:GeneralCareerPathClassificationDefinition

NewEngineer Experienced,NeverTitledSystemsEngineer

ExperiencedTitledSystemsEngineer

YearsofExperience

Lessthan9years

Equaltoorgreaterthan9years

Equaltoorgreaterthan9years

PositionsTitle’s

- 0yearstitledasSystemsEngineer

Greaterthan0yearstitledasSystems

Engineer

SEproficiency,asdefinedinAtlas,consistsofsixdifferentareasbasedonHELIXinterview data. Many individuals in the HELIX dataset completed a self-assessment,ratinghowtheybelievetheyperformeachSEproficiencyatthetimeoftheassessment.ThesixdifferentareasaredefinedinTable2.

Table2:AtlasProficiencyAreaDefinitionsArea Definition

Math/Science/GeneralEngineering

Foundationalconceptsfrommathematics,physicalsciences,andgeneralengineering.

System’sDomain&OperationalContext

Relevantdomains,disciplines,andtechnologiesforagivensystemanditsoperation.

SystemsEngineeringDiscipline

Foundationofsystemsscienceandsystemsengineeringknowledge.

SystemsEngineeringMindset

Skills,behaviors,andcognitionassociatedwithbeingasystemsengineer.

InterpersonalSkills

Skillsandbehaviorsassociatedwiththeabilitytoworkeffectivelyinateamenvironmentandtocoordinateacrosstheproblemdomainandsolutiondomain.

TechnicalLeadershipSkillsandbehaviorsassociatedwiththeabilitytoguideadiverseteamofexpertstowardaspecifictechnicalgoal.

Witheachgeneralcareerpathclassificationandproficiencyself-assessmentwecanbeginidentifyingrelationshipsbetweenthetwo.Relationshipsareevaluatedandpresented in twoways: (1)viamosaicchartsand (2)usingnon-parametricstatisticalanalysis,similarlytoTheBusinessCaseforSystemsEngineeringStudy:ResultsoftheSystemsEngineeringEffectivenessSurvey0,Section4.2.

When creating mosaic charts, the only difference to The Business Case forSystems Engineering Study: Results of the Systems Engineering EffectivenessSurvey 0, Section 4.2.1 is that career path classification definitions were


26

arbitrarily chosen instead of divided using theweighted summed indices. It isalsoimportanttonotethattheweightedsummedindicesweredeterminedusingthe cumulative score for SE proficiency and not for each individual area of SEproficiency.

Non-parametric statistical analysiswas conducted just as in The Business CaseforSystemsEngineeringStudy:ResultsoftheSystemsEngineeringEffectivenessSurvey 0, Section 4.2.2. Goodman and Kruskal’s Gamma and p-value is ameasureofassociationwhichexpressesthestrengthoftherelationshipbetweentwoordinalvariables.Notionally,Gammavaluesmaybeinterpretedas:0.0 ≤ 𝐺𝑎𝑚𝑚𝑎 < 0.2 → 𝑊𝑒𝑎𝑘 𝑟𝑒𝑙𝑎𝑡𝑖𝑜𝑛𝑠ℎ𝑖𝑝0.2 ≤ 𝐺𝑎𝑚𝑚𝑎 < 0.3 → 𝑀𝑜𝑑𝑒𝑟𝑎𝑡𝑒 𝑟𝑒𝑙𝑎𝑡𝑖𝑜𝑛𝑠ℎ𝑖𝑝0.3 ≤ 𝐺𝑎𝑚𝑚𝑎 < 0.4 → 𝑆𝑡𝑟𝑜𝑛𝑔 𝑟𝑒𝑙𝑎𝑡𝑖𝑜𝑛𝑠ℎ𝑖𝑝0.4 ≤ | 𝐺𝑎𝑚𝑚𝑎 | → 𝑉𝑒𝑟𝑦 𝑠𝑡𝑟𝑜𝑛𝑔 𝑟𝑒𝑙𝑎𝑡𝑖𝑜𝑛𝑠ℎ𝑖𝑝

Important to note, p-values are always cited with each Gamma statistic. P-valuesaretypicallyusedforrejectingthenullhypothesis;the lowerthep-valuethelesslikelythemagnitudeoftherelationshipistobeachanceoccurrence.Itisconventionalthatvaluesofp<0.05orp<0.01beusedasabasisforrejectingthenullhypothesis.

ForfulldetailsonMr.Partacz’scapstoneproject,seehisfullreport(2017).WithMr.Partacz’shelp indata collection andanalysis, theHelix teamwas able to cross-reference careerpathsgenerated for many of the systems engineers in the sample with their proficiency self-assessments, a critical step in further validating importance of the career path assessmentapproachandprovidingmoreinsightsonhowcareerpathpatternsalignwithproficiency.ThisworkisreportedintheCareerPathGuidebook.

2.7HELIXDATA

Helix research uses a bottom-up approach, based on the data being analyzed. Hence, it isessential to gather data that is sufficient in quantity and quality to enable effectivedevelopment of Atlas, and to provide reliable insights and recommendations that can beconfidentlyusedforthedevelopmentofeffectivesystemsengineers.

2.7.1DATASOURCES

Theprimary sourceof data forHelix research is face-to-face semi-structured interviewswithparticipants at their place of work. Additional information about the participant and theorganizationwerealsocollectedasavailable.AnotherdatasourcethatHelixgainedaccessto


27

was the application data for the INCOSE Systems Engineering Professional (SEP) certificationprogram.

2.7.1.1HelixInterviewData

FromJune2013,whenHelixconducteditsfirstsitevisitfordatacollection,untilJanuary2018,atotalof363participantswereinterviewedfrom22organizations.Typically,2to3membersoftheHelixteaminterviewedanywherefrom1to6participantsinasingleinterviewsession.

Interview participants, if willing, also provided their resumes with details about theireducationalbackground,workexperiences,andanyotherinformationtheywishedtoprovide.

Follow-upinterviewswereconductedovertelephonewithwillingparticipants,toexploretopicsthat could not be covered in the initial face-to-face interviews or to collect additionalinformationbasedontheirresumes.Follow-upinterviewswerealsousedtovalidateresultsofHelixanalysis.

In both the initial interviews aswell as follow-up interviews, transcripts were createdwhenaudio recording was permitted; when not permitted, summaries were prepared from notestakenduringtheinterviews.Thesetranscriptsandsummariesfromatotalofabout270hoursofinterviewsformthebulkofdatathatHelixanalyzed.

2.7.1.2INCOSESEPApplicationData

INCOSEprovides threedifferent levelsof SEP certification:Associate (ASEP), Certified (CSEP),andExpert(ESEP).ApplicantsfromallovertheworldseekingINCOSEcertificationapplyfortheappropriate level based on their systems engineering experiences, knowledge, andaccomplishment. INCOSE provided to Helix, under a Non-Disclosure Agreement, over 3000applicationformsreceivedfromapplicantsduringtheperiodMay2004toMay2014.Thoughtheapplicationdatawasavailableinelectronicform,itwasnotinaformatthatwouldreadilysupport analysis. Significant timeandeffortwas spent in extraction, cleaning, and tabulatingthedatatoenablefurtheranalysis.

AnalysisofINCOSEdatadidnotdirectlycontributetothebuildingofAtlas,butprovidedsomevalidationandadditionalinsightsfortheanalysisoftheinterviewdata.


28

2.7.2DEMOGRAPHICS

2.7.2.1DemographicsofSamplePopulation

Understandingthesamplepopulationisimportant,sincetheinterviewdataisreflectiveofthepopulation fromwhich it hasbeen collected, and in turn, thatdata is thebasis forAtlas.AnunderstandingoftheINCOSEapplicantsrevealsthebreadthofthedatathatitcontains.

2.7.2.2DemographicsofHelixDataset

Understandingthesamplepopulationisimportant,sincetheinterviewdataisreflectiveofthepopulationfromwhichithasbeencollected,andinturn,thatdata isthebasisfor identifyingcareer paths. Following the rubric for understanding the seniority of systems engineerspresented in Figure5, the results arepresented inFigure5. Seniorparticipants coveralmosttwo-thirds of the population while the remaining one-third is almost split almost evenlybetweenjuniorandmid-Levelparticipants.

Figure5.DistributionofsenioritylevelsacrossHelixdataset

Figure 6 illustrates the distribution of participants based on the “general career pathclassifications”usedinPartacz(2017).Itdividesthesamplebyindividualswhoarerecognized–andrecognizethemselvesassystemsengineers–andthosewhodonot.Athirdcategory“newengineer” denotes any individual with less than nine years of experience. Note that this isdifferentthantheHelixseniorityclassifications,whichdonotdependontime.

It can be observed thatmore than two-thirds of Helix participants are Experienced SystemsEngineers.NewEngineersareslightlybehindExperiencedSystemsEngineerswithonly31%oftheparticipantsbeingallocatedtoNewSystemsEngineers.Ontheotherhand,analmostevendistribution occurred among Experienced Systems Engineers who have never been officiallytitled“systemsengineer”andthosewhohave.

Senior65%

Mid-Level18%

Junior17%

SeniorityLevelDistribution


29

Figure6.DistributionofcareerpathclassificationacrossHelixdataset(fromPartacz2017)

Figure7denotes thedistributionof genderacross theHelixdataset. It canbeobserved thatmorethanthree-fourthsofparticipantsaremalesystemsengineers.Ineachorganization,theHelixteamrequestedadditionalinformationontheoverarchingsystemsengineeringworkforce– as opposed to only the sampled individuals.Most organizations could not or chose not toprovide this information. The Helix team does not know if the demographics of the samplereflect the overarching gender demographics of the populations or is a result of theway inwhichorganizationsselectedindividualsforparticipation.

Figure7.DistributionofgendersacrossHelixdataset

ToprovideamoredetailedcontextaboutHelixfindings,itishelpfultounderstandthedomaininwhich the systemsengineers interviewedperformtheiractivities.As it canbeobserved inFigure8,fromeveryfourparticipants,threearerelatedtoDefensetypeoforganizations.Therest of participants are involved in the domains of Healthcare, Transportation,Telecommunications,orInformationTechnology.

NewEngineer,31%

Experienced,NeverTitledSystems

Engineer,35%

Experienced,TitledSystemsEngineer,34%

GeneralCareerPathClassificationDistribution

Male81%

Female19%

Participants'GenderDistribution


30

Figure8.DistributionofIndividualsbyOrganizationDomain

Another classification of the type of participant organizations is their commercial affiliation.Helix classified commercial affiliation into: Government, Industry and Federally FundedResearchandDevelopmentCenters(FFRDC).AsitcanbeobservedinFigure9,morethanhalfof the participants belong to industry organizations. The rest of the dataset is distributedamong government entities and FFRDC, the former covering more than 20%, while directgovernmentorganizationsslightlylessthan20%.

Figure9.IndividualsbyOrganizationType

0%10%20%30%40%50%60%70%80%90%

Percen

tage

OrganizationDomain

IndividualbyOrganizationDomain

0%

10%

20%

30%

40%

50%

60%

70%

Government Industry FFRDC

Percen

tage

OrganizationType

IndividualsbyOrganizationType


31

2.7.2.3DemographicsofINCOSESEPApplicants

Fromtheover3000applicationforms,about2500uniqueapplicantswereidentifiedforfurtheranalysis.TheseapplicantswerepredominantlyfromtheU.S,buttherewereothersfromAsiaandEuropeaswell,asindicatedinTable3.

Table3.GeographicalDistributionofINCOSESEPApplicantsRank Country #Applicants %Total1 U.S. 1847 74%2 India 179 7%3 Germany 151 6%4 France 101 4%5 U.K. 49 2%6 Sweden 41 <2%7 Spain 36 1%

Other 100 4%

Informationfromallthe2504uniqueapplicantswasusedforanalysisofeducationbackground;asubsetofthoseapplicantswasanalyzedforexperiences.

2.8INTERPRETATIONANDGENERALIZATIONUSINGTHEDATASET

Helixiscarefulwhenusingthedatatounderstandwhetherandhowfindingsandconclusionsaboutthedatasetcanbegeneralizedtothewiderpopulationofsystemsengineers.Theteamrecognizesthattherearesomelimitationsbasedonthesample.Thoughitisrelativelylargeatnearly300individuals,thereisnoclearestimateofhowmanysystemsengineersareworkingintheUS,letalonetheworld;thismakesitdifficulttounderstandhowstatisticallyrepresentativethesamplemaybe.Likewise,all interviewsconductedtodatehavebeenintheUS;thoughafew individuals provided insight into, for example, education outside the US, and someorganizationshadunitsoutsidetheUS,currentfindingsreflectaUScontext.Likewise,thoughHelix has expanded beyond its initial defense roots to include healthcare, transportation,telecommunications, and other industries, Figure 3 clearly shows that the majority ofindividualsparticipating inthesampleare fromthedefense industry.Giventhese limitations,theHelixteamiscarefulnottoover-interpretthedata.

However, evenwith the limitations of the sample, the team believes the overall size of thesample and the diversity in terms of industries, organization types, and senioritymakes anyfindings and conclusions drawn from the data extremely useful, even if they are not fullygeneralizable. The Helix team first publishedAtlas 0.25 in 2014; since then the coverage of


32

industriesandorganizationshasgrownandtheteamhasnearlydoubledthesizeofthesample.Andoverthistime,therehavebeensomeupdatesandedits,butnomajorissuesorbreakswiththe theory have been discovered. This, again, builds confidence that the existing sample issufficienttoenableusefulandinsightfulwork.Astheteamcontinuesworkonimplementationand application of Atlas (see Future Directions), these activities should generate greaterconfidenceinthegeneralizabilityofthedata.


33

3HELIXACTIVITIESIN2017

ThissectionprovidesabriefoverviewoftheworkoftheHelixteamin2017.Ascitedbelow,theprimaryresultsofthestudyarefoundincompaniondocuments.

3.1WORKFOCUSEDONEFFECTIVESYSTEMSENGINEERS

TheHelixteamcompletedconductedthreesitevisits in2017,addingthreeorganizationsand76 additional individuals to the dataset (see section 2.7 for a description of the updateddataset). Inaddition,theteamperformedadditionalanalysestocreatethreenewdocumentsfortheHelixlibrary:

• Atlas1.1:TheTheoryofEffectiveSystemsEngineers–(SERC-2018-TR-101-A)Thisisanincremental evolution ofAtlas that reflects feedback from the community, additionalanalysis, and maturation of the team’s thinking in 2017. In particular, Atlas includesminor updates on the values systems engineers provide, the roles systems engineersplay, theproficiencymodel for systemsengineers, and thepersonal characteristics ofsystemsengineers.Henceforth,thiswillbereferredtoas“Atlas1.1”.

• AtlasCareerPathGuidebook–(SERC-2018-TR-101-B)Thisdocumentprovidesanalysesof the Helix dataset, providing common patterns in systems engineers’ careers. TheGuidebookalsoprovidessomeinsightsonquestionscommonlyaskedoftheHelixteamaround career paths and the team’s responses. Finally, additional work on linkingproficiencies to career paths has been completed and is reflected in the guide.Henceforth,thiswillbereferredtoasthe“Guidebook”.

• Atlas1.1. ImplementationGuide:MovingfromTheoryIntoPractice–(SERC-2018-TR-101-C)WheneverAtlas ispresented,therearemanyquestionsabouthowtotakethetheory and apply it in practice. TheGuide provides examples fromorganizations thathave implemented parts ofAtlas, and guidance created by the Helix team based onmany interactionswithorganizations around implementation aswell as the extensiveHelixdataset.Henceforth,thiswillbereferredtoasthe“ImplementationGuide”.


34

Figure10.Relationshipbetween“Helix”and“Atlas”

TherelationshipsbetweenthesedocumentsarehighlightedinFigure10,above.

In 2017, the Helix team conducted additional work on effective systems engineeringcapabilities, culminating in Atlas 1.1, which is an incremental update reflecting additionalanalysis of existingdata aswell as additional data collection in 2018. Though the changes inAtlas1.1arerelativelyminor(asreflectedinthe“.1”versionnumber),theyneverthelessreflectnot only additional data collection and analyses, but also incorporate feedback from thecommunity.TheHelix teampresented theirworkat several communityevents, including theIISE annual conference, the INCOSE International Symposium, theNDIA Systems EngineeringConference, and the SERC Sponsored Research Review (SSRR). At each of these events, theteamgainedfeedbackfromthecommunity,collectingfrequentlyaskedquestions,uncoveringareasofconfusion,andidentifyingareasforimprovement.Thechangesinclude:

• Revisionof theAtlas 1.1 graphic that explains all of the key variables included in thetheory.Thecontentdidnotchange,buttheteambelievestheupdatebetterhighlightsthetwocriticalactors–individualsandorganizations.

• Reorderingof the values systemsengineersprovide to reflect the frequencyatwhichtheyoccurredinthedataset.

• Updating the “Requirements Owner” and “Systems Architect” roles. The activitiesaround functional architecture were moved from Requirements Owner to SystemsArchitect which both better reflect the realities of the grouping of these activities in


35

practice, but are groupings which better align with the mental models of mostindividualswhohaveengagedwiththeHelixteamin2017.

• Therewereseveralminoredits totheproficiencymodel.Theproficiencyareasstayedthe same, though the area formerly titled “Systems Engineering Mindset” is now“SystemsMindset”.Withinthisarea,thecategoryformerlytitled“flexibility”hasbeenrenamed “adaptability”. This not only better reflects the comments in the Helixinterviews–whichrevolvedaroundtheabilityofanindividualtocopewithachange–butalsoreducesconfusion.Thedistinctionbetweenproficienciesandpersonalenablingcharacteristics is nuanced, and the term “flexibility” caused confusion about theclassification of the category. In addition, the titles of categories in the “TechnicalLeadership” proficiency area were updated to increase clarity. The previous titlesimpliedoverlap;e.g.“ManagingStakeholderswithDiverseandConflictingNeeds”and“ConflictResolutionandBarrierBreaking”seemedtooverlap,thoughtheirtopicsweredifferent. Though they are related, they are distinct. The Helix team renamed“Managing Stakeholder with Diverse and Conflicting Needs” to “Managing DiverseStakeholders”.

• Therewereminoreditstopersonalcharacteristics,particularlytheupdateofdefinitionsfor“inquisitiveness”and“life-longlearning”tohelpclarifythedistinctionbetweenthetwo.

In addition to the work on Atlas 1.1, the Helix team continued the work of Dr. Hutchison,building on her dissertation dataset on career paths by creating career paths for additionalindividuals inthesampleaswellascollectingnewcareerpathdata.AsAtlaswasrefined,theteamupdatedtheanalysesaroundcareerpathstoreflectthesechanges.Thecareerpathdatawasthenreanalyzedforpatterns.TheresultsoftheseanalysescanbefoundintheCareerPathGuidebook.

In addition to these analyses, the Guidebook contains insights from the team on how tointerpret and apply some of this guidance. A series of frequently asked questions – and theteam’sanswertothesequestions–isincorporatedintotheguidebook.

Finally, Mr. Matthew Partacz based his capstone project for his master’s in systemsengieneeringatStevensontheHelixproject.Mr.Partaczexaminedtherelationshipbetweencareerpathsandproficiencyandlookedforalinkbetweentheseandprojectperformance.Mr.Partacz’s report can be found on the Helix webpage (sercuarc.org/projects/helix) but keyfindingsrelatedtocareerpatharesummarizedintheGuidebook.

3.1.1ADDITIONALANALYSISOFPROCESSDATA

In2017,theHelixteamexaminedtheexistingdatatounderstandhowthedatacouldsupportthisbroaderscope.ThesummaryofthatworkisthattheHelixteamhassomeinformationon


36

Structure, though it varies widely between organizations, little on governance as mostparticipatingorganizations chosenot to share their policies around systemsengineering–orsimplydidnothavesuchpolicies.Thereisverylittleinformationonthedatasetonprocess.Astheteamwasfocusedonindividualsystemsengineers,processwasnotthefocus.Inthelittledataavailable,therearethreefindings:

• Process can be important, but is not a substitute for experience. Overall, systemsengineerswho bring experience, insight, and foresight are valued. Systems engineerswhoonlyfollowaprocessarenot.

• Inexperienced systems engineers tend to lean very heavily on process anddocumentation,whichcancauseprogrammangersorpeerstoquestionthevaluestheyprovide.

• Individualsatmostorganizationsinthesamplereportedthattheiremployersreactedtoissues or failures by adding to their processes.Most of these individuals complainedaboutthe“bloat”intheirprocessesaswellasthelackofclearguidanceonorauthoritytotailortheprocesses.

Theabovearenotlikelytobesurprisingtomostpeoplewhohaveworkedinoraroundsystemsengineering. The Helix dataset to date primarily consists of these types of qualitativeassessments, which are useful but do not contribute to the same type of analyses as wereconductedon, forexample,proficiencies.Theonlyadditionaldataarethespecificprocessanorganizationmightfollow,suchasDoD5000.02.Goingforward,theHelixteamneedstogatheradditional data around the processes used in organizations and how they impact theorganization’sabilitytosuccessfullydevelop,maintain,orupdatesystems.

TheapproachestheHelixteamplanstousetoaddresstheseissuesaroundprocessin2018arediscussedinSection4,below.

3.1.2ADDITIONALANALYSISOFORGANIZATIONALDATA

Inordertominethedataforrelevantattributesaroundorganizationaldata,theteamidentifiedanddefinedsevenprimarycharacteristicsoforganizationalculturesbasedonworkbySchein,(2010);CameronandQuinn,(2011),andthePMBOKguidedefinitionsoforganizationstructure.SeeAppendixCfordetaileddefinitionsofthesevencharacteristics.

1. StatusaffordedSystemsEngineers(High,Low,InTransition)2. Structure (SE is Functionally Centralized vs. Distributed – Strong, Balanced, orWeak

Matrix,Function,Project)3. Professionalism(High,Low,InTransition)4. Formality(High,Low)5. Influence(High,Low)


37

6. Collaboration(High,Low:Strength,Breadth)7. Change(types:planned,dynamic;SEinvolvement)

First,theteamselectedfourtranscriptsfromfourdifferentorganizationstocodeforthesevencharacteristicstoa)determinehoweasyordifficultitwouldbetoapplytherubricandb)getafirst indicationofthepresenceorabsenceofinterpretabledataintheexistingdataset.Fromthis analysis it was determined that a) it was possible to reliably code for the sevencharacteristics across two coders, and b) in these transcripts, some culture indicators werepresentwhileotherswerenot.Inparticular,SEstatus,structure,professionalismandformalitywereobservedtosomedegree,whiletherewerenoclearindicatorsofinfluence,collaborationandSEinvolvementinorganizationchangepresentinthissubsample.Second, the teamexpanded the test of the usefulness ofmining existing data by coding theseven characteristics in all transcripts for one organization. By examining 325 pages oftranscriptsfrom33interviewsfromoneorganization,57pageswereextractedthatcontainedsomereferencetooneormoreculturecharacteristics.Conclusionsfromtheseadditionalanalysesoforganizationdatainclude:

• Theinterviewscontainsomerelevantdataonorganizationcharacteristics,thoughforseveralcharacteristicsthedataissparse.

• Whereitispossibletocodeforspecificcharacteristics,itisdifficulttointerpretthemeaningofthepresenceorabsenceofspecificcharacteristicswithoutdiscussionsincetheimpactofculturecharacteristicsdependsonhowmembersofthecultureinterpretthem.Meaning,relevance,andimpactmustbedeterminedincontextbytheparticipantsintheculture.

• Theseanalysesenabledtheteamtoidentifyspecificinterviewprobesthatcanelicitmoremeaningfuldatainfutureinterviews.Theseprobeswerefurthertestedinthethreeadditionalsitevisitssubsequentlyperformedin2017.

TestingtheMeaningandValueoftheOrganizationConstructs

Based on the analyses of existing data and experience with the three new organizationsinterviewedin2017,theHelixteamrefinedthecharacteristicsandaddedtwoadditionalareasof interests to test for relevance at the 4th Annual Helix Workshop in October 2017. (SeeWorkshopReport,Hutchisonetal.2017)Participantsindividuallycompletedananalysisoftheirownorganizationsbasedonthesevenculturecharacteristicsandquestionsrelatedto(a)howtheroles,status,andbehaviorsoftheNon-SEtalentareas(ME,EE,CS,etc.)affectSEgrowthand effectiveness in their organizations, and b) how the customer's (or customers') culture,values,andexpectationsofSEaffectSEgrowthandeffectivenessintheirorganizations.Thegroupdiscussionabout themeaning, importanceandapplicabilityof the chosenculturalattributestotheirorganizationsrevealedthattheattributesaremeaningfulandimportantand


38

thatthepresenceandimpactoftheattributesvariesgreatlyacrossorganizations.ParticipantsencouragedtheteamtocontinuetoexploretheseaspectsoforganizationcapabilityandtheirimpactonSEgrowthandeffectiveness.Theworkshopvalidatedthatorganizationscanincreasetheirawarenessabout the impactof theirownorganizationculturesonSEbyusinga simpleandrelevantlensthroughwhichtoobservetheircultureandbehavior.Overall,theworkin2017increasestheteam’sconfidencethat:

1. Tools for observing and describing attributes of organization culture, structure andgovernance can increase awareness and options for growth within an individualorganization.

2. Describing the variability and inter-relationships of various culture, governance, andstructure attributes across organizations and their impact on Systems Engineeringcapabilities can advance our understanding and development of the SystemsEngineeringprofession.

3.1.3ADDITIONALANALYSISOFMETHODSANDTOOLSDATA

TheHelix team reviewed the information in the current dataset around systemsengineeringmethods and the tooling to support systems engineering. A list of themost common toolsmentionedinthedatasetcanbefoundinTable4.

Table4.CitedToolsandMethodsfromtheHelixDatasetMostCommonlyCited(>50%ofInterviews)

OccassionallyCited(10%<x<50%)

RarelyCited(<10%ofinterviews)

RationalRhapsodyDOORS

MagicDraw

MBSEMatlabRiskRadarTeamCenterMKSPLM3DCADXMicrosoftProjectMicrosoftPowerPointMicrosoftExcelSysMLArtisanStudioRiskReconSEERVitechCoreRedMineO-NET


39

Inaddition toexamining themethodsand tools in thedataset, theHelix teamalso reviewedhowindividualsdiscussedthesetools intheir interviews.Becausethiswasnotamajor focus,thedataissparse,butthefollowingarecommonpatternsseentodate:

• Individualsinalmosteveryorganizationdescribedtoolingasanissueorchallenge.

• In some organizations, the individuals felt that the organization had either not yetidentified sufficient tools or did not believe in investing in tools specific to systemsengineering.

• In others, the tools had been identified and were in use in the organization, butindividualsdidnothaveaccesstothem.Thiswasoftenbecausetherewerenotenoughlicenses for thenumberof individualswhobelievedthetoolswouldbeuseful in theirworkorbecausethosetoolswereonlyutilizedinsomeareasoftheorganization.

• Modelingwas a commonpoint of discussion, particularly in termsof how it couldbetransformational to the systems engineering work. Different terms were used todescribe this (model-based systemsengineering,model-oriented systemsengineering,model-based engineering, etc.), but individuals were excited about the possibility ofusingmodelstohelpguidetheworkandcapturetheresults,ratherthanthedocument-basedapproachthatprimarilydominatedthelandscape.

o Some organizations reported having a model-based systems engineeringinitiative. The individuals in the organization were all excited about this butexpressed that they did not believe the modeling and tools were sufficientlymature tosupport thisyet. In someorganization, individuals reported that thetools “workedwell enough” but that the alignment between themodel-basedapproach and the organization’s process meant that significant level of effortwas still required toproduce the expecteddocument-based artifacts. Sowhiletheapproachwasseenasbeneficial, itwasalsoconsideredamajorburdentotheprograms.

o In one organization, an entire programwas developed using themodel-basedapproach. The benefits of this approach were widely recognized within theorganization,andmanyindividualsreportedadesiretodothisapproachontheirown programs. However, the shift to a model-based approach in thisorganization included having several individuals dedicated specifically tomodeling, which significantly added to costs. The organization and individualswantedthistobroaden,butthefundingchallengeassociatedwiththishastoberesolved.

TheaboverepresentsthecurrentstateofthedatainAtlas.Futureworkwillincludeexaminingthis as part of the “infrastructure” of the organization, which will be described further inSection4.


40

3.1.4ADDITIONALDATACOLLECTION

In2017,theHelixteamadded76newindividualsand3neworganizationstothedataset.Thisbringthedatasettoatotalof363participantswereinterviewedfrom23organizations.

During the year 2017, data from3 neworganizationswas added into theHelix dataset. Thedatasetsawan incrementof76newinputsthatwereanalyzedaggregatelyto identifycareerpathsinsystemsengineering.

Figure11 illustrates the comparisonof seniority levelsamongHelixparticipants.As it canbeobserved, the input data is consistent with precious pattern. Before the year 2017, thedemographics were the following: junior (19%),mid-level (15%) and senior (66%). Once theinformation of new participants was included, the percentage of junior systems engineersdecrease to (17%).Mid-level increased2% toa totalof 17%.Therewasnoobserved changewith respect to thepercentageof senior systemsengineers,bothclassifications reported thetotaldemographicsofsystemsengineerstobe66%.

Figure11.Comparisonofseniorityleveldistribution

InregardtothetypesoforganizationtheHelixteamhascollaborated,Figure12denotesthecomparisonamongpreviousanalyzeddataagainsttheneworganizationsincludedduring2017.Federally FundedResearch andDevelopment Centers (FFRDC)type of organization double inpercentageduringthe2017. Industry typeoforganizationsdecreasedfrom65%to58%afterthe including the new data. Lastly, government entitieswere consistentwith previous yearsdata.

0%10%20%30%40%50%60%70%

Junior Mid-Level Senior

Percen

tage

Seniority

ComparisonofSeniorityLevel

Before 2017


41

Figure12.Comparisonoforganizationtypedistribution

3.2UPDATESTOTHERESEARCHQUESTIONS

From2012 through2016, theHelix team focusedon threeprimary researchquestions. Theywere:

1. Whatarethecharacteristicsofsystemsengineers?

2. Whatmakessystemsengineerseffectiveandwhy?

3. Howcanorganizationsimprovetheeffectivenessoftheirsystemsengineers?

WiththepublicationofAtlas1.0 inDecember2016,theHelixteambelievedthatquestions1and2wereanswered reasonablywell; theycouldbe improvedupon,but the foundationsofAtlashadremainedlargelystableforoverayearandadditionaldatacollectedreinforcedratherthancontradictedthefindings.

Despitetheprogressmadetodate,additionalworkwasrequiredtoensurethatHelixandAtlasfulfilled theirpotential impactwith thecommunity. Forexample,Atlasprovidesgreat insightfor individualsystemsengineers,but is itpossibletounderstandtheeffectivenessofsystemsengineers in teams or a collective systems engineering workforce. The Helix team receivedfrequentquestionsfromthecommunitynotonlyabouthowto implementAtlasbutwhetherHelixwouldalsoaddressthe“nextlevel”-thebroadercontextaroundthesystemsengineeringcapabilitiessupportedbythesystemsengineersinanorganization.With this inmind, in 2017, the research teamupdated the researchquestions to reflect thisshiftinfocus.Theresearchquestionsbecame:

0%

10%

20%

30%

40%

50%

60%

70%

Government Industry FFRDC

Percen

tage

OrganizationType

ComparisonofOrganzizationType

Before 2017


42

1. How can organizations improve the effectiveness of their systems engineeringworkforce? This question has two areas of consideration: first, how can theeffectiveness of the workforce itself be understood? Is it simply a collection ofindividualsorare theresynergieswhen lookingacross theworkforceasawhole?TheHelix teamhasworkedwith several organizations to understandhow they canutilizeAtlas in practice and to date, each of them treats their workforce as a collective ofindividuals, not as a holistic entity. There is additional research required to betterunderstand whether this is accurate or whether in systems thinking parlance, “thewholeistrulygreaterthanthesumofitsparts”.Thiswillprovideimportantinsightsasorganizationslooktohowtogrowtheirworkforce,notjustindividualmembersofit.

This leads to thesecondpartof thequestion,which ishowcanorganizationsactuallymake an impact in this area? In 2017, the Helix team collected additional data onorganizationalinitiatives,allofwhichalignedwithpreviousfindingsinAtlas1.0.

2. Howdoestheeffectivenessofthesystemsengineeringworkforceimpacttheoverallsystems engineering capability of an organization? The unspoken hypothesis here isthat having an effective systems engineering workforce should result in effectivesystems engineering capabilities. However, there are many factors which impact asystems engineering capability andwhile theHelix team anticipates that the systemsengineering workforce plays a critical role, as with the Atlas theory, the team alsoanticipates that several other factors will impact the ability of even a very skilledworkforcetotranslateskill intoeffectiveaction. Inordertounderstandthis, theteamhastoanswerthethirdquestion:

3. Whatcriticalfactors,inadditionaltoworkforceeffectiveness,arerequiredtoenablesystems engineering capability? Based on the Helix work through 2016, the teamdeveloped an expected set of variables that would impact systems engineeringcapability,asillustratedinFigure13.

Figure13.ExpectedVariablestoImpactSystemsEngineeringCapability

IndividualSEr FunctionsinaMulti-DisciplinaryTeam

SEWorkforce

Organization’sSystemsEngineeringCapability

Culture

Structure•  TechInfrastructureandSupport

Governance•  Policy•  Process


43

The Helix team investigated potential uses of the existing interview data to address theexpanded goal of understanding aspects of organizations that support effective systemsengineering capabilities. Although the interviewswere primarily focused on the individual, itwasrecognizedthatmany intervieweesreferredtoaspectsoforganizationculture,structure,andgovernancewhendescribingtheirowngrowthandeffectivenessassystemsengineers.

3.3RELATEDWORK

ThissectionhighlightsadditionalworkdoneonHelixin2017,specificallyworkbytheSERConaHelixtoolandabookwrittenbyArtPyster,NicoleHutchison,andDevaHenry:

• TheSERCbelievesthatHelixandAtlasprovidecriticalinsightsforindividualswhowishtounderstandtheirsystemsengineeringproficiencies,theircareerpaths,andhowthetwolinktogether.In2016,theHelixteampublishedanExceltemplatetofacilitateself-assessmentsofbothproficiencyandcareerpath,aswellaspaper-basedtools.However,forwide-scale use ofAtlas, the SERC believed that aweb-based toolwould bemorecritical.TheSERCinvestedresourcesin2017tobuildatooltosupportself-assessment.At the time of this report, the first iteration of the tool is nearly complete and willprovidethefollowingcapabilities:

o Individual proficiency self-assessments, allowingmultiple assessments coveringmanypoints intime, includingpromptinguserstocreatea“target”proficiencyprofile, and enabling tailoring of the proficiency model to fir the individual’sworkingenvironment;

o Proficiency graphics to allow individuals to easily compare proficiencies overtime,and

o Individual career path self-assessments, enabling data collection andcharacterizationonthecriticalvariablesofcareerpathsidentifiedinAtlas.

Futureiterationsareplannedtoenablevisualizationsofcareerpathsovertimeaswellas options for organizations, which would enable them to create an organization-specific tailoring of the proficiency model and collect data across their systemsengineeringworkforce.

• In2017,ArtPyster,theformerPIfortheHelixproject,NicoleHutchison,thecurrentPIfor the project, and Deva Henry, a researcher who worked on the project almostcontinuouslyforyears,developedaproposalforabookbasedaroundtheHelixfindings.This book, which heavily references the published Helix reports, also provides theinsights and opinions of the authors based on their long historywith the research aswell as additional on-the-record interviews, case studies, and views on the future ofsystemsengineeringandsystemsengineers.


44

ThisbookwillbesubmittedinJanuary2018anditisanticipatedthatitwillbepublishedinlate2018.


45

4FUTUREDIRECTIONS:HELIXIN2018ANDBEYOND

Aspartofunderstandingwhatmakessystemsengineerseffective,theHelixteamhascollecteddataonthewaysthatorganizationstrytoimprovetheirsystemsengineers’proficiency(calledorganizationaldevelopment initiatives)andsomebasic trendshavebeen identified.Thisarearequires further study and is critical to ensure that organizations that identify issues in theirsystems engineering workforce or organizational support for systems engineering canunderstandthepotentialimpactofandoptionsforimprovement.

Atlas1.1outlineswhatimpedesandenablessystemsengineers,fromaspectsoftheindividualsthemselves – such as proficiency, career path, roles, and positions – to aspects of theorganization, such as culture and the specific ways that systems engineering is valued andpromoted(ornot).Theinformationonthiscontextforsystemsengineersinthecurrentdatasetis helpful, in particular in highlighting where these areas can inhibit systems engineers’effectiveness.However,thedatasetisnotasrobustasisrequiredinthisareaiforganizationswant to use Atlas to make real changes to improve their systems engineering workforce.Additionaldatamustbecollectedonorganizationalculture,governance(includingprocesses)andstructure(includinginfrastructureandsupportforsystemsengineering).

TheHelix team, in order to draw clear conclusions around the organizational characteristics,will need to greatly increase the organizational variability of the dataset. This will require achange in data collection approach.While some site visitswith large numbers of individualsmaystillbenecessary,theHelixteamwillalsoneedtocreatearesearchprotocolthatwillallowthemtocollecttargetedorganizationaldatawithalowerlevelofefforttoincreasethenumberofparticipatingorganizations.

TheHelixteam,inordertobetterunderstandworkforceversusindividualeffectiveness,mustadd new data on individual systems engineers and collect aggregate data on the systemsengineering workforce from additional organizations. In addition, new analysis needs to beconducted to determine how to treat “the systems engineering workforce” as distinct from“individualsystemsengineers.”

In2017,theSystemsEngineeringResearchCenterdevelopedtoolingtosupportindividualself-assessments based on the Atlas findings. The team needs to evolve the tooling to allowincreasedcapability, includingcreatinganoption fororganizations to tailor this for theirownemployeesandseesummariesoftheresults.Thiswillaidtheteamin increasingcollectionofdatathatenablestheaggregateanalysisofindividualsasaworkforce.

TheHelixteamhasbegunexploringautomatedmethodstogleanadditional insightsfromtheexisting dataset. The team needs to continuework on this in 2017 to reduce the burden ofmanualcodinganalysisonthemorethan6,000pagesoftext.

The Helix team will also conduct an extensive literature review of systems engineeringcapability and build on thiswork to develop clearmethods for understanding and assessing


46

howsystemsengineersaresupportedorimpededinprovidinganeffectivesystemsengineeringcapability.

Inadditiontothetasksabove,theHelixteamwillbeginbuildingmodelstosupportthiswork.Thiswillincludemodelingtoolsthatwillhelpenableorganizationstoassessmenttheircurrentlevel of support for systems engineering and systems engineers. These models shouldeventuallybeabletohelporganizationsexplorethechangestheycouldmaketoimprovetheefficacy of their systems engineering workforce and increase their systems engineeringcapability.(Itislikelythatthisworkwillcontinuein2019.)

As the researchers continue data collection and expand the variety of organizationsparticipating, the team will gather additional data on the efficacy of organizationaldevelopmentinitiativesthatareintendedtogrowsystemsengineers.

The team shall develop a method for assessing the effectiveness of systems engineeringcapability in relation to the characteristics of the organization and the systems engineeringworkforceandvalidatethisthroughdatacollectionatavarietyoforganizations.

Theresearcherswouldaimtodevelopfiveitemstosupportthisworkin2018:

1) Aninitialframeworkforunderstandingtheeffectivenessofsystemsengineering(nominallyAtlasORG0.5);

2) Asetoftoolstosupportbroaderusebybothindividualsandorganizations;

3) Aseriesofdraftmodelstosupportexplorationofsystemsengineeringcapabilitywithorganizations;and

4) Atechnicalreportprovidingthedataandanalysissupportingeachofthethreeitemsabove.

5) Draftmodelsdevelopedtosupporttheresearch.

4.1ENVISIONEDENDSTATEFORHELIX

TheendgoalforHelix,whichwilllikelybecompletedin2019,istodevelopatheoryofeffectivesystems engineering capability – predicated on the hypothesis that if an organization has asufficientlyskilledsystemsengineeringworkforceandtheorganizationalcharacteristicssuchasculture,governance,andinfrastructurealignwithsystemsengineering–alongwiththetoolstosupport organizations’ self-assessment of their capabilities and their ability to change thosecapabilities. An appropriate analogy here is the “policy flight simulator”, which according toRouse is “designed for the purpose of exploring alternative management policies at levelsrangingfromindividualorganizationstonationalstrategy.”(2014)AsRouseexplains,“theideais for organizational leaders to be able to interactively explore alternative organizational


47

designscomputationallyratherthanphysically.Suchexplorationsallowrapidconsiderationofmanyalternatives,perhapsasakeystepindevelopingavisionfortransforminganenterprise.”

By developing a theory of what enables an organization to have an effective systemsengineeringcapability(e.g.AtlasORG),theHelixteamwillidentifythekeyfactorsthatimpactthiscapabilityandtherelationshipsbetweenthem.Thefoundinghypothesisforthisworkwillbe,“If anorganizationhasanappropriately sizedand skilled systemsengineeringworkforceandtheorganizationalcharacteristicsaresupportiveofsystemsengineering,thentheorganizationwillhaveaneffectivesystemsengineeringcapability.”

Withthistheoryandtherichdatasetbehindit,theteamwillbeabletobuildtoolstosupportdata collection around critical variables, and models to support this policy flight simulatorapproach. With it, organizations can assess their current capability and the factors thatinfluence it and run simulations to determine what organizational changes may enable anincreasedsystemsengineeringcapability.

Thoughprocess isanticipatedtobeoneofthevariablesthatHelixwillstudy,thisworkisnotintended to focusexclusivelyonmaking ‘betterprocesses’.Therearealreadyapproaches forprocessimprovementsuchasCMMI,SixSigma,andstatisticalprocesscontrol.Whiletheremaynotbemanyexamplesofthesetechniquesappliedtosystemsengineeringprocesses,theHelixteam nonetheless does not desire to create a process improvement methodology. Instead,areasof“process improvement”would includebetteraligningprocesswithothervariables inthemodel,suchasculture,governance,orworkforcecapabilities.


48

5CONCLUSION

TheworkonHelixin2017wascriticalpreparationfortheHelixworktocomeandhasfurtheredthedevelopmentofacommunityofinterestinunderstandingandbuildingexceptionalsystemsengineersandsystemsengineeringcapabilities.

Accomplishmentsthatprovideaclearfoundationforthenextstagesinthisresearchinclude:

• The work performed mapping general theoretical constructs about organizationalculture and governance to the systems engineering organizational context provides astarting point for designing future data collection, analysis strategies, and modelingapproaches.

• By examining additional areas of the existing dataset, the team has a clearunderstandingofthestatusofthecurrentdataandwhereadditionaldatacollectionisrequired.

• In updating the research methods and interview questions, testing these with threeadditional site visits, and reviewing the results, the team believes it has appropriatewaysofaskingfortherequiredadditionaldata.

Theresearchprocess itself isdevelopingacommunityofprofessionalswhocan influencethedirectionandeffectivenessofsystemsengineeringindiverseorganizationsinthefuture.

• Bycontinuingthedialogueaboutsystemsengineeringcareerpaths,successstories,andapplications of the prior research, and through the team’s evolving understanding ofcriticalcultureandgovernanceattributes,HelixisbuildingacommunityofcollaboratorswhoareevolvingthesystemsengineeringprofessioninparallelwiththeHelixwork.

• Themethodstheteamusestogeneratethisdialogueincludeinterviewquestionsthatspark self-awareness, site-visit summaries that provide organization leaderswith newperspectives, cross-industry workshops, conference presentations, documents andreferences.

Ultimately,justasthegrowthandeffectivenessofindividualsystemsengineersdependuponthecommitmentofanorganizationtogrowingitssystemsengineeringworkforcecapability,thevalueoftheHelixworkbeyondsupportingthedevelopmentofindividualsystemsengineerswillbebestrealizedthroughtheengagementofourwidercommunityofleadersofthesystemsengineeringprofession.TheHelixteamiscommittedtocontinuingthisdialoguein2018andbeyond.


49

WORKSCITED

Cameron, K. S. and Quinn, R. E. (2011). Diagnosing and changing organizational culture, 3rdEdition.SanFrancisco,CA:Jossey-Bass.

Davidz,H.L.2006.EnablingSystemsthinkingtoAcceleratetheDevelopmentofSeniorSystemsEngineers. Dissertation, Doctor of Philosophy in Engineering Systems, MassachusettsInstituteofTechnology.

Elm,JosephP.,andDennisR.Goldenson.TheBusinessCaseforSystemsEngineeringStudy:Resultsof the Systems Engineering Effectiveness Survey. Report no. CMU/SEI-2012-SR-009.SoftwareEngineeringInstituteCERTDivision,CarnegieMellonUniversity.

Firth,M.(1979). Impactofworkexperienceonthevalidityofstudentevaluationsofteachingeffectiveness.JournalofEducationalPsychology,71(5),726-730.

Ford, J.K.,Smith,E.M.,Sego,D. J.,&Quinones,M.A. (1993). Impactoftaskexperienceandindividualfactorsontraining-emphasisratings.JournalofAppliedPsychology,78(4),583-590.

Kirschenbaum,S. (1992). Influenceofexperienceon information-gathering strategies. JournalofAppliedPsychology,77(3),343-352.

Kor,Y.Y.2003.“Experience-BasedTopManagementTeamCompetenceand

Partacz, M., N. Hutchison, D. Verma. 2017. Building a Better Business Case for SystemsEngineering:TheRelationshipbetweenaSystemsEngineer’sCareerPath,ProficiencyandProjectPerformance.Hoboken,NJ:StevensInstituteofTechnology.

PMI.2013.AGuide to theProjectManagementBodyofKnowledge (PMBOK®Guide),5thed.NewtownSquare,PA,USA:ProjectManagementInstitute(PMI).

Pyster,A.,D.H.Olwell,T.L.J.Ferris,N.Hutchison,S.Enck, J.F.Anthony,D.Henry,andA.Squires(eds).2015.GraduateReferenceCurriculumforSystemsEngineering(GRCSE™),version1.1.Hoboken,NJ,USA:TheTrusteesoftheStevensInstituteofTechnology©2015.Availableat:http://www.bkcase.org/grcse-10/

Rouse,W. 2014. “Human interaction with policy flight simulators.” Applied Ergonomics. 45(1):January2014,pp.72-77.

Schein, E. H.(2010). Organizational culture and leadership, 4th Edition. San Francisco, CA:Jossey-Bass.

Schmidt, F.L., J.E. Hunter, J. E., and A.N. Outerbridge. (1986). Impact of job experience andability on job knowledge, work sample performance, and supervisory rating of jobperformance.JournalofAppliedPsychology,71(3),432-439.


50

Sheard,S.1996.“TwelveSystemsEngineeringRoles.”ProceedingsoftheInternationalCouncilonSystemsEngineering(INCOSE)SixthAnnual InternationalSymposium,7-11July1996,Boston,MA.

Sheard,S.2000."The12SystemsEngineeringRolesRevisited."PaperpresentedattheINCOSEMid-AtlanticRegionalConference.April2000.Reston,VA,USA.p5.2-1-5.2-9.

Stuart,R.W.andP.A.Abetti. (1990). Impactofentrepreneurialandmanagementexperienceonearlyperformance.JournalofBusinessVenturing,5(3),151-162.


51

APPENDIXA:LISTOFHELIXPUBLICATIONS

Clifford,M. andN.Hutchison. 2017. “Helix:Understanding SystemsEngineeringEffectivenessthroughModeling.”ProceedingsoftheNationalDefenseIndustrialAssociation(NDIA)SystemsEngineeringConference,25October2017,Washington,DC.

Henry,D.,Hutchison,N.,Pyster,A.,Dominick,P.,Lipizzi,C.,Kamil,M.,Manchanda,S.“Summaryof Findings from the Helix Project (2013-14) – An Investigation of the DNA of theSystems Engineering Workforce,” National Defense Industries Association SystemsEngineeringConference,Springfield,VA,October27-30,2014

Henry, D., N. Hutchison, A. Pyster, P. Dominick, C. Lipizzi, M. Kamil, S. Manchanda. 2014.“SummaryofFindingsfromtheHelixProject(2013-14)-AnInvestigationoftheDNAof the Systems Engineering Workforce”. Proceedings of the National DefenseIndustrial Association (NDIA) Systems Engineering Conference, Springfield, VA, USA,October28-30,2014.

Henry,D.,N.Hutchison,A.Pyster.2015. “Summaryof Findings from theHelixProject (2013-2014): An Investigation of the DNA of the Systems Engineering Workforce.”Proceedings of the National Defense Industrial Association (NDIA) SystemsEngineeringConference,October29-November1,2014,CrystalCity,VA.

Hutchison,N.2017.Atlas:AGuidetoGrowingEffectiveSystemsEngineers.ProceedingsoftheInstitute for Industrial and Systems Engineers (IISE) Conference, 20-23 May 2017.Pittsburgh,PA,USA.IISE2017Conference#2546.

Hutchison,N. 2015.A Framework to Classify Experiences and Enable Career Path Analysis toSupport Maturation of Effective Systems Engineers in the Defense Industry. PhDDissertation.Hoboken,NJ:StevensInstituteofTechnology.October2015.

Hutchison, N. and A. Pyster. 2015. “The Helix Project: Analysis of INCOSE SE CertificationProgramApplicants.”Presentation to the INCOSECorporateAdvisoryBoard, INCOSEInternationalWorkshop,January2015,LosAngeles,CA.

Hutchison,N.,A.Pyster,D.Henry.2017."Atlas:UnderstandingWhatMakesSystemsEngineersEffective in the US Defense Community." Systems Engineering. Expected to bepublished2017.

Hutchison,N.,D.Henry,A.Pyster,andP.Dominick.2014.“SystemsEngineeringCareerAnalysis:Supporting a Theory of Systems Engineers.” Proceedings of the INCOSE Europe,Middle-East, Africa Systems Engineering Conference (EMEASEC), Cape Town, SouthAfrica,October27-30,2014.

Hutchison,N.,D.Henry,A.Pyster,P.Dominick.2014(b).“SystemsEngineeringCareerAnalysis:SupportingaTheoryofSystemsEngineers.”Proceedingsof theEurope,MiddleEast,and Asia Sector Systems Engineering Conference, 27-30 October 2014, Cape Town,SouthAfrica.

Hutchison,N.,D.Henry,A.Pyster.2014(a).“EarlyFindingsfromInterviewingSystemsEngineersWho Support the US Department of Defense.” Proceedings of the InternationalCouncilonSystemsEngineering (INCOSE) International Symposium,30 June–4 July2014,LasVegas,NV.


52

Hutchison,N.,D.Henry,A.Pyster,R.Pineda.2014.“EarlyFindingsfromInterviewingSystemsEngineerswhoSupporttheUSDepartmentofDefense.”2014InternationalCouncilonSystems Engineering International Symposium, June 30-July 3, 2014, Las Vegas,Nevada.

Hutchison,N.,J.Wade,S.Luna.2017.“TheRolesofSystemsEngineersRevisited.”Proceedingsof the 27th Annual INCOSE International Symposium, 15-20 July 2017, Adelaide,Australia.

Jauregui,C.,A.Pyster,D.Henry,N.Hutchison,andC.Wright.2016.“InsightsontheExperiencesandEducationofINCOSE-CertifiedExpertSystemsEngineeringProfessionalsandChiefSystemsEngineers.”Tobepublished in theProceedingsof the InternationalCouncilon Systems Engineering (INCOSE) 26thInternational Symposium, 18-21 July 2016,Edinburgh,Scotland.

Lipizzi, C., S.Manchanda,M. Kamil, A. Pyster, D. Henry, N. Hutchison. 2015. “The EducationBackground of INCOSE Systems Engineering Professional Certification ProgramApplicants.”ProceedingsoftheInternationalCouncilonSystemsEngineering(INCOSE)InternationalSymposium,Seattle,WA,July2015.(Acceptedforpublication)

Pyser, A., S. Rifkin, D. Henry, K. Lasfser, N. Hutchison, D. Gelosh. 2013. "The Helix Project:Understanding theSystemsEngineerswhoSupport theUSDepartmentofDefense."Asia-PacificConferenceonSystemsEngineering(APCOSE),Yokahama,Japan,Sept.8-11,2013.

Pyster, A. 2015. “The Helix Project: Analysis of INCOSE SE Certification Program Applicants.”Special Presentation to the International Council on Systems Engineering (INCOSE)CorporateAdvisoryBoard(CAB),January25,2015aspartoftheINCOSEInternationalWorkshop,January25-27,LosAngeles,CA.

Pyster,A.,D.Henry,N.Hutchison,C. Jauregui, J.Armstrong,M.Clifford.2015.Reporton theSecond Helix Workshop: Exploring the Theory of Systems Engineers’ Effectiveness.Hoboken,NJ:SystemsEngineeringResearchCenter,StevensInstituteofTechnology.

Pyster, A., N. Hutchison, D. Henry, C. Barboza. 2015. “Helix:WhatMakes Systems EngineersEffective?” Proceedings of the Conference on Systems Engineering Research (CSER),March17-19,2015,Hoboken,NJ.

Pyster, A., R.L. Pineda, D. Henry, N. Hutchison. 2013. Helix – Phases 1 and 2. Hoboken, NJ:SystemsEngineeringResearchCenter,StevensInstituteofTechnology.SERC-2013-TR-038-2.

Squires, A.,Wade, J., Hutchison, N. 2016. “Building a Pathway to Systems Education for theGlobal Engineer.” To be published in the Proceedings of the American Society forEngineering Education (ASEE) 123rdAnnual Conference, 26-29 June 2016, NewOrleans,Louisiana,USA.


53

APPENDIXB:UPDATEDINTERVIEWQUESTIONSFORHELIX

IndividuallyFocusedGettingtoKnowYouIND01 Howdidyougetintosystemsengineering?IND02 Doyouconsideryourselfasystemsengineer?IND03 Defineyourcurrentposition.IND04 Howwouldyourateyourownclarityaboutyourroleasasystemsengineerrightnow

–high,medium,orlowclarity?Why?ExploringOrganizationalCharacteristicsGRP01 Howdosystemsengineersintegrateintoteamshere?(Whatisthedayinthelifeofa

systems engineer like with respect to who they work with and what they dotogether?)

GRP02 Howdoyouthinkothersonyourteamwouldratetheclarityofyourroleasasystemsengineerrightnow–high,medium,orlowclarity?Why?

GRP03 Wearegoingtodosomefreewordassociations.ThisiswhenIsayawordorphraseand you write down, without filter, the first three things that come to mind. Let’spracticealoud:apple;car.OK,sothinkingaboutyourexperience inthisorganization,pleasewrite down the first threewords that come tomindwhen I say each phrase[worksheet forparticipants]: systemsengineering;systemsengineeringmanagement;seniorleadership;[organizationname].

GRP04 Draw a picture that shows how systems engineering is really done here, in yourexperience.(Followup:Thinkingabout thecultureof theorganization,whataspectsof“thewaywedothingsaroundhere”contributetoSEgrowthandsuccessandwhichhinderSEgrowthandsuccesshere?)

GRP05 Howwouldyoudescribethestatusofsystemsengineeringinthisorganization?Whatisyourrationale?(Followup:Aresystemsengineersconsideredleaders?)

GRP06 Whatkindsofdecisionsarehighlyinfluencedor“owned”bysystemsengineersinthiscompany?(Followup:Arethereanykindsofdecisionswhereyouarenotatthetableandthinksystemsengineersshouldbeinvolved?)

GRP07 Whenyouthinkofsystemsengineeringasaprofession,wouldyousayyoufeelveryconnected,moderatelyconnected,slightlyconnectedornotconnectedtothebroadersystemsengineeringcommunity?Whydoyousaythat?

GRP08 What’s changing in your organization and what role do SE’s have in driving thosechanges?

GRP09 Howdo you think that the nature of thework you do influences your approach tosystemsengineering?

GRP10 Howdoprojectsgetstaffed?OrganizationalChange


54

ORG08 Ifyoucouldchangeone thing in thisorganization thatwould improveyoursystemsengineering capabilities, what would it be? [Note: time/money/power are not anissue.]

OrganizationallyFocusedORG01 WhatisSystemsEngineering?ORG02 Whatdoestheorganizationvalueaboutsystemsengineering?ORG03 Wheredoessystemsengineeringfitintheorganizationalstructure?ORG04 How does the organization expect systems engineers to be integrated into its

engineeringteams?ORG05 Whatisthemanagementstyleintheorganization?HowistheSEworkforceincludedin

themanagementprocess?ORG06 What practices does the organization currently implement to minimize systems

engineeringturnover?(i.e.retentionincentives,careerdevelopmentoptions)ORG07 Is there a gap between the skills of your systems engineering workforce and your

organizationalneed?(Followup:Whatareyoudoingtofillthatgap?)ORG08 Ifyoucouldchangeone thing in thisorganization thatwould improveyoursystems

engineering capabilities, what would it be? [Note: time/money/power are not anissue.]


55

APPENDIXC:CULTURALCHARACTERISTICS–DEFINITIONSANDRELATEDQUESTIONS

1.StatusaffordedSystemsEngineers(High,Low,InTransition)• PerceivedbySEs• EspousedasvaluedbyleadershipinAnnualReportsandcompanydocs• Formalavailabilityandactualuseofsystemsengineeringrewards• Examplesofinformalrecognition• KeyleadershiprolesfilledbySystemsEngineers

2.Structure(SEFunctionallyCentralizedvs.Distributed–Matrix,Function,Project)• Evidenceofmanagingthepolaritiesofbothfunctionalanddistributedstructures:

strongfunctionalsupport(training,mentoring,careerladders)andstrongcross-functionalcollaboration(integrationinprojectteams,cross-businesscontributions)

StrongMatrixOrganizationStructure• Instrongmatrixorganizations,mostofthepowerandauthorityisheldbyproject

managers.Projectmanagershaveafulltimerole,haveafulltimeprojectmanagementadministrativestaffunderthemandcontroltheprojectbudget.Thestrongmatrixstructurehasalotofthecharacteristicsofa“projectized”organization.

• ThefunctionalmanagerwillhaveaverylimitedrolewithintheStrongMatrixOrganization.

BalancedMatrixOrganizationStructure• Inbalancedmatrixorganizations,powerandauthorityaresharedbetweenthe

functionalmanagersandtheprojectmanagers.Althoughprojectmanagershaveafulltimerole,theyhaveaparttimeorotherwiselimitedprojectmanagementadministrativestaffunderthem.Inthistypeofstructure,bothmanagerscontroltheprojectbudget.

WeakMatrixOrganizationStructure• Inweakmatrixorganizations,projectmanagerswillhavelimitedpowerandauthority.

Theywillhaveaparttimeroleandnoadministrativestaffwillreporttothem.Theirrolewillbemorelikeacoordinatororanexpediter.Here,thefunctionalmanagercontrolstheprojectbudget.

• Aweakmatrixorganizationstructureresemblesthecharacteristicsofafunctionalorganizationstructure.

3.Professionalism(High,Low,InTransition)• Self-Regard:self-reportedprideinfunctionalexpertiseandcontributions,participationin

industrySEgroups• Other-Regard:encouragedtosubmitpapersandparticipateinindustrygroupsby

leadership,budgetforprofessionalactivities• Other-Regard:Viewedbynon-systemsengineeringcolleaguesasactinginintegritywith

systemsengineeringvaluesandethics

4.RoleFormality(High,Low)


56

• SEsdoprescribedjobapplyingprescribedskillsvs.SEsfulfillmultipleroles,someofwhicharenotrelatedtosystemsengineering

• Degreeofspecificityofroles• Degreeofsharedunderstandingorclarityaboutroles• TypesofSEtraining,rotations,mentorships,apprenticeships,ladder,titles,hiring

5.Influence(High,Low)

• Reach(narrow–broad)• Withinteam,cross-functional,crosslevel,crosscompany

• DecisionOwnership• Whichdecisionsdotheyown?Whichdotheycontributeto?Fromwhichare

theyexcluded?

6.Collaboration(High,Low:Strength,Breadth)• Teamparticipation

• percentoftime?co-location/distancecollaboration:time/supportfor• actasteamleaders,membersor“SMEconsultants”(structure?)

• Typesofcross-functionaldailywork• Typesofjointlearning,e.g.,participationinafter-actionreviews• Diversityofteammembership(functions,levels,customerinvolvement…)• Self-describedandcompany-espousedvaluesre:teamwork,jointproblemsolvingand

conflictresolution,groupcreativityandinnovation7.Change• Whatformalinitiativesareongoing?

• Whosponsorsthem?Whoparticipates?• RolesofSEsandSELeadersinthechange?• LinkstocorporateorSEstrategies?• Whatorganization,function,cultureorprocesschangeinitiativesdoSEssponsoror

lead?• Informalchange

• DoSEsandNon-SEsviewSEsaschampionsforchange(possibleareas:bestpractices,teaming,communication,technicalinnovation…)

rt-173: helix 2017 helix technical report · participation in the helix interviews provided us data...

Documents