lessons from space shuttle disasters for avoiding it ... · lessons from space shuttle disasters...

LessonsFromSpaceShuttleDisastersFor

AvoidingITProjectDisasters

JohnHelm

Executive Timeslice, LLCwww.ExecutiveTimeslice.com

Jhelm Associates, LLCwww.JHelmAssociates.com

http://jhelmassociates.com/resources.html?item=spaceProjCopyright(c)2013byJohnHelm.ThisworkismadeavailableunderthetermsoftheCreativeCommonsAttribution-ShareAlike 3.0Unsupportedlicense.

ApreviousversionofthistalkwaspresentedatAgile2013,Nashville,TN,5-9August2013

Motivation

Atleast45yearsofITProjectHell

• Significantimprovementsstubbornlyelusive– SoftwareEngineering:AReportonaconferencesponsoredbytheNATOScienceCommittee.Garmisch,Germany.07-11Oct1968.PeterNaur andBrianRandell,Eds.

• RiskAdjustedPerformanceshows– Junkbondsoftenbetterinvestments– See:“JunkBondsVersusITProjects”*

1*http://jhelmassociates.com/resources.html?item=junkProj

FourUserStories(3StoriesAndOneSmallEpic,really:^)

• AsanAgileProfessional,Iwantto1. ApplyfindingsfromDecisionSciencetomyworksoIcanbe

moreeffectiveo Note:PresentHolisticCriticalDecisionMakingFramework

2. Avoidadeathof10,000smellsbecausedyingthatwaysuckso Note:DoChallengercasestudyo Note:Setstageforthinkingtools

3. Learnaboutnewtoolstomakebetterexecutiondecisionsandhavebetterfact-basedconversationswithmystakeholderso Note:Introduceunpredictabilitymeasurementandthinkingtools

4. SeeAgilebedeployedmoresuccessfullyo Note:DoColumbiacasestudy

2

WhyThinkAboutThinking?

3

WhyThinkAboutThinking?

4

Themostdangerous

decisiontrapsaretheonesweareunawareof!

ArtandScienceofDecisionMakingIndividualDecisionMaking

5

Cognitiverulesofthumbusedsubconsciously

Subconscioustendencytothinkinacertainway

Mentalprocessesthatmake

thingsworse

ArtandScienceofDecisionMakingGroupDecisionMaking

6

ArtandScienceofDecisionMakingOrganizationalModulation

7

ArtandScienceofDecisionMakingHeuristics

8

ArtandScienceofDecisionMakingCognitiveBiases

9

ArtandScienceofDecisionMakingCognitiveBiases:ConfirmationBias

10

Photo:BillCunningham,CapricornControl

TheTRUTHIsOutThere…

NoomiRapace,SwedishActress

AllSwedesAreBlond…

ArtandScienceofDecisionMakingCognitiveBiases:ConfirmationBias

• ConfirmationBiasisbuiltintous

• Quantitativestudiesarenoprotection

• Example:inferenceofacausalrelationshipfromcorrelateddata

Read:WhyMostPublishedResearchFindingsAreFalse

11

ByJohnIoannidishttp://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.0020124

ArtandScienceofDecisionMakingCognitiveBiases:FramingandProspectTheory

12

Yoursurgeryhasan75%chancetosaveyourlife

HalfEmpty

HalfFull

Yoursurgeryhasa25%chanceitwillkillyou

FrameintermsofLOSS• Easiertomotivate• Choicesconstrainedtofamiliar

FrameintermsofOPPORTUNITY• Hardertomotivate• Moreflexibility&innovation

ArtandScienceofDecisionMakingHyperbolicDiscounting

13

• Preferenceforsmallpleasurestodaythataredetrimentaltoourfuture

– Skipfullcoveragetestingtoday,2dayoutagenextyear?

– $50todayv.$100tomorrow?– $50todayv.$1006months?– $50todayv.$1001year?

• Ourbrainsarewiredtorespondtofutureuncertaintybydiscountingthefuture– Coke&friestoday,bypasssurgeryin10yrs?– 500,000bypasssurgeries/year

– Only10%undotheirunhealthylifestyle

• Codingwhilebehindscheduleforrelease…– Cutandpastetoday,spaghetticodenextyear?

ArtandScienceofDecisionMakingHyperbolicDiscounting

14

• Discounting is the assessment of present value of a future gain– At 4% inflation, $100 in 10 years should be worth $67.55 today

• The 4% is called the discount rate; 0.6755 is the discount factor– According to the rational approach

Each period is discounted the same– Theresultisanexponentialdiscount

function

• ButpeopleDON’Tdiscounteachperiodthesame!– Thenearfutureisvaluedmuchmore

relativeto“far” future– Theresultingdiscountfunctionis

calledHYPERBOLIC(morecorrectly,it’squasi-hyperbolic)

ArtandScienceofDecisionMakingHyperbolicDiscounting

15

• Discount function graphs are unhelpfully abstract• Whatmattersforpeopleisthe

anticipationofsatisfaction!• Moreinterestingisthetrade-off

betweenasmallrewardsoonvs.alargerrewardfartherinthefuture

Delaydependentpreferencereversal!

MatthewG.Miller,RayJ.Dawson,KieranB.Miller,MalcolmBradley(2008).NewInsightsintoITProjectFailure&HowtoAvoidIt.Presentedat22ndIPMAWorldCongress- Rome(Italy)November9-11,2008,inStream6.AsofMay2013,selfpublishedathttp://www.mgmiller.co.uk/files/paper.pdf

The measured success rate (10% allowances) compared to how the project managers perceived their projects

16

ArtandScienceofDecisionMakingOverconfidence/AttributionBias

41.84%

79.32%

ArtandScienceofDecisionMakingThinkingTraps

17

ArtandScienceofDecisionMakingFunctionalFixedness

18

The Candle ProblemKarl Dunker, 1945Sam Glucksberg, 1962

The task: – Fix a lit candle to the

wall such that no wax hits the table using only

– Book of matches– A box of thumbtacks– A candle (of course)

How long to do?– Box empty– Tacks in box– No Pressure– Under Pressure

ArtandScienceofDecisionMakingFunctionalFixedness

19

• Financialincentives– Improvedperformanceofsimpledoingtask– HURT performanceofcreativethinkingtask

– Moreincentive➜ worsecreativethinkingresults

MeanTimeToSolvePuzzle

TacksOutofBox(Easy) TacksInsideBox(Hard)

Low PerformancePressure 4.99min 7.41min

HighPerformance Pressure 3.67min 11.08 min

ArtandScienceofDecisionMakingCognitiveDissonance

• Cognitivedissonance:– Unpleasantemotionsarisingfromsimultaneousmutuallyexclusivebeliefs

– Rationalizationsdevelopedtodefendbeliefwewishtohold• Aronsonmodel:RoleofEgoistoprotectidentity,whateverittakes

• Canmotivatemanyirrationaldecisionsandbehaviors– Denial– IllogicalRationalization(specialpleading)– EscalationofCommitment

20

ArtandScienceofDecisionMakingCognitiveDissonance

21

GreatFloodScheduledDec21,1954

ArtandScienceofDecisionMakingCognitiveDissonance

22

Believers,giveitallaway!YouwillbesavedbyUFO

GreatFloodScheduledDec21,1954

ArtandScienceofDecisionMakingCognitiveDissonance

23

Believers:giveitallaway!YouwillbesavedbyaUFO

GreatFloodScheduledDec21,1954

4:45am“PressRelease”

“Thelittlegroup,sittingallnightlong,hadspreadsomuchlightthatGodhadsavedtheworldfromdestruction.”

Theinconsideratebuggersfailedtoshowup!!!

4:00amCataclysmDay

ArtandScienceofDecisionMakingGroupDecisionMaking

24

ArtandScienceofDecisionMakingGroupDecisionMaking

• OftenmanyheadsAREmuchbetterthanone– Groupspoolandtapdiversetalents– Surfaceandcorrectimperfectindividualdecisions

• “WhoWantsToBeaMillionaire?” audiencecorrect91%

• BUT,often“ProcessLosses”trashthemagic

25

DesiredCondition ProcessLoss

Diversityofskills andviews Everybodylikemindedor samebackground

Decentralization/Delegation Centrally directeddecisionmaking

Aggregation Can’tintegrateindividualcontributions

Safety/Independence Mightgetfiredforideas orspeakingup

ArtandScienceofDecisionMakingGroupDecisionMaking

26

ArtandScienceofDecisionMakingOrganizationalModulation

• Twodimensionsoforganizationexamination– StructuralPerspective(e.g.NormalAccidentTheory)– BehavioralPerspective(Many)

• DecisionmakingissuescanarisefromBOTH– Structuralfeatures

• Clumsyorganizationalrelationships• Toomanytimezones….

– Behavioraltendencies• Corporateidentity• Culture• Values

27

GroupdecisionsDONOT occurinavacuum

ArtandScienceofDecisionMakingOrganizationalModulation

• Structuralpropertiesoforganizationsinclude– ComponentCoupling(LooseorTight)– InteractiveComplexity(LinearorNon-linear)

• Organizationalcontextshapesbehavior– Identity:whoweare

• Constrainsorbiasessolutionsets

– ValuesandCulture:unspoken“rules”everyonemustknow• Constrainsanddirectsbehavior

28

ArtandScienceofDecisionMakingOrganizationalModulation

29

ArtandScienceofDecisionMakingDecisionsandProjects

30

ProjectManagementApproachResourceChoicesCommunicationProtocolsCompliance/Governance

ToolChoicesTechnologySelectionsArchitectureDeliveryPracticesCut&PasteorRefactor?

FourUserStories

• AsanAgileProfessional,Iwantto

2. Avoidadeathof10,000smellsbecausedyingthatwaysucks3. Learnnewtoolstomakebetterexecutiondecisionsandhave

betterfact-basedconversationswithmystakeholders4. SeeAgilebedeployedmoresuccessfully

31

DecisionsAndTheChallengerDisaster27YearsAgo

32

DecisionsAndTheChallengerDisasterThe"L-1Meeting"DecisionToLaunch

33

“Ifthedecisionmakershadknownallofthefacts,itishighlyunlikelythattheywouldhavedecidedtolaunch51-LonJan28,1986.”(Pg.82,RogersCommissionReport)

Strongevidencesuggestsmattersnotthissimple

TheRogersCommission:Communicationfailuresenabledflawedgroupdecision:

DecisionsAndTheChallengerDisasterProcessLossesintheL-1Meeting?

GroupthinkUnlikely

34

DesiredCondition ProcessLoss? Why?

Diversity ofviews No Variousinternal andexternalexperts

Decentralization No NASA staffandmany independentcontractors

Aggregation No Vigorousandopendebateofdifferingviews

Safety/Independence No Managersdidnot“pullrank”Decisionwas“rule-based”

DecisionsAndTheChallengerDisasterDidStress/FunctionalFixednessplayarole?

35

Discussionfocusedontwokeylaunches,circledinred

DecisionsAndTheChallengerDisasterDidStress/FunctionalFixednessplayarole?

36

• Dataofproblemsonly• Discussionfocusedon

– SRM-15– SRM-22

DecisionsAndTheChallengerDisasterDidStress/FunctionalFixednessplayarole?

• Dataall24Launches– Damagelaunchesinred

37

DecisionsAndTheChallengerDisasterNASA:StructuralConsiderations

• StructuralObservations– Complexmatrixorg(NASA-contractorecosystem)

• Geographicallydispersed

– Establishedhierarchicalbureaucracy

• Littleevidencethatstructurewasissue– Longandimpressivehistoryofsuperbexecution– Effectivemanagementofcomplexecosystemandsupplychain

– Impressivesafetyrecordfor“twomillionparts– allbuiltbythelowestbidderonagovernmentcontract”

38

DecisionsAndTheChallengerDisasterNASA'sOrganizationalIdentity,ValuesandCulture

39

Theycalledita

SHUTTLE

“Beginningwiththenextflight,theColumbiaanditssistershipswillbefullyoperational”– Reagan,after4th flight

“Itwillrevolutionizetransportationintonearspacebyroutinizingit”

– Nixon,1972

DecisionsAndTheChallengerDisasterNASA'sOrganizationalIdentity,ValuesandCulture

• PoliticsandFunding– WrongFrame➡ Structureandcultureofroutineoperations

• BehavioralObservations– Rigidrulesandprotocols(nolevelhopping)

• Poorinformationflow

– Obsessionwithscheduleanddeadlines– Starkdistinctionbetweenengineersandmanagers

• “Takeoffyourengineer'shatandputonyourmanager'shat”

– Penchantforextensivesupportingdata– Insufficientacknowledgementoftheunknowns

40

CriticalThinkingandDecisionMakingNormalizationofDeviance

• DianeVaughnstudiedNASA’sorganizationalcontextandhistory– ObservedO-Ringproblemswerenotnew

• Issuescroppedupovertheyears• Historyandcontextmust haveinfluencedthelaunchdecision

• Vaughnarguedthelaunchdecisionisbestunderstoodinhistoricalcontext– O-ringerosionunexpected– Happenedonce,nodisaster– O-ringerosionbegantooccurregularly– Rationalizedredundancywassufficient– Graduallytheunexpectedbecame,expected,thenaccepted

41

CriticalThinkingandDecisionMakingNormalizationofDeviance

Vaughnproposedaslowholisticprocessatwork1. Smalldeviationsfromstandardsornormsarerationalized,

oftenunder(perceived)coercivepressure2. Nothingbadhappens,supportingthecorrectnessofthe

rationalization3. Withrepetitionsthe“deviation”eventuallybecomesthe

newnorm

NORMALIZATIONOFDEVIANCE

Notincompetence,justhumanity!SeeAlso:PracticalDrift; read“FriendlyFire”,Snook

42

FourUserStories

• AsanAgileProfessional,Iwantto

3. Learnnewtoolstomakebetterexecutiondecisionsandhavebetterfact-basedconversationswithmystakeholders

4. SeeAgilebedeployedmoresuccessfully

43

QuantitativeUnpredictabilityManagement

AmIdoomedifIcan’tpronounceCynefin?

44

QuantitativeUnpredictabilityManagement

45

Deterministic

Nondeterministic

SystemtobeManaged

ApproximateModel

QuantitativeUnpredictabilityManagement

• NormalizationofDeviance➡ 10,000smells– Everyriskierbehavior– Increasinglypoorjudgment– Escalatingvulnerabilityto“BlackSwans”

• LikeO-ringleaks,canseethemdeveloping

• UseEstimationErrorDistributions• Lookforlongtails

46

QuantitativeUnpredictabilityManagement

• Youcan’tmanagewhatyoudon’tmeasure• Track

47

RelativeEstimationError= Est − ActualEst

DecisionsandProjectsMeasureUnpredictability

48

ALongTail• Gaussiandatashouldhave1

or2cases(0.35%)• Actuallyhave40cases(8.6%)

DecisionsandProjectsMeasureUnpredictability

Uses:SensemakingwithCynefin

49

Youarehere!ALongTail

• Gaussiandatashouldhave1or2cases(0.35%)

• Actuallyhave40cases(8.6%)

DecisionsandProjectsMeasureUnpredictability

Longtailsdeprivesystemsofconsistent predictabilityWaterfallrequiresLOTSpredictability

50

-400 -300 -200 -100 0 100 0

20

40

60

80

100

120

Relative Estimation Error [%]

Estimation Accuracy “Distribution”

Not es: 465 User Stories; Single Scrum Team; 39 sprin ts in 2 yrs

DecisionsandProjectsMeasureUnpredictability

Uses1. SensemakingwithCynefin2. Conversationswithbusiness3. Seeing

– Deteriorationincodebase/system– Effectofchanges

51

DecisionsandProjectsSourceofMuri andMura?

• BlackSwanunderestimation– CreatesMuri andMura– Underminestrustbuilding

• Escalatingstressandfatiguedefinenegativefeedback

52

DecisionsandProjectsProtectiveToolsAndTechniques• BiasGuards

– Becomeawareandunderstandbiasesexist

• Trytolearnyours– SWOT-styledecisionanalysis– Managestress

• GroupEffectiveness– Understandandapplyframing

• Leadersmustframeespeciallycarefully• Usemultipleframes

– Monitorgroupdynamicsforprocesslosses• Watchforgroupthink• Stimulateconstructivedebate(scenarios/pre-postmortems)

– Includedecisionreviewsinretrospectives

– Fixmembershipproblemspromptly!

53

DecisionsandProjectsProtectiveToolsAndTechniques

• NormalizationofDeviance

UseTHEFORCE

• CognitiveDissonance– EngageaTruthsayer– Focuson(attachidentityto)process,notoutcome– Keepwrittenrecords– Usehonestmetrics

• UnderstandtheCultureCode– Seetheorganizationthatis– Ifactions≠words,ignorewords

54

DecisionsandProjectsDecisionAccounting/ChecksandBalances

• Setupchecksandbalances– Don'tonlyselfassess

• Inviteanoutsideauditor

• Useseparategroupsto– Approveprojects– Monitororcancelprojects

55

FourUserStories

• AsanAgileProfessional,Iwantto

4. SeeAgilebedeployedmoresuccessfully

56

Take-Aways andOpportunitiesTheColumbiaDisaster

57

• Observations– NASAnever interviewedVaughn– Evidenceof

• Confirmationbias• Processlossesindebrismeetings

• TheColumbiadisasterinvestigationboard:TheFoamDidIt,ButTheOrganizationLetItHappen

• NASA– FailedtolearnfromtheChallengerexperience– Failedtomeaningfullychangeitsbehavior

58

Take-Aways andOpportunitiesTheColumbiaDisaster

Take-Aways andOpportunitiesBewaretheHypecycle!

• ProjectManagementremainsunsatisfactory– Stubbornlysosince1968NATOConference

• Agileishelping!

• Havewe“CrossedtheChasm”?

• Withincreasingadoptioncomesincreasingrisk

• Failedadoptiondamages“thebrand”

• Embracecriticalthinkingandcriticaldecisioning

59

Take-Aways andOpportunitiesLet'sUseGandhi'sFormula

“Bethechangeyouwishtoseeintheworld”Gandhi

Letthatchangebeenrichedbycriticaldecisionmaking

60

Let'sGetStarted

61

• WhySmartPeopleMakeBigMoneyMistakesandHowtoCorrectThem,Belsky andGilovich

• PredictablyIrrational,DanAriely

• FriendlyFire:TheAccidentalShootdownofU.S.BlackHawksoverNorthernIraq,ScottSnook

Let'sGetStarted• MistakesWereMade(ButNotbyMe),Tavris andAronson

• SnakesinSuits:WhenPsychopathsGotoWork,Babiak andHare

62

• TheWisdomofPsychopaths:WhatSaints,Spies,andSerialKillersCanTeachUsAboutSuccess,Dutton

Namaste

QuestionsandDiscussion

63

AppendixandExtrasThiokolRecordofO-RingErosionandBlowby

64

AppendixandExtrasThiokolRecordofO-RingErosionandBlowby

65

AppendixandExtrasBoxplotRangesOverTheGaussianDistribution

66

ENDOFDECK

67