thinking ahead to system verification and system validation · verified and validated. at this...
TRANSCRIPT
Copyright©2016byRequirementsExperts. 1
Thinking Ahead to System Verification and System Validation LouisS.WheatcraftRequirementExperts
(281)[email protected]
Note:[Feb2016]Thisisanupdatetothispaperwhichwasoriginallypublishedin2012.Theupdatesarearesultofamaturingoftheauthor’sthoughtsonthesubjectandaclearerunderstandingofwhatismeantwhenusingtheterms“verification”and“validation”.ThemajorchangeistheadditionofamoredetaileddefinitionofthetermsandtheadditionofFigures1&2.Mostoftherestofthechangesweretomaketherestofthetextconsistentwiththeproposeddefinitionstomakeclearwhatthecontextiswhenusingtheseterms.
AbstractSecondonlytore-work,aproject’slargestcostsareinvolvedinsystemverificationandsystemvalidation.Failingtoadequatelyplanforsystemverificationandsystemvalidationfromthebeginningoftheprojectplacestheprojectathighriskforcostoverrunsandscheduleslips.Tomitigatetheserisksprojectteamsmustoutlinetheirconceptsforsystemverificationandsystemvalidationduringthescopedefinitionactivities;addressingthefollowingquestions:
• “Whowillverifythesystemmeetstherequirementsthatdrovethedesign,andwhereandwhenwilltheseactivitiesbedone?”
• “Whowillvalidatethatthedesigned,builtorcoded,systemwillmeetitsintendedpurposeinitsoperationalenvironment,andwhereandwhenwilltheseactiviiesbedone?”
• “Whatfacilities,supportequipment,simulators,emulators,ormodelsarerequiredforsystemverificationandsystemvalidation?”
• “Whataretheinterfacesinvolvedduringsystemverificationandsystemvalidation?”Whenyoustartwritingyourrequirements,addressingsystemverificationastherequirementsarewritteninsuresabettersetofrequirementsandcanreducethecostofsystemverificationandsystemvalidation.Aseachrequirementiswritten,thefollowingquestionsneedtobeaddressed:
• “Isthisrequirementverifiable,.i.e.,istherequirementwrittensuchthatitsintentisclearandasystemverificationactivitycanbeperformedthatwillprovethesystemmeetstherequirement?”
• “Whatwillconstituteproofthatthesystemhasbeenverifiedtomeettherequirement?”
• “Whatprimarymethodwillbeusedtoverifythesystemmeetsthisrequirement?”
Failingtoaddressthesequestionsearlycanresultinunpleasantsurprisesduringyoursystemverificationandsystemvalidationactivities,puttingtheprojectatriskofrework,scheduleslips,andbudgetoverruns.Theemphasisofthispaperistopresentbestpractices,tools,andprovenmethodsprojectteamscanusetoplanforsystemverificationandsystemvalidationfromthebeginningoftheirprojectandthusavoidtherisksofnotdoingso.
Copyright©2016byRequirementsExperts. 2
Verification and Validation
AquestionIhearoftenis“Whatisdifferencebetweenverificationandvalidation?”
Ingeneral,verificationreferstothebasics(structure)oftheitembeingverified,makingsureitmeetsrequirementsthatdrivethecreationoftheitem,whetheritberulesonwritingwell-formedrequirements,standardsandbestpractices(externalandinternal)onthedesign,orrequirementsonthesystem.Thenvalidationgoesbeyondthebasics(structure)tohowwelltheitem(requirements,design,system)communicatesoraddressesstakeholderneedsandexpectations.
Myuseofthewordsysteminthispaperreferstothesystemofinterest,nomatterwhichlevelthesystemofinterestmayexistwithinthearchitecture(system,subsystem,assembly,component).Requirementsaredevelopedduringtheconceptphaseandrepresentaconceptualviewofthesystemofinterest.Bydesign,Imeanthetransformationoftheconceptualviewofthesystemascommunicatedbythebaselinerequirementsetintoaphysicalrealizationofthesystem.Thisprocessevolvesfromapreliminarydesign,toafinaldesign,andthentodevelopment,whichtransformsthedesignintothephysicalsystem(hardware)orcode(software).
Figure1.Verificationandvalidationaretheprocessesofconfirmingthatartifactsgeneratedduringthetransformationprocessesareacceptable.
Whilethetermsverificationandvalidationarewidelyused,Ifindthatthetruemeaningoftheconceptsthesetermsrepresentareoftenmisunderstoodandthewordsoftenusedinterchangeablywithoutmakingclearthecontextinwhichtheyareusedresultinginambiguity.Withasingleorganization,whenengineersareaskedtodefinethetermsyouwillgetdifferent,oftenconflicting,definitions.Toavoidthisambiguity,eachtermneedstobeprecededbya
Stakeholderneedsandexpectations
TransformationRequirementDevelopment
TransformationDesign Design
OrganizationalRequirement
WritingGuidelines
TransformationBuild,Code,Buy,orReuse
System
VerificationVerificationVerification
Validation
Validation
Validation“Arewebuildingtherightthing?”
“Dowehavetherightdesign?”
OrganizationalDesign
GuidelinesVerification
“Didwebuildtherightthing?”
“Didwebuilditright?”
“Didwedesignitright?”
“Aretherequirementswrittencorrectly?”
“Didwedesignitcorrectly?”
OrganizationalCode,BuildGuidelines“Didwebuildit
correctly?”
Verification
Requirements
Copyright©2016byRequirementsExperts. 3
modifier(i.e.,thesubject)whichclearlydenotesthepropercontextinwhichthetermisbeingused,specificallyrequirementverificationorrequirementvalidation;designverificationordesignvalidation;systemverificationorsystemvalidationasshowninFigure1.
Arequirementistheresultofaformaltransformationofoneormorestakeholderneedsandexpectations in to the requirement statement. Correspondingly, design is a result of formaltransformation of requirements in to an agree-to design, and a system is a formaltransformationofadesignintothatsystemasshowninFigure1.
AsillustratedinFigure1,theprocessofcreatingarequirementinvolves:• analysingoneormoreselectedneedsandexpectationstoobtainthenecessaryelements
tobeincludedintherequirement;• selectingaformatfortherequirementstatement;• and identifying the characteristics of the desired result against the organizational
guidelinesandrulesbywhichtherequirementstatementistobewritten.
Inthiscontext,RequirementVerificationconfirms,byinspection,thattherequirementcontainsthe necessary elements and possesses the characteristics of awell formed requirement andconforms to the rules set forth in the organization’s requirement development guidelines.RequirementValidation confirms, by inspection and analysis, that the resulting requirementmeets the intent of the stakeholder need from which the requirement is derived. Thus arequirement statement (and sets of requirements) is/are confirmed by both verification andvalidation.
Basedonthisdiscussion,tohelpremovetheambiguityintheuseoftheterms“verification”and“validation”,thefollowingdefinitionsforrequirementverificationandrequirementvalidationareproposedintermsofthecontextofrequirements.
• RequirementVerification:theprocessofensuringtherequirementmeetstherulesandcharacteristicsdefinedforwritingagoodrequirement.Thefocusisonthewordingandstructure of the requirement. “Is the requirement worded or structured correctly inaccordancewiththeorganization’sstandards,processes,andchecklists?”
• RequirementValidation: confirmation the requirement is an agreed-to transformationthat clearly communicates the stakeholder needs and expectations in a languageunderstood by the developers. The focus is on the message the requirement iscommunicating. “Does the requirement clearly and correctly communicate thestakeholder expectations and needs?” “Are we doing the right things?” or “Are webuildingtherightthing?”
RequirementverificationandrequirementvalidationactivitiesshouldbedonecontinuouslyasyoudeveloptherequirementsaswellasdoneaspartofbaselineoftherequirementsetactivitiesperformedduringtheSystemRequirementsReview(SRR)orsimilartypeofgatereview.
Copyright©2016byRequirementsExperts. 4
Note: Most organizations do not make a distinction between requirement verification vsrequirementvalidation.Rathertheyuseonlythephrase“requirementvalidation”tomeanboth.Using the phrase “requirement verification” often confuses the conversation in that manyinterpret“requirementverification”tohavethesamemeaningas“systemverification”.
Figure1alsocarriestheseconceptsforwardtodesignandrealizationofthesystemunderdevelopment.
Onceyouhaveasetofrequirementsbaselined,therequirementsaretransformedintoadesignofthesystem.Mostorganizationshaveasetofguidelinesor“goldenrules”thatguidethedesignprocess.Theseguidelinesrepresentbestpracticesandlessonslearnedthedesignteamisexpectedtofollow.Aspartofthedesignprocess,thedesignteammaydevelopprototypesorengineeringunits.Theywillusethesetorunteststofinetunetheirdesign.
Afterthedesigniscompleteforthesystem,thereisusuallyagatereviewwherethedesignisbothverifiedandvalidated(e.g.SystemDesignReview(SDR),PreliminaryDesignReview(PDR),CriticalDesignReview(CDR).Inthiscontext,designverificationhastwoaspects:1)Doesthedesignclearlyrepresenttherequirementsthatdrovethedesign?and2)Didthedesignteamfollowtheorganization’sguidelinesfordesign?Alsoaspartofthegatereviewdesignvalidationisaddressedtodeterminewhetherornottheresultingdesignforthesystem,whenimplemented,willresultintheintendedpurposebeingmetintheoperationalenvironmentandthestakeholder’sexpectationsandneedsbeingmet.
Frequently,duringthesegatereviews,thedesignteam“pushesback”onsomeoftherequirementsthatproveddifficulttomeetorweredeemednotfeasible(cost,schedule,technology),andproposeschangestotherequirements.Whenthishappens,notonlydotherequirementsneededtobechanged,butalsothestakeholderexpectationsandneedsfromwhichtherequirementwasderivedmayneedtobeexamined,whichcouldresultinascopechange.
Designverificationanddesignvalidationactivitiesshouldbedoneaspartofacontinuousprocessduringthedesignphaseaswellasduringthethebase-liningofthedesigninthegatereview(s).
Basedonthisdiscussion,tohelpremovetheambiguityintheuseoftheterms“verification”and“validation”,thefollowingdefinitionsfordesignverificationanddesignvalidationareproposed.
• DesignVerification:theprocessofensuringthedesignmeetstherulesandcharacteristicsdefinedfortheorganization’sbestpracticesassociatedwithdesign.Thefocusisonthedesign process. “Did we follow our organizations guidelines for doing the designcorrectly?”Thedesignprocessalso includesensuring thedesignreflects thedesign-torequirements.Thus,designverificationisalsoaconfirmationthedesignisanagreed-totransformationofthedesign-torequirementsintoadesignthatclearlyimplementsthose
Copyright©2016byRequirementsExperts. 5
requirements correctly. “Does the design clearly and correctly represent therequirements?”“Didwedesignthethingright?”
• DesignValidation:confirmationthedesignwillresultinasystemthatmeetsitsintendedpurposeinitsoperationalenvironment.Willthedesignresultinasystemthatwillmeetthestakeholderexpectations(needs)thatweredefinedduringthescopedefinitionphasethatoccurredatthebeginningoftheproject?Thefocusisonthemessagethedesigniscommunicating.“Howwelldoesthedesignmeettheintentoftherequirements?”“Dowehavetherightdesign?”“Arewedoingtherightthings?”“Willthisdesignresultinthestakeholderexpectationsandneedsbeingmet?”
Oncethedesignisbaselined,thedesignistransformed;viabuild,code,buy,orreuse;intothesystemofinterest.Similartothediscussionforthedesignprocess,mostorganizationshaveasetofguidelinesor“goldenrules”thatguidethebuild(manufactureorcode)process.Theseincludeworkmanshipandqualitycontrolrequirementsontheorganization.
Afterthesystemhasbeenbuiltorcoded,therewillbeagatereviewwherethesystemisbothverifiedandvalidated.Atthisstageofthesystemlifecycle,theconceptsofsystemverificationandsystemvalidationtakeonamoreformalmeaning.Thus,theSystemsEngineering(SE)lifecycleprocessesincludetheprocessesofSystemVerificationandSystemValidation.Eachoftheserepresentasetofactivities(test,demonstration,inspection,analysis)thatcumulatewithoneormoregatereviewsassociatedwiththeacceptanceofthesystembythecustomer.ThusSystemVerificationisaformalSEprocessandhasalegalaspectwherethedeveloperisprovingthebaselinedrequirements,whichareatypeofcontractandarelegallybinding,havebeenmeet.
Inthiscontext,systemverificationhastwoaspects:1)Doesthebuiltorcodedsystemofinterestclearlyrepresenttherequirementsthatdrovethedesign?Didwebuildtherightthing?and2)Didthebuildorcodeteamfollowtheorganizationsguidelinesformanufacturingandcoding?
Followingsystemverification,systemvalidationisperformed.Again,SystemValidationisaformalSEprocessandhasalegalaspectwherethedeveloperisprovingwhetherornotthebuiltorcodedandverifiedsystem,resultsintheintendedpurposebeingmetintheoperationalenvironmentandthestakeholder’sexpectationsandneedsbeingmet.Likethesystemrequirements,thebaselinedstakeholderneedsandrequirementsdefinedduringthescopedefinitionphasecanalsobeconsideredpartofacontractandarelegallybinding.
Basedonthisdiscussion,tohelpremovetheambiguityintheuseoftheterms“verification”and“validation”,thefollowingdefinitionsforsystemverificationandsystemvalidationareproposed.ThesedefinitionsarealsoillustratedinFigure1.
• SystemVerification:SystemVerificationisdoneafterdesignandbuildorcoding,makingsurethedesignedandbuiltorcodedsystemmeetsitsrequirements.Thefocusisonthebuiltorcodedsystemandhowwellitmeetstheagreed-torequirementsetthatdrove
Copyright©2016byRequirementsExperts. 6
the design and fabrication. Methods used for system verification include: test,demonstration,inspection,oranalysis.“Didwebuildthethingright?”Alsoincludedinsystemverificationisadeterminationthattheteamresponsibleforbuildingorcodingthesystem of interests followed the organization’s rules, guidelines, and best practicesassociatedwithmanufacturingandcoding.Thefocusisonthemanufacturingorcodingprocesses. “Did we follow our organizations guidelines for manufacturing or codingcorrectly?”
• SystemValidation:Systemvalidationoccursaftersystemverificationandmakessurethedesigned, built, and verified system meets its intended purpose in its operationalenvironment.Thefocusisonthecompletedsystemandhowwellitmeetsstakeholderexpectations (needs) thatweredefinedduring the scopedefinitionphase that shouldhaveoccurredatthebeginningoftheproject.“Didwebuildtherightthing?”
Systemverificationandsystemvalidationprocessesandactivitiesaredirectlyrelatedtothecontractualobligationconceptforarequirementstatementandsetofrequirements.Itisthroughtheseactivitiesthatweprovewehavemetboththeagreed-torequirementsandtheagreed-toneedsoftheentitieswhoarethesourceoforownthem.Thisisoftenaccomplishedaspartofcertificationandacceptanceactivities.
VerificationandvalidationandtheSystemsEngineering“V”model
Whiletheprevioussectionpresenteddefinitionsthatareusefulinhelpingaddresstheissuesofambiguityintheuseofthetermsverificationandvalidation,thesystemengineeringprocessismorecomplex.SystemsEngineeringisaniterativeandrecursiveprocess.Requirementsdevelopmentanddesignoccurtop-downasshownontheleftsideoftheSE“V”asshowninFigure2.
Figure2:SystemsEngineer“V”Model
Operation andSupport
Concept Development Production
Development
StakeholderNeeds
Verifies
SystemRequirements)
StakeholderRequirements
SubsystemRequirements
Unit/componentRequirements
SystemIntegrationVerification
SubsystemIntegrationVerification
Unit/componentVerification
Verifies
Verifies
Validates PRODUCT
ProductLifecycle
Start
Operations
Engineering“Vee”Model
SystemsLifeCycleModel
Design
Disposal
VerificationMethods:- Test- Demonstration- Inspection/Observation- Analysis(includesmodels,
simulations, similarity)
Acceptance/OperationalEvaluation andValidation
Copyright©2016byRequirementsExperts. 7
AsshowninFigure2,Systemsengineeringstartswiththeconceptstagewherestakeholderneeds,expectations,andrequirementsareelicited,documented,andbaselined.Thisispartofdefiningthescopeofthesystemtobedeveloped.Thenthestakeholderneeds,expectationsandrequirementsaretransformedintoasetofsystemrequirementswhicharebaselinedviarequirementsverificationandrequirementsvalidationasdiscussedabove.Oncethesystemrequirementsarebaselined,designresultsinasystemarchitectureinwhichthesubsystemsaredefined.Thisdesignisbaselinedviathedesignverificationanddesignvalidationprocessdiscussedabove.
Foreachsubsystem,theabovecycleofstakeholdersubsystemneeds,expectations,andstakeholderrequirementsaredefinedandtransformedintosubsystemrequirements,whicharebaselinedviarequirementsverificationandrequirementsvalidationasdiscussedabove.Oncethesubsystemrequirementsarebaselined,designresultsinasubsystemarchitectureinwhichtheunits/componentsaredefined.Thisdesignisbaselinedviathedesignverificationanddesignvalidationprocessdiscussedabove.Thisprocesscontinuesuntiltheorganizationmakesabuy,build,code,orreusedecision.
Systemverificationandsystemvalidationoccurbottom-upasshownontherightsideoftheSE“V”asshowninFigure2.
Onceallthecomponentsthatmakeupthesubsystemsaredeveloped,unit/componentverificationandunit/componentvalidationtakeplaceasdescribedabove.Oncetheseactivitiesarecomplete,theunits/componentsareintegratedtogetherandthentheresultingsubsystemsareverifiedandvalidatedasdescribedabove.Oncethesubsystemverificationandsubsystemvalidationactivitiesarecompletethesubsystemsareintegratedtogetherandthensystemverificationactivitiesarecompleted.
Followingsystemverificationactivities,systemvalidationactivitiesareperformed.Thiscouldbedoneintheformofacceptanceand/oroperationevaluationandvalidationactivities.Intheend,proofwillbedocumentedthatcanbeevaluatedbythecustomertodeterminesystemvalidationactivitieshavebeencompletedsuccessfully,stakeholderneedshavebeenmet,andthesystemwilloperateasintendedinitsoperationalenvironment.
Followingthecustomerevaluation,thesystemcanbeacceptedandownershiptransferredtothecustomer.
Thefocusoftherestofthispaperisonsystemverificationandsystemvalidationplanning–addresstherightsideoftheSystemsEngineering“V”.(Formoreinsightintorequirementverificationandrequirementvalidation,seereferences1&2.)
A Winning Product vs. Risk
Attheendofaproject,allprojectteamswanttobeabletosaytheyhavebuilttherightthingandthattheproductmeetsstakeholderexpectationswhileaccomplishingitsintendedpurposeinitsoperationalenvironment.Icallthisdeliveringawinningproduct.Awinningproductis
Copyright©2016byRequirementsExperts. 8
definedhereinas:“Aproductthatdeliverswhatisneeded,withinbudget,withinschedule,andwiththedesiredquality.”Thegoalofallprojectsshouldbetodeliverawinningproduct.
Asimpleandpracticaldefinitionofriskis:“Anythingthatcanpreventyoufromdeliveringawinningproduct!”Failingtoplanforsystemverificationandsystemvalidationfromthebeginningoftheprojectisamajorrisktotheprojectthatcanresultinmassivecostoverrunsandscheduleslipsaswellasaproductthatdoesnotmeetrequirementsandfailstomeetstakeholderexpectations.Giventheseimpacts,allprojectmanagersneedtomitigatethisriskfromthebeginningoftheirproject.Recognizingthisiscriticaltobeingabletodeliverawinningproduct.
Requirementsbestpracticesincludetheplanningforsystemverificationandsystemvalidationactivitiesfromthebeginningoftheprojectandcontinuingtoaddresssystemverificationandsystemvalidationactivitiesthroughouttheproductlifecycle.Toreducetherisktoyourprojectassociatedwithfailingtoaddressverificationandvalidationactivities,itiscriticalthatyoufollowthesebestpracticesbyincludingsystemverificationandsystemvalidationplanningfromthebeginningofyourprojectstartingwithscopedefinitionandcontinuingthroughouttherequirementdefinition,management,anddesignprocesses.
Thefollowingsectionsintroducethesebestpracticesaswellasdiscusssomeofthetoolsandactivitiesthatwillhelpyouplanforsystemverificationandsystemvalidation.
Impact of requirement defects on system verification and system validation cost and schedule Requirementsarethesingleelementthattiesalltheproductdevelopmentlifecycleprocessestogether.Defectiverequirementshaveadirectandsignificantimpactonyourproject’scostandschedulewhenperformingsystemverificationandsystemvalidationactivities.
AsshowninFigure3[Hooks2001],thecosttofixrequirementdefects(re-work)increasesexponentiallyastheproductlifecycleprogresses.Thischartillustratestheorderofmagnitudecostsoffindingandfixingdefectsasweprogressintheproductlifecycle.Whatthisfigureshowsisthatthecostoffindingandfixingrequirementsdefectsinthecodingphaseis10timesmoreexpensivethanfindingandfixingthemduringtherequirementsphase.FindingdefectsduringDevelopmentTest,15-40timesmore,AcceptanceTest,30-70timesmoreandso-forth.InthecontextofthelifecyclephasesdepictedinFigure3,systemverificationandsystemvalidationactivitiesoccurduringdevelopmentandacceptancetestingaswellasduringinitialoperations.
Copyright©2016byRequirementsExperts. 9
Figure 3: Cost to fix requirement defects
AsshowninFigure3[Hooks2001],thecosttofixrequirementdefects(re-work)increasesexponentiallyastheproductlifecycleprogresses.Thischartillustratestheorderofmagnitudecostsoffindingandfixingdefectsasweprogressintheproductlifecycle.Whatthisfigureshowsisthatthecostoffindingandfixingrequirementsdefectsinthecodingphaseis10timesmoreexpensivethanfindingandfixingthemduringtherequirementsphase.FindingdefectsduringDevelopmentTest,15-40timesmore,AcceptanceTest,30-70timesmoreandso-forth.InthecontextofthelifecyclephasesdepictedinFigure3,systemverificationandsystemvalidationactivitiesoccurduringdevelopmentandacceptancetestingaswellasduringinitialoperations.
Note:Projectmanagersoftendon’tunderstandtheseconceptsnortheresourcesneededtoaccomplishtheactivitiesassociatedwiththeproductlifecyclephases.Ihaveseencaseswherethecosttodosystemverificationandsystemvalidationactivitiesendedupcostingbetween50%-70%ofthetotaldevelopmentcostforanewsystem.[Hooks2001].Thehigherpercentagesoccurduetohavingapoorsetofrequirementsandthustheresultingrework.Projectmanagerswhoonlybudgetformaybe20%ofthedevelopmentcostsforsystemverificationandsystemvalidationareinforarudeawakingandaresettingtheirprojectupforfailure.
Earlyidentificationofdefectiverequirementspreventsre-workandresultsinsignificantcostsavings.Conversely,allowingthesedefectstogoundetecteduntilsystemverificationandsystemvalidationwillresultinsignificantcostandschedulerisk.
Consideringyoursystemverificationandsystemvalidationstrategyearlyinthedevelopmentlifecycleprovidesafoundationforestimatingyourcostandschedulefortheseactivities.
Resistthetemptationtoreduceyoursystemverificationorsystemvalidationactivitieswhenyouareoverbudgetorhavescheduleproblems.AsshowninFigure3,todosocouldresultin
Copyright©2016byRequirementsExperts. 10
muchlargercostsandscheduleslipsduetoreworkifrequirementdefectsarenotdiscovereduntilsystemvalidationoroperations.
FailingdorequirementverificationandrequirementvalidationandbaseliningyourrequirementsbeforedesigncanresultinrequirementverificationandrequirementvalidationANDdesignverificationanddesignvalidationactivitiesbeingperformedtogether,duringwhatwassupposedtobejustdesignverificationanddesignvalidationactivities.Thiscouldresultinadesignthatmeetstherequirements(passesdesignverification)butiftheyarethewrongordefectiverequirements,thedesignwillfaildesignvalidation.
Evenworse,failingdorequirementverificationandrequirementvalidationandbaselinebeforedesignandbuildcanresultinrequirementverificationandrequirementvalidationANDdesignverificationanddesignvalidationactivitiesbeingperformedatthesametimeyourfocusshouldbeonsystemverificationandsystemvalidation.Whathappensifyouverifythatyoursystemmeetsitsrequirements,buttherequirementsetisdefectivewithambiguous,missing,orincorrectrequirements?Designverificationanddesignvalidationmaynotexposethesedefects.
Becausethesetofrequirementsisdefective,theydon’tcorrectlyandcompletelycommunicatestakeholderneedsandexpectationsresultinginyoursystemfailingtopasssystemvalidation.Ifyoursystemfailssystemvalidation,itwillnotmeetthestakeholderneedsexpectationsandwillnotaccomplishitsintendedpurposeinitsoperationalenviornment.Youwillnothavedeliveredawinningproduct.
Inthecommercialsector,warrantycostsareamajorconcern.Warrantycostsasafunctionofsalescaneatintoacompany’sprofits.Allowingdefectiverequirementsinyourrequirementsetandfailingtodoadequaterequirementverificationandvalidation,designverificationandvalidation,andsystemverificationandvalidationpriortoproductlaunchincreasestheriskofproblemsnotbeinguncovereduntilaftertheproductisprovidedtocustomersandreleasedintothemarketplace.Ahighdefectrateresultsnotonlyinhighwarrantycostsandreducedprofitability,butalsocanimpactacompany’sreputationresultinginreducedsales.
Note:Theconceptof“systemvalidation”discussedaboveispartoftheformalproductdevelopmentlifecycleaddressedinmosttextbooks.Whatmostpeopledon’trecognizeisthatthereisalsoan“informal”systemvalidationthattakesplaceafteraproductisreleasedforuse.Thisinformalsystemvalidationiscommunicatedtothedevelopingorganizationisvariousformsinadditiontoproductfailuresduringoperationswhichleadstocostlywarrantywork.Theseinformalsystemvalidationactionsinclude,returns,negativecustomercommentsandreviews,recalls,decliningreputation,decreasedsales,salesprojectionsnotbeingmet,etc.Allofthesethingshaveadirectimpactonprofitsandmissionsuccess.Intoday’sworldoflimitedfunding,profitmarginsaregettingtighterandtighter.Companiescannotaffordtofailinformalsystemvalidationanymorethantheycanaffordtofailformalsystemvalidation.
Therisksofcostoverrunsandscheduleslipsassociatedwithfailingtodevelopdefectfreerequirementsandfailingtoplanaheadforsystemverificationandsystemvalidationactivities
Copyright©2016byRequirementsExperts. 11
aremitigatedbyadoptinggoodrequirementdevelopmentpracticesandplanningaheadforsystemverificationandsystemvalidationactivitiesduringthescopedefinitionandrequirementdevelopmentandmanagementlifecyclephases.
The importance of thinking ahead to system verification and system validation Unfortunately,therearealltoomanyexamplesofaproductbeingdeliveredthatmetthe“customer”requirements,yetdidnotmeetstakeholderexpectationsandwereeithernotusedorrequiredsignificantchangestobeuseful.Whatwentwrong?
Addressing System Verification and System Validation activities during Scope Definition Phase
Oneareatolookatisinthescopedefinitionphaseoftheproject.Howwellhasthescopeofyourprojectbeendefined?Doesitclearlyreflectstakeholderneeds,expectations,andrequirements?Inthefinalanalysis,systemvalidationisbasedonmakingsuretheproductaddressesthestakeholderneeds,expectations,andrequirementsdefinedduringthescopedefinitionphase,agreed-tobytherelevantstakeholders,andbaselinedinthesystemScopeReview(SR)orMissionConceptReview(MCR)beforetherequirementsarewritten.
Thescopeofaprojectconstitutesthevision:the“Need”todeveloporprocureaproductorservice;thegoalsandobjectivesofthestakeholders;informationaboutthestakeholders,especiallycustomersandusersoftheproductorservice;driversandconstraints,andconceptsforhowtheproductwillbedevelopedorpurchased,tested,verified,validated,deployed,used,maintained,anddisposedof.Thescopealsoincludestheidentificationofproductboundaries(interfaces).
Thinkingaheadtosystemverificationandsystemvalidationbeginsduringthescopedefinitionphase.Duringthisphase,inadditiontodefiningstakeholderexpectations,theprojectisdevelopingdraftsoftheirProgram/ProjectManagementPlan(PMP),SystemsEngineeringManagementPlan(SEMP),andtheMasterIntegration,Verification,andValidation(MIVV)Plan.SchedulesandbudgetsarebeingformulatedandtheWorkBreakdownStructure(WBS)isbeingdefined.Becausesystemverificationandsystemvalidationactivitiesrequireasignificantamountoftheproject’sresources,planningfortheseactivitiesmustbeginduringthescopedefinitionphase.
SpecialScopeDefinitionPhaseconsiderations:
• DefineNeed,goals,andobjectives(NGOs)
The“Need”foraprojectdefinesthe“why”–whyarewedoingthis?Whatarewetryingtoaccomplish?The“Need”isbasedonananalysisofaproblemthattheprojectissupposedtosolveforsomestakeholderorgroupofstakeholders.Goalsarethosethingsthatyouplantoaccomplishthatwillresultinmeetingthe“Need”includingMeasuresofEffectiveness(MOEs)-thethingsyouplantousetomeasuresuccess.Objectivesandresultingtechnicalrequirements
Copyright©2016byRequirementsExperts. 12
areMeasuresofPerformance(MOPs),includingKeyPerformanceParameters(KPPs)thatshowthatyouhavemetthegoals.Objectivesshowthatyouhave“gottenthere”,i.e.,yourproductaccomplisheswhatwasexpectedofitbythestakeholdersandcanfulfillitsintendedpurposeinitsoperationalenviorment.
Requirementvalidationaddresseshowwelltherequirementscommunicatethestakeholderneedsandexpectationsandsystemvalidationactivitiesultimatelyaddresshowwellthedesignedandverifiedproducthasmettheproject’sNGOs.TheprojectteamcannotclaimtheydeliveredawinningproductiftheNeed,goals,andobjectiveswerenotmet.(FormoreinformationondefiningNGOsandotherscopedefinitionbestpractices,seereferences1,2,13,14,and15.)
• Indentifyandinvolverelevantstakeholders
Stakeholdersincludekeyrepresentativesfromvariousorganizationsandgroupsthathavea“stake”or“interest”inyourproject.Fromasystemverificationandsystemvalidationperspective,thestakeholdersthatwillhaveanythingtodowithanyofthevaricationandvalidationactivitiesneedtobeidentifiedandinvolvedfromthebeginningofyourproject.Keystakeholdersthatwillbe“signingoff”andacceptingtheproductmustagreethattheplannedsystemverificationandsystemvalidationactivitieswillresultinsufficient“proof”suchthattheywillapproveandaccepttheproduct.Duringscopedefinition,apreliminarydraftofyourMIVVPlanshouldbedeveloped.
• Identifydriversandconstraints
Driversandconstraintsarethosethingsthatareimposedfromoutsideyourprojectthatyouhavelittlecontrolover,butarerequiredtoshowcompliance.Driversandconstraintsrepresentamajorsourceofrequirements.Systemverificationandsystemvalidationactivitiesmustbecompletedsuccessfullytoshow“proof”thatthesystemiscompliantwiththeidentifieddriversandconstraints.Failingtoshowcompliance,willresultinthesystemfailingcertificationforuse.
Youneedtomakesureyouincludeappropriateindustrystandardsandanyotherstandardsmandatedbygovernmentregulatoryagenciesaspartoftheprojectdriversandconstraints.ForsystemstargetedforuseintheUnitedStates,chemicalcontainmentandemissionsrequirementsfromtheEnvironmentalProtectionAgency(EPA),operatorprotectionrequirementsfromtheOccupationalSafetyandHealthAdministration(OSHA),clinicaltrialsrequirementsfromthefoodandDrugAdministration(FDA),materialsflammabilityrequirementsfromtheFederalAviationAdministration(FAA)orNationalAeronauticsandSpaceAdministration(NASA),oraccessibilityrequirementsmandatedbytheAmericanswithDisabilitiesAct(ADA)areasamplingofthetypeofoutsidedriversthatmustbereflectedintherequirementsetforyoursystem.
Ifyouaredevelopingsystemsforuseinternationally,youneedtofindandincorporatetherelevantstandardsandregulationsinyourrequirements.Yoursystemverificationandsystem
Copyright©2016byRequirementsExperts. 13
validationplanningmustincludeproducingthedatarequiredbytheregulatoryagenciesto“prove”compliancewiththeirrequirements.
Whatifyouaredevelopingamedicalproduct?Whatsystemverificationandsystemvalidationrequirementswillthegovernmentandmedicalinsurersplaceonyourproduct?Whatextrareviewsbyoutsideexpertsandwhatliabilityinsurancewillberequiredbeforeyoucanconducttestsinvolvinghumanandanimalsubjects?
Otherproductsmayrequireoutsideexpertsforcertification.DoesyourproductrequireanUnderwriters’Laboratories(UL)certificationormaybeyouhaveapressurevesselthatrequiresanAmericanSocietyofMechanicalEngineers(ASME)Class1Certification?
Evenifnotexplicitlystatedbythecustomerorotherstakeholders,beawareofandresearchtherelevantoutsidedriversandconstraints.Complianceisexpectedandfailuretoaddressrelevantdriversandconstraintscanresultinfailureduringsystemverificationandsystemvalidationandproductrejectionbythecustomerevenifthecustomerdidnotexplicitlycommunicatethesedriversandconstraintstoyou.
• Assessthetechnologymaturityofthesystemtobedeveloped
Akeydriverandconstraintistechnology.Tomeetyourobjectives,especiallyhighpriorityMOPsandKPPs,anassessmentofthecurrentmaturitylevelofallthetechnologydevelopmentneedsfortheprojectisnecessary.OutofthisprocessassignaTechnologyReadinessLevel(TRL)foryoursystemanddevelopaTechnologyMaturationPlan.AllrequirementstiedtotheMOPsandKPPsarehighpriority.IftheTRLassociatedwiththerequirementsthatimplementyourMOPsandKPPsislow,theyarealsohigh-riskrequirementsandmustbetracedcloselybyprojectmanagement.Planningforsystemverificationandsystemvalidationofhighpriorityandhigh-riskrequirementsisextremelyimportant.Theassociatedactivitiesandresultingresourceneedscanhavebigimpactsonyourproject’scostandschedule.IncludeinyourplanningaplanformaturationofkeytechnologiesrequirementtomeetyourNGOs.Thisisnecessaryasamajorriskmitigationactivitytoprotectagainstcostandscheduleoverruns.(FormoreinformationonTRLsandthedevelopmentoftechnologydrivenproductsseereference16.)
• Definefeasiblescenariosandconceptstomeetthestakeholderexpectations
Amajoractivityofscopedefinitionisthedevelopmentofasetofscenariosandsystemconceptsthataddresstheneedsandexpectationsofallthestakeholdersandaddresseachoftheproduct’slifecycles--forbothnominalandoff-nominalconditions.ScenariosandsystemconceptsforsystemverificationandsystemvalidationneedtobedefinedanddocumentedsoyouhavetheinformationneededtodevelopyourWBS,budget,schedule,anddraftMIVVplan.Amajorreasonforaproject’scostoverrunsandschedulingdelayscanbecontributedtothefailureoftheprojecttoadequatelyplanaheadforsystemverificationandsystemvalidationactivities.
Copyright©2016byRequirementsExperts. 14
Whendevelopingscenariosthataddresssystemverificationandsystemvalidationactivities,payparticularattentiontoothersystemsthat“enable”youtoaccomplishyoursystemverificationandsystemvalidationactivities.Youmayneeduniquesupportortestequipment,specialfacilities,ormodelstoperformyoursystemverificationandsystemvalidationactivities.Thesesystemsarereferredtoasenablingsystems.Theseareproductsthatneedtobeidentified,defined,designed,modified,built,procured,verified,andvalidatedintimetomeetyoursystemverificationandsystemvalidationschedule.Willthesesystemsbereadytosupportyoursystemverificationandsystemvalidationactivitieswhenyouneedthem?Earlysystemverificationandsystemvalidationassessmentsduringthescopedefinitionphasewillhelpyouidentifytheseenablingsystemsaswellasadditionalprojectrequirementsneededtosupportsystemverificationandsystemvalidationactivitiesandallowyoutoincludetheminyourprojectbudgetandschedule.
Yoursystemmayneedspecificfeaturesdedicatedtosupportyoursystemverificationandsystemvalidationactivitiesespeciallywhenusingtestordemonstrationasmethodsforsystemverificationandsystemvalidation.Additionalinterfacesmaybeneededtogiveyouaccesstodatathatisneededforsystemverificationandsystemvalidation,butnotnormaloperations.Thiscouldresultintheneedforadditionalconnectors,testpoints,andwiringharnessestoconnecttospecialtestinstrumentationorexternalpower,extradatatodisplayorrecord,oradatabasetogiveyouvisibilityintoaninternalprocess,aninspectionportal,orabrackettoholdapartinatestfixture.[Hooks2001]
Requirementsthatareverifiedbyanalysisusingmodelscanresultinadditionaldevelopmentefforttoproducethosemodels.Modeltypesincludephysical,graphical,mathematical,andstatistical[Larsen2009].Insomecases,themodelsneededforsystemverificationandsystemvalidationcanbeveryexpensivetodevelopandtakeupalargeportionoftheproject’sresources.Inaddition,youwillneedtovalidatethemodel.Todothis,youmayhavetodevelopasimulationofyoursystem.Thissimulationwillbeusedtorunteststocollectdataneededtovalidatethemodelbeforethemodelcanbeusedtoverifyyoursystem’srequirementsorforsystemvalidation.
Whatistheneededfidelityofthesystemyouplantouseforsystemverificationandsystemvalidation?Foryoursystem,youneedtoassesswhichrequirementscanbeverifiedusinganengineeringtestmodel,qualificationunit,delivery-likehardware,oractualdeliveryorflighthardware.Whatfidelityofequipmentisneededduringdevelopment?Qualification?Acceptance?Operations?Eachofthesetypesofequipmentorversionsofyourproductneedstobeincludedinyourbudgetandschedule.
Theseareexamplesoffeaturesoradditionalsystemsthatareneededtomakesystemverificationandsystemvalidationpossibleinsomecasesandreducecostsinothercases.Addressingtheequipmentandfacilitiesneededtoperformsystemverificationandsystemvalidationearlyavoidspossibledelaysinactualproductdelivery.Likemodelsdiscussedabove,youmustverifyandvalidatethisequipmentpriortoyouruseduringyoursystemverificationandsystemvalidationactivities.Planningforsystemverificationandsystemvalidationearly
Copyright©2016byRequirementsExperts. 15
permitstheexecutionofconcurrentactivitiesthatwillallowyoutocompleteyoursystemverificationandsystemvalidationperyourscheduleandavoiddeliveryscheduledelaysandresultingcostoverruns.
• Defineproductboundariesandexternalinterfaces
Aninterfaceisaboundaryacrosswhichsystemsinteract.Seriousproblemscan,andalltoooftendo,ariseattheinterfaces.Yourprojectisparticularlyvulnerablewheninterfacingwithexternalsystemsoverwhichyouhavelittleornocontrol.Becauseofthis,yourproductisatmostriskattheinterfaces.Identifyinginterfacesfacilitatesdefinitionofyoursystem’sboundariesandclarifiesthedependenciesyoursystemhasonothersystemsanddependenciesothersystemshaveonyoursystem.Indentifyinginterfaceshelpsyouensurecompatibilitybetweenyoursystemandthosewithwhichitinteracts.Identifyinginterfacesalsohelpstoexposepotentialprojectrisks.
Becauseofthis,itisextremelyimportantthatyoudefineyourconceptsforhowyouwillmakesure(verify)yoursystemmeetsallinterfacerequirementsandvalidatethatyoursystemwillworkwithalltheexternalsystemswithwhichitmustinteractintheintendedoperationalenvironmentANDisprotectedfromoutsidethreatsacrossyourinterfaces.
Inyourplanning,besuretoaddressbothinternalandexternalinterfaces.Anemulatororsimulatormaybeneededtocompleteyoursystemverificationandsystemvalidationactivities.Asdiscussedearlier,thesemustbeeitherdevelopedorprocuredandverifiedandvalidatedpriortouseduringyoursystem’sintegration,systemverification,andsystemvalidationactivities.
Addressing System Verification during the Requirement Definition Phase Identificationoftheproposedsystemverificationmethodisanessentialattributeofawell-writtenrequirement.Identificationofthesystemverificationmethodasyoudevelopyourrequirementsimprovesrequirementquality,ensuresyourrequirementsareverifiable,providesthebasisforestimatingsystemverificationcostandschedule,andcanhelpreducethecostoftheseactivities.Carefulassessmentoftheverificationmethodandassociatedimplementationactivitiescanresultinamodificationtotheassociatedsystemverificationactivitiestoreducethesystemverificationcostandtimewhilestillmeetingyoursystem’sNeed,goalsandobjectives.
Duringthescopedefinitionphasediscussedabove,Idiscussedsystemverificationandsystemvalidationfromthe10,000-footlevel.Basicconceptsandscenariosforcompletingyoursystemverificationandsystemvalidationactivitiesweredefined.ThisinformationwasusedtodevelopahighlevelbudgetandschedulefortheseactivitiesinyourPMP,defineyourhighlevelprocessesintheSEMP,andprovidedadetailedprocessdefinitionintheMIVVPlan.
However,thedevilisinthedetails.Asrequirementsaredeveloped,theverificationmethodthatwillbeusedtoshowthesystemmeetstherequirementsmustbeaddressed.Onceyourrequirementshavestabilized,youalsoneedtostartdefiningtheactivitiesneededtoverifyyour
Copyright©2016byRequirementsExperts. 16
systemmeetsitsrequirements.Let’slookatthecharacteristicsofgoodrequirementsfromasystemverificationperspective.
• Requirementisneeded
Amandatorycharacteristicofarequirementisthatitisneeded.Ifarequirementisnotneeded,whyisitintherequirementset?Byitsveryexistence,arequirementhasadollarvalueassociatedwithit.“Eachrequirementcarriesacost.Itisthereforeessentialthatacompletebutminimumsetofrequirementsbeestablishedfromdefinedstakeholderrequirementsearlyintheprojectlifecycle.Changesinrequirementslaterinthedevelopmentcyclecanhaveasignificantcostimpactontheproject,possiblyresultingincancellation.”[INCOSE2010].
Allrequirementsneedtobemaintained,managed,andverified.Requirementsthatarenotnecessaryresultinincreasedmanagementcostoftherequirementsthat,inturn,increasesoverallprojectcostsandleavesfewerresourcesforneededrequirements.Un-neededrequirementsresultinworkbeingperformedthatisnotnecessary,takingresourcesawayfromtheimplementationofthoserequirementsthatareneeded.Rememberthatintegration,systemverification,andsystemverificationcostscanbe50%ormoreofyourdevelopmentbudget.Youdon’twanttospendtimeandeffortprovingthesystemmeetsrequirementsthatwerenotneeded.Inaddition,implementingrequirementsthatarenotnecessarycanresultindegradedsystemperformanceaswellasintroducingapotentialsourceoffailureandconflict.Werecommendyoutakea“zero-based”approachtodefiningyourrequirementset.Bythiswemeanthatyoudefinetheminimumsetthatisnecessaryandsufficienttomeetthestakeholderneedsandexpectations.Thenifanyonewantstoaddadditionalrequirements,theyhaveto“fight”togettheadditionalrequirement(s)addedtotheset.Insistthateachrequirementthatissubmittedhasrationaledocumentedwiththerequirementthatclearlycommunicatestheneedforthatrequirement.
Totestfortoseeifarequirementisneeded,ask:
1) Whyistherequirementneeded?Whyisitintherequirementset?Ifthereisnorationale,oraweakrationaleforitsexistence,deleteit.
2) Whatwouldhappenifwedeletedthisrequirement–wouldthedeveloperaddressthisconcernanyway?Ifnothingwouldhappenorthedeveloperwouldhavetoaddressthisconcernanywaytomeetotherrequirements,thenwhyisitintheset?Deleteit.
• Requirementisverifiable
Anothermandatorycharacteristicofarequirementisthatitmustbeverifiable.Whyaskforsomethingifyoucan’tproveithasbeenimplementedasintended?Arequirementthatisnotverifiablemayresultintheimplementationofanincorrectsolutionorthesystemmaybeverifiedasdoingsomethingotherthanwasintended.Ifthetrueintentoftherequirementisnotclear,stakeholderexpectationsmaynotbemet.Insistthateachrequirementthatissubmittedhastherecommendedverificationmethoddocumentedwiththerequirement.
Totestwhetherarequirementcanbeverified,ask:
Copyright©2016byRequirementsExperts. 17
1) Howcanthisrequirementbeverified?Ifnomethodcanbeidentified,canitbere-writtensothatitcanbeverified?
2) Dowewanttoverifythisrequirement?Ifnot,deleteit.3) Isthiswhatwereallywanttoverify?Ifnot,changeittowhatyoudowanttoverifyor
deleteit.
• Requirementisattainable/achievable
Athirdmandatorycharacteristicofarequirementisthatitmustbeattainable/achievablegiventheexistingbudget,schedule,andleveloftechnicalmaturity.Arequirementthatisnotattainablewillfailsystemverification.Ifyouknowyouwillnotbeabletomeettherequirementandthusknowyouwillfailtoverifythatthesystemmeetstherequirement,whystateit?Statingarequirementthatisnotattainablewillresultinwastedeffort,costandscheduleimpacts,aswellasunmetstakeholderexpectations(performance).Iftherequirementisaperformancevalue,youneedtodoanassessmentastothematurityofthetechnologyneededtomeettherequirement.Ifthecurrentmaturityislow,therequirementrepresentsarisktoyourprojectandyouneedaplantomitigatethatriskandmaturethetechnology.Thiscouldrepresentbothacostandscheduleriskaswell.
• Requirementiswrittencorrectly
Ifarequirementispoorlywritten,itwillmostlikelynotbeverifiable.Ifitcontainsambiguousterms,itcanbeunderstoodinmorethanonewayandisthusnotverifiable.Ifitisnotclear,concise,orusespoorgrammarandyouareleavingroomformultipleinterpretations,itisnotverifiable.Ifitisstatednegatively,itmaynotbeverifiable.(Itisdifficulttoprovethatasystemnever,everdoessomething.)Iftherequirementcontainsmorethanonethought,whichthoughtdoyouverify?Thisisespeciallyproblematicifeachthoughtneedsadifferentmethodofverification.
Toguardagainstpoorlywrittenrequirementsmakesureyourteamistrainedtowritedefectfreerequirementsanddefineyourrequirementdevelopmentprocessasonebasedupontheuseofthe“RulesofWritingGoodRequirements.”(Seereference2,aswellasAppendixCoftheNASASystemsEngineeringHandbook,“HowtoWriteGoodRequirements”.[NASA2007].)Inaddition,youcangetacopyofthe“INCOSEGuidetoWritingRequirements,2015”.
• Requirementisdocumentedatthelevelwhereitwillbeverified.
Properflowdown(allocation)ofrequirementstosystemarchitecturallevelsisveryimportanttoeffectivesystemverificationandsystemvalidation.Allrequirementsmustbedocumentedatthelevelwheretheywillbeimplementedandverified.
AsshowninFigure2,theSE“V”model,requirementdefinitionanddesignaretop-downactivitiesduringwhichrequirementsforthesystemasawholearedefinedandthendecomposedintosub-systemsdefinedduringthedesignactivitywhileallocatingrequirementstoeachsubsequentlevelofthearchitecture,descendingdowntheleftsideofthe“V”.
Copyright©2016byRequirementsExperts. 18
Oncethesystemhasbeendecomposedintothevariousunits/componentsthatmakeupthesystem;integration,systemverificationandsystemvalidationactivitiesbegin,ascendinguptherightsideofthe“V”.Duringintegration,lowerlevelunits/componentsareintegratedtogethertoformthesubsystems.Priortointegrationintosubsystems,thelowerlevelunits/componentsareverifiedandvalidatedandcompatibilityisassessedtoensuretheselowerlevelentitiesmeettherequirementsthatdrovetheirdesignandwillsuccessfullycombineintothesubsystemsatdefinedinterfaces.Oncethesubsystemshavebeenformed,theprocessrepeats.
Priortointegrationintothesystem,thesubsystemsareverifiedandvalidatedandcompatibilityisassessedtoensurethesubsystemsmeettherequirementsthatdrovetheirdesignandwillsuccessfullycombineintothesystematthedefinedinterfaces.
Once,thesubsystemshavebeenintegratedtogethertoformthesystem,systemverificationactivitiesareperformedtoensuretheintegratedsystemsmeetstherequirementsthatdrovethesystemdesign.
Oncetheentiresystemhasbeenverified,thesystemisreadyforsystemvalidationandacceptancebythecustomer.Insomecases,finalsystemvalidationandsystemacceptanceactivitiesarecombined.
Specifyingarequirementatalevelhigherthanwhereitwillbeverified,maymakeitdifficultorimpossibletoverifyatthatlevel.Whenwritingrequirements,alwaysask:“Doesthisrequirementapplytothelevelyouarewritingrequirementsfor?”Therequirementmaybevalid,butnotforthatlevel.Iftherequirementisnotbeingstatedatthecorrectlevel,moveittothecorrectlevel.
• Requirementisallocated(floweddown)tothenextlevelofthesystemarchitecture
Theflowdownofrequirementsfromoneleveltothenextlevelofthearchitectureisanimportantactivitythatneedstobedonecorrectly.Thisflowdownactivityiscalledallocation.Allocationistheprocessofapportioningresourcesorassigningresponsibilityforimplementationofrequirementsfromanupperlevelinthehierarchytopartsatthenextlevelofthesystemarchitecture.Allrequirementsmustbeallocateduntilthefinallevelofthearchitectureisdefined.Onceallocated,thenextlevelwillderivechildrenrequirements,thatwhenimplementedatthatlevel,willcontributetotheparentrequirementbeingmet.
Failingtoallocateyourrequirementscanresultinrequirementsnotbeingimplementedatthenextlevelofthearchitecture.Failingtoallocaterequirementscanalsoresultinmissinglowerlevelrequirements(derivedchildrenrequirementsthatarenecessaryandsufficienttomeettheparentrequirement.)This,inturn,couldresultinthesystemfailingeithersystemverificationorsystemvalidation.
• Requirementaddressesaninternalinterface
Akeypartoftheallocationprocessistheidentificationanddefinitionofinternalinterfaces
Copyright©2016byRequirementsExperts. 19
amongend-itemswithinyoursystemaswellasexternalinterfacesbetweenyoursystemandtheoutsideworld.Whenfunctionalrequirementsareallocatedtomorethanonepartofthearchitectureatthenextlevel,theremaybeaninterface.Ifso,theinterfacemustbeidentifiedanddefined.Therespectiverequirementsdocumentedforeachpartthenwillincludetheirrespectiveinterfacerequirements.Interfacerequirementsrequirespecialattentiontoensurethattheyarewrittensoastopermitsystemverification.(Theinclusionofareferencetowheretheinterfaceisdefinedisacommonpartofallinterfacerequirements.Thisdefinitionisneededsothattheinterfacerequirementcanbeverified.TheinterfacedefinitionscanbecontainedinanInterfaceAgreementDocument(IAD)orInterfaceControlDocument(ICD).)(Formoreinformationonidentifyinganddefininginterfacesandwritinganddocumentinginterfacerequirements,seereference17.)
• Requirementistraceabletoaparentorsource
Traceabilityistheconceptthatallrequirementsneedtobelinked(traced)totheirparentorsource.Tracingtoaparentorsourcesisalsoamethodtodetermineifarequirementisneeded.Whenthereisnotracetoaparentorsource,itcouldmeanthattherequirementisnotneededandthereisgoldplating.“Goldplating”istheactofaddingfeaturestoasystemthatarenotneeded.Goldplatingisamajorsourceofrequirementscreepandcanimpactthecostandscheduleofyourproject.Thisisespeciallytruewhenitcomestosystemverificationandsystemvalidationactivities.
Oncerequirementshavebeenallocatedtoapartatthenextlevelofthearchitecture,childrequirementsarederivedsuchthatwhenimplementedresultsinimplementationoftheparentorsourcerequirement.Assessmentofwhetherornotthechildrenrequirementslinked(traced)totheparentorsourcerequirementarenecessaryandsufficienttomeettheintentoftheparenttoisneededtodeterminewhetherornotaparentrequirementisproperlyimplemented.Withoutproperallocationandtraceabilitydocumentation,thisassessmentcannotbedone.
Understandingandthencorrectimplementationanddocumentationoftheconceptsofallocationandtraceabilityarethekeystoasuccessfulverificationandvalidationprocess.AsshowninFigure2,SystemsEngineeringVmodel,integrationisabottoms-upprocess.Whenintegratingatonelevel,youneedtomakesuresystemverificationandsystemvalidationarecompleteatthepreviouslevel.Successfulsystemverificationatthelowerlevelisaprerequisitetosuccessfulsystemverificationatthenextlevel.Thisisespeciallytruewhenitcomestointerfaces.
Akeypointisthatbeforeverifyingthesubsystemorsystemmeetsarequirement,ifthatrequirementhaschildrenrequirements,verificationthatthepreviouslevelmeetsthechildrenrequirementsmusthavebeencompletedbeforeverifyingtheparentrequirementhasbeenmet.Proofthatthechildrenrequirementshavebeenmetisneeded(aprerequisite)beforeverifyingthattheparenthasbeenmet.Allocationandtraceabilityallowthistohappen.
Copyright©2016byRequirementsExperts. 20
Whenplanningsystemverificationatonelevel,alwaysincludeastepinyourprocesstolookatthesystemverificationdocumentationatthepreviousleveltomakesureithasbeensuccessfullycompletedbeforeyoudosystemverificationatyourcurrentlevel.
Planning your system verification and system validation activities Thissectionprovidesmoreofthe“how”associatedwithplanningforsystemverificationandsystemvalidationbyintroducingsomeofthetoolsandactivitiesyoucanusethatwillhelpyouplanforandaccomplishsystemverificationandsystemvalidation.Twoexcellentsourcesthatgointomuchmoredetailcanbefoundinreferences3&4.
Thinking about system verification and system validation while developing your requirements
Asyouwriteyourrequirements,addressthecharacteristicsofgoodrequirementsdescribedintheprevioussection.Ensurethatyourrequirementsare:needed,verifiable,attainable,correctlywritten,atthecorrectlevel,allocated,andtraced.Alsoensurethatallinternalandexternalinterfacesaddressed.
Addressingsystemverificationastherequirementsarewrittenensuresabettersetofrequirements.Thefollowingquestionsneedtobeaddressedaseachrequirementiswritten:
1) “Whatwillbetheprimarymethodusedtoverifythisrequirement?”
TheprimarymethodsofverificationincludeTest,Demonstration,Inspection,andAnalysis.(Note:SomeorganizationsfurtherdefinethevariousmethodsofAnalysistoincludeanalysisbysimilarityandbymodeling.)“Eachrequirementshouldbeverifiablebyasinglemethod.Arequirementrequiringmultiplemethodstoverifyshouldbebrokenintomultiplerequirements.“[INCOSE2010].Somepeoplewillstatemorethanoneverificationmethod,e.g.,TestandAnalysis.ThisisnotneededasitisunderstoodthatverificationbyTestincludessomeanalysisactivitiesandAnalysismayincludesometestingactivities.Thebestpracticeistostateasingleprimarymethodforverification.Note:Makesureyouunderstandthedifferencebetweentest,inspection,andanalysisasactivities,vsTest,Inspection,andAnalysisasmethodsofsystemverification.Theconceptsarecompletelydifferent,yetoftenthisdifferenceisoftennotunderstood.
Riskisamajorconsiderationwhenchoosingthesystemverificationmethod.Thisisaveryimportantissuefromaprojectmanagementstandpoint.SystemverificationbyTestprovidesthemostconfidencethatthesystemmeetsarequirement.WewouldlikeuseTestasourprimarymethodforallrequirements,butwemaynotbeableto.InsomecasesTestisnotappropriate.Inothercases,inmaynotbepossibleuntilthesystemisintegratedandinitsrealoperationalenvironment.Inmanycaseswecan’taffordthecostortimetodoafull-uptestforallourrequirements.Inothercases,aspecialfacilitytosimulateyouroperationalenvironmentorequipmenttotestyourinteractionswithothersubsystemsorsystemsmaynotbeavailablenorcosteffectivetodevelop.
Copyright©2016byRequirementsExperts. 21
Alloftheseconsiderationsneedtobeaddressedwhendecidingonthesystemverificationmethod.Therefore,youmustdoariskassessmentanddeterminewhichrequirementsrepresentthehighestrisktoyourprojectifnotproperlyimplemented.Oftentherequirementsthatposethehighestrisktoprojectsuccessarerequirementsthattracetothehighpriorityobjectivesdefinedduringscopedefinitionaswellasthesafetyandmissionassurancerequirements.
Whenarequirementposesalowerrisktotheproject,oneoftheotherverificationmethodsmaybeacceptable.Inmyexperienceanalysisisthemostoverusedmethodandpossesthehighestriskwhengoodrationaleforusinganalysisisn’tdefined.Reducingverificationactivitiesduetocostoverrunsorscheduleslipscanaddsignificantrisktoyourproject.
Therationaleforwhyyouareusingaspecificverificationmethodneedstobeincludedinyourplanforeachrequirement.
Becausechoosingthesystemverificationandsystemvalidationmethodisreallyacost,schedule,andriskdecision,theProjectManagerandLeadSystemsEngineerneedtobeinvolvedinthedecisionastowhichsystemverificationmethodwillbeusedandwhichactivitieswillbeincludedaspartofthesystemverificationmethodagreedto.
2) “Whatactivitiesneedtobeperformedaspartofthesystemverificationorsystemvalidation?”
Developascenariooroperationalconcepttodeterminewhichactivities(test,inspection,analysis)needtobeperformedaspartofthechosenverificationorvalidationmethodbeingsuretoaddresswhereandbywhom.Thisprocesswilluncovertheinterface,facility,supportequipment,andinstrumentationissuesassociatedwithyoursystemverificationandsystemvalidationactivities.Wherewillthesystemverificationorsystemvalidationactivitiesbeperformed?WillthesystemunderverificationorvalidationrequiredifferentpowerorcoolingduringaTestorDemonstrationthanneededduringnormaloperations?Willitneedsimulateddatastreams?Howwillyouaddresstheinterfaces–isanemulatororsimulatorneeded?Whatexternalcommandandcontrolcapabilityisneeded?Whatdataisneededtocontroltheactivity?Whatdatamustberecordedandhowfastmustthesystemoutputthisdata?Whowillbeinvolvedinthesystemverificationorsystemvalidationactivity?Whomustobservethetestordemonstrationactivity,signoffthattheactivitywasperformedperprocedure,performtheanalysisofthedataresultingfromthesystemverificationorsystemvalidationactivity,orinspectthesystem?Whattrainingorcertificationsdothepersonnelneedtohavetoparticipateintheirassignedrole?Whatsafetyconsiderationsarethere?Whatresourcesareneededtocompletethisactivity?Howlongwillittake?Howmuchwillitcost?
Thesystemverificationandsystemvalidation(aka“test”)engineerswillaskthesequestions.Projectmanagersneedtoknowtheresourcesneeded,thecost,andscheduleneedstoaccomplishthesystemverificationorsystemvalidationactivitytomakesurethey
Copyright©2016byRequirementsExperts. 22
havesufficientfundsandtimeaccountedforintheprojectbudgetandschedule.Itisbesttoaddressthesequestionsbothasyouaredefiningyourprojectscopeandasyouaredevelopingyourrequirementsbeforeyouunknowinglyspendprojectresourcesondesigningorbuildingyoursystembasedonanunverifiablerequirementorperformingansystemverificationorsystemvalidationactivitythatcannotbeaccomplishedwithintheproject’sbudgetandschedule.
Thisisaknowledge-basedprocessthatevolveswiththedesignofyoursystem.Someofthesequestionsmaynotbeabletobeansweredbeforesystemdesign.Requirementdevelopment,management,anddesignareanongoingactivities.Evenifyoucannotanswerthesequestionsbeforesomedesigneffort,youcancertainlydosoduringorafterdesignbutbeforemanufacture,code,orbuild.
3) “Whenandatwhatlevelwillyoursystemverificationandsystemvalidationactivitiesbeperformed?”
Aspartofyoursystemverificationandsystemvalidationplanningandconceptdevelopmentefforts,considertiminganddevelopa“when”strategy.Rememberintegration,systemverification,andsystemvalidationarebottoms-upprocessactivities.Whenandwhereinyourproductintegrationlifecyclewillyouverifytherequirementhasbeenmet?
Balancecomponent,subsystem,andsystemlevelverificationactivities.Youmaybetemptedtosaveverificationtimeandmoneybydoingonlysubsystemorsystemlevelverification.Thisstrategyisveryrisky;youmaynotfindcriticalproblemsuntilverylateintheproductintegrationprocessorduringsystemvalidationorcustomeracceptance.Itmaybehard(timeconsuming)totracetheproblemtoitssourceforacomplexsystem.Onceyoufindthesourceoftheproblem,itwillbemoredifficulttoresolve,especiallyifmorethanonecomponentisinvolvedandyouneedtode-integratethesystemaspartoftheanomalyresolution.Asingleproblememergingduringsystem-levelverificationorvalidationmaycancelallsavingsprojectedfromeliminatingcomponentorsubsystemverificationactivities.
Alternatively,budgetandschedulelimitsmaymaketheverificationofeverycomponent-levelrequirementcost-prohibitiveandunnecessary.Again,itisamatterofriskmitigation.Ifthecomponentlevelrequirementtracestohigher-levelrequirementsdealingwithhighpriorityobjectivesormissionorsafety,youwillwanttoverifythosecomponentsmeettheirrequiremens.Alternatively,youmaybeabletoaccepttheriskandwaittodosystemverificationatahigherlevelofthearchitecture-ifpossible.Anotherapproachistousealesscostlyandtimeconsumingmethodofsystemverificationforthelowerriskcomponentrequirements,e.g.,“Analysis”or“Demonstration”ratherthan“Test”.
4) “WhatdoIneedtodotoverifythesysteminthecontextofthenextlevel“super”systeminwhichmysystemisapartofinitsintendedoperationalenvironment?”
Copyright©2016byRequirementsExperts. 23
Failingtoverifytheintegratedsystemorfailingtoverifythesystemwithitsexternalinterfacesandinitsoperationalenvironmentcanresultinmajorembarrassmentwhenthedeliveredsystemfailstoperformasexpectedanddoesn’taccomplishitsintendedpurposeinitsoperationalenvironment(validation).Eventhougheverypart,component,orsubsystemhasbeenverified,youneedtoensuretheyworktogetherasawhole,theyworkwiththeirexternalinterfaces,andtheyworkintheintendedoperationalenvironmentbeforesystemvalidationactivitiesandbeforeyoudeliverthesystemtoyourcustomer.VerifyinginternalandexternalinterfacesoftenrequiresTestasthepreferredmethodofverification.(UsingAnalysisbysimilarityisveryrisky!)Thecostofthistestingcanbehigh,butfailuretosuccessfullycompletethistestingcanleadtomajor,andevencatastrophic,problems.
5) “Whatconstitutesproofthattherequirementhasbeenverified?”
Proofisderivedfromdocumentationandestablishmentofconfidence.Documenttheexpectationsfortheevidenceneededandlevelofconfidenceinthisevidencebeforeyouandothersarewillingto“signoff”ontheverificationdocumentationfortherequirement.Consideralsowhatevidenceisneededbeforethecustomerwillaccepttheproductfordelivery.
Failingtoaddressthesequestionsearlycanresultinunpleasantsurpriseslateinyoursystemdevelopmentlifecycle,puttingyourprojectatrisk.
Addressing system verification, system validation, certification, and acceptance in Vendor/Supplier Agreements and Statements of Work Ifanend-itemistobeoutsourcedtoavendor/supplier,therequirementsforeachend-itemaretypicallydocumentedinaSystemRequirementDocument(SRD)oraTechnicalDataPackage(TDP).AlongwiththiswillbeaStatementofWork(SOW)thatcontainsrequirementsonthevendor/supplierconcerningdesign,development,systemverification,systemvalidation,andacceptance.TheSOWcontainsprocessandworkmanshiprequirementsleviedonthevendor/supplieronthetaskstheyneedtoperformaspartofthecontracttofabricate,test,transport,andinstallaspecificend-item.TheSRDcontainsrequirementsintendedtocommunicatetothedesignteamthestakeholderneedsandexpectationsinanunambiguousandcleartechnicallanguage,whichcanbethoughofasthe“design-to”requirements.TheTDPincludesequipmentlists,partslists,drawings,andend-itemspecificationrequirementsonthesystemtobesuppliedthatcanbethoughtofasthe"build-to"requirements.Insomecases,theSRDorend-itemrequirementsmaybecontaineddirectlyintheSOW.Themethodofhowthesystemwillbeverifiedthatitmeetsalloftheserequirementsneedstobedefinedaspartofthevendor/supplieragreementprocess.
KeydeliverablesspecifiedintheSOWneedtoincludeanallocationmatrixthatshowshowthevendor/supplierallocatesyourrequirementstothepartsthevendor/supplierisresponsibleforprovidingaswellasatraceabilitymatrixshowinghowthevendor’s/supplier’sderivedrequirementstracetoyourparentrequirements.Inaddition,youneedtoaddresswhatproof
Copyright©2016byRequirementsExperts. 24
youwillneedfromthevendor/supplierastohowtheyhavemetyourrequirementsandwhatisneededforthecustomertoacceptthesystemfromthevendor/supplier.
Together,theSOWandSRDorTDPresultsinavendor/supplieragreementwhichisacontractbetweenthecustomerandvendor/supplier.Thiscontractneedstoclearlystateyourexpectationsforthesystemverificationandsystemvalidationapproachandscopeoftheeffort.Ifthesystemiscomplex,testingeverypossiblecaseorpathmaybecost-prohibitive.Whataretherisksandwheredoyouwanttospendfundsforsystemverificationandsystemvalidation?Missing,vague,oropen-endedsystemverificationandvalidationplanscanoverrunthesupplier’sbudgetorleaveyou,thecustomer,withoutconfidenceinthesystem.Addressingsystemverificationandsystemvalidationinvendor/supplieragreementsprotectsallpartiesinthecontract.
Thedocumentationthatprovidesproofthattheseactivitieshavebeensuccessfullycompletedneedstobeclearlylistedascontractdeliverables.Thisdocumentationisusedaspartofyouroverallsystemverificationandvalidationactivities.Youcanacceptthesupplier’sdocumentationasproof,orforcriticalitems,youmaywanttoperformyourownsystemverificationandsystemvalidation.Insometexts,acceptingasupplier’sdocumentationasproofthedeliveredcomponentorsystemmeetsitsrequirementsisclassifiedas“verificationbycertification”andistreatedasavalidmethodforsystemverificationandsystemvalidation.
Ifyouareavendor/supplierandyourcustomerswillbeperformingtheirownacceptanceactivities,assessyoursystemverificationandvalidationactivitiesagainsttheiracceptanceplan.Areyoumissingrequirementsthatareobvioustothecustomer?Ifyouarethecustomer,youracceptanceplanmustbecomprehensiveenoughtosatisfyyourorganizationthatthesystemwillperformintheoperationalenvironmentasadvertisedorasdescribedinyoursupplieragreement/contract.Ineithercase,bothorganizationsneedtoensuretheiracceptanceplanreflectstheactualrequirementsanddoesnotintroducepreviouslyunstatedrequirements.[Hooks2001]
Tools to help with your system verification and system validation planning
Thissectionaddressessomeofthekeytoolsthatareusedtohelpwithsystemverificationandsystemvalidationplanning.
• ProgramorProjectManagementPlan(PMP):Definestheoverallprojectactivities,criticalmilestonesandevents,andprocessidentification.OrganizationandWorkBreakdownStructure(WBS)aswellasapreliminarybudgetandschedulearedefined.Thisinformationneedstoincludeyourhigh-levelsystemverificationandsystemvalidationactivities.
• SystemsEngineeringManagementPlan(SEMP):ExpandsonthePMPfromaSystemsEngineeringperspective.TheSEMPestablishestheoverallplanforthetechnicaldevelopmentofyoursystem.TheultimateobjectiveoftheSEMPistoprovideadisciplinedframeworktomeetcost,technicalperformance,quality,andschedule
Copyright©2016byRequirementsExperts. 25
objectivesfortheprojectorprogram.ItisimportantthattheSEMPestablishtheoverallsystemverificationandsystemvalidationphilosophyfortheprojectorprogram.
• MasterIntegrationVerificationandValidation(MIVV)Plan:ExpandsandimplementsthesystemverificationandsystemvalidationphilosophyoftheprojectorprogramasdefinedinthePMPandSEMP.TheMIVVPlandefinesthedetailedprocessesandapproachthatwillbeusedforsystemverificationandsystemvalidation.TheinformationintheMIVVPlanisusedtomanageandplansystemverificationandsystemvalidationactivities.TheMIVVPlanfurtherdefinestheintegration,systemverification,andsystemvalidationschedulecontainedinthePMPandSEMP.
• SystemRequirementsDocumentorSpecification(SRD/SRS):Containsthesystem’srequirements.Theproposedverificationmethodisakeyattributeofeachrequirement,andneedstobecapturedanddocumentedwiththerequirement.Youcanalsoincludethelevel,lifecyclephase,andsuccesscriteriafortheverificationactivities.ThisinformationwillbeusedtocreatetheRequirementsVerificationMatrix.
• RequirementsVerificationMatrix(RVM):Amatrixthatlistseachrequirement,verificationmethod(Test,Inspection,Demonstration,Analysis),responsibleorganization,level(component,subsystem,system),phase(design,manufacture,integration),locationorfacility,andoftenincludesashortdescriptionofthesuccesscriteriaforsuccessfulverification.TheRVMisoftenincludedaspartoftheSRD/SRS.IftheinformationneededtocreatetheRVMisincludedaspartoftherequirementattributes,thenyourRequirementManagementTool(RMT)willbeabletogeneratetheRVMforyou.
• SystemIntegration,VerificationandValidation(SIVV)Plan:Implementstheproject’sMIVVPlansystemverificationandvalidationactivitiesforagivencomponent,subsystem,orsystemasdocumentedintheSRD/SRSandassociatedRVM.TheSIVVPlancontainsadetailedidentificationofthetaskstobeperformedaspartofverificationandvalidationofthatsystem.Resourcesincludingtestequipment,facilities,andpersonnelareaddressed.AdetailedscheduleconsistentwiththeIntegration,Verification,andValidationScheduledefinedintheMIVVPlanisincluded.
Productstohelpdevelopandimplementthisplanarediscussedbelow.
• VerificationRequirementDefinitionSheet(VRDS):ForeachrequirementintheSRD/SRS/RVM,theVRDStransformstheRVM“what’s”into“hows”–“Howdoyouplanonverifyingtherequirementslistedin,andmeetthesuccesscriteriadefinedin,theRVM?”TheVRDSsdocumentthedetailsofthespecificverificationactivitiesidentifiedintheRVM.AnexampleofandinstructionsforcompletingaVRDSisincludedinAppendixA.
TheVRDScapturesalloftheverificationrequirementsandothersupportinginformationforaspecificrequirementverification.ForeachSystem,thesetofVRDSsdocumenttheverificationrequirementsfromtheperspectiveoftheverificationactivitiesthatare
Copyright©2016byRequirementsExperts. 26
performedtoensurethesystemmeetsitsrequirements.ThesetofVRDSarecontainedinaVerificationRequirementDocument(VRD).
(Note:SomeprojectsdocumentverificationrequirementsinaseparatesectionoftheSRD/SRS.Idonotadvocatethisapproach.Iadvocateaseparatedocument,VerificationRequirementsDocument(VRD)wheretheverificationrequirementsaredocumentedviatheVRDSsdefinedherein.TheconceptofaseparateVRDcontainingVRDSsforeachrequirementwasusedbyNASA’sConstellationProgramAres1Xproject.Seereference11.)UsingaseparateVRDtodocumentthisinformationallowsyoutobaselineyourrequirementsetandminimizeschangestoyourSRDastheinformationintheVRDmatures.
• TaskDefinitionSheets(TDS):Asanaidtoverificationactivityplanning,eachsystemleaddevelopsalistoftaskstobeperformedtoaccomplishtheneededsystemverificationactivitiesdefinedintheVRDS.Foreachofthosetasks,aTaskDefinitionSheet(TDS)isdeveloped.TheTDSscontaintheinformationneededtodevelopproceduresandanintegratedscheduleforcompletingallthedefinedtasksinvolvedinverification,validation,acceptance,certification,andreadiness.OneTDScanaddressmultiple,relatedVRDSs.WhichVRDSswillbelistedinthetaskobjectivessectionoftheTDS.ThereshouldbeoneTDSforeachverificationorvalidationtaskidentifiedintheSIVVPlan.AnexampleofandinstructionsforcompletingaTDSisincludedinAppendixB.
• TaskPerformanceSheet)(TPS):TheTPSorotherapprovedWorkAuthorizationDocument(WAD)containstheactualprocedureusedtoperformthesystemverificationorvalidationactivitiesdefinedintheTDS.Thisisthedocumentactuallyusedtoaccomplishthesystemverificationorsystemvalidationactivityanddocumenttheresults.TheTPSissignedoffbytheappropriatepersonnelwhoareidentifiedintheapplicableTDS(s).ThecompletedTPScontainsthedatathatrepresentstheproofthatthesystemmeetsarequirementorsetofrequirementsasdefinedintheVRDSsorvalidatedasdefinedintheMIVVPlanandSIVVPlan.
AsystemwillhavemultipleTPSs.TheremaybeoneTPSforverificationbyInspectionwhileanend-itemisatthevendorfacility.CompletionofthatTPSisthenoneoftheitemsrequiredpriortoend-itemacceptanceandshipmenttothecustomer.VerificationbyAnalysismaybeperformedviaasingleTPS–aseparateTPSforeachverificationbyAnalysis.VerificationorvalidationbyTestmaygrouplikerequirementstogetherinthesameTPS,i.e.,leaktestsformultiplejoints/welds.
TPSscalloutchecklistsandproceduresandcontainspecificactionstobeperformed.Fromasystemverificationorsystemvalidationactivitystandpoint,TPSsarepreparedbasedontheinformationcontainedintheSIVVPlan,VRDSs,andTDSthattheTPSisbeingpreparedtosatisfy.AllTPSlineitemsinvolvingasystemverificationorsystemvalidationactivityforaspecificrequirementmustincludeamandatoryinspectionpoint(MIP)thatincludesaQualityAssurance/QualityEngineer(QA/QE)stamp.
Copyright©2016byRequirementsExperts. 27
Wrap up and Parting Thoughts Systemverificationandsystemvalidationarealargepartofsystemdevelopment.Secondonlytorework,aproject’sbiggestcostsareinvolvedinsystemverificationandsystemvalidation.Failingtoaddresssystemverificationandsystemvalidationearlyintheprojectlifecyclecanresultinunpleasantsurprisestowardstheendoftheproductlifecyclewhensystemverificationandsystemvalidationactivitiesbegin,puttingyourprojectathighriskofcostoverrunsandscheduleslips.
Westartedwithadiscussiononthetermsverificationandvalidationandmadethepointthatthesetermsareambiguousunlessamodifierisused(requirement,design,orsystem)thatclearlycommunicatesthecontextinwhichthetermrefers.
Theemphasisoftherestofthispaperwastopresentbestpracticesandtoolsandprovenmethodsprojectteamscanusetoaddressplanningforsystemverificationandsystemvalidationfromthebeginningofaprojectandavoidtherisksofnotdoingso.
Thecasewasmadeconcerningtheimportanceofplanningforsystemverificationandsystemvalidationstartingatthebeginningofthesystemlifecyclefromariskmitigationstandpointandtheimportanceofprojectmanagementensuringthatrequirementdevelopment,management,anddesignprocessesaddresssystemverificationandsystemvalidationactivityrisks,cost,andschedule.
Inconclusion,avoidtherisksassociatedwithfailingtoadequatelyplanforsystemverificationandsystemvalidationthroughouttheprojectlifecyclebyaddressingthefollowingbestpractices:
• Duringscopedefinition,involvethosepersonnelassociatedwithsystemverificationandsystemvalidationwiththedevelopmentofasetofscenariosoroperationalconceptsusedtosuccessfullycompleteneededsystemverificationandsystemvalidationactivities.ThisknowledgeisdocumentedinthePMP,SEMP,andMIVVPlan.
• Addresssystemverificationandsystemvalidationearlyinthesystemdevelopmentlifecycleasariskmitigationactivityfromaprojectmanagementperspective.
• Ensuringthatyoubeginwithverifiablerequirementsisthekeytoreducingrisklaterinyoursystemdevelopmentlifecycle.Whiledevelopingrequirements,alwaysthinkaheadtosystemverification.Thisimproveseachrequirement’squalityandreducestheriskofunpleasantsurpriseslaterinthesystemdevelopmentlifecycle.
• Usethetoolsdiscussedinthepaper(PIVVplan,RVM,VRDS,TDS,andTPS)tofacilitateplanningforandcompletingsystemverificationandsystemvalidationactivities.
• Ensureprojectmanagementisinvolvedinkeydecisionsassociatedwithbalancingtherisksagainstcostandschedulewhenselectingsystemverificationandsystemvalidationmethodsandthendeterminingatwhatlevel,phase,facilityorlocation,thesystemwillbeverifiedthatitmeetsitsrequirementsitwasdesignedto.
Copyright©2016byRequirementsExperts. 28
Addressingsystemverificationandsystemvalidationearlyiskeytodeliveringawinningproduct–thefirsttime!!!
Copyright©2016byRequirementsExperts. 29
References and Further Reading 1. Hooks, I. F. and Farry, K. A., Customer-Centered Products: creating successful products
through smart requirements management; AMACOM Books, NY, NY, 2001. 2. Requirements Experts, Requirements Development and Management, Seminar Workbook.
2012. 3. Larson, Kirkpatrick, Sellers, Thomas, and Verma, Applied Space Systems Engineering,
McGraw-Hill, 2009. 4. Engel, A., Verification, Validation, and Testing of Engineered Systems, Wiley, 2010 5. INCOSE, Systems Engineering Handbook - a guide for system life cycle processes and
activities, Version 3.2, INCOSE-TP-2003-002-03.1, January 2010, ed, Cecilia Haskins. 6. ISO/IEC 15288, System Engineering-System Life Cycle Processes, October 2002. 7. NASA, System Engineering Handbook, SP-2007-6105, Rev. 1, December 2007.
<http://education.ksc.nasa.gov/esmdspacegrant/Documents/NASA%20SP-2007-6105%20Rev%201%20Final%2031Dec2007.pdf>.
8. NASA, Systems Engineering Processes and Requirements, NPR 7123.1A, March 2007 <http://nodis3.gsfc.nasa.gov/displayDir.cfm?Internal_ID=N_PR_7120_005D>.
9. NASA, Space Flight Program and Project Management Requirements, NPR 7120.5D, March 2007. <http://nodis3.gsfc.nasa.gov/displayDir.cfm?Internal_ID=N_PR_7120_005D>.
10. NASA, Space Flight Program and Project Management Handbook, NPR 7120.5, February 2010. < http://www.nasa.gov/pdf/423715main_NPR_7120-5_HB_FINAL-02-25-10.pdf >.
11. NASA, Ares 1-X System Verification Requirements Document (VRD0 for Ares 1-x Flight Test Vehicle (FTV), AI1-SYS-VRD, version 2.02, December 9, 2008
12. Software Engineering Institute, CMMI for Development – Improving processes for better products, Version 1.3, CMMI Product Team, Carnegie Mellon, August 2006.
13. Wheatcraft, L. S., and Hooks, I. F., Scope Magic, 2001. <http://www.complianceautomation.com>.
14. Wheatcraft, L. S., The Importance Of Scope Definition Prior to Developing Space System Requirements. INCOSE INSIGHT, Vol. 4 Issue 4, January 2002. <http://www.complianceautomation.com>.
15. Wheatcraft, L. S., Delivering Quality Products That Meet Customer Expectations. Published in CrossTalk, The Journal of Defense Software Engineering, January 2003, Vol. 16 No. 1. <http://www.complianceautomation.com>.
16. Wheatcraft, L. S., Developing Requirements for Technology-Driven Products. Presented at INCOSE 2005, July 2005. <http://www.complianceautomation.com>.
17. Wheatcraft, L. S., Everything You Wanted to Know About Interfaces – But Were Afraid to Ask, Presented at INCOSE, July 2010.
18. Wheatcraft, L. S., Triple Your Chances of Project Success - Risk and Requirements. Presented at INCOSE 2011, June 2011 and NASA PM Challenge 2012, February 2012.
Copyright©2016byRequirementsExperts. 30
BIOGRAPHY Lou Wheatcraft has over 40 years experience in the aerospace industry, including 22 years in the United States Air Force. Over the last 15 years, Lou has worked for Compliance Automation (dba Requirements Experts), where he has conducted over 160 seminars on requirement development and management for NASA, Department of Defense (DoD), and industry. Lou has had articles published in the International Council of Systems Engineering (INCOSE) INSIGHT magazine and in DoD’s magazine, CrossTalk.
Lou has made presentations at NASA’s PM Challenge, INCOSE’s International Symposium, and at the local Project Management Institute (PMI) and INCOSE Chapter Meetings. Lou has a BS degree in Electrical Engineering, an MA degree in Computer Information Systems, an MS degree in Environmental Management, and has completed the course work for an MS degree in Studies of the Future.
Lou is a member of INCOSE, co-chair of the INCOSE Requirements Working Group, a member of PMI, the Software Engineering Institute, the World Futures Society, and the National Honor Society of Pi Alpha Alpha. Lou is the recipient of NASA’s Silver Snoopy Award and Public Service Medal and was nominated for the Rotary Stellar Award for his significant contributions to the Nation’s Space Program.
Lou also maintains a blog on requirement development and management topics. You can access his blog articles at http://www.reqexperts.com/blog
If you have further questions, please feel free to contact the author at: [email protected]
Appendix A: Verification Requirement Definition Sheet (VRDS)
Copyright©2016byRequirementsExperts. 31
[Program or Project Name] [System/Subsystem/Component Name]
Verification Requirement Definition Sheet [Project Logo]
[Requirement Number]
Verification Requirement Number: VR [Rqmt Number] Parent Requirement(s): [Rqmt Number(s)] RequirementText:
System Verification Requirements Verification method (VM): [ ] Test [ ] Demonstration [ ] Inspection [ ] Analysis Descriptionofverificationactivitiestobeperformed:The[VM]shallconsistof…………
SuccessCriterion:Verificationshallbeconsideredsuccessfulwhenthe[VM]shows…...
Rationale:[ReasonfortheVMtobeused]:
System Verification Implementation
Lifecycle Phase: System: [ ] Manufacturing [ ] Build up Integration: [ ] Sys-Sys Interfaces [ ] System Acceptance
Applicable documents: TPS xxxxxxxxx; Drawing: Num: xxxxxxxxxxx Rev: x Date: xx/xx/xxxx Completed children requirement VRDSs: xxxxxxxxxxx
Nonconformance history: N/A
Closure data/documentation required: Signed off TPS and data. Electronic/written confirmation that the requirement verification activity has been successfully completed and the defined success criteria has been achieved. EventsprecedingtheSystemVerificationActivity:xxxxxxxxxEstimatedDurationoftheSystemVerificationActivity:[TBS]
Closure Of the VR [Rqmt Number]
Date Closed: mm/dd/yyyy [ ] Safety Critical (Y/N) xxxx Quality Engineer:
[Name]
xxxx System PM:
[TBS]
xxxx Chief Engineer:
[Name]
Customer Systems Engineer
[Name] Form:VerificationRequirementDefinitionSheet
Appendix A: Verification Requirement Definition Sheet (VRDS)
Copyright©2016byRequirementsExperts. 32
InstructionsforfillingoutaVRDSThissectionprovidestheinformationneededtofillouttheVRDSforms.Theseformsareusedtospecifythesystemverificationrequirementsastailoredforthesystemverificationactivitiestoprovethesystemmeetstherequirementsthatdrovethesystem’sdesign.• VRDSTitleThetitlecontainsthenameoftheComponent/Subsystem/Systemtherequirementconcerns• RequirementtheSystemVerificationisaddressingThenumberingofeachVRDSiscomprisedoftherequirementnumbercontainedinSystemSpecwithaVR-insertedatthebeginningofthenumber.Includetherequirementtextalongwithanynotesorrationale.• VerificationMethod
• ThemethodisselectedfromDemonstration,Analysis,Inspection,andTestasstatedintheRVMs.
• TheVerificationMethodidentifiesthemethodtoobtainthemeasureofthesuccesscriterion.Supportingactivitiesmaybecalledout(e.g.,“analysisandtest,”incaseswherethesuccesscriterionisaMonteCarlosimulationsupportedbyinputfromtestresults).(Generally,mostifnotallofthefouractivitieswillbeinherentinanyverificationmethod,soitispreferredtofocusonthefinalmethodthatgeneratesthemeasureofthesuccesscriterion,evenwhenmultipleactivitiesareinvolved.)(Rememberthatthemethodsandactivities,maysharethesamenames,buttheconceptsaretotallydifferent.)
• VerificationRequirementsTheverificationrequirementsarerequirementsleviedontheorganizationresponsibleforperformingtheverification.Verificationrequirementsaddresstheactivitiesthatmustbeperformedtoobtaintheinformation(proof)neededtoverifythesystemmeetstherequirementaswellasastatementofthesuccesscriterion.TheserequirementswillbeimplementedviaaTPSorotherformofWorkAuthorizationDocument(WAD).
• Descriptionofverificationactivitiestobeperformed• Oneormore“shall”statements(requirements)statingthemeasure(a
parameter,quantity,orcondition)oftheverificationsuccesscriterionandhowthismeasureisobtained.
• Thispartstateshow(whichactivities)willbeusedtoobtain(generally,notcallingoutaspecifictool)valuesorstatesforthismeasure.
• Thispartstatesrequiredassumptionsandconditionsfortheverification.• Necessaryconditions,assumptions,andotherrequiredstatesorinputsfor
obtainingthemeasureofthesuccesscriterionarestatedclearlyin“shall”statements(requirements).
• Anynecessaryconstraints,certifications,orrequiredtrainingforverificationexecutionpersonnelareidentifiedin“shall”statements(requirements).
Appendix A: Verification Requirement Definition Sheet (VRDS)
Copyright©2016byRequirementsExperts. 33
• SuccessCriterion• Includeasingle“shall”statement(requirement)statingthesuccesscriterion.This
successcriterionshouldbethesameasstatedintheVRMs.• Thesuccesscriterionincludesthevalueorstateagainstwhichthesuccess
measureobtainedfromtheverificationprocessactivitiesiscomparedtoindicatesuccessfulverification(e.g.;measuredweightshallbelessthan133pounds13ounces;ortelemetryreceivedinthecontrolroom).
• Thesuccesscriterionneedstobemeasurableorobservablebysomeprocess.• Asimple“yes/no”answerisallthatisrequiredtodetermineifthesuccess
criterionhasbeenmet.• Themeasureforthesuccesscriterionisstatedforthe“asbuilt”
implementationofthesystemrequirement.• Nospecialskillsorsubjectmatterexpertisearerequiredtomakethedecision
whetherthesuccesscriterionhasbeensatisfied.• Rationale
Therationaleisan“attribute”oftheverificationrequirements.OnerationaleiswrittenontheVRDSformforeachSystemRequirement,butitappliestoalloftheverificationrequirementsspecifiedonthesheet.Thefocusoftherationalestatementincludesanexplanationandjustificationfor“why”thespecificverificationmethod,successcriterion,measure,constraints,conditions,andassumptionshavebeenselected.Therationaleisespeciallyimportantwhenaspecificverificationmethod“choice”isnottheobviousorexpectedone.Forexample,aspecificsystemrequirementmightbeonethatisuniversallyconsideredtorequirerigorousverificationtesting.However,itmightbeverifiedbyalessstringentmethod(e.g.analysisorinspection)iftherisksofnottestinghavebeenfullyconsideredandacceptedbytheproject.
• VerificationImplementationTheVerificationImplementationportionoftheVRDScontainsthefollowingtypesofinformation:
• LifecyclePhaseTheVerificationPhasestatesinwhichsystemintegrationlifecyclephasethesystemwillbeverifiedtohavemettherequirement.
• ApplicableDocumentsAnydocumentsdealingwiththesystemverificationforthisrequirement.Theseincludethespecificprocedures(TPSorotherWAD)usedtodotheverificationinthecaseofatestordemonstration.Thesealsoincludelowerlevelchildrenrequirementswhoseverificationactivitiesmustbecompletedandsignedoffbeforethesystemverificationactivitiesfortherequirementcanbeperformed.
Appendix A: Verification Requirement Definition Sheet (VRDS)
Copyright©2016byRequirementsExperts. 34
• NonconformancehistoryNonconformancehistoryincludesthehistoryofpreviousactivitieswhentheabilityoftheSystemtomeetthisrequirementwasnotabletobeproven.
• Closuredata/DocumentationRequiredStatewhatclosuredataand/ordocumentationisrequiredtoprovethesystemhasmeetthisrequirement.
• Event(s)precedingVerificationActivityStateanyeventsthatmustbecompletedandindicatethesystemstateorconfigurationthatmustbeachievedpriortobeingabletoperformsystemverification.Thisincludesverificationthatchildrenrequirementshavebeenmet,installationofthesysteminthefacility,assembly,inspections,checkouts,etc.
• EstimatedDurationoftheVerificationActivityProvideanestimateofthetimeneededtocompletetheactivitiesassociatedwithsystemverificationforthisrequirement.Incaseswhereoneactivitywillinvolvethesystemverificationformultiplerequirements,statethetimetocompletethatactivity.
• ClosureoftheVerificationRequirementThissectioniscompletedoncethesystemverificationactivityhasbeencompletedandtheclosuredataandclosuredocumentationhasbeencompletedanddocumentedinthissectionoftheform.Enterthedateclosed(datetheformissignedbyallfourparties)andindicatewhetherthisverificationincludesasafetycriticalrequirement.EachofthedefinedpartiesneedtosignofftheVRDSindicatingtheiracceptanceoftheverificationactivityresults(proofthesuccesscriteriahasbeenmet)andtheirconfirmationthatsystemmeetsthestatedrequirement.
Appendix B: Task Definition Sheet (TDS)
Copyright©2016byRequirementsExperts. 35
[Program or Project Name] [System/subsystem/component Name]
Task Definition Sheet [Project Logo]
Task (Inspection/Test/Checkout) Title:
Task Number: Project [System 3 Ltr designation]-XXX TaskDescription:(Purposeofthistask–whyisitneeded?Includeadescriptionoftheequipment/components/assemblies/subsystem/systemsinvolvedinthetask.)
TaskObjectives:(Expectedoutcomesasaresultofdoingthistask.IncludeVRDSsthatareaddressedbythistask.)
Hazardous operation? [ ] Y [ ] N Hazard Analysis Needed? [ ] Y [ ] N
Type of Task: [ ] Mechanical [ ] Integration [ ] Interface [ ] Power [ ] Command/Data [ ] Pressure/Leak [ ] Other: Task Methodology: [ ] Test [ ] Demonstration [ ] Inspection [ ] Analysis LifecyclePhase:System:[]Manufacturing[]BuildupIntegration:[]Sys-SysInterfaces[]SystemAcceptance
Schedule:(Dateswhentaskwillbeperformed.)Start:End:Duration:PredecessorTask(s):(Tasks/installationsthatmustbecompletebeforethistaskcanbeperformed.)
Successor/ParentTask(s):(Tasks/installationsthatrequirethistaskbecompletedbeforetheycanbeperformed.Isthistaskpartofalarger(parent)task?Whichtask?)
Constraints:(Anythingthatlimitstheperformanceofthetask.Alsoincludeimpact/constraintsonotheractivitiesduringtheperformanceofthistask.e.g.,nootherworkinthefacilitywhilethistaskisbeingperformed.)
Codes/Standards: (list any codes or standards this task must be compliant with):
Resources Needed: (utilities (steam, power, LN2, air), other Systems, Portable Equipment, Leak Test Unit, etc. during the performance of this task.)
Documentation: Checklists/Procedures: [ ] Existing [ ] To be Developed [ ] Use Vendor Procedure
Organizations/Personnel: (who will be involved in the performance of this task?) [ ] Project Engineer, [ ] Test Director, [ ] Supplier/Vendor, [ ] Safety, [ ] Quality, Technician, [ ] Other:
Training/Certifications: (State any training/certifications required by personnel involved in the task.)
Form:TaskDefinitionSheet
Appendix B: Task Definition Sheet (TDS)
Copyright©2016byRequirementsExperts. 36
InstructionsforfillingoutaTDSThissectionprovidestheinformationneededtocompletetheTDSformsusedtospecifythetoplevelinformationneededtodevelopanintegratedverification,acceptance,andreadinessschedule.• Task(Inspection/Test/Checkout)Title:Includetitlethatclearlycommunicatesthenature
ofthetask.• TaskNumber:Includeauniqueidentificationnumberforthetask.Sequentiallynumber
eachTDSforyoursystem.• TaskDescription:Clearlystatethepurpose/reasonforthistask–whyisitneeded?Include
adescriptionoftheequipment/components/assemblies/subsystem/systemsinvolvedinthetask.Whatdoesthetaskconsistof?(startthesentencewithaactiveverb-test,checkout,inspect,etc.
• TaskObjectives:Listtheexpectedoutcomesfromtheperformanceofthistask.Ifverificationisanexpectedoutcome,listtherequirementsthatwillbeverifiedasaresultofdoingthistask.(Note:UpdateanyapplicableVRDSswiththetasknumberofthisTDStoprovidebi-directionaltraceabilitybetweentheverificationrequirementsandthetaskthatwillresultinverification.)
• Hazardousoperation?Checktheappropriateblocktoindicateahazardousoperationandwhetherahazardanalysisisneeded.
• TypeofTask:Checktheappropriateblocktoindicatethetypeoftask(Mechanical,Integration,Interface,Power,Command/Data,Pressure/Leak.Ifnoneoftheseisappropriate,writeinthetypeoftaskinthe"Other"field.
• TaskMethodology:Checktheboxthatbestindicatestheoverallmethodologytobeusedtocompletethistask.(Test,Demonstration,Inspection,Analysis).
• LifecyclePhase:Indicatethelifecyclestagewhereyouplantocompletethistask.• Schedule:Indicatethedateswhenyouplantoperformthetaskandexpectedoverall
duration(hours/days).ThesedatesmustbeconsistentwiththeotheractivitiesincludedintheProject’sMasterSchedule.
• PredecessorTask(s):Indicatewhichprerequisitetasksmustbecompletedbeforethistaskcanbeperformed.
• Successor/ParentTask(s):Indicatewhichtaskscannotbecompletedbeforethistaskisperformed.Indicatewhetherornotthistaskpartofalarger(parent)task?Ifso,indicatewhichtask.
• Constraints:Listanyconstraintsontheperformanceofthistask.Includeanythingthatlimitstheperformanceofthetask.Alsoincludeimpacts/constraintsonotheractivitiesduringtheperformanceofthistask.e.g.,nootherworkintestareawhilethistaskisbeingperformed.
• Codes/Standards:Listanycodesorstandardswithwhichthistaskmustbecompliant.• ResourcesNeeded:Listanyresourcesneededtoperformthistask(utilities:steam,power,
GN2,air)aswellasanyotherSystems,PortableEquipment,LeakTestUnit,etc.thatareneededduringtheperformanceofthistask.
Appendix B: Task Definition Sheet (TDS)
Copyright©2016byRequirementsExperts. 37
• Documentation:Checklists/Procedures:Indicatewhetherthereisarequirementtodevelopnewchecklistsand/orproceduresbeforedoingthistaskorwhetherexistingchecklistsorprocedurescanbeused.Indicatewhetheryouareplanningtouseanyvendorprocedurestocompletethistask.
• Organizations/Personnel:Indicatewhichkeyorganizationsneedtobeinvolvedintheplanningandperformanceofthistask.(ProjectEngineer,TestDirector,Supplier/Vendor,Safety,Quality,Technician,TestOperations,Others?
• Training/Certifications:Stateanytraining/certificationsrequiredbypersonnelinvolvedinthetask.