top iot testing challenges webinar with jon hagar

TopIoTTes(ngChallengesWebinar

JonD.Hagar,Consultant,[email protected]

XBOSoft Dedicated to Software Quality Improvement

Founded in 2006

We speed products to market with our expert: •  Software QA consulting •  Software testing

Global team with offices in San Francisco & Beijing

“Thorough, accurate and fast”

Copyright2017JonD.Hagar–"So4wareTestAHackstoBreakMobileandEmbeddedDevices" 2

House Rules §  Participants other than the speakers are muted

§  Ask questions in the GoToWebinar control on the right side of your screen or through Twitter @XBOSoft

§  Questions may be asked throughout the webinar - we’ll try to answer them at the end

§  You’ll receive info on recording after the webinar

Webinar Hashtag: #IOTTesting Copyright2017JonD.Hagar–"So4ware

TestAHackstoBreakMobileandEmbeddedDevices" 3

Meet Our Speakers


Jon Hagar Grand Software Testing • Author of Software Test Attacks

to Break Mobile and Embedded Devices

•  In software engineering since the beginning of time

• Combinatorial Testing Guru • Embedded testing expert • Lead project editor of ISO 29119

software testing standard •  IOT & mobile testing thought

leader

Philip Lew CEO and Founder, XBOSoft • Software quality

processes • UX design and evaluation • Mobile UX • Agile testing methods • Risk management

Lotsofopportuni8es(challenges)forTesters•  Alltheproblemsofhardware,embedded,ITandmobileso4ware

•  Worsethanthat...Headlines:TheDevicesKill

4-10trillionUSDinnext10+years

Copyright2017JonD.Hagar–"So4wareTestAHackstoBreakMobileandEmbeddedDevices"

5

IoTisthePartoftheIndustrialRevolu4on4.0

Internet of Things To Make Up Almost Half of IT Budgets By 2020

•  SpecializedIoThardware

•  Crea(ngandtes(ngmul(pleuserinterfaces—theUXchallenge

•  Third-partysoAwarefunc(onality

•  SecurityandprivacyofIoT

•  Notcovereverything:hardware,ops,systems,…..

Today’sChallenges

6


6

Embedded

IoT

Mobile-Smart Personal Computers

Big Iron (Cloud)

Many Options Huge Numbers of Devices (billions)

Numbers of Devices (millions)

Cyber-PhysicalSystems(today)

Table:IoTChallenges2Softwarefactors Opportunities PotentialSolutionsinclude

MappingtoAttacks**

Securityandprivacy Hackingtests,crowdsourcetesting chapter9

DatausageandprocessingAI,deeplearning,statsuseintesting IoTbook1,15

Ubiquitoususability UIV&V chapter7

Thirdpartyvendorimpacts Functionaltesting,regressiontesting,setbaseddesign,V&V 27

Softwarespecializedtohardware Testattheedge,Fieldtesting,V&V 5,6,7,8

Shortlifecycleswithinterfacestohardwarelifecycle Testplanningandstrategy IoTbook1

Developerwhodonottest(ordonottestwell) Independentcodecoveragetesting chapter2

Hardtofindcodeerrors Staticcodeanalysis,V&V 1

Portableoperatingsystemconfusion Compatibility,combinatorialtesting 32,IoTbook1

Allocationofprocessingtoedge,app,network,and/orcloud SystemV&V IoTbook2

TechnologySpace

PhysicalSystems(circa100,000BC)

7 Copyright2017JonD.Hagar–"So4wareTestAHackstoBreakMobileandEmbeddedDevices"

CyberSystems(1950s)

Embedded

IoT

Mobile-Smart Personal Computers

Big Iron (Cloud)

Many Options Huge Numbers of Devices (billions)

Numbers of Devices (millions)

Cyber-PhysicalSystems(today)

Today’sTestPa<ernsWillCon4nue

•  Requirementsverifica8onchecking-So4wareCoverage

•  Risk–basedanalysisandtes8ng

•  ExploratoryTes8ng(andAd-hoc)

Butyouneedmore

Copyright2017JonD.Hagar–"So4wareTestAHackstoBreakMobileandEmbedded

Devices"Copyright2017JonD.Hagar-“So4wareTestAHackstoBreakMobileandEmbeddedDevices” 8

Challenge:IoTHasUniqueSpecializedHardware

•  DifferentonWeborITwherehardwareis“generic”

•  IoT=Sensors=Controllers=Communica8on(Comm)linesandvendors


IoTHardware:WeWillNeedModellingandDeviceManagement•  Addresssystems,so4ware,andhardwaretest

•  Developerandindependenttestmodels

•  Improvesunderstandinginteams

•  InterfacestoCAD/CAMsystems?

10 Copyright2017JonD.Hagar–"So4wareTest

AHackstoBreakMobileandEmbeddedDevices"

•  Model-basedtes8ngoffersasolu8on(anduseisgrowing)

•  IoT“highintegrity”areaswillneedit

•  Model-basedtes8ngcansupport:

• Genera8onoftestcases• Earlyanalysisvalida8on• Simula8on• Verifica8on• Oraclesorjudges

Model-basedTes,ngforHighRiskDevices


•  Configura8onManagementandControl•  RecallsandUpdates•  So4wareUpdates:PushvsPull•  Ownership•  ProductEvolu8on•  DataManagementandAnaly8cs

DeviceManagement


•  MobileDevices

Challenge:HardwareandUxChallengeCrea4ngTeststoDealwithComplexity

13

•  UserInterfaces

•  IoTDevices

•  StakeholderData

•  CommChannels

•  IoTHomeProtocol

How many Tests? to address, data, configurations, devices Comm, UI, integration, etc….. 10 x 2 x 13 x 6 x 6 x 7 = 65,520 tests


ComplexityofCombina4ons-UsingtheACTSCombinatorialTool

14

Parameters: MobileDevices

[Device1,Device2,Device3,Device4,Device6,Device7,Device8,Device9,Device10]

IoTHomeProtocol [true,false]

IoTDevices

[Refrig,Stove,mircrowave,TV,frontdoor,Garagedoor,Homegaurd,Stereo,TempControl,Lights,Drapes,WaterHeater,windowopeners]

UxInterfaces [0,1,2,3,4,5]

Commproviders [Cell1,Broadband,cable,Cell2,Spacebased,Vendorgodzilla]

Data [1,0,-1,99999,-99999,100,-200]

TestCase# AndriodAppPlaqorm IoTsHome IoTDevices Routers Commproviders Data

0 Device1 false Refrig 1 Broadband 0 1 Device2 true Refrig 2 cable -1 2 Device3 false Refrig 3 Cell2 99999

3 Device4 true Refrig 4 Spacebased -99999

4 Device6 false Refrig 5 Vendorgodzilla 100

5 Device7 true Refrig 0 Cell1 -200

119 Test Sample


Challenge:ThirdPartyHardwareandSoAware

Data,Op,ons,Func,ons

IoTWillBeMadeUpofPiecePartsFromManyVendors

•  Off-the-shelf– Hardware– So4ware

•  Manyprovidersof“services”•  Piecepartsmaychangeduringdevandops•  Temp8ngtotrustBUTVerifica8onevalua8onisaMUST(usingthedata)


•  HowmanyvendorsofcomponentsandopswillbeinaCAR?

•  WillGovernmentpolicysaveus?– DefinesSAE5levelsofautonomy– Devandtest“requirements”– Willbeatleastasstrongasaircra4requirements– Who“owns”thethirdpartytes8ng?

•  Cars=LifeandDeathThirdPartyitemsmustbeTESTED

Example:USADOTSelf-DrivingCarPolicy(andstandards?)


•  ISODevsystemandso4warestandards•  ISO15288•  ISO12207•  ISO26262•  ……………………………………………

•  IEEE1012V&Vplan

•  Productstandards– Comm,lowpower,interfaces,networks,processors,plaqorms,“ecosystems”,………..

System,SoUwareandProductStandards


Love and Hate

Too Many to Name

FYI:ISO29119SoGwareTestStandard


Challenge:ThirdPartyVendorsWillBuryUsin

Data=>“Standards”=>Dev/Tes,ngOp,ons

IoTtoGenerateHugeAmountsofData(Petabyte,Exabyte,ZeHabyte,OraYoHabyte)

Currentanaly8csfocusisonmarke8ng/sales

Ifuserisatestergenera8ngdata…….

Dev–Testwillneedtousedataanaly8cs

Butforwhat?

BETTERTESTING


AsofLastYear:TestersandDataDidNOTmix

SODA–SelfOrganizingDataAnaly8cs

AINeuralNetsDeepLearningStat’s

• Tools• TesterItalkedtoDIDNOTCAREaboutusingdata!!!!

22Copyright2017JonD.Hagar–"So4wareTestAHackstoBreakMobileandEmbeddedDevices"

StepsinTes4ngThirdParty

•  Riskevalua8onofvendors/op8ons•  Selec8onofvendorop8ons

– Assess(test)thedecisionupfront– Setbaseddesignevalua8ons

•  Selectmorethanone???

– Dataanaly8cs•  Expectregressiontestsonchanges•  Verifyrequirementsandexplora8onasanongoingeffort


Whatkeepsmeupatnight(andlivinginabunker)Challenge:IoTSecurityandPrivacy(Manyexpertsthinkthesearetoppriority)

Safety-Security/PrivacyBugs

•  LeyngusersbetheTesters•  Commonprogrammingerrors(developerlevel)

•  Forgeyngaboutorlosingthehardware•  Systemenvironment•  Holesinstressandunusualcases•  IoTdenialofservicehack201625 Copyright2017JonD.Hagar–"So4wareTest

AHackstoBreakMobileandEmbeddedDevices"

A<ack-basedExploratoryTes4ngWhatisana<ack?

•  ApaHern(fortes8ng)basedonacommonmodeoffailureorinforma8onneedseenoverandover–  Maybeseenasanega8ve,whenitisreallyaposi,ve–  Goesa4erthe“bugs”thatmaybeintheso4ware–  Mayincludeoruseclassictesttechniquesandtestconcepts

•  SeeLeeCopeland’sbookontestdesignandmanyothergoodtestbooks

•  APaHern(morethanaprocess)whichmustbemodifiedforthecontextathandtodothetes8ng

•  TesterslearnmentalaOackpaHernswhenworkingovertheyearsinaspecificdomain


Devices"Copyright2017JonD.Hagarexcerptedfrom“So4wareTestAHackstoBreakMobileandEmbeddedDevices” 26

•  AOack28Penetra8onAOackTest•  AOack28.1Penetra8onSub–AOacks:Authen8ca8on—Password•  AOack28.2Sub–AOackFuzzTest•  AOack29:Informa8onThe4—StealingDeviceData •  AOack29.1SubAOack–Iden8tySocialEngineering •  AOack30:SpoofingAOacks •  AOack30.1Loca8onand/orUserProfileSpoofSub–AOack•  AOack30.2GPSSpoofSub–AOack

SecurityA<acks(from“So4wareTestAHackstoBreakMobileandEmbeddedDevices”)


InternalSecurityTes4ngHacksAreOnlyaStart

•  ExternalCrowdSourceSecurityHacking-GOOD

•  MoreinternalsaOacks–BETTER•  Badguysfind“zero”dayusingaOacks

•  Dataanaly8csandscien8ficexplora8on-BEST


MoreExamplesSo4wareAOacksforExploratoryTes8ngExcerptedfrom“So4wareTestAHackstoBreakMobileandEmbeddedDevices”

29

SoAwareTestAOackType AOackFinds NotesontheAOack

DeveloperlevelaOacks Codeanddatastructureproblems

Almostaquarteroferrorsinmobileandembeddedcanbefoundbystructuraltes(ng

ControlsystemaOacks HardwareandsoAwarecontrolsystemerrors

Manycri(calerrorsinmobileandembeddedarecenteredinthecontrollogic,forexampleanalog-to-digitalanddigital-to-analogcomputa(onproblems

Hardware-soAwareaOacks HardwareandsoAwareinterfaceissues ThesoAwareshouldbetestedtoworkwithanyuniquehardware

Communica(onaOacks Digitalcommunica(onsproblems

SoAwarecommunicateswithhardware,network,andothersoAwarewithcomplexinterfacesthatshouldbetested

TimeaOacks Time,performance,sequence,andscenarioerrors

SystemsoAwarecanhavecri(cal(mingandperformancefactorsthattes(ngcanprovidevaluableinforma(onabout

UserinterfaceaOacks Problemsbetweenmanandmachine TheusabilityofdevicesandsoAwarearecri(caltosuccess

Smart/Mobile/HardwareaOacks Issuesspecifictosmartdeviceconfigura(onsincludingcloudissues

Cloud-hybridcompu(ngcomprisesamajorityofthenewsoAwaresystemsbeingdeployed

SecuritytesthackingaOacks SoAwareerrorsthatcanexposedevicestosecuritythreats

SecurityofdevicesorsystemsisincreasinginimportanceandaOacksinclude,forexample,GPSandiden(tyspoofing

Genericfunc(onalverifica(onaOacks

Requirementsandinteroperabilityerrors BasicchecksthattestersshouldconductonsystemsandsoAware

Sta(ccodeanalysisaOacks Hardtofinderrorsthatclassictes(ngoAenmisses

CanoAenbedonebythedevelopmentgroupbutsome(mesthetestgroupmustrunthisanalysis


Mobile-IoTChallengeSummary•  Todefeatanenemy,youmustknowthebug•  TheIoTtestdataislimited,

– Whatexistshasimplica8onstotesters– DoNOThidefromtheDATA

•  Improveourtes8ngwithPaOerns–  IoTDev-TesteBook=search:LeanPub.com–Hagar(free)–  MoreChallengesatMobile/IoTConferenceSanDiegoApril23-28–  “SoAwareTestAOackstoBreakMobileandEmbeddedDevices”

•  So4warewillbeinverynearlyeverything– Goodtes8ngmaybealimi8ngfactor

30Copyright2017JonD.Hagar–"So4wareTestAHackstoBreakMobileandEmbeddedDevices"

References(my’sfavorites)•  “SoGwareTestAOackstoBreakMobileandEmbeddedDevices”

–JonHagar

–  IoTTestsBookin2017-LeanPub.com-Hagar

•  “How to Break Software” James Whittaker, 2003

–  And his other “How To Break…” books •  “A Practitioner’s Guide to Software Test Design” Copeland, 2004 •  “Computer Related Risks”, Neumann, 1995 •  “Safeware: System Safety and Computers” Leveson, 1995 •  Honorable mentions:

–  James Bach –  Cem Kaner –  Many others


Devices"Copyright2017JonD.Hagarexcerptedfrom“So4wareTestAHackstoBreakMobileandEmbeddedDevices” 31

MoreResources

•  www.stickyminds.com – Collection of test info •  www.embedded.com – info on attacks •  www.sqaforums.com - Mobile Devices, Mobile Apps - Embedded

Systems Testing forum •  Association of Software Testing

–  BBST Classes http://www.testingeducation.org/BBST/

•  Your favorite search engine

•  My web sites and blogs


Post your webinar questions on Twitter @XBOSoft Registrants will receive an email with information on where to view the recording and slides from today’s webinar. Join us to keep updated on all our webinars, reports and white papers: facebook.com/xbosoft +xbosoft linkedin.com/company/xbosoft

Check out our blog: http://xbosoft.com/software-quality-blog/ Download our free white papers: http://xbosoft.com/knowledge-center/ Email us with ideas for future webinars or questions regarding our services! [email protected]

Thank you!

Q+A

www.xbosoft.com