catching bugs in the internet of things - from art to science · tugraz institute of software...

tugrazInstitute of Software Technology

Catching Bugs in the Internet of Things -from Art to Science

Bernhard K. Aichernig

Institute of Software TechnologyGraz University of Technology, Austria

Bozen - Bolzano , 17 Nov 2016

B.K. Aichernig Catching Bugs in the Internet of Things - from Art to Science1 / 24


Dependability of the IoT

We need a science to make the IoT as dependable as the power grid.



Dependable Things

Things of high quality come with a warranty.

Are we ready to provide a warranty on our connected things withrespect to

I security?I safety?I correctness?I reliability?I availability?I maintainability?



Agenda

I Quality of Things in the IoTI Dependable Things ProjectI Model-based TestingI Learning-based TestingI Results on MQTT Brokers



Limited Warranties

I Cisco: “... In no event does Cisco warrant that theSoftware is error free or that Customer will be able tooperate the Software without problems or interruptions....”(http://www.cisco.com/public/limited-warranty.html)

I Skydrop Sprinkler Controller: “This warranty does notcover consumable parts, including batteries, unlessdamage is due to defects in materials or workmanship ofthe Product, or software (even if packaged or sold withthe product).”(https://www.skydrop.com/warranty/)

Marketing withApp


http://www.cisco.com/public/limited-warranty.html

https://www.skydrop.com/warranty/


Pollution of the Internet

I Distributed denial-of-service attack from IoT (21 Oct 2016)I on domain name provider DYNI Twitter, Pinterest, Reddit and PayPal went down for most of a dayI DYN estimated 100,000 malicious thingsI Source of Attack: Mirai botnet

I Mirai is malware that attacks vulnerable IoT devicesI scans for standard factory default usernames and passwordsI infected things listen to control server of the botnet

Bruce Schneier. Your WiFi-connected thermostat can take down thewhole Internet. We need new regulations. The Washington Post, 3 Nov2016.



Mirai Infection Map

Source: https://intel.malwaretech.com/botnet/mirai(14 Nov 2016, 13:18)


https://intel.malwaretech.com/botnet/mirai


1st LEAD Project of TU Graz

dependablethings.tugraz.at

I Excellence initiative funded by TU GrazI 2 faculties, 10 key researchers + 10 PhD studentsI Initial phase: 2016–2018I Research questions:

I Systematic design for dependability?I Which provable guarantees can be given?I Which practical assumptions can be made?I Models of environment and system?I Verifying models and assumptions?I Compensation across components?


dependablethings.tugraz.at


Multidisciplinary Team of Key Researchers

I SP1: Dependable Wireless

I Wolfgang Boesch: Microwave EngineeringI Kay Römer: Embedded NetworkingI Klaus Witrisal: Wireless Signal Processing

I SP2: Dependable Computing

I Marcel Baunach: RT Operating SystemsI Stefan Mangard: Embedded Security

I SP3: Dependable Composition

I Bernhard Aichernig: Model-Based TestingI Roderick Bloem: Formal VerificationI Franz Pernkopf: Machine Learning

I SP4: Dependable Networked Control

I Martin Horn: Control TheoryI Gernot Kubin: Information Theory



Catching Bugs: Objective I

Writing good test cases is hard!

Don’t write test cases,

generate them!

(John Hughes)



Model-based Testing

Requirements

Model

Test-CaseGenerator

AbstractTest Cases

Test DriverSystemUnder Test

formalise

testspass

satisfies

Automated tasks:

I model verification

I test-case generation

I test-case concretion

I test-case execution

I assignment of verdicts

Manual tasks:

I (requirements analysis)

I model creation

I model validation

I concretion implementation



MoMuT ToolsMoMuT

I is a family of tools implementing Model-based Mutation Testing.

I is jointly developed and maintained by AIT and TU Graz

I has been applied in industry: AVL, Thales Railways, Infineon, Volvo.

I supports different modelling styles:

I MoMuT::UMLI MoMuT::OOASI MoMuT::TAI MoMuT::Reqs

www.momut.org

Bernhard K. Aichernig, Jakob Auer, Elisabeth Jöbstl, Robert Korosec, Willibald Krenn, RupertSchlick, Birgit Vera Schmidt: Model-Based Mutation Testing of an Industrial MeasurementDevice. TAP 2014: 1-19


http://www.momut.org/


Modelling is Hard: AVL489 Particle CounterAVL489

isReady

isBusy

Pause_0

send SPAU state /entrysend StatusBusy; set Busy /exit

Standby_1

send STBY_state /entrysend StatusBusy; set Busy /exit

Active

Purging_Pause_12

send SPUL_state /entry

Purging_Standby_12

send SPUL_state /entry

Response_14

send SEGA_state /entry

Leakage_11

send SLEC_state /entry

Integral_9

send SINT_state /entrysend StatusBusy; set Busy /exit

Measurement_2

send SMGA_state /entrysend StatusBusy; set Busy /exit

ZeroGas_10

send SNGA_state /entrysend StatusBusy; set Busy /exit

Manual

set Manual /entry

Remote

unset Manual /entry

DilutionSelection [ not Manual and not Busy ] / set Dilution

LeakageTest, ResponseCheck [ not (oclIsInState(Standby_1)) and not Manual and not Busy ] / send RejectNA

SetPurge [ not (oclIsInState(Pause_0) or oclIsInState(Standby_1)) and not Manual and not Busy ] / send RejectNA

SetZeroPoint [ not oclIsInState(Active::Measurement_2) and not Manual and not Busy ] / send RejectNA

StopIntegralMeasurement [ not oclIsInState(Active::Integral_9) and not Manual and not Busy ] / send RejectNA

StartMeasurement [ not (oclIsInState(Standby_1) or oclIsInState(Active::Integral_9)) and not Manual and not Busy ] / send RejectNA

StartIntegralMeasurement [ not (oclIsInState(Active::Measurement_2) or oclIsInState(Active::Integral_9)) and not Manual and not Busy ] / send RejectNA

when Busy

30 [ not (oclIsInState(Active::Response_14) or oclIsInState(Active::Purging_Standby_12) or oclIsInState(Active::Leakage_11) or oclIsInState(Active::ZeroGas_10) or oclIsInState(Active::Purging_Pause_12)) ] / set not Busy - send StatusReady

LeakageTest, ResponseCheck, SetPurge, SetZeroPoint, StopIntegralMeasurement, SetStandby, StartMeasurement, StartIntegralMeasurement, SetPause, DilutionSelection [ not Manual ] / send RejectBusy

SetStandby [ not Busy and not Manual ]

SetPurge [ not Busy and not Manual ]

SetPause [ not Busy and not Manual ]



10


SetPurge [ not Busy and not Manual ]

LeakageTest [ not Busy and not Manual ]

StartMeasurement [ not Busy and not Manual ]

ResponseCheck [ not Busy and not Manual ]

10

10

10


StartIntegralMeasurement, StopIntegralMeasurement, StartMeasurement [ not Busy and not Manual ]StartIntegralMeasurement [ not Busy and not Manual ]

SetZeroPoint [ not Busy and not Manual ] 10

/ send Offline

SetRemote / send Online

DilutionSelection, LeakageTest, ResponseCheck, SetPurge, SetZeroPoint, StopIntegralMeasurement, SetPause, SetStandby, StartMeasurement, StartIntegralMeasurement / send RejectOF

SetManual

SetManual / send Offline

SetRemote



Catching Bugs: Objective II

Don’t create models,

learn them!



Learning of Models

Requirements

Requirements

Model

Test-CaseGenerator

AbstractTest Cases

Test DriverSystemUnder Test

ReferenceSystem

Learner

formalise

testspass

satisfies

satisfies /“defines”

conforms to



Minimally-Adequate-Teacher Framework (1) –Learning a Regular Language

TeacherLearningAlgorithm

Equivalence Query (Hypothesis Model)

Yes / Counterexample

Membership Query

Query Answer

BuildHypothesis

Angluin’s L∗-AlgorithmDana Angluin. Learning regular sets from queries and counterexamples.Information and Computation, 75:2, 1987.

Example - Language L over alphabet {0, 1}







Membership Query

Query Answer

BuildHypothesis

Example - Language L over alphabet {0, 1}I L contains strings with even number of 0- and 1-symbols, i.e.

L = {ε, 00, 11, 0000, 0011, 1111, 0110, 1001, . . .}I Learn DFA accepting L in black-box setting







Membership Query

Query AnswerBuild

Hypothesis

Example - Language L over alphabet {0, 1}Learner Teacherε ∈ L ? yes0 ∈ L ? no1 ∈ L ? no00 ∈ L ? yes01 ∈ L ? no







Membership Query

Query AnswerBuild

Hypothesis


q0start q1

0

1

0

1

Counterexample: 11 ∈ L but not accepted







Membership Query

Query AnswerBuild

Hypothesis


q0start q1

q2 q3

0

1 0 1

0

1

0

1. . . after further queries . . . → Equivalent



Minimally-Adequate-Teacher Framework (2) –Learning a Software System

Teacher

Model-BasedTestingTool

SystemUnder

Learning

LearningAlgorithm

Equivalence Query(Hypothesis Model)


Perform Tests

All Pass /Failed Test

Output Query

Query Output

Inputs

Outputs

Outputs Inputs

I Output queries replace membership queriesI Teacher wraps system under learningI Generate tests of hypothesis to falsify hypothesis



Mealy Machines

I Deterministic finite automata with inputs and outputsI Example with inputs I = {Ping,Connect} and outputs

O = {Pong,ConnectionClosed ,ConnAck}

q0start q1

Ping/ConnectionClosed

Connect/ConnAck

Ping/Pong

Connect/ConnectionClosed

I No accepting statesI Suited for reactive systems



Applicability of Learning

I Works nicely in theoryI Many extensions:

timed, parametrised, non-deterministic . . . systemsI Useful in practise? System size limited by

I Expensive testing→ Harsh abstraction is necessary

I Can we still catch bugs in the IoT?

→ Try using an existing learning tool: LearnLib1

1http://learnlib.deB.K. Aichernig Catching Bugs in the Internet of Things - from Art to Science

19 / 24

http://learnlib.de


Finding Bugs in MQTT – BasicsI Publish/Subscribe IoT-protocolI Architecture:

MQTT BrokerMQTT Broker

MQTT-SNForwarder

MQTT-SNGateway

MQTT-SNGateway

sensor

sensor

sensor

sensor

encapsulatedMQTT-SN

MQTT-SN

MQTT-SN

MQTT-SN

MQTT-SN

MQTT

MQTT client

MQTT client

MQTT client

MQTT

MQTT

MQTT



Finding Bugs in MQTT - Approach

Implementation I Implementation J

Abstract Model MI Abstract Model MJ

Differences

Checkout(MI ) = out(MJ)

Standards Document

Bugs

Learn Learn

Analyse Manually

Repeat for all Pairs of 5 Implementations



A Simple Bug in MQTT

s0 + / Closed

s1

Con / C_Ack Con / Closed Discon / Closed

+ / +

s2

Sub / S_Ack

Con / Closed Discon / Closed

UnSub / US_Ack

+ / +

Mosquitto

s0

s1

Con / C_Ack

s2

+ / Empty

s3

Discon / Closed

+ / +Con / Empty

Discon / Closed s4

Sub / S_Ack + / Empty

Discon / ClosedCon / C_Ack

+ / Closed

UnSub / US_Ack

Discon / Closed

+ / +

HBMQTT

outmos(Connect · Connect) = ConnectAck · ConnectionClosed

outhbmqtt(Connect · Connect) = ConnectAck · Empty



Finding Bugs in MQTT - Results

I Investigated five OS implementationsI Apache ActiveMQ, emqttd, HBMQTT, Mosquitto, VerneMQ.

I Found 18 bugs in four of them, despiteI necessary abstraction → model with less than 20 statesI ignoring time-dependent behaviourI partially non-deterministic behaviourI long test duration: thousands of tests, up to 600ms per input

I Solution: use more expressive models → more testsI Smarter testing first (ongoing work)



Conclusions

I Pollution in the IoTI insecure, incorrect and unknown Things.

I Integration Testing is hard → automation neededI Modelling is hard → automata learningI Learning-based TestingI 4 out of 5 MQTT brokers were faultyI Future work

I better test selection = faster learningI non-functional properties: response time, energy consumptionI load testing, fuzzing (robustness testing)I industrial cooperation


catching bugs in the internet of things - from art to science · tugraz institute of software...

Documents