automatisierung im iot-testlab - asqf · 2020-01-22 · akzent 1 akzent 2 akzent 3 akzent 4 akzent...

Akzent 1

Akzent 2

Akzent 3

Akzent 4

Akzent 5

Akzent 6

Hinter-

grund /

Text 2

Michael Wagner, Sascha Kretzschmann, Axel Rennoch

ASQF Quality Day, Berlin, November 29, 2017

AUTOMATISIERUNG IM IOT-TESTLAB

Akzent 1

Akzent 2

Akzent 3

Akzent 4

Akzent 5

Akzent 6

Hinter-

grund /

Text 2

2

BERLIN CENTER FOR DIGITAL TRANSFORMATION

Akzent 1

Akzent 2

Akzent 3

Akzent 4

Akzent 5

Akzent 6

Hinter-

grund /

Text 2

3

• IoT test objects, goals, and configuration

• IoT test automation

– IoT test language TTCN-3

– IoT-Testware project

• IoT-T testlab

• IoT-Quality Engineering outlook

AGENDA

Akzent 1

Akzent 2

Akzent 3

Akzent 4

Akzent 5

Akzent 6

Hinter-

grund /

Text 2

Where are we?

INTRODUCTION

Akzent 1

Akzent 2

Akzent 3

Akzent 4

Akzent 5

Akzent 6

Hinter-

grund /

Text 2

5

• Mirai botnet, October 2016:

botnet using insecure configured IoT-devices

attack causes blackout and disruption

(e.g. Amazon, Netflix, Twitter, Github)

• Attack to Deutsche Telecom Router, November 2016:

1.200.000 router briefly stopped

vulnerable router subject to be integrated to Mirai botnet

• Wannacry, cyber attack on steel mill in Germany, …

• Recently WLAN WPA2 (KRACK: Key Reinstallation Attack), …

MOTIVATION FOR QUALITY

Akzent 1

Akzent 2

Akzent 3

Akzent 4

Akzent 5

Akzent 6

Hinter-

grund /

Text 2

6

TRENDS IN IOT

Akzent 1

Akzent 2

Akzent 3

Akzent 4

Akzent 5

Akzent 6

Hinter-

grund /

Text 2

7

Wide portfolio of competences required

Devices (sensors, HW, embedded SW)

Platforms (Cloud, platform domain knowledge)

Applications (SW, dashboard, business logic)

CHALLENGES IN IOT

IoT platforms 360+ worldwide

IoT protocols Rich selection

IP-based

non-IP based

Cellular netw.

Cellular: 4G, NB-IoT, Cat-M1,EC-GSM…[5G]; non-cellular: Wifi, LoRa, Sigfox ,Zigbee, BLE, … connectivity

MQTT MQTT-SN CoAP AMQP3 OPC UA

XMPP HTTP1 Web

Socket2 1|2|3

application IoT application logic

TCP UDP

SMS

DTLS DTLS

IPv4/IPv6(6loWPAN)

transport TLS/SSL

IoT services layer services …

Connectivity options

Throughput

Latency

Power efficiency

Packet size

Akzent 1

Akzent 2

Akzent 3

Akzent 4

Akzent 5

Akzent 6

Hinter-

grund /

Text 2

8

TEST COVERAGE BY SDO-S

Cellular netw.

Cellular: 4G, NB-IoT, Cat-M1,EC-GSM…[5G]; non-cellular: Wifi, LoRa, Sigfox ,Zigbee, BLE, …

MQTT MQTT-SN CoAP AMQP3 OPC UA

XMPP HTTP1 Web

Socket2 1|2|3

IoT application logic

TCP UDP

SMS

DTLS DTLS

IPv4/IPv6(6loWPAN)

TLS/SSL

IoT services layer … C

I

P I

C

C C

C Conforman- ce tests

I Interopera- bility tests

P PlugFest/In- teroperability test event

P

C Certification

C

http://www.open

mobilealliance.or

g

T

T Compliance tool

C C

I

C

C

C

Akzent 1

Akzent 2

Akzent 3

Akzent 4

Akzent 5

Akzent 6

Hinter-

grund /

Text 2

9

© Fraunhofer FOKUS

QUO VADIS IOT?

Cathedral

OR

Bazaar?

Akzent 1

Akzent 2

Akzent 3

Akzent 4

Akzent 5

Akzent 6

Hinter-

grund /

Text 2

10

• IoT devices,

Mikrocontroller (MCU),

Gateways (Bosch XDK, IoT starterkits)

• IoT platforms

RIOT, relayr, Thread, mbed…

service layer (oneM2M, FiWare)

• IoT protocols

Constrained Application Protocol (CoAP)

MQ Telemetry Transport (MQTT)

IoT challenges: complexity, asynchronism, long operation phase, resource constraints

STARTING: TEST OBJECTS

LPWAN

LoRa

oneM2M

Akzent 1

Akzent 2

Akzent 3

Akzent 4

Akzent 5

Akzent 6

Hinter-

grund /

Text 2

11

INTEGRATION OF SEVERAL TESTING APPROACHES

IoT Testing

Software Testing

System Testing

Security Testing

Test Automation

Protocol Testing

Akzent 1

Akzent 2

Akzent 3

Akzent 4

Akzent 5

Akzent 6

Hinter-

grund /

Text 2

12

• Less resources needed (time and money)

• Avoid human mistakes due to manually testing

• During test development and execution

• Speed-up of regression tests and product time-to-market

TEST AUTOMATION

Akzent 1

Akzent 2

Akzent 3

Akzent 4

Akzent 5

Akzent 6

Hinter-

grund /

Text 2

13

MULTIPLE TEST CONFIGURATION (SAMPLES)

Akzent 1

Akzent 2

Akzent 3

Akzent 4

Akzent 5

Akzent 6

Hinter-

grund /

Text 2

14

• Toolset (selection of available means)

Protocol tester/monitor (Eclipse Titan, Wireshark)

Test devices (RFID kit, Bluetooth test device)

GUI tester (Selenium, SikuliX, Chrome headless)

Web services tester (soapUI)

…

• Public Testsuites (in preparation)

Application of a standardized notation

Abstract and platform-independent

TESTWARE

Akzent 1

Akzent 2

Akzent 3

Akzent 4

Akzent 5

Akzent 6

Hinter-

grund /

Text 2

What do we use?

IOT TEST LANGUAGE

Akzent 1

Akzent 2

Akzent 3

Akzent 4

Akzent 5

Akzent 6

Hinter-

grund /

Text 2

16

• TTCN-3 is the Testing and Test Control Notation

• Internationally standardized testing language for formally defining test

scenarios.

• Designed purely for testing

CHALLENGE TEST AUTOMATION

testcase Hello_Bob () {

p.send(“How do you do?“);

alt {

[]p.receive(“Fine!“);

{setverdict( pass )};

[else]

{setverdict( inconc )} //Bob asleep!

} }

Akzent 1

Akzent 2

Akzent 3

Akzent 4

Akzent 5

Akzent 6

Hinter-

grund /

Text 2

17

• Distributed testing

• Automatic execution (TTCN-3 -> Java/C++) and logging

• Import and use of external data types (ASN.1, IDL, XML, JSON)

TESTING AND TEST CONTROL NOTATION (TTCN-3)

testcase Hello_Bob () { p.send(“How do you do?“); alt { []p.receive(“Fine!“); {setverdict( pass )}; [else] {setverdict( inconc )} //Bob asleep! } }

SUT

System

Tester

TTCN-3

API / Network

API / Communication

Akzent 1

Akzent 2

Akzent 3

Akzent 4

Akzent 5

Akzent 6

Hinter-

grund /

Text 2

18

• One test technology for different tests

Distributed, platform-independent testing

Integrated graphical test development, documentation and analysis

Adaptable, open test environment

• Areas of Testing

Conformance and functional testing

Interoperability and integration testing

Real-time, performance, load and stress testing

Security testing

Regression testing

• Used for system and product qualification and certification (e.g. handset certification)

DESIGN PRINCIPLES OF TTCN-3

Akzent 1

Akzent 2

Akzent 3

Akzent 4

Akzent 5

Akzent 6

Hinter-

grund /

Text 2

19

TTCN-3 HISTORY

TTCN (1992)

• published as ISO standard

• Tree and Tabular Combined Notation

• used for protocol tests: GSM, N-ISDN, B-ISDN

TTCN-2/2++ (1997)

• enhancements by ETSI MTS

• module concept, concurrency

• used for conformance tests

Akzent 1

Akzent 2

Akzent 3

Akzent 4

Akzent 5

Akzent 6

Hinter-

grund /

Text 2

20

TTCN-3 HISTORY (CONT.)

TTCN-3 (2000)

• Further development by ETSI MTS

• Testing and Test Control Notation

• Standardised test specifications:

SIP, SCTP, M3UA, IPv6

HiperLan, HiperAccess, WiMAX

3GPP LTE, 3GPP NB-IoT

OMA

TETRA

MOST, AUTOSAR

EUROCONTROL

oneM2M

Akzent 1

Akzent 2

Akzent 3

Akzent 4

Akzent 5

Akzent 6

Hinter-

grund /

Text 2

21

IMPLEMENTATION

Test Specification

TE

Systems

Under

Test

+ TTCN-3

Test

Engine

Test System

Systems

Under

Test

Systems

Under

Test

Systems

Under

Test

Communication /

Invocation

TCI Adaptors

TRI Adaptors

Automated Test

Execution and Reporting

Akzent 1

Akzent 2

Akzent 3

Akzent 4

Akzent 5

Akzent 6

Hinter-

grund /

Text 2

22

• Test technology address various protocols and interfaces

• Control of real and/or virtual devices

(special hardware-interfaces, simulators)

TTCN-3 FOR IOT

Test cases: loudness, playCDTitle

MOST Bus

Head Unit

Tester

Speaker

Amplifier / Tuner

CD Changer

(SUT)

(SUT) Tester

PLC (SUT)

Elevator

Gateway (SUT)

MQTT

Tester

TTCN-3

Earlier days Today

Test cases: remote control

Tester

TTCN-3

Akzent 1

Akzent 2

Akzent 3

Akzent 4

Akzent 5

Akzent 6

Hinter-

grund /

Text 2

What else?

CONTRIBUTION TO IOT TESTING

Akzent 1

Akzent 2

Akzent 3

Akzent 4

Akzent 5

Akzent 6

Hinter-

grund /

Text 2

24

THE CONTEXT

Akzent 1

Akzent 2

Akzent 3

Akzent 4

Akzent 5

Akzent 6

Hinter-

grund /

Text 2

25

• Supplement to running Eclipse projects

Paho, OM2M, Titan

• New project at Eclipse Foundation:

https://projects.eclipse.org/projects/technology.iottestware

TTCN-3 test suites for CoAP, MQTT, OPC-UA, LoRa?

• Assured licenses for users

• Currently in cooperation with

relayr GmbH, Ericsson, LAAS/CNRS, itemis AG, Spirent Communications,

Easy Global Market, Iskratel/Sintesio, …

THE ECLIPSE PROJECT

Akzent 1

Akzent 2

Akzent 3

Akzent 4

Akzent 5

Akzent 6

Hinter-

grund /

Text 2

26

IOT-TESTWARE

Akzent 1

Akzent 2

Akzent 3

Akzent 4

Akzent 5

Akzent 6

Hinter-

grund /

Text 2

27

SAMPLE TESTSUITE STRUCTURE: MQTT

Broker as SUT

All mandatory message data fields

Regular and illegal data

(Fixed/variable header, payload)

Protocol features

General

Connect/disconnect (session)

Subscribe/unsubscribe

Immediate publish

Last will and Testament (LWT)

Heartbeats keepAlive values

Topic

Error handling

Client as SUT

…

Akzent 1

Akzent 2

Akzent 3

Akzent 4

Akzent 5

Akzent 6

Hinter-

grund /

Text 2

28

TEST DEVELOPMENT SAMPLE: MQTT

TESTZIEL-KATALOG

Test configurations

Test Suite Structure

Test purpose (catalogue)

Test implementation (TTCN-3)

Akzent 1

Akzent 2

Akzent 3

Akzent 4

Akzent 5

Akzent 6

Hinter-

grund /

Text 2

29

TTCN-3 REPOSITORY

https://github.com/eclipse/iottestware.git

Akzent 1

Akzent 2

Akzent 3

Akzent 4

Akzent 5

Akzent 6

Hinter-

grund /

Text 2

30

MQTT BROKER EVALUATION (JULY 2017)

Broker Version PASS FAIL INCONCLUSIVE

# % # % # %

Mosquitto 1.4.14 40 88,89% 3 6,67% 2 4,44%

VerneMQ 1.1.0 39 86,67% 3 6,67% 3 6,67%

HiveMQ broker.hivemq.org 39 86,67% 4 8,89% 2 4,44%

EMQ 2.0 36 80,00% 7 15,56% 2 4,44%

lannister ? 31 68,89% 12 26,67% 2 4,44%

ActiveMQ 5.14.5 31 68,89% 12 26,67% 2 4,44%

RSMB ? 26 57,78% 17 37,78% 2 4,44%

RabbitMQ 3.5.7 21 46,67% 24 53,33% 0 0,00%

Mosca 2.5.1 19 42,22% 24 53,33% 2 4,44%

Moquette 0.10 16 35,56% 29 64,44% 0 0,00%

HBMQTT 0.9 15 33,33% 30 66,67% 0 0,00%

Akzent 1

Akzent 2

Akzent 3

Akzent 4

Akzent 5

Akzent 6

Hinter-

grund /

Text 2

31

THE TEST EXECUTION TOOL

Akzent 1

Akzent 2

Akzent 3

Akzent 4

Akzent 5

Akzent 6

Hinter-

grund /

Text 2

32

ECLIPSE TITAN PROJECT

Akzent 1

Akzent 2

Akzent 3

Akzent 4

Akzent 5

Akzent 6

Hinter-

grund /

Text 2

33

• Vulnerability scanner:

in particular for web applications, zero-day/fuzzing, consideration of data

bases, traffic/network analyser, program code scanner

• Penetration tester, e.g. “SQL injection”

• Intrusion detection tools

• Load test/Scalability

• Further utilities: Model-based testing (UML testing profile) and risk modelling

TESTWARE: SECURITY

Akzent 1

Akzent 2

Akzent 3

Akzent 4

Akzent 5

Akzent 6

Hinter-

grund /

Text 2

34

• Focus on open source test tools (Eclipse)

• Support of test suites configuration

• Providing several end devices

• Remote test service (online)

Future certification

„light weight“ selection of criteria

„self certification“ if tests are successful

consideration of operational phase

use of standardized test purpose catalogs (in preparation at ETSI)

TESTLAB (TESTING AND CERTIFICATION)

Akzent 1

Akzent 2

Akzent 3

Akzent 4

Akzent 5

Akzent 6

Hinter-

grund /

Text 2

35

PROJECT

WWW.IOT-T.DE

standardization

methods

customer

(requirements and

applications)

test and certification

pilots

new requirements

test and evaluation IoT-Testware

All project partners

Akzent 1

Akzent 2

Akzent 3

Akzent 4

Akzent 5

Akzent 6

Hinter-

grund /

Text 2

What comes next?

SUMMARY AND OUTLOOK

Akzent 1

Akzent 2

Akzent 3

Akzent 4

Akzent 5

Akzent 6

Hinter-

grund /

Text 2

37

Advanced + mature testing technology:

(open source) community:

Standardized basis (for certification):

APPROACH

Akzent 1

Akzent 2

Akzent 3

Akzent 4

Akzent 5

Akzent 6

Hinter-

grund /

Text 2

38

New Working Group (TST) will develop IoT test catalogues and specifications (not covered elsewhere).

The types of testing include conformance, interoperability, security and performance testing.

The initial technical focus will be:

IoT network layer

(communication protocols, node connectivity, edge computing etc.),

IoT layer (data accumulation and aggregation),

Application layer (interfaces, business processes etc.).

ETSI TC MTS

Akzent 1

Akzent 2

Akzent 3

Akzent 4

Akzent 5

Akzent 6

Hinter-

grund /

Text 2

39

Advanced testing technology:

(Open source) IoT-Testware (code):

Standardized IoT test purposes:

Certified Professionals for IoT

APPROACH

Akzent 1

Akzent 2

Akzent 3

Akzent 4

Akzent 5

Akzent 6

Hinter-

grund /

Text 2

41

Axel Rennoch, axel.rennoch@fokus.fraunhofer.de, phone +49 30 3463-7344

CONTACTS

Thank you for your attention!

https://www.fokus.fraunhofer.de/sqc