preview iotmap testing in an iot environment

Testing inan IoT

EnvironmentTOM VAN DE VEN, JAAP BLOEM

& JEAN-PASCAL DUNIAU

IoTMap

Free

preview

©2016 Sogeti Nederland B.V., Vianen

Editing, typesetting & ePub-production LINE UP boek en media bv, GroningenCover design Axioma Communicatie, Baarn

ISBN 978 90 75414 85 1 (book) 978 90 75414 86 8 (ePub)

No part of this publication may be reproduced and/or made public (for any purposes whatsoever) by means of printing, photocopying, microfilm, sound tape, type of electronic system, or any other data retrieval system without prior written permission from Sogeti Nederland B.V.

TrademarksTMap, TMap NEXT and TPI are registered trademarks of Sogeti Nederland B.V.

Foreword

Mapping out the Internet of Things is essentially a matter of testing. Take Google’s self-driving car. The vehicles have travelled a few million miles since the project started in 2009 but every day they drive twice as far in the lab. Before the rubber of new soft-ware hits the road, every single change is thoroughly tested in the simulator by virtually driving the total mileage history of the fleet. Autonomously and manually that is!

You may think this is typical of today’s nascent state the Internet of Things is in but continuously checking behavior and delivering software updates is already the norm during the entire life cycle of systems. From thermostats to smartwatches, turbines, toothbrushes, connected cars, and complete production plants. IoT simply means auto-mation to the max.

The road to success for Internet of Things applications and systems is paved with continuous testing. That much is clear. IDC estimates there are currently 13 billion connected things. Over forty percent of worldwide IoT revenue currently comes from manufacturing, transportation, smart cities, and consumer applications. This is gradu-ally changing.

The ongoing disruption of value chains via machine-to-machine communication is forc-ing organizations to completely rethink and retool their business. The Internet of Things and the Industrial Internet of Things have started to reshape product design, customer engagement, decision making, marketing strategies and after sales.

Within five years all industries will have IoTified their business models. The transforma-tion towards mature IoT ecosystems delivering secure services will consist of connectiv-ity, platforms, applications, and devices, blended together in so-called “cyber-physical systems of systems.”

4 Testing in an IoT Environment

Analytics, which is at the core of continuous testing, will drive intelligent services related to consumer, government, and enterprise oriented domains, ranging from shop-ping and vehicles to healthcare, energy, manufacturing, entertainment, and more.

The enabler for increased growth is pervasive wireless connectivity to the Internet from every location. As much as ninety percent of all IoT data will eventually be hosted on cloud based service platforms. The question is not whether companies will be ready – there is no way of escaping this Innovation of Technology that is already developing.

The future belongs to those who create it, so start testing now! Sogeti’s IoTmap will help you map out your journey.

Michiel Boreel, Group CTO Sogeti

5

Table of Contents

Foreword 3

1. Introduction 71.1 Reading Guide 81.2 TMap Suite 101.3 The Internet of Things grows exponentially; testing must keep up 11

2. Background 132.1 Customer Service eXperience: nine phases, five steps 142.2 Connect-Talk-Think-Act 172.3 Internet of Things 18

3. IoT Model 233.1 IoT Layers 243.2 Blueprint embedded system 27

4. An IoT Test Approach in Five Steps 294.1 Test the IoT layers 334.2 Combine IoT test expertise 344.3 Choose IoT test environment(s) 414.4 Choose the right quality attributes 474.5 Choose building blocks 48

5. Quick Tech Testing 515.1 Shift domain knowledge testing to test craftsmanship 525.2 1-2-3 test! 545.3 From pragmatic testing to a high-tech test plan 555.4 Now is the time for automation 565.5 High-tech test vision 575.6 The IoT perspective 58


6. High-Tech Test Automation 596.1 Implement test automation in five steps 636.2 RedPrint for a high-tech test automation framework 656.3 Phased model for implementation of high-tech automated testing 67

7. Building Blocks 717.1 Crowd testing 727.2 Exploratory testing 767.3 Artificial Intelligence and testing 807.4 Evolutionary algorithms 827.5 Statistical Usage Testing 867.6 End-to-end testing 937.7 Storytelling 967.8 Interoperability testing 1017.9 Testing of compliance to standards 106

8. Testing for Physics and Systems 1098.1 Introduction to TEPS 1118.2 Apply TEPS to IoT Testing 1138.3 TEPS 7 step approach 1158.4 The feedback loop 1248.5 Capitalization: continuous improvement 1258.6 Synthesis table of the process 1268.7 TEPS key areas 128

Appendix A IoT Quality Attributes 139Appendix B (High-Tech) Test Automation Checklist 141Appendix C TEPS Checklist 145Appendix D Example of TEPS in Use 151

Abbreviations/Definitions 153References 155Acknowledgements 159

1. Introduction


1.1 Reading Guide

The book you have before you gives insight into testing an IoT solution. Subjects range from defining a test strategy to specific building blocks but also touch on different test expertise areas needed to execute an IoT test project.

To guide you in reading this book a small summary is given for each chapter in Table 1. Different readers will have their own interest for specific chapters.

91. Introduction

Table 1 The reading guide gives a summary per chapter with a suggestion for whom it is

of interest.

Chapter Summary Of interest to:

1 Introduction Describes the reason this book is written. This book is an extension of the TMap Suite to the Internet of Things domain.

Everyone

2 Background Background information on the Internet of Things (IoT).

Everyone

3 IoT Model Definitions on IoT related terms and a model of an IoT solution. The base for defining the IoT test strategy.

Test manager/coordinator Project leader

4 An IoT Test Approach in Five Steps

Setting up an IoT test approach is explained in five clear steps. Each step is described and put into an IoT context. The further details of each step can be found in the next chapters of this book. This is described in more detail for some IoT test expertise areas.

Project leader Test manager/coordinator

5 Quick Tech Testing

When focusing on the Thing of the IoT solution, we often find ourselves in high-tech environments. Quick Tech Testing is a method to quickly progress to a mature and structured way of testing in high-tech environments.

Product development manager Test manager/coordinator

6 High-Tech Test Automation

With a Thing we introduce sensors, actors, electronics, and other hardware in the test scope. This is an extension for test automation in high-tech environments.

Test manager/coordinator Test engineer Development engineer

7 Building Blocks Building blocks are needed in order to fulfill the IoT test strategy. Where needed existing techniques are placed in an IoT context and new techniques are explained.

Test manager/coordinator Test engineer Development engineer

8 Testing for Physics and Systems

Testing for Physics and Systems is a seven step approach to testing in high-tech environments. It expands the first quick steps to structured testing, realized with Quick Tech Testing.

Product development manager Test manager/coordinator


1.2 TMap Suite

TMap is a household name in testing since 1995. The continuous improvement of the method has now led to the development of the TMap Suite. The TMap Suite consists of the following items:

▸ The New Approach: TMap HD – Human Driven. A quality driven test approach for modern, agile organizations. This is described in the novel Neil’s Quest for Quality.

▸ The tmap.net website. This website contains the building blocks of TMap. Together with the building blocks in this book you can build your own testing method. On TMap.net templates and checklists to support your IoT test strategy are available for download.

▸ TMap Next is a test method rooted in more traditional, process based development methods.

▸ This book expands the TMap Suite to the IoT and high-tech domain. Together with the other elements from the Suite a complete test strategy for your project, match-ing your product development methodology, can be set up.

Figure 1 Building Blocks working with the TMap Suite.

111. Introduction

1.3 The Internet of Things grows exponentially; testing must keep up

Around 2006, after Pervasive Computing, Ubiquitous Computing and indeed the Inter-net of Things, the notion of Cyber-Physical Systems (CPS) was introduced to eradicate confusion, to repair all damage done and to reconcile viewpoints. But terminology is hard to get rid of, and CPS developed as just another buzzword, next to the Industrial Internet, the Industrial Internet of Things, Industry 4.0, and – let’s not forget – the amaz-ing Internet of Everything.

We are experiencing devices we have seen in science fiction movies decades ago become a reality. Our clothes can sense our well-being, parts of body scans can be done by smart phones, predictive maintenance is done at an oil rig at open sea and cars are driving themselves. These are but examples of the high-tech devices or sys-tems we are working with today. These high-tech systems are measuring high volumes of data and exchange it with fellow systems and beyond. Systems collecting and exchanging data are of all ages. We see the amount of devices and the amount of data exchanged growing to epic proportions. This book categorizes all of this under the term Internet of Things (IoT in short).

From the science fiction examples, two types of IoT solutions can be distinguished. A set more associated with consumer electronics and a set more associated with indus-trial solutions. The latter is often referred to as the Industrial Internet of Things (IIoT). Where IIoT focuses more on high-volume “things” and big data the elements that make an IoT solution are the same. In this book we only refer to the term IoT in the context of testing.

The rapid growth of products associated with IoT (see Figure 2) is having conse-quences in how we do system design. Systems we now develop are (parts of) an IoT solution or are strongly related to the IoT. IoT brings a new experience to new and existing products. A new set of properties are available in IoT devices and we can connect them all. Combining new properties together with all the connectivity options gives us an infinite amount of situations one IoT device can be in.


Devices carrying an IoT experience and the infinite amount of possibilities are enough reason to look into the testing IoT solutions. This book is trying to find new views and ideas on testing in the IoT environment.

GROWTH IN THE INTERNET OF THINGSThe number of devices will exceed 50 billion by 2020

billionsof devices

1988

10

01992

1M1992 0.5B

20038.7B2012

11.2B201314.2B

201418.2B2015

22.9B2016 28.4B

2017 34.8B2018

42.1B2019 50.1B

2020

IoTINCEPTION

2009

1996 2000 2004 2008 2012 2016 2020

20

30

40

50

Figure 2 Forecast of the growth of IoT Things up to 2020 (source: Cisco).

2. Background


Before diving into technical details on building blocks and approaches to Internet of Things solutions, some context is needed. This book contains a list of definitions of terminology associated with the Internet of Things. Chapter 2 describes the relevant concepts regarding the Internet of Things to create a starting point for forming that all-important Internet of Things test strategy.

2.1 Customer Service eXperience: nine phases, five steps

Although most processes aiming at a better Connected Service eXperience (CSX) are only semi-automated at present – involving sales people, field service technicians, and customer care employees – they already make a huge difference in improved customer satisfaction. Across industries, the following nine main Connect-Talk-Think-Act phases are typical of many practices today:

1 A connected machine or device generates data that is transmitted using a machine data gateway.

2 The data is aggregated and analyzed. When an anomaly is detected that can impact performance, a case is created.

3 The knowledge base or expert system is queried to identify the “next best action”, e.g. skill set and part replacement.

4 The integration with the knowledge base or expert system identifies the service level and response time.

5 The integration with the HR system identifies the available co-worker with the proper skill set.

6 The integration with the inventory or PLM system (Product Lifecycle Management) ensures that the required part or service is available or ordered.

7 A ticket is created and a task is assigned.8a A field service technician receives an alert on his mobile device. He informs a co-

worker who performs the task and records the result on his mobile device, or …8b sales or customer service receive a task in their CRM system (Customer Relationship

Management). Then, sales sends an offer and/or customer support contacts the customer. The results are recorded in the CRM system.

9 The resolutions are analyzed, and the knowledge base or expert system (see 3) is updated.

152. Background

Digitally optimized connected services typically relate to knowledge bases, expert systems, PLM, and CRM systems – inside the enterprise and beyond its boundaries – tapping into partner ecosystems.

Connect-Talk-Think-Act (see section 2.2) is a colloquial translation of the corresponding 5A cycle that includes these subsequent steps of data Acquisition, data Aggregation, data Analysis, the Assignment of tasks, and the required Action to be taken:

Step A1 – Acquisition. Sensor modules extract system logs or operating data from connected machines, devices or components and transmit the data. Some of these “connected things” generate data that is not transmitted, but which requires additional action. Ideally, performance and usage data is acquired and transmitted every n seconds.

Step A2 – Aggregation. The operating data from multiple connected things is col-lected, aggregated and stored. The data is visualized. Most machine data stored nowadays is latent.

Step A3 – Analysis. Ideally, the data is being analyzed in real-time to identify issues. We distinguish between data visualization and analysis. When data is visualized, a person must monitor the data and decide if it is critical. This method is unreliable and subjective. Using historical and/or predictive tools, anomalies and issues can be auto-matically identified and trigger alerts set. Only a fully automated approach is reliable, objective, and timely.

Step A4 – Assignment. When the data indicates an anomaly or when an issue is identified that might cause a problem or represents an opportunity to eliminate dis-satisfaction in advance, ideally a case is automatically created. The most appropriate resolution is selected from the knowledge base or expert system, and assigned to the proper actor for handling.

Step A5 – Action. Interventions are performed to resolve the issue, and the process is recorded and analyzed. The results should be fed into the knowledge base or expert system.

In many cases, the actor is currently a sales representative, field service technician, customer care agent or the Thing itself as in the following scenarios:


i Customer Care provides the operator with tips and instructions.ii Sales & Marketing proactively sends the customer offers and orders, and/or …iii Technical Service is dispatched to perform maintenance or repair.iv The system sends messages, instructions, and possibly software updates directly to

the machine.

Consider these five A’s as the core of a more detailed Data Analysis Lifecycle. Research by Sogeti from 2016 reveals the following breakdown as companies are con-tinuing to embrace Internet of Things based solutions (Figure 3).

is tracking and monitoring indicators.

is into controlling and adapting behavior remotely.

13% 16%27% 44%

Embracing IoTLEVEL 1 LEVEL 2 LEVEL 3 LEVEL 4

is developing new products & services.

is using algorithms to predict: usage, failures et cetera.

Research by Sogeti, 2016.

Figure 3 Embracing the Internet of Things.

172. Background

2.2 Connect-Talk-Think-Act

Every major progress comes at a cost and the IoT is no exception. Learning by doing, we experience today what it means to Connect objects, machines, devices, compo-nents, and people; what these actors should Talk about; how we can let them Think with us and among themselves; and what Actions we consider necessary and desir-able in specific business contexts.

This Connect-Talk-Think-Act paradigm is the concise way of explaining

A what the implementation of IoT solutions would mean in a specific context,B which complexities and challenges arise, andC how the simple CTTA mantra governs the more technical 5A scope of data Acquisi-

tion, Aggregation and Analysis, the automated Assignment of tasks, and the Actions to be taken.

Sensor modulesextract system logsor operating datafrom devices,machinesor vehiclesand transmit them.

Data streams frommultiple devices,machines or vehiclesare collected and stored.

Tasks are assigned:actions are performed

& recorded.

Data is interpreted:sales or service

opportunities surface.

Acquisition

Analysis

Assignment AggregationA C T A L K

CONNEC

HINK

Figure 4 Connect-Talk-Think-Act paradigm in one overview.


2.3 Internet of Things

The ongoing miniaturization of electronic circuits, omnipresent digital networks; the rapid development of high-sensitivity sensors, advanced data processing, diminishing cost, and efficient power consumption are the main instrumental drivers behind a huge increase of interest in what is commonly known as the Internet of Things (IoT).

In industrial as well as consumer domains, a complete set of affordable IoT technolo-gies currently enables organizations in manufacturer and operator roles to automati-cally and constantly monitor the behavior of products and processes, as well as their interaction with other components and artifacts, with people, and with the narrow and broader environments they are in – from connected turbines to toothbrushes, from ambient lighting to medical equipment, and from smart energy solutions to fleet man-agement, to name a few popular areas.

In practice we see organizations look at the Internet of Things from three distinct per-spectives. Firstly, the technical complexity of attaching sensors to products, aggregat-ing the data from those sensors, analyzing them to produce relevant insights, deciding what actions are required plus the assignments of tasks. Secondly, industry specific use cases and solutions like preventive or predictive maintenance, or asset manage-ment. Thirdly, the digital transformation of the entire organization, determining a future enabled by the new technologies.

The vast and largely unexploited area between and beyond the traditional IT and OT axes of business process and industrial automation – commonly known as the Internet of Things or the Industrial Internet – is now truly open to enthusiastic exploration and reaping commercial benefit. In this “Anything Internet” space, both IT and OT meet in sensorized products and processes, ranging from connected consumer gear or “wear-ables” to industrial robots working closely with humans.

IoT can be considered as a fundamental Innovation of Technology, aka the IoT Tech Triad (Figure 5). It consists of three main components:

▸ IT (Information Tech – white collar)

▸ OT (Operational Tech – blue collar)

▸ and IoT (Internet of Things – ‘Anything Internet’ in between and beyond)

192. Background

Customer touch points

Performance management

Digital Globalization

Top line growth

Worker enablement

New Digital Business

Customer understanding

Process digitization

Digitally-modified business

CustomerExperience

OperationalProcess

BusinessModel

Digital Transformation (Capgemini/MIT)• The 5 A’s:• Technologies• & Architecture• Acquisition of data

• Aggregation of data

• Analysis of data

• Assignment of tasks

• Actions to be taken

• Security & Testing

• Industry-specific• Use Cases & Solutions• Predictive maintenance

• Smart monitoring

• People & asset tracking/mgt

• On-field operations

• Supply chain efficiency

• Assemble line optimization

• Product & service innovation

• …

Principles & Implementationof Digital Transformation

I O T

Figure 5 IT, OT and IoT triad.

These components are well positioned to blossom out as the single most important Innovation of Technology (IoT) since the World Wide Web, aiming at a hitherto unseen Connected Service eXperience (CSX) throughout industries.

Technically this requires enhanced security and testing, advanced apps, API’s and algorithms, and a clear architecture to incorporate legacy systems. Businesswise the IoT Tech Triad’s Innovation of Technology will be exploited by organizations with a his-tory that embrace a startup mentality in their operations. To them, the business innova-tion triad of Product Leadership, Operational Excellence, and Customer Intimacy will be one.

From a testing perspective, IoT in this context surely signifies Interoperability Testing. Actually, both the Cloud and the IoT Tech Triad on the one hand, plus Interoperability Testing and DevOps or continuous delivery & deployment on the other make up the new Digital Transformation bridge that all organizations have to cross. Sooner or later,


and preferably fast. But the Bridge Too Far syndrome looms and before you know it the construction you chose may crumble under your feet so to speak, as technological disruption deals its final blow. Too little too late! Wrong time, wrong place! And, please realize: this is not “just IT,” this is Business Technology, your company’s lifeblood!

Structurally, on the business level, over the last 20 years – starting with Treacy & Wiersema’s The Discipline of Market Leaders – it has all developed as is shown in Figure 6.

I O T

Customer Intimacy

Operational Excellence

Prod

uct Leadership

Inte

rn

et of ThingsIoT

Information Technology < ITOT > O

perational Technology

Service

Design

Disrupt

Innovation of Tech 1995-2015

Figure 6 Development of IT and OT; total innovation of IoT.

People today already own an average of ten IP-enabled devices, reflecting that more and more connected objects will be smart. The traditional distinction between network and device is starting to blur as the functionalities of the two become indistinguishable.

212. Background

Shifting the focus from the IoT network to the devices costs less, scales more gracefully, and leads to immediate revenues.

The systemic nature of innovation calls for coordination as the Internet (Of Things) will consist of countless cyber-physical systems. Successful innovation will depend on coor-dinating the huge range of stakeholders, systems and services in interaction-intensive environments with a permanent and seamless mix of online and real-world experi-ences and offerings. The overlay of virtual and physical will be enabled by these four technology strands:

▸ Layered and augmented reality interfaces for smartphones, wearables, industrial equipment, and all other “things”.

▸ Continuous data exchange via low-cost sensor networks and context‐aware applications.

▸ Ubiquitous connectivity and computing by technologies such as cloud and mobile.

▸ Real-time access to intelligence about virtual and physical processes and events by open, linked and big data.

The “softwarization” of networks has profound implications as the threshold for market entrants becomes drastically reduced because new software-defined operators have their platforms fully developed, executed and operated in software. This brings dra-matic capex and opex cost reduction.

In the new IoT context two basic conditions must be fulfilled. We need:

▸ Standards to prevent different interpretations and interoperability issues.

▸ Test specifications and methodologies to ensure validation and conformity to standards.

The overall challenge in IoT testing is to examine whether existing test methodologies are still valid and whether we need new methods and techniques to cope with the ele-ments that make an IoT solution (Figure 7).


Figure 7 Schematic example of elements that make an IoT solution (source: IBM).

Real-timeEnvironmental

sensors

Integratedwebcam

powerplug

PIR &dooropening smartCard reader

smartEnginegateway

Remotedisplay

devices

Internet/VPN

Testing inan IoT

EnvironmentTOM VAN DE VEN, JAAP BLOEM

& JEAN-PASCAL DUNIAU

TESTING

IN A

N IO

T ENVIR

ON

MEN

T

“ THE FUTURE BELONGS TO THOSEWHO CREATE IT, SO START TESTING NOW! SOGETI’S IOTMAP WILL HELP YOU MAP OUT YOUR JOURNEY.”– MICHIEL BOREEL

SOGETI GROUP CTO

IoTMap

The Internet of Things (IoT) gives us solutions made of a mix of expertise such as high-tech “Things”, mobile solutions and business intelligence. Not only do we need a wide range of test expertise for IoT testing but with “Things” we introduce sensors, actors, elec-tronics and other hardware to the test scope.

IoTMap gives insight into testing an IoT solution. Setting up an IoT test approach is explained in fi ve clear steps. Each step is described and put into IoT context. We put existing building blocks from the TMap® Suite in an IoT perspective and add some new ones. This book provides all the components to implement (more) structured (high-tech) testing in IoT environments.

Sogeti_Handboek_IoT_Cover-Stand.indd 1 17-03-16 12:01

preview iotmap testing in an iot environment

Documents