IN DEGREE PROJECT ELECTRICAL ENGINEERING,SECOND CYCLE, 30 CREDITS

, STOCKHOLM SWEDEN 2018

Machine Learning for Enabling ActiveMeasurements in IoT Environments

WENQING YAN

KTH ROYAL INSTITUTE OF TECHNOLOGYSCHOOL OF ELECTRICAL ENGINEERING AND COMPUTER SCIENCE

1

Machine Learning for Enabling Active

Measurements in IoT Environments

Wenqing Yan

Ericsson Supervisor: Dr. Andreas Johnsson

KTH Supervisor: Prof. Rolf Stadler

Examiner: Prof. Gyorgy Dan

9 december 2018

Abstrakt

Med den snabba utvecklingen av Internet of Things (IoT) s tillhandahller ntope-ratrerna allt fler nya tjnster inom IoT. Fr traditionella ntverkstjnster r operatrervana vid att anvnda aktiva mtningar p IP-lagret fr att bedma ntverkets end-to-end prestanda och p s stt skerstlla kvalitn p tjnsten. Med liknande metodik villde ocks mta och bedma ntverksprestanda i IoT-system. Aktiva mtningar kandock negativt pverka energi- och ntverksbelastningen, vilka r kritiska resurser frIoT-applikationer.

Den hr uppsatsen undersker ett nytt tillvgagngsstt fr att mta ntverksprestandadr underliggande statistik, ssom topologiinformation, radio-, och paketinforma-tion, anvnds fr att frutse olika mtt, ssom paketfrluster och frdrjningar. Predik-tionsfunktionaliteten r baserad p Machine Learning. I denna uppsats diskuterasven hur denna funktionalitet kan implementeras som en del av IoT proxy frntverkshanteringssystem.

Utvrdering av frutsgningsfunktionaliteten baseras p omfattande experiment iEWSN’17 testmiljn. Aktiva mtningar genomfrs genom att periodiskt skicka pro-be paket fr att uppskatta ntverksprestanda. Samtidigt samlas mtdata in frn varjentverksnod. Efter att ha samlat in data, s anvndes en ML-pipeline byggd fr attlra sig frhllandet mellan mtdata frn noderna och ntverksprestanda. Uppsatsenutvrderar specifikt tv metoder baserade p Machine Learning, nmligen linjra ochtrdbaserade regressionsalgoritmer.

Resultaten fr olika scenario och topologier visar att det nya tillvgagngssttet kanuppskatta 10% fr frdrjning och 16% fr genomsnittlig packetfrlust.

Abstract

With the explosion of Internet of Things (IoT) technology, network operators tryto provide more and more new services related to IoT. For traditional services,operators are accustomed to using IP-layer active measurements for assessingend-to-end network performance to ensure the quality of service (QoS). Sim-ilarly, they also want to use the same methods to assess end-to-end networkperformance in IoT systems. However, due to the resource-constrained IoTenvironment, intrusive active measurements may induce energy and networkoverhead, which are sensitive topics for IoT applications.

The thesis investigates a new approach where network performance metrics,such as packet loss and round-trip time, are predicted from network and en-vironmental features, such as topology information, packet statistics, and en-vironmental sensing features. The overarching goal is to lower the impact ofactive measurements adjusting into resource-constrained IoT systems. The pre-diction functionality is based upon supervised machine learning algorithms. Inthis thesis, we discuss how this functionality can be implemented as part of theIoT network management system with an active measurement proxy.

Evaluation of the predictive functionality is based on extensive experimentationon the EWSN’17 testbed. Probe packets are sent periodically to estimate net-work performance metrics. Simultaneously, device and infrastructure metricsare collected from each network node. After collecting the data, an ML (Ma-chine learning) pipeline is used to learn the relation between accessible featuresand target service metrics. To predict service-level metrics, the thesis specif-ically evaluates two prediction models based on statistical learning methodsincluding linear and tree-based regression algorithms.

The results for different scenarios and topologies show that the new approach canaccurately estimate the service-level network performance metrics for wirelesssensor networks with error rates lower than 10% for RTT and 16% for 20minaverage loss (NMAE).

Acknowledgements

I am appreciated for having the opportunity to carry out this Master’s degreeproject at the Machine Intelligence and Automation department of EricssonResearch.

I would first like to thank my thesis supervisor Dr. Andreas Johnsson fromEricsson for his continuous guidance and encouragement through my half-yearwork. He is the pilot of my first research journey, who opened the door ofacademic mysteries for me.

I would also like to express my gratitude to my KTH supervisor Prof. RolfStadler for his strict requirements and patient inculcation. He taught me whatis craftsmanship in academic research and how to be a rigorous researcher. It ismy pleasure to follow him both during the thesis work and Master’s educationin KTH.

Also, I would like to thank my examiner Prof. Gyorgy Dan for this useful adviceand careful consideration of my thesis work.

Finally, I would like to express my gratitude to my family and friends for en-couraging me in all of my pursuits.

1

Contents

1 Introduction 7

1.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

1.2 Sustainability and Ethics . . . . . . . . . . . . . . . . . . . . . . 10

1.3 Research Questions . . . . . . . . . . . . . . . . . . . . . . . . . . 10

1.4 Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

1.5 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

1.6 Structure of the Thesis . . . . . . . . . . . . . . . . . . . . . . . . 12

2 Background 13

2.1 Network Management . . . . . . . . . . . . . . . . . . . . . . . . 13

2.2 Wireless Sensor Networks . . . . . . . . . . . . . . . . . . . . . . 15

2.3 Machine Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

2.3.1 Lasso . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

2.3.2 Decision Tree . . . . . . . . . . . . . . . . . . . . . . . . . 17

2.3.3 Random Forest . . . . . . . . . . . . . . . . . . . . . . . . 17

2.3.4 Feature Reduction . . . . . . . . . . . . . . . . . . . . . . 18

2.3.5 Manual Feature Selection . . . . . . . . . . . . . . . . . . 19

3 Overview of the New Approach:Active-measurement Proxy Based on Machine Learning 20

4 Method 22

2

4.1 Experimental Design . . . . . . . . . . . . . . . . . . . . . . . . . 22

4.2 Machine Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

5 Testbed and Experiment Description 25

5.1 Testbed Description . . . . . . . . . . . . . . . . . . . . . . . . . 25

5.1.1 Testbed . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

5.1.2 Contiki OS . . . . . . . . . . . . . . . . . . . . . . . . . . 26

5.1.3 RPL in the Testbed . . . . . . . . . . . . . . . . . . . . . 27

5.1.4 TWAMP for Active Measurements . . . . . . . . . . . . . 27

5.1.5 jammer . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

5.2 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . 28

5.2.1 Configuration Parameters . . . . . . . . . . . . . . . . . . 28

5.2.2 Probe Sending Strategy . . . . . . . . . . . . . . . . . . . 30

5.2.3 Setup Plan . . . . . . . . . . . . . . . . . . . . . . . . . . 30

6 Data Description 32

6.1 Feature Set X . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

6.1.1 Feature Description . . . . . . . . . . . . . . . . . . . . . 34

6.1.2 Data Preprocessing . . . . . . . . . . . . . . . . . . . . . . 36

6.2 Target Set Y . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

6.2.1 Target Description . . . . . . . . . . . . . . . . . . . . . . 40

6.2.2 Data Preprocessing . . . . . . . . . . . . . . . . . . . . . . 40

6.3 Data Visualization . . . . . . . . . . . . . . . . . . . . . . . . . . 42

6.3.1 Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

6.3.2 Target . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

7 Evaluation 48

7.1 Model Configuration Details . . . . . . . . . . . . . . . . . . . . . 48

7.1.1 Hyperparameters . . . . . . . . . . . . . . . . . . . . . . . 48

3

7.1.2 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . 49

7.1.3 Feature Sets . . . . . . . . . . . . . . . . . . . . . . . . . . 49

7.2 Batch Learning for Different Topologies without jammer . . . . . 50

7.2.1 Target RTT . . . . . . . . . . . . . . . . . . . . . . . . . . 50

7.2.2 Target loss . . . . . . . . . . . . . . . . . . . . . . . . . . 51

7.3 Batch Learning for Scenarios with Jammer . . . . . . . . . . . . 53

7.3.1 Target RTT . . . . . . . . . . . . . . . . . . . . . . . . . . 55

7.3.2 Target loss . . . . . . . . . . . . . . . . . . . . . . . . . . 55

8 Discussion 61

8.1 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . 61

8.2 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

9 Conclusion and Future Work 65

9.1 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

9.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

Appendices 68

A Evaluation 69

A.1 RTT Prediction without Jammer . . . . . . . . . . . . . . . . . . 69

A.2 Loss Prediction without Jammer . . . . . . . . . . . . . . . . . . 70

A.3 RTT Prediction with Jammer . . . . . . . . . . . . . . . . . . . . 71

A.4 Loss Prediction with Jammer . . . . . . . . . . . . . . . . . . . . 72

4

Glossary

6LoWPAN IPv6 over Low-Power Wireless Personal Area Networks. 26

ETX Expected Transmission Count. 27

GPIO General Purpose Input/Output. 25

IoT Internet of Things. 1, 7, 14, 20, 24, 27, 61, 65

ISO International Organization for Standardization. 13

LLNs Low-Power and Lossy Networks. 26, 36

LQI Link Quality Indicator. 33

MAE Mean Absolute Error. 49

ML Machine learning. 9, 16, 22, 34, 51, 61

MRHOF Minimum Rank with Hysteresis Objective Function. 27

MSE Mean Square Error. 49

NBL Neighbor List. 36

NMAE Normalized Mean Absolute Error. 49, 62

NMSE Normalized Mean Square Error. 49

OFs Objective Functions. 27

OS Operating System. 26

OWAMP One-way Active Measurement Protocol. 14

QoS Quality of Service. 8, 13

RFE Recursive Feature Elimination. 18

RPL Routing protocol for LLNs. 27, 66

5

RSSI Received Signal Strength Indicator. 27

RTT Round-Trip Time/Delay. 14, 20, 25, 40, 50, 61, 65

SLA Service Level Aggreement. 13

TWAMP Two-way Active Measurement Protocol. 14, 21

WSN wireless sensor network. 7, 20, 22, 25, 32, 51, 61, 65

6

Chapter 1

Introduction

This thesis describes the research by the author on the subject of active per-formance measurement in wireless sensor network (WSN) and was carried outat the Machine Intelligence and Automation department of Ericsson Researchin Kista. This chapter gives an overview of the context of the thesis and briefbackground knowledge, describes the objectives and contributions, and presentsthe structure of the thesis.

1.1 Background

Today, information can be delivered anywhere using transmission technologies.Networked society [1], which is a new version of the communication world coinedby Ericsson, a global information techonology campany, has become known asthe future development direction. It is used to describe a future ecosystemwhere widespread Internet connectivity drives changes involved individuals andcommunities. In the networked society, IoT is one main technology branch,integrating physical and virtual Things into the information network [2]. Ev-ery device which benefits from an Internet connection will be linked into IoTnetwork, which means the number of networked devices will explode. Ericssonpredicts there will be around 28 billion connected devices by 2021 involved inIoT system. In the foreseeable future, IoT will provide access to information notonly from standard devices, such as desktops, laptops, smartphones and tablets,but also non-internet-enabled physical devices and everyday objects [3].

One important component for IoT is Wireless sensor networks (WSNs), mainlyused to interconnect intelligent sensors for sensing and monitoring the surround-ing information [2]. Besides, some IoT systems with actuators can change theenvironment situation based on sensed information. Traditionally, the positionof sensor nodes in WSNs does not need to be pre-determined, because previousWSNs are normally deployed in inaccessible terrains or disaster relief operations,where the nodes are randomly placed. This situation requires the network to

7

possess self-organizing capability and has high levels of redundancy to achieverobustness. Most often WSNs do not provide any performance assurances be-cause of the limitation of resources and redundancy of sensor nodes [4].

In general, traditional WSNs are a best-effort system with little considerationfor Quality of Service (QoS). However, the controlled performance scenarios inmany areas such as industrial factories and health monitoring now show anoverwhelming trend of needs. These application scenarios put forwards specificrequirements on performance, for example, industrial factory demands stricttime limits, reliability and security as a part of its QoS needs; a health monitor-ing application most certainly has accuracy and timeliness among its essentialQoS necessities.

QoS requirements are urgent, but fulfilling QoS requirements in WSNs is chal-lenging. There are two reasons. First, WSNs use open air to transmit signals.The open wireless environment is dynamic so WSNs are transitory and change-ful, which brings challenges to provide stable services. However, from the clientspoint of view, they expect stable and reliable services. As an result, it is imper-ative to solve the conflict between consistent services and the dynamic natureof the IoT environment [2]. Second, WSNs present severe hardware limitationssuch as energy, storage, processing power, communication capabilities and band-width, raising the difficulty of implementing these networks with QoS [4]. Ingeneral, achieving QoS in WSN is much more difficult [5].

In order to guarantee network QoS expectations, the network performanceshould be monitored, effectively measured and managed. Sometimes, severeproblems may lead to bad performance or even application failures in the net-work [6]. When the system malfunctions, it needs to detect and repair theproblem. Therefore, the WSN system needs a management mechanism, whichnot only enable continuously monitor and ensure that the required QoS is undercontrol but also offer support to the system’s failure analysis.

As mentioned earlier, a new range of applications and scenarios like those exist-ing in industrial factories and health care show diverse demands of performance.For example, a healthcare monitoring WSN may demand reliability, security, lowlatency and wide bandwidth. A smart industry warehouse targeting precisionmost certainly has accuracy among its essential QoS necessities. A WSN forreal-time logistic monitoring has the typical delay in its QoS parameters. Butperformance is not only the capability to accomplish a certain task but also howwell it is done in comparison to preset standards of accuracy [7]. These new-typeWSNs with controlled performance should have some different characteristics,which are related to the special QoS demands and largely depend on the ap-plications implemented in the network. As a result, there are big differencesbetween different WSNs with controlled performance.

To manage the network, obviously, the system needs network performance eval-uating metrics. There are several different performance metrics in WSNs. Someof them can be gathered easily, such as time, location, and link quality estima-tion parameters (CH, RSSI, LQI), where we use symbol X to represent them.However, others are hidden performance data, which are either measured or

8

derived and not included in the raw data. Most of them are core assessmentmetrics to evaluate the performance, such as round-trip time (RTT), averagepacket loss and energy consumption, which are not directly shown in the normalsensor data, and we use Y to express this metric set. In network management,the process to get the feature set Y instead of using direct measurements isdefined as the observability expansion. The direct method to get them in trans-port and radio access networks is using active measurement by sending probepackets into the network. More details about the processes are shown in Figure1.1. One node acts as the controller sending probe packets to the destinationnode. The destination node acts as a reflector, and will reflect the packet to thecontroller after receiving. During the back and forth process, four timestampsare recorded, from which performance metrics, such as RTT, can be derived toevaluate the network.

Figure 1.1: Example interaction using active measurement.

Active probing is an appendage for the original network. It requires additionalenergy, and occupies the network resource. Nevertheless, WSN devices only havelimited processing capability, memory, as well as power. If the additional load isadded on WSN, network capacity may reach its limits, thus the performance ofthe original sensing task will be affected and even destroyed. Generally speaking,energy consumption and network overhead are always the center of attentionin WSN research. In WSN management mechanism, it is expected that theadditional network management component will take up little network resources,so reducing the overhead and controlling energy consumption of this additionalcomponent is the fundamental goal of this research. This project aims at findinga good balance between observability and overhead in WSNs.

A large body of recent network management research is aiming towards observ-ability expanders such as predictions, inference, process models, and measure-ments, to obtain the wanted observability under the condition of limiting theadditional network load. Machine learning (ML) prediction models to achievethe network observability in WSNs unable only reduce the energy consumptionbut also the network overhead. It is an innovative underexploited research field.In the paper [8], authors prove the feasibility of implementing probe-based meth-ods to reach the observability expansion in WSNs, which provide the possibilityof further research.

This thesis project is a continuation of that paper and we plan to transformthe investigation into performance metrics prediction models, trying to findout a suitable model M to predict hidden performance metrics Y with avail-able metrics X. The data is collected from the EWSN’17 WSN testbed [9]at Uppsala University, and the primary task of this project is to analyze therelations between feature metrics and our target metrics, building up severalprediction models, and evaluating the performance of each model. The targetof smart wireless sensor network management research is to realize the new

9

pattern of IoT, which possesses a series of intelligent characteristics includingself-configuration, self-optimization, self-protection, and self-healing.

Generally speaking, the ultimate goal is to improve the current managementof WSNs and reduce overhead associated with management tasks. Enablingthe expansion of new types of WSNs with controlled performance will openan all-new innovative scope of applications and is the main motivation of thework presented in this thesis. To make it possible, improving observability ofperformance metrics in WSNs must be studied, new types of performance datacollection framework need to be created. This will be a meaningful step toachieve WSNs with controlled performance.

1.2 Sustainability and Ethics

WSNs have attracted worldwide attention in recent years. Now the ubiqui-tous sensor networks bring the radical changes to different fields, such as healthcare, smart agriculture, intelligent building etc.. The sensors in WSN are smallequipment with limited processing resource, memory and battery capability.Finding the solution for WSN to achieve varieties of functionalities by minimiz-ing resource is an open research problem. This project serves as one innovativesolution for implementing off-the-shelf network management protocols in WSN.With the help of regression forecast model, the scheme proposed can fulfill thelimitation of WSN by reducing the measurement traffic flows. It proves the nor-mal operation of network resource monitoring mechanism in WSN, with a sus-tainable mechanism consuming less resource especially for energy consumption.As for the results, this research ensure the possibility of intelligent managementand efficient utilization of network resources in the WSN. Speaking of ethics,common topics related to data security and user privacy. Besides, one newinteresting problem is if network information such as WSN real-time topologybelongs to one part users’ privacy, should be protected and can not get accessrandomly. We do not have the answer yet, and it could be considered as openquestion to discuss in the future.

1.3 Research Questions

After presenting the motivation and the context of the thesis, specific researchquestions are proposed below, which define and formalize the problem addressedin this thesis.

Research Question 1: With infrastructural measurements, can we find anaccurate supervised machine learning prediction model M : X → Y , where Xcorresponds to the WSN features and Y is the round-trip time and loss betweendevices?

Research Question 2: Which off-line supervised ML model is better suited

10

for the collected dataset, Lasso or Random Forest?

Research Question 3: What are the key features for performance predic-tion?

Research Question 4: Is the approach to predict RTT and loss applicableacross multiple IoT scenarios?

1.4 Approach

The goal of this project is to provide a simple machine learning framework withoff-line models to predict network performance parameters. In this project,firstly, experimental research is used to find out the data collection strategy, byrewriting the testbed data collection script with C language. Then, experimentswhich represent different scenarios are carried out to collect X and Y data.After data collection, machine learning processes are implemented to explorethe relation between the features X and Y . A set of ML supervised modelsis developed and evaluated to find the suitable off-line learning model, anddifferent data sets are used to prove the generalization of the ML pipeline.

1.5 Contributions

The focus of this Master’s thesis project is to study machine-learning modelsfor predicting network performance metrics from device statistics and assess theaccuracy of such models. In order to realize the goal, we finished a series of workaround this project. As a result of the tasks done, the following contributionsare listed:

Contribution 1: Implementation of experiment and data collection function-ality in the EWSN’17 testbed.

Contribution 2: Execution of a large set of experiments, and collection ofdata traces, of which some are publicly available on mldata.org.

Contribution 3: Implementation of ML pipeline from the raw data tracesto model evaluation for RTT and loss prediction. The Random Forest modelcan achieve about 7% error for average RTT and one-hour average loss predic-tion, and around 15% for 20-minute average loss in several scenarios with andwithout jammer.

Contribution 4: A study of feature importance showing that topology fea-tures are the most important for performance prediction.

Contribution 5: An analysis about limitations of the current approach andpreparation for the next-step research.

11

1.6 Structure of the Thesis

The thesis is organized in eight chapters and one appendix:

• Chapter 1 presents the motivation for the undergone research, the thesisgeneral content, the research questions, objectives and contributions ofthis project, and at last the structure of the thesis.

• Chapter 2 introduces the background knowledge relevant to the thesis,including the research field, related work, and tools used in the work.

• Chapter 3 gives an overview of the future WSN management system.

• Chapter 4 proposes specific research processes and detailed methodologiesused in this project.

• Chapter 5 describes the testbed setup and configuration of the experi-ments.

• Chapter 6 shows the overview of the data, specific strategies to do the datapreprocessing, and the detailed description about the extracted features.

• Chapter 7 presents the result of all the prediction models.

• Chapter 8 discusses related problems and the future development of thisresearch.

• Chapter 9 concludes this thesis. It presents an overview of the work doneand the major contributions and results.

12

Chapter 2

Background

While the wireless sensor networks (WSNs) did not originally consider the QoS,a range of newborn applications such as modern agriculture, smart home, in-dustrial plant, and medical intra-body communication demand new approachesfor controlled and observable performance, forcing a new approach - WSNs withcontrolled performance. In this chapter, some research for network managementto ensure QoS in a traditional network is introduced, a general view of WSNsis presented to understand the problem at hand, and we list the challenges forWSNs with controlled performance to indicate the meaning of our work.

2.1 Network Management

Traditional networks support varieties of services with different QoS demands.Network operators need some sort of management system for monitoring andmaintenance so that they know what kind of service quality they can assume forcustomers. In the International Organization for Standardization (ISO) networkmodel, five pivotal areas of network management are defined, including FaultManagement, Configuration Management, Performance Management, SecurityManagement and Accounting Management [10]. Our research focuses on thePerformance Management aspect. Normally, network operator provides a writ-ten agreement to customers, called Service Level Aggreement (SLA) related tothe expected performance level of network services. The SLA contains a seriesof criterion as well as some specific value of network performance metrics. Tomake sure SLA is respected, performance needs to be monitored, measured andreported.

A network operator uses both active and passive measurements to monitor andmeasure network performance. In passive measurements. the performance datais collected by passively listening to network traffic, whereas in active mea-surements, probe packets are sent with special purpose over the network togather the performance data. For example, ping with ICMP protocol is the

13

most well-known active measurement tool to get the transmission response time[11]. Passive measurements do not generate extra traffic to disturb the networkperformance, but it requires affiliated network functionalities to gather the per-formance metrics. In contrast, active measurements create additional trafficwhich may disturb the normal traffic flow. However, since they can server forspecific purposes, they can be used to measure more metrics, including those arenot feasible through passive measurements [12]. For traditional wired networksor WSNs, active measurements are widely used for providing network observ-ability. There are several standard protocols used in active measurements, suchas One-way Active Measurement Protocol (OWAMP) [13] and Two-way Ac-tive Measurement Protocol (TWAMP) [14] [15]. In our experiments, we applyTWAMP and target to the round-trip performance metrics.

Active measurement is a probe-based method. A probe packet is an artificialpacket generated depending on the specific measurement requirement. Duringthe transmission, some information (usually is the timestamp) will be recordedin this packet. Finally the data contained in the packet can be analyzed and per-formance target metrics are collected, such as calculating performance metricsincluding delay and loss can be extracted. TWAMP and OWAMP are advancedIP-network measurement protocols. Now, active measurements are wildly usedby telecommunication operators in traditional network services. With the de-velopment of new IoT technologies, they expect that somehow the new IoTmanagement system can provide a similar toolkit using IP basic active mea-surements for achieving observability. Therefore, combined with the operator’srequirements, we focus on IP-layer active measurements. In the following ex-periments, the system utilizes it to send the TCP test probes periodically andrequest the Round-Trip Time/Delay (RTT) back [14]. TWAMP is designedfor round-trip measurements. The basic idea is to bounce probe packets be-tween controllers and reflectors. It can be used to collect varieties of networkperformance metrics, such as one-way delay, RTT, and packet loss.

We use RTT as an example. TWAMP can measure RTT with greater accu-racy by using timestamps. There is no demand for host clock synchronization.The controller plays the role of Control-Client and Session-Sender, and roles ofServer and Session-Reflector are implemented in another host referred to as theresponder [14]. As shown in Figure 2.1, four timestamps are recorded duringthe transmission, including controller sending time t1, reflector receiving timet2, reflector sending time t3 and controller receiving back packet time t4. Thus,the measurement of RTT is t4− t1. Paper [8] compares the accuracy of RTT es-timation with TWAMP and GPS precise timestamps, and proves the feasibilityto use sensor built-in clocks to gather RTT data.

Compared with traditional network management tools, one new research branchrevolving around a new conception - network observability [8] shows up. Theconcept of observability was introduced by Rudolf E. Klmn for linear controlsystem [16]. In this concept, system internal states are inferred from externaloutputs. The fundamental target of network management is to analyze theinner states of network. It is very difficult to complete the process by makinginferences with this theory because of the inherent complexity of a network.However, with the development of machine learning, we obtain the necessary

14

Figure 2.1: Example interaction using TWAMP active measurements to get theRTT.

tools to analyze a complex system, which makes it possible to approach networkmanagement with newly-presented concepts. New challenges about networkmonitoring are denoted as network performance observability. The promisingapproach is using available data to predict the covered performance metrics.In some papers, the concept - observability expanding is used to define thisprocess [8]. Similar prediction-based examples include the research by Woloszet al. where the throughput metric is predicted from a link estimation indicator[17]; also the work done by Rerngvit et al. about predicting cloud servicesmetrics using statistical learning [18][19]; another one is described in [20], wherethey use online machine learning to localize network performance problems.

2.2 Wireless Sensor Networks

Figure 2.2: Wireless sensor network [8]

15

With the expansion of IoT applications, now IoT traffic always has strict QoSrequirements. QoS is very critical for some applications such as typical delayand delay variation for real-time health monitoring and real-time surveillance[21]. To fulfill these requirements, WSNs need network management systemto monitor the performance in general also to detect and localize bottlenecks.The objective QoS requirements about the quality of information sensed andprovided by the service in WSNs are distinguished from normal QoS, and thenew term - QoSensing from [7] is introduced.

Compared with traditional network management, to satisfy QoSensing demandsof different applications, WSNs own some particularities which challenge QoSens-ing provisionings such as resource constraints, platform heterogeneity, dynamicnetwork topology and mixed traffic [22]. First of all, common sensor nodes arelimited in power supply, storage space and computational capabilities, since allthe components are compressed to small size. Besides, sensors even use energy-saving schemes to prolong their life. Sensor nodes are not active all the timewhich worsens their performance even more. The sink nodes/gateways are dif-ferent, normally with less limitation and more powerful capacities. In general,WSNs have weaker transmission performance compared with a wired network.In consequence, when implementing active measurements in WSN management,we need to keep a watchful eye on how many footprints will leave in terms of en-ergy consumption, CPU, and network overhead. Beyond this, WSNs topologiesare highly dynamic because of the instability of the wireless transmission envi-ronments as well as the weakness of the transmitted signal. Thus, the dynamicsmake it more challenging to guarantee QoSening for WSNs.

The proposed solution to network observability in WSNs obtain several techni-cal benefits towards the inevitable limitations. One typical advantage is withML, the energy consumption and overhead for the nodes are lowered. WSNs canapply ML solutions to predict QoSensing instead of sending amounts of mon-itoring probes to do the active measurements. The extra prediction functionis located in the sink nodes which have powerful capabilities and sufficient re-sources, such as power, storage and CPU. Another benefit is that for observingWSN performance, this proposal provides a uniform interface. The system canuse standardized network measurement protocols, such as TWAMP, OWAMPand SNMP and there is no need to design new auxiliary protocols.

2.3 Machine Learning

Instead of gathering all the performance metrics in WSNs by direct measure-ments, we utilize statistical learning to predict them, whereby the behavior ofa system is learned from currently available observations. Specifically, in thisproject, we collect network and environment statistics from a WSN testbed andapply regression methods to predict WSN performance metrics.

ML includes a series of algorithms and approaches, which can learn from andmake predictions on data. In the training process, there are some example inputsX and desired outputs Y , and the goal is to learn the mapping M : X → Y .

16

Then, we can do a prediction with the help of M . ML tasks can be classifiedinto several groups depending on desired outputs. In classification, the outputsare labels, such as discrete numbers; in regression, continuous values are theoutputs; in clustering, no outputs label Y are given to the learning algorithm.We focus on regression algorithms, and three regression methods used in thiswork are outlined - Lasso, Decision Tree and Random Forest.

2.3.1 Lasso

The Lasso regression method is an interpretable linear model performing bothvariable selection and regularization. It was introduced by Robert Tibshirani in1996 [23], and mitigates the overfitting problem with the additional cost termλ∥∥βj∥∥1

, in which λ is the regularization parameter and a greater value means

the result tends to yield a model with smaller linear coefficients∥∥βj∥∥1

, andsome of them are even close to zero, compared with the normal linear regressionmodel. In this framework, the solution to Lasso regression is computed with acoordinate descent algorithm [24].

2.3.2 Decision Tree

Decision tree and Random Forest belongs to the tree-based methods. Theypartition the feature space into a set of rectangles, and in which fits a simplemodel such as a constant. There are a lot of different algorithms for tree-basedregression and classification, such as ID3, C4.5, C5.0 and CART. In this project,we use the CART algorithm [24] to find the regression solution. The researchquestion to regression tree is finding the best binary partition in terms of min-imum sum of squares

∑(yi − f(xi))

2, which is solved with a greedy algorithmThe tree construction process is repeated on all of the resulting regions; duringeach tree construction step, consider a splitting variable j and split point s,define the pair of half-planes as R1 R2, and seek the solution (j, s) to the targetequation 2.2, where c is the response in each region, which is a constant. [25].

R1(j, s) = {X|Xj ≤ s};R2(j, s) = {X|Xj > s} (2.1)

minj,s

minc1

∑xi∈R1(j,s)

(yi − c1)2 + minc2

∑xi∈R2(j,s)

(yi − c2)2

(2.2)

2.3.3 Random Forest

Random Forest is an ensemble learning method by constructing multiple regres-sion trees in the training phase. The output of the model is the mean estimatedvalue of Y of the individual trees. Each tree is built with a different training

17

subset and a different random-selected feature set [26]. Random Forest canmitigate the common overfitting problem in decision tree algorithms.

2.3.4 Feature Reduction

The data may contain either redundant or irrelevant features, which affects themodel training time and may even result in overfitting problems and deterio-rates the prediction accuracy. Feature selection can take out some features andreturn a subset of features without incurring much loss of information. In thisproject, we tried several feature selection methods, including Recursive Fea-ture Elimination (RFE), feature selection based on the feature coefficients andunivariate feature selection.

RFE

The recursive feature elimination algorithm [24] selects features by recursivelyconsidering smaller and smaller sets of features based on the weights of thefeatures in the estimator. In the beginning, the estimator is trained on the totalfeature set and ranks the importance of each feature according to the coefficient(e.g. the coefficients of a linear model) or the feature importance attribute (e.g.the importance matrix in the decision tree or Random Forest). Then, the systemprunes the least important features recursively, until the size of the pruned setreaches the expected number.

Feature Selection from Model

A lot of ML estimators assign weights to features, such as the coefficients ofa linear model and the feature importance of tree-based models. Before thereal training process, the system can pre-train data and get the feature weightmetrics. According to the corresponding value, we can remove the unimpor-tant features [24]. There are different reduction strategies, such as using thenumerical thresholds or setting the specific number of vital features.

Univariate Feature Selection

Univariate feature selection [24] selects the best features based on a univariatestatistical test. There are different scoring functions that can be used in thetests, such as linear regression tests, mutual information tests, and chi-squaredtests for classification tests.

18

2.3.5 Manual Feature Selection

Manual feature selection is selecting out irrelevant data, poorly documenteddata and less important features by manually trimming the list of potentialpredictor variables. Sometimes, we choose the feature subset through domainknowledge to help us understand the relation between features and targets.

19

Chapter 3

Overview of the NewApproach:Active-measurement ProxyBased on Machine Learning

This thesis is a part of a larger project with focus on model-based active mea-surements in IoT environments. Main challenges for IoT are all associated withthe limited resources. In order to solve this problem referring network manage-ment, we put forward the new structure for IoT with an active-measurementproxy. The main functionality of the proxy is to intercept measurement packets.One example is in Figure 3.1: the operator acts as the controller; once it wantsto know the current state of the network performance, one probe is sent fromthe controller to the target network; when the probe passes the IoT gateway, theproxy emulates the measurement-probe response using an ML model, insteadof direct forwarding it to the reflector; then, the proxy sends the answer backto the controller.

There are four main components in the active-measurement proxy, including,TWAMP Interception, IoT Feature Monitoring, ML Model and TWAMP Con-struction module. The TWAMP Interception module captures incoming TWAMPpackets with a destination IP in the target WSN. Instead of forwarding the mea-surement probe, this proxy extracts the necessary information, such as sourceIP, destination IP, packet size and so on for the performance prediction andreflection sending. The IoT Feature Monitoring module collects data from theWSN, such as the radio signal metrics, topological data, and sensor/actuatorresults, for the next performance prediction process. The ML module predictsthe target performance metrics requested by the controller, such as RTT, packetloss with the inputs from the IoT Feature Monitoring and TWAMP Interceptionmodules. The last TWAMP Construction Module uses the information gath-ered from the TWAMP interception module and prepares the reflection packet,

20

Figure 3.1: Example interaction between a TWAMP controller, the active-measurement proxy, and IoT entities.

emulating a response from the IoT device at the destination address.

21

Chapter 4

Method

The work of this project revolves around several parts, including experimen-tal design, experiment execution, data collection, and ML. In this section, wemainly focus on two critical parts - the experimental design and the machinelearning process.

4.1 Experimental Design

The experimental data is collected from the EWSN’17 WSN testbed [9] locatedat Uppsala University. We execute multiple experiments on the testbed tocollect data. From an overall point of view, this project proceeds according tothe experimental design process shown in Figure 4.1.

Figure 4.1: Experimental Design Process.

• Design Experiment After defining the research questions, the projectgoes to the first main part - design of experiments. It is a process concern-ing the minimum number of experiments necessary to develop an empiricalmodel of research questions, and also refers the methodology for settingup the necessary experiments. To be specific, it is necessary to define thedata collection strategy, rewrite the testbed data collection script, anddesign the experiment setup to simulate different scenarios.

• Experiment Execution In this part, we run several experiments usingdesigned configurations on EWSN’17 testbed to collect data.

22

• Machine Learning Process This part is the second main part of thisproject - the machine learning process, trying to build up a predictionpipeline with ML models. Details will be introduced in the next section.

• Draw Conclusions The final process is analyzing the result and drawingconclusions to answer the initial research questions.

4.2 Machine Learning

The ML process is around the loop that includes data acquisition, data pre-processing, statistical analysis, building machine learning models and validation.The core of the project is in line with data science procedures, which is aniterative process. The basic processes are shown in Figure 4.2. The backwardarrows correspond to the fact that the whole process is iterative.

Figure 4.2: Machine Learning Work Flow.

In the following, there is a detailed description of this work flow.

• Acquisition Firstly, it is necessary to go through the Acquisition processto obtain the raw data. Data is the fuel of data science, so this process isthe foundation.

• Preprocessing & Filtering The second step is trying to reduce thenoise, gets rid of the data which is not directly needed and extracts theuseful information from the raw data.

• Statistical Analysis Before building some models with the data, thepotential relationship hidden in data needs to be known in this step. Thisprocess is an indispensable part of data mining.

• Building Machine Learning Models Next comes implementing MLalgorithms, which is another essential part of data mining, and the pre-diction models are obtained from this step. In this project, we implement

23

three models, including Lasso, Decision Tree and Random Forest intro-duced in the background.

• Validation Then, the model is tested against data that has never beenused for model establishing.

• Insight This process is an in-depth understanding step, where further re-search is conducted to learn results, and interpret them with valid reasons.

• Deployment Finally, there is a deployment step, and we need to imple-ment these research results in some real applications.

For this IoT project, repeatedly doing experiments and continuously collectingdata are necessary and time-consuming. After data collection, the predictingmodels need to be built and tested. Besides, every time we get some newdata, retraining data, tuning parameters for better performance are necessaryprocesses. According to the results, changing configuration and designing newexperiments for more data in different scenarios are quite normal.

In addition, the detailed process of data preparation (Preprocessing & Filter-ing) is important, also, it is conducive to model choosing. It includes thedata exploration, cleaning, and preparation. There are basically seven steps:Variable identification, Univariate analysis, Bi-variate analysis, Missing valuestreatment, Outlier treatment, Variable transformation, Variable creation. Dur-ing these procedures, several problems need to be solved, particularly, aboutmissing value, outlier detection, and feature engineering [27].

24

Chapter 5

Testbed and ExperimentDescription

5.1 Testbed Description

5.1.1 Testbed

The EWSN’17 testbed [9] located at Pollacksbacken, Uppsala University, is usedto explore the prediction of RTT and loss in order to reduce the overhead fromactive measurements. There are 20 MTM-CM5000-MSP nodes (TelosB replicas)installed in the testbed, but two of them(#127 and #133) are broken, so in themap there are only 18 nodes. Each sensor node, also known as a mote, is aconstrained execution environment that has the complete capabilities to gathersensory information, perform processing and communicate with other motes.

The motes are arranged in a public building that constitutes a challenging envi-ronment for wireless communication due to thick stone walls and rebars. Figure5.1 shows the location map. This is a real experimental environment and thereis also an uncontrolled impact from the people in and out of this building, aswell as their wireless devices. Therefore, the WSN transmission environment isdynamic and results in fluctuation of network performance.

For the purpose of experiment administration and monitoring, each mote isattached to a separate monitoring module based on Raspberry Pi 2 (RPi2)MODEL B, named D-Cube [9]. D-Cube sits on the top of the basic WSNtestbed infrastructure and provides capabilities for easy and large-scale repro-gramming of motes, power profiling, and high-precision event detection, withoutinterfering with the motes nor the wireless medium during execution of experi-ments [8]. There are two kinds of connection between a mote and the monitoringmodule. One is through general General Purpose Input/Output (GPIO) pins todo passive GPIO tracing and independently monitor the activities of the target

25

mote. For our experiments, the monitoring frequency is set to a default value of62.5kHz, which enables event detection to reach 16 µs high resolution. The otherconnection is for logging and reprogramming via a USB active cable from a cen-tral control station. The same code runs on all motes, but different motes playdifferent roles depending on the node identifier, that is whether the mote shouldact as a controller or reflector. For the experiments discussed in this thesis, weset one controller as the gateway for the sensor network, and the controller isselected from [#125,#126,#134]. The other motes act as reflectors.

Figure 5.1: Placement of Motes in EWSN’17 testbed.

All the motes in the testbed use the Contiki operating system and communi-cate over IPv6 enabled by IETF IPv6 over Low-Power Wireless Personal AreaNetworks (6LoWPAN), running on top of IEEE 802.15.4. Each mote uses RPLin the network layer to complete the routing process and utilize TWAMP, in-troduced in Section 2.1, to actively measure network performance.

5.1.2 Contiki OS

Since the devices in WSNs are often resource constrained, this new environ-ment is also called Low-Power and Lossy Networks (LLNs). Staying with thelimitations of the constrained devices, LLNs presents special requirements forthe design of an Operating System (OS). Contiki [28] is one promising solution,and it is a lightweight operating system for sensor nodes, providing dynamicloading and unloading of individual programs and services with an event-drivenkernel [29]. For lower layers (data link and physical layers) the IEEE 802.15.4standard is implemented in Contiki. Besides, Contiki supports both IPV4 andIPV6 standard which increase interoperability and universality and one way tomake the IP protocol compatible with LLNs is to introduce an adaptation layer -loWPAN. In the EWSN’17 testbed, we use IPV6 standard with 6loWPAN layer.The MTU for the network is 127 bytes, and to avoid packet fragmentation theprobe-packet size is set to 100 bytes, including all the headers and payload.

26

5.1.3 RPL in the Testbed

Routing protocol for LLNs (RPL) (RFC6550) [30] is designed for WSNs’s con-strained environment to fulfill the demands with respects to limited bandwidth,memory, and energy resources. The Contiki OS implements a fairly stable ver-sion of RPL. RPL uses Objective Functions (OFs) to state the outcome of theprocess used by an RPL protocol for each node to choose suitable routes. Differ-ent routing computation factors are utilized in different OFs to form a networktopology such as Expected Transmission Count (ETX) 1, RSSI 2 and other linkestimation metrics 3. As for the experiments mentioned in this thesis, the Con-tiki default Minimum Rank with Hysteresis Objective Function (MRHOF) isused as the OF, in which ETX is the routing computation factor, and moredetails are shown in RFC 6719 [31].

5.1.4 TWAMP for Active Measurements

We implemented IETF TWAMP light (RFC 5357) for a Contiki 3.0 environment[28]as the predominant OS for IoT devices, to enable active measurements betweenthe motes. The implementation supports GPIO signaling upon packet receivingand transmitting events, which allows the monitoring modules to detect eventsoccurring at the mote. The basic idea is to bounce probe packets between con-trollers and reflectors. TWAMP is designed among other things for round-tripmeasurements with greater accuracy by using time-stamps. There is no needfor host clock synchronization. The controller plays the role of the Control-Client and Session-Sender, and the roles of Server and Session-Reflector areimplemented in another host referred to as the responder [14]. Four timestampsare recorded during the transmission (as shown in the Figure 2.1), includingcontroller sending time T1, reflector receiving time T2, reflector sending timeT3 and controller receiving back packet time T4. Thus, the measurement ofRTT is T4 − T1. The paper [8] compares the accuracy of RTT estimation withTWAMP and GPS-precise time stamps, and it proves the feasibility to use asensor built-in clock to gather RTT data.

5.1.5 jammer

Since the reliability of a sensor network is affected by radio interference, thereis a strong need for predicting the performance of the WSN protocol with inter-ference to test the robustness of the prediction model. In our jammer test, one

1ETX is the number of expected transmissions of a packet necessary for it to be receivedwithout error at its destination. This number varies from one to infinity. An ETX of oneindicates a perfect transmission medium, where an ETX of infinity represents a completelynon-functional link.

2In telecommunications, Received Signal Strength Indicator (RSSI) is a measurement ofthe power present in a received radio signal.

3Links Estimation is an integral part of reliable communication in wireless networks. Var-ious link estimation metrics have been proposed to effectively measure the quality of wirelesslinks

27

JamLab node [32] is added to the existing network infrastructure. JamLab usesa separate removable off-the-shelf mote to record and playback interference pat-terns with low overhead. In experiment #4, the position is fixed and changed bymanually moving from one position to another in the last experiment. CC2420supports 11 distinct output power levels in the range of [-55,0] dBm by settingthe value of PA POWER (the comparison table between PA POWER and realout power is explained in [32]). The JamLab motes we use in all the jammingexperiments are set to 11 (-10 dBm). The jammer interference model is set toemulate a Wi-Fi network saturated with YouTube video streaming.

5.2 Experimental Setup

We describe here some of the details. For the experiments discussed in thisthesis, we co-locate the TWAMP controller and the RPL root (i.e. the gateway)for the sensor network. The controller is selected from [#125,#126,#134].The other motes act as reflectors. Except for the controller position, five extraparameters can be set up, and we use four sets of setup plans to collect the data.The details of each parameter are discussed below.

5.2.1 Configuration Parameters

The parameters need to be configured are as follows:

• TWAMP Node: The controller in this experiment also acts as the gate-way of the WSN, which is the root of the network logic topology related tothe routing path of the packet transmission. We can change the networktopology by choosing a different gateway location. For example, in ourexperiment, selecting the edge motes number #134 or #126, the networkshows a tree topology. Setting a central mote number 126 as the gate-way, the network forms a star topology, which is also the most commontopological structure in WSNs.

• Probe sending time interval: In this project, a probe-based activemeasuring method is implemented to test the network performance. Thetime between two continuous probes is the probe sending time interval∆t. In the experiment, we set a relatively long interval of 6s in order toavoid network congestion and overload, because we find that most RTTdata are shorter than 6s.

• Probe repeat times: Since after analyzing the raw data we gather fromthe EWSN’17 testbed, we find that noise is a big problem during the targetperformance data collection procedure. In the following experiment, we seta new probe sending strategy: the controller sends probes in cycles and ineach loop it will send to the other seventeen reflectors successively. Insteadof one packet, the controller sends sequentially n packets to each reflector

28

node. It means after ∆t ∗ n, the controller changes the destination nodeto the next one. Figure 5.2 illustrates the detailed process of the sendingstrategy. In all the experiments, n is set to 5.

Figure 5.2: Probe sending strategy.

• NBR reporting frequency: The method we use to collect the topol-ogy information is passive reporting. NBR reporting interval ∆tnbr is setand each reflector sends its current neighbor list back to the controllerperiodically. In all the experiments, ∆tnbr is set to 60s.

• Wireless channel: The IEEE 802.15.4 standard is applied for wirelesscommunication in the EWSN’17 testbed, which mainly adopts the 2.4-GHz band. A total of 16 channels are available in this band, numbered11 to 26, each with a bandwidth of 2 MHz and a channel separation of5 MHz [33]. Since other protocols also share the same band, such asBluetooth and Wifi, only channel 15, 20, 25 and 26 are not overlappingwith other protocols. In the following test setup both overlapping channelsand non-overlapping channels are chosen to increase the feature diversity,also create a more complex environment for data collection.

• Jammer location: The real IoT communication environment is a morecomplicated situation. Sometimes, there are some significant interferencesources in the network. Therefore, in order to make our experiments morerealistic and close to the actual scenario, jammer motes are added to thetestbed. There are several typical positions pointed in the testbed Figure5.3 as the placement of the jammer node, which is shown in red. The reddot line shows one jammer moving route in the experiment from J-2 toJ-1 through J-3, J-4, J-5. J-6, and J-7.

29

Figure 5.3: Placement of jammers in EWSN’17 testbed.

Figure 5.4: Probe Sending Loop.

5.2.2 Probe Sending Strategy

As described above, with repeated probe sending method, sending time interval∆t is set to 6s; probe repeat times n is 5; the number of nodes in the testbed is20 (one is the controller and the other 19 are reflectors). Therefore, the completeprobe sending timeline is shown in 5.4. Every 6s the controller sends one probeout; every 30s the controller changes the probe destination to the next one;every 600s is one loop and after that, the destination is changed back to thefirst one; the cycle continues repeating until the end of the experiment.

5.2.3 Setup Plan

Five experimental plans are shown in Table 5.1. Since star topologies are themost common configuration in real WSNs, both No.4 and No.5 jammer experi-ments set # 126 as the controller (# 126 locates in the middle of all the sensors.If set as the controller, the network topology will be close to star shape). Theconfiguration plan No.4 is composed of three parts with fixed testbed config-uration, but different jammer positions (J-1, J-2) and one without jammer asthe blank control experiment. The two tests with jammer labeled as #4-v1 and#4-v2, and the blank control test without jammer is named #4-v3. No.5 isa configuration plan with moving jammer. The jammer is moved every hourfrom position J-1 to J-7 through the path shown in the Figure 5.3 (J-1, J-2, J-3,J-4, J-5, J-6, J-7 then remove jammer). This experiment is modeling a Wi-Fihandheld mobile device as the interference source.

30

Table 5.1: Experiment Setup

Experimental Setup

Exp Label Controller node Wireless channel Jammer position

#1 #134 26,21,16,11 /

#2 #125 26,21 /

#3 #126 26,21 /

#4-v1 #126 26 J-1

#4-v2 #126 26 J-7

#4-v3 #126 26 /

#5 #126 26 J-1,2,3,4,5,6,7

31

Chapter 6

Data Description

The data is collected from the sensors in the EWSN’17 WSN testbed, imple-menting the testbed configuration plans introduced in the Section 5.2.3. Eachpacket transmitting in the network is sent from the source (src) node (TWAMPcontroller) to the destination (dst) node (TWAMP reflector). Every time thereflector receives a packet, the system gets one record from the controller. Inaddition to this, each node in the WSN, except the controller, will report thetopology data periodically, which is regarded as the neighbor data. There are 20nodes in the testbed, but after collecting the data, we find two nodes are broken.Therefore, we use the data from 18 active nodes (nodes #127 and #133 are notincluded).

From the machine learning point of view, raw data needs to be cleaned, collatedand transformed in the preprocessing and filtering phases (Section 4). Finally,we will get the well-structured feature set and input it into the statistical pre-diction models.

We implement the configuration introduced in Section 3.2 and get the data fromthree types of experiments (without jammer, with fixed jammer and with movingjammer). The experiment execution time details are shown in Table 6.1. Theexperiment configuration plans are clarified in the Section 5.2.3 - ExperimentDescription.

Figure 6.1: Data Processing Flow.

For each experiment, the system gathers two types of data files, including one

32

Table 6.1: Running Time of the Experiments

Exp Label Running Time Running Date# 1 10 days 2018/03/24-04/02# 2 6 days 2018/04/03-2018/04/08# 3 6 days 2018/06/20-06/25# 4-v1 24 hours 2018/06/02# 4-v2 24 hours 2018/06/29# 4-v3 24 hours 2018/06/06# 5 7 hours 2018/06/19

packet records file from the controller node and seventeen neighbor list reportingfiles from seventeen reflectors. The data processing for all the experimental datain this project is explained in the Figure 6.1. Firstly, these two files need to bemerged and get the raw dataset, then it will pass the preprocessing step, finallythe feature and target metrics will be extracted for the model training phase.The raw data size of each file shows in the Table 6.2 and Table 6.3, in which weuse the number of records as the unit of the measurement. For example, thereare 75’641 controller packet records in the dataset experiment #1; in the sameexperiment, for neighbor data from node 121, there are 14’391 records.

Table 6.2: WSN Raw Data Size - Controller Records

Experiment Label Number of Records

# 1 75’614# 2 47’335# 3 23’961

# 4-v1 3’648# 4-v2 7’361# 4-v3 11’398

# 5 2’023

There are 44 metrics in the raw data including source IP (srcIP), destinationIP (nodeIP), sequence number (seqNo), packet recording date (logDate) andtime (logTS), wireless channel (CH), last-hop received signal strength (RSSI),last-hop Link Quality Indicator (LQI), light intensity (Light), round-trip time(RTT) as well as 17-node rank value (rank) and number of neighbors (nbr).These are the unprocessed raw metrics. The system will clean these metrics,transform them and extract the features from them.

33

Table 6.3: WSN Raw Data Size - Neighbor List Records (in terms of samples)

# 1 # 2 # 3 # 4-v1 # 4-v2 # 4-v3 # 5

node 121 14’391 8’635 8’634 67 488 677 127node 122 14’367 8’637 8’638 373 485 702 165node 123 5’481 4’932 606 90 485 685 131node 124 11’264 8’056 6’076 371 263 670 98node 125 14’389 / 1 8’635 366 298 669 99node 126 6’094 6’207 / / / / /node 128 14’055 8’637 8’640 193 447 648 168node 129 14’389 8’631 8’632 51 483 702 126node 130 14’389 8’639 8’639 86 464 676 117node 131 14’377 8’628 8’564 371 413 707 162node 132 5’414 4’335 695 84 475 678 120node 134 / 8’385 8’636 28 366 592 0node 135 14’394 8’637 864 2 241 474 0node 136 14’394 8’639 8’639 366 492 704 168node 137 8’547 7’912 3’748 373 493 713 175node 138 11’411 5’899 7’809 84 491 699 42node 139 10’258 8’491 8’483 369 480 697 147node 140 11’120 8’031 8’238 374 497 705 178

6.1 Feature Set X

6.1.1 Feature Description

The final feature set in Table 6.4 is extracted from raw metrics through the datapreprocessing step. In this process, some metrics are changed or transformed,and the others are consistent with their original form in the raw dataset. Thesystem will get the prepared training set, which is the direct input of ML algo-rithm, including the X feature set and the target set Y.

The feature set X is composed of three kinds of statistics: the basic packetstatistics, the sensing statistic and the network topology statistics. The targetset Y includes two target network performance metrics - ’RTT’ and ’loss’ andit will be introduced in the following section. Table 6.4 shows the classificationof the feature set X.

Basic Packet Statistics

This category is related to the basic information about the WSNs, which canbe divided into three groups - WSNs nodes placement information (src x, src y,

1 Controller does not return the neighbor list information, since the rank value is fixed inthe RPL routing protocol

34

Table 6.4: WSN Feature List

Feature List

Basicsrc x src y node x node y distance timedate srcID nodeID CH RSSI LQI

Sensing Light

NBLrank list (the rank value of each destination node)

NBL list (the number of neighbors of each destination node)

node x, node y, distance), package information (time, date, srcIP, NodeIP) andwireless environment information (CH, RSSI, LQI). These features will now beintroduced in detail.

src x, src y: physical topology feature, geographic coordinates of the sourcenode.

node x, node y: physical topology feature, geographic coordinates of the des-tination node.

distance: physical topology feature, real straight-line distance between thesource node and the destination node.

time: the time frame of destination node receiving the packet.

date: the day of destination node receiving the packet (from Monday to Sun-day).

srcID: the label number of source node (details are in the Section 5 - TestbedDescription).

nodeID: the label number of the destination node (details are in the Section 5- Testbed Description).

CH: the transmission wireless channel of the packet. The testbed employsIEEE802.15.4 standard [34], 2.4GHz band. A total of 16 channels are availablein this band, numbered 11 and 26, each with a bandwidth of 2MHz and a channelseparation of 5MHz.

RSSI: (Received Signal Strength Indication) one of the link estimation metrics.It is a DC analog level, which is an average of the last 8 symbols received signalpower and often derived in the intermediate frequency (IF) stage before the IFamplifier [35].

LQI: wireless environment feature, the cleaned Link Quality Indicator for eachpacket’s last hop. LQI is one of the link estimation metrics 2, which is a digital

2’The LQI gives an estimate of how easily a received signal can be demodulated by accu-mulating the magnitude of the error between ideal constellations and the received signal overthe 64 symbols immediately following the sync word.’ https://forum.pololu.com/t/meaning-

35

value often provide by the Chipset vendor. A high value LQI indicates a betterlink than a low value does, and this value is dependent on the modulationformat.

Sensing Statistic

The sensing statistic contains the information collected from the sensors in thetestbed. WSNs realize the fundamental task of monitoring and recording thephysical environment conditions with wireless technology using the open spaceas transmission media. Therefore, The communication quality depends largelyon the surrounding conditions. In this thesis, the framework only contains onesensing feature - Light. Every time the destination node receives one packet,the system records the light intensity at that time point, with the light sensorfrom the controller node 3.

Light: the sensing feature, light intensity at the controller node.

Network Topology Statistics

Network topology is the arrangement of communication between network ele-ments such as nodes, link, etc. This framework includes two kinds of topologicalfeatures to illustrates how data flows in a network. The testbed Contiki systemautomatically forms a IPV6 wireless network with the help of RPL (RoutingProtocol for LLNs, in which each node owns a Neighbor List (NBL) with theinformation about the number of neighbors, the rank value of each neighbor,the metric of each surrounding link, as well as the rank value of the node itself.In the experiments, each reflector sensor reports the neighbor list periodically,so the system can monitor the changes of WSN testbed topology. The detailsof the topology features are as follows.

rank: logical topology feature, core fundamental metric determining how thepacket chooses the routing path, as well as how the topology forms.

nbr: logical topology feature, number of neighbors for each node.

6.1.2 Data Preprocessing

The prepared feature set is obtained from the Preprocessing & Filtering pro-cedure, introduced in the Section 4 which is also the start point of ML dataprocessing. Through this step, the system needs to find out the irrelevant, noisyor unreliable data. It has a significant impact on the prediction performance ofsupervised learning algorithm, since the ultimate product of data pre-processingis the final training set, which is the direct input of the ML algorithm. As we allknow, the quality of food is largely related to the raw material. Similarly, data

of-link-quality-indicator/3358/23Hamamatsu S1087 Series, Visible Range (560 nm peak sensitivity wavelength)

36

is the fundamental material of ML. Therefore, ”The representation and qualityof algorithm data is first and foremost before running and analysis” [36].

Normally, there are several steps in data pre-processing, including data cleaning,normalization, transformation, filtering, feature extraction, and selection, etc.[37]. Not all these steps are necessary for each dataset, which is determined bythe specific situation of the raw data. In this report, we will not specify thesesteps, and just roughly divide them into two parts (Data cleaning and FeatureEngineering) to present the content of data pre-processing. In addition, both thedata related to the feature set X and target set Y need the data preprocessing,and the target metrics will be introduced in the next section.

Data Cleaning

The data-collecting procedure is often loosely controlled, resulting in unstruc-tured raw data having out-of-range and missing values, etc. Machine learningprediction models are sensitive to these problems, which can produce misleadingresults. Thus, in the data cleaning, the system filters out problematic data, ap-plying outliers detection methods. For example, the basic variable-by-variabledata cleaning method is used for the majority of metrics in this project to detectpossible data quality problems with the metadata 4 [37]. Looking through allthe raw data, the metric ’LQI’ has an illegal value problem, also there is oneunavoidable problem - incomplete data.

LQI: LQI is an integer link estimation metric. In Contiki, it should be in therange from 50 to 110. The system sets the permissible range to filter out therecords outside the range.

Missing Feature ValuesIncomplete data is a normal problem in dealing with the data collected fromthe real world. In this framework, the system applies the method of ignoringinstances with unknown feature value, and directly deletes the records with atleast one unknown metric value [37].

Feature Engineering

Since the original data in the raw files are straightforward and rough, thereis useful information hidden in them. In data science research, the success ofMachine Learning algorithms depends on how we present the characteristics ofthe data 5. The performance of algorithm depends largely on how to get thesuitable data format for predictive modeling. This is the main task of feature

4Metadata means ”data about data”. Metadata is defined as the data providing informa-tion about one or more aspects of the data; it is used to summarize basic information aboutdata which can make tracking and working with specific data easier.

5 Mohammad Pezeshki, answer to What are some general tips on feature selection andengineering that every data scientist should know? https://www.quora.com/What-are-some-general-tips-on-feature-selection-and-engineering-that-every-data-scientist-should-know

37

engineering 6. In this section, the raw metrics are modified and transformed tothe features listed in the Table 6.4. The ultimate goal of feature engineering isanswering the question: what is the best representation of the sample data? Inthe following, the final processing strategies are introduced.

We use the following pattern to show the relation between raw metrics andtransformed metrics:Sample: Raw metric → Preprocessed metric

srcIP and nodeIP → srcID and nodeID:’srcIP’ and ’nodeIP’ are the IP address of the source and destination node IP.Since the testbed implements IPV6 to achieve the communication and each nodeowns a static IP address, there is a one-to-one corresponding relations betweenthe IP address and the node label. The testbed contains eighteen active nodes,which are labeled by eighteen numbers (the label of each node is shown in theSection 5 - Testbed Description).

srcIP and nodeIP → src x, src y, node x and node y:In the normal WSNs with controlled performance scenarios, sensors are dis-tributed orderly and the location of sensors is fixed which can be known inadvance. The relation between the coordinates and the IP address is also one-to-one corresponding (the coordinate of each is introduced in the Section 5 -Testbed Description).

src x, src y, node x and node y → distance:For better presenting the physical network topology, we define a new feature todepict the real straight-line distance between the source node and the destina-tion node, calculated with the coordinates of nodes by the following equation6.1. √

(srcx − nodex)2 + (srcy − nodey)2 (6.1)

logTS → time slot:The raw logTS metric is the time stamp of the controller receiving the backpacket, such as ”19:23:02.149496”. The ’time’ feature is extracted from the’logTS’ to give additional insight into WSN performance. The testbed is locatedin the teaching area, where the people activities follow the school schedule, andhuman activities play an important role among the inference in the wirelesstransmission. Therefore, instead of the specific time stamp, we want to use atime map to understand the people behavior. Time map: (The twenty-fourhours are divided into eight time slots) slot 1:0-3, slot 2: 3-6, slot 3: 6-9, slot 4:9-12, slot 5: 12-15, slot 6: 15-18, slot 7: 18-21, slot 8: 21-24.

logDate → date:’logDate’ is the date of receiving the packet. Similar to ’time’ feature, we extract

6”Feature engineering is the process of transforming raw data into features that better rep-resent the underlying problem to the predictive models, resulting in improved model accuracyon unseen data.”–Dr. Jason Brownlee

38

the ’date’ feature from the raw ’logDate’ metric (”2018-05-04”), and change thespecific date to seven days in a week (from Monday to Sunday) 7.

Merge topology data stream into probing data stream:First, from the experimental configuration described in the Section 5.2.3, amongthe eighteen active nodes in the WSN testbed, there is one controller, which col-lects the packet records, and the other seventeen nodes report their neighborlists periodically. In this framework, the system regards the seventeen nodes’rank metrics as seventeen features. nbr metric follows the same strategy. Sec-ond, it is introduced that the neighbor list reporting interval is ∆tnbr, but theprobe sending interval is ∆t(∆tnbr 6= ∆t), so there are two streams of data withdifferent time granularity shown in Figure 6.2. The system needs to duplicatetopology reporting stream so that each record of the probing stream matchesone record of the topology stream. Finally, after matching, the dataset will gettotally 47 metrics in feature set X.

Figure 6.2: Two data streams merge diagram.

7With the help to pandas time series functionality, find the specific day of a time stamp.https://pandas.pydata.org/pandas-docs/stable/timeseries.html#

39

6.2 Target Set Y

In this thesis, we focus on predicting RTT and loss, which corresponds to thetarget YRTT and Yloss.

6.2.1 Target Description

The target set Y includes two target network performance metrics - ’RTT’ and’loss’. In Vasco WSNs requirements taxonomic research [7], the authors liststhat the main requirements of the general communication performance are thedelay tolerance, loss tolerance, capacity, reliability, energy efficiency, criticalityand fault tolerance. In this thesis, the framework aims at RTT, reflecting thedelay tolerance, and the round trip loss, which shows the loss performance ofthe network. The specific definitions of the targets are as follows.

RTT: cleaned round-trip (individual) delivery delay, which is the time spentby a packet from controller to a reflector together with the response from thereflector to the controller [5].

loss: round-trip loss per node, the fraction of loss packets considering all thepackets sent by the controller to the specific reflector and then sent back to thecontroller, in a specific time interval.

6.2.2 Data Preprocessing

Similar to the feature set X, the raw data related to the target also needs togo through the Preprocessing & Filtering phase and get the prepared target setY as a part of the training set. In this section, we also divide them into twoparts (Data Cleaning and Feature Engineering) to introduce the strategies ofour data pre-processing.

Data Cleaning

In the target set Y raw metrics, there is only one problem - the distinct illegalvalue problem in ’RTT’.

RTT: An inlier lies in the interior of a statistical distribution. From the distri-bution of RTT, it is founded that there are some extremely large values, far awaydeviating from the average of ’RTT’. These outliers occupy less than 5% of thetotal data. After calculating the statistical distribution of different data sets,the ’RTT’ outliers threshold is set to 10s, with which the system can usuallyfilter out about 1 % large-value data regarded as outliers.

40

Feature Engineering

In this section, the specific methods to transform and extract the target set Y(’RTT’, ’loss’) will be introduced.

A similar pattern mentioned before is used to show the relation between rawmetrics and worked metrics.Sample: Raw metric → Preprocessed metric

RTT → RTT:Since noise is a big problem during the data collection procedure in the EWSN’17testbed, the controller sends sequentially n packets to each reflector node (de-tails show in the Section 5.2.3 - Experiment Description). As shown in theFigure 6.3, some probes will be lost during the transmission. Then, in thereal situation, every time from each reflector, the controller receives n′ packets(0 ≤ n′ ≤ n). The system will calculate the mean ’RTT’ value or p percentilevalue of the n′ packets as the new ’RTT’ feature of them to reduce the influenceof noise.

Figure 6.3: Computing average RTT over received probes

seqNo→ loss: In the raw data, ’seqNo’ is a value to record the order of sendingpackets in one experiment. With the probe sending interval ∆t, we can knowthe supposed number packets being sent by the controller to each node Ns T inthe specific time interval ∆T (Ns T = ∆T

∆t ). Then, using the sequence numberlist in the packet records dataset, the number of received packets for each nodeNr T within the time interval ∆T can be counted. Figure 6.4 shows an example.Thereby, the round-trip loss per node can be computed with equation 6.2. Thedetailed python code is attached in the appendix.

Figure 6.4: Computing average loss over recevied probes

41

loss = 1− Nr T

Ns T= 1− Nr T

∆T∆t

= 1− Nr T •∆t

∆T(6.2)

As mentioned in Figure 6.4, ∆t is 6s and one loop is 600s. We choose threetime intervals to compute the average loss which are 6h, 1h and 20min. We useexperiment # 1 data (running time is 10 days) as a example to show the statisticdetails of each situation in Table 6.5. In 20min time interval each reflector inthe testbed only can receive and reflect at most ten packets, so there are onlyeleven possible values per-node 20min average loss(0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6,0.7, 0.8, 0.9, 1). If we using 1h as the time interval to calculate, each reflectorcan receive at most thirty probes, so there are thirty-one possible values forper-node average loss over 1h, ranging from 0 to 1. Thereby the data diversityfor 1h average loss is higher than 20min average loss. Therefore, we can observethat as for the same data set, with longer calculation interval, loss has morepossible values (higher diversity) but fewer samples.

Table 6.5: Loss Statistics Example for Experiment # 1

¿ Interval # Records fora node per timeinterval

# Loss examples Range of loss for a node

6h 180 40 (0, 1180 ,

2180 ...

179180 , 1)

1h 30 240 (0, 130 ,

230 ...

2930 , 1)

20min 10 720 (0, 110 ,

210 ...

910 , 1)

6.3 Data Visualization

After the feature extraction process introduced above, in this section, we willselectively show several data visualization figures and provide a overview im-pression about the metrics in our dataset.

6.3.1 Features

Daylight

Daylight is the only sensing metric in our dataset and it has obvious periodicityand seasonality. The activities of people and animal often have similar variationrule to light intensity, which are the considerable influence factors to the WSNperformance. Therefore, daylight is an interesting feature to our research. Afteranalyzing the data collected from the experiment # 125, we find the periodicalcharacteristic that is consistent with common sense, shown in the Figure 6.5

42

(light feature from any of the experiments all have this character). The hori-zontal axis is the time line, and the vertical axis is the daylight intensity. Theline reflects the phenomenon of every day sun rising up and falling down.

Figure 6.5: The periodical characteristic of the light feature.

Rank

Rank is the one feature for WSN topology construction. In RPL routing pro-tocol, each node uses the rank value of the neighbors as the decision basis toselect the parent, composing the routing map. According to the setup descrip-tion in Section 5.2.3, different controller position will form different topologies.In the following figures, we choose three nodes (node #135, node #124 andnode #128) which are separately close to the three controller candidates (node#134, node #125 and node #126) as the observation points, to understand howthe rank of different node varies in each topology.

Figure 6.6 shows the rank value collected from the experiment #134; Figure 6.7is for the experiment # 125; and Figure 6.8is for the star-topology experiment#126. Looking through the three time series figures and reviewing the testbedmap in the Figure 5.1, we can find that there are three phenomena: firstly, rankis a accumulated metric, hence, the farther away the geographical location is,the larger the rank value, which is consistent with the RPL routing protocol;secondly, for the nodes with large rank value (i.e. that are far away from thecontroller), the rank value is more dynamic, which means the probability of thepath change at the edge of the topology is higher; finally, the three lines inFigure 6.6 and Figure 6.7 have similar positions because both of their networkshave tree topology, but Figure 6.8 is different, there are two line really close toeach other. The difference shows that the rank value can reflect the topologyfeature.

43

Figure 6.6: The time series of the experiment #134 topology rank feature.

Figure 6.7: The time series of the experiment #125 topology rank feature.

Figure 6.8: The time series of the experiment #126 topology rank feature.

44

6.3.2 Target

There are two target metrics, RTT and round-trip average loss. In this section,the influence of different data processing is shown.

RTT

As mentioned in the Section 6.2.2, in order to improve the accuracy of prediction,the noise in RTT data needs to be reduced. In our framework, except keepingthe single RTT value as the target, we use two aggregation methods to processthe RTT data: one is like the description in Section 6.2.2, calculating the meanvalue of the packets reflected from the same reflectors; the other, instead of usingthe mean value, computes the 80th percentile. From the experiment #134 RTT,time series is shown in Figure 6.9. The blue subplot shows the obvious noisein the original RTT. Compared with the RTT without any data processingstrategy, the curves of average and 80th percentile strategy are smoother, whichillustrates the effect of noise reduction. Beyond that, from the Figure 6.10, wecan find that the additional target processing methods do not largely changethe primary RTT distribution, especially the mean value strategy owning almostthe same density plot as the original RTT.

Figure 6.9: The time series of different RTT strategy.

45

Figure 6.10: The kernel density plot for RTT, averaged RTT, and 80th percentileRTT.

Loss

In all experiments, we use the same sending strategy introduced in Chapter5. According to the sequence number in each packet, we define three differenttime intervals (6h, 1h, 20min) to calculate the average round-trip loss. In thefollowing two figures, we show the experiment # 134 data in two ways (timeseries and kernel density plot).

The difference of time interval can be seen obviously from the time series inFigure 6.11, as the loss value change more frequently with shorter time intervalfor the same experiment. For example, the average 20min loss value is changingmore frequently than 6h average loss.

From the distribution of loss shown in histogram Figure 6.12, we can see themain body of average loss with three different computing intervals are almostat the same position. It means the majority spread are consistent. However,it is obvious that average loss values over 6h are more continuous, and thereis a certain amount of values distribute in the high loss range (above 0.5).In the contrast, using the same dataset, shorter computing intervals incur thediscreteness of loss, hardly any get the value larger than 0.5. We checked the rawdataset and find that sometimes, massive data loss occured during the executionof experiments, which are normally longer than 10 minutes and even continueseveral hours, forming some data faults or gaps. From the histogram it seemslike with shorter computing time like 20min, these gaps are seldomly detected.

In the testbed, controller records information are based on the receiving packets.

46

Every time the controller receive a probe, it will get one record. Thus, thepacket loss leads to massive record loss. With shorter loss computing intervals,it is hard to detect the existence of massive data loss. However, with longerintervals, especially when the loss computing interval longer than the data faulttime interval, it easier to detect the data faults with a high data loss rate.Therefore, the data with higher loss rate for 6h computing interval displays theinformation about data faults in the experiments.

Figure 6.11: The time series of different loss strategy.

Figure 6.12: The histogram of average loss with different computing intervals.

47

Chapter 7

Evaluation

This Chapter provides an evaluation of machine learning (ML) models M : X →Y , predicting performance metrics Y from network statistics X. Firstly thedetails about model configuration are introduced, continuing with the evaluationof each ML model and the analysis of the results.

7.1 Model Configuration Details

7.1.1 Hyperparameters

In ML, the term of hyperparameter is used to distinguish from the model param-eters learned from the data. Model parameters are learned through the modeltraining process, which is also the crux of ML. Through model training, we tryto fit a model to the data. Hyperparameter forms a different set of parametersnot during the training phase. It is usually determined in the pre-training pro-cess, which is a preparation for the actual model training. Different ML modelshave different hyperparameter sets. This project focuses on the comparison be-tween the Random Forest and Lasso models. In the following, we list the mainhyperparameters for each of them.

• Random Forest

1. ’max depth’ controls how deep the trees are.

2. ’max features’ determines the number of features to consider whenlooking for the best split.

3. ’number of estimators’ is the number of trees in the forest.

• Lasso’alpha’ determines the amount of regularisation.

48

The model parameters are optimized by the training process. Hyperparametersare tuned in the pre-training process. In this project, GridSearchCV is used [24],with the Normalized Mean Absolute Error (NMAE) as the scoring method, tofind out the best hyperparameter composition [24].

7.1.2 Evaluation

After training, we get the complete ML pipeline, then the evaluation is neededto assess the performance. In order to avoid overfitting of supervised learning,each model is trained with the 10-fold cross-validation method [24] to evaluateestimator performance. As we know, in the fitting process, the model parame-ters are optimized to make the model fit the training data as well as possible.Training with different data will result in different models. If we grasp two in-dependent subsets of the data as the training set and test set, it will generallyturn out that the model does not fit the test data as well as it fits the trainingdata. Cross-validation is a method to estimate the size of the difference andget a more reliable model performance evaluation metric. For each model, boththe mean value and standard deviation of cross-validation evaluation scores areused to indicate the model performance.

In statistics, many metrics can illustrate the accuracy of error rate for models,such as Mean Absolute Error (MAE), NMAE, Mean Square Error (MSE), andNormalized Mean Square Error (NMSE). We tried all of them to assess modelperformance and got similar conclusions. For all the models in this thesis report,we just show NAME results, defined in Equation 7.1, where y is the actual valueand yi is the estimation value.

NMAE =1n

∑ni=1‖yi − yi‖

1n

∑ni=1 yi

=

∑ni=1‖yi − yi‖∑n

i=1 yi(7.1)

7.1.3 Feature Sets

Five feature sets are used including the set with all available features (’all’), themanual set with only 17 ’rank’ features and ’nodeID’ (’RANK’) and sets chosenby three feature selection methods. For feature selection, the three methodsare introduced in the Chapter 2.1: RFE, feature selection from the model and’Univariate’ feature selection. As for RFE, we use the function RFECV inthe scikit-learn library [24] to realize feature ranking with recursive featureelimination and cross-validated selection of the best number of features; forfeature selection from the model method, we add one pre-training process forfeature selection; for ’Univariate’ feature selection, we choose f-regression using’Univariate’ linear regression test to select the four highest scoring features.

49

7.2 Batch Learning for Different Topologies with-out jammer

This section focuses on the prediction of RTT and average packet loss in threedifferent topologies without JamLab devices, corresponding to experiments #1,#2, #3 in Table 5.1. We use two machine learning models (the Random Forestregressor (’RF’) and the Lasso linear regressor (’LR’)) and five feature sets to dothe batch learning (including all features ’All’; three feature selection methods’SelectFromModel’, ’RFE’, ”Univariate”; ’Rank’ represents the feature ’rank’and nodeID feature combination). The last strategy is designed to help usunderstand the influence of topology feature ’rank’. The tables 7.1, 7.2, 7.3 and7.5 only show three critical feature set strategies (’All’, ”Univariate”, ’Rank’),and the complete results are displaied in Appendix A.

7.2.1 Target RTT

From Table 7.1, we can easily see that it is hard to directly predict singleRTT without any RTT preprocessing. However, once we use some aggregatedstatistics as prediction targets, e.g. average or the percentile, the predictionerror decreases more than 5 times in many cases. For example, dataset #1using all features, the NMAE for RTT without any preprocessing is 0.546, butwith average RTT is 0.0849, and with 80th percentile is 0.0832. The two RTTaggregation methods (average and 80th percentile) have similar performance.One possible reason may be there are many outliers in the raw data. Or rather,the raw RTT data value varies dramatically, so these seemingly unusual valuesare not actually outliers. However, the model can not predict these parts well.Both two aggregation methods make efforts to decrease the sensitive to outliers,so there is no big difference between average value and the 80th percentile value.

The most common type of topology in real scenarios is star shape, which isthe topology of experiment #3 with the controller at #126 node. We choosethe best model (average RTT as the target and ’Univariate’ feature reduction)owning the lowest NMAE error rate 0.0652 to show the detailed estimationperformance in Figure 7.1, where the real target value Ytrue as the x axis andthe estimation value Yesti as the y axis. We can see that in Random Forest figure,the predictive points are close the ideal prediction line (the black bold line in thefigures), different from the Lasso model where most points are far away deviatefrom the ideal prediction line. Besides, in the Table 7.1, the lowest NMAEis around 6% with Random Forest, but the Lasso model performs no betterthan 44%. Therefore, compared with the Lasso regression model, the RandomForest has better performance regarding both prediction accuracy and variance.Besides, in the Figures we use different colors to point out the reflectors. Wefound that the prediction performance is similar between one-hop and multi-hop nodes. Normally, the prediction of multi-hop nodes is more difficult thanone-hops because in ad hoc wireless sensor networks the topology is dynamicand hard to monitor. However, from the results we can see the current featureset already can cover the necessary information for multi-hop predictions.

50

The topology feature ’rank’ is an important feature. From the Table 7.1, withaverage RTT as the target, we can see that in experiments #1 and #2, thefeature set ’Rank’ (including the ’rank’ and ’nodeID’ features) performs bestamong several feature selecting strategies. In dataset #1 compared with otherfeature sets, ’Rank’ feature set has the lowest 0.0678 error rate, so as for dataset#2, the NMAE for the ’Rank’ feature set is 0.0687. In dataset # 3 althoughthe ’Rank’ set is not the best, it also has a very good performance with 0.0652NMAE. Therefore, it proves that in the feature set X, ’rank’ is one critical met-ric. In the scenarios without jammer, only with ’rank’ feature and destination’nodeID’, it is enough to get a good predictive accuracy.

Experiments #1, #2, #3 use three different topologies including two trees withroot at the edge (#1 and #2) and one star with root in the central(#3). Fromthe Table 7.1, we can deduce that Random Forest model has similar performancefor all these three data sets. It indicates that the Random Forest model hasgeneral applicability in different WSN topologies.

7.2.2 Target loss

From the Table 7.2 for loss prediction, we can get similar results as for RTTprediction.

The average loss metric can be predicted by ML models.

Through Table 7.2, Random Forest performance is far way better than Lassolinear model among three experiments with different topologies. In experiment#1, using the best combination with ’Rank’ feature set and Random Forestmodel, NMAE value is 0.0098 for 6h average loss, 0.0670 for 1h average loss and0.124 for 20min average loss, but for Lasso model, average error rate is higherthan 0.5.

The topology feature ’rank’ is still the important feature, which we can deducefrom the Table 7.1. Only using the ’Rank’ feature set with the destination’nodeID’ and ’rank’ feature, Random Forest can get good prediction accuracy.For 20min loss interval, ’Rank’ feature set has the best performance in all threedatasets, #1 is 0.124, #2 is 0.131 and #3 0.168.

Comparing the result of experiments with different topologies using the sameloss computing interval, the performance is similar. For example, to predict 1haverage loss using the ’Rank’ feature set, #1 NMAE is 0.0670, #2 NMAE is0.0680, and #3 error rate is 0.0855.

We choose three ’true value’ vs ’prediction value’ figures 7.3, 7.4 and 7.5 forexperiment # 3 with a star topology and ’Rank’ feature set to show the detailedevaluation performance. We can observe the effect of different loss computingintervals. Combined with the previous explanation with Table 6.5, it is easilyfound that with the same sending strategy, a longer time interval means morepossible values. As shown in the Table 7.2, three distinct time intervals are usedto calculate the average loss. Referring the data in the Table 7.2, it shows that

51

Figure 7.1: Experiment #3 RTT prediction with Random Forest model and’Univariate’ feature reduction.

Figure 7.2: Experiment #3 RTT prediction with Lasso model and ’Univariate’feature set

with shorter interval (20min), the performance is worse than the longer interval(1h and 6h). The lowest NMAE error rate for 20min loss is about 15%, which is15 times worse than the 6h loss. That is to say, the current prediction pipelineis not good enough to predict the loss with small-time granularity.

52

Figure 7.3: Experiment #3 Average loss prediction over 6h with Random Forestmodel and ’Rank’ feature set

Figure 7.4: Experiment #3 Average loss prediction over 1h with Random Forestmodel and ’Rank’ feature set

7.3 Batch Learning for Scenarios with Jammer

For the experiments with Jamlab activated, there are two kinds of jammertypes: fixed jammer in experiment #4 and moving jammer in experiment #5(details are in Chapter 5.2.3). There are three parts of the experiments #4

53

Figure 7.5: Experiment #3 Average loss prediction over 20min with RandomForest model and ’Rank’ feature set

with the same controller position, wireless channel and execution time exceptthe jammer setup. The #4-v1 and #4-v2 are two tests with one fixed jammer.The #4-v3 is a control-trial without any jammer. It is a good contrast test tocheck the influence of jammer attendance. In the following result discussion,the attention is focused on the comparison between controlled trial #4-v3 andother trials with Jamlab.

As the Random Forest is much better than the Lasso model, in this section weonly focus on the former model. All tests for the scenarios with jammer areshort experiments shown in the Table 6.1. Therefore, to get more loss samples,we omit the 6h computing interval and focus more on the 1h as well as 20min. Inaddition, RTT preprocessing strategies are the same still using the three metricsfor target RTT prediction, including RTT, and two aggregated metrics (averageRTT and 80th percentile RTT).

The feature sets are the same, implementing all three feature reduction meth-ods, all features set and ’Rank’ feature set. In the following model evaluationtable, only part of the results are displayed. Table 7.3 is the RTT predictionperformance, Table 7.5 is for 1h and 20mins average loss. Tables 7.4, 7.6 and7.7 show the feature details of different feature sets with either the best and theworst error rate. The complete model evaluation table for all models is shownin Appendix A.

54

7.3.1 Target RTT

According to the evaluation results in the Table 7.3, comparing with the con-trolled trial #4-v3, we can see that the Random Forest model has similar per-formance, around 8% NMAE in two fixed jammer data #4-v1 and #4-v2, butworse accuracy 13.3% in moving jammer dataset #5. This may result from theexperiment running time, since the all data sets for test #4 are longer than #5,and fewer samples will deteriorate the Random Forest model performance.

Moreover, unlike the without jammer scenarios, the ’Rank’ feature set is not abetter choice anymore. In all the experiments with jammer, the performance ofmodels using ’Univariate’ feature set is always better than ’Rank’ feature set.For example, #4-v1 ’Rank’ set NMAE is 0.0963 and ’Univariate’ feature set is0.0865; #5 ’Rank’ set is 0.150, but ’Univariate’ is 0.133. From the Table 7.4,we can find the possible reason behind this observation. In more complicatedscenarios with activated jammer, only using the topology feature is not enoughfor RTT prediction with good accuracy. The importance of other basic fea-tures such as ’distance’, ’node x’, ’node y’, ’RSSI’ and ’LQI’, show up. Besides,reviewing all features selected by univariate reduction method, excepting themanually added rank metrics, the ’nodeID’ metric is always involved, whichshows the importance of this metric.

7.3.2 Target loss

Looking through the loss Table 7.5, we can find that Random Forest performanceis variable. For example, to predict 1h average loss with ’Univariate’ featurereduction, NMAE for #4-v1 is 0.0349, #4-v2 is 0.0950, #5 is 0.0681 and withoutjammer #4-v3 is 0.0874. The moving jammer prediction result is not worse thanthe fixed jammer trials.

As the experiments without jammer, a shorter interval brings worse predictionperformance. The NMAE for 1h average loss all are below 10% and 20minaverage loss are below 20%.

As for the feature set, from Tables 7.6 and 7.7 we can see that the ’Rank’ featureset is not the best strategy anymore, and sometimes it is even the worst featuresets like for experiments #4-v1 and #5 with the 1h loss as the target. Someother features like ’time’, ’date’, distance’, ’node x’, ’node y’, ’distance’ and’RSSI’ play more important roles. Besides, which is the best feature set? Theanswer varies and all three feature reduction methods show up in the tables asthe best choices.

55

Table 7.1: RTT Prediction Results, without JamLab activated.

DataSource

RTTStrategy

Features RFNMAE

RF Std LRNMAE

LR Std

#1

RTTAll 0.546 0.0098 0.619 0.0126

Univariate 0.546 0.0098 0.632 0.0121

Rank 0.545 0.0114 0.823 0.0129

AverageRTT

All 0.0849 0.00288 0.454 0.0110

Univariate 0.0816 0.00323 0.473 0.0065

Rank 0.0678 0.00289 0.711 0.01314

80thPercentile

RTT

All 0.0832 0.00517 0.456 0.00918

Univariate 0.0802 0.00329 0.474 0.0076

Rank 0.06615 0.00397 0.693 0.0110

#2

RTTAll 0.565 0.0184 0.624 0.0115

Univariate 0.568 0.0170 0.636 0.0082

Rank 0.563 0.0179 0.798 0.0146

AverageRTT

All 0.105 0.0066 0.445 0.0115

Univariate 0.0988 0.00853 0.462 0.0147

Rank 0.0687 0.00464 0.668 0.0142

80thPercentile

RTT

All 0.105 0.0061 0.442 0.0078

Univariate 0.0936 0.00481 0.460 0.0131

Rank 0.0666 0.00619 0.652 0.0065

#3

RTTAll 0.600 0.0334 0.660 0.0187

Univariate 0.600 0.0268 0.660 0.0201

Rank 0.599 0.0154 0.707 0.0208

AverageRTT

All 0.0789 0.01243 0.468 0.01374

Univariate 0.0593 0.01189 0.467 0.01690

Rank 0.0652 0.00428 0.535 0.01399

80thPercentile

RTT

All 0.087 0.00896 0.488 0.01782

Univariate 0.0624 0.00794 0.488 0.01485

Rank 0.0693 0.01139 0.538 0.0212

Notes: The data in bold are discussed in the text

56

Table 7.2: loss Prediction Results, without JamLab activated.

DataSource

lossStrategy

Features RFNMAE

RF Std LRNMAE

LR Std

#1

6h lossAll 0.0117 0.00126 0.559 0.0076

Univariate 0.0279 0.00283 0.576 0.0096

Rank 0.0098 0.00176 0.796 0.0097

1h lossAll 0.0643 0.00492 0.660 0.01721

Univariate 0.0814 0.00510 0.670 0.0125

Rank 0.0670 0.00298 0.876 0.0110

20minloss

All 0.149 0.00469 0.823 0.0144

Univariate 0.152 0.0096 0.830 0.0165

Rank 0.124 0.0067 1.001 0.0182

#2

6h lossAll 0.0115 0.00191 0.500 0.0079

Univariate 0.0224 0.00335 0.507 0.0126

Rank 0.0173 0.00249 0.654 0.0141

1h lossAll 0.0743 0.00484 0.652 0.0.0162

Univariate 0.0850 0.00659 0.667 0.0135

Rank 0.0680 0.00367 0.792 0.0203

20minloss

All 0.177 0.0080 0.866 0.0264

Univariate 0.158 0.0108 0.884 0.0148

Rank 0.131 0.0078 0.985 0.0268

#3

6h lossAll 0.0063 0.00238 0.761 0.02517

Univariate 0.0063 0.00366 0.876 0.0317

Rank 0.0238 0.00535 0.998 0.0231

1h lossAll 0.0774 0.00840 1.020 0.0380

Univariate 0.0881 0.01097 1.033 0.0330

Rank 0.0855 0.01171 1.053 0.0310

20minloss

All 0.225 0.0238 1.302 0.0823

Univariate 0.169 0.0230 1.302 0.0615

Rank 0.168 0.0229 1.358 0.0798

Notes: The data in bold are discussed in the text

57

Table 7.3: Average RTT prediction results using Random Forest model, withJamLab activated

Data Source Feature Set RF NMAE RF Std

#4-v1All 0.119 0.0344

Univariate 0.0865 0.0364Rank 0.0963 0.03602

#4-v2All 0.0987 0.01210

Univariate 0.0785 0.01968Rank 0.0878 0.01896

#4-v3All 0.0856 0.02717

Univariate 0.0741 0.01899Rank 0.0789 0.02495

#5All 0.152 0.0294

Univariate 0.133 0.0231Rank 0.150 0.0235

Notes: The data in bold are discussed in the text

Table 7.4: Best and worst feature set for average RTT prediction using RandomForest model, with JamLab activated

jammerType

ExpLabel

Features RFNMAE

RF Std Selected Features

Fixed

#4-v1Univariate 0.0865 0.03636 ’nodeID’,’distance’,

’node x’,’time’,’rank’SelectFromModel 0.131 0.0385 ’node x’,’LQI’,’Light’,

’time’,’rank’

#4-v2Univariate 0.0785 0.01968 ’nodeID’,’node x’,’node y’,

’distance’,’rank’SelectFromModel 0.0992 0.01852 ’node x’,’node y’,’time’,

’LQI’,’rank’

#4-v3Univariate 0.0741 0.01899 ’nodeID’,’RSSI’,’node x’,

’distance’,’rank’SelectFromModel 0.0968 0.02871 ’distance’,’RSSI’,’LQI’,

’Light’,’rank’

Moving #5Univariate 0.133 0.0231 ’nodeID’,’LQI’,’node x’,

’distance’,’rank’SelectFromModel 0.157 0.0368 ’time’,’RSSI’,’LQI’,

’node x’,’rank’

Note: in bold represent the feature set with the best accuracy (the lowestNMAE error rate), the others represent the feature set with the best

performance (the highest NMAE error rate)..

58

Table 7.5: loss prediction results using Random Forest model, with JamLabactivated

Data Source Target Feature Set RF NMAE RF Std

#4-v1

1h lossAll 0.0436 0.00838

Univariate 0.0349 0.01457Rank 0.0569 0.02118

20min lossAll 0.165 0.0265

Univariate 0.124 0.0249Rank 0.162 0.0392

#4-v2

1h lossAll 0.0728 0.00792

Univariate 0.0950 0.02040Rank 0.0939 0.01712

20min lossAll 0.191 0.0383

Univariate 0.158 0.0235Rank 0.134 0.0435

#4-v3

1h lossAll 0.0934 0.01077

Univariate 0.0874 0.02999Rank 0.0859 0.02873

20min lossAll 0.242 0.0490

Univariate 0.180 0.0507Rank 0.184 0.0456

#5

1h lossAll 0.0566 0.01779

Univariate 0.0681 0.02706Rank 0.106 0.0252

20min lossAll 0.166 0.0363

Univariate 0.165 0.0648Rank 0.186 0.0350

Notes: The data in bold are discussed in the text

Table 7.6: Average loss over 1h prediction results, with JamLab activated.

jammerType

ExpLabel

Features RFNMAE

RF Std Selected Features

Fixed

#4-v1Univariate 0.0349 0.01457 ’time’,’date’,’node x’,

’distance’,’rank’Rank 0.0569 0.02118 ’nodeID’,’rank’

#4-v2SelectFromModel 0.0698 0.00974 ’node x’,’node y’,’time’,

’RSSI’,’rank’Univariate 0.0950 0.02040 ’nodeID’,’node x’,’node y’,

’distance’,’rank’

#4-v3Univariate 0.0874 0.02999 ’RSSI’,’date’,’node x’,

’distance’,’rank’SelectFromModel 0.118 0.0313 ’node y’,’LQI’,’Light’,

’RSSI’,’rank’

Moving #5All 0.0566 0.01779

Rank 0.1058 0.02519 ’nodeID’,’rank’

Note: Models in bold represent the feature set with the best accuracy (thelowest NMAE error rate), the others represent the feature set with the best

performance (the highest NMAE error rate).

59

Table 7.7: Average loss over 20min prediction results, with JamLab activated.

jammerType

ExpLabel

Features RFNMAE

RF Std Selected Features

Fixed

#4-v1Univariate 0.124 0.0249 ’nodeID’,’time’,’node x’,

’distance’,’rank’SelectFromModel 0.167 0.0453 ’node’ x’,’time’,’distance’,

’LQI’,’rank’

#4-v2RFE 0.148 0.0246 ’time’,’node x’,’node y’,

’rank’All 0.191 0.0383

#4-v3Univariate 0.180 0.0507 ’nodeID’,’RSSI’,’node x’,

’distance’,’rank’SelectFromModel 0.286 0.0426 ’node y’,’LQI’,’Light’,

’RSSI’,’rank’

Moving #5RFE 0.154 0.0625 ’RSSI’,’time’,’node x’,

’rank’SelectFromModel 0.191 0.0340 ’node x’,’time’,’RSSI’,

’LQI’,’rank’

Note: in bold represent the feature set with the best accuracy (the lowestNMAE error rate), the others represent the feature set with the best

performance (the highest NMAE error rate).

60

Chapter 8

Discussion

In this chapter, the results of the experiments and conclusions about the out-comes are discussed.

8.1 Experimental Results

• The most important contribution entails that it is possible tofind a model M which can accurately predict network perfor-mance metrics Y (RTT and round-trip average loss) from theenvironment and network statistics X. The model M is an MLpipeline composed of ML supervised regression models, features X pre-processing and targets Y aggregation strategies. It appears to be a strongtechnique which can be used in a future WSN management architecturementioned in Chapter 3. The new network management solution - ActiveMeasurements with Proxy can solve the difficulties of implementing activemeasurements in resource-constrained environments such as WSNs. Thisis also a critical step for the future IoT network development, which en-courages us to think more about using ML (Machine Learning) predictiontechnologies in different network management fields.

• Second, in order to accurately predict network performance, ag-gregation over the target samples is necessary.

For RTT prediction, instead of choosing the real-time single RTT value asthe target, we use aggregation methods such as average or 80th percentile.The reason is that there is much noise in the RTT raw data even afterfiltering out abnormal values. Therefore, we used a new probe-sendingstrategy shown in Figure 5.2 and applied the average method to reducenoise. From the results shown in Chapter 7, it is shown that predictinga single RTT value is hard, but it is possible to predict the average of apercentile aggregation RTT value with an error better than 8%.

Referring to the loss prediction, loss is an aggregation value computedusing the packet sequence number. It depends on the interval chosen to

61

calculate the loss. The current ML pipeline mentioned above performsbetter to predict loss with a longer interval. We are not sure about thereason, but there are some clues. We find in our dataset, sometimes datais lost for hours one time, which means the controller cannot receive anybackward probes for several hours. This big data lossing gap might causethe inaccuracy of loss prediction with shorter time interval. Up to now,we do not have enough evidence to diagnose the probing failure cause. Itmight be the testbed malfunction, or experimental bug. However, fromthe prediction result, it is obvious that current feature set X can notcapture the data lossing gap well.

• Third, the Random Forest model performs better than Lasso.In terms of model choosing, compared with the Lasso linear model, theRandom Forest regressor performs better across all traces, feature sets,test scenarios and target metrics. This means that the relation betweennetwork statistics X and performance metrics Y is not a simple linearrelationship, and a tree model like Random Forest can depict the relationbetter.

• Forth, topology features such as ’rank’ are important for networkperformance prediction. In scenarios without activated Jamlab, evenwith a small feature set (only ’rank’ and ’NodeID’), it is possible to predictthe current service-level metrics with an error rate lower than 7% forthe average RTT, about 8% for the 1h average loss, and below 17% for20min average loss. Several other manual feature sets are tried, such asonly 12 basic packet statistics, only topology statistics ’nbr’, with andwithout sensing feature. Then, it turns out that topology feature ’rank’with the combination of destination node ’NodeID’ are one critical featuresubset for the prediction, with which it is possible to get accurate resultsespecially for the without jammer scenarios (NMAE below 7% for averageRTT ,below 9% for 1h average loss, and below 17% for 20min average lossshown in Figures 7.1 and 7.2 ) .

For the experiments with Jamlab, the minor feature set only including’rank’ and ’NodeID’ is not enough for good accuracy. This may be be-cause the complexity of senarios with an active jammer. The jammer is aman-made interference and affects the sensor network topology. The sce-narios without Jamlab are more stable than the other experiments. In thejammer activated environment, the importance of other features such asgeographic coordinates of the sensor nodes and the probing time show up,which can reflect the behavior of jammers and help the ML model to findout the relation between X and Y . This indicates that the ’rank’ topol-ogy feature is critical information to predict the network performance,but several other features are also necessary to achieve a sound predictionaccuracy, especially in more complex situations.

• At last, from the results we can conclude that the ML pipelinewith Random Forest model allows for accurate prediction in dif-ferent scenarios. In this project, we set up seven different experimentswith varying controller positions and jammer situations to simulate differ-ent scenarios. The pipeline with Random Forest has similar performance

62

in different topologies, with or without jammer. However, there is somedifference in model performance between target RTT and loss.

To be specific, for average RTT prediction, from the results it is obvi-ous that in no jammer scenarios the model performs better than in fixedjammer scenarios. Furthermore, the moving jammer scenario is about 5percent (NMAE) worse than the above two situations. Since the movingjammer scenario is the most complicated environment, the WSN topologykeeps changing with movement of the jammer. Besides, the experiment #5for moving jammer is the shortest test with the smallest dataset. There-fore, both of these factors may be to blame for bad accuracy. It is possiblethat the Random Forest can have a good performance in the experiment#5 moving jammer scenario with a longer run time. The results mayimply that in more complex scenarios, the used features cannot capturethe relation characteristics as well as in simpler scenarios. Although fordifferent scenarios the model performs a little bit differently, generally,NMAE errors for any scenarios to predict average RTT all are below alevel of 15%. Therefore, we can conclude that the current pipeline withthe Random Forest model more or less has the universality for averageRTT prediction.

For loss prediction, the results change a little bit. The Random Forestmodel has similar performance among three different types of experiments(without jammer, with fixed jammer, or with moving jammer). FromFigures 7.3 and 7.5, we see that the NMAE errors are smaller than 10%for the 1h average loss and below 18% for 20min average loss. For thesame loss target and different datasets, there is no statistically significantdifference between the NMAE scores.

Overall, the current ML pipeline with the Random Forest model is appli-cable with good result for WSNs network performance prediction. For lossprediction, it has better results, which may be because the current featureset is enough for loss prediction but not for RTT prediction. Therefore,if we want to improve RTT prediction accuracy, exploring more relatedfeatures is a potential solution.

8.2 Limitations

Generally speaking, the results show that the approach we put forward is feasiblefor predicting service-level metrics in the WSNs, even though our approach haslimitations.

• The first limitation is the trade-off between the cost of repeatedprobes and prediction accuracy. As introduced in in the Section 5, inorder to adopt the average RTT, the controller sends sequentially n probesto each reflector node. This will increase the network load. However,decreasing the traffic load is the original research challenge. Therefore,about the compromise between target preprocessing strategies and cost,we haven’t done much research in this project.

63

• The second limitation related to the reliability of service met-rics. Compared with direct active measurements, the performance met-rics gotten from our approach is an evaluation value predicted throughthe machine learning model, which has a prediction error.

• The third limitation is about asynchrony between the data col-lection schedule and prediction process. As shown in the Section 5,the collection of some data has its own separate schedules, like topologyfeatures (every ∆tnbr to report neighbor list information). Meanwhile,active measurements may have a different probe-sending interval ∆t. Sothese two processes might be out-of-step and cannot be matched. Some-times, the freshness of data may be a problem. For example, the predictionis not made by the latest feature set X. This may be one possible reasonto deteriorate the reliability of the prediction.

64

Chapter 9

Conclusion and FutureWork

This chapter presents an overview of the work done during this thesis, highlightsthe contributions and also gives possible direction for the continuation of thework produced.

9.1 Conclusion

In this study we develop and evaluate machine learning models for predictingnetwork performance metrics in an IoT environment based on infrastructure anddevice features. These models can be implemented in an active measurementproxy, described in the thesis, aiming at reducing the overhead stemming fromactive measurements. This thesis investigates the applicability of a statisticallearning models to predict the network performance such as RTT and averageloss for WSN and provide a simple machine learning framework with off-linemodels.

The project focuses on a IEEE 802.15.4 network with the Contiki OS and choosesthe IETF standard for active measurements. Seven experiments with differentconfigurations are designed to simulate different network scenarios. We rewrotethe testbed configuration scipt in C and then did a series of experiments in areal-world testbed to generate data traces. After that, based on the collecteddata, machine learning models for the delay (average RTT) and loss (round-tripaverage loss) predictions are developed and evaluated.

Taken together, the results show that the approach generalizes across multiplescenarios. For stable scenarios without jammer, no matter which topology, thethesis approach enables to get good prediction accuracy: NMAE lower than 10%both for average RTT and 1h average loss; about 16% for 20min average loss.For data from experiments with jammer, there is some different performance

65

between delay and loss. The prediction accuracy of loss prediction showed tobe independent of the jammer situations (i.e. if with fixed, moving jammer orwithout jammer). However, RTT prediction is related to jammer configurations,with jammer performance is worse than without, and for the most complexmoving jammer scenario with added interference, the error rate is 5% higherthan other scenarios.

Compared with the linear Lasso model, tree-based Random Forest achievesmuch better performance. Furthermore, a discussion on feature importance im-plies that network topology statistics (i.e. RPL ’rank’ feature) is most valuablefor service-level metrics prediction, especially in stable environments. For morecomplex scenarios even lower error rates were achieved when including positionand time-related features.

66

9.2 Future Work

What we can do in the future.

First future work includes extending the realism of the testbed. Now exceptactive measurement probes, the current testbed does not carry any load relatedto a service. In real-world application scenarios, there are different traffic sourcesin the network. Would the thesis’ approach still work in a real case wheredifferent services may run concurrently on the same physical network? Thisneeds to be explored. Furthermore, different WSN tasks target diverse sensingmetrics which are connected with the sensing feature group in our predictionframework. For all experiments in the current thesis, only the daylight intensityis included. Some other environmental metrics like temperature and humidity,also are interesting features for the service-level metrics prediction.

Secondly, in this thesis, we primarily use batch learning for modeling RTT andloss. We know that WSNs are changeable network environments with dynamictopology. To take advantage of these characteristics, future work must copewith additional ML techniques that utilize the dynamicity of IoT systems, suchas online learning methods.

Data collection and target prediction are two critical modules for this framework.The coordination and cooperation between them are the keys to the successof this approach. For different use cases with different service requirements,choosing a matched data collection strategy is an important issue.

At last, as introduced in the Chapter 3, the active-measurement proxy shouldbe fully implemented in an IoT setting. This includes interception of pack-ets, estimation of delay and loss, and the creation of packet responses in theactive-measurement the proxy. During the design of proxy, improving the dataprocessing to optimize the system operation speed and other capabilities is alsoa major potential extension of the work.

67

Appendices

68

Appendix A

Evaluation

A.1 RTT Prediction without Jammer

The full evaluation table for all the RTT prediction models. In the tablesshowing the result with Jammer, color red represents the feature set with thelowest error rate, and color yellow is the feature set with the highest error.

69

Table A.1: RTT Prediction Results, without JamLab activated.

Datasource

RTTStrategy

Features RFNMAE

RF Std LRNMAE

LR Std

#1

RTT

All 0.546 0.0098 0.619 0.0126

SelectFromModel 0.550 0.0102 0.629 0.0081

RFE 0.514 0.0072 0.627 0.0172

Univariate 0.546 0.0098 0.632 0.0121

Rank 0.545 0.0114 0.823 0.0129

AverageRTT

All 0.0849 0.00288 0.454 0.0110

SelectFromModel 0.0757 0.00572 0.472 0.0087

RFE 0.0873 0.00408 0.465 0.0072

Univariate 0.0816 0.00323 0.473 0.0065

Rank 0.0678 0.00289 0.711 0.01314

80Percentile

RTT

All 0.0832 0.00517 0.456 0.00918

SelectFromModel 0.0760 0.00492 0.466 0.0037

RFE 0.0867 0.00853 0.462 0.0147

Univariate 0.0802 0.00329 0.474 0.0076

Rank 0.06615 0.00397 0.693 0.0110

A.2 Loss Prediction without Jammer

70

#2

RTT

All 0.565 0.0184 0.624 0.0115

SelectFromModel 0.569 0.0165 0.636 0.0175

RFE 0.562 0.0146 0.633 0.01813

Univariate 0.568 0.0170 0.636 0.0082

Rank 0.563 0.0179 0.798 0.0146

AverageRTT

All 0.105 0.0066 0.445 0.0115

SelectFromModel 0.118 0.0070 0.462 0.0108

RFE 0.106 0.0047 0.450 0.0098

Univariate 0.0988 0.00853 0.462 0.0147

Rank 0.0687 0.00464 0.668 0.0142

80Percentile

RTT

All 0.105 0.0061 0.442 0.0078

SelectFromModel 0.110 0.0064 0.460 0.0115

RFE 0.104 0.0076 0.454 0.0056

Univariate 0.0936 0.00481 0.460 0.0131

Rank 0.0666 0.00619 0.652 0.0065

#3

RTT

All 0.600 0.0334 0.660 0.0187

SelectFromModel 0.601 0.0328 0.660 0.0174

RFE 0.605 0.0280 0.664 0.0196

Univariate 0.600 0.0268 0.660 0.0201

Rank 0.599 0.0154 0.707 0.0208

AverageRTT

All 0.0789 0.01243 0.468 0.01374

SelectFromModel 0.0675 0.01028 0.468 0.0122

RFE 0.0648 0.01032 0.468 0.0125

Univariate 0.0593 0.01189 0.467 0.01690

Rank 0.0652 0.00428 0.535 0.01399

80Percentile

RTT

All 0.087 0.00896 0.488 0.01782

SelectFromModel 0.0889 0.00695 0.488 0.0154

RFE 0.0697 0.00875 0.488 0.0134

Univariate 0.0624 0.00794 0.488 0.01485

Rank 0.0693 0.01139 0.538 0.0212

A.3 RTT Prediction with Jammer

71

Table A.3: Loss Prediction Results, without JamLab activated.

Datasource

LossStrategy

Features RFNMAE

RF Std LRNMAE

LR Std

#1

6h Loss

All 0.0117 0.00126 0.559 0.0076

SelectFromModel 0.0706 0.00596 0.566 0.0104

RFE 0.0194 0.00145 0.562 0.0085

Univariate 0.0279 0.00283 0.576 0.0096

Rank 0.0098 0.00176 0.796 0.0097

1h Loss

All 0.0643 0.00492 0.660 0.01721

SelectFromModel 0.0824 0.00603 0.665 0.0154

RFE 0.0725 0.00465 0.662 0.0134

Univariate 0.0814 0.00510 0.670 0.0125

Rank 0.0670 0.00298 0.876 0.0110

20minLoss

All 0.149 0.00469 0.823 0.0144

SelectFromModel 0.177 0.0059 0.827 0.0223

RFE 0.157 0.0067 0.823 0.0254

Univariate 0.152 0.0096 0.830 0.0165

Rank 0.124 0.0067 1.001 0.0182

#2

6h Loss

All 0.0115 0.00191 0.500 0.0079

SelectFromModel 0.0168 0.00402 0.509 0.0226

RFE 0.0157 0.00439 0.500 0.0116

Univariate 0.0224 0.00335 0.507 0.0126

Rank 0.0173 0.00249 0.654 0.0141

1h Loss

All 0.0743 0.00484 0.652 0.0.0162

SelectFromModel 0.0808 0.00890 0.662 0.0150

RFE 0.0825 0.00782 0.652 0.03071

Univariate 0.0850 0.00659 0.667 0.0135

Rank 0.0680 0.00367 0.792 0.0203

20minLoss

All 0.177 0.0080 0.866 0.0264

SelectFromModel 0.539 0.0175 0.879 0.0179

RFE 0.191 0.0150 0.866 0.0200

Univariate 0.158 0.0108 0.884 0.0148

Rank 0.131 0.0078 0.985 0.0268

A.4 Loss Prediction with Jammer

72

#3

6h Loss

All 0.0063 0.00238 0.761 0.02517

SelectFromModel 0.0091 0.00336 0.982 0.0232

RFE 0.0075 0.00380 0.870 0.0435

Univariate 0.0063 0.00366 0.876 0.0317

Rank 0.0238 0.00535 0.998 0.0231

1h Loss

All 0.0774 0.00840 1.020 0.0380

SelectFromModel 0.0882 0.01102 1.033 0.0452

RFE 0.0866 0.01495 1.026 0.04534

Univariate 0.0881 0.01097 1.033 0.0330

Rank 0.0855 0.01171 1.053 0.0310

20minLoss

All 0.225 0.0238 1.302 0.0823

SelectFromModel 0.225 0.02450 1.302 0.0592

RFE 0.226 0.00169 1.303 0.0718

Univariate 0.169 0.0230 1.302 0.0615

Rank 0.168 0.0229 1.358 0.0798

73

Table A.5: Average RTT Prediction Results, with JamLab activated.

JammerType

ExpLabel

Features RFNMAE

RF Std Selected Features

Fixed

#4-v1

All 0.119 0.0344SelectFromModel 0.131 0.0385 ’node x’,’LQI’,’Light’,

’time’,’rank’RFE 0.113 0.0316 ’time’, ’node x’

Univariate 0.0865 0.03636 ’node x’,’distance’,’node x’,’time’,’rank’

Rank 0.0963 0.03602 ’NodeID’,’rank’

#4-v2

All 0.0984 0.01210SelectFromModel 0.0992 0.01852 ’node x’,’node y’,’time’,

’LQI’,’rank’RFE 0.0786 0.01922 ’time’, ’node x’, ’node y’

Univariate 0.0785 0.01968 ’NodeID’,’node x’,’node y’,’distance’,’rank’

Rank 0.0878 0.01896 ’NodeID’,’rank’

#4-v3

All 0.0856 0.02717SelectFromModel 0.0968 0.02871 ’distance’,’RSSI’,’LQI’,

’Light’,’rank’RFE 0.0824 0.04141 ’RSSI’, ’distance’

Univariate 0.0741 0.01899 ’NodeID’,’RSSI’,’node x’,’distance’,’rank’

Rank 0.0789 0.0250 ’NodeID’,’rank’

Moving#5

All 0.152 0.0294SelectFromModel 0.157 0.0368 ’time’,’RSSI’,’LQI’,

’node x’,’rank’RFE 0.155 0.0246 ’RSSI’, ’LQI’, ’time’,

’node x’Univariate 0.133 0.0231 ’NodeID’,’LQI’,’node x’,

’distance’,’rank’Rank 0.150 0.0235 ’NodeID’,’rank’

74

Table A.6: 1h Average Loss Prediction Results, with JamLab activated.

JammerType

ExpLabel

Features RFNMAE

RF Std Selected Features

Fixed

#4-v1

All 0.0436 0.00838SelectFromModel 0.0376 0.116 ’node x’, ’time’, ’RSSI’,

’distance’RFE 0.0429 0.01901 ’RSSI’, ’time’, ’node x’,

’distance’Univariate 0.0349 0.01457 ’Time’,’date’,’node x’,

’distance’,’rank’Rank 0.0569 0.02118 ’NodeID’,’rank’

#4-v2

All 0.0728 0.00792SelectFromModel 0.0698 0.00974 ’node x’,’node y’,’time’,

’RSSI’,’rank’RFE 0.0704 0.01034 ’RSSI’, ’LQI’, ’time’,

’node x’, ’node y’Univariate 0.0950 0.02040 ’NodeID’,’node x’,’node y’,

’distance’,’rank’Rank 0.0939 0.01712 ’NodeID’,’rank’

#4-v3

All 0.0934 0.01077SelectFromModel 0.118 0.0313 ’node y’,’LQI’,’Light’,

’RSSI’,’rank’RFE 0.105 0.0289 ’srcIP’, ’NodeID’, ’CH’,

’RSSI’, ’LQI’, ’time’,’date’, ’src x’, ’src y’,’node x’, ’node y’, ’dis-tance’, ’Light’

Univariate 0.0874 0.02999 ’RSSI’,’date’,’node x’,’distance’,’rank’

Rank 0.0859 0.02873 ’NodeID’,’rank’

Moving#5

All 0.0566 0.01779SelectFromModel 0.0890 0.02031 ’node y’, ’LQI’, ’Light’,

’RSSI’RFE 0.0779 0.03930 ’RSSI’, ’time’, ’node x’,

’rank’Univariate 0.0681 0.02706 ’RSSI’, ’date’, ’node x’,

’distance’Rank 0.1058 0.02519 ’NodeID’,’rank’

75

Table A.7: 20min Average Loss Prediction Results, with JamLab activated.

JammerType

ExpLabel

Features RFNMAE

RF Std Selected Features

Fixed

#4-v1

All 0.165 0.0265SelectFromModel 0.167 0.0453 ’node’ x’,’time’,’distance’,

’LQI’,’rank’RFE 0.0129 0.0320 ’time’, ’node x’, ’dis-

tance’Univariate 0.124 0.0249 ’NodeID’,’time’,’node x’,

’distance’,’rank’Rank 0.134 0.0435 ’NodeID’,’rank’

#4-v2

All 0.191 0.0383SelectFromModel 0.185 0.0197 ’node x’, ’node y’, ’LQI’,

’time’RFE 0.148 0.0246 ’time’,’node x’,’node y’,

’rank’Univariate 0.158 0.0235 ’NodeID’,’node x’,

’node y’,’distance’Rank 0.162 0.0392 ’NodeID’,’rank’

#4-v3

All 0.242 0.0490SelectFromModel 0.286 0.0426 ’node y’,’LQI’,’Light’,

’RSSI’,’rank’RFE 0.248 0.0415 ’NodeID’, ’RSSI’, ’LQI’,

’date’, ’src x’, ’src y’,’node x’, ’node y’, ’dis-tance’, ’Light’

Univariate 0.180 0.0507 ’NodeID’,’RSSI’,’node x’,’distance’,’rank’

Rank 0.184 0.0456 ’NodeID’,’rank’

Moving#5

All 0.166 0.0363SelectFromModel 0.191 0.0340 ’node x’,’time’,’RSSI’,

’LQI’,’rank’RFE 0.154 0.0625 ’RSSI’,’time’,’node x’,

’rank’Univariate 0.165 0.0648 ’NodeID’,’RSSI’,

’node x’, ’distance’Rank 0.186 0.0350 ’NodeID’,’rank’

76

Bibliography

[1] Ericsson. (1999) Networked society insights. [Online]. Available: https://www.ericsson.com/en/trends-and-insights/networked-society-insights

[2] L. Da Xu, W. He, and S. Li, “Internet of things in industries: A survey,”IEEE Transactions on industrial informatics, vol. 10, no. 4, 2014.

[3] “White paper - cellular networks for massive iot - enabling low power widearea applications,” Ericsson, Tech. Rep. Uen 284 23-3278, January 2016.[Online]. Available: https://www.ericsson.com/assets/local/publications/white-papers/wp iot.pdf

[4] I. F. Akyildiz, W. Su, Y. Sankarasubramaniam, and E. Cayirci, “Wirelesssensor networks: a survey,” Computer networks, vol. 38, no. 4, 2002.

[5] V. Pereira, J. S. Silva, and E. Monteiro, “A framework for wireless sensornetworks performance monitoring,” in World of Wireless, Mobile and Mul-timedia Networks (WoWMoM), 2012 IEEE International Symposium on a.IEEE, 2012.

[6] U. Wetzker, I. Splitt, M. Zimmerling, C. A. Boano, and K. Romer,“Troubleshooting wireless coexistence problems in the industrial internetof things,” in Computational Science and Engineering (CSE) and IEEE IntlConference on Embedded and Ubiquitous Computing (EUC) and 15th IntlSymposium on Distributed Computing and Applications for Business Engi-neering (DCABES), 2016 IEEE Intl Conference on. IEEE, 2016.

[7] V. N. S. S. Pereira, “Performance measurement in wireless sensor networks,”Ph.D. dissertation, Departamento de Engenharia Informtica,Faculdade deCincias e Tecnologia,Universidade de Coimbra, February 2016.

[8] A. Johnsson and C. Rohner, “On performance observability in iot systemsusing active measurements,” in IFIP/IEEE Network Operations and Man-agement Symposium (NOMS), April 2018.

[9] M. Schuß, C. A. Boano, M. Weber, and K. Romer, “A competition to pushthe dependability of low-power wireless protocols to the edge,” in Proceedingsof the European Conference on Wireless Sensor Networks (EWSN), 2017.

[10] “Network management system: Best practices white paper,” Cisco, Tech.Rep., July 2007. [Online]. Available: https://www.cisco.com/c/en/us/support/docs/availability/high-availability/15114-NMS-bestpractice.html

77

[11] R. Rosen, Internet Control Message Protocol (ICMP). Berke-ley, CA: Apress, 2014. [Online]. Available: https://doi.org/10.1007/978-1-4302-6197-1 3

[12] V. Mohan, Y. J. Reddy, and K. Kalpana, “Active and passive networkmeasurements: a survey,” International Journal of Computer Science andInformation Technologies, vol. 2, no. 4, 2011.

[13] M. J. Zekauskas, A. Karp, B. Teitelbaum, S. Shalunov, and J. W.Boote, “A one-way active measurement protocol (OWAMP),” RFC 4656,September 2006. [Online]. Available: https://tools.ietf.org/html/rfc4656

[14] K. Hedayat, R. Krzanowski, A. Morton, K. Yum, and J. Babiarz, “Atwo-way active measurement protocol (TWAMP),” RFC 5357, March 2008.[Online]. Available: https://tools.ietf.org/html/rfc5357

[15] S. Baillargeon, C. Flinta, and A. Johnsson, “Ericsson two-way activemeasurement protocol (twamp) value-added octets,” no. ISSN: 2070-1721,November 2012. [Online]. Available: https://tools.ietf.org/html/rfc6802

[16] R. Kalman, “On the general theory of control systems,” IRE Transactionson Automatic Control, vol. 4, no. 3, 1959.

[17] K. Wolosz, U. Bodin, and L. Riliskis, “A measurement study for predictingthroughput from lqi and rssi,” Lulea University of Technology, 2012.

[18] R. Yanggratoke, J. Ahmed, J. Ardelius, C. Flinta, A. Johnsson, D. Gillblad,and R. Stadler, “Predicting service metrics for cluster-based services usingreal-time analytics,” in Network and Service Management (CNSM), 201511th International Conference on. IEEE, 2015, pp. 135–143.

[19] Y. Rerngvit, A. Jawwad, A. John, F. Christofer, J. Andreas, G. Daniel,and S. Rolf, “A service-agnostic method for predicting service metrics inreal time,” International Journal of Network Management, vol. 28, no. 2,2018.

[20] A. Johnsson, C. Meirosu, and C. Flinta, “Online network performancedegradation localization using probabilistic inference and change detection,”in Network Operations and Management Symposium (NOMS). IFIP/IEEE,2014.

[21] G. Armitage, J. Kennedy, S. Nguyen, J. Thomas, and S. Ewing, “Householdinternet and the need for speed: evaluating the impact of increasingly onlinelifestyles and the internet of things,” Transport, vol. 500, 2017.

[22] F. Xia, “Qos challenges and opportunities in wireless sensor/actuator net-works,” Sensors, vol. 8, no. 2, 2008.

[23] R. Tibshirani, “Regression shrinkage and selection via the lasso,” Journalof the Royal Statistical Society. Series B (Methodological), 1996.

[24] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel,M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Pas-sos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, “Scikit-learn: Machine learning in Python,” Journal of Machine Learning Research,vol. 12, 2011.

78

[25] J. Friedman, T. Hastie, and R. Tibshirani, The elements of statistical learn-ing : Data mining, inference, and prediction (Springer series in statistics).New York: Springer, 2001.

[26] L. Breiman, “Random forests,” Machine learning, vol. 45, no. 1, 2001.

[27] A. Vidhya. A comprehensive guide to data exploration. [Online]. Available:https://www.analyticsvidhya.com/blog/2016/01/guide-data-exploration/

[28] “Contiki: The open source os for the internet of things,” accessed: 2018-07-13. [Online]. Available: http://www.contiki-os.org/index.html#why

[29] A. Dunkels, B. Gronvall, and T. Voigt, “Contiki - a lightweight and flexibleoperating system for tiny networked sensors,” in Local Computer Networks,2004. 29th Annual IEEE International Conference on. IEEE, 2004.

[30] T. Winter, P. Thubert, A. Brandt, J. Hui, R. Kelsey, P. Levis, K. Pister,R. Struik, J. P. Vasseur, and R. Alexander, “RPL: Ipv6 routing protocol forlow-power and lossy networks,” RFC 6550, March 2012. [Online]. Available:https://tools.ietf.org/html/rfc6550

[31] O. Gnawali and P. Levis, “The minimum rank with hysteresisobjective function,” RFC 6719, September 2012. [Online]. Available:https://tools.ietf.org/html/rfc6719

[32] C. A. Boano, T. Voigt, C. Noda, K. Romer, and M. Zuniga, “Jamlab: Aug-menting sensornet testbeds with realistic and controlled interference gener-ation,” in Information Processing in Sensor Networks (IPSN), 2011 10thInternational Conference on. IEEE, 2011.

[33] A. B. M. Z. Antonio Lin Colina, Alvaro Vives and E. Pietrosemoli, “Iot in 5days,” March 2015, [Online; posted 02-June-2018]. [Online]. Available: http://wireless.ictp.it/school 2015/book/# ieee 802 15 4 channels and pan id

[34] “IEEE802.15.4,” http://www.ieee802.org/15/pub/TG4.html, accessed:2018-06-13.

[35] M. Sauter, From GSM to LTE: an introduction to mobile networks andmobile broadband. Wiley, 2011.

[36] D. Pyle, “Data preparation for data mining,”https://pdfs.semanticscholar.org/470a/828d5e3962f2917a0092cc6ba46ccfe41a2a.pdf,accessed: 2018-05-12.

[37] S. Kotsiantis, D. Kanellopoulos, and P. Pintelas, “Data preprocessing forsupervised leaning,” International Journal of Computer Science, vol. 1,no. 2, 2006.

79

www.kth.se

80TRITA-EECS-EX-2018:788