real-time control in industrial iot - diva...

Malardalen UniversitySchool of Innovation Design and Engineering

Vasteras, Sweden

Thesis for the Degree of Master of Science in Intelligent EmbeddedSystem 30.0 credits

REAL-TIME CONTROL ININDUSTRIAL IOT

Alma [email protected]

Pavlos [email protected]

Examiner: Moris BehnamMalardalen University, Vasteras, Sweden

Supervisor: Saad MubeenMalardalen University, Vasteras, Sweden

Company supervisor: Kristian SandstromABB Corporate Research, Vasteras, Sweden

Company supervisor: Hongyu Pei Breivold,ABB Corporate Research, Vasteras, Sweden

June 15, 2015

Malardalen University Master Thesis

Acknowledgements

We would like to thank our supervisor from MDH Dr. Saad Mubeen, our supervisors from ABBProfessor Kristian Sandstrom and Dr. Hongyu Pei Breivold for the collaboration and guidanceduring the thesis. We would also like to thank our examiner Dr. Moris Behnam for his precioushelp and advice.

i


Abstract

Great advances in cloud computing have drawn the interest of industry. Cloud infrastructures areused, mainly, for monitoring the shop floor. In recent years cloud technologies combined with IoTtechnologies have initiated the effort to close loops between industrial applications and cloud infras-tructures. This thesis examines the effect of including remote servers, both local and centralized,in the closed control loop. Specifically, we investigate how delays and jitter affect the closed controlloop. A prototype is developed to include servers on a control loop and inject delays and jitter inthe network. Furthermore, delay mitigation mechanisms are proposed and, using the prototype, anumber of experiments are performed to evaluate them. The mitigation mechanisms focus mainlyon delays and jitter that are larger than the period of closed control loop. The proposed mechanismsimprove the closed control loop response but still fall short compared to the performance of the closecontrol loop when it executes locally, without any servers included. We also, show that local serverscan be included in the closed control loop without significant degradation in the performance of thesystem.

ii


Table of Contents

1 Introduction 11.1 Problem statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Thesis goal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.3 Thesis outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

2 Background 22.1 Internet of Things . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22.2 Cloud computing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32.3 Fog computing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2.3.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32.3.2 Cloud vs Fog . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42.3.3 Fog and similar technologies . . . . . . . . . . . . . . . . . . . . . . . . . . . 42.3.4 Fog architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52.3.5 Another definition of Fog computing . . . . . . . . . . . . . . . . . . . . . . 6

2.4 Offloading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.4.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.4.2 Computational offloading approaches . . . . . . . . . . . . . . . . . . . . . 7

2.5 Industrial automation systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82.5.1 Automation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82.5.2 Cloud-based automation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102.5.3 Fog in industrial applications . . . . . . . . . . . . . . . . . . . . . . . . . . 102.5.4 Smart factory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

3 Research method 12

4 System development 144.1 Architecture overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144.2 Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

4.2.1 System design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144.2.2 Delay mitigation mechanisms . . . . . . . . . . . . . . . . . . . . . . . . . . 17

4.3 End device set-up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

5 Experimental evaluation 205.1 Experimental set-up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205.2 Local Response . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

5.2.1 Constant delays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215.2.2 Jitter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225.2.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

5.3 Delay mitigation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265.3.1 Prediction methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265.3.2 Adaptive PI controller . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

6 Related work 356.1 Cloud and Fog architectures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 356.2 Offloading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 356.3 Cloud-based automation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 356.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

7 Conclusion 37

8 Future work 38

9 References 39

i


1 Introduction

Technology has gone through a big boost in the recent years. The arrival of cloud computing hasmade sharing and accessing data an everyday occurrence. It has made working with big streamsof data easy and manageable. It supports the storage of such data with flexibility.

The development of Internet of things (IoT) [1] has brought connectivity among the mostdiverse devices and has introduced a new kind of data flow. The future of IoT will bring even moreconnectivity and interaction among the simplest and most common devices (e.g., light bulbs andtoasters). The IoT is therefore bringing more and more devices into the connected world, whilecloud computing is able to support more and more users and their increasing amount of data. Itis clear that both of these technologies are on the rise and are here to stay. It is only natural thatthese two technologies start coming together at some point.

The automation Industry is an area that uses both of these technologies. Having the enddevices connected and networking together enables them to send and receive the information fromeach other. This means that practically the devices can demand the information when they needit instead of waiting for a scheduled response. It simplifies task scheduling as well as the scalabilityof the system. Having a cloud supervise all the data flow is beneficial for fault detecting andmaintenance monitoring. Cloud computing can also be used as a way to backup a great amountof data. Furthermore, clouds can be used on a higher level between multiple plants to improveefficiency and collaboration between them at different sites.

1.1 Problem statement

Cloud computing and IoT are typically used at different levels. IoT, which is used at lower levels,deals mostly with the end devices, their connections and the protocols they use. Cloud computingis used at higher levels with the end users and deals with handling and storage of data. In otherwords, IoT works with low latency and, in the industrial automation, real-time environment. Onthe other hand, the cloud is connected to the system through a Wide Area Network (WAN) andtherefore, it has unpredictable latencies. Thus, the following question comes to mind:“Is it possibleto construct a stable and reliable control system while using cloud computing as a part of it?” Ifthe cloud is brought closer to the devices, making a more secure and stable connection, there isno doubt that the automation industry will benefit from it. It would enable a centralized controlwhile minimizing costs. But moving parts of the control system is not simple since it can affectthe stability, robustness and reliability of the system.

1.2 Thesis goal

The goal of this thesis is to investigate ways of using the cloud in a closed loop control system andthe impacts it will have on the system itself. The thesis includes building a prototype to asses theeffects of having the cloud in the control system and evaluating the results.

1.3 Thesis outline

The rest of the thesis is organized as follows: Section 2 explains the terms and relevant technologieswhich are used throughout the document. Section 3 presents the research method that is used inthis thesis. Section 4 explains the prototype system and its architecture. Section 5 presents theexperimental evaluation. Section 6 summarizes the related work done in this and related fields.Finally, Section 7 concludes the thesis and Section 8 presents the recommendations for the futurework.

1


2 Background

2.1 Internet of Things

The internet of things (IoT) is a term that first showed up in 1999 and in the recent years quicklygained wide popularity. It depicts an idea of various devices, such as sensors and embedded devices- the things, being connected to each other through the internet and using it as a means of sharingdata and information. The things might have some level of intelligence that enables them to beaware of the surroundings and act accordingly. This simple idea offers a lot of potential to awide range of fields and applications (health monitoring, home automation, ambient intelligence,augmented reality applications, smart cars, smart cities, ...) so it is not a big surprise it has gainedsuch a huge following. But because it covers such a wide range it makes it hard to narrow downor generalize the fundamental characteristics of the IoT concept. There is no set definition of theterm and different authors emphasize different aspects of it. Some focus on the networking or theconnectivity of the devices and the way this connectivity is achieved, others focus on the fact thatthe things are embedded devices with limited resources and power saving features. One of themore general definitions, taken from [2] is:

“The Internet of Things (IoT) is the interconnection of uniquely identifiable embed-ded computing devices within the existing Internet infrastructure. Typically, IoT isexpected to offer advanced connectivity of devices, systems, and services that goes be-yond machine-to-machine communications (M2M) and covers a variety of protocols,domains, and applications.”

As the idea of many connected devices grew into separate fields, many new concepts startedforming. This, however, created a big pool of terms that are sometimes used interchangeably suchas Web of Things (WoT), Internet of Everything (IoE), Embedded Web, Future Internet of Things(FIoT). Some even equalize it with more specific terms such as Machine-to-Machine (M2M) oreven Industrie 4.0. It is important to note that the IoT is the most general term and although itmight be related to a large part of what all of these terms specify. However, all these other termsare created for some specific sub-area which it focuses on and shouldn’t be confused with IoT.

Since the IoT networks simple embedded devices, limited by their CPU capabilities, powersources and memory, it naturally found its way into many applications. Some fields, like automationand ambient intelligence, were not part of the original idea, especially since they don’t require aninternet connection. However, the potential of a great number of networked embedded systems islarge, whether to expand the limited functionalities of the devices or to better utilize the massiveamount of data they create. This also led to the development of cyber-physical systems (CPS),automated systems that connect the operations of the physical reality to communication andcomputing infrastructures [3]. While traditional embedded systems focus more on the computingabilities, the CPS focus more on the connection between the physical and computational elements.

Research studies from Gartner [4] predict that there will be around 26 billion devices connectedthrough the IoT by 2020. Such a number will have a huge impact on many things, not only ontechnology development [5]. For example, the IoT will require the use of IPv6 since using IPv4would not be enough to support this amount of devices. Another problem that is considered insome papers is the amount of electronic waste that will build up, as well as the availability ofmaterials that are used for making electronics (since very little is recyclable from old circuits).Another challenge is adapting to the impact of the changes that will come with the inevitablechanges in existing business models. Job structures will change as many labour jobs becomereplaceable by machines and new maintenance jobs are formed instead. But the biggest challengewith IoT is privacy and security. The existing internet infrastructure is far from perfect when itcomes to keeping the data safe. Having a big number of devices harvesting sensitive data frompublic places raises a lot of privacy and security concerns which current legislations cannot handle(for example data ownership). This implies that there is a need for some form of governance. Thatis, a governing body is needed to create and enforce policies that deal with such problems [5].

There is a great need for standardization as well, to ensure interoperability not only for theexisting IoT frameworks but also for the future ones. There have been some efforts to introduce

2


an architectural reference model, like the IoT-A1 project. In any case all agree that this area willexpand rapidly in the next few years and that it will have both social and economical impacts onthe industry, the workers and generally the countries that adopt it.

2.2 Cloud computing

Cloud computing was first introduced around 2000 and rapidly became popular. It was fullydeveloped through the first decade of the twenty first century and reached the peak of inflated ex-pectations by the end of the decade. Grid and cluster computing architectures are the predecessorsof the cloud architecture. Cloud architecture is based on aggregation of hardware components thatare accessed and used as a unified entity. The details of the aggregation are hidden to the userand thus the complexity of scaling up is negligible. Scalability along with pay-as-you-go chargingpolicy applied by cloud providers introduced a new operational and business model. Utilizing cloudcomputing, enterprises can take advantage of powerful infrastructures and, in parallel, minimizethe operational, acquisition and maintenance costs by exploiting the pay-as-you-go policy. Con-sequently cloud computing drew the attention of industries that were interested in computationalresources and big data analysis.

The great interest of companies in cloud computing increased the number of cloud providers,resulting with more than 200 cloud providers around the world in 2014, providing a big variety ofservices2. Cloud services are divided in three main categories: Infrastructure as a Service (IaaS),Platform as a Service (PaaS) and Software as a Service (SaaS). IaaS provides computational powerand storage as a service (CPU, HDD etc.). Resources can be scaled up or down on demand in orderto adapt the fluctuating demands. PaaS targets software developers and is indented to help themduring the development process. Developers can utilize the service to develop, test and maintainsoftware in a collaborative environment. SaaS offers commercial software applications hosted inremote data centers that can be accessed from everywhere. Amazon Web Services and Rackspace3

are examples of IaaS services, Google App Engines and Microsoft Azure Services4 are examples ofPaaS services and finally SaaS example applications are Google Mail and Google Docs.

Cloud computing comes with significant advantages for industries but not for all kinds ofapplications. Firstly, latency-sensitive applications cannot be hosted in cloud infrastructure dueto unpredictable and high latencies introduced by Wide Area Networks (WAN). Secondly, dataprivacy is one of the major drawbacks of cloud computing for industries. In order to reach thecloud infrastructure, a multi-hop route should be utilized. Multi-hop introduces a lot of privacyand security issues because the connection relies on a commercial network. Consequently, theprobability for data attacks en route increases. Finally, the cloud infrastructure itself faces safetyproblems including insider attacks and exposure to a wide range of malicious elements.

2.3 Fog computing

2.3.1 Definition

The term fog computing was first introduced by Cisco in 2012 [6] to overcome the drawbacks ofthe cloud architecture. Since fog computing has not reached maturity yet, it is still a fuzzy term.Cisco’s initial definition does not reveal all the advances that fog introduces [7]. Consequently,several definitions of fog computing exist in literature [8, 7] along with the opinion that fog isa market buzz word [9] and that cloud providers introduced similar purpose architectures beforeCisco invented this term. Furthermore, fog computing is described in [10, 7] not as a new technol-ogy but rather as a combination of mature technologies that have already been used for a while,such as P2P networks and Wireless Sensor Networks (WSN). Nevertheless, fog computing shouldnot be considered a trivial extension of cloud computing [11, 12]. Fog computing brings computa-tional power to the edge of the network and thus makes Ubiquitous Computing (UC) possible [11]and enables cloud architectures to include smart sensors and intelligent devices. Moreover, fog

1http://www.iot-a.eu/public2http://en.wikipedia.org/wiki/Category:Cloud computing providers3http://www.rackspace.com/4http://azure.microsoft.com/en-us/

3


computing leverages the rapid increase of IoT platforms, e.g., if you need bigger area coverage orto include more end-devices you add fog nodes [13].

Nowadays mobile devices need more and more computational power to execute computationallyintensive tasks for applications like augmented reality applications. These applications also requirereal-time response to ensure Quality of Service (QoS). Even if these applications are not timecritical, their quick responses are highly desirable. Mobile devices, as well as embedded devices,are limited with computational power, battery and size [14, 15]. Although there has been a greatadvancement in their performance concerning CPU frequency in recent years, battery life does notface the same progress [16] and longer battery life is highly demanded by the users[17]. Moreover,the gap in terms of computational power between mobile devices and, their counterparts, stationarydevices is not likely to narrow in the foreseen future because technological advances in hardwareare applied in both kind of devices [18]. Fog servers can alleviate this problem by bringing thecloud to the edge of the network which implies more computation power closer to the user. Theedge nodes are 10 to 100 thousand times faster than the end-devices with 100 times less networklatency compared to cloud [18, 16].

2.3.2 Cloud vs Fog

There are some main differences between fog and cloud (summarized in Table 1) that make fog nota competitor of cloud but a technology that has come to form a new operational scheme of morecapabilities. Cloud computing combined with fog is the future platform for IoT applications [6, 19]where applications can take advantage of the locality of the fog and the global-centralized natureof the cloud. Fog computing is a vital part of the IoT platform architecture because of its supportof scalability. In the past decades the humans have been the ones that produce data. Nowadays, inthe IoT era the devices spread in a wide range, form clusters, and produce big data [9, 20]. Alongwith the existing three dimensions of big data: volume, velocity and variety, it adds a fourth one -the geo-distribution [21]. Embedded devices, as mentioned, are resource constrained, but in orderto perform intelligent tasks they need to communicate and aggregate information from deviceswithin the cluster. So there is a need for intelligent platform placed in the edge of the network toincrease computational power of IoT devices. That implies that fog nodes should be distributedand placed in the near vicinity of the user compared to the centralized cloud infrastructure. Beingclose to the user, just one hop away [22, 23, 24], enables fog nodes to support latency sensitiveapplications. On the other hand, due to the unpredictable and significant latencies introduced byWAN [8, 25, 26, 27, 28, 29, 15], cloud computing is not appropriate for applications that requireeither predictable or quick responses. Since WAN latencies are not expected to decrease in thenear future [30], fog nodes placed on the edge of the network enable IoT platforms to supporta wider range of applications. A significant problem that cloud providers have to deal with, asmentioned before, is data privacy and security. Fog nodes alleviates this problem because datais kept in the local network and is not transferred between numerous servers. This decreases theprobability of man in-the-middle attacks [23, 31] on the data en route until it reaches the fog, asopposed to reaching the cloud infrastructure[22, 7]. The gain from the data kept in local networkshas another advantage though, reducing Internet traffic. The security improvements are furtherenhanced by the location awareness of fog nodes. In the cloud context, applications and data areplaced “somewhere in the cloud” [32]. Fog nodes are hosted inside the premises of a plant, in a neardata center, intelligent routers that can host general purpose logic [25], smart gateways [8], accesspoints or even set-top-boxes [23]. Hosting fog nodes in such places also minimizes the possibilityof insider attacks that cloud infrastructures are more exposed to. Another solution would imply tokeep data encrypted in data centers, which adds an overhead in process data and results in higherlatencies due to the encryption/decryption procedure [7]. Finally, fog nodes store data ephemerallywhere cloud exhibits a more permanent storage.

2.3.3 Fog and similar technologies

Fog computing has a lot of similarities with cloudlets [33], edge computing [34] and mobile localclouds [35] and there is no clear distinction between them since all three are introduced mainly tomove the cloud closer to the edge. One can address some differences due to the different contextin which they are used. A lot of diversity exists in the difference among these terms. In [22] all the

4


Requirement Cloud Computing Fog Computing

Latency High LowDelay Jitter High Very lowLocation of server nodes Within the Internet At the edge of the local networkDistance between the client and server Multiple hops One hopSecurity Undefined Can be definedAttack on data en route High probability Very low probabilityLocation awareness No YesGeographical distribution Centralized DistributedNumber of server nodes Few Very largeSupport for Mobility Limited SupportedReal-time interactions Supported SupportedType of last mile connectivity Leased line Wireless

Table 1: Comparison between Cloud and Fog(taken from [7])

aforementioned terms are identical due to the fact that they serve the same purpose in the network.In [36], Mobile Cloud Computing (MCC) architectures are separated in two categories. The firstis a client-server architecture where the MCC provides resources to all clients. This approach isidentical to the fog computing architecture presented in [6] where the fog term is introduced forthe first time. The same architecture is analyzed in the context of cloudlet for assisting troopsin battlefields [16], face recognition [18] and augmented reality applications [27]. In the secondapproach, all the devices in the network share their resources. The latter architecture is used inthe context of cloudlets in [37]. The only difference is that the central node can be a normaldesktop computer or any device that can orchestrate the sharing without facing energy problems.This architecture does not exist yet for fog computing. Hong et al. [25] state that the cloudlets donot support distributed geo-spatial applications in contrary to fog nodes. Despite the divergencein architectures, all three technologies are made to offer computational power or save energy onresource-constrained devices and serve real-time applications. In this thesis, fog node, local cloudand edge node are used interchangeably.

2.3.4 Fog architecture

The architecture of fog nodes introduced by Cisco in [21, 6] includes three tiers. Fog nodes ingestdata produced by local embedded devices and sensors. The first tier of the architecture is usedfor M2M interactions and supports real-time applications. The M2M interaction is the key aspectto increase the intelligence of “things” [11, 12]. By utilizing the above architecture, fog is capableof providing sub-second responses [22]. In addition, high computational characteristics can servecomputational intensive applications to resource-constrained devices such as smart phones. In [38],fog computing is used for one of the most computationally-intensive tasks, namely brain-stateclassification, where it achieves low response times to enable augmented brain-computer interaction.Several approaches that utilize local servers to boost computational power of mobile devices havebeen proposed, for example, for surveillance [18] or face or object recognition [39, 40]. In all theseapproaches, performance is increased by 2-3 orders of magnitude and consequently the QoS of themobile devices is increased and battery consumption is kept low. Finally, the first tier can filter orpre-process data that is received by the mobile devices and sensors and send it to the higher tiersfor further processing, as shown in Figure 1. This is known as hierarchical data processing [13].

The second and third layer of the fog architecture also deal with machine to machine inter-action(M2M), real-time analytics, visualization and reporting, e.g., human to machine interac-tions(H2M). The second layer provides seconds to sub-minute responses. Consequently, the secondlayer can handle soft real-time tasks, M2M interactions and H2M interactions. The last layer isresponsible for analytics. It provides response times from minutes up to days for transactionalanalytics. Two different architectural concepts regarding the third layer can be developed: in-dependent fog devices connected with the cloud and interconnected fog devices (smart grids) toexploit the advantages of cooperation [23, 12]. It should be noticed here that the higher the tier,

5


the wider the coverage range. Cloud is responsible for the global coverage which is used for intel-ligent business analytics based on long period, wide area gathered data. Regarding data storage,fog should support different kinds of storage ranging from temporary storage in the lower level, tosemi-permanent storage in the higher level.

Figure 1: Different use of the same data

2.3.5 Another definition of Fog computing

The fog computing term is also used to describe a safety approach regarding data in cloud comput-ing proposed in [41]. One should not be confused with the term fog computing that is investigatedin this thesis because their meanings are completely distinct. In [41], they propose a securitytechnique based on user behavior profiling and decoy technology in order to detect the attackerand protect the user from misuse of their data. User behavior profiling is a common mechanism totrack users profile and detect abnormal behaviour. User behaviour profiling is also used in banksto detect unusual credit card transactions. There is a lot of research for user profile behaviourmonitoring and detection of abnormal behaviour but it is out of the scope of this thesis. The sec-ond feature included in this approach is called decoy technology which is used to detect an attackand serve the attacker with fake files to minimize the harm of the attack. Fake files are placed inconspicuous locations that are highly possible to be accessed by the attacker. When these files areaccessed, a possible attack takes place and the system feeds the attacker with all the fakes files.Consequently, it helps the cloud to prevent big data loss.

2.4 Offloading

2.4.1 Definition

As embedded devices become more and more important and useful in everyday life [7], the demandfor more capabilities rises. As mentioned before, in the foreseeable future we can expect stationarydevices to be 2-3 orders of magnitude better in performance than mobile devices. One way tonarrow that gap is to exploit other resourceful devices. A resourceful device can be any device thathas a resource which can be used by the embedded device to increase its performance. Resourcescan be computational power, memory, battery or even data. The procedure of moving tasks toother devices is the so-called offloading [40]. Depending on the device characteristics, offloading can

6


be used to save energy [17], decrease response time by moving computation and data to a powerfuldevice [39, 42, 43, 44], or request data from other devices in order to increase accuracy or easethe computation [45]. There are a few works that investigate offloading solutions for optimizingmultiple parameters, e.g. energy and response time [46, 47].

In this work we focus more on computational offloading (sometimes also referred to as cyberforaging or surrogate computing [40]) which deals with moving tasks to powerful, in terms of com-putational power, servers and receiving results from the servers. In the computational offloadingconcept, the target device used to offload tasks can be another mobile device. In this work, weconsider that the offloading target is a stationary powerful device as the aforementioned fog orcloud server. These devices are also called surrogate servers [48, 40]. The benefits of offloading,apart from increasing the performance of the system, is that it exploits the advantages of a cen-tralized system. When a cluster of devices is connected to a powerful server that provides someintensive computation algorithms such as image processing, any updates or improvements madeon the server side can impact the whole system.

2.4.2 Computational offloading approaches

Offloading algorithms are used to make the decision which task should be offloaded so the systemsrequirements are met. Computational offloading algorithms focus on meeting the timing constraintsof their tasks [14] or minimizing the response times to support smaller sampling periods or toincrease, for instance, throughput [43] and improve QoS. Computational offloading algorithms canbe divided in two categories: static and dynamic.

Static offloading algorithms let the developer decide which task/part of the code is suitable foroffloading before runtime. Static offloading is an NP-complete problem that is proved by inductionfrom the subset sum problem. The detailed proof can be found in [43]. However, given whichtasks to offload, an optimal ordering can be found in linear time using Johnson’s rule [49], whichminimizes the total response time. Tasks that have hardware dependencies, such as I/O drivers,cannot be offloaded. These tasks can be found by static analysis [50]. Moreover, tasks that cancause security issues if they are offloaded or human interaction tasks that need to interact with theuser should not be offloaded either [28]. Static offloading adds small overhead to the system andis advantageous when the parameters of the system can be accurately estimated or computed, butit cannot adapt to significant changes [40].

On the other hand, dynamic offloading algorithms decide which tasks to offload at run-time.There are several approaches for dynamic offloading exhibiting different advantages and disadvan-tages. In [48], regression is used to predict when the tasks’ execution time goes over a certain limit,and starts preparing offloading to prevent violating the timing constraint. Another way is to makethe offload decision when the task is ready to execute according to its input. In this case the systemis based on a-priori knowledge of the possible input properties that could lead to a deadline miss.For instance, the system can handle up to a set image size in pixels, so when it obtains somethingbigger it decides to offload [39]. Yuan Zhang et al. [50] propose a component-based approach wherea call graph of the system is built and then further partitioned to sub-graphs. Afterwards, thesystem decides whether to offload the whole sub-graph or execute it locally. Partitioning of thegraph in sub-graphs is an NP-hard problem, so heuristics are used to find a near optimal solu-tion [50, 35]. In [51, 52], an estimation of the tasks’ execution time is made using measurementsor historical data and thus creating a dynamic profile of the task to make the offloading decision.Dynamic offloading can be adapted to changes in the network such as fluctuating bandwidth. Butit introduces bigger overhead to the system compared to static offloading [40, 42].

On the server side, execution time can be estimated either by profiling or by some worst-caseguarantee given by some cloud providers. If the server is unreliable and the response time cannotbe estimated based on profile, an expected response time can be set according to the computationalpower of the server. If this time is overpassed, a local execution can start [53]. If there are multipleservers available for offloading, the system can have performance history and estimate the nextpossible response time. If a probabilistic scheduler is utilized, servers with better performance canhave a bigger chance to serve offloaded tasks [29]. The latter approach outperforms the greedysolution where the server with the better performance is always selected for remote execution. Thisresults to overload on specific servers, whilst other servers have very low utilization and thus waste

7


resources. Finally, in the case that a server has to support multiple clients, the Total BandwidthServer (TBS) is used to provide a fair part of computational power to all clients [14].

2.5 Industrial automation systems

The industry has, with the constant emergence of new technology, gone through great changesthroughout history. Like the conveyor belt revolutionized the industrial process, utilizing technol-ogy in automation and control made a great impact on the industry in the past few decades. Sincea dramatic upswing in the development of current technology has been going on recently, it isexpected it will outpour into many fields that will find use of it. Industrial automation is expectedto be one of them.

However changing industrial automation systems is not an easy task since these systems aretypically quite large and complex. It is not uncommon to find a mix of devices in such systems,both old and modern ones, covering many different functionalities. Legacy systems are still usedsince they were built to last, and it can be a problem to integrate them with new technologies [54].Another challenge is keeping the strict industrial requirements. Often new technologies do notsupport such requirements and need to be adapted or further developed before they can be usedin the industry.

Industrial automation systems are built to be robust, durable, reliable, often fault tolerant, real-time and safety critical, relying on strict industrial standards. A fault tolerant system is a systemthat can continue working correctly if a failure happens. This is especially important for ensuringavailability. Typically, fault tolerance is enabled through redundancy of parts, components, infor-mation or data that can then replace the one that failed to complete its task. A real-time systemis a system that reacts to it’s environment within a certain time frame - a set deadline. In thesesystems, providing the correct result is just as important as providing it on time. Responding in atimely manner makes the system predictable and reliable, which is why the timing correctness isimportant. Depending on the consequences of a missed deadline, real-time systems are classified ashard real-time if it is absolutely unacceptable to miss a deadline, and soft real-time if an occasionaldeadline miss can be tolerated.

2.5.1 Automation

Automation or automatic control means using a specialized computer system for controlling phys-ical processes, equipment, machinery and other applications, while reducing human intervention.Automating processes can optimize resource use, such as energy and materials and minimize theirwaste. It also increases production efficiency and precision which increases quality of the productor service. Automation also saves labour, and takes over heavy operations.

Control is achieved through control loops. When a measurement is done, some calculation orcomparison needs to be done in order to make a decision and then perform a certain action basedon that decision. If the decision making process involves giving feedback for corrections, it is calleda closed loop; otherwise, it is an open loop.

The ISA 95 [56] standard describes automation systems as layered architecture, pictured inFigure 2. The bottom of the architecture is where the sensors and actuators are, sensing andchanging the physical processes. This is also referred to as layer L0. The next layer, L1, iswhere the control of the sensors and actuators lies, whether it is microcontrollers, programmablelogic controllers (PLC) or some other specialized computer system. The connection between thislayer and the sensors and actuators is also referred to as the field-level network. The next level,L2, is where operators monitor the process variables and control loops. This level can includesuch systems as Human-Machine Interfaces (HMI) or Supervisory Control And Data Acquisition(SCADA), different kind of alarms or event triggers and logging of process information. Thecommunication between the controllers of L1 and the HMI/SCADA devices of L2 is called thecontrol network. The third level, L3, is for optimizing and coordinating software and controlloops of the whole plant. The top level, L4 is the enterprise management. It connects the plantproduction with outside factors such as market demands and material cost. It also performsanalysis based on which optimization goals for L3 are determined.

8


Figure 2: ISA95 architecture of automation systems, adapted from [55]

9


Some processes can be fully automated, but others may be complicated to fully automate ifthe human interaction they include is hard to mimic using computers. Since automation is usefulfor industries there will always be a tendency towards full automation.

2.5.2 Cloud-based automation

With the emergence and popularity of cloud computing and IoT, the further development of in-dustries is moving towards cloud-based solutions in automation and control systems. New researchin industries has primarily gone into the direction of supervision, monitoring and high-level con-trol [57, 58, 59, 60, 54, 55], which is a natural path since the higher levels (L4, L3) have lessconstraints than the lower ones (L2, L1) [55]. Most of the existing works draw their ideas from theservice-oriented architecture (SOA) or web-based architecture (WOA) concepts [54]. This meansthat the automation functions are provided as services on the cloud. The users then have easyaccess to all the different functionalities of the system without any requirement for specializedhardware, such as HMI. There are some solutions emerging for the L2 level as well [61], but verylittle is done so far for L1 [62, 55]. The problem here lies in the fact that the constraints, espe-cially the real-time requirements, get harder in the lower layers. This is a problem because usingan internet connection introduces unpredictability into these systems that require predictability.Many solutions propose a local cloud to minimize this problem and some are able to offer soft-realtime guarantees. Currently, some of the existing applications for cloud-based automation are asfollows.

• Monitoring and SCADA systems - Most commonly the system is a user-based applicationthat enables the operators to access the plant and monitor it remotely form a mobile device.Different access is provided based on user privilege.

• Analytics - Large systems with many devices produce a lot of information which is usefulfor fault detecting analysis. Some solutions use clouds for storage and analyse big chunksof data later (batch computing) while others analyse it as it comes, without storing it first(stream computing).

• Management at different levels - Most dominant is the idea of Cloud Manufacturing, a newand advanced computing and service-oriented manufacturing model.

• Storage - Clouds can be used for storing data that might not need frequent use but needs tobe archived, like different kinds of documents or logs and historical data from processes anddevices.

• Collaboration - Using cloud computing for collaboration in the development of automatedsystems is beneficial, especially when remote teams need to work on its development.

Using a cloud-based solution in the industry offers many benefits. Other than the great compu-tation power it can offer, the cloud is extremely flexible in terms of setting up, adding or removingnodes. Some solutions propose using cloud computing as a complement to the existing infras-tructure, for example to help out during peak times of the system [63]. In that case there is noneed to allocate extra resources that will not be fully utilized most of the time. Instead, the extracomputational power could be reserved when needed and released when done. Since the industryuses large systems, typically it takes a lot of time from the start of setting up to the time thesystem is used. This is because proper testing needs to be done before it can be used [55]. Chang-ing or upgrading can also be problematic if it requires the system to go down (offline) while it isdone. The cloud-based system skips all these problems and also reduces cost in maintenance andlabor needed during setups and hardware testing. As a result, much less space for hosting all themachinery and wires is needed.

2.5.3 Fog in industrial applications

The characteristics of fog architecture best fits industrial applications. Augmented security, hier-archical data processing, low latencies and low jitter makes fog interesting for industry [21]. Firstof all, low latencies and jitter conform with the need for predictable response times that is a red

10


line for industrial applications. In complement, hierarchical data processing is ideal for industrialautomation systems that are based on closed control loops. Fog nodes can be used for filtering,aggregating or summarizing data from the machine to the cloud or from the cloud to the sys-tem/machine [13]. This implies that only useful data reaches the cloud and the system [13, 18];whereas, all the irrelevant data is filtered out by the fog node. Data security and privacy is alsoreally important to ensure safety and avoid abnormal machine behaviour, as well as to protectcrucial industrial data that can be valuable to competitors. Moreover, connecting fog nodes formsa smart grid which can switch to renewable or cheaper energy sources to save energy and reduceoperational costs [36]. To the best of our knowledge, there are no industrial case studies that usefog nodes.

2.5.4 Smart factory

Implementing the IoT in industries would completely change how the factories operate. Notonly will it increase the efficiency and production of the factory but also change the work andbusiness models that are used and the jobs required (taking over the labor tasks and introducingmaintenance and programming ones). In Germany, a workgroup was formed to investigate thepossibilities of incorporating the concepts of cyber-physical systems, internet of things and internetof services. They have predicted that the changes will be significant enough to call it a newindustrial revolution, introducing their idea of the smart factory in Industrie 4.0 [64]. Thereare many names emerging for similar ideas, that are sometimes used interchangeably, such asFactory of Things (FoT), Factory of the Future (FoF), Industrial internet. Many countries havecreated workgroups and associations for research on how to adjust the industry to the new ideas,e.g., Industrie 4.0 working group in Germany [64], Smart Industry initiative in Netherlands [65],Future of manufactoring project in UK [66], European Factories of the Future Research Assosiation(EFFRA) [67], Industrial Internet Consortium [68] and many big companies (Cisco5 and Intel6)have started defining their own position in this field by introducing new ideas and further developingtheir own IoT solutions. It is a new challenge but it is evident that the future of industry is nowgoing in this direction and that the potentials are great.

5http://www.cisco.com/web/solutions/trends/iot/overview.html6http://www.intel.com/content/www/us/en/internet-of-things/intel-and-iot.html

11


3 Research method

The research method used in this thesis is called the System development research [69]. It consistsof five steps (as shown in Figure 3), namely:

1. Construct a conceptual framework

2. Develop a system architecture

3. Analyze and design the system

4. Building the (prototype) system

5. Observe and evaluate the system.

Figure 3: Flow diagram of research method, adapted from [69]

The system development research method is used to answer research questions when a theo-retical solution cannot be found or an innovative way of utilizing existing technologies needs tobe demonstrated. In this case a system is developed to validate the proposed technique or thesolution. This method is also known as proof-by-demonstration.

In the beginning, the research questions must be formulated and stated clearly. Clarifyingconceptual framework, relevant disciplines and goals is also a part of this step and it helps tomaintain the focus during the research process. Moreover, according to the goals, the high-levelsystem functionalities should be determined along with requirements. Relevant disciplines in thebibliography should be studied in order to extract useful information, like different ways to ap-proach the problem. This step includes the theory building, i.e., forming new ideas and concepts,conceptual frameworks, new methods or models which will be checked for correctness later.

In the second step, a high-level architecture of the system must be defined. The different partsof the architecture must be identified as well as their functionalities. The dynamic interactions be-tween the different components should be specified as well. In this research approach, that requiresdevelopment, all the constraints of the components and their interactions should be considered inorder to result in a system that can fulfill the stated objectives. The requirements must be definedduring this step so they are quantitative and measurable for evaluation during the last phase.Finally, every assumption that has to be made should be made in this step so the system can bedeveloped according to the defined requirements, assumptions and constraints.

In the next step, technical and scientific knowledge of the studied domain is applied to producedifferent approaches and alternative solutions. Additionally, evaluation criteria should be deter-mined in order to evaluate all the possible solutions and make the decision of the final design.Afterwards, the detailed functionalities of every component is designed in order for the system tomeet the requirements.

System implementation research method is used, as mentioned, to investigate the performanceof a design and to explore its potential functionalities, advantages and disadvantages. Duringthe implementation, researchers get an insight of the frameworks, tools and components used.The knowledge acquired from the implementation can be used to refine the system or help futureexpansion of the system.

After the system is built, case and field studies are conducted and evaluated by experimentsaccording to the predefined goals of the system. Experimental results and the experience gained

12


from the development of the system may lead to formulate new theories to explain results. Resultscan also lead to extend or develop new models and functionalities. The experience gained from thewhole research process can be generalized and used in the future in similar system development.

13


4 System development

In this chapter we present the design of our prototype. First we explain the difference betweena close control loop without a server and a close control loop that includes a server. Then wedescribe the system architecture and our proposed solution for mitigating delays.

4.1 Architecture overview

In our thesis the effect of adding a remote server to a closed control-loop needs to be investigated.A typical closed control-loop consists of a controller, an actuator, a sensor and a process to becontrolled. As it can be seen in Figure 4, the sensor produces a value by measuring the environmentof the process and sends the information to the controller. The controller takes the measured valueand, according to the desired output value, it computes a new output value. Afterwards, thecontroller feeds the output value to the actuator. The actuator makes the corresponding changeon the process and after that the sensor can obtain an updated value.

Figure 4: Closed control loop

Our focus is the migration of the controller to remote servers. The system in Figure 4 isused as a reference model in the experiments. When a remote server is added in the control loop,latencies, jitter and packet loss can be introduced between the controller and the sensor or actuator.As Figure 5 shows, the controller is connected to the actuator and the sensor via a network. Thenetwork can be either local (LAN) or global (WAN). For the former, the remote server is consideredto be a local cloud (fog node) which resides close to the rest of the control loop. For the WANcase, the remote server is connected with the actuator and the sensor via internet, resulting insignificant latencies between them.

Figure 5: Network Control System

4.2 Design

In this chapter the details of different hardware and software solutions used for developing theprototype are described.

4.2.1 System design

The system is based on existing hardware and software solutions. The Arduino starter kit7 is usedas the end device (the “thing”) to handle real-time tasks like a process automation system. The

7http://arduino.cc/en/Main/ArduinoStarterKit

14


starter kit includes Arduino Uno equipped with an ATMega3288 microprocessor, prototype board,motors, sensors and LEDs. Arduino is a good choice for prototyping because there is a vast amountof libraries available online from serial port communication to more high-level communication suchas Ethernet, IP stacks and TFT-LCD touch screens. A disadvantage of Arduino libraries is thatthey are not well documented. Arduino uses a C-based programming language and compiles withgcc9. Arduino is not made for process automation in practise because gcc is not meant for safety-critical systems [70], as it includes more than a 100 unpredicted cases, i.e., in some cases thecompiler makes unpredicted compilation. In summary, the starter kit is a good choice for ourpurposes of fast prototyping but not the ideal for a real world machine. Building a back-end inreal world machines is not possible due to limited time budget and to a more general approach ofthe subject.

The HP Compaq Elite 8300 Series computer is used as a server. It contains a i5-3470S quad-core Processor with a speed of 2.9 GHz and 3.7 GiB RAM. The server supports Intel’s virtual-ization technology (Intel VT-x) which minimizes the virtualization overhead for I/O and memory.Moreover, Intel VT enables virtual machines (VM) to run without performance degradation orcompatibility issues, as if they were running natively on a CPU. Regarding memory, VT-x enablesmemory isolation and monitoring for each virtual machine separately. The server is approximately600 time faster than the Arduino Uno which makes it good enough to be used as a server. Theserver runs a Ubuntu Server 14.04 LTS that hosts kernel virtual machines (KVM). The VM runsthe Trusty distribution of Linux on a generic kernel.

As mentioned before, the network that connects the controller with the rest of the control loopcan be either LAN or WAN. For the latter an emulator needs to be added to the system to introduceWAN characteristics such as delays, jitter and packet loss. For this purpose WANEM10 is utilized.WANEM is a software tool that can interfere with the incoming traffic and introduce the desirablenetwork behaviour. WANEM can add delay, jitter packet loss, packet reordering, bandwidth limit,corruption, duplication and disconnection between nodes that are connected to it. WANEM is alsoable to add different rules for different routes in the network. In order to utilize WANEM one onlyneeds to redirect the messages between nodes through the machine running WANEM.

WANEM runs on a virtual machine created with Oracles Virtual Box11 on Intel’s NUC 12.Intel NUC is a small, really powerful computer that can use GPOS and high-level libraries as anormal personal computer. Intel NUC has various models available, most of which include i-seriesCPUs, DDR3 memories, USB, HDMI and Ethernet interface. Intel NUC can be used either to hosta SDN real-time network or for fog computing because of its potential computation capabilities.Intel NUC is directly connected to Arduino, using Ethernet to maximize throughput. Since NUCcan host a GPOS such as Linux or Windows, all the low-level libraries are already implemented inthese OSs. The advantage of NUC is that it is expandable in certain ways and it can connect toHDD/SSD, USB3.0 and PCIe. Another advantage is that Intel is putting some effort in the IoTarea, including the use of closed loops with gateways providing feedback to smart factories.

In order to implement the aforementioned design in a controlled environment, the schema ofFigure 6 is used. All the components are connected to an Ethernet Switch (HP 1410-16G Switch13).When Internet needs to be added in the closed control loop, all the traffic from and to ArduinoUno is redirected and passed through WANEM. In the case of LAN, the Arduino Uno connectsdirectly to the server. For all the experiments there are two different approaches for the controlloop implementation which are two simple and initial ideas of how the control loop works whenthe controller is moved to a server. The first one, referred as polling, reads the sensor, sends thevalue to the server and waits until it gets the answer. Afterwards it applies the value and readsthe sensor again. A descriptive pseudo code is provided in Algorithm 1.

8http://www.atmel.com/products/microcontrollers/avr/megaavr.aspx9https://gcc.gnu.org/

10http://wanem.sourceforge.net/11https://www.virtualbox.org/12http://www.intel.com/content/www/us/en/nuc/overview.html13http://h30094.www3.hp.com/product/sku/10256671/mfg partno/J9560A

15


Algorithm 1 Polling

1: procedure Polling controller2: begin:3: sensorValue← readSensor();4: sendMsgToServer(sensorV alue);5: loop:6: if resultAvailable() then7: result← readResult();8: applyValue(result);9: goto begin;

10: goto loop;

The second approach is the non-polling approach. The loop of this approach has three steps,the first step is to read the sensor, the second is to send the value to the server and the third is tocheck if there is a response from the server. If there is a response, the system reads the responseand applies the value. If there is no response, the system moves to the first step. The pseudo codethat describes the non-polling version is provided in Algorithm 2.

Algorithm 2 Non-polling

1: procedure Non-polling controller2: begin:3: sensorValue← readSensor();4: sendMsgToServer(sensorV alue);5: if resultAvailable() then6: result← readResult();7: applyValue(result);

8: goto begin;

The architecture to be evaluated includes a back-end machine connected to remote servers.Servers can be local, representing fog nodes or global and act as cloud servers. On the machinethere is a closed control-loop that reads a sensor, computes a value according to the input andapplies the result to an actuator. As shown in Figure 4, the computation of the value is donelocally on the machine. In our design we add the server to the closed control-loop. In this case,depicted in Figure 6, the machine sends the value to the server, the server computes the valueand sends the computed value back to the machine and finally the machine applies the receivedvalue. Depending on where the server is in the network, latencies, jitter and even packet loss isintroduced from the machine to the server and from the server to the machine. In the experimentsthere is no bandwidth limit.

16


Figure 6: Architecture overview

4.2.2 Delay mitigation mechanisms

Since the controller of the control loop is offloaded, and the control system influenced by networkproblems, it is of interest to have a local delay mitigation mechanism. Two different mitigationapproaches are proposed here, one for each communication algorithm.

In order to improve the polling method, a timeout is introduced to stop the system for wait-ing forever on the server response and instead try and predict the missing server values. Thetimeout equals to the average round trip time (RTT) in the server communication. The RTT isthe total time it takes a message to go from the client to the server and the server response to reachthe client. Prediction methods are sometimes used in dynamic offloading to estimate changes insome parameters rather than checking every time. This includes predicting changes in parameterssuch as the load per node or available bandwidth, as is done in [71]. If the server does not respondin the expected time, the system predicts the value and uses that instead. The late responsesfrom the server get buffered when they arrive and the system can either use those values whenthey arrive or disregard them. Both options are considered here, where applying the late values islabeled as buffering, and disregarding them is labeled as no buffering.

Two prediction models are used here:

1. Exponential moving average

2. Double exponential smoothing model

The exponential moving average [72] (also called exponential smoothing) is a simple weightedmoving average of the data values and the previous statistic values. It is calculated as presented inequation (1), where st is the statistical value, st−1 is the statistical value calculated in the previousround, xt is the current observation and A is the smoothing factor. The smoothing factor, A, isset between 0 and 1 and weights the data values.

st = Axt + (1−A)st−1 (1)

The current value st is an interpolation between the observation xt and the previous valuest− 1, where A determines how close the interpolated value is to the latest observation. Theforecast for the next period is then the current value, as shown in equation (2)

Ft+1 = st (2)

While the two prediction models mentioned are similar, the double exponential smoothingintroduces another term that takes the change of the slope, or trend, into account. It is based

17


on the “Holt model” forecast [73]. The data value is calculated by the formula (3). In additionto the exponential moving average (see equation (1)), it has a parameter bt−1 that represents thetrend calculated in the previous round. The trend value is calculated in every round by using theequation (4). Apart from using the statistical data values st and st−1, as well as the previouslycalculated trend bt−1, it also uses another smoothing factor B, set between 0 and 1. The firstsmoothing factor, A, acts as a weight for the data values while the other one, B, weights the trendof the slope.

st = Axt + (1−A)(st−1 + bt−1) (3)

bt = B(st − st−1) + (1−B)bt−1 (4)

The forecast can now be made for xt+m at the time

t + m,m > 0.

The forecast is calculated as presented in equation (5)

Ft+m = st + mbt (5)

Both methods depend on values from the previous round which doesn’t make it obvious howto start. There are a few different recommendations on how one can set the initial values, sincenot every approach works well for all cases. In our case, the initial values are set up based on thefirst values of the observation xt. In the case of the exponential moving average, the initial valueis

s1 = x0,

while for the double exponential smoothing the initial values are

s1 = x0

b1 = x1 − x0.

Both methods also require setting the smoothing parameters, A and B, to a value between 0 and1. While there is no formal way to do it, they can be calculated by using optimization techniquesfor getting the minimal error between the data and predicted values. In our case, the parametersare set using a trial and error method. Since having lower values of the A parameter means a slowerresponse of the system, in our case higher values are preferred. The B parameter reflects how muchchange in the trend the model is expecting. A small value expects very little change and the otherway around. In our case, the system works better for higher values of B since it adapts better tothe rise of the curve in the beginning. The parameters in our system are set at A = 0.8 and B = 0.9.

In order to improve the non-polling method, an adaptive PI controller is utilized. When thenon-polling method is used and the delay increases more than the sampling period, the controllermakes the decisions based on old messages because messages are queued in the network. Specif-ically, the controller’s sampling period is approximately 16 milliseconds (ms) thus the controllerreceives a new value every 16 ms. When the RTT is bigger than 16 ms, the controller sends anumber of messages before it receives an answer. The number of messages sent before receivingan answer depends on the RTT. For example with 20 ms of RTT two messages are queued in thenetwork. The result of the queued messages is to double the I part of the PI controller. Concretely,the integral part of the controller is computed by formula

integrali = integrali−1 + errori, for i > 0,

where integral0 = 0 and errori is the difference between the target value and the value read fromthe sensor [74]. Therefore, when the first message is received from the controller the integral partis

integral1 = integral0 + error1 = error1.

18


Afterwards, the second message is received with the same sensor value, since as mentioned beforethe previous computed value has not applied yet. Therefore, for the second message we haveerror2 = error1 and so

integral2 = integral1 + error2

= integral0 + 2 ∗ error1= 2 ∗ error1= 2 ∗ integral1.

From the last equation we conclude that for every queued message the integral part of the controllerbecomes a multiple of the initial value resulting in performance downgrade especially before thesettling time. Settling time is the time elapsed to stabilize the output from a change of the input.On the other hand, the P part of the controller remains the same even when messages are queued inthe network since it does not depend on previous values. In order to mitigate the ever increasingintegral part of the PI controller, we apply a smoothing factor to the computed value of thecontroller. The actuator receives a value, applies the smoothing factor then applies the value tothe plant. The smoothing factor is computed by the formula

a =sampling period

sampling period + RTT

where RTT is the round trip time and sampling period is the sampling period that is used when thecontroller executes locally. The smoothing factor increases as the RTT increases. The smoothingfactor applied is a type of adaptive controller but its definition is intuitive. Its benefits are examinedby conducting a number of experiments in Section 5.

4.3 End device set-up

The control loop consists of a photo-resistor (LDR-VT90N2) and an LED (LEDWC503B-WAN-CBADA151). The controller can change the brightness of the LED until it reaches the desired value.The LED values can be set between 0 and 1024, and the target value used in our experiments is700. The sampling period is approximately 14 ms. In order to accomplish this a PI controller isutilized with proportional gain Kp = 0.001 and and integral gain Ki = 0.002. The Ziegler-Nicholsheuristic tuning method is used to obtain the initial values of the parameters and then manual finetuning to obtain the final values. The Figure 7 illustrates the circuit diagram of the control-loop.As it can be noticed the LED is connected in series with a 330Ω resistor to a digital output ofthe microcontroller. Also the photo-resistor is connected in series with a 1kΩ resistor to a 5 Voltssupply. The microcontroller reads the voltage drop across the photo-resistor through an analoginput.

Figure 7: Circuit schematics

19


5 Experimental evaluation

This chapter contains the experiments that are conducted on the developed system. First wediscuss the experimental setup. Then we discuss the results with constant and variable delays inthe local controller. After that, an analysis of the delay mitigation mechanisms (the two predictionmethods and the adaptive PI controller) is presented and compared to the local controller responseunder the same delays.

5.1 Experimental set-up

In the first set of experiments we investigate the effects of adding constant delays in the control-loop for both the polling and non-polling approaches. WANEM is used to inject the delays in thenetwork. Firstly, the delays are set at 0, 1 and 10 ms, to investigate the effect of delays shorterthan the sampling period in the control-loop, for both approaches. The second part of the firstset of experiments is set to 50, 100 and 150 ms for the polling approach, and 15, 20 and 23 msfor the non-polling approach. The purpose of this set of experiments is to investigate the effect ofdelays longer than the sampling period to the control loop. In the second set of experiments, weinvestigate the effect of variable delays and jitter in the control-loop. WANEM is used to set thejitter to 1, 2, 5 and 10 ms for both approaches, in order to investigate the effect of small jitters inthe control loop. Similarly, the second part of this set of experiments is to investigate the effectof jitter longer than the sampling period in the control loop. Jitter is set to 25, 50 and 75 ms forboth approaches.

In the next two set of experiments, we apply the delay mitigation mechanisms, one for thepolling method and one for the non-polling method. The second set of experiments uses the twoprediction methods to test how they can improve the polling method. Both methods are testedunder the delays and jitter that affect the control loop in the first set of experiments. The resultsare then compared between the two prediction methods and with the polling method without anyprediction.

In the third set of experiments the Adaptive PI controller is used. The experiments targetdelays and jitters that exceed the sampling frequency. As mentioned earlier, the effect of delayssmaller than the sampling period is not significant. Delays and jitter are set to the same values asin the first set of experiments in order to compare the results.

5.2 Local Response

In the first set of experiments, we investigate the effects of network latencies on the controller inboth polling and non-polling approaches. In Figure 8 the controller is executed locally. As we cansee the overshoot, the difference between the maximum value and the target value, is negligibleand the settling time is 89 ms.

Figure 8: Local response

20


5.2.1 Constant delays

In the polling approach the latencies affect the sampling frequency of the control loop. In Figure 9,three different communication latencies are examined. The direct communication with the server,labelled as fog, where the latency is less than half a millisecond and represents the communicationwith a local server where no latencies are introduced. In the other two cases, the latencies are setto 1 and 10 ms respectively. One can notice that the disturbance in the response curve is not big,especially for the fog and 1 millisecond communication latency. As expected, the delays that areless than the sampling frequency can be tolerated without significant change at the performanceof the control loop. Therefore it can be concluded, that offloading the controller to local serversdoes not have any effect on the response.

Figure 9: Latency less than the initial sampling period

In Figure 10, the communication delays are more than the initial sampling period. In thesethree experiments the latency is dominating the control procedure and it is approximately 4, 8 and12 times higher that the sampling period, respectively. It can be easily noticed that the disturbanceis obvious, the overshoot is increasing and for 150 ms latency the system becomes unstable. Theresult is reasonable since controllers are tuned to work on a specific sampling frequency and maketimely decisions.

Figure 10: Latency more than the initial sampling period

In Figure 11 the non-polling approach is used. As it can be noticed in this approach, whenlatencies are kept below the sampling period there is no great effect on the response. In the non-polling approach the degradation in the performance is severe when messages are queued in thenetwork. As we can notice in Figure 11, when the delay is 10 ms the controller sends two messageswith the same value before it receives a response. This results doubling the I part of the PI controllerand thus causes bigger changes to the actuator which downgrades systems’ performance. That is,

21


the system receives the first message, it computes a value and then it receives a second messagewith the same value. That makes the controller to compute a bigger value in order to decrease theerror, since the previous value does not affect the system. Obviously, this interpretation from thecontrollers’ side is wrong since the first computed value has not applied yet.

Figure 11: Latency less than initial sampling period

In Figure 12, the fast degradation of the controller’s performance can be seen. When thelatency is increased to 23 ms four messages are queued in the network. As a result, the systembecomes unstable. The queued messages cause an increase in the I part of the controller beforethe computed value is applied. Additionally, the controller makes its decisions not on how the lastcomputed value affects the system, but on how an older value affects the system. In this context,the age of the value is proportional to the network delays. Finally, as it can be derived from thefigure below that the controller still stabilizes with 3 messages queued in the network.

Figure 12: Latency more than initial sampling period

5.2.2 Jitter

In the second set of experiments we investigate how jitter affects the performance of the controller.In Figure 13 the effect of four different values of jitter is depicted for the polling approach. Asit can be noticed, the effect of jitter that is smaller than the sampling period does not affect theperformance of the controller significantly. However, the jitter increases the overshoot but thesystem still manages to stabilize.

22


Figure 13: Jitter less than the initial sampling period for the polling approach

In Figure 14, it can be clearly seen that as the jitter increases and exceeds initial samplingperiod, the overshoot increases significantly. Additionally, one can notice the settling time increasesand reaches, approximately, 600 ms. The performance of the controller is degraded and it cannotbe considered acceptable compared to the local response characteristics.

Figure 14: Jitter more than the initial sampling period for the polling approach

In Figures 15 and 16, the performance of the non-polling approach for communication lineswith jitter is investigated. As it can be clearly seen, the non-polling approach malfunctions whenmessages are queued in the network. Even if one message is queued in the network the degradationis considerable. Overshoot is more that 100 when one message is queued in the network and itrises fast as the jitter becomes bigger that the sampling period.

23


Figure 15: Jitter less than the initial sampling period for the non-polling approach

Figure 16: Jitter more than the initial sampling period for the non-polling approach

5.2.3 Discussion

By comparing the delays’ effect on the controller with the jitter, we can conclude that both affectthe settling time since it takes more time to get enough responses to stabilize the system. Also,both the delay and jitter affect the overshoot of the controller. Additionally, the polling approachtolerates delays better than the non-polling approach. In the next four figures the changes ofovershoot and settling time are depicted for the two approaches. Obviously the overshoot and thesettling time for the non-polling approach increases faster as compared to the polling approach.Moreover, from the graphs we can verify that offloading the controller to fog nodes does not affectthe system. Consequently, the local servers can be included in the control loop without performancedegradation.

24


Figure 17: Overshoot using the polling approach

Figure 18: Overshoot using the non-polling approach

Figure 19: Settling time using the polling approach

25


Figure 20: Settling time using the non-polling approach

5.3 Delay mitigation

In next set of experiments we try different methods to compensate delays and jitter in the controlloop. The goal of the experiments, as said before, is to increase the reliability of the system whenmessages are delayed. For the polling approach, since messages are not received in every period,a prediction model is used, whereas in the non-polling approach a smoothing factor is applied tothe received value to mitigate the ever-increasing computed value from the controller as a resultof the queued messages.

5.3.1 Prediction methods

The prediction methods used here are, as mentioned before, the double exponential smoothingmodel and the exponential moving average. Both of them are applied in the polling method whenthe waiting time for the server exceeds a timeout of 18ms, as that is the average RTT between thedevice and the server. The communication between the server and the device is capable of bufferingthe messages between them. In the case of higher delays the buffered messages become older andolder as the system continues to use them. But if the system does not buffer the messages thenit makes use of less values form the server in total. The prediction methods, however, depend onbeing updated with new values. Both prediction models are tested with and without buffering themessages. The tests show that buffering and not buffering gives similar results in the lower delays,but not buffering takes more time on the same number of samples, since it reaches the timeoutmore often than buffering. In the case of tests with constant delays, for the double exponentialsmoothing the not buffering approach starts to become unstable at 75ms delay (see Figure 21). Forthe exponential moving average the curve also loses its smoothness at 75ms delay (see Figure 22).This is because both of these methods start using more predicted than calculated control valuesat this point, around 65.68%. In the case of tests with variable delays, the no buffering approachgives results that are very close to the buffering approach, but still uses more predicted values andtakes longer time. In the analysis presented from here on, only the buffering method is considered.

26


Figure 21: Double exponential smoothing without buffering messages at a 75ms delay

Figure 22: Exponential moving average without buffering messages at a 75ms delay

5.3.1.1 Constant delays

When the delays are higher than the sampling period, the system has the most trouble to settlein the start. We can notice these irregularities in both prediction methods in Figure 23, where theapplied delay is 75ms. This occurs when the prediction method needs to act but hasn’t gatheredsufficient data yet. However, the methods are still able to handle it better than the original signal.In this case both methods used about 8.82% of predicted messages, but most of them are in thebeginning. The overshoot of the double exponential smoothing is slightly higher (97) than theovershoot of the exponential moving average (86).

27


Figure 23: Exponential moving average and double exponential smoothing compared to the pollingapproach at a 75ms delay

A similar situation goes on in the case of the 100ms delay, pictured in Figure 24. The predictionsmade in the beginning are noticeable on the curves again. This time about 10.78% of the valuesare predicted for both methods. the overshoot is practically the same at 87 for double exponentialsmoothing and 86 for the exponential moving average. The settling time is longer this time, butthat is understandable since the delay is longer as well.


As for the 150ms delay, pictured in Figure 25, both methods manage to settle after a consid-erable time. In this case the percentage of predicted messages is 14.70%. The overshoot is almostthe same at 119 for double exponential smoothing and 116 for exponential moving average.

28



We can notice that both the overshoot and settling times increased with higher delays, but bothmethods managed to settle although after some significant time. Still, the results are reasonableconsidering the applied delays are 4, 8 and 12 times the sampling time. The methods managed toperform because the amount of predicted values stayed lower than the amount of calculated controlvalues. The system would, however, perform better if most of the predictions didn’t happen in thestart.

5.3.1.2 Jitter

The next set of test covers variable delays with jitter. That means that the delay is different inevery communication cycle and it might vary from 1ms up to the specified value for each test.This also means that the system has less load in the beginning compared to the constant delays,which is why all the curves are much smoother in these tests. Figure 26 represents the case when a25ms jitter is applied. We can see that both prediction methods have barely any overshoot(3 and4) and that the exponential moving average settles slightly faster. It’s obvious that the system isless stressed in this case (compared to constant delays) as the amount of predicted values used isonly 3.92%.

Figure 26: Exponential moving average and double exponential smoothing compared to the pollingapproach at a 25ms jitter

Figure 27 shows a similar behaviour when 50ms jitter is applied, even though it uses morepredicted values. In this case, it predicts 16.66% of the used values, which is more as compared to

29


when the constant delay of 150ms was applied. The system performs well because the higher delaysare scattered. Therefore the predicted values are scattered as well, instead of being concentratedin the start (like they are in the case of constant delays). The percentage of predicted values alsoincreases because the buffering has less effect in this case compared to when constant delays areapplied. For constant delays, when the system empties the buffer it has to wait until the nextserver message is received, which is after the duration of the complete delay. This means thesystem has to wait longer in the beginning for a received message as it empties the buffer faster;whereas later on, as more time passes, more messages will be stored in the buffer and it will takelonger for the system to empty the buffer. For variable delays, the system does not have a patternlike this, which is why it behaves better even though it uses more predicted values.


In Figure 28, a jitter of 75ms is applied. We can notice small disturbances in the rising partsof the prediction curves, as in the experiments with constant delay, caused by predicted values inthis stage. However, we can also notice that they affect the rise of the curve much less than inthe case of constant delays. This is because in the case of constant delays the methods have topredict multiple consecutive values. This is a problem since both the prediction methods need tobe updated with new values. So even though the amount of predictions reaches 26.47% in thiscase, the performance is still comparable to that of lower disturbances in the system.


30


5.3.1.3 Discussion

In summation, the prediction methods manage to stabilize the system with increased communi-cation delays. The prediction methods, considered in the experiment, are challenged in the casewhen consecutive predicted values are needed, especially in the start up of the control system. Theprediction works better with variable delays because the predicted values that are used are morescattered, as opposed to constant values where the predictions are concentrated at the beginning.The double exponential smoothing shows slightly better performance in the case of constant delayswith slightly lower settling times. On the other hand, the exponential moving average predictionmethod is slightly better in the case of jitter, with a slightly faster rise in the start.

5.3.2 Adaptive PI controller

As it is mentioned before, the smoothing factor is computed by the formula shown in (4.2.2). Thefollowing experiments show the affect of delays and jitter on the system when using the adaptivePI controller.

5.3.2.1 Constant delays

In this experiment, the network latency is set to 10 ms, and the smoothing factor is set to 0.42.In Figure 29 the response of the system using the smoothing factor over the initial response isdepicted. It can be noticed that the overshoot is decreased significantly, from 102 to 38, and thesettling time is droppd from 482 to 311 ms.

Figure 29: Smoothed line for a 10ms delay

In the next experiment the latency has been increased to 20 ms resulting in a dominatinglatency in the control loop. From Figure 30, one can notice the degradation in the performancefor the initial response caused by the delay. In contrast, when the smoothing factor is appliedto the received values, the overshoot and settling time are decreased considerably. Concretely,overshoot is 142 before the smoothing factor is applied; whereas, after it is 56 after the smoothingfactor is applied. Hence, the smoothing factor is decreased by 3 times; whereas, the settling timeis decreased from 1415 to 601 ms.

31


Figure 30: Smoothed line for 20ms delay

In Figure 31 the delay is set to 25 ms and the system becomes obviously unstable. As it canbe noticed, the application of the smoothing factor retains the system stability with an overshootof 59 and the settling time of 686 ms.


In Figure 32, the delay is double compared to the previous experiments. By applying thesmoothing factor, the system still manages to stabilize after 1309 ms with an overshoot of 70.From the last set of experiments, the benefit of the smoothing factor in the response of the systemcan be easily noticed, although it considerably deviates from the local response.

32



5.3.2.2 Jitter

In the next three figures (33,34,35), jitter is applied to the communication lines, resulting invariable delays, from 0 to 10, from 0 to 20 and from 0 to 75 ms respectively. The smoothing factoris computed by using the formula in (4.2.2) where RTT is set to the maximum possible delay.Specifically, for the 10 ms jitter, the smoothing factor is set to 0.45; for 25 ms of jitter to 0.29and for 75 ms of jitter to 0.1. As it can be seen, the response of the system highly benefits fromthe smoothing factor. The overshoot for 10 and 20 ms of jitter is 0 when the smoothing factor isapplied. On the other hand, the overshoot is 108 and 134 respectively when the smoothing factoris not applied. The settling time is also improved, for the 10 ms jitter by 247 ms(from 374 to 100)and for 20 ms of jitter by 185 ms(from 384 to 199).

Figure 33: Smoothed line for 10ms jitter

33



In Figure 35, the jitter is set to 75 ms and the system becomes unstable. As it can be noticedby applying the smoothing factor the system manages to become stable after 1264 ms with anovershoot of 62.


5.3.2.3 Discussion

In summation, the adaptive PI method manages to stabilize the system when the delays exceedthe sampling period. The adaptive PI method, considered in the experiment, manages to stabilizethe system even when the normal controller cannot. In the case of jitter, the adaptive PI controlleroutperforms the normal controller in all cases. The only drawback is that, in the case of jitter, theRTT should be set to the higher value which probably cannot be precisely predicted. Finally, theadaptive PI controller improves the system behaviour compared to when the adaptive PI controlleris not used, but system response is downgraded compared to the local system response.

34


6 Related work

6.1 Cloud and Fog architectures

Cloud architectures leverage the 4th industrial revolution by introducing augmented capabilities,new operational schemes and features for industries. So far, the existing research focuses on moni-toring and managing industrial data that comes mainly from shop floor [60]. Cloud infrastructuresare used for big data analysis and business intelligence. A few works exist that consider closedloops between automation systems and cloud [62, 75]. In this thesis, cloud computing and fogcomputing are added to the closed control loops in automation systems. We have exploited theadvantages of such architectures and investigated the effects of including cloud and fog nodes toclosed control loops.

Fog architecture is introduced to overcome cloud drawbacks. It is suggested for industrialpurposes, but so far, only one industrial application is using fog nodes or similar technologies inclosed control loops. Furthermore, there is no industrial case study that evaluates the fog combinedwith cloud (fog-cloud) architecture; and how a system can benefit from a smart combination ofthese two platforms. Regarding non-industrial applications only a few simulation results existfor this architecture, such as [18, 30, 16], but there is no actual implementation. Local cloudarchitectures are mostly used for quick responses and not as intermediate nodes between the enddevice and cloud to filter data. In summary, there is not much research on how the cloud can becombined with fog in order to benefit the end device. In our thesis we investigate the effect ofadding fog or cloud servers in a close control loop.

6.2 Offloading

Several approaches exist on how a system can use near and centralized resourceful servers toincrease performance [40, 52]. In these cases a resource-constrained embedded device takes ad-vantages of powerful stationary machines to increase performance or throughput. In our case,we consider industrial systems that are able to reach the desired performance metrics. In somenon-industrial approaches embedded devices are not capable of executing their tasks to meet thedesired QoS without offloading [39]. We make an assumption, that the industrial systems arecapable of executing all their tasks without any problem and are able to achieve the desired QoS.The latter assumption is realistic because adding cloud infrastructure to the closed-loop, does notgreatly change the performance characteristics of the existing automation systems. It only addsan overhead in the industrial controller for the communication with the remote infrastructures,which is one of the parameters that is taken into account in this thesis. Preparation for offloading,like code upload and protocol overhead, is considered negligible in some commercial applicationscompared to other parameters of Round Trip Time (RTT), like communication bandwidth [47]. Inour case the offloading overhead is included in measurements as part of the RTT. In commercialapplications, devices offload tasks in order to increase the performance. So if offloading results indecreased performance, the device makes the decision to not offload. We do not consider increasein performance as a factor to make the decision but our goal is different. We investigate a morecontrol-as-a-service scenario where control is offered as a cloud service. We explore the advantagesand disadvantages of executing the service in cloud, fog or locally are to be investigated. Ouroffloading goal is not to increase performance but to offer a centralized control of an automationsystem. Commercial applications rely mostly on wireless communications for the last hop (betweenthe end device and the first server). Although there are several wireless industrial networks, we usewired connections for the last hop. The connectivity aspect of the problem is out of the scope ofthis thesis. Finally, due to the fact that control is offered as a service, the automation system doesnot have to make the decision for offloading the tasks and the burden of the offloading decision innot a problem for our system.

6.3 Cloud-based automation

Most of the research for cloud-based automation focuses on monitoring and maintenance levels.Very few works exist in the process control levels [60, 62]. The research that considers the option

35


of moving low-level functionalities argue that not all of them can be moved to a cloud system. Thisis understandable for the functionalities that are tied to the hardware. But none of the papersspecify any way to distinguish what can and cannot be moved. The explanations are often veryvague and conclude that it depends on what the system consists of. Some of the solutions also tryto adapt the cloud to the system and vice versa in order to optimize the way the system works andto minimize the latency effects of using the cloud. In [55], a delay compensator is modeled thatcorrects the output of the system and therefore adjusts its behaviour to a variable latency. Thedelay compensator tries to minimize the impact of cloud delays and adjusts the system responseso that it continues to work smoothly. They demonstrate that the system remains stable evenwith remote server connections. They also propose a fault-tolerant system using redundancy inthe cloud. While our work also utilizes a delay mitigating mechanism, using the cloud for faulttolerance is out of the scope of this thesis.

In [76], an analysis of the tasks that use cloud computing is made, based on which the tasks areclassified. The tasks are classified into short tasks with early deadlines and long tasks with heavyresources. Each class is then provided with different cloud resources and the tasks are scheduledwith a different algorithm that better suits the class. Similarly, in [58] the two main functionalitiesare separated on two different clusters. One is reserved for fast data handling without storingor too much processing, while the other is reserved for tasks with heavy resources that requiretime-consuming analysis. In their work they aim to optimize the cloud resources for the two mainfunctionalities and therefore save energy. Cloud computing is ideal for cases like this because itcan offer such diverse resources without any extra effort. This idea is useful for big systems. Butin our work the system is made up of a simplified control system so there is no need for any taskanalysis or providing different infrastructure for parts of the system. In [63], the authors make useof the flexibility of the cloud, in the sense that it is easy to allocate or deallocate resources at anytime. They propose a system which predicts how many resources the system will need in the nexttime frame. Based on that prediction, cloud nodes are allocated or deallocated. The idea is that itis easier to complement the existing system with allocated cloud nodes than to expand the systemwith more hardware. This idea is quite useful for the systems where there is a need for scalability.Since our system is a simple one, it remains out of the scope of our work.

Another paper [62] proposes virtualization of PLC controllers. Their set up uses a local privatecloud with internet connection through a local network. They choose this because it offers bestsecurity and performance capabilities when using clouds. They show that the cloud-based PLCbehaves similar to a hardware PLC, but response times worsen with the decrease of task samplingtimes (with sampling times less than 128ms). The paper does something similar to our approach,although in our work the local private cloud is combined with a remote one. Furthermore, weperform more extensive testing comparatively.

6.4 Discussion

From the literature review some valuable and important conclusions can be drawn. First of all,there is not much research done for remote servers that are included in closed control loops. So far,cloud infrastructures are used for monitoring industrial machines, improving business intelligenceand reporting. Moreover, cloud computing is used for high-level procedures and not in low-levelprocedures that require fast responses. In order to overcome these problems, local servers areutilized and they can be found in literature with different names as fog computing, edge computingor cloudlet that exhibit lots of similarities. These technologies are not mature yet, and a clearseparation line cannot be drawn between them. The local server schemes present great potentialfor the applications that require real-time behaviour and fast responses. Moreover, there is a lotof research done in offloading techniques, both static and dynamic, for partitioning software andexecuting the code remotely, to a server or a resourceful device that can provide improvement inperformance. Offloading techniques are targeted at the systems in which communication delays arenegligible compared to execution time. Offloading techniques have helped the development of IoTdevices and collaboration among them. The amount of IoT devices is going to rise significantlyin the foreseen future, bring more devices connected to the world and increase the intelligenceof every day appliances. IoT is, according to researches, an important part of the 4th industrialrevolution, known as Industrie 4.0.

36


7 Conclusion

In this thesis we have investigated the usability of servers inside a closed control loop. A prototypewith a simple fallback mechanism is developed and tested in different conditions. The proposedsystem showed interesting results in various test cases.

The presented system works fine for local networks and small delays, when the latencies aresmaller than the sampling period. We can conclude that having a local cloud or fog nodes is aviable solution for a scenario like ours. Remote cloud nodes are best kept for less critical softreal-time tasks that can handle delays with more flexibility.

Since our system includes small sampling periods, it doesn’t tolerate big changes in the responsetimes. Even the delays that could be considered small affect the system if they are multiples ofthe sampling period. We also noted that the offloading does not make sense in the cases whenthe communication time dominates the processing time. Having a timeout for the server connec-tion prevents the system from becoming unresponsive but the prediction methods still depend onreceiving the values from the server.

The forecasting methods, double exponential smoothing and exponential moving average, canbe used for small compensations. They could be used for compensating a few misses every oncein a while as long as the system does not have to rely on them for too many consecutive periods.They are also not very reliable when starting up the system before the system reaches its stablepoint. However, they do produce sufficient results when they are not the dominating source of newvalues.

The proposed adaptive PI controller uses an intuitive idea for mitigating delays inside thenetwork. Although there is no formal proof for the robustness of the algorithm, but it improvesthe system’s response. In order to prove the robustness of the method, there is a need of amathematical proof or examine the behaviour with a more complex system. The mathematicalproof is an advanced control theory subject. On the other hand, testing it with a more complexsystem is left for future work. The adaptive PI controller manages to improve the performanceof the system but the delays still affect the system’s response. In conclusion, the adaptive PIcontroller improves the system’s response compared to a normal PI controller but the system’ssettling time and overshoot are still affected compared to the local response.

37


8 Future work

Since there has not been plenty of work done in this area, there are many possibilities left to tryout in the future. More testing could be done concerning how the system is affected when the loadsof both the end devices and the server nodes change. This would be a more realistic situation sincethe loads are not typically constant on either end. While the prototype used in this work is verysimple. The two approaches presented in handling the system (polling and non-polling) could beexpanded into some more sophisticated system. For less reliable connections, the setup could alsobe expanded with dynamic offloading solutions. In the future work, some other, more complexsystem, for example modeled with a second order differential function, could be explored. For anymodeled system that is affected by delays, a delay compensator might be a possible solution, as isthe case in [55]. Another interesting future work is to implement an advanced delay compensator.

38


9 References

[1] K. Ashton, “That ’Internet of Things’ Thing, in the real world things matter more thanideas.” http://www.rfidjournal.com/articles/view?4986, June 2009. [Online; Accessed06-February-2015].

[2] J. Holler, V. Tsiatsis, C. Mulligan, S. Avesand, S. Karnouskos, and D. Boyle, From Machine-to-machine to the Internet of Things: Introduction to a New Age of Intelligence. AcademicPress, 2014.

[3] N. Jazdi, “Cyber physical systems in the context of Industry 4.0,” in IEEE InternationalConference on Automation, Quality and Testing, Robotics, pp. 1–4, IEEE, 2014.

[4] Gartner, “Gartner says the Internet of Things installed base will grow to 26 billion units by2020 .” http://www.gartner.com/newsroom/id/2636073. [Online; Accessed 20-March-2015].

[5] A. Serbanati, C. M. Medaglia, and U. B. Ceipidor, Building blocks of the internet of things:State of the art and beyond. INTECH Open Access Publisher, 2011.

[6] F. Bonomi, R. Milito, J. Zhu, and S. Addepalli, “Fog computing and its role in the internetof things,” in Proceedings of the 1st edition of the MCC workshop on Mobile cloud computing,pp. 13–16, ACM, 2012.

[7] L. M. Vaquero and L. Rodero-Merino, “Finding your Way in the Fog: Towards a Comprehen-sive Definition of Fog Computing,” SIGCOMM Computer Communication Review, vol. 44,no. 5, pp. 27–32, 2014.

[8] M. Aazam and H. Eui-Nam, “Fog Computing and Smart Gateway Based Communicationfor Cloud of Things,” in International Conference on Future Internet of Things and Cloud(FiCloud), pp. 464–470, 2014.

[9] F. Doidge, “Lost in the fog,” Computer Reseller News, pp. 19–20,22, 2014.

[10] M. Hajibaba and S. Gorgin, “A Review on Modern Distributed Computing Paradigms: CloudComputing, Jungle Computing and Fog Computing,” CIT. Journal of Computing and Infor-mation Technology, vol. 22, no. 2, pp. 69–84, 2014.

[11] H. Madsen, G. Albeanu, B. Burtschy, and F. L. Popentiu-Vladicescu, “Reliability in the utilitycomputing era: Towards reliable Fog computing,” in 20th International Conference on Systems,Signals and Image Processing (IWSSIP), pp. 43–46, 2013.

[12] I. Stojmenovic, “Fog computing: A cloud to the ground support for smart things and machine-to-machine networks,” in Australasian Telecommunication Networks and Applications Con-ference (ATNAC), pp. 117–122, 2014.

[13] M. Yannuzzi, R. Milito, R. Serral-Gracia, D. Montero, and M. Nemirovsky, “Key ingredientsin an IoT recipe: Fog Computing, Cloud computing, and more Fog Computing,” in 19th IEEEInternational Workshop on Computer Aided Modeling and Design of Communication Linksand Networks (CAMAD), pp. 325–329, 2014.

[14] A. Toma and C. Jian-Jia, “Server resource reservations for computation offloading in real-timeembedded systems,” in 11th IEEE Symposium on Embedded Systems for Real-time Multimedia(ESTIMedia), pp. 31–39, 2013.

[15] T. Verbelen, P. Simoens, F. De Turck, and B. Dhoedt, “Cloudlets: bringing the cloud tothe mobile user,” in Proceedings of the 3rd ACM workshop on Mobile cloud computing andservices, pp. 29–36, ACM, 2012.

[16] T. Soyata, R. Muraleedharan, J. Langdon, C. Funai, S. Ames, M. Kwon, and W. Heinzelman,“COMBAT: mobile-Cloud-based cOmpute/coMmunications infrastructure for BATtlefield ap-plications,” in SPIE Defense, Security, and Sensing, pp. 84030K–84030K–13, InternationalSociety for Optics and Photonics, 2012.

39

http://www.rfidjournal.com/articles/view?4986

http://www.gartner.com/newsroom/id/2636073


[17] E. Cuervo, A. Balasubramanian, D.-k. Cho, A. Wolman, S. Saroiu, R. Chandra, and P. Bahl,“MAUI: making smartphones last longer with code offload,” in Proceedings of the 8th ACMInternational conference on Mobile systems, applications, and services, pp. 49–62, ACM, 2010.

[18] T. Soyata, R. Muraleedharan, C. Funai, K. Minseok, and W. Heinzelman, “Cloud-Vision:Real-time face recognition using a mobile-cloudlet-cloud acceleration architecture,” in IEEESymposium on Computers and Communications (ISCC), pp. 59–66, 2012.

[19] S. Lawson, “Fog Computing Links Clouds to the Internet of Things,” vol. 27, no. 9, 2014.Cisco Systems Inc. Copyright - Copyright 2012 CXO Media Inc.

[20] C. Mims, “Keywords: Forget ’the Cloud’; Computing’s Future Lies in ’the Fog’,” May 2014.Name - Cisco Systems Inc; World Economic Forum Copyright - (c) 2014 Dow Jones & Com-pany, Inc. Reproduced with permission of copyright owner. Further reproduction or distribu-tion is prohibited without permission. Last updated - 2014-05-29.

[21] F. Bonomi, R. Milito, P. Natarajan, and J. Zhu, “Fog computing: A platform for internet ofthings and analytics,” in Big Data and Internet of Things: A Roadmap for Smart Environ-ments, pp. 169–186, Springer, 2014.

[22] M. Firdhous, O. Ghazali, and S. Hassan, “Fog Computing: Will it be the Future of Cloud Com-puting?,” in Proceedings of the 3rd International Conference on Informatics & Applications,Kuala Terengganu, Malaysia, pp. 8–15, 2014.

[23] V. A. Kumar and E. Prasad, “Fog Computing: Characteristics, Advantages and Security-Privacy,” International Journal of Computer Science and Management Research, vol. 3,no. 11, 2014.

[24] B. Ottenwalder, B. Koldehofe, K. Rothermel, and U. Ramachandran, “MigCEP: operatormigration for mobility driven distributed complex event processing,” in Proceedings of the 7thACM international conference on Distributed event-based systems, pp. 183–194, ACM, 2013.

[25] K. Hong, D. Lillethun, U. Ramachandran, B. Ottenwlder, and B. Koldehofe, “Mobile fog: Aprogramming model for large-scale applications on the internet of things,” in Proceedings ofthe 2nd ACM SIGCOMM workshop on Mobile cloud computing, pp. 15–20, ACM, 2013.

[26] Z. Jiang, D. S. Chan, M. S. Prabhu, P. Natarajan, H. Hao, and F. Bonomi, “ImprovingWeb Sites Performance Using Edge Servers in Fog Computing Architecture,” in 7th IEEEInternational Symposium on Service Oriented System Engineering (SOSE), pp. 320–323, 2013.

[27] T. Verbelen, P. Simoens, F. De Turck, and B. Dhoedt, “Adaptive deployment and configurationfor mobile augmented reality in the cloudlet,” Journal of Network and Computer Applications,vol. 41, no. 1, pp. 206–216, 2014.

[28] W. Lingyun and A. Canedo, “Offloading industrial human-machine interaction tasks to mobiledevices and the cloud,” in IEEE Conference on Emerging Technology and Factory Automation(ETFA), pp. 1–4, 2014.

[29] S. Ting, Y. Mei, J. Yingtao, L. Xiang, and L. Qing, “An adaptive probabilistic scheduler foroffloading time-constrained tasks in local mobile clouds,” in 6th International Conference onUbiquitous and Future Networks (ICUFN), pp. 243–248, 2014.

[30] C. Ling, M. Chen, W. Zhang, and F. Tian, “AR cloudlets for mobile computing,” InternationalJournal of Digital Content Technology and its Applications, vol. 5, no. 12, pp. 162–169, 2011.

[31] I. Stojmenovic and W. Sheng, “The Fog computing paradigm: Scenarios and security issues,”in Federated Conference on Computer Science and Information Systems (FedCSIS), pp. 1–8,2014.

[32] S. Kulkarni, S. Saha, and R. Hockenbury, “Preserving privacy in sensor-fog networks,” in9th International Conference for Internet Technology and Secured Transactions (ICITST),pp. 96–99, 2014.

40


[33] M. Satyanarayanan, P. Bahl, R. Caceres, and N. Davies, “The case for vm-based cloudlets inmobile computing,” Pervasive Computing, IEEE, vol. 8, no. 4, pp. 14–23, 2009.

[34] G. A. Lewis, “Mobile computing at the edge (keynote),” in Proceedings of the 1st ACM Inter-national Conference on Mobile Software Engineering and Systems, pp. 69–70, ACM, 2014.

[35] Z. Qin, J. Zhang, and X. Zhang, An Effective Partition Approach for Elastic ApplicationDevelopment on Mobile Cloud Computing, vol. 7296 of Lecture Notes in Computer Science,ch. 4, pp. 46–53. Springer Berlin Heidelberg, 2012.

[36] T. Nishio, R. Shinkuma, T. Takahashi, and N. B. Mandayam, “Service-oriented heterogeneousresource sharing for optimizing service latency in mobile cloud,” in Proceedings of the 1stinternational workshop on Mobile cloud computing & networking, pp. 19–26, ACM, 2013.

[37] T. Verbelen, P. Simoens, F. De Turck, and B. Dhoedt, Adaptive Application Configurationand Distribution in Mobile Cloudlet Middleware, vol. 65 of Lecture Notes of the Institute forComputer Sciences, Social Informatics and Telecommunications Engineering, ch. 13, pp. 178–191. Springer Berlin Heidelberg, 2013.

[38] J. K. Zao, T. T. Gan, C. K. You, S. J. R. Mendez, C. E. Chung, Y. T. Wang, T. Mullen, andT. P. Jung, “Augmented brain computer interaction based on fog computing and linked data,”in International Conference on Intelligent Environments (IE), pp. 374–377, 2014.

[39] Y. Nimmagadda, K. Kumar, L. Yung-Hsiang, and C. S. G. Lee, “Real-time moving objectrecognition and tracking using computation offloading,” in International Conference on Intel-ligent Robots and Systems (IROS), IEEE/RSJ, pp. 2449–2455, 2010.

[40] K. Kumar, J. Liu, Y.-H. Lu, and B. Bhargava, “A Survey of Computation Offloading forMobile Systems,” Mobile Networks and Applications, vol. 18, no. 1, pp. 129–140, 2012.

[41] S. J. Stolfo, M. B. Salem, and A. D. Keromytis, “Fog Computing: Mitigating Insider DataTheft Attacks in the Cloud,” in IEEE Symposium on Security and Privacy Workshops (SPW),pp. 125–128, 2012.

[42] M.-R. Ra, A. Sheth, L. Mummert, P. Pillai, D. Wetherall, and R. Govindan, “Odessa: enablinginteractive perception applications on mobile devices,” in Proceedings of the 9th internationalconference on Mobile systems, applications, and services, pp. 43–56, ACM, 2011.

[43] A. Toma and J.-J. Chen, “Computation offloading for real-time systems,” in Proceedings ofthe 28th Annual ACM Symposium on Applied Computing, pp. 1650–1651, ACM, 2013.

[44] A. Toma, C. Jian-Jia, and L. Wei, “Computation offloading for sporadic real-time tasks,” in20th IEEE International Conference on Embedded and Real-Time Computing Systems andApplications (RTCSA), pp. 1–10, 2014.

[45] H. Viswanathan, E. K. Lee, I. Rodero, and D. Pompili, “Uncertainty-aware Autonomic Re-source Provisioning for Mobile Cloud Computing,” IEEE Transactions on Parallel and Dis-tributed Systems, vol. PP, no. 99, pp. 1–1, 2014.

[46] M. A. Elgendy, A. Shawish, and M. I. Moussa, “An Enhanced Version of the MCACC to Aug-ment the Computing Capabilities of Mobile Devices Using Cloud Computing,” InternationalJournal of Advanced Computer Science and Applications(IJACSA), Special Issue on ExtendedPapers from Science and Information Conference, 2014.

[47] Elgendy, M. A. and Shawish, A. and Moussa, M. I., “MCACC: New approach for augmentingthe computing capabilities of mobile devices with Cloud Computing,” in Science and Informa-tion Conference (SAI), pp. 79–86, 2014.

[48] L. L. Ferreira, G. Silva, and L. M. Pinho, “Service offloading in adaptive real-time systems,”in 16th IEEE Conference on Emerging Technologies & Factory Automation (ETFA), pp. 1–6,2011.

41


[49] A. Toma and C. Jian-Jia, “Computation Offloading for Frame-Based Real-Time Tasks with Re-source Reservation Servers,” in 25th Euromicro Conference on Real-Time Systems (ECRTS),pp. 103–112, 2013.

[50] Z. Yuan, L. Hao, J. Lei, and F. Xiaoming, “To offload or not to offload: An efficient codepartition algorithm for mobile cloud computing,” in 1st IEEE International Conference onCloud Networking (CLOUDNET), pp. 80–86, 2012.

[51] S. Imai and C. A. Varela, “Light-weight adaptive task offloading from smartphones to nearbycomputational resources,” in Proceedings of the ACM Symposium on Research in AppliedComputation, pp. 146–152, ACM, 2011.

[52] V. Pandey, S. Singh, and S. Tapaswi, “Energy and Time Efficient Algorithm for Cloud Offload-ing Using Dynamic Profiling,” Wireless Personal Communications, vol. 80, no. 4, pp. 1687–1701, 2015.

[53] W. Liu, J.-J. Chen, T.-W. Kuo, Q. Deng, et al., “Computation offloading by using timingunreliable components in real-time systems,” in Proceedings of the 51st ACM/EDAC/IEEEDesign Automation Conference (DAC), pp. 1–6, IEEE, 2014.

[54] J. Delsing, F. Rosenqvist, O. Carlsson, A. W. Colombo, and T. Bangemann, “Migration ofindustrial process control systems into service oriented architecture,” in 38th Annual IEEEConference on Industrial Electronics Society (IECON), pp. 5786–5792, IEEE, 2012.

[55] T. Hegazy and M. Hefeeda, “Industrial automation as a cloud service,” IEEE Transactionson Parallel and Distributed Systems, vol. PP, no. 99, pp. 1–1, 2015.

[56] ISA, “International standard for the integration of enterprise and control systems.”http://www.isa-95.com/. [Online; Accessed 20-May-2015].

[57] R. Langmann and L. Meyer, “Automation services from the cloud,” in 11th IEEE InternationalConference on Remote Engineering and Virtual Instrumentation (REV), pp. 256–261, IEEE,2014.

[58] H. Sequeira, P. J. Carreira, T. Goldschmidt, and P. Vorst, “Energy Cloud: real-time cloud-native Energy Management System to monitor and analyze energy consumption in multipleindustrial sites,” in Proceedings of the 7th IEEE/ACM International Conference on Utilityand Cloud Computing (UCC), pp. 529–534, 2014.

[59] O. Givehchi and J. Jasperneite, “Industrial automation services as part of the Cloud:First experiences,” Proceedings of the Jahreskolloquium Kommunikation in der Automation(KommA), Magdeburg, 2013.

[60] O. Givehchi, H. Trsek, and J. Jasperneite, “Cloud computing for industrial automation systemsA comprehensive overview,” in 18th IEEE Conference on Emerging Technologies & FactoryAutomation (ETFA), pp. 1–4, 2013.

[61] L. Wang and A. Canedo, “Offloading industrial human-machine interaction tasks to mobiledevices and the cloud,” in 19th IEEE Conference on Emerging Technology and Factory Au-tomation (ETFA), pp. 1–4, IEEE, 2014.

[62] O. Givehchi, J. Imtiaz, H. Trsek, and J. Jasperneite, “Control-as-a-service from the cloud: Acase study for using virtualized PLCs,” in 10th IEEE Workshop on Factory CommunicationSystems (WFCS), pp. 1–4, 2014.

[63] A. Ishii and T. Suzumura, “Elastic stream computing with clouds,” in Proceedings of the 4thIEEE International Conference on Cloud Computing (CLOUD), pp. 195–202, IEEE, 2011.

[64] Industrie 4.0 Working Group, “Recommendations for implementing the strategic initiativeIndustrie 4.0 .” http://www.plattform-i40.de/finalreport2013, 2013. [Online; Accessed06-February-2015].

42

http://www.isa-95.com/

http://www.plattform-i40.de/finalreport2013


[65] TNO, Ministry of Economic Affairs, VNO-NCW and the Chambers of Commerce, FME,“Smart Industry Duthch Industry fit fiot the Future.” http://www.smartindustry.nl/

wp-content/uploads/2014/07/Opmaak-Smart-Industry.pdf, 2014. [Online; Accessed 06-February-2015].

[66] Government Office for Science, London, UK, “The Future of Manufacturing: A newera of opportunity and challenge for the UK .” http://www.ifm.eng.cam.ac.uk/uploads/

Resources/Future_of_Manufacturing_Report.pdf, 2013. [Online; Accessed 06-February-2015].

[67] European Factories of the Future Research Association (EFFRA) . http://www.effra.eu/.[Online; Accessed 20-March-2015].

[68] Industrial Internet Consortium . http://www.industrialinternetconsortium.org. [Online;Accessed 20-March-2015].

[69] J. F. Nunamaker Jr and M. Chen, “Systems development in information systems research,” inProceedings of the 23rd Annual Hawaii International Conference on System Sciences, vol. 3,pp. 631–640, IEEE, 1990.

[70] R. Palelli, “Recommendations of Ada for Safetry-Critical software.” http://russp.us/

Ada-recommend.pdf. [Online; Accessed 06-February-2015].

[71] C.-S. Shih, S.-M. Wang, J. Chen, and Y.-H. Wang, “Workload migration framework for stream-ing applications on smartphones,” in Proceedings of the 20th IEEE International Conferenceon Embedded and Real-Time Computing Systems and Applications (RTCSA), pp. 1–8, IEEE,2014.

[72] R. G. Brown, Smoothing, forecasting and prediction of discrete time series. Courier Corpora-tion, 2004.

[73] J. Guerard, Introduction to financial forecasting in investment analysis. Springer Science &Business Media, 2013.

[74] K. J. Astrom and T. Hagglund, Advanced PID control. ISA-The Instrumentation, Systems,and Automation Society; Research Triangle Park, NC 27709, 2006.

[75] Y. Chen, Z. Du, and M. Garcıa-Acosta, “Robot as a service in cloud computing,” in 5thIEEE International Symposium on Service Oriented System Engineering (SOSE), pp. 151–158, IEEE, 2010.

[76] Y. Zhang, X. Fu, and K. Ramakrishnan, “Fine-Grained Multi-Resource Scheduling in CloudDatacenters,” in 20th IEEE International Workshop on Local and Metropolitan Area Networks(LANMAN), pp. 1–6, 2014.

43

http://www.smartindustry.nl/wp-content/uploads/2014/07/Opmaak-Smart-Industry.pdf

http://www.smartindustry.nl/wp-content/uploads/2014/07/Opmaak-Smart-Industry.pdf

http://www.ifm.eng.cam.ac.uk/uploads/Resources/Future_of_Manufacturing_Report.pdf

http://www.ifm.eng.cam.ac.uk/uploads/Resources/Future_of_Manufacturing_Report.pdf

http://www.effra.eu/

http://www.industrialinternetconsortium.org

http://russp.us/Ada-recommend.pdf

http://russp.us/Ada-recommend.pdf

real-time control in industrial iot - diva...

Documents