Fog Computing Middleware for DistributedCooperative Data Analytics

Jose Clemente∗, Maria Valero∗, Javad Mohammadpour∗, Xiangyang Li† and WenZhan Song∗∗College of Engineering, University of Georgia.†University of Science and Technology of China.

Abstract—This paper presents an innovative fog computingmiddleware for distributed cooperative data analytics (DCDA) inthe Internet of Things (IoT). The existing IoT systems gather datain a central place (e.g., cloud) for post-processing and high-levelsituation awareness. Such a collection-and-computing paradigmcannot sustain due to exponential sensor and data growth - thecommunication bandwidth is limited by spectrum and cannotcatch up the data growth rate. In addition, many criticalcyber-physical applications demand time-sensitive decisions, andprivacy concerns prefer to avoid data collection. Distributedcooperative data analytics for high-level situation awareness canavoid costly data collection and meet the time and privacyrequirements, and has received much attention in the past severalyears. In this work, we propose a DCDA architecture andmiddleware to support flexible and scalable DCDA, and test andvalidate the proposed middleware in seismic and ambient noiseimaging case studies. The evaluation results in a fog computingtestbed demonstrate that the DCDA middleware allows scalableand fault-tolerant data analytics and is suitable for real-timesituation awareness under the bandwidth and communicationconstraints.

Index Terms—Fog computing, Middleware, Data analytics,Distributed applications.

I. INTRODUCTION

The Internet of Things (IoT) have experienced an exponentialgrowth in the past few years with billions of devices connectedto the Internet [1]. At the same time, the storage and thecomputation capabilities of these devices approximately doubleevery eighteen months to support intensive data analytics,while the communication bandwidth does not grow at thesame speed and is limited by the spectrum availability [2].Bandwidth limitations make it infeasible to send all datato a central place (e.g., cloud) for post-processing. Latencybecomes an issue when IoT systems need to transfer data in realtime. To overcome the bottlenecks of bandwidth and latencydeficiencies and take full advantage of powerful computationof edge devices, the fog computing (also called fogging or edgecomputing) paradigm was introduced in [3]. This paradigmbrings the data processing, networking, storage and analyticscloser to the devices and applications.

In this paper, we propose a novel fog computing middlewareto support scalable and flexible distributed cooperative dataanalytics (DCDA) at the edge of a network. The proposedmiddleware allows processing, computing, analyzing and shar-ing information at the edge for high-level situation awareness.The proposed DCDA middleware provides services that allowthe analysis of the most time-sensitive data at the network

edge, minimize latency, conserve network bandwidth, allowcollecting and processing data across a wide geographic areaunder different environmental conditions.

The proposed DCDA middleware manages three differentlevels of data processing and analytics illustrated in Fig. 1 forthe cooperation among edge devices connected to the network.On the low level, DCDA middleware intends to generate andanalyze real-time data for protection and control systems,e.g., alerts and notifications that require immediate actionsin the network. On the intermediate level, DCDA handlesoperation data, as well as historical data to generate real-timeanalytics for visualization systems and processes. An exampleis the real time monitoring of data on smart grids, connectedvehicles, seismographs, etc. Finally, on the high level, themiddleware aims to generate transactional analytics for decision-making process, e.g., real-time 2D and 3D visualization, eventsdetection in the network and compromised functionality.

DCDA middleware enables a mesh network composedof many devices to share resources and capabilities. Everyedge device, here called node, is unique and independent ofother nodes. Every node handles a set of functionalities toaddress different problems and its own database which storescompressed data. Nodes may cooperate with others to solvea global optimization problem when they are sharing a task,or provide determined services to other nodes when they areworking in cooperative mode. For instance, two nodes mightwork together to solve a linear least-squares system of equationsby sharing and analyzing information among them. On the otherhand, one node (A) might provide assistance to another node(B) that needs to use a specific function that is available in nodeA. This mechanism provides a comprehensive fog middlewarefor distributed applications that focus on data analytics.

In the literature, there exist research studies that focus ondistributed computation for solving problems at the edge ofthe network. We have pioneered the development of suchcomputing methodology for seismic applications [4]–[9]. Thisworks presents a generalized middleware framework that allowsapplications to execute different kinds of algorithms, communi-cate partial results with neighbor nodes, share resources amongthem and solve a wide variety of problems at the edge of anetwork.

To the best of our knowledge, this is the first comprehensivefog computing middleware for distributed cooperative dataanalytics in real time that can be used on real-world sensorsin a wide-range geographical location. The rest of the paper is

Fig. 1. Levels of data processing and analytics inside DCDA middleware.

organized as follows. Section II presents a needed backgroundon distributed cooperative data analytics. Section III shows thesystem design and formulation of our proposed middleware.Section IV presents two different case studies, and we carry outexperiments using real sensor data in Section V. The conclusionand future work are presented in Section VI.

II. DISTRIBUTED COOPERATIVE DATA ANALYTICS

Cooperation between edge devices has become a topicof significant interest in recent years [10]. Challenges ofdistributed computation and decision making in sensor networksare not trivial because of potentially harsh, uncertain, anddynamic environments, along with energy and bandwidthconstraints. In this section, we describe the main challengesof developing a distributed cooperative middleware and howwe propose data analytics at the edge of a network.

A. Challenges and Solutions

In the design of a middleware for distributed cooperativesolutions in a network, multiple challenges have to be consi-dered. First, in cooperative signal and information processing,we have to consider the trade-off between performance andresource utilization. Sharing more data usually provides a highperformance in distributed algorithms but also requires morecommunication resources (energy and bandwidth). Second, theinformation shared between one node and another needs to becombined with local information in a certain way. This fusionof information may range from simple rules to model-basedtechniques. Finally, another important issue in cooperativeanalytics is visualization. Users should be able to see in realtime what is happening at the edge of the network and howpartial results are contributing to final results.

DCDA of this paper attempts to overcome the aforemen-tioned challenges by using a mechanism for exchangingprocessed information instead of raw data, allowing differenttypes of algorithms that combine information from differentnodes, and provide a simple interface to interactively querythe nodes. Also, the proposed DCDA provides an architecturein which every node has local databases for raw and processeddata allowing fast querying and information visualization inreal time. We also design an intuitive graphic interface thathelps in monitoring the edge network.

III. MIDDLEWARE DESIGN

DCDA middleware is designed to allow nodes workingtogether on a specific task and/or share resources when they areworking in the cooperative mode. In this section, we exemplifythe architecture, formulation and implementation used in themiddleware design.

A. Middleware Architecture

The general architecture of the proposed middleware isshown in Fig. 2. Every node has a relatively small but powerfulcomputational unit (e.g., Beaglebone Black, Raspberry Pi).Nodes are able to store raw and processed information in thedatabase that is designed for compressed data. Inside eachprocessing unit, nodes are able to manage input data, generatereal-time information, and analyze and develop analytics. Everynode is also capable to communicate and cooperate withneighbors using wireless, Xbee or bluetooth. Additionally,DCDA middleware allows users to observe in real time thestate of the network by a simple visualization and analyticsinterface.

Fig. 2. Middleware general architecture.

B. Middleware Model Formulation

DCDA middleware model is based on a mesh network (F )composed of fog nodes (Υ). Υj represents a specific fog nodej in F , where j = 1, ..,m and m is the total number of nodes.

Definition 1. We define Υj as a finite sequence of terms suchas Υj = (Ij , Sj , Uj , τj ,Λj ,Γj[p]),

where Ij is the unique id number of Υj in F and Sj is the stateof node Υj which indicates if Υj is joined in the distributedcomputation. Notice that Sj is useful to determine if the nodeis up or down in certain stage of the computation. Furthermore,Uj is the degree or number of neighbors of Υj , τj denotesthe timestamp, Λj represents hardware information of Υj , andfinally, Γj[p] represents a list of p libraries available inside Υj .

Definition 2. The hardware information Λj of each fog nodeΥj is defined as a finite number of elements that describe thecurrent status of processor, memory and battery, such as

Λj = (Pj ,Mj , Bj , ρj , αj , Cj),

where Pj is the total capacity of the processor in Υj in Hz,Mj denotes the total memory in Υj by time τj in Gb, Bj

represents the battery storage available in Υj by time τj , ρjis the available percentage of the processor in Υj by time τj .αj is the percentage of memory available in Υj by time τj ,and finally, Cj denotes the communication type (for instance,WiFi, Bluetooth, XBee, etc.)Definition 3. We define Γj as the specific library functionavailable in a node Υj ,

Γj = (ψj , κj , µj , ωj),

where ψj is the id of the library, κj and µj are booleanparameters that indicate if the library is in the node Υj and ifit is available for computations, respectively, and ωj denotes thetype of library (mathematical operation, filter, image processing,etc.)

Nodes in the proposed middleware can work together onthe same task (task sharing mode) or can cooperate amongthemselves to provide services to each other (cooperationmode).

Task sharing mode: In task sharing mode, a set of m nodesneed to solve a specific problem by sharing information amongthem. For instance, consider this general problem: every nodej privately holds an objective Fj : Rn → R, which describesthe data acquisition process at node j. The goal is to find theglobal consensus solution x ∈ Rn to minimize the optimizationproblem min

x∈Rn

{F (x) :=

∑mj=1 Fj(x)

}. This general problem

is applied for distributed travel time tomography [11].Under these circumstances, our middleware allows the nodes

to construct a mechanism to combine the information of theneighbors with the local one. This mechanism consists ofa ‘handshaking’ in which a node shares its information Υj

for constructing weighted matrices to assign the contributionpercentage of a node in the computation. Note that here,the information of Sj and Uj is useful to know whether anode is contributing in the computation and its degree. Afterthe ‘handshaking’ process is complete, the nodes start tocommunicate using a communication type (Λj(Cj)

of nodeΥj) to obtain the final result.

Cooperation mode: In cooperation mode, one node Υj mayneed to compute some analytics, in a determined timestamp,using libraries and algorithms available in other nodes Υi1, Υi2,..., where i1, i2, .. ∈ Nj and Nj is the neighborhood of Υj .Notice that |Nj | is the number of neighbors of node Υj andhere denoted by Uj . The node Υj decides which neighbor nodeit will help to compute the algorithm according to equation (1)

Ej = maxi=1:Uj

(Γi[x](κi)· Γi[x](µi)

) ·( 1

lij+ (Λi(Pi)

· Λi(ρi))+

(Λi(Mi)· Λi(λi)

)),

(1)where x is the specific library to execute and lij is the latencybetween node i and node j. Notice that lij is calculated usingthe timestamps between nodes i and j. Also, the node Υi withthe maximum Ej is the neighbor node with more resourcesavailable to meet Υj requirements. Υj transfers the compressedand preprocessed data to Υi and obtains a result as answer.

C. Middleware Implementation

Every node in DCDA middleware possesses the followingstructure:

1) Database: We use MySQL database at every node ofDCDA middleware. However, any other database can be easilyadapted to our middleware. We have one database for storingstream data (transactional database) and another for processeddata (analytic database). The transactional database storesthe node measurements in real time. Tables in transactionaldatabase contain a field called traceBuf that saves a binaryarray with a header, a buffer structure and the data itself.The binary data is compressed using zlib library achievinga reduction in size of 80% compared to original raw data.The analytic database is configurable and depends on thespecific analysis that the node wants to perform over data.For example, we can create a table to store the location and/ormagnitude of an earthquake that has been detected by a mesh ofsensors deployed in a wide geographical area. Furthermore, ourmiddleware database is flexible to store any kind of analyzeddata that comes from different types of analytic algorithms.

2) Library Manager and Adaptor: The library managerwithin a fog node is responsible for ensuring the correctalgorithm selection in the Task sharing mode. For example,if DCDA is configured to execute an algorithm for solving adistributed optimization problem, the library manager ordersthe processing unit to solve the problem and communicatethe results with other neighbors at each iteration. The libraryadaptor is mainly used in the Cooperation mode to selectwhich library and functions will be used to cooperate withother nodes.

3) Processing Unit: The processing unit is responsible forexecuting a determined algorithm within the fog node. Thisunit is also in charge of communicating with neighbors toshare information at each iteration or communicate the finalresults.

4) Middleware Visualizer: We integrate a visualization toolto DCDA middleware in order to monitor results in real time.This tool is based on a Java Servlet framework that is capableof: 1) Connecting via IP addresses to different edge devicesto visualize node states, characteristics and data; 2) Readingdirectly from the device database to perform off-line dataanalysis; 3) Visualizing partial and final results of algorithmsthat are performed on different nodes in real time; 4) Visualizingdata analytics over the network. This tool is expandable toincorporate more features using a simple language such asJava.

IV. CASE STUDIES

In order to validate the implementation of the proposedDCDA middleware, we use two case studies of subsurfaceimaging and monitoring in real time: Travel Time Tomography[5] and Ambient Noise Tomography [12]. We chose subsurfaceimaging because the process of monitoring in real time iscrucial for understanding subsurface structures and dynamicsthat may pose risks or opportunities for oil/gas exploration andproduction, civil infrastructure, etc.

Travel Time TomographyTravel time tomography involves many steps as illustrated

in Fig. 3. Those include: (i) Seismic sensors or edge devices(green circles) that measure the vibration after the occurrence ofan earthquake and calculate the arrival time of the p-wave [13].This process is called arrival time picking; (ii) Next, theearthquake location is estimated; (iii) The magnitude of theearthquake is then calculated by combined information of alledge devices; (iv) Lastly, the rays are traced from earthquake tonodes and the tomography inversion is performed. We have usedthe details of the work presented in [5] for distributed traveltime tomography. In this paper, we use distributed algorithmsfor calculating the arrival time picking as described in [13], well-known algorithms for earthquake location (Greiger’s method)[14] and magnitude calculation method described in [15].

Fig. 3. Travel time tomography.Ambient Noise Tomography

Ambient noise tomography (ANT) uses vibrations of theambient noise to estimate the local phase speed of the waves.ANT involves the steps shown in Fig. 4. Those include:(i) Seismic sensors or edge devices (green circles) measurethe vibration of the ambient noise; (ii) Edge devices calculatethe cross-correlation of the signal waves with their neighborsand perform a frequency-time analysis to obtain travel timemeasurements of the ambient noise signal; (iii) Using a methodknown as Eikonal Tomography [16], a speed map is built.The formulation of the method can be found in [16] and thedistributed implementation is described in [12].

V. EXPERIMENTAL RESULTS

A. EquipmentWe applied our proposed solution to seismic and ambient

noise tomography case studies and evaluated using a fog

Fig. 4. Ambient noise tomography.

computing testbed and real seismic devices deployed in thefield. We developed a fog computing testbed consisting of64 BeagleBone Black (BBB). These computing units areinterconnected via wireless network forming a cluster. EachBBB operates a Linux based OS and has 512MB DDR3 RAM,16GB flash storage, and 1 GHz ARM Cortex A8 processor.They are credit card sized, low-power computing units thatcost under USD $70. For field testing, we have used 25 R1+devices which are small, light-weight seismograph nodes. R1+is designed for data acquisition in the field and is equippedwith an internal, industry-standard Linux processor. R1+ alsohas an external antenna for Xbee communication that allows acommunication range up to 45 km with a high gain antenna.It also can communicate via wireless protocols.

B. Dataset

For travel time tomography, we use real data acquired by R1+devices. We deployed these devices in a wide range area insidethe University of Georgia campus. We artificially generatedearthquake signals using a hammer and a metal pad over thefield. The data recorded by the instruments was processedin real time by taking advantage of our proposed DCDAmiddleware. For ambient noise tomography, we used ambientnoise data collected from 500 stations of the EarthScopeTransportable Array (USArray)1 database deployed over ourBBB cluster. The data was compiled between September andNovember of 2007. We use distributed cooperative computa-tion for generating ambient noise velocity maps inside ourmiddleware.C. Middleware vs. Cloud Time Evaluation

Fig. 5 shows the results of running distributed algorithmsfor travel time tomography and ambient noise tomography.We measured the speedup between computing travel time andambient noise tomography in our middleware and the cloud.Fig. 6(a) illustrates the improvements in terms of computationtime achieved using the DCDA middleware. Note that forsome processes such as arrival time picking and ambient noiseimaging, the cloud is more than twice slower compared to ourmiddleware. This is due to the amount of data to transfer tothe cloud for these processes. For processes with less amountof data to transfer, the speedup of our middleware is not toohigh. This scenario confirms our premise that for real-timeapplications, computing in the edge of the network is fasterand reduces latency. Furthermore, with DCDA middleware weonly transfer processed data which implies an efficient use ofavailable bandwidth in the network.

D. Scalability, Robustness and Flexibility

The proposed DCDA middleware can be augmented tohandle increasing workloads. We can easily add more nodesto the network and integrate them into the cooperation modeor task sharing mode. Most importantly, we can increase thenumber of algorithms performed within our middleware. Wereduce the bandwidth bottleneck by ensuring the processed

1http://www.earthscope.org/science/observatories/usarray

(a) (b)Fig. 5. (a) 2D and 3D locations and magnitude of different detected earthquakes (red circles) using DCDA middleware. (b) Ambient noise velocity mapconstructed using our DCDA middleware.

(a) (b) (c)Fig. 6. (a) Time evaluation and speedup comparison between the proposed DCDA middleware and the Cloud. (b) Performance of DCDA middleware underdifferent cases of node/link failure. (c) Battery usage in hours using DCDA Middleware vs. Cloud.

data flow only between nodes. This also guarantees the systemscalability.

One of the important characteristics of DCDA middlewareis its fault tolerance, and here we validate it by simulatingnode and link failure. We evaluated these properties of DCDAmiddleware in Figure 6(b) using the magnitude calculationalgorithm. We run each experiment with three different cases:Case 1) No Failure; Case 2) 20% of the nodes fail for 10% ofthe time; Case 3) 40% of the nodes fail for 10% of the time.From Fig. 6(b), we observe that there is no significant effect onthe error regarding a scenario with no package loss. However,some algorithms require all nodes to be running and hence itsspeed depends on the slowest node. This property is extremelyimportant when choosing algorithms for edge computing.

Finally, DCDA middleware is flexible to handle differentkinds of distributed algorithms. However, the performanceis directly dependent on the algorithm design. Because themiddleware is based on the Linux distribution, it can be installedon a wide variety of sensors and devices.

0

0.5

20

1

1.5

20

Nu

mb

er

of

Me

ssa

ge

s

×104

2

y

15

2.5

10

x

105

0 0

(a)

0

50

20

100

20

Nu

mb

er

of

Me

ssa

ge

s

150

y

1510

x

200

105

0 0

(b)

Fig. 7. (a) Communication cost in the cloud. (b) Communication cost in theproposed DCDA middleware.

E. Energy ConsumptionWe conducted tests to measure the amount of energy

consumed by fog nodes when computing information usingthe middleware and when sending raw data to be processed atthe cloud. Fig. 6(c) shows the battery usage for different nodesand its average. Even though with the middleware the nodesneed to compute in-situ and communicate with neighbors, theprocess consumes less power than to transmit raw data tothe cloud because the communication cost is higher than thecomputation cost.

F. Communication CostWe measured the communication cost based on the number

of packages transmitted during computation. Fig. 7(a) showsthat at the cloud the communication cost is high because allnodes send raw data for further processing. In contrast, Fig.7(b) illustrates the communication cost using the proposedDCDA middleware. Notice that the cost in DCDA middlewareis very low since nodes communicate only processed data,and picks occur only in the nodes that receive this data forprocessing final data analytics.

VI. CONCLUSION AND FUTURE WORK

In this paper, we presented a fog computing middlewarefor distributed cooperative data analytics (DCDA) in realtime. The proposed middleware can eliminate the need tosend raw data to the cloud and hence overcome bandwithproblems and leverage internal resources to offer fast andbalanced computation. The experimental results show that ourmiddleware is scalable, flexible and energy efficient for real-time distributed applications. We plan to incorporate morefeatures for allowing any type of distributed applications andtackle issues related to security in fog computing.

REFERENCES

[1] D. Evans, “The Internet of Things: How the next evolution of the Internetis changing everything,” CISCO white paper, vol. 1, pp. 1–11, 2011.

[2] Cisco Systems Inc., “The Internet of Things: Extend the cloud to wherethe things are,” 2016.

[3] I. Stojmenovic and S. Wen, “The fog computing paradigm: Scenarios andsecurity issues,” in Computer Science and Information Systems (FedCSIS),2014 Federated Conference on. IEEE, 2014, pp. 1–8.

[4] G. Kamath, P. Agnihotri, M. Valero, K. Sarker, and W.-Z. Song, “Pushinganalytics to the edge,” in 2016 IEEE Global Communications Conference:Selected Areas in Communications: Internet of Things (Globecom2016SAC IoT), Washington, USA, 2016.

[5] G. Kamath, L. Shi, W.-Z. Song, and J. M. Lees, “Distributed travel-timeseismic tomography in large-scale sensor networks,” Journal of Paralleland Distributed Computing, vol. 89, 2016.

[6] L. Zhao, W.-Z. Song, L. Shi, and X. Ye, “Decentralized seismictomography computing in cyber-physical sensor systems,” Cyber-PhysicalSystems, Taylor and Francis, 2015.

[7] G. Kamath, L. Shi, E. Chow, and W.-Z. Song, “Distributed tomographywith adaptive mesh refinement in sensor networks,” International Journalof Sensor Network, 2015.

[8] G. Kamath, W.-Z. Song, P. Ramanan, L. Shi, and J. Yang, “Dristi:Distributed real-time in-situ seismic tomographic imaging,” in 14thInternational Conference on Ubiquitous Computing and Communications(IUCC), Liverpool, UK, 2015.

[9] L. Shi, W.-Z. Song, M. Xu, Q. Xiao, J. M. Lees, and G. Xing, “Imagingvolcano seismic tomography in sensor networks,” in The 10th AnnualIEEE Communications Society Conference on Sensor and Ad HocCommunications and Networks (IEEE SECON), 2013.

[10] N. Bessis, F. Xhafa, D. Varvarigou, R. Hill, and M. Li, Internet ofThings and inter-cooperative computational technologies for collectiveintelligence. Springer, 2013, vol. 460.

[11] L. Zhao, W.-Z. Song, X. Ye, and Y. Gu, “Asynchronous broadcast-baseddecentralized learning in sensor networks,” International Journal ofParallel, Emergent and Distributed Systems, 2017.

[12] M. Valero, G. Kamath, J. Clemente, F.-C. Lin, Y. Xie, and W.-Z.Song, “Real-time ambient noise subsurface imaging in distributedsensor networks,” in The 3rd IEEE International Conference on SmartComputing (SMARTCOMP 2017), 2017.

[13] G. Liu, R. Tan, R. Zhou, G. Xing, W.-Z. Song, and J. M. Lees,“Volcanic earthquake timing using wireless sensor networks,” in The12th ACM/IEEE International Conference on Information Processing inSensor Networks (IPSN13), 2013.

[14] L. Geiger, “Probability method for the determination of earthquakeepicenters from the arrival time only,” Bull. St. Louis Univ, vol. 8, no. 1,pp. 56–71, 1912.

[15] J. Ristau, “Comparison of magnitude estimates for New Zealandearthquakes: Moment magnitude, local magnitude, and teleseismic body-wave magnitude,” Bulletin of the Seismological Society of America,vol. 99, no. 3, pp. 1841–1852, 2009.

[16] F.-C. Lin, M. H. Ritzwoller, and R. Snieder, “Eikonal tomography: surfacewave tomography by phase front tracking across a regional broad-bandseismic array,” Geophysical Journal International, vol. 177, no. 3, pp.1091–1110, 2009.