Research ArticleFog-Based Distributed Networked Control for ConnectedAutonomous Vehicles

Zhuwei Wang , Yuehui Guo , Yu Gao , Chao Fang , Meng Li , and Yang Sun

Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China

Correspondence should be addressed to Chao Fang; fangchao.bupt@gmail.com

Received 15 August 2020; Revised 10 September 2020; Accepted 12 October 2020; Published 3 November 2020

Academic Editor: Li Zhu

Copyright © 2020 Zhuwei Wang et al. This is an open access article distributed under the Creative Commons Attribution License,which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

With the rapid developments of wireless communication and increasing number of connected vehicles, Vehicular Ad HocNetworks (VANETs) enable cyberinteractions in the physical transportation system. Future networks require real-time controlcapability to support delay-sensitive application such as connected autonomous vehicles. In recent years, fog computingbecomes an emerging technology to deal with the insufficiency in traditional cloud computing. In this paper, a fog-baseddistributed network control design is proposed toward connected and automated vehicle application. The proposed architecturecombines VANETs with the new fog paradigm to enhance the connectivity and collaboration among distributed vehicles. A casestudy of connected cruise control (CCC) is introduced to demonstrate the efficiency of the proposed architecture and controldesign. Finally, we discuss some future research directions and open issues to be addressed.

1. Introduction

Nowadays, networking technology has become the key ele-ment in various application areas with the increasing numberof connected devices. Networked control systems (NCSs),which consist of spatially distributed sensors, actuators, andcontrollers in the close-loop communication network, arethe powerful enablers to meet the demands for both informa-tion dissemination and data analysis [1, 2]. This characteris-tic of distributed architecture design makes NCSs easier toimplement, maintain, and share information resources,meeting the requirements of future connected system [3].With the development of wireless communication technol-ogy, the transmission rate and reliability of informationexchange through wireless link gradually meet the needs offuture NCSs. In particular, the use of wireless communica-tion eliminates unnecessary wiring among system compo-nents and can be upgraded easily [4]. Therefore, in wirelessnetworked control systems (WNCS), such as wireless sensornetworks and wireless multihop networks, more flexible sys-tem architecture design, more easily resource utilization, andincreased safety can be achieved [5].

However, the usage of wireless networks in the datatransmission introduces message delay and packet loss whichcannot meet the emerging requirements for real-time controlapplications [6]. For example, higher requirements areemerged such as sufficiently reliability and low latency. Previ-ous cloud-based architecture has a high response delay and isnot suitable for delay-sensitive applications. Delays maycause the performance degradation and even more serioussafety issues. In addition, for real-time control applications,the requirements of transmission reliability and stabilityintroduce further research challenges. The complex networkstructures, large network scales, flexible communicationtopologies, and variable communication environments canalso destabilize a WNCS and degrade system performance.In addition, traditional single-controller-based closed-loopcontrol architecture is difficult to meet the needs of real-time control applications [7–10]. In order to improve overallsystem performance and efficiency, multicontroller design isadopted with the advantages in modularity, scalability, androbustness. With the growing number of connected vehiclesrequire more communication, computing, and storageresources in the network.

HindawiWireless Communications and Mobile ComputingVolume 2020, Article ID 8855655, 11 pageshttps://doi.org/10.1155/2020/8855655

Considering the current situation of WNCS, traditionalcloud-based system architectures have pronounced limita-tions with extracommunication cost and waste on network-ing resources. Therefore, it is necessary to introduce edgecomputing and use cooperative features of wireless networksto optimize system performance and stability [11, 12]. In thiscase, all underlying distributed controllers are able to shareinformation within a certain neighboring area and cooperateto make decisions alleviating computation burden and designcomplexity [13]. Fog computing expands the cloud comput-ing paradigm, which is seen as a promising technique tospread the computing resources from the cloud computingservers to the edge in order to balance the load [14]. The inte-gration of cloud, fog, and edge nodes at the lowest perceptionlayer takes the advantages of low latency and flexible networktopology [15]. Among all the service types, the WNCS basedon fog computing provides a new solution for the develop-ment of real-time control applications. Fog computing makesit available for the central data processing and storage on-lineat the cloud and distributed caching and computingresources at the edge [16].

In recent years, with the increasing number of vehicles onthe road, intelligentization and networking of vehicles havebeen widely concerned by automobile manufacturers andresearch institutions [17, 18]. With the new round of techno-logical revolution from the Internet to the Internet of Things(IoT) [19], the construction of Intelligent TransportationSystems (ITSs) has become a tendency to deal with the seri-ous problems of traffic safety and road congestion [20–22].In order to achieve safe, energy-saving, and efficient control,the ability of flexibly information sharing, environmentalawareness, intelligent decision-making, coordinated control,and execution is needed among vehicles [23]. ApplyingWNCS technologies to the intelligent control of vehiclescan strengthen the information exchange, make full use ofsystem resources, and promote the realization of ITS. Sincefog platform can provide computing, storing, and network-ing services, it is considered as a promising solution in theconnected vehicles scenario [24–26].

In the safety aspect, the collision avoidance (CA) systemplays a vital role in preventing the driver-caused accidents[27, 28]. Especially in the cruise control problem, applyingwireless communication among vehicles, rather thancompletely relying on sensors equipped on each vehicle, hasthe potential to provide better system performance [29, 30].In order to deal with above challenges, more flexible designfor the vehicular platoon becomes an emerging trend. Con-nected cruise control (CCC), as a potential solution, has beenproposed to hold a smooth traffic flow with flexible connec-tivity structure and communication topology [31, 32].

Recently, advanced driver assistance systems (ADAS)have attracted widespread attention in order to improve theflow rate and safety of the traffic flow [17, 33]. In the cruisecontrol problem, the CA technology plays a vital role in pre-venting traffic accidents, and the use of wireless communica-tion technology to replace sensors directly equipped betweentraditional vehicles can further provide better system perfor-mance [28, 30]. Therefore, the design of more flexible vehicleplatoon system based on wireless communication technology

has become as an emerging trend [27, 29]. The CCC system,guarantee a stable traffic flow through flexible communica-tion topologies and connection structures, has become apotential solution for future development [31, 32]. In thispaper, we investigate the CCC system as a case study byimplementing the proposed architecture. The main contribu-tions of this paper are listed as follows.

(i) We propose a novel fog-based distributed networkarchitecture toward real-time control applications.The critical system variables involved within thecommunication and control system that may influ-ence system performances are analyzed

(ii) We formulate the CCC problem as a case study ofthe proposed fog-based distributed network archi-tecture, and the control scheme is analyzed thor-oughly with the cloud, fog, and connected vehicles.Then, a linear quadratic optimization problem isformulated to regulate vehicle's longitudinal motionwhen the time delay is considered

(iii) Based on the fog control nodes, the whole vehiclesystem can successfully split into multiple distrib-uted vehicle platoons that each has an automatedvehicle at the tail. Then, a two-step control schemeis proposed to solve the optimal CCC problem. (i)At the offline step, a backward recursion approachis used to derive the optimal control gain. (ii) Atthe online step, the optimal control strategy can bereal-time calculated based on the current systemstates and local cached information

(iv) In a distributed control scheme, the optimal CCCstrategy can be derived as a linear function of currentplatoon state information and the last controlstrategy

The remainder of this article is organized as follows.We propose a novel fog-based distributed network archi-tecture toward real-time application in the following sec-tion. Next, a practical case study is provided to validatethe high efficiency of the proposed architecture. After-ward, some discussions are presented in future researchdirections and open issues. Finally, the conclusion ismade.

2. Proposed Architecture

With the recent revolution in wireless networking applica-tions, such as IoT, Tactile Internet [34], and Vehicular AdHoc Networks (VANETs) [35], fog computing becomes apromising enabler to carry out a massive amount of commu-nication, computation, storage, and networking servicesbetween edge devices and traditional center cloud servers[36]. More specifically, fog computing is more suitable forreal-time control applications with the advantages of lowlatency, location awareness, wider geographical distribution,wider range of mobility, suitable for more nodes, and supportfor network heterogeneity [37].

2 Wireless Communications and Mobile Computing

2.1. Overall Architecture Design. In this paper, we propose afog-based distributed network architecture toward real-timecontrol applications. Figure 1 depicts the overall systemmodel consisting of three logical layers, the cloud, distributedfog, and edge smart devices. The cloud can serve multiple fognodes, and a fog node can serve several dozens of edge nodes.The edge nodes can be sensors, embedded smart devices,mobile terminals, and vehicles. Each fog node can not onlyact as an independent service provider but also work togetherwith other fog nodes to achieve collaborative control and ser-vices to edge nodes. The fog layer can be further classifiedinto two categories, determined by their capability. Normalfog nodes are governed by the more powerful nodes. In theproposed architecture, the fog nodes have the ability of com-municating, processing, and caching to complete twokinds of functions. On the one hand, the fog nodes realizethe control of the edge local network, such as resourceallocation and transmission path planning [38]. On theother hand, the fog nodes act as remote controllers, gener-ate control strategies in real time according to the stateinformation of the controlled platform and network char-acteristics to implement close-loop feedback control.Therefore, the fog nodes can both govern the wireless net-work as well as the real-time control of the controlled plat-form, which can effectively avoid entirely relying on thecloud. This scheme relieves the problem of high latency, andthe usage of distributed cooperative control alleviates thetransmission instability caused by wireless channels in tradi-tional single-controller systems. The above fog-based systemarchitectures can be applied to delay-sensitive systems insmart grid, vehicular network, wireless sensor and actuatornetwork, and smart medical systems.

Figure 2(a) shows the typical WNCS model with the con-trolled plant, sensors, actuators, controllers, and relay nodes.Compared with the architecture in Figure 1, the plant alongwith sensors and actuators is seen as the edge node; therouters and controllers are seen as the fog nodes. The plantstates are sampled at periodic intervals by the sensors capableof wireless communication. In order to realize the closed-loop control, each packet consisting of the plant states istransmitted to the controller over the wireless link; the con-troller computes the control signal based on the sensor mea-surements, then sent it to the actuator to ensure desirabledynamic and steady-state response.

2.2. Critical System Variables. In the proposed system, thereare several critical variables involved with the combinationof control system and wireless communication as shown inFigure 2(b). The practical physical systems are continuous-time systems where typical WNCS considers a discrete-timeplant model with digital controllers. In the WNCS, periodicsampling is widely used, and shorter sampling periodincreases the network traffic causing higher message delayand message loss probability.

In particular, the network-induced delays mainly consistof the communication delays among sensors, controllers, andactuators, varying with communication channel quality.With different applications using various communicationnetworks, the delays may possess diverse characteristics suchas random or deterministic, constant, or time-varying. Theuncertainty of delays significantly degrades the system per-formance. In addition, inevitably channel fading or shadow-ing in the dynamic wireless communication increases thepacket loss probability, and retransmission is unfitted in

Cloud

Internet

Fog

Wirelesslink

Smartdevices

Figure 1: Overall system model of fog-based WNCS.

3Wireless Communications and Mobile Computing

real-time applications. In this case, seamless channel andbandwidth managements are required to optimize the net-work performance. In WNCS, the communication channelis often seen as an uncertain linear system with additiveGaussian noise. In order to design control strategies with par-tial state observation, estimation of the unknown variablesbased on measurements observed over time can be used.

In the proposed architecture, novel design is needed tooptimize the system performance of control and communi-cation. Since the raw sensor data in local networks are con-verged into the fog nodes with unavoidable invalid data, thefog nodes are responsible of filtering and temporarily storingthe data. Fog nodes are also capable of the management ofthe local network with resource allocation, transmission pathplanning, and congestion control. In addition, the dynami-cally changes of nodes result in the time-varying networktopology. In this case, it is difficult to maintain the system sta-bility with single centralized controller [8]. In the moderncontrol system, multiple distributed controller collaborationcan enhance the control performance and provide better sys-tem stability.

3. Case Study: Connected Cruise Control

Today’s automakers, Internet and leading technology com-panies are striving to develop innovative technologies to pro-vide a fully integrated and highly intelligent vehicleexperience. Electric and self-driving cars, the advanced stateof artificial intelligence (AI) technology, and connectivitythrough wireless communication are revolutionizing thefuture of mobility [39, 40]. Connected vehicles are the emerg-ing tendency in automobile manufacturing industries pro-viding advanced quality of service (QoS) and attractiveperformance enhancements. The VANET has been widely

perceived as a promising concept for the realization of ITSwith both safety and efficiency promotion. Vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) communi-cations are implemented in VANETs to support road safety,navigation, entertainments, and other application services.

In this section, we further investigate the system architec-ture toward real-time autonomous driving applications. Inparticular, to fulfill low latency requirements, traditionalcloud-based network architecture may be insufficient withthe growing number of vehicles. Based on the fog-basedWNCS architecture shown in Figure 1, Figure 3 presents itsapplication in the vehicular network scenario. In this archi-tecture, vehicles are seen as edge devices equipped with sen-sors and On Board Units (OBUs). The perception of real-time traffic conditions is fulfilled by roadside sensors andvehicle’s onboard sensors. Vehicles are capable of wirelesscommunication broadcasting their kinematic data and trans-portation information in the local network. Each vehicleplays the role of packet sender, receiver, or even router withintheir reach via wireless medium. Rather than moving at ran-dom, vehicles tend to move in an organized fashion with theslowly varying network topology. RSUs and RSU controllers(RSUCs) are seen as fog nodes to provide more powerfulpacket transmission and computing functions as well as stor-age capabilities in the network in a timely, highly efficient,and coordinated way. RSUs are managed by the RSUC whichhas more resources for computing, storage, and communica-tion through Internet to the cloud. The traditional core cloudfacilitates the large-scale data processing and storage on-lineand provides global view with big data and AI technology.

3.1. Problem Formulation and Control Design. As one of thewidely used applications of Automatic Driving AssistanceSystems, cruise control systems can effectively improve road

Actuators Plant Sensors

SensorRouter

Controller

Wirelessnetwork

(a)

Sampling period

Plant state

TimeNoise

Control signal

New plant state

Error

Sensor

Sensor signal

Loss

Controller

Actuator

Sensor-to-controllerdelay

Processingtime

Controller-to-actuatordelay

(b)

Figure 2: (a) WNCS model. (b) Timing diagram for the control loop over a wireless communication network.

4 Wireless Communications and Mobile Computing

traffic efficiency, reduce traffic accidents, improve vehicle fueleconomy, and promote the realization of the ITS [41]. TheCCC system, as an integration of the sensor-based cruisecontrol system and wireless communication, can make fulluse of system resources to effectively avoid collision alongwith smooth drive [31]. Figure 4 depicts the considered sce-nario where vehicles generally move at a relatively constantspeed, such as following a paved highway, and can be con-trolled as a whole platoon. The vehicular platoon consistsof two kinds of vehicles: the human-driven vehicles and theconnected and automated vehicles (CAVs). The CAVs areable to communicate with each other within a designatedarea through V2V communication and communicate withRSUs through V2I communication [20, 42]. The system is alocation-based distributed control system, and the connectiv-ity structure is basically stable or changing slowly. The localwireless network includes a RSU and vehicles governed bythe RSU, and the transmission path through V2V and V2Iis planned dynamically in RSUs to collaborate the resourcesand balance the load. The close-loop control of the CAV isrealized with the collaboration of the cloud, fog, and vehicles.The cloud can provide big data processing and storage capa-bility which is connected through the Internet. The fog,namely, the RSUs and RSUCs, brings powerful communicat-ing and computing capabilities to the connected vehicles viaV2I communication. The RSUs are responsible for localinformation processing and filtering, thereby effectivelyrelieving the workload on the cloud, making it more suitablefor real-time control applications. The large amount of mov-ing vehicles is capable of sensing the real-time traffic infor-mation. In particular, at each sampling interval, the stateinformation of all vehicles in the platoon is sent to the RSU.Then, the vehicle platoon model is formulated, and the opti-mal control strategy for the CAV is calculated by the RSU.The cooperative control is achieved by the information shar-

ing among the RSUs of the control strategies. In addition, thetransmission path through V2V and V2I is planned dynam-ically in RSUs to allocate the resources and balance the load.Finally, the control signal is downloaded to the CAV to reg-ulate the longitudinal motion based on its real-time states,and the close-loop control is realized.

In this subsection, the optimal control design is investi-gated for the CCC system with communication delays in aheterogeneous platoon including CCC and human-drivenvehicles. First, the mathematical model of the vehicle dynam-ics for the connected vehicles is formulated. Here, we investi-gate the longitudinal control of CAVs where the velocity anddistance between two consecutive vehicles are considered.The CAV’s dynamics is formulated as

_hj tð Þ = vj+1 tð Þ − vj tð Þ,

_vj tð Þ = uj t − τð Þ, ð1Þ

where ujðtÞ, namely, the acceleration, is the control signal of j-th CAV, τ is the time delay introduced by the V2V commu-nication, and vjðtÞ and hjðtÞ are the velocity and headway,respectively.

The behavior of the human-driven vehicle can be formu-lated as [31, 43].

_hi tð Þ = vi+1 tð Þ − vi tð Þ, ð2Þ

_vi tð Þ = αi V hi tð Þð Þ − vi tð Þð Þ + βi vi+1 tð Þ − vi tð Þð Þ, ð3Þwhere VðhÞ denotes the range policy, βi and αi are the systemparameters determined by the human driver, and viðtÞ andhiðtÞ are the velocity and headway of i-th human-drivenvehicle.

Fog 1Fog 2

RSUCRSU

RSU

Cloud

12IV21V2V

Figure 3: Fog-based vehicular network scenario.

5Wireless Communications and Mobile Computing

Considering n vehicles in the whole system as shown inFigure 4, we assume there are m CAVs, and then, the wholesystem can be split into m vehicular platoons with singleCAV in each platoon. The j-th vehicular platoon has nj vehi-

cles that ∑j nj = n. We define the state vector as x =½~h1, ~v1,⋯, ~hn, ~vn�

T, where ~h and ~v denote the state errors to

the desired states h∗ and v∗ Then, base on (1) and (3), themath-ematical model for the whole system can be formulated as

_x tð Þ = Ax tð Þ + Bu t − τð Þ, ð4Þ

where the control signal u is an m-dimensional vector corre-sponding to m CAVs, and the coefficient matrices can beexpressed as

A

A1

A2

⋱

Am

2666664

3777775, =

B1

B2

⋮

Bm

2666664

3777775, ð5Þ

and the block matrices are

Aj =

0 −1 0 1 0 0 ⋯ 0 00 0 0 0 0 0 ⋯ 0 00 0 0 −1 0 1 ⋯ 0 00 0 Γ2 Φ2 0 β2 ⋯ 0 0⋮ ⋮ ⋱ ⋱ ⋱ ⋱ ⋱ ⋮ ⋮

0 0 ⋯ ⋯ 0 0 ⋯ 0 −10 0 ⋯ ⋯ 0 0 ⋯ Γnj

Φnj

2666666666666664

3777777777777775

,Bj = 0, 1, 0,⋯,0½ �T , ð6Þ

where Γi = αi f∗,Φi = −αi − βi, and i = 2, 3,⋯, nj.

Then, the corresponding discrete-time system model canbe formulated as

xk+1 = Akxk + B1kuk + B2

kuk−1, ð7Þ

where

xk = x kTð Þ, uk = u kTð Þ, Ak = eAT ,

B1k =

ðT−τ0

eAtdtB, B2k =

ðTT−τ

eAtdtB,ð8Þ

and T is the sampling period, and τ is the network-induceddelay. Here, the short-delay case is considered that τ issmaller than the sampling period.

The objective is to design an optimal control strategy toreduce energy consumption and improve traffic capacity. Inthe uniform traffic flow, each vehicle is tracking the desiredvelocity and headway. Thus, in order tominimize the deviationsof CAV’s headway, velocity, and acceleration, the optimizationproblem of the control system is typically designed as

min JN = xTNQxN + 〠N−1

k=0xTk Qxk + uTk Ruk� �

,

s:t:xk+1 = Akxk + B1kuk + B2

kuk−1, ð9Þ

whereN is the finite time horizon andQ and R are the symmet-ric positive definite weight matrices. Solving the optimizationproblem, we can obtain the control signals for the CAVs to reg-ulate their longitudinal motion.

In the proposed architecture, RSUs and RSUCs can pro-vide powerful communicating and computing capabilities ina coordinated way, enabling connected vehicles in the local

Cloud

Internet

RSU RSU

Localnetwork

Vehicle platoon

V2V communicationnetwork

#1 #2 # n–1 # n

RSUC

h1 hn–1

𝜈n–1 𝜈n𝜈C 𝜈2

The controlled CAV

Figure 4: A platoon of connected vehicles.

6 Wireless Communications and Mobile Computing

wireless network to interact with each other. Here, vehiclestend to move in an organized fashion in order to maximizethe throughput of the traffic flow. Therefore, a uniformdesired velocity is to be realized for the whole system. Sincethe RSUs can share the local information with each other,the desired states for the traffic flow can be determined bythe RSUCs with global view. Then, with multiple RSUsimplemented in the system to provide seamless experiencefor all vehicles, we can split the whole system into severalparts with a single RSU serving one corresponding vehicularplatoon. In this case, when each platoon reaches the desiredstates, the total cost can beminimized, and the traffic capacityis maximized. Thus, the whole system can split intom vehic-ular platoons as x = ½x1, x2,⋯, xm�T . In particular, each vehi-cle platoon has multiple vehicles with a CAV at the tail, andother vehicles are human-driven. For the given j-th vehicular

platoon, there are total nj vehicles that we have xj =½~h1, ~v1,⋯, ~hnj

, ~vnj�T .

In this case, for the decentralised control system, the opti-mization problem in (9) can be equivalent to a noncoopera-tive control game for each vehicle platoon as

min J j,N = xTj,NQjxj,N + 〠N−1

k=0xTj,kQjxj,k + uTj,kRjuj,k

� �,

s:t:xj,k+1 = Aj,kxj,k + B1j,kuj,k + B2

j,kuj,k−1, ð10Þ

where for CAV j, j = 1, 2,⋯,m; Aj,k, B1j,k, and B2

j,k are the 2nj × 2nj, 2nj × 1, and 2nj × 1 coefficient matrices as parts ofthe original 2n × 2n matrix Ak, 2n × 1 matrix B1

k, and 2n × 1matrix B2

k in (7), respectively; the control signal ujðtÞ is forthe j-th CAV; and Qj and Rj are the corresponding symmet-ric positive definite weight matrices.

In order to solve the optimal control strategy uj,k, we

define a new state vector zj,k = ½xj,k, uj,k−1�T , and we have

zj,k+1 = Fj,kzj,k +Gj,kuj,k, ð11Þ

where the coefficient matrices are

Fj,k =Aj,k B2

j,k

0 0

" #,

Gj,k =B1j,k

I

" #: ð12Þ

Then, by using zk, we can rewrite the optimization prob-

lem in (10) as

min J j,N = zTj,N �Qjzj,N + 〠N−1

k=0zTj,k �Qjzj,k + uTj,kRjuj,k

� �,

s:t:zj,k+1 = Fj,kzj,k +Gj,kuj,k,ð13Þ

where

�Qj =

Qj 0 ⋯ 00 0 ⋯ 0⋮ ⋮ ⋱ ⋮

0 0 ⋯ 0

2666664

3777775: ð14Þ

Then, as a part of cost function J j,N , the residual costfunction is defined as

V j,k = zTj,N �Qjzj,N + 〠N−1

i=k

zj,i

uj,i

" #T �Qj 00 Rj

" #zj,i

uj,i

" #: ð15Þ

Based on our previous works [44], the residual cost func-tion can be derived as a quadratic function of platoon statesthat

V j,k = zTj,kHj,kzj,k: ð16Þ

Then, the optimal control strategy is derived as

uj,k = −Wj,kzj,k, ð17Þ

where

Wj,k = GTj,kHj,k+1Gj,k + Rj

h i−1GT

j,kHj,k+1Fj,k,

Hj,k = FTj,kHj,k+1Fj,k + �Qj −WT

j,kGTj,kHj,k+1Fj,k,Hj,N = �Qj:

ð18Þ

The optimal controller design can be summarized as inAlgorithm 1. The message flows for the control system areshown in Figure 5. In particular, numerous RSUs are seenas multiple remote controllers in the WNCS model. CAVsequipped with sensors and actuators connect to RSUsthrough wireless link. This distributed and collaborate designpromotes the system reliability and stability. Fog nodes dealwith real-time responsiveness while the cloud server focuseson delay-tolerate data processing and storage.

The control scheme is designed in the following steps.First, at each sampling interval, the state information of thewhole platoon is captured by vehicles and uploaded to theRSUs via V2I communication. The RSUs are responsiblefor local information processing and filtering. To enhancethe capacity and efficiency of the system, the selected trafficflow information is transmitted to the cloud server. Based

7Wireless Communications and Mobile Computing

on the historical and real-time data with global insight, thecloud employs a powerful AI to learn the generic trends of traf-fic flow in the near future. In the cloud, the desired speed ofvehicles can be determined to realize dynamic traffic flowman-agement. The computing results in the cloud are then down-loaded to the RSUs. Then, the vehicle platoon model and theoptimal control problem are formulated in the RSUs. The off-line part of the control strategy proposed in Algorithm 1,namely, the coefficient Wj,k, can be calculated by the RSUwhich is then transmitted to the CAV. Finally, the real-timecontrol signal is computed by the on-board controller basedon the coefficientWj,k and the real-time measurements of vehi-cle’s states, namely, the on-line part of Algorithm 1, to regulatevehicle’s longitudinal motion. In the next time period, theupdated vehicle states are feeded back to the RSU to realizethe close-loop control.

3.2. Simulations and Results. In this subsection, we providesimulations of the connected vehicle system to study the per-formance of the proposed control scheme. Here, the typicalscenario of the three-vehicle platoon is considered [31, 43],which can be easily extended to more vehicle scenarios. Inparticular, each platoon consists of 2 normal human-drivenvehicles and a single CAV at the tail. In addition, in orderto validate the performance of the distributed control, 3 dis-tributed platoons with different initial states are consideredin the simulation, which can also be easily extended to morecomplex distributed control system. For the first platoon, theinitial states for the CAV are set to be hð0Þ = 12 [m] and vð0Þ = 5 [m/s]. For the second platoon, the initial states forthe CAV are set to be hð0Þ = 15 [m] and vð0Þ = 8 [m/s]. Forthe last platoon, the initial states for the CAV are set to behð0Þ = 25 [m] and vð0Þ = 20 [m/s]. The global desired veloc-ity for the traffic flow is set as v∗ = 15 [m/s], and the desiredintervehicle distance is set to be h∗ = 20 [m]. The samplingperiods and communication delays are different in each pla-toon, which are set as T = 0:2s, τ = 0:1s; T = 0:2s, τ = 0:3s;

and T = 0:4s, τ = 0:3s to verify the effectiveness of the pro-posed algorithm when the communication topology changes.

Figures 6 and 7 show the headway h and velocity trajecto-ries v of the CAVs in 3 different vehicle platoons, respec-tively. It can be seen that the CAVs’ velocity and headwayare tracking the desired states in the presence of various timedelays, and the remained state errors are close to almost zero.Figure 8 shows the acceleration u of the CAVs, all of which godown to near zero within about 5 seconds. The results depictthat all 3 CAVs gradually reach the same desired states withthe proposed control strategy. Therefore, all of the 3 platoonscan simultaneously track the desired velocity no matter whatthe initial states are. This indicates that the proposed networkarchitecture and control algorithm can regulate the uniformtraffic flow with multiple distributed controlled vehicleplatoons.

4. Future Research Challenges and Open Issues

Current research has achieved great success in the design oflow latency and high efficiency network system, but manyof the technologies are not fully resolved in practical applica-tions. In the following, we conclude the most talked keyresearch directions and challenging issues to be addressed.

4.1. Tradeoff between Communication and Control. Based onthe analysis of the critical system variables involved withwireless communication, it is obvious that the condition ofthe communication network directly influences the controlperformance. In the existing research of WNCS, due to thecomplexity of the control system itself, researchers often sep-arate the design of communication network and controlstrategies independently. However, in practical applications,the network characteristic tends to change dynamically, andthe corresponding control strategies also need to be changed.If the network resource management and control strategyoptimization can be jointly designed, the performance andthe system stability can be further improved. In the indepen-dent design from the control aspect, control strategies arenormally carried out with certain communication require-ments, which results in the loss of serviceability in other envi-ronments. Even the communication system can satisfy thecontrol requirements, predefined communication variablesmay result in the waste of wireless resources with low systemefficiency. Therefore, it is essential to jointly design moreintegrated control algorithms that consider both communi-cation and control so that the whole system can be compati-ble with various delay-sensitive and safe-critical applications.However, in order to realize the integration of control sys-tems with communication, multiple interaction and coordi-nation among system components should be reevaluated.For example, higher information rate brings better controlperformance, which at the same time leads to traffic conges-tion and increases computational burden. In the view ofresource allocation of communication networks, since thewireless resource consumption observably increases withgrowing QoS requirements, the joint design approach mayuse various communication techniques to satisfy differentQoS requirements. In the case study, we consider the

Off-line:1: Initialize Hj,N = �Qj, j ∈ ½1,m�2: fork =N − 1 : −1 : 0 do.3: Calculate Wj,k using

Wj,k = ½GTj,kHj,k+1Gj,k + Rj�−1GT

j,kHj,k+1Fj,k4: Calculate Sj,k using

Hj,k = FTj,kHj,k+1Fj,k + �Qj −WT

j,kGTj,kHj,k+1Fj,k:

5: end forOn-line:1: Determine the desired states h∗, v∗2: Initialize xjð0Þ, uj,k = 0, k ≤ 03: for =k = 0 : 1 : N − 1do4: Update zj,k = ½xj,k, uj,k−1�T5: Calculate uj,k using uj,k = −Wj,kzj,k6: end for

Algorithm 1. Optimal CCC strategy.

8 Wireless Communications and Mobile Computing

network-induced delay and packet dropout in the design ofthe control strategy. Further research is needed in the con-troller design with other consideration such as transmissionpath planning, energy consumption, and resource allocation.In addition, the dynamic nature of the vehicles results in thetime-varying network topology. In this case, it is difficult tomaintain the system stability, and the controller designremains challenging.

4.2. Security and Privacy Problems. Security issues should becarefully considered in the whole system, including radioaccess, data transmission, and controller design. However,current solutions based on classic secrecy and authenticationmethods do not meet the needs of real-time control applica-tions. Existing centralized security protection protocolsmight not be applicable in the distributed network systemand may bring significant delay in delay-sensitive applica-tions. Therefore, it is a great challenge to achieve both highsecurity and control performance under limited resources.In the connected vehicle scenario, the sensor-based andV2X-enabled automated vehicles rely heavily on electronicequipments with sensitive personal information needed tobe kept secure, such as the position data, social networkinformation, and multimedia records. In this case, some spe-cific procedures need to be taken into consideration to avoidpotential cybersecurity attacks [45]. Therefore, informationsecurity and privacy problems need further investigation.

4.3. Application Considerations. In the whole network, fognot only acts as middle-ware between the cloud and edgebut also provides quick and efficient control services.Depending on the type of service, the fog servers take theresponsibility of deciding the kind of tasks to be processedlocally and that to be offloaded to the cloud [12]. In particu-lar, fog has relatively weaker storage and computationresources than the cloud, so whether a certain application’sdata are stored locally or transmitted to the cloud becomesa problem. In addition, what kind of network structure issuitable for what type of applications, what kind of commu-nication technology will be used between fogs, and howcooperation will be realized between fogs are major researchissues that need further investigation. Different use cases

V2I: vehicles upload sensor data to RSUs

RSU: collect vehicle state information

RSU:Caching vehicle states and control strategies

Regulate transmission path of V2V/V2ICollaborate locally and controlled by RSUC

Uploading

Downloading

Internet

Cloud:Receive and store vehicles states

Globally vehicle data analysisOverall regulation of the traffic flow

Compute and download control information

Vehicles: onboard sensors collect motional data

V2V: forward sensor data to other vehicles

Vehicles: receive control strategy and actuate RSU: compute control strategy based on system states

V2I: RSUs download control strategy to vehicles

V2V / V2Icommunication

Vehicles:Platoons running at a constant speed

Collaborate through V2V communication

Figure 5: The procedure of CCC.

40 1 2 3 5 6 7 8 9 1012

14

16

18

20

22

24

26

h

t

T = 0.2 s, 𝜏 = 0.1 sT = 0.2 s, 𝜏 = 0.3 sT = 0.4 s, 𝜏 = 0.3 s

Figure 6: Headway of the CAVs in different platoons.

0 1 2 3 4 5 6 7 8 9 105

10

15

20

𝜐

t

T = 0.2 s, 𝜏 = 0.1 sT = 0.2 s, 𝜏 = 0.3 sT = 0.4 s, 𝜏 = 0.3 s

Figure 7: Velocity of the CAVs in different platoons.

9Wireless Communications and Mobile Computing

come with diverse requirements. How to customize the net-work to provide corresponding network resources with spe-cific control applications remains challenging.

With the removal of wiring in WNCS, permanentlypower supply is often removed, so additional limitations onthe energy consumption of the system components arise.The tradeoff between control performance and network life-time should be considered thoroughly. More fog nodes meanmore energy consumption, less computing resource result inlonger time delay. How to address the tradeoff in energy con-sumption and computation in order to improve the perfor-mance of green communications also needs further research.

5. Conclusion

In this paper, we propose a novel fog-based distributednetwork architecture toward real-time control applica-tions. Three logical layers, the cloud, distributed fog,and edge smart devices, are considered. We discuss thecritical system variables of communication and controlsystems, including sampling period, message delay, mes-sage dropout, and noisy channel. Fog enriches the net-work resources making real-time control possible. Acase study of CCC is introduced to demonstrate the fea-sibility of the proposed architecture in the field of vehic-ular networks. The optimal control design for the CAV isproposed to realize uniform traffic flow. Simulations indi-cate that the performances are improved with the lowlatency and real-time control capabilities. Finally, we dis-cuss some future research directions and open issues tobe addressed.

Data Availability

The data used to support the findings of this study are avail-able from the corresponding author upon request.

Conflicts of Interest

The authors declare that there is no conflict of interestregarding the publication of this paper.

Acknowledgments

This work was supported by the National Natural ScienceFoundation of China (No. 61701010), the Beijing Social Sci-ence Fund (19GLC051), the Beijing Nova Program of Scienceand Technology (Z191100001119094), and the ScientificResearch Plan of Beijing Municipal Commission of Educa-tion under Grant KM201910005026.

References

[1] O. Bello and S. Zeadally, “Intelligent device-to-device commu-nication in the internet of things,” IEEE Systems Journal,vol. 10, no. 3, pp. 1172–1182, 2016.

[2] L. Zhu, F. R. Yu, B. Ning, and T. Tang, “Cross-layer handoffdesign in mimo-enabled wlans for communication-based traincontrol (cbtc) systems,” IEEE Journal on Selected Areas inCommunications, vol. 30, no. 4, pp. 719–728, 2012.

[3] X. Ge, F. Yang, and Q.-L. Han, “Distributed networked controlsystems: a brief overview,” Information Sciences, vol. 380,pp. 117–131, 2017.

[4] A. W. Al-Dabbagh and T. Chen, “Design considerations forwireless networked control systems,” IEEE Transactions onIndustrial Electronics, vol. 63, no. 9, pp. 5547–5557, 2016.

[5] P. Park, S. Coleri Ergen, C. Fischione, C. Lu, and K. H. Johans-son, “Wireless network design for control systems: a survey,”IEEE Communications Surveys & Tutorials, vol. 20, no. 2,pp. 978–1013, 2018.

[6] G. Zhao, M. A. Imran, Z. Pang, Z. Chen, and L. Li, “Towardreal-time control in future wireless networks:communication-control co-design,” IEEE CommunicationsMagazine, vol. 57, no. 2, pp. 138–144, 2019.

[7] J. Wang, J. Liu, and N. Kato, “Networking and communica-tions in autonomous driving: a survey,” IEEE CommunicationsSurveys & Tutorials, vol. 21, no. 2, pp. 1243–1274, 2019.

[8] Z. Wang, X. Wang, L. Liu, M. Huang, and Y. Zhang, “Decen-tralized feedback control for wireless sensor and actuator net-works with multiple controllers,” International Journal ofMachine Learning and Cybernetics, vol. 8, no. 5, pp. 1471–1483, 2017.

[9] Z. Wang, L. Liu, and Z. Wang, “Stochastic optimal linear con-trol of wireless networked control systems with delays andpacket losses,” IET Control Theory & Applications, vol. 10,no. 7, pp. 742–751, 2016.

[10] L. Zhu, Y. Li, F. R. Yu, B. Ning, T. Tang, and X. Wang, “Cross-layer defense methods for jamming-resistant cbtc systems,”IEEE Transactions on Intelligent Transportation Systems (EarlyAccess), pp. 1–13, 2020.

[11] T. X. Tran, A. Hajisami, P. Pandey, and D. Pompili, “Collabo-rative mobile edge computing in 5G networks: new paradigms,s c e n a r i o s , a n d c h a l l e n g e s , ” 2 0 1 6 ,https://arxiv.org/abs/1612.03184.

[12] M. Aazam, S. Zeadally, and K. A. Harras, “Fog computingarchitecture, evaluation, and future research directions,” IEEECommunications Magazine, vol. 56, no. 5, pp. 46–52, 2018.

0 1 2 3 4 5 6 7 8 9 10−3

−2

−1

0

1

2

3

4

5

T = 0.2 s, 𝜏 = 0.1 sT = 0.2 s, 𝜏 = 0.3 sT = 0.4 s, 𝜏 = 0.3 s

𝜐

t

Figure 8: Acceleration of the CAVs in different platoons.

10 Wireless Communications and Mobile Computing

[13] Y. Liu, J. E. Fieldsend, and G. Min, “A framework of fog com-puting: architecture, challenges, and optimization,” IEEEAccess, vol. 5, pp. 25445–25454, 2017.

[14] M. Li, P. Si, and Y. Zhang, “Delay-tolerant data traffic tosoftware-defined vehicular networks with mobile edge com-puting in smart city,” IEEE Transactions on Vehicular Technol-ogy, vol. 67, no. 10, pp. 9073–9086, 2018.

[15] S.-C. Hung, H. Hsu, S.-Y. Lien, and K.-C. Chen, “Architectureharmonization between cloud radio access networks and fognetworks,” IEEE Access, vol. 3, pp. 3019–3034, 2015.

[16] L. Zhou, L. Yu, S. Du, H. Zhu, and C. Chen, “Achieving differ-entially private location privacy in edge-assistant connectedvehicles,” IEEE Internet of Things Journal, vol. 6, no. 3,pp. 4472–4481, 2019.

[17] K. Bengler, K. Dietmayer, B. Farber, M. Maurer, C. Stiller, andH.Winner, “Three decades of driver assistance systems: reviewand future perspectives,” IEEE Intelligent Transportation Sys-tems Magazine, vol. 6, no. 4, pp. 6–22, 2014.

[18] L. Zhu, Y. He, F. R. Yu, B. Ning, T. Tang, and N. Zhao, “Com-munication-based train control system performance optimiza-tion using deep reinforcement learning,” IEEE Transactions onVehicular Technology, vol. 66, no. 12, pp. 10705–10717, 2017.

[19] H. F. Atlam, R. J. Walters, and G. B. Wills, “Fog computingand the Internet of Things: a review,” Big Data and CognitiveComputing, vol. 2, no. 2, pp. 10–18, 2018.

[20] J. Liu, J. Wan, D. Jia et al., “High-efficiency urban traffic man-agement in context-aware computing and 5g communica-tion,” IEEE Communications Magazine, vol. 55, no. 1,pp. 34–40, 2017.

[21] J. E. Siegel, D. C. Erb, and S. E. Sarma, “A survey of the con-nected vehicle landscape — architectures, enabling technolo-gies, applications, and development areas,” IEEETransactions on Intelligent Transportation Systems, vol. 19,no. 8, pp. 2391–2406, 2018.

[22] Z. Wang, Y. Gao, C. Fang, L. Liu, S. Guo, and P. Li, “Optimalconnected cruise control with arbitrary communication delays,”IEEE Systems Journal, vol. 14, no. 2, pp. 2913–2924, 2020.

[23] X. Wang, Z. Ning, X. Hu et al., “A city-wide real-time trafficmanagement system: enabling crowdsensing in social internetof vehicles,” IEEE Communications Magazine, vol. 56, no. 9,pp. 19–25, 2018.

[24] E. Baccarelli, P. G. V. Naranjo, M. Scarpiniti, M. Shojafar, andJ. H. Abawajy, “Fog of everything: energy-efficient networkedcomputing architectures, research challenges, and a casestudy,” IEEE Access, vol. 5, pp. 9882–9910, 2017.

[25] T. S. Darwish and K. Abu Bakar, “Fog based intelligent trans-portation big data analytics in the internet of vehicles environ-ment: motivations, architecture, challenges, and criticalissues,” IEEE Access, vol. 6, pp. 15679–15701, 2018.

[26] W. Zhang, Z. Zhang, and H.-C. Chao, “Cooperative fog com-puting for dealing with big data in the internet of vehicles:architecture and hierarchical resource management,” IEEECommunications Magazine, vol. 55, no. 12, pp. 60–67, 2017.

[27] C. Bila, F. Sivrikaya, M. A. Khan, and S. Albayrak, “Vehicles ofthe future: a survey of research on safety issues,” IEEE Trans-actions on Intelligent Transportation Systems, vol. 18, no. 5,pp. 1046–1065, 2017.

[28] M. Mazzola, G. Schaaf, A. Stamm, and T. Kurner, “Safety-crit-ical driver assistance over LTE: toward centralized ACC,” IEEETransactions on Vehicular Technology, vol. 65, no. 12,pp. 9471–9478, 2016.

[29] T. Zeng, O. Semiari, W. Saad, andM. Bennis, “Integrated com-munications and control co-design for wireless vehicular pla-toon systems,” in 2018 IEEE International Conference onCommunications (ICC), pp. 1–6, Kansas City, MO, USA, 2018.

[30] L. Zhang and G. Orosz, “Beyond-line-of-sight identification byusing vehicle-to-vehicle communication,” IEEE Transactionson Intelligent Transportation Systems, vol. 19, no. 6,pp. 1962–1972, 2018.

[31] G. Orosz, “Connected cruise control: modelling, delay effects,and nonlinear behaviour,” Vehicle System Dynamics, vol. 54,no. 8, pp. 1147–1176, 2016.

[32] T. Zhang, Y. Zou, X. Zhang, N. Guo, and W. Wang, “A cruisecontrol method for connected vehicle systems considering sidevehicles merging behavior,” IEEE Access, vol. 7, pp. 6922–6936,2019.

[33] D. Jia, K. Lu, J. Wang, X. Zhang, and X. Shen, “A survey onplatoon-based vehicular cyber-physical systems,” IEEECommu-nication Surveys and Tutorials, vol. 18, no. 1, pp. 263–284, 2016.

[34] G. P. Fettweis, “The tactile internet: applications and chal-lenges,” IEEE Vehicular Technology Magazine, vol. 9, no. 1,pp. 64–70, 2014.

[35] J. Wan, D. Zhang, S. Zhao, L. T. Yang, and J. Lloret, “Context-aware vehicular cyber-physical systems with cloud support:architecture, challenges, and solutions,” IEEE Communica-tions Magazine, vol. 52, no. 8, pp. 106–113, 2014.

[36] P. Patil, A. Hakiri, and A. Gokhale, “Cyber foraging and off-loading framework for internet of things,” in 2016 IEEE 40thAnnual Computer Software and Applications Conference(COMPSAC), vol. 1, pp. 359–368, Atlanta, GA, USA, 2016.

[37] C. Yu, B. Lin, P. Guo,W. Zhang, S. Li, and R. He, “Deploymentand dimensioning of fog computing-based internet of vehicleinfrastructure for autonomous driving,” IEEE Internet ofThings Journal, vol. 6, no. 1, pp. 149–160, 2019.

[38] Q. Fan and N. Ansari, “Towards workload balancing in fogcomputing empowered IoT,” IEEE Transactions on NetworkScience and Engineering, vol. 7, no. 1, pp. 253–262, 2020.

[39] S. Zhang, J. Chen, F. Lyu, N. Cheng, W. Shi, and X. Shen, “Vehic-ular communication networks in the automated driving era,”IEEE Communications Magazine, vol. 56, no. 9, pp. 26–32, 2018.

[40] X. Ge, H. Cheng, G. Mao, Y. Yang, and S. Tu, “Vehicular com-munications for 5G cooperative small-cell networks,” IEEETransactions on Vehicular Technology, vol. 65, no. 10,pp. 7882–7894, 2016.

[41] A. Vinel, L. Lan, and N. Lyamin, “Vehicle-to-vehicle commu-nication in C-ACC/platooning scenarios,” IEEE Communica-tions Magazine, vol. 53, no. 8, pp. 192–197, 2015.

[42] J. Guanetti, Y. Kim, and F. Borrelli, “Control of connected andautomated vehicles: state of the art and future challenges,”Annual reviews in control, vol. 45, pp. 18–40, 2018.

[43] M. Huang, W. Gao, and Z.-P. Jiang, “Connected cruise controlwith delayed feedback and disturbance: an adaptive dynamicprogramming approach,” International Journal of AdaptiveControl and Signal Processing, vol. 33, no. 2, pp. 356–370, 2019.

[44] M. Huang, L. Liu, X. Wang, and Z. Wang, “Optimal state feed-back control for wireless networked control systems withdecentralised controllers,” IET Control Theory & Applications,vol. 9, no. 6, pp. 852–862, 2015.

[45] M. Amoozadeh, A. Raghuramu, C. N. Chuah et al., “Securityvulnerabilities of connected vehicle streams and their impacton cooperative driving,” IEEE Communications Magazine,vol. 53, no. 6, pp. 126–132, 2015.

11Wireless Communications and Mobile Computing