Wide-Area Communication and Control: A Cyber-PhysicalPerspective

Aranya Chakrabortty1

1 Introduction

For several decades the traditional mindset for controlling large-scale power systems has been lim-ited to local output feedback control, which means that controllers installed within the operatingregion of any utility company typically use measurements available only from inside that region forfeedback, and, in fact, more commonly only from the vicinity of the controller location. Examplesof such controllers include Automatic Voltage Regulators (AVR), Power System Stabilizers (PSS),Automatic Generation Control (AGC), FACTS control, HVDC, and so on. However, the US North-east blackout of 2003, followed by the timely emergence of sophisticated GPS-synchronized digitalinstrumentation technologies such as Wide-Area Measurement Systems (WAMS) led utility ownersto understand how the interconnected nature of the grid topology essentially couples their controllerperformance with that of others, and thereby forced them to look beyond this myopic approach oflocal feedback and instead use wide-area measurement feedback [1]. Over the past few years severalresearchers have started investigating such data-driven wide-area control designs using H∞ control[2, 3, 4], LMIs and conic programming [5], wide-area protection [6], model reduction and controlinversion [7, 8], adaptive control [9], LQR-based optimal control [10, 11, 12, 13], etc., complimentedwith insightful case studies of controller implementation for various real power systems such asthe US west coast grid [14, 15], Hydro Quebec [16], Nordic system [17, 18], and power systemsin China [19], Australia [20], and Mexico [21]. A tutorial on the ongoing practices for wide-areacontrol has recently been presented in [22], while cyber-physical implementation architectures forrealizing these controls have been proposed in [23, 24].

One of the biggest roadblocks for implementing wide-area control in a practical grid, however,is that the current power grid IT infrastructure is rigid and low capacity as it is mostly based ona closed-mission specific architecture. The current push to adopt the existing TCP/IP based openInternet and high-performance computing technologies such as the NASPInet [25] would not beenough to meet the requirement of collecting and processing very large volumes of real-time dataproduced by such thousands of PMUs. Secondly, the impact of the unreliable and insecure commu-nication and computation infrastructure, especially long delays and packet loss uncertainties overwide-area networks, on the development of new WAMS applications is not well understood. Forexample, as shown in [26]-[28] uncontrolled delays in a network can easily destabilize distributedestimation algorithms for wide-area oscillation monitoring using PMU data from geographically dis-persed locations. Finally, and most importantly, very little studies have been conducted to leveragethe emerging IT technologies, such as cloud computing, software defined networking (SDN), andnetwork function virtualization (NFV), to accelerate the development of WAMS [29]. Anothermajor challenge is privacy of PMU data as utility companies are often shy in sharing data froma large number of observable points within their operating regions with other companies. Equallyimportant is cyber-security of the data as even the slightest tampering of Synchrophasors, whetherthrough denial-of-service attacks or data manipulation attacks, can cause catastrophical instabili-ties in the grid. What we need is a cyber-physical architecture that explicitly brings out potentialsolutions to all of these concerns, that is how data from multitudes of geographically dispersedPMUs can be shared across a large grid via a secure communicate medium for successful executionof critical transmission system operations, how the various binding factors in this distributed com-munication system can pose bottlenecks, and, how these bottlenecks can be mitigated to guaranteethe stability and performance of the grid.

Motivated by these challenges, in this tutorial we review the current state-of-the-art practicefor wide-area communication and control from a cyber-physical system (CPS) viewpoint, the ways

1Electrical & Computer Engineering, North Carolina State University, Email: achakra2@ncsu.edu

1

these practices pose limitations against optimal grid performance, how modern communicationtechnologies such as SDN, NFV, and cloud computing can be used to overcome these limitations,and how new ideas of distributed control and optimization need to be integrated with variousoperational protocols and wide ranges of uncertainties of these wide-area networks to ensure thatthe closed-loop grid operates in a stable, reliable, robust, and efficient way. We also discuss theneed for advanced modeling, simulation, and control of wide-area communication to ensure cyber-security and resilience of wide-area controllers.

The remainder of the chapter is organized as follows. In Section II we recall standard differential-algebraic models of power system networks followed by wide-area control designs in Section III.Section IV presents the potential cyber-physical architectures for implementing these wide-area con-trollers over a distributed communication network. Section V highlights research directions on howthese controllers can be made aware of the various operational uncertainties in the communicationnetwork such as delays. Section VI summarizes the potential benefits of modern communicationtechnology such as SDN, NFV and cloud computing in realizing the envisioned CPS architecture.Section VII highlights the importance for making wide-area controllers secure against cyber-attacks.Section VIII presents the current challenges facing simulation of extreme-scale wide-area control,while Section IX describes the importance of hardware-in-loop CPS testbeds that can be used fortesting, verification and validation of such emulated control loops before they are deployed in thefield. Section X concludes the chapter.

2 Power System Models

Consider a power system network with n synchronous generators. Each generator is modeled bya flux-decay model assuming that the time constants of the d- and q-axis flux are fast enough toneglect their dynamics, that the rotor frequency is around the normalized constant synchronousspeed, and that the amortisseur effects are negligible. The model of the ith generator can be thenwritten as [30]:

δi = ωi − ωs (1)

Miωi = Pmi − (ViIqi cos(θi − δi) + ViIdi sin(δi − θi))− di(ωi − ωs) (2)

Tqi E′qi = −E′qi + (xdi − x′di)Idi + Efdi (3)

Tdi E′di = −E′di + (xqi − x′qi)Iqi (4)

TAi Efdi = −Efdi +KAi(Vref,i − Vi) + ui(t). (5)

for i = 1, . . . , n. Equations (1)-(2) are referred to as the swing equations while (3)-(5) as theexcitation equations. The states δi, ωi, E

′qi, E

′di, and Efdi respectively denote the generator phase

angle (radians), rotor velocity, the quadrature-axis internal emf, the direct-axis internal emf, andthe field excitation voltage of the ith generator. The voltage at the generator terminal bus isdenoted in the polar representation as Vi(t) = Vi(t)∠θi(t). Vref,i is the constant setpoint for Vi.The generator current in complex phasor form is written as Idi+jIqi = Ii∠φi. ωs is the synchronousfrequency, which is equal to 120π rad/sec for a 60-Hz power system. Mi is the generator inertia,di is the generator damping, and Pmi is the mechanical power input from the ith turbine, all ofwhich are considered to be constant. Tdi, Tqi, and TAi are the excitation time constants; KAi isthe constant voltage regulator gain; xdi, x

′di, xqi and x′qi are the direct-axis and quadrature-axis

salient reactances and transient reactances, respectively. All variables, except for the phase angles(radians), are expressed in per unit. Equations (1)-(5) can be written in a compact form as

xi(t) = g(xi(t), zi(t), ui(t), αi) (6)

where xi = [δi ωi E′qi E

′di Efdi]

′ ∈ R5 denotes the vector of state variables, zi = [Vi θi Idi Iqi]′ ∈ R4

denotes the vector of algebraic variables, ui ∈ R is the control input, and αi is the vector of theconstant parameters Pmi, ωs, di, Tqi, Tdi, TAi, Mi, KAi, Vref,i, xdi, xqi, x

′di, and x′qi, all of which

are assumed to be known. The definition of the nonlinear function g(·) follows from (1)-(5).

2

The model (6) is a completely decentralized model since it is driven by variables belonging tothe ith generator only. It is, however, not a state-space model as it contains the auxiliary variableszi. The states xi can be estimated for this model in a completely decentralized way if one has accessto zi(t) at every instant of time. This can be assured by placing PMUs within each utility area suchthat the generator buses inside that area becomes geometrically observable, measuring the voltageand currents at the PMU buses, and thereafter computing the generator bus voltage Vi∠θi andcurrent Ii∠φi (or equivalently, Idi and Iqi) from those measurements. As the PMU measurementswill be corrupted with noise, the estimated values of zi(t) will not be perfect. They should ratherbe denoted as zi(t) = zi(t) + ni(t), where ni(t) is a Gaussian noise. An unscented Kalman filter(UKF) is next designed as

˙xi(t) = g(xi(t), zi(t), ui(t), αi), xi(0) = xi0 (7)

producing the state estimates xi(t) for the ith generator at any instant of time t. For details of theconstruction of this UKF, please see [31]. The estimator can be installed directly at the generationsite to minimize the communication of signals, and made to run continuously before and after anydisturbance.

The network equations that couple (xi, zi) of the ith generator in (6) to the rest of the networkcan be written as

0 = Idi(t)Vi(t) sin(δi(t)− θi(t)) + Iqi(t)Vi(t) cos(δi(t)− θi(t)) + PLi(t)

−m∑j=1

Vi(t)Vj(t)(Gij cos(θij(t)) +Bij sin(θij(t))) (8)

0 = Idi(t)Vi(t) cos(δi(t)− θi(t))− Iqi(t)Vi(t) sin(δi(t)− θi(t)) +QLi(t)

−m∑j=1

Vi(t)Vj(t)(Gij cos(θij(t))−Bij sin(θij(t))), (9)

where m is the total number of buses in the network. Here, θij = θi−θj , PLi and QLi are the activeand reactive power load demand at bus i, and Gij and Bij are the conductance and susceptanceof the line joining buses i and j. As shown in [30], the variables zi(t) in (6) can be eliminatedusing (8)-(9) by a process called Kron reduction. The resulting dynamic model is linearized abouta given operating point, and the small-signal model for the power system is written as

x(t) = Ac x(t) +Bc u(t) (10)

y(t) := ∆ω(t) = C x(t). (11)

In this model x(t) ∈ R5n and u(t) ∈ Rn now represent the small-signal deviation of the actualstates and excitation inputs of the n generators from their pre-disturbance equilibrium. We definey(t) ∈ Rn as a performance variable, based on which the closed-loop performance of the overallsystem as well as the saturation limits on u(t) can be judged. Ideally, this can be chosen as thevector of all electro-mechanical states. For simplicity, y if often chosen as the vector of the small-signal generator frequency ∆ω(t) as frequency is the most effective electro-mechanical state forevaluating damping. The input u(t) is commonly used for designing feedback controllers such asPower System Stabilizers (PSS), which takes local feedback from the generator speed, and passesit through a lead-lag controller for producing damping effects on the oscillations in phase angle andfrequency. PSSs, however, are most effective in adding damping to the fast oscillation modes in thesystem, and perform poorly in adding damping to the slow or inter-area oscillation modes [22]. Thegoal is to design a supplementary controller u(t) for the model (10)-(11) on top of the local PSS byusing state-feedback from either all or selected sets of other generators. If the state vector of everygenerator is fed back to every other generator, then the underlying communication network musthave a complete graph, i.e. an all-to-all connection topology. If the state vectors of only a selectedfew generators are fed back to the controllers of some other generators, then the communicationnetwork may have a sparse structure. In either case, since long-distance data transfer is involved,we refer to this controller as a wide-area controller.

3

3 Controller Design

Several papers such as [10, 11, 32] have posed the wide-area control problem as a constrainedoptimal control problem of the form:

minK

J =1

2

∫ ∞0

(xT (t)Qx(t) + uT (t)Ru(t))dt

s.t., x(t) = Ax(t) +Bu(t)

u(t) = Kx(t), K ∈ S, Q > 0, R ≥ 0, (12)

where the state estimate x(t) follows from (7). The choice of the objective function J depends onthe goal for wide-area control. For power oscillation damping, this function is often simply justchosen as the total energy contained in the states and inputs as in (12). For wide-area voltagecontrol, it can be chosen as the setpoint regulation error for the voltages at desired buses [33],while for wide-area protection it can be chosen as the total amount of time taken to trigger relaysso that fault currents don’t exceed their maximum values [6]. The set S is a structure set thatdetermines the topology of the underlying communication network. In the current state-of-the-art,most synchronous generators operate under a completely decentralized feedback from its own speedmeasurement only. Thus in today’s power grid the structure set S is reflected in K as

K =

K1 0 0 . . . 00 K2 0 . . . 0

...0 0 0 . . . Kn

, Ki =[

0 1 0 0 0]

(13)

where Ki is the PSS controller for the ith generator, i = 1, . . . , n, whose state vector follows from(6). However, as pointed out before, decentralized feedback can damp only the high-frequencyoscillations in the power flows. Their impact on inter-area oscillations that typically fall in the rangeof 0.1 to 2 Hz is usually small [22]. Inter-area oscillations arise due to various coherent clusters inthe power network oscillating against each other following a disturbance. If left undamped theycan result in catastrophic failures in the grid. In fact, both the 1996 blackout on the west coastand the 2003 blackout on the eastern grid of the United States were largely caused because of thelack of communication between generators resulting from the decentralized structure of K in (13).Triggered by the outcomes of these events, over the past few years utility companies have graduallystarted moving away from the structure in (13) to a slightly more global structure as

K =

K1 0 0 . . . 00 K2 0 . . . 0

...0 0 0 . . . Kr

(14)

where Ki ∈ Rni×5ni , ni being the number of generators within the operating region of the ith utilitycompany, and r being the toal number of such companies. All the elements of the block matrix Ki

may be non-zero, meaning that the generators inside an area communicate their state informationwith each other to compute their control signals. The resulting controller K is block-diagonal,and, therefore, a better wide-area controller than (13) since here at least the local generatorsare allowed to interact. An ideal wide-area controller, however, would be one whose off-block-diagonal entries are non-zero as well, meaning that generators across the operating areas of differentcompanies are allowed to exchange state information. In the worst-case, K can be a standardLQG controller, and, therefore, a dense matrix with every element non-zero, which means thatthe communication topology is a complete graph. In reality, however, all-to-all communicationcan be quite expensive due to the cost of renting communication links in the cloud [34], if notunnecessary. Papers such as [11, 32, 35] have proposed various graph sparsification algorithms

4

based on l1-optimization to develop controllers that require far less number of communication linkswithout losing any significant closed-loop performance. Papers such as [10], on the other hand,has proposed the use of modal participation factors between generator states and the inter-areaoscillation modes to promote network sparsity in a more structured way. Papers such as [36, 37, 38,39] have proposed various projection and decomposition-based control designs by which a significantportion of the communication network admits a broadcast-type architecture instead of peer-to-peerconnectivity, thereby saving on the number of links. The main challenge of implementing thesetypes of wide-area controllers, however, is the fact that utilities are still shy in terms of sharingtheir PMU measurements and state information with other utilities due to which they prefer tostick to the block-decentralized structure of (14). Besides these network-centric approaches, othercontrol designs based on more traditional approaches such as adaptive control [7], robust control[3], and hybrid control [9] have also been proposed for wide-area oscillation damping in presence ofvarious model and operational uncertainties.

Several questions arise naturally from the design problem stated in (12). For example,

1. in practice, given the size and complexity of any typical grid, the exact values of the matricesA and B are highly unlike to be known perfectly. Moreover, the entries of these matrices canchange from one event to another. Therefore, just one dedicated model created from a one-time system identification may not be suitable. Thus, one pertinent question is - how can oneextend the model-based design in (14) to a more measurement-based approach where onlinePMU measurements from different parts of the grid can be used to estimate the small-signalmodel of the grid, preferably in a recursive way, based on which the control signals can beupdated accordingly? This estimation should also be preferably carried out in a distributedway over the sparse communication topology generated from the controller. While traditionalnotions of adaptive control can be highly useful here, the speed of estimation may suffer ifthe entire model needs to be estimated. Newer ideas from reinforcement learning, adaptivedynamic programming, and Q-learning can be useful alternatives in this case as they tend tooptimize the objective function directly by bypassing the need for estimating the model.

2. Another question is - whether it is necessary to base the design of the wide-area controlleru(t) on the entire state-space model (14), or does it suffice to design it using a reduced-ordermodel only? For example, recent papers such as [36] have shown that one can make use ofsingular perturbation based model reduction for designing u(t) in a highly scalable way forconsensus networks using a hierarchical control architecture. This is especially true if thegrid exhibits spatial clustering of generators due to coherency. Therefore, following [36] oneidea can be to estimate a reduced-order network model, where every network node representsan equivalent generator representing an entire cluster, design an aggregated control input foreach equivalent generator, and then broadcast the respective inputs to every generator insidethat cluster for actuation. Open questions for these types of designs would include proofsfor stability, sensitivity of closed-loop performance to estimation and model reduction errors,derivation of numerical bounds for closed-loop performance as a function of the granularityof model reduction, and so on.

3. The third question is - how can one robustify the wide-area controller (14) against the typicaluncertainties in both the physical layer and the communication layer? Uncertainties in thephysical grid model, for example, can easily arise from the lack of knowledge of the mostupdated model parameters in the set αi in (6). With increase in renewable penetration andthe associated intermittencies in generation profiles, operational uncertainties are graduallyincreasing in today’s grid models [40]. Similarly, significant uncertanties will exist in thecyber-layer models as well including uncertainties due to delays, congestion, queuing, routing,data loss, synchronization loss, quantization, etc. A natural concern, therefore, is - how canthe design in (14) be made aware of these uncertainties so as to optimize the closed-loopperformance of variables for the entire CPS grid?

5

Many other CPS-centric design and implementation questions related to scalability, centralizedversus distributed implementation, speed of computation, big data analytics in the loop, and co-dependence of (14) on other state estimation and control loops (for example, those using SCADA)can also arise. All of these questions deserve dedicated attention from researchers with backgroundsin control theory, power systems, signal processing, machine learning, computer science, commu-nication engineering, economics, and information theory. In the following sections, we highlightseveral of these CPS research challenges for wide-area control .

4 Cyber-Physical Implementation of Wide-Area Controllers

Once designed, wide-area controllers such as (12) need to be implemented in a distributed wayby transmitting outputs measured by PMUs over hundreds of miles across the grid to designatedcontrollers at the generation sites. Depending on the number of PMUs, the rate of data transfercan easily become as high as hundreds of Terabytes per second. The time-scale associated withtaking these control actions can be in fractions of seconds. Therefore, controlling latencies anddata quality, and maintaining high reliability of communication are extremely important for theseapplications. The media used for wide-area communications typically use longer-range, high-powerradios or Ethernet IP-based solutions. Common options include microwave and 900 MHz radiosolutions, as well as T1 lines, digital subscriber lines (DSL), broadband connections, fiber net-works, and Ethernet radio. The North American Synchrophasor Initiative (NASPI), which is acollaborative effort between the U.S. Department of Energy (DOE), the North American Elec-tric Reliability Corporation (NERC), and various electric utilities, vendors, consultants, federaland private researchers and academics is currently developing an industrial grade, secure, stan-dardized, distributed, and scalable data communications infrastructure called the NASPI-net tosupport Synchrophasor applications in North America. NASPI-net is based on an IP Multicastsubscription-based model. In this model the network routing elements are responsible for handlingthe subscription requests from potential PMU data receivers as well as the actual optimal pathcomputation, optimization, and re-computation and rerouting when network failures happen. Theschematic diagram of the NASPI-net is shown in Fig. 1. An excellent survey of NASPI-net can befound in [41].

Figure 1: Architecture of NASPInet [25]

6

The implementation of the state estimator (7) and the controller (12) in the NASPI-net maybe done in the following way. PMU measurements yi(t) from each utility company i, as shown inFig. 1, are first gathered in a local phasor data concentrator (PDC) at the local substation using alocal-area network, assuming that the PDC is located at a reasonably close geographical distancefrom all the PMUs in that service area. The generator bus variables zi(t) are computed fromyi(t) using Kirchoff’s laws at this PDC, and the decentralized state estimator (7) may be run togenerate xi(t). This state estimate then enters the NASPI-net data bus through a phasor gateway(PGW). The PGW has high levels of security encryption so as to prevent the flow of any maliciousdata between the local PDC and the data bus. The state estimates are then communicated viastandard Internet protocols to the designated controllers of other generators so that they can beused for computing the control signal uj(t) for the jth generator, which finally gets actuated usingthe excitation control system located at this generator.

A slightly different cyber-physical architecture for implementing these types of controllers hasrecently been proposed in [23]. This architecture, shown in Fig. 2, is very similar to the NASPI-net except that here the estimation of the states, and the computation of the control signals arenot done at the PDC or at the generator, but entirely inside a cloud computing network. Forexample, PMU measurements from inside a service area are still gathered at its local PDC, butthis PDC does not generate the state estimate. It rather ensures that all the measurements areproperly synchronized with respect to each other, that their measurement noise is within acceptablelimits, and that the measurements do not consist of any bad data due to GPS errors, or errors inthe phase-locked loop inside the PMUs. The PDC then relays all the measurements to a localservice-based private cloud owned by the utility company, wherein they are gathered in a virtualcomputer or virtual machine (VM), created on the fly using the available computation resourcesin the cloud. The measurements are digitally represented as a periodic stream of data points. Thegeographical location of the VMs can be close to that of the generators in that area so that thelatency from PDC-to-cloud communication is small. The state estimator (7) can then be employedin this VM. The local clouds themselves are connected to each other through an Internet of clouds,as shown in the figure. This means that the VMs in every local cloud, depending on the sparsitystructure demanded by the controller u(t) = Kx(t), are connected to other remote VMs throughan advanced, secure, third-party wide-area communication network such as SDN, an example ofwhich can be Internet2. The VMs can then exchange their estimated states xi(t), and computetheir control signals ui(t) through this network in a completely distributed way.

In reality, depending on the number of PMUs inside any service territory, the local cloud of autility company may have multiple VMs, each of which receive a designated chunk of the local PMUmeasurements from the local PDC, as shown in the figure. These VMs communicate with theirneighboring VMs inside the cloud as well as to those across other clouds exchanging PMU data, andcomputing control signals via pre-determined sparse feedback control laws such as (14). The controlsignals are, thereafter, transmitted back from the local cloud to the actuators of the correspondinggenerators inside their respective service regions. The resulting system is referred to as a cloud-in-the-loop wide-area control system. End-to-end delay specifications for wide-area control usingthis type of wide-area communication have been presented in [24]. Two main advantages of thisarchitecture compared to NASPI-net is that all the computations are done inside a third-partycloud, thereby preserving the privacy of PMU data from direct exchange between the utilities,and that the communication between the clouds no longer has to be based on standard Internetprotocols but can rather use more advanced networking technologies such as SDN with additionallayers of network controllability.

For either architecture, the shared communication network will have delays arising from routingand queuing, besides the usual transmission delays due to the geographical distance between theVMs. Three classes of delays may be defined, namely - small delays τs that arise due to messagequeuing inside any virtual machine, thereby delaying the availability of the measured state variableassigned to that machine for computing the corresponding control signal; medium delays τm thatarise due to communication between two virtual machines that are part of the same local cloud,

7

`

Area 1

Area 2 Area 3

Area 4

Power

Transmission

Network

Local

PDC 1

)(3 tx

)(2 tx

)(1 tx

)(4 tx

)(5 tx

)(6 tx

)(7 tx

)(9 tx

Local

PDC 2

Local

PDC 3

)(8 tx

)(10 tx

Internet of Clouds

VM

VM

VM

Local cloud in Area 1

VM

VM

VM

Local cloud in Area 4

Local cloud in

Area 2

)(1 tx

)(2 tx

)(3 tx

VM VM

)(4 tx )(5 tx

Local cloud in

Area 3

VM VM

)(6 tx)(7 tx

Local

PDC 4

)(8 tx

)(9 tx

)(10 tx

Figure 2: Wide-area control using a cloud-in-the loop architecture [23]

and large delays τl that arise due to communication between any two virtual machine that arelocated in two different local clouds. In the recent papers [26, 27] the stochastic end-to-end delayexperienced by messages in an Internet-based wide-area communication link was modeled usingthree components:

1. the minimum deterministic delay, say denoted by m,

2. the Internet traffic delay with Probability Density Function (PDF), say denoted by φ1, and

3. the router processing delay with PDF, say denoted by φ2.

The PDF of the total delay at any time t was written in terms of these three components as

φ(t) = p φ2(t) + (1− p)∫ t

0φ2(u)φ1(t− u)du, t ≥ 0, (15)

Here, p is the probability of the open-period of the path with no Internet traffic. The routerprocessing delay can be approximated by a Gaussian density function

φ2(t) =1

σ√

2πe−

(t−µ)2

2σ2 , (16)

8

where µ > m. The Internet traffic delay is modeled by an alternating renewal process with expo-nential closure period when the Internet traffic is on. The PDF of this delay is given by

φ1(t) = λe−λt, (17)

where λ−1 models the mean length of the closure period. The cumulative distribution function(CDF) of this delay model can then be derived as

P (t) =

∫ t

−∞φ(s)ds =

1

2[erf(

µ√2σ

) + erf(t− µ√

2σ)]+

(p− 1)

2e(

12λ2σ2+µλ)e−λt[erf(

λσ2 + µ√2σ

) + erf(t− λσ2 − µ√

2σ)]. (18)

Random numbers arising from this CDF can be used for simulating delays of the form τs, τm andτl, as defined above. The challenge, however, is to translate these single-path models to multi-path, multi-hop, shared network models where background traffic due to other parallelly runningapplications over shared resources may pose serious limitations in latencies. Recent references suchas [42, 43] have provided interesting theoretical tools such as Markov jump process, Poisson process,multi-fractal models, and Gaussian fractional sum-difference models for modeling delay, packetloss, queuing, routing, load balancing, and traffic patterns in such multi-channel communicationnetworks.

5 Cyber-Physical Codesigns

The next question is - how can the controller in (12) be co-designed with the information aboutτs, τm and τl? The conventional approach is to hold the controller update until all the messagesarrive at the end of the cycle. However, this approach may result in poor closed-loop performance,especially for damping of the inter-area oscillations. In recent literature several researchers havelooked into delay mitigation in wide-area control loops [44]-[47], including the seminal work of Vittaland co-authors in [3] whereH∞ controllers were designed for redundancy and delay insensitivity. Allof these designs are, however, based on worst-case delays, which makes the controller unnecessarilyrestrictive, and may degrade closed-loop performance. In [23, 48] this problem was addressed byproposing a delay-aware wide-area controller, where the feedback gain matrix K was made anexplicit function of delays using ideas from arbitrated network control theory [49].

For example, considering the three delays τs, τm and τl defined in the previous section, onemay update the control input at any VM as new information arrives instead of waiting till theend of the cycle. If tweaking the protocols is difficult, then another alternative strategy will be toestimate the upper bounds for the delays using real-time calculus [50, 51]. The approach is referredto as arbitration, which is an emerging topic of interest in network control systems. Based on theexecution of the three protocols, one may define two modes for the delays - namely, nominal andoverrun. If the messages meet their intended deadlines, they are denoted as nominal. If they donot arrive by that deadline, they are referred to as overruns. Defining two parameters τth1 and τth2such that τth1 ≤ τth2, one may define nominal, skip, and abort cases for computing the wide-areacontrol signal as:

• If the message has a delay less than τth1, we consider the message as the nominal message ofthe system and no overrun strategy will be activated.

• If the message suffers a delay greater than τth1 and less than τth2, the message will be com-puted; however, the computations of the following message will be skipped.

• If the message suffers a delay greater than τth2, the computations of the message will beaborted, and the message is dropped. This strategy is motivated by assuming that themessages will be significantly delayed, and are no longer useful.

9

Accordingly, a feasible way to formulate the execution rules can be: (1) if τth1 ≤ τth2 ≤ τwcet, whereτwcet is the worst-case delay in the network, both abort and skip can happen, (2) Abort Only: ifτth1 = τth2 < τwcet, the message will be dropped if they miss their first deadline, and (3) SkipOnly: if τth1 ≤ τwcet and τth2 ≥ τwcet. One idea will be to set τth2 = τwcet to develop a constructivestrategy to determine τth1. This step can be a significant improvement over conventional networkcontrol designs in terms of both closed-loop performance and resource utilization.

Figure 3: Discrete-time delays τ1 < τ2 < τ3. The vector indicated next to the arrow represents thestate used to compute input uij .

To justify this approach, we cite an example from the control design presented recently in [23].The sampling interval h of the PMUs between two consecutive control updates was broken downinto three smaller intervals at which the inputs were updated as new measurements arrive, as shownin Figure 3. If any state is unavailable, it is replaced by its predicted value. For the first generator,for example, the input u1(k) is further divided to

[u11(k) u12(k) u13(k)

], where uij(k) denotes

the input of the ith generator adjusted using the measurements of jth generator. Repeating thesame logic for all generators, it was shown in [23] that the discrete-time model of the system canbe written as

x(k + 1) = Ax(k) +n∑i=1

m(i)∑j=1

Bij1uij(k) +

n∑i=1

Bii2uik(i)(k − 1), (19)

where m(i) shows the number of times that the inputs are updated in each generator, and k(i) isthe index of the largest delay, or equivalently as

x(k + 1) = Ax(k) +B2u(k − 1) +B1u(k) (20)

In other words, the excitation controller needs feedback from the current state samples as well asthe past input samples to stabilize the closed-loop swing dynamics with communication delays.An open problem is to derive the equivalent expressions of u(k) for various typical protocols usedfor wide-area communication, develop tractable and scalable ways for tuning the control gains toguarantee closed-loop stability and performance while promoting sparsity in the network structureas indicated by the set S in (14), and, most importantly, validating these communication andcontrol laws using realistic cyber-physical testbeds.

6 SDN, NFV, and Cloud Computing

As mentioned earlier, typically the Internet cannot provide the required latency and packet lossperformance for grid operation under high data-rates. Moreover, the network performance is highlyrandom, and therefore, difficult to model accurately. That is why the cloud-based WAMS archi-tecture proposed in Fig. 2 is currently garnering a lot of attention from power system engineers.However, limited studies have been conducted so far to leverage all possible benefits of cloud

10

computing, software-defined networking (SDN), and network function virtualization (NFV), to ac-celerate this development [29]. With the recent revolution in networking technology, these newcommunication mechanisms can open up several degrees of freedom in programmability and vir-tualization for computation and communication platforms. However, customized SDN control andprotocols, and sufficient experimental validation using realistic testbeds are still missing in almostall wide-area control applications.

As mentioned in the previous section, latency and data loss rate are important factors in theperformance of all wide-area control and protection applications. Software such as RTDMS (real-time dynamics monitoring system), PGDA (phasor grid data analyzer), and GridSim are usedfor online oscillation monitoring using Synchrophasors. A list of related open-source software canbe found in [52]. These simulation engines need to be integrated into executable actions so thatresults from the monitoring algorithms can be exported to a custom SQL database that can beset to trigger alerts or alarms whenever damping levels of oscillatory modes fall below pre-specifiedthresholds. These alarm signals need to be communicated to the operator through a reliablecommunication network so that the operator can take manual actions to bring the damping backto acceptable levels [53]. In recent years, simulation platforms such as ExoGENI-WAMS [54] havebeen developed to emulate such communication platforms. The computation and communicationplanes are entirely shifted away from the physical infrastructure, similar to the architecture proposedin Fig. 2. Another example of a CPS simulator is GridSim [55]. The data delivery componentin this simulation platform, also referred to as GridStat, is a publish-subscribe middleware, whichallows for encrypted multicast delivery of data. GridStat is designed to meet the requirements ofemerging control and protection applications that require data delivery latencies on the order of 10to 20 ms over hundreds of miles with extremely high availability.

Similar to GridStat, the VMs in any cloud-in-the-loop CPS simulator may consist of two com-munication planes, namely a data plane and a management plane. The data plane is a collection offorwarding engines (FEs) designed to quickly route received messages on to the next VMs. The FEsare entirely dedicated to delivering messages from publishers to subscribers. Routing configurationinformation is delivered to the FEs from the management plane. The forwarding latency throughan FE implemented in software is generally on the order of 100 micro-seconds, and with networkprocessor hardware it is less than 10 micro-seconds. The management plane, on the other hand, isa set of controllers, called QoS brokers, that manage the FEs of the data plane for every VM. TheQuality-of-Service (QoS) brokers can be organized in a hierarchy to reflect the natural hierarchyin the physical infrastructure of the grid model. When a subscriber wishes to receive data from apublisher, it communicates with a QoS broker that designs a route for the data, and delivers therouting information to the relevant FEs and VMs, creating the subscription.

However, simulation platforms such as GridStat are just starting points for research. Muchmore advanced cloud computing and SDN protocols as well as their emulation software are need tobe developed for the future grid. One outstanding challenge is to develop a reliable communicationsoftware that will enable timely delivery, high reliability and secure networking for these emulators.Timeliness of message requires guaranteed upper bounds on end-to-end latencies of packets. Legacynetworking devices do not provide such guarantees, neither for commodity Internet connections norfor contemporary proprietary IP-based networks that power providers may operate on. Moreover,direct communication lacks re-routing capabilities under real-time constraints, and resorts to his-toric data when communication links fail. A promising solution for both timeliness and link-failurescan be the idea of Distributed Hash Tables (DHT), which was recently introduced for wide-areacontrol applications in [56].

7 Cyber-Security

Another critical CPS challenge for both wide-area monitoring and control is the issue of cyber-security. While security is an universally growing concern for applications at every layer of the grid,ranging from distribution grids to power markets, the challenge for WAMS is especially critical

11

1

2

13

144 5 6

8

7 9 10

11 123

PDC 1

PDC 2 PDC 3

PDC 4

) ,( 11 cH

PMU

PMU

PMU PMU PMU

PMU

Area 1

Area 2

Area 3

Area 4

) ,( 22 cH ) ,( 33 cH

) ,( 44 cH

3y2y

6y 8y

11y 13y

Central PDC

4

4 321

kkkk

k aaaaz

+++=

kkaa 11 D+

ka2 k

a3

ka4

Power System Network

kz

kz

kz

kzCompromised

PDC

Manipulated

Information

Figure 4: Cyber-attacks on wide-area monitoring and control [59]

since the stakes here are much higher. The integration of cyber components with the physicalgrid introduces new entry points for malicious attackers. These points are remotely accessible atrelatively low risks to attackers compared to physical intrusions or attacks on substations. They canbe used to mount coordinated attacks to cause severe damages to the grid, resulting in catastrophicblackouts with billions of dollars worth of economic loss. One eye-opening example of such an attackwas the Stuxnet. Attacks can be originated on the cyber layer as well to trigger cascading eventsleading to damages on physical facilities, leading to major outages. While mathematical modelshave been developed to model electrical faults and device failures, there are far less reliable waysof modeling and simulating realistic scenarios for different types of cyber attacks happening in apower grid. Several universities and national laboratories have only recently started developingsimulation testbeds to emulate these vulnerability scenarios. Research demonstration events suchas Cyber Security Awareness Month, which is a student-run cyber security event in the US, hasbeen introduced by the Department of Homeland Security [58]. Many organizations are working onthe development of smart grid security requirements including the NERC Critical InfrastructureProtection (NERC-CIP) plan, the National Infrastructure Protection Plan (NIPP), and NationalInstitute of Standards and Technology (NIST). The goal is for power system operators to workwith these standards organizations to develop simulation software that can model, detect, localize,and mitigate cyber vulnerabilities in the grid as quickly as possible. A more detailed overview ofthese methods will be provided in the forthcoming chapter by Wang.

Given the size, complexity, and enormous number of devices present in a typical grid, developingone unique solution for securing the grid from cyber-attacks is probably impossible. Even the attackspace can easily become huge, ranging from denial-of-service (DoS) attack on communication links,to disabling of physical assets, to data corruption, to GPS spoofing, to eavesdropping. Everyphysical application, whether it be state estimation, oscillation monitoring, automatic generationcontrol (AGC), or wide-area damping control, would need its own way of dealing with each ofthese attacks. One common solution to make all of these applications more resilient is to switchfrom centralized to distributed implementation, as alluded to in the previous sections. Although

12

distributed communication opens up more points for an attacker to enter through, it also providesresilience through redundancy. For example, the distributed architecture shown in Fig. 4 wasrecently used in [59] for wide-area oscillation estimation purposes. The idea was to distributed theestimation algorithm among multiple estimators (or VMs, if implemented using the cloud-in-the-loop architecture of Fig. 2) that exchange estimates recursively through a supervisor. If any of theestimators are disabled by a cyber-attack, the other estimators or the supervisor can be quicklyassigned to take up the role of the lost estimator. A similar concept can also be used for the wide-area control design in (14). For example, a LQR wide-area controller was designed in [10] using asparse communication graph G. The nonlinear model of the IEEE 39-bus power system model wassimulated with this sparse state-feedback controller using Power System Toolbox (PST) in Matlab.A fault was induced at t = 0, and the small-signal speed deviations of the synchronous generatorswere recorded, as shown in Fig. 5. The closed-loop system behavior is observed to be stable. Att = 10 seconds, a DoS attack induced on the communication link connecting generators 1 and 8,which means that these two generators are no longer capable of exchanging state information fortheir control actions. Instability is noticed immediately, with the frequency swings diverging withincreasing amplitude at a frequency of roughly 0.05 Hz. This can be seen in Fig. 5 onwards fromt = 10 sec. At t = 60 sec, the communication links connecting generators (2, 6) and (3, 6) areadded, and the corresponding control gains are recomputed. The system is seen to regain stability,indicating that the attack has been successfully mitigated. The frequency deviations are all seento converge to zero over time.

Figure 5: Time-response of generator speeds before, during, and after DoS attack

This example shows the importance of developing formal recovery rules of wide-area controllersin face of attacks. It also highlights the need for developing effective simulation packages forinvestigating the impact of attack scenarios. Simulations, in fact, can reveal the most importantpairs of generators that must communicate to maintain stable operating condition before and afteran attack, and also the lesser important pairs that, either due to large geographical separation orweak dynamic coupling, do not necessarily add any significant contribution to stability. Softwarepackages for illustrating other types of attack scenarios such as data manipulation attacks, jamming,eavesdropping, and GPS spoofing also need to be developed [60, 61, 62].

8 Simulation of Wide-Area Controllers

Besides design and implementation, an equally important challenge for WAMS engineers is todevelop simulation platforms where extreme-scale power grid models, and their associated wide-area monitoring and control algorithms can be simulated for various contingencies against various

13

system-level uncertainties. As described in the introduction, the grid itself is changing because ofthe proliferation of new technologies. New sources of renewable generation that are intermittentand have no rotating inertia, are being added rapidly. New types of loads such as electric vehiclesand smart buildings are also proliferating. Power electronic converters and controllers are beingintroduced to connect these new generation sources, load and storage devices to the grid. The gridis being overlaid with more PMUs, communications, computers, information processors, and con-trollers. The challenge is that to fully showcase the capability of any wide-area controller all thesenew technologies must be modeled and simulated together. Growth in computing power showsno signs of slowing down, so we do not foresee any limitations on model size or algorithm speed.However, the ability to utilize such powerful simulations depends on how easy their handling canbe made to the engineers. Moreover, power systems today are gradually becoming an integral partof other interconnected infrastructures such as gas networks, transportation networks, communi-cation networks, water networks, economics, and food chain networks. Thus, inter-operability ofsimulation programs will become a key to minimizing the manual effort needed to set up and runco-simulations of these systems with PMU-assisted monitoring and control loops in grid models.We highlight two potential ways by which these co-simulations can be handled.

8.1 Parallel Computing

An obvious approach to speed up simulations is to utilize parallel or distributed computing. Al-though specially written programs for particular parallel architectures can provide high speed-ups,the rapidly changing hardware and software makes it impossible to keep modifying the simulationprograms to keep up. The trend today is to use multiprocessor computers with compilers thatcan distribute the computation optimally to multiple processors. For wide-area monitoring andcontrol simulations, some applications are much more amenable to parallelization than others. Forexample, any simulation that requires running many contingencies can run these hundreds of con-tingency cases in separate processors. The dynamics of individual generators can be run in parallelbut the network that connects the generators has to be solved simultaneously. It turns out, how-ever, that the algebraic equations representing the network cannot be parallelized very efficiently,and becomes the main bottleneck for speeding up power system simulations. Parallel computingcan also be used for gain-scheduling of robust wide-area controllers.

8.2 Hybrid Simulations

In order to implement fast wide-area control strategies it is highly desirable to have a faster thanreal-time simulation of that model in hand. One promising solution is hybrid mixed-signal hardwareemulation. In these emulations hardware-based digital simulation and analog simulation can be usedin an integrative way to achieve a massively parallel and scale-insensitive emulation architecture.There have been several attempts at building hardware accelerators in the past [63, 64]. Forexample, the grid models can be emulated in hardware by a coupled set of oscillators, resistors,capacitors and active inductors. A higher frequency can be used so as to suit the scale of on-chipelements and permit faster than real-time operation. These elements then need to be built onchip, and connected with a customizable switch matrix, allowing a large portion of the grid tobe modeled in real-time. Researchers have proposed the use of Verilog-AMS model, which is ahardware modeling language that includes features for Analog and Mixed Signal (AMS) elements[65]. It can incorporate equations to model analog sub-components. The AMS model can bedesigned to emulate open-loop models of very large-scale power systems with tens of thousands ofbuses with built-in equations for AC power flow, electro-magnetic and electro-mechanical dynamicsof synchronous generators and induction generators, AC load models, power oscillation dampingcontrol, voltage control, droop control, AGC, and PSS. Several challenges that still stand on theway for developing at-scale faster than real-time simulations are:

1. How to synthesize the transmission network without unnecessary and unrealistic assumptions,and approximations using state-of-the-art microelectronics design technology?

14

2. How to develop a scalable mixed-signal emulation architecture capable of large power systemswith tens of thousands nodes?

3. How to design configurable units of emulation on a chip so that any large power system withrealistic transmission connections can be realized via software-based configuration?

Research activities are underway on building VLSI chips that can scale to large grid models forfaster than real-time emulators, especially directed towards transient wide-area AC emulations. In[64], for example, predictive simulations were shown to be very useful for real-time path rating.

9 Simulation Testbeds

Gaining access to realistic grid models and PMU data owned by utility companies can be difficultdue to privacy and non-disclosure issues. More importantly, in many circumstances even if realdata are obtained they may not be sufficient for studying the detailed operation of the entire systembecause of their limited coverage. To resolve this problem several smart grid simulation testbedshave recently been developed to facilitate hardware-in-loop simulation of different grid applicationswithout the need for gaining access to real data. Selected examples of such testbeds in the UnitedStates include CPS testbeds at Washington State University using GridStat [55], cloud-assistedwide-area control testbed at North Carolina State University using ExoGENI [54], cyber-securitytestbeds at Iowa State [66], TCIPG in University of Illinois [67], DETER-lab testbed at Universityof Southern California [68], CPS testbeds at Idaho National Lab, Cornell University, and PacificNorthwest National Labs, and a big data hub at Texas A&M [69]. A comprehensive list of manyother smart grid testbeds and their CPS capabilities was recently presented in the survey paper [70].Two key questions that most of these testbeds are trying to answer are - (1) is it possible to designsufficiently general CPS standards and protocols to support a mass plug-and-play deployment ofa wide-area grid without sacrificing reliability, data privacy and cyber-security, and (2) if so, thenwhat standards and protocols are required to transform today’s grid into an end-to-end enabler ofelectric energy services.

9.1 Hardware Components

Generally speaking, the physical component of these testbeds are comprised of Real-Time DigitalSimulators (RTDS) and Opal-RT. These are power system simulator tools that allow real-timesimulation of both transmission and distribution models with a time-step of 50 microseconds.The RTDS comes with its own proprietary software known as RSCAD, which allows the user todevelop detailed dynamic models of various components in prototype power systems. The RTDSalso comes with digital cards that allow external hardware to interface with the simulation. Forexample, the Gigabit Transceiver Analog Output (GTAO) card allows the user to view low-levelsignals proportional to voltages and currents at different buses of the system in real-time. TheGTAO card generates voltage and current waveforms, and communicates them to sensors suchas relays, circuit breakers, and PMUs. The PMUs measure these signals, and send the resultingdigitized phasor data calculations to the PDCs. The PDC time-stamps and collects the data fromall the PMUs, and sends them to the server for display and archival, when requested. The hardwareand the software layers of these testbeds are integrated with each other to create a substation-likeenvironment within the confines of research laboratories. The two layers symbiotically capturepower system dynamic simulations as if these measurements were made by real sensors installed atthe high-voltage buses of a real transmission substation.

9.2 Cyber and Software Components

The cyber-layer, on the other hand, is generally emulated by either a local-area network or alocal cloud service. The ExoGENI-WAMS testbed at NC State [54], for example, is connected to astate-funded, metro-scale, multi-layered advanced dynamic optical network testbed called Breakable

15

Experimental Network (BEN) that connects distributed cloud resources in local universities [71]. Itallows one to set up dynamic multi-layer connections of up to 10 Gbps between different sites. Onemay simulate different types of disturbance events in power system models in RTDS, collect theemulated grid responses via PMUs and other sensors, communicate these data streams via BEN,and run virtual computing nodes at various sites in the ExoGENI cloud overlaid on top of BENto execute distributed estimation and control algorithms. Some open questions for these types ofset-ups in the future are, for example - where to deploy the computing facilities to better facilitatedata collection and processing, and how to design better communication topologies.

9.3 A Network of Remote Testbeds

One pertinent question is whether these different simulation testbeds at different locations shouldbe conjoined with each other to create a much bigger nationwide network of CPS testbeds. Andif yes, then what are the most common challenges for such remote testbed federation? Developingprotocols for usability by different users, and potential safety hazards are two important challenges,for example. Researchers are also contemplating making their testbeds open to public for accelerat-ing research in the power and CPS community, but a robust economic and ethics model for sharingaccess to private resources still needs to be developed. Should there be a common centralized sim-ulation testbed for accessing power system model and data, one must also resolve standardizationissues, communication issues, maintenance costs, and strategies for sustainability.

9.4 Interoperability: Databases and User Interfaces

One of the major challenges for the users of the hundreds of existing simulations is that none of themare compatible with each other. Using two different simulations require keeping up two differentdatabases of input data, and being familiar with two different sets of graphical outputs. This is notonly true for different types of applications but also of the same application marketed by differentvendors. Thus, in the current state of art it is impossible to integrate these different simulationprograms. The easiest way to encourage inter-operability is to standardize the databases. Thedata that go into these databases are proprietary to the utilities, and if the utilities can agree onusing a standard database the simulation vendors will have to adopt it. In the USA, the NationalInstitute for Standards and Technology (NIST) has been tasked to develop such standards. Anearlier standard called the Common Information Model (CIM) is now an IEC standard, and isslowly being adopted at different rates in different countries. A similar effort should be made tostandardize user interfaces for wide-area monitoring and control. If the database and user interfacefor a simulation is standardized, the ability to integrate different simulators as mentioned abovewould become much simpler.

10 Summary and Conclusions

This tutorial serves as a technical invitation to engineers for entering the challenging and attractiveresearch field of wide-area control of power systems. We presented an overview of the main researchideas related to the cyber-physical aspects of this topic, and established a strong dependence ofcontrol on various properties of the underlying communication, computing, and cyber-security. Ev-idently, there are several challenges that need to be surmounted in order to implement the proposeddesigns, requiring a strong knowledge of stochastic modeling, estimation theory, distributed control,system identification, model reduction, robust control, optimization, and related topics in signalprocessing and computer science. It is, therefore, our hope that this topic will be viewed by bud-ding and established control theorists as a challenging and attractive opportunity. We hope thatthe compelling societal importance of power and energy systems will also serve as an additionalmotivation to enter this endeavor.

While this tutorial overviewed the general research landscape of WAMS, in the following threechapters we present a more detailed overview of some specific research challenges associated with

16

this technology. A preview of these chapters is as follows.

The chapter by Chakrabortty enlists six research challenges that need to be resolved beforewide-area control can transition from a concept to a reality. These challenges include scalability ofmodeling and control, system identification and online learning of models from streaming or storedPMU data followed by model validation, wide-area communication and its associated uncertaintiesand architectural bottlenecks, cost allocation strategies for renting links in wide-area communica-tion networks where game-theoretic algorithms can play a significant role, cyber-security of WAMS,and finally, WAMS simulation testbeds where real-time interaction between hardware PMUs andsoftware emulation of communication networks can be testbed with high fidelity.

The chapter by Wang highlights several research problems on the signal processing aspects ofWAMS. It presents an overview of the data-centric challenges that the power industry is currentlyfacing while it transitions from SCADA-based monitoring and control to wide-area monitoring andcontrol using gigantic volumes of PMU data. Data quality is an inevitable issue for the control-roomincorporation of these data. Because of the communication networks that were not conventionallydesigned to carry high-speed PMU data and the early deployment of older PMUs, data losses anddata quality degradations happen quite often in practice. Different applications have diverse re-quirements on the data quality. This chapter provides a detailed list of challenges associated withdata quality, and also cites recommendations on how trust scores can be assigned to data for takingcontrol actions.

The chapter by Sun addresses the topic of wide-area protection and mitigation of cascadingfailures using PMU data. In the current-state of art cascading failures in the grid are prevented byseparating the grid into disconnected islands. The chapter proposes the use of controlled islandingusing real-time PMU data by which system efficiency can still be maintained at an acceptablelevel. It specifically highlights the three main challenges on when to island, where to island, andhow to island. For the latter it proposes the use of advanced numerical algorithms for predictingtransient instability using PMU data, and lists a number of open challenges on how these predictivecommands can be made use of in taking appropriate control and decision-making actions.

References

[1] A. G. Phadke, J. S. Thorp, Synchronized Phasor Measurements and Their Applications,Springer, 2008.

[2] N. R. Chaudhuri, D. Chakraborty, and B. Chaudhuri, “Damping Control in Power SystemsUnder Constrained Communication Bandwidth: A Predictor Corrector Strategy,” IEEE Trans-actions on Control Systems Technology, vol. 20(1), pp. 223-231, Jan 2012.

[3] S. Zhang and V. Vittal, “Design of Wide-Area Power System Damping Controllers Resilient toCommunication Failures,” IEEE Transactions on Power Systems, vol. 28(4), Nov. 2013.

[4] N. R. Chaudhuri, B. Chaudhuri, S. Ray, R. Majumder, “Wide-area phasor power oscillationdamping controller: A new approach to handling time-varying signal latency,” IET Generation,Transmission & Distribution, vol. 4(5), pp. 620-630, 2010.

[5] R. A. Jabr, B. C. Pal and N. Martins, “A Sequential Conic Programming Approach for theCoordinated and Robust Design of Power System Stabilizers,” IEEE Transactions on PowerSystems, vol. 25, no. 3, pp.1627-1637, August 2010.

[6] M. Zima, M. Larsson, P. Korba, C. Rehtanz, and G. Andersson, “Design Aspects for Wide-AreaMonitoring and Control Systems,” Proceedings of the IEEE, vol. 93(5), pp. 980-996, 2005.

17

[7] A. Chakrabortty, “Wide-Area Damping Control of Power Systems Using Dynamic Clusteringand TCSC-Based Redesigns,” IEEE Transactions on Smart Grid, vol. 2(3), Nov. 2012.

[8] A. Chakrabortty, “Wide-Area Damping Control of Large Power Systems Using a Model Refer-ence Approach,” 50th IEEE Conference on Decision and Control, FL, 2011.

[9] J. H. Chow and S. G. Ghiocel, “An Adaptive Wide-Area Power System Damping Controllerusing Synchrophasor Data,” Control and Optimization Methods for Electric Smart Grids, pp.327-342, 2012.

[10] A. Jain, A. Chakrabortty, and E. Biyik, “An Online Structurally Constrained LQR Design forDamping Oscillations in Power System Networks,” American Control Conference, Seattle, WA,2017.

[11] F. Dorfler, M. Jovanovic, M. Chertkov, and F. Bullo, “Sparsity-Promoting Optimal Wide-AreaControl of Power Networks,” IEEE Transactions on Power Systems, vol. 29(5), pp. 2281-2291,2014.

[12] X. Wu, F. Dorfler, and M. Jovanovic, “Input-Output Analysis and Decentralized OptimalControl of Inter-Area Oscillations in Power Systems,” IEEE Transactions on Power Systems,vol. 31(3), pp. 2434-2444, 2016.

[13] M. E. Raoufat, K. Tomsovic, and S. M. Djouadi, “Virtual Actuators for Wide-Area DampingControl of Power Systems,” IEEE Transactions on Power Systems, vol. 31(6), 2016.

[14] C. W. Taylor, D.C. Erickson, K.E. Martin, R.W. Wilson, and V. Venkatasubramanian, “Wide-Area Stability and Voltage Control System: R & D and Online Demonstration,” Proceedings ofthe IEEE, vol. 93(5), pp 892-906, May 2005.

[15] M. D. Weiss, J. Zhang, and A. Chakrabortty, “Wide-Area Control of Power Systems usingCloud-in-the-Loop Feedback,” IEEE Global Conference on Signal and Information Processing(GlobalSIP), Washington DC, Dec. 2016.

[16] I. Kamwa, R Grondin, and Y. Hebert, “Wide-Area Measurement Based Stabilizing Control ofLarge Power Systems - A Decentralized/Hierarchical Approach,” IEEE Transactions on PowerSystems, vol. 16(1), pp. 136-153, 2001.

[17] N. R. Chaudhuri, A. Domahidi, B. Chaudhuri, R. Majumder, P. Korba, S. Ray, and K.Uhlen, “Power Oscillation Damping Control Using Wide-Area Signals: A Case Study on NordicEquivalent System,” IEEE PES T&D Conference and Exposition, 2010.

[18] R. Majumder, B. C. Pal, C. Dafour and P. Korba, “Design and Real-time Implementation ofRobust FACTS controller for Damping Inter-area Oscillation,” IEEE Transactions on PowerSystems, vol. 21, no. 2, pp. 809-816, May 2006.

[19] C. Lu, X. Wu, J. Wu, P. Li, and Y. Han, L. Li, “Implementations and Experiences of Wide-area HVDC Damping Control in China Southern Power Grid,” in proceedings of the IEEE PESGeneral Meeting, San Diego, CA, 2012.

[20] Y. Pipelzadeh, B. Chaudhuri, T. C. Green, “Wide-area power oscillation damping controlthrough HVDC: A case study on Australian equivalent system,” IEEE PES General Meeting,2010.

[21] E. M. Martinez, L. Vanfretti, and F. R. Sevilla, “Automatic Triggering of the Interconnec-tion between Mexico and Central America using Discrete Control Schemes,” IEEE PES ISGTEurope, 2013

18

[22] A. Chakrabortty and P. Khargonekar, “Introduction to Wide-Area Monitoring and Control,”American Control Conference, Washington DC, 2013.

[23] D. Soudbaksh, A. Chakrabortty, and A. Annaswamy, ‘A Delay-aware Cyber-Physical Archi-tecture for Wide-area Control of Power Systems,” Control Engineering Practice, 2017.

[24] N. Anh, L. Vanfretti, J. Driesen, and D. V. Hertem, “A Quantitative Method to Determine ICTDelay Requirements for Wide-Area Power System Damping Controllers,” IEEE Transactionson Power Systems, vol. 30(4), pp. 2023-2030, 2015.

[25] North American Synchrophasor Initiative (NASPI) - www.naspi.org

[26] J. Zhang, S. Nabavi, A. Chakrabortty, and Y. Xin, “ADMM Optimization Strategies for Wide-Area Oscillation Monitoring in Power Systems under Asynchronous Communication Delays,”IEEE Transactions on Smart Grid, vol. 7(4), pp. 2123 - 2133, 2016.

[27] J. Zhang, S. Nabavi, A. Chakrabortty, and Y. Xin, “Convergence Analysis of ADMM BasedPower System Mode Estimation Under Asynchronous Wide-Area Communication Delays,”IEEE PES General Meeting, Denver, CO, July 2015.

[28] S. Nabavi, J. Zhang, and A. Chakrabortty, “Distributed Optimization Algorithms for Wide-Area Oscillation Monitoring in Power Systems Using an Inter-Regional PMU-PDC Architec-tures,” IEEE Transactions on Smart Grid, vol. 6(5), pp. 2551 - 2559, Sep. 2015.

[29] M. Jarschel, S. Oechsner, D. Schlosser, R. Pries, S. Goll and P. Tran-Gia, “Modeling andPerformance Evaluation of an Openflow Architecture,” Proceedings of the 23rd InternationalTeletraffic Congress, 2011.

[30] P. M Anderson and A. A. Fouad, Power System Control and Stability, 2nd Edition, WileyInterscience, 2002.

[31] A. K. Singh and B. C. Pal, “Decentralized Dynamic State Estimation in Power Systems UsingUnscented Transformation,” IEEE Transactions On Power Systems, vol. 29(2), 2014.

[32] F. Lian, A. Chakrabortty, and A. Duel-Hallen, “Game-Theoretic Multi-Agent Control andNetwork Cost Allocation Under Communication Constraints,” IEEE Journal on Selected Areasin Communications, vol. 35(2), pp. 330 - 340, 2017.

[33] A. Chakrabortty, J. H. Chow, and A. Salazar, “Interarea Model Estimation for Radial PowerSystems Transfer Paths with Intermediate Voltage Control using Synchronized Phasor Mea-surements,” IEEE Transactions on Power Systems, vol. 24, no. 3, pp. 1318-1326, Aug. 2009.

[34] S. Malik and F. Huet, “Virtual Cloud: Rent Out the Rented Resources,” International Con-ference for Internet Technology and Secured Transactions, 2011.

[35] M. Wytock and Z. Kolter, “A Fast Algorithm for Sparse Controller Design,” ArXiv preprint,2013.

[36] A. M. Boker, T. R. Nudell, and A. Chakrabortty, “On Aggregate Control of Clustered Con-sensus Networks,” American Control Conference, pp. 5527-5532, Chicago, IL, USA, 2015.

[37] T. Sadamoto, T. Ishizaki, and J. I. Imura, “Hierarchical Distributed Control for NetworkedLinear Systems,” 53rd IEEE Conference on Decision and Control, pp. 2447-2452, 2014.

[38] S. Hara, J. I. Imura, K. Tsumura, T. Ishizaki, and T. Sadamoto, “Glocal (Global/Local) Con-trol Synthesis for Hierarchical Networked Systems,” IEEE Conference on Control Applications(CCA), pp. 107-112, 2015.

19

[39] N. Xue and A. Chakrabortty, “H2-Clustering of Closed-loop Consensus Networks under aClass of LQR Design,” American Control Conference, 2016.

[40] R. Harvey, Y. Xu, Z. Qu, and T. Namerikawa, “Dissipativity-based Design of Wide-AreaGeneration Control for Large-Scale Power Systems with High Penetration of Renewables,” IEEEConference on Control Technology and Applications, Kohala Coast, Hawaii, USA, Aug. 2017.

[41] P. T. Myrda, J. Taft, and P. Donner, “Recommended Approach to a NASPInet Architecture,”45th Hawaii International Conference on System Science, (HICSS), 2012.

[42] J. G. Deshpande, K. C. Budka, and M. Thottan, Communication Networks for Smart Grids,Springer, 2014.

[43] D. Anderson, W. S. Cleveland, B. Xi, “Multifractal and Gaussian fractional sumdifferencemodels for Internet traffic,” Performance Evaluation, vol. 107, Jan. 2017.

[44] H. Wu, H. Ni, and G. T. Heydt, “The Impact of Time Delay on Robust Control Design inPower Systems,” IEEE Power Engineering Society Winter Meeting, pp. 1511-1516, 2002.

[45] J. He, C. Lu, X. Wu et al., “Design and Experiment of Wide Area Hvdc SupplementaryDamping Controller Considering Time Delay in CSG,” IET Generation, Transmission andDistribution, vol. 3, No. 1, pp. 17-25, 2009.

[46] B. Chaudhuri, R. Majumder, B. Pal, “Wide-Area Measurement-Based Stabilizing Control ofPower System Considering Signal Transmission Delay,” IEEE Trans. on Power Systems, v. 19,No. 4, Nov. 2004.

[47] J. W. Stahlhut, J. Browne, G. T. Heydt, and V. Vittal, “Latency viewed as a stochasticprocess,” IEEE Trans. Power Systems, vol. 23, no. 1, Feb. 2008.

[48] D. Soudbaksh, A. Chakrabortty, and A. Annaswamy, “Computational Codesigns for Wide-Area Control of Large Power Grids,” 55th IEEE Conference on Decision and Control, LosAngeles, Dec. 2014.

[49] A. M. Annaswamy, D. Soudbakhsh, R. Schneider, D. Goswami, and S. Chakraborty, “Arbi-trated network control systems: A co-design of control and platform for cyber-physical systems,”in Control of Cyber-Physical Systems, ser. Lecture Notes in Control and Information Sciences,D. C. Tarraf, Ed. Springer International Publishing, 2013, vol. 449, pp. 339–356.

[50] S. Chakraborty, S. Kunzli, and L. Thiele, “A general framework for analysing system propertiesin platform-based embedded system designs,” in DATE, 2003.

[51] D. Soudbakhsh, L. Phan, O. Sokolsky, I. Lee, and A. Annaswamy, “Co-design of Control andPlatform with Dropped Signals,” in The 4th ACM/IEEE International Conference on Cyber-Physical Systems (ICCPS’13), April 2013.

[52] SmarTS Lab, Royal Institute of Technology (KTH), Sweden: https://github.com/

SmarTS-Lab

[53] M. Chenine, L. Vanfretti, S. Bengtsson, and L. Nordstrom, “Implementation of an Experi-mental Wide-Area Monitoring Platform for Development of Synchronized Phasor MeasurementApplications,” IEEE Power and Energy Society General Meeting, 2011.

[54] A. Chakrabortty and Y. Xin, “Hardware-in-the-Loop Simulations and Verifications of SmartPower Systems Over an Exo-GENI Testbed,” proceedings of 2nd GENI Research and EducationalExperiment Workshop, GREE-2013, Utah, Mar. 2013.

[55] GridStat: www.gridstat.net

20

[56] T. Qian, A. Chakrabortty, F. Mueller, and Y. Xin, “A Distributed Storage System for Multi-Resolution Virtual Synchrophasors,” IEEE PES General Meeting, Washington DC, July 2014.

[57] Stuxnet: https://en.wikipedia.org/wiki/Stuxnet

[58] National Cyber Security Awareness Month: https://www.it.iastate.edu/blog/view/36

[59] J. Zhang, P. Jaipuria, A. Hussain, and A. Chakrabortty, “Attack-Resilient Estimation ofPower System Oscillation Modes using Distributed and Parallel Optimization: Theoretical andExperimental Methods,” Conference on Decision and Game Theory for Security (GameSec),Los Angeles, CA, 2014.

[60] F. Pasqualetti, F. Dorfler, and F. Bullo, “Cyber-physical Attacks in Power Networks: Models,Fundamental Limitations and Monitor Design,” 50th IEEE Conference on Decision and Controland European Control Conference, Orlando, FL, 2011.

[61] G. Dan, H. Sandberg, M. Ekstedt, and G. Bjrkman, “Challenges in Power System InformationSecurity,” IEEE Security & Privacy, vol. 10(4), pp. 62-70, 2012.

[62] Y. Wang and A. Chakrabortty, “Distributed Monitoring of Wide-Area Oscillations in the Pres-ence of GPS Spoofing Attacks,” IEEE Power and Energy Society General Meeting (PESGM),2016.

[63] I. Nagel, “Analog Microelectronic Emulation for Dynamic Power System Computation,” PhDThesis, EPFL, 2013.

[64] S. Jin, Z. Huang, R. Diao, D. Wu, and Y. Chen, “Parallel Implementation of Power SystemDynamic Simulation,” IEEE PES General Meeting, 2013.

[65] E. J. Wyers, M. Steer, C. Kelley, and P. Franzon, “A Bounded and Discretized Nelder-MeadAlgorithm Suitable for RFIC Calibration,” IEEE Transactions on Circuits and Systems I: Reg-ular Papers, vol. 60(7), pp. 1787-1799, Jul. 2013.

[66] Power Infrastructure Cybersecurity Laboratory: http://powercyber.ece.iastate.edu/

penetintro.html

[67] TCIPG: Trustworthy Cyber Infrastructure for the Power Grid, https://tcipg.org

[68] The DETER Project: https://deter-project.org

[69] Global Network for Synchrophasor Solutions: http://gnssconsortium.org

[70] M. H. Cintuglu, O. A. Mohammed, K. Akkaya, and A. S. Uluagac, “A Survey on Smart GridCyber-Physical System Testbeds,” IEEE Communications Surveys & Tutorials, vol. 19(1), FirstQuarter 2017.

[71] GENI: www.geni.net

21