Implementation Challenges in Real-Time Middleware for DistributedAutonomous Systems

Vincenzo LiberatoreDivision of Computer Science

Case Western Reserve University10900 Euclid Avenue

Cleveland, Ohio 44106-7071, USAE-mail: vl@case.edu

URL: http://vincenzo.liberatore.org/NetBots/

1 Introduction

Exploration missions will achieve a sustainable human-robotic presence in space and on the surface of the Moonand of Mars. Sustained operations require higher perfor-mance, which in turn implies a higher degree of asset au-tonomy. For example, construction, maintenance, or in-situ resource utilization will be accomplished by efficientlyexecuting complex tasks in the absence of long-haul tele-operation from Earth. Moreover, assets will communicatewith each other and form a distributed system. For example,multiple units will collaborate on, say, a construction task.The resulting system is both autonomous (i.e., it can oper-ate in the absence of direct Earth tele-operation) and dis-tributed (i.e., multiple assets communicate and collaboratewith each other). Robustness and economy further demandthe re-usability of tested middleware whenever possible. Fi-nally, the distributed autonomous systems will include as-sets that operate in real-time: examples are surface robots,rovers, CEV on-board actuation units, or space constructionrobots.

In this paper, we describe the transversal challenges inimplementing real-time middleware components for dis-tributed autonomous systems. The emphasis is on reusablecomponents that address issues that are shared among dis-tributed autonomous real-time systems. Specific applica-tions are not the focus of this paper.

Real-time middleware has been extensively used in ter-restrial systems. For example, Real-Time CORBA (im-plemented, say, in TAO-ACE [21]) aids in the develop-ment of distributed real-time applications. Other middle-ware paradigms are for example TMO (Time-triggered andMessage-triggered Objects [11]) and Agents (as in Jack,used for example in UAV and surface autonomous vehi-cles, or as in Aglets [12], which we have employed for re-

mote manipulation tasks [1]). Terrestrial real-time middle-ware has reached high technology maturity and solves forthe most part three major issues: (1) a consistent and well-defined API for applications (e.g., a well-defined methodol-ogy for RPC or for resource discovery), (2) a way to specifyreal-time constraints, such as soft or hard real-time dead-lines, and (3) an implementation that maps the real-timerequirements into process or thread scheduling. However,the current state of the art fails to address network-relatedreal-time issues.

In this paper, we discuss four critical network-centricreal-time middleware services and the challenges in theirimplementation. First, applications specify network QoS interms of, say, delay, jitter, losses, or bandwidth. The appli-cation requirements are then translated into network QoS.However, the translation is often difficult. For example,if the underlying network supports, say, assured forward-ing [20], then it is not straightforward for the middlewareto map application requirements into network Classes ofService. Furthermore, general network QoS may requiresubstantial coordination for traffic shaping or for resourcereservation, depending on the specific QoS model. An al-ternative is to adopt Local QoS [13], at the cost of reducedend-point control over the network. The second application-independent service is the so-called rate control module,which defines the rate at which communication end-pointsinject packets into a network. Although rate control is well-understood in the case of bulk data transfers [22] or of cer-tain multimedia applications (audio, video) [7], it is an openproblem for distributed autonomous systems consisting ofsensing, actuation, and control units. Third, sensor sam-ples and control signals are subject to losses and jitter in thenetwork infrastructure even if QoS is supported. The end-point middleware should recover from losses and jitter via aplay-back buffer. Although play-back has been extensively

1

studied in digital audio, play-backs are an open problemin sense-and-respond systems. Finally, middleware mustbe hosted on space assets that are constrained in terms ofcomputation, memory, power, or communication. A majorproblem is to map high-level middleware services to suchdevices.

2 Background

2.1 Middleware

The terrestrial Internet has grown explosively in the lastdecade. The Internet success is due in large part to a sim-ple network architecture combined with an open interfacethat lets developers invent and deploy novel applicationsand services. The Internet model is radically different fromthe traditional practice in space networks, where commu-nication software has often been developed specifically foreach application or mission. However, the TCP/IP interfaceaddresses fairly low level communication issues. Socketsare similar to low-level disk I/O and force application de-velopers to structure communication around the disk I/Oparadigm. On the other hand, a distributed application maybenefit from more complex abstractions, such as remoteprocedure calls, where locally running code will invoke afunction to be executed transparently on a remote server.Fortunately, the open-interface Internet architecture makesit possible to implement more complex primitives on topof TCP/IP. The resulting system is middleware, a set of li-braries that abstract the details of the underlying commu-nication protocols. Middleware implements common func-tionality once, thereby significantly speeding up the timeand cost of developing distributed applications. Further-more, middleware implements state-of-the-art distributedalgorithms while hiding their complexity from applicationdevelopers. Commercial off-the-shelf middleware includesfor example CORBA and Java RMI [23].

Although middleware has been primarily used in the ter-restrial setting to support business applications, much re-search has also been devoted to middleware for real-timedistributed embedded systems [21]. In this case, middle-ware must incorporate time as a first-class citizen becausethe stability, safety, and performance of applications de-pends on real-time middleware properties.

2.2 Networked Control

The literature commonly denotes the physical systemto be controlled as plant. A controller acquires data fromthe sensors that monitor the plant. The controller uses thesensor readings to issues commands to actuators so as tochange appropriately the plant state. The textbook exampleis the regulation of ambient temperature. A thermostat (the

controller) reads the temperature in the ambient (the plant)from a thermometer (the sensor) and modulates the heating(the actuator) so as to bring the room to the desired temper-ature.

In general, the plant state depends on the actuator oper-ations and also on exogenous disturbances. Furthermore,the sensors are subject to measurement errors. A networkedcontrol system (NCS) involves a plant that is controlled re-motely, and networked control is complicated by delays, jit-ter, and packet losses. In the first place, network latency candelay the reception of sensor data at the controller site andthe application of a control signal at the plant site. Further-more, packets can be lost either because they are droppedout in the network infrastructure or because they are dis-carded by the communication end-points, for example, ifthey arrive late.

3 Network Quality-of-Service

Real-time middleware must satisfy Quality-of-Service(QoS) requirements. The QoS requirements can be clas-sified depending on whether they focus on network pa-rameters (latency, jitter, loss rates, end-to-end bandwidth),distributed systems concerns (predictability, scalability, de-pendability, and security) or programming languages andinterfaces (programming abstractions for QoS). QoS re-quirements are translated into QoS provisioning by the mid-dleware to meet the application requirements [8].

Unfortunately, QoS provisioning is made difficult by twofundamental architectural principles of the terrestrial Inter-net [5]. A first principle is autonomy and decentralized con-trol. A decentralized architecture is responsible in part forpreventing the adoption of QoS mechanisms that rely onthe concurrent administration of multiple nodes or on theconcerted efforts of multiple providers. Conversely, spaceexploration networks fall within the administrative controlof a handful of government agencies, and make it relativelyeasier to coordinate administrative responsibilities. A sec-ond principle is best-effort service, which implies an unre-liable service with no provisioning for QoS. A best-effortarchitecture can employ over-provisioning to guarantees astatistical form of QoS because an over-provisioned net-work is characterized by bandwidth larger than the arrivalrates, so that queuing delays and loss rates are small. More-over, over-provisioning also leads to redundancy and fault-tolerance. On the other hand, best-effort networks do notapply policing or isolation among flows and, as a result,QoS can suffer. Specifically, over-provisioned best-effortnetworks work well in the typical case, but do not provideguarantees and expose flows to the risk of poor QoS. Fur-thermore, over-provisioning is more difficult to use in spacenetworks due to the exorbitant cost of deploying space as-sets.

2

0

20

40

60

80

100

120

140

160

180

200

0 10 20 30 40 50 60 70 80 90fast queue length (KB)

time (s)

(a) YAQS

0

500

1000

1500

2000

2500

3000

3500

0 10 20 30 40 50 60 70 80 90fast queue length (KB)

time (s)

(b) YAQS and FIFO

FIFOYAQS

0

100

200

300

400

500

600

700

60 61 62 63 64 65fast queue length (KB)

time (s)

(c) YAQS and FIFO (zoom)

FIFOYAQS

Figure 1. (Fast) queue length (trace 6). Chart (a) gives YAQS fast queue length, chart (b) comparesYAQS and FIFO, and chart (c) zooms on the interval [60, 70], where the FIFO queue starts its highestascent. The vertical axis scale differs among charts [13].

On the whole, terrestrial networks have followed thetwo fundamental Internet design principles of decentralizedcontrol and best-effort, and mostly guarantee QoS throughover-provisioning. However, space networks are not subjectto the same types of constraints. First, space explorationnetworks have a more centralized administrative structure.Second, over-provisioning is not a sustainable solution. Asa result, space networks are a better fit for QoS provision-ing, including selected mechanisms to achieve scheduling,isolation, and policing. For example, a surface networkcould establish a priority scheme to differentiate amongdata, video, audio, and control flows.

Although planetary networks can support QoS mecha-nisms, a planetary network is also an autonomous systemthat must be ready to operate without direct Earth supervi-sion. In particular, a distributed autonomous system shouldbe able to arbitrate among competing QoS demands evenwithout human supervision. For example, suppose that acertain flow has low priority until an unexpected event ren-ders it critical to the mission. The distributed autonomoussystem should adapt to the change and adapt QoS levelsaccordingly. By contrast, the flow cannot suffer poor QoSuntil a remote Earth administrator realizes the danger andrearranges priorities. In other words, QoS administrationmust also be autonomous and distributed.

An approach to autonomous and distributed QoS is LocalQoS, a set of fully distributed mechanisms to protect flowsand ensure QoS. Local QoS is typically an addition (or evena replacement) to explicit and coordinated QoS provision-ing.

In previous work, we have addressed local QoS with ascheduling algorithm that maintains short queues in best-effort networks that are generally over-provisioned but oc-casionally congested [13]. The algorithm requires no sup-port for distributed coordination. The YAQS (Yet An-other Queuing Strategy) mechanism attempts to obtain shortqueuing delays in a best-effort router that is crossed by ag-

gressive flows. The algorithm involves the following two el-ements. First, per-flow queue occupancy is maintained withhigh accuracy and limited state. Second, when the queueoccupancy of a certain flow exceeds a threshold value, sub-sequent packets belonging to that flow are segregated in adifferent queue. The forwarding decision is then made byan approximately fair schedule between the regular and thesegregated queue. The intuition is that flows with smallbuffer occupancy should benefit from a shorter queue oncethe more aggressive flows have been separated.

Figure 1 gives the (fast) queue length of YAQS and FIFOon a Internet trace. YAQS is a clear improvement over FIFOin terms of fast queue length. In fact, the YAQS line is butnoise close to the horizontal axis when compared to FIFO.The comparison is better seen by zooming in, as in Figure1(c).

Simulations afford us the opportunity to compare YAQSwith an Active Queue Management mechanism such asRED [6]. Figure 2 gives the (fast) queue length of YAQS,FIFO, and RED during the activity of an aggressive flow.YAQS is a clear improvement over FIFO and RED in termsof fast queue length. In fact, the YAQS line looks like noisewhen compared to FIFO or RED. RED improves over Drop-Tail, and it prevents long queues after an initial transientwhere it behaves like FIFO (i.e., until time 1.9s). How-ever, RED did lead to long queues initially due to its slowerdynamics. The aggressive flow goodput is 24Mbps underYAQS, 29Mbps under FIFO, and 17Mbps under RED. Insummary, YAQS was the most effective method to main-tain short queues, and simultaneously the aggressive flowachieved higher goodput than under RED.

Future work should develop the ideas of local QoS. Forexample, YAQS should be compared with Stochastic Fair-ness Queuing (SFQ) [17].

A complementary approach is to hide latency throughoverlay QoS. In an overlay network, end-points are orga-nized at the application layer into a virtual connectivity net-

3

0

5

10

15

20

25

30

0 0.5 1 1.5 2 2.5 3fast queue length (KB)

time (s)

(a) YAQS

0

100

200

300

400

500

600

700

0 0.5 1 1.5 2 2.5 3(fast) queue length (KB)

time (s)

(b) YAQS and FIFO

FIFOYAQS

0

100

200

300

400

500

600

700

0 0.5 1 1.5 2 2.5 3(fast) queue length (KB)

time (s)

(c) YAQS and RED

REDYAQS

Figure 2. (Fast) queue length in simulations. Chart (a) gives YAQS fast queue length, chart (b) com-pares YAQS and FIFO, and chart (c) compares YAQS with RED. The vertical axis scale differs betweencharts [13].

work on top of the underlying IP connectivity. OverlayQoS dynamically manages an overlay network to guaran-tee higher levels of service to applications. For example, areliable overlay can reduce the periods of network partition[4] that we found to be a significant problem for networkedcontrol performance. We expect that overlays and bufferswill act on different time scales, and thus both should beinvestigated and used.

A related issue is the the use of multihoming, as in thereal-time version of SCTP. Multihoming allows hosts to bereached from different end-to-end paths, thus improving thereliability of the distributed applications. In a real-time dis-tributed application, signals would be replicated via forwarderror correction over multiple paths, and would increase theapplication’s performance and continued operations [18].Future research should also involve a comparison of SCTPmultihoming with resilient overlays.

4 Rate Control

A second issue is the rate at which embedded devicesexchange data. The issue of bandwidth management hasgained considerable research attention in NC, see for ex-ample [25, 3] and the references contained therein. Mostresearch has focused on bandwidth scheduling in limited(local) area networks, e.g., in a car, in an airplane, or in afactory. Several reasons hinder the extension of such band-width scheduling schemes to Wide Area Networks (WANs),such as a full planetary network. Previous work usuallyrequires time synchronization among the different devicesin the network. The allocation schemes are either static ordynamic. Static schemes determine allocation before run-ning and consequently lack flexibility and adaptability todynamic changes. On the other hand, dynamic schemes of-ten required centralized implementations.

In our earlier work, we have developed a dynamic,decentralized bandwidth allocation scheme for networked

control systems that have their control loops closed overWANs [2]. The algorithm is depends on a formulation ofthe bandwidth allocation problem as a convex optimizationprogram that can be solved in a fully distributed manner.

The idea behind our approach is to have the controlsystems vary their sampling periods (thus their bandwidthconsumption) based on the congestion level fed back fromthe network, and to preserve high performance level. Ourscheme has the following features:

• It allocates the bandwidth to ensure stability of all con-trol systems, if feasible.

• It allocates the bandwidth in a way to attain the maxi-mum aggregate performance of all control systems.

• It makes use of network bandwidth efficiently; con-trols congestion, thus minimizes delays and losses, andachieves fairness by fulfilling performance objectivesof different control loops.

• It provides a fully distributed, asynchronous, and scal-able solution. Each node executes an independent al-gorithm using local information with no central man-aging entity. The approach scales up with the numberof controlled systems and the size of the network.

• It is dynamic and flexible. It dynamically reallocatesbandwidth as different control systems acquire and re-lease the network.

A NCS typically benefits from higher sampling rates.For example, the physical behavior tends to track moreclosely the intended reference behavior if the sampling rateis higher. In extreme circumstances, the sampling rate is solow that the physical system becomes unstable, in whichcase even small perturbations can cause massive break-downs. Hence, the sampling rate must strike a balance be-tween network utilization and intended physical behaviors.

4

The sampling rate is thus a critical tuning factor in NCSs.The effect of the transmission rate, r, on the physical systemdynamics is often captured by a utility function, Ui(r). Theutility value Ui(r) expresses the degree to which a partic-ular system i can benefit from sampling rate r. In general,the utility function is a monotonically increasing function ofthe rate r, which reflects the fact that higher sampling rateslead to better control performance. In practice, the utilityfunction is also often a strictly concave function of r, whichreflects a law of diminishing returns as the rate increases.

The rate control objective can be formulated as follows:determine a transmission rate of each NCS so as to maxi-mize the sum of utilities subject to (i) each system stabilityconstraint, and (ii) each link’s capacity constraint:

max∑

i

Ui(ri), (1)

s. t.∑

i∈S(l)

ri ≤ Cl, l = 1, . . . , L,

and ri ≥ rmin,i,

where S(l) is the set of NCSs whose communication loopsuse link l, Cl is the capacity of link l, and L is the totalnumber of links in the network. In this formulation, we as-sume that the communication loop for each NCS can uselink l only once. This assumption is always valid if all linksare full duplex (in which case, forward and backward traf-fic do not interfere). The optimization problem (1) can bedecomposed into separable sub-problems [2]. The solutionwas then be implemented in a distributed fashion, wherebyindividual controlled systems and links execute indepen-dent algorithms. This solution is achieved by considering adual version of the basic optimization problem that incorpo-rates the Lagrange multipliers for link capacity constraints.The innovation involves the combination of an optimizationcongestion control approach with a PI AQM method in thecase of networked control, and the exact analysis of the PI’sstability region.

A basic issue is the derivation of utility functions that arerelevant to networked control. A possible approach is to de-rive a per-plant utility function depending on how close theplant is approaching the instability region at the chosen rate,and using these utility functions in our framework. Futurework should explore an innovative way to measure plant in-stability that is based on the time-scales approach for solv-ing differential equations [9]. The advantage of time-scalesis that the utility functions can change over time dependingon the “degree of instability” currently exhibited by a plant.

In the development of congestion control algorithms, itis important to consider the effects of slow convergence ofa congestion control algorithm, which can have substantialimpact on the performance of connected plants. In counter-ing the slow convergence rates, it is also important to not to

Figure 4. Queue oscillations.

vary the sampling rates of the NCSs as this may have detri-mental effects on system performance. A promising ap-proach would be based on equation-based congestion con-trol [7] or on TCP-friendly SIMD congestion control [10].

Sense-and-respond congestion control differs from TCPcongestion control in that rate regulation must be based onrelatively fewer packets. On the other hand, the numberof packets corresponds to the speed at which an individ-ual plant receives information about the queue. As a result,the queue cannot be controlled at frequencies that are com-parable or higher than the packet frequency. In turn, thepaucity of packets causes visible queue oscillation and flowsynchronization, as in the simulations in Figure 4. The os-cillations are a problem in congestion control but are unre-markable in TCP congestion control even though the twomethods are based on the same original techniques becauseTCP’s packet rate is much higher and tends to mask the os-cillations with noise. Future work should investigate meth-ods to suppress this oscillations, and our on-going worksuggests that a particularly effective technique would be tointroduce a non-linear element prior to the PI AQM.

5 Play-Back

A play-back method smooths out delays, jitter, andpacket losses by applying control signals to the environmentduring predictable time intervals [20]. Figure 3 exemplifiesa possible signal exchange. In this example, the plant sendssamples to the controller at regular intervals of length T .A sample was dropped out after t3 but the plant kept sam-pling every T seconds and, in this example, the next signalwas successfully delivered. The unbuffered control is com-monly adopted in related work [15, 19] and applies receivedcontrol signals from the time when they are received untilthe reception of a new signal. The unbuffered mechanismsuffers the problem that no signal is applied during a pre-dictable time interval. A traditional play-back buffer holdsthe signal u1, u2, and u4 that are received before their play-back time and applies the corresponding control signal only

5

u1

u2

u3

u4

1t t

2t

3t

4

u2

u2(c) u

3(c) u

4u

1

u1

u3

+ τt2

+ L + τt4

u2

1t + τ

t3+ τ

t2+ τ + L

u1(c)

1u , u

2u

2(c), u

3u

3(c), u

4(c)4

u,

u4

integratedplay−back buffer

t

t

plant

controller

unbuffered control

pure play−back buffer

Figure 3. The timeline of network events in the remote control of a plant. In this example, τi andLi = T are constants [14].

at the planned instant. The buffer discards u3 because itis late and applies u2 until otherwise instructed by a newcontrol signal. Traditional buffers suffer from the long ap-plication interval of signals such as u2.

In our earlier work, we have developed a play-back algo-rithm for tolerant and adaptive networked control [14]. Fig-ure 3 shows one of the main improvements over traditionalbuffers. The improved algorithm times out and switches tou

(c)2 when u3 is late and to u

(c)3 when a sample is lost and

no control signal is forthcoming.

Our playback algorithm employs a reverse play-backbuffer and the estimation of play-back times, an expirationtime to curb an aggressive controller, a contingency controlto deal with loss episodes, and an observer that explicitlyaccounts for network vagaries. The intuition is that the con-trol signals should be applied during a predictable interval.If the control is applied too early, the plant will not havereached yet the state for which the control was meant. Sym-metrically, a control signal ui can negatively affect a plantif it is applied continuously well after the sampling time tibecause the plant state would in general have changed sig-nificantly in the meanwhile.

In our algorithm, the controller sends to the plant twocontrol signals, which will be called the regular control (de-noted by ui) and the contingency control (denoted by u

(c)i ).

Additionally, the controller sends also a play-back delay τi

and the duration Li of the regular control. The play-backtime τi is used to implement a relaxed play-back buffer: theregular control is applied at time ti + τi, or immediately ifthe play-back time has already passed. Thus, the play-backdelay only fixes the earliest time at which a control can beapplied. After a duration Li since the time when the controlwas applied, if a new control has not been received, the plantswitches to the contingency action u

(c)i . The duration Li

and the contingency control u(c)i prevent the plant from ap-

plying the regular control for an unpredictably long periodof time, thereby damaging the system. Therefore, the con-

tingency control should be a low performance but presum-ably safe action. Additionally, the controller sends the planta sampling period Ti, which denotes the time interval untilthe plant collects the next sample from the sensor and sendsit to the controller. A predictable sampling schedule Ti ef-fectively establishes time-out protection against losses. Theplay-back algorithm is integrated with sampling (throughthe dynamic setting of sampling time) and control (throughexpiration times and performance metrics).

The sampling rate 1/T normalized by the packet sizeis the instantaneous bandwidth used by networked control.Since T is related to bandwidth and τ to communication de-lays, T and τ are largely independent of each other. In par-ticular, nothing prevents T � τ , in which case a sequenceof samples and control signals are typically in flight in thenetwork. Such a scenario is called a control pipe. Notionssimilar to pipes were also discussed in [16, 24]. Althoughthe pipe is, in some sense, stored in the network, the plantdoes not keep explicit state for the signals in the pipe. Themain advantage of using control pipes comes from the factthat shorter sampling periods tend to improve control per-formance. A second advantage of pipes is that it enables acontroller to quickly countermand a previous control signal,in case of transient delay spikes.

The algorithm was extensively simulated and emulated.The evaluations demonstrated that the algorithm producedlow variability around the set point and it was robust to vary-ing levels of connectivity. Extensive simulations showedthat the combined approach was able to remove 75% of theadditional (non-inherent) plant variability. Furthermore, theplay-back control performed roughly as well as any pro-portional controller on an ideal network with pure delays.The algorithm was robust to parameter choice and its per-formance gracefully degraded when network connectivityworsened (Figure 5). The algorithm addressed several prob-lematic issues in networked control systems. A control sig-nal ui is applied during a time interval whose bounds are de-fined by the play-back delay and by the expiration time. The

6

1e+00

1e+01

1e+02

1e+03

1e+04

1e+05

1e+06

1e+07

1e+08

1e+09

1e+10

1e+11

1e-06 1e-05 1e-04 1e-03 1e-02 1e-01 1e+00

m2

log (1-q)

open loop

play-backunbuffered

Figure 5. Root-mean-square m2 as a functionof the probability q of remaining in the lossystate [14].

expiration time was especially effective in the case of lossesin that it prevented the continued application of an old con-trol that was appropriate at the time it was originally appliedbut that later on would take the plant output away from itsreference value. Symmetrically, the play-back buffer pre-vented the application of a signal when it was not appro-priate yet for the current plant state. The introduction of aplayback delay was very effective in mitigating the effectsof jitter, at the price of a minimal additional average delay.

A major open question is the integration of play-backbuffers with congestion control. In the first place, the play-back algorithm also fixes the sampling period of T as a bal-ance of two competing objective: a smaller T leads to betterplant performance, but a smaller T also increases the num-ber of control pipe imperfections that are caused by jitter.The algorithm dynamically attempts to decrease T until Treaches a value so small that it would start causing controlpipe imperfections. An integrated algorithm can take oneof following two approaches. First, T could be generatedby the rate control algorithm, but a final check is made toensure that T is higher than a jitter-aware lower bound. Thesecond approach is to remove the lower bound altogether byusing more aggressive buffers at the plant site. In principle,if the plant had an infinite buffer, the effect of jitter wouldvanish. However, larger buffers also pose significant ques-tions: its memory consumption is heavier, and more sig-nificantly, the prediction of the future plant state becomesextremely complicated. As a consequence, the observer’sprediction could be far less accurate. In summary, it is notclear whether the better solution is to keep the pipe simplewhile restricting the smallest value of the sampling periodT .

The play-back algorithm often generates control pipes inwhich multiple samples and control signals are outstand-ing. Meanwhile, the plant uses timers to clock the sample

generation process. Control pipes present an opportunityfor congestion control to be based on a congestion win-dow, thus becoming self-clocked and dispensing of timers.Future work should include an investigation of congestioncontrol based on a congestion window. Finally, the commu-nication flow is driven by the plant timers, and alternativesampling schemes should be investigated.

6 Middleware Implementation

Terrestrial middleware is modular and flexible, but of-ten demanding in terms of computational resources or band-width. A major open question is to map the clean designsof distributed real-time embedded systems on constrainedresources, such as those that are going to be deployed onspace assets.

A number of pragmatic issues must also be addressed be-fore the algorithms can be integrated into a working system.In the first place, samples and controls should be sent overRTP. Although we are confident that the RTP protocol is ap-propriate for real-time flows, we had problems with severalof its implementations. Future work should undertake an in-vestigation of existing RTP libraries and implement modifi-cations as needed. We expect that the modifications will beconfined to the implementations and that no changes wouldbe needed in the RTP/RTCP protocols. Similarly, our pre-liminary investigation suggests that SCTP implementationsmay suffer from problems similar to RTP implementations.

Acknowledgments

Work supported in part under NSF CCR-0329910, De-partment of Commerce TOP 39-60-04003, and NASANNC04AA12A. We would like to thank Ahmad Al-Hammouri for Figure 4.

References

[1] A. Al-Hammouri, A. Covitch, D. Rosas, M. Kose,W. S. Newman, and V. Liberatore. Compliant controland software agents for Internet robotics. In WORDS,2003.

[2] Ahmad Al-Hammouri and Vincenzo Liberatore. De-centralized and dynamic bandwidth allocation in net-worked control systems. In 14th International Work-shop on Parallel and Distributed Real-Time Systems(WPDRTS), 2006.

[3] L. Almeida, L. Pedreiras, and P. Fonseca. The FTT-CAN protocol: why and how. IEEE Transactionson Industrial Electronics, 49(6):1189–1201, Decem-ber 2002.

7

[4] D. G. Andersen, H. Balakrishnan, M. F. Kaashoek,and R. Morris. Resilient overlay networks. Operat-ing Systems Review, 35(5):131–145, 2001.

[5] V. Cerf and R. Kahn. A protocol for packet networkinterconnection. IEEE Transactions on Communica-tions Technology, COM-22(5):627–641.

[6] Mikkel Christiansen, Kevin Jeffay, David Ott, andF. Donelson Smith. Tuning RED for Web traffic. InProc. Sigcomm, pages 139–150, 2000.

[7] S. Floyd, M. Handley, J. Padhye, and J. Widmer.Equation-based congestion control for unicast appli-cations. Computer Communication Review, 30(4):43–56, 2000.

[8] Christopher D. Gill, Jeanna M. Gossett, Joseph P. Loy-all, Douglas C. Schmidt, David Corman, Richard E.Schantz, and Michael Atighetchi. Integrated adaptiveQoS management in middleware: An empirical casestudy. Real-Time Systems, 29(2–3):101–130, March2005.

[9] I.A. Gravagne, J.M. Davis, J.J. DaCunha, and R.J.Marks. Bandwidth reduction for controller area net-works using adaptive sampling. In International Con-ference on Robotics and Automation, 2004.

[10] Shudong Jin, Liang Guo, Ibrahim Matta, and AzerBestavros. A spectrum of TCP-friendly window-basedcongestion control algorithms. IEEE/ACM Transac-tions on Networking, 11(3), June 2003.

[11] K.H. (Kane) Kim. APIs for real-time distributedobject programming. Computer, 33(6):72–80, June2000.

[12] Danny B. Lange and Mitsuru Oshima. Program-ming and Deploying Java Mobile Agents with Aglets.Addison-Wesley, 1998.

[13] Vincenzo Liberatore. Local flow separation. In TheTwelfth IEEE International Workshop on Quality ofService (IWQoS), pages 87–95, 2004.

[14] Vincenzo Liberatore. A play-back algorithm for net-worked control. In Proceedings of the TwentififthAnnual Joint Conference of the IEEE Computer andCommunications Societies (INFOCOM 2006), 2006.

[15] Vincenzo Liberatore et al. Net-worked Control Systems Repository.http://home.cwru.edu/ncs/.

[16] B. Lincoln and B. Bernhardsson. Optimal control overnetworks with long random delays. In Proc. Inter-national Symposium on Mathematical Theory of Net-works and Systems, 2000.

[17] P. E. McKenney. Stochastic fairness queueing. In IN-FOCOM 1990, 1990.

[18] T. Nguyen and A. Zakhor. Path diversity with forwarderror correction (PDF) system for packet switched net-works. In Twenty-Second Annual Joint Conferenceof the IEEE Computer and Communications Societies(IEEE INFOCOM), pages 663–672, 2003.

[19] J. Nilsson. Real-Time Control Systems with Delays.PhD thesis, Lund Institute of Technology, 1998.

[20] Larry L. Peterson and Bruce S. Davie. Computer Net-works. Morgan Kaufmann, 2000.

[21] D. Schmidt, A. Gokhale, T. Harrison, D. Levine, andC. Cleeland. TAO: A high-performance endsystem ar-chitecture for real-time CORBA. IEEE Communica-tions Magazine, February 1997.

[22] W. Richard Stevens. TCP/IP Illustrated, volume 1.Addison-Wesley.

[23] Andrew S. Tanenbaum and Maarten Van Steen. Dis-tributed Systems. Principles and Paradigms. Prentice-Hall, Upper Saddle River, NJ, 2002.

[24] A. Tzes, G. Nikolakopoulos, and I. Koutroulis. De-velopment and experimental verification of a mobileclient-centric networked controlled system. To appear.

[25] Manel Velasco, Josep M. Fuertes, Caixue Lin, PauMarti, and Scott Brandt. A control approach to band-width management in networked control systems. In30th Annual Conference of the IEEE Industrial Elec-tronics Society (IECON04), 2004.

8