software technology for implementing reusable, distributed control

Control systems such as those used in satel-lites, spacecraft, automobiles, chemicalprocessing plants, and manufacturing relyheavily on the software that is used to im-plement them. In fact, engineers at BoeingCo. and Honeywell Inc. have estimated

that 60-80% of the development of a complex control systemis the software development effort, while only 20-40% is theactual control system design. Thesoftware effort includes program-ming individual software modulesas well as writing code for commu-nications between components.Much of the software developmenttime is spent on making thesoftware stable and reliable, such as tracing possible haz-ard conditions and inserting software fault protectionmethods. When the operation of a control system ishighly critical due to human safety factors or the high costof failure in damaged capital or products, the software de-signers must expend extra effort to validate and verify theirsoftware before it can be released. In flight-critical opera-tions, validation and verification are part of the flight certifi-cation process.

Many new software technologies exist that can facilitatethe development and deployment of complex control sys-tems. For example, component-based architectures [1], [2]promote code reuse, thereby decreasing development andvalidation time. Moreover, these architectures encourageflexible “plug-and-play” extensibility and evolution of sys-tems. Distributed object computing allows heterogeneouscomponents to interoperate across diverse platforms and

network protocols [3], [4]. Openstandards for software mean thatproducts from different vendorscan interoperate transparently tothe user. New advances are beingmade to enable dynamic recon-figuration and evolution of sys-

tems while they are still running [5], [6]. New com-munication technologies are being developed to allow net-worked embedded devices to connect to each other and toself-organize [7].

Historically, there has been a disconnect between thecontrol engineers who design the controls and the softwareengineers who implement them. A thorough understandingof the methodologies used in both disciplines would makethe overall process more cohesive. This article gives a

February 2003 IEEE Control Systems Magazine 211053-5888/03/$17.00©2003IEEE

Heck ([email protected]), Wills, and Vachtsevanos are with the School of Electrical and Computer Engineering, Georgia Insti-tute of Technology, Atlanta, GA 30332-0250, U.S.A.

By Bonnie S. Heck,Linda M. Wills, and

George J. Vachtsevanos

JEN

EN

ELS

ON

ATC

NG

RA

PH

ICS

INC

.

tutorial overview of specific new software technologies thatare useful for implementing and reusing complex controlsystems. The focus is on distributed controls, which utilizemultiple processors, particularly those that need to be re-configured either online or offline. This article first dis-cusses technologies for component-based design and reuseof complex control systems, including object-oriented ap-proaches, patterns, and frameworks. It then describes dis-tributed computing and middleware technology forsupporting the composition and interaction among controlsystem components that are distributed. The article alsopresents advances in real-time distributed computing thatare needed to support the real-time demands of control sys-tems in coordinating distributed interactions. Finally, it sur-veys the current technology that is commercially availableand on the research horizon for building reusable distrib-uted control systems. The new software technologies dis-cussed in this article can enhance the performance of theseexisting commercial systems and may form the basis for thenext generation of distributed control systems.

Component-Based ArchitecturesConsider the engineering term component, which means apart or a subsystem in a complex engineering system.Typically, components are designed to be generic enough tobe used in multiple different applications. An engineermight design a system by composing various components,ideally commercial-off-the-shelf (COTS). Similarly, softwareengineers strive to build software components [1], [2],which are modules that can be used in several applicationsand can be composed together to build larger applications.Some may be available as COTS. Components that intercon-nect across a network are called distributed components.

An advantage of software components is that they pro-mote code reuse across many different applications,thereby reducing development and validation time. To con-struct components with this property, a designer must firstassess a range of different applications and then determinegeneric patterns, which are processing needs and computa-tional solutions to them that are similar across all of the ap-plications [8]. The designer then creates softwarecomponents implementing the common solutions.Typically, the components can be specialized further in theindividual applications.

The software architecture of an entire control systemmay be component based. As an example, consider the com-ponents in a standard hierarchical control system: sensors,low-level control algorithms (such as stabilizing controllersin aircraft or set-point controllers in process control),high-level control algorithms (such as supervisory controlor fault-tolerant control), and actuators. Many of these com-ponents have both hardware and software elements (e.g., ahardware sensor usually has software drivers used to trans-mit measurements to a processor). Similarly, there are stan-dard control algorithms, which could be used in many

applications (such as PID controllers, Kalman filters, fuzzy-logic-based controllers, neural network controllers, etc.),that could be coded as software components that are ge-neric and yet can be tuned to a specific application.

Component-based design of control systems has severaladvantages over the current practice of design, especiallyfor streamlining the transition from simulation to implemen-tation. In particular, it makes sense to build a simulation thatcontains separate software modules for each component,including one that represents the plant dynamics. Settingand abiding by standards for the interfaces between thecomponents facilitates rapid prototyping from the simula-tion stage to the implementation stage. For example, thecode for all the controller and signal processing/filteringcomponents would be the same from the simulation to theimplementation, while the component representing the ve-hicle dynamics model would be replaced with the actual ve-hicle. Similarly, the component(s) representing sensormodels would be replaced with the actual sensor driversand hardware. Ideally, the signal processing/filtering com-ponent would not know if the information it is getting comesfrom the simulation code or from the sensor itself. There-fore, it would not need to change if the simulated sensorwere replaced with the actual physical sensor.

As a promising future direction for control system de-sign, suppose software components that model physical de-vices (such as sensors and actuators) are available fromdifferent vendors for their products. If these componentsadhere to specific and uniform standards that govern theirinterfaces, then they may be fully compatible in simulations.Standard control algorithms may also be available that ad-here to these standards. A designer selects likely compo-nents, sets the control parameters, and integrates thecomponents with a simulation of the plant. This type of de-sign process is beneficial in several ways: the initial simula-tion is a more accurate representation of the final imple-mentation, since it includes all the components and thecomponent interactions; the designer can test the perfor-mance of sensors or actuators of different types in simula-tion before making a final selection; and the final systemintegration is streamlined since the software componentscan be replaced with the actual hardware components andtheir software drivers.

Object-Oriented Approachto Reusable Control SystemsSoftware components may be objects (such as a Java appli-cation or a C++ program), or they may be non-object-ori-ented modules (such as C code or legacy Fortran code).Objects contain attributes, called instance variables, repre-senting an object’s state, and operations, called methods,that access and manipulate the state. An object’s state canonly be accessed and changed by using the object’s meth-ods. The methods form a standard interface for an objectthat allows other objects to interact with it. This is a prop-

22 IEEE Control Systems Magazine February 2003

erty called encapsulation, which isuseful for implementing components.The inner workings of the object arekept hidden from the outside world;only the interface is visible. The pro-grammer can make changes and up-grades to the implementation of anobject’s methods without requiringchanges to how other objects are im-plemented as long as the interface re-mains the same. In object-orientedlanguages, an object is defined byspecifying a class, which defines the in-stance variables and methods of theobjects that are instances of the class.For example, in the design of a control system, a sensor classmay be defined as shown in Figure 1, which has methods forinitialization, data acquisition, and filtering. These are func-tions that are commonly used in interacting with a sensorobject. In Figure 1, three additional classes are defined forspecialized types of sensors: global positioning system(GPS) sensors, altimeters, and inertial measurement unit(IMU) sensors. These are related as subclasses to the Sen-sor class, meaning that they share the interface defined forthe Sensor class, but they specialize it in some way. Theymay specialize it by providing a more specific implementa-tion for some of its methods or by adding additional in-stance variables or methods specific to the subclass. Anynumber of object instances can be created for a class. Ourexample may be used, for instance, in a system in whichthere are two altimeters with differing ranges, each an in-stance of the Altimeter class.

The relationships between the classes are graphicallyshown in Figure 1 using a unified modeling language (UML)class diagram. The UML [9] is an object-oriented modelinglanguage that is becoming a de facto standard in the soft-ware industry. The UML provides a set of commonly usedgraphical notations for modeling various aspects of a soft-ware system, including requirements (e.g., use-case dia-grams), temporal behavior (e.g., interaction sequencingand state charts), and implementation (e.g., class dia-grams). In the class diagram, each rectangle represents aclass whose name is at the top and whose methods arelisted in the bottom portion of the rectangle. Instance vari-ables, if any, are included in an additional middle section ofthe rectangle.

The use of subclasses in our example illustrates two keyconcepts in object-oriented programming that facilitate re-use: inheritance and polymorphism. When a subclass, suchas GPS_Sensor, is defined, it automatically inherits themethods defined in the parent class Sensor. This means thatattributes and operations common to a set of subclassescan be defined once and shared efficiently across the sub-classes. An example is the FilterData method, which may beimplemented in the Sensor class as a common first-order fil-

ter to reduce noise in the sensor data. When each of the sub-classes is defined, the FilterData method comes for free inthat it does not need to be redefined for each subclass. Themethod is defined once in the parent Sensor class and re-used by the subclasses.

For flexibility, it is important to be able to tailor or spe-cialize subclasses so that they can extend or adapt the func-tionality of the parent class. In particular, subclasses can betailored by adding new methods and instance variablesand/or overriding the methods defined in the parent class.Thus, a method in the parent class may be implemented dif-ferently in the different subclasses. For example, the specif-ics of initialization may differ from sensor to sensor. So theinitialize method may need to be defined and implementeddifferently in each of the subclasses, as shown in Figure 1.Nevertheless, objects that interact with any sensor objectare able to invoke the initialize method on any type of sensorand the correct implementation of the initialization will betransparently used. This is known as polymorphism, sincethe method can take on multiple different forms for thesame function. The key benefit for reuse is that code that in-terfaces with the sensor class, such as a navigation compo-nent, does not need to be rewritten when a new sensorsubclass is added. As long as the new sensor class satisfiesthe interface, it can be used just like any other sensor. Ob-jects with the same interface can be substituted for one an-other at run time, a concept also referred to as“substitutability.” Inheritance and polymorphism promotereuse by pulling out the generic parts of the code intoclasses while allowing customizations to be made in sub-classes. Other specific examples of object-oriented controlsystem software can be found in [10].

Because of the advantages of object-oriented program-ming, especially that of encapsulation, it is common prac-tice to wrap code around a non-object-oriented(procedural) component to make the component appear tobe an object to the rest of the world. This allows legacy pro-cedural code to be reused in object-oriented systems.

Although object-orientation can increase the reusabilityof a control system, as the software evolves, it can be vulner-able to problems arising from the undisciplined use of im-

February 2003 IEEE Control Systems Magazine 23

Sensor

initialize(double )FilterData( )AcquireData( )...

value

GPS Sensor Altimeter IMU Sensor

initialize(double )AcquireData( )...

value initialize(double )AcquireData( )...

value initialize(double )AcquireData( )...

value

Figure 1. Example class hierarchy for sensor objects.

plementation inheritance. If a parent, or “base,” class ischanged, it might have unexpected consequences for inde-pendently created subclasses that reused aspects of thebase class. This is called the fragile base class problem [2].Implementation dependencies due to subclassing can limitflexibility and reusability in new domains, since any part ofthe inherited implementation that is not applicable in thenew domain could require changes to the base class or re-structuring of the class hierarchy [8]. One way to avoidthese problems is to constrain the design to inherit onlyfrom abstract classes that have no method implementa-tions (only interfaces). This is done in object-orientedframeworks, which are described in the next section. An-other common approach is to use object composition in-stead of class inheritance to reuse functionality. Objects arereused by assembling them into more complex objects; atrun time, objects obtain references to other objects and for-ward (or “delegate”) messages to them to perform support-ing functionality. This requires that the interfaces of theobjects be well defined and general enough that each “part”object can be composed with many others. The advantageis that the objects can be changed and replaced independ-ently without affecting the objects with which it is com-posed as long as the new object adheres to the interface. Inpractice, the components typically available to a designerare never exactly what is needed, so it is common to use in-heritance along with object composition to allow flexibilityand maximize reuse [8].

Composing Components:Enhancing Reuse with FrameworksMost commercial tool environments for building controlsystems make available repositories of prebuilt andpreverified reusable components (such as common types ofPID controllers, filters, and matrix computations) for effi-cient construction of systems. This is an important type ofreuse. However, there are tremendous opportunities for re-use at an architectural level as well. For example, commoncontroller configurations are encountered and used timeand time again (e.g., the common sensor-controller-actua-tor feedback loop, feedforward loops, and common evalua-tion mechanisms that compare expected and actualenvironmental factors). Similarly, common patterns [8] forconstructing dynamically reconfigurable systems are start-ing to emerge, particularly in the hybrid controls area. Forexample, hybrid systems often contain a common patternfor gradually transitioning from one control strategy to an-other [11], [12]. It may encapsulate a strategy dictating howquickly one algorithm can be switched for another orwhether a redundant component needs to work concur-rently with the component it is replacing before the swap oc-curs to allow the new component to “come up to speed” ortransfer state. If generalized and made available for reuse,this pattern can be specialized with user-specified decisionlogic for managing the reconfiguration and applying the ap-propriate blending strategies to smooth the transients dur-ing the transition.

What is reusable in these cases are the “skeletons” of thedesign—how the components are composed or integrated.For example, the generic part can be implemented by an ob-ject-oriented framework [13], which is a set of abstractclasses. An abstract class has a standard interface but con-tains no concrete object instances or method implementa-tions. These abstract classes form a framework that can beapplied to a particular application by adding concrete sub-classes that specialize the abstract classes and by plugging inapplication-specific method implementations. Reusethrough object-oriented frameworks has several advantages,such as reducing development costs and design errorsacross multiple similar applications, increasingunderstandability and maintainability of the systems, and fa-cilitating system evolution. A key idea behind designing forreuse using frameworks is that the designer explicitly identi-fies the places where the design is anticipated to change inthe future. These have been called hot spots [14] and plug-inpoints [15] to express the idea that the design is built to with-stand evolution in predefined ways. By performing a domainanalysis [16] of a family of related control system designs, it ispossible to identify the common, generic, reusable structureof each mechanism and to “encapsulate the variation” [8]that is expected. The places where the code can be varied (orspecialized to a specific application) are localized to the con-crete methods that the application engineer is expected towrite and plug into the framework. Examples of control-spe-


See the following Web sitesfor more information.

OSACAwww.mel.nist.gov/proj/interop.htm

Object Management Group (OMG)www.omg.com

QNXwww.qnx.com

VxWorkswww.windriver.com

Real-Time CORBA (TAO)www.cs.wustl.edu/~schmidt/TAO.html

Mathworkswww.mathworks.com

RTIwww.rti.com

Sun Microsystems Javajava.sun.com

cific frameworks include the AOCS Framework (developedfor satellite control systems) [17], Realtime Framework (de-veloped for robotics and automation) [18], NEXUS (for robot-ics) [19], and OSACA (for automation) [71].

One of the most critical active areas of research in com-ponent-based reuse is how to compose components in sys-tems that have stringent resource and dependabilityconstraints, such as high performance, reliability, availabil-ity, consistency, survivability, timeliness, and security [20].The loose coupling that is needed for flexible reuse andreconfigurability favors using composition mechanismsthat decouple interaction. This may be done through asyn-chronous or anonymous communication mechanisms, suchas using a shared data space in a blackboard architecture orusing a publish/subscribe model (described in the next sec-tion) [21]. At the same time, it is critical that resource anddependability constraints be enforced in the use of these in-teraction mechanisms. How to specify and verify these con-straints is an important research problem. Failure to dealwith these constraints in the composition of components(not just enforcing them on individual components) canlead to unpredictable behavior and resource conflicts, suchas frame overruns [21].

Recently, a broader and more powerful notion of frame-works has been defined [22] to encompass not only compo-nents and how they are composed but also the set ofconstraints in their interaction and the benefits resultingfrom these constraints. A framework in this definition speci-fies rigorously the types of components integrated, theknowledge they have of one another, protocol mechanisms,and the vocabulary used in component interaction (e.g.,possible message types). The advantage of this richer no-tion of frameworks is that it may enable distributed controlsystem architectures to self-adapt dynamically and robustlyas changes in the networked environment occur and as newcomponents are plugged into the system at run time. Thisprovides a rich context in which to specify timing and re-source constraints so that they can be enforced as an inte-gral part of the design process.

Design for reuse by factoring out similarities across ap-plications and identifying generic components is essentialin reducing the complexity and cost of developing controlsystems. In today’s embedded controls applications andpervasive computing environments, it is often the case thatthese components are distributed (both geographicallyacross different processors and logically across differentprocesses in a multitasking environment). Distributed com-puting software technology has been developed to supportthe composition of and interaction among components thatare distributed. This technology is described next.

Distributed Computing OverviewComputer systems are typically set up using one of four dif-ferent organizations. The system may have one very largecentralized machine (such as a mainframe) with smart ter-

minals that can be used to access the data or functions onthat machine. While most companies and universities em-ployed this organization during the 1970s and 1980s, the useof centralized processing has fallen off dramatically. An al-ternative approach to centralized processing is that of com-pletely decentralized processing, where a large number ofsmall units (such as desktop machines) independently per-form operations with very little interaction. This provides agreat deal of flexibility for individual users but makes datasharing difficult. A third possible organization is hierarchi-cal, which integrates relatively similar small-scale unitsthrough a centralized machine, often called a server, thatperforms many higher-level functions. This organization iscommonly employed in university computer clusters. It isalso used in Web applications where the Web server is thecentralized machine and the clients are the remote ma-chines that log into the Web server to access data.

The fourth organization for computer systems is distrib-uted. In this organization, each of the computers performsfunctions vital to the overall operations and is integratedwith the other machines in the system [23]. It is differentiatedfrom decentralized processing in that the machines are moretightly coupled to complete higher-level tasks. It is a flatter ar-chitecture than the hierarchical since the higher-level func-tions are performed by the individual processors and not bya high-level, centralized machine. Distributed processing canbe considered a form of parallel processing, except that theprocessing can be distributed over separate computers, notjust separate processors. An example from the businessworld is one in which a company has one large-scale com-puter for marketing, one for inventory (including shippingand receiving), and one for accounting. Clearly, all of thesemachines would need to perform distinct operations, and yetthey need to be integrated to share data. A customer placingan order on the system may not realize that he/she is usingthree different computers that are working together to exe-cute the order. Distributed computing is more robust than us-ing centralized or hierarchical architectures since onemachine can go down without necessarily disrupting the op-erations being performed on the other machines. Also, up-grades can be performed in a more modular fashion and thuswill be less disruptive and done in smaller chunks. Distrib-uted computing has gained acceptance, primarily due to ad-vances in communication networks that make the transfer ofdata fast and reliable and due to advances in creatingmiddleware, which hides low-level details about networkprogramming. As a result, distributed architectures are nowthe most common for computer systems in businesses, uni-versities, and government.

Distributed computing is ideal for complex control sys-tems as well. These systems typically have multiple pro-cessors, with each performing distinct functions thatneed to be integrated. An example of distributed comput-ing for the controls of an uninhabited aerial vehicle (UAV)is shown in Figure 2. There may be multiple processors on


board the vehicle: one for avionics (containing sensor andactuator drivers and sensor processing), one for low-levelcontrol function (such as stability augmentation), and onefor high-level control functions (such as trajectory planningand fault detection). The mission control processor is lo-cated remotely, either on the ground or in another vehicle(perhaps the mother ship). The onboard processors areconnected via a LAN, while the remote processor is con-nected via a wireless connection (such as Ethernet for closerange or serial for longer range). See [24] for details of an ex-ample architecture for UAV flight controls.

Network CommunicationsIntegrating software components (or, more generally, appli-cation software) over a network requires extensive knowl-edge about data transfer between software applications andacross networks. Consider first the task of setting up a com-munications network. In hardware, the ports and the com-munication lines must be connected. In software, thesockets (which are software analogs of ports) must be con-figured, a software interface for opening and closing socketsmust be written, and the communication protocols must beset, including routing, the method to break up the data intopackets, and error checking. Ethernet is a common networksolution for high-speed, high-volume communicationamong a large number of nodes (that is, machines hooked tothe network). A problem with using Ethernet for control ap-plications is that the data transfer rate and the size of pack-ets are not deterministic [25]. As a result, there is noguarantee of arrival time for data packets and no guaranteeof information flow rate. If a control system closes the loopover a network, then this nondeterminism gives rise tostochastically varying time delays. Or if high-priority calls(such as fault isolation and reconfiguration strategies) weredelayed over the network, then the system’s health may beat risk. Protocols are being developed to attack some ofthese issues [26].

An alternative to Ethernet is to use a LAN with protocolsthat are better suited to smaller, more frequent data packet

transfers found in control applications. Often calledfieldbus systems, these control networks include ASI,BITBUS, CAN, DeviceNet, Hart, IEC-Fieldbus, InterBus-S,LonWorkds, Pi-NET, Profibus, Rackbus, Secros, SDS, TON,and WordFIP [27]. CAN (control area network) is usedwidely in automotive applications and is an event-drivenprotocol; that is, it assumes that data packets are sent in asporadic manner, and it has a decision structure for decid-ing the route and the priority for transmitting different pack-ets. In contrast, time-driven protocols (TTPs) set specificschedules for transmitting different data and nominally re-peat these schedules on a periodic basis. In a comparison,Kopetz [28] determined that time-triggered protocols arewell suited for safety-critical systems with hard real-time de-pendability constraints (by making worst-case timing as-sumptions), and CAN is more suitable for soft real-timesystems where average-case performance and flexibility arethe primary design considerations. A comparison of Ether-net (with CSMA/CD), a CAN, and a token ring network forcontrol applications is given in [29].

Numerous communication protocols exist for dedicatedcontrols-related LANs, as mentioned earlier; however, thereis a need to develop an Internet-based protocol that is wellsuited for controls applications. Although a great deal ofwork is being done to develop protocols tailored for multi-media applications over the Internet, the same is not truefor Internet-based protocols for control systems. For exam-ple, asynchronous transfer mode (ATM), which uses fixeddata packets, making it more deterministic than protocolsused with Ethernet networks [25], was first established tobe efficient with streaming video and audio as well as stan-dard data communications. Other work in the multimediacommunity aimed at making streaming audio and video reli-able and steady has produced communication solutionssuch as voice over IP or Firewire. Some of these multimediasolutions may work their way into controls applications, butthere could be work aimed at making a WAN protocol de-signed specifically for control systems. In fact, this sugges-tion was made at the recent NSF Workshop on FutureDirections in Control Systems [30]. As hinted by this briefdescription of network protocols, programming at the net-work level requires detailed knowledge about networks thattypical control engineers lack. Learning enough to imple-ment even a simple control law takes considerable time.This issue is being addressed by the use of distributed com-puting middleware, as described next.

MiddlewareMiddleware is software that provides a substrate throughwhich software components can communicate with eachother [2]. It sits between the operating system and the appli-cation software and transparently handles many mundanelow-level details generally required for data transfer be-tween applications and over the network. As such, the useof middleware for integrating a networked control system


On-BoardProcessors

Low-Level Control Processor

High-Level Control Processor

SensorsActuators

R-1 Integrated Avionics System

LAN

WirelessConnection

MissionControlStation

Figure 2. Distributed computing architecture for an uninhabitedaerial vehicle.

would substantially decrease the complexity of program-ming, since the low-level network details are hidden fromthe controls engineer.

Middleware can be used when the application software iscolocated on the same processor (local) or on a distributedcomputing system linking many processors across a net-work (remote). For example, consider the middleware usedon a Microsoft Windows-based desktop machine that facili-tates the interaction of different software applications run-ning in different windows. As long as vendors adhere tocertain standards, their application software is generallycompatible with other software running on the platform.The programs can be run at the same time, and data can bepassed from one program to another. A popular middlewareused for desktop units is Microsoft’s Component ObjectModel (COM) [23]. Distributed Component Object Model(DCOM) is a version of this middleware used to integrate ap-plication software running on different Windows machinesand connected via a network.

Application-specific middleware can provide good exam-ples of code reuse and show the benefits of compo-nent-based architectures. For example, avionics systemshave numerous sensors and measurement devices thatneed to be integrated. Rather than build completely special-ized avionics packages for each aircraft, the code is sepa-rated into parts that are common among all the aircraft andparts that need to be specialized for different types of air-craft. The communication software is something that can bemade common across all applications. For this reason, Boe-ing has developed a middleware that can provide a commoncommunication substrate for all of their measurement de-vices [31]. Thus, design of a new avionic system starts withthis middleware, which is then customized by adding com-ponents for the particular devices used for a particular air-craft. The overall software development time is reduceddramatically by reusing code across different applications.

To be effective for use in distributed computing, middle-ware should have certain properties: itmust be able to automate common net-work programming tasks, it must havea standard interface so that all applica-tion software that uses the standardcan be integrated easily, and it mustsupport interoperability in heteroge-neous environments. Interoperabilityallows application programs that arewritten in different languages or fordifferent target operating system andhardware platforms to work togetherseamlessly. The ultimate goal for con-trol systems would be to use middle-ware that has plug-and-play capability.For the sake of adaptability, it wouldbe helpful to support run-time recon-figuration of the software components

(such as updating or changing algorithms online in realtime).

While COM (and DCOM) are popular for desktop comput-ing applications, they lack the interoperability needed forintegrating software between machines that are not Win-dows based. Data transfer between different machine plat-forms is not trivial; for example, the memory byte orderingis different between Windows-based platforms and Unixplatforms, and word size differences on 32-bit versus 64-bitprocessor architectures cause incompatibilities. Two othercommon types of middleware are CORBA (common objectrequest broker architecture) and Java-based middleware.

CORBA is a software standard developed by a consor-tium called the Object Management Group (OMG). As of thiswriting, there are 800 member companies of OMG that de-velop and adopt products that abide by the standards setforth by OMG. A basic feature of CORBA is the object re-quest broker (ORB), which handles remote method calls. Asillustrated in Figure 3, when an object (such as a navigationsystem) calls a method (e.g., MeasureAltitude) of anotherobject (such as a remote sensor) distributed elsewhere onthe network, the ORB intercepts the call and directs it. Theclient object does not need to know the location of the re-mote server object, a principle known as location transpar-ency that greatly simplifies the programming of distributedapplications. The way this works is that a programmer spec-ifies an interface for each object using a standard interfacedefinition language (IDL). These interfaces are compiledinto what are referred to as client IDL “stubs,” which act asproxies for the actual objects, and server object IDL “skele-tons.” All components are registered with and interactthrough the ORB. When a client object invokes a method(e.g., MeasureAltitude), it does so as if it is making a methodcall to a local object, but it is actually invoking it on a proxythat is the IDL stub. The method call goes through the ORB,which locates the server object (e.g., the Altimeter). If theserver object is remote, the ORB needs to send the request


Navigator(Client)

MeasureAltitude(x) Altitude9. Client

ReceivesReturn Value

1. Client SendsRequest

IDL Stub

ORB...11010...

Network

IIOP Protocol

3. Marshals Parameters 4. Unmarshals Parameters8. Unmarshals Return Value 7. Marshals Return Value

2. ORB Locates Server

Altimeter(Server Object)

MeasureAltitude(x);

5. InvokesMethod

6. ServerReturnsAltitude

IDL Skeleton

ORB...11010...

Figure 3. Distributed communication through the CORBA standard.

to the remote object’s ORB over the network. This involvesmarshalling the parameters, which means translating datavalues from their local representations to a common net-work protocol format, such as the Internet Inter-ORB Proto-col (IIOP) standard defined by OMG. On the server objectside, the parameters are demarshalled and the method in-vocation is passed to the IDL skeleton, which invokes themethod on the actual server object. The server returns therequested value in a similar fashion through the ORB. Notethat the client does not have to be aware of where the serverobject is located, its programming language, its operatingsystem, or any other system aspects that are not part of anobject’s interface.

The use of CORBA in control engineering is rather new;see [31]-[33] for details on the development of a platformthat is based on a CORBA substrate. However, some of thehigh-level features of CORBA make it very attractive for fu-ture control engineering products used to integrate sys-tems. Its main benefits are that it has a well-accepted stan-dard for component interfaces, it allows for interoperabilityof software components written in a range of different lan-guages (including Java and C++) and/or running on differentplatforms, and it handles all the network communicationprotocols to seamlessly interconnect multiple componentswithin and across control systems. Components can also beintegrated that are not object oriented (such as legacycode) by wrapping the code in a layer that provides an ob-jected-oriented interface. Services are continually beingadded to the ORB structure that extend the capabilities ofCORBA.

The third common way of integrating components is withJava-based technologies. Java is an object-oriented lan-guage that is platform independent, so it has good promisefor use in middleware development. Java is based on a lay-ered architecture, with new open-source APIs being devel-oped on a continuing basis. Some of the APIs that enableJava to be used for distributed applications are the Enter-prise Java Beans API, the Remote Method Invocation (RMI)API, the RMI over IIOP API, and the Java ORB API. Java Beansis a way to make software components called Beans out ofstandard Java applets or applications (that is, unlike ap-plets or applications, which cannot interact with one an-

other, Beans can be composed together to form a larger ap-plication). The Enterprise version allows these Beans to ex-ist on different machines and to be connected across anetwork. The RMI API provides the means for distributedobjects to interface through remote method calls (similar toremote procedure calls but specialized to Java objects). TheRMI IIOP API provides the remote method call capability us-ing a CORBA ORB. The Java ORB API provides a Java-basedCORBA ORB. It should be noted that Java components caninteract with non-Java components (such as those writtenin C) through the use of the Java Native Interface (JNI). JNIworks by providing a standard for writing Java code to wraparound non-Java code; the combined code then can be

treated as a Java object.Although many of these Java APIs

provide services useful in middle-ware, more complete packages areavailable such as the Jini Technology,J2EE (Java 2 Platform Enterprise Edi-tion), and J2ME (Java 2 Platform MicroEdition). Jini is a software architecturefor integrating hardware over a net-work. The real strength of Jini is thediscovery service, whereby new de-vices can be added to the network atrun time and be recognized and regis-

tered. J2EE is a component-based technology useful for inte-grating business applications. Both Jini and J2EE have theoption of performing remote method calls using the Java RMItechnology, CORBA services, or the Extensible Markup Lan-guage (XML) [34]. Thus, these technologies can be used ascomplementary technologies to CORBA. J2ME is an opti-mized Java virtual machine meant for running on small em-bedded platforms, and it uses the Java RMI technology. Thereis a strong continuing effort in the Java community to buildsupport for real-time applications of Java, such as theReal-Time Specification for Java [35] and the requirementsfor Real-Time Java [36]. For example, a version of J2ME isavailable for VxWorks, a commercial real-time operating sys-tem developed by Wind River Systems. Since this fieldchanges rapidly, the reader is encouraged to visit the Sun Mi-crosystems Java Web site (java.sun.com) for updates.

A competing technology for J2EE is Microsoft’s .NET [38],which is a framework that supports Internet-based connec-tivity between machines by providing the communicationsmiddleware for doing remote procedure calls (as well asother services). For communications, .NET relies on SOAP(simple object access protocol), an XML-over-HTTP proto-col [37]. There is some question as to the suitability of HTTPfor real-time applications with the additional overhead ofXML [38]. Another drawback is .NET’s reliance on theMicrosoft Windows operating system.

Middleware is generally constructed to provide commu-nication between application software that adheres to oneof the following communication models:


The software engineering conceptsintroduced make implementingcontrol systems more flexible so thatthey can be dynamically reconfiguredwith ease and can be upgraded oradapted in a flexible manner.

• Point-to-point (or peer-to-peer): In this model, all com-munication is one-to-one between pairs of entities. Acommon example for this type of communication is atelephone call between two parties.

• Client-server: Client-server is a many-to-one communi-cation model where many clients can be connected si-multaneously to one server. The server acts as acentralized source of data.

• Publish-subscribe (or producer-consumer): The pub-lish-subscribe model allows many components to in-teract with one another in a many-to-manyarchitecture by introducing an intermediate level be-tween the components that mediates the communica-tions. Components can be designated as publishers,which produce data, and subscribers, which requiredata as input. An example of a publisher is a sensorthat produces measurements. An example of a sub-scriber is a control algorithm that requires the mea-surements from the sensor. In a complex controlsystem, there may be many publishers and many sub-scribers. For example, a process control system mayhave distributed components using many processors,or at least many separate algorithms all using similardata, and numerous sensors producing data.

As an example of these models, consider the architecturefor implementing a control system for an uninhabited aerialvehicle, as shown in Figure 4. The control components arethe software modules that perform the tasks of missionplanning, target tracking, obstacle avoidance, and attitudecontrol. The sensor components consist of the device driv-ers as well as the sensor hardware. In this case, the sensorsused are a camera for vision information, a forward-lookinginfrared (FLIR) camera for nighttimevision, a GPS sensor, and an IMU. Thearchitecture shown in Figure 4(a) usesa point-to-point model where the com-munication is between pairs of compo-nents. As a result, there is a tightcoupling between components that isachieved by hard-coding calls betweencontrol components and sensor com-ponents. Such tight coupling of soft-ware components has been standardpractice in control implementationdue to the efficiency gained at runtime, but it makes upgrades and evolu-tion of software difficult. For example,suppose the FLIR sensor is added at alater time than the other three sensors.Then adding this sensor requires mod-ifying three different control compo-nents, which all require image data.

Now consider the same example im-plemented using a publish-subscribemiddleware as shown in Figure 4(b).

Here the middleware provides a communication abstrac-tion called an event channel that acts as a mediator betweencomponents. The vision component registers with theevent channel as a publisher of image data, while the mis-sion planning, target tracking, and obstacle avoidance com-ponents are subscribers to that data. This removes the needto hard-code a call from a specific subscriber to a specificsource of data in order for components to interact. With theevent channel, the interconnection structure between com-ponents is much more flexible. Subscribers and publishersregister with the event channel to receive (or provide) dataat a given rate. Subscribers get the data from whatever pub-lishers are currently registered to provide that data. If a newsensor (e.g., an FLIR) is added, then it can replace the oldsensor by registering the new sensor driver with the eventchannel as a publisher. Other publishers of the same type ofdata may be removed by changing their registration. Noother changes to the interconnection structure betweencomponents are necessary.

Multiple publishers of a particular type of data can beregistered. The event channel must have strategies for de-termining which data to send to the subscribers of that data.One strategy used by a real-time publish-subscribemiddleware, NDDS, is to give each publisher a weight, andthe middleware forwards whichever data set comes fromthe publisher with the highest weight [39], [40]. (Theweights can be reset in real time.) Other strategies exist fordetermining an average of the data from the different pub-lishers [41]. Thus, the event channel is a means to facilitatecommunication between components in a flexible enoughmanner that is easily adjusted to accommodate evolution ofthe system software. CORBA and DCOM use a client-server


MissionPlanning

TargetTracking

ObstacleAvoidance

AttitudeController

Vision FLIR GPS IMU

(a)

(b)

MissionPlanning

TargetTracking

ObstacleAvoidance

AttitudeController

Vision FLIR GPS IMU

Image Data Image Data Image Data, Navigation Data Navigation Data

Image Data Image Data Navigation Data Navigation Data

Event Channel

Figure 4. Decoupling components through the publish-subscribe communication model.(a) In the point-to-point model, adding an FLIR component requires many changes. (b) In thepublish-subscribe model, adding an FLIR component requires local change.

model, but an event service has been layered on the CORBAspecification that emulates a publish-subscribe model.

In summary, distributed computing middleware pro-vides the ability to automate common network program-ming tasks, provides a standard interface for componentintegration, and facilitates interoperability across differentmachine platforms and networks. Complex control systemsthat require distributed interaction among diverse reusablecomponents can benefit from this type of technology. How-ever, it is also critical that the interactions among compo-nents in these systems be coordinated in real time. The nextsection discusses new software technology that supportsreal-time interactions among distributed components.

Real-Time ComputingThe essence of real-time systems is that they are able to re-spond to external stimuli within a certain predictable pe-riod of time. Building real-time computing systems ischallenging due to requirements for reliability and effi-ciency, as well as for predictability in the interaction amongcomponents. Real-time operating systems (RTOSs) such asQNX and VxWorks facilitate real-time behavior by schedul-ing processes to meet the timing constraints imposed by theapplication.

Control systems are among the most demanding ofreal-time applications. There are constraints on the allow-able time delays in the feedback loop (due to latency and jit-ter in computation and in communication), as well as thespeed of response to an external input such as changing en-vironmental conditions or detected faulted conditions. Ifthe timing constraints are not met, the system may becomeunstable. This differs from multimedia applications, for ex-ample, where the user would at worst be irritated by thepoor quality of an audio transmission or a jerky video or ani-mation sequence.

These stringent timing constraints are particularly chal-lenging to meet when used in a multitasking environment.Multitasking is useful in implementing multirate control andfor allowing concurrent component execution in a complexsystem, such as enabling a fault detection module to runconcurrently with a nominal control module. Since themultitasked processes must share CPU time, the corre-sponding tasks must be scheduled. Typically, such pro-cesses are scheduled statically assuming periodicoperation, say τ ms per computation cycle. In this strategy,the worst-case execution time (WCET) is calculated for eachprocess. That amount of time is allotted for that processduring the τ ms computation cycle (or frame). Since WCET isvery conservative, this strategy can result in a small utiliza-tion of the CPU, thereby wasting computing resources. Theclassic technique for static real-time scheduling commonlyused in real-time operating systems is rate monotonicscheduling (RMS) [42]. It is a fixed-priority scheduling tech-nique in that it assigns each process a fixed priority basedon how frequently the process needs to run (i.e., its rate). At

run time, the static scheduler always chooses the processwith the highest priority to run at any given point, preempt-ing the running process if necessary [43].

Scheduling may be performed based on event triggers oron time triggers. Event triggers are external stimuli, such asoperator inputs, that introduce interrupts which are han-dled as soon as they occur, based on priority. Loop rates canbe enforced in this manner by using a hardware timer to sup-ply periodic interrupts that are handled by sampling thesensor data and computing the corresponding control sig-nal. The processing must be completed before the nexttimer interrupt; this time period is often described as aframe or as a computation cycle. Other interrupts, such asan operator input, may occur during any frame and must behandled using predefined logic on priorities. An alternativearchitecture is one based on time triggers rather than exter-nal event triggers. In this case, the frame is subdivided intotime periods for different tasks to be completed based onthe WCET. The tasks are scheduled at the beginning of eachframe. External interrupts are handled by waiting until thebeginning of a frame and then scheduling the correspondinghandling process to be completed during the next frame,making the process more predictable than those producedby event triggers. This is an advantage for periodic pro-cesses such as control systems that have strict rate require-ments. In a comparison of time-triggered architectures toevent-triggered architectures, Kopetz [44] concluded thatevent-triggered architectures are better suited tononperiodic processes that require flexibility in structure(e.g., to accommodate reconfiguration of the control),whereas time-triggered architectures are better suited toperiodic tasks that have strict requirements on their rates(including prediction). In particular, time-triggered archi-tectures may be less efficient in that they schedule pro-cesses based on WCET. However, if there are manyinterrupts, event-triggered architectures become less effi-cient because of the added overhead needed for interrupthandling. If events are very sensitive to latency (such as re-action to faulted conditions), then event-triggered architec-tures may be preferable. The alternative is to make theframes in time-triggered architectures smaller to shortenthe time to react to interrupts, but this reduces efficiencydue to the overhead in scheduling needed at the beginningof each frame.

To improve processor utilization, several researchersemploy dynamic scheduling algorithms [3], [45], [46] foruse in real-time systems. Dynamic scheduling allowsrun-time changes in the control system architecture whilemaintaining real-time operation. Since a fixed schedule isnot used, higher utilization of resources can be achievedwhen nonperiodic tasks are used. While dynamic schedul-ing is common on desktop machines, the timing constraintsfor real-time systems are more challenging. The concept ofquality of service (QoS) has been borrowed from networkcommunications and applied to dynamic scheduling for


real-time systems. In this application, QoS resource require-ments are defined for each task. These typically include pa-rameters such as update rates for different signals,priorities in case two events occur simultaneously (such asa fault occurrence flag having higher priority than a nominalcontrol update signal), WCET, and typical execution time(i.e., the amount of time a task usually takes). The dynamicscheduling algorithm uses these parameters to determinethe order in which to start tasks so that they have the com-putational resources they need to complete within theirdeadlines or at their required rates.

Figure 5 shows examples of different schedules createdusing two popular dynamic scheduling techniques: earliestdeadline first (EDF) and minimum laxity first (MLF). In EDF,the task with the earliest deadline is scheduled first [47]. InMLF, the task with the least “laxity” is scheduled to run first,where laxity is the difference in the time remaining before atask’s deadline and the task’s execution time [48].

Run-time reconfiguration of a control system may re-quire changing the QoS parameters in real time so that tasksthat might have previously been running at a lower prioritythan others can be placed at a higher priority to respond toa failure or some unexpected condition. For example, an ae-rial vehicle’s navigation component may need data at ahigher rate, quality, and priority during an extreme maneu-ver than during other types of maneuvers, such as hovering.QoS parameters may also need to be adjusted due tochanges in the system resources, such as the addition ofnew processes, and due to computer system failures.

To support these needs, new dynamic scheduling algo-rithms [4], [45], [46] and adaptive resource managementtechniques [49]-[52] are being developed. Adaptive re-source management allocates processor and network re-sources based on changing client demands and based onchanges in available system resources. Feedback mecha-nisms have been employed by several researchers to adjustscheduling of resources to ensure that the actual QoS pa-

rameters are maintained within acceptable QoS parameterregions; see, for example, the control-based middleware de-veloped by Li and Nahrstedt [53] and Honeywell’s adaptiveresource management system (RT-ARM) [49], [50]. Some ofthese methods have been integrated by Boeing as part ofthe open control platform (under the DARPA Software-En-abled Control program) [31], [32]. By using dynamic sched-uling algorithms in the event channel scheduler, run-timechanges can be made in QoS requirements for events by al-lowing QoS parameter changes to be specified and enforcedat run time.

Combining dynamic scheduling and adaptive resourcemanagement enables a particularly valuable class of algo-rithms, called anytime algorithms or imprecise computa-tion [54], [55]. These are continually running algorithmsthat can produce results at any time but continually refinetheir results as time passes and the system state changes.Anytime algorithms are valuable when dealing with variableamounts of available computational resources. A feature ofthese algorithms is that they monotonically converge on asolution. If they are given much processing time, they con-verge to a very accurate solution. Given a smaller amount ofprocessing time and/or resources, they provide a morecrude, but acceptable, answer. This contrasts with othertypes of algorithms that would not be able to produce anyresult if their execution time is reduced. Examples of con-trols applications in which these algorithms could be usedare route planning and failure prediction, particularly thoseusing iterative methods such as genetic algorithms or meth-ods that gradually improve as they learn, such as neural-net-work-based techniques.

An alternative approach, but one that also uses adaptiveresource management, is to allow the scheduler to switchcomponents to ones that are less demanding when it deter-mines that there are insufficient resources to use the nomi-nal component. Honeywell employed this technique in con-junction with RT-ARM for multiagent coordination [54].

Honeywell developed a multiresolutiontrajectory planning algorithm withvarying levels of computational needs.The algorithm is useful in its most de-manding form, which gives the best re-sults, under nominal conditions, whilethe least compu- tationally demandingform, which gives only crude results, isused when there are diminished re-sources.

When the processing is distributedover multiple machines, the influenceson real-time performance becomemuch more complex. For example, indetermining the latency of a remotemethod call made across a network,the communication latency must beadded to the processing latencies in-


Earliest Deadline First (EDF):

Minimum Laxity First (MLF):

C A D B

A C D B

100 110 120 130 140 150 160 170 Time ( s)µ

100 110 120 130 140 150 160 170 Time ( s)µ

Task A: <Execution Time: 15 s Deadline: 125µ , >Task B: <Execution Time: 15 s Deadline: 175µ , >Task C: <Execution Time: 5 s Deadline: 120µ , >Task D: <Execution Time: 30 s Deadline: 155µ , >

Figure 5. Example schedules from two well-known dynamic scheduling algorithms.

herent in the participating processors. Therefore, the net-work communications QoS must be coordinated with theindividual processor QoS. Static scheduling, which is done apriori using worst-time execution time and worst-casetransport delays, is common practice. The time-triggeredarchitecture must rely on the notion of a global clock whenapplied to a distributed system, so that the timers on all theprocessors are synchronized to within a certain small errorof one another [44]. Dynamic scheduling for distributedprocesses is currently an open research topic due to thecomplexity of interactions between the communicationscheduling and the processor scheduling; see, for example,[56] and [57].

Methods exist to reduce communication latency in dis-tributed systems. In particular, there are numerous meth-ods for scheduling communications to reduce latency. Forexample, a dynamic scheduling algorithm for communica-tions is proposed in [58]. Another method that may be usedto reduce latency in a real-time distributed system is replica-tion [44], [47], which is normally used to provide fault toler-ance through redundancy in a computational system butprovides efficiency advantages as well. Replication refers tothe practice of creating copies (or “replicas”) of objects atmultiple computational nodes. These nodes must be syn-chronized with respect to time and data to maintain consis-tency between objects and their replicas. If a replica of anobject is available locally, then a remote method call may beredirected to the local replica, thereby avoiding the commu-nication delay. There is a tradeoff, of course, between main-taining replica consistency and reducing communicationdelays in that the use of replication may result in too muchextraneous network traffic if the replica update rate is toohigh compared to the rate at which the replica is accessed.

A specification for Real-Time CORBA [4] has been re-leased by OMG that defines standard features for managingprocessor utilization, network, and memory resources.These standard features support end-to-end predictable be-havior for fixed-priority operations (i.e., operations withprespecified static priorities) between CORBA clients andservers. By defining standard policies and capabilities forresource management, the Real-Time CORBA specificationhelps ensure QoS predictability for fixed-priority real-timeapplications. An open-source implementation of Real-TimeCORBA, called TAO, is publicly available at http://www.cs.wustl.edu/~schmidt/TAO.html.

Commercial Productsand Market TrendsCommercial software tools for control systems are aimed ateither design and analysis or at implementation. The mostcommon design and analysis tool is MATLAB from TheMathWorks. MATLAB is often used in conjunction with theSimulink package to provide a block-diagram-based graphi-cal user interface along with some expanded simulation ca-pabilities. To ease system implementation, the Real-Time

Workshop toolbox can be used to generate C code automat-ically from a Simulink block diagram. The MathWorks isworking toward making the resulting code more efficient, interms of both processing time and size (an important fea-ture for embedded processors). Another MathWorks prod-uct, Stateflow, which is well suited to hybrid systems, alsohas an autocode generator that can be purchased as an op-tion. Unfortunately, MathWorks products tend to generatemonolithic code rather than component-based code. Thismakes it more difficult to validate or update the code. As analternative, designers can use MATLAB to generate individ-ual portions of the code, which can then be made into com-ponents. For example, the designer can obtain the code for alow-level controller operating under nominal conditions, aswell as separate code for a controller operating underfaulted conditions. The designer can then integrate theseseparate components using middleware.

Distributed control systems have been available com-mercially for many years. The majority of these systemsconsist of industrial-grade processors or PLCs connectedto hardware devices by dedicated fieldbus networks (CAN,DeviceNet, etc.). The applications are in process control,environmental control, automotive control, mechatronicsystems, and power systems, to name a few. This market isdominated by a few large corporations such as RockwellAutomation (including the subsidiary Allen-Bradley),Honeywell, Siemens, Invensys, and ABB, which supply97.5% of the market [59]. These control suppliers providethe entire system: the measurement and actuator devices,the processors, the network interfaces and cables, and ser-vice. Very few third-party vendors (suppliers of special-ized components such as sensors) have broken into thismarket because the interfaces to these networked systemsare proprietary.

The desire for open standards has prompted the in-creased use of PC-based controllers, which has become thelargest growth area in the distributed control system market[59]. PC-based controllers are cheaper than the indus-trial-grade processors and have a more open architecture.This open architecture means that third-party vendors areable to supply more of the components, resulting in largecontrol houses having a smaller market share for PC-basedcontrol (75% as compared to 97.5% in the standarddistributed control system market). Most control housesnow offer interface cards for PCs to connect to their stan-dard fieldbus networks.

Another trend in the market is the move toward nonpro-prietary communication protocols such as TCP/IP; in par-ticular, much work is being done to develop Ethernet-basednetworked control systems. For example, two commercialmiddleware products for controls systems are specificallyfor networks running TCP/IP: products from AdvancedRealtime Control Systems, Inc. (ARCS) [60] and NDDS fromReal-Time Innovations, Inc. (RTI) [39], [40]. ARCS productsuse Java-based technologies interfaced with Simulink for


code generation. NDDS, which uses a real-time pub-lish-and-subscribe communication model, performs well forrelatively fast periodic events, as is needed, for example, bya low-level stability augmentation controller used in flightcontrols. RTI also markets ControlShell, a graphical toolthat allows integration of control systems in an ob-ject-oriented fashion. Consult the RTI Web site(www.rti.com) for several white papers that further de-scribe the use of the publish-and-subscribe communicationmodel in controls, including [40]. In answer to this move inthe market, many of the large control houses that make dedi-cated, proprietary networked systems are adding interfacesfrom their devices to Ethernet-based systems so that peoplecan create interconnected networks such as a fieldbus net-work system to a Web-based system. This works well in con-necting a local control operation to a plantwide monitoringsystem such as one based on SCADA devices [61].

Research Tools on the HorizonA few notable research tools are emerging that supportmodeling and construction of distributed, heterogeneoussystems. One such system is Ptolemy II, which focuses onmodeling embedded systems that rely on a combination oftechnologies or, more formally, “models of computation”that define how components execute and interact with eachother [62], [63]. For example, differential equations can beused to model analog circuits, and finite state machines canbe used to model control logic. Ptolemy II allows multiplemodels of computation to be used to model hierarchicalcompositions of hetereogeneous components. This is nec-essary in designing complex, hybrid embedded systemsthat include, for example, mixed analog/digital technologiesor continuous adaptive controllers whose activation is gov-erned by higher-level discrete coordination logic. Ptolemysupports a systematic and principled approach to design-ing real-world embedded systems that are compositions ofheterogeneous components. Since the focus is on the inter-actions among components, communication protocols andtiming properties/constraints become primary design con-siderations, which is critical in developing resource-con-strained systems.

While Ptolemy focuses on modeling the interaction se-mantics governing compositions of diverse components, acomplementary system, called Cactus [64]-[66], focuses onproviding customizable middleware services for distrib-uted systems that implement useful interaction styles. Cac-tus allows the QoS attributes (including fault tolerance andtimeliness) of these services to be tailored to the require-ments of a given application. The Cactus system synthe-sizes middleware software as a layer between the operatingsystem and the application. The approach is to provide a setof “micro-protocol” software modules, each of which imple-ments a well-defined service property such as atomicity, ac-cess control, or replication or retransmission forfault-tolerant communication. Desired, customized ser-

vices are then created by composing micro-protocols (iftheir logical properties are compatible) to address a givenapplication’s specific requirements. What are reused in thiscase are fundamental fine-grained distributed computingabstractions that can be combined to form configurablemiddleware services, customizable to a given application.

Giotto [67], [68] is a tool-supported methodology for de-veloping embedded control systems, particularly forsafety-critical applications that have hard real-time con-straints. It provides a programming model based on thetime-triggered paradigm [69]. A key innovation is thatGiotto supports developing a system model of functionalityand timing that is independent of the real-time platform (i.e.,separate from platform-specific computation and communi-cation scheduling). Giotto provides an abstraction layerthat lies between the high-level mathematical model of thecontrol design and the platform-specific executable code.Giotto first converts a mathematical design model to ahigh-level embedded program (termed an embedded soft-ware model). This software model specifies the interactions(including timing requirements) between software pro-cesses, as well as the timing requirements for the softwarewith respect to the physical system being controlled.Giotto’s retargetable compiler then converts this softwaremodel into executable code that meets the scheduling andcommunication constraints when run on a specific targetmachine. The positioning of the Giotto software model at anintermediate layer provides a place for validation and verifi-cation of software that is easier to manage than at the plat-form-specific code level.

Another notable system is the generic modeling environ-ment (GME), which supports reuse of system modeling con-cepts themselves [70]. GME is a design environment thatprovides a set of generic modeling and visualization con-cepts that are common to a wide range of domains. Thesecan be used to efficiently create a modeling environment fora specific domain (e.g., the signal processing domain or re-configurable control systems) by customizing the genericmodeling concepts to support a given domain language.The customized GME models are then used to automaticallysynthesize applications in the given domain.

SummaryThe software engineering concepts introduced in thisarticle make implementing control systems more flexible sothat they can be dynamically reconfigured with ease andcan be upgraded or adapted in a flexible manner. The desireto reduce software development and validation time has ledto design procedures that reuse architectural patterns aswell as software components. Common architectural pat-terns are captured in frameworks that identify commoncompositions of components that have proven to be valu-able in a particular domain. In addition, new distributed andreal-time computing technologies are emerging to help or-chestrate the distributed interaction of the diverse compo-


nents in today’s complex control systems. The applicationof these software technologies to controls systems is rela-tively new but has tremendous potential for growth. Some ofthe leading applications that motivate research on this topicare robotics, automation, and flight control, where the in-creasing complexity of the design process and the need forrapid run-time changes to the design pose challenges thatcan only be met by these new technologies.

AcknowledgmentsThis work is supported under the DARPA Software-EnabledControl program under contracts 33615-98-C-1341 and33615-99-C-1500, managed by the Air Force Research Labo-ratory (AFRL). We gratefully acknowledge DARPA’s Soft-ware-Enabled Control program, AFRL, and Boeing PhantomWorks for their continued support. We have benefitedgreatly from the guidance of Helen Gill (NSF), John Bay(DARPA), and Bill Koenig (AFRL) and from interactions withScott Clements, Cameron Craddock, Murat Guler, Eric John-son, Suresh Kannan, Chris Lee, J.V.R. Prasad, FreemanRufus, Sam Sander, Daniel Schrage, BalasubramanianSeshasayee, and Ilkay Yavrucuk.

References[1] G.T. Heineman and W.T. Councill, Component-Based Software Engineering:Putting the Pieces Together. Reading, MA: Addison-Wesley, 2001.[2] C. Szyperski, Component Software: Beyond Object-Oriented Programming.Reading, MA: Addison-Wesley, 1998.[3] D. Levine, S. Mungee, and D. Schmidt, “The design and performance ofreal-time object request brokers,” Comput. Commun., vol. 21, pp. 294-324, Apr.1998.[4] D. Schmidt and F. Kuhns, “An overview of the real-time CORBA specifica-tion,” IEEE Computer, vol. 33, pp. 56-63, June 2000.[5] P. Oreizy, N. Medvidovic, and R.N. Taylor, “Architecture-based runtimesoftware evolution,” in Proc. Int. Conf. Software Engineering (ICSE’98), Kyoto,Japan, 1998, pp. 117-186.[6] P. Oreizy, M. Gorlick, R.N. Taylor, D. Heimbigner, G. Johnson, N.Medvidovic, A. Quilici, D. Rosenblum, and A. Wolf, “An architecture-based ap-proach to self-adaptive software,” IEEE Intell. Syst., vol. 14, pp. 54-62,May/June 1999.[7] J. Waldo, “The Jini architecture for network-centric computing,” Commun.ACM, vol. 42, no. 7, pp. 76-82, July 1999.[8] E. Gamma, R. Helm, R. Johnson, and J. Vlissides, Design Patterns: Elementsof Reusable Object-Oriented Software. Reading, MA: Addison-Wesley, 1995.[9] G. Booch, J. Rumbaugh, and I. Jacobson, The Unified Modeling LanguageUser Guide. Reading, MA: Addison-Wesley, 1999.[10] D. Auslander, J.R. Ridgely, and J.D. Ringgenberg, Control Software for Me-chanical Systems, Object-Oriented Design in a Real-Time World. Upper SaddleRiver, NJ: Prentice-Hall, 2002.[11] M. Guler, S. Clements, L. Wills, B. Heck, and G. Vachtsevanos, “Generictransition management for reconfigurable hybrid control systems,” in Proc.20th American Control Conf. (ACC-2001), Arlington, VA, June 2001.[12] M. Guler, L. Wills, S. Clements, B. Heck, and G. Vachtsevanos, “A patternfor gradual transitioning during dynamic component replacement in extremeperformance UAV hybrid control systems,” in OOPSLA 2001 Workshop Pat-terns and Pattern Languages Object-Oriented Distributed Real-Time and Em-bedded Syst., Tampa, FL, Oct. 2001 [Online]. Available: http:/www.cs.wustl.edu/~mk1/realtimepatterns/oopsla2001/submissions/muratguler.pdf[13] R. Johnson and B. Foote, “Designing reusable classes,” J. Object-OrientedProgramming, vol. 1, no. 2, pp. 22-35, June/July 1988.[14] W. Pree, “Essential framework design patterns: Why ‘hot spot’ identifica-tion helps,” Object Mag., vol. 7, no. 1, pp. 34-37, Mar. 1997.[15] D. D’Souza and A. Wills, Objects, Components, and Frameworks with UML:The Catalysis Approach. Reading, MA: Addison-Wesley, 1999.[16] R. Prieto-Diaz and G. Arango, Domain Analysis and Software SystemsModeling. Los Alamitos, CA: IEEE Comput. Soc. Press, 1991.

[17] T. Brown, A. Pesetti, W. Pree, T. Henzinger, and C. Kirsch, “A reusable andplatform-independent framework for distributed control systems,” in Proc.IEEE/AIAA 20th Digital Avionics Syst. Conf. (DASC’2001), Daytona, FL, Oct.2001, vol. 2, pp. 1-11.[18] A. Traub and R. Schraft, “An object-oriented realtime framework for dis-tributed control systems,” in Proc. IEEE Int. Conf. Robotics and Automation, De-troit, MI, May 1999, pp. 3115-3121.[19] J. Fernandez and J. Gonzalez, “NEXUS: A flexible, efficient and robustframework for integrating software components of a robotic system,” in Proc.1998 IEEE ICRA, Leuven, Belgium, 1998, pp. 524-529.[20] D. Hammer, C. Becchetti, P. Narasimhan, A. Schurr, and B. Selic, “What arethe most urgent research problems of component-based software engineer-ing for resource-constraint systems?” in Proc. 6th Int. Workshop Object-Ori-ented Real-Time Dependable Syst. (WORDS’01), Rome, Italy, Jan. 2001, pp.97-98.[21] D. Hammer and M.R.V. Chaudron, “Component-based software engineer-ing for resource-constraint systems: What are the needs?” in Proc. 6th Int.Workshop Object-Oriented Real-Time Dependable Syst. (WORDS’01), Rome, It-aly, Jan. 2001, pp. 91-96.[22] E. Lee, “What’s ahead for embedded software?” IEEE Computer, vol. 33,pp. 18-26, Sept. 2000.[23] A. Tanenbaum and M. van Steen, Distributed Systems: Principles and Para-digms. Upper Saddle River, NJ: Prentice-Hall, 2002.[24] E. Johnson and S. Mishra, “Flight simulation for the development of an ex-perimental UAV,” in AIAA Modeling and Simulation Technol. Conf., Monterey,CA, no. AIAA-2002-4975, Aug. 2002.[25] A. Tanenbaum, Computer Networks. Englewood Cliffs, NJ: Prentice-Hall,1996.[26] S. Schneider, G. Pardo-Castellote, and M. Hamilton, “Can Ethernet be realtime?” [Online]. Available: http://www.rti.com/products/ndds/leads.html[27] G. Schickhuber and O. McCarthy, “Distributed fieldbus and control net-work systems,” IEEE Comput. Control Eng. J., vol. 8, pp. 21-32, Feb. 1997.[28] S. Kopetz, “A comparison of CAN and TTP,” Annu. Rev. Control, vol. 24, pp.177-188, 2000.[29] F.-L. Lian, J. Moyne, and D. Tilbury, “Performance evaluation of controlnetworks: Ethernet, ControlNet, and DeviceNet,” IEEE Contr. Syst. Mag., vol.21, pp. 66-83, 2001.[30] C. Georgakis and W. Levine, “Workshop on the research needs in model-ing, dynamics, monitoring and control of complex engineering systems,” Re-port, Anchorage, AK, May 11-12, 2002 [Online]. Available:http://www.isr.umd.edu/ISR/Complex/[31] J. Paunicka, B. Mendel, and D. Corman, “The OCP—An open middlewaresolution for embedded systems,” in Proc. American Control Conf., Arlington,VA, 2001, pp. 3345-3350.[32] J. Paunicka, D. Corman, and B. Mendel, “A CORBA-based middleware so-lution for UAVs,” in Proc. 4th IEEE Int. Symp. Object-Oriented Real-Time Distrib-uted Computing (ISORC 2001) Madgeburg, Germany, May 2001.[33] L. Wills, S. Kannan, S. Sander, M. Guler, B. Heck, J. Prasad, D. Schrage, andG. Vachtsevanos, “An open software infrastructure for reconfigurable controlsystems,” IEEE Contr. Syst. Mag., vol. 21, pp. 49-64, June 2001.[34] M. Morrison, D. Brownell, and F. Broumphrey, XML Unleashed.Indianapolis, IN: Sams Publishing, 1999.[35] Java Community Process, “Real-time specification for Java,” [Online].Available: http://jcp.org/jsr/detail/1.jsp[36] L. Carnahan and M. Ruark, Eds., “Requirements for real-time extensionsto the Java platform,” NIST Pub. 500-243 [Online]. Available:http://www.itl.nist.gov/div897/ctg/real-time/rtj-final-draft.pdf[37] J. Farley, “.NET from the enterprise perspective: SOAP,” O’Reilly News, 12Sept. 2000 [Online]. Available: http://java.oreilly.com/news/soap_0900.html[38] J. Farley, “Microsoft .NET vs. J2EE: How do they stack up?,” O’Reilly News,Aug. 2000 [Online]. Available: http://java.oreilly.com/news/farley_0800.html[39] G. Pardo-Castellote and S. Schneider, “The network data delivery service:Real-time data connectivity for distributed control applications,” in Proc.1994 IEEE Int. Conf. Robotics and Automation, San Diego, CA, vol. 4, pp.2870-2876.[40] G. Pardo-Castellote, S. Schneider, and M. Hamilton, “NDDS: The real-timepublish-subscribe middleware” [Online]. Available: www.rti.com/prod-ucts/ndds/leads.html[41] D. Bakken, Z. Zhan, C. Jones, and D. Karr, “Middleware support for votingand data fusion,” in Proc. Int. Conf. Dependable Systems and Networks,Goteborg, Sweden, July 2001 pp. 453-462.[42] C. Liu and J. Layland, “Scheduling algorithms for multiprogramming in ahard real-time environment,” J. ACM, vol. 20, pp. 46-61, Jan. 1973.


[43] A. Tanenbaum, Modern Operating Systems. Upper Saddle River, NJ:Prentice-Hall, 2001.[44] H. Kopetz, Real-Time Systems Design Principles for Distributed EmbeddedApplications. Norwell, MA: Kluwer, 2001.[45] C. Gill, D. Levine, and D. Schmidt, “The design and performance of areal-time CORBA scheduling service,” Int. J. Time-Critical Comput. Syst. (Spe-cial Issue on Real-Time Middleware), vol. 20, no. 2, 1999.[46] D. Levine, C. Gill, and D. Schmidt, “Dynamic scheduling strategies for avi-onics mission computing,” in Proc. 17th DASC. AIAA/IEEE/SAE Digital Avion-ics Syst. Conf., vol. 1, 1998, pp. C15/1-8.[47] A. Tanenbaum, Distributed Operating Systems. Englewood Cliffs, NJ:Prentice-Hall, 1996.[48] D.B. Stewart and P.K. Khosla, “Real-time scheduling of sensor-based con-trol systems,” in Real-Time Programming, W. Halang and K. Ramamritham,Eds. Tarrytown, NY: Pergamon, 1992.[49] M. Cardei, I. Cardei, R. Jha, and A. Pavan, “Hierarchical feedback adapta-tion for real time sensor-based distributed applications,” in Proc. 3rd IEEE Int.Symp. Object-Oriented Real-Time Distributed Computing, 2000 (ISORC 2000)Newport Beach, CA, Mar. 2000, pp. 181-188.[50] B. Doerr, T. Venturella, R. Jha, C. Gill, and D. Schmidt, “Adaptive schedul-ing for real-time, embedded information systems,” in Proc. 18th Digital Avion-ics Syst. Conf., St. Louis, MO, Oct. 1999, pp. 2.D.5/9.[51] C.S.R. Murthy and G. Manimaran, Resource Management in Real-Time Sys-tems and Networks. Cambridge, MA: MIT Press, 2001.[52] A.M. van Tilborg and G.M. Koob, Foundations of Real-Time Computing:Scheduling and Resource Management. Norwell, MA: Kluwer, 1991.[53] B. Li and K. Nahrstedt, “A control-based middleware framework for qual-ity-of-service adaptation,” IEEE J. Select. Areas Commun., vol. 17, pp.1632-1650, Sept. 1999.[54] M. Agrawal, D. Cofer, and T. Samad, “Real-time adaptive resource man-agement for multi-model control,” in Proc. American Control Conf., Arlington,VA, 2001, pp. 3451-3456.[55] S. Natarajan, Ed., Imprecise and Approximate Computation. Norwell, MA:Kluwer, 1995.[56] G. Manimaran and C.S.R. Murthy, “An efficient dynamic scheduling algo-rithm for multiprocessor real-time systems,” IEEE Trans. Parallel Distrib. Syst.,vol. 9, pp. 312-319, 1998.[57] F. Petrini and W.C. Feng, “Scheduling with global information in distrib-uted systems,” in Proc. Int. Conf. Distributed Computing Syst., Taipei, Taiwan,Apr. 2000, pp. 225-232.[58] G. Walsh and H. Ye, “Scheduling of networked control systems,” IEEEContr. Syst. Mag., vol. 21, pp. 57-65, Feb. 2001.[59] Venture Development Corp., “Global markets and user needs for indus-trial distributed/remote I/O, section edition” [Online]. Available:h t t p : / / i c s . p e n n n e t . c o m / A r t i c l e s / A r t i c l e _ D i s p l a y. c f m ? S e c-tion=OnlineArticles&SubSection=Display&PUBLICATION_ID=26&ARTI-CLE_ID=134229.[60] Advanced Realtime Control Systems Inc. [Online]. Available:http://www.arcsinc.com/progref-manual.pdf.[61] Industrial Automation Insider [Online]. Available:http://www.abpubs.demon.co.uk/scada.htm.[62] X. Liu, J. Liu, J. Eker, and E. Lee, “Heterogeneous modeling and design ofcontrol systems,” submitted for publication in Software-Enabled Control: In-formation Technology for Dynamical Systems, T. Samad and G. Balas, Eds. NewYork: IEEE Press, 2003.[63] E. Lee, “Computing for embedded systems,” presented at the IEEE Instru-mentation and Measurement Technology Conf., Budapest, Hungary, May 2001.[64] M. Hiltunen and R. Schlichting, “The Cactus approach to buildingconfigurable middleware services,” in Proc. Workshop Dependable SystemMiddleware and Group Communication (DSMGC 2000), Nuremberg, Germany,Oct. 2000 [Online]. Available: http://www.cs.arozona.edu/cactus/public.html[65] W. Chen, M. Hiltunen, and R. Schlichting, “Constructing adaptive soft-ware in distributed systems,” in Proc. 21st Int. Conf. Distributed Computing Syst.(ICDCS-21), Mesa, AZ, Apr. 2001, pp. 635-643.[66] M. Hiltunen, X. Han, and R. Schlichting, “Real-time issues in Cactus,” inProc. Workshop Middleware Distributed Real-Time Syst. Services, Dec. 1997,pp. 214-221.[67] T.A. Henzinger, C.M. Kirsch, M.A.A. Sanvido, and W. Pree, “From controlmodels to real-time code using Giotto,” IEEE Contr. Syst. Mag., vol. 23, pp.50-64, Feb. 2003.[68] T. Henzinger, B. Horowitz, and C. Kirsch, “Embedded control systems de-velopment with Giotto,” in Proc. Int. Conf. Languages, Compilers, and Tools Em-bedded Syst. (LCTES), Snowbird, UT, 2001, pp. 64-72.

[69] T. Henzinger, B. Horowitz, and C. Kirsch, “Giotto: A time-triggered lan-guage for embedded programming,” in Proc. 1st Int. Workshop Embedded Soft-ware (EMSOFT ‘01), Taho City, CA, 2001, pp. 166-184.[70] A. Ledeczi, M. Maroti, A. Bakay, G. Karsai, J. Garrett, C. Thomason, G.Nordstrom, J. Sprinkle, and P. Volgyesi, “The generic modeling environment,”in Proc. IEEE Int. Workshop Intelligent Signal Processing (WISP’2001), Buda-pest, Hungary, May 2001 [CD-ROM].[71] P. Lutz, W. Sperling, D. Fichtner, and R. Mackay, “OSACA—The vendorneutral control architecture,” in Proc. European Conf. Integration in Manufac-turing, Dresden, Germany, Sept. 1997, pp. 247-256.

Bonnie S. Heck is a professor of electrical and computer en-gineering at the Georgia Institute of Technology. She receivedher bachelor’s degree in electrical engineering from the Uni-versity of Notre Dame in 1981. She received a master’s degreein mechanical and aerospace engineering from PrincetonUniversity in 1984 and a Ph.D. in electrical engineering fromGeorgia Tech in 1988. She was an engineer at Honeywell Inc.from 1983 to 1985. She has been a member of the IEEE ControlSystems Society since 1988 and has served in several organi-zational capacities since that time, including serving on theBoard of Governors. Her research interests include industrialapplications, power electronics, real-time computing andcontrol, and nonlinear control design.

Linda M. Wills is an assistant professor of electrical andcomputer engineering at the Georgia Institute of Technol-ogy, where she holds the Demetrius T. Paris Professorship.She received her S.B. (1985), S.M. (1986), and Ph.D. (1992)degrees from the Massachusetts Institute of Technology.She is general chair of the IEEE International Workshop onRapid System Prototyping (RSP2003) and served as pro-gram chair of RSP2001. She has also served as general chairand program chair of the Working Conference on ReverseEngineering. She is a member of IEEE, ACM, and the IEEEComputer Society. Her research interests are in automatedsoftware reengineering and reuse, reconfigurable embed-ded software, and software parallelization for portable mul-timedia systems.

George J. Vachtsevanos is a professor of electrical andcomputer engineering at the Georgia Institute of Technol-ogy. He received a B.E.E. degree from the City College of NewYork, an M.E.E. degree from New York University, and a Ph.D.degree from the City University of New York. He directs theIntelligent Control Systems Laboratory at Georgia Tech,where faculty and students are conducting research in intel-ligent control, diagnostics and prognostics for condi-tioned-based maintenance, and vision-based inspectionand control of industrial and bioengineering systems andmanufacturing systems. He has published more than 250 pa-pers in his areas of expertise. He is a Senior Member of IEEE.He serves as the associate editor of the International Journalof Intelligent and Robotic Systems and is a consultant to gov-ernment agencies and industrial organizations.


software technology for implementing reusable, distributed control

Documents