design of dynamically reconfigurable real-time software ...in contrast, dynamically reconfigurable...

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 23, NO. 12, DECEMBER 1997 759

Design of Dynamically ReconfigurableReal-Time Software Using Port-Based Objects

David B. Stewart, Member, IEEE, Richard A. Volpe, Member, IEEE,and Pradeep K. Khosla, Fellow, IEEE

Abstract —The port-based object is a new software abstraction for designing and implementing dynamically reconfigurable real-timesoftware. It forms the basis of a programming model that uses domain-specific elemental units to provide specific, yet flexible,guidelines to control engineers for creating and integrating software components. We use a port-based object abstraction, based oncombining the notion of an object with the port-automaton algebraic model of concurrent processes. It is supported by animplementation using domain-specific communication mechanisms and templates that have been incorporated into the ChimeraReal-Time Operating System and applied to several robotic applications. This paper describes the port-based object abstraction,provides a detailed analysis of communication and synchronization based on distributed shared memory, and describes aprogramming paradigm based on a framework process and code templates for quickly implementing applications.

Index Terms —Dynamic reconfiguration, real-time operating system, software architecture, object-based design, port-automatontheory, reusable software, component-based design, evolutionary design, digital control systems, robotics.

—————————— ✦ ——————————

1 INTRODUCTION

HE port-based object (PBO) is a new software abstrac-tion for designing dynamically reconfigurable real-time

software (DRRTS). It forms the basis of a programmingmodel that provides very specific guidelines for controlengineers to create and integrate DRRTS components, yet isflexible for many types of control applications. The PBO issupported by an implementation based on domain-specificreal-time operating system (RTOS) mechanisms. Together,the PBO and RTOS mechanisms form a software frame-work that supports the design and implementation of sen-sor-based control systems.

The software framework was developed as part of theChimera RTOS Project [37] in the Advanced ManipulatorsLaboratory at Carnegie Mellon University (CMU). It is anoffshoot of a project to develop reconfigurable robots [29].We refer to the theory behind this framework as the Chi-mera Methodology. That methodology, and the correspond-ing RTOS mechanisms needed to support it, are the subjectof this paper

The following goals for a robotics programming envi-ronment were initially set forth by several robotic projectsat CMU:

• Support reconfigurable robots;• Integrate multiple sensors;

• Enable real-time sampling rates of up to 1,000 Hz;• Change controllers dynamically;• Execute code transparently on multiple processors; and• Promote collaboration in the lab through code sharing.

The Chimera Methodology is the solution to meeting theabove goals. The key contributions of our solution are thefollowing:

• A detailed definition of a port-based object, whichcombines the port-automaton algebraic model ofconcurrent processes with the software abstractionof an object, to obtain a model for creating and inte-grating dynamically reconfigurable real-time soft-ware components.

• Operating system services, including a PBO frame-work process, a multiprocessor state variable com-munication mechanism, and automated timing andanalysis of a configuration of PBOs, that create aframework for straightforward implementation ofapplications that use PBOs.

In addition, perhaps the most important criterion toachieving our goals is to hide the real-time programmingand analysis details. The target users of our framework arecontrol engineers, who do not have extensive backgroundin real-time systems or software engineering. Their strengthlies in developing control systems; they want a high-leveltool that is easy to use and frees them from the low levelimplementation details such as programming timers, ana-lyzing schedulability, synchronizing processes, or commu-nicating in a multiprocessor environments..

The background and the terminology used in this paperare given in Section 2. The architectural components of theframework are considered in Section 3. The focus of Sec-tion 4 is the domain-specific communication in a multi-processor environment. Details of the PBO framework

0098-5589/97/$10.00 © 1997 IEEE

¥¥¥¥¥¥¥¥¥¥¥¥¥¥¥¥

• D.B. Stewart is with the Department of Electrical Engineering, and Insti-tute for Advanced Computer Studies, University of Maryland, CollegePark, MD 20742. E-mail: [email protected].

• R.A. Volpe is with the Jet Propulsion Laboratory, California Institute ofTechnology, Pasadena, CA 91109. E-mail: [email protected].

• P.K. Khosla is with the Department of Electrical and Computer Engineer-ing and The Institute for Complex Engineered Systems, Carnegie MellonUniversity, Pittsburgh, PA 15213. E-mail: [email protected].

Manuscript received 7 July 1993; revised 22 Nov. 1996.Recommended for acceptance by A. Shaw.For information on obtaining reprints of this article, please send e-mail to:[email protected], and reference IEEECS Log Number 101183.

T

760 IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 23, NO. 12, DECEMBER 1997

process are given in Section 5. Favorably, the resultingframework has many advantages that go beyond theseoriginal goals. These other advantages, which fuel our cur-rent research in this area, are summarized in Section 6.

2 BACKGROUND

2.1 The Origin of the Chimera Project:Reconfigurable Robots

Robotic manipulators, such as those found on the produc-tion line of car assembly plants, have always been createdwith a fixed posture. However, depending on the task to beperformed, some postures are unsuitable. For example, amanipulator is like a person’s arm. Consider the difficultyin reaching under your car with your hand. It would bemuch easier to reach if your arm and hand were connectedto the end of your foot. The same type of problem occurswhen using robots. Different robots are better able to per-form different tasks. But using multiple robots is expensive.To solve this problem, CMU developed the ReconfigurableModular Manipulator System (RMMS) [29]. In this system,modular joints and links were created, such that they can beassembled quickly (i.e., within minutes), to create robotswith various configurations. The genesis of the ChimeraProject was to create the software environment for theRMMS [28].

We also identified other times when software needed tobe reconfigured alongside the hardware. This includedadding and removing sensors, changing the computinghardware from single processor to multiprocessing envi-ronments, and reusing generic software components withnonreconfigurable robots.

The need for dynamic reconfiguration comes from theneed to change control algorithms on-the-fly, to supportmore intelligent control strategies. However, in reviewingthe capabilities of our framework, other major advantagesof dynamically reconfigurable systems emerged, includinghaving the ability to implement real-time systems usingtime-based decomposition, and the ability to performmaintenance on-the-fly, especially for real-time systemsthat cannot be shutdown, or are too expensive to reboot.Additional benefits of our framework which go beyond theinitial goals are discussed in Section 6.

2.2 Modular Components vs. ReconfigurableComponents

Modular software is characterized by many guidelines, suchas a simple structure, data encapsulation, functional and in-formational cohesion, separation of the interface specifica-tion, and the internal behavior [21], [27], [33]. The degree ofmodularity refers to a subjective measurement used to de-scribe the extent that a software module follows these guide-lines. For example, a system decomposed into modules maybe classified as “somewhat modular” or “highly modular,”depending on a software engineer’s assessment of how wellthe module meets the defined criteria [6].

Reconfigurable components are modular componentswith the highest degree of modularity. Most important, it is amodule designed to have replacement independence. In amodular system, there is often only one way to piece all the

components together, because the interfaces of modules thatneed to be integrated are designed according to the othermodules they interact with. For example, if you write a C orC++ module, and #include a .h file of another module, thenyour module becomes dependent on the interfaces of thatother module. In contrast, interface specifications for recon-figurable components are designed according to a predefinedstandard, not according to the interfaces of other moduleswith which it will be integrated. Interactions between com-ponents are through these standard interfaces only.

As an example, Purtilo created the Polylith Software Busfor designing reconfigurable distributed systems [22]. Soft-ware components were made to interface with the bus, andnot with other modules. The bus was implemented as amessage passing transmission layer. However, the un-bounded execution and blocking times of the transmissionlayer prevents the approach from being used directly forreal-time systems. Nevertheless, Polylith demonstrates asoftware method for achieving replacement independence.

2.3 Static Configurability vs. DynamicReconfigurability

An important distinguishing feature between our approachand many other efforts in configurable systems is that ofstatic configurability versus dynamic reconfigurability of soft-ware components.

In statically configurable systems, reusable softwaremodules are selected and integrated off-line, and only exe-cuted after configuration is complete. Examples of thesemethods include real-time object-oriented program-ming [5], [30], software synthesis [1], [26], [31], also knownas automatic code generation, interface adaptation [4], [13],[18], [23], and the Polylith Software Bus mentioned above.The static nature of these systems is a result of the need tocreate or generate “glue” code to integrate the componentsfor each different configuration, then compile and link theapplication with this new code.

In contrast, dynamically reconfigurable systems can bemodified on-line, without the need to recompile and relinkthe application nor shutdown and reboot the system. Forexample, the Regis environment [17] uses detailed softwaremodels and operating system services to obtain dynamicreconfigurability. Like Polylith, modules are defined ac-cording to a standard interface, rather than to other mod-ules. Regis then uses the Darwin configuration language,based on the Conic [19] interface adaptation method, to in-teractively structure the components using input and out-put communication objects. Regis, however, has not beenapplied to real-time system design. One of its primary lim-iting factors for use in a real-time environment is that likethe Polylith transmission layer, communication objects arebased on message passing, with no considerations for exe-cution or blocking times of processes. STER [2] is anothermethod based on Conic, which can be used to create recon-figurable real-time systems. However, STER sacrifices theability to perform dynamic reconfiguration in favor of pro-viding real-time guarantees.

The research we present in this paper uses a similar con-cept as Regis and Polylith, by designing components ac-cording to a predefined standard interface, rather than to

STEWART ET AL.: DESIGN OF DYNAMICALLY RECONFIGURABLE REAL-TIME SOFTWARE USING PORT-BASED OBJECTS 761

the interface of other modules. Like Regis, detailed softwaremodels are defined, and operating system services are cre-ated that directly support those models. Our work, how-ever, concentrates on many of the real-time system issuesnot addressed by the Polylith and Regis environments. Themajor differences in our work include a software abstrac-tion that is specific to control processes, predictable com-munication and synchronization based on distributedshared memory, a detailed software structure that enablesthe automated timing of real-time properties of an applica-tion, and a programming interface that is designed espe-cially for control engineers. In Section 5.2, we also showthat static configurability is a subset of dynamic reconfig-urability, and thus our solution provides the same benefitsas those that only support static configurability.

2.4 Reconfigurable Software vs. Generic SoftwareReconfigurable software does not necessarily imply genericsoftware, as it is possible to have both hardware dependentand application dependent components that are not ge-neric, but are reconfigurable. In this section, we define ourclassifications of reconfigurable software components. Ex-amples of components are given later in Section 3.3.

A generic component is a module that is neither hardwaredependent nor application dependent. The component canbe configured for different types of hardware, and can beused in different applications.

Hardware dependent (HD) components are software mod-ules that can only be executed when specific hardware ispart of the system. Hardware dependent components canbe of two types:

Hardware dependent interface components are used to con-vert hardware dependent signals into hardware independ-ent data, such that other generic components can interfacewith these modules. The HD interface component is an in-terface to the application hardware such as robotic actua-tors, switches, sensors, and displays. They differ fromRTOS I/O device drivers, because as processes with theirown thread of control, they have the same standard inter-face as other software components, rather than being de-fined as system calls which are called by other processes.

Hardware dependent computation components providesimilar functionality as generic components, but with betterperformance or added functionality, due to hardware-specific optimizations or modifications of the generic com-ponent. Unlike the interface components, they do not com-municate directly to hardware; they are simply dependenton having specific hardware as part of the system.

Application dependent components are modules used toimplement the specific details of an application. As thename implies, these components are not reusable acrossdifferent applications. Ideally, these components are elimi-nated, since they must be redeveloped for each new appli-cation. Modules initially defined as application compo-nents, however, can often be transformed into generic com-ponents by converting hard-coded information into vari-able input. The input can then be obtained from the userthrough a teleoperating device or keyboard, from a con-figuration file, or from an external subsystem.

3 ARCHITECTURAL VIEW OF THE PORT-BASEDOBJECT

There are several approaches to attacking the problem ofcreating a general programming environment for sensor-based control that meets all the goals listed in Section 1.One approach is to create the most general architecture inwhich every possible situation that may arise is considered.Generalizing every possible scenario is impractical. A re-fined version of this approach is to create a domain specificsoftware architecture that can handle every possible situa-tion that may arise within the domain. NASREM [3] is anexample of such an architecture for the robotics domain.Our first attempt at creating a real-time processing envi-ronment was based on the NASREM model [40]. However,from our experience, using the NASREM model proved tobe very difficult, partially due to the complexity resultingfrom trying to accommodate every possible scenario into arigid hierarchical structure, and partially because there isno easy way to map from the theoretical architecture to apractical implementation.

We use an alternate approach, based on domain-specificelemental units. A framework is designed that uses theseelemental units as building blocks to incrementally createlarger, more complex applications. Various domain specificsoftware architectures can then be created using theframework, depending on how these building blocks areultimately assembled.

We select the independent process as our elemental processmodel.1 An independent process does not have to communi-cate or synchronize with any other component in the system,and thus integration is simple. A system that is comprisedonly of independent components, however, is very limiting,as there are no means to share data or resources. Neverthe-less, this extreme emphasizes a desire to keep the frameworksimple. Rather than trying to achieve the most general modelof a task, we attempt to get as close to this “ideal” simple caseof an independent process. The simpler each component, thesimpler it will be to integrate them.

Streenstrup, Arbib, and Manes formalized the algebra ofindependent concurrent processes with their port-automaton theory [32]. They model a concurrent process asan independent automaton, which operates on the state ofthe environment.

When a process needs information, it obtains the mostrecent data available from its input ports. This port can beviewed metaphorically as a window in your house; what-ever you see out the window is what you get. There is nosynchronization with other processes and there is noknowledge as to the origin of the information that is ob-tained from this port.

When a process generates new information that might beneeded by other processes, it sends this information to itsoutput ports. An output port is like a door in your home;you can open it, place items outside for others to see, thenclose it again. As with the input ports, there is no synchro-nization with other processes, and there is no knowledge asto which processes might look at this information.

1. Although the term process is used throughout this paper, implementa-tion in our RTOS is done using lightweight processes, which are also calledthreads in many operating systems.


Lyons and Arbib [16] applied this model specifically torobotics, and showed that a stable control system can beachieved using a formally specified language they termedRobot Schemas. As compared to the Robot Schemasmodel, we extend the port-automaton model to includemultiple types of ports, and we create a framework fordirectly implementing software components using thiscomputational model.

In addition to the independent process, we select the ob-ject as an elemental software abstraction. As stated byWegner, an object is the atomic unit of encapsulation, withoperations that control access to the data [42]. The termobject does not imply “object-oriented design,” which is anextension to objects to include classes and inheritance. Thereferences to objects in this proposal are thus classified asobject-based design, as defined by Wegner’s distinction ofthat term and object-oriented design [43].

3.1 The Port-Based ObjectWe combine the algebraic model of a port automaton withthe software abstraction of an object, to create the port-based object (PBO), shown in Fig. 1. In our diagrams, wedraw a PBO as a round-corner rectangle, with input andoutput ports drawn as arrows entering and leaving the sideof the rectangle. Configuration constants are drawn as ar-rows entering/leaving the top of the rectangle. Resourceports are shown as arrows entering/leaving the PBO fromthe bottom.

Fig. 1. Architectural view of a port-based object.

A PBO is an independent concurrent process, whosefunctionality is defined by the methods of a standard ob-ject. Communication with other modules is restricted toits input ports and output ports, as defined by the port-automaton theory. There is no explicit synchronizationwith other processes. The configuration constants are usedto reconfigure generic components for use with specifichardware or applications.

In addition to input and output ports, we also define re-source ports, which are needed to create an environment formultisensor integration. The resource ports connect to sen-sors and actuators, via I/O device drivers, which are notPBOs. The details of accessing the sensor or actuator arethus encapsulated within the PBO, resulting in an HD inter-face component (as defined in Section 2.4).

By modeling PBOs to have optional configuration con-stants and resource ports, we have been able to use thesame PBO model for all types of reconfigurable compo-

nents. A sample library of PBO objects for robotic manipu-lators is shown in Table 1. The library represents a subset ofPBOs that were created in our laboratory at CMU.

An important note about the functional descriptions ofthe modules is that the framework is designed independent ofthe granularity of functionality in each PBO. The software ar-chitect who decomposes an application into modules de-fines the granularity; our framework then provides themechanisms for quickly realizing each of these modules byusing the PBO model to implement them as reconfigurableobjects.

Similarly, the framework does not define the type or semanticsof the port variables. A variable type mechanism [38] is usedso that data transmitted over the ports can be any type. Forexample, it can be raw data, such as input from an A/D orD/A converter; processed data, such as positions and ve-locities; or processed information, such as structures de-scribing types and locations of objects in the environment.

In our implementation, a configuration file is used tospecify the information shown in the PBO Name and Portscolumn of Table 1. For example, the configuration filepuma.rmod for module puma is shown in Fig. 2. The MODULEline specifies the name of the object module file name(extension omitted) of the code used for this PBO. MultiplePBOs can specify the same code, for cases where multipleversions of the same object, possibly each with its ownunique configuration, are desired. The DESC line gives aone-line description of the module. The INVAR and OUTVARlines define the input and output ports, respectively, theINCONST and OUTCONST lines define the configuration con-stants. Since ASCII characters for the configuration file re-strict us, we created our own conventions for mappingnames, as shown in Table 2. Note that these conventions arepersonal preferences, and not specifications of the frame-work. TASKTYPE is either periodic or aperiodic. The FREQ line isthe initial frequency at which a periodic process corre-sponding to this PBO executes; the frequency can bechanged dynamically. The SVARALIAS line is used for map-ping internal and external names, as described in the nextsection. The LOCAL line marks the beginning of module-specific information. Any information after this line is notused by the PBO framework, but rather passed on to theinitialization code of the PBO. Lines beginning with # arecomments and are ignored.

3.2 ConfigurationsAs defined by Dorf [7], “a control system is an interconnec-tion of components forming a system configuration thatwill provide a desired system response.” Each componentcan be mathematically modeled using a transfer function,which computes an output response for any given inputresponse. The port-automaton theory provides an algebraicmodel for these types of control systems. By incorporatingthe model into the PBO, our PBOs also provide this samemodel suitable for control engineers. PBOs are configuredto form a control system in the same way as a control engi-neer configures a system using transfer functions and blockdiagrams. This approach allows us to satisfy an importantcriterion for our DRRTS framework, to make a frameworkfor control engineers, rather than for software or real-timesystem engineers.


TABLE 1EXAMPLE OF SOFTWARE COMPONENTS IN A ROBOTIC MANIPULATOR PBO LIBRARY

Fig. 2. File puma.rmod: sample configuration file for puma PBO.

TABLE 2SUMMARY OF INVARS, OUTVARS, AND CONFIGURATION

CONSTANTS FOR SAMPLE LIBRARY OF TABLE 1


A configuration is a set of PBOs which are interconnectedto provide the required open- or closed-loop system. In ourimplementation, the set of PBOs can be specified in one offour ways using a graphical software assembly tool, using acommand-line interface, through a network from an externalplanning subsystem, or embedded using the C programminglanguage.

A configuration is valid only if for every PBO selected,any data that it requires at its input ports is produced byone of the other PBOs as output. As per the port-automatontheory, the control engineer does not have to be concernedwith how data gets from the output of one PBO to the inputof another PBO. The communication is embedded in theframework, such that it is transparent to the control engi-neer. A configuration also cannot have two PBOs that pro-duce the same output; otherwise there may be a conflict asto which output should be used at a given time.

Port names are used to perform the bindings betweeninput and output ports. Whenever two PBOs exist withmatching input and output ports, the framework creates acommunications link from the output to the input. If neces-sary, the output can be fanned into multiple inputs. Ourframework uses an internal/external name separation forthe ports, such that the name used to code the PBO can beindependent of the name used for linking that object toother PBOs. The mapping between internal and externalname is done in the SVARALIAS lines of the configuration filefor each PBO. The left value represents the name used inthe configuration, and the right value is the name used in-ternally by the PBO. If an SVARALIAS line is not specified,then the default is that both the internal and external namesare the same.

The correctness of a configuration is verified analyticallyusing set equations, where the elements of the sets are thestate variables. If Xj is a set representing the input variablesof module j, Yj is a set representing the output variables ofmodule j, then a configuration is legal only if the followingtwo conditions are true:

< <j

k

j j

k

jX Y

��

�� ¯��

��

��

��1 1

(1)

Y Y O i j i j k i ji j>4 9 � � � » �, for all , such that ,1 (2)

where k is the number of modules in the configuration.Equation (1) represents our first condition that there must bea corresponding output port for every input port. Equation(2) ensures that two PBOs do not produce the same output.

3.3 Configuration ExamplesIn this section, we illustrate through examples how tocreate configurations out of PBOs stored in a library.Details of designing individual PBOs are given later inSection 5. In our implementation, a configuration can beassembled graphically using the Onika visual program-ming environment [8].

3.3.1 Cartesian Control of the Reconfigurable ModularManipulator System

Fig. 3a shows a configuration, using modules from oursample library shown in Table 1, to perform teleoperated

Cartesian control of the RMMS. The configuration of theRMMS robot is not known beforehand. Rather, its configu-ration is read from EPROMs embedded in the robot duringinitialization. From that configuration, the rmms moduleoutputs the NDOF and DH configuration constants. Those con-stants are used as input to the gfwdkin and ginvkin modules,which can be configured for any robot based on NDOF and DH

[11]. A teleoperation interface is provided by the 6-DOFtrackball, and the cinterp module is used to generate inter-mediate trajectory points for the robot, because the tballmodule typically executes at a much lower frequency thanthe other modules.

The software framework does not pose any constraints onthe frequency of each PBO. Rather, as defined by the port-automaton theory, every PBO is an independent concurrentprocess that can execute at any frequency. Whenever thatprocess needs data from its input ports, it retrieves the mostrecent data available. When it completes its processing, itthen places any new data onto its output ports.

A configuration can be executed in either a single- ormultiprocessor environment. In a multiprocessor environ-ment, the control engineer only needs to specify whichprocessor to use for each PBO. The communication betweenPBOs and synchronization of their processes is otherwiseidentical, and fully transparent to the control system engi-neer, as detailed in Section 4.

3.3.2 Cartesian Teleoperation of a Puma 560Suppose that a Puma 560 robot is to be used instead of theRMMS. The rmms module can be replaced with the pumarobot interface module, as shown in Fig. 3b. Since the Pumais a fixed configuration robot, its NDOF and DH parameters areconstant. Instead of reading these values from the robot,they can instead be hard-coded into the puma module, andoutput as configuration constants. There is no need tochange any other module, since the gfwdkin and ginvkinmodules will configure themselves during initialization forthe Puma based on the new values of NDOF and DH.

3.3.3 Improving Performance of a Puma 560Generic components are useful for enabling rapid proto-typing, but they may not always be computationally effi-cient. For example, the computation of the forward kine-matics based on the DH configuration constants and usingmatrix operations will naturally be slower than performingsimilar computations for a specific robot, such as the Puma560. For a fixed-configuration robot, the DH parameters areconstant, and unnecessary computations (such as multiplyby zero or 1, or computing sin(S/2)) can be eliminated.

To improve the performance of an application, an HDcomputation components can be created. The pfwdkin andpinvkin modules are examples of such components. Theycompute the forward and inverse kinematics specifically fora Puma 560, and they execute faster than their genericcounterparts. It is then desirable to replace gfwdkin withpfwdkin, and ginvkin with pinvkin, as shown in Fig. 3c,whenever the puma HD interface component is used.

In order for an HD computation component to replace ageneric component, it must provide at least the same outputsand must not require any additional inputs as compared tothe generic component. Even when an HD component is


used, it does not eliminate the usefulness of the generic com-ponent. For example, in order to improve fault tolerance ofan application, the generic components can still be used asstandby modules, or as shown in Fig. 3d, it can execute inparallel with the HD computation components, albeit atlower frequencies, in order to provide consistency checks.

3.3.4 Autonomous Execution of a Puma 560As an example of an application component, suppose that acustom autonomous trajectory module ctraj is created toreplace the teleoperation module tball, as shown in Fig. 3e.The component can be integrated into the system by defin-ing it as a port-based object.

Even though a module is application dependent, it doesnot have to be hardware dependent. Thus, if the hardwarefor the application is changed, the application componentdoes not necessarily have to change. Fig. 3f shows this byreplacing rmms with puma, but not changing the trajectoryof the robot’s end effector, as defined by ctraj.

3.3.5 Cartesian Control of a Torque-Mode RobotAs a more elaborate example of a configuration, a telero-botic Cartesian visual servoing subsystem is shown in Fig. 4(note that these modules are from a different PBO librarythan the one defined in Table 1). Input can come from ei-ther a user through a trackball or from an external visionsubsystem. The port interfaces of each PBO can easily bedetermined simply by looking at the inputs and outputs ofeach object. Despite the seemingly complex communicationpaths between objects, communication remains transparentfrom the control system engineer, and lines are simplydrawn between matching input and output ports.

The configuration of PBOs is not the only part of a sub-system. As shown in Fig. 5, PBOs can interface with devicedrivers, external subsystems, software libraries, special pur-pose processors, and user interfaces. In this paper, however,we focus on the configurations of port-based objects, and theassociated communication, synchronization, and analysis.

(a) (b)

(c)(d)

(e) (f)

Fig. 3. Example of component-based design using port-based objects. (a) example of Cartesian teleoperation of the RMMS using GCs; (b) exam-ple of Cartesian teleoperation of a Puma 560 using GCs; (c) example of Cartesian teleoperation of a Puma 560 using HDCSs; (d) example offault-tolerant components, with co-existence of old and new software components; (e) example of application-specific autonomous execution ofthe RMMS using GCs; (f) example of application-specific autonomous execution of a Puma 560 using HDCCs.


4 INTEROBJECT COMMUNICATION

Integrating software components such that all communica-tion is performed in a predictable and timely manner isperhaps the most difficult aspect of creating reconfigurablereal-time systems. In order to support the port-based objectmodel, we need a communication mechanism that meetsthe following requirements:

• Support the port-automaton model of independentprocesses. That is, read the input ports at the begin-

ning of each cycle to obtain the most recent dataavailable, and write to the output ports at the end ofeach cycle.

• Disallow synchronization or communication betweenprocesses except through ports.

• Target data transfers with low volume but at highfrequency (1,000 Hz).

• Have a simple and straightforward binding schemefor communication links to dynamically reconfigure asubsystem in bounded time.

Fig. 4. Example of module integration: Cartesian teleoperation.

Fig. 5. Complete software infrastructure for a sensor-based control subsystem.


• Fan an output into multiple inputs.• Support transparent multiprocessing, so that the

processes that are communicating can be either on thesame or different processors, without any differencein the communication.

• Allow communication between processes that may beexecuting at different frequencies.

We designed a communication mechanism that meets all ofthese requirements, based on the combined use of local anddistributed shared memory. The domain-specific solutiontakes advantage of several of the characteristics of controlsystems, such as the assumption on transfer rates and theneed to only read the most recent data, rather than all data.The solution provides a level of performance and predict-ability that has not previously been achieved using moregeneral message-passing mechanism.

Port communication has often (and more typically) beenimplemented using some form of message passing. This al-ternative was considered, but not selected for several reasons:

• Using messages, we could not support the port-automaton theory because the most recent data is notalways readily available. For example, if the processproducing the data is faster, then the messages maybe queued, and the message received by the con-sumer might not contain the most recent data.

• Fanning an output to multiple inputs is difficult be-cause it requires a message to be duplicated for eachinput or requires a more complex mechanism to ensurethat messages are not deleted until all processes need-ing it have used it. Duplicating messages based on thenumber of recipients also violates the port-automatontheory, which states that a process is unaware of thedestination of the data on its output ports.

• The overhead with sending messages, especially in amultiprocessor environment, is much higher than thatachievable using shared memory. This factor is espe-cially important considering some data must be trans-ferred 1,000 times per second.

These drawbacks of message passing systems led to ourdesign of a mechanism based on distributed shared mem-ory. Our work focuses on loosely coupled shared memoryarchitectures.

4.1 State Variable CommunicationThe communication between PBOs is performed via statevariables stored in global and local tables, as shown inFig. 6. Every I/O port and configuration constant is definedas a state variable (SVAR) in the global table, which isstored in shared memory.

A PBO can only access the local table, where only thesubset of data from the global table that is needed by thatPBO is kept. Since every PBO has its own local table, nosynchronization is needed to read from or write to it. APBO process can, thus, execute independently of otherprocesses by using the data in its local table. Consistencybetween the global and local tables is maintained by theSVAR mechanism, as detailed in the remainder of this sec-tion. As an example, Fig. 7 shows the contents of the globaland local tables for the sample configuration that was illus-trated in Fig. 3a.

Support for SVAR communication has been built into ourframework, such that updates of the local and global tablesoccur at predetermined times only. Configuration constantsare updated only during initialization of the PBO. The statevariables corresponding to input ports (which we callINVARS) are updated prior to executing each cycle of a peri-odic PBO, or before processing each event for an aperiodicPBO. During its cycle, a PBO may update the state variablescorresponding to output ports (which we call OUTVARS) atany time. These values are only updated in the global tableafter the PBO completes its processing for that cycle or event.All transfers between the local and global tables are blocktransfers (i.e., using a routine like UNIX’s memcpy()). Ensur-ing the integrity of the data is a matter of ensuring that theblock transfers are performed as critical sections.

Although there is no explicit synchronization or com-munication among processes, we must ensure that accesses

Fig. 6. Structure of state variable table mechanism for port-based object integration.

Fig. 7. Contents of global and local tables for sample configuration shown in Fig. 3a.


to the same SVAR in the global table are mutually exclu-sive, which creates potential implicit blocking. The lockingmechanism we use enables us to maintain the autonomousexecution model of the PBO, while also ensuring the integ-rity of the communication.

4.2 Locking the Global SVAR TableThe method we use to lock the global table and preserve theautonomous execution model of the PBO is based on theassumption that the amount of data communicated via theports on each cycle of a PBO is relatively small. That is, eachINVAR or OUTVAR is only a few tens to a few hundreds ofbytes. This is in contrast to communication systems whereimages may require many thousands or millions of bytesper cycle. The exact value of what we mean by “small” de-pends on a particular configuration, and is quantified aspart of our analysis in Section 4.4. For this reason, ourframework, which defines objects that are specific to thecontrol systems domain, cannot directly be applied to ageneral communication system. However, as was shown inFig. 4, we can use the external subsystem interface to inter-act with a communication system.

Processes executing on different CPUs can access theglobal table, and therefore a multiprocessor solution forlocking the table is required. The type of multiprocessorsynchronization we require for the global SVAR table hasbeen addressed in [24]. The shared memory protocol (SMP) ispresented as an extension of the single processor priorityceiling protocol [25]. The protocol involves defining globalsemaphores for locking the global shared memory andplacing priority ceilings on accessing the semaphores tobound the waiting time of higher priority jobs.

Unfortunately, there are several problems that preventthe use of SMP within our framework:

• This method assumes that the local scheduling oneach processor is the rate monotonic algorithm, withstatic priorities. As discussed in [39],it is desirable touse mixed or dynamic priority algorithms for sched-uling reconfigurable systems, for which the protocolis not suitable.

• One assumption of the SMP is that the delay to accessthe backplane bus from any processor is negligiblecompared to the task execution times. Unfortunatelythis is usually not the case; buses like the VMEbus areimplemented using a static-priority processor as-signment that is not under the control of software.Therefore, the time to wait for the bus can be signifi-cant if a process is on a low-priority CPU.

• There is significant overhead associated with imple-menting SMP that prevents its use with control appli-cations requiring frequencies over 1,000 Hz. Thecomplexity and overhead of SMP can be reduced sig-nificantly for the port-based communication by se-lecting a single lock for the entire table, instead of aseparate lock for each state variable. Selecting a singlelock for the entire table is not as restrictive as it seems,since a shared bus connects the shared memory to lo-cal memory. Even if multiple tasks have separatelocks only one of them can physically access theshared memory at once; other tasks must wait for thebus even while in their critical section.

An alternate solution for synchronizing access to theglobal state variable table is to use spin-locks [20]. When atask must access the global table, it first locks the processoron which it is executing. Locking the CPU ensures that thetask does not get swapped out while holding the criticalglobal resource. The task then tries to obtain a global lockby performing an atomic read-modify-write instruction,which is supported by most hardware processors. If thelock is obtained, the task reads or writes the global tablethen releases the lock, still while being locked into the localCPU. It then releases its lock on the local processor. If thelock cannot be obtained because it is held by another task,then the task spins on the lock. It is guaranteed that the taskholding the global lock is on a different processor, and willnot be preempted, thus it will release the lock shortly.

In theory, locking the CPU can lead to possible misseddeadlines or priority inversion. However, considering thepractical aspects of real-time computers, it is not unusualthat a real-time microkernel locks the CPU for up to100 Psec in order to perform system calls such as handlingtimer interrupts, scheduling, and performing full contextswitches [37]. Furthermore, many RTOS are created suchthat periods and deadlines of processes are rounded to thenearest multiple of the system clock since more accuratetiming is not available to the scheduler. In these systems, ifthe total time that a CPU is locked to transfer a state vari-able is small as compared to the resolution of the systemclock, then there is negligible effect on the predictability ofthe system due to this mechanism locking the local CPU.

In comparing this method to SMP, the lock can be viewedas a single global semaphore, and since all tasks can access it,its priority ceiling is constant, and is equal to the maximumtask priority in the system. Since there is only one lock thereis no possibility of deadlock. A task busy-waits with the localprocessor locked until it obtains the lock and goes through itscritical section. In Section 4.4 it is shown that for configura-tions where the volume of data transferred between objects issmall, there is a bounded waiting time for obtaining theglobal lock, even on hardware such as the VMEbus that onlyhas fixed priority bus arbitration.

4.3 InitializationThe global table is initialized in shared memory based onthe contents of an SVAR configuration file. The file includesthe names and types of all the INVARS, OUTVARS, and con-figuration constants. A default configuration should beprovided in conjunction with each PBO library, which de-fines the name, type, and size of each variable. For example,Table 2 shows the information that would be included inthe SVAR configuration file for the library that was shownin Table 1. For any application, a programmer may updatethis default configuration file by adding new variables to itif they make use of the SVARALIAS facility to change some ofthe port names, or by deleting variables that are not re-quired for the application.

A local table is only created and initialized when a PBO iscreated, and contains those SVARs given in the .rmod con-figuration file (as was shown in Fig. 2). Information about thetype and size of those SVARs is retrieved from the globaltable, which is why it did not have to be specified in the .rmodfile. For each SVAR, a pointer is also stored in the local table,


which points to the corresponding SVAR in the global table,and used during the block transfer operations.

Configuration constants create a necessary order of ini-tialization for PBOs. Any PBO that has an output configu-ration constant (which we call OUTCONST), must be initial-ized before any other PBO with the same constant as aninput (called an INCONST) is created. INVARS and OUTVARS,on the other hand, do not have a necessary order, since incontrol systems they often form a closed loop system, andthus the order of initialization of PBOs is not obvious. Be-yond the necessary order for constants, ordering initializa-tion of PBOs in a closed loop system to ensure stability ofthe control system is generally application specific, andthus not discussed further within our framework. Ourframework, however, can support any order of initializa-tion of the PBOs, as dictated by the application.

4.4 Analysis of SVAR MechanismIn this section, we show that on a fixed priority hardwareplatform such as the VMEbus, it is possible to provide pre-dictable high performance communication using the SVARmechanism. The analysis assumes that data from the inputports is transferred once from the global table to the localtable at the beginning of a process’ cycle, and data destinedfor the output ports is transferred from the local table to theglobal table upon completion of the process’s cycle. ThePBO framework ensures that communication occurs atthese specified times, as described in Section 5.

4.4.1 Transfer TimesTo ensure predictable communication, the time required totransfer data between the local and global tables for eachtask must be computed. Let tIP be the time required to trans-fer the INVARS and tOP be the time required to transfer theOUTVARS of a PBO P, assuming no waiting for the bus.These values are computed as:

t V n V R x

t V n V R y

IP IP ai

n

Pi

OP OP ai

n

Pi

IP

OP

� �

� �

Ç

Ç

11

11

( )

( )

(3)

where V1 is the overhead for locking and unlocking the ta-

ble, excluding waiting time for the bus. Va is the overhead

of transferring each additional variable; nIP/nOP are thenumber of INVARS/OUTVARS for object P; x xPi Po/ numberof transfers required for the INVAR/OUTVAR i of object P; andR(x) = time required for x transfers. V1, Va, and R(x) are de-pendent on the speed of the hardware. These values can bemeasured initially for each type of hardware supported,then used by a configuration manager for estimating com-munication times. As an example, V1, Va, and R(x) weremeasured in our laboratory. The breakdown of times for anIronics IV3230 single board computer [9] with a 25 MHzMC68030 processor on a VMEbus is shown in Table 3. AVMETRO 25 MHz VBT-321 VMEbus analyzer [41] wasused to time the communication, and provided a resolutionof better than 1 Psec. The global state variable table wasstored within the dual-ported memory of a second IV3230.

Note that the value of R(x) is not linear. This is due to theunderlying block copy routine, which has a better averagetime per transfer for larger transfers. Through interpolation,different transfer sizes can be estimated, and more meas-urements of R(x) with different values of x can give moreaccurate results. However, for purposes of discussion andexamples in this paper, the values shown are sufficient.

The values in Table 3 can be substituted into (3) to esti-mate the transfer times for each port-based object. As anexample, consider the joint control of a robot with built-incontrollers, whose configuration is shown in Fig. 8, andassume that NDOF = 6. The values of tIP and tOP for eachmodule were estimated. These estimates were then com-pared to actual transfer times measured with the VMETROanalyzer. As can be seen in Table 4, the estimates and actualtimes are sufficiently close to use the estimates for furtheranalysis. This aspect is important since it is not desirable,and perhaps not feasible, to measure the communication ofevery software module for every type of hardware. Forsimplicity, (3) assumes that V1 is always present, even if n is0, such as nIP for jtball. In practice, if there are no transfersto be made, the global table is not locked. As a result, theactual measured time is very small and accounts for over-head of testing if a transfer must be made.

4.4.2 Waiting Time for Global Table LockUntil now, we assumed there is no contention for the globaltable’s lock. Next, we compute the worst-case waiting timefor the lock by each task. Let Lpj be the maximum time thattask p on processor j will hold the global table lock. There-fore Lpj = max(tIp, tOp). Let Mj be the longest time that theglobal lock is held by any task on processor j, then

M Lj pj p

N j ��

��max 1 (4)

where Nj is the number of tasks on processor j.Ideally, if multiple tasks are trying to obtain the lock, the

one with the highest priority succeeds. Unfortunately, on ashared bus where each processor has a fixed priority, suchas the VMEbus that is not using round-robin bus arbitra-tion, that is not the case. Instead, the task inherits the prior-ity of the processor. For the remainder of the analysis, as-sume that the hardware is a fixed-priority VMEbus, suchthat the lowest numbered processor has highest priority.For different hardware configurations, the following analy-sis may have to be redone, and perhaps a different form oflocking for the global table may be appropriate.

Any task on processor k attempting to lock the global ta-ble must wait for tasks on all higher priority processors.Furthermore, the task may also have to wait for a task cur-rently holding the lock on a lower priority processor. Basedon the locking mechanism described in Section 4.2, only onetask on any processor can request the lock at once, andtherefore there is no contention with other tasks on thesame processor.

In Section 4.2, an assumption was made that the statevariable table mechanism was valid as long as the amountof data to be transferred is small. That assumption is nowquantified, as the volume of data affects the maximumwaiting time of each task. Let Wk be the worst case waiting


time for any task on processor k. Since this is waiting timeand not blocking time (a waiting task is in the running state,a blocked task is suspended) Wk can be added to the worst-case execution time of a task. It is computed as

W W Wk kLO kHI � (5)

where WkLO and WkHI are the maximum time the task mayhave to wait for a task to release the lock on a lower orhigher priority processor, respectively. WkLO is computedsimply as the longest time any single task on a lower prior-ity processor may hold the lock. Therefore,

W MkLO j j kr

�

max 14 9 (6)

where r is the number of processors.Next, WkHI is computed. For k = 1, there are no higher

priority processors, thus W1HI = 0 and W1 = W1LO. For k > 1,the potential locking of the table for all tasks on processors1 to k � 1 must be considered. Under the assumption thatthe volume of data is small, the bandwidth required totransfer all the data is much less than the total bandwidthof the bus. Therefore, in the worst case, all tasks on higher-priority processors may require the lock at the same time.WkHI is thus computed as the sum of the waiting time of alltasks on higher-priority processors as:

W t tkHIj

k

i

N

I ij O ij

j

�

�

Ç Ç1

1

1, ,4 9 . (7)

The notation tI,ij is the same as tIP, where a process P is re-ferred to by the processor number i and task ID j. As anexample, (5), (6), and (7) were applied to the sample con-figuration that was shown in Fig. 8, with estimated lockingtimes as were shown in Table 4. Task periods and cycletimes (before adding maximum waiting time) were arbi-trarily assigned to illustrate the computations. Assumingthat puma_pidg and grav_comp are on processor 1, and diffand jtball are on processor 2, the resulting computations areshown in Table 5. The adjusted execution time should beused in a schedulability analysis.

In our applications, the average case is significantlylower than the worst case. To compensate, a soft real-timescheduling algorithm can be used to schedule the less criti-cal tasks on lower priority processors [39], while hard real-time critical tasks should be placed on the high-priorityprocessors. Another consideration for assigning tasks toprocessors is the volume of data that needs to be trans-ferred. The computations of Wk show that it is preferablefor tasks producing a low volume of data to be placed onhigher priority processors, since that significantly reduces

TABLE 3BREAKDOWN OF VMEBUS TRANSFER TIMES AND COMMUNICATION OVERHEAD

TABLE 4COMPARISON OF ESTIMATED AND ACTUAL TRANSFER TIMES

TABLE 5SAMPLE COMPUTATIONS OF WORST EXECUTION TIME (ALL TIMES IN msec)


WkHI for tasks on lower priority processors. A different as-signment of tasks to processors can lead to very differentresults. A configuration manager can perform the aboveanalysis for various configuration possibilities, in order tooptimize the global allocation of processes to processors.

5 FRAMEWORK PROCESS FOR IMPLEMENTINGPORT-BASED OBJECTS

The analysis in previous sections was performed with anunderlying assumption that the framework handles com-munication and synchronization. In this section, we look atthe details of the PBO, and show how we have imple-mented the framework as part of the RTOS. This allows acontrol engineer to easily implement individual softwarecomponents using the PBO model.

5.1 An Inside-Out Programming ParadigmCreating the code for PBOs is an “inside-out” programmingparadigm as compared to traditional coding of real-timeprocesses, as shown in Fig. 9. The grey area shows RTOScode, while the black areas show the user code. The tradi-tional approach is used by most current RTOS; processesare created, each with their own main() (or equivalent func-tion name). The process executes user code and controls theflow of the program. It invokes the operating system, typi-cally via a system call, whenever an OS service is required.OS services include communication, synchronization, pro-gramming timers, and creating new processes.

The PBO method, on the other hand, provides a consis-tent structure for every process, and thus operating systemservices such as communication, synchronization, sched-uling, and process management are performed in a predict-able manner. Only when necessary, the operating systemcalls a PBO’s method to perform user-defined functions.This predefined structure also allows the RTOS to continu-ally measure the execution time of the PBO code, using anautomated task profiling mechanism as described in [35].

A PBO process is realized by creating a single, standardprocess, which we call the framework process (pboframe()).Every process in the system uses this same framework, andtakes a PBO as an argument. The PBO defines the module-specific user code, the I/O ports and configuration con-stants, the type of process (e.g., periodic process or aperiodicserver), and the timing parameters such as frequency, dead-line, and priority.

5.2 The Framework ProcessThe framework process implements a finite state machinewith four states, as shown in Fig. 10. The states are shownas bold ellipses, and are NOT_CREATED, ON, OFF, and ERROR.

State transitions are shown in the diagram as process flowdiagrams. A state transition is triggered by a signal (drawnas solid bars). Signals may originate from interrupts, aplanning module, an external subsystem, or from the userthrough a graphical user interface. In response to a signal, atransfer is made between the local and global SVAR tablesto read INCONSTS or INVARS, then one of the user-definedfunctions is called, followed by another transfer betweenthe local and global SVAR tables, to write OUTCONSTS orOUTVARS. State variable transfers are shown as ovals, whilethe user-defined functions, which form the PBO, are shownas rectangles.

The PBO method that is called depends on the state ofthe process and the signal that is received. For example, if aPBO is in the ON state, and it receives a wakeup signal, thenit will execute the cycle method and remain in the ON state.On the other hand, if the PBO is in the ON state, and re-ceives the kill signal from an a configuration manager, thenit will execute the off method, followed by the kill method,then enter the NOT-CREATED state.

The framework process, as shown, evolved over severalyears as we designed and tested many variations, in order toobtain a common program structure for all software compo-nents. The diagram represents the most recent revision in theevolution of the structure. The detailed PBO framework im-plies many design decisions, as described next

5.2.1 Notes about Framework ProcessDespite the seeming complexity of the framework, dis-secting it into pieces shows that it is indeed rather simple.In the steady state, PBO processes are all in the ON state,and executing their cycle method once per cycle or event,going back to sleep until the next wakeup signal. Note thatthe only difference between a periodic process and anaperiodic server is the source of the wakeup signal. For aperiodic process, the wakeup signal is received from thetimer. For the aperiodic processes, the process blocks on asemaphore, message, or event, as defined by the syncmethod of the PBO.

The autonomous nature of the PBO allows the mostpopular scheduling algorithms, such as the rate monotonicstatic priority [15], earliest-deadline-first [15], or maximum-urgency-first dynamic priority scheduling [39] algorithms,to be used to schedule PBOs. The control systems designeronly needs to specify the frequency of the cycle routine foreach PBO. Determining the optimal set of frequencies is acontrol systems issue, however, and beyond the scope ofthis paper.

As a result of the underlying timing error detection andhandling mechanism built into the Chimera RTOS [36],aperiodic servers can use the same fundamental structureas periodic processes. The framework can define aperiodicprocesses as either deferable or sporadic servers, and usethem with either the rate monotonic static priority ormaximum-urgency-first dynamic priority scheduling algo-rithms to ensure predictable scheduling.

The remainder of the framework handles the initializa-tion and termination, reconfiguration, and error handlingfor the PBO. To support dynamic reconfiguration, a two-stage initialization and termination is used. High-overhead

Fig. 8. Joint control of a robot manipulator with built-in PID controller.


Fig. 9. Comparison of POSIX real-time processes and port-based object programming paradigms.

Fig. 10. Finite state machine for the port-based object framework process. State transitions are shown as process flow diagrams.


initialization of a new process can be performed upon sys-tem startup, or in the background, in preparation for beingactivated. The initialization includes creating a process’scontext, dynamically allocating its memory, creating a localtable and translating I/O port symbols into pointers to theglobal table, and calling the user-defined init method. Theprocess then waits in the OFF state, and can be viewed asbeing in a standby mode for a dynamic reconfiguration.When an on signal is received, the local table is updated toreflect the current state of the system, and execution begins.

Perhaps one of the keys to supporting dynamic reconfigu-ration is handling the initialization of the port variables. Be-fore calling the on method, it is necessary to read the OUTVARS

in addition to the INVARS. This solution resulted after manytrials with initialization, trying to determine the best way toensure that when a process is activated, its view of the envi-ronment is correct. Since the process is not previously exe-cuting, it is possible that some other process is generatingthose OUTVARS. In general, a process that outputs OUTVARS

knows the same values on the subsequent cycle because theyare in the local table. However, in a process’ first cycle, this isnot the case. Ultimately, rectifying the situation is a simplematter of also reading the OUTVARS during activation, thusupdating the local table to properly reflect the state of thesystem. The on method of the process is then called, such thatif necessary, these OUTVARS can be updated, before the proc-ess enters the ON state.

The time for a process P to be activated, Con,P is bounded,if the user-defined on method is bounded. It is,

C t t Con P IP OP on P OS, , � � � �2 ' (8)

where tIP and tOP are defined in (3), �Con P, is the execution

time of the user-defined on method of process P, and 'OS isthe operating system overhead for sending a signal andperforming a context switch. Typical values for Con P, in our

system were in the order of (100 Psec + �Con P, ).The deactivation of a process is similarly time-bounded.

Thus, the time to perform a dynamic reconfiguration is thesum of the time to activate new processes and deactivaterunning processes that form the differential between the oldand new configurations. Since it is generally possible to acti-vate and deactivate several processes within a millisecond, itis possible to execute one controller module one cycle, fol-lowed by executing a different controller on the next cycle.

As a precautionary measure in case of transient overloadduring a dynamic reconfiguration, an illegal configurationflag is temporarily set, which indicates that not all requiredoutput is being produced. The framework provides thismechanism primarily for use of HD interface componentsthat send output to actuators. The most critical processes inthe system, which should have a priority higher than theprocesses being reconfigured, can test this flag, and if it isset, can choose to ignore its INVARS, and go into locally sta-ble execution. For example, a robot interface module canselect to keep the velocity of the joints constant, until thedynamic reconfiguration is complete, signaled by the illegalconfiguration flag being reset. Determining when it is safeto perform a dynamic reconfiguration is beyond the scopeof the framework. The framework provides the mecha-

nisms only. Developing policies that ensure stable execu-tion during a reconfiguration is usually application specific.In our experiments, we use a conservative approach of en-suring that the robot is temporarily at rest (i.e., velocity andacceleration are both zero before dynamic reconfigurationbegins). Further research is required in order to developmore aggressive policies.

Error detection and handling is implemented using aglobal error handling mechanism, as described in [37].Whenever an error occurs, an error signal is generated, anda user-defined error handler is called. The frameworkautomatically initializes this error handler to be the errormethod of the PBO. The purpose of the error method is toattempt automated recovery. If that fails, the process goesinto the ERROR state, indicating that user intervention is re-quired. Once the user attempts to fix the problem, a clearsignal is generated. The clear method checks that the errorwas indeed fixed. If it was, then the process returns to theOFF state and can be reactivated; if not, the process remainsin the ERROR state. See [14] for further research into errorhandling for PBOs.

Whenever one or more processes are in the ERROR state,the illegal configuration flag described above is set. Thisagain indicates to critical modules that there exists at leastone other module in the configuration not producing out-put as required, and therefore the INVARS of that processmay not have been properly computed.

During a dynamic reconfiguration, a new process can becreated in the background, such that it updates some of theconfiguration constants. Before this process can be acti-vated, some of the running processes might need to be ini-tialized, if they have configured themselves previouslybased on different INCONSTS. In this case, a reinit signal issent to those processes that have the updated OUTCONST asan INCONST. The new process can only be activated afterprocesses have reinitialized.

Statically configurable processes are subsets of dynami-cally reconfigurable processes. These processes go directlyto the ON state after initialization. Therefore, if the OFF stateis removed, and any transition emanating from the OFF

state removed, the result is a framework process that can beused for statically configurable systems. In our applicationsrequiring only static configurability, however, we still de-fine the modules as dynamically reconfigurable, but sendan on signal immediately after the init signal.

5.3 Coding a Port-Based ObjectThe structure of a PBO is designed so that a control engi-neer can simply define module-specific code, and not beconcerned with any of the details of creating a real-timeprocess. A template can be created for any specific PBO,given an .rmod file, as was shown in Fig. 2. The control en-gineer then only has to fill in the blanks, which is to definethe methods of the PBO to perform the module-specificfunctionality. As an example, Fig. 11 shows the templateand control engineer’s code for the tball module. Theregular font shows the template code for tball, given theinformation in Table 1, while the bold font shows codewritten by the control engineer.


The encapsulated data of the object is stored in thestructure called tballLocal_t. Part of this structure is gener-ated by the framework, to include pointers to the localSVAR table. The remainder of the structure is user-definable, so that the control engineer can place any“global” variables for their PBO. The data in this structureis made available to every method. The methods all havespecific names, which is of the form xxxYyyy. xxx repre-sents the module name, and Yyyy represents the methodname, as was depicted in Fig. 10.

The rigid process structure provides strict guidelines forcontrol engineers, telling them exactly where to put whatkind of code. It removes any guesswork, reduces the amountof code they must write, and guarantees that synchronizationand communication works from the beginning.

6 DISCUSSION

While our DRRTS solution meets the requirements set forthin Section 1, we have since realized many other, and in manycases more important, potential benefits of the solution:

Rapid development through component-based design.Component-based design minimizes the amount of newcode that must be written when an application is devel-oped. A framework for DRRTS eliminates the need to writeany “glue” code, thus going a step beyond reuse of modu-lar components.

Hardware/software co-design. One form of system-leveldesign of embedded applications uses a mixture of hard-ware and software components [10]. The co-design ap-proach allows the hardware and DRRTS components to betightly coupled throughout the design process.

Tele-configuration of services. Telecommunication com-panies, such as cable, satellite TV, and telephone, are in-creasingly transmitting control information over the datatransmission medium. DRRTS components can be used todynamically change the configuration, hence the servicesprovided, or to perform remote upgrades and maintenanceof existing services.

Evolutionary design. Complex systems may requirecontinuous hardware and software upgrades during thelifetime of the system in response to technological ad-

Fig. 11. Sample .c code for tball PBO. Template code uses regular font; user code is in bold .


vancements, environmental change, or alteration of systemgoals. DRRTS components are designed to undergo suchevolution, as individual modules can be replaced incre-mentally and independently.

Flexibility for fine-tuning after implementation. ADRRTS framework offers considerable flexibility for finetuning an application to “make it work.” It can have recon-figuration options such as switching between static anddynamic scheduling algorithms, using time-based decom-position to improve CPU utilization, and easily convertinginterrupt handlers to aperiodic servers and vice versa.

Increased reliability through automated analysis. Theinternal structure of a DRRTS component is well defined,based on a theoretical model. This structure allows for theautomation of such things as performance measurement,configuration verification, and scheduling analysis.

These additional benefits fuel our current research effortinto developing advanced RTOS technology for supportingdynamically reconfigurable systems.

REFERENCES

[1] B. Abbott et al., “Model-Based Software Synthesis,” IEEE Software,vol. 10, no. 3, pp. 42-52, May 1993.

[2] J.M. Adan and M.F. Magalhaes, “Developing ReconfigurableDistributed Hard Real-Time Control Systems in STER,” Algo-rithms and Architectures for Real-Time Control, Proc. IFAC Workshop,pp. 147-152, Oxford: Pergamon Press, Sept. 1991.

[3] J.S. Albus, H.G. McCain, and R. Lumia, NASA/NBS Standard Ref-erence Model for Telerobot Control System Architecture (NASREM),Technical Note 1235, Nat’l Inst. of Standards and Technology, Gaith-ersburg, Md., Apr. 1989.

[4] B.W. Beach, “Connecting Software Components with DeclarativeGlue,” Proc. Int’l Conf. Software Eng., pp. 11-15, Melbourne, Aus-tralia, IEEE Press, 1992.

[5] T.E. Bihari and P. Gopinath, “Object-Oriented Real-Time Systems:Concepts and Examples,” Computer, vol. 25, no. 12, pp. 25-32, Dec.1992.

[6] B.A. Blake and P. Jalics, “An Assessment of Object-OrientedMethods and C++,” J. Object-Oriented Programming, vol. 9, no. 1,pp. 42-48, Mar.-Apr. 1996.

[7] R.C. Dorf, Modern Control Systems, third edition. London: Addi-son-Wesley, 1980.

[8] M.W. Gertz, D.B. Stewart, and P.K. Khosla, “A Human MachineInterface for Distributed Virtual Laboratories,” IEEE Robotics andAutomation Magazine, vol. 1, no. 1, pp. 5-13, Dec. 1994.

[9] IV3230 VMEbus Single Board Computer and MultiProcessing EngineUser’s Manual, Computer Systems Division, Ironics Inc., Ithaca,New York, 1991.

[10] A. Kalavade, “System-Level Codesign of Mixed Hardware-Software Systems,” doctoral dissertation, Univ. of California, Ber-keley, 1995.

[11] L. Kelmar and P.K. Khosla, “Automatic Generation of Forwardand Inverse Kinematics for a Reconfigurable Modular Manipula-tor System,” J. Robotics Systems, vol. 7, no. 4, pp. 599-619, Aug.1990.

[12] D. Kinny, M. Georgeff, and A. Rao, “A Methodology and Model-ing Technique for Systems of BDI Agents: Agents BreakingAway,” Lecture Notes in Artificial Intelligence LNAI 1038. Berlin:Springer-Verlag, 1996.

[13] D.A. Lamb, “IDL: Sharing Intermediate Representations,” ACMTrans. on Programming Languages and Systems, vol. 9, no. 3, pp.297-318, July 1987.

[14] J. Lang, “A Distributed and Time-Bounded Exception HandlingMechanism for Dynamically Reconfigurable Real-Time Software,”masters thesis, Dept. Electrical Engineering, Univ. of Maryland,College Park, 1997.

[15] C.L. Liu and J.W. Layland, “Scheduling Algorithms for Multipro-gramming in a Hard Real Time Environment,” J. ACM, vol. 20,no. 1, pp. 44-61, Jan. 1973.

[16] D.M. Lyons and M.A. Arbib, “A Formal Model of Computationfor Sensory-Based Robotics,” IEEE Trans. Robotics and Automation,vol. 5, no. 3, pp. 280-293, June 1989,.

[17] J. Magee, N. Dulay, and J. Kramer, “Structuring Parallel and Dis-tributed Programs,” IEE Software Eng. J., vol. 8, no. 2, pp. 73-82,Mar. 1993.

[18] J. Magee et al., “Configuring Object-Based Distributed Programsin REX,” IEE Software Eng. J., vol. 8, no. 2, pp. 139-149, Mar. 1992.

[19] J. Magee, J. Kramer, and M. Sloman, “Constructing DistributedSystems in Conic,” IEEE Trans Software Eng., vol. 15, no. 6, pp.663-675, 1989.

[20] L.D. Molesky, C. Shen, and G. Zlokapa, “Predictable Synchroni-zation Mechanisms for Multiprocessor Real-Time Systems,” The J.Real-Time Systems, vol. 2, no. 3, pp. 163-180, Sept. 1990.

[21] D.L. Parnas, P.C. Clements, and D.M. Weiss, “The ModularStructure of Complex Systems,” IEEE Trans. Software Eng., vol. 11,no. 3, pp. 259-266, Mar. 1985.

[22] J. Purtilo, “The Polylith Software Bus,” ACM Trans. ProgrammingLanguages and Systems, vol. 16, no. 1, pp. 151-174, Jan. 1994

[23] J.M. Purtilo and J.M. Atlee, “Module Reuse by Interface Adapta-tion,” Software: Practice and Experience, vol. 21, no. 6, pp. 539-556,June 1991.

[24] R. Rajkumar, “Real-Time Synchronization Protocols for SharedMemory Multiprocessors,” Proc. 10th Int’l Conf. Distributed Com-puting Systems, pp. 116-123, Paris, 1990.

[25] L. Sha, R. Rajkumar, and J.P. Lehoczky, “Priority InheritanceProtocols: An Approach to Real-Time Synchronization,” IEEETrans. on Computers, vol. 39, no. 9, Sept. 1990.

[26] S. Ritz, M. Pankert, and H. Meyr, “High Level Software Synthesisfor Signal Processing Systems,” Proc. Int’l Conf. Application SpecificArray Processors, Berkeley, Calif., pp. 4-7, 1992.

[27] S.R. Schach, Software Eng., second edition. Asken Associates, 1993.[28] D.E. Schmitz et al., “CHIMERA: A Real-Time Programming Envi-

ronment for Manipulator Control,” IEEE Int’l Conf. Robotics andAutomation, Phoenix, Ariz., pp. 846-852, 1989.

[29] D.E. Schmitz, P.K. Khosla, and T. Kanade, “The CMU Reconfig-urable Modular Manipulator System,” Proc. Int’l Symp. and Exposi-tion on Robots (designated 19th ISIR), Sydney, Australia, pp. 473-488, 1988.

[30] B. Selic, G. Gullekson, and P. Ward, Real-Time Object-OrientedModeling. New York: John Wiley & Sons, 1994.

[31] T.E. Smith and D.E. Setliff, “Towards an Automatic SynthesisSystem for Real-time Software,” Proc. Real-Time Systems Symp.,San Antonio, Texas, pp. 34-42, 1991.

[32] M. Steenstrup, M.A. Arbib, and E.G. Manes, “Port Automata andthe Algebra of Concurrent Processes,” J. Computer and System Sci-ences, vol. 27, no. 1, pp. 29-50, Aug. 1983.

[33] W.P. Stevens, G.J. Myers, and L.L. Constantine, “Structured De-sign,” IBM Systems J., vol. 13, no. 2, pp. 115-139, 1974.

[34] D.B. Stewart and G. Arora, “Dynamically Reconfigurable Embed-ded Systems–Does it Make Sense?” Proc. Real Time ApplicationWorkshop, Montreal, Oct. 1996.

[35] D.B. Stewart and P.K. Khosla, “Policy-Independent RTOS Mecha-nisms for Timing Error Detection, Handling, and Monitoring,”Proc. IEEE High Assurance Systems Eng. Workshop, Niagara, Ont.Canada, Oct. 1996.

[36] D.B. Stewart and P.K. Khosla, “Mechanisms for Detecting andHandling Timing Errors,” Comm. ACM, vol. 40, no. 1, pp. 87-94,Jan. 1997.

[37] D.B. Stewart, D.E. Schmitz, and P.K. Khosla, “The Chimera IIReal-Time Operating System for Advanced Sensor-based ControlApplications,” IEEE Trans. Systems, Man, and Cybernetics, vol. 22,no. 6, pp. 1,282-1,295, Nov./Dec. 1992.

[38] D.B. Stewart and P.K. Khosla, “Chimera 3.1: The Real-TimeOperating System for Reconfigurable Sensor-Based ControlSystems,” program documentation, Advanced ManipulatorsLaboratory, The Robotics Inst. and Dept. Electrical and ComputerEng., Carnegie Mellon Univ., Pittsburgh, http:// www.ee.umd.edu/serts/

bib/ manuals/Chimera.html .[39] D.B. Stewart, “Real-Time Software Design and Analysis of Recon-

figurable Multi-Sensor Based Systems,” doctoral dissertation,Carnegie Mellon Univ., Dept. Electrical and Computer Eng.,Pittsburgh, 1994, http://www.ee.umd.edu/serts/bib/thesis/dstewart.html.

[40] D.B. Stewart, D.E. Schmitz, and P.K. Khosla, “CHIMERA II: AReal-Time Multiprocessing Environment for Sensor-Based RobotControl,” Proc. IEEE Int’l Symp. Intelligent Control, Albany, N.Y.,pp. 265-271, Sept. 1989.


[41] VBT-325 The VME+ Analyzer System User’s Manual, VMETRO Inc.,Houston, 1995.

[42] P. Wegner, “Dimensions of Object-Oriented Programming,” Com-puter, vol. 25, no. 10, pp. 12-20, Oct. 1992.

[43] P. Wegner, “Concepts and Paradigms of Object-Oriented Pro-gramming,” OOPS Messenger, vol. 1, no. 1, pp. 7-84, Aug. 1990.

David B. Stewart received his MS degree (1989)and his PhD degree (1994) in electrical andcomputer engineering from Carnegie MellonUniversity. He received a BEng degree (1988,great distinction) in computer engineering fromConcordia University. Dr. Stewart is an assistantprofessor with the Department of Electrical Engi-neering at the University of Maryland, CollegePark. He has a joint appointment with the Uni-versity of Maryland Institute for Advanced Com-puter Studies (UMIACS) and is a faculty affiliate

with the Institute for Systems Research (ISR). He is director of theSoftware Engineering for Real-Time Systems (SERTS) Laboratory. Hisresearch lies in the areas of advanced real-time operating systemstechnology and component-based software for embedded systems. In1991, he was a visiting researcher at the Jet Propulsion Laboratory,California Institute of Technology.

In May 1997, Dr. Stewart received the George Corcoran Award inrecognition of teaching and education leadership at the University ofMaryland, College Park Campus, effective contribution at the nationallevel, and creative and other scholarly activities related to electricalengineering education. He is a recipient of the Natural Sciences andEngineering Research Council of Canada 1967 Science and Engi-neering Scholarship. He has also been awarded the Chait Medal andComputer Engineering Medal from Concordia University; the Prize ofExcellence from the Professional Order of Engineer’s of Quebec, Can-ada; the Myer F. Pollock Scholarship for academic excellence; theConcordia University Entrance Scholarship; the Independent Order ofForester’s University Entrance Scholarship; the National Honor Societyof Canada Scholarship; and the Josten’s Foundation Scholarship. Dr.Stewart is a member of the IEEE.

Richard A. Volpe received his MS degree(1986) and his PhD degree (1990) in appliedphysics from Carnegie Mellon University, wherehe was a U.S. Air Force Laboratory graduatefellow. His thesis research concentrated on real-time force and impact control of robotic manipu-lators. In December 1990, he became a seniormember of the technical staff at the Jet Propul-sion Laboratory, California Institute of Technol-ogy. Until late 1993, he was a member of theRemote Surface Inspection Project, investigating

sensor-based control technology for telerobotic inspection of the Inter-national Space Station. After late 1993, he led the development ofRocky 7, a next generation mobile robot prototype for extended-traverse sampling missions on Mars. Recently he received a NASAExceptional Achievement Award for this work, and has joined thespaceflight team for this 2001 rover mission. His research interestsinclude real-time sensor-based control, robot design, path planning,and computer vision. Volpe is a member of the IEEE.

Pradeep K. Khosla received the BTech degree(honors) from IIT (Kharagpur, India), and bothMS and PhD degrees from Carnegie MellonUniversity. He is currently professor of electricaland computer engineering and robotics, andfounding director of the Institute for ComplexEngineered Systems (which includes the formerEngineering Design Research Center—an NSFERC) at Carnegie Mellon University. Prior tojoining Carnegie Mellon, he worked with TataConsulting Engineers and Siemens in the area of

real-time control. From January 1994 to August 1996, he was on leavefrom Carnegie Mellon and served as a DARPA program manager inthe Software and Intelligent Systems Technology Office (SISTO), De-fense Sciences Office (DSO), and Tactical Technology Office (TTO),where he managed advanced research and development programs inthe areas of Information based design and manufacturing, web basedInformation technology infrastructure, real-time planning, real-timesoftware, sensor-based control, and robotics.

Professor Khosla’s research has focused on the theme of “creatingcomplex electro-mechanical and information systems through compo-sition of and collaboration amongst building blocks.” Specifically, hisinterests are in the area of integrated design-manufacturing systems,collaborating robots, agent-based architectures for design and control,software composition for real-time systems, reconfigurable autono-mous systems, and human systems interface. He is particularly inter-ested in the application domains of design and manufacturing, andunstructured environments. He is involved in electrical and computerengineering, design, and robotics education, both at the graduate andthe undergraduate level. He was a member of the committee thatformulated a curriculum for the PhD program in robotics at CarnegieMellon. He was also a member of the Wipe the Slate Clean Committeethat created a new four year undergraduate ECE degree curriculum atCMU. In support of the new curriculum, he developed and imple-mented the notion of virtual laboratory, and an introductory freshmanlevel course “Introduction to Electrical and Computer Engineering” thatemphasizes the notion of teaching in context.

Professor Khosla was the program vice-chair for the 1989 IEEEInternational Conference on Systems Engineering; general chair forthe 1990 IEEE International Conference on Systems Engineering;program vice chair of the 1993 International Coference on Roboticsand Automation; general co-chair of the 1995 Intelligent Robotics Sys-tems (IROS) conference; and program vice-chair for the 1997 IEEERobotics and Automation Conference. He has served as a member ofthe AdCom of the IEEE Systems, Man and Cybernetics Society, tech-nical editor of the IEEE Transactions on Robotics and Automation, andcair of the education committee of the IEEE Robotics and AutomationSociety.

Professor Khosla is a recipient of the Inlaks Fellowship, UnitedKingdom; the Carnegie Institute of Technology Ladd award for excel-lence in research in 1989; two NASA Tech Brief awards; and waselected as an IEEE fellow in January 1995. Professor Khosla's re-search has resulted in two books and more than 200 journal articles,conference papers, and book contributions. Professor Khosla servedas a member of the technical advisory board of the Next GenerationController Project (WPAFB, U.S. Air Force and Martin Marietta),RIMCC, and the Sample Acquisition Analysis and Preservation Project(Jet Propulsion Laboratory, NASA). He has been an invited participantto the Department of Commerce workshops on the Intelligent Manu-facturing Systems program, and USA-Japan R&D consortia and col-laboration. He is a consultant to several industries in the USA and aco-founder and chairman of the board of K2T inc.—a high tech com-pany based in Pittsburgh.

design of dynamically reconfigurable real-time software ...in contrast, dynamically reconfigurable...

Documents