CoatrolEr4. Prc~ti~, Vol. 1, No. 1, pp. 65-70,1993 0967-0661D3 $6.00 + 0.00 Printed in Great Britain. ALl fights reserved. © 1993 Pergamon Press I.~

REAL-TIME UNIX: WHAT PERFORMANCE CAN WE EXPECT?

H. Rzehak

Univer$itdt der Bundeswehr Miinchen, Werner-lleisenberg-Weg 39, D-8014 Neubiberg. Germany

Abstract. The paper deals with necessary enhancements of tIaditional UNIX systems in order to meet the requirements for real-time applications and gives an insight on sources of delays caused by the operating system. A survey of the real-time extensions for POSIX, the ISO/IEC standard operat- ing system, is given. Some key parameters and basic concepts for implementing good real-time per- fonnance are discussed. Figures for commonly used metrics are provided.

Kevwords. Real-time operating systems; POSIX real-time extensions; real-time performance metrics; reentrant system calls; concurrency control

1. I N T R O D U C T I O N

Recently UNIX-systems have become available which claim to have real-time properties, and real-time exten- sions for POSIX, the ISO/IEC standard operating system based on UNIX (ISO/IEC 9945-1, 1990), are going to be defined. This gives reason not only to look at the functional requirements in terms of required system calls, but also to ask for real-time performance of such UNIX derivatives. Slogans like "Fully preemptable Kernel" are used to emphasize real-time capabilities and even more serious performance-metrics may be confusing in the case of poor knowledge of basic system concepts. The aim of the following sec- tions is to give a better understanding of what real-time performance depends on.

2. F U N C T I O N A L R E Q U I R E M E N T S

Real-time operating systems axe state of the art and have been used for many applications for a couple of years. Specialists mainly agree upon functional requirements provided by system calls, and differences in functionality mainly result from the scope of appli- cations (e.g. whether an application requires a file system). Using the draft on real-lime extensions for POSIX (IEEE P1003.4, 1992) as an example the tradi- tional functionality of UNIX (multiprocessing, inter- process communication, file handling, standard input and output) will be enhanced by the following features for real-time applications (listed in the order of the draft):

Realtime Signals Extensions Synchronized Input and Output

Asynchronous Input and Output Semaphores Process Memory Locking Memory Mapped Files and Shared Memory Priority Scheduling Clocks and Timers Message Passing Realtime Files

In order to tailor operating systems for applications not requiring the full functionality real-time profiles and their minimum hardware requirements are defined. The actual state of work on POSIX extensions for support- ing real-time applications distinguishes between four profiles (IEEE P1003.13, 1991):

Minimal real-time system:

This profile is typically used in embedded systems and somewhere else called an operating system kernel. The programming model is that of a single address space. This corresponds to a single POSIX process. Concurrent activities may be established through more than one POSIX thread (IEEE P1003.4a, 1992). Minimal hardware requirements axe a single processor with its memory, but no memory management unit or common I/O devices.

ReaL-time controller system:

The minimal real-time system is enhanced by a file system interface, character serial I/O and exception handling (POSIX Signals). One or more serial channels (RS-232-1ike) have to be added to the minimal hardware but no mass storage device, since the file system may be implemented in m e m o r y .

65

66 H. Rzehak

Dedicated real-time system:

In addition to the minimal real-time system this profile supports multiple POSIX processes, a common interface for i]evice drivers and a non hierarchical f'de system interface. For memory management hardware memory locking is provid- ed. Minimal hardware assumes one or more pro- cessors with or without memory management unit to be present.

Multi-purpose real-time system

This profile supports full functionality of POSIX real-time extensions. The hardware model covers processors with memory management unit, high speed mass storage devices, network support and display devices.

3. R E A L - T I M E P E R F O R M A N C E

In addition to the def'med functionality the draft on real- time extensions for POSIX defines performance metrics and a performance documentation option provides a document with measurements of these per- formance metrics. This is necessary to achieve a required level of service in a bounded response time. Furthermore these metrics are a base for comparing different implementations.

In general the metrics should serve for calculating

the overhead introduced by a service or

upper bounds for delays of running programs caused by the operating system.

Some methodologies and related metrics for measuring real-time performance have been proposed in Furth et al. (1991a). This paper is focused on the event reaction time and related topics, because the timely reaction to (external) events is the main requirement for real-time applications.

An often referenced metric is the Process Dispatch Latency Time of the operating system - sometimes misleading shortly cared Interrupt Latency Time. But the crucial issue on real-time operating systems is to avoid all unexpected delays of the running processes. As we will see the Process Dispatch Latency Time is not the only delay to be considered for timely reactions.

3.1 Event Reaction Time

An external event is commonly signaled by an interrupt and we assume that the associated event reaction is part of a user task. Assumed that the interrupt has high prio- rity to be immediately served and the currently running task has lower priority as the user task containing the event reaction, we define the time between the occur- rence of the external event and the execution of the flu'st statement of the associated event reaction within the user task as the Process Dispatch Latency Time (PDLT). If this fin:st statement sets a bit in an externally observable register, the PDLT is easy to measure. For real-time applications we are interested in the upper bound of the PDLT.

To give an answer we assume that the running task has executed a system call just before the interrupt occurs. If the called system function is not preemptable, the transfer of contxol to the task containing the event reac- tion is delayed until control is returned from the system function to user code. Fig. 1 gives a sketch of this situation.

®

Task T 1

osfmac( )

Entry

E x t e m a l Event

Return

®

OS-Kernel

/ , osfunc, /

E x c c u t c

further tasks

" " - • u Interrupt Routine

awake(task ti)

Task T . !

OS-funct ions must not be interrupted in case of non reentrancy

®

I l I

Fig. 1 Calling an event reaction

Exe.cute T a s k T 1

U s e r C o d e

Real-time UNIX

Exte rna l E v e n t

I~rocasss Dispatch _ I Latency Time - - ' ~

® ®

In t e r rup t Serv ice Ro u t in e

E x ecu t e T a s k T i ( even t related)

" . . . . - ~ = t ime

Continue Task T 1

67

Fig. 2 Definition of the Process Dispatch Latency Time

In Fig. 2 the transitions of control are marked on the time scale. The PDLT is the time between point (1) and (2); the worst case is given by the dynamically longest system function. A traditional UNIX implementation is not suitable for real-time applications because the kernel functions are not preemptable leading to an extraordinarily high PDLT. Now real-time UNIX systems claim to have fully preemptable kernel func- tions and therefore their PDLT is very small, but this is only half of the truth. For a better understanding we have to know why the preemption of kernel functions may be a problem.

3.2 Reentrant System Calls

In general kernel functions manipulate entries in data structures of the operating system kernel necessary to manage the various objects and resources of the operat- ing system. Changes necessary to perform such a func- tion cannot be done in a single indivisible instruction and therefore during execution the contents of these data structures may be temporarily inconsistent. If an interrupt occurs in such a situation some or even all kernel functions may not be executable because they would have to operate on inconsistent data. The brute force way to overcome the problem is to forbid kernel preemptions. A traditional UNIX kernel uses this method which is fairly simple and not so bad except for real-time applications.

Allowing preemptions to a certain extent a kernel func- tion may be called the second time until returning from the first call. Therefore we need reentrant system calls. Summarizing, for kernel preemptions we have to maintain the following conditions:

A sequence of instructions changing data from one consistent state to another consistent one, com- monly called a critical region, must not be entered the second time until the first entry call has reached the exit.

If system data used by two critical regions have a common subset these two critical regions must be entered as mutually exclusive.

Several concepts may be used to meet these conditions. With respect to real-time requirements the most elaborate one is constructing the kernel functions with critical regions and using semaphores to prevent reen- trant execution of critical regions and to avoid conflict- ing concurrent execution. Such a kernel is sometimes called fully preemptable.

Fig. 3 gives an example with three critical regions. It is shown how a preemption with a reentrant function call works.

It should be noticed that the construction of such a kernel is not an easy task. As we will see below, for good real-time performance the critical regions should be (dynamically) as short as possible. A traditional UNIX kernel cannot be adapted by minor changes. Another problem is the use of drivers written for a traditional UNIX system, because their kernel level synchronization does not fit into the concept described here. Some compatibility modes may be offered for this purpose at the cost of real-time performance (Furth et al., 1991a).

In the case of a fully preemptable kernel the PDLT does not depend on whether user or system code will be executed at the cccurence of the external event. There- fore measured values are reasonably small. But as shown in Fig. 3 an unexpected delay may happen later on, if during the event reaction a system call invokes the interrupted kernel function. This is the time between the points t 3 and t 6 in Fig. 3 necessary to terminate the critical region.

At first glance the imposed latency time is at most the time necessary to execute the dynamically longest criti- cal region. But this is not the worst case. Consider the situation at t 3 in Fig. 3. If another external event happens causing the same function call with a higher priority (e.g. prio = 22) this call has to wait twice: for leaving P1 by prio = 27 and leaving P2 by prio = 32.

We can continue this construction until a call has to wait for leaving all sections by other processes. The necessary nesting of system calls and events becomes more and more unlikely or even impossible.

68 H. Rzehak

P1

P2

) semal

P3

Pl ; P2; P3:

Mutual Exclusive Region

Call with prio = 32

semal (

( Event Causing ~ . Function Call pC" . . . . . with prio = 27

sema3 ("

(

Call with prio = 27

I, m No Invocation of this Region

~ Invocation , ' " in Progress

) (

)

t 0 1

semal

sema2

sema3

)

t t t 2 3 4

(3 () (

t7 10

Fig. 3 Preemptive execution of a reentrant system function

Looking at the results of introducing kernel preemp- tions we are not sure if we have reduced the worst case value for the latency time because it depends on the application if a certain nesting of system calls and events is possible. But by using reentrant system calls we have achieved that higher latency times have become much more unlikely. We consider this to be an improvement.

We can sum up our results as follows:

For a fully preemptable kernel of an operating system with reentrant functions an arbitrary delay may be imposed into an event reaction. We call this the Maximum Kernel Latency Time (MKLT). It is not part of the Process Dispatch Latency Time. The MKLT depends on the application. An approxima- tion is the execution time of the (dynamically) longest critical region of the kernel. Even this approximation is neither easy to determine nor to measure without knowledge of the internal struc- ture of the kernel. Sometimes reported statistical results may give a first impression but worst case values may be much higher than mean values.

There is another possible solution for implementing kernel preemptions: If a second call takes place on a higher priority the first call (on a lower priority) will be aborted instead of reentering a function. For continuing the preempted process the first call has to be restarted. Compared with the usual solution of reentrant functions the latency time due to the restart of the function appears on the lower priority level. This is an advan-

tage.

On the other hand processor time is wasted and there- fore latency times are imposed on all lower priority processes and not only on the preempted one. For that reason aborting a kernel function seem to give a pre- ferred solution only if we have a single very time criti- cal process.

3.3 Published Results

Table 1 shows some measured values of the PDLT obtained from Furth et al. (1991a). All systems claim to be real-time UNIX systems. Some seem to give fairly good real-time performance.

OS ! lardware

| IP-UX HP 900)/825

] larris Night CX/RT Flawk 3400

Concurrent/ RTU Masscomp 6300

AEG -MODCOMP REAL/IX Tri-D 9730

(MC 68030-25)

Process Dispatch Lat~mcy

MIN ps MAX p's

655 1991

90 430

1605 11605

50 360

Table 1 Process Dispatch Latency Time for some systems

In contrast to the values for HP-UX in Table 1 Hewlett- Packard has published values for the kernel latency

Real-time UNIX

time of HP-UX (Doughty et al., 1987a; Doughty, 1987b). These values are shown in Table 2.

90 % ofpatl~ through the kernel

99 % of paths through the kernel 129 ms 3.4 ms

max. paths through the kernel 112} ms 14.6 rm

Preemption OFF Preemption ON

4On-~ 1.4n=

Table 2 Kernel Latency Time for HP-UX

Intended Cycle Tune T0[ms ]

W~om L ~ DeviationofTo[%] } Mi..

Mean Value [ms]

With Load:

DevialionofTo[qO ] } kl~ Mean vd~ [ms]

RTOS-UH

(S~ h=l-Tm l~ngl)

7-OOO

- 11.775 + 11.75

7-O0O

fRegT'm USiX3

2.OOO 7-O833

- 0.25 - 1.28 0.00 + 1.28

7-0~33

-4.96 +4.96

2.0833

Haglwa¢: MVME 14"/with MC 68030-25 Mlk

69

For I-IP-UX we see that the MKLT is 14.6 ms, whereas the maximum PDLT is 1.991 ms. As described in Doughty et al. (1987a) and Doughty (1987b) the kernel of HP-UX is not fully preemptable because the concept of critical regions guarded by semaphores (see section 3.2) is mixed with the concept of preemption points. This leads to a higher PDLT.

Other available data (Kriechbaum, 1991; Gralla, 1991; MODCOMP, 1991) support the general assumption that the Maximum Kernel Latency Time is much higher than the Process Dispatch Latency Time.

To compare real-time UNIX systems with specialized real-time operating system kernels Table 3 and Table 4 show some application oriented characteristics for two operating systems measured on the same hardware (obtained from Gralla, 1991). The considered UNIX system does not look so bad. The results depend on the considered load (terminal I/O in Table 3 and Table 4). We expect less-good results if network applications are included in the load.

Max. h'nlerrula Frequency (kHz)

Inlerrupt Resolution Time (/as )

Proceu Dispatch Latency Time "~ [ pSI Min without Load .J Max.

Dispatch Latency Time "~ [ ps] Process Min. with Load J Max.

RTOS-UH (Special Real. Time Kernel)

14.7

6.6

55.2 63.9

57.3 110

REAL/IX

~,ed-~me UNlX~

6.67

150.0

128 171

128.8 228

Paudwa~: MVME 147 with MC 68030.25 MHz

Table 3 Event Reaction Characteristics

Table 4 Characteristics for Cyclic Tasks

4. H I E R A R C H Y O F L E V E L S

Powerful operating systems are no longer a monolithic entity. Reducing complexity the internal structure con- sists of a hierarchy of levels of which the lowest is the kernel. It provides only a minimal set of functions from which the rest of the operating system can be con- structed. The critical regions of the kernel can be kept short. Some user services may be performed by kernel functions directly, other services by functions on a higher level using kernel functions indirectly. Examples are file access and management services or network services.

A close look at existing implementations shows that relatively long critical regions will be found exclusively in I/O-, file system- and network service-calls. For a hierarchically structured operating system these calls do not invoke kernel functions directly and in this case it is possible to keep critical regions of the kernel fairly short. This reduces latency times for those processes using only kernel functions but not in general because we have the problem of critical regions on higher levels too.

Similar to the kernel services on higher levels use internal data structures to manage their objects and resources. These internal data structures may be temporarily inconsistent too. Sharing services between several user tasks leads to the same problems as descri- bed in section 3.2. These problems are resolved in the same way establishing critical regions guarded by kernel semaphores. Clearly we have a latency time on a higher level if a preempted region should be entered again during a service call of a process with higher priority. Services on higher levels are not so closely linked together. They form classes with no c o m m o n

critical regions for any two classes and the latency time on higher levels concerns only user tasks using the same service class. Especially tasks using only kernel services cannot be delayed by locked critical regions on higher level.

The problem of latency times on higher levels is weaker to some extent, but the problem still remains. We have seen that for the kernel latency time it is hard

70 H. Rzehak

to get some figures by measurements without know- ledge of the internal structure of the kernel. This is also true for latency times on higher levels. Information from vendors is quite unusual.

5. C O N C L U S I O N S

It is fairly well understood what services should be provided by an operating system for real-time applica- tions, and it is time to standardize it. The POSIX real- time extensions are going to establish such a standard. Although the standardization process is not completed real-time UNIX systems available today are more or less on the line of this standard.

Focusing on the Process Dispatch Latency Time and other unexpected delays we have discussed some crucial issues concerning real-time performance of operating systems and collected some results of measurements for UNIX systems claiming to have real- time properties. These results show that some of these UNIX Systems are well suited for a class of real-time applications. Compared with dedicated small real-time operating system kernels we have to take into account that UNIX offers a much more extensive functionality which costs some overhead, but simplifies user programs if needed. Disregarding operating economy the use of real-time UNIX systems makes sense in many cases.

It is difficult to compare performance data from diffe- rent vendors, because most of the time the metrics used for measurement are not exactly the same or the operat- ing conditions during measurements differ. The defini- tion of performance metrics and its mandatory support for POSIX implementations is a step in the right direc- tion. Our analysis shows that unexpected delays of considerable duration caused by the operating system may happen in user tasks but are not included in these metrics. Unfortunately these unexpected delays are not easy to measure without inside knowledge of the kernel.

6. R E F E R E N C E S

Doughty, S.M.; S.F. Kary; S.R. Kusmer and D.V. Larson (1987a). UNIX for Real Time; Proceedings of the UniForum 1987, Washington D.C., January, pp. 219-230

Doughty, S.M. (1987b). Adding Real Time Capabilities to the UNIX Operating System; Proceedings of the PEARL 87 Workshop tiber Realzeitsysteme, Boppard, 3.-4. Dez., pp. 151- 165

Fogelin, J.: The VxWorks Real-Time Kernel; Wind River Systems, Alameda

Furht, B. et al. (1991a). Real-time UNIX Systems: Design and Application Guide; Kluwer Academic Publishers

Furht, B.(1991b). The Reality of a Real-Time UNIX Operating System; Proceedings of the convention Echtzeit '91 (Ed. H. Rzehak), Sindelfingen, 11.-13. Juni, pp. 281-296, Ludwig Drebinger GmbH, Mtinchen

Gralla, C. (1991). Quantitativer Vergleich yon drei Echtzeitbetriebssystemen fur die digitale Regelung; Proceedings of the PEARL 91 Workshop tiber Realzeitsysteme, Boppard, 28.- 29. Nov., pp. 141-155, Informatik- Faehberichte, Springer-Verlag

Kamenoff, N.I.; N.H. Weidermann (1991). Hartstone Benchmark: Requirements and Definitions; IEEE Real-Time Systems Symposium, San Antonio, Dec. 4-6, pp. 199-208

Kriechbaum, W. (1991). Adding Real-Time Capabilities to a Standard UNIX Implementation: The AIX Version 3.1.5 Approach; Proceedings of the convention Echt- zeit '91 (Ed. H. Rzehak), Sindelf'mgen, 11.-13. Juni 1991, pp. 63-71, Ludwig Drebinger GmbH, Mtinehen

Levi, S.T. and A.K. Agrawala (1990). Real-Time System Design; McGraw-Hill Publishing Company, 1990

Rajkumar, R. (1991). Synchronization in Real-Time Systems: A Priority Inheritance Approach; Kluwer Academic Publishers

REAL/STAR 1000 and 2000 Series Benchmark Report; MODCOMP Benchmarking Group, Modular Computer Systems, Ft. Lauderdale, 1991

Rzehak. H. (1991 ). Echtzeitdatenverarbeitung - Grundlagen und Methoden far die Praxis; Elektronik, Nr. 5 - Nr. 7; Franzis-Verlag Mtinchen

Rzehak. H. (1992). Distributed Systems for Real Time Applications-Using Manufacturing Automation as an Example; In: M. Schiebe, S. Pferrer (Eds.), Real-Time Systems, Engineering and Applications, Kluwer-Verlag

Rzehak. H. (1992). Distributed Real Time Systems for Manufacturing Automation; IFAC Symposium on Intelligent Components and Instruments for Control Applications (SICICA '92), Malaga, May 20-22

Stieger, K. (1991). Eine PEARL-Ablaufumgebung unter Ausnutzung zweier unterschiedlicher Betriebssysteme; Proceedings of the convention Echtzeit '91 (Ed. H. Rzehak), Sindelfingen, 11 .- 13. Juni, pp. 203-212, Ludwig Drebinger GmbH, Mnnchen

Trennhaus, W. (1989). Das Mikrocomputersystem SX mit seinem Echtzeit-UNIX-Betriebssystem SORIX: Ein System far die Realisierung von verteilten Automatisierungs-Aufgaben; PEARL 89 Workshop tiber Realzeitsysteme, Boppard, 6.-8. Dez., pp. 157-171, Informatik- Fachberichte, Springer-Verlag

Zeltwanger, H. (1991). Trend zu Echtzeit-Unix und Crossentwicklung unter Unix; Elektronik 5/1991 S. 186-190; Franzis-Verlag Mtinehen

Portable Operating System Interface for Computer Environments; ISO/IEC 9945-1:1990 and IEEE Std 1003.1-1990 (POSIX.1)

Realtime Extensions for Portable Operating Systems; IEEE PlOO3.4/DI2 Feb. 1992

Realtime System API Extension; IEEE PIOO3.4b/DI Nov. 1991

Standardized Application Environment Profile-POSIX Realtime Application Support (AEP); IEEE P1003.13/D4 Nov. 1991

Threads Extensions for Portable Operating Systems; IEEE PlOO3.4a/D6 Feb. 1992