recording of scheduling and communication … · web viewrecording of scheduling and communication...
TRANSCRIPT
Master’s Thesis in Computer Science, with a focus on Real-Time Telecom Systems
Recording of Scheduling and Communication Events on Complex
Telecom Systems
AUTHORSMuhammad Imran Mughal
Razwan Javed
SUPERVISORJohan Kraft
19th June 2008
School of Innovation, Design and EngineeringMälardalen University
Note
Due to business secrecy, the companies involved in this thesis may not be revealed. Company A is a consultancy firm, providing services for Company B, a major telecom company.
Abstract
Monitoring and debugging for a real-time system is a complicated problem due to the lack of a set of advanced tools and adequate operating system capability. Software tools can cover the wide range of the software development life cycle from the requirement analysis phase to debugging and maintenance phases. However, the tools available for development of traditional PC software are not sufficient for development of real-time systems. Real-time software tools and effective kernel support are essential to facilitate the development of real-time software as this allows developers of real-time systems to better understand and troubleshoot their systems.
The Industrial Research and Innovation Lab at MRTC (Mälardalen Real-Time Center) contains real telecom system, hardware and software. The software system is based on the real-time operating system OSE. The telecom applications that run on this system are often highly complex and hard to troubleshoot since they are often distributed on different nodes and have strict performance requirements, which make monitoring hard.
Company A and Company B are therefore interested in tools for monitoring the CPU usage and behavior of the system at runtime, for instance the communication and timing of the various processes in the system. At MRTC they are doing closely related research, but they have not yet worked with OSE based systems. This thesis has been formulated to find a suitable technical solution to evaluate the Tracealyzer tool on OSE. We have investigated existing tools and implemented a recorder for the Tracealyzer tool and evaluated it under realistic conditions. The evaluation showed that the CPU usage was quite low, about 1% at 30 % CPU load.
Acknowledgements
This is a master thesis in the field of Computer Science and Electronics. The work present in this thesis was done at Industrial Research and Innovation Lab at MRTC (Mälardalen Real-Time Center) at Mälardalen University in Västerås.
We are very thankful to “Company A” for contributing to this project by allowing us to do measurements on their hardware.
Great thanks go to Ola Liljedahl, Director of Product Enablement and Product Management, ENEA USA for helping us with different investigations about OSE and answering a lot of questions through emails.
We would like to express our gratitude to our supervisor Johan Kraft for his excellent guidance and countless support during this work. He dedicated much time supervising us and pointed us to right direction. His advice has been of vital importance during our work.
We also wish to thank for our lab colleague Mikael Krekola for his companionship and discussions and also for technical support and kind help for this thesis work. We are also thankful to Daniel Flemström, the Lab manager at MDH.
Finally we want to express our gratitude to our parents for their love and support.
Table of Contents
Introduction..........................................................................................................................11.1 Background................................................................................................................11.2 Problem Definition....................................................................................................21.3 Purpose......................................................................................................................31.4 Objectives..................................................................................................................31.5 Method.......................................................................................................................31.6 Thesis Overview........................................................................................................4
Theoretical Background....................................................................................................62.1 Company B’s platform..............................................................................................62.2 Embedded and Real-Time Systems...........................................................................62.3 Real-Time Operating Systems (RTOS).....................................................................72.4 The OSE Operating System.......................................................................................8
2.4.1 History of OSE...................................................................................................82.4.2 The OSE Family.................................................................................................82.4.3 OSE Concepts.....................................................................................................9
2.4.3.1 Kernel..........................................................................................................92.4.3.2 Process Management...................................................................................92.4.3.3 Interprocess Communication (IPC)...........................................................112.4.3.4 Memory Management................................................................................112.4.3.5 OSE Kernel Handlers................................................................................12
Monitoring and Debugging of Real-Time Systems.......................................................143.1 Monitoring Systems.................................................................................................14
3.1.1 Terms and Notations.........................................................................................143.1.2 Application Domains........................................................................................163.1.3 Probe Effect......................................................................................................173.1.4 Monitoring Levels............................................................................................17
3.1.4.1 Process-Level Monitoring.........................................................................173.1.4.2 Function-Level Monitoring.......................................................................17
3.1.5 Monitoring Targets...........................................................................................183.1.6 Monitoring Approaches....................................................................................18
3.2 Debugging Systems.................................................................................................193.2.1 Static Debugging..............................................................................................193.2.2 Dynamic Debugging.........................................................................................20
3.2.2.1 Cyclic Debugging......................................................................................203.2.2.2 Event-Based and Post-Mortem Debugging...............................................203.2.2.3 Relative Debugging...................................................................................20
3.2.3 Debugging with Monitoring Support................................................................21Related Work...................................................................................................................22
4.1 The Tracealyzer.......................................................................................................234.2 Wind River Workbench...........................................................................................254.3 OSE Illuminator.......................................................................................................264.4 LTT and relayfs.......................................................................................................274.5 ART Real-Time Monitor.........................................................................................29
4.6 TraceX.....................................................................................................................314.7 Dynamic Probes (DProbes).....................................................................................324.8 Kernel Probes (kProbes)..........................................................................................324.9 DTrace.....................................................................................................................324.10 Conclusions............................................................................................................32
Recording of Scheduling and Communication Events in OSE...................................345.1 Technical Requirements of Tracealyzer..................................................................345.2 How to Get Tracealyzer Requirements in OSE.......................................................345.3 Methods for Recording of Scheduling Events in OSE............................................36
5.3.1 The Kernel Handler Implementation Method...................................................365.3.2 OSE Kernel Modification.................................................................................37
5.4 Methods Evaluation.................................................................................................375.4.1 Analysis............................................................................................................375.4.2 Evaluation and Choice......................................................................................38
Implementation................................................................................................................406.1 Design Issues...........................................................................................................406.2 Tools and Project Structure Setting.........................................................................416.3 Memory Configuration............................................................................................426.4 Kernel Handlers.......................................................................................................43
Investigation of CPU Overhead of Recorder................................................................487.1 Execution time of Probe......................................................................................487.2 Measure number of context switches..................................................................487.3 CPU usage of create_handler_process.................................................................487.4 CPU overhead of Recorder..................................................................................48
Conclusions and Future Work.......................................................................................508.1 Solution....................................................................................................................508.2 Limitations...............................................................................................................518.3 Future Work.............................................................................................................51
References..........................................................................................................................52
Index of Figures
(Figures 1 and 2 removed from public version of report)
Figure 3: Real-time embedded systems
Figure 4: Deadline example in distributed system
Figure 5: Possible process states in OSE
Figure 6: Memory organization in OSE
Figure 7: Tracealyzer tool with a task switch view
Figure 8: CPU Usage graph
Figure 9: Wind River Workbench’s graphical interface and log files view
Figure 10: The Event Viewer displays a trace of all sent signals in the current state.
Figure 11: The CPU Profiler displays the CPU load of an application.
Figure 12: Graphical Viewing Tool
Figure 13: Real-Time Monitor’s Visualizer
Figure 14: Visualized System Behavior
Figure 15: TraceX Visualizer
Figure 16: Kernel handler implementation flow chart
Figure 17: Project Structure
Figure 18: Kernel handler and user area activation
Figure 19: User area configuration
Figure 20: Create handler flow chart
Figure 21: Kill handler flow chart
Figure 22: create_handler_process
Figure 23: Explanation of functions in create_handler_process flow chart
Figure 24: Swap in handler flow chart
1
Chapter 1
Introduction
1.1 Background
The development of embedded real-time systems has grown so much nowadays that they can be found everywhere in our everyday life. There are many industrial areas, in which the application of real-time systems is vitally necessary. Some examples are: telecommunication, computer-aided production in factories, computer-controlled power and atomic plants, vehicular, aerospace industry, etc.
Company A develops parts of Company B’s base software platform for telecom systems. The platform is targeting distributed systems with a network of cooperating nodes, which together make up the system functionality.
The development of real-time systems needs much more time than the development of non real-time systems, because testing and debugging of real-time systems are much more difficult due to several reasons. It is usually not possible to use debuggers in a real-time distributed system. If a single CPU is stopped by the debugger, processes running on other CPUs will probably fail due to a timeout. Monitoring is solution which does not halt the system, but only records the behavior for later presentation. A common problem of monitoring embedded system is performance; the CPU usage of the recorder may be unacceptable. Another aspect of the difficulty in monitoring comes from the invasive nature of the real-time monitoring activity in the distributed environment. It not only interferes with the processor scheduling, but also with communication scheduling and activities [43]. The lack of kernel support makes monitoring/debugging for a distributed real-time system more complicated.
There are two types of errors we often encounter while we monitor or debug a distributed real-time program. One is related to logical error and the other is related to timing error. Both errors are very difficult to track down in a real-time environment. Particularly in a distributed real-time environment, a lack of instantaneous, accurate global state or event ordering creates extra complexity in analyzing program behavior.
Monitoring is often used to measure CPU usage. Two types of CPU usage measurement tools are commonly available: hardware tools, such as logic analyzers, and software tools, such as profilers. The problem with hardware monitors is that they provide too detailed information which makes it hard to get an overview of the behavior and they are very expensive. Software profilers typically collect data in one of two basic ways: instrumentation and sampling. With instrumentation, profilers inject code into key points of the system (such as at the top of every method call); this code then records the event at
2
run-time for subsequent analysis by the profiler. With sampling, the system remains unmodified and is instead analyzed periodically by the profiler in runtime, allowing inspection of such metrics as the amount of CPU time used, for processes and/or functions.
Although they have proven successful for analyzing desktop software and parallel computers, instrumentation and sampling can be enormously problematic for real-time systems. The fundamental flaw is that the very act of observation disturbs the system’s real-time characteristics in a way that may cause missed deadlines and scheduling conflicts that would not occur under normal execution. If using instrumentation, this problem can be avoided by leaving the instrumentation in the release system, so that it becomes a part of the tested system.
This approach is used by the Tracealyzer, a viewer and analysis tool for embedded real-time systems. It presents the system behavior (from a recording) at a higher level of abstraction compared to debuggers, with focus on process scheduling and communication. This is very helpful for understanding the behavior of complex embedded systems. Tracealyzer is also very useful for getting an overview of a system's resource usage, for instance CPU-usage in total and per process. The Tracealyzer uses instrumentation manually inserted in the base software platform. The application software does not have to be modified.
In this chapter the reader will be introduced to the key features of the thesis along with problem definition and methods that will be used to fulfill these. This chapter ends with a thesis overview that briefly describes the content of each chapter.
1.2 Problem Definition
The CPU-usage measurement tools used by Company A are not accurate enough as they use a sampling approach and give different result from time to time. Company A is in need for a tool that could be used to monitor their real-time systems. They especially need the ability to accurately measure the CPU usage for each process and the entire system. There are several existing commercial tools for logging information from systems and showing the result either graphically or textually that should be investigated. The Tracealyzer tool is the primary candidate in them due to availability and experience of the tool on other systems. An important issue is the amount of CPU time that is required for the monitoring, i.e. the overhead.
1.3 Purpose
The main business goal with Tracealyzer, from Company A’s point of view, is the possibility to use Tracealyzer for testing activities. This thesis will investigate the possibility to use of the Tracealyzer tool for visualizing the behavior, timing and CPU usage of software running on the OSE platform. The focus lies on CPU usage and
3
visualization of context switch for processes so the engineers can see if a process consumes more CPU cycles than expected. This can facilitate troubleshooting and give developers a better understanding of the system.
1.4 Objectives
The stated problem has been broken down into the following questions:
1. What information is required by the Tracealyzer recorder to measure CPU-usage and display the system behavior?
2. Is it possible to record the information needed by the Tracealyzer from OSE to measure CPU-usage, preferably without changing the OSE kernel code?
3. What is the impact of this type of recording, in a typical case and in a high load situation?
a. What is the execution time of a probe?b. What are the rates of different types of events?
2. Is the impact of the recording acceptable, if not, how can it be reduced?a. Is it possible to select what events to record, to reduce the number of
probes that are executed?b. Is it possible to reduce the execution time of a probe?
1.5 Method
The thesis project was initially commenced with a study for acquaintance with the Tracealyzer tool, how recordings are visualized by Tracealyzer, technical area of real-time operating system. The OSE real-time operating system from ENEA, which is used by the involved companies, was studied in particular. Discussions on possible methods for recording of scheduling events in OSE were made with Company A and ENEA personnel with great experience in OSE. These discussions lead to a study of the handlers in kernel in OSE.
To measure CPU-usage and display the internal behavior of the system a software recorder was implemented on OSE, compatible with the Tracealyzer tool. Finally the recorder was evaluated on the test cases provided by Company A. The purpose of the recorder evaluation was to see how much it will affect the behavior of the target system and how much CPU will be used by this recorder.
4
1.6 Thesis Overview
This master thesis report will cover the basic information on operating systems, real time operating systems and specific information about the OSE RTOS, from ENEA. Those basics are necessary for comprehension of the thesis task. The structure of the thesis report is outlined as follows:
Chapter 1: Thesis Introduction, Problem Definition and Method
Chapter 2: Theoretical Background
Chapter 3: Monitoring and Debugging of Real-Time Systems
Chapter 4: Related Work
Chapter 5: Recording of Scheduling and Communication Events in OSE
Chapter 6: Implementation
Chapter 7: Investigation of CPU Overhead of Tracealyzer Recorder
Chapter 8: Conclusions and Future Work
5
Chapter 2
Theoretical Background2.1 Company B’s platform(removed from public version of report)
2.2 Embedded and Real-Time SystemsGrowing demands on functionality in today's electronic products is leading to an increasing shift towards developing systems in software and programmable hardware in order to increase design flexibility. Most of today's electronic products are based around programmable integrated circuits. These integrated circuits are usually embedded as part of a complete device including hardware and mechanical parts. In contrast to embedded systems a general-purpose computer, such as a personal computer, can do many different tasks depending on programming. A general definition of embedded systems is:
“Embedded systems are computing systems with tightly coupled hardware and software integration that are designed to perform a dedicated function. The word embedded reflects the fact that these systems are usually an integral part of a larger system, known as the embedding system. Multiple embedded systems can coexist in an embedding system [6]”
A real-time system is a system that is required to react to stimuli from the environment (including the passage of physical time) within time intervals dictated by the environment. Real-time systems can be defined as
“A system where correctness of the system behavior depends not only the logical results of the computations, but also on the physical instant at which these results are produced [6]”
This definition emphasizes the notion that time is one of the most important entities of the system, and there are timing constraints associated with systems tasks. Such tasks have normally to control or react to events that take places in the outside world, which are happening in “real time”. It should be noted here that real-time computing is not equivalent to fast computing. Fast computing aims at getting the results as quickly as possible, while real-time computing aims at getting the results at a prescribed point of time within defined time tolerances. Thus, a deadline (for this point of time) can be associated with the task that has to satisfy this timing constraint specifying either its start or completion time. If the meeting of the deadline is desirable but not mandatory, the task is said to be a soft real-time task. On the contrary, if the task has to meet the deadline, because otherwise it will cause fatal errors or undesirable consequences, the task is called hard real-time task (e.g. an airbag control system in the cars). Such systems, where lives are in danger on failure, are called “safety-critical”
6
A good way to understand the relationship between real-time systems and embedded systems is to view them as two intersecting circles, as shown in Figure 1. It can be seen that not all embedded systems exhibit real-time behaviors nor are all real-time systems embedded. However, the two systems are not mutually exclusive, and the area in which they overlap creates the combination of systems known as real-time embedded systems [6].
Figure 3: Real-time embedded systems [6].
2.3 Real-Time Operating Systems (RTOS)The definition of a real-time system is that the program execution has certain timing deadlines that have to be kept. Deadlines in real-time systems are often divided into two types: Hard and Soft. Where in hard deadline real-time systems, missing a deadline causes the system to crash in every case. For soft deadline real-time systems, deadlines can be missed without crashing the system. Another way to look at it is that the type of the deadline depends on the consequences for missing it, e.g. lowered efficiency in a car engine or an unstable measuring system.
Keeping real-time constraints can be difficult even on single processor systems. Therefore on distributed real-time systems it’s even more difficult too keep the deadlines. A distributed system is a system with several processors, running separate from each other. The deadlines are kept through a carefully considered scheduling algorithm. An example of missing a deadline is shown in figure 4. Here a reply message arrives too late.
7
Figure 4: Deadline example in distributed system, taken from [7]
In the figure process A needs some service form process B in order to continue the execution. The process sends a message to B requesting the information. B receives the message and sends a reply back to process A. If the reply message from B is not received by process A within the deadline, either an exception is raised (soft realtime systems) or the system requirements are not kept (hard real-time systems).
2.4 The OSE Operating System
2.4.1 History of OSEIn the mid 1980’s Enea Data AB developed a real-time operating system, which they decided to call “OSE”. Today, OSE is developed and marketed by Enea OSE Systems AB, a subsidiary of Enea Data AB. Among Enea OSE Systems’ customers some of the World’s leading companies in the telecom industry are found.
2.4.2 The OSE FamilyThe OSE system is actually more than one operating system; it is a family of related operating systems. First, there is the OSE Auric, which is intended for small real time systems with strict memory constraints. Second comes the OSE Basic whose use is in static real-time systems. Then comes the OSE Classic that is intended for compact real-time systems, perhaps with multiple CPUs. Last comes the OSE Delta which is intended for use in larger fault-tolerant distributed systems. This thesis focuses on OSE Delta.
8
2.4.3 OSE ConceptsThis chapter is intended to give the reader an idea of what OSE is and how it can be used to solve both simple and complex problems.
2.4.3.1 KernelThe kernel of the OSE system is of what might be called a micro-kernel. This means that the kernel is stripped from most unnecessary services, what is necessary or not is of course a matter of opinion. What this tries to say is that most services such as the file system and the heap management etc. is located in processes outside the kernel.
In the OSE environment there exists two types of kernels. One is meant to run on an embedded system environment, a target, the other is meant to run in a host environment, such as the Solaris OS or the Win32 environment.
2.4.3.2 Process ManagementThe most fundamental building block in an OSE system is the process, since it is through the use of processes that a system allocates CPU time. OSE processes correspond to light weight processes, threads and tasks in other operating systems. Every process is assigned a name and numeric identifier. The same name may be assigned to several processes independently, but the numeric identifier, called the process id or “pid” for short, is assigned by the system and is unique for each process running [8].
Depending on when a process is declared and created it will belong to one of two process group, the Static process group or the Dynamic process group.
Static process – is a process declared in the system configuration files, and hence started at startup of the system. It is not allowed to die or get killed while the system is running.
Dynamic process – is a process that is not started at the system startup. A dynamic process is created during the execution of the system, and may be recreated at a later occasion as well. It is also allowed for such a process to die or get killed while the system is running.
Process TypesIn the OSE system there exists five different kinds of processes for the programmer to use, to enable prioritized tasks during execution and abstract programming.
Interrupt processes – An interrupt process run as soon as there is a hardware or software interrupt, to which it is assigned. These are the most prioritized processes [8].
Timer-interrupt processes – On the timer-interrupt level among the interrupt levels these processes exist. A timer-interrupt process runs at cyclic intervals.
Prioritized processes – A prioritized process will run as long as it does not suspend itself or any higher priority process is ready to run. These are scheduled in round-
9
robin fashion within a priority level. The next process to run in a level will get access to the CPU and other resources, as soon as the running process suspends itself
Background processes – These processes run in a round-robin time sliced environment. As long as no other prioritized process executes and the process’ time slice has not yet expired, the background process will execute.
Phantom processes – These processes are not actually processes, in the same way as the other processes, they are a way to perform abstraction on process level. It redirects incoming signals to the one or more processes according to a local redirection table.
Process PrioritiesThere exist 32 priority levels for each of the prioritized process types [8], with a priority of 0 as the most prioritized and of 31 as the least. The interrupt processes, as well as the timer-interrupt processes, should only execute a small block of code that for example sends a signal, and then exits, to decrease the time when hardware and software interrupts are marked out. It is possible to assign a certain priority to each process.
Process StatesA process in OSE is always in one of the following states [8]:
Running - The CPU is currently assigned to this process. In a single processor system, only one process can be in this state at a time.
Ready - All processes ready to run are placed in a ready queue. At each process-switch, the first process in the ready queue is scheduled for execution. Each process priority level has its own ready queue. Because a process of higher or equal priority is currently running, the processes in a ready queue may not be allowed to run.
Figure 5: Possible process states in OSE from Enea OSE Documentation [8]
Waiting - The process is either waiting for some event to occur or it is stopped. Waiting processes do not require the CPU.
10
Process Scheduling principlesThese are the different scheduling principles used in OSE [8]:
Pre-emptive - The operating system can pre-empt the current process at any time, even if the current process is executing a system call, i.e. the operating system can change execution to another process at any time. All processes in OSE use this scheduling method.
Cyclic - Processes can be scheduled to run at certain time intervals. This scheduling method is used for timer-interrupt processes in OSE.
Priority Based - The process with the highest priority will run as long as no interrupts are in service and the process is not waiting for some event to occur. This scheduling method is used for all prioritized processes in OSE.
2.4.3.3 Interprocess Communication (IPC)The way to pass data between processes in an OSE system is by signals. In an OSE system a signal is a message that is sent directly from one process to another. A message contains an ID, and the addresses of the sender and the receiver, as well as data. Once a signal is sent, the sending process can no longer access it, i.e. ownership of a signal is never shared. The receiving process may specify what signal it wants to receive at any particular moment.
The process can wait or poll the signal queue for a specific signal. The kernel manages the ownership of signals. The known ownership makes it possible for a system debugger to keep track of the communication between processes, e.g. it makes it possible to set a break point on a specific signal during system debugging. Normally when transferring signals, the sending process only sends a pointer to a signal buffer. The receiving process uses this pointer to access the signal buffer. This has the advantage of making the system fast, but there is also the danger of the receiving process destroying the pool of the sending process. However, if a signal is sent between processes that are located in different segments the user can choose between having only the pointer transferred and having the signal buffer copied. Signals will die with the owning process and be cleaned up by the OS itself.
2.4.3.4 Memory ManagementIn the OSE real-time kernel, different levels of memory management support both small systems and large, complex systems with memory protection and other controlled accesses to pooled resources [9], such as heap memory. The system pool is located in the kernel memory segment. All processes can allocate memory from the system pool. The OSE real-time kernel provides resource pools at multiple levels: blocks, local pools and memory protected segments to ensure better fault isolation and more reliable system operation.
11
Figure 6: Memory organization in OSE taken from OSE documentation [8]
To avoid fragmentation in the memory pools, OSE allocates memory to the processes in predefined memory-chunks. This also ensures to keep the system deterministic, as it make its easier to keep information of how much and which blocks of memory is free.
2.4.3.5 OSE Kernel HandlersIt is possible to configure handlers into the kernel for certain purposes. These handlers are called by the kernel at points where it may be desirable for a user to provide information to the kernel. Any number of handlers may be declared. If a user area is declared, a pointer to it is passed to the handlers. There may be several handlers, in which case they will all be run in undefined order every time a new process is created, scheduled or killed.
There is a create handler that is called each time a process is created. Swap handlers are called when a process is scheduled. Each handler that is present must be configured into the kernel with a statement in the kernel's configuration file, ospp.con. A typical example of the syntax for a kernel configuration statement is:
CREATE_HANDLER(<name>)
This means that the user wants to specify that the kernel should contain a create handler. The <name> field in the configuration example should be changed to the name of the subroutine containing the create handler's code. Any handler may be written in the C programming language or in assembler. Consider the following example of a configuration statement:
CREATE_HANDLER(create_hnd)
12
This means that there is a function named "create_hnd" that is to be linked with the kernel and called during each process creation.
13
Chapter 3
Monitoring and Debugging of Real-Time Systems
The objective of this chapter is the presentation of the basic concepts for monitoring of real-time systems.
3.1 Monitoring SystemsDuring monitoring process run-time information are gathered from the target system, which represent the behavior of the target system. Monitoring systems are used in run-time for extraction, analyzing, presentation and (probably) acting on gathered information from systems that are monitored [10,11].
In the current research we can find different definition of monitoring made by different researchers. In [12] Schmid defines monitoring as the process of extracting and gathering information regarding the behavior of a particular system. In [11] Joyce et al. define monitoring as the process of dynamic collection, interpretation and presentation of information concerning objects or software processes under scrutiny.
3.1.1 Terms and NotationsIt is the objective of this section to present the terms and notations that are used in the real-time monitoring.
Target and Monitoring System: The system that is monitored by the monitoring system is called target system [10]. The application that is executed on the target system is called target application. The monitoring system is a system that is used for monitoring of monitored systems (target systems) [13].
Sensor and Probes: Urting and Berbers in [14] define a sensor as an entity (software or hardware) that observes a small part of the system, and which is responsible for gathering of monitoring information. Schroeder in [15] defines a sensor as an entity that observes the behavior of a small part of the application system state space. Sensors can include additional functionality. For example, the user can define conditions that must be satisfied before sensors start their operation for gathering monitoring information. The sensor collects (gathers) monitoring information when it is triggered. Schroeder in [15] denotes that the triggering process can be done either by the change to the entity the sensor observes or by a request from the monitoring system. Depending on how the sensor is triggered, there are two different types of operations that can be done by the sensors: sampling and tracing. Kaelbling and Ogle in [16] define sampling as an asynchronous operation, which is started on-demand by a monitoring system for gathering of monitoring information. If a monitoring system decides to collect monitoring information from a set of sensors, it sends a message to
14
them, and they return the current values from entities they are observing. On the other side, tracing is a synchronous operation, during which the sensor reports the new value of the entity it observes, when the state of the entity is changed. Dias and Richardson in [17] state that by sampling, information about the execution state is collected synchronously (in a specific time rate), or asynchronously (through the direct request of the monitoring system), while by tracing, information is collected when an event of interests occurs in the system [18]. Sampling sensors can be applied in applications where no need for collection of all events is imperative. However, in cases where collection of all events must be guaranteed, then tracing sensors must be used. In this case the sampling rate of the sensor must be at least higher than the fastest expected change rate of the entity the sensor observes [14]. Sensors are widely used by software monitoring systems. An example is given in [19] by Mahrenholz, where the use of two different types of sampling sensors is presented: block and expression level sensors. He uses block sensors for collection of entries into and/or exits from basic code blocks (e.g., functions, etc), while expression level sensors are used for collection of the pass of a single code position or the state of a variable at this point. In [18] Ogle et al. differentiate sensors from probes. Whereas sensors are small pieces of code residing within the program being monitored, the probes are code fragments residing within the resident monitor (rather than application), i.e., the resident monitors are responsible for gathering of run-time information from target systems. The main advantage of probes over sampled sensors is that the application code need not be changed for probing, so that the information to be probed can be defined dynamically.
Events, Actions and Event Histories: In [20] Schroeder determines that monitoring systems collects monitoring information in form of events. The relevant events occurring in a target system are time stamped, collected, and analyzed by a monitoring system at run-time. Furthermore, the collected events are stored into event histories for analyzing of the past history and the output of the analyzing process can be used for feedback to the target system or correctness checking of the target system [21].
Intrusive vs. Non-intrusive Monitoring: A monitoring system is called intrusive, if it uses resources of the target system during gathering of runtime information. On the other hand, a monitoring system is called non intrusive, if it does not use any resources of the target system during information gathering.
Time vs. Event-Driven Monitoring: Time-driven monitoring systems periodically gather monitoring information from target systems (i.e., the state of target systems is periodically sampled by the monitoring system), while event-driven monitoring systems collect occurrences of events of interests from target systems.
Synchronous vs. Asynchronous Monitoring: A monitoring system is called synchronous if collected monitoring data are processed by monitoring system running on the target system. A monitoring approach is called asynchronous if collected monitoring data are processed by a monitoring system running on a separate task or machine.
Run-Time Monitoring: In general, we can define the run-time monitoring as a process that has to collect relevant events (i.e., monitoring-data at the intended abstraction level) from the target system and to process these collected events in run-
15
time. Urting and Berbers in [14]determine that run-time monitoring systems consist of three important parts:
o Annotation Formalism: an RTL like formalism for annotating timing constraints,
o Probes: are responsible for intercepting events and time-stamping them, ando Satisfiability Checker: is responsible for checking of the RTL constraints
making use of the events generated by the probes.
Deterministic and Reproducible Monitoring: Deterministic and reproducible monitoring is very important, because it enables the user during debugging of the target system to deterministically reproduce the behavior of the target system at the intended abstraction level which is observed during monitoring process. This is very helpful in case of faults, because it helps the user to locate the faults on the target system. In order to achieve that, the monitoring system must be able to collect all needed information, from which the run-time behavior of a target system at the intended abstraction level can be represented. However, the exact reproduction of the target behavior strongly depends on the architecture (and nature) of the target system. Deterministic and reproducible monitoring of a target system that can be regarded as deterministic is easier than the monitoring of a target system that cannot be regarded as deterministic.
3.1.2 Application DomainsApplication domains of monitoring systems are divided in following categories: testing and debugging, feedback monitoring, correctness checking, and performance evaluation.
Testing and DebuggingIn the last few years different debugging techniques were developed for debugging of real-time systems. These techniques can be classified into static and dynamic debugging techniques. The static debugging techniques are based on the static analysis of the source code. They try to detect parts of code that probably contain anomalies and (or) bugs. The dynamic debugging techniques are based on the monitoring techniques. The monitoring system collects run time information from the target system and the developer can analyze the collected run-time data in off-line, in order to find the bugs in the target system.
Feedback MonitoringThere are some cases in which the output produced by the monitoring system is used as a feedback by the target system to make decisions for further processing. This kind of monitoring application has been applied especially to assist the operating system’s scheduler aiding it to schedule task sets.
Correctness CheckingMonitoring systems can be used to ensure the consistency of the target system with its formal specification [20]. These monitoring systems can be considered as a feedback systems too, because they could trigger the target system to start some recovery actions if the monitoring system detects an invariant violation [21]. However, their primary goal is to observe and make sure that target systems are running conform their formal specifications.
16
Performance EvaluationMonitoring systems gather run-time information that can be used for assessment of system performance.
3.1.3 Probe EffectDuring the information gathering, the monitoring system can perturb or influence the target system. This is called probe effect or monitoring perturbation, which is a quantitative representation of the perturbation that the monitoring system causes on the target system. Probe effect changes the ordering and timing of events on the target system, on which the system's correctness depends. Since the probe effect is caused by the addition or removal of code instrumentation, not the presence of code instrumentation, the probe effect can be avoided by leaving the “probes” in, so it becomes a part of the tested system. This requires that the monitoring is always active, which is good for debugging but will decrease the system performance.
3.1.4 Monitoring LevelsMonitoring systems provide presentation of gathered monitoring information at different abstraction levels, in order to assist users for better understanding of the run-time behavior of target systems. In this section only two monitoring levels are discussed that are process and function abstraction levels.
3.1.4.1 Process-Level MonitoringFor monitoring program execution at the process level, Tsai et al. [13]consider a process as a black box which can be in one of the three states: running, ready, or waiting. Furthermore, at the process-level monitoring, Tsai et al. distinguish the events that affect the program execution (e.g., interprocess communication and synchronization events), from those events that affect execution at lower levels (e.g., assigning a value to a variable, procedure calls, etc.). Tsai et al. in [13] have presented two reasons for the necessity of monitoring of target systems at the process abstraction level:
Nondeterminism: Processes are the minimum program unit that can exhibit nondeterministic behavior. Furthermore, the interactions between different processes on one hand and the interactions between processes (of real-time applications) and their environment on the other hand, are frequently sources of possible faults. If the user can isolate anomalies on an individual process, then he can use monitoring at lower-levels of abstraction for successive fault isolation
Reconstruction: The execution behavior for interprocess communication and the interaction between the software processes using the collected process level events can be reconstructed
3.1.4.2 Function-Level MonitoringThe user can monitor the target system at lower abstraction levels for localization of possible faults, after they are identified in a process. One of these monitoring levels is the monitoring at the function abstraction level, at which the user can try to localize the
17
faulty modules or functions. Tsai et al. in [22] point out that functions (or modules) at the function level of abstraction are the basic units of the program model and each function can be viewed as a black box that interacts with others by calling them, or being called by them, with a set of parameters as arguments. At the function-level monitoring there are only two events of interests: function call and function return. Both of these events have their corresponding key values. Function call has following key values: calling function identification, called function identification, passed-in parameters, and time [23].
3.1.5 Monitoring TargetsThe goal of this section is the presentation of requirements and difficulties, which a monitoring (or debugging) system during monitoring (or debugging) of target systems has to fulfill or to overwhelm. The target systems that are presented below are considered to be real-time systems, i.e., they have to fulfill their timing constraints.
Single-Task (Sequential) Target Systems: The monitoring (or debugging) of sequential real-time applications do not differ from monitoring (or debugging) of sequential non real-time applications. However, the most important difference are the timing constraints that must be fulfilled by real-time applications, and hence, for debugging of real-time applications the traditional breakpoint debugging technique cannot be used. Real-time applications have to react in timely manner on stimuli generated by their environments, and if the debugging system stops the execution of a real-time application, it cannot stop the physical time, and the target real-time application is not anymore consistent with its environment in the time domain [24]. Thane in [25] indicates that another reason, why traditional debuggers cannot be used for monitoring of sequential real-time applications, is the impossibility to directly reproduce the inputs to the system that depends on the time, when the program is executed. Such examples are readings of sensors and the local real-time clock.
Multitasking Target Systems: Thane in [25] determines that during debugging (or monitoring) of multitasking applications, in contrast to the debugging of sequential applications, the mechanisms for reproducing of task interleaving must be supported by the monitoring (or debugging) system.
Distributed Target Systems: Tsai et al. in [26] define a real-time distributed computing system as a system that consists of a collection of communicating and cooperating processors or computers (nodes) that work toward a common goal, and on which critical timing constraints are imposed. Kirschbaum et al. in [27] determine that the special constraint of monitoring distributed real-time systems is to keep the interference with the target system so small, that the change in the system performance due to the monitor will neither effect the order nor the timing of events. Although in the most distributed systems there is no total ordering defined over events that occur on different nodes, monitored data must be collected from several sites and integrated to obtain a coherent view of the system [28].
3.1.6 Monitoring ApproachesMonitoring systems use different approaches for gathering of monitoring data, from which the influence of the monitoring system caused on the target system directly
18
depends. During gathering of monitoring data the target system must be probably instrumented, and depending on this fact, the monitoring approaches can be classified into: hardware, software, and hybrid approaches.
Hardware Monitoring: Hardware monitoring systems use dedicated hardware for gathering of significant events from target systems. The dedicated hardware snoops the busses of a target system and tries to match the bus signals. After successfully matching of significant events, they are stored for post- processing.
Software Monitoring: In contrast to hardware monitoring, software monitoring systems use only software for gathering of monitoring information and therefore, these monitoring systems share the computational resources with their target systems. The target systems must be instrumented with sensors or probes. These sensors or probes are executed on target systems and they are used for detection of significant events, which are stored in the memory of target systems.
Hybrid Monitoring: Hybrid monitoring systems try to combine the advantages of both above presented monitoring approaches. Hybrid monitoring systems typically use software probes for detection of significant events; however the collection of these events is the job of a specialized hardware, which is not directly dependent on the target system.
3.2 Debugging SystemsDebugging is defined by the ANSI/IEEE glossary as the process of locating, analyzing and correcting suspected faults", where fault is defined as an accidental condition that causes a program to fail to perform its required function. Watson in [29] determines that the following steps are a commonly used systematic approach for addressing of each error:
Gather Information: This step is used to establish the nature of the error, the behavior that is being observed, data structures likely to be involved, and a broad indication of the error location, such as the relevant module or subsystem.Analyze and Locate: Once sufficient information about the error has been obtained, it must be analyzed to determine the cause of the error.Correct the Error: After the cause of the error has been identified, a solution must be proposed.
Debugging techniques can be classified into static and dynamic techniques, depending on the fact, if the target system needs to be executed during debugging process or not.
3.2.1 Static DebuggingThe static debugging techniques are based on the static analysis of the source code. They try to detect parts of code that probably contain anomalies and (or) bugs. The static debugging techniques are not adequate for debugging of real-time systems, because they cannot deliver any information over the timing constrains and the behavior of the target system in the time domain.
19
3.2.2 Dynamic DebuggingIn contrast to the static techniques, the dynamic debugging techniques are applied at the run-time during execution of the target system. Watson in [29] presented three different classes of dynamic debugging techniques:
3.2.2.1 Cyclic Debugging One of the widely used dynamic debugging techniques is the cyclic debugging.Stewart and Gentlman in [30] determine that cyclic debugging refers to a repeatedly stopping execution of a program to examine the program state and then either continuing the execution or restarting it in order to stop some earlier point [30]. Tsai et al. in [13] point out that there are three techniques that are used to perform cyclic debugging:
Memory Dumps: This technique collects the program status, including program object code, register and memory contents and dumps it into a file, when the program execution is terminated abnormally or by user's request. However, this technique requires from the user to have a strong background in machine-level language to examine the dumped code.
Tracing: The tracing technique utilizes special tracing facilities in the compiler or operating system to continuously track and display every step of the program's execution, including control flow, data flow, variable contents, and the function calling sequence. Tracing gives the programmers a sense of the step-by-step flow of the program's execution.
Breakpoints: The breakpoint technique utilizes predicates that are inserted by the programmer or by the system to suspend program execution. Once a predicate is satisfied, the program execution is suspended and the relevant run-time information, such as variable values, stack contents and register values, can be displayed.
3.2.2.2 Event-Based and Post-Mortem DebuggingEvent-based debuggers view a parallel program execution as a sequence of events that are collected into a large event history that can later (post-mortem) be browsed, analyzed or even replayed to force deterministic execution of a non-deterministic program [30].
3.2.2.3 Relative DebuggingA relative debugger is a system which allows the user to compare the state of two executing programs [31]. Relative debugging is not applicable for debugging of real-time systems, because the real-time systems are usually not implemented for different platforms.
Most of the above presented debugging techniques are not suitable for monitoring of real-time and especially distributed real-time systems. Especially the breakpoint technique, which is the most often used cyclic debugging technique for debugging of non real-time systems, cannot be applied for debugging of real-time and specifically distributed real-time systems for two reasons:
20
Consistency Problem: Real-time systems are used for controlling of physical processes (their environment), whose state is changed with the progression of the physical time. If the execution of a real-time system is stopped by the debugger, then the real-time system is not anymore consistent with its environment. An alternative to this problem is the use of environmental simulators.Global Time: Another problem is the absence of the perfectly synchronized global time by many distributed real-time systems, which makes impossible the simultaneous stop of execution of the distributed real-time system.
3.2.3 Debugging with Monitoring SupportAs depicted above the traditional debugging techniques are not suitable for debugging of real-time and especially distributed real-time systems. Therefore, debugging approaches with monitoring support must be used for debugging of real-time systems. Tsai in [13] presents the following classification of debugging approaches with monitoring support:
Real-time Display: The execution of a program is monitored and displayed in a continuous real-time mode. In this approach the target program cannot be suspended or repeated, and no modification to the target program is allowed.
Real-time Debugging: The execution of a target program is monitored, analyzed, and debugged in a continuous real-time mode. In this approach the execution of the target program cannot be suspended or repeated, but it can be modified automatically in real-time during execution.
Interactive Debugging: The execution of a target program is monitored and will be suspended if a predefined situation is encountered. In this approach the target program can be suspended and modified, but the execution is not repeatable.
Deterministic Replay: A trace of the program execution is recorded during execution and can be replayed deterministically after the execution. This approach will be presented in more detail in the next section.
Dynamic Simulation: A trace or log is recorded during a target program's execution and is used for dynamic simulation to evaluate the run-time behavior of a target program in a different run with different test data.
21
Chapter 4
Related WorkIn this section, we described existing monitoring approaches for operating systems and evaluate their applicability to real-time systems.
Technical Terms and Abbreviations
Term/Abbreviation DescriptionLTT Linux Trace Toolkit
ARTS A Real-Time System
BSD Berkeley Software Distribution
API Application Programming Interface
DProbes Dynamic Probes
kProbes Kernel Probes
kernel The core software component of a system, providing the system with access to resources.
22
4.1 The TracealyzerTracealyzer is a viewer and analysis tool for recordings of embedded systems [32]. The recordings are obtained using a software recorder that is integrated in the base platform of the system.
The Tracealyzer’s main view
Figure 7: Tracealyzer tool with a task switch view
The software recorder records the task switches (i.e. the task scheduling), important system calls that impact scheduling (like IPC, delays and priority changes) as well as generic data probes and stores the recorded events in a ring-buffer, that always contains the most recent information. Apart from recording the task switches, the software recorder also allows recording of generic data values, using data probes. It stores the recorded events in a ring-buffer that always contains the most recent information. An event only requires 4 bytes, so a configuration allowing 100000 events only requires about 400 KB. This allows for recording of many seconds of execution. The exact length
23
depends on the rate of the recorded events. For instance 5000 events (e.g. context-switches) per second, will give a trace of 20 seconds in this case. The buffer can easily be extended to allow for even longer recordings, if desired. The Tracealyzer tool presents the recorded data in parallel with the process execution trace, which makes this a nice method for visualizing resource usage, state information or application specific events.
Tracealyzer have been introduced in the software development at ABB Robotics, a world leading developer of industrial robots and robot control systems. The Tracealyzer recorders have successfully been implemented for ABB Robotics on operating systems such as VxWorks and RTXC. Due to the simplicity and low resource usage of the recorder, ABB Robotics lets their recorder remain active in their control system for industrial robots which facilitates troubleshooting of post-release problems significantly. In the ABB system, there is no probe effect from this recording, since the recorder is an integrated part of the system and always active.
CPU usage graph: The CPU Usage Graph presents the CPU usage over the whole recording, which gives a nice overview. The CPU usage is presented in an accumulated manner, so for each time interval both the CPU usage of the individual processes and the total CPU usage is illustrated. It is possible to view select only specific processes, to study their CPU usage in isolation. Moreover, it is possible to study the details of interesting time intervals.
24
Figure 8: CPU Usage graph
4.2 Wind River WorkbenchWind River “Workbench” formerly known as “Windview” is a Tornado tool that provides the function of a run-time analysis tool. Wind River Workbench monitors the target system and logs critical information like timing, events (system events and user defined events) and task state transitions. User defined events, set with the “wvEvent()" function can be referenced in the corresponding Wind River Workbench output in order to accurately determine timing. The output data can be analyzed in Wind River Workbench’s graphical interface.
Figure 9: Wind River Workbench’s graphical interface and log files view[1]
Some of the features of Wind River Workbench are indicated in the figure above. It is a powerful tool that allows users to study the VxWorks environment in detail, such as task executions, semaphore handling and memory status.
There are also some disadvantages with Wind River Workbench. It caused unknown amount of overhead work every time a new task is created while collecting data on target
25
system. Another disadvantage is that the log files only can be analyzed in the Tornado environment. Wind River Workbench is a commercial tool with license fee of about $14000 for each user.
4.3 OSE IlluminatorOSE Illuminator is a comprehensive suite of software tools for application-level debugging and analysis of embedded applications. Application-level debugging is performed at a higher abstraction level than source code debugging, allowing the user to debug based on the events of the application, such as messages sent between processes and context switches.
It provides a user-friendly graphical interface by integrating plug-ins for debugging and system information. It is debugging distributed multi-CPU systems as well as single CPU systems. It provides a nice overview of how application is using system resources such as memory and CPU time.
Illuminator consists of five major plug-in tools:
The System Browser is used to see currently running OSE targets and their blocks, block information, processes and process information.
The Event and Actions Handler is used to trace and monitor system events, catch and halt specified processes.
Figure 10: The Event Viewer displays a trace of all sent signals in the current state [42]
The Application Loader enables system developers to supervise, load, and unload programs dynamically from an OSE system running on target.
The Post-Mortem Debugger allows the system designer to perform detailed debugging and analysis of a system after a failure.
26
The Profiler shows the CPU and memory usage of the system.
Figure 11: The CPU Profiler displays the CPU load of an application. [42]
4.4 LTT and relayfsThe Linux Trace Toolkit (LTT) [33,34] is the most-used tracing facility in Linux. Although LTT is not currently part of the kernel, it is included in many of the popular distributions, including UnitedLinux, MontaVista, Lineo, Debian, ELinos, and Denx, and efforts are underway to have it included it in the kernel. In the last year, several aspects of the technology described above have been integrated into LTT with positive results.
More recently, relayfs[35], a mechanism for transferring data from kernel to user space in Linux, has also incorporated aspects of K42’s tracing technology [36]. An order of magnitude performance improvement was achieved when this technology was applied to Linux. The three primary aspects providing this performance improvement were the lockless logging of events, preprocessor buffers, and more efficient timestamp acquisition. Cheap user-mapped buffers are currently under development.
The Linux Trace Toolkit (LTT) [34] consists of patches to the Linux kernel that instrument a fixed number of locations inside the kernel. These sensors can be switched
27
on to collect subsystem-related events. A kernel module stores data and provides it to user-space applications for further processing.
Figure 12: Graphical Viewing Tool [34]
28
4.5 ART Real-Time MonitorThe objective of ART Real-Time Monitor [43] is to visualize the system’s internal behavior for the designers of ARTS.
Overall StructureThe ART Real-time Monitor is divided into three functional units: the Event Tap, the Reporter, and the Visualizer. The following is a brief description of each
Figure 13: Real-Time Monitor’s Visualizer
The Event Tap is the part of the operating system code which records the information about interesting events, which, in this case, are than changes of process state. The Tap is embedded inside the kernel code which performs process switching.
The Reporter sends the event message from the target system to the Visualizer on a remote host. The time requirement of the communication portion of the Reporter is incorporated into the given task set so that the interference by the Reporter’s activity can be analyzed. By creating a separate task for the Reporter, the interference is predictable in our schedulability analysis.
The Visualizer utilizes event messages from the Reporter, visualizes them and then provides the user with an interface for remote debugging. An event messages is a collection of raw data, from which it is hard to find meaningful information. The Visualizer interprets these raw data so as to visualize an execution diagram. After receiving an event message from the Reporter, the Visualizer not only creates an execution history diagram, but also calculates the important statistics such as total CPU utilization for periodic and aperiodic tasks, the number of successful completion, missed deadlines by abortion or by cancellation, and the number of events per second. The Visualizer is also capable of replaying an execution diagram for later analysis. All received messages are fifed, and the Visualizer replays the same execution diagram by referring the filed information. Statistics are also provided while replay is going on.
29
Figure 14: Visualized System Behavior
4.6 TraceXTraceX is designed to work with Express Logic's ThreadX RTOS, which constructs a database of system and application “events” on the target system during run-time. These
30
events include: thread context switches, preemptions, suspensions, terminations, system interrupts, application-specific events, all ThreadX API calls.
Figure 15: TraceX Visulaizer
TraceX software helps embedded developers visualize and better understand the behavior of their real-time systems (RTOSs). TraceX collects a database of system and application “events” on the target system during run-time. These events include thread context switches, preemptions, suspensions, terminations, and system interrupts, all of which generally escape detection in a standard debugging environment. Events are logged in the database by ThreadX under program control, with time-stamping and active thread identification so they can be displayed later in the proper time sequence. Event logging may be stopped and restarted by the application program dynamically, for example, when an area of interest is encountered. This avoids cluttering the database and using up target memory when the system is performing correctly. TraceX is a commercial tool that is available from Express Logic, for use on Windows hosts, for all target architectures supported by ThreadX RTOS, for a license price of $1,000.
4.7 Dynamic Probes (DProbes)Dynamic Probes (DProbes) [37] allow users to dynamically instrument arbitrary locations within an application or the Linux kernel. Whenever a so-called probe point is hit, the
31
framework executes user-defined probe handlers. DProbes require probe handlers to be written in a special language. Dynamic probes have the advantage that they do not pose any overhead on the system, if monitoring is turned off. DProbes mainly provide means for dynamic instrumentation.
4.8 Kernel Probes (kProbes)Kernel Probes (kProbes) [38] introduce the DProbes concept to the Linux kernel. They are dynamically inserted into a running kernel by loading a kernel module. Any location inside the kernel can be instrumented by replacing instructions with a trap event opcode. On x86 architectures, INT3 is used for this purpose. When such an instruction is hit, the kProbes framework performs all necessary tasks for event generation. It executes handlers, single-steps the original instruction, and finally returns control to the code that raised the exception. In addition to arbitrary kProbes, function-entry probes (jProbes) and function-return probes (kRetProbes) are available. kProbes provide means for dynamic instrumentation in the Linux kernel.
4.9 DTraceSun’s Solaris operating system uses DTrace for tracing. DTrace also does not have any probe effect if tracing is turned off for dynamic probes. Static probes have no-ops in place if deactivated, which are patched with calls to DTrace trampoline code on activation. In addition to kProbes, DTrace is able to probe user-space applications. Probes are written in a specialized language called D. Although DTrace offers versatile aggregation support, it is not especially suited for real-time systems, for example, probe code is executed with interrupts turned off in the kernel.
4.10 ConclusionsIn this chapter, we have described a number of existing visualization tools. The Tracealyzer compares favorably with these tools in terms of features, performance and/or availability.
Previous work for tracing operating systems such as AIX[39], IRIX[40], Linux[34], LLT, ART Real-time monitor, Dynamic Probes and Kernel Probes have had limitations including using fixed-length events, only allowing tracing via system calls to transfer data between in-kernel sensors and user space monitors, requiring locking to log events, and using inefficient timestamp acquisition. Dynamic probing additionally uses exceptions to handle probes. These means add an additional overhead to the probe effect caused by the instrumentation. The Tracealyzer addresses these problems and a recorder for the Tracealyzer can be made quite uncomplicated, robust and very efficient.
TraceX has almost similar features as Tracealyzer. It provides a nice overview of thread context switches, preemptions, suspensions, terminations, system interrupts, application-specific events and all ThreadX API calls. TraceX is a commercial tool that is available from Express Logic with source code, for use on Windows hosts, for all target
32
architectures supported by ThreadX, for a license price of $1,000. The advantage of tracealyzer over TraceX is the better visualization and the statistics support
Wind River Workbench is a powerful tool for debugging. The disadvantages with Wind River Workbench are; unknown amount of CPU overhead for each context switch, it works only for systems using VxWorks, not OSE as in this case, and log files can only be analyzed in the Tornado environment which costs about $14.000 for each user and source code is not available. Tracealyzer compares favorably with Wind River Workbench in features that it imports logs in a very simple file which is documented and available, has better statistics support and is free of cost for research/educational purposes.
Mostly visualization tools are dependent on a specific operating system while others are dependent on a specific hardware platform. For example LTT, Kernel Probes and Dynamic Probes are implemented for Linux Kernel, DTrace for Sun Solaris and TraceX is designed to work with Express Logic's ThreadX RTOS, Wind River Workbench is designed for VxWorks, OSE Illuminator for OSE operating system. The Tracealyzer is however possible to use on different platforms, as the file format is open. Recorders have been successfully implemented for VxWorks (from Wind River) and RTXC (from Quadros) and can easily be reused on other system using these operating systems. It is believed that OSE and also Windows CE have the necessary functionality for Tracealyzer recorders.
Here we see several tools exists for monitoring the behavior of real-time systems like Windview, OSE Illuminator, ART Real-time Monitor, ThreadX but no tool seems more suitable for this purpose than the Tracealyzer.
33
Chapter 5
Recording of Scheduling and Communication Events in OSE
This chapter presents what information is required by the Tracealyzer to measure the CPU-usage and to display the system behavior, different methods that can be used to implement the software recorder for Tracealyzer and an evaluation of different methods for recording of scheduling events.
5.1 Technical Requirements of TracealyzerTo be able to visualize the behavior of the running system, the system must have the appropriate recording functionality. In operating systems there are a series of well known events that affect behavior, examples include context switch, I/O interrupt, IPC, etc. The ability to rapidly trace all these frequent events in the minimum time is key to understanding system behavior. As Tracealyzer recordings are obtained using a software recorder that is integrated in the base platform of the system. The software recorder must be implemented according to a file format provided with Tracealyzer.
Generally, the Tracealyzer recorder uses a feature in the Operating System to associate a specific routine with the task-switch (context-switch), which is executed each time the operating system performs a task-switch [32]. This task-switch routine contains a probe that registers each task-switch event in the system. Each execution of the task-switch probe stores a timestamp together with an ID of the next process execute and the scheduling status code of the previously executing process. The scheduling status code contains the reason behind the task-switch, e.g. preemption by a process with higher priority ( “READY”), blocking by a semaphore or a message receive (“SUSPEND”) or a waiting for a specified time (“DELAY”). From the collected task-switch information, it is possible to generate an execution trace, which accurately describes how the processes have executed over time.
Typically the mandatory features from OS that are required during implementation of Tracealyzer recorder are:
Timestamp Process-ID of the process that begin to execute The OS scheduling status of the old process The current scheduling priority of the process that is about to start
5.2 How to Get Tracealyzer Requirements in OSETo get information about timestamp three system calls below are built-in in OSE timing functionality. Time is usually represented by a number of system ticks that is incremented
34
after a certain time interval. Different system hardware configurations have different duration of tick.
get_ticks() system_tick() get_systime()
The get_ticks() returns system time in ticks since the system start. Since context switching will occur several times in one system tick, this system is will be insufficient. As the resolution is required at microseconds level so this system call is not sufficient. The system_tick() call can be used to returns the number of microseconds per system tick. It is used when converting a time represented by number of ticks to a number of microseconds. As microsecond level resolution is needed the most realistic choice for time measurement is to use get_systime() call. It returns the number of system ticks since system start and the number of microseconds since the last tick in the microseconds parameter. Unfortunately the microsecond counter provide rarely accurate timing reference because the resolution presented by the get_systime() is hardware dependent. More about this restriction can be found in OSE system programming interface reference manual.
To get required information about the processes following system call can be used:
get_pcb()
The get_pcb() call returns a buffer obtained from the caller’s pool containing the OS_pcb structure. i.e. struct OS_pcb *get_pcb (PROCESS pid); The OS_pcb structure contains information such as name, priority and status of a process.
35
5.3 Methods for Recording of Scheduling Events in OSEIn this section different methods that can be used to implement the Tracealyzer recorder will be discussed with advantages and disadvantages.
5.3.1 The Kernel Handler Implementation MethodIt is possible to write customized routines to change the behavior of, or add functionality to, some of the OSE system calls. It is sometimes also necessary to write code in order to configure the kernel. Additional operations to be performed at context switching can be added as an extension in OSE through so called kernel handlers [8]. There is a create handler that would be called at process creation, swap handlers called at context switching and a kill handler. When a process is swapped in or out extra operations to be performed can be invoked by the user.
Activating these swap handlers is one possible method of recording scheduling. Swap in handler is called each time the process is scheduled so we can associate a specific routine in swap in handler in OSE that Tracealyzer recorder uses to record events.
The disadvantage using this method for recording of scheduling is the increased context switch latency. Unless the extra operations are performed very fast an unacceptable system overhead could be the consequence. There is therefore a limitation concerning allowed system calls from inside a kernel handler. The OSE core user’s guide manual declares that no system call could be made from kernel handlers [8]. However that is not completely true. The get_pcb() call is illegal only in swap handlers not in create handler. Also we can use other system calls in create handlers for instance send(). Which system calls are available depends on specific system and which are unsuitable depends on the system requirements.
As we know for the recording of scheduling get_pcb(), get_systime(), system_tick() system calls are necessary. In order to get necessary information from OS for Tracealyzer recorder get_pcb() needs to be called from outside the swap handler.
The allowed system calls and speed requirement brings challenges to the implementation, particularly storage of measurement results. This problem partly can be solved by utilization of “user area” core component extending
Figure 16: Kernel handler implementation flow chart [41]
36
process specific memory. This memory is directly accessible from kernel handlers however not outside them. This memory can be used to store a pointer to the container position for fast access.
5.3.2 OSE Kernel ModificationThe kernel can trace all signal interactions and process switching including interrupt processes. The recording functionality can be included as kernel service [41]. This would probably be the more accurate and possibly a bit fast way to record the scheduling and communication events. The kernel implementation of context switch could be updated to include extra operations for recording of scheduling and communication events.
Such an implementation does not differ significantly from using kernel handlers. A kernel modification would only move the code otherwise placed in the kernel handlers to the kernel directly. There would not be any difference in accuracy between these two alternatives. Whether a kernel handler or kernel modification implementation becomes satisfying depends mostly on the code quality. It is however desired that only experienced programmers are allowed modifying the kernel.
The problem is that some interfaces are private and unavailable to the user. Modifying the OSE kernel also requires the development of a new OSE kernel distribution. Another disadvantage when including the functionality inside the kernel is less flexibility. If for some reason you would not like to fully utilize the scheduler, perhaps several scheduling alternatives are available, the choice of using context-switch recording might be included, by mistake.
5.4 Methods EvaluationTwo alternative methods for recording of scheduling have been discussed in section 5.2. These alternatives now will be considered for implementation. This section will explain the parameters needed to evaluate the solutions and determine one of the most suitable implementation.
5.4.1 AnalysisIn order to analyze the feasible solutions for recording of scheduling, parameters determining the most preferable solution for implementation must be declared. These parameters are [41]:
Accuracy Time resolution Context switching latency Flexibility Memory overhead Complexity
37
Accuracy is a parameter describing how exact the measurement results are, telling whether the measurements take place at the exact time of a context switch or not. If there is delay between actual swap and the time stamp received the accuracy is affected. This parameter also considers whether the result will be instantly accessible to the scheduler or if there is delay before the results are handled.Time Resolution represents the unit in which time is measured. For instance time measured in microseconds has better and more sufficient time resolution than if measured in system ticks only. Time resolution also affects the measurement accuracy.Context switching latency is the parameter telling how much extra time per swap that is required to perform the measurements. Dynamic scheduling optimizing resource allocation in order to improve the system performance, would loose its purpose if the execution time measurement causes a severe system slow down due to high context switch latency.Flexibility is a parameter for determining the hardware determining the hardware dependency. A flexible solution can be used on different hardware configuration and processors.Memory Overhead is caused by unnecessary or extensive memory utilization. Memory is a limited resource and should therefore be spent carefully.Complexity is the parameter for considering implementation time and difficulty as well as development cost.
5.4.2 Evaluation and ChoiceThe memory overhead can be expected to be very similar in the two cases as the same temporary values and non-temporary results need to be stored regardless which method is used. Considering complexity the kernel modification methods is probably more complex and harder to implement, since the kernel source-code has to be understood and changed. Moreover, the kernel source-code is not always available.
Accuracy is not a problem for either method, since the recording is performed during the context-switch, there is no delay. Both methods may obtain timestamps on micro-second resolution, using OSE´s get_systime system call. Even better resolution can be obtained using hardware timers, but this is not necessary and the disadvantage with hardware timers is that the solution will be hardware dependent. The timer implementation will then have to be made for a number of system hardware configurations in order to ensure compatibility with all hardware configurations that might be desired. That would be all target boards that OSE is compatible with.
That leaves two parameters particularly important to investigate. The characteristics of each method are graded with one, two or three stars. One star indicates that the method does not behave as desired; two stars indicate that the method meets the requirements and three stars indicate that method has very good characteristics
38
Method Latency Flexibility
Kernel Handlers * * * * *
Kernel Modification * * * *
Table 1: Methods Evaluation and Choice
A kernel modification is not flexible since it requires development of a new OSE distribution. This is why, if one of other two methods is appropriated for implementation, the kernel modification method will be ruled out.
The context switching latency of the kernel handler and kernel modification implementation both depend on how well the programmer can speed optimize the code. However, a kernel modification is probably faster, since the recorder can be called directly, without first calling a handler function. The CPU time overhead in a system implementing any of these methods for execution time measurement depends on the rate of the context-switches. Measurements of the CPU overhead are presented in Chapter 7.
Since the kernel modification method is not suitable, the implementation will be based on kernel handlers.
39
Chapter 6
Implementation
According to the conclusions made in section 5.4 a kernel handler implementations is the most appropriate alternative solution for realization. Still we need to address the particular swap handler design issues mentioned in section 5.3.1.
This section will describe in detail the implementation of a Tracealyzer recorder through kernel handlers. Design issues will be considered and choices will be explained and test application with the flowing results will be presented and analyzed in chapter 7.
6.1 Design IssuesImplementing the kernel handler method for Tracealyzer recorder requires considerations concerning the kernel handlers specific design issues. In section 5.1 we mentioned that how trace recorder records scheduling and which mandatory information/parameters are required by recorder from OS kernel. The problem being the first design question 1 is how to get information about processes when no system call is allowed from kernel handlers.
Design Question 1:How can we get information about processes e.g. names, status, priority required to implement Tracealyzer recorder?
The get_pcb() system call interrogates the status of the specified process or block. It returns a buffer obtained from the caller’s pool containing the OS_pcb structure. i.e. struct OS_pcb *get_pcb (PROCESS pid);
The OS_pcb structure contains information such as name, priority and status of a process. Unfortunately, the returned data is a copy of the current state, not a direct reference to the kernel’s data structure, so the result of a get_pcb call can not be “reused” later.
Section 5.1 presents the typical requirements of trace recorder implementation. As we know trace recorder uses these feature/requirements in OS and associate them with a routine that is executed each time the OS performs a task-switch. The trace recorder requirements bring challenges to the implementation when no system call is allowed from kernel handlers. It could be more easy and efficient to implement the trace recorder if we can get these requirements directly from swap handlers and use them further. To overcome this design issue section 6.4 presents how and where these requirements for trace recorder were achieved and used.
40
Design Question 2:How can we reduce memory consumption?
The native process-id in OSE is 4 bytes, but memory consumption can be reduced if a shorter process-id, using only 2 bytes (pid16) can be associated with a process by storing this in user area.
6.2 Tools and Project Structure Setting(removed from public version of report)
6.3 Memory ConfigurationThe user area process specific memory is extended by 50 bytes out of which 2 bytes are specific for each process as defined in the kernel configuration file (ospp.con). This size must correspond to the size of implemented variables to be stored in user area. For this application there would be only shorter process-id (pid16) that is associated with the PID to reduce memory consumption.
During implementation of recorder the user area process specific memory was first extended by 2 bytes for pid16. As a result we got some strange results in the form of bad process id (pid16). It was believed that some other process also was using the memory area reserved for pid16. We therefore introduced a dummy variable with 48 bytes first in the user area, in order to move our pid16 variable to a probably unused location.
Figure 19: User area configuration
As seen in code of figure 18 the variable sized to exactly 50 bytes. The OSE specific type definitions can be found in the system programming interface manual. The user area contains 16 bit unique identifier (pid16) that is associated to each process to reduce the memory consumption instead of 32 bit native process identifier from OSE. As OSE PID is not used by the Tracealyzer, the Tracealyzer will use pid16 to identify OSE processes.
Only necessary information required is stored globally. The user area will be initialized in the create handler i.e. pid16 will be assigned a short process id by calling a routine in create handler and then this short process id will be used in swap in handler for process identification and information.
typedef struct user_area{
char dummy[48]; /* char = 48 bytes for some dummy data */
unsigned short pid16; /* unsigned short = 2 bytes */
} user_area_t;
41
6.4 Kernel HandlersThe discussion in section 6.1 considering design issues has determined the contents of each kernel handler respectively. Let the detailed explanation of the kernel handler implementation begin with the create handler.
The recorder was implemented according to the file format provided with Tracealyzer by keeping in view the requirements of Tracealyzer recorder. As we know create handler is activated once for each process during the process creation. Because of time aspect it is preferable to keep the contents of the swap handlers as short as possible. Therefore we would like to perform as many of the operations as possible inside the create handler instead of inside the swap handlers.
Still we can not get any information about processes from create handler because create handler is called before the process is fully initialized, so get_pcb does not give all information desired. So get_pcb() unfortunately can not be performed inside the create handler. To get required information about processes, special process, called create_handler_process() is created as early as possible in the system startup that is configured to only receive signals. In the create handler a unique 16 bits identifier (pid16) is associated as an index to all native OSE PID (i.e. is 32 bits) to reduce memory consumption and to make it easier to look through all OSE processes in the future using this as an index.
By an initial control in the create handler, checking if create_handler_process() is created, we can send OSE PID as signal from create handler to our process to get required information about OSE process and update this in the processes list. Since whenever create handler will be called for new process, a unique identifier will be assigned to this process and sent along with OSE PID to create_handler_process() and the required information about this OSE process will be stored/updated in the processes list if the process is still alive. The initial control in the create handler was necessary because it is not possible to create any process before the create handler is called. So we might loose some OSE internal kernel processes that were ignored by this control. All the ignored processes were updated later on the first activation of create_handler_process().
As no system call is allowed in the swap in handler we can not get priority of the current process and status of old process as required by the Tracealyzer to display the context switch. Due to this issue the Tracealyzer was changed to rely on priority instead of status. From this information, it is possible to calculate when jobs are finished, unless the priority is not changed.
The Swap-in handler calls the recorder for each process that is about to execute. Each execution of the swap in handler stores a timestamp together with an ID of the next to execute. The priority of the process was recorded in create_handler_process() along with the processes names and stored in the processes list. Tracealyzer was changed to rely on priority instead of status to display the context switch because it is not common to see priority changes.
42
Timestamp is calculated as a difference in time between the previous event and the current event. The time stamp is to fit in 14 bits (i.e. above 16384) otherwise an extra time stamp event is added to the recording, where 32 bits are used to store the timestamp. From the collected task-switch information, the Tracealyzer can present an execution trace, which accurately describes how the processes have executed over time. The data from the task-switch probe was stored in a ring-buffer, i.e. the oldest data is overwritten when the buffer is full. Since the amount of memory available for recording is limited, the alternative to a ring-buffer is to stop recording when the buffer is full, which is not desirable. A ring-buffer always contains the most recent data. This is necessary as it might be required to have the recorder active for long periods before interesting behavior occurs. When the behavior of interest has been observed, the recorder can be stopped and the content of the buffer is written to a file. The file can then be transferred to a PC for analysis.
In the implementation of the software behavior recorder, the size of the ring-buffer is by default set to 100.000 events. As each event requires 4 bytes of memory for storage, this corresponds to a memory allocation of 400 KB. However, it is not a problem to use buffer sizes larger than 100.000 events as long as there is memory available.
The file that is the result of the recording uses a publicly available file format, called TRC, (named using the consonants from the word “trace”). The TRC format was developed together with the recorder and is very simple. A TRC file contains an execution trace, i.e. a list of time-stamped events generated by the code instrumentation. The time-stamps are read from a clock with microsecond resolution.
43
Figure 20: Create handler flow chart
Figure 21: Kill handler flow chart
Leave Create Handler
Send pid16 to create_handler_process
If create_handler_proce
ss is started
Enter Create Handler
Allocate slot in processes_list for created process (ID: pid16)
YesNo
Leave Kill Handler
Enter Kill Handler
Set process ID as a killed process i.e. alive = FALSE
44
Figure 22: create_handler_process flow chart
create_handler_process
receive signal sig
If first_time == 1
update_process (sig.pid) i = 0
If i < process_count
update_process ( i )
i = i + 1
Yes
No
Yes
No
45
Figure 24: Swap in handler flow chart
YesNo
Store extra time stamp event in Ring Buffer i.e XTS event
Leave Swap in Handler
Get timestamp i.e, the time difference since last swap-in
If timestamp < 16384
Encode 32-bit
Store task switch event (timestamp and pid16)
event
Enter Swap in Handler
46
Chapter 7
Investigation of CPU Overhead of Recorder
In this chapter an investigation of the CPU cost of the recorder will be presented based on the real test cases from Company A.
7.1 Execution time of ProbeThe effect on the system’s temporal behavior caused by specific software probe is dependent on two factors, the rate of which the probes are executed in the system and the time it takes to execute a probe. The execution rate of a probe depends on the context in which it is used. The measurement of the probe execution time can be accomplished by configurating the system to execute only a single task containing two immediately adjacent probes, in a loop. The difference in time-stamps between two adjacent probes provides an estimate on the probe execution time. The execution time of probe in this implementation is found to be 0.6 µs.
7.2 Measure number of context switchesThe number of context switches per second can be measured by dividing the number of events reported by the Tracealyzer with trace length (also reported by the Tracealyzer)
7.3 CPU usage of create_handler_processA special process create_handler_process has been created to store information about different OSE processes that should also be investigated in reference to CPU usage. The Tracealyzer has a feature to generate a report with a summary of the CPU usage and timing properties, such as maximum or average execution time, of all or selected processes. This feature can be used to measure the CPU usage of create_handler_process. As the create_handler_process is created with quite low priority 20 so it will not affect the higher priority processes. The CPU usage of this process depends how often processes are created. Most processes are created in the system startup, about 1000. The execution time of this process handling a single signal was on average 14 µs, so this process will use about 14 µs of CPU time during the startup (on the low priority 20).
7.4 CPU overhead of RecorderThe CPU overhead of recorder can be measured as:
CPU overhead of recorder (U)
= Measure execution time of probe (C)
* Measure no. of context switches (F)
Calculations:
47
One test case from Company A will be presented where the CPU overhead of recorder is calculated. In this test case some fake traffic was started by traffic generator “Loco”. As a result the total CPU usage was found to be 30% and a 91045 events was recorded in 5.1 seconds. The CPU overhead of recorder in this case can be calculated as follows:
Rate of context-switch event (F) = 91045 / 5.1 = 17851 events/sec
Execution time of context-switch probe (C) = 0.6 µs
CPU overhead = F * C = 17851 * (0.6 / 1000000) = 1.07 %
48
Chapter 8
Conclusions and Future Work
This chapter concludes the results obtained from the implementation. The first, and most important, conclusion of this project is that we have showed that it is possible to use record scheduling events on OSE using Tracealyzer. But there are still a number of issues that can be improved; these will be further discussed in this chapter in the future work.
8.1 SolutionThe implementation of software behavior recorder provides a simple solution for monitoring the internal behavior of the system. All the events are stored in structured binary file that can be processed offline using Tracealyzer. This makes it easy for the software engineers at Company A to improve their systems according to the demands of Company B. The CPU usage feature provides a good statistics over the CPU usage for each process in the system.
In the beginning of the project, we set up a number of questions that needed to be answered:
1. What information is required by the Tracealyzer recorder to measure CPU-usage and display the system behavior?
2. Is it possible to record the information needed by the Tracealyzer from OSE to measure CPU-usage, preferably without changing the OSE kernel code?
3. What is the impact of this type of recording, in a typical case and in a high load situation?
a. What is the execution time of a probe?b. What are the rates of different types of events?
4. Is the impact of the recording acceptable if not, how can it be reduced?a. Is it possible to select what events to record, to reduce the number of
probes that are executed?b. Is it possible to reduce the execution time of a probe?
The answer to the first question is presented in section 5.1 with some typical features required by Tracealyzer that are necessary to show the complex system behavior.
The answer to the second question is yes. Two methods have been discussed and evaluated to record information from OSE using Tracealyzer with advantages and disadvantages in section 5.3. After making evaluation it is decided that kernel handlers method is the most suitable to implement Tracealyzer recorder to show the system behavior and measure CPU usage.
49
Answer to questions 3 and 4 are presented in chapter 7 where CPU overhead of recorder is evaluated on the base of test cases from Company A. It is seen that overhead of the trace recorder is about 1% which according to Company A is acceptable.
8.2 Limitations A number of problems occurred that we had to solve such as how to get processes names, priorities, status required by Tracealyzer when no system call is allowed from kernel swap handlers.
Since we couldn’t get the OS scheduling status of the old process and the current scheduling priority of the process in the swap in handler as required by Tracealyzer to show the context switch, the priorities were stored along with the processes names during the processes creation. So we had to rely on these priorities because it is not common that they change, but it is possible.
The Tracealyzer was changed to rely on priorities instead of OS scheduling status, which works assuming that the priorities are not changed. If priority change occurs, there is a risk that the side-ways indentation in the Tracealyzer process trace, which shows if the context-switch is a due to preemption or a blocking, might be wrong.
8.3 Future WorkThere are some issues that could be considered in the future work.
This solution doesn’t remove the killed processes from the active processes list. It should be considered in the future work.
The Tracealyzer can display two types of probe readings, custom probes – without interpretation, and interprocess communication (IPC) events, which are displayed graphically in the trace. This could be used to log signal events, i.e. interprocess communication. The Tracealyzer supports visualization of such events.
50
References
[1] WindView User’s Guide 2.0.1 Edition 1. Part #: DOC-12393-ZD-02, 6 May 99
[6] Real-Time Concepts for Embedded Systems by Qing Li and Carolyn Yao ISBN:1578201241.
[7] D. Bucar, “Reducing Interrupt Latency using the Cache” master thesis in Electrical Engineering Stockholm January 2001.
[8] Manual from ENEA Embedded Technology AB, 2006, OSE Kernel User’s Guide
[9] ENEA OSE Real-Time Kernel Datasheet (advertisement), OSE Real-Time Kernel. Available at www.enea.com (14th April 2008)
[10] B. Plattner. Real-Time Execution Monitoring. IEEE Transactions on Software Engineering, SE-10(6):756{754, Nov. 1984.
[11] J. Joyce, G. Lomow, K. Slind, and B. Unger. Monitoring Distributed Systems. ACM Trans. Computer Systems, 5(2):121{150, May 1987.
[12] U. Schmid. Monitoring Distributed Real-Time Systems. Real-Time Systems Journal, 7(1):33{56, 1994.
[13] J. J. Tsai, Y. Bi, S. J. Yang, and R. A. Smith. Distributed Real-Time Systems - Monitoring, Visualization, Debugging, and Analysis. John Wiley and Sons, Inc., USA, 1996. ISBN 0-471-16007-5.
[14] D. Urting and Y. Berbers. Runtime Verification of Timing Constraints. Research Report CW 345, Department of Computer Science, K.U. Leuven, Belgium, July 2002.
[15] B. A. Schroeder. On-Line Monitoring: A Tutorial. IEEE Computer, 28(6):72{78, June 1995.
[16] M. J. Kaelbling and D. M. Ogle. Minimizing Monitoring Costs: Choosing Between Tracing and Sampling. In Proceedings of the Twenty- Third Annual Hawaii International Conference on System Sciences, volume 1, pages 314{320,1990.
[17] M. S. Dias and D. J. Richardson. The Role of Event Description in Architecting Dependable Systems. In Workshop on Architecting Dependable Systems, ICSE 2002, May 2002. Orlando, Florida.
51
[18] D. Ogle, K. Schwan, and R. Snodgrass. Application-Dependent Dynamic Monitoring of Distributed and Parallel Systems. IEEE Transactions on Parallel and Distributed Systems, 4(7):762{778, 1993.
[19] D. Mahrenholz. Minimal Invasive Monitoring. In Proceedings of Fourth IEEE International Symposium on Object-Oriented Real-Time Distributed Computing, pages 251{258, 2001.
[20] J. Joyce, G. Lomow, K. Slind, and B. Unger. Monitoring Distributed Systems. ACM Trans. Computer Systems, 5(2):121{150, May 1987.
[21] S. C. Raju, R. Rajkumar, and F. Jahanian. Monitoring Timing Constrains in Distributed Real-Time Systems. In Proceedings of the 13th IEEE Real-Time Systems Symposium, pages 57{67, 1992.
[22] J. J. Tsai, Y. Bi, S. J. Yang, and R. A. Smith. Distributed Real-Time Systems – Monitoring, Visualization, Debugging, and Analysis. John Wiley and Sons, Inc., USA, 1996. ISBN 0-471-16007-5.
[23] J. Tsai, K.-Y. Fang, H.-Y. Chen, and Y.-D.Bi. A Noninterference Monitoring and Replay Mechanism for Real-Time Software Testing and Debugging. IEEE Transaction on Software Engineering, 16:897{916, 1990.
[24] I. Smaili. A Real-Time Monitoring System for the Time-Triggered Architecture. In 8th IEEE Real-Time and Embedded Technology and Applications Symposium (RTAS 2002), Poster Addendum to the Proceedings, pages 9{16, Sept. 2002. San Jose, CA, USA.
[25] H. Thane. Monitoring, Testing and Debugging of Distributed Real-Time Systems. PhD thesis, Royal Institute of Technology, Stockholm, May 2000.
[26] J. J. Tsai, K.-Y. Fang, and H.-Y. Chen. A Noninvasive Architecture to Monitor Real-Time Distributed Systems. Computer, 23(3):11{23, Mar. 1990.
[27] A. Kirschbaum, J. Becker, and M. Glesner. Run-Time Monitoring of Communication Activities in a Rapid Prototyping Environment. In Proceedings of the 9th International Workshop on Rapid System Prototyping, pages 52{57, 1998.
[28] P. S. Dodd and C. V. Ravishankar. Monitoring and Debugging Distributed Real-Time Programms. Software-Practice and Experience, 22(10):863{877, Oct. 1992.
[29] G. R. Watson. The Design and Implementation of a Parallel Relative Debugger. PhD thesis, School of Computer Science and Software Engineering, Monash University, Wellington Road, Clayton, VIC 3168, Australia, Oct. 2000.
52
[30] D. A. Stewart and W. M. Gentlman. Non-Stop Monitoring and Debugging on Shared-Memory Multiprocessors. In Proceedings of the 2nd International Workshop on Software Engineering for Parallel and Distributed Systems, pages263{269, May 1997. Boston, MA.
[31] D. Abramson and G. Watson. Relative Debugging for Parallel Systems. In Proceedings of PCW 97, Sept. 1997. Canberra, Australia.
[32] Johan Andersson. Modeling the Temporal Behavior of Complex Embedded Systems. A Reverse Engineering Approach. Mälardalen University, 2005
[33] Karim Yaghmour. Ltt web page.http://www.opersys.com/LTT/index.html.
[34] Karim Yaghmour. Measuring and characterizing system behavior using kernel-level event logging. In Proceedings of the 2000 USENIX Annual Technical Conference, June 2000.
[35] Tom Zanussi, Karim Yaghmour, Robert W. Wisniewski, Michel Dagenais, and Richard Moore. An efficient unified approach for trasmitting data from kernel to user space. In OLS 2003 – Ottawa Linux Symposium, page to appear, July 23-26 2003.
[36] http://domino.research.ibm.com/comm/research_projects.nsf/pages/k42.index.html (17th April 2008)
[37] Richard J. Moore. A Universal Dynamic Trace for Linux and Other Operating Systems. In Proceedings of the FREENIX Track: 2001 USENIX Annual Technical Conference, Berkeley, CA, USA, 2001.
[38] R. Krishnakumar. Kernel Korner: Kprobes - a Kernel Debugger. Linux J., 2005.
[39] IBM Corporation. Aix version 3.1 for risc system/6000 performance monitoring and tuning guide. Technical Report SC23-2365- 00, IBM Corporation.
[40] Robert W. Wisniewski and Luis F. Stevens. A model and tools for supporting parallel real-time applications in unix environments. In Proceedings of The 12th IEEE Real-Time Technology and Applications Symposium, pages 126–133, Chicago Illinois, May 15-17 1995.
[41] Liz Malin Ling. Master’s Thesis. Execution Time Measurements of Processes on OSE Real-time Operating System. Linkoping University. September 2007.
[42] Enea Embedded Technology AB, 2006. OSE Illuminator User’s Guide.
53
[43] Hideyuki Tokuda, Makoto Kotera and Clifford W. Mercer. A Real-Time Monitor for a Distributed Real-Time Operating System. Computer Science Department Carnegie Mellon University Pittsburgh, Pennsylvania 15213