fall 2013 silicon valley university confidential 1 introduction to embedded systems dr. jerry shiao,...

Fall 2013SILICON VALLEY UNIVERSITY

CONFIDENTIAL 1

Introduction to Embedded Systems

Dr. Jerry Shiao, Silicon Valley University

SILICON VALLEY UNIVERSITY CONFIDENTIAL 2Spring 2014

Section 14 Real-Time Patch Linux 2.6 Real-Time Modifications

Linux Community wants to convert the Linux Kernel into true Real-Time Operation System (RTOS) without a microkernel.

Linux RTOS Needs: Interrupt Service Routines must NOT unconditionally

preempt any process running on the CPU. Critical Sections as needed by processes. Unbounded Priority Inversion not allowed.

Ingo Molnar, approached the Linux RTOS problem by changing the Linux system with Real-Time features that would improve user’s experience.

High-Resolution timers. Kernel lock validation. Generic interrupts for all architectures. Robust futexes. Priority Inheritance.



Interrupt as Threads Interrupt Service Routine (ISR) has highest priority

events in the Operating System and only another ISR can interrupt an ISR, but only if interrupts are enable.

ISR pushes work to Kernel Thread, Tasklet, or Softirq. Kernel Thread: Work Queue, queues up work from ISR that

will be executed by worker kernel thread, keventd. Works in a FIFO order.

Softirq: Service routine performed after ISR returns and before interrupted process is resumed. Kernel Thread, ksoftirqd, executes queued softirqs.

Same softirq run simultaneously on different CPUs. Reentrant: Need to use locks to protect against concurrent accesses.

Tasklet: Similar to softirq, performed after ISR returns and before resuming interrupted process.

Same tasklet cannot run simultaneously on different CPUs. Not reentrant. Implemented as a softirq



Interrupt Signal Will Always Preempts CPU and executes the ISR.

Interrupt Inversion: ISR takes time away from higher priority process to handle

lower priority process. Lower Interrupt Latency and Interrupt Inversion:

Transform all ISRs, softirqs, and tasklets into Kernel Threads.

Converts Interrupt Handlers into Kernel Threads. Masking of the Interrupt Line. Wake Interrupt Service Thread (one for each type of

interrupt). Interrupt Service Thread registered by the Device Driver.

Original Process Higher Priority, than Interrupt Service Thread will not Preempt.



Interrupt Inversion:Interrupt Service Routine takes time away from a high priority process in order to perform work that is of lower priority.

The ISR is lower priority than the process that is currently running.

Interrupt Latency:Time before the high priority process gets the CPU back.


Section 14 Real-Time Patch

Linux 2.6 Real-Time ModificationsReal Time Patch:ISR wakes up a Kernel Thread that will run the function registered by the Device Driver.

Interrupt Latency:Shortens the time of Interrupt Inversion to a bare minimum.



Linux 2.6 Real-Time ModificationsReal Time Patch:The Threads that are created to service the interrupts are named “IRQ_n”, where “n” is the number of the Interrupt Vector.

Interrupt Thread for Interrupt Vector 14, the IDE disk driver.



Linux 2.6 Real-Time Modifications

ps command shows the pid = 1722 has priority set to 80.

chrt command sets the Real-Time scheduling attributes of the process ( pid ).-p: Existing pid = 1722, set the priority = 80.-f: Set scheduling policy to SCHED_FIFO.



Interrupts and CPU Affinities. Each thread kept on the CPU where the thread started

execution in order to prevent expensive cache flushes. IRQ Thread determined by CPU Affinity of Interrupt.# cat /proc/interrupts | grep ide0

14: 13602 1720 IO-APIC-edge ide0

# ps ax | grep IRQ-14

790 ? S< 0:00 [IRQ-14]

# cat /proc/irq/14/smp_affinity

1

# echo 2 > /proc/irq/14/smp_affinity

# cat /proc/irq/14/smp_affinity

2

Get the Interrupt Vector for the IDE 0 Controller ( 14 ).

Get the Process ID of ISR Thread 14.

Get the CPU Affinity of IRQ Interrupt Vector 14 ( CPU 0 ).

Set the CPU Affinity of IRQ Interrupt Vector 14 ( CPU 1 ).



IRQ Thread determined by CPU Affinity of Interrupt.

# taskset -p 790

pid 790's current affinity mask: 2

# echo 1 > /proc/irq/14/smp_affinity

# taskset -p 790


# ls -lR / > /dev/null

# taskset -p 790


Get IRQ Thread’s CPU Affinity.

The IRQ Thread’s CPU Affinity does not change until the interrupt is triggered.

Trigger some IDE interrupts. After the interrupts, the IRQ Thread’s Affinity follows the IRQ Affinity.

Set the CPU Affinity of IRQ Interrupt Vector 14 ( CPU 0 ).


Section 14 Real-Time Patch Linux 2.6 Real-Time Modifications Softirqs As Threads. Softirq runs with interrupts enabled. Possible Preempt higher priority Kernel Threads. Make ALL softirqs run under ksoftirqd Thread.

Problem: ALL softirqs grouped into one priority (the ksoftirqd Thread).

Solution: Softirq Thread is created for each softirq routine. Softirq Thread bound to a CPU.Softirq Thread has name: softirq-name/n

“n” is the CPU where Softirq Thread is set to run.


Section 14 Real-Time Patch Linux 2.6 Real-Time Modifications Softirqs As Threads.

$ ps -e -o pid,rtprio,comm | grep sirq

5 50 sirq-high/0

6 50 sirq-timer/0

7 50 sirq-net-tx/

8 50 sirq-net-rx/

9 50 sirq-block/0

10 50 sirq-tasklet

11 50 sirq-sched/0

12 50 sirq-hrtimer

13 50 sirq-rcu/0

17 50 sirq-high/1

18 50 sirq-timer/1

19 50 sirq-net-tx/

20 50 sirq-net-rx/

21 50 sirq-block/1

22 50 sirq-tasklet

23 50 sirq-sched/1

24 50 sirq-hrtimer

25 50 sirq-rcu/1

Softirq Thread has name: softirq-name/n

“n” is the CPU where Softirq Thread is set to run.

Priority of the softirqs are set at 50 by default.


Section 14 Real-Time Patch Linux 2.6 Real-Time Modifications Softirq Timer Thread (softirq-hrtimer) Control Timing Events:

Crucial to be executed on time. timer_create POSIX System Call:

Registers function to be called by hrtimer subsystem. High-priority process MUST be lower priority

than softirq-hrtimer thread. If not, high-priority process will NOT release CPU and

softirq-hrtimer thread will starve. Dynamic softirq-hrtimer thread: Automatically

change its priority to the high-priority process that requested the timer.



Linux 2.6 Real-Time Modifications Priority Inheritance:

Low-priority task has priority raised until completes critical section (releases shared resource).

Nested resource locks can lead to deadlock. Avoid deadlock by allowing each task to own one shared

resource. Do not allow nested locks.

Overhead raising priority and then lowering priority.

Task M

Task L

Normal Execution

Critical Section Execution

- Task M request resource.- Task L priority raised.

Executes critical section.

- Task L Completes critical section. Priority lowered.

- Task M receives resource. Executes critical section.


Section 14 Real-Time Patch Linux 2.6 Real-Time Modifications Linux Configuration

make menuconfig Preemption Mode No Forced Preemption (Server) Linux Kernel runs in 2.4 Kernel Mode.

No Preemption of the Kernel when CPU is in Kernel Mode. High-Priority Process MUST wait until current process context

switch back to User Mode or blocks on I/O. Low Overhead: Low Context Switching

Less scheduling, more time for process to run on CPU. Cache is valid. Translation Lookaside Buffer (TLB) is valid. Large batch processes, need throughput, and run serially.

Disadvantages: Low reaction time. Negative affect on application requiring low latencies and user

interaction.


Section 14 Real-Time Patch Linux 2.6 Real-Time Modifications Linux Configuration

Kernel code has points of voluntary preemption, where preemption is safe.

make menuconfig Preemption Mode Voluntary Kernel Preemption

Kernel cannot call schedule( ) when interrupts are disabled. “might_sleep” function checks for interrupts are disabled or

spin lock is held. Place “might_sleep” function in Kernel process handling

system call. The Kernel process will call schedule( ) when condition is

valid and allow higher priority process to get the CPU.



Linux originally designed around fairness, sharing resources among its uses and to perform under sustained load conditions.

General purpose complexity too unpredictable for real-time. Preemption

Linux 2.6 system call was preemptable. A process forced to release CPU.

Real Time Scheduler Linux 2.6 scheduler determined time slice for each process

and which process to run based on runqueue per priority level. Active and Expired priority arrays.

High Resolution POSIX Timers. Timer hardware removed timer update dependency on

Jiffies. Minimize execution of interrupt critical sections

(interrupt latency).

SILICON VALLEY UNIVERSITY CONFIDENTIALSpring 2014


Real-Time Patch To Native Linux KernelLinux Kernel in Real-Time Environment:

Preemption Improvements. Reduce time in non-preemptible code (interrupt context) to

minimize interrupt latency and scheduling latency. Low latency patches in Kernel, add points where Kernel thread

relinquishes non-preemption.

Unbounded Priority Inversion avoided.

1 3

2 6

5

4

Normal Execution


Unbounded Priority Inversion

Task H

Task M

Task L

Priority



Real-Time Patch To Native Linux Kernel Priority Ceiling:

Shared resource has predefined priority ceiling. Task acquires shared resource, task priority temporarily

raised to priority ceiling. Prevents deadlocks by stopping nested locks. Static analysis to determine priority ceiling of each

shared resource. Every task must be known.

Task H

Task M

Task L

Normal Execution


- Task L priority raised to priority ceiling.- Executes critical section.

- Task H request resource. Priority raised to priority ceiling.

- Executes critical section.




Handle IRQs as Threads. Lowers latency caused by softirqs and tasklets.

Softirqs preempts all threads, including kernel. Default priority IRQ threads (PID = IRQ_n, n is interrupt

vector) is 50. chrt utility used to modify the priority of the ISR thread.

Interrupts can be prioritized, even though hardware does not support prioritized interrupts.

User process can be prioritized higher than IRQ thread.

CPU affinity of IRQ thread based on CPU affinity of the interrupt. Prevents expensive cache flushes.




Real-Time Patch has infrastructure for most hard real-time.

Size of the code is too large and general purpose design makes it difficult to meet all RT Patch needs.

Only necessary drivers are loaded (bulk of Linux are drivers).

Drivers must be audited to not disable interrupts for long periods.

Kernel configuration options must be optimized. Only enable options that are needed.

http://people.redhat.com/mingo/realtime_preempt

fall 2013 silicon valley university confidential 1 introduction to embedded systems dr. jerry shiao,...

Documents