© 2006 rightnow technologies, inc. tribute to synchronization jeff elser december 11 th, 2006
Post on 20-Dec-2015
218 views
TRANSCRIPT
© 2006 RightNow Technologies, Inc.
Tribute to Synchronization
Jeff ElserDecember 11th, 2006
Page 2
Outline
•Why synchronize?– Synergy.
•Historical solutions•Synchronization Primitives•Brief word on concurrency•Semaphores in depth
– semid_ds– sem– Working with semaphores
•My project•References
Page 3
Prehistoric Solutions
Page 4
Dekker's Algorithm
boolean lockA = lockB = false;int turn = A; // or B
//Thread A:
lockA = true;
while(lockB)
{
if (turn != A)
{
lockA = false;
while(turn != A){}
lockA = true;
}
} lockA = false;
turn = B;
//Thread B:
lockB = true;
while(lockA)
{
if (turn != B)
{
lockB = false;
while(turn != B){}
lockB = true;
}
} lockB = false;
turn = A;
Page 5
Peterson's Algorithm
boolean lockA = lockB = false;int turn = A; // or B
//Thread A:
lockA = true;turn = B;while( lockB && turn != A ){}
lockA = false;
//Thread B:
lockB = true;turn = A;while( lockA && turn != B ){}
lockB = false;
Page 6
Synchronization Primitives
•Completion variables•Per-CPU variables•Atomic operation•Memory barrier•Spin lock•Seqlocks•Read-copy-update•Semaphore•Local interrupt disabling•Local softirq disabling
Page 7
Completion Variables
• Basically a simplified semaphore• wait_for_completion• complete• complete_all• Linux Kernel Development (2nd Edition) (Novell Press)
(Paperback) by Robert Love • http://www.amazon.com/Linux-Kernel-Development-Robe
rt-Love/dp/0672327201• Couldn’t find where the Linux OS actually used these –
they may just be for Linux developers
Page 8
Synchronization Primitives – cont.
• Per-CPU Variables– Array of data structures that can be accessed by only 1 CPU– Only useful when data can be split across CPUs– Fails if kernel preemption is enabled (race conditions)– How does this work with the O(1) scheduler?
• Atomic operation– Read-modify-write
o atomic_read(v)o atomic_set(v, i) – (test & set)o atomic_add(i, v)
– Another set operates on bit masks– Not all are hardware supported. Some are just locked
sections of code.
Page 9
Synchronization Primitives – cont.
• Memory barrier– Avoids code optimization problems by creating a barrier that
ensures that all code before the barrier gets executed before code after the barrier
• Spin locks– Busy wait while blocked– Useful for multi-processor systems when the critical code
will execute very fasto 1 = unlocked, 0 = locked
– Read/Write Spin Locks
Page 10
Synchronization Primitives – cont.
• Seqlocks– Read/Write spin locks that give high priority to writers
• Read-Copy Update– Allows many readers and writers– No locks or counters– Writer just copies the old data, updates it and changes the
pointer– Old data cannot be freed until all readers are finished
Page 11
Synchronization Primitives – cont.
• Semaphores– Kernel and System V IPC (User)– count
o If count > 0 the resource is freeo count = 0 means the resource is 100% utilized but the wait
queue is emptyo If count < 0 the resource is 100% utilized and there are
processes in the wait queue
– wait – pointer to the wait queue– sleepers – flag that is set if there are processes that are
waiting blocked– More to come!
Page 12
Synchronization Primitives – cont.
• Local Interrupt Disabling– Creates a critical section in the kernel so that even
hardware interrupts won’t break the execution of a set of statements
– CPU specific
• Local softirq disabling– Disable deferrable functions w/o disabling interrupts– Increment the softirq counter to disable; decrement the
softirq counter to enable
Page 13
++ Concurrency == ++ Synergy
• Maximize the number of I/O devices operating– Disable interrupts for very short periods
• Maximize the number of productive CPUs– Avoid using spin locks if possible
Page 14
Sema-what?
Page 15
More on Semaphores (semid_ds)
/* One semid data structure for each set of semaphores in the system. */ struct semid_ds
{ struct ipc_perm sem_perm; /*
permissions .. see ipc.h */ time_t sem_otime; /* last semop
time */ time_t sem_ctime; /* last change
time */ struct sem *sem_base; /* ptr to
first semaphore in array */ struct wait_queue *eventn; struct wait_queue *eventz; struct sem_undo *undo; /* undo
requests on this array */ ushort sem_nsems; /* no. of
semaphores in array */ };
•sem_perm – This is an instance of the ipc_perm
structure, which is defined for us in linux/ipc.h. This holds the permission information for the semaphore set, including the access permissions, and information about the creator of the set (uid, etc).
•sem_otime – Time of the last semop() operation
•sem_ctime – Time of the last change to this
structure (mode change, etc)
•sem_base – Pointer to the first semaphore in the
array (see next structure)
•sem_undo – Number of undo requests in this array
•sem_nsems – Number of semaphores in the
semaphore set (the array)
Page 16
More on Semaphores (sem)
•/* One semaphore structure for each semaphore in the system. */
struct sem {
short sempid; /* pid of last operation */
ushort semval; /* current value */
ushort semncnt; /* num procs awaiting increase in semval */
ushort semzcnt; /* num procs awaiting semval = 0 */
};
•sem_pid – The PID (process ID) that performed
the last operation
•sem_semval – The current value of the semaphore
•sem_semncnt – Number of processes waiting for
resources to become available
•sem_semzcnt – Number of processes waiting for
100% resource utilization
Page 17
Working with Semaphores
int semget ( key_t key, int nsems, int semflg );
int semop ( int semid, struct sembuf *sops, unsigned nsops);
struct sembuf
{
ushort sem_num; /* semaphore index in array */
short sem_op; /* semaphore operation */
short sem_flg; /* operation flags */ };
Page 18
Working with Semaphores – cont.
int semctl ( int semid, int semnum, int cmd, union semun arg );
union semun {
int val; /* value for SETVAL */
struct semid_ds *buf; /* buffer for IPC_STAT & IPC_SET */
ushort *array; /* array for GETALL & SETALL */
struct seminfo *__buf; /* buffer for IPC_INFO */ void *__pad;
};
Page 19
Working with Semaphores – cont.
• static inline void down(struct semaphore * sem)• {• might_sleep();• __asm__ __volatile__(• "# atomic down operation\n\t"• LOCK_PREFIX "decl %0\n\t" /* --sem->count */• "js 2f\n"• "1:\n"• LOCK_SECTION_START("")• "2:\tlea %0,%%eax\n\t"• "call __down_failed\n\t"• "jmp 1b\n"• LOCK_SECTION_END• :"=m" (sem->count)• :• :"memory","ax");• }
Page 20
Working with Semaphores – cont.
• //Jump only for contention• static inline void up(struct semaphore * sem)• {• __asm__ __volatile__(• "# atomic up operation\n\t"• LOCK_PREFIX "incl %0\n\t" /* ++sem->count */• "jle 2f\n"• "1:\n"• LOCK_SECTION_START("")• "2:\tlea %0,%%eax\n\t"• "call __up_wakeup\n\t"• "jmp 1b\n"• LOCK_SECTION_END• ".subsection 0\n"• :"=m" (sem->count)• :• :"memory","ax");• }
Page 21
Semaphore Commands
•IPC_STAT – Retrieves the semid_ds structure for a
set, and stores it in the address of the buf argument in the semun union.
•IPC_SET – Sets the value of the ipc_perm
member of the semid_ds structure for a set. Takes the values from the buf argument of the semun union.
•IPC_RMID – Removes the set from the kernel.
•GETALL – Used to obtain the values of all
semaphores in a set. The integer values are stored in an array of unsigned short integers pointed to by the array member of the union.
•GETNCNT – Returns the number of processes
currently waiting for resources.
•GETPID – Returns the PID of the process which
performed the last semop call.
•GETVAL – Returns the value of a single
semaphore within the set.
•GETZCNT – Returns the number of processes
currently waiting for 100% resource utilization.
•SETALL – Sets all semaphore values with a set
to the matching values contained in the array member of the union.
•SETVAL – Sets the value of an individual
semaphore within the set to the val member of the union.
Page 22
Wait queue
• Processes are added to the end of the queue– down– down_interuptable– down_trylock
• Processes are removed from the front of the queue– up
o Add the process to the run queueo Next time the process is scheduled it is removed from the wait
queue
• Processes in the wait queue can be interrupted– Scheduler, when it runs next, will suspend waiting processes
that are interrupted
Page 23
Project Outline
•The Big Pictures•The changes I made•The results
Page 24
My project – The Big Picture
• Releasing processes from the wait queue– Moved to run queue– Leave the rest to the scheduler
• Why release in FIFO?– Probably the result of a coin flip.
• Shouldn’t we use the dynamic priorities?– Seems like a good idea.
• Release All/Most/Some/More than one – Trust that the scheduler will pick the very best one and run
it first.– Then the rest can be added back on the wait queue
o In order of their dynamic priorities
– sem_queue
Page 25
If it Ain’t Broke…Lets find out how it works and tweak it.• static inline void up(struct semaphore * sem)• {• int i;• for(i=0; i<10; i++)• {• __asm__ __volatile__(• "# atomic up operation\n\t"• LOCK_PREFIX "incl %0\n\t" /* ++sem->count */• "jle 2f\n"• "1:\n"• LOCK_SECTION_START("")• "2:\tlea %0,%%eax\n\t"• "call __up_wakeup\n\t"• "jmp 1b\n"• LOCK_SECTION_END• ".subsection 0\n"• :"=m" (sem->count)• :• :"memory","ax");• }• }
Page 26
Results
•The while loop broke it.•Played with a for loop.
– ~10 booted fine– Larger numbers broke it.
•After doing Student’s t-Test we find that there is a 97% confidence that the before and after data sets came from the same population.
Page 27
Results – cont.
Execution times
75
75.1
75.2
75.3
75.4
75.5
75.6
75.7
75.8
75.9
76
<10 Control
Se
co
nd
s
Page 28
SUM
• Primary goal: dig into the synchronization in Linux and learn lots about semaphores– Success
• Secondary goal: alter the semaphore in a way that would leave the computer operational– Success
• Tertiary goal: pass the class without losing my job, girlfriend, or sanity (with “sanity” being loosely defined)– Pending (I think it’ll turn out ok as long as I get her a nice
Christmas present)
Page 29
Future work
• Examine how the scheduler interacts with the wait queue– Rather than release from the front of the queue would it be
better to let the scheduler choose which process to release from the queue?
o Too much overhead? (Maybe we could even have another O(1) scheduler.)
o Gain in performance with respect to priority levels?
Page 30
References
• D. Bovet and M. Cesati. Understanding the Linux Kernel, Third Edition.O'Reilly and Associates, Inc., 2002.
• The Linux Documentation Project: http://www.tldp.org/LDP/lpg/node46.html
• R. Kline. Peterson's Algorithm, 2006. http://www.cs.wcupa.edu/~rkline/OS/Peterson.html
• The Linux Tutorial: http://www.linux-tutorial.info• http://lxr.linux.no/source/include/asm-i386/semaphore.h• http://lxr.linux.no/source/ipc/sem.c• Student’s t-Test:
http://www.physics.csbsju.edu/stats/t-test.html
Page 31
Special Thanks To:
•Everybody that is still awake•The synchronized swimmers from University of Minnesota
Page 32
Questions?