concurrent queues and stacks - brown...
TRANSCRIPT
Concurrent Queues and Stacks
Companion slides for
The Art of Multiprocessor Programming
by Maurice Herlihy & Nir Shavit
Art of Multiprocessor Programming 22
The Five-Fold Path
• Coarse-grained locking
• Fine-grained locking
• Optimistic synchronization
• Lazy synchronization
• Lock-free synchronization
Art of Multiprocessor Programming 33
Another Fundamental Problem
• We told you about
– Sets implemented by linked lists
– Hash Tables
• Next: queues
• Next: stacks
Art of Multiprocessor Programming 44
Queues & Stacks
• pool of items
Art of Multiprocessor Programming 55
Queues
deq()/enq( )
Total order
First in
First out
Art of Multiprocessor Programming 66
Stacks
pop()/
push( )
Total order
Last in
First out
Art of Multiprocessor Programming 77
Bounded
• Fixed capacity
• Good when resources an issue
Art of Multiprocessor Programming 88
Unbounded
• Unlimited capacity
• Often more convenient
Art of Multiprocessor Programming 9
Blocking
zzz …
Block on attempt to remove from empty stack or queue
Art of Multiprocessor Programming 10
Blocking
zzz …
Block on attempt to add to full bounded stack or queue
Art of Multiprocessor Programming 11
Non-Blocking
Ouch!
Throw exception on attempt to remove from empty stack or queue
Art of Multiprocessor Programming 1212
This Lecture
• Queue
– Bounded, blocking, lock-based
– Unbounded, non-blocking, lock-free
• Stack
– Unbounded, non-blocking lock-free
– Elimination-backoff algorithm
Art of Multiprocessor Programming 1313
Queue: Concurrency
enq(x) y=deq()
enq() and deq()
work at different
ends of the object
tail head
Art of Multiprocessor Programming 1414
Concurrency
enq(x)
Challenge: what if
the queue is empty
or full?
y=deq()
Art of Multiprocessor Programming 1515
Bounded Queue
Sentinel
head
tail
Art of Multiprocessor Programming 1616
Bounded Queue
head
tail
First actual item
Art of Multiprocessor Programming 1717
Bounded Queue
head
tail
Lock out other
deq() calls
deqLock
Art of Multiprocessor Programming 1818
Bounded Queue
head
tail
Lock out other
enq() calls
deqLock
enqLock
Art of Multiprocessor Programming 1919
Not Done Yet
head
tail
deqLock
enqLock
Need to tell whether
queue is full or
empty
Art of Multiprocessor Programming 2020
Not Done Yet
head
tail
deqLock
enqLock
Max size is 8 items
size
1
Art of Multiprocessor Programming 2121
Not Done Yet
head
tail
deqLock
enqLock
Incremented by enq()
Decremented by deq()
size
1
Art of Multiprocessor Programming 2222
Enqueuer
1
Lock enqLock
head
tail
deqLock
enqLock
size
Art of Multiprocessor Programming 2323
Enqueuer
1
Read size
OK
head
tail
deqLock
enqLock
size
Art of Multiprocessor Programming 2424
Enqueuer
1
No need to
lock tail
head
tail
deqLock
enqLock
size
Art of Multiprocessor Programming 2525
Enqueuer
1
Enqueue Node
head
tail
deqLock
enqLock
size
Art of Multiprocessor Programming 2626
Enqueuer
size
12
getAndincrement()
head
tail
deqLock
enqLock
Art of Multiprocessor Programming 2727
Enqueuer
8Release lock
2
head
tail
deqLock
enqLock
size
Art of Multiprocessor Programming 2828
Enqueuer
2
If queue was empty,
notify waiting dequeuers
head
tail
deqLock
enqLock
size
Art of Multiprocessor Programming 2929
Unsuccesful Enqueuer
8
Uh-oh
Read size
…head
tail
deqLock
enqLock
size
Art of Multiprocessor Programming 3030
Dequeuer
2
Lock deqLock
head
tail
deqLock
enqLock
size
Art of Multiprocessor Programming 3131
Dequeuer
2
Read sentinel’s
next field
OK
head
tail
deqLock
enqLock
size
Art of Multiprocessor Programming 3232
Dequeuer
siz
2
Read value
head
tail
deqLock
enqLock
size
Art of Multiprocessor Programming 3333
Dequeuer
2
Make first Node
new sentinel
head
tail
deqLock
enqLock
size
Art of Multiprocessor Programming 3434
Dequeuer
1
Decrement
size
head
tail
deqLock
enqLock
size
Art of Multiprocessor Programming 3535
Dequeuer
1Release
deqLock
head
tail
deqLock
enqLock
size
Art of Multiprocessor Programming 3636
Unsuccesful Dequeuer
0
Read sentinel’s
next field
uh-oh
head
tail
deqLock
enqLock
size
Art of Multiprocessor Programming 3737
Bounded Queue
public class BoundedQueue<T> {
ReentrantLock enqLock, deqLock;
Condition notEmptyCondition, notFullCondition;
AtomicInteger size;
Node head;
Node tail;
int capacity;
enqLock = new ReentrantLock();
notFullCondition = enqLock.newCondition();
deqLock = new ReentrantLock();
notEmptyCondition = deqLock.newCondition();
}
Art of Multiprocessor Programming 3838
Bounded Queue
public class BoundedQueue<T> {
ReentrantLock enqLock, deqLock;
Condition notEmptyCondition, notFullCondition;
AtomicInteger size;
Node head;
Node tail;
int capacity;
enqLock = new ReentrantLock();
notFullCondition = enqLock.newCondition();
deqLock = new ReentrantLock();
notEmptyCondition = deqLock.newCondition();
}
enq & deq locks
Art of Multiprocessor Programming 39
(push)
Art of Multiprocessor Programming 4040
Digression: Monitor Locks
• Java synchronized objects and ReentrantLocks are monitors
• Allow blocking on a condition rather than spinning
• Threads: – acquire and release lock
– wait on a condition
Art of Multiprocessor Programming 4141
public interface Lock {
void lock();
void lockInterruptibly() throws InterruptedException;
boolean tryLock();
boolean tryLock(long time, TimeUnit unit);
Condition newCondition();
void unlock;
}
The Java Lock Interface
Acquire lock
Art of Multiprocessor Programming 4242
public interface Lock {
void lock();
void lockInterruptibly() throws InterruptedException;
boolean tryLock();
boolean tryLock(long time, TimeUnit unit);
Condition newCondition();
void unlock;
}
The Java Lock Interface
Release lock
Art of Multiprocessor Programming 4343
public interface Lock {
void lock();
void lockInterruptibly() throws InterruptedException;
boolean tryLock();
boolean tryLock(long time, TimeUnit unit);
Condition newCondition();
void unlock;
}
The Java Lock Interface
Try for lock, but not too hard
Art of Multiprocessor Programming 4444
public interface Lock {
void lock();
void lockInterruptibly() throws InterruptedException;
boolean tryLock();
boolean tryLock(long time, TimeUnit unit);
Condition newCondition();
void unlock;
}
The Java Lock Interface
Create condition to wait on
Art of Multiprocessor Programming 4545
The Java Lock Interface
public interface Lock {
void lock();
void lockInterruptibly() throws InterruptedException;
boolean tryLock();
boolean tryLock(long time, TimeUnit unit);
Condition newCondition();
void unlock;
}
Never mind what this method does
Art of Multiprocessor Programming 4646
Lock Conditions
public interface Condition {
void await();
boolean await(long time, TimeUnit unit);
…
void signal();
void signalAll();
}
Art of Multiprocessor Programming 4747
public interface Condition {
void await();
boolean await(long time, TimeUnit unit);
…
void signal();
void signalAll();
}
Lock Conditions
Release lock and
wait on condition
Art of Multiprocessor Programming 4848
public interface Condition {
void await();
boolean await(long time, TimeUnit unit);
…
void signal();
void signalAll();
}
Lock Conditions
Wake up one waiting thread
Art of Multiprocessor Programming 4949
public interface Condition {
void await();
boolean await(long time, TimeUnit unit);
…
void signal();
void signalAll();
}
Lock Conditions
Wake up all waiting threads
Art of Multiprocessor Programming 5050
Await
• Releases lock associated with q
• Sleeps (gives up processor)
• Awakens (resumes running)
• Reacquires lock & returns
q.await()
Art of Multiprocessor Programming 5151
Signal
• Awakens one waiting thread
– Which will reacquire lock
q.signal();
Art of Multiprocessor Programming 5252
Signal All
• Awakens all waiting threads
– Which will each reacquire lock
q.signalAll();
Art of Multiprocessor Programming 5353
A Monitor Lock
Cri
tical S
ecti
on
waiting room
lock()
unlock()
Art of Multiprocessor Programming 5454
Unsuccessful Deq
Cri
tical S
ecti
on
waiting room
lock()
await()
deq()
Oh no,
empty!
Art of Multiprocessor Programming 5555
Another One
Cri
tical S
ecti
on
waiting room
lock()
await()
deq()
Oh no,
empty!
Art of Multiprocessor Programming 5656
Enqueuer to the Rescue
Cri
tical S
ecti
on
waiting room
lock()
signalAll()enq( )
unlock()
Yawn!Yawn!
Art of Multiprocessor Programming 5757
Yawn!
Monitor Signalling
Cri
tical S
ecti
on
waiting room
Yawn!
Awakened thread
might still lose lock to
outside contender…
Art of Multiprocessor Programming 5858
Dequeuers Signalled
Cri
tical S
ecti
on
waiting room
Found it
Yawn!
Art of Multiprocessor Programming 5959
Yawn!
Dequeuers Signaled
Cri
tical S
ecti
on
waiting room
Still empty!
Art of Multiprocessor Programming 6060
Dollar Short + Day Late
Cri
tical S
ecti
on
waiting room
Art of Multiprocessor Programming 6161
public class Queue<T> {
int head = 0, tail = 0;
T[QSIZE] items;
public synchronized T deq() {
while (tail – head == 0)
wait();
T result = items[head % QSIZE]; head++;
notifyAll();
return result;
}
…
}}
Java Synchronized Methods
Art of Multiprocessor Programming 6262
public class Queue<T> {
int head = 0, tail = 0;
T[QSIZE] items;
public synchronized T deq() {
while (tail – head == 0)
wait();
T result = items[head % QSIZE]; head++;
notifyAll();
return result;
}
…
}}
Java Synchronized Methods
Each object has an implicit
lock with an implicit condition
Art of Multiprocessor Programming 6363
public class Queue<T> {
int head = 0, tail = 0;
T[QSIZE] items;
public synchronized T deq() {
while (tail – head == 0)
wait();
T result = items[head % QSIZE]; head++;
notifyAll();
return result;
}
…
}}
Java Synchronized Methods
Lock on entry,
unlock on return
Art of Multiprocessor Programming 6464
public class Queue<T> {
int head = 0, tail = 0;
T[QSIZE] items;
public synchronized T deq() {
while (tail – head == 0)
wait();
T result = items[head % QSIZE]; head++;
this.notifyAll();
return result;
}
…
}}
Java Synchronized Methods
Wait on implicit
condition
Art of Multiprocessor Programming 6565
public class Queue<T> {
int head = 0, tail = 0;
T[QSIZE] items;
public synchronized T deq() {
while (tail – head == 0)
this.wait();
T result = items[head % QSIZE]; head++;
notifyAll();
return result;
}
…
}}
Java Synchronized Methods
Signal all threads waiting
on condition
Art of Multiprocessor Programming 6666
(Pop!) The Bounded Queue
public class BoundedQueue<T> {
ReentrantLock enqLock, deqLock;
Condition notEmptyCondition, notFullCondition;
AtomicInteger size;
Node head;
Node tail;
int capacity;
enqLock = new ReentrantLock();
notFullCondition = enqLock.newCondition();
deqLock = new ReentrantLock();
notEmptyCondition = deqLock.newCondition();
}
Art of Multiprocessor Programming 6767
Bounded Queue Fields
public class BoundedQueue<T> {
ReentrantLock enqLock, deqLock;
Condition notEmptyCondition, notFullCondition;
AtomicInteger size;
Node head;
Node tail;
int capacity;
enqLock = new ReentrantLock();
notFullCondition = enqLock.newCondition();
deqLock = new ReentrantLock();
notEmptyCondition = deqLock.newCondition();
}
Enq & deq locks
Art of Multiprocessor Programming 6868
Bounded Queue Fields
public class BoundedQueue<T> {
ReentrantLock enqLock, deqLock;
Condition notEmptyCondition, notFullCondition;
AtomicInteger size;
Node head;
Node tail;
int capacity;
enqLock = new ReentrantLock();
notFullCondition = enqLock.newCondition();
deqLock = new ReentrantLock();
notEmptyCondition = deqLock.newCondition();
}
Enq lock’s associated
condition
Art of Multiprocessor Programming 6969
Bounded Queue Fields
public class BoundedQueue<T> {
ReentrantLock enqLock, deqLock;
Condition notEmptyCondition, notFullCondition;
AtomicInteger size;
Node head;
Node tail;
int capacity;
enqLock = new ReentrantLock();
notFullCondition = enqLock.newCondition();
deqLock = new ReentrantLock();
notEmptyCondition = deqLock.newCondition();
}
size: 0 to capacity
Art of Multiprocessor Programming 7070
Bounded Queue Fields
public class BoundedQueue<T> {
ReentrantLock enqLock, deqLock;
Condition notEmptyCondition, notFullCondition;
AtomicInteger size;
Node head;
Node tail;
int capacity;
enqLock = new ReentrantLock();
notFullCondition = enqLock.newCondition();
deqLock = new ReentrantLock();
notEmptyCondition = deqLock.newCondition();
}
Head and Tail
Art of Multiprocessor Programming 7171
Enq Method Part Onepublic void enq(T x) {
boolean mustWakeDequeuers = false;
enqLock.lock();
try {
while (size.get() == Capacity)
notFullCondition.await();
Node e = new Node(x);
tail.next = e;
tail = tail.next;
if (size.getAndIncrement() == 0)
mustWakeDequeuers = true;
} finally {
enqLock.unlock();
}
…
}
Art of Multiprocessor Programming 7272
public void enq(T x) {
boolean mustWakeDequeuers = false;
enqLock.lock();
try {
while (size.get() == capacity)
notFullCondition.await();
Node e = new Node(x);
tail.next = e;
tail = tail.next;
if (size.getAndIncrement() == 0)
mustWakeDequeuers = true;
} finally {
enqLock.unlock();
}
…
}
Enq Method Part One
Lock and unlock
enq lock
Art of Multiprocessor Programming 7373
public void enq(T x) {
boolean mustWakeDequeuers = false;
enqLock.lock();
try {
while (size.get() == capacity)
notFullCondition.await();
Node e = new Node(x);
tail.next = e;
tail = tail.next;
if (size.getAndIncrement() == 0)
mustWakeDequeuers = true;
} finally {
enqLock.unlock();
}
…
}
Enq Method Part One
Wait while queue is full …
Art of Multiprocessor Programming 7474
public void enq(T x) {
boolean mustWakeDequeuers = false;
enqLock.lock();
try {
while (size.get() == capacity)
notFullCondition.await();
Node e = new Node(x);
tail.next = e;
tail = tail.next;
if (size.getAndIncrement() == 0)
mustWakeDequeuers = true;
} finally {
enqLock.unlock();
}
…
}
Enq Method Part One
when await() returns, you
might still fail the test !
Art of Multiprocessor Programming 7575
public void enq(T x) {
boolean mustWakeDequeuers = false;
enqLock.lock();
try {
while (size.get() == capacity)
notFullCondition.await();
Node e = new Node(x);
tail.next = e;
tail = tail.next;
if (size.getAndIncrement() == 0)
mustWakeDequeuers = true;
} finally {
enqLock.unlock();
}
…
}
Be Afraid
After the loop: how do we know the
queue won’t become full again?
Art of Multiprocessor Programming 7676
public void enq(T x) {
boolean mustWakeDequeuers = false;
enqLock.lock();
try {
while (size.get() == capacity)
notFullCondition.await();
Node e = new Node(x);
tail.next = e;
tail = tail.next;
if (size.getAndIncrement() == 0)
mustWakeDequeuers = true;
} finally {
enqLock.unlock();
}
…
}
Enq Method Part One
Add new node
Art of Multiprocessor Programming 7777
public void enq(T x) {
boolean mustWakeDequeuers = false;
enqLock.lock();
try {
while (size.get() == capacity)
notFullCondition.await();
Node e = new Node(x);
tail.next = e;
tail = tail.next;
if (size.getAndIncrement() == 0)
mustWakeDequeuers = true;
} finally {
enqLock.unlock();
}
…
}
Enq Method Part One
If queue was empty, wake
frustrated dequeuers
Art of Multiprocessor Programming 7878
Beware Lost Wake-Ups
Cri
tical S
ecti
on
waiting room
lock()
Queue empty
so signal ()
enq( )
unlock()
Yawn!
Art of Multiprocessor Programming 7979
Lost Wake-Up
Cri
tical S
ecti
on
waiting room
lock()
enq( )
unlock()
Yawn!
Queue not
empty so no
need to signal
Art of Multiprocessor Programming 8080
Lost Wake-Up
Cri
tical S
ecti
on
waiting room
Yawn!
Art of Multiprocessor Programming 8181
Lost Wake-Up
Cri
tical S
ecti
on
waiting room
Found it
Art of Multiprocessor Programming 8282
What’s Wrong Here?
Cri
tical S
ecti
on
waiting room
Still waiting ….!
Art of Multiprocessor Programming 83
Solution to Lost Wakeup
• Always use
– signalAll() and notifyAll()
• Not
– signal() and notify()
83
Art of Multiprocessor Programming 84
(pop)
Art of Multiprocessor Programming 8585
Enq Method Part Deux
public void enq(T x) {
…
if (mustWakeDequeuers) {
deqLock.lock();
try {
notEmptyCondition.signalAll();
} finally {
deqLock.unlock();
}
}
}
Art of Multiprocessor Programming 8686
Enq Method Part Deux
public void enq(T x) {
…
if (mustWakeDequeuers) {
deqLock.lock();
try {
notEmptyCondition.signalAll();
} finally {
deqLock.unlock();
}
}
}Are there dequeuers to be signaled?
Art of Multiprocessor Programming 8787
public void enq(T x) {
…
if (mustWakeDequeuers) {
deqLock.lock();
try {
notEmptyCondition.signalAll();
} finally {
deqLock.unlock();
}
}
}
Enq Method Part Deux
Lock and
unlock deq lock
Art of Multiprocessor Programming 8888
public void enq(T x) {
…
if (mustWakeDequeuers) {
deqLock.lock();
try {
notEmptyCondition.signalAll();
} finally {
deqLock.unlock();
}
}
}
Enq Method Part Deux
Signal dequeuers that
queue is no longer empty
Art of Multiprocessor Programming 8989
The enq() & deq() Methods
• Share no locks
– That’s good
• But do share an atomic counter
– Accessed on every method call
– That’s not so good
• Can we alleviate this bottleneck?
Art of Multiprocessor Programming 9090
Split the Counter
• The enq() method
– Increments only
– Cares only if value is capacity
• The deq() method
– Decrements only
– Cares only if value is zero
Art of Multiprocessor Programming 9191
Split Counter
• Enqueuer increments enqSize
• Dequeuer increments deqSize
• When enqueuer hits capacity– Locks deqLock
– Sets size = enqSize - DeqSize
• Intermittent synchronization
– Not with each method call
– Need both locks! (careful …)
Art of Multiprocessor Programming 9292
A Lock-Free Queue
Sentinel
head
tail
Art of Multiprocessor Programming 9393
Compare and Set
CAS
Art of Multiprocessor Programming 9494
Enqueue
head
tail
enq( )
Art of Multiprocessor Programming 9595
Enqueue
head
tail
Art of Multiprocessor Programming 9696
Logical Enqueue
head
tail
CAS
Art of Multiprocessor Programming 9797
Physical Enqueue
head
tailCAS
Art of Multiprocessor Programming 9898
Enqueue
• These two steps are not atomic
• The tail field refers to either
– Actual last Node (good)
– Penultimate Node (not so good)
• Be prepared!
Art of Multiprocessor Programming 9999
Enqueue
• What do you do if you find
– A trailing tail?
• Stop and help fix it
– If tail node has non-null next field
– CAS the queue’s tail field to tail.next
• As in the universal construction
Art of Multiprocessor Programming 100100
When CASs Fail
• During logical enqueue
– Abandon hope, restart
– Still lock-free (why?)
• During physical enqueue
– Ignore it (why?)
Art of Multiprocessor Programming 101101
Dequeuer
head
tail
Read value
Art of Multiprocessor Programming 102102
Dequeuer
head
tail
Make first Node
new sentinel
CAS
Art of Multiprocessor Programming 103103
Memory Reuse?
• What do we do with nodes after we
dequeue them?
• Java: let garbage collector deal?
• Suppose there is no GC, or we prefer
not to use it?
Art of Multiprocessor Programming 104104
Dequeuer
head
tail
CAS
Can recycle
Art of Multiprocessor Programming 105105
Simple Solution
• Each thread has a free list of unused
queue nodes
• Allocate node: pop from list
• Free node: push onto list
• Deal with underflow somehow …
Art of Multiprocessor Programming 106106
Why Recycling is Hard
Free pool
head tail
Want to
redirect
head from
gray to red
zzz…
Art of Multiprocessor Programming 107107
Both Nodes Reclaimed
Free pool
zzz
head tail
Art of Multiprocessor Programming 108108
One Node Recycled
Free pool
Yawn!
head tail
Art of Multiprocessor Programming 109109
Why Recycling is Hard
Free pool
CAS
head tail
OK, here
I go!
Art of Multiprocessor Programming 110110
Recycle FAIL
Free pool
zOMG what went wrong?
head tail
Art of Multiprocessor Programming 111111
The Dreaded ABA Problemhead tail
Head reference has value A
Thread reads value A
Art of Multiprocessor Programming 112112
Dreaded ABA continued
zzz
head tail
Head reference has value B
Node A freed
Art of Multiprocessor Programming 113113
Dreaded ABA continued
Yawn!
head tail
Head reference has value A again
Node A recycled and reinitialized
Art of Multiprocessor Programming 114114
Dreaded ABA continued
CAS
head tail
CAS succeeds because references match,
even though reference’s meaning has changed
Art of Multiprocessor Programming 115115
The Dreaded ABA FAIL
• Is a result of CAS() semantics
– Oracle, Intel, AMD, …
• Not with Load-Locked/Store-Conditional
– IBM …
Art of Multiprocessor Programming 116116
Dreaded ABA – A Solution
• Tag each pointer with a counter
• Unique over lifetime of node
• Pointer size vs word size issues
• Overflow?
– Don’t worry be happy?
– Bounded tags?
• AtomicStampedReference class
Art of Multiprocessor Programming 117117
Atomic Stamped Reference
• AtomicStampedReference class
– Java.util.concurrent.atomic package
address S
Stamp
Reference
Can get reference & stamp atomically
Art of Multiprocessor Programming 118118
Concurrent Stack
• Methods
– push(x)
– pop()
• Last-in, First-out (LIFO) order
• Lock-Free!
Art of Multiprocessor Programming 119119
Empty Stack
Top
Art of Multiprocessor Programming 120120
Push
Top
Art of Multiprocessor Programming 121121
Push
TopCAS
Art of Multiprocessor Programming 122122
Push
Top
Art of Multiprocessor Programming 123123
Push
Top
Art of Multiprocessor Programming 124124
Push
Top
Art of Multiprocessor Programming 125125
Push
TopCAS
Art of Multiprocessor Programming 126126
Push
Top
Art of Multiprocessor Programming 127127
Pop
Top
Art of Multiprocessor Programming 128128
Pop
TopCAS
Art of Multiprocessor Programming 129129
Pop
TopCAS
mine!
Art of Multiprocessor Programming 130130
Pop
TopCAS
Art of Multiprocessor Programming 131131
Pop
Top
Art of Multiprocessor Programming 132132
public class LockFreeStack {
private AtomicReference top =
new AtomicReference(null);
public boolean tryPush(Node node){
Node oldTop = top.get();
node.next = oldTop;
return(top.compareAndSet(oldTop, node))
}
public void push(T value) {
Node node = new Node(value);
while (true) {
if (tryPush(node)) {
return;
} else backoff.backoff();
}}
Lock-free Stack
Art of Multiprocessor Programming 133133
public class LockFreeStack {
private AtomicReference top = new
AtomicReference(null);
public Boolean tryPush(Node node){
Node oldTop = top.get();
node.next = oldTop;
return(top.compareAndSet(oldTop, node))
}
public void push(T value) {
Node node = new Node(value);
while (true) {
if (tryPush(node)) {
return;
} else backoff.backoff()
}}
Lock-free Stack
tryPush attempts to push a node
Art of Multiprocessor Programming 134134
public class LockFreeStack {
private AtomicReference top = new
AtomicReference(null);
public boolean tryPush(Node node){
Node oldTop = top.get();
node.next = oldTop;
return(top.compareAndSet(oldTop, node))
}
public void push(T value) {
Node node = new Node(value);
while (true) {
if (tryPush(node)) {
return;
} else backoff.backoff()
}}
Lock-free Stack
Read top value
Art of Multiprocessor Programming 135135
public class LockFreeStack {
private AtomicReference top = new
AtomicReference(null);
public boolean tryPush(Node node){
Node oldTop = top.get();
node.next = oldTop;
return(top.compareAndSet(oldTop, node))
}
public void push(T value) {
Node node = new Node(value);
while (true) {
if (tryPush(node)) {
return;
} else backoff.backoff()
}}
Lock-free Stack
current top will be new node’s successor
Art of Multiprocessor Programming 136136
public class LockFreeStack {
private AtomicReference top = new
AtomicReference(null);
public boolean tryPush(Node node){
Node oldTop = top.get();
node.next = oldTop;
return(top.compareAndSet(oldTop, node))
}
public void push(T value) {
Node node = new Node(value);
while (true) {
if (tryPush(node)) {
return;
} else backoff.backoff()
}}
Lock-free Stack
Try to swing top, return success or failure
Art of Multiprocessor Programming 137137
public class LockFreeStack {
private AtomicReference top = new
AtomicReference(null);
public boolean tryPush(Node node){
Node oldTop = top.get();
node.next = oldTop;
return(top.compareAndSet(oldTop, node))
}
public void push(T value) {
Node node = new Node(value);
while (true) {
if (tryPush(node)) {
return;
} else backoff.backoff()
}}
Lock-free Stack
Push calls tryPush
Art of Multiprocessor Programming 138138
public class LockFreeStack {
private AtomicReference top = new
AtomicReference(null);
public boolean tryPush(Node node){
Node oldTop = top.get();
node.next = oldTop;
return(top.compareAndSet(oldTop, node))
}
public void push(T value) {
Node node = new Node(value);
while (true) {
if (tryPush(node)) {
return;
} else backoff.backoff()
}}
Lock-free Stack
Create new node
Art of Multiprocessor Programming 139139
public class LockFreeStack {
private AtomicReference top = new
AtomicReference(null);
public boolean tryPush(Node node){
Node oldTop = top.get();
node.next = oldTop;
return(top.compareAndSet(oldTop, node))
}
public void push(T value) {
Node node = new Node(value);
while (true) {
if (tryPush(node)) {
return;
} else backoff.backoff()
}}
Lock-free Stack
If tryPush() fails,
back off before retrying
Art of Multiprocessor Programming 140140
Lock-free Stack
• Good
– No locking
• Bad
– Without GC, fear ABA
– Without backoff, huge contention at top
– In any case, no parallelism
Art of Multiprocessor Programming 141141
Big Question
• Are stacks inherently sequential?
• Reasons why
– Every pop() call fights for top item
• Reasons why not
– Stay tuned …
Art of Multiprocessor Programming 142142
Elimination-Backoff Stack
• How to
– “turn contention into parallelism”
• Replace familiar
– exponential backoff
• With alternative
– elimination-backoff
Art of Multiprocessor Programming 143143
Observation
Push( )
Pop()
linearizable stack
After an equal number
of pushes and pops,
stack stays the sameYes!
Art of Multiprocessor Programming 144144
Idea: Elimination Array
Push( )
Pop()
stack
Pick at
random
Pick at
random Elimination
Array
Art of Multiprocessor Programming 145145
Push Collides With Pop
Push( )
Pop()
stack
continue
continue
No need to
access stack Yes!
Art of Multiprocessor Programming 146146
No Collision
Push( )
Pop()
stack
If no collision,
access stack
If pushes collide or
pops collide
access stack
Art of Multiprocessor Programming 147147
Elimination-Backoff Stack
• Lock-free stack + elimination array
• Access Lock-free stack,
– If uncontended, apply operation
– if contended, back off to elimination array
and attempt elimination
Art of Multiprocessor Programming 148148
Elimination-Backoff Stack
Push( )
Pop()TopCAS
If CAS fails, back off
Art of Multiprocessor Programming 149149
Dynamic Range and Delay
Push( )
Pick random range and
max waiting time based
on level of contention
encountered
50-50, Random Slots
0
20000
40000
60000
80000
100000
120000
2 4 8 121 62 4 324 04 85 66 4
Threads
ops/ msec Exchanger
Treiber
Asymmetric Rendevous
pop1()pop2()
Push( )
Pops find first vacant slot
and spin. Pushes hunt for
pops.
15 94 26head:
Asymmetric vs. Symmetric
0
20000
40000
60000
80000
100000
120000
2 4 8 12 16 24 32 40 48 56 64
op
s/m
se
c
Threads
50% Push/50% Pop/Random ids
Exchanger
Rendezvous
Treiber
(a) Instructions/op
0
100
200
300
400
500
600
2 4 8 12 16 24 32 40 48 56 64
Threads
instr
ucti
on
s/o
p
Exchanger
Rendezvous
Treiber
(b) Cache misses/op
0
5
10
15
20
25
30
2 4 8 12 16 24 32 40 48 56 64
Threads
instr
ucti
on
s/o
p
Exchanger
Rendezvous
Treiber
(c) CAS/op
0
5
10
15
20
25
30
2 4 8 12 16 24 32 40 48 56 64
Threads
instr
ucti
on
s/o
p
Exchanger
Rendezvous
Treiber
50% Push/50% Pop/Random ids
0
20000
40000
60000
80000
100000
120000
2 4 8 12 16 24 32 40 48 56 64
Threads
op
s/m
sec
Exchanger
Rendezvous
Treiber
50% Push/50% Pop
0
20000
40000
60000
80000
100000
120000
140000
2 4 8 12 16 24 32 40 48 56 64
Threads
op
s/m
se
c
Rendezvous (random ids)
Rendezvous(backoff,
random ids)
Rendezvous (seq ids)
Rendezvous(backoff, seq
ids)
Effect of Backoff and Slot Choice
Effect of Slot Choice
Random choice Sequential choice
Darker shades mean more exchanges
110
115
120
125
130
135
140
145
150
2 4 8 12 16 24 32 40 48 56 64
ins
tru
cti
on
s/o
p
Threads
Rendezvous instruction count:
sequential vs. random ids
Push (random ids) Pop (random ids)
Push (seq ids) Pop (seq ids)
Rendezvous array probes:
sequential vs. random ids
0
2
4
6
8
10
2 4 8 12 16 24 32 40 48 56 64
Threads
# p
ro
bes
Push (random ids) Push (seq ids)
Pop (random ids) Pop (seq ids)
Effect of Slot Choice
Rendezvous Pop spin iterations:
sequential vs. random ids
0
2
4
6
8
1 2 3 4 5 6 7 8 9 10 11
Threads
# s
pin
s
Pop (random ids) Pop (seq ids)
Art of Multiprocessor Programming 157157
Linearizability
• Un-eliminated calls
– linearized as before
• Eliminated calls:
– linearize pop() immediately after
matching push()
• Combination is a linearizable stack
Art of Multiprocessor Programming 158158
Un-Eliminated Linearizability
push(v1)
timetime
push(v1)
pop(v1)pop(v1)
Art of Multiprocessor Programming 159159
Eliminated Linearizability
pop(v2)push(v1)
push(v2)
timetime
push(v2)
pop(v2)push(v1)
pop(v1)
Collision
Point
Red calls are
eliminated
pop(v1)
Art of Multiprocessor Programming 160160
Backoff Has Dual Effect
• Elimination introduces parallelism
• Backoff to array cuts contention on lock-
free stack
• Elimination in array cuts down number
of threads accessing lock-free stack
Art of Multiprocessor Programming 161161
public class EliminationArray {
private static final int duration = ...;
private static final int timeUnit = ...;
Exchanger<T>[] exchanger;
public EliminationArray(int capacity) {
exchanger = new Exchanger[capacity];
for (int i = 0; i < capacity; i++)
exchanger[i] = new Exchanger<T>();
…
}
…
}
Elimination Array
Art of Multiprocessor Programming 162162
public class EliminationArray {
private static final int duration = ...;
private static final int timeUnit = ...;
Exchanger<T>[] exchanger;
public EliminationArray(int capacity) {
exchanger = new Exchanger[capacity];
for (int i = 0; i < capacity; i++)
exchanger[i] = new Exchanger<T>();
…
}
…
}
Elimination Array
An array of Exchangers
Art of Multiprocessor Programming 163163
public class Exchanger<T> {
AtomicStampedReference<T> slot
= new AtomicStampedReference<T>(null, 0);
Digression: A Lock-Free
Exchanger
Art of Multiprocessor Programming 164164
public class Exchanger<T> {
AtomicStampedReference<T> slot
= new AtomicStampedReference<T>(null, 0);
A Lock-Free Exchanger
Atomically modifiable
reference + status
Art of Multiprocessor Programming 165165
Atomic Stamped Reference
• AtomicStampedReference class
– Java.util.concurrent.atomic package
• In C or C++:
address Sreference
stamp
Art of Multiprocessor Programming 166166
Extracting Reference & Stamp
public T get(int[] stampHolder);
Art of Multiprocessor Programming 167167
Extracting Reference & Stamp
public T get(int[] stampHolder);
Returns reference to
object of type T
Returns stamp at
array index 0!
Art of Multiprocessor Programming 168168
Exchanger Status
enum Status {EMPTY, WAITING, BUSY};
Art of Multiprocessor Programming 169169
Exchanger Status
enum Status {EMPTY, WAITING, BUSY};
Nothing yet
enum Status {EMPTY, WAITING, BUSY};
Art of Multiprocessor Programming 170170
Exchange Status
Nothing yet
One thread is waiting
for rendez-vous
Art of Multiprocessor Programming 171171
Exchange Status
enum Status {EMPTY, WAITING, BUSY};
Nothing yet
One thread is waiting
for rendez-vous
Other threads busy
with rendez-vous
Art of Multiprocessor Programming 172172
public T Exchange(T myItem, long nanos)
throws TimeoutException {
long timeBound = System.nanoTime() + nanos;
int[] stampHolder = {EMPTY};
while (true) {
if (System.nanoTime() > timeBound)
throw new TimeoutException();
T herItem = slot.get(stampHolder);
int stamp = stampHolder[0];
switch(stamp) {
case EMPTY: … // slot is free
case WAITING: … // someone waiting for me
case BUSY: … // others exchanging
}
}
The Exchange
Art of Multiprocessor Programming 173173
public T Exchange(T myItem, long nanos)
throws TimeoutException {
long timeBound = System.nanoTime() + nanos;
int[] stampHolder = {EMPTY};
while (true) {
if (System.nanoTime() > timeBound)
throw new TimeoutException();
T herItem = slot.get(stampHolder);
int stamp = stampHolder[0];
switch(stamp) {
case EMPTY: … // slot is free
case WAITING: … // someone waiting for me
case BUSY: … // others exchanging
}
}
The Exchange
Item and timeout
Art of Multiprocessor Programming 174174
public T Exchange(T myItem, long nanos)
throws TimeoutException {
long timeBound = System.nanoTime() + nanos;
int[] stampHolder = {EMPTY};
while (true) {
if (System.nanoTime() > timeBound)
throw new TimeoutException();
T herItem = slot.get(stampHolder);
int stamp = stampHolder[0];
switch(stamp) {
case EMPTY: … // slot is free
case WAITING: … // someone waiting for me
case BUSY: … // others exchanging
}
}
The Exchange
Array holds status
Art of Multiprocessor Programming 175175
public T Exchange(T myItem, long nanos) throws
TimeoutException {
long timeBound = System.nanoTime() + nanos;
int[] stampHolder = {0};
while (true) {
if (System.nanoTime() > timeBound)
throw new TimeoutException();
T herItem = slot.get(stampHolder);
int stamp = stampHolder[0];
switch(stamp) {
case EMPTY: // slot is free
case WAITING: // someone waiting for me
case BUSY: // others exchanging
}
}}
The Exchange
Loop until timeout
Art of Multiprocessor Programming 176176
public T Exchange(T myItem, long nanos) throws
TimeoutException {
long timeBound = System.nanoTime() + nanos;
int[] stampHolder = {0};
while (true) {
if (System.nanoTime() > timeBound)
throw new TimeoutException();
T herItem = slot.get(stampHolder);
int stamp = stampHolder[0];
switch(stamp) {
case EMPTY: // slot is free
case WAITING: // someone waiting for me
case BUSY: // others exchanging
}
}}
The Exchange
Get other’s item and status
Art of Multiprocessor Programming 177177
public T Exchange(T myItem, long nanos) throws
TimeoutException {
long timeBound = System.nanoTime() + nanos;
int[] stampHolder = {0};
while (true) {
if (System.nanoTime() > timeBound)
throw new TimeoutException();
T herItem = slot.get(stampHolder);
int stamp = stampHolder[0];
switch(stamp) {
case EMPTY: … // slot is free
case WAITING: … // someone waiting for me
case BUSY: … // others exchanging
}
}}
The Exchange
An Exchanger has three possible states
Art of Multiprocessor Programming 178178
Lock-free Exchanger
EMPTY
Art of Multiprocessor Programming 179179
EMPTY
Lock-free Exchanger
CAS
Art of Multiprocessor Programming 180180
WAITING
Lock-free Exchanger
Art of Multiprocessor Programming 181181
Lock-free Exchanger
In search of
partner …
WAITING
Art of Multiprocessor Programming 182182
WAITING
Lock-free Exchanger
Slot
Still waiting …
Try to exchange
item and set
status to BUSY
CAS
Art of Multiprocessor Programming 183183
BUSY
Lock-free Exchanger
Slot
Partner showed
up, take item and
reset to EMPTY
item status
Art of Multiprocessor Programming 184184
EMPTYBUSY
Lock-free Exchanger
Slot
item status
Partner showed
up, take item and
reset to EMPTY
Art of Multiprocessor Programming 185185
case EMPTY: // slot is free
if (slot.CAS(herItem, myItem, EMPTY, WAITING)) {
while (System.nanoTime() < timeBound){
herItem = slot.get(stampHolder);
if (stampHolder[0] == BUSY) {
slot.set(null, EMPTY);
return herItem;
}}
if (slot.CAS(myItem, null, WAITING, EMPTY)){
throw new TimeoutException();
} else {
herItem = slot.get(stampHolder);
slot.set(null, EMPTY);
return herItem;
}
} break;
Exchanger State EMPTY
Art of Multiprocessor Programming 186186
case EMPTY: // slot is free
if (slot.CAS(herItem, myItem, EMPTY, WAITING)) {
while (System.nanoTime() < timeBound){
herItem = slot.get(stampHolder);
if (stampHolder[0] == BUSY) {
slot.set(null, EMPTY);
return herItem;
}}
if (slot.CAS(myItem, null, WAITING, EMPTY)){
throw new TimeoutException();
} else {
herItem = slot.get(stampHolder);
slot.set(null, EMPTY);
return herItem;
}
} break;
Exchanger State EMPTY
Try to insert myItem and
change state to WAITING
Art of Multiprocessor Programming 187187
case EMPTY: // slot is free
if (slot.CAS(herItem, myItem, EMPTY, WAITING)) {
while (System.nanoTime() < timeBound){
herItem = slot.get(stampHolder);
if (stampHolder[0] == BUSY) {
slot.set(null, EMPTY);
return herItem;
}}
if (slot.CAS(myItem, null, WAITING, EMPTY)){
throw new TimeoutException();
} else {
herItem = slot.get(stampHolder);
slot.set(null, EMPTY);
return herItem;
}
} break;
Exchanger State EMPTY
Spin until either
myItem is taken or timeout
Art of Multiprocessor Programming 188188
case EMPTY: // slot is free
if (slot.CAS(herItem, myItem, EMPTY, WAITING)) {
while (System.nanoTime() < timeBound){
herItem = slot.get(stampHolder);
if (stampHolder[0] == BUSY) {
slot.set(null, EMPTY);
return herItem;
}}
if (slot.CAS(myItem, null, WAITING, EMPTY)){
throw new TimeoutException();
} else {
herItem = slot.get(stampHolder);
slot.set(null, EMPTY);
return herItem;
}
} break;
Exchanger State EMPTY
myItem was taken,
so return herItem
that was put in its place
Art of Multiprocessor Programming 189189
case EMPTY: // slot is free
if (slot.CAS(herItem, myItem, EMPTY, WAITING)) {
while (System.nanoTime() < timeBound){
herItem = slot.get(stampHolder);
if (stampHolder[0] == BUSY) {
slot.set(null, EMPTY);
return herItem;
}}
if (slot.CAS(myItem, null, WAITING, EMPTY)){
throw new TimeoutException();
} else {
herItem = slot.get(stampHolder);
slot.set(null, EMPTY);
return herItem;
}
} break;
Exchanger State EMPTY
Otherwise we ran out of time,
try to reset status to EMPTY
and time out
Art of Multiprocessor Programming 190Art of Multiprocessor
Programming© Herlihy-Shavit
2007
190
case EMPTY: // slot is free
if (slot.compareAndSet(herItem, myItem, WAITING,
BUSY)) {
while (System.nanoTime() < timeBound){
herItem = slot.get(stampHolder);
if (stampHolder[0] == BUSY) {
slot.set(null, EMPTY);
return herItem;
}}
if (slot.compareAndSet(myItem, null, WAITING,
EMPTY)){throw new TimeoutException();
} else {
herItem = slot.get(stampHolder);
slot.set(null, EMPTY);
return herItem;
}
} break;
Exchanger State EMPTY
If reset failed,
someone showed up after all,
so take that item
Art of Multiprocessor Programming 191191
case EMPTY: // slot is free
if (slot.CAS(herItem, myItem, EMPTY, WAITING)) {
while (System.nanoTime() < timeBound){
herItem = slot.get(stampHolder);
if (stampHolder[0] == BUSY) {
slot.set(null, EMPTY);
return herItem;
}}
if (slot.CAS(myItem, null, WAITING, EMPTY)){
throw new TimeoutException();
} else {
herItem = slot.get(stampHolder);
slot.set(null, EMPTY);
return herItem;
}
} break;
Exchanger State EMPTY
Clear slot and take that item
Art of Multiprocessor Programming 192192
case EMPTY: // slot is free
if (slot.CAS(herItem, myItem, EMPTY, WAITING)) {
while (System.nanoTime() < timeBound){
herItem = slot.get(stampHolder);
if (stampHolder[0] == BUSY) {
slot.set(null, EMPTY);
return herItem;
}}
if (slot.CAS(myItem, null, WAITING, EMPTY)){
throw new TimeoutException();
} else {
herItem = slot.get(stampHolder);
slot.set(null, EMPTY);
return herItem;
}
} break;
Exchanger State EMPTY
If initial CAS failed,
then someone else changed status
from EMPTY to WAITING,
so retry from start
Art of Multiprocessor Programming 193193
case WAITING: // someone waiting for me
if (slot.CAS(herItem, myItem, WAITING, BUSY))
return herItem;
break;
case BUSY: // others in middle of exchanging
break;
default: // impossible
break;
}
}
}
}
States WAITING and BUSY
Art of Multiprocessor Programming 194194
case WAITING: // someone waiting for me
if (slot.CAS(herItem, myItem, WAITING, BUSY))
return herItem;
break;
case BUSY: // others in middle of exchanging
break;
default: // impossible
break;
}
}
}
}
States WAITING and BUSY
someone is waiting to exchange,
so try to CAS my item in
and change state to BUSY
Art of Multiprocessor Programming 195195
case WAITING: // someone waiting for me
if (slot.CAS(herItem, myItem, WAITING, BUSY))
return herItem;
break;
case BUSY: // others in middle of exchanging
break;
default: // impossible
break;
}
}
}
}
States WAITING and BUSY
If successful, return other’s item,
otherwise someone else took it,
so try again from start
Art of Multiprocessor Programming 196196
case WAITING: // someone waiting for me
if (slot.CAS(herItem, myItem, WAITING, BUSY))
return herItem;
break;
case BUSY: // others in middle of exchanging
break;
default: // impossible
break;
}
}
}
}
States WAITING and BUSY
If BUSY,
other threads exchanging,
so start again
Art of Multiprocessor Programming 197197
The Exchanger Slot
• Exchanger is lock-free
• Because the only way an exchange can
fail is if others repeatedly succeeded or
no-one showed up
• The slot we need does not require
symmetric exchange
Art of Multiprocessor Programming 198198
public class EliminationArray {
…
public T visit(T value, int range)
throws TimeoutException {
int slot = random.nextInt(range);
int nanodur = convertToNanos(duration, timeUnit));
return (exchanger[slot].exchange(value, nanodur)
}}
Back to the Stack: the
Elimination Array
Art of Multiprocessor Programming 199199
public class EliminationArray {
…
public T visit(T value, int range)
throws TimeoutException {
int slot = random.nextInt(range);
int nanodur = convertToNanos(duration, timeUnit));
return (exchanger[slot].exchange(value, nanodur)
}}
Elimination Array
visit the elimination array
with fixed value and range
Art of Multiprocessor Programming 200200
public class EliminationArray {
…
public T visit(T value, int range)
throws TimeoutException {
int slot = random.nextInt(range);
int nanodur = convertToNanos(duration, timeUnit));
return (exchanger[slot].exchange(value, nanodur)
}}
Elimination Array
Pick a random array entry
Art of Multiprocessor Programming 201201
public class EliminationArray {
…
public T visit(T value, int range)
throws TimeoutException {
int slot = random.nextInt(range);
int nanodur = convertToNanos(duration, timeUnit));
return (exchanger[slot].exchange(value, nanodur)
}}
Elimination Array
Exchange value or time out
Art of Multiprocessor Programming 202202
public void push(T value) {
...
while (true) {
if (tryPush(node)) {
return;
} else try {
T otherValue =
eliminationArray.visit(value,policy.range);
if (otherValue == null) {
return;
}
}
Elimination Stack Push
Art of Multiprocessor Programming 203203
public void push(T value) {
...
while (true) {
if (tryPush(node)) {
return;
} else try {
T otherValue =
eliminationArray.visit(value,policy.range);
if (otherValue == null) {
return;
}
}
Elimination Stack Push
First, try to push
Art of Multiprocessor Programming 204204
public void push(T value) {
...
while (true) {
if (tryPush(node)) {
return;
} else try {
T otherValue =
eliminationArray.visit(value,policy.range);
if (otherValue == null) {
return;
}
}
Elimination Stack Push
If I failed, backoff & try to eliminate
Art of Multiprocessor Programming 205205
public void push(T value) {
...
while (true) {
if (tryPush(node)) {
return;
} else try {
T otherValue =
eliminationArray.visit(value,policy.range);
if (otherValue == null) {
return;
}
}
Elimination Stack Push
Value pushed and range to try
Art of Multiprocessor Programming 206206
public void push(T value) {
...
while (true) {
if (tryPush(node)) {
return;
} else try {
T otherValue =
eliminationArray.visit(value,policy.range);
if (otherValue == null) {
return;
}
}
Elimination Stack Push
Only pop() leaves null,
so elimination was successful
Art of Multiprocessor Programming 207207
public void push(T value) {
...
while (true) {
if (tryPush(node)) {
return;
} else try {
T otherValue =
eliminationArray.visit(value,policy.range);
if (otherValue == null) {
return;
}
}
Elimination Stack Push
Otherwise, retry push() on lock-free stack
Art of Multiprocessor Programming 208208
public T pop() {
...
while (true) {
if (tryPop()) {
return returnNode.value;
} else
try {
T otherValue =
eliminationArray.visit(null,policy.range;
if (otherValue != null) {
return otherValue;
}
}
}}
Elimination Stack Pop
Art of Multiprocessor Programming 209209
public T pop() {
...
while (true) {
if (tryPop()) {
return returnNode.value;
} else
try {
T otherValue =
eliminationArray.visit(null,policy.range;
if ( otherValue != null) {
return otherValue;
}
}
}}
Elimination Stack Pop
If value not null, other thread is a push(),
so elimination succeeded
Art of Multiprocessor Programming 210210
Summary
• We saw both lock-based and lock-freeimplementations of
• queues and stacks
• Don’t be quick to declare a data structure inherently sequential
– Linearizable stack is not inherently sequential (though it is in the worst case)
• ABA is a real problem, pay attention
Art of Multiprocessor Programming 211211
This work is licensed under a Creative Commons Attribution-
ShareAlike 2.5 License.
• You are free:
– to Share — to copy, distribute and transmit the work
– to Remix — to adapt the work
• Under the following conditions:
– Attribution. You must attribute the work to “The Art of
Multiprocessor Programming” (but not in any way that
suggests that the authors endorse you or your use of the
work).
– Share Alike. If you alter, transform, or build upon this work,
you may distribute the resulting work only under the same,
similar or a compatible license.
• For any reuse or distribution, you must make clear to others the
license terms of this work. The best way to do this is with a link
to
– http://creativecommons.org/licenses/by-sa/3.0/.
• Any of the above conditions can be waived if you get permission
from the copyright holder.
• Nothing in this license impairs or restricts the author's moral
rights.