finding concurrency problems in core ruby libraries
TRANSCRIPT
Louis DunnePrincipal Software Engineer @ Workday
Finding Concurrency Problems in Core Ruby
Libraries
Structure of the Talk
• Signals & Stacktraces
• Reproduce / Examine / Experiment
• Root Cause Analysis
• Results From MRI, JRuby, RBX
• Lessons from Other Languages
Signals & Stacktraces
Signal Handlers in 2.0
• Can't use a mutex• Hard to share state safely
reader, writer = IO.pipe
# writer.puts won’t block %w(INT USR2).each do |sig| Signal.trap(sig) { writer.puts(sig) } end
Thread.new { signal_thread(reader) }
Signals & Stacktraces
# signal_thread... sig = reader.gets.chomp # This will block
...
Thread.list.each do |thd| puts(thd.backtrace.join) end
Signals & Stacktraces
Reproduce, Examine, Experiment
ReproduceExamine
Experiment
Reproduce It
• A lot of effort but essential
• The easier you can reproduce it
• The easier you can debug it
Examine The Code
• Multi-threaded principles
• Anything obvious?
• You still need to experiment and prove your case
Experiment
• Start running experiments
• See if your expectations match reality
• Keep a written log
Reproduce
Reproduce
Reproduce
• Start simple...
• Start 100 clients doing a list operation
while true; do date echo "Starting 100 requests..." for i in {1..100}; do <rest-client-list-operation> & done wait done
Reproduce
Reproduce
• On my laptop → No lockup
• On a real server → No lockup
• Need to try both
• More concurrency
• A dependency verification thread
• Run this every second
• Test again for 30 minutes → No lockup
Reproduce
• We deploy to an OpenStack cluster
• What if we do nothing and return early
• Run the test again for 30 minutes
→ No lockup
Reproduce
Reproduce
Timeout.timeout(job.timeout_seconds) do run_job(job) end
• Set job.timeout_seconds to 1
→ Deadlock!
Reproduce
.../monitor.rb:185:in `lock'
.../monitor.rb:185:in `mon_enter'
.../monitor.rb:210:in `mon_synchronize'.../logger.rb:559:in `write'
Examine
Examine
Examine def write(message) begin @mutex.synchronize do # write-the-log-line end rescue Exception # log-a-warning end end
Anything wrongwith this?
Examine
• mon_synchronize• mon_enter• mon_exit• mon_check_owner
Examine def mon_synchronize mon_enter begin yield ensure mon_exit end end Looks OK
Examine
def mon_enter if @mon_owner != Thread.current @mon_mutex.lock @mon_owner = Thread.current end @mon_count += 1 end
@mon_mutex = Mutex.new@mon_owner = nil@mon_count = 0
Grrrrr...
Rant
• I’ve debugged locks involving reentrant mutexes more times than I can remember
• If you ever feel like using a reentrant mutex, please I beg you, don’t do it
• There’s almost always a way to structure your code so that you can use a regular mutex
Examine
def mon_enter if @mon_owner != Thread.current @mon_mutex.lock @mon_owner = Thread.current end @mon_count += 1 end Anything wrong
with this?
Examine
def mon_exit mon_check_owner @mon_count -=1 if @mon_count == 0 @mon_owner = nil @mon_mutex.unlock end end Looks OK
if @mon_owner != Thread.current raise...
• Take a look at the first line in mon_enter: if @mon_owner != Thread.current
• Modified by multiple threads
• Read by other threads without being locked
• Read access needs a mutex too
Examine
Aside: Double Checked Locking
• Many people have gotten this wrong
• Doug Schmidt & Co, ACE C++
• Pattern-Oriented Software Architecture(Volume 2, April 2001)
• Popularised a pattern that was completely broken: Double Checked Locking
• A variable shared between multiple threads...
• ...Modified by one or more threads
• You need to use a mutex around the modification (of course)
• But you also need to a mutex around any READ access to that variable
Aside: Takeaway
GIL?
Aside: Takeaway
This is because of…
• Instruction pipelining
• Multiple levels of chip caches
• Out of order memory references
• The memory model of the platform
• The memory model of the language
Examine
def mon_enter if @mon_owner != Thread.current @mon_mutex.lock @mon_owner = Thread.current end @mon_count += 1end
def mon_exit mon_check_owner @mon_count -=1 if @mon_count == 0 @mon_owner = nil @mon_mutex.unlock endend
Examine
So that's two concerning things so far:
1. Logger's rescue of Exception
2. Read access to @mon_owner outside of any mutex
Experiment
Experiment
Experiment 1
The Change:• Puts all access to @mon_owner and @mon_count (& the Thread ID)
The Result:• Deadlock• I saw @mon_count changing from 0 to 2
The Change:• Keep track of @mon_count and @mon_owner
in a list in memory (& the Thread ID)• Puts the list when we dump the stacktraces
Experiment 2
The Result: • Deadlock• @mon_count changing from 0 to 2 (same)
The Change:• @mon_owner and @mon_count don’t really
need to be shared among threads• Use thread local variables instead
Experiment 3
The Result:• Deadlock• @mon_count jumps from 0 to 2 occasionally
The Change:• When a thread acquires the monitor mutex @mon_count should always be zero
• So check to see if it’s ever non-zero
Experiment 4
Experiment 4 def mon_enter if @mon_owner != Thread.current @mon_mutex.lock @mon_owner = Thread.current if @mon_count != 0 puts '=========XXXXXXXXXX=======' end end @mon_count += 1 end
Experiment 4
The Result:• Test again → No Deadlock → No log line
• OK that's really odd…
• But you can't rely on a negative, so then I removed those lines and ran again
• Now it locks
Experiment 4
The Result:• Add back the lines → Doesn't lock
• Remove the lines → Deadlocks quickly
• Hmm, ok that's definitely odd, feels like a memory visibility issue
The Change:• Download and build a debug version of MRI
• Looking in thread.c I found: rb_threadptr_unlock_all_locking_mutexes() with the following warning commented out:
Experiment 5
/* rb_warn("mutex #<%p> remains to be locked by terminated thread", mutexes); */
The Result:
• Deadlocks
• Saw threads exiting with that warning about locked mutexes
Experiment 5
Experiment 6
The Change:• Examining mon_enter and mon_exit we can
see that when the lock is taken @mon_count should always be zero
• But we saw @mon_count jumping from 0 to 2 so let’s try putting in @mon_count = 0 explicitly
Experiment 6
def mon_enter if @mon_owner != Thread.current @mon_mutex.lock @mon_owner = Thread.current @mon_count = 0 end @mon_count += 1 end
Experiment 6
The Result:• Doesn't lock, left it running for hours
• Take out the @mon_count = 0 and it locks
• But remember checking if @mon_count != 0 had the same effect
Experiment 6
• So it seems that adding@mon_count = 0
"fixes" the problem
• I still want to understand the cause
• I’d like a reproducible test case that doesn’t rely on our service
Experiment 7
• With new threads coming and going, some exiting normally, some timing out, all emitting log messages
• What about if we try to log heavily within a timeout block and time it out in a bunch of threads
def run count = 1 begin Timeout.timeout(1) do loop do @logger.error("#{Thread.current}: Loop #{count}") count += 1 end end rescue Exception @logger.error("#{Thread.current}: Exception #{count}") endend
Experiment 7
• So with this code I get:
... `join': No live threads left. Deadlock? (fatal)
• Happens every time after a few seconds
Experiment 7
Experiment 7
• Since it says all threads are dead, what happens if there is another thread just sitting there doing nothing?
• Add
Thread.new { loop { sleep 1 } }
Experiment 7
• Run the code again → Deadlock
• All threads stuck in the same location as before:
.../monitor.rb:185:in `lock'
.../monitor.rb:185:in `mon_enter'
.../monitor.rb:210:in `mon_synchronize'
.../logger.rb:559:in `write'
• So now I have a simple test case that reproduces the issue every time
• I can also confirm that adding @mon_count = 0 into mon_enter "fixes" the problem
Experiment 7
Examine Again
• At some point during all of this I showed this to a colleague who suggested I look for recent changes in this code within the Ruby repo
• We checked the Ruby git repo...
Examine Again
commit 7be5169804ee0cfe1991903fa10c31f8bd6525bdAuthor: shugo <shugo@b2dd03c8-39d4-4d8f-98ff-823fe69b080e>Date: Mon May 18 04:56:22 2015 +0000
* lib/monitor.rb (mon_try_enter, mon_enter): should reset @mon_count just in case the previous owner thread dies without mon_exit.[fix GH-874] Patch by @chrisberkhout
Root Cause Analysis
Root CauseAnalysis
Root Cause Analysis
• With a little more thought I realised what the root cause of this problem is…
• It’s the Timeout module and how corrupts state in the monitor object
Root Cause Analysis class Monitor def synchronize mon_enter begin yield ensure mon_exit end end end
Timeout.timeout(seconds) do logger.write end class Logger def write @mon.synchronize do write-log end end end
1
24
5
3
Root Cause Analysis
Thread 1 (T1)• Timeout.timeout(seconds)
• Start a new thread T2• logger.write
• mon.synchronize– write-the-log
• Kill T2
Thread 2 (T2)
• Keeps a reference to T1
• sleep(seconds)
• Raise a Timeout exception against T1
1
2
class Monitor def synchronize mon_enter begin yield ensure mon_exit end end end
Root Cause Analysis
• What about right here?• mon_enter is invoked• mon_exit is not
def mon_enter if @mon_owner != Thread.current @mon_mutex.lock @mon_owner = Thread.current end @mon_count += 1 end
Root Cause Analysis
Root Cause Analysis
def mon_exit mon_check_owner @mon_count -=1 if @mon_count == 0 @mon_owner = nil @mon_mutex.unlock end end
Finally!It all makes sense
Demo Time!
https://github.com/lad/ruby_concurrency• thread_deadlock.rb• thread_starve.rb
Show Me The Code
Results From Different Ruby VMs
MRI 1.8.7 MRI 1.9.3 MRI 2.1.5 MRI (HEAD)
Deadlock Yes Yes Yes No (*)
Starvation YesCan’t say.
Always deadlocks
Yes Yes
Mid Jan, 2016 (2.3+)
Results From Different Ruby VMs
JRuby 1.6.8
JRuby 1.7.11
JRuby 1.7.19
JRuby 9.0.0.0.pre1
Deadlock YesNo Deadlock or Starvation.
Though only because the Timeout exception is not raisedStarvation Yes
Results From Different Ruby VMs
RBX 2.4.1 RBX 2.5.2
Deadlock VM Crashes Yes.Though mostly thread starvation
Starvation VM Crashes Yes
My Assertion
It is fundamentally unsafe to interrupt a running thread in the general case
Side Effects
Other Languages
Lessons FromOther Languages
Thread Cancellation in C
• The C pthread API offers additional features
• Has the concept of thread cancellation:
• Enable / Disable thread cancellation requests
• User defined, per thread cleanup handlers
Thread Cancellation in JavaWhy is Thread.stop deprecated?• Because it is inherently unsafe
• Stopping a thread causes it to unlock all the monitors that it has locked
• If any of the objects previously protected by these monitors were in an inconsistent state, other threads may now view these objects in an inconsistent state
http://docs.oracle.com/javase/1.5.0/docs/guide/misc/threadPrimitiveDeprecation.html
“As of JDK8, Thread.stop is really gone. It is the first deprecated method to have
actually been de-implemented. It now just throws UnsupportedOperationException”
Doug Lea, Java Concurrency Guru
http://cs.oswego.edu/pipermail/concurrency-interest/2013-December/012028.html
Java Thread Cancellation
Ruby rant: Timeout::ErrorJan 2008
(http://goo.gl/PLxR76)
Ruby's Thread#raise, Thread#kill, timeout.rb, and net/protocol.rb libraries are broken
February 2008
(http://goo.gl/DI8GMX)
Ruby timeouts are dangerousMarch 2013
(https://goo.gl/3EoTM6)Ruby’s Most Dangerous API
May 2015
(http://goo.gl/2RkFbn)Why Ruby’s Timeout is dangerous (and Thread.raise is terrifying)
Nov 2015
(http://goo.gl/xLvuWG)
Reliable Ruby timeouts for M.R.I. 1.8https://github.com/ph7/system-timer
Fixing Ruby's standard library Timeouthttps://github.com/jjb/sane_timeout
A safer alternative to Ruby's Timeout that uses unix processes instead of threadshttps://github.com/david-mccullars/safe_timeout
Better timeout management for rubyhttps://github.com/ryanking/deadline
What Does The Community Say?
Threading Quick Links
John Ousterhout:http://web.stanford.edu/~ouster/cgi-bin/papers/threads.pdf
Ed Lee:http://www.eecs.berkeley.edu/Pubs/TechRpts/2006/EECS-2006-1.pdf
• Best practices, smart people, code locked up after running successfully with minimal changes for four years
Takeaways
• Try to avoid writing multi-threaded code
• Try to avoid reentrant mutexes
• Always use a mutex for read access to shared state
• Don’t use the Timeout module. It’s a broken concept