Is Ruby LoggerThread(Process)-

safe?

RubyConf 2013 @MiamiSEO Naotoshi (@sonots)

2013/11/09

Naotoshi SEOlike Now-toshi Say-oh

@sonots (twitter, github)Dev Engineer for OperationsBecame as a commiter of Fluentd

•realtime pipelining of log streams

•plugins written in ruby

Log Management Tool•syslog-ng / rsyslog•splunk•scribed•flumeNG

•logstash

Only collection

No Plugin

Java Plugin

Ruby Plugin (Not Gem)

Not Free

•fluentd Ruby Plugin (Gem Support)

Try thefluentd!It’s very nice tool, check it out!

http://fluentd.org/

Today’sTalk

Logger

We needed aLogger whichworks safely

in multiprocessfor Fluentd

Examined Ruby Logger

• 1) Will logs not be mixtured in multi-threads?

• 2) Will logs not be mixtured in multi-processes?

• 3) Does log rotation work safely in multi-threads?

• 4) Does log rotation work safely in multi-processes?

1) Will logs not be mixtured in multi-threads?

require 'logger'require 'parallel'

logger = Logger.new("/tmp/test.log")Parallel.map(['a', 'b'], :in_threads => 2) do |letter| 10000.times do logger.info letter * 5000 endend

Code

Result

$ egrep -e 'ab' -e 'ba' /tmp/test.log[empty]

OK

WHY?

def write(message) begin @mutex.synchronize do ... @dev.write(message) ... end

logger.rb#L559

Ruby’s Logger is taking mutex to be thread-safe.

2) Will logs not be mixtured in multi-processes?

require 'logger'require 'parallel'

logger = Logger.new("/tmp/test.log")Parallel.map(['a', 'b'], :in_processes => 2) do |letter| 10000.times do logger.info letter * 5000 endend

Code

Result

$ egrep -e 'ab' -e 'ba' /tmp/test.log[empty]

OK

Really? Why?

But, Linux system call write(2) itself is atomic as long as writing to a local file.

Atomic/non-atomic: A write is atomic if the whole amount written in oneoperation is not interleaved with data from any other process. This isuseful when there are multiple writers sending data to a single reader. Applications need to know how large a write request can be expected tobe performed atomically. This maximum is called {PIPE_BUF}. This volumeof IEEE Std 1003.1-2001 does not say whether write requests for morethan {PIPE_BUF} bytes are atomic, but requires that writes of {PIPE_BUF} orfewer bytes shall be atomic.

Mutex does not work to exclusively lock in multi-processes environment.

OK

3) Does log rotation work safely multi-threads?

require 'logger'require 'parallel'

logger = Logger.new("/tmp/test.log", 3, 1024 * 10)Parallel.map(['a', 'b'], :in_threads => 2) do |letter| 10000.times do logger.info letter * 5000 endend

Code

Result

$ ls -l /tmp/test*-rw-r--r-- 1 sonots sonots 10806 9月 29 23:03 /tmp/test.log-rw-r--r-- 1 sonots sonots 10806 9月 29 23:03 /tmp/test.log.0-rw-r--r-- 1 sonots sonots 10806 9月 29 23:03 /tmp/test.log.1

No Error, OK

4) Does log rotation work safely in multi-processes?

require 'logger'require 'parallel'

logger = Logger.new("/tmp/test.log", 3, 1024 * 10)Parallel.map(['a', 'b'], :in_processes => 2) do |letter| 10000.times do logger.info letter * 5000 endend

Code

Result

log writing failed. closed streamlog shifting failed. closed streamlog writing failed. closed streamlog shifting failed. closed stream...

Oops, Bad!

WHY?

def write(message) begin @mutex.synchronize do ... check_shift_log ... end

logger.rb#L559

Ruby’s Logger is taking mutex to be thread-safe, but was not treating anything for multi-processes.

We(@frsyuki and me) Fixed

• The approach is to use flock(2) without creating another *.lock file

Pull requested to ruby

My first contribution to ruby

Merged!!

Thanks to @naruse for his review and advice!!

Will be releasedwith 2.1.0!!

on 12/25

•Ruby Logger was not safe in multi-processes, but it is safe now!!!

•Try ruby-trunk orhttps://github.com/sonots/

process_safe_logger

CONCLUSION

Try thefluentd!It’s very nice tool, check it out!

http://fluentd.org/