ZeroMQAnatomy & JeroMQ

ZeroMQ

MQ 가 아니무니다..

TCP on Steroids

ZMQ Socket

ROUTER DEALER XPUB XSUB PUSH

PULLREP REQ PUB SUB

ZeroMQ AnatomyR w

io_thread_t

ZMQ Socket

Mailbox

Signalerypipe_t

<cmd_t>

Session

Socket

Session

Socket

poller

rfd

fd

ypipe_t <msg_t> inpipe

ypipe_t <msg_t> outpipe

ZMQSocket

ZMQSocket

ZMQSocket

ZMQSocket

SessionSessionSession

Context

Mailbox

r

Msgunion { ... struct { unsigned char data [29]; unsigned char size; unsigned char type; unsigned char flags; } vsm; struct { content_t *content; unsigned char unused [29 + 1 - sizeof (content_t*)]; unsigned char type; unsigned char flags; } lmsg;

DATA S T F

T F

S DATA

vsm

lmsg

writer thread

reader thread

YQueue

yqueue_t <msg_t, 256> yqueue_t <command_t, 16>

struct chunk_t { T values [N]; chunk_t *prev; chunk_t *next; };

atomic_ptr_t <chunk_t> spare_chunk

...begin

back

• Only single thread can read from begin_chunk• Only single thread can write to yqueue_t <msg_t> to back chunk• Only single thread can write to yqueue_t <command_t> at a time

YPipe

atomic_ptr_t <T> c

Reader

Writer

T* r

T* wT* f

bool flush () { if ( c.cas ( w, f ) != w ) { c.set ( f ) ; w = f ; return false ; // reader is sleeping } w = f ; return true ; }

bool check_read () { if ( queue.front () != r ) return true ;

r = c.cas ( queue.front () , NULL ) ; if ( queue.front () == r ) return false ; // nothing to read

return true; }

read() => queue<T>.pop

write() => queue<T>.push

r w f

Signalersocketpair ( AF_UNIX, SOCK_STREAM, 0, sv[2] )

int w = sv[0]; // writer fd

int r = sv[1]; // reader fd, Used at IOThread.poll

Reader

recv () { :: recv ( r , dummy[1] );}wait ( timeout ) { poll ( r , timeout );}

Writersend () { :: send ( w, dummy[1] );}

Mailboxypipe_t <command_t, 16> cpipe

signaler_t signaler

Reader

Writer

recv (timeout) {signaler.wait (timeout)

return cpipe.read ()}

send (cmd) {lock ()cpipe.write (cmd)cpipe.flush ()unlock ()

signaler.send ()}

ZMQ SocketZMQ Socket

mailbox_t

signaler_typipe_t

<cmd_t>

Session

Socket

Session

Socketr

fd

fd

ypipe_t <msg_t> inpipe

ypipe_t <msg_t> outpipe

• Session is create when connect / accept• Worker thread owns ZMQ socket• IOThread owns Sessions

• recv msg from socket and en-queue inpipe• recv msg from outpipe and send to socket

• zsocket.send en-queue to outpipe• zsocket.recv de-queue from inpipe if any

main/worker

IOThreadio_thread

ZMQ Socket

mailbox

Session

Socket

poller

ZMQSocket

ZMQSocket

ZMQSocket

ZMQSocket

Session

mailbox

zsocket.write(msg)outpipe.write(msg)activate_read

session.read_activatedengine.activate_out engine.out_event

SessionSessionSession

Stream Engine

engine.in_eventoutpipe.write(msg)

activate_readzsocket.read_activatedmsg = inpipe.read

in_event () { // read from socket size = 8192 // *** buffer = decoder . get_buffer (size) ::recv (buffer, size) decoder . process_buffer(buffer) session . flush() }out_event () { // write to socket size = 8192 // *** buffer = encoder . get_data (size) ::send (buffer, size)}

msg = parse (buffer) session.write (msg)

Stream Engine

msg = session.read ()buffer.fill (msg)

Summary

inline

lock-free

well-defined threading model

less system calls

• malloc / free• bulk socket recv / send• balance copy overhead

JeroMQ - jeromq.org

Hand-made

Compatible with ZeroMQ-3

tcp:// inproc://

ipc:// pgm://

same dev experience

95%~ performance

JeroMQ - diffJava NIO

SelectableChannel(Direct) ByteBuffer

SignalerPipe.SourceChannel / Pipe.SinkChannelMixed with counter

YQueue - circular linked listPlain Socket ProxyPersistence - Kafka alike

MappedByteBufferZeroCopy

Future

Zero Persistence Queue

http://github.com/miniway/zper

Faster than Kafka

50% Less code

Configurable topology

Java 7 - AsynchronousChannel

Conclusion

Why reinvent the wheel

“The only limitation is your imagination and sobriety”

Q/A