JMS – HIGH LEVEL MESSAGING

Felix EhmCERN BE-CO

Content Introduction

JMS in the Controls System

Deployment and Operation

Conclusion

Introduction

Introduction to JMS – The API Java Messaging Service

Messaging API defined in 2002Decouples Source and DestinationTopic / Queue Concepts for named Destinations

Introduction to JMS – A Broker Technology originated from the financial sector

“Few” sources and large set of readers

A Broker is the central entityAccepts messages from 1..* publisher and

re-distributes to 0..* subscribersProvides Quality of Service for Messages

○ Persistency, Transactions, Buffering, Expiration Policies, Slow Client Handling

Design to distribute data reliably and scalableLoad balancing / fail-over / embedded

deployment scenarios possible

Broker

Introduction to JMS - A Broker Many broker implementations available

ProprietaryIBM WebsphereMQ, SonicMQ, TIBCO, RedHat MRG, …

OpenSourceApache ActiveMQ, Apache Apollo, RabbitMQ, HornetMQ, …

Broker may be implemented in another language than Java (e.g. Qpid & RabbitMQ)

Communication protocol is non-standardBrokers of different vendors are not interoperableAMQP & STOMP address this issue

Introduction to JMS - Request/Reply “Direct communication” via Broker using

Request – Reply mechanism

Due to intermediate Broker two hops are required

Requestor ReplierBroker

RequestMSG

ReplyMSG

Request Queue

Temporary Reply Queue

creates

sends

sends

Introduction to JMS - Selectors Filter messages from one Destination using

Messages SelectorsEach message can be enriched

with AttributesSubscriber receives filtered

messages via defined selector

Broker

Subscriber

selector:BUILDING ≠ A

HEADERSCYCLE=ATIME=9845

BODY

JMS

Message

BUILDING=A

BUILDING=B

BUILDING=C

Topic/Queue

Introduction to JMS - Persistency JMS defines persistent and non-persistent

messages Broker stores message to

disk before acknowledging to sender

Avoids loss of data when broker crashes Drawbacks

Higher disk loadSlower message processing

Sender1. send

3. ack 2. store

JMS in the Controls System

CERN’s Accelerator Complex

CERN’s Accelerator Complex

JMS in the Controls System Purpose

Reliable transport of data between (Java) ProcessesHigh level controls applications + middle tier servers

○ Alarm System○ Data Rendering Services○ (Control) GUIs○ Logging Services○ Beam Security System(s)○ Access Systems, ….

Today vital part of Beam Operations Product choice:

No JMS - No Beam !

JMS in the Controls System - History

HistorySonicMQ since 2001 for CERN’s Alarm

and CO Monitoring systemOpenSource ActiveMQ since 2005

○ Middle tier servers update GUIs○ Middle tier to Middle tier

Phased out in 2013

JMS for Accelerator Controls Early days only Java clients via JAPC

japc-ext-remote contains JMS communication

Today multi-language clientsJAPC clients remainJava using ActiveMQ client libraries directlyC++/Python using STOMP

Deployment and Operation

29 production brokers Single or broker cluster

Large spectrum of usage patternsfrom few topics to 40K topicsfrom few subscribers to >23 K subscribersfrom couple msg / minute to 10K msg / secfrom few bytes/msg to 3 Mbyte/msg

Deployment and Operation

Deployment and OperationSome numbers : 300 Applications 4’400 Connections 40’000 Subscriptions 85’000 Topics 8 Million msg/h IN,

3.5M msg/h OUT

Archived Uptime in 2012: 99.998%

Not all data which is produced is also

consumed

Deployment and Operation Distribute Load

Interconnect two BrokersConsumers/Producer can

choose any BrokerSharing of work

Smooth Upgrades possibleClient library version != Broker version OK

DrawbacksConsumer might not get all messages in case of

network errorRouting overhead

The STOMProtocol

The STOMProtocol Simple Text Oriented Message Protocol Defines messaging API + wire format Many open source clients libraries available

Python, C/C++, Java, Perl, Ruby, PHP, Erlang, … Targets the messaging interoperability among

language, platforms and brokers Supported by many brokers Very easy to use

Example to send STOMP in Pythonmysocket = socket.socket(socket.AF_INET,socket.SOCK_STREAM)mysocket.connect( ("jms-co-dev", 61680) )

text = '''SEND\n\destination:/queue/CERN.DEPLOYMENT\n\HOST:%s\n\INSTALLPATH:%s\n\PRODUCT:%s\n\TIME:%s\n\persistent:false\n\n\x00''‘

mysocket.sendall(text)mysocket.sendall("DISCONNECT\x00")mysocket.close()

The AMQProtocol

The AMQProtocol Advanced Message Queue Protocol Defines messaging API + wire format Next evolution after JMS

Consortium of RedHat, Credit Suisse, Microsoft, JPMorgen, Barclays, VMWare, Cisco, …

Released as Version 1.0More and more brokers implement it

Direction for the future ?

Conclusion JMS is vital part of Accelerator Controls System

Performs very well for publish - subscribe scenarios

ActiveMQ was a good choiceRunning reliably Low maintenance and flexible deploymentsProvides many metrics for diagnostic and monitoring

Multi-language applications can use service

Central (redundant) service

Result Example Messages received/sec over time

time

AVG

Mes

sage

s / s

ec

Result Example Histogram of delayed message distribution

Delay time [msec]

# M

essa

ges

Result Example Number of “late” (>300ms) messages for test

duration

time

Del

ayed

Mes

sage

s

Message Delay Summary Measuring delayed Message

~ 99.32% of messages delivered < 300msRest < 1 sec

Doubling numbers of messages (~9500 msg/sec)100% between 5 and 10 secondsFurther studies for improvement needed

40% less CPU load thanks to optimization

Result Example CPU load

time

% C

PU lo

ad

Consumer connects

Consumer disconnects

Broker Memory Consumption Initial Memory is proportional to

ConnectionsSubscriptions

0 5000 10000 20000 30000 400000

50

100

150

200

250

300

350

400

Heap Memory Usage Live Threads

0 10000 20000 40000 60000 800000

100

200

300

400

500

600

700

800

Heap Memory Usage Live Threads

1000 Subscription / Connection 500 Subscriptions / Connection