it messaging services - cern it-cm-mm 28-01... · 28 jan 2016 it messaging services 11 kafka uses...
TRANSCRIPT
IT Messaging Services
Lionel Cons, Luca Magnoni – CERN IT/CM
28 Jan 2016 IT Messaging Services 1
Messaging
28 Jan 2016 IT Messaging Services 2
10,000 Feet Overview
28 Jan 2016 IT Messaging Services 3
Messaging Concepts
JMS defines some key concepts:
• message as body, header and properties
☞ think data and meta-data
• topic aka publish/subscribe model
☞ similar to television broadcasting
• queue aka point-to-point model
☞ similar to electronic mail
Other models exist such as virtual queues.
28 Jan 2016 IT Messaging Services 4
Messaging Protocols
There are several messaging protocols:
• AMQP (Advanced Message Queuing Protocol)
• MQTT (Message Queuing Telemetry Transport)
• OpenWire
• STOMP (Simple Text Oriented Messaging Protocol)
• XMPP (Extensible Messaging and Presence Protocol)
• ØMQ aka ZeroMQ
28 Jan 2016 IT Messaging Services 5
Examples with STOMP
28 Jan 2016 IT Messaging Services 6
→ CONNECT
login: ...
passcode: ...
← CONNECTED
→ SEND
destination: ...
some-header: ...
... body ...
→ CONNECT
...
← CONNECTED
→ SUBSCRIBE
destination: ...
← MESSAGE
...
← MESSAGE
...
What about Flume?
28 Jan 2016 IT Messaging Services 7
SinkChannelSource
• Source can be jms (inc. with STOMP provider)
• Flume can be a message consumer
• Flume is not a message broker
• no on-demand subscriptions
What about Logstash?
28 Jan 2016 IT Messaging Services 8
OutputInput
• Input can be stomp (or rabbitmq, xmpp, zeromq)
• Output can be stomp too (or ...)
• Logstash can be a message consumer and/or a message producer
• Logstash is not a message broker• no on-demand subscriptions
Filter
What about Kafka?
28 Jan 2016 IT Messaging Services 9
“Apache Kafka is publish-subscribe messaging
rethought as a distributed commit log.”
Key Kafka concepts:
• commit log, can be partitioned and replicated
• consumer group, allows JMS semantics:
• topic: all consumers use different groups
• queue: all consumers use the same group
• retention, i.e. when messages get deleted
Kafka Retention
28 Jan 2016 IT Messaging Services 10
Stream 1
Stream 2
available messages
Kafka Protocol
28 Jan 2016 IT Messaging Services 11
Kafka uses its own protocol that puts more
burden on the consumers that need to keep
track of where they are in the commit logs.
OTOH, they can read again older messages.
Kafka can be seen as a kind of message
broker with non-standard protocol and features.
Kafka’s maturity needs to be assessed. The
new consumer API is marked “beta quality”.
Messaging Software in 2012
28 Jan 2016 IT Messaging Services 12
Broker Qpid MRG HornetQ ActiveMQ Apollo RabbitMQ
Language Java C++ Java Java Scala Erlang
Main Protocols
AMQP AMQP proprietarySTOMP
OpenWireSTOMPAMQPMQTT
OpenWireSTOMPAMQPMQTT
AMQPSTOMP(MQTT)
Owner (*) Red Hat FuseSource(Progress)
VMware
September 7, 2012 : Red Hat completed its acquisition of FuseSource
Red Hat
IT Messaging Services
28 Jan 2016 IT Messaging Services 13
Hardware Layer
We currently use physical machines for the
brokers (and VMs for the monitoring nodes).
New types of VMs should be tried:
• high-end VMs with fast Ceph volumes
• high-end VMs with dedicated local SSD drives
To be tested and benchmarked…
28 Jan 2016 IT Messaging Services 14
Operating System Layer
We currently use Java 7 on SL 6.
We need to prepare for Java 8 and CentOS 7.
History taught us that there is no such thing as
a painless upgrade.
28 Jan 2016 IT Messaging Services 15
Software Layer
We currently use Red Hat’s A-MQ which is
basically Apache ActiveMQ patched, tested
and supported by Red Hat.
We currently use A-MQ 6.2.1 which is based
on ActiveMQ 5.11.
Since our clients only use STOMP, we could
migrate to other software such as RabbitMQ
or Apache ActiveMQ Artemis.
28 Jan 2016 IT Messaging Services 16
Messaging Services Growth
One year ago:
• 5 (+ 5) clusters: Agile Infrastructure, ATLAS,
Dashboard, IT/DSS and MIG
• 23 brokers on 16 machines
Today:
• 8 (+ 7) clusters with the addition of: IT/DB,
MCollective and Network Monitoring (perfSONAR)
• 32 brokers on 16 machines
28 Jan 2016 IT Messaging Services 17
Messaging Services Numbers
28 Jan 2016 IT Messaging Services 18
Messaging Services Users
Many different use cases. For instance:
• AI: events, monitoring notifications
• ATLAS: Rucio, event indexes
• Dashboard: FTS, Transfer, SAM, XRootD
• DSS: CASTOR/EOS logs
• MCollective
• MIG: monitoring
• Network Monitoring: perfSONAR
28 Jan 2016 IT Messaging Services 19
AI Monitoring Use Case
Very low volume (only notifications) but very
diverse clients: PES, River, SNow, Spectrum...
Many clients: any machine using AI monitoring.
Enabling X.509 authentication:
• 50k+ DNs authorized
• pushed ActiveMQ’s limits thanks to Red Hat
• we could go even further by submitting a new,
more flexible X.509 authentication module
28 Jan 2016 IT Messaging Services 20
MCollective Use Case
MCollective needs a “network of brokers”.
Requirements are challenging:
• 30k concurrent connections (vs. 3k in total before)
• 300k subscriptions (vs. 1k in total before)
• working with Red Hat for the design and tuning
First prototype delivered and is being tested.
Several MCollective shortcomings found and
reported.
28 Jan 2016 IT Messaging Services 21
Messaging Team
Currently: ~1 FTE across two persons.
Wide range of activities:
• operations and evolutions (e.g. CC7)
• support (with the help of Red Hat for A-MQ)
• consultancy
• testing and validation
• software maintenance and development
Contact: http://cern.ch/mig – [email protected]
28 Jan 2016 IT Messaging Services 22
Messaging Software
Diverse software:
• libraries: STOMP for Perl, messaging for Python...
• building blocks: directory queue, stompclt...
• management: MBSD, MBCG, Puppet hostgroup...
• monitoring: MBM, Metis...
• testing: MBTF...
In total:
• ~70k SLOC mainly in Perl, Python, Java and C
• 14 packages in EPEL + ~18 in CERN’s Koji
28 Jan 2016 IT Messaging Services 23
Metis
We are very happy with our Esper based
solution named Metis.
For more information, see our “Advanced
Monitoring With Complex Stream Processing”
presentation at the IT Technical Forum.
Most of Metis is generic and should be reused
in other areas where advanced monitoring is
needed.
28 Jan 2016 IT Messaging Services 24
Possible Next Steps
1. Investigate Kafka-based messaging services
2. Improve and extend our building blocks
3. Test high-end VMs with better I/O:
• fast Ceph volumes
• local SSD drives
4. Extend Metis beyond messaging monitoring
5. Evaluate ActiveMQ Artemis
6. Prepare for the upcoming system upgrade
• CentOS 7 and Java 8
28 Jan 2016 IT Messaging Services 25