reliable system integration and scaling with wso2 message broker
TRANSCRIPT
Reliable System Integration and Scaling with WSO2 Message Broker
Ramith JayasingheSenior Technical Lead
WSO2
Messaging Models/Concepts• RPC style
- Request/Response, Synchronous, Non-persistent - e.g. Web Services, Thrift, REST
• Point to Point (Queues)
- A message is delivered only once to a single consumer
• Publish/Subscribe (Topics)
- Broadcast a message to all the subscribers
Messaging Models/Concepts• Durable Subscriptions
- Same as topic subscriptions- Subscriber offline - Messages get queued- Subscriber back online - Messages get delivered- Un-subscription needed to remove queuing
Messaging Models/Concepts• Transactional ( send/consume -> rollback, reject, commit)• Guaranteed Delivery, QOS ( MQTT, AMQP), Persistence• Dead letter channel• Hierarchical topics
- Games.FootBall.UK, Games.FootBall.USA, - Games.Cricket.SriLanka, Games.Football.UK
Distributed Queues
• Provides strict or best effort support for inorder delivery• There are no guarantee about the global order seen
across subscribers
WSO2 Message Broker
• Design Objectives:- Higher performance (by not routing messages over the
network multiple hops)- High availability and fault tolerance.- Leaner deployment layout ( and clustering story)
• v 3.0.0 - Significant architectural changes.• v 3.1.0 - Latest version
- Minor improvements - PostgreSQL support
WSO2 MB v 3.x.x Design Objectives
• Avoid message routing over the network multiple times.• Use storage to share messages between nodes
Allow database to scale as separate layer.
• Use distributed coordination to control the behavior (i.e. Hazelcast)
WSO2 MB v 3.x.x Design Objectives
• Content and metadata are saved separately into a database shared between
all cluster nodes.
• All nodes knows about all subscribers ( - via Hazelcast coordination)
• Each node will logically divide messages into group (known as Slots).
WSO2 Message Broker Design
More information about disruptor: https://lmax-exchange.github.io/disruptor/
WSO2 Message Broker Design - Inbound
All events coming to broker are added to ‘inbound disruptor’ such as:
● Messages● Message acknowledgements ● Create/Delete queues topics● Subscribe/Unsubscribe● UI Events
Inbound disruptor:● Sequence inbound events ( - disruptor
behaviour)● Generate internal Message IDs ( length =
java - long)● persist/delete messages● Execute other control functions.
Note: #1. Internal message Ids are time based. Therefore, MB cluster needs to be time synced.
#2. Inbound path can be tuned for best performance ( broker.xml).
WSO2 Message Broker Design - Outbound
All message going out of broker are added to ‘outbound disruptor”
Slot Delivery Worker pool:1) Connects to coordinator node ( - get
messages to deliver)2) Fetches metadata of messages and add
to outbound disruptor
Outbound disruptor:1) Fetch message payloads (in parallel)2) Forward fetched messages to transport.
Note: Slot delivery thread pool and outbound disruptors can be tuned for best performance (broker.xml).
WSO2 Message Broker Design - Distributed Queues
• Content and metadata are saved separately into a database shared between
all cluster nodes.
• All nodes knows about all subscribers ( - via Hazelcast coordination)
• Each node will logically divide messages into group (known as Slots).
WSO2 Message Broker Design - Distributed Queues
• Cluster elects a leader ( known as Coordinator); Usual the first node to start in cluster.
• Coordinator keeps track of all the slots available (via Hazelcast)
• Nodes with subscriptions acquire a slot (from Coordinator) and deliver them to
subscribers.
WSO2 Message Broker v 3.1.0 - Features
● Message Queuing Telemetry Transport (MQTT)
● Shared Subscriptions
● RDBMS based message store (MSSQL, Oracle, H2, MySQL, PostgreSQL)
● Real time statistics/graphs (on each server) via carbon-matrices
● Improved internals
MQTT support in WSO2 MB• Lightweight, pub-sub protocol designed for IOT use-cases.
• Hierarchical topic structures and wildcard subscriptions
• All quality of service levels and retained messages
Shared Durable Subscriptions
Extension to JMS 1.1, until JMS 2.0 support is released. Purpose: Ability to load balance.
broker.xml: <allowSharedTopicSubscriptions>true</allowSharedTopicSubscriptions>
Back pressure / Flow Control - Inbound• Every system has its limits.
• Safeguards server from sudden bursts of message loads
• Flow control based on:
– Total inbound events
– Inbound events per connection/channel
– Global error conditions (e.g. offline databases)
<flowControl> <!-- This is the global buffer limits which enable/disable the flow control globally --> <global> <lowLimit>800</lowLimit> <highLimit>8000</highLimit> </global>
<!-- This is the channel specific buffer limits which enable/disable the flow control locally.
--> <bufferBased> <lowLimit>100</lowLimit> <highLimit>1000</highLimit> </bufferBased> </flowControl>
General Improvements• Real time statistics/graphs
– Quickly figure out bottlenecks in a production deployments.
– Database read/write latencies– Inbound/Outbound event rates– Number of Connections/Channels etc.
• Performance Improvements – Disruptor based thread model– Improved internals ( topic matching, slot based delivery,
caching)
Improvements in MB v 3.1.0• Support for PostgreSQL as the message store.• LZ4 Compression for storage
– Dramatic performance improvement large/text oriented message payloads
• Forcefully disconnect Consumers– If a subscriber misbehaves, admin can disconnect the client
forcefully.
Roadmap• Support for AMQP 1.0 and JMS 2.0
• Move to Carbon 5
• Improved GUI
• Improvements to Storage Scheme
• Performance improvements.
• Websockets + Stomp
What makes WSO2 MB different● Simplified clustering story (- Queues/Topics)
- Users don’t have to worry about how brokers transmit messages in the cluster.
● RDBMS for persistence →Easy to manage and use
● Simpler Cluster deployments → Minimum deployment effort + maintenance.
Use Case – Guaranteed Delivery with WSO2 ESB
• Guarantees In-Order Delivery.• Throttle messages ( for slow consumers/systems).• Ensure reliable messaging
An Extended Use case - Where MB fits
