fabric8 mq

Achieving Horizontal scaling for Enterprise Messaging using Fabric8

Rob Davies June 2015

RED HAT | Fabric8 Rocks! 2

Rob Davies

•  Director, Engineering xPaaS •  Technical Director of Fuse

Engineering •  Over 20 years experience of

developing large scale solutions for telcos and finance

•  Creator of ActiveMQ and ServiceMix •  Committer on open source projects,

including Apache Camel …


Agenda

• Introduction to Fabric8 • Enterprise messaging • Fabric8MQ • Demo


Fabric8

•  Open source, ASL 2.0 licensed project, built to provide the management infrastructure for Microservices deployments

•  Management: console, logging, metrics, API Management

•  Continuous Delivery - workflow •  Integration Platform as a Service – Camel route

visualization, API registry, Messaging as a Service, … •  Tools – Kubernetes/OpenShift build integration,

Kubernetes component test support, CDI extensions


Fabric8:

GCE AWS CoreOS Mesos vSphere Fedora Vagrant Azure

Kubernetes

Centralized Logging Metrics Deep Application

Management

API Registry

API Man Fabric8MQ Camel Fabric8

Java tools

social SonaQube Gogs Jenkins Nexus


Agenda



Why use messaging ?

•  Robustness to change •  Time independence •  Hide latency •  Event driven •  Platform and language

independence Broker

Message in

Message out after enrichment by Process B

Process C

Process B

Process A Queue:Foo

Queue:Bar


Enterprise Message brokers

•  Designed to support many different messaging patterns

•  highly available •  Support clustering •  Support store and forward •  But – are usually very static in nature


•  Top Level Apache Software Foundation Project

•  Wildly popular, high performance, reliable message broker

•  Connects to nearly everything

• Native Java, C/C++, .Net,

• AMQP 1.0, MQTT 3.1, STOMP (1.0-1.2) and OpenWire

• STOMP enables Ruby, JS, Perl, Python, PHP, ActionScript …

•  Embedded and standalone deployment options


Message Channels and Routing

•  Message Channels •  Named communication between interested parties •  JMS calls them ‘Destinations’

•  Can “tune-in” to multiple channels using wildcards •  Can fine-tune message consumption with selectors •  Can route a message based on content


Message Channels = JMS Destinations

Broker Consumer

Consumer Destination WIDGET

Destination ORDER

Producer

Consumer


Message Routing: Selectors

Producer Destination

Consumer Color = red

Consumer Color = blue


Message Routing: Destination Wildcards

•  * matches a subject •  > matches sub-tree

Topic: BAR.BEER.WHITE

Topic: BAR.WINE.WHITE

Producer

Consumer Topic:BAR.WINE.WHITE

Consumer Topic:BAR.BEER.WHITE

Consumer Topic:BAR.*.WHITE

Consumer Topic:BAR.>


Message Groups

Producer

Consumer

Consumer

Consumer

Queue:Prices

Message message = session.createTextMessage(“hi JBCNConf”); message.setStringProperty("JMSXGroupID", ”JavaRocks"); producer.send(message);


High Availability clustering:

Master Client Slave Slave

Only one broker is active (the master) at any one time.

1. send

3. receipt

2. store


Network of Brokers:

Producer

Highly reliable, store and forward

Consumer

1. send

Broker1

2. store

Broker2

5. store

4. Network send

6. Network receipt

3. receipt

7. delete

6. Network send

7. acknowledge

8. delete


Networks of Brokers: some limitations

•  Performance – messages are always stored by the broker to disk before forwarding to another broker – this can increase latency by at least a factor of 10

•  Networks are chatty – routing information is passed from broker to broker every time a consumer starts or stops

•  Networks are a bottleneck – networks are implemented as one TCP/IP connection – this is natural choke point for high throughput scenarios

•  Messages can be orphaned – If a broker goes down, locally stored messages won’t get delivered until the broker is recovered


Networks of Brokers: Worst bit …

Producer Consumer Broker1 Broker2

This is configured differently to …


Networks of Brokers: Worst bit …

Producer Consumer Broker1 Broker2

To this

Broker3


ActiveMQ is static:


ActiveMQ Key Features

Fast Storage

Connections

A-MQ

Embedded Camel OpenWire, STOMP MQTT AMQP WebSockets

Scaling Connections A problem

KahaDB/LevelDB

Lots and Lots of these


To Scale A-MQ needs to focus …

Fast Storage

A-MQ

Remove all the cruft – allow ActiveMQ to focus on message routing …


Messaging for cloud based deployments

•  Needs to support many thousands of clients •  Flexible, brokers need to be spun up and down, based

on demand •  Client connections may need to be multiplexed, to

decrease the load on individual message brokers •  Popular messaging protocols support •  Flexible routing needs to be supported


Agenda



Fabric8 MQ is built on Vert.x

•  Lightweight, multi-reactive, application platform •  Inspired from Erlang/OTP •  Polygot •  High performance •  Asynchronous, non-blocking: – high level of

concurrency


Vert.x is just fast …


Fabric8 MQ Message Flow:

Protocol Conversion

Camel Routing

API Management Multiplexer

Destination Sharding

Broker Control


Fabric8 MQ Protocol Conversion

•  We convert to ActiveMQ OpenWire. This saves ActiveMQ doing this, allowing it to expend all its resources to pushing messages in and out of Queues

•  Protocols currently supported: •  OpenWire •  Stomp •  MQTT •  AMQP 1.0


Camel Routing:

Camel is embedded to allow flexible routing:

Topic:Foo

Message Message Message

Queue:Foo

Channel


API Management

•  Fabric8 uses APIMan for its API Management – see http://apiman.io

•  Utilized within Fabric8 to provide: •  Central common place for managing all APIs (including

Message Queues) •  Control and Access policies •  Rate Limiting •  Metrics and Billing


Multiplexing

•  ActiveMQ performs better the lower the number of connections – less contention

•  IoT applications have long lived connections, but small amount of traffic

•  All configurable – you can decide how fine or coarse grained you want multiplexing

•  Fabric8MQ scales up/down multiplexers as required


Destination Sharding: Benefits

•  There is an overhead associated with a Destination – restrict the number of Destinations per Broker to improve performance

•  Co-locate producers and consumers of a Destination to the same broker reduces latency and improves overall performance

•  Increased scalability for your messaging solution

•  This is done automatically by Fabric8 MQ


Destination Sharding: Fabric8 MQ Internal Model

•  Discover and track all allocated brokers

•  Fabric8 MQ regularly updates its internal model by examining internals of brokers running via Jolokia allowing multiple Fabric8-MQ instances to run (co-operatively scalable)

•  Scales up and down brokers – moving destinations and clients between brokers

•  Can dynamically set limits on number of brokers, and broker limits

Broker1 Connections count

Depth Consumers Producers

Queue://foo.1


Queue://foo.2


Queue://foo.3

Broker2 Connections count


Queue://bar.1


Queue://bar.2


Queue://bar.3


Destination Sharding: Internal Rules

•  Fired when state changes

•  Operates over the internal Model

•  Rules fired in priority order

•  Only one rule fired per state change

Scale Up

•  Connection limits exceeded OR

•  Destination Limits Exceeded AND

•  NO spare capacity

Broker Control

Scale Down

•  At least ONE broker AND

•  Total load across all brokers less than 50%

Distribute Load

•  Broker limits are Exceeded AND

•  Spare capacity

Ask to move clients and destinations from most loaded to least loaded

Distribute clients and destinations off least loaded and remove broker

Request a new Broker

Broker Control is done via Kubernetes


ActiveMQ hates unexpected Acks…


Fabric8MQ has to 1.  Work out which destinations to migrate 2.  Stop dispatching to consumers 3.  Wait until the all in flight messages are acked 4.  Do the migration 5.  Update the internal destination map 6.  Then resume processing


Kubernetes

•  Created by Google from experience with GCE

•  Lightweight, simple and accessible orchestration system for Docker containers

•  Schedules onto nodes in a compute cluster and actively manages workloads to ensure state is “correct”

•  Provides easy management and Discovery

Kubernetes Master

Node Node Node

User containers

Scheduled and packed dynamically onto nodes


Kubernetes Model

•  Pods – collection of one or more Docker containers – the minimum deployment unit in Kubernetes

•  Replication Controllers – stored on Kubernetes Master, they determine how many replicas of Pods should be running

•  Service – abstraction that defines a logical set of pods, and defines a way of exposing their ports on a unique IP address

•  Namespaces – Pods, Services, Replication Controllers, Label Selector – all exist within a defined Namespace. This allows for multiple environments to be supported by the same compute cloud – or multitenancy

Kubernetes Master

Node

Container Pod Container

Pod

Node

Container Pod Container

Pod

Replication Controller

Fabric8-MQ-Service


Fabric8 MQ Broker Control:

•  Fabric8 MQ controls a headless Replication Controller to spin up and down ActiveMQ Brokers

•  Monitors the state of running ActiveMQ brokers via Jolokia – and fires changes to the internal rules engine.

•  Communicates with the Replication Controller to spin up – or delete Pods containing ActiveMQ Brokers

Node

Pod

ActiveMQ Broker

AMQ Replication Controller

Node

Pod

ActiveMQ Broker

Node

Pod

ActiveMQ Broker


Fabric8 MQ – not a message broker – but a scalable messaging system

Many concurrent connections, one out

Connections

Vert.x Core

Embedded Camel, Integration with APIMan

OpenWire, STOMP MQTT AMQP WebSockets – all done asynchronously

Scaling Connections NOT a problem

Destination Sharding


Fabric8 MQ Independently scalable:

Node

Pod

ActiveMQ Broker

AMQ Replication Controller

Node

Pod

ActiveMQ Broker

Node

Pod

ActiveMQ Broker

Fabric8MQ Replication Controller

Vert.x Vert.x

Vert.x

Multiplexer Multiplexer

Multiplexer Fabric8MQ

Vert.x Vert.x

Vert.x



Vert.x Vert.x

Vert.x




What I’m gonna try and attempt to do ...

Fabric8MQ

MQTT Device

MQTT Device MQTT

Device MQTT Device

MQTT Device MQTT

Device

MQTT Device MQTT

Device MQTT Device


Destination Sharding: Internal Rules

Limits are set low: 5 Destinations per

Broker

Scale Up

•  Connection limits exceeded OR

•  Destination Limits Exceeded AND

•  NO spare capacity

Broker Control

Scale Down

•  At least ONE broker AND

•  Total load across all brokers less than 50%

Distribute Load

•  Broker limits are Exceeded AND

•  Spare capacity

Ask to move clients and destinations from most loaded to least loaded

Distribute clients and destinations off least loaded and remove broker

Request a new Broker

Broker Control is done via Kubernetes


Client code

final String host = System.getenv("FABRIC8MQ_SERVICE_HOST");

final String portStr = System.getenv("FABRIC8MQ_SERVICE_PORT");

Kubernetes injects services as Environment variables


Questions and Discussion

fabric8 mq

Software

red hat fabric8 rocks

prices message message

red consumer color

message routing

message consumption

fabric8 open source

fabric8 rob davies

white consumer topic