„enterprise event bus“ unified log (event) processing architecture

30
BASEL BERN BRUGG LAUSANNE ZÜRICH DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MÜNCHEN STUTTGART WIEN 2014 © Trivadis „Enterprise Event Bus“ Unified Log (Event) Processing Architecture DOAG DevCamp 2015 Guido Schmutz Trivadis AG April 2015 „Enterprise Event Bus“ Unified Log (Event) Processing Architecture 1

Upload: guido-schmutz

Post on 21-Jan-2017

1.173 views

Category:

Technology


1 download

TRANSCRIPT

Page 1: „Enterprise Event Bus“ Unified Log (Event) Processing Architecture

2015 © Trivadis

BASEL BERN BRUGG LAUSANNE ZÜRICH DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MÜNCHEN STUTTGART WIEN

2014 © Trivadis

„Enterprise Event Bus“Unified Log (Event) Processing ArchitectureDOAG DevCamp 2015

Guido Schmutz

Trivadis AG

April 2015„Enterprise Event Bus“ Unified Log (Event) Processing Architecture

1

Page 2: „Enterprise Event Bus“ Unified Log (Event) Processing Architecture

2015 © Trivadis

A little story of a “real-life” customer situation

Traditional system interact with its clients and does its work

Implemented using legacy technologies (i.e. PL/SQL)

New requirement:

• Offer notification service to notify customer when goods are shipped

• Subscription and inform over different channels

• Existing technology doesn’t fit

May 2015Blueprints for the analysis of social media

2

delivery

LogisticSystem

Oracle

MobileApps

Sensor ship

sort

2

Rich(Web)ClientApps

DB

schedule

Logic(PL/SQL)

delivery

Page 3: „Enterprise Event Bus“ Unified Log (Event) Processing Architecture

2015 © Trivadis

A little story of a “real-life” customer situation

May 2015Blueprints for the analysis of social media

3

delivery

LogisticSystem

Oracle

MobileApps

Sensor ship

sort

3

Rich(Web)ClientApps

DB

schedule

Notification

Logic(PL/SQL)

Logic(Java)delivery

SMS

Email

Eventsare“owned”bytraditionalapplication(aswellasthechannelstheyaretransportedover)

Page 4: „Enterprise Event Bus“ Unified Log (Event) Processing Architecture

2015 © Trivadis

A little story of a “real-life” customer situation

Rule Engine implemented in Java and invoked from OSB message flow

Notification system informed via queue

Higher Latency introduced (good enough in this case)

integrate in order to get the information!

Oracle Service Bus was already there

May 2015Blueprints for the analysis of social media

4

delivery

LogisticSystem

OracleOracle

ServiceBus

MobileApps

Sensor AQship

sort

4

Rich(Web)ClientApps

DB

schedule

Filter

Notification

Logic(PL/SQL)

JMS

RuleEngine(Java)

Logic(Java)delivery

shipdelivery

delivery true SMS

Email

Page 5: „Enterprise Event Bus“ Unified Log (Event) Processing Architecture

2015 © Trivadis

A little story of a “real-life” customer situation

May 2015Blueprints for the analysis of social media

5

delivery

LogisticSystem

OracleOracle

ServiceBus

MobileApps

Sensor AQship

sort

5

Rich(Web)ClientApps

DB

schedule

Filter

Notification

Logic(PL/SQL)

JMS

RuleEngine(Java)

Logic(Java)delivery

shipdelivery

delivery true SMS

Email

Treat events as first-class citizens

Events belong to the “enterprise” and not an individual system => Catalog of Events similar to Catalog of Services/APIs !!

Event (stream) processing can be introduced and by that latency reduced!

Page 6: „Enterprise Event Bus“ Unified Log (Event) Processing Architecture

2015 © Trivadis

Why Stream Processing?

Responselatency

StreamProcessing

Milliseconds tominutes

RPC

Synchronous Later.Possiblymuchlater.

April 2015Apache Storm vs. Spark Streaming - Two Stream Processing Platforms compared

6

Page 7: „Enterprise Event Bus“ Unified Log (Event) Processing Architecture

2015 © Trivadis

Agenda

1. Designing Stream Processing Solutions

2. Implementing the Enterprise Event Bus (Unified Log)

3. Unified Log (Event) Processing Architecture in Action

April 2015„Enterprise Event Bus“ Unified Log (Event) Processing Architecture

7

Page 8: „Enterprise Event Bus“ Unified Log (Event) Processing Architecture

2015 © Trivadis

What is Stream Processing?

April 2015„Enterprise Event Bus“ Unified Log (Event) Processing Architecture

8

EventStream

eventCollecting

event

Persist(Queue)

EventStream

eventCollecting

event

Processing

eventProcessing

result

result

EventStream

event Collecting/Processing

result

Page 9: „Enterprise Event Bus“ Unified Log (Event) Processing Architecture

2015 © Trivadis

What is Stream Processing?

Infrastructure for continuous data processing

Computational model can be as general as MapReduce but with the ability to produce low-latency results

Data collected continuously is naturally processed continuously

aka. Event Processing / Complex Event Processing (CEP)

April 2015Apache Storm vs. Spark Streaming - Two Stream Processing Platforms compared

9

Page 10: „Enterprise Event Bus“ Unified Log (Event) Processing Architecture

2015 © Trivadis

How to design a Streaming Processing System?It usually starts very simple … just one data pipeline

April 2015„Enterprise Event Bus“ Unified Log (Event) Processing Architecture

10

EventStream ConsumereventApplication

Page 11: „Enterprise Event Bus“ Unified Log (Event) Processing Architecture

2015 © Trivadis

New Event Stream sources are added …

April 2015„Enterprise Event Bus“ Unified Log (Event) Processing Architecture

11

EventStream Consumer

2nd EventStream

3rd EventStream

nth EventStream

event

event

event

event

Application

2ndApplication

3rdApplication

Nth

Application

Page 12: „Enterprise Event Bus“ Unified Log (Event) Processing Architecture

2015 © Trivadis

New Processors are interested in the events …

April 2015„Enterprise Event Bus“ Unified Log (Event) Processing Architecture

12

EventStream Consumer

2nd EventStream

3rd EventStream

nth EventStream

2nd Consumerevent

event

event

event

Application

2ndApplication

3rdApplication

Nth

Application

Page 13: „Enterprise Event Bus“ Unified Log (Event) Processing Architecture

2015 © Trivadis

… and the solution becomes the problem

April 2015„Enterprise Event Bus“ Unified Log (Event) Processing Architecture

13

EventStream Consumer

2nd EventStream

3rd EventStream

nth EventStream

2nd Consumer

3rd Consumer

Nth Consumer

event

event

event

event

Application

2ndApplication

3rdApplication

Nth

Application

Page 14: „Enterprise Event Bus“ Unified Log (Event) Processing Architecture

2015 © Trivadis

… and the solution becomes the problem

April 2015„Enterprise Event Bus“ Unified Log (Event) Processing Architecture

14

EventStream Consumer

2nd EventStream

3rd EventStream

nth EventStream

2nd Consumer

3rd Consumer

Nth Consumer

event

event

event

event

Application

2ndApplication

3rdApplication

Nth

Application

Page 15: „Enterprise Event Bus“ Unified Log (Event) Processing Architecture

2015 © Trivadis

… and the solution becomes the problem

April 2015„Enterprise Event Bus“ Unified Log (Event) Processing Architecture

15

NewCustomers

OperationalLogs

ClickStream

MeterReadings

event

event

event

event

CDCCollector

LogCollector

ClickStreamCollector

SenorCollector

Hadoop/DataWarehouse

RecommendationSystem

LogSearch

FraudDetection

Page 16: „Enterprise Event Bus“ Unified Log (Event) Processing Architecture

2015 © Trivadis

Decouple event streams from consumers

April 2015„Enterprise Event Bus“ Unified Log (Event) Processing Architecture

16

„UnifiedLog“

RememberEnterpriseServiceBus(ESB)?

EnterpriseEventBus EventStreamProcessorEventStreamSource

NewCustomers

OperationalLogs

ClickStream

MeterReadings

CDCCollector

LogCollector

ClickStreamCollector

SenorCollector

Hadoop/DataWarehouse

RecommendationSystem

LogSearch

FraudDetection

WhatistheideaofaUnifiedLog?

Page 17: „Enterprise Event Bus“ Unified Log (Event) Processing Architecture

2015 © Trivadis

Unified Log – What is it?

By Unified Log, we do not mean this ….137.229.78.245 - - [02/Jul/2012:13:22:26 -0800] "GET /wp-admin/images/date-button.gif HTTP/1.1" 200 111137.229.78.245 - - [02/Jul/2012:13:22:26 -0800] "GET /wp-includes/js/tinymce/langs/wp-langs-en.js?ver=349-20805 HTTP/1.1" 200 13593137.229.78.245 - - [02/Jul/2012:13:22:26 -0800] "GET /wp-includes/js/tinymce/wp-tinymce.php?c=1&ver=349-20805 HTTP/1.1" 200 101114137.229.78.245 - - [02/Jul/2012:13:22:28 -0800] "POST /wp-admin/admin-ajax.php HTTP/1.1" 200 30747137.229.78.245 - - [02/Jul/2012:13:22:40 -0800] "POST /wp-admin/post.php HTTP/1.1" 302 -137.229.78.245 - - [02/Jul/2012:13:22:40 -0800] "GET /wp-admin/post.php?post=387&action=edit&message=1 HTTP/1.1" 200 73160137.229.78.245 - - [02/Jul/2012:13:22:41 -0800] "GET /wp-includes/css/editor.css?ver=3.4.1 HTTP/1.1" 304 -137.229.78.245 - - [02/Jul/2012:13:22:41 -0800] "GET /wp-includes/js/tinymce/langs/wp-langs-en.js?ver=349-20805 HTTP/1.1" 304 -137.229.78.245 - - [02/Jul/2012:13:22:41 -0800] "POST /wp-admin/admin-ajax.php HTTP/1.1" 200 30809

… but this

• a structured log (records are numbered beginning with 0 based on order they are written)

• aka. commit log orjournal

April 2015„Enterprise Event Bus“ Unified Log (Event) Processing Architecture

17

0 1 2 3 4 5 6 7 8 9 10 11

1st record Nextrecordwritten

Page 18: „Enterprise Event Bus“ Unified Log (Event) Processing Architecture

2015 © Trivadis

Central Unified Log for (real-time) subscription

Take all the organization’s data (events) and put it into a central log for subscription

Properties of the Unified Log:• Unified: “Enterprise”, single deployment• Append-Only: events are appended, no update in place => immutable• Ordered: each event has an offset, which is unique within a shard• Fast: should be able to handle thousands of messages / sec• Distributed: lives on a cluster of machines

April 2015

„Enterprise Event Bus“ Unified Log (Event) Processing Architecture18

0 1 2 3 4 5 6 7 8 9 10 11

reads

writes

Collector

ConsumerSystemA(time=6)

ConsumerSystemB(time=10)

reads

Page 19: „Enterprise Event Bus“ Unified Log (Event) Processing Architecture

2015 © Trivadis

Unified Log Processing Architecture

Stream processing allowsfor computing feedsoff other feeds

Derived feedsare no differentthan original feedsthey are computed off

Single deployment of“Unified Log”

logically different feeds

April 2015„Enterprise Event Bus“ Unified Log (Event) Processing Architecture

19

MeterReadings Collector

Enrich/Transform

AggregatebyMinute

Raw MeterReadings

Meter&Customer

Meterby CustomerbyMinute

Customer AggregatebyMinute

MeterbyMinute

Persist

MeterbyMinute

Persist

RawMeterReadings

….

Page 20: „Enterprise Event Bus“ Unified Log (Event) Processing Architecture

2015 © Trivadis

Agenda

1. Designing Stream Processing Solutions

2. Implementing the Enterprise Event Bus (Unified Log)

3. Unified Log (Event) Processing Architecture in Action

April 2015„Enterprise Event Bus“ Unified Log (Event) Processing Architecture

20

Page 21: „Enterprise Event Bus“ Unified Log (Event) Processing Architecture

2015 © Trivadis

Apache Kafka - Overview

• A distributed publish-subscribe messaging system

• Designed for processing of real time activity stream data (logs, metrics collections, social media streams, …)

• Initially developed at LinkedIn, now part of Apache

• Does not follow JMS Standards and does not use JMS API

• Kafka maintains feeds of messages in topics

April 2015„Enterprise Event Bus“ Unified Log (Event) Processing Architecture

21

Kafka Cluster

Consumer Consumer Consumer

Producer Producer Producer0 1 2 3 4 5 6 7 8 9 1

011

12

0 1 2 3 4 5 6 7 8 9

0 1 2 3 4 5 6 7 8 9 10

11

12

Anatomy of a topic:

Partition 0

Partition 1

Partition 2

Writes

old new

Page 22: „Enterprise Event Bus“ Unified Log (Event) Processing Architecture

2015 © Trivadis

Apache Kafka - Performance

Kafka at LinkedIn

Up to 2 million writes/sec on 3 cheap machines§ Using 3 producers on 3 different machines

April 2015„Enterprise Event Bus“ Unified Log (Event) Processing Architecture

22

10+billionwritesperday

172kmessagespersecond

(average)

55+billionmessagesperday

toreal-timeconsumers

http://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines

Page 23: „Enterprise Event Bus“ Unified Log (Event) Processing Architecture

2015 © Trivadis

Apache Kafka - Partition offsets

Offset: messages in the partitions are each assigned a unique (per partition) and sequential id called the offset

• Consumers track their pointers via (offset, partition, topic) tuples

April 2015„Enterprise Event Bus“ Unified Log (Event) Processing Architecture

23

ConsumergroupC1

Page 24: „Enterprise Event Bus“ Unified Log (Event) Processing Architecture

2015 © Trivadis

Apache Kafka – two Options for Log Cleanup

April 2015„Enterprise Event Bus“ Unified Log (Event) Processing Architecture

24

1. Retaining a window of data• Ideal for event data• Window can be defined in time (days) or space (GBs) – defaults to 1 week

2. Retain a complete log (log compaction)• Ideal for keyed data• Keep a space-efficient complete

log of changes• Log compaction runs in the

background• Ensures that always at least the

last known value for each messagekey within the log of data is retained

Page 25: „Enterprise Event Bus“ Unified Log (Event) Processing Architecture

2015 © Trivadis

Unified Log Alternatives

• Amazon Kinesis (http://aws.amazon.com/kinesis/)• Confluent (http://confluent.io/)• Redis Pub/Sub (http://redis.io/topics/pubsub)• Kestrel (http://robey.github.io/kestrel/)• ZeroMQ (http://zeromq.org/)• RabbitMQ (http://www.rabbitmq.com/)• Oracle GoldenGate (http://bit.ly/g-gate)• JMS compliant Server

• Apache ActiveMQ (http://activemq.apache.org/)• Weblogic JMS

(http://www.oracle.com/technetwork/middleware/weblogic/overview/index.html)

• IBM Websphere MQ (http://www-03.ibm.com/software/products/de/ibm-mq)• …

April 2015„Enterprise Event Bus“ Unified Log (Event) Processing Architecture

25

Page 26: „Enterprise Event Bus“ Unified Log (Event) Processing Architecture

2015 © Trivadis

Agenda

1. Designing Stream Processing Solutions

2. Implementing the Enterprise Event Bus (Unified Log)

3. Unified Log (Event) Processing Architecture in Action

April 2015„Enterprise Event Bus“ Unified Log (Event) Processing Architecture

26

Page 27: „Enterprise Event Bus“ Unified Log (Event) Processing Architecture

2015 © Trivadis

Unified Log Processing Architecture in Trivadis CRA

April 2015„Enterprise Event Bus“ Unified Log (Event) Processing Architecture

27

Tweets FilterandUnify

PersistTweet

FilteredTweets

SplitText

Words

CountoverTime

CountbyMinute

AnalyzeSentiment

Tweet&Sentiment

RemoveStopwords

Tweet

TweetsConsumer

TwitterFilterStream

SensorLayer DistributionLayer

SpeedLayer

Kafka Storm

Cassandra Elasticsearch

Page 28: „Enterprise Event Bus“ Unified Log (Event) Processing Architecture

2015 © Trivadis

Unified Log Processing Architecture in Trivadis CRA

April 2015„Enterprise Event Bus“ Unified Log (Event) Processing Architecture

28

Tweets FilterandUnify

PersistTweet

FilteredTweets

SplitText

Words

CountoverTime

CountbyMinute

AnalyzeSentiment

Tweet&Sentiment

RemoveStopwords

Tweet

TweetsConsumer

TwitterFilterStream

SensorLayer DistributionLayer

Splitter

KafkaSpout

WordRemover

Splitter

WordRemover

Shuffle Fields

Kafka

Kafka

WordRemover

StormTopology

SpeedLayer

Kafka Storm

Cassandra Elasticsearch

Page 29: „Enterprise Event Bus“ Unified Log (Event) Processing Architecture

2015 © Trivadis

Weitere Informationen...

April 2015„Enterprise Event Bus“ Unified Log (Event) Processing Architecture

29

Page 30: „Enterprise Event Bus“ Unified Log (Event) Processing Architecture

2015 © Trivadis

BASEL BERN BRUGG LAUSANNE ZÜRICH DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MÜNCHEN STUTTGART WIEN

Fragen und Antworten...

2013 © Trivadis

April 2015„Enterprise Event Bus“ Unified Log (Event) Processing Architecture