faбrik - timesopen: sockets and streams - sept. 2012

NYT faбrik

TimesOpen – 12 September 2012

2

Who

Architect Infrastructure Group Previous

– US Army– Harvard– Tech companies– United Nations

3

What we’ll cover

A story faбrik overview Code and demo

4

Takeaways

Application Developer– Demand ‘events’ (no polling, no thanks, been there, done that)– Demand infrastructure that scales and you don’t have to worry about

Infrastructure Engineer– Decouple, flatten, simplify– Outsource complexity

5

The Story: Harvard U, 1986+

Bad: – No Internet (pre-web)– data locked in mainframe– large central clerical staff– monolithic central systems

Good:– Vision of an Information Utility– desire to innovate– lots of desktop computers (30,000)– email everywhere although over diverse networks and technologies

6

The Story: Harvard U, 1986+

Solution: – Relational database (decouple data from application)– Email backbone (decouple producers from consumers)– Event-driven desktop applications (flatten)– Identical code on mainframe (simplify)

Result:– Data warehouse unlocked (before the term was coined)– Central clerical staff functions upgraded/dispersed– Old central systems replaceable and, ultimately, replaced– Happy users!

7

faбrik – Why?

?

8

NYT Mission

Enhance society by creating, collecting and distributing high quality news, information and entertainment

- Distributing: publish / subscribe- Collecting: gather / analyze- High Quality: fast, reliable, accurate

9

faбrik

Asynchronous Messaging Framework

For client devices as well as our apps

Enabled by:– Websockets– Robust message handling software– Amazon Web Services

Focusing on simple, common services

10

faбrik – solve the problem!

App

11

faбrik – basic

App

App

App

Message Broker

12

faбrik – basic

App

App

App

Message Broker

Amazon Web Services

• EC2• S3• Identity &

Access Mgt• DynamoDB• Route 53…

13

faбrik – basic++

OtherApp

AppBuddy

Service Buddy

Message Broker

Message Broker

Service Buddy

“Retail”

“Wholesale”

14

faбrik: Current Implementation

Open source– Erlang/OTP– RabbitMQ– Nodejs – Sockjs (websockets +)– Python (gevent, select/epoll, tornado, futures, …)– ZeroMQ (new)

Automated build/deployment

15

faбrik – active/active cluster

Service Buddy

‘a’

Service Buddy

‘b’

Zone ‘a’ Zone ‘b’

Region Wherever

16

faбrik – active/active cluster

Service Buddy

‘a’

Service Buddy

‘b’

Zone ‘a’ Zone ‘b’

Region Wherever

Got your back!

Got yours too!

17

faбrik

18

faбrik

19

faбrik

20

faбrik – Layers

21

faбrik – Layers

22

faбrik – Layers

23

faбrik – Layers

24

25

26

Pub / Sub

27

Pub / Sub

28

Pub / Sub

29

Pub / Sub

30

Pub / Sub

31

Gather / Analyze

32

Gather / Analyze

33

Gather / Analyze

34

Gather / Analyze

35

Gather / Analyze

36

Gather / Analyze

37

Gather / Analyze

38

faбrik

Demo

Code

App