i’mbrian! · 14 confiden)al"informaon"of"talksum,"inc."...
TRANSCRIPT
1 Confiden)al Informa)on of Talksum, Inc.
I’m Brian!
Who Is This Guy?
2 Confiden)al Informa)on of Talksum, Inc.
• Principal Architect at Talksum
• Open Source Contributor - ZeroMQ
- Rsyslog
Who Is This Guy?
3 Confiden)al Informa)on of Talksum, Inc.
Talksum Data Stream Router
Next Age of Data Management
November 2013
4 Confiden)al Informa)on of Talksum, Inc.
The Talksum Data Stream Router
Apache Common Logging – Files SNMP -‐ UDP
Unix Logs – RFC3164 UDP/TCP Ne[low – UDP – NG v.5, 8, 9, 10
Pa)ent Records (HL7) XML/ASN.1 Transporta)on (BSM) SAE J2735
I2C, CAN, SNMP, Serial
XML, JSON, File, HTTP REST
Twiger, RSS, CAP (Weather Alerts)
Refined Data Stream
Refined Data Stream
Refined Data Stream
Indexed, Mapped, Reduced Ordered, Sorted Data Streams
Bulk Data Streams (Lightly Ordered and Filtered)
Talksum Data Stream
Router (TDSR)
• Data Normaliza)on
• Parsers • Filters • Metrics and Counts
• Inline ETL/PTL • Asynchronous Outputs
• Protocol Verifica)on
Customer A: Summarized Data
System Logs
Applica=on Data
Sensor and Industrial
Data
3rd Party Data B2B/M2M
Social and Public Data
Customer B: Aggregated Data
Customer C: Dynamic Stream
Applica=on Logs
• SQL Warehouse • Bulk Data Stores • File Storage
• Object Data Stores • Indexed Data Caches • NoSQL Data Warehouses
5 Confiden)al Informa)on of Talksum, Inc.
We turn alphabet soup into coherent streams of informa)on so that you can get to work solving problems.
In Other Words…
6 Confiden)al Informa)on of Talksum, Inc.
Cross Domain Rou)ng with ZeroMQ
November 2013
7 Confiden)al Informa)on of Talksum, Inc.
“A territory over which rule or control is exercised”
“A sphere of ac)vity, concern, or func)on”
-‐ freedic)onary.com
Domains
8 Confiden)al Informa)on of Talksum, Inc.
• Energy • Transporta)on
• Biomedical
• Informa)on Systems
Some Domains!
9 Confiden)al Informa)on of Talksum, Inc.
• Applica)on Threads • Applica)on Processes
• Systems on a network
• Networks themselves
Some More Domains!
10 Confiden)al Informa)on of Talksum, Inc.
ØMQ \zeromq\:
• A network socket library that acts as a concurreny framework
• Carries messages across mul)ple transports
• Enables networking pagerns • Provides Asyncronous I/O • Large and ac)ve open source community
• Bindings for 40+ programming languages
• Works on most opera)ng systems
What is ZeroMQ
11 Confiden)al Informa)on of Talksum, Inc.
What is ZeroMQ
WHAT IF I TOLD YOU
WE CAN HAVE LOCK-FREE IN MEMORY MESSAGE PASSING BETWEEN THREADS?
Here there be dragons
12 Confiden)al Informa)on of Talksum, Inc.
What Is ZeroMQ To Us
ZeroMQ is a tool which facilitates the construc)on of cross domain communica)on pagerns
13 Confiden)al Informa)on of Talksum, Inc.
When we can apply a pagern from one domain to a different domain, we can do more with less work.
Pagerns!
14 Confiden)al Informa)on of Talksum, Inc.
Network Monitoring & Op)miza)on
Unix Logs – RFC3164 UDP/TCP Ne[low – UDP – NG v.5, 8, 9, 10
Refined Data Stream
Refined Data Stream
Indexed, Mapped, Reduced Ordered, Sorted Data Streams
Bulk Data Streams (Lightly Ordered and Filtered)
Talksum Data Stream
Router (TDSR)
• Data Normaliza)on
• Parsers • Filters • Metrics and Counts
• Inline ETL/PTL • Asynchronous Outputs
• Protocol Verifica)on
Exis=ng BI Tools
System Logs
NOC Aler=ng
• SQL Warehouse • Bulk Data Stores • File Storage
• Object Data Stores • Indexed Data Caches • NoSQL Data Warehouses
Customer: Large European ISP/Email Communica)ons Provider Use Case: Ingest Ne[low data, parse and aggregate in real )me, monitors and alerts, op)mize network topology Status: Deploying beta appliance
15 Confiden)al Informa)on of Talksum, Inc.
Financial
XML, JSON, File, HTTP REST
Twiger, RSS, CAP (Weather Alerts)
Refined Data Stream
Refined Data Stream
Indexed, Mapped, Reduced Ordered, Sorted Data Streams
Bulk Data Streams (Lightly Ordered and Filtered)
Talksum Data Stream
Router (TDSR)
• Data Normaliza)on
• Parsers • Filters • Metrics and Counts
• Inline ETL/PTL • Asynchronous Outputs
• Protocol Verifica)on
Aler=ng & No=fica=on
3rd Party Data Trading Desks
Social and Public Data
Market Dashboard
• SQL Warehouse • Bulk Data Stores • File Storage
• Object Data Stores • Indexed Data Caches • NoSQL Data Warehouses
Customer: Major Financial Stock Exchange Use Case: Ingest unstructured financial market data, parse and filter for quality, aggregate, integrate with exis)ng data warehouse Status: Acquiring data sample for POC
16 Confiden)al Informa)on of Talksum, Inc.
Request Reply
17 Confiden)al Informa)on of Talksum, Inc.
Round Robin Load Balancing
18 Confiden)al Informa)on of Talksum, Inc.
Unicast
19 Confiden)al Informa)on of Talksum, Inc.
Mul)cast
20 Confiden)al Informa)on of Talksum, Inc.
Fair Queuing
21 Confiden)al Informa)on of Talksum, Inc.
Explicit Rou)ng
22 Confiden)al Informa)on of Talksum, Inc.
• Within a thread
• Between threads
• Between processes on the same system
• Between processes on different systems
Inter Domain Messaging
23 Confiden)al Informa)on of Talksum, Inc.
Why Talksum Contributes to Zeromq
• We are all facing a lot of problems. • We believe in the value of empirical evidence. • Solu)ons emerge from all of us working together across our various domain boundaries.
• Par)cipate!
24 Confiden)al Informa)on of Talksum, Inc.
• hgp://www.zeromq.org/ -‐ Main ZeroMQ Web Site
• hgp://zguide.zeromq.org/ -‐ ZeroMQ User Guide
• hgp://github.com/zeromq -‐ ZeroMQ Git Repository
• hgp://api.zeromq.org/ -‐ ZeroMQ Reference Manual
• hgp://zeromq.org/whitepapers:architecture – Architecture Whitepaper
• hgp://czmq.zeromq.org/ -‐ ZeroMQ C Binding
• hgp://zeromq.github.io/pyzmq/ -‐ ZeroMQ Python Binding
Resources