Monitoring Electronic Trading Environments using Spark

Fergal Toomey and Pierre LacaveCorvil

Monitoring Goals

•Performance and technical risk

•Client experience

•Compliance

Challenges

•Distributed across sites and firms

•Latency sensitive

•Multiple protocols and data formats

EXCHANGES

Matching Engine Trading

Gateways

Market Data

Validation/ Routing

Consolidation/ Distribution

MARKET DATA SERVICES

BROKERAGE SERVICES

TRADERS

•Transformations:•Change of custody•Different symbologies•Parent/child orders•Variable routing

Data Sources

Data Volumes

Platform

Corvil Streaming

DataCorvil

Streaming DataCorvil Streaming

Data

Message BusDistributed Real-Time Processing

StorageData Sources

Exploration and Visualization

Metrics Calculation

Windowing & Correlation

Raw Data•Enriched (correlated data)•Searchable

Custom Metrics•Latencies•Counts•Filterable by key dimensions•Rolled up

Clients MarketsA B C D

ID: 1

CLIENT: C1SYMBOL: ABC

ID: APARENTID: 1

SYMBOL: ABC

ID: ZPARENTID: A

SYMBOL: ABC

ID: 10PARENTID: Z

SYMBOL: ABCMARKET: NYSE

Clients Markets

Events Correlation

Filter by destination market in point A?

A B C D

ID: 1

CLIENT: C1SYMBOL: ABC

ID: APARENTID: 1

SYMBOL: ABC

ID: ZPARENTID: A

SYMBOL: ABC

ID: 10PARENTID: Z

SYMBOL: ABCMARKET: NYSE

Clients Markets

Events CorrelationA B C D

ID: 1

CLIENT: C1SYMBOL: ABCMARKET: NYSE

ID: APARENTID: 1

CLIENT: C1 SYMBOL: ABCMARKET: NYSE

ID: ZPARENTID: A

CLIENT: C1 SYMBOL: ABCMARKET: NYSE

ID: 10PARENTID: Z

CLIENT: C1SYMBOL: ABCMARKET: NYSE

Clients Markets

Events CorrelationA B C D

Events Windowing

Batch #1 Batch #2 Batch #3

window #1

window #2

A B C D

Spark Streaming window

Reconstruct batch based on original time

Spark Streaming micro-batch

Direct Data Access

Access metrics via SQL / JDBC

BI tools, self-developed application, etc

Calculation done in HBASE

Data Extracts

• Filtering pushed down to HBASE• Aggregation done In SPARK (SparkSQL)

CSV extraction to HDFS

Extract Consumption

THANK YOU.www.corvil.com