big data for finance – challenges in high-frequency trading
TRANSCRIPT
THINK BIG INTERNATIONAL
Matt Cooke - Principal Manager
Martin Oberhuber - Senior Data Scientist
© 2015 Think Big, a Teradata Company
WHO IS THINK BIG? © 2015 Think Big, a Teradata
Company
THINK BIG
© 2015 Think Big, a Teradata Company
• First and leading professional services firm exclusively focused on big data
• End to End Services: Strategy, Design, Implementation, IP/Software, Support and Managed Services
• Academy to scale delivery capability
• Extend and integrate open source with UDA
• Team-based delivery with Solution Center
• Trusted Analytics Services Provider to Fortune 1000
Proven, Team-based MethodologyExperiment-Driven Short Sprints with Quick Release Cycles
We will be the trusted Big Analytics provider to the Fortune 1000and become the #1 Global Brand in Big Data analytics consulting.
Our Mission
eCommerce2 of Global Top 5
Internet Transaction SecurityGlobal #1
Retail2 of Global Top 5
Brokerage & Mutual Funds2 of Global Top 5
Social NetworkingGlobal #1
Asset ManagementGlobal #1
Credit Issuer2 of Global Top 5
Semiconductor2 of Global Top 5
Banking4 of Global Top 10
Data Storage Devices3 of Global Top 5
Financial Data Services2 of Global Top 5
Disk ManufacturingGlobal #1
Financial ExchangesGlobal #2
Telecommunications2 of Global Top 5
Media & Advertising2 of Global Top 5
THINK BIG CLIENTSTrusted Analytics Service Provider to the Fortune 1000
© 2015 Think Big, a Teradata Company
Think Big Academy
© 2015 Think Big, a Teradata Company
VELOCITY METHODOLGY
Managed Services
Data Engineering
Big Data Program Mgt
• Solution Focus• Planning & Design• Team Prioritization• Engineering
• Engineering• Software Dev• Agile Sprint(s)• Optimization
• Quality Assurance & Test• Managed Support• Break Fix• Sustaining Engineering
• New Models• New Analytics• New Insights• New Data Requirements
• Big Data Approach• Use Cases• Refine Roadmap• Org & Process
• Data Science• Discovery• R&D• Machine Learning
Big Data Strategy
Business Analytics
Big Data Lab
Hands on Training• Data Science• Data Engineering• Operations
Think Big engages with it’s clients business, technical, analyst and support teams in an agile inspired VELOCITY methodology to continuously develop big data solutions.
© 2015 Think Big, a Teradata Company
ENGAGEMENT MODEL
Think Big offers end-to-end Big Data strategy, implementation and support services focused on helping customers quickly achieve ROI on their Big Data investments
STRATEGY IMPLEMENTATION SOLUTION SUPPORT
ENTERPRISE DATA LAKE SOFTWARE FRAMEWORKS
ManagedServices
Big DataAnalyticsRoadmap
Data Lake Optimisatio
n
EstablishData Lake
AnalyticSolutions
© 2015 Think Big, a Teradata Company
Big Data for Finance – Challenges in High-Frequency Trading
Graphic by Stamen
Using computer algorithms to rapidly trade securities• Positions are held for seconds to minutes
• Reaction times to market changes are sub-millisecond.
• HFT accounts for more than 60% of all trading volume in some markets
© 2015 Think Big, a Teradata Company
WHAT IS HIGH-FREQUENCY TRADING?
•SpeedLatencies in electronic trading are usually measured in
microsecondsHFT firms co-locate with exchanges to reduce latency
•Strong predictionsDue to market efficiency it is challenging to come up with robust predictive models
© 2015 Think Big, a Teradata Company
WHAT IS IMPORTANT?
Fibre Optic MicrowaveRefractive Index (v) 1.5 1.0003Round-trip time Chicago/New Jersey
~8ms ~13ms
© 2015 Think Big, a Teradata Company
LOW LATENCY
© 2015 Think Big, a Teradata Company
MICROWAVE VS FIBRE OPTIC
time price order_flag size1336732593.051448 571.00 Ask_cancel 1081336732593.096281 571.06 Ask_add 9221336732593.138566 571.19 Bid_add 2301336732593.179509 571.26 Bid_add 7311336732593.249253 571.28 Trade 2801336732593.321581 571.33 Bid_cancel 9331336732593.369489 571.36 Ask_cancel 6761336732593.396394 571.37 Trade 4891336732593.403784 571.39 Bid_cancel 7801336732593.471040 571.48 Trade 4651336732593.485026 571.54 Bid_cancel 6681336732593.585481 571.55 Ask_cancel 8141336732593.699121 571.63 Ask_cancel 2861336732593.704077 571.74 Ask_add 4241336732593.820406 571.82 Ask_cancel 7891336732593.865808 571.88 Bid_cancel 2581336732593.912195 571.89 Bid_cancel 5791336732593.916676 571.91 Ask_add 2411336732593.941828 571.95 Bid_add 5281336732593.965397 571.99 Trade 300
• Liquid stocks >10 million messages per day
• NYSE produces ~1TB of market data per day
• Market data from all relevant exchanges is collected
© 2015 Think Big, a Teradata Company
MARKET DATA
time price order_flag size1336732593.051448 71.00 Ask_cancel 1081336732593.096281 71.06 Ask_add 9221336732593.138566 71.19 Bid_add 2301336732593.179509 71.26 Bid_add 7311336732593.249253 71.28 Trade 2801336732593.321581 71.33 Bid_cancel 9331336732593.369489 71.36 Ask_cancel 6761336732593.396394 71.37 Trade 4891336732593.403784 71.39 Bid_cancel 7801336732593.471040 71.48 Trade 4651336732593.485026 71.54 Bid_cancel 6681336732593.585481 71.55 Ask_cancel 8141336732593.699121 71.63 Ask_cancel 2861336732593.704077 71.74 Ask_add 4241336732593.820406 71.82 Ask_cancel 7891336732593.865808 71.88 Bid_cancel 2581336732593.912195 71.89 Bid_cancel 5791336732593.916676 71.91 Ask_add 2411336732593.941828 71.95 Bid_add 5281336732593.965397 71.99 Trade 300
© 2015 Think Big, a Teradata Company
Order Book
Spread
Cancel
Add
Priority
Price
Best Bid
Best Ask
Ask
Bid
Matching Engine
Exchange 1
ScoringPlatform
News, Twitter…
Compute Cluster, e.g.
Hadoop, Flat files
Data Science
Matching Engine
Exchange 2
ScoringPlatform
MarketData
Orders
Market & scoringdata, orders
Ingestion
Data
Data
Model deployment
Model storage
© 2015 Think Big, a Teradata Company
ARCHITECTURE
Questions© 2015 Think Big, a Teradata Company
?
WE ARE HIRING© 2015 Think Big, a Teradata Company