© Man 2015

Building a time series database… 10^12 rows and counting

James Blackburn

@jimmybb

@ManAHLTech

2

Agenda

Data Storage At AHL1. AHL2. Data: size and shape3. Implementation: Arctic4. Performance5. Conclusion

3

AHL Systematic Fund Management

© Man 2015

4© Man 2015

AHL Systematic Fund Management

5© Man 2015

AHL Systematic Fund Management

6© Man 2015

AHL Systematic Fund Management

Quant researchers • Interactive work – latency sensitive• Batch jobs run on a cluster – maximize throughput• Historical data• New data• ... want control of storing their own data

Trading system• Auditable – SVN for data• Stable• Performant

7

Overview – Data consumers

8

AHL’s Data Pipeline

…[2-2092492678] FFIM6.NaE 2015-11-11 13:38:18.330 4 UPDATE TIMACT: '13:38', BID: 6192.0, QUOTIM: '13:38:18', QUOTIM_MS: 49098000.0, BIDSIZE: 1.0, EXCHTIM: '13:38:18'[2-2092492759] FFIM6.NaE 2015-11-11 13:38:18.676 16 UPDATE TIMACT: '13:38', ASKSIZE: 1.0, QUOTIM: '13:38:18', QUOTIM_MS: 49098000.0, ASK: 6200.5, EXCHTIM: '13:38:18'[2-2092493019] FFIM6.NaE 2015-11-11 13:38:20.333 14 UPDATE TIMACT: '13:38', BID: 6192.5, QUOTIM: '13:38:20', QUOTIM_MS: 49100000.0, BIDSIZE: 1.0, EXCHTIM: '13:38:20'[2-2092493079] FFIM6.NaE 2015-11-11 13:38:20.685 2 UPDATE TIMACT: '13:38', ASKSIZE: 1.0, QUOTIM: '13:38:20', QUOTIM_MS: 49100000.0, ASK: 6201.0, EXCHTIM: '13:38:20'…

9

Tick Data

© Man 2015

10

TimeSeries – single stock

© Man 2015

Data sizes we deal with…• ~1MB x 1000s 1x a day price data 10k rows (30 years)• ~0.5GB x 1000s 1-minute data 4M rows (20 years)• ~1GB x 1000s 10k x 10k data matrices 100M cells (30 years)• ~30TB Tick data 100k msgs/s

1.2B msgs/day

… and different shapes• Time series of prices• Event data• News data• Metadata• What’s next?

11

Data sizes

12

lib.read(‘US Equity Adjusted Prices')Out[4]: <class 'pandas.core.frame.DataFrame'>DatetimeIndex: 9605 entries, 1983-01-31 21:30:00 to 2014-02-14 21:30:00Columns: 8103 entries, AST10000 to AST9997dtypes: float64(8631)

Problems - Scale

13

lib.read(‘US Equity Adjusted Prices')Out[4]: <class 'pandas.core.frame.DataFrame'>DatetimeIndex: 9605 entries, 1983-01-31 21:30:00 to 2014-02-14 21:30:00Columns: 8103 entries, AST10000 to AST9997dtypes: float64(8631)

Equity Prices: 77M float64s 600MB of data ~= 5Gbits! 600 MB

Problems - Scale

14

SQL

© Man 2015

cx_Oracle

~ 200us per Row.

10k rows => 2.2s

15

SQL

© Man 2015

cx_Oracle

~ 200us per Row.

10k rows => 2.2s

How do we query 4 million rows?

Many different existing data stores• Relational databases• Tick databases• Flat files • HDF5 files• Caches

16

Overview – Databases

Many different existing data stores• Relational databases• Tick databases• Flat files • HDF5 files• Caches

17

Can we build one system to rule them all?

Overview – Databases

18

Implementation

https://github.com/manahl/arctic/

Requirements

• Easy to use – and we mean easy

• Fast – as fast as local files

• Scalable – unbounded in data-size and number of clients

• Agile – any data shape; new shapes; iterative development

• Complete – all data behind the simple API

19

Project Requirements

Goals• 20 years of 1 minute data in <1s• 200 instruments x all history x once a day data <1s

• Single data store for all data types• 1x day data Tick data

• Data versioning + Audit

20

Project Goals

Data bucketed into named Libraries• One minute• Daily• User-data: jbloggs.EOD• Metadata Index

Pluggable Library types:• VersionStore• TickStore• Pickle Store• … pluggable …

https://github.com/manahl/arctic/blob/master/howtos/how_to_custom_arctic_library.py

21

Arctic Libraries

22© Man 2015

23© Man 2015

24© Man 2015

Document ~= Python Dictionary / Java HashMap

Flexible schema Rapid prototyping

OpenSource database

Great support

#1 NoSQL DB (#4 overall) http://db-engines.com/en/ranking

25

Why MongoDB

Arctic key-value store

26

from arctic import Arctic

a = Arctic('research') # Connect to the data store

a.list_libraries() # What data libraries are availablelibrary = a[‘jbloggs.EOD’] # Get a Librarylibrary.list_symbols() # List symbols

library.write(‘SYMBOL’, <TS or other data>) # Writelibrary.read(‘SYMBOL’, version=…) # Read, with an optional version

library.snapshot('snapshot-name') # Create a named snapshot of the libraryLibrary.list_snapshots()

https://github.com/manahl/arctic/blob/master/howtos/how_to_use_arctic.py

Arctic API

27© Man 2015

28

Arctic - TickStore

Arctic(‘localhost’).initialize_library(‘tickdb’, ‘TickStoreV3’)

29

Implementation – slicing a chunk

© Man 2015

30

Implementation – a chunk

{ ID: ObjectId('52b1d39eed5066ab5e87a56d'), SYMBOL: 'symbol' INDEX: Binary(‘…, datetime, …', 0), COLUMNS: { ASK: { DATA: Binary(‘<compressed>', 0), DTYPE: '<f8', ROWMASK: Binary('...', 0) }, ... } START: DateTime(2015-01-01), END: DateTime(2015-11-12), SEGMENT: 1386933906826L, SHA: 1386933906826L, VERSION: 3,}

31

Implementation – TickStore

Sym1

Sym2

32© Man 2015

33

Arctic - VersionStore

Arctic(‘localhost’).initialize_library(‘library’)

34

Implementation – VersionStore

Snap A

Snap B

Sym1, v1

Sym1, v3

Sym2, v4

Sym2, v5

© Man 2014 35

Architecture – final system

ReutersR

MD

S M

essa

ge B

us

Bloomberg

Banks

Kafka Queue

Kafka Queue

16 (micro-)shard cluster

Master + 1 replicaLinux

8 cores256 GB RAM

96TB Disk

Infiniband network LZ4 compressed data

MongoDB Cluster

-> Arctic -> -> Arctic ->

36

Performance

Flat files on NFS – Random market

37

Results – Performance Once a Day Data

HDF5 files – Random instrument

38

Results – Performance One Minute Data

Random E-Mini S&P contract from 2013

© Man 2013 39

Results – TickStore

40

Results – TickStore II

© Man 2015 40

Infinibandsaturated

25x greater tick throughput

With just 2 machines!

Random E-Mini S&P contract from 2013

41

Results – System Load

OtherTick Mongo (x2)N Tasks = 32

42

Performance II

43

TickStore message input

© Man 2015

44

TimeSeries Query Throughput

© Man 2015

Low latency:- 1xDay data: 4ms for 10,000 rows (vs. 2,210ms from SQL) - OneMinute / Tick data: 1s for 3.5M rows Python (vs. 15s – 40s+ from OtherTick)- 1s for 15M rows Java

Parallel Access:- Cluster with 256+ concurrent data access- Consistent throughput – little load on the Mongo server

Efficient:- 10-15x reduction in network load- Negligible decompression cost (lz4: 1.8Gb/s)

45

Conclusion

46

The Future

© Man 2015

- Python 3 support- Mac Support- VersionStore write performance

- OpenSource other native clients- JavaScript- Java- C#

- Contributions Welcome!

47

Questions

@ManAHLTech

@jimmybbJames Blackburn

We’re hiring!recruitment@ahl.com?