2015-08-22 stream billions of data to google big query
Post on 20-Feb-2017
325 Views
Preview:
TRANSCRIPT
teddy@aftership.com
Teddy ChanCEO& Co-Founder
Shipment tracking APISupporting 270 290+ carriers
Free Shipping APIsSingle API for 5 carriers
How do we find you?
Google BigQuery is great
- Charge only when you do query
- Super cheap in storage
- Super cheap in sending the data
Perfect for log everything
3 billions records / month
1,200 records / sec
API Log BigQuery
Goal
API Log BigQuery
It does not work!
- http RESTful- BigQuery API call limit
API Log Redis Streamer BigQuery
How do we solve?
Why Redis?
- TCP persistent connection- Fast, really FAST
What does the STREAMER do?
- Read data from redis- Send BATCH data to BQ- Pre / Post process data
Demo
http://www.slideshare.net/aftership
FAQ
OPEN SOURCEhttps://github.com/AfterShip/demo_streamer2bigquery
top related