building a scalable system for shipping tracking

BUILDING A SCALABLE SYSTEM FOR SHIPPING TRACKING

PREMSHREE PILLAI,

Sunday, October 5, 14

Sunday, October 5, 14

Sunday, October 5, 14

SR. ENGINEER, SHIPPINGPREVIOUSLY, YAHOO!

@premshree

Sunday, October 5, 14

https://www.etsy.com/listing/110699162/seeker-of-happiness-canvas-tote-bag-you

RE-IMAGINE COMMERCEIN WAYS THAT BUILD A MORE FULFILLING AND LASTING WORLD

Sunday, October 5, 14

Sunday, October 5, 14

CLEVELAND, OHIO

https://www.etsy.com/listing/199433520/1893-antique-map-of-cleveland-ohio

Sunday, October 5, 14

Sunday, October 5, 14

HONG KONG

https://www.etsy.com/listing/199835452/hong-kong-china-morning-street-view

Sunday, October 5, 14

Sunday, October 5, 14

MALI PLAC NA TAVANU, ZAGREB

Sunday, October 5, 14

Sunday, October 5, 14

♥ POTTERY

Sunday, October 5, 14

Sunday, October 5, 14

SCOTTSDALE, AZ

Sunday, October 5, 14

ANDOVER, MA

Sunday, October 5, 14

LEFKIS, GREECE

Sunday, October 5, 14

https://www.etsy.com/listing/150742570/white-nesting-bowl-set-modern-ceramic

1 MILLION SELLERS40 MILLION MEMBERS26 MILLION ACTIVE LISTINGS

Sunday, October 5, 14

Sunday, October 5, 14

Sunday, October 5, 14

Sunday, October 5, 14

Sunday, October 5, 14

Sunday, October 5, 14

Where is myPACKAGE?

https://www.flickr.com/photos/oneeighteen/2597603677

Sunday, October 5, 14

Where is myPACKAGE?

https://www.flickr.com/photos/oneeighteen/2597603677

♥

Sunday, October 5, 14

IN TRANSIT 200,000+

https://www.etsy.com/listing/153268931/custom-world-map-absract-print-poster

Sunday, October 5, 14

OUT FOR DELIVERY 125,000+

https://www.etsy.com/listing/153268931/custom-world-map-absract-print-poster

Sunday, October 5, 14

SHIPPING CARRIERS 150+

https://www.etsy.com/listing/153268931/custom-world-map-absract-print-poster

Sunday, October 5, 14

SHIPPING CARRIERS

Sunday, October 5, 14

FedEx

DHL

USPS

Hrvatska pošta

...

Sunday, October 5, 14

Sunday, October 5, 14

LET’S DIVE IN

Sunday, October 5, 14

Sunday, October 5, 14

Sunday, October 5, 14

Sunday, October 5, 14

Sunday, October 5, 14

Sunday, October 5, 14

Sunday, October 5, 14

Sunday, October 5, 14

Sunday, October 5, 14

Sunday, October 5, 14

Sunday, October 5, 14

tracking_events

Sunday, October 5, 14

SHIPPING APIS

Sunday, October 5, 14

GET http://production.shippingapis.com/ShippingAPI.dll?API=TrackV2&XML=%3CTrackFieldRequest+USERID%3D%22022XXXXH6696%22%3E%3CTrackID+ID%3D%22LZ13XXX3064US%22%3E%3C%2FTrackID%3E%3C%2FTrackFieldRequest%3E

Sunday, October 5, 14

GET http://production.shippingapis.com/ShippingAPI.dll?API=TrackV2&XML=%3CTrackFieldRequest+USERID%3D%22022XXXXH6696%22%3E%3CTrackID+ID%3D%22LZ13XXX3064US%22%3E%3C%2FTrackID%3E%3C%2FTrackFieldRequest%3E

Sunday, October 5, 14

Sunday, October 5, 14

Sunday, October 5, 14

Sunday, October 5, 14

Sunday, October 5, 14

Sunday, October 5, 14

TRACKING STATES

Sunday, October 5, 14

Sunday, October 5, 14

Sunday, October 5, 14

Sunday, October 5, 14

Sunday, October 5, 14

INFO_RECEIVED

PACKAGE_RECEIVED

IN_TRANSIT

DELAYED

OUT_FOR_DELIVERY

...

DELIVERED

Sunday, October 5, 14

STATE MAP

Sunday, October 5, 14

INFO_RECEIVED

“acceptance”

“processed at usps origin facility"

“accepted at usps origin sort facility”

“picked up”

Sunday, October 5, 14

INFO_RECEIVED

“tendered at fedex office”

“picked up"

“in fedex possession”

Sunday, October 5, 14

IN_TRANSIT

“dispatched from usps sort facility”

“sorting complete"

“local holiday - will attempt delivery on next working day”

“customs clearance processing

Sunday, October 5, 14

SELECT * FROM trackings WHERE last_access_date < ?

Sunday, October 5, 14

SELECT * FROM trackings WHERE last_access_date < ?

Sunday, October 5, 14

? = now() - 15

Sunday, October 5, 14

Sunday, October 5, 14

WHAT’S THE PROBLEM?

Sunday, October 5, 14

SIZE

Sunday, October 5, 14

SELECT * FROM trackings WHERE last_access_date < ? LIMIT ?

Sunday, October 5, 14

STRAGGLERS

Sunday, October 5, 14

Sunday, October 5, 14

SHARDING 101

Sunday, October 5, 14

Sunday, October 5, 14

Sunday, October 5, 14

Sunday, October 5, 14

Sunday, October 5, 14

Sunday, October 5, 14

Sunday, October 5, 14

Sunday, October 5, 14

Sunday, October 5, 14

SELECT * FROM trackings WHERE shard = ? AND last_access_date < ?

Sunday, October 5, 14

Sunday, October 5, 14

QUEUE

Sunday, October 5, 14

FAUX QUEUE

Sunday, October 5, 14

Sunday, October 5, 14

QUEUE_3_DAY

QUEUE_1_DAY

QUEUE_3_HOUR

QUEUE_1_HOUR

...

QUEUE_STOPPED

Sunday, October 5, 14

QUEUE MAP

Sunday, October 5, 14

INFO_RECEIVED => QUEUE_1_DAY

PACKAGE_RECEIVED => QUEUE_1_DAY

IN_TRANSIT => QUEUE_1_DAY

OUT_FOR_DELIVERY => QUEUE_1_HOUR

...

DELIVERED => QUEUE_STOPPED

Sunday, October 5, 14

INFO_RECEIVED => QUEUE_3_HOUR

PACKAGE_RECEIVED => QUEUE_3_HOUR

IN_TRANSIT => QUEUE_3_HOUR

OUT_FOR_DELIVERY => QUEUE_1_HOUR

...

DELIVERED => QUEUE_STOPPED

Sunday, October 5, 14

QUEUE AGE

Sunday, October 5, 14

QUEUE_3_DAY => 1*60*60*24*3

QUEUE_1_DAY => 1*60*60*24

QUEUE_3_HOUR => 3*60*60

QUEUE_1_HOUR => 1*60*60

Sunday, October 5, 14

SELECT * FROM trackings WHERE shard = ? AND queue = ? AND last_access_date < ?

Sunday, October 5, 14

SELECT * FROM trackings WHERE shard = ? AND queue = ? AND last_access_date < ?

Sunday, October 5, 14

last_access_date < time() - age

Sunday, October 5, 14

ASYNC

Sunday, October 5, 14

Sunday, October 5, 14

IN SUMMARY

Sunday, October 5, 14

API

STATE MAP

SHARDING

CRON/ASYNC

FAUX-QUEUE

Sunday, October 5, 14

https://www.etsy.com/listing/200465416/mustache-card-youre-awesome

Sunday, October 5, 14

Hvala!https://www.etsy.com/listing/157215176/croatia-watercolor-painting-art-print

Sunday, October 5, 14

https://www.etsy.com/listing/192250796/i-mustache-you-a-question-embroidery-4

Sunday, October 5, 14