hermes: free the data! distributed computing with mongodb
TRANSCRIPT
![Page 1: Hermes: Free the Data! Distributed Computing with MongoDB](https://reader036.vdocuments.site/reader036/viewer/2022062523/5877563d1a28ab84388b7509/html5/thumbnails/1.jpg)
Hermes: Free the Data!Distributed Applications with MongoDB
Presented by Warren Chang, VP Engineering @ Borderfree
![Page 2: Hermes: Free the Data! Distributed Computing with MongoDB](https://reader036.vdocuments.site/reader036/viewer/2022062523/5877563d1a28ab84388b7509/html5/thumbnails/2.jpg)
2
Hermes {hur'-meez}Hermès
Hermes 69230
![Page 3: Hermes: Free the Data! Distributed Computing with MongoDB](https://reader036.vdocuments.site/reader036/viewer/2022062523/5877563d1a28ab84388b7509/html5/thumbnails/3.jpg)
3
Iconic French Brand since 1837Known for luggage and handbags
![Page 4: Hermes: Free the Data! Distributed Computing with MongoDB](https://reader036.vdocuments.site/reader036/viewer/2022062523/5877563d1a28ab84388b7509/html5/thumbnails/4.jpg)
4
(hur'-meez)Greek God,Divine Messenger of the GodsLink between the mortals and Olympians
![Page 5: Hermes: Free the Data! Distributed Computing with MongoDB](https://reader036.vdocuments.site/reader036/viewer/2022062523/5877563d1a28ab84388b7509/html5/thumbnails/5.jpg)
5
HermesIt’s a conceptSimple, Persisted, Message Bus
![Page 6: Hermes: Free the Data! Distributed Computing with MongoDB](https://reader036.vdocuments.site/reader036/viewer/2022062523/5877563d1a28ab84388b7509/html5/thumbnails/6.jpg)
6
Event driven messaging Distributed Applications Flexible architecture Self-contained business rules
![Page 7: Hermes: Free the Data! Distributed Computing with MongoDB](https://reader036.vdocuments.site/reader036/viewer/2022062523/5877563d1a28ab84388b7509/html5/thumbnails/7.jpg)
7
Event driven messaging Distributed Applications Flexible architecture Self-contained business rules
![Page 8: Hermes: Free the Data! Distributed Computing with MongoDB](https://reader036.vdocuments.site/reader036/viewer/2022062523/5877563d1a28ab84388b7509/html5/thumbnails/8.jpg)
8
Event driven messaging Distributed Applications Flexible architecture Self-contained business rules
![Page 9: Hermes: Free the Data! Distributed Computing with MongoDB](https://reader036.vdocuments.site/reader036/viewer/2022062523/5877563d1a28ab84388b7509/html5/thumbnails/9.jpg)
9
Event driven messaging Distributed Applications Flexible architecture Self-contained business rules
![Page 10: Hermes: Free the Data! Distributed Computing with MongoDB](https://reader036.vdocuments.site/reader036/viewer/2022062523/5877563d1a28ab84388b7509/html5/thumbnails/10.jpg)
10
Lets take a step back
![Page 11: Hermes: Free the Data! Distributed Computing with MongoDB](https://reader036.vdocuments.site/reader036/viewer/2022062523/5877563d1a28ab84388b7509/html5/thumbnails/11.jpg)
11
Consulting
Publishing
Ad Tech
![Page 12: Hermes: Free the Data! Distributed Computing with MongoDB](https://reader036.vdocuments.site/reader036/viewer/2022062523/5877563d1a28ab84388b7509/html5/thumbnails/12.jpg)
12
![Page 13: Hermes: Free the Data! Distributed Computing with MongoDB](https://reader036.vdocuments.site/reader036/viewer/2022062523/5877563d1a28ab84388b7509/html5/thumbnails/13.jpg)
currencies60+
countries100+
customers170+
2014 GMV$550M
+
Borderfree helps online retailers go global
![Page 14: Hermes: Free the Data! Distributed Computing with MongoDB](https://reader036.vdocuments.site/reader036/viewer/2022062523/5877563d1a28ab84388b7509/html5/thumbnails/14.jpg)
14
2012Tech Stack Diversity
Java
![Page 15: Hermes: Free the Data! Distributed Computing with MongoDB](https://reader036.vdocuments.site/reader036/viewer/2022062523/5877563d1a28ab84388b7509/html5/thumbnails/15.jpg)
15
2013Tech Stack Diversity
JavaPHP
![Page 16: Hermes: Free the Data! Distributed Computing with MongoDB](https://reader036.vdocuments.site/reader036/viewer/2022062523/5877563d1a28ab84388b7509/html5/thumbnails/16.jpg)
16
2014Tech Stack Diversity
JavaPHPnode.jspythonscala
![Page 17: Hermes: Free the Data! Distributed Computing with MongoDB](https://reader036.vdocuments.site/reader036/viewer/2022062523/5877563d1a28ab84388b7509/html5/thumbnails/17.jpg)
17
THE CHALLENGE
![Page 18: Hermes: Free the Data! Distributed Computing with MongoDB](https://reader036.vdocuments.site/reader036/viewer/2022062523/5877563d1a28ab84388b7509/html5/thumbnails/18.jpg)
18
The Challenge J2EE Best Practices Enterprise Application Server
Weblogic Single Database (Oracle) Hibernate ORM 3 digit growth YOY Speed to market mentality Data Lock-in
![Page 19: Hermes: Free the Data! Distributed Computing with MongoDB](https://reader036.vdocuments.site/reader036/viewer/2022062523/5877563d1a28ab84388b7509/html5/thumbnails/19.jpg)
19
WHAT DO WE DO?
![Page 20: Hermes: Free the Data! Distributed Computing with MongoDB](https://reader036.vdocuments.site/reader036/viewer/2022062523/5877563d1a28ab84388b7509/html5/thumbnails/20.jpg)
20
The ChallengeWhat to do about the data? Decouple Stream Persist Monitor
![Page 21: Hermes: Free the Data! Distributed Computing with MongoDB](https://reader036.vdocuments.site/reader036/viewer/2022062523/5877563d1a28ab84388b7509/html5/thumbnails/21.jpg)
21
SOLUTIONS
![Page 22: Hermes: Free the Data! Distributed Computing with MongoDB](https://reader036.vdocuments.site/reader036/viewer/2022062523/5877563d1a28ab84388b7509/html5/thumbnails/22.jpg)
22
What we loved about it: Great messaging platform. Highly scalable Reliable
What didn’t work for us: Lack of Persistence Administrative overhead Additional Infrastructure Complex message handling
![Page 23: Hermes: Free the Data! Distributed Computing with MongoDB](https://reader036.vdocuments.site/reader036/viewer/2022062523/5877563d1a28ab84388b7509/html5/thumbnails/23.jpg)
23
What we loved about it: Highly Scalable Durable Messages Extendibility
What didn’t work for us: Expertise It is HARD! Additional Infrastructure Support
![Page 24: Hermes: Free the Data! Distributed Computing with MongoDB](https://reader036.vdocuments.site/reader036/viewer/2022062523/5877563d1a28ab84388b7509/html5/thumbnails/24.jpg)
24
Features Known & supported
infrastructure Highly Scalable Fault Tolerant Simple & Flexible
Caveats Initial sizing & design is
critical
MongoDBComponents: Capped Collection Tail-able Cursor Replica set Flexible Schema Aggregation Framework
![Page 25: Hermes: Free the Data! Distributed Computing with MongoDB](https://reader036.vdocuments.site/reader036/viewer/2022062523/5877563d1a28ab84388b7509/html5/thumbnails/25.jpg)
25
Hermes
![Page 26: Hermes: Free the Data! Distributed Computing with MongoDB](https://reader036.vdocuments.site/reader036/viewer/2022062523/5877563d1a28ab84388b7509/html5/thumbnails/26.jpg)
26
Hermes Breakdown{ Capped Collections }http://docs.mongodb.org/v2.6/core/capped-collections/
Highlights:• FIFO Queue• Fixed Size Collection• Guarantee sort by insertion order
($natural)• Tail-able Cursor support!• Very little support overhead
{Code}• use hermes
db.createCollection(“orders”, {‘capped’:true, ‘size’:4000000000})
![Page 27: Hermes: Free the Data! Distributed Computing with MongoDB](https://reader036.vdocuments.site/reader036/viewer/2022062523/5877563d1a28ab84388b7509/html5/thumbnails/27.jpg)
27
Hermes Breakdown{ JSON }Basic Schema
{_id: ObjectId()typ: (msg type: String),dt: MongoDate() data: {
/* flexible json object data here */ }
}
![Page 28: Hermes: Free the Data! Distributed Computing with MongoDB](https://reader036.vdocuments.site/reader036/viewer/2022062523/5877563d1a28ab84388b7509/html5/thumbnails/28.jpg)
28
Hermes Breakdown{ Tailable Cursor }http://docs.mongodb.org/manual/tutorial/create-tailable-cursor/
Highlights:• Non-exhausting cursor• Query-’less’ feed of data
{code}Python
cursor = self.coll.find(tailable=True, await_data=True)Or
cursor = self.coll.find({'_id':{'$gt’:<objectId>}}, tailable=True, await_data=True)
![Page 29: Hermes: Free the Data! Distributed Computing with MongoDB](https://reader036.vdocuments.site/reader036/viewer/2022062523/5877563d1a28ab84388b7509/html5/thumbnails/29.jpg)
29
Hermes Breakdown{ Replica Set }http://docs.mongodb.org/manual/core/replication/
Highlights:• Low overhead replication• Scalable redundancy• Optimize read/write efficiency
![Page 30: Hermes: Free the Data! Distributed Computing with MongoDB](https://reader036.vdocuments.site/reader036/viewer/2022062523/5877563d1a28ab84388b7509/html5/thumbnails/30.jpg)
30
Simple Example
![Page 31: Hermes: Free the Data! Distributed Computing with MongoDB](https://reader036.vdocuments.site/reader036/viewer/2022062523/5877563d1a28ab84388b7509/html5/thumbnails/31.jpg)
31
# setup Mongo Connection try: self.conn = pymongo.MongoClient(host=servers) self.db = self.conn[db] self.coll = self.db[coll] self.startPoint = startpoint except Exception as e: self.logger.error(e) self.logger.error(str(self.__class__.__name__) + " :: Connection to Database Failed!") exit(1)
def run(self): self.logger.info(str(self.__class__.__name__) + " starting tailable cursor.") if self.startPoint != None: # start from last queue position cursor = self.coll.find({'_id':{'$gt':self.startPoint}}, tailable=True, await_data=True) else: # start from top of queue cursor = self.coll.find(tailable=True, await_data=True) while cursor.alive: try: data = cursor.next() rec = data self.logger.debug("updt:" + rec['data']['UPDATED_DATE'] + " ordr:" + rec['data']['ORDER_ID'] + " objid: " + str(rec['_id'])) except StopIteration: self.logger.info(str(self.__class__.__name__) + " waiting ...")
![Page 32: Hermes: Free the Data! Distributed Computing with MongoDB](https://reader036.vdocuments.site/reader036/viewer/2022062523/5877563d1a28ab84388b7509/html5/thumbnails/32.jpg)
32
Core System Migration
![Page 33: Hermes: Free the Data! Distributed Computing with MongoDB](https://reader036.vdocuments.site/reader036/viewer/2022062523/5877563d1a28ab84388b7509/html5/thumbnails/33.jpg)
33
![Page 34: Hermes: Free the Data! Distributed Computing with MongoDB](https://reader036.vdocuments.site/reader036/viewer/2022062523/5877563d1a28ab84388b7509/html5/thumbnails/34.jpg)
34
Questions?