server discovery and monitoring with mongodb
TRANSCRIPT
2
MongoDB
MongoDB Query Language (MQL) + Native Drivers
MongoDB Document/JSON Data Model
WiredTiger MMAP
Man
agem
ent
Sec
urity
In-memory Encrypted 3rd party
Shared Clusters
Replica Sets
10
Sharded Cluster
Driver
Mongod Mongod
Mongod
Mongod Mongod
Mongod
Mongod Mongod
Mongod
mongos mongos
11
Driver Responsibilities
https://github.com/mongodb/mongo-python-driver
Driver
Authentication& Security Python<->BSON Error handling &
Recovery
WireProtocol
Topology Management Connection Pool
12
Driver Responsibilities
https://github.com/mongodb/mongo-python-driver
Driver
Authentication& Security Python<->BSON Error handling &
Recovery
WireProtocol
Topology Management Connection Pool
13
Example API Callsimport pymongoclient = pymongo.MongoClient( host=“localhost”, port=27017)database = client[ ‘test_database’ ]collection = database[ ‘test_collection’ ]
collection.insert_one({ "hello" : "world" , "goodbye" : "world" } )
collection.find_one( { "hello" : "world" } )
collection.update({ "hello" : "world" }, { "$set" : { "buenos dias" : "world" }} )
collection.delete_one({ "hello" : "world" } )
15
Client Side View
Secondaryhost2
Secondaryhost3
Primaryhost1
MongoClient
MongoClient( "host1, host2", replicaSet="replset" )
16
Client Side View
Secondaryhost2
Secondaryhost3
Primaryhost1
MongoClient
MonitorThread 1
MonitorThread 2
{ ismaster : False, secondary: True, hosts : [ host1, host2, host3 ] }
17
What Does ismaster show?
>>> pprint.pprint( db.command( "ismaster" )){u'hosts': [u'JD10Gen-old.local:27017', u'JD10Gen-old.local:27018', u'JD10Gen-old.local:27019'], u'ismaster' : False, u'secondary': True, u'setName' : u'replset',…}>>>
19
Client Side View
Secondaryhost2
Secondaryhost3
Primaryhost1
MongoClient
MonitorThread 1
MonitorThread 2 ✔
20
Client Side View
Secondaryhost2
Secondaryhost3
Primaryhost1
MongoClient
MonitorThread 1
MonitorThread 2 ✔
MonitorThread 3
21
Client Side View
Secondaryhost2
Secondaryhost3
Primaryhost1
MongoClient
MonitorThread 1
MonitorThread 2 ✔
MonitorThread 3
YourCode
22
Next Is Insert
c = MongoClient( "host1, host2", replicaSet="replset" )client.db.col.insert_one( { "a" : "b" } )
23
Insert Will Block
Secondaryhost2
Secondaryhost3
Primaryhost1
MongoClient
MonitorThread 1
MonitorThread 2 ✔
MonitorThread 3
YourCode
Insert
24
ismaster response from Host 1
Secondaryhost2
Secondaryhost3
Primaryhost1
MongoClient
MonitorThread 1
MonitorThread 2 ✔
MonitorThread 3
YourCode
Insert
ismaster
25
Now Write Can Proceed
Secondaryhost2
Secondaryhost3
Primaryhost1
MongoClient
MonitorThread 1
MonitorThread 2 ✔
MonitorThread 3
YourCode
Insert
✔
Insert
26
Later Host 3 Responds
Secondaryhost2
Secondaryhost3
Primaryhost1
MongoClient
MonitorThread 1
MonitorThread 2 ✔
MonitorThread 3
YourCode
✔
✔
27
Steady State
Secondaryhost2
Secondaryhost3
Primaryhost1
MongoClient
MonitorThread 1
MonitorThread 2 ✔
MonitorThread 3
YourCode
✔
✔
28
Life Intervenes
Secondaryhost2
Secondaryhost3
Primaryhost1
MongoClient
MonitorThread 1
MonitorThread 2 ✔
MonitorThread 3
YourCode
✔
✖
29
Monitor may not detect
Secondaryhost2
Secondaryhost3
Primaryhost1
MongoClient
MonitorThread 1
MonitorThread 2 ✔
MonitorThread 3
YourCode
✔
✖
Insert
ConnectionFailure
30
So Retry
Secondaryhost2
Secondaryhost3
MongoClient
MonitorThread 1
MonitorThread 2 ✔
MonitorThread 3
YourCode
✔
✖
Insert
31
Check for Primary
Secondaryhost2
Secondaryhost3
MongoClient
MonitorThread 1
MonitorThread 2 ✔
MonitorThread 3
YourCode
✔
✖
Insert
32
Host 2 Is Primary
Primaryhost2
Secondaryhost3
MongoClient
MonitorThread 1
MonitorThread 2 ✔
MonitorThread 3
YourCode
✔
✖
Insert
33
Steady State
Secondaryhost2
Secondaryhost3
Primaryhost1
MongoClient
MonitorThread 1
MonitorThread 2 ✔
MonitorThread 3
YourCode
✔
✔
34
What Does This Mean? - Connect
import pymongo
client = pymongo.MongoClient()
try: client.admin.command( "ismaster" )except pymongo.errors.ConnectionFailure, e : print( "Cannot connect: %s" % e )
35
What Does This Mean? - Queries
import pymongo
def find_with_recovery( collection, query ) : try:
return collection.find_one( query )
except pymongo.errors.ConnectionFailure, e :
logging.info( "Connection failure : %s" e ) return collection.find_one( query )
36
What Does This Mean? - Inserts
def insert_with_recovery( collection, doc ) : doc[ "_id" ] = ObjectId() try: collection.insert_one( doc ) except pymongo.errors.ConnectionFailure, e: logging.info( "Connection error: %s" % e ) try: collection.insert_one( doc ) except DuplicateKeyError: pass
39
connectTimeoutMS
Secondaryhost2
Secondaryhost3
MongoClient
MonitorThread 1
MonitorThread 2 ✔
MonitorThread 3
YourCode
✔
✖
Insert
connectTimeoutMS
40
serverTimeoutMS
Secondaryhost2
Secondaryhost3
MongoClient
MonitorThread 1
MonitorThread 2 ✔
MonitorThread 3
YourCode
✔
✖
Insert
serverTimeoutMS
41
More Reading
• The spec author Jess Jiryu Davis has a collection of links and his better version of this talkhttps://emptysqua.re/blog/server-discovery-and-monitoring-in-mongodb-drivers/
• The full server discovery and monitoring spec is on GitHubhttps://github.com/mongodb/specifications/blob/master/source/server-discovery-and-monitoring/server-discovery-and-monitoring.rst
43
insert_one
• Stages– Parse the parameters– Get a socket to write data on– Add the object Id– Convert the whole insert command and parameters to a SON object– Apply the writeConcern to the command– Encode the message into a BSON object– Send the message to the server via the socket (TCP/IP)– Check for writeErrors (e.g. DuplicateKeyError)– Check for writeConcernErrors (e.g.writeTimeout)– Return Result object
44
Bulk Insert
bulker = collection.initialize_ordered_bulk_op()bulker.insert( { "a" : "b" } )bulker.insert( { "c" : "d" } )bulker.insert( { "e" : "f" } )try: bulker.execute()except pymongo.errors.BulkWriteError as e : print( "Bulk write error : %s" % e.detail )
45
Bulk Write
• Create Bulker object• Accumulate operations• Each operation is created as a SON object• The operations are accumulated in a list• Once execute is called
– For ordered execute in order added– For unordered execute INSERT, UPDATEs then DELETE
• Errors will abort the whole batch unless no write concern specified