trading up: adding flexibility and scalability to bouygues telecom with mongodb
DESCRIPTION
TRANSCRIPT
Adding Flexibility and Scalability to
Bouygues Telecom with MongoDB
Pierre-Alban DEWITTE@__pad__Bouygues TelecomMongoDB World – 06/24/2014
Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Bouygues Telecom
13 M Customers
Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Who is speaking ?
@__pad__
Thanks !
Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Agenda
Adding Flexibility and Scalability to Bouygues Telecom with MongoDB
1. Existing problems2. Design choices3. Build4. A few steps back
Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Agenda
1. Existing problems
Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Existing problems
ConsumerCustomers
Existing Sytem
CompagnyCustomers
LDAPaccess
Micropayment system
Identity and Oauth management system
Network Connexion system
ETL
Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Existing problems
Two years delay
Schema customization very hard to design and implementPerformance issues
Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Existing problems
Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Agenda
1. Existing problems
2. Design choicesA. MongoDBB. TomcatC. Storm
Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Design choices - MongoDB
Schema less data integrationHigh availabilityShardingLess expensive data storage allowing duplication
Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Design choices - MongoDB
Team with development and architecture skillsSchema design experienceCapability to train new actors
Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
POC between Tomcat and NodeJS
We choose our battle, MongoDB !
Design choices - Tomcat
Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Design choices - Storm
Distributed real-time computation system
Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Design choices - Storm
ConsumerCustomer
MongoDB database
CompagnyCustomers
REST Micropayment system
Identity and Oauth management system
Network Connexion system
API
STORM
Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Agenda
1. Existing problems2. Design choices
3. BuildA. Storm topologyB. Schema design and refactorC. DevOps
Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Storm topology
Customer files are pushed every night
Read file line by line
1
…DetectFile
SpoutProcessLine
Bolt
2
line
Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Storm topology
Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Storm topology
Emitting tuple for each set of functional
collections
ProcessLineBolt
4
OauthUpdaterBolt
IdentityUpdaterBoltTransform a line into a
document & check delta
3
Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Storm topology
Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Storm topology
Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Design choices - Storm
Message failure handle
ack(Object msgId) failed(Object msgId)
Take care of system stability
Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Design choices - Storm
We scaled and easily !
Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Build - Schema design and refactor
Conception User- Name- Compagny- Billcycle- Payment
Mode
Acces Point- Name- Compagny- Billcycle- Payment
Mode
1
1..n
Conception
Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Build - Schema design and refactor
Developm
ent
Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Build - Schema design and refactor
Qualification
Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Build - Schema design and refactor
Developm
ent
Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Build - Schema design and refactor
« That works »
Qualification
Conception
Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Build - Schema design and refactor
Stre
ss T
est
Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Build - Schema design and refactor
ConceptionConception
Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Build - Schema design and refactor
ConceptionConception
Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Build - Schema design and refactor
Developm
ent
Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Build - Schema design and refactor
Developm
ent
Qualification
Conception
Stre
ss T
est
Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Build - Schema design and refactor
PERFECT !
Developm
entConception
Stre
ss T
es
Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Build - DevOps
Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Build - DevOps
Tooling first
Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Build - DevOps
Moving the line between Dev and Ops
Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Build - DevOps
Internal training by development teamRedaction of a play-book dedicated to opsPre production tests played by both teamEarly write of stress tests, played during qualification tests
Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
6 months later
Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Agenda
1. Existing problems2. Design choices3. Iterations during build
4. A few steps back
Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
A few steps back
Apache Tomcat Apache STORM
Flexibility and Scale with a 100% Open Source Stack !
Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
A few steps back
1. Choose your battle2. Use the right tool for
the right job3. Be prepare to fail and
to rework4. To move organization,
training is a key
Thank you for listening
Questions ?
Further talks @__pad__
Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Photo credits100 m start : http://fr.wikipedia.org/wiki/100_m%C3%A8tres_%28athl%C3%A9tisme%29#mediaviewer/Fichier:Mens_100m_finals_British_Champs_and_Olympic_Trials.jpg by Paul Foot from Birmingham, UKQuestion mark block : https://www.flickr.com/photos/jarbo/9379813470 by Jared Cherup Pelleteuse orange : http://fr.123rf.com/photo_3669950_un-grand-pelleteuse-orange-stationne-a-un-chantier-de-construction.html by Stephen McsweenyPunaise : https://www.flickr.com/photos/24362608@N05/3501112978/sizes/l/in/photolist-f5R8g4-6kiVfR-6koahu-6ko7US-6ko8Qh-d9NUKU-nDi2eb-ffg13f-8h56wx-cgAcib-hchCtD-decZ4p-6kocrL-6kj2J6-5doxaw-gbTPmK-nCFBBq-672Snf-2sCzvw-e1zAjM-8voRbc-c9Rh1m-amFFGt-2iZNBq-cFSnFU-81Ckk4-2XHZAy-9Ggpfw-m8FfKz-8hW6r5-4xSuxC-awXsp4-5aVn7g-fbXVHZ-9GAPLX-9qr7uU-fcddgA-cP1EN7-np2q92-6V8oBT-cenSsb-cyzaEd-8h5a2F-6V2b3F-ch1RME-fJ52s3-9XthVN-kBs5mW-6qSPaF-5cA9Qb/ par dractrain94 Une longue vue qui louche : https://www.flickr.com/photos/la_bretagne_a_paris/3847733265/ par Yann Caradec NASA-Apollo8-Dec24-Earthrise : http://www.hq.nasa.gov/office/pao/History/alsj/a410/AS8-14-2383HR.jpg par NASA / Bill Anders