operationalizing mongodb at aol
DESCRIPTION
MongoDC 2012TRANSCRIPT
Michael DelNegroPrincipal Database Administrator
AOL
“Operationalizing”
MongoDB@AOL
About Me
• DBA at AOL (Dulles) for 6+ years• Original DBMS Background is in Sybase• Now MySQL, MongoDB, PostgreSQL and
NoSQL (Neo4J, Redis, etc)• Check out www.meetup.com/NOVA-
MySQL
The First Mongo at AOL• A MapQuest Developer in the
Summer of 2010 Replaced a MySQL Datastore with Mongo
• That Developer left in early 2011 :-/• Was Handed Over to Operations and
Yours Truly
Never Forget Your First• Not Small: 40+ Hosts• Upgrade Challenges• Connection Floods• Support to the Rescue• Magnified the Reasons to
Operationalize
Operation “Operationalize”
Operation “Operationalize”
• Take Advantage of 10Gen Support• Establish Standards• Monitoring/Alerting• Backups• Information Sharing• Challenges
10Gen Support
• JIRA Documents Cases Enabling Information Sharing and Referencing
• Serves Us Well in Both “Emergencies” and “Hey Quick Question”
• Training Classes: Administration & Development
Standard Host Setup
• CentOS 6• XFS/EXT4• Deadline I/O Scheduler• dirty_ratio = 10 (down from 40)• dirty_background_ratio = 5 (down
from 10)• noatime
Standard Build Scripts
• Use 64bit Linux Binaries Stored in AOL Repository
• mongodb_install.sh– Parameters Include:• MongoDB Version• Replica Set Name• Sharded? (Y/N)• Port (27017 is standard)• Type (DB, Arbiter, Config, Mongos)
Standard Directory/Layouts
• Based on our standard MySQL layout• Binaries in /opt/local/pkg• /db/mongodb/<port> base directory
– Ex. /db/mongodb/27017– Can Override
• /db/mongodb/27017-repset1
• Under base directory– data– etc (config files)– logs– scripts (ex stop/start)
ORB
• AOL Technologies’ Configuration Management Database (CMDB)
• Integrated with many authoritative data repositories
• Unique namespace for many operations data points
• Data model for operations management• Projects, Assets, People, Applications,
Network Data, HCM• SQL Interface
Monitoring/Alerting
• Argus• Nagios• Component Test Tool• MMS (MongoDB Monitoring Service)
Argus
• Metric and Event– Collector– Thresholds –Management– Data Viewing
Argus
• Capture serverStatus Metrics (once a minute)• Capture Replication Lag• Great for Trending• Great for Capacity Planning• Great for Troubleshooting• We Also Use for Host Metrics (CPU, I/O, etc)• Administrated By a Small and very Busy Group– Requested Additions/Changes Can Be Slow
Nagios
• Great For Fault-detection/Alerting• Great For Show Me What Is Currently
Broken• Great For Service Availability Metrics• Flexible• Reduces Pressure On NOC• Integrates With Netcool, Ignore Tool• We Write Our Own Plug-Ins
Component Test Tool
• Runs Parallel HTTP GETs Across a Group of Hosts
• Matches Regular Expressions On the Page Content (or HTTP Headers)
• Displays Number (or Content) of Matches in a HTML Table
MMS
Backups
• Scripts Delivered With Install• Choice Between LVM (default) and
Standard (mongodump)• Ability to Backup Sharded Setup
Mongo @ AOL Currently
Mongo @ AOL Currently
• 30+ Active MongoDB Projects• 500+ Servers in Production• Developed in:
– Java– PHP– Ruby– C#– Erlang– Scala– Perl– C++
Information Sharing
• Internal Wikis/Websites• UnUniversity (Organized Brown Bags)• Architecture Reviews• 10Gen JIRA Access
Challenges
• Developers Love It • BUT Requires More Upfront Planning
Than They May Realize• Should NOT Be Our Default Datastore• Better Developer OnBoarding• Aggressive Upgrading Is Required• Coordination of Driver Upgrades
The Future
• “MongoDB As a Service”• Puppet or Chef• Internal Administration Console– Already has Visual Topologies
• Internal MongoDB Operations Class– First One is in July
Resources
• Recommended Presentation: Todd Dampier’s Rock-Solid Mongo Ops (mongolab)
• Check out Jon Reed’s MongoDB for Online Advertising at AOL (MongoDB UK 2012)
• www.meetup.com/Washington-DC-MongoDB-Users-Group/
Thank You!
• www.slideshare.net/radiocats• @radiocats• www.linkedin.com/in/mdelnegro• Thank you, SeniorGif