transitioning your architecture to scale
DESCRIPTION
TRANSCRIPT
![Page 1: Transitioning your architecture to scale](https://reader036.vdocuments.site/reader036/viewer/2022081602/54b6a3534a7959be738b4696/html5/thumbnails/1.jpg)
Transitioning your architecture to scale
David Tinker - CTO BrandsEye
@david_tinker#scaleconf
15 April 2013
![Page 2: Transitioning your architecture to scale](https://reader036.vdocuments.site/reader036/viewer/2022081602/54b6a3534a7959be738b4696/html5/thumbnails/2.jpg)
What is BrandsEye?
• We monitor online conversation (Twitter, Facebook, G+, websites, blogs, etc.) about brands (e.g. ABSA, Gautrain) and derive insights from the data
• We process between 1m and 3m tweets and other brand mentions per day
• Relevancy, sentiment analysis, country and language and other variables all automated and crowd sourced
• Clients use our web app to see the results and explore the data
![Page 3: Transitioning your architecture to scale](https://reader036.vdocuments.site/reader036/viewer/2022081602/54b6a3534a7959be738b4696/html5/thumbnails/3.jpg)
Simplified BrandsEye Schema
Account
urititleextractcountrylanguageetc.matched phrasesbrand sentiments
Mentionnameparent
Brand
querybrand
Search Phrase
![Page 4: Transitioning your architecture to scale](https://reader036.vdocuments.site/reader036/viewer/2022081602/54b6a3534a7959be738b4696/html5/thumbnails/4.jpg)
BrandsEye in June 2011• Single Java application (ear file) for everything• Mention collection, mention processing & client web app
• All accessing MySQL directly using Hibernate & JDBC
• Single MySQL server at Rackspace• Separate database per client account
• 2 app servers at Rackspace (Apache & JBoss)• Client web app load balanced
• Single instance for mention collection and processing
• Appropriate architecture when BE started (2006)• It was website mentions and not tweets back then and 20/day was a lot!
• Quick to get to market
![Page 5: Transitioning your architecture to scale](https://reader036.vdocuments.site/reader036/viewer/2022081602/54b6a3534a7959be738b4696/html5/thumbnails/5.jpg)
Problems in 2011
• Fragile• Any problem with mention processing or MySQL would stop mention
collection and lead to missed tweets (and grumpy clients)
• Re-deploying the app (e.g. for client web app change) would interrupt mention collection and processing
• Hard to recover from mention processing problems as mentions were not stored prior to processing
• Any change to any part of the application risked breaking some other part of it (no tests)
• Hard to change MySQL schema as all parts of the application needed to be changed and manually tested
• If our MySQL server dies we are down for a while and will lose data
• Slow• Everything used MySQL and our server (16 cores, 72G RAM) was taking
strain
• Poor latency - long time for a mention to appear in a client account
![Page 6: Transitioning your architecture to scale](https://reader036.vdocuments.site/reader036/viewer/2022081602/54b6a3534a7959be738b4696/html5/thumbnails/6.jpg)
Solution & Challenges
• Separate out and decouple mention collection, mention processing, the client app and the database
• Had to keep the business running at the same time
• Small team (1.5 - 4 developers) and limited budget so big bang new architecture not an option
=> Incremental approach which is still in progress!
The major components and technologies are described on the following slides with details on what worked and what didn’t work for us
![Page 7: Transitioning your architecture to scale](https://reader036.vdocuments.site/reader036/viewer/2022081602/54b6a3534a7959be738b4696/html5/thumbnails/7.jpg)
Simplified Architecture
ChickenMention Store
AnalyticsPublic API
Account Mention Data
PostgreSQL master+slave
BeefFeedproxy
Mention Collector
Redis Mongo
RabbitMQ
Account Mention Data
PostgreSQL master+slave
BrandsEyeJavascript app
JSONHTTP
PorkMention
ProcessingPipeline
Redis
MashAccount Meta
Data
PgSQL
3rd party apps
AMQP
JDBC
MentionsGravyThe BrandsEye
Crowd
MySQL
Mentions
AMQP
![Page 8: Transitioning your architecture to scale](https://reader036.vdocuments.site/reader036/viewer/2022081602/54b6a3534a7959be738b4696/html5/thumbnails/8.jpg)
Sunday Lunch
![Page 9: Transitioning your architecture to scale](https://reader036.vdocuments.site/reader036/viewer/2022081602/54b6a3534a7959be738b4696/html5/thumbnails/9.jpg)
Mention Flow
FeedproxyMention Collector RabbitMQ
PorkMention
ProcessingPipeline
ChickenMention Store
AnalyticsPublic API
Account Mention Data
PostgreSQL master+slave
GravyThe BrandsEye
Crowd
Rater (starving student)
![Page 10: Transitioning your architecture to scale](https://reader036.vdocuments.site/reader036/viewer/2022081602/54b6a3534a7959be738b4696/html5/thumbnails/10.jpg)
Feedproxy (Mention Collector)
• Java app deployed on 2 virtual servers• Collects mentions from many different sources using Redis sorted sets
for efficient polling and de-duping
• Buffers mentions in a MongoDB capped collection as JSON messages
• Writes them to a RabbitMQ queue on a remote server for processing
• Can replay mentions from a point in the past to recover from processing failures
• Has a web UI to display status and stats
![Page 11: Transitioning your architecture to scale](https://reader036.vdocuments.site/reader036/viewer/2022081602/54b6a3534a7959be738b4696/html5/thumbnails/11.jpg)
Feedproxy (2)
• Redis• Redis is a semi-persistent key/value store with sets, lists etc.
• Perfect for this application
• Uses clever fork trick with copy-on-write to save to disk periodically
• The data fits in memory easily and its not terribly bad if we lose the most recent 2 minutes or so
• Lightning fast and uses hardly any CPU
• As easy as using in memory data structures but the app continues where it left off after a re-deploy
• Have to watch out for “leaks” in Redis data structures
• Clustered version not available yet but you can do replication
![Page 12: Transitioning your architecture to scale](https://reader036.vdocuments.site/reader036/viewer/2022081602/54b6a3534a7959be738b4696/html5/thumbnails/12.jpg)
Feedproxy (3)
• MongoDB• JSON data store with indexing, querying and so on
• Uses memory mapped files for everything and relies on the OS
• Has capped collections (ring buffer) with fixed size
• Capped collection uses a lot of IO once it fills up even though we only use one index
• Gets “swapped out” so occasional retrieval of old mentions takes a long time - Mongo not so good on a machine doing other stuff as well
• “Expensive” way of buffering our mentions
• Have written a replacement but its not in production yet
![Page 13: Transitioning your architecture to scale](https://reader036.vdocuments.site/reader036/viewer/2022081602/54b6a3534a7959be738b4696/html5/thumbnails/13.jpg)
Feedproxy (4)
• RabbitMQ• RabbitMQ is a message broker
• You setup exchanges which distribute messages to queues for consumers to process
• Initially used it to buffer mentions on the mention collector server and a RabbitMQ shovel to get those to the RabbitMQ instance on the processing machine
• The shovel got “stuck” sometimes (every couple of weeks)
• Rabbit’s memory usage climbs linearly with the number of messages in its queues regardless of queue durability settings + it stops accepting messages when a “high watermark” of memory usage is reached
• Cannot “replay” mentions from a point in time in the past
• Not good as a durable store for messages
![Page 14: Transitioning your architecture to scale](https://reader036.vdocuments.site/reader036/viewer/2022081602/54b6a3534a7959be738b4696/html5/thumbnails/14.jpg)
Chicken API & Mention Store
ChickenMention Store
AnalyticsPublic API
Account Mention Data
PostgreSQL master+slave
BeefFeedproxy
Mention Collector
Redis Mongo
RabbitMQ
Account Mention Data
PostgreSQL master+slave
BrandsEyeJavascript app
JSONHTTP
PorkMention
ProcessingPipeline
Redis
MashAccount Meta
Data
PgSQL
3rd party apps
AMQP
JDBC
MentionsGravyThe BrandsEye
Crowd
MySQL
Mentions
AMQP
![Page 15: Transitioning your architecture to scale](https://reader036.vdocuments.site/reader036/viewer/2022081602/54b6a3534a7959be738b4696/html5/thumbnails/15.jpg)
Chicken API & Mention Store
• Provides a REST API to access mentions and analytics• Written using Grails, a Groovy+Java Ruby on Rails clone for Spring stack
• The only app with access to the account mention databases
• Translates our high level mention filter language into SQL
• Has good set of functional tests
• BrandsEye customers can use the API to build their own apps
• Supports multiple different mention stores with a single API
• We are busy transitioning from a single MySQL server to several PostgreSQL clusters
• Keeps stats in Redis
• Uses Apache SOLR for full text search (likely to be replaced with PostgreSQL)
• Stateless (will be load balanced soon)
![Page 16: Transitioning your architecture to scale](https://reader036.vdocuments.site/reader036/viewer/2022081602/54b6a3534a7959be738b4696/html5/thumbnails/16.jpg)
Chicken API & Mention Store• Online documentation
• We use semantic versioning• Major: Increment on breaking API changes
• Minor: Increment when new functionality added
• Patch: Increment for bug fixes
The Book of ChickenWelcome to the BrandsEye API, v1.17.10
https://api.brandseye.com/rest/accounts/BESC27AA/mentions?filter=Published inthelast month and Language isnt 'en'
select id, title, extract ... from mention join ... where published_date >= ? and
language <> ?Can also do groupby etc.
![Page 17: Transitioning your architecture to scale](https://reader036.vdocuments.site/reader036/viewer/2022081602/54b6a3534a7959be738b4696/html5/thumbnails/17.jpg)
Why PostgreSQL?• Synchronous replication since 9.1• Transaction on master only commits when slave has received the log
records
• Keeps pair of servers exactly in sync
• Replication done over dedicated gigabit network link
• Read-only queries can go to master or slave, writes only to master
• We wrote an app (running on 3rd machine) to monitor the pair and promote the slave / disable replication as needed (uses Hetzner failover IP addresses to make the dead machine inaccessible)
• MySQL now also has something similar but it all seems a bit ropey
• Other reasons• PostgreSQL has good full text search and arrays
• MySQL query planner doesn’t handle some simple subqueries + inexplicably fails to use indexes for others
• We need to stick with RDBMS as we do lots of ad-hoc queries
![Page 18: Transitioning your architecture to scale](https://reader036.vdocuments.site/reader036/viewer/2022081602/54b6a3534a7959be738b4696/html5/thumbnails/18.jpg)
Why Grails?• Quick to build apps• Groovy has syntax similar to Ruby and Python but runs on the JVM and
integrates seamlessly with Java code
• Most Java code is also valid Groovy code so great for a team with Java background
• Grails is very much like Rails but using familiar Java stuff (Spring, Hibernate etc.)
• Lots of plugins and they are easy to write
• Performance is available• Need something fast? Just write that little bit in Java
• We won’t have to switch stacks (e.g. from Ruby to Java) at some point for performance reasons
![Page 19: Transitioning your architecture to scale](https://reader036.vdocuments.site/reader036/viewer/2022081602/54b6a3534a7959be738b4696/html5/thumbnails/19.jpg)
Mash Account Meta Data
ChickenMention Store
AnalyticsPublic API
Account Mention Data
PostgreSQL master+slave
BeefFeedproxy
Mention Collector
Redis Mongo
RabbitMQ
Account Mention Data
PostgreSQL master+slave
BrandsEyeJavascript app
JSONHTTP
PorkMention
ProcessingPipeline
Redis
MashAccount Meta
Data
PgSQL
3rd party apps
AMQP
JDBC
MentionsGravyThe BrandsEye
Crowd
MySQL
Mentions
AMQP
![Page 20: Transitioning your architecture to scale](https://reader036.vdocuments.site/reader036/viewer/2022081602/54b6a3534a7959be738b4696/html5/thumbnails/20.jpg)
Mash Account Meta Data
• Provides a REST API for account meta data• Grails app using MySQL (moving to PostgreSQL)
• Brands, search phrases, processing rules etc.
• Notifies client applications of changes via RabbitMQ topic exchange
• Client apps typically cache the data until it changes or a timeout expires
• Most client apps use a Java library which handles the caching and maps the JSON to a data model
• Simple way to distribute the information to many de-coupled apps
![Page 21: Transitioning your architecture to scale](https://reader036.vdocuments.site/reader036/viewer/2022081602/54b6a3534a7959be738b4696/html5/thumbnails/21.jpg)
Pork Mention Processor
ChickenMention Store
AnalyticsPublic API
Account Mention Data
PostgreSQL master+slave
BeefFeedproxy
Mention Collector
Redis Mongo
RabbitMQ
Account Mention Data
PostgreSQL master+slave
BrandsEyeJavascript app
JSONHTTP
PorkMention
ProcessingPipeline
Redis
MashAccount Meta
Data
PgSQL
3rd party apps
AMQP
JDBC
MentionsGravyThe BrandsEye
Crowd
MySQL
Mentions
AMQP
![Page 22: Transitioning your architecture to scale](https://reader036.vdocuments.site/reader036/viewer/2022081602/54b6a3534a7959be738b4696/html5/thumbnails/22.jpg)
Pork Mention Processor• Consumes mentions (JSON messages) from
RabbitMQ queues• Grails app with basic UI for monitoring
• Annotates mentions with extra information (relevancy, sentiment, country, language etc.) using machine learning and other techniques
• Applies automated processing rules
• Sends & receives mentions from the BrandsEye crowd
• Writes mentions to Chicken (Mention Store)
• ACK mention if all good, otherwise NACK and re-process
• Lots of batching and shared models for performance and rate limiting reasons
• Groovy closures and other features result in compact maintainable code
• Small amount of performance centric code in Java + machine learning libraries
• Can process 1m+ mentions/hour, mostly waiting for Chicken
• Keeps stats in Redis
![Page 23: Transitioning your architecture to scale](https://reader036.vdocuments.site/reader036/viewer/2022081602/54b6a3534a7959be738b4696/html5/thumbnails/23.jpg)
RabbitMQ
• Good• ACK/NACK model is very convenient for development
• Can limit number of un-ACKed messages allowed to control app memory usage
• Exchange types and routing keys allow for flexible setup
• Easy to duplicate messages on the fly (e.g. for debugging in live)
• Nice admin console
• Limitations• Cannot cluster queues so even for clustered rabbit losing a machine
loses everything in its queues
• Memory usage climbs rapidly if you aren’t consuming messages
![Page 24: Transitioning your architecture to scale](https://reader036.vdocuments.site/reader036/viewer/2022081602/54b6a3534a7959be738b4696/html5/thumbnails/24.jpg)
Beef Client Web App
• Grails + Javascript using Backbone, Handlebars etc.• Communicates with Chicken using the same API we offer to clients
• Maintains brands, phrases etc. by communicating with Mash
• All REST using JSON
• Makes it possible for us to refactor all of the backend apps (e.g. change database schemas) so long as we keep the API the same
![Page 25: Transitioning your architecture to scale](https://reader036.vdocuments.site/reader036/viewer/2022081602/54b6a3534a7959be738b4696/html5/thumbnails/25.jpg)
Monitoring
• We use Wormly for server and app alerts
• Each app has a simple web console that can be checked for “ERROR”
• Logs are aggregated using Graylog2
![Page 26: Transitioning your architecture to scale](https://reader036.vdocuments.site/reader036/viewer/2022081602/54b6a3534a7959be738b4696/html5/thumbnails/26.jpg)
![Page 27: Transitioning your architecture to scale](https://reader036.vdocuments.site/reader036/viewer/2022081602/54b6a3534a7959be738b4696/html5/thumbnails/27.jpg)
SCM and Packaging• We use Git & Bitbucket• Same as Github but much cheaper for a small team with many repos
• Master must always be deployable, we don’t use branches much
• Apps are packaged as executable war files• Servlet container (Jetty) embedded in war
• java -jar crackling.war and it will come up listening on its port
• Built on the target machine (no CI yet ...)
• Apache or Nginx in front
• Good documentation• Each app has a README.md describing its purpose, how to build and
run/publish it, API endpoints, dependencies (and how to install them) etc.
![Page 28: Transitioning your architecture to scale](https://reader036.vdocuments.site/reader036/viewer/2022081602/54b6a3534a7959be738b4696/html5/thumbnails/28.jpg)
Hetzner & AWS vs Rackspace
• Price• Rackspace is at least 6x more expensive that Hetzner for a similar
(supposedly better quality) machine
• Rackspace cloud servers are also much more expensive than Amazon servers (6x or more) and they charge full price even when your server isn’t running
• So we are moving towards “more machines that can fail” instead of “a few really reliable machines”
• Other factors• Hetzner give failover IPs that you can change with an API for
implementing HA stuff
• Servers have 2 NICs so you can create private nets for replication
• Traffic is free (great for backups) up to 10000G/month
• We need physical hardware for at least the database servers for performance
![Page 29: Transitioning your architecture to scale](https://reader036.vdocuments.site/reader036/viewer/2022081602/54b6a3534a7959be738b4696/html5/thumbnails/29.jpg)
Tech Summary
• Redis• Fast solid software, lots of use cases for data sets that fit in memory
• Watch out for leaks
• MongoDB• Capped collections slower than you expect
• Really wants to be the only thing installed on the server
• MySQL• Not very good at optimizing queries or using indexes, dodgy replication
• PostgreSQL• Synchronous replication is cool
• Advanced query optimizer
• RabbitMQ• Great for short term routing of messages, watch out for memory usage
![Page 30: Transitioning your architecture to scale](https://reader036.vdocuments.site/reader036/viewer/2022081602/54b6a3534a7959be738b4696/html5/thumbnails/30.jpg)
Conclusion
• We can now easily scale BrandsEye to handle any number of clients and volume of mentions
• All of this is still in progress
• We have lots of other little apps not described here interacting using the same tech
• We have done a lot of work with chef for our new servers but its not handling everything yet
![Page 31: Transitioning your architecture to scale](https://reader036.vdocuments.site/reader036/viewer/2022081602/54b6a3534a7959be738b4696/html5/thumbnails/31.jpg)
Questions?
$100 discount to Scaleconf people!