hbasecon 2015- hbase @ flipboard
TRANSCRIPT
HBASE @ FLIPBOARD
What is Flipboard?
Publications
People
Topics
Scale
100+ million users
250,000+ new users/day
15+ million magazines
Why HBase?• Transferrable Hadoop operational expertise
• MapReduce!
• Write throughput
• Better elasticity than MySQL, our other primary data store
• Strong consistency
• Column-oriented, as opposed to simple K/V
What do we use it for?
• User-generated magazines, likes, comments
• Vanity metrics for those magazines- daily + all-time counters/HLLs
• Follow graph
• RSS feeds
• More and more every day…
Magazine Storage• Stored in a single HBase table
• Magazines live in one column family (“magazine”)
• Articles in temporal order in another CF (“article”)
• Logically, everything shared is tagged with magazine ID (prefix compression helps here)
• Makes the calculation of everything a user shared efficient
User Magazines
magazine:<magazineid> magazine:<magazineid> magazine:<magazineid>
sha1(userid) MagazineData (serialized JSON) MagazineData (serialized JSON) MagazineData (serialized JSON)
Magazine CF of Collection Table Listing magazines a user has created is a single read
Data is stored in serialized JSON for language interoperability but is parsed and serialized by plain old java objects
User Magazines
[Reverse Unix TS]:[magazineid]
[Reverse Unix TS]:[magazineid]
sha1(userid) Article Data(Serialized JSON)
Article Data(Serialized JSON)
Articles CF of Collection Table Kept in temporal order so that most recently shared articles are first
Access patterns are usually newest first. HBase Filters are used to slice wide rows.
User Magazines
like:[userid] reflip:[new article id] comment:[timestamp][userid]
Article ID JSON (who/when) JSON (where it was
reflipped) JSON (comment/person)
Social Activity
One cell per like, since you can only do it once per user Can be many comment and reflip cells by one user per article
Alternative orderings can be computed from Elasticsearch indexes
User Magazines
magazine:<magazineid> contributor:<magazine>:<userid>
sha1(userid) JSON metadata JSON metadata
Multiple Contributors Magazine CF contains magazines that user can share into.
Contributor CF contains user’s magazines that others are allowed to share into.
User Magazines
<metric>:<day> <metric>_count
magID long count for day alltime count
Per magazine metrics live in Stats CF
Atomic increments for counters, both a per day count and a total count: Total Articles Contributors
etc.
User Magazines
<unique>:<day> <unique>_count
magID HLL for individual day Premerged HLL
Unique readers kept in each magazine’s row as a serialized HyperLogLog
Allows for merging unique data over day ranges or displaying all time
Social Graph
follow:userid follower:userid stats:<counter>
sha1(userid) JSON person that I follow
JSON person that follows me
long count of followers/following
Stored in friends table, follower/followers/stats CFs; metadata in MySQL
Alternative indexes in Elasticsearch
HBase Table Access Patterns• Tables optimized for application access patterns (“design for the questions, not the
answers”)
• Fetching an individual magazine- collection table, magazine CF, [magazine ID] -> cell
• Fetching an individual article - article table, article:[article ID] cell
• Fetch an article’s stats - article table, article:stats cells
• Fetching a magazine’s articles: collection table, article CF, with cell limit and column qualifier starts with magazine id
• Fetching a user’s magazines- collection table, magazine CF, [magazine ID] in the CQ
Client Stats
• Articles: sum(magazine stats:article_count for each magazine)
• Magazines: count(collection:magazine) cells • Followers: friends:stats:follower_count +
sum(magazine stats:subscriber_count for each magazine)
More Client Stats
• Summary stats use counters from the article table, detailed stats (who liked the article?) read cells
• We can cache the feed of items, but the stats/like state is calculated per user
• likes: article:stats:like_count • reflips: article:stats:reflip_count • comments: article:stats:comment_count
Even More Client Stats
AsyncHBase Usage• Our fork adds column filters on wide rows- we’d like to get these upstream
• Stats requests require scatter/gather reads for several tables, sometimes over multiple HBase clusters
• HBaseAsync requests are grouped into a single Deferred
• Most requests are a get on a single row, no multi row scans
• Most requests wait once until the results are returned or a deadline expires
• If data is returned late or HBase regions are not available, partial calculations are allowed (we just display the stats we’ve got)
Handling HBase Failures• Most patterns are read before write which causes early failure
• We can tolerate some data loss (atomic increments, vanity stats)
• Individual servers track inflight requests to HBase, slow puts and gets, and report to Graphite
• Various levels of caching allow HBase recovery/region reassignment without end users noticing
• Read Only mode - writes are stopped at the application layer
• Ability to switch to replica under duress
Current HBase Fleet
• 15 clusters
• ~100 tables
• ~250TB in HDFS
• ~250 RegionServers
• Busiest clusters: 100,000+ qps, 1000 regions
HBase Fleet, continued
• All in EC2 😳
• Nothing in VPC, yet
• Each cluster lives within an AZ
• 1 durable cluster doing cross-AZ HBase-level replication
• 1 cluster running Stargate (it works, but we’re not in love with it)
HBase History at FlipboardOldest current production instances launched in 2011
(This cluster is going away soon 😀)
HBase Version Distribution• 0.90: dwindling, thankfully
• 0.94: Moved to Snappy; Stargate cluster for RSS storage; Python writers, Go readers
• 0.96: First CDH5/Java 7/Ubuntu Precise clusters; magazines live in one of these
• 0.98: pre-calculated user homefeeds, more
• 1.0, 1.1: Soon…
Which instance types?• Started off with m1.xlarges for the 1.6TB of ephemeral (spinning)
disk; when we started using HBase, AWS didn’t have SSDs
• Moved to hi1.4xlarges (16 cores, 60GB RAM, 2x1TB local SSD)
• Moved to i2s (next-gen SSD instances, made for databases) as soon as AWS let us launch them!
• ❤ i2s; some 2x, some 4x
AWS tips• Use instance storage, not EBS
• Rely on HDFS to keep your bits replicated instead of using EBS
• Cross-AZ latency is minimal, but traffic is expensive!
• Push HDFS snapshots to S3 (we trigger this from Jenkins)
• If you jack up your network-related timeouts to handle AWS’ network flakiness, your MTTR rises, so be careful…
• Upgrade often, you’ll get more sleep!
Clients
• Java, Scala- AsyncHBase, which we love. Added column filtering for wide rows.
• Python + Go: protobufs over HTTP via Stargate, which works
• We use HAProxy everywhere, so we use that to load balance requests to Stargate servers
What’s next for HBase at Flipboard?• Moar HBase
• 1.0, now that CDH has it in 5.4.0; 1.1 when CDH gets it, hopefully soon…
• Region replicas (HBASE-10070) will help with use cases that can tolerate timeline consistency; 1.1 will have many improvements here
• Compaction throttling! (HBASE-8329)
• Java 8 + G1, Ubuntu Trusty, 3.13 kernel
• EC2 placement groups + VPC enhanced networking, once we’re in (no charge for these)
• HTrace (we use a little Zipkin, would love to get more HBase visibility)
• Multitenancy improvements in Apache HBase will help us put more customers on a cluster
Wish List $
HydraBase! (HBASE-12259)
Async client in Apache HBase so it keeps pace (HBASE-12684 is a start!)
Native Go client for 0.96+
Thanks!
• Matt Blair (@mb on Flipboard, @mattyblair on Twitter)
• Jason Culverhouse (@jsonculverhouse on both)
• Sang Chi (@sangchi on Flipboard, @sandbreaker on Twitter)