cassandra day sv 2014: building a flexible, real-time big data applications platform on apache...
DESCRIPTION
The talk is about the process of adding support for Cassandra in Kiji, our open-source platform for building big-data applications. I start off by describing the Kiji project, how it enables folks to build big-data applications, and (hopefully) get everyone excited about it. Then I talk about the Kiji data model, its origins in HBase (we initially built Kiji on top of HBase), how we updated it to also support Cassandra, what some if the issues were, etc. I get into some detail about our use of the Java driver and its async API, how we translate operations in Kiji into CQL statements, and some enhancements we've made to the Hadoop InputFormat and OutputFormat. I think this talk will be interesting to folks in general, and in particular will be useful for anyone who has an HBase background and is now working with Cassandra. The Kiji Project is a modular, open-source framework that enables developers to efficiently build real-time Big Data applications. Kiji is built upon popular open-source technologies such as Cassandra, HBase, Hadoop, and Scalding, and contains components that implement functionality critical for Big Data applications, including the following: Support for evolvable schemas of complex data types Batch training of machine learning models with Hadoop Real-time scoring with trained models Integration with Hive and R A REST endpoint Recently, we have updated Kiji to use Cassandra as a backing data store (previously, Kiji worked only with HBase). In this talk, we describe the process of integrating Cassandra and Kiji. Topics we cover include the following: The Kiji architecture and data model Implementing the Kiji data model in Cassandra using the Java driver and CQL3 Integrating Cassandra with Hadoop 2.x Building a flexible middleware platform that supports Cassandra and HBase (including projects that use both simultaneously) Exposing unique features of Cassandra (e.g., variable consistency) to Kiji usersTRANSCRIPT
Building a Flexible, Real-time Big Data Applications Platform
on Cassandra with Kiji
Cassandra Day Silicon Valley07 April 2014
Clint KellyMember of Technical StaffWibiData
1
Overview
• The Kiji Project• The Kiji data model and KijiSchema• Mapping Kiji to Cassandra• Status and future work• Try it now!
2
Should there be any intro page that talks about WibiData anywhere?
The Kiji Project
3
4
!
Want to build this...
Have this...
5
!
Want to build this...
!
Have this...
6
Want to build this...
Open source components
• Batch processing– Extract, transform, load– Train machine learning models
• Scalable storage– Time-series data
• Serialization– Complex data types
7
Hadoop, C*, HBase, Avro
KijiSchema
KijiMR KijiREST
KijiHive KijiScoring
KijiExpress
KijiSchema
• Schemas and data serialization• Complex, atomic data types
8
Hadoop, C*, HBase, Avro
KijiSchema
KijiMR KijiREST
KijiHive KijiScoring
KijiExpress
record UserLog { long timestamp; int user_id; string url; long session_id;}
• Schema evolution• Table metadata
Kiji batch components
• Scala DSL ➔ describe MapReduce computations
• Machine learning library• Hive adapter
9
Hadoop, C*, HBase, Avro
KijiSchema
KijiMR KijiREST
KijiHive KijiScoring
KijiExpress
Kiji real-time components
• REST server• Scoring server
10
Hadoop, C*, HBase, Avro
KijiSchema
KijiMR KijiREST
KijiHive KijiScoring
KijiExpress
Kiji Summary
• Bridge between open-source technologies and real-time, big data applications
• Users are building real systems with Kiji now!– Personalized recommendation systems for retail– Energy usage and analytics reporting
11
The Kiji data model and KijiSchema
12
row
13
Table are composed of rows.
entity ID data
14
We call row keys “entity IDs.”
data0xfa “bob”
15
We support composite entity IDs (with hashed and unhashed components).
info0xfa “bob” songs
16
Data in rows is organized into “column families.”
songs:let it be
songs:help
songs:helterskelter
0xfa “bob” info:email
info:payment
17
Column families contain columns, named as “family:qualifier.”
songs:let it be
songs:help
songs:helterskelter
0xfa “bob” info:email
info:payment songs:
let it besongs:let it besongs:
let it besongs:let it be
1396560123
18
Individual columns can have many different timestamped versions.
songs:let it be
songs:help
songs:helterskelter
0xfa “bob” info:email
info:payment songs:
let it besongs:let it besongs:
let it besongs:let it be
1396560123
19
Data values can be complex records
record SongPlay { long song_id; int user_rating; long session_id; device_type device;}
20
Locality groups
Separate logical organization of data (column families) from physical attributes (caching, compression, etc.)
info songs_todayentity ID songs_prev_year
21
Locality groups
Separate logical organization of data (column families) from physical attributes (caching, compression, etc.)
Need this data ASAP for real-time scoring. Use this data only for
batch jobs.
info songs_todayentity ID songs_prev_year
info songs_todayentity ID songs_prev_year
“real_time” (in-memory, uncompressed, TTL = 1 day)
“batch” (compressed, TTL = 12mo)
22
Locality groups
Always refer to columns by logical name (“family:qualifier”).
Need this data ASAP for real-time scoring. Use this data only for
batch jobs.
KijiSchema summary
• Data model similar to Cassandra, HBase, BigTable
• Contains time dimension (not present in C*)• Logical and physical organization separate• Complex schemas with Avro
23
Mapping Kiji to Cassandra
24
Implementation notes
25
• Built for Cassandra 2.0.6+• Native protocol / Java driver (no Thrift)• Asynchronous API• Assume users have Hadoop, ZooKeeper
Mapping a Kiji table ➔ Cassandra
• Locality group ➔ Table• Entity ID ➔ Primary key
– Hashed components ➔ partition key– Unhashed components ➔ clustering columns
• Family, qualifier, timestamp ➔ clustering columns• Cell values ➔ blobs
26
songs:let it be
songs:help
songs:helterskelter
0xfa “bob” info:email
info:payment songs:
let it besongs:let it besongs:
let it besongs:let it be
1396560123
CQL for Kiji locality groupCREATE TABLE users_locality_group_fast ( userid bigint, user text, family text, qualifier text, timestamp bigint, value blob, PRIMARY KEY (userid, username, family, qualifier, timestamp)) WITH CLUSTERING ORDER BY ( username ASC, family ASC, qualifier ASC, timestamp DESC);
27
TODO: Show row diagram, arrows pointing to components?
28
cqlsh:kiji_music>SELECT * FROM kiji_table_users;
userid | username | family | qualifier | timestamp | value--------+----------+--------+----------------+-----------+--------------- 0xfa | bob | info | email | 139653249 | 1243970104327 0xfa | bob | songs | abbey road | 139656012 | 0981274331032 0xfa | bob | songs | help | 139625013 | 9074132704129 0xfa | bob | songs | help | 139621359 | 1923079210370 0xfa | bob | songs | help | 139625013 | 4745018223497 0xfa | bob | songs | helter skelter | 139621324 | 7710423974234
Physical organization of data on disk
29
songs:let it be
songs:help
songs:helterskelter
0xfa “bob” info:email
info:payment songs:
let it besongs:let it besongs:
let it besongs:let it be
13965601230xfa:bob:info:email:t0:[email protected]
0xfa:bob:info:payment:t1:AMEX1234...
0xfa:bob:songs:let it be:t5:...
0xfa:bob:songs:let it be:t4:…
0xfa:bob:songs:let it be:t2:…
0xfa:bob:songs:help:t2:…
0xfa:bob:songs:helter skelter:t1:…
Efficient queries = continuous scans!
Kiji queries ➔ CQL queries
All data in “info” column family for “bob” ➔SELECT qualifier, value FROM music WHERE userid=0xfa AND user=‘bob’ AND family=‘info’;
30
songs:let it be
songs:help
songs:helterskelter
0xfa “bob” info:email
info:payment songs:
let it besongs:let it besongs:
let it besongs:let it be
1396560123
Kiji queries ➔ CQL queries
Data in “info:email” and last play of “help” for “bob” ➔
SELECT value FROM music WHERE userid=0xfa AND user=‘bob’ AND family=‘info’ AND qualifier=‘email’;
SELECT value FROM music WHERE userid=0xfa AND user=‘bob’ AND family=‘songs’ AND qualifier=‘help’ LIMIT 1;
31
songs:let it be
songs:help
songs:helterskelter
0xfa “bob” info:email
info:payment songs:
let it besongs:let it besongs:
let it besongs:let it be
1396560123
Kiji queries ➔ CQL queries
All songs played by “bob” on April 2nd ➔SELECT qualifier, value FROM music WHERE userid=0xfa AND user=‘bob’ AND family=‘songs’ AND timestamp >= 1396396800 AND timestamp <= 1396483200 ALLOW FILTERING;😱😱
32
songs:let it be
songs:help
songs:helterskelter
0xfa “bob” info:email
info:payment songs:
let it besongs:let it besongs:
let it besongs:let it be
1396560123
Kiji queries ➔ CQL queries
33
songs:let it be
songs:help
songs:helterskelter
0xfa “bob” info:email
info:payment songs:
let it besongs:let it besongs:
let it besongs:let it be
1396560123
!Bad Request: PRIMARY KEY part timestamp cannot be restricted (preceding part qualifier is either not restricted or by a non-EQ relation)
Queries that do not map well to CQL
• Break up into multiple CQL queries– Hooray for Session#executeAsync!
• Filter on the client– Potentially very expensive, but functional– Provide warning to user
• Educate users about table layout– Layout in previous example is terrible for that query
• Most issues related to “time” dimension
34
MapReduce
• Wrote new InputFormat, OutputFormat• Hadoop 2.x• Multiple C* queries per RecordReader• Does not use Thrift
35
Project status and next steps
36
Initial release in ~ 2 weeks
37
• Cassandra as part of the Bento Box• Cassandra working in KijiSchema, KijiMR
Support in the coming months
• Cassandra integration with KijiREST, KijiScoring, KijiExpress, etc.
• Expose Cassandra-specific features to users– Variable consistency levels– Load-balancing policies– Diagnostics (e.g., route tracing)
• Kiji support in CQLSH– Decode Avro values
38
Thanks to Cassandra community
• Great help on mailing lists for users, dev, java driver
• Webinars, meetups, C* Summit all available online
• Free training from DataStax• Very easy to get up-to-speed
39
Try it now -- Kiji Bento Box
• Latest compatible versions of all components• Hadoop, ZooKeeper, HBase• Cassandra in ~2 weeks
40
www.kiji.org/getstarted
Mention hiring?
KijiSchema
• Schemas and data serialization• Complex data types (e.g.,
nested maps)• Schema evolution• Metadata• Composite row keys• Transparent paging• Data-definition language, REPL
41
Hadoop, C*, HBase, Avro
KijiSchema
KijiMR KijiREST
KijiHive KijiScoring
KijiExpress
42
Schema support
Support for complex schemas with Avrorecord UserLog { long timestamp; int user_id; string url;}
KijiSchema allows schema versioning
43
Column name translation
• “family:qualifier” -> “A:B”• Saves disk space• Improves performance• User-facing tools translate names• Possible to turn this off
Kiji queries ➔ CQL queries
All data in family “songs” for user “bob” ➔SELECT qualifier, value FROM music WHERE userid=0xfa AND user=‘bob’ AND family=‘songs’;
44
songs:let it be
songs:help
songs:helterskelter
0xfa “bob” info:email
info:payment songs:
let it besongs:let it besongs:
let it besongs:let it be
1396560123