where does apache geode fit in cqrs architectures?
TRANSCRIPT
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
Where Does Apache Geode Fit in CQRS Architectures?
Eitan Suez@eitan_suez
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
About Me• Eitan Suez• Pivotal Consultant Instructor
• Teach GemFire, Cloud Native, PCF• Prior to joining Pivotal, was Principal Consultant with ThoughtWorks• Long-time software developer, based in Austin, TX
2
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
Relationship with Apache Geode• Over the years have worked on many enterprise projects for a number of
customers• First hands-on experience with Geode when consulting at SouthWest Airlines..• ..in the role of technical lead on a multi-team project, where Geode played a
prominent role in the system architecture
3
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
My Journey• gfsh• OQL and the data browser• PDX serialization• Spring Data GemFire• Learn how to do automated functional testing with it
4
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
Don’t immediately realize what you’ve got• At first, we were so focused on building features• Regions were already defined by solutions architects, treated them as tables• Didn’t pay too close attention to the fact that we had:
• near-linear scale-out capabilities built-in with partitioned regions• fault-tolerance with redundant data copies• locators adding indirection, clients isolated from cluster specifics
5
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
Example: Queries against partitioned regions
6
Can go further with server-side functions
Client
Geode Distributed System
Server
Query ExecutorPartitioned
Region
Server
Partitioned Region
Server
Partitioned Region
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
Unique Combination of Features• A Database, but in-memory?• Can also double as a simple cache?• A key-value store, but supports queries?• Supports transactions• Events?
7
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
Impressive feature set• Briefly reviewed the traits of Apache Geode• It takes time to “wrap one’s head around” the whole of this product
8
So, what can you do with it?
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
Use Cases “in the Small”• Specific to Java stack: O/RM and Hibernate• can plug in as Hibernate L2 Cache• Peer-to-peer configuration
9
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
Use Cases “in the Small”Can be an out-of-process cache server, like Redis, or memcached
gemcached
These are fine, but does not take advantage of the full feature set
10
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
Canonical Architecture
11
Geode Distributed System
RegionsFunctionsLocator
Backing Store
Client Client
Events, Continuous Queries
RegionsFunctions
CacheLoader AsyncEventListener
Server
RegionsFunctions
Client
Queries, Transactions, Function Executions
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
Switching Gears• On a couple of projects over the last couple of years, have been exposed to
CQRS• At first it seemed strange, or overly complex. Didn't get it• Kept asking myself:
• Why not start out simpler?• Seems rather complicated• It’s more work
12
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
What is CQRS?• Stands for Command Query Responsibility Segregation• A Pattern
• deliberately not prescriptive regarding how you implement this separation• Separation all the way down to the database
• Germ of the idea came from Bertrand Mayer (of Eiffel fame), with concept of CQS
• Introduced, proposed by Greg Young• Active .NET community, Udi Dahan (among others)
13
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
CQRS..
14
..tells you what,not how,but to answer why,we are asked to look at what happens when you “go there”
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 / 15
When reads and writes are separate..
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
..can optimize reads and writes• With a single schema, you’re forced to optimize for one at the expense of the
other• With two schemas, one can be optimized for reads and the other for writes
(have your cake and eat it too)• relational model for writes• denormalized views for reads
16
Read-Optimized Write-Optimized3rd normal formdenormalized views
Data-Representation Spectrum
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
Reading when your data is normalized
17
Controller
Services Repositories
Relational Database
Queries w/Joins
Request
transformations
compositions
Result Sets
viewsConstantly reassembling views
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
With a denormalized schema..• no joins necessary• no transformations• no need to reconstruct a view model for each request
18
Apache Geode
Region: Customers
Region: Orders
Region: Products . . .
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
..more benefits• Can scale reads and writes independently
• many systems have a profile where reads outnumber writes at 100:1 ratio• Read and write sides can be implemented with entirely different tools and
technologies• Read-side can stay up when write-side is temporarily down
19
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
Command Side• Commands are semantic, in the language of the business, not REST CRUD:
AddToCart, AddPaymentMethod, ChangeAddress• Command handling can be asynchronous
• enqueue commands• can scale command handling
20
See: Udi DahanClarified CQRS
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
Event Sourcing• The LOG• Append-only, no mutation• Immutable storage, doesn't destroy history• Activity just a stream of events
• tables are projections, can be derived entirely from log• views can be recreated at will• multiple views
21
See: Martin KleppmannStream processing, Event sourcing, Reactive, CEP … and making sense of it all
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
The Log / Table Duality• Data in Motion vs Data at Rest• Entire History vs Snapshot in time• Source of truth vs derived information
• materialized views, caches, indexes, aggregations
22
See: Jay KrepsThe Log: What every software engineer should know about real-time data's unifying abstraction
See: Martin KleppmannStream processing, Event sourcing, Reactive, CEP … and making sense of it all
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
Replaying the Log..• ..in test environments to reproduce bugs• ..in dev environments to test an upcoming release• ..in production to “undo” a bug• ..in production for blue-green type deployments
Can transition to a new schema/representation of data in your regions because you've come up with a radically different user interface for navigating that information.
23
See: Greg YoungCQRS and Event Sourcing
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 / 24
Diagram by“Exploring CQRS and Event Sourcing”, msdn
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 / 25
✓ update caches when new events come in✓ invalidate caches proactively - ensure data
in caches remain fresh✓ inverts the cache loader concept ✓ serving data from fast, in-memory caches✓ regions contain “ViewModel” objectsEvents
Projection Updates Views in Regions
Geode Distributed SystemRegions
containingView Models
Read Side - relay view models to ui - little to no transformations
Events, Continuous Queries
Queries
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
SummaryApache Geode as the read store in a CQRS system is a particularly good fit:
• eager cache invalidation• scalable and fast reads via..
• regions store denormalized views• partitioned regions enable linear scale-out• in-memory data supports low-latency reads
26
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
References & Attributions• Martin Kleppmann
Stream processing, Event sourcing, Reactive, CEP … and making sense of it all• Rx, Erik Meijer
Your Mouse is a Database• Greg Young
CQRS and Event Sourcing• Jay Kreps
The Log: What every software engineer should know about real-time data's unifying abstraction• Udi Dahan
Clarified CQRS• Dominic Betts, Julian Dominguez, Grigori Melnik, Fernando Simonazzi, Mani Subramanian
CQRS Journey• Dannielle Burrow
Four Real World Use Cases For An In-Memory Data Grid
27
Unless o therw ise ind ica ted , these s l ides are © 2013-2016 P ivo ta l So f tware , Inc . and l i censed under a Creat ive Commons At t r ibu t ion-NonCommerc ia l l i cense: h t tp : / / c rea t ivecommons.org / l i censes /by-nc /3 .0 /
Learn More. Stay Connected.
@springcentralspring.io/blog
@pivotalpivotal.io/blog
@pivotalcfhttp://engineering.pivotal.io