introducing apache geode and spring data gemfire
TRANSCRIPT
2© 2014 Pivotal Software, Inc. All rights reserved. 2© 2014 Pivotal Software, Inc. All rights reserved.
Introducing Apache Geode with Spring Data GemFireJohn Blum / Luke Shannon
24 June 2015
3© 2014 Pivotal Software, Inc. All rights reserved.
Agenda
What is Apache Geode? Apache Geode Technical Specification High-level Implementation Details Spring Data GemFire Q&A
4© 2014 Pivotal Software, Inc. All rights reserved.
Pivotal GemFireWhat is Apache Geode?
In a nutshell…– Open Source core of Pivotal GemFire– Apache Incubator
▪ https://wiki.apache.org/incubator/GeodeProposal
5© 2014 Pivotal Software, Inc. All rights reserved.
What is Apache Geode?One definition is…A distributed, in-memory compute and data management platform that elastically scales to achieve high-throughput, low-latency access to big, fast data in order to power business critical, analytical applications in real-time.
Nodes
Ops / SecLinear scalability
Elastic capacity +/-
Latency optimizeddata distribution
6© 2014 Pivotal Software, Inc. All rights reserved.
What is Apache Geode? Technically…
– Distributed, in-memory database (IMDG) with ACID properties
– Transactional Key/Value Store (Cache)
– Glorified-version of ConcurrentHashMap
– Message Bus with guaranteed message delivery▪ Reliable, asynchronous event notifications
7© 2014 Pivotal Software, Inc. All rights reserved.
What Apache Geode Does? Enables new or existing Java applications to operate at cloud-scale
with predictable, low-latencies and be highly-available without sacrificing consistency.
Applications can transact and analyze Big Data in real-time to garner insights and achieve meaningful and impactful business results.
App
App
App
App
8© 2014 Pivotal Software, Inc. All rights reserved.
Apache Geode Characteristics In-Memory
Distributed
Consistent
Highly-Available / Fault Tolerant
High-Throughput / Low, Predictable Latency
High Scalability
Data-aware Parallel Compute
9© 2014 Pivotal Software, Inc. All rights reserved.
How are Apache Geode’s features implemented?
10© 2014 Pivotal Software, Inc. All rights reserved.
Memory Geode stores data in memory…
– JVM Heap– Off-Heap (Native Memory)
Geode (optionally) stores data to disk– Persistence + Overflow
11© 2014 Pivotal Software, Inc. All rights reserved.
ManagementMemory
Eviction (LRU)
Expiration (Region & Entry TTL/TTI)
Automatic resource management (critical & eviction Heap %)
Region Compression (Snappy)
JVM GC Tuning
12© 2014 Pivotal Software, Inc. All rights reserved.
IMDG An In-Memory Data Grid (IMDG), such as Apache Geode…
– Pools system resources across multiple nodes in a cluster to manage both application state and behavior.
– Memory, CPU, Network and (optional) Disk
13© 2014 Pivotal Software, Inc. All rights reserved.
High-Throughput / Low-Latency Geode uses concurrent, in-memory data structures with a highly
optimized distribution infrastructure to minimize context switching and contention.– synchronous/asynchronous replication for high read throughput
– partitioning for both high read and write throughput
14© 2014 Pivotal Software, Inc. All rights reserved.
Shared-Nothing Disk Persistence Architecture Manages data in local disk files on each node in the cluster
independently from other nodes– Members are self-sufficient; Members do not share memory or disk;– Data/Transactions appended to Oplog files– Compaction
Design goals (all about In-Memory)…– Minimize IO operations (Write-Behind vs. Write-Through)– Minimize locking– Minimize serialization
15© 2014 Pivotal Software, Inc. All rights reserved.
REPLICATE & PARTITION RegionsHigh Availability
Data is partitioned across nodes in the cluster and distributed uniformly– Rebalanced as nodes join/leave– Redundant copies
16© 2014 Pivotal Software, Inc. All rights reserved.
Persistence – Shared NothingServer 3Server 2
Bucket 3*primary*
Bucket 1secondary
Bucket 3secondary
Bucket 2*primary*
Server 1
Bucket 2secondary
Bucket 1*primary*
17© 2014 Pivotal Software, Inc. All rights reserved.
Persistence – Shared NothingServer 1 Server 3Server 2
Bucket 2secondary
Bucket 1*primary*
Bucket 3*primary*
Bucket 1secondary
Bucket 3secondary
Bucket 2*primary*
What if a server crashes?
18© 2014 Pivotal Software, Inc. All rights reserved.
Persistence – Shared NothingServer 1 Server 3Server 2
Bucket 2secondary
Bucket 1*primary*
Bucket 3*primary*
Bucket 1*primary*
Bucket 3secondary
Bucket 2*primary*
Bucket 2secondary
Primary Failover
RestoreRedundancy
Bucket 1secondary
19© 2014 Pivotal Software, Inc. All rights reserved.
Network Partitions
20© 2014 Pivotal Software, Inc. All rights reserved.
Partition RegionsConfigurable Consistency
Redundant Copies – 0 to 3
Colocation
Total Number of Buckets– 113 (Default)
Local Max Memory– 90% of Heap
Total Max Memory– All Members, Both Primary/Secondary
Copies
21© 2014 Pivotal Software, Inc. All rights reserved.
REPLICATE vs. PARTITIONConfigurable Consistency
Partition Regions– Updates routed through primary bucket– Holds lock on key when distributing updates
Replicate Regions– Possible conflicts due to out-of-order updates– Conflict checking (--concurrency-checks-enabled)
ReplicatedPartitioned Disabled
22© 2014 Pivotal Software, Inc. All rights reserved.
Apache Geode Topologies
23© 2014 Pivotal Software, Inc. All rights reserved.
Peer-to-Peer (P2P)Apache Geode Topologies
Configuration with the Cache embedded in the application process– Close proximity of application logic and data
24© 2014 Pivotal Software, Inc. All rights reserved.
Client/ServerApache Geode Topologies
Single-Hop Data Access
Continuously load balanced between servers to get the best response-times
`
Automatic fail-over when servers crash or become unresponsive
25© 2014 Pivotal Software, Inc. All rights reserved.
WANApache Geode Topologies
Site-to-Site Replication for DR
26© 2014 Pivotal Software, Inc. All rights reserved.
Use CasesApache Geode
Persistent OLTP/OLAP– System of Record (SOR)
Caching with JSR-107 (JCache) or Spring Cache Abstraction HTTP Session State Management
– Caching and Replication
Distributed L2 Caching for Hibernate Memcached Server (Gemcached)
27© 2014 Pivotal Software, Inc. All rights reserved.
Additional FeaturesApache Geode
Functions– Application logic is distributed, routed and run in parallel with close proximity to the data required by the
application.
Serialization (PDX)
Delta-propagation – difference of new/old value
JTA-compliant Transactions
Security – SSL, Auth
28© 2014 Pivotal Software, Inc. All rights reserved.
RoadmapApache Geode
HDFS persistence Off-Heap storage Lucene search/indexing Spark integration Cloud Foundry service
29© 2014 Pivotal Software, Inc. All rights reserved.
InterfaceApache Geode
Language Support– Native: Java, C/C++, C#– Non-native: JavaScript, Python,
Ruby, Scala, etc
API– Java HashMap (Region)– OQL / Function– Memcache client– REST API– Spring Data GemFire
Data Structure– User-defined Classes / POJOs– Portable Data Exchange (PDX) – Documents (JSON)
Powerful Data Access– Object Query Language (OQL)– Data-aware, Scatter-Gather
Functions– Publish/Scribe framework for RI/CQs
with reliable, async event queues
30© 2014 Pivotal Software, Inc. All rights reserved. 30© 2014 Pivotal Software, Inc. All rights reserved.
Spring Data GemFirehttp://projects.spring.io/spring-data-gemfire/
31© 2014 Pivotal Software, Inc. All rights reserved.
So… Why Spring?
32© 2014 Pivotal Software, Inc. All rights reserved.
“Simple things should be simple; complex things should be possible”
– Alan Kay
33© 2014 Pivotal Software, Inc. All rights reserved.
Spring Data GemFire (SDG)Consistently applies Spring's powerful, non-invasive programming model to simplify configuration and development with Apache Geode.
Spring Ecosystem Integration…– Spring Cache Abstraction / Transaction Management– Spring Data Commons + REST– Spring Integration (Inbound/Outbound Channel Adapters)– Spring XD (Input Source and Sinks)
34© 2014 Pivotal Software, Inc. All rights reserved.
Apache Geode combined with Spring Data GemFire can be used as JSR-107 (JCache) caching provider
+ +
35© 2014 Pivotal Software, Inc. All rights reserved.
GRAILS
Full-stack, Web
XD
Stream, Taps, Jobs
BOOT
Bootable, Minimal, Ops-Ready
Big, Fast,
FlexibleData Web,
Integration,Batch
WEB
Controllers, REST,WebSocket
INTEGRATION
Channels, Adapters,Filters, Transformers
BATCH
Jobs, Steps,Readers, Writers
BIG DATA
Ingestion, Export,Orchestration, Hadoop
DATA
NON-RELATIONALRELATIONAL
CORE
GROOVYFRAMEWORK SECURITY REACTOR
36© 2014 Pivotal Software, Inc. All rights reserved.
Use Cases / ConfigurationsSpring Data GemFire
Apache Geode Configuration / Bootstrapping– Replacement for cache.xml; Can be used with Cluster Configuration
Application Peer (Cache) Application Client (ClientCache)
37© 2014 Pivotal Software, Inc. All rights reserved.
Examples…SDG-based Configuration
38© 2014 Pivotal Software, Inc. All rights reserved.
with SpringBootstrapping Apache Geode
@SpringBootApplication@ImportResource("/spring-data-gemfire-cache.xml")@EnableGemfireRepositories@EnableTransactionManagementpublic class SampleDataGemFireApplication { public static void main(String[] args) { SpringApplication.run(SampleDataGemFireApplication.class, args); }}
Or…gfsh>start server –name=Example … --spring-xml-location=“<classpath-to-spring-application-context.xml>”
39© 2014 Pivotal Software, Inc. All rights reserved.
Basic (Region) Data AccessData Access with Spring
@Repositorypublic class CustomerDao {
@Resource(name = “Example”) private Region<?, ?> example; …}
Example…
40© 2014 Pivotal Software, Inc. All rights reserved.
GemfireTemplateData Access with Spring Data GemFire
Advantages…– Simple, Convenient Data Access (CRUD, OQL, Function)– Protects developer from GemFire/Geode API changes – Exception Translation into Spring DAO Exception Hierarchy– Transaction Management
<bean name=“customersTemplate” class=“org.springframework.data.gemfire.GemfireTemplate” p:region-ref=“Customers”>
41© 2014 Pivotal Software, Inc. All rights reserved.
GemfireRepositoryData Access with Repositories
public interface CustomerRepository extends GemfireRepository<Customer, Long> {
}
// Spring context.xml<gfe-data:repositories base-package="org.example.app.customer.dao.repo"> <repo:include-filter type="assignable" expression=”org.example.app.customer.dao.repo.CustomerRepository"/></gfe-data:repositories>
Example…
42© 2014 Pivotal Software, Inc. All rights reserved.
Implementation
Annotation-based Function Implementation/Execution
package org.example.app.functions;
import …
@Componentpublic class ExampleFunctions {
@GemfireFunction public Customer update(…) { … } …}
Example…
(Optional)…
<bean class=“org.example….ExampleFunctions”/>
43© 2014 Pivotal Software, Inc. All rights reserved.
Execution
Annotation-based Function Implementation/Execution
package org.example.app.function.executions;
import …
@OnRegion(“Example”)public interface ExampleFunctionExecution { Customer update(…);}
And…<gfe:annotation-driven/>
<gfe-data:function-executions base-package="org.example.app.function.executions">
44© 2014 Pivotal Software, Inc. All rights reserved.
Execution
Annotation-based Function Implementation/Execution
Additionally…– @OnServer(s), @OnMember(s)
Examples…
45© 2014 Pivotal Software, Inc. All rights reserved.
JSR-107 (JCache), Spring Cache AbstractionGeode Caching Provider
With @Cacheable And @CachePut, @CacheEvict(allEntries=[true|false]), @CacheConfig
– http://docs.spring.io/spring/docs/current/spring-framework-reference/htmlsingle/#cache-jsr-107
Or with JSR-107 using…– @CacheResult, @CachePut, @CacheRemove, @CacheRemoveAll,
@CacheDefaults
46© 2014 Pivotal Software, Inc. All rights reserved. 46© 2014 Pivotal Software, Inc. All rights reserved.
DemoBy Luke Shannon
47© 2014 Pivotal Software, Inc. All rights reserved.
On the horizon…
48© 2014 Pivotal Software, Inc. All rights reserved.
RoadmapSpring Data GemFire
Support for GemFire 8.2 & 9.0– HDFS, Off-Heap, Apache Lucene, Spark
Support for pre-defined data sets using Snapshot Service– SGG-408
Expiration annotation support with CustomExpiry– SGF-404
Spring Session Adapter for GemFire/Geode– SGF-373
49© 2014 Pivotal Software, Inc. All rights reserved.
Roadmap Continued…Spring Data GemFire
Repository Query method annotation extensions...– SGF-392
public interface ExampleRepository implements GemfireRepository<Customer, Long> {
@Trace @Limit(20) @Import(“org.example.app.domain.SomeOtherType”) List<Customer> findByLastname(String lastName);
}
50© 2014 Pivotal Software, Inc. All rights reserved.
Get Involved…
51© 2014 Pivotal Software, Inc. All rights reserved.
Apache GeodeContribute
In addition to developing Spring applications with Apache Geode, now is a perfect time to get involved in the Apache Geode community and contribute growing Apache project from the ground-up.– Ways to participate…
http://geode.incubator.apache.org/contribute/
52© 2014 Pivotal Software, Inc. All rights reserved.
Finally…
53© 2014 Pivotal Software, Inc. All rights reserved.
Washington, DC – September 14 – 17, 2015SpringOne2GX
Early Bird price registration ends August 14th, Register Now!– https://2015.event.springone2gx.com/register
Building High-Scalable Spring Applications with Distributed In-Memory Data Grids.– Luke Shannon & John Blum– Tuesday, September 15th, 2015– https://2015.event.springone2gx.com/schedule/sessions/
building_highly_scalable_spring_applications_with_in_memory_distributed_data_grids.html
54© 2014 Pivotal Software, Inc. All rights reserved.
References http://geode.incubator.apache.org/ http://geode.incubator.apache.org/contribute/ https://github.com/project-geode/docs/wiki https://github.com/apache/incubator-geode https://pivotal.io/big-data/pivotal-gemfire http://gemfire.docs.pivotal.io/latest/userguide/index.html#getting_started/topics/gemfire_overview.html http://gemfire.docs.pivotal.io/latest/javadocs/japi/index.html http://stackoverflow.com/questions/tagged/geode+or+gemfire http://projects.spring.io/spring-data-gemfire/ http://docs.spring.io/spring-data-gemfire/docs/1.6.0.RELEASE/reference/html/ http://docs.spring.io/spring-data-gemfire/docs/current/reference/html/ http://docs.spring.io/spring-data-gemfire/docs/current/api/ https://github.com/spring-projects/spring-data-gemfire https://github.com/spring-projects/spring-gemfire-examples http://stackoverflow.com/questions/tagged/spring-data-gemfire
55© 2014 Pivotal Software, Inc. All rights reserved. 55© 2014 Pivotal Software, Inc. All rights reserved.
Q&AAny questions…
56© 2014 Pivotal Software, Inc. All rights reserved. 56© 2014 Pivotal Software, Inc. All rights reserved.
Thank you!