apache geode (incubating) introduction with docker
TRANSCRIPT
Hands-on Introduction& Hackathon Kickoff
Ashvin Agrawal William Markito@william_markito@aasoj
Powered by
Pivotal Open Source Hub (POSH)
(incubating)
• Hackathon Details• Apache Geode Introduction
• History• Key features and components• Roadmap
• Hands-on lab• Build & run• Starting a cluster• Using docker for clustering• Your first app
• Q&A
Agenda
Hackathon details
Powered by
Pivotal Open Source Hub (POSH)
http://ambitious-apps.challengepost.com/
Introduction
A distributed, memory-based data management platform for data oriented apps that need:• high performance, scalability, resiliency and continuous
availability• fast access to critical data set• location aware distributed data processing• event driven data architecture
Introduction
One size fits all ?
Cost of sorting is nlog(n)
One size fits all ?
Cost of sorting is nlog(n)
• Data quality and quantity differences• Eventual consistency• Response time expectation • Scalability challenges: disk, memory,
network and external systems
• 1000+ systems in production (real customers)• Cutting edge use cases
Incubating… but rock solid
• 17 billion records in memory • GE Power & Water's Remote Monitoring & Diagnostics Center
• 3 TB operational data in-memory, 400 TB archived • China Railways
• 4.6 Million transactions a day / 40K transactions a second • China Railways
Incubating… but rock solid
• Performance optimized persistence
• Configurable consistency
• Elastic capacity
• Latency minimizing distribution
• Heterogenous deployment
Designed for High Performance
+/-
L2 ~10 ns, memory ~100 ns, network <1ms, disk ~10ms
• Cache• Region• Member• Client Cache• Functions• Listeners
Concepts
• Cache
• In-memory storage and management for your data
• Configurable through XML, Spring, Java API or CLI
• Collection of Region
Concepts
• Region
• Distributed java.util.Map on steroids (Key/Value)
• Consistent API regardless of where or how data is stored
• Observable (reactive)
• Highly available, redundant on cache Member (s).
Concepts
• Region
• Local, Replicated or Partitioned
• In-memory or persistent
• Redundant
• LRU
• Overflow
Concepts
LOCALLOCAL_HEAP_LRULOCAL_OVERFLOWLOCAL_PERSISTENTLOCAL_PERSISTENT_OVERFLOWPARTITIONPARTITION_HEAP_LRUPARTITION_OVERFLOWPARTITION_PERSISTENTPARTITION_PERSISTENT_OVERFLOWPARTITION_PROXYPARTITION_PROXY_REDUNDANTPARTITION_REDUNDANTPARTITION_REDUNDANT_HEAP_LRUPARTITION_REDUNDANT_OVERFLOWPARTITION_REDUNDANT_PERSISTENTPARTITION_REDUNDANT_PERSISTENT_OVERFLOWREPLICATEREPLICATE_HEAP_LRUREPLICATE_OVERFLOWREPLICATE_PERSISTENTREPLICATE_PERSISTENT_OVERFLOWREPLICATE_PROXY
• Persistent Regions
• Durability
• WAL for efficient writing
• Consistent recovery
• Compaction
Concepts
Server 1 Server N
• Member
• A process that has a connection to the system
• A process that has created a cache
• Embeddable within your application
Concepts
Client
Locator
Server
• Client cache
• A process connected to the Geode server(s)
• Can have a local copy of the data
• Can be notified about events on the servers
Concepts
• Functions
• Used for distributed concurrent processing (Map/Reduce, stored procedure)
• Highly available
• Data oriented
• Member oriented
Concepts
Concepts
• Functions
• Listeners
• CacheWriter / CacheListener
• AsyncEventListener (queue / batch)
• Parallel or Serial
• Conflation
Concepts
Hands on
• Clone & Build
Hands-on: Build & run
git clone https://github.com/apache/incubator-geodecd incubator-geode./gradlew build -Dskip.tests=true
• Start a server
cd gemfire-assembly/build/install/apache-geode ./bin/gfsh gfsh> start locator --name=locator gfsh> start server --name=server gfsh> create region --name=myRegion --type=REPLICATE
Hands-on: Docker
&
• Containers• FreeBSD Jails (2000)• Solaris Zones (2004)
• Docker (2013)
• Operating system level virtualization• Isolated user space instances
* https://linuxcontainers.org/
Hands-on: Docker
Container vs VM
“..while the hypervisor abstracts the entire device, containers just abstract the operating system kernel"
Hands-on: Docker & Compose
• Single instance
docker run -it apachegeode/geode:nightly gfsh
• Cluster
docker-compose up
• Scale
docker-compose scale server=3
Hands-on: Application
• Teeny URL• Fast response time• Statistics
• Hits• User agent ?• IPs ?
• URL will last for 5 minutes
• Distribute data & load• Highly scalable
createURL
getURLstats
• HDFS Persistence• Off-heap memory storage• Lucene Search• Spark Integration• Cloud Foundry service
Roadmap
• Code• New features• Bug fixes• Writing tests
• Documentation• Wiki• Web site• User guide
How to Contribute
• Community• Join the mailing list
• Ask or answer • Join our HipChat• Become a speaker• Finding bugs• Testing an RC/Beta
• JIRAhttps://issues.apache.org/jira/browse/GEODE
• Wikicwiki.apache.org/confluence/display/GEODE
• GitHubhttps://github.com/apache/incubator-geode
• Mailing listsmail-archives.apache.org/mod_mbox/incubator-geode-dev/
Links
Thank you
http://geode.incubator.apache.org
https://github.com/Pivotal-Open-Source-Hub