replication distilled: hazelcast deep dive - ensar... · hazelcast is pa/ec consistency is usually...
TRANSCRIPT
![Page 1: Replication Distilled: Hazelcast Deep Dive - Ensar... · Hazelcast is PA/EC Consistency is usually traded to availability and latency together. Hazelcast works in memory and mostly](https://reader030.vdocuments.site/reader030/viewer/2022040715/5e1d768d9a3f5774bb6f159d/html5/thumbnails/1.jpg)
Replication Distilled:Hazelcast Deep Dive
Ensar Basri KahveciHazelcast
![Page 2: Replication Distilled: Hazelcast Deep Dive - Ensar... · Hazelcast is PA/EC Consistency is usually traded to availability and latency together. Hazelcast works in memory and mostly](https://reader030.vdocuments.site/reader030/viewer/2022040715/5e1d768d9a3f5774bb6f159d/html5/thumbnails/2.jpg)
Hazelcast
▪ The leading open source Java IMDG▪ Distributed Java collections, concurrency primitives, ...▪ Distributed computations, messaging, ...
![Page 3: Replication Distilled: Hazelcast Deep Dive - Ensar... · Hazelcast is PA/EC Consistency is usually traded to availability and latency together. Hazelcast works in memory and mostly](https://reader030.vdocuments.site/reader030/viewer/2022040715/5e1d768d9a3f5774bb6f159d/html5/thumbnails/3.jpg)
In-Memory Data Grids
▪ Distributed caching▪ Keeping data in local JVM for fast access & processing▪ Elasticity, availability, high throughput, and low latency▪ Multiple copies of data to tolerate failures
![Page 4: Replication Distilled: Hazelcast Deep Dive - Ensar... · Hazelcast is PA/EC Consistency is usually traded to availability and latency together. Hazelcast works in memory and mostly](https://reader030.vdocuments.site/reader030/viewer/2022040715/5e1d768d9a3f5774bb6f159d/html5/thumbnails/4.jpg)
Replication
▪ Putting a data set into multiple nodes▪ Fault tolerance▪ Latency▪ Throughput
![Page 5: Replication Distilled: Hazelcast Deep Dive - Ensar... · Hazelcast is PA/EC Consistency is usually traded to availability and latency together. Hazelcast works in memory and mostly](https://reader030.vdocuments.site/reader030/viewer/2022040715/5e1d768d9a3f5774bb6f159d/html5/thumbnails/5.jpg)
Challenges
▪ Where to perform reads & writes?▪ How to keep replicas sync?▪ How to handle concurrent reads & writes?▪ How to handle failures?
![Page 6: Replication Distilled: Hazelcast Deep Dive - Ensar... · Hazelcast is PA/EC Consistency is usually traded to availability and latency together. Hazelcast works in memory and mostly](https://reader030.vdocuments.site/reader030/viewer/2022040715/5e1d768d9a3f5774bb6f159d/html5/thumbnails/6.jpg)
CAP Principle
▪ Pick two of C, A, and P▪ CP versus AP
![Page 7: Replication Distilled: Hazelcast Deep Dive - Ensar... · Hazelcast is PA/EC Consistency is usually traded to availability and latency together. Hazelcast works in memory and mostly](https://reader030.vdocuments.site/reader030/viewer/2022040715/5e1d768d9a3f5774bb6f159d/html5/thumbnails/7.jpg)
CP
![Page 8: Replication Distilled: Hazelcast Deep Dive - Ensar... · Hazelcast is PA/EC Consistency is usually traded to availability and latency together. Hazelcast works in memory and mostly](https://reader030.vdocuments.site/reader030/viewer/2022040715/5e1d768d9a3f5774bb6f159d/html5/thumbnails/8.jpg)
AP
![Page 9: Replication Distilled: Hazelcast Deep Dive - Ensar... · Hazelcast is PA/EC Consistency is usually traded to availability and latency together. Hazelcast works in memory and mostly](https://reader030.vdocuments.site/reader030/viewer/2022040715/5e1d768d9a3f5774bb6f159d/html5/thumbnails/9.jpg)
Consistency/Latency Trade-off
![Page 10: Replication Distilled: Hazelcast Deep Dive - Ensar... · Hazelcast is PA/EC Consistency is usually traded to availability and latency together. Hazelcast works in memory and mostly](https://reader030.vdocuments.site/reader030/viewer/2022040715/5e1d768d9a3f5774bb6f159d/html5/thumbnails/10.jpg)
Consistency/Latency Trade-off
![Page 11: Replication Distilled: Hazelcast Deep Dive - Ensar... · Hazelcast is PA/EC Consistency is usually traded to availability and latency together. Hazelcast works in memory and mostly](https://reader030.vdocuments.site/reader030/viewer/2022040715/5e1d768d9a3f5774bb6f159d/html5/thumbnails/11.jpg)
PACELC Principle
▪ If there is a network partition (P), we have to choose between availability and consistency (AC).
▪ Else (E), during normal operation, we can choose between latency and consistency (LC).
![Page 12: Replication Distilled: Hazelcast Deep Dive - Ensar... · Hazelcast is PA/EC Consistency is usually traded to availability and latency together. Hazelcast works in memory and mostly](https://reader030.vdocuments.site/reader030/viewer/2022040715/5e1d768d9a3f5774bb6f159d/html5/thumbnails/12.jpg)
Let’s buildthe core replication protocolof Hazelcast
![Page 13: Replication Distilled: Hazelcast Deep Dive - Ensar... · Hazelcast is PA/EC Consistency is usually traded to availability and latency together. Hazelcast works in memory and mostly](https://reader030.vdocuments.site/reader030/viewer/2022040715/5e1d768d9a3f5774bb6f159d/html5/thumbnails/13.jpg)
Primary Copy
▪ Operations are sent to primary replicas.▪ Strong consistency when the primary is reachable.
![Page 14: Replication Distilled: Hazelcast Deep Dive - Ensar... · Hazelcast is PA/EC Consistency is usually traded to availability and latency together. Hazelcast works in memory and mostly](https://reader030.vdocuments.site/reader030/viewer/2022040715/5e1d768d9a3f5774bb6f159d/html5/thumbnails/14.jpg)
Partitioning (Sharding)
▪ Partitioning helps to scale primaries.▪ A primary replica is elected for each partition.
![Page 15: Replication Distilled: Hazelcast Deep Dive - Ensar... · Hazelcast is PA/EC Consistency is usually traded to availability and latency together. Hazelcast works in memory and mostly](https://reader030.vdocuments.site/reader030/viewer/2022040715/5e1d768d9a3f5774bb6f159d/html5/thumbnails/15.jpg)
Updating Replicas
![Page 16: Replication Distilled: Hazelcast Deep Dive - Ensar... · Hazelcast is PA/EC Consistency is usually traded to availability and latency together. Hazelcast works in memory and mostly](https://reader030.vdocuments.site/reader030/viewer/2022040715/5e1d768d9a3f5774bb6f159d/html5/thumbnails/16.jpg)
Updating Replicas
partition id = hash(serialize(key)) % partition count
![Page 17: Replication Distilled: Hazelcast Deep Dive - Ensar... · Hazelcast is PA/EC Consistency is usually traded to availability and latency together. Hazelcast works in memory and mostly](https://reader030.vdocuments.site/reader030/viewer/2022040715/5e1d768d9a3f5774bb6f159d/html5/thumbnails/17.jpg)
Updating Replicas
![Page 18: Replication Distilled: Hazelcast Deep Dive - Ensar... · Hazelcast is PA/EC Consistency is usually traded to availability and latency together. Hazelcast works in memory and mostly](https://reader030.vdocuments.site/reader030/viewer/2022040715/5e1d768d9a3f5774bb6f159d/html5/thumbnails/18.jpg)
Updating Replicas
![Page 19: Replication Distilled: Hazelcast Deep Dive - Ensar... · Hazelcast is PA/EC Consistency is usually traded to availability and latency together. Hazelcast works in memory and mostly](https://reader030.vdocuments.site/reader030/viewer/2022040715/5e1d768d9a3f5774bb6f159d/html5/thumbnails/19.jpg)
Async Replication
▪ Each replica is updated separately.▪ High throughput and availability
![Page 20: Replication Distilled: Hazelcast Deep Dive - Ensar... · Hazelcast is PA/EC Consistency is usually traded to availability and latency together. Hazelcast works in memory and mostly](https://reader030.vdocuments.site/reader030/viewer/2022040715/5e1d768d9a3f5774bb6f159d/html5/thumbnails/20.jpg)
Anti-Entropy
▪ Backup replicas can fall behind the primary.▪ Non-sync backups are fixed with an active anti-entropy
mechanism.
![Page 21: Replication Distilled: Hazelcast Deep Dive - Ensar... · Hazelcast is PA/EC Consistency is usually traded to availability and latency together. Hazelcast works in memory and mostly](https://reader030.vdocuments.site/reader030/viewer/2022040715/5e1d768d9a3f5774bb6f159d/html5/thumbnails/21.jpg)
Replicas are not sync
▪ The client reads a key from the current primary replica.
![Page 22: Replication Distilled: Hazelcast Deep Dive - Ensar... · Hazelcast is PA/EC Consistency is usually traded to availability and latency together. Hazelcast works in memory and mostly](https://reader030.vdocuments.site/reader030/viewer/2022040715/5e1d768d9a3f5774bb6f159d/html5/thumbnails/22.jpg)
Network Partitioning
▪ The client reads the same key.
![Page 23: Replication Distilled: Hazelcast Deep Dive - Ensar... · Hazelcast is PA/EC Consistency is usually traded to availability and latency together. Hazelcast works in memory and mostly](https://reader030.vdocuments.site/reader030/viewer/2022040715/5e1d768d9a3f5774bb6f159d/html5/thumbnails/23.jpg)
Split-Brain
▪ Strong consistency is lost.
![Page 24: Replication Distilled: Hazelcast Deep Dive - Ensar... · Hazelcast is PA/EC Consistency is usually traded to availability and latency together. Hazelcast works in memory and mostly](https://reader030.vdocuments.site/reader030/viewer/2022040715/5e1d768d9a3f5774bb6f159d/html5/thumbnails/24.jpg)
Resolving the Divergence
▪ Merge policies: higher hits, latest update / access, …▪ Merging may cause lost updates.
![Page 25: Replication Distilled: Hazelcast Deep Dive - Ensar... · Hazelcast is PA/EC Consistency is usually traded to availability and latency together. Hazelcast works in memory and mostly](https://reader030.vdocuments.site/reader030/viewer/2022040715/5e1d768d9a3f5774bb6f159d/html5/thumbnails/25.jpg)
Let’s classify this protocolwith PACELC
![Page 26: Replication Distilled: Hazelcast Deep Dive - Ensar... · Hazelcast is PA/EC Consistency is usually traded to availability and latency together. Hazelcast works in memory and mostly](https://reader030.vdocuments.site/reader030/viewer/2022040715/5e1d768d9a3f5774bb6f159d/html5/thumbnails/26.jpg)
Hazelcast is PA/EC
▪ Consistency is usually traded to availability and latency together.
▪ Hazelcast works in memory and mostly used in a single computing cluster.
▪ Consistency - latency trade-off is minimal.▪ PA/EC works fine for distributed caching.
![Page 27: Replication Distilled: Hazelcast Deep Dive - Ensar... · Hazelcast is PA/EC Consistency is usually traded to availability and latency together. Hazelcast works in memory and mostly](https://reader030.vdocuments.site/reader030/viewer/2022040715/5e1d768d9a3f5774bb6f159d/html5/thumbnails/27.jpg)
Favoring Latency(PA/EL)
![Page 28: Replication Distilled: Hazelcast Deep Dive - Ensar... · Hazelcast is PA/EC Consistency is usually traded to availability and latency together. Hazelcast works in memory and mostly](https://reader030.vdocuments.site/reader030/viewer/2022040715/5e1d768d9a3f5774bb6f159d/html5/thumbnails/28.jpg)
Scaling Reads
▪ Reads can be served locally from near caches and backup replicas.
![Page 29: Replication Distilled: Hazelcast Deep Dive - Ensar... · Hazelcast is PA/EC Consistency is usually traded to availability and latency together. Hazelcast works in memory and mostly](https://reader030.vdocuments.site/reader030/viewer/2022040715/5e1d768d9a3f5774bb6f159d/html5/thumbnails/29.jpg)
Favoring Consistency(PC/EC)
![Page 30: Replication Distilled: Hazelcast Deep Dive - Ensar... · Hazelcast is PA/EC Consistency is usually traded to availability and latency together. Hazelcast works in memory and mostly](https://reader030.vdocuments.site/reader030/viewer/2022040715/5e1d768d9a3f5774bb6f159d/html5/thumbnails/30.jpg)
Failure Detectors
▪ Local failure detectors rely on timeouts.▪ Operations are blocked after the cluster size falls below
a threshold.
![Page 31: Replication Distilled: Hazelcast Deep Dive - Ensar... · Hazelcast is PA/EC Consistency is usually traded to availability and latency together. Hazelcast works in memory and mostly](https://reader030.vdocuments.site/reader030/viewer/2022040715/5e1d768d9a3f5774bb6f159d/html5/thumbnails/31.jpg)
Failure Detectors
▪ It takes some time to detect an unresponsive node.▪ Minimizes divergence and maintains the baseline
consistency.
![Page 32: Replication Distilled: Hazelcast Deep Dive - Ensar... · Hazelcast is PA/EC Consistency is usually traded to availability and latency together. Hazelcast works in memory and mostly](https://reader030.vdocuments.site/reader030/viewer/2022040715/5e1d768d9a3f5774bb6f159d/html5/thumbnails/32.jpg)
Isolated Failure Detectors
▪ Configure failure detectors independently for data structures
▪ Phi-Accrual Failure Detector
![Page 33: Replication Distilled: Hazelcast Deep Dive - Ensar... · Hazelcast is PA/EC Consistency is usually traded to availability and latency together. Hazelcast works in memory and mostly](https://reader030.vdocuments.site/reader030/viewer/2022040715/5e1d768d9a3f5774bb6f159d/html5/thumbnails/33.jpg)
CP Data Structures
▪ IDGenerator▪ Distributed impls of java.util.concurrent.*▪ PA/EC is not the perfect fit for CP data structures.
![Page 34: Replication Distilled: Hazelcast Deep Dive - Ensar... · Hazelcast is PA/EC Consistency is usually traded to availability and latency together. Hazelcast works in memory and mostly](https://reader030.vdocuments.site/reader030/viewer/2022040715/5e1d768d9a3f5774bb6f159d/html5/thumbnails/34.jpg)
Flake IDs
▪ Local unique id generation▪ Nodes get a unique node id during join.▪ K-ordered IDs
![Page 35: Replication Distilled: Hazelcast Deep Dive - Ensar... · Hazelcast is PA/EC Consistency is usually traded to availability and latency together. Hazelcast works in memory and mostly](https://reader030.vdocuments.site/reader030/viewer/2022040715/5e1d768d9a3f5774bb6f159d/html5/thumbnails/35.jpg)
CRDTs
▪ CRDTs: Conflict-free Replicated Data Types▪ Replicas are updated concurrently without coordination.▪ Strong eventual consistency▪ Counters, sets, maps, graphs, ...
![Page 36: Replication Distilled: Hazelcast Deep Dive - Ensar... · Hazelcast is PA/EC Consistency is usually traded to availability and latency together. Hazelcast works in memory and mostly](https://reader030.vdocuments.site/reader030/viewer/2022040715/5e1d768d9a3f5774bb6f159d/html5/thumbnails/36.jpg)
PN-Counter
![Page 37: Replication Distilled: Hazelcast Deep Dive - Ensar... · Hazelcast is PA/EC Consistency is usually traded to availability and latency together. Hazelcast works in memory and mostly](https://reader030.vdocuments.site/reader030/viewer/2022040715/5e1d768d9a3f5774bb6f159d/html5/thumbnails/37.jpg)
PN-Counter
![Page 38: Replication Distilled: Hazelcast Deep Dive - Ensar... · Hazelcast is PA/EC Consistency is usually traded to availability and latency together. Hazelcast works in memory and mostly](https://reader030.vdocuments.site/reader030/viewer/2022040715/5e1d768d9a3f5774bb6f159d/html5/thumbnails/38.jpg)
Sync Replication
▪ Concurrency primitives imply the true CP behavior.▪ Paxos, Raft, ZAB, VR▪ Re-implementing Hazelcast concurrency primitives with
Raft
![Page 39: Replication Distilled: Hazelcast Deep Dive - Ensar... · Hazelcast is PA/EC Consistency is usually traded to availability and latency together. Hazelcast works in memory and mostly](https://reader030.vdocuments.site/reader030/viewer/2022040715/5e1d768d9a3f5774bb6f159d/html5/thumbnails/39.jpg)
Recap
▪ http://bit.ly/hazelcast-replication-consistency▪ http://bit.ly/hazelcast-network-partitions▪ http://dbmsmusings.blogspot.com/2017/10/hazelcast-an
d-mythical-paec-system.html
![Page 40: Replication Distilled: Hazelcast Deep Dive - Ensar... · Hazelcast is PA/EC Consistency is usually traded to availability and latency together. Hazelcast works in memory and mostly](https://reader030.vdocuments.site/reader030/viewer/2022040715/5e1d768d9a3f5774bb6f159d/html5/thumbnails/40.jpg)
Thanks!You can find me at▪ @metanet▪ [email protected]