understanding memory management in spark for fun and profit
TRANSCRIPT
![Page 1: Understanding Memory Management In Spark For Fun And Profit](https://reader034.vdocuments.site/reader034/viewer/2022051502/58e4f8781a28abdf698b51a3/html5/thumbnails/1.jpg)
Understanding Memory Management in Spark For Fun And Profit
Shivnath Babu (Duke University, Unravel Data Systems)Mayuresh Kunjir (Duke University)
![Page 2: Understanding Memory Management In Spark For Fun And Profit](https://reader034.vdocuments.site/reader034/viewer/2022051502/58e4f8781a28abdf698b51a3/html5/thumbnails/2.jpg)
We are• Shivnath Babu
– Associate Professor @ Duke University– CTO, Unravel Data Systems
• Mayuresh Kunjir– PhD Student @ Duke University
![Page 3: Understanding Memory Management In Spark For Fun And Profit](https://reader034.vdocuments.site/reader034/viewer/2022051502/58e4f8781a28abdf698b51a3/html5/thumbnails/3.jpg)
A Day in the Life of a Spark Application Developer
![Page 4: Understanding Memory Management In Spark For Fun And Profit](https://reader034.vdocuments.site/reader034/viewer/2022051502/58e4f8781a28abdf698b51a3/html5/thumbnails/4.jpg)
spark-submit --class SortByKey --num-executors 10 --executor-memory 4G --executor-cores 16
Container [pid=28352,containerID=container_1464692140815_0006_01_000004] is running beyond physical memory limits. Current usage: 5 GB of 5 GB physical memory used; 6.8 GB of 10.5 GB virtual memory used. Killing container.
![Page 5: Understanding Memory Management In Spark For Fun And Profit](https://reader034.vdocuments.site/reader034/viewer/2022051502/58e4f8781a28abdf698b51a3/html5/thumbnails/5.jpg)
Searches on StackOverflow
![Page 6: Understanding Memory Management In Spark For Fun And Profit](https://reader034.vdocuments.site/reader034/viewer/2022051502/58e4f8781a28abdf698b51a3/html5/thumbnails/6.jpg)
Fix #1: Turn off Yarn’s Memory Policing
yarn.nodemanager.pmem-check-enabled=false
Application Succeeds!
![Page 7: Understanding Memory Management In Spark For Fun And Profit](https://reader034.vdocuments.site/reader034/viewer/2022051502/58e4f8781a28abdf698b51a3/html5/thumbnails/7.jpg)
But, wait a minute
This fix is not multi-tenant friendly!-- Ops will not be happy
![Page 8: Understanding Memory Management In Spark For Fun And Profit](https://reader034.vdocuments.site/reader034/viewer/2022051502/58e4f8781a28abdf698b51a3/html5/thumbnails/8.jpg)
Fix #2: Use a Hint from SparkWARN yarn.YarnAllocator: Container killed by YARN for exceeding memory limits. 5 GB of 5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead
![Page 9: Understanding Memory Management In Spark For Fun And Profit](https://reader034.vdocuments.site/reader034/viewer/2022051502/58e4f8781a28abdf698b51a3/html5/thumbnails/9.jpg)
What is this Memory Overhead?Node memory
ContainerOS overhead Executor memory
Sharednative
libsMemory mapped
filesThread Stacks
NIO buffers
![Page 10: Understanding Memory Management In Spark For Fun And Profit](https://reader034.vdocuments.site/reader034/viewer/2022051502/58e4f8781a28abdf698b51a3/html5/thumbnails/10.jpg)
A Peek at the Memory Usage Timeline
Executor JVM max heap
Container memory
Physical memoryused by Container
as seen by OS
Container is killed by Yarn
![Page 11: Understanding Memory Management In Spark For Fun And Profit](https://reader034.vdocuments.site/reader034/viewer/2022051502/58e4f8781a28abdf698b51a3/html5/thumbnails/11.jpg)
After Applying Fix #2Leaving more room
for overheads
spark-submit --class SortByKey --num-executors 10 --executor-memory 4G --executor-cores 16 --conf spark.yarn.executor.memoryOverhead=1536m
![Page 12: Understanding Memory Management In Spark For Fun And Profit](https://reader034.vdocuments.site/reader034/viewer/2022051502/58e4f8781a28abdf698b51a3/html5/thumbnails/12.jpg)
But, what did we do here?
We traded off memory efficiency for reliability
![Page 13: Understanding Memory Management In Spark For Fun And Profit](https://reader034.vdocuments.site/reader034/viewer/2022051502/58e4f8781a28abdf698b51a3/html5/thumbnails/13.jpg)
What was the Root Cause?
Each task is fetching shuffle files over NIO channel. The
buffers required are allocated from OS overheads
Container is killed by Yarn
![Page 14: Understanding Memory Management In Spark For Fun And Profit](https://reader034.vdocuments.site/reader034/viewer/2022051502/58e4f8781a28abdf698b51a3/html5/thumbnails/14.jpg)
Fix #3: Reduce Executor Cores
Less Concurrent Tasks Less Overhead Space
spark-submit --class SortByKey --num-executors 10 --executor-memory 4G --executor-cores 8
Application Succeeds!
![Page 15: Understanding Memory Management In Spark For Fun And Profit](https://reader034.vdocuments.site/reader034/viewer/2022051502/58e4f8781a28abdf698b51a3/html5/thumbnails/15.jpg)
But, what did we really do here?
We traded off performance and CPU efficiency for reliability
![Page 16: Understanding Memory Management In Spark For Fun And Profit](https://reader034.vdocuments.site/reader034/viewer/2022051502/58e4f8781a28abdf698b51a3/html5/thumbnails/16.jpg)
Let’s Dig Deeper
Why is so much memory consumed in Executor heap?
![Page 17: Understanding Memory Management In Spark For Fun And Profit](https://reader034.vdocuments.site/reader034/viewer/2022051502/58e4f8781a28abdf698b51a3/html5/thumbnails/17.jpg)
JVM’s View of Executor MemoryNode memory
ContainerOS overhead Executor memory
Direct byte buffersJVM Internal
(Code Cache + Perm Gen)Young Gen Old Gen
Off-heap Heap
![Page 18: Understanding Memory Management In Spark For Fun And Profit](https://reader034.vdocuments.site/reader034/viewer/2022051502/58e4f8781a28abdf698b51a3/html5/thumbnails/18.jpg)
JVM’s View of Executor Memory
![Page 19: Understanding Memory Management In Spark For Fun And Profit](https://reader034.vdocuments.site/reader034/viewer/2022051502/58e4f8781a28abdf698b51a3/html5/thumbnails/19.jpg)
Fix #4: Frequent GC for Smaller Heap
spark-submit --class SortByKey --num-executors 10 --executor-memory 4G --executor-cores 16 - --conf "spark.executor.extraJavaOptions=-XX:OldSize=100m -XX:MaxNewSize=100m"
![Page 20: Understanding Memory Management In Spark For Fun And Profit](https://reader034.vdocuments.site/reader034/viewer/2022051502/58e4f8781a28abdf698b51a3/html5/thumbnails/20.jpg)
But, what did we do now?Reliability is achieved at the cost of extra CPU cycles spent in GC, degrading performance by 15%
![Page 21: Understanding Memory Management In Spark For Fun And Profit](https://reader034.vdocuments.site/reader034/viewer/2022051502/58e4f8781a28abdf698b51a3/html5/thumbnails/21.jpg)
Can we do better?
So far, we have sacrificed either performance or efficiency for reliability
![Page 22: Understanding Memory Management In Spark For Fun And Profit](https://reader034.vdocuments.site/reader034/viewer/2022051502/58e4f8781a28abdf698b51a3/html5/thumbnails/22.jpg)
Fix #5: Spark can Exploit Structure in Data
spark-submit --class SortByKeyDF --num-executors 10 --executor-memory 4G --executor-cores 16
Tungsten’s custom serialization reduces memory footprint
while also reducing processing time
Application succeeds and runs 2x faster
compared to Fix #2!
![Page 23: Understanding Memory Management In Spark For Fun And Profit](https://reader034.vdocuments.site/reader034/viewer/2022051502/58e4f8781a28abdf698b51a3/html5/thumbnails/23.jpg)
ReliabilityEfficiency
PerformancePredictability
Objectives
Workloads
Memory management optionsBI ML Gra
phStr
eaming
Challenges in Memory
Management
![Page 24: Understanding Memory Management In Spark For Fun And Profit](https://reader034.vdocuments.site/reader034/viewer/2022051502/58e4f8781a28abdf698b51a3/html5/thumbnails/24.jpg)
Next• Key insights from
experimental analysis
• Current work
![Page 25: Understanding Memory Management In Spark For Fun And Profit](https://reader034.vdocuments.site/reader034/viewer/2022051502/58e4f8781a28abdf698b51a3/html5/thumbnails/25.jpg)
Yarn-level Memory Management
![Page 26: Understanding Memory Management In Spark For Fun And Profit](https://reader034.vdocuments.site/reader034/viewer/2022051502/58e4f8781a28abdf698b51a3/html5/thumbnails/26.jpg)
Yarn-level Memory Management
• Executor memory• OS memory overhead per executor• Cores per executor• Number of executors
Node memoryContainer
OS overhead Executor memory
![Page 27: Understanding Memory Management In Spark For Fun And Profit](https://reader034.vdocuments.site/reader034/viewer/2022051502/58e4f8781a28abdf698b51a3/html5/thumbnails/27.jpg)
Impact of Changing Executor MemoryFa
iled
java.lang.OutOfMemoryError: Java heap spaceat java.util.Arrays.copyOf(Arrays.java:2271)at
java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)...
at org.apache.spark.storage.BlockManager.dataSerialize(BlockManager.scala:1202)...
at org.apache.spark.CacheManager.putInBlockManager(CacheManager.scala:175)
Reliability EfficiencyPerformancePredictability
![Page 28: Understanding Memory Management In Spark For Fun And Profit](https://reader034.vdocuments.site/reader034/viewer/2022051502/58e4f8781a28abdf698b51a3/html5/thumbnails/28.jpg)
Spark-level Memory Management
![Page 29: Understanding Memory Management In Spark For Fun And Profit](https://reader034.vdocuments.site/reader034/viewer/2022051502/58e4f8781a28abdf698b51a3/html5/thumbnails/29.jpg)
Spark-level Memory ManagementNode memory
ContainerOS overhead Executor memory
spark.executor.memoryStorage
spark.storage.memoryFractionspark.storage.safetyFraction
Executionspark.shuffle.memoryFractionspark.shuffle.safetyFraction
Unmanaged
spark.executor.memory
Storage Execution
Unmanaged
Unified poolspark.memory.fraction
spark.memory.storageFraction
Legacy Unified
![Page 30: Understanding Memory Management In Spark For Fun And Profit](https://reader034.vdocuments.site/reader034/viewer/2022051502/58e4f8781a28abdf698b51a3/html5/thumbnails/30.jpg)
Spark-level Memory Management• Legacy or unified?
– If legacy, what is size of storage pool Vs. execution pool?• Caching
– On heap or off-heap (e.g., Tachyon)?– Data format (deserialized or serialized)– Provision for data unrolling
• Execution data– Java-managed or Tungsten-managed
![Page 31: Understanding Memory Management In Spark For Fun And Profit](https://reader034.vdocuments.site/reader034/viewer/2022051502/58e4f8781a28abdf698b51a3/html5/thumbnails/31.jpg)
Comparing Legacy and Unified
Increasing storage pool size from left to right
SortByKey
K-Means
Increasing storage pool size, Decreasing execution pool sizeUnified
Unified
![Page 32: Understanding Memory Management In Spark For Fun And Profit](https://reader034.vdocuments.site/reader034/viewer/2022051502/58e4f8781a28abdf698b51a3/html5/thumbnails/32.jpg)
Unified does as Expected, But…
Size of storage pool increases from left to right
Performance Predictability
Executors fail due to OOM errors while receiving shuffle blocks
java.lang.OutOfMemoryError: Java heap spaceat java.util.Arrays.copyOf(Arrays.java:2271)at
java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)...
at org.apache.spark.storage.BlockManager.dataSerialize(BlockManager.scala:1202)...
at org.apache.spark.network.netty.NettyBlockRpcServer.receive(NettyBlockRpcServer.scala:58)
legacy
![Page 33: Understanding Memory Management In Spark For Fun And Profit](https://reader034.vdocuments.site/reader034/viewer/2022051502/58e4f8781a28abdf698b51a3/html5/thumbnails/33.jpg)
Unified Memory Manager is:• A step in the right direction• Not unified enough
![Page 34: Understanding Memory Management In Spark For Fun And Profit](https://reader034.vdocuments.site/reader034/viewer/2022051502/58e4f8781a28abdf698b51a3/html5/thumbnails/34.jpg)
Spark-level Memory Management• Legacy or unified?
– If legacy, what is size of storage pool Vs. execution pool?• Caching
– On heap or off-heap (e.g., Tachyon)?– Data format (deserialized or serialized)– Provision for data unrolling
• Execution data– Java-managed or Tungsten-managed
![Page 35: Understanding Memory Management In Spark For Fun And Profit](https://reader034.vdocuments.site/reader034/viewer/2022051502/58e4f8781a28abdf698b51a3/html5/thumbnails/35.jpg)
Deserialized Vs. Serialized cache
Size of storage pool increases from left to right
Performance Predictability
Memory footprint of data in cache goes down by ~20% making more partitions fit in the storage pool
Efficiency
legacy
![Page 36: Understanding Memory Management In Spark For Fun And Profit](https://reader034.vdocuments.site/reader034/viewer/2022051502/58e4f8781a28abdf698b51a3/html5/thumbnails/36.jpg)
Another Application, Another Story!
Size of storage pool increases from left to right
Faile
dFa
iled
java.lang.OutOfMemoryError: Java heap spaceat java.util.Arrays.copyOf(Arrays.java:2271)at
java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)...
at org.apache.spark.storage.BlockManager.dataSerialize(BlockManager.scala:1202)...
at org.apache.spark.CacheManager.putInBlockManager(CacheManager.scala:175)
PredictabilityReliabilityExecutors fail due to OOM errors while serializing data
legacy
![Page 37: Understanding Memory Management In Spark For Fun And Profit](https://reader034.vdocuments.site/reader034/viewer/2022051502/58e4f8781a28abdf698b51a3/html5/thumbnails/37.jpg)
Spark-level memory management• Legacy or unified?
– If legacy, what is size of storage pool Vs. execution pool?• Caching
– On heap or off-heap (e.g., Tachyon)?– Data format (deserialized or serialized)– Provision for data unrolling
• Execution data– Java-managed or Tungsten-managed
![Page 38: Understanding Memory Management In Spark For Fun And Profit](https://reader034.vdocuments.site/reader034/viewer/2022051502/58e4f8781a28abdf698b51a3/html5/thumbnails/38.jpg)
Execution Data Management
All objects in Heap Up to 2GB objects in off-heap at any time
We have seen that Tungsten-managed heap improves the performance significantly. (Fix #5)
We did not notice much further improvements by pushing objects to off-heap
![Page 39: Understanding Memory Management In Spark For Fun And Profit](https://reader034.vdocuments.site/reader034/viewer/2022051502/58e4f8781a28abdf698b51a3/html5/thumbnails/39.jpg)
JVM-level Memory Management
![Page 40: Understanding Memory Management In Spark For Fun And Profit](https://reader034.vdocuments.site/reader034/viewer/2022051502/58e4f8781a28abdf698b51a3/html5/thumbnails/40.jpg)
JVM-level Memory Management
• Which GC algorithm? (Parallel GC, G1 GC, …)• Size cap for a GC pool• Frequency of collections• Number of parallel GC threads
![Page 41: Understanding Memory Management In Spark For Fun And Profit](https://reader034.vdocuments.site/reader034/viewer/2022051502/58e4f8781a28abdf698b51a3/html5/thumbnails/41.jpg)
Spark-JVM Interactions
Ratio of old generation size to average RDD size cached per executor varied
Keep JVM OldGen size at least as big as RDD cache
Keeping Spark storage pool size
PageRank
Keeping Spark storage pool size constant, the size of OldGen pool is increased from left to right K-means executors display more skew in data compared to PageRank
![Page 42: Understanding Memory Management In Spark For Fun And Profit](https://reader034.vdocuments.site/reader034/viewer/2022051502/58e4f8781a28abdf698b51a3/html5/thumbnails/42.jpg)
Current Work• Automatic root-cause
analysis of memory-related issues
• Auto-tuning algorithms for memory allocation in multi-tenant clusters
![Page 43: Understanding Memory Management In Spark For Fun And Profit](https://reader034.vdocuments.site/reader034/viewer/2022051502/58e4f8781a28abdf698b51a3/html5/thumbnails/43.jpg)
Get Free Trial Edition:bit.ly/getunravelUNCOVER ISSUESUNLEASH RESOURCESUNRAVEL PERFORMANCE