investing the effects of overcommitting yarn resources

Investigating the Effects of Overcommitting YARN Resources

Jason Lowe

[email protected]

Nathan Roberts

Kendall said he might have something ready before summit. Could just remove this slide for now and if one shows up, great.

Problem: Underutilized Cluster Resources

Jason Lowe

To sum up what we discussed, let's go with an enlargement of the right-hand graph since it follows the results graph much better. I would like to keep mem in there since it shows we have room to grow in both mem+cpu utilization. I would also remove the 125% and 150% from this graph since it's not doing any overcommit here.

Nathan Roberts

Would right hand days work better? Part of the problem is that we were dealing woth spark shells at this time so the memory one is not following very well. I could only plot CPU.

Jason Lowe

Can we enlarge this graph to fill more of the page so it's easier to read at distance? Also it's a bit surprising how cpu and mem don't seem to track YARN at all in this graph. I thought it was always tracking but with a gap?

Optimize The Jobs!

● Internal Downsizer tool quantifies job waste● Application framework limitations● Optimally tuned container can still have opportunities

Time

Con

tain

er U

tiliz

atio

n

UnderutilizedResources

What about Static Overcommit?

● Configure YARN to use more memory than node provides● Tried with some success● Performs very poorly when node fully utilized

Overcommit Prototype Design Goals

● No changes to applications● Minimize changes to YARN protocols● Minimize changes to scheduler internals● Overcommit on memory only● Conservative growth● Rapid correction

Overcommit Overview

ResourceManager NodeManagerUtilization report in heartbeat

■ Unaware of overcommit amount■ Self-preservation preemption

■ Adjusts internal node size■ Assigns containers based on new size

Container assignments

ApplicationMasters

Container launches

Nod

e M

emor

yN

ode

Util

izat

ion

ResourceManager Node Scaling

Time

Time

No Overcommit

Reduced Overcommit

Full Overcommit

Allocated Node Mem

Total Node Mem

Original Node Mem

ResourceManager Overcommit Tunables

Parameter Description Value

memory.max-factor Maximum amount a node will be overcommitted 1.5

memory.low-water-mark Maximum overcommit below this node utilization

0.6

memory.high-water-mark No overcommit above this node utilization 0.8

memory.increment-mb Maximum increment above node allocation 16384

increment-period-ms Delay between overcommit increments if node container state does not change

0

Parameters use yarn.resourcemanager.scheduler.overcommit. prefix

NodeManager Self-Preservation Preemption

Node Utilization

High Water Mark

Low Water Mark

● Utilization above high mark triggers preemption● Preempts enough to reach low mark utilization● Does not preempt containers below original node size● Containers preempted in group order

○ Tasks from preemptable queue○ ApplicationMasters from preemptable queue○ Tasks from non-preemptable queue○ ApplicationMasters from non-preemptable queue

● Youngest containers preempted first within a group

0%

100%

NodeManager Overcommit Tunables

Parameter Description Value

memory.high-water-mark Preemption when above this utilization 0.95

memory.low-water-mark Target utilization after preemption 0.92

Parameters use yarn.nodemanager.resource-monitor.overcommit. prefix

Results

Jason Lowe

Would like to blow these graphs up to take up the page so it's easier to see the legend and marks on axes.

Nathan Roberts

June week will be better, just waiting for the cluster to be free of upgrades or other noise. I'll try to add a work gained / work lost diagram today.

Jason Lowe

math looks ok assuming overcommitmb metric is per minute. I thought we were reporting more often than that, but maybe that's just to the aggregators and the aggregators are only reporting to simonweb every minute? As for the 502 GB-hours lost, it's hard to read from this graph which I suppose is the point. We lost 0.009% of the work that we gained.

Nathan Roberts

Double check my math on gbhr_gained vs gbhr_lost:gbhr_gained=overcommitmb/1024/60 (60 1 minute samples per hour)gbhr_lost = overcommitmbseclost / 1024 /3600In the Jay slides I had 272 GB-hours lost vs 3.35M GB-hours gained for a 3 day period. and in this chart it's 502 GB-hours lost for 5.5M GB-hours gained. So if math is wrong, I'm at least consistent.Could be a nice place for a magnifying-glass effect.

Results - Capacity_Gained vs Work_Lost

Lessons Learned

● Significant overcommit achievable on real workloads● Far less preemption than expected● Container reservations can drive overcommit growth● Coordinated reducers can be a problem● Cluster totals over time can be a bit confusing at first

Future Work

● YARN-5202● Only grows cluster as a whole not individual queues● Nodes can overcommit while others are relatively idle● CPU overcommit● Predict growth based on past behavior● Relinquish nodes during quiet periods● Integration with YARN-1011

YARN-1011

● Explicit GUARANTEED vs. OPPORTUNISTIC distinction● Promotion of containers once resources are available● SLA guarantees along with best-effort load

Acknowledgements

● Nathan Roberts for co-developing overcommit POC● Inigo Goiri for nodemanager utilization collection and reporting● Giovanni Matteo Fumarola for nodemanager AM container detection● YARN-1011 contributors for helping to shape the long-term solution

Questions?

Jason Lowe

[email protected]

investing the effects of overcommitting yarn resources

Technology