cluster schedulerの紹介
TRANSCRIPT
Cluster Schedulerの紹介
劉 春来 (りゅう しゅんらい)
2015-02-18@Container勉強会, 東京
Cluster Schedulingの話なぜこのトーク？
コンテナでいろんなworkloadsが流れます：batch系、service系(おおまかな二分法)↓実際にどこで実行するのかの問題(Cluster scheduling)↓PaaS検証の背景：multiple workloads, multiple tenantsのPaaS上マルチClustersのresource sharing問題
(Dynamic sharingのcluster scheduler)
Dynamic sharingって何？
まず反対のStatic partitioningを見てみよう
Static partitioningWeb cluster、DB cluster、Hadoop ClusterなどのClusterは独自のサーバー群を持っていてsharingしない
● hard to utilize machines● hard to scale elastically● hard to deal with failures
絵でわかる(p30~p40)：https://speakerdeck.com/benh/apache-mesos-nyc-meetup
Dynamic sharingThe graph fromhttp://people.csail.mit.edu/matei/talks/2011/nsdi_mesos.pptp4
Dynamic sharing
Running multiple frameworks in a single cluster can● maximize utilization ● sharing data between frameworks● simplify the infrastructure
Dynamic sharingの課題
Dynamic sharingのメリットは大きい一方で、Cluster schedulingは複雑化になります：
● a wide range of requirements and policies have to be taken into account
● clusters and their workloads keep growing and since the scheduler's workload is roughly proportional to the cluster size, the scheduler is at risk of becoming a scalability bottleneck.
代表のふたつ：Mesos とOmega● Mesosはresearch projectから生まれたOSS、paperあり、TwitterやAirbnbなど
の大規模運用実績ありTwitterはMesosで3万以上のserversを管理している(http://www.centurylinklabs.com/interviews/making-clustered-infra-look-like-one-big-server-with-mesosphere/)
● Omega(OSSではない):1) Googleのnext-generation cluster management platform(前身はBorgと
いうシステム、数年間の運用実績) 参照：https://www.usenix.org/cluster-management-google
2) Omegaというpaper：Googleのcluster scheduler, 2013※ Mesos paperの共著者のひとりもOmegaの共著者です。
Cluster schedulersの三つtype● Monolithic schedulers:
Omegaの前身であるBorgのscheduler、Apache Hadoop YARN(Omega Paperより)
● Two-level schedulers:Mesos、Hadoop-on-Demand
● Share-state schedulers:Omega
Scheduler architectures
Monolithic scheduleruse a single, centralized scheduling algorithm for all jobs.
Google's current(2013) cluster scheduler is effectively monolithic, acquired many optimizations over the years: provide internal parallelism and multi-threading to address head-of-line blocking and scalability.
Two-level scheduler(Mesos)
Mesos: controls resource allocations to schedulers
Schedulers: make decisions about what to run given allocated resources
Mesos architecture
Mesos: Example of resource offer
Two-level scheduler(Mesos)An obvious fix to the issues of static partition is to adjust the allocation of resource to each scheduler dynamically, using a central coordinator to decide how many resources each sub-cluster can have.
Mesos works best when 1) tasks are short-lived2) relinquish resources frequently3) job sizes are small compared to the size of the cluster
なぜgoogleは不採用？Monolithic schedulerとtwo-level schedulerはgoogleのニーズに満たせない:
0) Googleのニーズは何？
Clusterのworkloads
simple two-way split:● batch jobs: perform a computation and then finish. For
simplicity we put all low priority jobs and those marked as "best effort" or "batch" into the batch category
● service jobs: long-running service jobs that provide end user operations(e.g., web services) and internal infrastructure services(e.g. storage service, naming service, locking service)
Cluster traces from Google
● most(>80%) jobs are batch jobs● the majority of resources (55-80%) are
allocated to service jobs● service jobs typically run for much longer(20-
40% of them run for over a month) and have fewer tasks than batch jobs
※ YahooとFacebookのworkloadsも似ている
Googleのニーズ● Many batch jobs are short, and fast turnaround is important, so a lightweight, low-quality
approach to placement works just fine.● Long-running, high-priority service jobs must meet stringent availability and performance targets,
so careful placement of their tasks is needed to maximize resistance to failures and provide good performance.
● "head of line blocking" problem: while it is very reasonable to spend a few seconds making a decision whose effects last for several weeks, it can be problematic if an interactive batch job has to wait for such a calculation. This problem can be avoided by introducing parallelism.
つまりGoogleのニーズ：require a scheduler architecture that● can accommodate both types of jobs● flexibly support job-specific policies● and also scale to an ever-growing amount of scheduling work.
なぜgoogleは不採用？Monolithic schedulerとtwo-level schedulerはgoogleのニーズに満たせない:1) Monolithic scheduler:● It complicates an already difficult job: the scheduler has to minimize the
time a job spends waiting before it starts running.● It is surprisingly difficult to support a wide range of policies in a sustainable
manner using a single-algorithm implementation.This kind of software engineering consideration, rather than performance scalability implementation, was our primary motivation to move to an architecture that supported concurrent, independent scheduling components. performance scalabilityよりsoftware engineeringの考えですね！
なぜgoogleは不採用？Monolithic schedulerとtwo-level schedulerはgoogleのニーズに満たせない:2) Two-level scheduler:● No global view of the overall cluster state● Lock issue: pessimistic concurrency control● Assumptions that resource become available frequently and scheduler
decisions are quick, so works best when short tasks/relinquish resource frequently/small job size compared to the size of the cluster: but google's cluster workloads do not have these properties, especially in the case of service jobs
Share-state scheduler(Omega)● each scheduler can full access to the entire cluster● use optimistic concurrency controlThis immediately eliminate two of the issues of the two-level scheduler approach:➔ limited parallelism due to pessimistic concurrency
control➔ restricted visibility of resources in a scheduler
framework
Share-state scheduler(Omega)● No central resource allocator in Omega(be simplified to a persistent data store)● All of the resource-allocation take place in the schedulers.● "cell state": a resilient master copy of the resource allocation maintained in the cluster. Each
scheduler is given a private, local, frequently-updated copy of cell state for making scheduling decisions. The scheduler can see the entire state of the cell.
● Omega schedulers operate completely in parallel and do not have to wait for jobs in other schedulers and there is no inter-scheduler head of line blocking.
The performance viability of the share-state approach is ultimately determined by the frequency at which transactions fail and the costs of such failures.
The batch scheduler is the main scalability bottleneck, the Omega model can scale to a high workload while still providing good behavior for service jobs.
cluster schedulersの比較
Approach Resource Choice
Interference Alloc. granularity
Cluster-wide policies
Monolithic all available none(serialized) global policy strict priority(preemption)
Statically partitioned fixed subnet none(partitioned)
per-partition policy
scheduler-dependent
Two-level(Mesos) dynamic subnet pessimistic hoarding strict fairness
Shared-state(Omega) all available optimistic per-scheduler policy
free-for-all, priority preemption
MesosとPaaSの話PaaS検証の背景(p3)：multiple workloads, multiple tenantsのPaaS上マルチClustersのresource sharing問題
(Dynamic sharingのcluster scheduler)
PaaS上のworkloads：long running processes/one-off tasks/scheduled jobsservice jobsの割合はより高く、service jobsのschedulingはもっと重要
Mesos frameworks for Long running services:Aurora/Marathon/SingularityなどありますがOmegaのpaper(2013)が指摘したMesosの問題(特にService jobsの問題)Mesosの最新状況や各 frameworksの対応はどうになっているか
MesosとPaaSの話Kubernetesについて
Run Kubernetes on Mesos:https://github.com/mesosphere/kubernetes-mesos
Run Kubernetes on Hadoop YARN:http://hortonworks.com/blog/docker-kubernetes-apache-hadoop-yarn/
ReferencesMesos paper:http://mesos.berkeley.edu/mesos_tech_report.pdf
Mesos presentations:http://mesos.apache.org/documentation/latest/mesos-presentations/
Omega paper:http://eurosys2013.tudos.org/wp-content/uploads/2013/paper/Schwarzkopf.pdf
ご静聴、ありがとうございます