flux: practical job schedulingflux-framework.org/papers/flux-devday-2018-slides.pdfflux can run...
TRANSCRIPT
![Page 1: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/1.jpg)
LLNL-PRES-757227 This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under contract DE-AC52-07NA27344. Lawrence Livermore National Security, LLC
Flux: Practical Job Scheduling
Dong H. Ahn, Ned Bass, Al Chu, Jim Garlick, Mark Grondona, Stephen Herbein, Tapasya Patki, Tom Scogland, Becky Springmeyer
August 15, 2018
![Page 2: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/2.jpg)
LLNL-PRES-7572272
What is Flux?
▪ New Resource and Job Management Software (RJMS) developed here at LLNL
▪ A way to manage remote resources and execute tasks on them
![Page 3: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/3.jpg)
LLNL-PRES-7572273
What is Flux?
▪ New Resource and Job Management Software (RJMS) developed here at LLNL
▪ A way to manage remote resources and execute tasks on them
![Page 4: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/4.jpg)
LLNL-PRES-7572273
What is Flux?
▪ New Resource and Job Management Software (RJMS) developed here at LLNL
▪ A way to manage remote resources and execute tasks on them
![Page 5: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/5.jpg)
LLNL-PRES-7572273
What is Flux?
▪ New Resource and Job Management Software (RJMS) developed here at LLNL
▪ A way to manage remote resources and execute tasks on them
flickr: dannychamoro
![Page 6: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/6.jpg)
LLNL-PRES-7572274
What is Flux?
▪ New Resource and Job Management Software (RJMS) developed here at LLNL
▪ A way to manage remote resources and execute tasks on them
![Page 7: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/7.jpg)
LLNL-PRES-7572274
What is Flux?
▪ New Resource and Job Management Software (RJMS) developed here at LLNL
▪ A way to manage remote resources and execute tasks on them
![Page 8: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/8.jpg)
LLNL-PRES-7572274
What is Flux?
▪ New Resource and Job Management Software (RJMS) developed here at LLNL
▪ A way to manage remote resources and execute tasks on them
![Page 9: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/9.jpg)
LLNL-PRES-7572275
What about …?
![Page 10: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/10.jpg)
LLNL-PRES-7572276
What about …?
Closed-source
![Page 11: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/11.jpg)
LLNL-PRES-7572277
What about …?
Not designed for HPC
![Page 12: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/12.jpg)
LLNL-PRES-7572278
What about …?
Limited Scalability, Usability, and Portability
![Page 13: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/13.jpg)
LLNL-PRES-7572279
Why Flux?
![Page 14: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/14.jpg)
LLNL-PRES-7572279
▪ Extensibility — Open source — Modular design with support for user plugins
Why Flux?
![Page 15: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/15.jpg)
LLNL-PRES-7572279
▪ Extensibility — Open source — Modular design with support for user plugins
▪ Scalability — Designed from the ground up for exascale and beyond — Already tested at 1000s of nodes & millions of jobs
Why Flux?
![Page 16: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/16.jpg)
LLNL-PRES-7572279
▪ Extensibility — Open source — Modular design with support for user plugins
▪ Scalability — Designed from the ground up for exascale and beyond — Already tested at 1000s of nodes & millions of jobs
▪ Usability — C, Lua, and Python bindings that expose 100% of Flux’s functionality — Can be used as a single-user tool or a system scheduler
Why Flux?
![Page 17: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/17.jpg)
LLNL-PRES-7572279
▪ Extensibility — Open source — Modular design with support for user plugins
▪ Scalability — Designed from the ground up for exascale and beyond — Already tested at 1000s of nodes & millions of jobs
▪ Usability — C, Lua, and Python bindings that expose 100% of Flux’s functionality — Can be used as a single-user tool or a system scheduler
▪ Portability — Optimized for HPC and runs in Cloud and Grid settings too — Runs on any set of Linux machines: only requires a list of IP addresses or PMI
Why Flux?
![Page 18: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/18.jpg)
LLNL-PRES-7572279
▪ Extensibility — Open source — Modular design with support for user plugins
▪ Scalability — Designed from the ground up for exascale and beyond — Already tested at 1000s of nodes & millions of jobs
▪ Usability — C, Lua, and Python bindings that expose 100% of Flux’s functionality — Can be used as a single-user tool or a system scheduler
▪ Portability — Optimized for HPC and runs in Cloud and Grid settings too — Runs on any set of Linux machines: only requires a list of IP addresses or PMI
Why Flux?
Flux is designed to make hard scheduling problems easy
![Page 19: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/19.jpg)
LLNL-PRES-75722710
Portability: Running Flux
![Page 20: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/20.jpg)
LLNL-PRES-75722710
▪ Already installed on LC systems (including Sierra) — spack install flux-sched for everywhere else
Portability: Running Flux
![Page 21: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/21.jpg)
LLNL-PRES-75722710
▪ Already installed on LC systems (including Sierra) — spack install flux-sched for everywhere else
▪ Flux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource allocation from: itself (hierarchical Flux), Slurm, Moab, PBS, LSF, etc — flux start OR srun flux start
Portability: Running Flux
![Page 22: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/22.jpg)
LLNL-PRES-75722710
▪ Already installed on LC systems (including Sierra) — spack install flux-sched for everywhere else
▪ Flux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource allocation from: itself (hierarchical Flux), Slurm, Moab, PBS, LSF, etc — flux start OR srun flux start
▪ Flux can run anywhere that supports TCP and you have the IP addresses — flux broker -Sboot.method=config -Sboot.config_file=boot.conf — boot.conf:
Portability: Running Flux
session-id = "mycluster" tbon-endpoints = [ "tcp://192.168.1.1:8020", "tcp://192.168.1.2:8020", "tcp://192.168.1.3:8020"]
![Page 23: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/23.jpg)
LLNL-PRES-75722711
▪ Extensibility — Open source — Modular design with support for user plugins
▪ Scalability — Designed from the ground up for exascale and beyond — Already tested at 1000s of nodes & millions of jobs
▪ Usability — C, Lua, and Python bindings that expose 100% of Flux’s functionality — Can be used as a single-user tool or a system scheduler
▪ Portability — Optimized for HPC and runs in Cloud and Grid settings too — Runs on any set of Linux machines: only requires a list of IP addresses or PMI
Why Flux?
![Page 24: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/24.jpg)
LLNL-PRES-75722712
Usability: Submitting a Batch Job
![Page 25: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/25.jpg)
LLNL-PRES-75722712
▪ Slurm — sbatch –N2 –n4 –t 2:00 sleep 120
Usability: Submitting a Batch Job
![Page 26: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/26.jpg)
LLNL-PRES-75722712
▪ Slurm — sbatch –N2 –n4 –t 2:00 sleep 120
▪ Flux CLI — flux submit –N2 –n4 –t 2m sleep 120
Usability: Submitting a Batch Job
![Page 27: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/27.jpg)
LLNL-PRES-75722712
▪ Slurm — sbatch –N2 –n4 –t 2:00 sleep 120
▪ Flux CLI — flux submit –N2 –n4 –t 2m sleep 120
Usability: Submitting a Batch Job
▪ Flux API: import json, flux
jobreq = { 'nnodes' : 2, 'ntasks' : 4, 'walltime' : 120, 'cmdline' : ["sleep", "120"]}
f = flux.Flux () resp = f.rpc_send ("job.submit", json.dumps(jobreq))
![Page 28: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/28.jpg)
LLNL-PRES-75722713
Usability: Running an Interactive Job
![Page 29: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/29.jpg)
LLNL-PRES-75722713
▪ Slurm — srun –N2 –n4 –t 2:00 sleep 120
Usability: Running an Interactive Job
![Page 30: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/30.jpg)
LLNL-PRES-75722713
▪ Slurm — srun –N2 –n4 –t 2:00 sleep 120
▪ Flux CLI — flux wreckrun –N2 –n4 –t 2m sleep 120
Usability: Running an Interactive Job
![Page 31: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/31.jpg)
LLNL-PRES-75722713
▪ Slurm — srun –N2 –n4 –t 2:00 sleep 120
▪ Flux CLI — flux wreckrun –N2 –n4 –t 2m sleep 120
Usability: Running an Interactive Job
▪ Flux API: import sys from flux import kz
resp = f.rpc_send ("job.submit", json.dumps(jobreq)) kvs_dir = resp['kvs_dir']
for task_id in range(jobreq['ntasks']): kz.attach (f, "{}.{}.stdout".format(kvs_dir, task_id), sys.stdout)
f.reactor_run (f.get_reactor (), 0)
![Page 32: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/32.jpg)
LLNL-PRES-75722714
Usability: Tracking Job Status
![Page 33: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/33.jpg)
LLNL-PRES-75722714
Usability: Tracking Job Status
▪ CLI: slow, non-programmatic, inconvenient to parse — watch squeue –j JOBID — watch flux wreck ls JOBID
![Page 34: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/34.jpg)
LLNL-PRES-75722714
Usability: Tracking Job Status
▪ CLI: slow, non-programmatic, inconvenient to parse — watch squeue –j JOBID — watch flux wreck ls JOBID
▪ Tracking via the filesystem — date > $JOBID.start; srun myApp; date > $JOBID.stop
![Page 35: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/35.jpg)
LLNL-PRES-75722714
Usability: Tracking Job Status
▪ CLI: slow, non-programmatic, inconvenient to parse — watch squeue –j JOBID — watch flux wreck ls JOBID
▪ Tracking via the filesystem — date > $JOBID.start; srun myApp; date > $JOBID.stop
→ quota -vf ~/quota.conf Disk quotas for herbein1: Filesystem used quota limit files /p/lscratchrza 760.3G n/a n/a 8.6M
![Page 36: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/36.jpg)
LLNL-PRES-75722714
Usability: Tracking Job Status
▪ CLI: slow, non-programmatic, inconvenient to parse — watch squeue –j JOBID — watch flux wreck ls JOBID
▪ Tracking via the filesystem — date > $JOBID.start; srun myApp; date > $JOBID.stop
→ quota -vf ~/quota.conf Disk quotas for herbein1: Filesystem used quota limit files /p/lscratchrza 760.3G n/a n/a 8.6M
UQP StartupJob SubmissionFile CreationFile Access
I/O
I/ONon-I/ORuntime Stages
![Page 37: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/37.jpg)
LLNL-PRES-75722714
Usability: Tracking Job Status
▪ CLI: slow, non-programmatic, inconvenient to parse — watch squeue –j JOBID — watch flux wreck ls JOBID
▪ Tracking via the filesystem — date > $JOBID.start; srun myApp; date > $JOBID.stop
▪ Push notification via Flux’s Job Status and Control (JSC): def jsc_cb (jcbstr, arg, errnum): jcb = json.loads (jcbstr) jobid = jcb['jobid'] state = jsc.job_num2state (jcb[jsc.JSC_STATE_PAIR][jsc.JSC_STATE_PAIR_NSTATE]) print "flux.jsc: job", jobid, "changed its state to ", state
jsc.notify_status (f, jsc_cb, None)
![Page 38: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/38.jpg)
LLNL-PRES-75722715
▪ Extensibility — Open source — Modular design with support for user plugins
▪ Scalability — Designed from the ground up for exascale and beyond — Already tested at 1000s of nodes & millions of jobs
▪ Usability — C, Lua, and Python bindings that expose 100% of Flux’s functionality — Can be used as a single-user tool or a system scheduler
▪ Portability — Optimized for HPC and runs in Cloud and Grid settings too — Runs on any set of Linux machines: only requires a list of IP addresses or PMI
Why Flux?
![Page 39: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/39.jpg)
LLNL-PRES-75722716
Scalability: Running Many Jobs
![Page 40: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/40.jpg)
LLNL-PRES-75722716
▪ Slurm — find ./ -exec sbatch –N1 tar –cf {}.tgz {}\;
• Slow: requires acquiring a lock in Slurm, can timeout causing failures • Inefficient: uses 1 node for each task
— find ./ -exec srun –n1 tar –cf {}.tgz {}\; • Slow: spawns a process for every submission • Inefficient: is not a true scheduler – can overlap tasks on cores
Scalability: Running Many Jobs
![Page 41: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/41.jpg)
LLNL-PRES-75722716
▪ Slurm — find ./ -exec sbatch –N1 tar –cf {}.tgz {}\;
• Slow: requires acquiring a lock in Slurm, can timeout causing failures • Inefficient: uses 1 node for each task
— find ./ -exec srun –n1 tar –cf {}.tgz {}\; • Slow: spawns a process for every submission • Inefficient: is not a true scheduler – can overlap tasks on cores
Scalability: Running Many Jobs
![Page 42: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/42.jpg)
LLNL-PRES-75722716
▪ Slurm — find ./ -exec sbatch –N1 tar –cf {}.tgz {}\;
• Slow: requires acquiring a lock in Slurm, can timeout causing failures • Inefficient: uses 1 node for each task
— find ./ -exec srun –n1 tar –cf {}.tgz {}\; • Slow: spawns a process for every submission • Inefficient: is not a true scheduler – can overlap tasks on cores
Scalability: Running Many Jobs
![Page 43: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/43.jpg)
LLNL-PRES-75722716
▪ Slurm — find ./ -exec sbatch –N1 tar –cf {}.tgz {}\;
• Slow: requires acquiring a lock in Slurm, can timeout causing failures • Inefficient: uses 1 node for each task
— find ./ -exec srun –n1 tar –cf {}.tgz {}\; • Slow: spawns a process for every submission • Inefficient: is not a true scheduler – can overlap tasks on cores
Scalability: Running Many Jobs
▪ Flux API: for f in os.listdir(‘.’): payload[‘command’] = [“tar”, “-cf”, ”{}.tgz”.format(f), f] resp = f.rpc_send ("job.submit", payload)
![Page 44: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/44.jpg)
LLNL-PRES-75722716
▪ Slurm — find ./ -exec sbatch –N1 tar –cf {}.tgz {}\;
• Slow: requires acquiring a lock in Slurm, can timeout causing failures • Inefficient: uses 1 node for each task
— find ./ -exec srun –n1 tar –cf {}.tgz {}\; • Slow: spawns a process for every submission • Inefficient: is not a true scheduler – can overlap tasks on cores
Scalability: Running Many Jobs
▪ Flux API: for f in os.listdir(‘.’): payload[‘command’] = [“tar”, “-cf”, ”{}.tgz”.format(f), f] resp = f.rpc_send ("job.submit", payload)
![Page 45: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/45.jpg)
LLNL-PRES-75722716
▪ Slurm — find ./ -exec sbatch –N1 tar –cf {}.tgz {}\;
• Slow: requires acquiring a lock in Slurm, can timeout causing failures • Inefficient: uses 1 node for each task
— find ./ -exec srun –n1 tar –cf {}.tgz {}\; • Slow: spawns a process for every submission • Inefficient: is not a true scheduler – can overlap tasks on cores
Scalability: Running Many Jobs
▪ Flux API: for f in os.listdir(‘.’): payload[‘command’] = [“tar”, “-cf”, ”{}.tgz”.format(f), f] resp = f.rpc_send ("job.submit", payload)
Subject: Good Neighbor Policy You currently have 271 jobs in the batch system on lamoab.The good neighbor policy is that users keep their maximum submitted job count at a maximum of 200 or less. Please try to restrict yourself to this limit in the future. Thank you.
![Page 46: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/46.jpg)
LLNL-PRES-75722716
▪ Slurm — find ./ -exec sbatch –N1 tar –cf {}.tgz {}\;
• Slow: requires acquiring a lock in Slurm, can timeout causing failures • Inefficient: uses 1 node for each task
— find ./ -exec srun –n1 tar –cf {}.tgz {}\; • Slow: spawns a process for every submission • Inefficient: is not a true scheduler – can overlap tasks on cores
Scalability: Running Many Jobs
▪ Flux API: for f in os.listdir(‘.’): payload[‘command’] = [“tar”, “-cf”, ”{}.tgz”.format(f), f] resp = f.rpc_send ("job.submit", payload)
Constant Output Job StreamCapacitor SchedulerVariable Input Job Stream
![Page 47: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/47.jpg)
LLNL-PRES-75722716
▪ Slurm — find ./ -exec sbatch –N1 tar –cf {}.tgz {}\;
• Slow: requires acquiring a lock in Slurm, can timeout causing failures • Inefficient: uses 1 node for each task
— find ./ -exec srun –n1 tar –cf {}.tgz {}\; • Slow: spawns a process for every submission • Inefficient: is not a true scheduler – can overlap tasks on cores
▪ Flux Capacitor — find ./ -printf -n1 tar –cf %p.tgz %p | flux-capacitor — flux-capacitor --command_file my_command_file
• -n1 tar -cf dirA.tgz ./dirA • -n1 tar -cf dirB.tgz ./dirB • -n1 tar -cf dirC.tgz ./dirC
Scalability: Running Many Jobs
▪ Flux API: for f in os.listdir(‘.’): payload[‘command’] = [“tar”, “-cf”, ”{}.tgz”.format(f), f] resp = f.rpc_send ("job.submit", payload)
![Page 48: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/48.jpg)
LLNL-PRES-75722717
Scalability: Running Many Heterogeneous Jobs
![Page 49: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/49.jpg)
LLNL-PRES-75722717
Scalability: Running Many Heterogeneous Jobs
▪ Slurm — No support for heterogeneous job steps in versions before 17.11 — Limited support in versions after 17.11
![Page 50: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/50.jpg)
LLNL-PRES-75722717
Scalability: Running Many Heterogeneous Jobs
▪ Slurm — No support for heterogeneous job steps in versions before 17.11 — Limited support in versions after 17.11
https://slurm.schedmd.com/heterogeneous_jobs.html#limitations
![Page 51: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/51.jpg)
LLNL-PRES-75722717
Scalability: Running Many Heterogeneous Jobs
▪ Slurm — No support for heterogeneous job steps in versions before 17.11 — Limited support in versions after 17.11
▪ Flux Capacitor — flux-capacitor --command_file my_command_file
• -n1 tar -cf dirA.tgz ./dirA • -n32 make –j 32 • -N4 my_mpi_app • ...
![Page 52: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/52.jpg)
LLNL-PRES-75722718
Scalability: Running Millions of Jobs
![Page 53: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/53.jpg)
LLNL-PRES-75722718
▪ Flux Capacitor (Depth-1) — flux-capacitor --command_file my_command_file
Scalability: Running Millions of Jobs
Cluster
Node
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
Node
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
Node
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
Node
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
Flux InstanceCapacitor
![Page 54: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/54.jpg)
LLNL-PRES-75722718
▪ Flux Capacitor (Depth-1) — flux-capacitor --command_file my_command_file
▪ Hierarchical Flux Capacitor (Depth-2) — for x in ./*.commands; do
flux submit -N1 flux start \ flux-capacitor --command_file $xdone
Scalability: Running Millions of Jobs
Cluster
Node
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
Node
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
Node
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
Node
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
Flux Instance
Flux Instance Flux Instance Flux Instance Flux InstanceCapacitor Capacitor Capacitor Capacitor
![Page 55: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/55.jpg)
LLNL-PRES-75722718
▪ Flux Capacitor (Depth-1) — flux-capacitor --command_file my_command_file
▪ Hierarchical Flux Capacitor (Depth-2) — for x in ./*.commands; do
flux submit -N1 flux start \ flux-capacitor --command_file $xdone
▪ Flux Hierarchy (Depth-3+) — flux-hierarchy --config=config.json
--command_file my_command_file
Scalability: Running Millions of Jobs
Cluster
Node
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
Node
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
Node
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
Node
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
Flux Instance
Flux Instance Flux Instance Flux Instance Flux Instance
Capacitor
![Page 56: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/56.jpg)
LLNL-PRES-75722718
▪ Flux Capacitor (Depth-1) — flux-capacitor --command_file my_command_file
▪ Hierarchical Flux Capacitor (Depth-2) — for x in ./*.commands; do
flux submit -N1 flux start \ flux-capacitor --command_file $xdone
▪ Flux Hierarchy (Depth-3+) — flux-hierarchy --config=config.json
--command_file my_command_file
Scalability: Running Millions of Jobs
![Page 57: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/57.jpg)
LLNL-PRES-75722719
▪ Extensibility — Open source — Modular design with support for user plugins
▪ Scalability — Designed from the ground up for exascale and beyond — Already tested at 1000s of nodes & millions of jobs
▪ Usability — C, Lua, and Python bindings that expose 100% of Flux’s functionality — Can be used as a single-user tool or a system scheduler
▪ Portability — Optimized for HPC and runs in Cloud and Grid settings too — Runs on any set of Linux machines: only requires a list of IP addresses or PMI
Why Flux?
![Page 58: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/58.jpg)
LLNL-PRES-75722720
Extensibility: Modular Design
![Page 59: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/59.jpg)
LLNL-PRES-75722720
▪ At the core of Flux is an overlay network — Built on top of ZeroMQ — Supports RPCs, Pub/Sub, Push/Pull, etc
Extensibility: Modular Design
Msg Idioms (RPC/Pub-Sub)
Overlay Networks & Routing
Comms Message Broker
Flux Instance
![Page 60: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/60.jpg)
LLNL-PRES-75722720
▪ At the core of Flux is an overlay network — Built on top of ZeroMQ — Supports RPCs, Pub/Sub, Push/Pull, etc
▪ Modules provide extended functionality (i.e., services) — User-built modules are loadable too — Some modules also support plugins
Extensibility: Modular Design
Msg Idioms (RPC/Pub-Sub)
Overlay Networks & Routing
Comms Message Broker
Flux Instance
Sched Framework
Remote Execution
Polic
y Pl
ugin
A
Service Modules
Resource
Key-Value Store
Heartbeat
![Page 61: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/61.jpg)
LLNL-PRES-75722720
▪ At the core of Flux is an overlay network — Built on top of ZeroMQ — Supports RPCs, Pub/Sub, Push/Pull, etc
▪ Modules provide extended functionality (i.e., services) — User-built modules are loadable too — Some modules also support plugins
▪ External tools and commands can access services — User authentication and roles supported
Extensibility: Modular Design
Msg Idioms (RPC/Pub-Sub)
Overlay Networks & Routing
Comms Message Broker
Flux Instance
Sched Framework
Remote Execution
Polic
y Pl
ugin
A
Service Modules
Resource
Key-Value Store
Heartbeat
Commands
flux submit
flux-capacitor
![Page 62: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/62.jpg)
LLNL-PRES-75722721
Extensibility: Creating Your Own Module
![Page 63: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/63.jpg)
LLNL-PRES-75722721
▪ Register a new service “pymod.new_job” that ingests jobs and responds with a Job ID
Extensibility: Creating Your Own Module
![Page 64: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/64.jpg)
LLNL-PRES-75722721
▪ Register a new service “pymod.new_job” that ingests jobs and responds with a Job ID
Extensibility: Creating Your Own Module
import itertools, json, flux
def handle_new_job(f, typemask, message, arg): job_queue, job_ids = arg job_queue.append(message.payload) response = {‘jobid’ : job_ids.next()} f.respond(message, 0, json.dumps(response))
def mod_main(f, *argv): f.msg_watcher_create(flux.FLUX_MSGTYPE_REQUEST, handle_new_job,"pymod.new_job", args=([], itertools.count(0))).start() f.reactor_run(f.get_reactor(), 0)
![Page 65: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/65.jpg)
LLNL-PRES-75722721
▪ Register a new service “pymod.new_job” that ingests jobs and responds with a Job ID
▪ Load using flux module load pymod --module=path/to/file.py
Extensibility: Creating Your Own Module
import itertools, json, flux
def handle_new_job(f, typemask, message, arg): job_queue, job_ids = arg job_queue.append(message.payload) response = {‘jobid’ : job_ids.next()} f.respond(message, 0, json.dumps(response))
def mod_main(f, *argv): f.msg_watcher_create(flux.FLUX_MSGTYPE_REQUEST, handle_new_job,"pymod.new_job", args=([], itertools.count(0))).start() f.reactor_run(f.get_reactor(), 0)
![Page 66: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/66.jpg)
LLNL-PRES-75722722
Extensibility: Flux’s Communication Overlay
![Page 67: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/67.jpg)
LLNL-PRES-75722722
▪ Connect to a running flux instance — f = flux.Flux()
Extensibility: Flux’s Communication Overlay
![Page 68: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/68.jpg)
LLNL-PRES-75722722
▪ Connect to a running flux instance — f = flux.Flux()
▪ Send an RPC to a service and receive a response — resp = f.rpc_send (”pymod.new_job", payload)
jobid = json.loads(resp)[‘jobid’]
Extensibility: Flux’s Communication Overlay
![Page 69: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/69.jpg)
LLNL-PRES-75722722
▪ Connect to a running flux instance — f = flux.Flux()
▪ Send an RPC to a service and receive a response — resp = f.rpc_send (”pymod.new_job", payload)
jobid = json.loads(resp)[‘jobid’]
▪ Subscribe to and publish an event — f.event_subscribe(“node_down”)
f.msg_watcher_create(node_down_cb, raw.FLUX_MSGTYPE_EVENT, “node_down”).start()
— f.event_send(“node_down”)
Extensibility: Flux’s Communication Overlay
![Page 70: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/70.jpg)
LLNL-PRES-75722723
Extensibility: Scheduler Plugins
![Page 71: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/71.jpg)
LLNL-PRES-75722723
▪ Common, built-in scheduler plugins: — First-come First-Served (FCFS) — Backfilling
• Conservative • EASY • Hybrid
Extensibility: Scheduler Plugins
![Page 72: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/72.jpg)
LLNL-PRES-75722723
▪ Common, built-in scheduler plugins: — First-come First-Served (FCFS) — Backfilling
• Conservative • EASY • Hybrid
▪ Various, advanced scheduler plugins: — I/O-aware — CPU performance variability aware — Network-aware
Extensibility: Scheduler Plugins
![Page 73: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/73.jpg)
LLNL-PRES-75722723
▪ Common, built-in scheduler plugins: — First-come First-Served (FCFS) — Backfilling
• Conservative • EASY • Hybrid
▪ Various, advanced scheduler plugins: — I/O-aware — CPU performance variability aware — Network-aware
▪ Create your own!
Extensibility: Scheduler Plugins
![Page 74: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/74.jpg)
LLNL-PRES-75722723
▪ Common, built-in scheduler plugins: — First-come First-Served (FCFS) — Backfilling
• Conservative • EASY • Hybrid
▪ Various, advanced scheduler plugins: — I/O-aware — CPU performance variability aware — Network-aware
▪ Create your own!
▪ Loading the plugins — flux module load sched.io-aware — FLUX_SCHED_OPTS="plugin=sched.fcfs" flux start
Extensibility: Scheduler Plugins
![Page 75: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/75.jpg)
LLNL-PRES-75722724
▪ Flux-Framework code is available on GitHub
▪ Most project discussions happen in GitHub issues
▪ PRs and collaboration welcome!
Extensibility: Open Source
![Page 76: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/76.jpg)
LLNL-PRES-75722724
▪ Flux-Framework code is available on GitHub
▪ Most project discussions happen in GitHub issues
▪ PRs and collaboration welcome!
Extensibility: Open Source
Thank You!
![Page 77: Flux: Practical Job Schedulingflux-framework.org/papers/Flux-DevDay-2018-Slides.pdfFlux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource](https://reader034.vdocuments.site/reader034/viewer/2022051912/60034ec631d0d66ed11bea0a/html5/thumbnails/77.jpg)
Disclaimer This document was prepared as an account of work sponsored by an agency of the United States government. Neither the United States government nor Lawrence Livermore National Security, LLC, nor any of their employees makes any warranty, expressed or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately owned rights. Reference herein to any specific commercial product, process, or service by trade name, trademark, manufacturer, or otherwise does not necessarily constitute or imply its endorsement, recommendation, or favoring by the United States government or Lawrence Livermore National Security, LLC. The views and opinions of authors expressed herein do not necessarily state or reflect those of the United States government or Lawrence Livermore National Security, LLC, and shall not be used for advertising or product endorsement purposes.