Summary: | backfill scheduler priority threshold | ||
---|---|---|---|
Product: | Slurm | Reporter: | Doug Jacobsen <dmjacobsen> |
Component: | slurmctld | Assignee: | Moe Jette <jette> |
Status: | RESOLVED FIXED | QA Contact: | |
Severity: | 3 - Medium Impact | ||
Priority: | --- | CC: | jfbotts, jsrinivasan, tinad, tlbutler |
Version: | 15.08.8 | ||
Hardware: | Cray XC | ||
OS: | Linux | ||
Site: | NERSC | Alineos Sites: | --- |
Atos/Eviden Sites: | --- | Confidential Site: | --- |
Coreweave sites: | --- | Cray Sites: | --- |
DS9 clusters: | --- | HPCnow Sites: | --- |
HPE Sites: | --- | IBM Sites: | --- |
NOAA SIte: | --- | OCF Sites: | --- |
Recursion Pharma Sites: | --- | SFW Sites: | --- |
SNIC sites: | --- | Linux Distro: | --- |
Machine Name: | CLE Version: | ||
Version Fixed: | 16.05.0-pre2 | Target Release: | --- |
DevPrio: | --- | Emory-Cloud Sites: | --- |
Attachments: |
Add bf_min_prio_reserve option to SchedulerParameters
Add bf_min_prio_reserve option to SchedulerParameters |
Description
Doug Jacobsen
2016-03-17 05:38:20 MDT
It should be relatively simple. I don't anticipate any problem adding this feature to Slurm version 16.05 and perhaps give you an early patch for 15.08. Created attachment 2886 [details]
Add bf_min_prio_reserve option to SchedulerParameters
This patch is for Slurm version 15.08 and adds a "bf_min_prio_reserve" option to SchedulerParameters. If set, then only jobs with a priority equal to or higher than the configured value will have the backfill scheduler reserve resources for starting at a later time. The time required for processing jobs with a lower priority is vastly lower since the scheduling logic only needs to determine if the job can be started _now_ rather than simulating the termination of all running jobs to determine when and where it might start in the future. If most jobs have a priority lower than the threshold, your bf_max_job_part and bf_max_job_user parameters can be either vastly increased or eliminated.
I do not intend to add this new feature to version 15.08, but it will go into 16.05 with some additional logic for the non-backfill scheduler (which runs at more frequent intervals).
Created attachment 2887 [details]
Add bf_min_prio_reserve option to SchedulerParameters
Slight revision to previous patch based upon merge issues with v16.05
This is fantastic, Moe. Thank you! I'll spin this up on our test system now and start planning a transition if it works as it reads. -Doug ---- Doug Jacobsen, Ph.D. NERSC Computer Systems Engineer National Energy Research Scientific Computing Center <http://www.nersc.gov> dmjacobsen@lbl.gov ------------- __o ---------- _ '\<,_ ----------(_)/ (_)__________________________ On Fri, Mar 18, 2016 at 11:30 AM, <bugs@schedmd.com> wrote: > Moe Jette <jette@schedmd.com> changed bug 2565 > <https://bugs.schedmd.com/show_bug.cgi?id=2565> > What Removed Added > Attachment #2886 [details] is obsolete 1 > > *Comment # 3 <https://bugs.schedmd.com/show_bug.cgi?id=2565#c3> on bug > 2565 <https://bugs.schedmd.com/show_bug.cgi?id=2565> from Moe Jette > <jette@schedmd.com> * > > Created attachment 2887 [details] <https://bugs.schedmd.com/attachment.cgi?id=2887&action=diff> [details] <https://bugs.schedmd.com/attachment.cgi?id=2887&action=edit> > Add bf_min_prio_reserve option to SchedulerParameters > > Slight revision to previous patch based upon merge issues with v16.05 > > ------------------------------ > You are receiving this mail because: > > - You reported the bug. > > I've committed to Slurm version 16.05 the requested functionality here: https://github.com/SchedMD/slurm/commit/45560872d84358779690e1164de842c5061bbd36 Note the changes in v16.05 are a bit more comprehensive than those in the attached patch for v15.08 and there are other changes in the code base, so please work with the attached patch (which was made specifically for v15.08) until you upgrade to v16.05. I'm also going to lower the priority and closing the ticket now, but if you encounter problems feel free to re-open it. Patch also provided for v15.08. Hi Moe, I wanted to let you know that even though we've only had this enabled since Tuesday on cori, and for about 12 hours on edison (starting today at 9AM), the effect on our schedule was immediately noticeable. And our users are seeing the difference as well. Throughput and responsiveness have both improved. The scheduling cycle on cori dropped from 120s as measured by sdiag to 5s overnight and 25-30s during the day. We're reviewing far more jobs for backfill, and utilization is fantastic. Typically cori has been achieving 85-90% at max. Yesterday we hit 95.8% and are on track for 97% today -- all time highs for cori. I really appreciate you implementing this feature for us - thank you. Will let you know how it progresses over time. I did restructure the priority parameters a bit when I enabled it. I've ensured that "premium" priority jobs are run in a qos that has a base priority *at* the threshold level. I've set our debug jobs to run in a qos that starts at a priority that will take 60 minutes to age up to the threshold. Regular jobs 3 days below the priority threshold, and lower for low priority jobs. Other than a small contribution from fairshare, I've removed all "stacking" priority additions (e.g., TRES contributions), so it is almost entirely qos + age with a dash of fairshare for local ordering. -Doug ---- Doug Jacobsen, Ph.D. NERSC Computer Systems Engineer National Energy Research Scientific Computing Center <http://www.nersc.gov> dmjacobsen@lbl.gov ------------- __o ---------- _ '\<,_ ----------(_)/ (_)__________________________ On Fri, Mar 18, 2016 at 2:47 PM, <bugs@schedmd.com> wrote: > Moe Jette <jette@schedmd.com> changed bug 2565 > <https://bugs.schedmd.com/show_bug.cgi?id=2565> > What Removed Added > Status UNCONFIRMED RESOLVED > Resolution --- FIXED > Version Fixed 16.05.0-pre2 > > *Comment # 6 <https://bugs.schedmd.com/show_bug.cgi?id=2565#c6> on bug > 2565 <https://bugs.schedmd.com/show_bug.cgi?id=2565> from Moe Jette > <jette@schedmd.com> * > > Patch also provided for v15.08. > > ------------------------------ > You are receiving this mail because: > > - You reported the bug. > > |