As suggested, we use maintenance reservations to prevent jobs from starting that will be killed by the scheduled downtime. However, the "Reason" that is listed for jobs whose walltime request would extend into the reservation is (ReqNodeNotAvail, UnavailableNodes:...) and then it lists all down/drained nodes. Is there a way to make the text of this reason better align with the actual reason the job isn't running (e.g. "Upcoming Maintenance" or "Walltime Overlaps with Maintenance")?
Hi Could you send me slurm.conf, output from "scontrol show res" and scontrol show job <example job>. We are already working on 2 related problems and I need to know could if this is something new or some sort of another bug? Dominik
Created attachment 6483 [details] slurm.conf See attached for slurm.conf. scontrol show res: ReservationName=root_2 StartTime=2017-10-12T19:19:20 EndTime=2018-10-12T19:19:20 Duration=365-00:00:00 Nodes=c14n[01-04] NodeCnt=4 CoreCnt=32 Features=(null) PartitionName=(null) Flags=MAINT,SPEC_NODES TRES=cpu=32 Users=root Accounts=(null) Licenses=(null) State=ACTIVE BurstBuffer=(null) Watts=n/a ReservationName=root_3 StartTime=2017-11-25T00:00:00 EndTime=2018-11-25T00:00:00 Duration=365-00:00:00 Nodes=c19n12 NodeCnt=1 CoreCnt=8 Features=(null) PartitionName=(null) Flags=MAINT,SPEC_NODES TRES=cpu=8 Users=root Accounts=(null) Licenses=(null) State=ACTIVE BurstBuffer=(null) Watts=n/a ReservationName=root_4 StartTime=2018-04-02T08:00:00 EndTime=2019-04-02T08:00:00 Duration=365-00:00:00 Nodes=c01n[02-16],c02n[01-16],c03n[01-16],c04n[01-16],c05n[ 01-16],c06n[01-16],c07n[01-16],c08n[01-16],c09n[01-16],c10n[ 01-16],c11n[01-16],c12n[01-16],c13n[01-16],c14n[01-16],c15n[ 01-16],c16n[01-16],c17n[01-16],c18n[02-16],c19n[01-16],c20n[ 01-16],c21n[01-16],c22n[01-16],c23n[01-16],c24n[01-16],c25n[ 01-16],c26n[01-16],c27n[01-16],c28n[01-16],c29n[01-16],c30n[ 01-16],c31n[01-16],c32n[01-07,09-16],c33n[01-16],c34n[01-16] ,c35n[01-16],c36n[01-16],c37n[01-16],c38n[01-16],c39n[01-16] ,c40n[01-16],c41n[01-16],c42n[01-16],c43n[01-16],c44n[01-16] ,c45n[01-04],c46n[01-05,07-08],c47n[01-04,06-07],c48n[01-07] ,c49n[01-08],c50n[01-08],c51n[01-08],c52n[01-08],c53n[01-08] NodeCnt=765 CoreCnt=6392 Features=(null) PartitionName=(null) Flags=MAINT,IGNORE_JOBS,SPEC_NODES,ALL_NODES TRES=cpu=6392 Users=root Accounts=(null) Licenses=(null) State=INACTIVE BurstBuffer=(null) Watts=n/a Example scontrol show job: JobId=597372 JobName=t16z16defk14a14 UserId=zm56(12261) GroupId=storelvmo(10081) MCS_label=N/A Priority=11204 Nice=0 Account=geo_storelvmo QOS=normal JobState=PENDING Reason=ReqNodeNotAvail,_UnavailableNodes:c03n16, c06n02,c14n[01-04,10],c15n01,c16n06,c17n12,c19n12,c28n08, c35n13,c45n[01-04],c46n[02,04-05],c47n[01,03,06-07],c48n[02- 03],c49n05,c50n[01-02,07],c51n[02,07],c52n[06-07],c53n[01,04-06] Dependency=(null) Requeue=0 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0 RunTime=00:00:00 TimeLimit=7-00:00:00 TimeMin=N/A SubmitTime=2018-03-26T16:35:36 EligibleTime=2018-03-26T16:35:36 StartTime=Unknown EndTime=Unknown Deadline=N/A PreemptTime=None SuspendTime=None SecsPreSuspend=0 Partition=geo AllocNode:Sid=omega2:22355 ReqNodeList=(null) ExcNodeList=(null) NodeList=(null) NumNodes=16-16 NumCPUs=128 NumTasks=128 CPUs/Task=1 ReqB:S:C:T=0:0:*:* TRES=cpu=128,mem=524288,node=16 Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=* MinCPUsNode=1 MinMemoryNode=32G MinTmpDiskNode=0 Features=(null) Gres=(null) Reservation=(null) OverSubscribe=NO Contiguous=0 Licenses=(null) Network=(null) Command=./t16z16defk14a14_merradust_n0.083_d2.39_2004_ uvonly_1.4xwsahara_1.run WorkDir=/lustre/home/client/geo/storelvmo/zm56/cesmruns/ t16z16defk14a14_merradust_n0.083_d2.39_2004_uvonly_1.4xwsahara_1 StdErr=/lustre/home/client/geo/storelvmo/zm56/cesmruns/ t16z16defk14a14_merradust_n0.083_d2.39_2004_uvonly_1. 4xwsahara_1/slurm-597372.out StdIn=/dev/null StdOut=/lustre/home/client/geo/storelvmo/zm56/cesmruns/ t16z16defk14a14_merradust_n0.083_d2.39_2004_uvonly_1. 4xwsahara_1/slurm-597372.out Power= --------------- Kaylea Nelson, PhD | kaylea.nelson@yale.edu Computational Research Support Analyst Yale Center for Research Computing <http://research.computing.yale.edu/> On Tue, Mar 27, 2018 at 6:46 AM, <bugs@schedmd.com> wrote: > *Comment # 1 <https://bugs.schedmd.com/show_bug.cgi?id=4987#c1> on bug > 4987 <https://bugs.schedmd.com/show_bug.cgi?id=4987> from Dominik > Bartkiewicz <bart@schedmd.com> * > > Hi > > Could you send me slurm.conf, output from "scontrol show res" and > scontrol show job <example job>. > We are already working on 2 related problems and I need to know could if this > is something new or some sort of another bug? > > Dominik > > ------------------------------ > You are receiving this mail because: > > - You reported the bug. > >
Hi Commit https://github.com/SchedMD/slurm/commit/fc4e5ac9e0563c8 should change reason string to: "ReqNodeNotAvail,_May_be_reserved_for_other_job" If you want, we can now change this to enhancement. Described feature requests to some changes that can be done only in next major release. Dominik
Hi Dominik, Thanks, this will be really useful the regular reservations. However, is there away to have different reason string for reservations specifically flagged as Maint, such as "ReqNodeNotAvail,_reserved_for_maintenance"? I feel like this new text will still be very confusing to users when there are maintenance reservations. Thanks! (In reply to Dominik Bartkiewicz from comment #3) > Hi > > Commit https://github.com/SchedMD/slurm/commit/fc4e5ac9e0563c8 should change > reason string to: "ReqNodeNotAvail,_May_be_reserved_for_other_job" > > If you want, we can now change this to enhancement. > Described feature requests to some changes that can be done only in next > major release. > > Dominik
*** Bug 5138 has been marked as a duplicate of this bug. ***
Hi In commit https://github.com/SchedMD/slurm/commi/4b9a7589067d88 we added a new job pending reason of "ReqNodeNotAvail, reserved for maintenance". It will be in released 18.08. I will close this ticket. Dominik
