Minor issue: one of our monitoring script broke today after migrating from Slurm 17 to 18 on Sherlock. It looks like the -o '%b' (GRES) format option of squeue doesn't add a trailing space anymore. See below:
$ squeue -rh -o '%g %u %P %b %T %C %D %R' -j 33003323
rzia emmagg rzia N/ARUNNING 1 1 sh-106-12
$ squeue -rh -o '%g %u %P %b %T %C %D %R' -j 33236568
epop gabourie gpu gpu:1RUNNING 1 1 sh-112-04
Our current workaround is to specify a format length which is long enough to cover all gres types on the cluster:
$ squeue -rh -o '%g %u %P %10b %T %C %D %R' -j 33236568
epop gabourie gpu gpu:1 RUNNING 1 1 sh-112-04
$ squeue -rh -o '%g %u %P %10b %T %C %D %R' -j 33003323
rzia emmagg rzia N/A RUNNING 1 1 sh-106-12
Supplied patch in bug report 6141 fixes the issue with N/A printing, curious if fixes actual issue when TRES is present.
*** Bug 6141 has been marked as a duplicate of this bug. ***
Created attachment 8496 [details]
squeue tres print fix
Trey Dockendorf (email@example.com) fix / contribution from 6141.
Created attachment 8497 [details]
Fixes missing suffixes for TRES.
Hi Stephane, Trey,
> Supplied patch in bug report 6141 fixes the issue with N/A printing, curious
> if fixes actual issue when TRES is present.
Yes, the patch fixes both, the trailing spaces and the "DELIM" issues.
In fact, anything after the %b and before any other specified was actually ignored.
Trey's patch has been merged and properly authored at:
*** Bug 6144 has been marked as a duplicate of this bug. ***
We are currently on Slurm 18.08.5 and we are still seeing the issue when using the SQUEUE_FORMAT2/-O/--Format formatting options. -o works as expected after the fix.
# -o, works as expected
$ squeue -o "%i,%j"
# -O, missing delimiter
$ squeue -O "name,jobid"
# -o, works as expected
$ squeue -o "%i %j"
# -O, fails completely with any delimiter other than ","
$ squeue -O "name jobid"
squeue: error: Invalid job format specification: name jobid
> We are currently on Slurm 18.08.5 and we are still seeing the issue when
> using the SQUEUE_FORMAT2/-O/--Format formatting options. -o works as
> expected after the fix.
Actually what you are reporting in comment 14 is the expected behavior (or at least a known limitation) in the sense that as you can see in the manpage one of the differences between -o and -O is that -O "Requests a comma separated list of job information to be displayed".
So, for -O we can only specify the desired fields, but not any other delimiter of character in the list of desired fields.
Just each field, justification and size.
The same happens with sinfo's pair of options.
I can see that you may want this limitation to be removed to allow the same level of flexibility/delimiters between -o and -O though, so if you are still interested and because it's not actually related to the original bug, please file a new bug referencing your comment here as 5-Enhancement and there we will discuss it further.
Is that fine for you?
Sounds good, will do!
Closing as fixed again.