Bug 6120 - Missing trailing space with squeue -o %b (GRES)
Summary: Missing trailing space with squeue -o %b (GRES)
Status: RESOLVED FIXED
Alias: None
Product: Slurm
Classification: Unclassified
Component: User Commands (show other bugs)
Version: 18.08.3
Hardware: Linux Linux
: --- 4 - Minor Issue
Assignee: Albert Gil
QA Contact:
URL:
: 6141 6144 (view as bug list)
Depends on:
Blocks:
 
Reported: 2018-11-28 18:06 MST by Stephane Thiell
Modified: 2019-06-10 09:23 MDT (History)
3 users (show)

See Also:
Site: Stanford
Alineos Sites: ---
Bull/Atos Sites: ---
Confidential Site: ---
Cray Sites: ---
HPCnow Sites: ---
HPE Sites: ---
IBM Sites: ---
NOAA SIte: ---
OCF Sites: ---
SFW Sites: ---
SNIC sites: ---
Linux Distro: ---
Machine Name:
CLE Version:
Version Fixed: 18.08.4 19.05.0pre2
Target Release: ---
DevPrio: ---


Attachments
squeue tres print fix (1.73 KB, patch)
2018-12-03 09:01 MST, Jason Booth
Details | Diff
Fixes missing suffixes for TRES. (2.29 KB, patch)
2018-12-03 09:17 MST, Albert Gil
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Stephane Thiell 2018-11-28 18:06:22 MST
Hello Tim,

Minor issue: one of our monitoring script broke today after migrating from Slurm 17 to 18 on Sherlock. It looks like the -o '%b' (GRES) format option of squeue doesn't add a trailing space anymore. See below:


$ squeue -rh -o '%g %u %P %b %T %C %D %R' -j 33003323
rzia emmagg rzia N/ARUNNING 1 1 sh-106-12
$ squeue -rh -o '%g %u %P %b %T %C %D %R' -j 33236568
epop gabourie gpu gpu:1RUNNING 1 1 sh-112-04

Our current workaround is to specify a format length which is long enough to cover all gres types on the cluster:

$ squeue -rh -o '%g %u %P %10b %T %C %D %R' -j 33236568
epop gabourie gpu gpu:1     RUNNING 1 1 sh-112-04
$ squeue -rh -o '%g %u %P %10b %T %C %D %R' -j 33003323
rzia emmagg rzia N/A       RUNNING 1 1 sh-106-12

Thanks!
Stephane
Comment 3 Trey Dockendorf 2018-11-29 13:54:56 MST
Supplied patch in bug report 6141 fixes the issue with N/A printing, curious if fixes actual issue when TRES is present.
Comment 6 Jason Booth 2018-12-03 09:00:13 MST
*** Bug 6141 has been marked as a duplicate of this bug. ***
Comment 7 Jason Booth 2018-12-03 09:01:50 MST
Created attachment 8496 [details]
squeue tres print fix

Trey Dockendorf (tdockendorf@osc.edu) fix / contribution from 6141.
Comment 8 Albert Gil 2018-12-03 09:17:22 MST
Created attachment 8497 [details]
Fixes missing suffixes for TRES.
Comment 11 Albert Gil 2018-12-06 04:41:37 MST
Hi Stephane, Trey,

> Supplied patch in bug report 6141 fixes the issue with N/A printing, curious
> if fixes actual issue when TRES is present.

Yes, the patch fixes both, the trailing spaces and the "DELIM" issues.
In fact, anything after the %b and before any other specified was actually ignored.

Trey's patch has been merged and properly authored at:
https://github.com/SchedMD/slurm/commit/9b0399b84373b3360a61b68c6725b0770c7f9dab

Thanks Trey!


Albert
Comment 12 Danny Auble 2018-12-06 11:55:48 MST
*** Bug 6144 has been marked as a duplicate of this bug. ***
Comment 13 Stephane Thiell 2018-12-07 12:02:25 MST
Thanks!
Comment 14 Kaylea Nelson 2019-06-07 11:35:36 MDT
We are currently on Slurm 18.08.5 and we are still seeing the issue when using the SQUEUE_FORMAT2/-O/--Format formatting options. -o works as expected after the fix.

# -o, works as expected
$ squeue -o "%i,%j"
NAME,JOBID
mesau,32994447_[77-399%20]
schedule-,30983558
schedule-,30983557
schedule-,30983556
LFeHNTMSFeL_-1_m1_BP86_def2-TZVP_OptHess.mpi,32999232
LFeHNTMSFeL_-1_m9_BP86_def2-TZVP_OptHess.mpi,32999246
LFeHCTMSFeL_-1_m2_BP86_def2-TZVP_OptHess.mpi,32999250
LFeHCTMSFeL_-1_m10_BP86_def2-TZVP_OptHess.mpi,32999251
LFeSCHTMS_0_m1_BP86_def2-TZVP_OptHess_NotTight.mpi,33001227

# -O, missing delimiter
$ squeue -O "name,jobid"
NAME                JOBID
mesau               32994447
schedule-           30983558
schedule-           30983557
schedule-           30983556
LFeHNTMSFeL_-1_m1_BP32999232
LFeHNTMSFeL_-1_m9_BP32999246
LFeHCTMSFeL_-1_m2_BP32999250
LFeHCTMSFeL_-1_m10_B32999251
LFeSCHTMS_0_m1_BP86_33001227

# -o, works as expected
$ squeue -o "%i %j"
NAME JOBID
mesau 32994447_[77-399%20]
schedule- 30983558
schedule- 30983557
schedule- 30983556
LFeHNTMSFeL_-1_m1_BP86_def2-TZVP_OptHess.mpi 32999232
LFeHNTMSFeL_-1_m9_BP86_def2-TZVP_OptHess.mpi 32999246
LFeHCTMSFeL_-1_m2_BP86_def2-TZVP_OptHess.mpi 32999250
LFeHCTMSFeL_-1_m10_BP86_def2-TZVP_OptHess.mpi 32999251
LFeSCHTMS_0_m1_BP86_def2-TZVP_OptHess_NotTight.mpi 33001227

# -O, fails completely with any delimiter other than ","
$ squeue -O "name jobid"
squeue: error: Invalid job format specification: name jobid
Comment 16 Albert Gil 2019-06-10 05:15:50 MDT
Hi Kaylea,

> We are currently on Slurm 18.08.5 and we are still seeing the issue when
> using the SQUEUE_FORMAT2/-O/--Format formatting options. -o works as
> expected after the fix.

Actually what you are reporting in comment 14 is the expected behavior (or at least a known limitation) in the sense that as you can see in the manpage one of the differences between -o and -O is that -O "Requests a comma separated list of job information to be displayed".

So, for -O we can only specify the desired fields, but not any other delimiter of character in the list of desired fields.
Just each field, justification and size.

The same happens with sinfo's pair of options.

I can see that you may want this limitation to be removed to allow the same level of flexibility/delimiters between -o and -O though, so if you are still interested and because it's not actually related to the original bug, please file a new bug referencing your comment here as 5-Enhancement and there we will discuss it further.
Is that fine for you?

Thanks,
Albert
Comment 17 Kaylea Nelson 2019-06-10 08:42:40 MDT
Sounds good, will do!
Comment 18 Albert Gil 2019-06-10 09:23:03 MDT
Closing as fixed again.