Hi Ole,

> We have upgraded to Slurm 18.08

Good!
We will release soon 19.05.

> and would like to take advantage of the new
> GrpJobsAccrue and MaxJobsAccrue limits defined in
> https://slurm.schedmd.com/resource_limits.html.

Just for curiosity, is there any specific goal, problem or enhancement that you want to achieve with them?

> I can update a user's setting by:
> 
> sacctmgr modify user where name=user01 set GrpJobsAccrue=300
> 
> but I can't inquire the user's current GrpJobsAccrue setting since it isn't
> printed by:
> 
> sacctmgr -s show user user01
> 
> Question 1: How do I inquire the current GrpJobsAccrue and MaxJobsAccrue
> limits for a user or association?

Note that sacctmgr has a different set of possible "formats" for each "entity" that you ask for.
This is the list of entities:
https://slurm.schedmd.com/sacctmgr.html#lbAH

In your case you ask for the entity "user", so you can ask for these "formats":
- Generic for associations: https://slurm.schedmd.com/sacctmgr.html#lbAI
- User specific: https://slurm.schedmd.com/sacctmgr.html#lbBG

As you can see, both GrpJobsAccrue and MaxJobsAccrue are formats available to associations, so the command that you are looking for is:

$ sacctmgr modify user where name=joe set GrpJobsAccrue=300 MaxJobsAccrue=100
$ sacctmgr show assoc where user=joe format=User,GrpJobsAccrue,MaxJobsAccrue
      User GrpJobsAccrue MaxJobsAccrue 
---------- ------------- ------------- 
       joe           300           100 


> I'm considering setting a global cluster value of GrpJobsAccrue.
> Question 2: How do I define the cluster's value of GrpJobsAccrue?

First let me say that GrpJobsAccrue and in general all Grp* parameters are limits assigned to Associations/Accounts and QoSes.
Some other QoS values can also be assigned to Partitions (via slurm.conf).
And very few to "actual clusters" (via some defaults in slurm.conf).

In terms of sacctmgr for "clusters" you can only manage values like ControlHost, ControlPort..:
https://slurm.schedmd.com/sacctmgr.html#lbAO


BUT:
The Slurm Account/Associations scheme has two useful features for your case:
- Associations/Accounts have hierarchy
- A default Account/Association per cluster named "root" is always created and is the default parent for any other account/assoc

Therefore, ff you set GrpJobsAccrue and MaxJobsAccrue at the root account, they will be the default values for "the cluster".
This is what you want, right?
You can do it with:

$ sacctmgr update account root set GrpJobsAccrue=250

But in fact, sacctmgr is smart enough to understand you and this command also works (I didn't know it):

$ sacctmgr update cluster $CLUSTERNAME set GrpJobsAccrue=250

WARNING:
Please note that some limits in the "root" account are not actually enforced.
Please, check the documentation for each case to be sure that this note is not there:
"NOTE: This limit is not enforced if set on the root association of a cluster. So even though it may appear in sacctmgr output, it will not be enforced."
GrpJobsAccrue and MaxJobsAccrue are safe in this sense.

> Apparently no examples of the above questions are documented, and it might
> be nice to add examples to the resource_limits.html page.

Agree!
I'll keep the bug open to improve the docs.

Albert

Hi Albert,

Thanks a lot for your explanations, it's much appreciated!  I just have a quick comment for now:

> Note that sacctmgr has a different set of possible "formats" for each "entity" that you ask for.

This is the crucial information which I've been missing from the sacctmgr man page!!  Searching the man page for the string "format=" (which I didn't realize existed), it turns out that this syntax isn't defined anywhere!!  Only in the examples at the end there are some hits...

In my opinion, all sections of the sacctmgr man page with headings like:
LIST/SHOW .... FORMAT OPTIONS
are missing this crucial definition in line 1 the sections.  For example:

LIST/SHOW ASSOCIATION FORMAT OPTIONS

Print options by: sacctmgr show association format=option1,option2,...,optionN [args]

I plan to reread your answers more carefully within the next few days and respond more fully.

Thanks,
Ole

Hi Ole,


> This is the crucial information which I've been missing from the sacctmgr
> man page!!  Searching the man page for the string "format=" (which I didn't
> realize existed),

Yeah, its probably not well documented.

> it turns out that this syntax isn't defined anywhere!! 
> Only in the examples at the end there are some hits...

Well, it is at the GLOBAL FORMAT OPTION section:
https://slurm.schedmd.com/sacctmgr.html#lbBK

But I also think that it should appear right in the beginning of the manpage.


> In my opinion, all sections of the sacctmgr man page with headings like:
> LIST/SHOW .... FORMAT OPTIONS
> are missing this crucial definition in line 1 the sections.  For example:
> 
> LIST/SHOW ASSOCIATION FORMAT OPTIONS
> 
> Print options by: sacctmgr show association
> format=option1,option2,...,optionN [args]

I think that it is better to move really up the info of GLOBAL FORMAT OPTION and add the "format=<SPECS>" as a new bullet point into OPTIONS.
Less repetitive and even more useful/clear when reading the manpage.
Don't you agree?


> I plan to reread your answers more carefully within the next few days and
> respond more fully.

Go ahead please!

But let me take this opportunity to be more precise and, unlike I said in comment 2, these two commands are not *totally* equivalents:

$ sacctmgr update account root         set GrpJobsAccrue=250
$ sacctmgr update cluster $CLUSTERNAME set GrpJobsAccrue=250

Actually these two are:

$ sacctmgr update account root where cluster=$CLUSTERNAME set GrpJobsAccrue=250
$ sacctmgr update                    cluster $CLUSTERNAME set GrpJobsAccrue=250

Where $CLUSTERNAME is any of your clusters, of course.
If you only have one cluster, then all four are equivalents (that's my lack of precision in comment 2).


And finally also let me note that there is a small difference in how/where GrpJobsAccrue and MaxJobsAccrue are set.
MaxJobsAccrue needs to be defined per association not per QoS (like Fairshare, MaxJobs, MaxSubmitJobs and QOS).
So, it doesn't take the same advantage of the hierarchy like GrpJobsAccrue, or in fact it does it in a different way.
I mean MaxJobsAccrue also uses the hierarchy in terms that a new association/user created under a hierarchy will automatically set the parent MaxJobsAccrue value.
But MaxJobsAccrue needs to be set per association/user to actually work, the parent one is not used for the scheduler, unlike GrpJobsAccrue and others.
See "Association specific scheduling policies supported" in https://slurm.schedmd.com/resource_limits.html

But don't worry, sacctmgr is also smart enough and unlike when setting GrpJobsAccrue in the above commands, these following commands will not only change the root account, but also all its child associations:

$ sacctmgr update account root where cluster=$CLUSTERNAME set MaxJobsAccrue=250
$ sacctmgr update                    cluster $CLUSTERNAME set MaxJobsAccrue=250

(In reply to Albert Gil from comment #2)
> > and would like to take advantage of the new
> > GrpJobsAccrue and MaxJobsAccrue limits defined in
> > https://slurm.schedmd.com/resource_limits.html.
> 
> Just for curiosity, is there any specific goal, problem or enhancement that
> you want to achieve with them?

Some accounts submit large numbers of jobs as part of materials screening studies.  We don't want 1000's of jobs to accrue priority which will eventually block other user jobs which were submitted recently.

Our previous cluster used the MAUI scheduler which had some very useful MAXIJOB and MAXIPROC configurations, see http://docs.adaptivecomputing.com/maui/6.2throttlingpolicies.php#idle.  We would like to implement this very useful feature with the new Slurm xxxJobAccrue settings.


> In your case you ask for the entity "user", so you can ask for these
> "formats":
> - Generic for associations: https://slurm.schedmd.com/sacctmgr.html#lbAI
> - User specific: https://slurm.schedmd.com/sacctmgr.html#lbBG
> 
> As you can see, both GrpJobsAccrue and MaxJobsAccrue are formats available
> to associations, so the command that you are looking for is:
> 
> $ sacctmgr modify user where name=joe set GrpJobsAccrue=300 MaxJobsAccrue=100
> $ sacctmgr show assoc where user=joe format=User,GrpJobsAccrue,MaxJobsAccrue
>       User GrpJobsAccrue MaxJobsAccrue 
> ---------- ------------- ------------- 
>        joe           300           100 

Great!  It's the format= option which we need!

(In reply to Albert Gil from comment #4)
> > This is the crucial information which I've been missing from the sacctmgr
> > man page!!  Searching the man page for the string "format=" (which I didn't
> > realize existed),
> 
> Yeah, its probably not well documented.
> 
> > it turns out that this syntax isn't defined anywhere!! 
> > Only in the examples at the end there are some hits...
> 
> Well, it is at the GLOBAL FORMAT OPTION section:
> https://slurm.schedmd.com/sacctmgr.html#lbBK

That section is somewhat cryptic and buried way down in the man page.

> But I also think that it should appear right in the beginning of the manpage.

I think this would be a crucial improvement of the sacctmgr man page!

> > In my opinion, all sections of the sacctmgr man page with headings like:
> > LIST/SHOW .... FORMAT OPTIONS
> > are missing this crucial definition in line 1 the sections.  For example:
> > 
> > LIST/SHOW ASSOCIATION FORMAT OPTIONS
> > 
> > Print options by: sacctmgr show association
> > format=option1,option2,...,optionN [args]
> 
> I think that it is better to move really up the info of GLOBAL FORMAT OPTION
> and add the "format=<SPECS>" as a new bullet point into OPTIONS.
> Less repetitive and even more useful/clear when reading the manpage.
> Don't you agree?

Yes, that makes sense.  However, each of the sections LIST/SHOW .... FORMAT OPTIONS really don't make sense from a practical perspective.  There is no indication what the long list of format options is really used for, so at the very least you should refer to the "format=<SPECS>" new bullet point in OPTIONS.

> But let me take this opportunity to be more precise and, unlike I said in
> comment 2, these two commands are not *totally* equivalents:
> 
> $ sacctmgr update account root         set GrpJobsAccrue=250
> $ sacctmgr update cluster $CLUSTERNAME set GrpJobsAccrue=250
> 
> Actually these two are:
> 
> $ sacctmgr update account root where cluster=$CLUSTERNAME set
> GrpJobsAccrue=250
> $ sacctmgr update                    cluster $CLUSTERNAME set
> GrpJobsAccrue=250
> 
> Where $CLUSTERNAME is any of your clusters, of course.
> If you only have one cluster, then all four are equivalents (that's my lack
> of precision in comment 2).

Very informative.  We only have 1 cluster, so life is simple :-)

> And finally also let me note that there is a small difference in how/where
> GrpJobsAccrue and MaxJobsAccrue are set.
> MaxJobsAccrue needs to be defined per association not per QoS (like
> Fairshare, MaxJobs, MaxSubmitJobs and QOS).
> So, it doesn't take the same advantage of the hierarchy like GrpJobsAccrue,
> or in fact it does it in a different way.
> I mean MaxJobsAccrue also uses the hierarchy in terms that a new
> association/user created under a hierarchy will automatically set the parent
> MaxJobsAccrue value.
> But MaxJobsAccrue needs to be set per association/user to actually work, the
> parent one is not used for the scheduler, unlike GrpJobsAccrue and others.
> See "Association specific scheduling policies supported" in
> https://slurm.schedmd.com/resource_limits.html

This difference between GrpJobsAccrue and MaxJobsAccrue is really subtle, and one has to read your explanation carefully to appreciate the distinction.

Could you kindly expand the terse definitions of GrpJobsAccrue and MaxJobsAccrue in the sacctmgr and resource_limits pages so that readers would get the idea more easily?

> But don't worry, sacctmgr is also smart enough and unlike when setting
> GrpJobsAccrue in the above commands, these following commands will not only
> change the root account, but also all its child associations:
> 
> $ sacctmgr update account root where cluster=$CLUSTERNAME set
> MaxJobsAccrue=250
> $ sacctmgr update                    cluster $CLUSTERNAME set
> MaxJobsAccrue=250

Thanks again.  I'll have to consider whether to set a cluster default, or set individual values for the associations.  

We already have a local framework for managing user limits available at https://github.com/OleHolmNielsen/Slurm_tools/tree/master/slurmaccounts and this could be expanded to include GrpJobsAccrue and MaxJobsAccrue.

Hi Ole,

> > Just for curiosity, is there any specific goal, problem or enhancement that
> > you want to achieve with them?
> 
> Some accounts submit large numbers of jobs as part of materials screening
> studies.  We don't want 1000's of jobs to accrue priority which will
> eventually block other user jobs which were submitted recently.
> 
> Our previous cluster used the MAUI scheduler which had some very useful
> MAXIJOB and MAXIPROC configurations, see
> http://docs.adaptivecomputing.com/maui/6.2throttlingpolicies.php#idle.  We
> would like to implement this very useful feature with the new Slurm
> xxxJobAccrue settings.

That makes sense.
Maybe some info of bug 3844 could also help you.


> Great!  It's the format= option which we need!

Great!

> We already have a local framework for managing user limits available at https://github.com/OleHolmNielsen/Slurm_tools/tree/master/slurmaccounts and this could be expanded to include GrpJobsAccrue and MaxJobsAccrue.

Thanks for the extra info!
Looks cool.


Once I update the documentation, I'll let you know.


Albert

I have updated my account management tool to include the possibility of configuring GrpJobsAccrue and MaxJobsAccrue limits for each user association.

The tools are in https://github.com/OleHolmNielsen/Slurm_tools/tree/master/slurmaccounts:

* Run slurmusersettings to update your Slurm user settings.
* Run showuserlimits to display a Slurm user's limits.

Hi Ole,

> Once I update the documentation, I'll let you know.

We are merging several documentation improvements into a single internal task.
I close this bug for now, but I'll let you know once the documentation is updated.

Regards,
Albert

For the record, my showuserlimits tool from https://github.com/OleHolmNielsen/Slurm_tools/tree/master/showuserlimits has recently been updated so that it can display all types of Slurm limits, including for example:

showuserlimits -l MaxJobsAccrue
showuserlimits -l GrpJobsAccrue

The way this tool works is to execute "scontrol -o show assoc_mgr users=$username account=$account flags=Assoc" and extract the relevant information.

IMHO, this is a quite general solution to display any Slurm limit, although it's undoubtedly possible to make the tool even better :-)

Do you think that showuserlimits should be considered for a contribution to Slurm?

Thanks,
Ole

(In reply to Ole.H.Nielsen@fysik.dtu.dk from comment #10)
> For the record, my showuserlimits tool from
> https://github.com/OleHolmNielsen/Slurm_tools/tree/master/showuserlimits has
> recently been updated so that it can display all types of Slurm limits,
> including for example:
> 
> showuserlimits -l MaxJobsAccrue
> showuserlimits -l GrpJobsAccrue
> 
> The way this tool works is to execute "scontrol -o show assoc_mgr
> users=$username account=$account flags=Assoc" and extract the relevant
> information.
> 
> IMHO, this is a quite general solution to display any Slurm limit, although
> it's undoubtedly possible to make the tool even better :-)

Thanks for the info!

> Do you think that showuserlimits should be considered for a contribution to
> Slurm?

Although it's true that if you see the current contribs of Slurm it may land on it, actually we are planning to change the future of contribs.
It's still under discussion, but we will probably keep and move the language wrappers out of contribs, move as first citizens the pam staff, something similar with the pmi, and externalize most of the rest pointing to external repos.

So, as your contribution would be a script that runs slurm commands, I don't think that it could land anymore?
May be as such external links/submodules that we are planning to add in the future?

Anyway, I can see its utility and we/you can try to get the same as a subcommand of a current command or similar. I can see it becoming a contribution not in "contribs", but as a code contribution.


Thanks,
Albert

(In reply to Albert Gil from comment #11)
> Although it's true that if you see the current contribs of Slurm it may land
> on it, actually we are planning to change the future of contribs.
> It's still under discussion, but we will probably keep and move the language
> wrappers out of contribs, move as first citizens the pam staff, something
> similar with the pmi, and externalize most of the rest pointing to external
> repos.
> 
> So, as your contribution would be a script that runs slurm commands, I don't
> think that it could land anymore?
> May be as such external links/submodules that we are planning to add in the
> future?

Linking to external tools is fine with me.

> Anyway, I can see its utility and we/you can try to get the same as a
> subcommand of a current command or similar. I can see it becoming a
> contribution not in "contribs", but as a code contribution.

The "showuserlimits" script definitely fills a need for Slurm users.  I agree that it would be better to integrate this functionality into an existing Slurm command, and here "sacctmgr" is the obvious choice.

Let me roughly sketch how one might add new options to the command "sacctmgr show association cluster=xxx account=yyy user=zzz partition=www".

Currently one can specify the selection of a small set of limits using the "format=aaa" option.  Please note that the use of "sacctmgr format=..." is actually undocumented in the https://slurm.schedmd.com/sacctmgr.html page (except for a couple of usage examples).

I propose that the "format=aaa" option should be extended to allow the specification of *any* available limit, PLUS the user's current usage for each limit.

The current limit and usage number is actually printed by the Slurm command:

scontrol -o show assoc_mgr users=xxx account=camdvip flags=assoc

and displays limits in the format like "GrpTRES=cpu=1500(80)" (i.e., Limit(Value) pairs) for every limit and association of user xxx.

Actually, user xxx will also be subject to the limits of his parent account.  The parent limits can be printed by omitting the "users=xxx" option and looking only at the first line with an empty user field.  It would be good to print the user limits together with the parent limits, which is what the "showuserlimits" script does.

As an example, the "showuserlimits" script reveals the full set of limits in our 18.08 installation:

$ showuserlimits 
Association (Parent account):
	   ClusterName = 	niflheim
	       Account = 	camdvip
	      UserName = 	None, current value = Parent account
	     Partition = 	None, current value = Any partition
	            ID = 	25
	SharesRaw/Norm/Level/Factor = 	2147483647/0.00/565/0.00
	UsageRaw/Norm/Efctv = 	3752412418.72/0.11/0.11
	 ParentAccount = 	camd, current value = 16
	           Lft = 	967
	      DefAssoc = 	No
	       GrpJobs = 	None, current value = 29
	 GrpJobsAccrue = 	None, current value = 36
	 GrpSubmitJobs = 	None, current value = 72
	       GrpWall = 	None, current value = 1683530.91
	       GrpTRES = 
		     cpu:	Limit = None, current value = 3152
		     mem:	Limit = None, current value = 29122000
		  energy:	Limit = None, current value = 0
		    node:	Limit = None, current value = 127
		 billing:	Limit = None, current value = 3843
		 fs/disk:	Limit = None, current value = 0
		    vmem:	Limit = None, current value = 0
		   pages:	Limit = None, current value = 0

	   GrpTRESMins = 
		     cpu:	Limit = None, current value = 58267403
		     mem:	Limit = None, current value = 506526257641
		  energy:	Limit = None, current value = 0
		    node:	Limit = None, current value = 3105343
		 billing:	Limit = None, current value = 62429586
		 fs/disk:	Limit = None, current value = 0
		    vmem:	Limit = None, current value = 0
		   pages:	Limit = None, current value = 0

	GrpTRESRunMins = 
		     cpu:	Limit = None, current value = 5724829
		     mem:	Limit = None, current value = 47109030100
		  energy:	Limit = None, current value = 0
		    node:	Limit = None, current value = 250151
		 billing:	Limit = None, current value = 7120542
		 fs/disk:	Limit = None, current value = 0
		    vmem:	Limit = None, current value = 0
		   pages:	Limit = None, current value = 0

	       MaxJobs = 	
	 MaxJobsAccrue = 	
	 MaxSubmitJobs = 	
	     MaxWallPJ = 	
	     MaxTRESPJ = 	
	     MaxTRESPN = 	
	 MaxTRESMinsPJ = 	
	 MinPrioThresh = 	
Association (User):
	   ClusterName = 	niflheim
	       Account = 	camdvip
	      UserName = 	xxx, UID=123456
	     Partition = 	None, current value = Any partition
	            ID = 	58
	SharesRaw/Norm/Level/Factor = 	3/0.01/565/0.50
	UsageRaw/Norm/Efctv = 	975630.49/0.00/0.00
	 ParentAccount = 	
	           Lft = 	1018
	      DefAssoc = 	Yes
	       GrpJobs = 	None, current value = 0
	 GrpJobsAccrue = 	None, current value = 0
	 GrpSubmitJobs = 	None, current value = 0
	       GrpWall = 	None, current value = 241.08
	       GrpTRES = 
		     cpu:	Limit = 2000, current value = 0
		     mem:	Limit = None, current value = 0
		  energy:	Limit = None, current value = 0
		    node:	Limit = None, current value = 0
		 billing:	Limit = None, current value = 0
		 fs/disk:	Limit = None, current value = 0
		    vmem:	Limit = None, current value = 0
		   pages:	Limit = None, current value = 0

	   GrpTRESMins = 
		     cpu:	Limit = None, current value = 9737
		     mem:	Limit = None, current value = 87251849
		  energy:	Limit = None, current value = 0
		    node:	Limit = None, current value = 243
		 billing:	Limit = None, current value = 16068
		 fs/disk:	Limit = None, current value = 0
		    vmem:	Limit = None, current value = 0
		   pages:	Limit = None, current value = 0

	GrpTRESRunMins = 
		     cpu:	Limit = 1000000, current value = 0
		     mem:	Limit = None, current value = 0
		  energy:	Limit = None, current value = 0
		    node:	Limit = None, current value = 0
		 billing:	Limit = None, current value = 0
		 fs/disk:	Limit = None, current value = 0
		    vmem:	Limit = None, current value = 0
		   pages:	Limit = None, current value = 0

	       MaxJobs = 	200, current value = 0
	 MaxJobsAccrue = 	30, current value = 0
	 MaxSubmitJobs = 	200, current value = 0
	     MaxWallPJ = 	
	     MaxTRESPJ = 	
	     MaxTRESPN = 	
	 MaxTRESMinsPJ = 	
	 MinPrioThresh =

Hi Ole,

For a proper management of your request for enhancement and/or your contribution, please fill up a specific new bug with this information (and a link to this one if you want).
We will discuss it internally and let you know.

Thanks for your detailed information and your willingness to contribute,
Albert