Bug 2423 - UsageFactor to effect Accounting
Summary: UsageFactor to effect Accounting
Status: OPEN
Alias: None
Product: Slurm
Classification: Unclassified
Component: Accounting (show other bugs)
Version: 15.08.6
Hardware: Linux Linux
: --- 5 - Enhancement
Assignee: Unassigned Developer
QA Contact:
URL:
Depends on:
Blocks:
 
Reported: 2016-02-07 23:47 MST by Michael Ver Haag
Modified: 2016-06-02 07:01 MDT (History)
1 user (show)

See Also:
Site: Johns Hopkins University
Alineos Sites: ---
Atos/Eviden Sites: ---
Confidential Site: ---
Coreweave sites: ---
Cray Sites: ---
DS9 clusters: ---
HPCnow Sites: ---
HPE Sites: ---
IBM Sites: ---
NOAA SIte: ---
OCF Sites: ---
Recursion Pharma Sites: ---
SFW Sites: ---
SNIC sites: ---
Linux Distro: ---
Machine Name:
CLE Version:
Version Fixed:
Target Release: ---
DevPrio: ---
Emory-Cloud Sites: ---


Attachments
Patch to allow job submission limits to reflect usage_factor (3.63 KB, patch)
2016-03-04 04:45 MST, Kevin Hildebrand
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Michael Ver Haag 2016-02-07 23:47:06 MST
According to the QOS documentation it seems that the UsageFactor only applies to the multifactor Priority Plugin. 
This appears to be the way that this is functioning. 
We are attempting to utilize an strict allocation policy of hours per quarter with a preemptable scavenger qos with usageFactor=0 for users who have depleted their allocations. We would like to be able to still be able to observer real utilization vs accounted utilization 

Is there a way that this can be accomplished with the current code or will we need to request a feature? If so will that need to be a sponsored feature?
Comment 1 Danny Auble 2016-02-11 03:59:57 MST
Hey Michael, I'll look into what this would take.  I'll get something back to you in a bit.  It will probably be a configuration option in the slurmdbd.conf.

The development will probably only support the current usagefactor of the QOS, so if the QOS usagefactor changes over time it will only use the latest value.

Would you want this to work for sacct as well as sreport, or just sreport?  I'm guessing both.
Comment 2 Michael Ver Haag 2016-02-11 05:29:38 MST
Our primary concern is that users submitting jobs to a qos with usageFactor=0 do not end up pending with (AssocGrpCPUMinutesLimit) 
Secondarily jobs run in the scavenger qos do not get "billed" for usage
ie The amount that an account is "billed" for usage is base multiplied by the 
usageFactor
Finally, As far as reporting is concerned, we would need any one tool(probably sreport) that would allow us to get total utilization and billed utilization per user and account. We need to report in near real time "remaining hours" to users and quarterly to allocation provisioners to inform reallocation of resources.   

Current usageFactor should be fine as the usage factors do not change frequently(if ever) 
Our current reporting tools are based on sreport cluster AccountUtilizationByUser  and scontrol show cache so we do not have a priority for sacct
Comment 3 Danny Auble 2016-02-11 05:39:10 MST
Ah, I see, the usageFactor currently does get applied to all the time based limits on the association as well as QOS.  It just doesn't get applied to the numbers sacct and sreport report.

Are you seeing something else as far as this is concerned, or are you just reporting what you understood the docs to say?

I just checked and this is how the UsageFactor option reads...

UsageFactor Usage factor when running with this QOS (i.e. .5 would make it use only half the time as normal in accounting and 2 would make it use twice as much.) This only applies to the usage used in determining priority and doesn't modify the real usage in the database. 

In this case the usage used to determine priority is the same usage applied to limits as well.  If you have suggestions how to make that clearer please let me know your ideas :).

Are you still interested in at least sreport reporting the time with this usage factor applied?  Keep in mind you would easily be able to report more time than is possible on your cluster if the usage factor is above 1.
Comment 4 Michael Ver Haag 2016-02-11 07:01:24 MST
Below is what I am experiencing.
TLDR jobs submitted to a qos with usagefactor=0 are not running

[root@mgmt1 ~]# sacctmgr show associations -o format=account,grptresmin 
   Account   GrpTRESMins
---------- -------------
   verhaag        cpu=24
[root@mgmt1 ~]# sacctmgr show qos -o format=name,usagefactor
      Name UsageFactor
---------- -----------
    normal    1.000000
 scavenger    0.000000

[mverhaa1@jhu.edu@mgmt1 ~]$ cat sleepy.sh
#!/bin/bash

#SBATCH
#SBATCH --qos=scavenger
#SBATCH --time=00:01:00
#SBATCH --nodes=1
#SBATCH --partition=scavenger
#SBATCH --account=verhaag
#SBATCH --cpus-per-task=24
sleep 60
[mverhaa1@jhu.edu@mgmt1 ~]$  scontrol show cache |grep verhaag -A 13 |tail -n 26
    SharesRaw/Norm/Level/Factor=1/0.04/2/0.00
    UsageRaw/Norm/Efctv=0.00/0.00/0.00
    ParentAccount=marcc(5) Lft-Rgt=87-80 DefAssoc=No
    GrpJobs=N(0)
    GrpSubmitJobs=N(0) GrpWall=N(0.00)
    GrpTRES=cpu=N(0),mem=N(0),energy=N(0),node=N(0),gres/gpu=N(0)
    GrpTRESMins=cpu=24(0),mem=N(0),energy=N(0),node=N(0),gres/gpu=N(0)
    GrpTRESRunMins=cpu=N(0),mem=N(0),energy=N(0),node=N(0),gres/gpu=N(0)
    MaxJobs= MaxSubmitJobs= MaxWallPJ=
    MaxTRESPJ=cpu=3600
    MaxTRESPN=
    MaxTRESMinsPJ=
ClusterName=marcc Account=verhaag UserName=mverhaa1@jhu.edu(1058) Partition= ID=860
    SharesRaw/Norm/Level/Factor=1/0.04/1/0.00
    UsageRaw/Norm/Efctv=0.00/0.00/4294967294.00
    ParentAccount= Lft-Rgt=88-81 DefAssoc=Yes
    GrpJobs=N(0)
    GrpSubmitJobs=N(0) GrpWall=N(0.00)
    GrpTRES=cpu=N(0),mem=N(0),energy=N(0),node=N(0),gres/gpu=N(0)
    GrpTRESMins=cpu=N(0),mem=N(0),energy=N(0),node=N(0),gres/gpu=N(0)
    GrpTRESRunMins=cpu=N(0),mem=N(0),energy=N(0),node=N(0),gres/gpu=N(0)
    MaxJobs= MaxSubmitJobs= MaxWallPJ=
    MaxTRESPJ=cpu=3600
    MaxTRESPN=
    MaxTRESMinsPJ=
[mverhaa1@jhu.edu@mgmt1 ~]$ squeue |grep mverhaa1
[mverhaa1@jhu.edu@mgmt1 ~]$ sbatch sleepy.sh
Submitted batch job 3907033
[mverhaa1@jhu.edu@mgmt1 ~]$  scontrol show cache |grep verhaag -A 13 |tail -n 26
    SharesRaw/Norm/Level/Factor=1/0.04/2/0.00
    UsageRaw/Norm/Efctv=0.00/0.00/0.00
    ParentAccount=marcc(5) Lft-Rgt=87-80 DefAssoc=No
    GrpJobs=N(0)
    GrpSubmitJobs=N(0) GrpWall=N(0.00)
    GrpTRES=cpu=N(0),mem=N(0),energy=N(0),node=N(0),gres/gpu=N(0)
    GrpTRESMins=cpu=24(24),mem=N(132258),energy=N(0),node=N(1),gres/gpu=N(0)
    GrpTRESRunMins=cpu=N(0),mem=N(0),energy=N(0),node=N(0),gres/gpu=N(0)
    MaxJobs= MaxSubmitJobs= MaxWallPJ=
    MaxTRESPJ=cpu=3600
    MaxTRESPN=
    MaxTRESMinsPJ=
ClusterName=marcc Account=verhaag UserName=mverhaa1@jhu.edu(1058) Partition= ID=860
    SharesRaw/Norm/Level/Factor=1/0.04/1/0.00
    UsageRaw/Norm/Efctv=0.00/0.00/4294967294.00
    ParentAccount= Lft-Rgt=88-81 DefAssoc=Yes
    GrpJobs=N(0)
    GrpSubmitJobs=N(0) GrpWall=N(0.00)
    GrpTRES=cpu=N(0),mem=N(0),energy=N(0),node=N(0),gres/gpu=N(0)
    GrpTRESMins=cpu=N(24),mem=N(132258),energy=N(0),node=N(1),gres/gpu=N(0)
    GrpTRESRunMins=cpu=N(0),mem=N(0),energy=N(0),node=N(0),gres/gpu=N(0)
    MaxJobs= MaxSubmitJobs= MaxWallPJ=
    MaxTRESPJ=cpu=3600
    MaxTRESPN=
    MaxTRESMinsPJ=
[mverhaa1@jhu.edu@mgmt1 ~]$ sbatch sleepy.sh
Submitted batch job 3907034
[mverhaa1@jhu.edu@mgmt1 ~]$ squeue |grep mverhaa1
           3907034 scavenger sleepy.s mverhaa1 PD       0:00      1 (AssocGrpCPUMinutesLimit)
Comment 5 Danny Auble 2016-02-11 08:41:48 MST
Michael, I can confirm what you see.  I can also see in the code where it isn't happening.  Let me see what I can do for you on the primary concern.
Comment 6 Danny Auble 2016-02-11 09:12:35 MST
Michael, indeed this is working as documented/currently intended.  I do agree with you this is a good idea though.  I'll work on what needs to be done with this and get your a SOW to look at.  Most of it seems straight forward though.  This wouldn't be put into 15.08 either, but most likely in 16.05.
Comment 7 Kevin Hildebrand 2016-03-04 04:44:43 MST
I'd like to add that we are also affected by this problem- I'd originally submitted a patch for this in Bug 1048.
I've updated the patch (attached here) to work with 15.08.8 which should resolve this for us.

Kevin
Comment 8 Kevin Hildebrand 2016-03-04 04:45:48 MST
Created attachment 2821 [details]
Patch to allow job submission limits to reflect usage_factor
Comment 9 Danny Auble 2016-03-29 07:15:49 MDT
Thanks Kevin, I'll take a look at this later and see if we can get it in before 16.05.