There are some great tools for assigning discounts or penalties to jobs before they are allocated resources:

* QOS.UsageFactor
* Partition.TRESBillingWeights
* etc.

But what if I want to change the "cost" of a job *after* the fact? I might want to avoid penalizing users who ran jobs which failed due to hardware reasons. Or I might want to charge extra for jobs which require node reboots to cleanup afterwards. Either way, I want to be able to adjust how the job affects their current fairshare priority for queued jobs.

Are there any existing solutions for this?

The only solutions I've found so far are:

1. 'sacctmgr modify ... set RawUsage=0' - obviously this is too big of a hammer. I only want to edit a single job, and I might want to *increase* the usage for the job - not decrease it.

2. For clusters using "banking" (limits on GrpTRESMins and PriorityDecayHalfLife=0), you can essentially accomplish this by editing the limit after the fact (increasing the limit for a refund, decreasing it for a penalty). But we don't use that accounting strategy at our site.

I think something like this would mostly get me what I want:

> $SLURM_TOOL modify job=1234 set EffectiveWallTime=+1:00:00
> $SLURM_TOOL modify job=1234 set EffectiveWallTime=0
I like this option because the syntax is pretty easy to interpret. I think it would be complex to implement in the backend because you would need to interpolate back from NOW to JOB_END and estimate how much that job is contributing to the current RawUsage (as adjusted by the PriorityHalfLifeDecay). And you might need to keep track of the last time there was a RawUsage=0 reset to avoid editing the usage if it has already been removed. Seems tricky, but not impossible.


Another, simpler-to-implement option would be something like this:

> sacctmgr modify association where user=lyeager account=admin set RawUsage=+100
That's still a little tricky to implement because you'd need to figure out how to adjust 'usage_raw', AND 'used_jobs', AND 'usage.usage_tres_raw', etc. internally. But most of the burden would be on me, the admin, to figure out what the RawUsage actually means. Though, since I'd usually be using this in the job epilog, I wouldn't typically need to worry much about the half-life decay.


I'm hoping I've missed something and that there's already some way for me to edit historical jobs? Please let me know if so.