Bug 3094

Summary: Limit interactive jobs on partition
Product: Slurm Reporter: Daniel P Davis <daniel.p.davis>
Component: ConfigurationAssignee: Alejandro Sanchez <alex>
Status: RESOLVED INFOGIVEN QA Contact:
Severity: 4 - Minor Issue    
Priority: --- CC: alex, charles.wright
Version: 16.05.0   
Hardware: Linux   
OS: Linux   
Site: EM Alineos Sites: ---
Atos/Eviden Sites: --- Confidential Site: ---
Coreweave sites: --- Cray Sites: ---
DS9 clusters: --- HPCnow Sites: ---
HPE Sites: --- IBM Sites: ---
NOAA SIte: --- OCF Sites: ---
Recursion Pharma Sites: --- SFW Sites: ---
SNIC sites: --- Linux Distro: ---
Machine Name: CLE Version:
Version Fixed: 16.05.0 Target Release: ---
DevPrio: --- Emory-Cloud Sites: ---

Description Daniel P Davis 2016-09-16 07:52:18 MDT
We would like to limit interactive jobs to a particular partition.  IS that possible?
Comment 2 Alejandro Sanchez 2016-09-16 08:55:39 MDT
Hi Daniel, right now there's no partition/qos limit to accomplish this. We can mark it as a possible future enhancement if needed.
Comment 5 Alejandro Sanchez 2016-09-16 09:30:18 MDT
Daniel, note that the job_submit plugin has access to the job_desc->script field to discern interactive from non-interactive jobs. You could check that field to control the submissions to specific partitions as desired. Dunno exactly what you want to limit:

- "Number of interactive jobs submitted to specific partition, computing also past jobs"
- "Number of interactive jobs RUN/PEND to a specific partition at the same time"
- "Reject any submission to specific partition if job is interactive"
- ...?
Comment 7 Daniel P Davis 2016-09-16 10:14:14 MDT
We are trying to reject interactive job submission to one of our partitions.
Comment 8 Daniel P Davis 2016-09-16 10:52:02 MDT
Can you point me to documentation/examples on how to use the job_submit plugin?
Comment 9 Alejandro Sanchez 2016-09-16 11:03:35 MDT
Sure, this is the documentation:

http://slurm.schedmd.com/job_submit_plugins.html

You should configure the JobSubmitPlugins=lua in your slurm.conf. There are a few examples in the contribs/lua directory:

https://github.com/SchedMD/slurm/tree/master/contribs/lua

I think something in this direction is what you want:

--[[ Example Lua script ]]--

function slurm_job_submit( job_desc, part_list, submit_uid )
        local partition = "part1"
        if ((job_desc.script == nil or job_desc.script == '') and job_desc.partition == partition) then
                slurm.log_info("slurm_job_submit: interactive job submitted by user_id:%d to partition:%s rejected", job_desc.user_id, job_desc.partition)
                return slurm.FAILURE
        end
        return slurm.SUCCESS
end

function slurm_job_modify( job_desc, job_rec, part_list, modify_uid )
        return slurm.SUCCESS
end

slurm.log_info("initialized")
return slurm.SUCCESS
Comment 11 Daniel P Davis 2016-09-19 07:52:43 MDT
This is great info.  Only one thing is unclear for me. Is the job_submit plugin build as part of the default installation, or something I need to compile myself?
Comment 12 Alejandro Sanchez 2016-09-19 08:23:28 MDT
By default Slurm does not load any Job Submit Plugins. To accomplish this task, set JobSubmitPlugins=lua in slurm.conf and reconfigure it. Then Slurm will dynamically execute the lua script named "job_submit.lua" located in the default script directory (typically the subdirectory "etc" of the installation directory). Lua is an interpreted scripting language and needs no compilation after source edition.
Comment 13 Daniel P Davis 2016-09-20 06:36:49 MDT
Thanks.
Comment 14 Charles Wright 2017-03-21 08:57:00 MDT
FYI,
I just tried the script that comes with 16.05.10 and got this
slurmctld[4133]: error: lua: /etc/slurm/job_submit.lua: /etc/slurm/job_submit.lua:5: ')' expected near '!'

It appears that != in lua is "~=" http://lua-users.org/wiki/LuaTypesTutorial

diff job_submit.lua job_submit.lua-new 
5c5
<   if ((job_desc.script == nil or job_desc.script == '') and job_desc.partition != partition) then
---
>   if ((job_desc.script == nil or job_desc.script == '') and job_desc.partition ~= partition) then