Hi,

I think appending 'CR_LLN' to SelectTypeParameters or setting 'LLN' to a specific partition will give you what you want. In short, this will schedule resources to jobs on the least loaded nodes but increase fragmentation. Please, let me know if this fits your node minimization goal.

(In reply to Alejandro Sanchez from comment #3)
> I think appending 'CR_LLN' to SelectTypeParameters or setting 'LLN' to a
> specific partition will give you what you want. In short, this will schedule
> resources to jobs on the least loaded nodes but increase fragmentation.
> Please, let me know if this fits your node minimization goal.

We're trying to achieve the opposite of the CR_LLN, which would apparently increase fragmentation.  The CR_LLN is defined in https://slurm.schedmd.com/slurm.conf.html and it doesn't appear to pack jobs compactly onto as few nodes as possible, but in stead spread jobs onto more nodes than the minimum necessary number.

I think what we're looking for is similar to the Moab/Maui scheduler's 
  NODEALLOCATIONPOLICY MINRESOURCE
defined as: "This algorithm priorities nodes according to the configured resources on each node. Those nodes with the fewest configured resources which still meet the job's resource constraints are selected." (see http://docs.adaptivecomputing.com/maui/5.2nodeallocation.php#MINRESOURCE)

This sounds to me like a common goal suitable for most HPC sites.

Let's see if I understand what you want properly. You have this partition with 179 24-core nodes. Let's imagine that 2 of these nodes are half-allocated, so that 1 node has 12 CPUs allocated and another node has 12 CPUs allocated too, and the rest of the nodes are idle. Now you submit a new job to this partition with -n24, what you want is that these 24 tasks fill up these first two nodes so that the overall node utilization in the cluster is minimized, right? Not that the number of nodes using for the job is minimized (that would be LLN). Am I correct with your goal now?

(In reply to Alejandro Sanchez from comment #5)
> Let's see if I understand what you want properly. You have this partition
> with 179 24-core nodes. Let's imagine that 2 of these nodes are
> half-allocated, so that 1 node has 12 CPUs allocated and another node has 12
> CPUs allocated too, and the rest of the nodes are idle. Now you submit a new
> job to this partition with -n24, what you want is that these 24 tasks fill
> up these first two nodes so that the overall node utilization in the cluster
> is minimized, right? Not that the number of nodes using for the job is
> minimized (that would be LLN). Am I correct with your goal now?

I don't think so.  What we want is for jobs to be scheduled to the minimum number of nodes possible, so that we avoid fragmentation of the cluster.  So a new job with -n24 should be scheduled to exactly 1 node, not 2 or 3 (assuming there exists at least 1 node with all 24 cores free).  This is not what we have experienced in the two examples given above, where jobs got scheduled to multiple nodes unnecessarily.

(In reply to Alejandro Sanchez from comment #5)
> Let's see if I understand what you want properly. You have this partition
> with 179 24-core nodes. Let's imagine that 2 of these nodes are
> half-allocated, so that 1 node has 12 CPUs allocated and another node has 12
> CPUs allocated too, and the rest of the nodes are idle. Now you submit a new
> job to this partition with -n24, what you want is that these 24 tasks fill
> up these first two nodes so that the overall node utilization in the cluster
> is minimized, right? Not that the number of nodes using for the job is
> minimized (that would be LLN). Am I correct with your goal now?

Also, LLN seems to cause fragmentation as described in the slurm.conf page:

CR_LLN
    Schedule resources to jobs on the least loaded nodes (based upon the number of idle CPUs). This is generally only recommended for an environment with serial jobs as idle resources will tend to be highly fragmented, resulting in parallel jobs being distributed across many nodes.

Ok, I'm gonna ask you some info to better analyze what's going on, for example with this submission:

"sbatch -n2 -J DCDFTv2b -p xeon24 -t 720 script.sh 
  Gets scheduled on 2 24-core nodes in stead of 1 node."

$ scontrol setdebugflags +steps
$ sinfo -p xeon24 -N -O "nodelist,statecompact,cpusstate,memory,allocmem,weight"
$ squeue -p xeon24
$ sbatch -n2 -J DCDFTv2b -p xeon24 -t 720 script.sh
$ sinfo -p xeon24 -N -O "nodelist,statecompact,cpusstate,memory,allocmem,weight"
$ squeue -p xeon24
$ scontrol show job <jobid>
$ scontrol setdebugflags -steps

And also the script.sh and the slurmctld.log part related to the job submission Step debug flag messages?

I want to see the initial nodes state right before the job runs, and what's the state once it is allocated. If you think the node state is gonna change between the submission and the job start, let me know. The thing is to analyze how the job is allocated resources and see if this is expected behavior with your config.

Independently of that, the "right" request for jobs that want a specific node count is to ask for that node count, otherwise the scheduler may spread them out.

(In reply to Alejandro Sanchez from comment #8)
> Ok, I'm gonna ask you some info to better analyze what's going on, for
> example with this submission:
> 
> "sbatch -n2 -J DCDFTv2b -p xeon24 -t 720 script.sh 
>   Gets scheduled on 2 24-core nodes in stead of 1 node."
> 
> $ scontrol setdebugflags +steps
> $ sinfo -p xeon24 -N -O
> "nodelist,statecompact,cpusstate,memory,allocmem,weight"
> $ squeue -p xeon24
> $ sbatch -n2 -J DCDFTv2b -p xeon24 -t 720 script.sh
> $ sinfo -p xeon24 -N -O
> "nodelist,statecompact,cpusstate,memory,allocmem,weight"
> $ squeue -p xeon24
> $ scontrol show job <jobid>
> $ scontrol setdebugflags -steps
> 
> And also the script.sh and the slurmctld.log part related to the job
> submission Step debug flag messages?
> 
> I want to see the initial nodes state right before the job runs, and what's
> the state once it is allocated. If you think the node state is gonna change
> between the submission and the job start, let me know. The thing is to
> analyze how the job is allocated resources and see if this is expected
> behavior with your config.
> 
> Independently of that, the "right" request for jobs that want a specific
> node count is to ask for that node count, otherwise the scheduler may spread
> them out.

We're not currently seeing the job fragmentation issue on our cluster.  It may perhaps be due to a changing load pattern of the queue.  I suggest that we close this case for now.  If the problem recurs, I'll follow your procedure for obtaining debugging data.

Ole,

The logic that is picking the resources for the job is in the _eval_nodes() function in src/plugins/select/cons_res/job_test.c.

After the available nodes and cores are identified, the _eval_nodes() function is not preferring idle nodes, but sequentially going through the lowest weight nodes and accumulating cores. Here are some possible solutions:

1. The node count can be specified at job submit time or using a job_submit plugin. This requires more work on the part of the user and could prevent the job from running if idle nodes are not available (i.e. depending upon the node count, idle nodes could be required rather than preferred).

2. The job can be submitted with the --exclusive option to get dedicated nodes. This is probably simpler than specifying a node count, but also could prevent the job from running if idle nodes are not available.

3. Add a new job option that does the opposite of "--spread-job". If set, that would use the same logic as CR_LLN, but only for jobs that specifically request that option. If we were to add that option, it could be set using a job_submit plugin for jobs requesting larger task counts an the user would not need to change behavior.

If we add logic for option 3, that would probably only go into Slurm version 17.11 (a patch might be provided for version 17.02) and it would probably not work with the topology/tree plugin.

Please, let me know if either options 1 or 2 suffices for you. Otherwise if we go for option 3 I'll go ahead and reclassify this as a sev-5.

Hi Alejandro,

This is a situation we're currently struggling with at Swinburne too, we would like nodes for jobs picked by least available resource that still fits the jobs requirements.  This way single core jobs will fill up spare cores on otherwise fragmented nodes (hopefully) leaving entire nodes free for larger parallel jobs.

In the interim we are looking to corral jobs that use single nodes to a partition that only has access to a subset of nodes and require jobs that use more than 1 node of 32 cores to request a multiple of the permitted 32 cores per node for non-GPU jobs (all done via the submit filter).

I'm not sure what the best way to pursue that with you folks though, whether you wish to reopen this ticket to keep the context or for me to create a new one?

All the best,
Chris

I'd create a new one.