The slurm power saving feature has been turned on, so that idle nodes are powered off when not in use. It has been noticed that new jobs are scheduled to nodes that have recently been powered on, rather than nodes that should be prioritised due to the node "Weight". It looks like there are already bugs files in the slurm bugtracker (9548 & 9734) , currently marked as "under consideration".
Before the customer started using the powersave feature, all other things being equal, jobs would be scheduled to nodes[001-051] in preference to gnodes[001-016], as the Weight was lower. After turning on the powersave feature, the weight appears to be ignored. Everything else works as expected (eg specific feature or resource requests).
I'm going to mark this as a duplicate: (In reply to Broderick Gardner from bug#9734 comment #1) > Yes, as it is currently designed, nodes in a powered-down state are in a > lower tier altogether, making weights useless for cloud. I'm looking into > what the scope for an enhancement would be in this regard. Please take a look at bug#9734 for continuing updates. *** This ticket has been marked as a duplicate of ticket 9734 ***