systemd is going to start restricting the number of processes/threads for services: https://www.suse.com/releasenotes/x86_64/SUSE-SLES/12-SP2/#fate-320358 It looks like setting TasksMax in our services file will suffice, although it is currently unclear to me what happens if you have this set in a service file with an older systemd running - if it's silently ignored then this should be safe to add throughout, otherwise we're heading down a path of needing to identify specific systemd versions or capabilities.
Created attachment 4337 [details] Proposal of adding TasksMax to systemd service files Hello, We had problems with this TasksMax not being set in SUSE 12 SP2 so we set it to "TasksMax=infinity" and it solved the problem. systemd version in SUSE is systemd-228-117.12.x86_64 I tested it in a CentOS 7 with an older systemd: systemd-219-30.el7_3.6.x86_64 And it is complaining: /var/log/messages:Apr 11 18:09:54 moll-archive24 systemd: [/usr/lib/systemd/system/slurm.service:12] Unknown lvalue 'TasksMax' in section 'Service' The TasksMax value were introduced in systemd in version 226. All versions below 225 will complain despite the service starts correctly. Moreover this value is used when kernel is 4.3 or higher. - The feature is announced here: https://github.com/systemd/systemd/blob/master/NEWS#L1688 - Pull request introduced here: https://github.com/systemd/systemd/pull/1239 - Default values set here: https://github.com/systemd/systemd/pull/1886 My first suggestion is to add this TasksMax to the service files as newer versions of systemd will start being deployed. My second suggestion is not to touch this parameter and keep defaults, but comment it in the administrator documentation or maybe in the tunning guide. Best regards, Barcelona Supercomputing Center Felip M
Thanks for the background Felip, that helps a lot. It does look like we'd need a way to detect the systemd version at install time, and conditionally enable it on newer platforms.
Created attachment 4372 [details] Patch for conditionally modify systemd unit files Hi Alex, I am sending a patch that creates a macro in auxdir/x_ac_systemd.m4 in order to check wether the installed systemd corresponds to a 226 version or greater. If so, then TasksMax parameter is included in slurmd and slurmctld unit files. You must run autogen.sh to recreate Makefile.in's and configure. Hope it helps. Best regards, Felip M
Created attachment 4373 [details] Patch for conditionally modify systemd unit files Corrected patch, systemd version to check must be greater or equal to 227, not 226.
Thanks Felip. We're gonna take a look at this and will come back to you.
A modified version of this has been checked into master commits b19fbf3ba579 2ef36b04ea47 dc54d5462b9d ran autogen.sh 6a9f84978b82d Please reopen if you feel/see anything out of place.
Sorry those hashes are df2306ff5f 02aa5a1a20 30982991b8 868aefdc97
*** Ticket 4005 has been marked as a duplicate of this ticket. ***
