Currently Slurm interprets a memory[+swap].failcnt > 0 as an Out of Memory condition. When this happens, the Kernel might be able to reclaim unused pages, and the application may or may not fail. This does not necessarily mean that the OOM-Killer actually killed any process. It would be nice though to provide a feedback back to user/admins when the oom-killer kills any process. There are mechanisms to register a notifier through the cgroup.event_control control file, so that the application can be notified through eventfd when OOM-Killer actually kills the process. If we manage to catch that event, we could then log that back to the slurmd.log, to the user stdout/stderr and potentially add a new Job/Step state OOM-Killed or similar, aside from the current OutOfMemory which just means that a spike of memory usage hit the limit. There are two more potential things that could improve the feedback. The first is to somehow capture the syslog messages logged by the Kernel when the oom-killer kills a process and report that info back. That was requested in bug 4520 for CRAY systems but could potentially be considered for vanilla Linux systems too. The other one thing that could improve feedback is to not only report that a job/step hit a memory[+swap] limit at some point, but also catch the information on memory.stats when this happens and report that back. I think that would let users better analyze the different factors that contributed to the memory footprint (page cache, rss, swap, etc.) under OOM conditions. Useful references: 1. https://www.kernel.org/doc/Documentation/cgroup-v1/memory.txt 2. https://github.com/BYUHPC/uft/blob/master/oom_notifierd/oom_notifierd.c 3. https://groups.google.com/forum/#!msg/slurm-users/f359KT-3lsU/yRm8vkxeBwAJ Related bugs: bug 3820, bug 4520.
Part of the enhancement has been solved here: https://github.com/SchedMD/slurm/commit/943c4a130f39dbb1fb Perhaps modify the API so that we get rid of the SIG_OOM and instead we add a new member(s) to reflect oom-kill event and/or memory hitting the limit, perhaps displaying the second as SystemComment.
Try to detect kernels with different oom counts available in the event file: https://patchwork.kernel.org/patch/9737381/ and use this instead of the manual eventfd() monitoring.
*** Ticket 6765 has been marked as a duplicate of this ticket. ***
