Would you please attach the slurmctld.log and slurmd.log from one of the nodes? Also, let us know if you see errors like the following:

> RPC:REQUEST_PING : Protocol authentication error

Created attachment 24857 [details]
Slurmctld log

Please also note that slurmd's on an older version will run into this issue. Mixing new slurmctld with older slurmd's will cause the nodes to have Protocol authentication error. The new slurmd should work just fine. Please let us know if the new slurmd's have communication issues.

Created attachment 24858 [details]
slurmd log from node running old version 21.08.6

Created attachment 24859 [details]
slurmd from node running new version 21.08.8

We are not seeing RPC:REQUEST_PING messages, but we are seeing "Protocol authentication error" messages like this:

[2022-05-05T13:19:28.606] error: slurm_receive_msgs: [[c004]:6818] failed: Protocol authentication error

I thought that updates within a major release were supposed to be protocol compatible such that 21.08.6 and 21.08.8 would work fine together such that we could do a rolling update. Is that never the case or does that break with this particular update?

Mike

We have found the issue, and it relates to mixed fixed+unfixed 21.08. Currently, this is broken. We do apologize. I will have Chad follow up with you shortly (in the next hour or two) with a patch and / or  link to 21.08.8-2.

(In reply to Jason Booth from comment #7)
> We have found the issue, and it relates to mixed fixed+unfixed 21.08.
> Currently, this is broken. We do apologize. I will have Chad follow up with
> you shortly (in the next hour or two) with a patch and / or  link to
> 21.08.8-2.
Please see the related note just posted to slurm-announce and slurm-users. You can download the fixed release from https://www.schedmd.com/downloads.php.

I wanted to test the patch so I downloaded 21.08.8-2 and compiled the RPMs and got them installed on all the nodes that I had already updated with 21.08.8-1 and I am still seeing the same problem. 

Is this due to jobs running that were started under 21.08.8-1 or is there maybe still a problem with the patch?

(In reply to Michael Robbert from comment #9)
> Is this due to jobs running that were started under 21.08.8-1 or is there
> maybe still a problem with the patch?

We are actively working to determine the answer to that question.

Is it possible to set debugflags=network on the controller and node052. A restart of the slurmd will be required. `scontrol setdebugflags +network` can activate the logging. Have it log until the 'RPC:REQUEST_PING : Protocol authentication error' errors are logged.

Please also provide the logs from slurmd on node501, node505, node602 from at least May.

I ran `scontrol setdebugflags +network` on the controller. I also tried running on a compute node. I also tried adding the line "DebugFlags=Network" to slurm.conf on a few compute nodes and then restarted the slurmd there. None of that changed the messages that I'm seeing in the logs. 
Further we do not have nodes with the names "node501, node505, node602". The nodes in the partition that start with node* are 001 through 140.

Should I proceed to update the rest of the nodes in the cluster to see if that clears it up?

Mike

Please attach the logs of the nodes with network flag active. It is not ideal to leave it active on the controller for long periods.

Here are the logs from node052 running 21.08.6 with Debugflags=Network active. You can see the restart where I activated it and the restart where I deactivated it. During this time the controller showed several nodes that went offline and online again, but this node logged nothing.

[2022-05-06T01:40:09.033] [11591547.extern] done with job
[2022-05-06T08:42:48.988] Slurmd shutdown completing
[2022-05-06T08:42:49.119] slurmd version 21.08.6 started
[2022-05-06T08:42:49.120] slurmd started on Fri, 06 May 2022 08:42:49 -0600
[2022-05-06T08:42:49.752] CPUs=16 Boards=1 Sockets=2 Cores=8 Threads=1 Memory=64400 TmpDisk=32200 Uptime=3492078 CPUSpecList=(null) FeaturesAvail=(null) FeaturesActive=(null)
[2022-05-06T08:49:25.697] Slurmd shutdown completing
[2022-05-06T08:49:25.811] slurmd version 21.08.6 started
[2022-05-06T08:49:25.812] slurmd started on Fri, 06 May 2022 08:49:25 -0600
[2022-05-06T08:49:26.444] CPUs=16 Boards=1 Sockets=2 Cores=8 Threads=1 Memory=64400 TmpDisk=32200 Uptime=3492474 CPUSpecList=(null) FeaturesAvail=(null) FeaturesActive=(null)

When you asked for node052 I wasn't sure if maybe you could have meant c052 so I did the same thing there and had the same results. That node is running the latest code 21.08.8-2

[2022-05-06T07:42:35.775] [11600775.0] task/cgroup: _memcg_initialize: job: alloc=187740MB mem.limit=187740MB memsw.limit=unlimited
[2022-05-06T07:42:35.775] [11600775.0] task/cgroup: _memcg_initialize: step: alloc=187740MB mem.limit=187740MB memsw.limit=unlimited
[2022-05-06T08:44:45.768] Slurmd shutdown completing
[2022-05-06T08:44:45.877] slurmd version 21.08.8-2 started
[2022-05-06T08:44:45.877] slurmd started on Fri, 06 May 2022 08:44:45 -0600
[2022-05-06T08:44:47.153] CPUs=36 Boards=1 Sockets=2 Cores=18 Threads=1 Memory=385414 TmpDisk=9990 Uptime=3492448 CPUSpecList=(null) FeaturesAvail=(null) FeaturesActive=(null)
[2022-05-06T08:48:54.091] Slurmd shutdown completing
[2022-05-06T08:48:54.209] slurmd version 21.08.8-2 started
[2022-05-06T08:48:54.210] slurmd started on Fri, 06 May 2022 08:48:54 -0600
[2022-05-06T08:48:55.486] CPUs=36 Boards=1 Sockets=2 Cores=18 Threads=1 Memory=385414 TmpDisk=9990 Uptime=3492696 CPUSpecList=(null) FeaturesAvail=(null) FeaturesActive=(null)

It does appear that neither of this nodes is showing up in the list of nodes that is flapping up and down. I'll now try the same thing on one of the nodes that is flapping.

Created attachment 24883 [details]
slurmd with errors Debugflags=network active

I am not seeing any logging from node125 which is flapping, even with Debugflags=network active. I am attaching the slurmd.log from a node that is throwing errors, but isn't flapping. I activated and then deactivated the network debug flag this morning for several minutes, but I didn't see much difference in the output. That output is in the attachment.

I feel like the debug flag for network isn't doing what what it should. 

Mike

Please also provide output of:
> scontrol show nodes

(In reply to Michael Robbert from comment #15)
> I feel like the debug flag for network isn't doing what what it should. 

Do you have the logs from slurmctld at this time too with network flag active?

Note that this timestamp is just zero:
> ENCODED: Wed Dec 31 17:00:00 1969

Could be that munge is very unhappy and refusing to sign/verify anything on this node. Can you please also restart munged on the nodes with this error?

Created attachment 24884 [details]
Output of "scontrol show nodes"

Created attachment 24885 [details]
slurmctld logs from today. Debugflags=network enabled at 08:40:56

(In reply to Michael Robbert from comment #19)
> Created attachment 24885 [details]
> slurmctld logs from today. Debugflags=network enabled at 08:40:56

Looks like the debugflag was not enabled. There should be a ton of NET: logs but I don't see even a single one.

Please also set log level to at least on the controller:
> scontrol setdebug debug

Please also note that a work-around for the issue is to set:
> SlurmdTimeout=0

Note that a restart of all daemons is required for it to be effective.

Setting the log level to debug has enabled NET: messages. On the compute nodes setting slurmddebug to debug has gotten the messages to show up there as well. 

Now that we have that debugging working is there a particular node that you'd like to see network deubugging from? node052 is what you asked for last night.

I only found the munge errors on a small handful of nodes. Restarting munge on those nodes didn't help. I double checked the md5sum of our munge.key and the verified that the time on those nodes is in sync with the rest of the cluster.

(In reply to Michael Robbert from comment #24)
> Setting the log level to debug has enabled NET: messages. On the compute
> nodes setting slurmddebug to debug has gotten the messages to show up there
> as well. 
Great

> Now that we have that debugging working is there a particular node that
> you'd like to see network deubugging from? node052 is what you asked for
> last night.
I only need it from one of the flapping nodes.

> I only found the munge errors on a small handful of nodes. Restarting munge
> on those nodes didn't help. I double checked the md5sum of our munge.key and
> the verified that the time on those nodes is in sync with the rest of the
> cluster.
How many threads is munged running with on those nodes?

It doesn't look like we are specifying a number of threads for munged so I am assuming it is the default. Is there a way to query to know for sure?

I did find a total of 5 nodes that were logging munge errors. All of those nodes were still running slurmd 21.08.6 so I tried updating them to 21.08.8-2 and the munge errors stopped after that update.

It looks like the node availability stopped flapping too.

I can't think of anything special about these 5 nodes. There were 3 from the compute partition and 2 from the aun partition so it was 2 different node types. Other nodes that didn't get the update aren't having the same problem. 

Any thoughts? 
Should I go ahead and continue with updating the rest of the nodes?

In case it helps, these were the only nodes with the Munge errors:
c[001,022,061],node[037,109]

(In reply to Michael Robbert from comment #26)
> It doesn't look like we are specifying a number of threads for munged so I
> am assuming it is the default. Is there a way to query to know for sure?

Then I suggest adding `--num-threads=10` as an arg.

> I did find a total of 5 nodes that were logging munge errors. All of those
> nodes were still running slurmd 21.08.6 so I tried updating them to
> 21.08.8-2 and the munge errors stopped after that update.
> 
> It looks like the node availability stopped flapping too.
Good.
 
> I can't think of anything special about these 5 nodes. There were 3 from the
> compute partition and 2 from the aun partition so it was 2 different node
> types. Other nodes that didn't get the update aren't having the same
> problem. 

The bug in 21.08.8-1 is a cross-compatibility issue so it could just have been these nodes had a particularly long enough running job to cross all the versions. Since the bug is fixed, having all nodes uniform should be sufficient to be rid of the node flapping. If not, please provide more logs.

> Should I go ahead and continue with updating the rest of the nodes?

Yes, we have not seen this issue happen on clusters that have all daemons updated and restarted.

(In reply to Michael Robbert from comment #26)
> Should I go ahead and continue with updating the rest of the nodes?

Any updates? The problem should have been resolved by updating all nodes.

Sorry, I thought I had updated the ticket. When we updated all the nodes the problem went away. We are all good now!

Great. Closing out ticket.