Hi Harry,
If you are running a cluster with more than one broker node, and the brokers’ logging levels are set to DEBUG, you should see something like this when one of the broker nodes leaves the cluster:
2022-12-27 20:29:18,150 INFO - Cluster nodes found by discovery: [HceI5|16] (1) [HceI5].
2022-12-27 20:29:18,870 DEBUG - Previous running nodes [BBkvg, HceI5]
2022-12-27 20:29:19,252 INFO - Cluster size = 1, members : [HceI5].
2022-12-27 20:29:19,258 DEBUG - Removed state of node BBkvg.
2022-12-27 20:29:19,259 DEBUG - Node left cluster: 'BBkvg'
In my example, I have a simple cluster containing 2 broker nodes. One node with the name “BBkvg” was shut down; the remaining node “HceI5”'s log showed the above output when it realized “BBkvg” had left the cluster.
Later when a new node “hm276” joins the cluster, “HceI5”'s log shows output like this:
2022-12-27 20:30:49,943 INFO - Cluster nodes found by discovery: [HceI5|17] (2) [HceI5, hm276].
2022-12-27 20:30:49,951 DEBUG - New node discovered: 'hm276'
2022-12-27 20:30:49,951 DEBUG - State of node hm276 changed to UNKNOWN.
2022-12-27 20:30:49,952 DEBUG - Current cluster node states: {HceI5=RUNNING, hm276=UNKNOWN}
2022-12-27 20:30:51,979 DEBUG - State of node hm276 changed to NOT_JOINED was UNKNOWN.
2022-12-27 20:30:51,979 DEBUG - Current cluster node states: {HceI5=RUNNING, hm276=NOT_JOINED}
2022-12-27 20:30:51,986 DEBUG - Sending NOT_JOINED state notification for hm276 to all nodes.
2022-12-27 20:30:52,308 DEBUG - State of node hm276 changed to JOINING was NOT_JOINED.
2022-12-27 20:30:52,308 DEBUG - Current cluster node states: {HceI5=RUNNING, hm276=JOINING}
2022-12-27 20:30:52,837 DEBUG - Sending JOINING state notification for hm276 to all nodes.
2022-12-27 20:30:53,555 DEBUG - Replication of data to nodes [hm276] finished.
2022-12-27 20:30:54,959 DEBUG - State of node hm276 changed to RUNNING was JOINING.
2022-12-27 20:30:54,960 DEBUG - Current cluster node states: {HceI5=RUNNING, hm276=RUNNING}
2022-12-27 20:30:54,961 INFO - Cluster size = 2, members : [HceI5, hm276].
2022-12-27 20:30:54,971 DEBUG - Sending RUNNING state notification for hm276 to all nodes.
If the logging level is INFO, you would see only the lines in the output above that show “INFO” after the date/timestamps (which would still give you an idea that nodes had left and joined the cluster.
I encourage you to try this with your test cluster to see the results. You can kill or stop one of the broker nodes in the cluster and observe similar results in the logs of one of the other broker nodes that is still running.
If you are running only one broker, you will not be able to replicate the above test since there are no other nodes for it to interact with via the “heartbeat” or “tcp health check” features.
I hope this helps!
Thanks,
Seth - HiveMQ Support