Yes it has to be the firewall, but I’ve gone through the configuration and the pure IP tables of it 20 times and I just can’t pinpoint what could cause the problem.
I looked at the ICMP_IN option and this is set, ICMP_OUT as well:
# Allow incoming PING. Disabling PING will likely break external uptime
# monitoring
ICMP_IN = "1"
# Set the per IP address incoming ICMP packet rate for PING requests. This
# ratelimits PING requests which if exceeded results in silently rejected
# packets. Disable or increase this value if you are seeing PING drops that you
# do not want
#
# To disable rate limiting set to "0", otherwise set according to the iptables
# documentation for the limit module. For example, "1/s" will limit to one
# packet per second
ICMP_IN_RATE = "1/s"
Could this rate have anything to do with it?
We first used the latest version: hivemq-ce-2024.1 via docker. When we noticed the problems we downgraded to the version that was (succesfully) running on the old server: hivemq-ce-2020.2, which is what we’ve got running now. This version is running directly on Linux now.
We purposely added the whole IP range to csf.allow:
172.17.0.0/16 # MQTT clients
We also added the MQTT port to incoming and outgoing TCP and UDP ports (1883):
# Allow incoming TCP ports
TCP_IN = "35000:35999,20,21,22,25,53,853,80,110,143,443,465,587,993,995,2222,1883,1884"
# Allow outgoing TCP ports
TCP_OUT = "143,20,21,22,25,53,853,80,110,113,443,587,993,995,2222,1883,1884,3050"
# Allow incoming UDP ports
UDP_IN = "20,21,53,853,80,443,1883,1884"
# Allow outgoing UDP ports
# To allow outgoing traceroute add 33434:33523 to this list
UDP_OUT = "11335,443,20,21,53,853,113,123,1883,1884"
Connection tracking is turned off.
It checks processes but only warns and doesn’t kill them.
You can see the LFD logs from around that time here:
labs.netvibes.nl/2024/lfd-hivemq.log
But it doesn’t seem to show anything except some SSH blocks and sees some potentially suspisious processes. But as I said before it doesn’t kill any of them.