HiveMQ

Cluster Support in Community Edition (CE)

Hi There,

I want to know whether the Clustering is supported in Community Edition?

As per this link: Discover the 3 different editions of HiveMQ it is mentioned that Clustering is not supported in Community Edition.

I have downloaded the Community Edition from this URL: https://github.com/hivemq/hivemq-community-edition/releases/download/2021.2/hivemq-ce-2021.2.zip

Thanks & Regards,
-Venkat

Hi @vgkkonda,

the mentioned page tells the truth, there is no clustering for CE.

Greetings,
Michael from the HiveMQ team

Hi Michael / HiveMQ Team,

Thanks for the response.

Does it mean that CE will have single point of failure? There is no distribution or scalable feature available in CE?

Thanks & Regards,
-Venkat

Hi @vgkkonda,

Does it mean that CE will have single point of failure?

Correct

There is no distribution or scalable feature available in CE?

Correct (though you can scale vertically but I guess this wasn’t the question)

I’m not sure what your use case is but HiveMQ cloud has a freemium version (with clustering) that you could use.

Greetings,
Michael from the HiveMQ team

Hi Michael,

My question was related to avoid single point of failure and also towards horizontal scaling rather vertical scaling.

I want to bring to your notice below:

I did the following to verify the Cluster feature of HiveMQ CE:

  1. Created two AWS RHEL 8 EC2 Instances, created Network Load Balancer in AWS (reference:
    hivemq /blog/running-hivemq-cluster-aws-auto-discovery/ ) in my personal AWS Account.

  2. Installed Java 11 and HiveMQ in both the EC2 Instances, configured config.xml in both using the hivemq /docs/hivemq/4.7/user-guide/cluster.html

  3. Then started HiveMQ in both EC2 Instances.

  4. Then from my Laptop “Laptop1”, installed MQTT CLI, I connected as a client for publishing to a topic called “test” using AWS Load Balancer DNS Name.

  5. Then from my another laptop “Laptop2”, installed MQTT CLI, I connected as a client for subscribing to a topic “test” to the Load Balancer DNS Name.

  6. Sent one message or published one message from Laptop1, the message was received at subscribing client.

7. Then I stopped HiveMQ in one of the EC2 instances, then both the clients got disconnected.

  1. Again, I connected the Publishing Client from Laptop1 to the Load Balancer DNS Name and sent one message. Note at this time only one EC2 instance had Hive MQ running and I was connected to Load Balancer DNS Name. And Laptop2 was disconnected.

  2. Now again, I connected the Subscribing Client from Laptop2 to the Load Balancer DNS Name and I was able to receive the message due to the effect of Persistent Session. Note at this time only one EC2 instance had Hive MQ running and I was connected to Load Balancer DNS Name.

  3. Commands for Subscribing Client from Laptop2:
    mqtt-cli.exe sub -i subclient1 -h LoadBalancerHiveMQ-xxxxxx.elb.ap-south-1.amazonaws.com -p 1883 --no-cleanStart --debug --verbose -k 3600 -se 3600 -t test -q 2 -J

  4. Commands for Publishing Client from Laptop1:
    mqtt-cli.exe shell
    con -i pubclient1 --no-cleanStart --debug --verbose -k 3600 -se 3600 -h LoadBalancerHiveMQ-xxxxxx.elb.ap-south-1.amazonaws.com -p 1883
    pub -t test -q 2 -m ‘test1’

  5. MQTT CLI which is installed in both the Laptops reference URL are:
    github: hivemq/mqtt-cli/releases/download/v4.7.1/mqtt-cli-4.7.1-win.zip

  6. Both laptops were running Windows OS and installed Java 11.

Thanks & Regards,
-Venkat

Hi @vgkkonda,

HiveMQ CE does not support clustering.
You need to use either professional or enterprise edition to have the advantage of clustering.
As suggested in the very post you quoted.

Best,
Florian

Hi @vgkkonda,

That’s why I put it into parenthesis, as your statement was correct.

I again tell you HiveMQ CE has no clustering feature, so something in your test lead to the behaviour described in step 9.

Can you post the content of the “hivemq/log/event.log” of both instances so we can figure out how the clients connected and how that may explain the seen behaviour at step 9.

Just to be sure you only used the mentioned blog post to set up the load balancer? Aka starting HiveMQ prints “Starting HiveMQ Community Edition Server” as first log (in the hivemq/log/hivemq.log) when you start it up.

Greetings,
Michael from the HiveMQ team

Hi @michael_w

Below are the logs from hivemq.log file from one of the AWS EC2 Instance:

2021-09-28 18:13:11,417 INFO - Starting HiveMQ Community Edition Server
2021-09-28 18:13:11,431 INFO - HiveMQ version: 2021.2
2021-09-28 18:13:11,431 INFO - HiveMQ home directory: /home/ec2-user/hivemq-ce-2021.2
2021-09-28 18:13:11,508 INFO - Log Configuration was overridden by /home/ec2-user/hivemq-ce-2021.2/conf/logback.xml
2021-09-28 18:13:11,915 INFO - This HiveMQ ID is fSEBe
2021-09-28 18:13:15,882 INFO - Starting HiveMQ extension system.

Below are the logs from migration.log from one of the AWS EC2 Instance:

2021-09-28 18:13:11,949 INFO - Checking for migrations (HiveMQ version 2021.2).
2021-09-28 18:13:11,953 INFO - Read metadata file: MetaInformation{hivemqVersion=‘2021.2’, clientSessionPersistenceVersion=‘040000’, queuedMessagesPersistenceVersion=‘040500’, subscriptionPersistenceVersion=‘040000’, retainedMessagesPersistenceVersion=‘040500_R’, publishPayloadPersistenceVersion=‘040500_R’, retainedMessagesPersistenceType=‘FILE_NATIVE’, publishPayloadPersistenceType=‘FILE_NATIVE’}
2021-09-28 18:13:12,010 INFO - Nothing to migrate found.
2021-09-28 18:13:12,010 INFO - Checking for value migrations (HiveMQ version 2021.2).
2021-09-28 18:13:12,011 INFO - Read metadata file: MetaInformation{hivemqVersion=‘2021.2’, clientSessionPersistenceVersion=‘040000’, queuedMessagesPersistenceVersion=‘040500’, subscriptionPersistenceVersion=‘040000’, retainedMessagesPersistenceVersion=‘040500_R’, publishPayloadPersistenceVersion=‘040500_R’, retainedMessagesPersistenceType=‘FILE_NATIVE’, publishPayloadPersistenceType=‘FILE_NATIVE’}
2021-09-28 18:13:12,011 INFO - Nothing to migrate found.
2021-09-28 18:13:14,626 INFO - Read metadata file: MetaInformation{hivemqVersion=‘2021.2’, clientSessionPersistenceVersion=‘040000’, queuedMessagesPersistenceVersion=‘040500’, subscriptionPersistenceVersion=‘040000’, retainedMessagesPersistenceVersion=‘040500_R’, publishPayloadPersistenceVersion=‘040500_R’, retainedMessagesPersistenceType=‘FILE_NATIVE’, publishPayloadPersistenceType=‘FILE_NATIVE’}
2021-09-28 18:13:14,627 INFO - Write metadata file: MetaInformation{hivemqVersion=‘2021.2’, clientSessionPersistenceVersion=‘040000’, queuedMessagesPersistenceVersion=‘040500’, subscriptionPersistenceVersion=‘040000’, retainedMessagesPersistenceVersion=‘040500_R’, publishPayloadPersistenceVersion=‘040500_R’, retainedMessagesPersistenceType=‘FILE_NATIVE’, publishPayloadPersistenceType=‘FILE_NATIVE’}
2021-09-28 18:13:14,627 INFO - Finished migration.

Thanks & Regards,
-Venkat

Ah not the logs from the migration.log, please post the content from the event.log

And please post the event.log from both instances

Hi @michael_w

Please find below the logs from event.log from EC2 Instance 1.

2021-09-28 18:00:01,845 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:00:11,844 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:00:21,844 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:00:31,845 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:00:41,845 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:00:51,844 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:01:01,845 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:01:11,845 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:01:21,844 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:01:31,846 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:01:41,844 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:01:51,844 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:02:01,844 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:02:11,844 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:02:21,844 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:02:31,845 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:02:41,845 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:02:51,844 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:03:01,846 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:03:11,845 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:03:21,844 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:03:31,844 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:03:41,844 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:03:51,845 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:04:01,846 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:04:11,845 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:04:21,844 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:04:31,844 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:04:41,844 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:04:51,844 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:05:01,851 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:05:11,848 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:05:21,845 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:05:31,844 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:05:41,845 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:05:51,844 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:06:01,846 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:06:11,845 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:06:21,844 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:06:31,844 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:06:41,845 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:06:51,844 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:07:01,844 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:07:11,844 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:07:21,844 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:07:31,845 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:07:41,844 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:07:51,845 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:08:01,844 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:08:11,845 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:08:21,844 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:08:31,844 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:08:41,844 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:08:51,844 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:09:01,844 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:09:11,844 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:09:21,845 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:09:31,845 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:09:41,845 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:09:51,846 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:10:01,844 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:10:11,844 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:10:21,847 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:10:31,844 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:10:41,844 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:10:51,844 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:11:01,845 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:11:11,844 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:11:21,844 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:11:31,845 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:11:41,844 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:11:51,844 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:12:01,165 - Client ID: subclient1, IP: 117.236.183.78 disconnected ungracefully.
2021-09-28 18:12:01,166 - Client ID: pubclient1, IP: 117.236.183.78 disconnected ungracefully.

Hi @michael_w

Please find below the logs from event.log from EC2 Instance2.

2021-09-28 18:13:16,372 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:13:26,248 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:13:36,233 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:13:46,233 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:13:56,233 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:14:06,233 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:14:16,233 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:14:26,233 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:14:36,235 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:14:46,232 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:14:56,232 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:15:06,235 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:15:16,234 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:15:26,233 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:15:36,235 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:15:46,233 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:15:54,291 - Client ID: subclient1, IP: 117.236.183.78, Clean Start: false, Session Expiry: 3600 connected.
2021-09-28 18:15:56,232 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:16:06,234 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:16:16,239 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:16:26,237 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:16:36,234 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:16:46,233 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:16:56,232 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:17:06,233 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:17:16,233 - Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.
2021-09-28 18:17:21,464 - Client ID: subclient1, IP: 117.236.183.78 disconnected gracefully.

Hi @vgkkonda,

step 1-7 work because the load balancer directed both clients to instance1. That is also why both clients get disconnected when you shut down the instance.

2021-09-28 18:12:01,165 - Client ID: subclient1, IP: 117.236.183.78 disconnected ungracefully.
2021-09-28 18:12:01,166 - Client ID: pubclient1, IP: 117.236.183.78 disconnected ungracefully.

As for the steps 8 and 9 it is not visible that the publisher connected to that instance at all, are these the full event.logs?

My current assumptions are:

  • you either connected the subscriber once before to instance 2 that’s why it had also a session there
  • you first connected the subscriber then let the publisher publish

I’m leaning more to the first point.

P.S.: Please remove the “Client ID: UNKNOWN, IP: UNKNOWN disconnected ungracefully.” logs as those are most likely the health checks of the AWS load balancer and are not important here. This makes it easier to read the event.log

Greetings,
Michael from the HiveMQ team

Hi @michael_w

Below are the event log details from Instance1 after removing Unknown logs.

2021-09-28 17:41:16,626 - Client ID: subclient1, IP: 117.236.183.78, Clean Start: false, Session Expiry: 3600 connected.
2021-09-28 17:41:34,540 - Client ID: pubclient1, IP: 117.236.183.78, Clean Start: false, Session Expiry: 3600 connected.
2021-09-28 17:41:46,675 - Client ID: subclient1, IP: 117.236.183.78 disconnected gracefully.
2021-09-28 17:42:10,485 - Client ID: subclient1, IP: 117.236.183.78, Clean Start: false, Session Expiry: 3600 connected.
2021-09-28 18:12:01,165 - Client ID: subclient1, IP: 117.236.183.78 disconnected ungracefully.
2021-09-28 18:12:01,166 - Client ID: pubclient1, IP: 117.236.183.78 disconnected ungracefully.

Below are the event log details from Instance2 after removing Unknown logs.

2021-09-28 17:38:02,110 - Client ID: subclient1, IP: 117.236.183.78, Clean Start: false, Session Expiry: 3600 connected.
2021-09-28 17:40:13,640 - Client ID: pubclient1, IP: 117.236.183.78, Clean Start: false, Session Expiry: 3600 connected.
2021-09-28 17:40:42,099 - Client ID: pubclient1, IP: 117.236.183.78 disconnected ungracefully.
2021-09-28 17:40:42,099 - Client ID: subclient1, IP: 117.236.183.78 disconnected ungracefully.
2021-09-28 18:15:54,291 - Client ID: subclient1, IP: 117.236.183.78, Clean Start: false, Session Expiry: 3600 connected.
2021-09-28 18:17:21,464 - Client ID: subclient1, IP: 117.236.183.78 disconnected gracefully.

Thanks & Regards,
-Venkat

Hi @vgkkonda,

my assumption was right, as you can see in the logs (of both instances actually) the subscriber connected before the publisher published the message, therefore the session already existed so the message got queued.

Greetings,
Michael from the HiveMQ team