Error when starting HiveMQ service - Configured bucket 'hivemq' doesn't exist. Skipping initial discovery

RB1 · March 17, 2023, 2:31pm

Hello,

When starting HiveMQ service (/etc/init.d/hivemq restart), I see an error in the log file regarding the s3 bucket:
2023-03-17 13:32:44,163 ERROR - Configured bucket ‘hivemq’ doesn’t exist. Skipping initial discovery.

My bucket name is ‘hivemq-demo-rajiv’ but somehow picking up ‘hivemq’ - not sure why? Attached error log and s3 configuration files.

Appreciate any help - Thanks.

Best,
Rajiv

[root@ip-10-10-1-14 ~]# cat /opt/hivemq/extensions/hivemq-s3-cluster-discovery-extension/hivemq-s3-cluster-discovery-extension.xml
############################################################

S3 Bucket

############################################################

Region for the S3 bucket used by hivemq

see AWS service endpoints - AWS General Reference for a list of regions for S3

example: us-west-2

s3-bucket-region:us-east-1

Name of the bucket used by HiveMQ

s3-bucket-name:hivemq-demo-rajiv

[root@ip-10-10-1-14 ~]#

[root@ip-10-10-1-14 ~]# cat /opt/hivemq/log/hivemq.log
2023-03-17 13:32:40,219 INFO - Starting HiveMQ Enterprise Server
2023-03-17 13:32:40,222 INFO - HiveMQ version: 4.12.0
2023-03-17 13:32:40,222 INFO - HiveMQ home directory: /opt/hivemq
2023-03-17 13:32:40,223 INFO - Log Configuration was overridden by /opt/hivemq/conf/logback.xml
2023-03-17 13:32:40,557 INFO - Successfully loaded configuration from ‘/opt/hivemq/conf/config.xml’.
2023-03-17 13:32:40,615 WARN - Soft limit for open files (500000) is lower than the recommended limit (1000000). Please increase the open file limit to at least the recommended limit.
2023-03-17 13:32:40,615 WARN - Hard limit for open files (500000) is lower than the recommended limit (1000000). Please increase the open file limit to at least the recommended limit.
2023-03-17 13:32:40,618 INFO - This node’s ID is he9gs
2023-03-17 13:32:40,743 INFO - Clustering is enabled
2023-03-17 13:32:42,815 INFO - No valid license file found. Using trial license, restricted to 25 connections.
2023-03-17 13:32:43,254 INFO - This node uses ‘4’ CPU cores.
2023-03-17 13:32:43,258 INFO - Starting HiveMQ extension system.
2023-03-17 13:32:43,300 INFO - Starting extension with id “hivemq-allow-all-extension” at /opt/hivemq/extensions/hivemq-allow-all-extension
2023-03-17 13:32:43,316 WARN -
################################################################################################################

This HiveMQ deployment is not secure! You are lacking Authentication and Authorization.

Right now any MQTT client can connect to the broker with a full set of permissions.

For production usage, add an appropriate security extension and remove the hivemq-allow-all extension.

You can download security extensions from the HiveMQ Marketplace (HiveMQ Extensions Marketplace: Integrate HiveMQ into your business).

################################################################################################################
2023-03-17 13:32:43,317 INFO - Extension “Allow All Extension” version 1.0.0 started successfully.
2023-03-17 13:32:43,317 INFO - Starting extension with id “hivemq-s3-cluster-discovery-extension” at /opt/hivemq/extensions/hivemq-s3-cluster-discovery-extension
2023-03-17 13:32:43,328 INFO - Extension “S3 Cluster Discovery Extension” version 4.0.1 started successfully.
2023-03-17 13:32:43,431 INFO - Using TCP cluster transport on address 10.10.1.14 and port 7800
2023-03-17 13:32:43,437 INFO - Using extension cluster discovery
2023-03-17 13:32:44,163 ERROR - Configured bucket ‘hivemq’ doesn’t exist. Skipping initial discovery.
2023-03-17 13:32:46,164 INFO - he9gs: no members discovered after 2628 ms: creating cluster as first member
2023-03-17 13:32:46,171 INFO - Cluster nodes found by discovery: [he9gs|0] (1) [he9gs].
2023-03-17 13:32:46,186 INFO - No user for HiveMQ Control Center configured. Starting with default user
2023-03-17 13:32:46,186 INFO - Starting HiveMQ Control Center on address 0.0.0.0 and port 8080
2023-03-17 13:32:46,366 INFO - Control Center Audit Logging started.
2023-03-17 13:32:46,366 INFO - Started HiveMQ Control Center in 180ms
2023-03-17 13:32:46,381 INFO - Starting TCP listener on address 0.0.0.0 and port 1883
2023-03-17 13:32:46,401 INFO - Started TCP Listener on address 0.0.0.0 and on port 1883.
2023-03-17 13:32:46,401 INFO - Started HiveMQ in 6189ms
2023-03-17 13:33:44,454 ERROR - Configured bucket ‘hivemq’ doesn’t exist. Skipping reload callback.
[root@ip-10-10-1-14 ~]#

Daria_H · March 17, 2023, 3:28pm

Hi @RB1 ,

Please try the latest version of the extension and see if you observe the same error with it. You can download it here: Release 4.1.1 · hivemq/hivemq-s3-cluster-discovery-extension · GitHub

Kind regards,
Dasha from HiveMQ Team

RB1 · March 17, 2023, 3:45pm

Hi Dasha,

Thanks for the note. I am expected to use the version I have. QQ - do the following configuration file looks OK? If possible, are you able to send me a sample working copy of hivemq-s3-cluster-discovery-extension.xml file?

[root@ip-10-10-1-14 ~]# cat /opt/hivemq/extensions/hivemq-s3-cluster-discovery-extension/hivemq-s3-cluster-discovery-extension.xml
############################################################

S3 Bucket

############################################################

Region for the S3 bucket used by hivemq

see AWS service endpoints - AWS General Reference for a list of regions for S3

example: us-west-2

s3-bucket-region:us-east-1

Name of the bucket used by HiveMQ

s3-bucket-name:hivemq-demo-rajiv

[root@ip-10-10-1-14 ~]#

Thank you,
Rajiv

Diego · March 17, 2023, 9:02pm

Hello Rajiv,

Please follow these steps below in order to get the S3 Cluster Discovery Extension to work.

Download the latest version of S3 Cluster Discovery Extension.

cd /opt/hivemq/extensions/

wget https://github.com/hivemq/hivemq-s3-cluster-discovery-extension/releases/download/4.1.1/hivemq-s3-cluster-discovery-extension-4.1.1.zip

Copy the content of the zip file to the extensions folder of your HiveMQ nodes.

unzip hivemq-s3-cluster-discovery-extension-4.1.1.zip

Create environment variables credentials to be able to access the S3 bucket from your HiveMQ nodes (easy way).

export AWS_ACCESS_KEY_ID=[your-access-key]
export AWS_SECRET_ACCESS_KEY=[your-secret-key]

Modify the s3discovery.properties file for your needs. Make sure to set the correct region in which your S3 bucket resides.

s3-bucket-region:us-east-1
s3-bucket-name:hivemq
file-prefix:hivemq/cluster/nodes/
file-expiration:360
update-interval:180
credentials-type:environment_variables

Change the discovery mechanism of HiveMQ to extension. Example at HiveMQ Clusters :: HiveMQ Documentation

Please refer to the extension documentation for additional information at GitHub - hivemq/hivemq-s3-cluster-discovery-extension: HiveMQ extension for dynamic clustering with AWS S3 discovery

Best regards,
Diego from HiveMQ Team

RB1 · March 18, 2023, 1:27pm

Hi Diego,

My original question - why is it stuck on the bucket name ‘hivemq’? Can I name this bucket anything else per s3 naming convention (e.g. hivemq-demo-rajiv’)? See below logs.
I have created a role for ec2 instances with full s3 access. Therefore I shouldn’t be needing my aws credentials. I’ll go down that path onve we figure out the bucket naming issue.

Appreciate your support.

Best,
Rajiv

################################################################################################################
2023-03-18 13:21:06,460 INFO - Extension “Allow All Extension” version 1.0.0 started successfully.
2023-03-18 13:21:06,461 INFO - Starting extension with id “hivemq-s3-cluster-discovery-extension” at /opt/hivemq/extensions/hivemq-s3-cluster-discovery-extension
2023-03-18 13:21:06,468 INFO - Extension “S3 Cluster Discovery Extension” version 4.0.1 started successfully.
2023-03-18 13:21:06,607 INFO - Using TCP cluster transport on address 10.10.1.14 and port 7800
2023-03-18 13:21:06,621 INFO - Using extension cluster discovery
2023-03-18 13:21:07,235 ERROR - Configured bucket ‘hivemq’ doesn’t exist. Skipping initial discovery.
2023-03-18 13:21:09,236 INFO - uc5Ef: no members discovered after 2541 ms: creating cluster as first member
2023-03-18 13:21:09,242 INFO - Cluster nodes found by discovery: [uc5Ef|0] (1) [uc5Ef].
2023-03-18 13:21:09,258 INFO - No user for HiveMQ Control Center configured. Starting with default user
2023-03-18 13:21:09,258 INFO - Starting HiveMQ Control Center on address 0.0.0.0 and port 8080
2023-03-18 13:21:09,436 INFO - Control Center Audit Logging started.
2023-03-18 13:21:09,437 INFO - Started HiveMQ Control Center in 179ms
2023-03-18 13:21:09,452 INFO - Starting TCP listener on address 0.0.0.0 and port 1883
2023-03-18 13:21:09,472 INFO - Started TCP Listener on address 0.0.0.0 and on port 1883.
2023-03-18 13:21:09,473 INFO - Started HiveMQ in 6116ms
2023-03-18 13:22:07,528 ERROR - Configured bucket ‘hivemq’ doesn’t exist. Skipping reload callback.
2023-03-18 13:23:07,823 ERROR - Configured bucket ‘hivemq’ doesn’t exist. Skipping reload callback.
2023-03-18 13:24:08,110 ERROR - Configured bucket ‘hivemq’ doesn’t exist. Skipping reload callback.
2023-03-18 13:25:08,399 ERROR - Configured bucket ‘hivemq’ doesn’t exist. Skipping reload callback.
2023-03-18 13:26:08,689 ERROR - Configured bucket ‘hivemq’ doesn’t exist. Skipping reload callback.
2023-03-18 13:27:08,977 ERROR - Configured bucket ‘hivemq’ doesn’t exist. Skipping reload callback.
^C
[root@ip-10-10-1-14 ec2-user]#

Daria_H · March 18, 2023, 1:57pm

Hi @RB1 ,

The name of the configuration file should be s3discovery.properties rather than hivemq-s3-cluster-discovery-extension.xml

I hope it helps,
Kind regards,
Dasha from HiveMQ Team

Diego · March 20, 2023, 9:56am

@Hello @RB1

My original question - why is it stuck on the bucket name ‘hivemq’? Can I name this bucket anything else per s3 naming convention (e.g. hivemq-demo-rajiv’)? See below logs.

It must be your already configured S3 bucket name. It’s because you are not setting the correct bucket name on the s3discovery.properties file

I have created a role for ec2 instances with full s3 access. Therefore I shouldn’t be needing my aws credentials. I’ll go down that path onve we figure out the bucket naming issue.

If you are using IAM instance profile that grants access to Amazon S3 you need to configure the s3discovery.properties file to use “Instance Profile Credentials Authentication”.

Sample s3discovery.properties file that should work in your scenario.

s3-bucket-region:us-east-1
s3-bucket-name:hivemq-demo-rajiv
file-prefix:hivemq/cluster/nodes/
file-expiration:360
update-interval:180
credentials-type:instance_profile_credentials

Please note that this only works if your EC2 instance has configured the right IAM Role to access S3 bucket! You can validate access to this S3 bucket by installing AWS CLI on your EC2 instance and by running the following AWS CLI command

aws s3 ls s3://hivemq-demo-rajiv

Best regards,
Diego from HiveMQ Team