following this article “Building an elastic high availability MQTT broker cluster on AWS” when I restart the instance, there is no message to show that successful cluster establishment.
instead, the image below was what I got, it seems like it cannot find any members
Hi @jeffrey,
can you check that your config is correct? I see a WARN log that cluster discovery is not set to extension:
The blog post lists the example, I still post it here (mind replacing the IP_ADDRESS):
<cluster>
<enabled>true</enabled>
<transport>
<tcp>
<bind-address>IP_ADDRESS</bind-address>
<bind-port>7800</bind-port>
</tcp>
</transport>
<discovery>
<extension/>
</discovery>
</cluster>
Greetings,
Michael from the HiveMQ team
Hi Michael
Thanks for your reply
I did set to , but I saw a WARN log was “can not read the opt/hivemq/conf/config.xml” but O have double checked all the spelling and pretty sure that there’s no wrong spelling. Also Wanna double check for the IP ADDRESS is that private ip or public IP? I put the private ip address of that instance is that corrent?
Thanks
Best Regards
Jeffrey Lin
Please post your config.xml (omit sensitive information) and restart HiveMQ with log level DEBUG there you should get the reason why it couldn’t read the config.xml. You can adapt the log level in the conf/logback.xml from
<root level="${HIVEMQ_LOG_LEVEL:-INFO}">
to
<root level="${HIVEMQ_LOG_LEVEL:-DEBUG}">
As for
Also Wanna double check for the IP ADDRESS is that private ip or public IP? I put the private ip address of that instance is that corrent?
If you can ping from one instance to the other instance using the private IP then yes use the private IP.
Greetings,
Michael from the HiveMQ team
thank you so much Michael, I would have a try!
This will not help, we need the DEBUG log after the ERROR log, there the reason is logged for the ERROR message.
Also please post your config.xml in code (use the “Preformatted Text”) and not as a image so I can copy it and check if it works on my machine. I took a look at the config.xml though and it looks ok. So my current guess is that there is an issue with permissions.
Greetings,
Michael
Oh sorry about that, I will find it and post later, currently on my way home🤣
<?xml version="1.0"?>
<hivemq>
<listeners>
<tcp-listener>
<port>1883</port>
<bind-address>0.0.0.0</bind-address>
</tcp-listener>
</listeners>
<cluster>
<enabled>true</enabled>
<transport>
<tcp>
<bind-address>172.31.44.31</bind-address>
<bind-port>7800</bind-port>
</tcp>
</transport>
<discovery>
<extension/>
</discovery>
</cluster>
<anonymous-usage-statistics>
<enabled>true</enabled>
</anonymous-usage-statistics>
<control-center>
<listeners>
<http>
<port>8080</port>
<bind-address>0.0.0.0</bind-address>
</http>
</listeners>
</control-center>
</hivemq>
2022-10-14 07:47:17,415 DEBUG - Setting retained messages enabled to true
2022-10-14 07:47:17,416 DEBUG - Setting wildcard subscriptions enabled to true
2022-10-14 07:47:17,416 DEBUG - Setting subscription identifier enabled to true
2022-10-14 07:47:17,416 DEBUG - Setting shared subscriptions enabled to true
2022-10-14 07:47:17,416 DEBUG - Setting maximum qos to EXACTLY_ONCE
2022-10-14 07:47:17,416 DEBUG - Setting topic alias enabled to true
2022-10-14 07:47:17,416 DEBUG - Setting topic alias maximum per client to 5
2022-10-14 07:47:17,416 DEBUG - Setting the number of max queued messages per client to 1000 entries
2022-10-14 07:47:17,416 DEBUG - Setting queued messages strategy for each client to DISCARD
2022-10-14 07:47:17,416 DEBUG - Setting the expiry interval for client sessions to 4294967295 seconds
2022-10-14 07:47:17,416 DEBUG - Setting the expiry interval for publish messages to 4294967296 seconds
2022-10-14 07:47:17,416 DEBUG - Setting the server receive maximum to 10
2022-10-14 07:47:17,416 DEBUG - Setting keep alive maximum to 65535 seconds
2022-10-14 07:47:17,416 DEBUG - Setting keep alive allow zero to true
2022-10-14 07:47:17,416 DEBUG - Setting the maximum packet size for mqtt messages 268435460 bytes
2022-10-14 07:47:17,417 DEBUG - Setting global maximum allowed connections to -1
2022-10-14 07:47:17,417 DEBUG - Setting the maximum client id length to 65535
2022-10-14 07:47:17,417 DEBUG - Setting the timeout for disconnecting idle tcp connections before a connect message was received to 10000 milliseconds
2022-10-14 07:47:17,417 DEBUG - Throttling the global incoming traffic limit 0 bytes/second
2022-10-14 07:47:17,417 DEBUG - Setting the maximum topic length to 65535
2022-10-14 07:47:17,417 DEBUG - Setting allow server assigned client identifier to true
2022-10-14 07:47:17,417 DEBUG - Setting validate UTF-8 to true
2022-10-14 07:47:17,417 DEBUG - Setting payload format validation to false
2022-10-14 07:47:17,417 DEBUG - Setting allow-problem-information to true
2022-10-14 07:47:17,417 DEBUG - Setting control center audit log enabled to true
2022-10-14 07:47:17,418 DEBUG - Setting replica count to 2
2022-10-14 07:47:17,418 DEBUG - Setting cluster TCP health-check disabled
2022-10-14 07:47:17,418 DEBUG - Setting cluster heartbeat disabled
2022-10-14 07:47:17,418 DEBUG - Setting HiveMQ Control Center enabled to true
2022-10-14 07:47:17,418 DEBUG - Setting HiveMQ Control Center default login enabled to true
2022-10-14 07:47:17,418 DEBUG - Setting HiveMQ Control Center session idle time to 14400 seconds
2022-10-14 07:47:17,426 DEBUG - Setting overload protection enabled to true
2022-10-14 07:47:17,427 DEBUG - Setting anonymous usage statistics enabled to false
2022-10-14 07:47:17,427 DEBUG - Setting HiveMQ REST API enabled to false
2022-10-14 07:47:17,427 DEBUG - Setting HiveMQ REST API authentication enabled to false
2022-10-14 07:47:17,427 DEBUG - Setting expired messages topic enabled to false
2022-10-14 07:47:17,427 DEBUG - Setting dropped messages topic enabled to false
2022-10-14 07:47:17,427 DEBUG - Setting dead messages topic enabled to false
2022-10-14 07:47:17,427 DEBUG - Setting client event history enabled to false
2022-10-14 07:47:17,427 DEBUG - Setting client event history lifetime to 604800000 ms
2022-10-14 07:47:17,428 INFO - This node's ID is gajPF
2022-10-14 07:47:17,428 INFO - Clustering is disabled
2022-10-14 07:47:17,817 DEBUG - 476.50 MB allocated for qos 0 inflight messages
2022-10-14 07:47:18,437 DEBUG - Initializing payload reference count and queue sizes for client_queue persistence.
2022-10-14 07:47:18,554 DEBUG - Diagnostic mode is disabled
2022-10-14 07:47:19,535 DEBUG - Native Epoll is available on this platform
2022-10-14 07:47:19,549 DEBUG - Set extension executor thread pool size to 4
2022-10-14 07:47:19,549 DEBUG - Set extension executor thread pool keep-alive to 30 seconds
2022-10-14 07:47:19,568 INFO - No valid license file found. Using trial license, restricted to 25 connections.
2022-10-14 07:47:19,744 DEBUG - Throttling incoming traffic to 0 B/s
2022-10-14 07:47:19,744 DEBUG - Throttling outgoing traffic to 0 B/s
2022-10-14 07:47:19,792 DEBUG - Building initial topic tree
2022-10-14 07:47:19,867 DEBUG - Started JMX Metrics Reporting.
2022-10-14 07:47:19,879 DEBUG - Registered JVM metrics with prefix com.hivemq.jvm.
2022-10-14 07:47:19,962 INFO - This node uses '4' CPU cores.
2022-10-14 07:47:19,970 INFO - Starting HiveMQ extension system.
2022-10-14 07:47:20,013 INFO - Starting extension with id "hivemq-allow-all-extension" at /opt/hivemq/extensions/hivemq-allow-all-extension
2022-10-14 07:47:20,021 WARN -
################################################################################################################
# This HiveMQ deployment is not secure! You are lacking Authentication and Authorization. #
# Right now any MQTT client can connect to the broker with a full set of permissions. #
# For production usage, add an appropriate security extension and remove the hivemq-allow-all extension. #
# You can download security extensions from the HiveMQ Marketplace (https://www.hivemq.com/extensions/). #
################################################################################################################
2022-10-14 07:47:20,022 DEBUG - Simple authenticator added by extension 'hivemq-allow-all-extension'.
2022-10-14 07:47:20,022 INFO - Extension "Allow All Extension" version 1.0.0 started successfully.
2022-10-14 07:47:20,022 INFO - Starting extension with id "hivemq-s3-cluster-discovery-extension" at /opt/hivemq/extensions/hivemq-s3-cluster-discovery-extension
2022-10-14 07:47:20,037 WARN - A Cluster discovery callback is present, but HiveMQ ist not configured to use extension discovery. Please check your configuration.
2022-10-14 07:47:20,037 DEBUG - Registered S3 discovery callback successfully.
2022-10-14 07:47:20,037 INFO - Extension "S3 Cluster Discovery Extension" version 4.0.1 started successfully.
2022-10-14 07:47:22,145 INFO - gajPF: no members discovered after 2001 ms: creating cluster as first member
2022-10-14 07:47:22,171 INFO - No user for HiveMQ Control Center configured. Starting with default user
2022-10-14 07:47:22,171 INFO - Starting HiveMQ Control Center on address 127.0.0.1 and port 8080
2022-10-14 07:47:22,318 INFO - Control Center Audit Logging started.
2022-10-14 07:47:22,319 INFO - Started HiveMQ Control Center in 148ms
2022-10-14 07:47:22,319 DEBUG - Adding TCP Listener on bind address 0.0.0.0 and port 1883. Name: tcp-listener-1883. Proxy Protocol supported: false
2022-10-14 07:47:22,331 INFO - Starting TCP listener on address 0.0.0.0 and port 1883
2022-10-14 07:47:22,346 INFO - Started TCP Listener on address 0.0.0.0 and on port 1883.
2022-10-14 07:47:22,346 INFO - Started HiveMQ in 5144ms
You are still not giving me the DEBUG log following the ERROR message.
It should look like this:
ERROR - Could not read the configuration file {}. Using default config
DEBUG - Original error message:
In the meantime I tried your config and it successfully loaded the config, so it’s definitely not an syntax issue.
So my guess right now are permissions. Maybe you downloaded the zips as root but try to run the service as hivemq user or something along those lines. Can you run ls -lat and find out which permissions are set for the config.xml and check if the user with with you start HiveMQ has the needed permissions.
But as I said the DEBUG log after the ERROR should be very helpful to find out the issue:
ERROR - Could not read the configuration file {}. Using default config
DEBUG - Original error message:
2022-10-14 07:47:17,412 ERROR - Could not read the configuration file /opt/hivemq/conf/config.xml. Using default config
2022-10-14 07:47:17,413 DEBUG - Original error message:
javax.xml.bind.UnmarshalException: null
at javax.xml.bind.helpers.AbstractUnmarshallerImpl.createUnmarshalException(Source.java:310)
at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.createUnmarshalException(Source.java:548)
at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal0(Source.java:234)
at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal(Source.java:206)
at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal(Source.java:261)
at hmq.ca.d.e(Source.java:134)
at hmq.ca.d.a(Source.java:86)
at hmq.bW.a.a(Source.java:58)
at com.hivemq.HiveMQServer.a(Source.java:108)
at com.hivemq.HiveMQServer.d(Source.java:236)
at com.hivemq.HiveMQServer.main(Source.java:78)
Caused by: org.xml.sax.SAXParseException: The processing instruction target matching "[xX][mM][lL]" is not allowed.
at java.xml/com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:204)
at java.xml/com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:178)
at java.xml/com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:400)
at java.xml/com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:327)
at java.xml/com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1471)
at java.xml/com.sun.org.apache.xerces.internal.impl.XMLScanner.scanPIData(XMLScanner.java:745)
at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanPIData(XMLDocumentFragmentScannerImpl.java:1049)
at java.xml/com.sun.org.apache.xerces.internal.impl.XMLScanner.scanPI(XMLScanner.java:713)
at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:891)
at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:605)
at java.xml/com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:112)
at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:534)
at java.xml/com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:888)
at java.xml/com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:824)
at java.xml/com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
at java.xml/com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1216)
at java.xml/com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:635)
at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal0(Source.java:228)
... 8 common frames omitted
sorry the community said I couldn’t reply too frequently, I tried to send you an email but not sure did you receive it
Hi @jeffrey,
first time seeing this error, what I found online and how I could reproduce it was that my config.xml file didn’t start straight with the XML declaration tag:
<?xml version="1.0"?>
But had a whitespace or line break before it, like:
<?xml version="1.0"?>
So please check your config.xml and make sure to remove anything before the XML declaration tag.
Let me know if it fixed the issue.
Greetings,
Michael
Hi michael
seems that problem is fixed but a new one comes
what is hivemq not read? i didn’t change that file actually
2022-10-17 01:37:53,973 DEBUG - Reading properties file '/opt/hivemq/extensions/hivemq-s3-cluster-discovery-extension/s3discovery.properties'.
2022-10-17 01:37:55,020 ERROR - Configured bucket 'hivemq' doesn't exist. Skipping reload callback.
2022-10-17 01:38:55,020 DEBUG - Reading properties file '/opt/hivemq/extensions/hivemq-s3-cluster-discovery-extension/s3discovery.properties'.
2022-10-17 01:38:56,053 ERROR - Configured bucket 'hivemq' doesn't exist. Skipping reload callback.
for the hivemq-s3-cluster-discovery-extension.xml file, do I need to fill them up with “<>” ?
############################################################
# S3 Bucket #
############################################################
#
# Region for the S3 bucket used by hivemq
# see http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region for a list of regions for S3
# example: us-west-2
#
s3-bucket-region: ap-southeast-2
#
# Name of the bucket used by HiveMQ
#
s3-bucket-name: 712hivemqs3bucket
this is the error file
do I need to change the bucket here as well? as I didn’t see they changed in the article
############################################################
# AWS Credentials #
############################################################
#
# Use environment variables to specify your AWS credentials
# the following variables need to be set:
# AWS_ACCESS_KEY_ID
# AWS_SECRET_ACCESS_KEY
#
#credentials-type:environment_variables
#
# Use Java system properties to specify your AWS credentials
# the following variables need to be set:
# aws.accessKeyId
# aws.secretKey
#
#credentials-type:java_system_properties
#
# Uses the credentials file wich can be created by calling 'aws configure' (AWS CLI)
# usually this file is located at ~/.aws/credentials (platform dependent)
# The location of the file can be configured by setting the environment variable
# AWS_CREDENTIAL_PROFILE_FILE to the location of your file
#
#credentials-type:user_credentials_file
#
# Uses the IAM Profile assigned to the EC2 instance running HiveMQ to access S3
# Notice: This only works if HiveMQ is running on an EC2 instance !
#
#credentials-type:instance_profile_credentials
#
# Tries to access S3 via the default mechanisms in the following order
# 1) Environment variables
# 2) Java system properties
# 3) User credentials file
# 4) IAM profiles assigned to EC2 instance
#
credentials-type:default
#
# Uses the credentials specified in this file.
# The variables you must provide are:
# credentials-access-key-id
# credentials-secret-access-key
#
#credentials-type:access_key
#credentials-access-key-id:{access_key_id}
#credentials-secret-access-key:{secret_access_key}
#
# Uses the credentials specified in this file to authenticate with a temporary session
# The variables you must provide are:
# credentials-access-key-id
# credentials-secret-access-key
# credentials-session-token
#
#credentials-type:temporary_session
#credentials-access-key-id:{access_key_id}
#credentials-secret-access-key:{secret_access_key}
#credentials-session-token:{session_token}
############################################################
# S3 Bucket #
############################################################
#
# Region for the S3 bucket used by hivemq
# see http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region for a list of regions for S3
# example: us-west-2
#
s3-bucket-region:us-east-1
#
# Name of the bucket used by HiveMQ
#
s3-bucket-name:hivemq
#
# Prefix for the filename of every node's file (optional)
#
file-prefix:hivemq/cluster/nodes/
#
# Expiration timeout (in seconds).
# Files with a timestamp older than (timestamp + expiration) will be automatically deleted
# Set to 0 if you do not want the extension to handle expiration.
#
file-expiration:360
#
# Interval (in seconds) in which the own information in S3 is updated.
# Set to 0 if you do not want the extension to update its own information.
# If you disable this you also might want to disable expiration.
#
update-interval:180
#
# Endpoint config to use other compatible S3 cloud storage services.
#
s3-endpoint:s3.amazonaws.com
#s3-endpoint-region:
#
# Setting this flag will de-/activate path style access for all request.
# If not set, the path style will be set dynamically.
#