Hivemq docker and kubernetes

triples13 · June 1, 2019, 1:30pm

hi i am following the link given for hivemq cluster formation with docker and kubernetes to build a cluster, there is one publisher and three subscriber , publisher is publishing 10k messages to each individual subscriber , subscribers starts receiving the messages as soon as publisher starts publishing them , it is working completely fine with standalone hivemq broker but after i clustered only 1 of the subscriber at a time starts receiving the messages

triples13 · June 1, 2019, 6:45pm

sometimes only 1 client gets the message, sometimes 2 clients, very different behavior i am getting in the cluster compared to standalone clients

sauroter · June 2, 2019, 8:39am

Hi @triples13,

nice to see you again

Sounds to me like your cluster was not fully formed. Can you please provide an excerpt (like the first 100 lines) from the HiveMQ log file, directly after start.

The default path to the log is HIVEMQ_HOME/log/hivemq.log.

Greetings
Georg

triples13 · June 2, 2019, 9:39am

Hi @sauroter i managed to get the logs please find the logs, i can see it’s still showing clustering is disabled , i have followed all the steps mentioned here
https://www.hivemq.com/blog/hivemq-cluster-docker-kubernetes/

also just want to check whether normal load balancing will work for MQTT broker it’s mentioned here https://stackoverflow.com/questions/31360172/load-blancing-mqtt-broker and also i think load balancer should persist the client connection data or hivemq does it by default?

Thanks for helping me out

2019-06-01 12:23:58,202 INFO - Starting HiveMQ Enterprise Server
2019-06-01 12:23:58,214 INFO - HiveMQ version: 4.0.0
2019-06-01 12:23:58,214 INFO - HiveMQ home directory: /opt/hivemq
2019-06-01 12:23:58,259 INFO - Log Configuration was overridden by /opt/hivemq/conf/logback.xml
2019-06-01 12:23:58,856 INFO - This node’s ID is xypvV
2019-06-01 12:23:58,861 INFO - Clustering is disabled
2019-06-01 12:24:04,348 INFO - Created user preferences directory.
2019-06-01 12:24:06,848 INFO - No valid license file found. Using trial license, restricted to 25 connections.
2019-06-01 12:24:06,848 INFO - This node uses ‘1’ CPU cores.
2019-06-01 12:24:06,856 INFO - Starting HiveMQ extension system.
2019-06-01 12:24:06,917 INFO - Starting File RBAC extension.
2019-06-01 12:24:07,351 INFO - Extension “File Role Based Access Control Extension” version 4.0.0 started successfully.
2019-06-01 12:24:09,444 INFO - No user for HiveMQ Control Center configured. Starting with default user
2019-06-01 12:24:09,444 INFO - Starting HiveMQ Control Center on address 127.0.0.1 and port 8080
2019-06-01 12:24:09,644 INFO - Started HiveMQ Control Center in 200ms
2019-06-01 12:24:10,235 INFO - Enabled protocols for TCP Listener with TLS at address 0.0.0.0 and port 8883: [TLSv1.2, TLSv1.1, TLSv1]
2019-06-01 12:24:10,236 INFO - Enabled cipher suites for TCP Listener with TLS at address 0.0.0.0 and port 8883: [TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA]
2019-06-01 12:24:10,247 INFO - Starting TLS TCP listener on address 0.0.0.0 and port 8883
2019-06-01 12:24:10,288 INFO - Started TCP Listener with TLS on address 0.0.0.0 and on port 8883

sauroter · June 2, 2019, 12:55pm

Hi @triples13,

when reading your log, two potential sources of trouble spring to mind:

Are you using the right image? hivemq:dns-<version>?
Do you have the HIVEMQ_HOME/extension folder mounted and no DNS Cluster Discovery Extension present?

I’am basing this on the fact that in your log HiveMQ only loads the File Role Based Access Control Extension and not the DNS Cluster Discovery Extension.

Additionally the config file must contain a passage the looks somewhat as described here: https://www.hivemq.com/docs/4.1/hivemq/cluster.html#extension.

Regarding the load balancing:

The information provided by @dobermai, the CTO of HiveMQ is still largely correct. HiveMQ will work with load balancing like HAProxy or AWS ELB.

Have a nice day

Georg

triples13 · June 2, 2019, 2:53pm

hi @sauroter to answer your question first

Are you using the right image? hivemq:dns-<version>
yes i am using the right image i am building a custom image on top of hivemq/hivemq3:dns-latest image
Do you have the HIVEMQ_HOME/extension folder mounted and no DNS Cluster Discovery Extension present
*yes i had the extension folder but the discovery extension was not present i had to copy that while building custom image , i thought dns docker image would be enough and extension would not be needed.

after enabled cluster in config file i was able to see the intended behavior but sill i am not able to see the control center(web UI) neither on the web loadbalancer:8080 ip neither on individual pods ip: 8080

2019-06-02 13:51:48,035 INFO - Starting HiveMQ Enterprise Server
2019-06-02 13:51:48,048 INFO - HiveMQ version: 4.0.0
2019-06-02 13:51:48,048 INFO - HiveMQ home directory: /opt/hivemq
2019-06-02 13:51:48,082 INFO - Log Configuration was overridden by /opt/hivemq/conf/logback.xml
2019-06-02 13:51:48,455 INFO - This node’s ID is 9PPmp
2019-06-02 13:51:48,455 INFO - Clustering is enabled
2019-06-02 13:51:54,049 INFO - Created user preferences directory.
2019-06-02 13:51:57,610 INFO - No valid license file found. Using trial license, restricted to 25 connections.
2019-06-02 13:51:57,610 INFO - This node uses ‘1’ CPU cores.
2019-06-02 13:51:57,612 INFO - Starting HiveMQ extension system.
2019-06-02 13:51:57,875 WARN - A Cluster discovery callback is present, but HiveMQ ist not configured to use extension discovery. Please check your configuration.
2019-06-02 13:51:57,912 INFO - Extension “DNS Cluster Discovery Extension” version 4.0.1 started successfully.
2019-06-02 13:51:57,921 INFO - Starting File RBAC extension.
2019-06-02 13:51:57,938 INFO - Using UDP cluster transport on address 172.17.0.5 and port 8000
2019-06-02 13:51:57,966 INFO - Using multicast cluster discovery
2019-06-02 13:51:58,024 WARN - JGRP000015: the send buffer of socket MulticastSocket was set to 5.00MB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux)
2019-06-02 13:51:58,025 WARN - JGRP000015: the receive buffer of socket MulticastSocket was set to 5.00MB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)
2019-06-02 13:51:58,025 WARN - JGRP000015: the send buffer of socket MulticastSocket was set to 5.00MB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux)
2019-06-02 13:51:58,026 WARN - JGRP000015: the receive buffer of socket MulticastSocket was set to 5.00MB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)
2019-06-02 13:51:58,199 INFO - Extension “File Role Based Access Control Extension” version 4.0.0 started successfully.
2019-06-02 13:52:00,063 INFO - Cluster nodes found by discovery: [9PPmp|0] (1) [9PPmp].
2019-06-02 13:52:00,137 INFO - No user for HiveMQ Control Center configured. Starting with default user
2019-06-02 13:52:00,138 INFO - Starting HiveMQ Control Center on address 127.0.0.1 and port 8080
2019-06-02 13:52:00,314 INFO - Cluster nodes found by discovery: [9PPmp|1] (2) [9PPmp, dGK7J].
2019-06-02 13:52:00,342 INFO - Started HiveMQ Control Center in 204ms
2019-06-02 13:52:00,909 INFO - Enabled protocols for TCP Listener with TLS at address 0.0.0.0 and port 8883: [TLSv1.2, TLSv1.1, TLSv1]
2019-06-02 13:52:00,909 INFO - Enabled cipher suites for TCP Listener with TLS at address 0.0.0.0 and port 8883:
2019-06-02 13:52:00,918 INFO - Starting TLS TCP listener on address 0.0.0.0 and port 8883
2019-06-02 13:52:00,937 INFO - Started TCP Listener with TLS on address 0.0.0.0 and on port 8883
2019-06-02 13:52:00,938 INFO - Started HiveMQ Enterprise in 12910ms
2019-06-02 13:52:01,598 INFO - Cluster size = 2, members : [9PPmp, dGK7J].
2019-06-02 13:52:01,939 INFO - Cluster nodes found by discovery: [9PPmp|2] (3) [9PPmp, dGK7J, VnaJI].
2019-06-02 13:52:03,280 INFO - Cluster size = 3, members : [9PPmp, VnaJI, dGK7J].

docker file looks like

ARG TAG=dns-latest

(1)

FROM hivemq/hivemq4:${TAG}

(2)

ENV MY_CUSTOM_PLUGIN_ENV myvalue

(3)

ENV HIVEMQ_CLUSTER_PORT 8000

(4)

#COPY your-license.lic /opt/hivemq/license/your-license.lic
COPY --chown=hivemq:hivemq /conf/config.xml /opt/hivemq/conf/config.xml
COPY --chown=hivemq:hivemq extensions /opt/hivemq/extensions
COPY --chown=hivemq:hivemq /bin/run.sh /opt/hivemq/bin/run.sh

EXPOSE 8080

(5)

RUN chmod +x /opt/hivemq/bin/run.sh

(6)

ENTRYPOINT ["/opt/hivemq/bin/run.sh"]

fraschbi · June 2, 2019, 3:20pm

There appears to be an issue with your docker image.

2019-06-02 13:51:48,048 INFO - HiveMQ version: 4.0.0

-> This is not the latest version. And you are right, the DNS image already comes with the dns discovery extension.

but sill i am not able to see the control center(web UI) neither on the web loadbalancer:8080 ip neither on individual pods ip: 8080

As you can see by the log statement you posted.

Starting HiveMQ Control Center on address 127.0.0.1 and port 8080

This means you cannot access the Control Center via the pods IPs.
You need to make sure you have a service set up for routing from the load balancer to the pods on 8080, as is mentioned in the blog post you quoted in your initial post.
Like this:

kind: Service
apiVersion: v1
metadata:
  name: hivemq-control-center
spec:
  selector:
    app: hivemq-cluster1
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 8080
  sessionAffinity: ClientIP
  type: LoadBalancer

Regards,
Florian from The HiveMQ Team.

triples13 · June 2, 2019, 6:27pm

hi @fraschbi

There appears to be an issue with your docker image.
2019-06-02 13:51:48,048 INFO - HiveMQ version: 4.0.0

i have tagged the image as in the post i am using hivemq/hivemq4:dns-latest as base image

You need to make sure you have a service set up for routing from the load balancer to the pods on 8080, as is mentioned in the blog post you quoted in your initial post.

i already have a service set up as mentioned by you

but when i try to access 10.106.115.34:8080 i don’t get any response

abdullah.imal · June 3, 2019, 7:31am

Hi @triples13,

It seems like that the 8080 port of the Docker container is mapped to the port 31921 on your host system. You should be able to connect to the Control Center when you use 10.106.115.34:31921.

Kind regards,
Abdullah

sauroter · June 3, 2019, 11:07am

Hi @triples13,

by the way. The vanishing discovery extension comes from this line:

With this you are replacing the whole extensions folder of HiveMQ with your own, this of course replaces the prepackaged discovery extension as well. If you want to avoid this behavior you can copy only the contents of your extension folder.

Greetings
Georg