I just replaced our moquette mqtt library with the hiveMq library. While I really liked the API of the hivemq edition over the other one. I just deployed the new library in a shadow environment. Immediately I noticed that the publish method is WAAY slower than in the old library.
After adding a timer around the forwardToMqtt method i see this:
With the light blue line bing the new version and the purple one being the old one.
This seemed strange to me, as there are not even any clients connected in the shadow environment and there are in the old one. I saw that the publish method returned a completableFuture, but I decided to wrap it in the ManagedExecutorsService anyway.
Now it is a lot faster, but the executorService is really occupied, after adding a Gauge to the task queue size I saw this:
Which seems really strange to me, there are about 400K messages to be published which stays steady, but then all of a sudden it completes them all in relatively little time.
Except for this there is not going on anything else, except for some scheduled methods (but that is only once a minute, and is a small task).
Can somebody explain to me how such a thing could be caused and what I can do to fix such a thing? It seems really strange to me that the executorService is waiting so long to start processing as there should be more than enough threads available. Is there a thing I can fix such a thing?
Is there a way to publish A LOT of messages (with retain) quickly, when there are no subscribers. Currently this is VERY slow (~1 ms/message).
I even tried turning on in memory persistence, but that barely makes any difference. Is it supposed to be so slow? I’m not using that many extra threads.
No, I’m using the HiveMq community edition, but I’m listening to an external source from which I call Services.PublishService(…)
However, I see that publishing roughly 1M of payloads it takes roughly 7 minutes or so. That sounds like it is way to long. For now I was able to reduce it by first consuming all the payloads and combining them and just publish roughly 200K payloads in parallel. This reduces it to roughly 2-3 minutes, which still seems like to long to me.
Even switching to in-memory persistence didn’t do anything, so its not the write time.
We recommend considering HiveMQ Swarm for conducting tests in these scenarios. HiveMQ Swarm stands out as a sophisticated IoT testing and simulation tool, providing the essential capabilities for load and reliability testing. HiveMQ Swarm allows testing millions of MQTT clients, processing millions of MQTT messages, and handling hundreds of thousands of MQTT topic names. It empowers you to assess the resilience and capacity of your entire IoT system effectively. Please give it a try and let us know if you need any help or have any questions.
