I’m building a system to ingest messages via subscription from an MQTT broker and, in order to test my application, I need to publish messages at volume to the broker. The actual data that I’ll be ingesting is highly structured so I’ve built a java application incorporating the HiveMQ Java Client (async mode) to generate and publish that data. However, I seem to have run into some sort of bottleneck and I’ve more or less run out of avenues to explore to try and figure this out myself. For whatever reason, I’m unable to get above a publish rate of about 250 messages per second (I’d like to get into the range of 10K to 30K messages per second). I’m using instances of Java’s ScheduledExecutorService in parallel to actually manage the send rate and, when I modify this code to not publish to the broker but to simply bump a counter variable up a tick, I’m easily able to achieve this rate. I’m using a Mosquitto for my broker and from what I’ve read online the broker should be easily capable of handling message rates in the range I’m trying to achieve. This leads me to believe that the source of the “bottleneck” is somewhere in the HiveMQ Client Library. Also, at this time, all of this software and infrastructure is running on my local environment.
So, I basically have two questions:
Should I be able to achieve the desired message rate (10K to 30K messages per second) using the HiveMQ Java Client Library?
Is there some configuration option I’m unaware of (a thread pool size perhaps?) that I need to manage to allow me to achieve the message send rates I want to achieve.
Based on the details you’ve provided, handling a publish rate of 30k messages per second with a single MQTT client in a local environment may be challenging, primarily due to limitations in network and processing resources.
While I’m not fully familiar with the throughput capabilities of the Mosquitto broker, I have conducted a simple performance test using HiveMQ Swarm publish spike scenario and HiveMQ broker on my laptop (via Docker containers). In that scenario, I was able to achieve a publish rate of ˜12k messages per second with a single MQTT client. From my experience, this doesn’t seem to indicate a bottleneck with the HiveMQ Java Client Library itself.
If you’re facing performance challenges, it might be worth investigating your local system’s capacity, Mosquitto broker performance, or exploring parallel publishing strategies.
Can you provide more guidance on “parallel publishing strategies”? I’ve been trying for some time to prevent formation of a bottleneck for message publishing. I have one application that is doing the publishing so I assume I need only one client instance. However, if I start too many asynchronous publishing tasks I either encounter an error (e.g. no packet id) or not all messages are received (despite QoS 1). I’m using the C# client.
I have posted a related issue to the GitHub project|:
