This question was asked by Dan Davies during the MQTT 5 webinar.
Answer:
It is impossible to answer this question in a general manner, as the amount of messages that can be consumed by a single client is largely dependant on the specific implementation.
High performance implementations like the Hive MQTT Client Library are specifically designed to handle a 5 digit amount of messages per second, while other libraries may struggle with handling more than 1000 msg/s.
Generally speaking it is a good idea to utilize Shared Subscriptions for “root wildcard backend subscriber” scenarios, as it allows client load balancing.
Hi @hivemq-support,
Dear Florian,
Sorry for being a topic necromancer, but there is no topic I found that is closer to my problem, and this question perfectly fits my problem.
So we’re implemented a broker system with HiveMQ 4 CE (hivemq-ce-2020.2) as our central broker. I added a memory queue as a Java Extension to the system, and then via this memory queue, I use postgresql’s COPY FROM to bulk load all received messages into an SQL database the fastest way possible…
Currently, we have reached a limit of saving 30 000 - 33 000 messages / seconds into the database (ONE publisher with Hive MQTT Client) …In the future we will have multiple senders (5-50), altogether the broker should handle a combined total of 100-300 thousand messages / seconds… Messages are medium-sized IoT messages, 50-100 bytes / messages
I have the following questions:
-
What is the maximum limit for a single MQTT client? Is it possible to send+receive more than 33K msg/sec for a single connection, with HiveMQ Java Client?
-
What server configuration settings do you recommend for maximizing the number of incoming messages? Lets say I am willing to commit ALL RAM and ALL CPU to the hiveMQ process?
-
What client configuration settings do you recommend for maximizing the number of transmitted messages?
-
for the 100-300k msg/sec rate, do you recommend a HiveMQ cluster, or one central HiveMQ broker could be enough? I saw https://www.hivemq.com/benchmark-10-million/ , it says “3 billion msg/hour” ==> 833 333 msg/sec … BUT 10,000,000 clients ; this yields a very low number of messages per client…
Thank you for your help
Zsolt Szabo
Óbuda Unversity
Budapest, Hungary
Hey Zsolt,
Welcome to the community!
Just curious: Are you working on a bachelor or master thesis around MQTT?
Don’t worry about the topic but please feel free to open a new one next time you have a question that does not fit.
I will go through your questions and answer them as best I can:
-
The biggest limiting factor is the latency between the broker and the client. If bandwidth and latency support that throughput, I do not see why this couldn’t be achieved.
-
You should always make sure that HiveMQ has uninterrupted access to all system resources. In terms of settings you can do some tweaks to the TCP socket configuration.
By increasing the server receive maximum, you can make sure the server can handle more in-flight messages. Ultimately this helps with handling burst messages.
You can set this value via the HiveMQ config. -
Similarly to the server receive maximum there is a client receive maximum. You should also make sure that client gets sufficient resources.
-
For anything over 100k msg/s I would recommend using a HiveMQ cluster, yes. This is especially true, if high availability and more scalability in the future are a concern.
Hope this helps.
Let me know, if you have additional questions.
Kind regards,
Florian
Hi Florian,
Thanks for your detailed reply, we will try to configure our software accordingly.
Its not a thesis work, its a research project arount healthcare / mass ventillators for covid-like situations ( http://massventil.org/en/massventil-project/ ) that might be used in critical environments in the future… (We are FAR from completion, currently its only a prototype with an mqtt communication + 2-3 prototype ventillators on ambu cpr manikins) …
We plan to have capabilities for 10-50 patients and 1000-7000 msg/sec/patient, so altogether 10-50 mqtt clients + 1 broker + 10k-300k msg/sec
Yours,
Zsolt