Hello HiveMQ Community,
I am Markus and I am facing a problem with duplicate messages being received by the MQTT broker (HiveMQ). Here is the situation:
We have multiple sensors that are sending data over Bluetooth to Raspberry Pi-based IoT gateways. The IoT gateways then forward the received data via MQTT to the MQTT broker. The issue is that one sensor’s message is often captured by multiple IoT gateways, resulting in duplicate messages being received by the broker.
I am wondering if using QoS 2 (i.e. “exactly once” delivery) could help us deduplicate the received messages. Is this a potential solution to this problem, or are there other options that I should consider?
Any insights or recommendations would be greatly appreciated. Thank you in advance for your help!
Hi Markus.kohn.gwa,
If each gateway acts as a client of the broker, each should be using its own unique packet id for the MQTT packet sent to the broker. QoS 2 may not work as deduplication in this scenario since it relies on unique packet identifiers for the broker to recognize and handle packets for QoS levels > 0.
Per some of the below articles describing the concepts:
https://www.hivemq.com/blog/mqtt-essentials-part-4-mqtt-publish-subscribe-unsubscribe/ (packet identifier basics)
https://www.hivemq.com/blog/mqtt-essentials-part-6-mqtt-quality-of-service-levels/ (quality of service info)
For QoS 2 to work as deduplication in this scenario, the gateways would all have to be sending their packets with the same packet id. This still wouldn’t be sufficient though given packet ids can be re-used once released after a QoS 2 packet cycle, and thus you’d potentially be sending the same message payloads from other gateways with the now released packet id, still creating the duplicate scenario you are wanting to avoid.
My preference in such a setup would be to just have the broker pass everything and deal with deduplication on the receiving/ingesting side, but it may not be workable for your use case. Others here may have different (and better!) ideas.
Thanks,
Seth - HiveMQ Support
*edits for conciseness
1 Like
Thank you so much for your detailed response, Seth. I will definitely check out the provided articles to better understand the concepts of packet identifiers and QoS levels.
Your suggestion to deal with deduplication on the receiving/ingesting side is definitely something I will consider for our use case. I appreciate your recommendation and am open to any additional suggestions or ideas from the community.
Thank you again for your help and for taking the time to provide such a thorough response.
Best regards,
Markus