Marking the end of a series of messages

MQTT
1

I am sending chunks of data in a series of messages. Is there a standard to mark the end of the message series?

2

Hi @Kurvel

Welcome to the HiveMQ Community! We’re excited to have you join us, especially because of your interest in MQTT and the HiveMQ broker. It’s great to see new users like you.

In MQTT, there is no official standard in the protocol itself to mark the “end” of a series of messages. However, there are common patterns and best practices to achieve this. Below are some approaches you can use, depending on your specific use case:

1. Use a Special Message (End Marker)

  • Send a final message in the series with a special payload that indicates the end of the sequence, such as "END", "EOF", or any agreed-upon keyword.
  • Example:
    Message 1: {"chunk": 1, "data": "part1"}
Message 2: {"chunk": 2, "data": "part2"}
Message 3: {"chunk": 3, "data": "END"}
  • This requires the consumer to recognize the marker and handle it appropriately.

2. Add Metadata in the Payload

  • Include metadata in each message payload to indicate its position and total count in the series (e.g., message number and total number of messages).
  • Example:
    Message 1: {"chunk": 1, "total": 3, "data": "part1"}
Message 2: {"chunk": 2, "total": 3, "data": "part2"}
Message 3: {"chunk": 3, "total": 3, "data": "part3"}
  • When the consumer receives the message where chunk == total, it knows the series is complete.

3. Use MQTT Topics to Indicate Sequence

  • You can include sequence information in the MQTT topic itself.
  • Example topics:
    data/chunk/1
data/chunk/2
data/chunk/END
  • The END topic indicates the end of the series.

4. Leverage Retained Messages

  • Publish the end marker as a retained message on a dedicated topic to indicate that the series is complete.
  • Example:
    • Topic: data/end_marker
    • Retained Message: "END"

5. Set a Timeout

  • If you cannot explicitly mark the end, you can implement a timeout mechanism. If no new messages are received within a certain time frame, assume the series is complete. This is less reliable but might work for simpler use cases.

Best of luck with your implementation!

Dasha from The HiveMQ Team

3

Hi @Daria_H

Thanks for your reply! Interesting approaches. I am sure one of them will suit our use case.