HiveMQ CE Subscribers Become Idle Under Heavy Load with Shared Subscriptions and QoS=AtLeastOnce

HiveMQ Community Edition
1

Description:

Under scenarios involving high message throughput and frequent topic publishing/subscribing, we observe that subscriber clients stop receiving messages from the HiveMQ CE broker at some point. Notably, the connection remains established and there are no any errors and warnings in the logs, but message flow ceases. When the consumer client is restarted, messages are temporarily resumed, but after some time they stop again. This behavior is specifically observed when using MqttQualityOfServiceLevel = AtLeastOnce in combination with shared subscriptions.

The Link to the repository where you can find all code examples, conf, logs and traces, videos and screenshots.

What We Have Observed & Collected:

  1. Reproduction Code & Setup:
  • We have shared code samples and a repository link demonstrating how to reproduce this issue.
    • We have three different consumer clients written on three different technologies, and the issue is reproducible in all cases. hivemq-ce-idle-issue/tree/main/src
      • NOTE: Our main stack is C# with .NET Core.
  • We have a screen-recording video illustrating the steps to trigger the behavior. hivemq-ce-idle-issue/blob/main/hivemq_idle.mp4
  1. Broker Configurations:
  • We have tested with both a default HiveMQ CE broker configuration and a custom configuration hivemq-ce-idle-issue/tree/main/custom%20broker%20conf. The issue occurs in both scenarios.
  1. Logs & Diagnostics:
  • We have broker log files and diagnostic traces from two separate test runs that exhibit the idle subscriber behavior. hivemq-ce-idle-issue/tree/main/diagnostics%20and%20logs
  1. Testing Environment & Components:

  • Publishers:

    • Implemented in .NET
    • Connection settings: cleanStart = false, protocolVersion = 5
    • Number of client publishers: 20
    • Topics: 5 shared topics

  • Subscribers:

    • Implemented using .NET, JavaScript, and Python clients
    • Connection settings: cleanStart = false, protocolVersion = 5
    • Number of subscribers: 1
    • Topic Subscriptions: 5 shared subscriptions
  1. When Does It Occur?
  • When publishing and reading messages at high rates.
  • Under heavy load conditions, like during resource-constrained conditions on the broker’s hosting environment (e.g., VM memory and CPU nearing limits).
  1. Additional Testing:
  • This issue does not occur in HiveMQ Enterprise Broker.
  • This issue does not occur with the default Mosquitto Broker.

Steps to Reproduce:

  1. Set up a HiveMQ broker (either default or custom configuration). ``
  • docker run -d --name hivemq_ce_latest_8g -p 8080:8080 -p 1883:1883 -e JAVA_OPTS="-Xmx8g" hivemq/hivemq-ce:latest
  1. Launch 20 publisher clients with cleanStart = false and protocolVersion = 5, publishing messages to 5 shared topics at a high rate.
  1. Launch a single subscriber client (in .NET, JS, or Python) with cleanStart = false and protocolVersion = 5, subscribing to the same 5 shared topics using QoS=AtLeastOnce.
  2. Observe that after some time the subscriber stops receiving messages (even though the connection remains active and there are no errors).

Expected Behavior:

Subscribers should consistently receive messages as long as the connection is active and the broker has messages on the subscribed topics, regardless of load conditions.

Actual Behavior:

After an initial period of normal message flow, subscribers become idle and do not receive any further messages until they are restarted.

Request:

We kindly request guidance on why this behavior occurs and how to mitigate it. Any insights, suggestions, or known issues related to this scenario would be greatly appreciated.

Thank you for your assistance!

PS: Because I have restrictions, I couldn’t insert links normally … new users can only put 2 links in a post

1 Like
2

If anyone is interested, a discussion of this issue can be found here: