MQTT's data directory/file are increased steadily

HiveMQ Community Edition
1

Dear all,

When simulating the subscribe/pusher a lot of messages with qos 2, I found that data directory are increased steadily and never decrease the size although stoping the hivemq.
It will cause the issue over disk memory. Could you guys help to decrease the size? for ex: after pushed message successfully…

data/persistence/publish_payload_store
data/persistence/client_queue/

2

one more information: The CleanUpService are running at background service, but the size still NOT decrease.

3

Hi @Phu ,

Great that you are interested in MQTT and HiveMQ, welcome to the community!

The data/persistence directory stores persistent data of the broker, such as
Retained messages
Subscriptions of persistent-session clients
Queued messages of persistent-session clients
The current state of each individual message transmission that is not yet completed

If you want to keep all persistence data, you must keep the data directory as it is. If you are Ok to lose all your persistent data, you can delete the data directory after you stop the broker. Once the broker is started again, it will create a new data directory.

In your first message, you indicated that the data directory is increasing, when you simulated (published?) a lot of messages with QoS 2. I am wondering, did you also see the data directory increasing with QoS 1 and QoS 0 messages, or this is only with QoS 2?

Kind regards,
Dasha from HiveMQ Team

4

Thank @Daria_H

My simulation test:

1 client subscribe to topic be call topic1.
1 client push 10 messages to topic1 successfully with Qos2. Then client received all 10 messages.

I double check these directories in persistence and found that the size of that are always increase. I keep clients and brokers alive and 1 hour later, I double check size again, but size is NOT decrease.

I search some message’s infomation and found that it still keep in payload persistence dir. or something like that

5

Hi @Phu ,

Thanks for describing your testing process.

  1. I did not understand, if you have tested this with QoS 0 and QoS 1 or not, could you please confirm that?
  2. To get a better overview of the activity of your client, try to check the event.log file at $HIVEMQ_HOME/log/event.log. You can filter the log for the Client ID of the1 client that you are testing with.

For example, when the client connects, the event.log will contain the line like:

Client ID: Publicio100, IP: 127.0.0.1, Clean Start: true, Session Expiry: 0 connected.

The line contains the Client ID, IP, the Clean Start flag, and the Session Expiry interval.
When a client disconnects, the event.log will contain the line like this:

Client ID: Publicio100, IP: 127.0.0.1 disconnected gracefully.

Or like this:

Client ID: MonitoringSubscriber, IP: 127.0.0.1 disconnected ungracefully.

When the client session is expired, the event.log will contain the line like this:

Client ID: Publicio100 session has expired at 2023-01-10T09:52:56.124Z. All persistent data for this client has been removed.

Please refer to the documentation for a detailed explanation of the event.log.

I hope it helps.
Kind regards,
Dasha

1 Like
6

Thank @Daria_H ,

I will explain these steps to execute:

  1. subscriber has subscribed to topic and keep alive
    using python script like : client.connect(g_broker_host, g_broker_port, keepalive=30)

2023-01-11 11:41:21,144 - Client ID: Web_58407, IP: 192.168.31.93, Clean Start: true, Session Expiry: 0 connected.
2023-01-11 11:41:21,149 - Client ID: Web_58408, IP: 192.168.31.93, Clean Start: true, Session Expiry: 0 connected.

The size of directory

hivemq du -s data/persistence/client_queue/
9176	data/persistence/client_queue/
➜  hivemq du -s data/persistence/publish_payload_store
39040	data/persistence/publish_payload_store
  1. publisher connect and push the 100 messages using Qos 2 to topic successfully and subscriber has received messages
      mqttClient.toAsync().publishWith()
                .topic(topic)
                .payload(payload)
                .qos(MqttQos.EXACTLY_ONCE) => **Qos 2.**
                .send()
  1. Check directory size again ( has waiting time)
 du -s data/persistence/publish_payload_store
39296	data/persistence/publish_payload_store
➜  hivemq du -s data/persistence/client_queue/
11224	data/persistence/client_queue/
  1. Stop the subscriber and stop the publisher
2023-01-11 11:56:55,092 - Client ID: Srv_f87b4f55-c3d1-4efd-ae10-bca53e654050, IP: 192.168.31.93 disconnected ungracefully.
2023-01-11 11:57:45,559 - Client ID: Web_58408 session has expired at 2023-01-11 04:56:54. All persistent data for this client has been removed.
2023-01-11 12:15:31,035 - Client ID: Srv_f87b4f55-c3d1-4efd-ae10-bca53e654050 session has expired at 2023-01-11 04:56:55. All persistent data for this client has been removed.
  1. Check directory size again ( has waiting time)

The size of directories are not decrease

hivemq du -s data/persistence/publish_payload_store
39936	data/persistence/publish_payload_store
➜  hivemq du -s data/persistence/client_queue/
11224	data/persistence/client_queue/

Thanks for your support.

7

Hi Phu,

Would you be able to share your HiveMQ broker’s config.xml as well?
​
Thanks,
Seth - HiveMQ Support