Persistent session and queued messages

1

Hi,

I wanted a few clarification’s related to Persistent sessions.

I am trying “persistent sessions” and I generated some data so that the queue size is 1000 for 1 client which is disconnected and clean session set to 'false". I also noted in the Control center console the used disk space went up by 230 MB after generating this data. Then I connected the subscribing client to the broker and all the messages in the queue were delivered to the client and the queue size went down to 0. But I noticed the disk usage was still showing the same number even after the queue was cleared. I would like to know where is the persistent data stored and why didn’t the used disk space number come down after delivering all the data stored in persistence by the broker . I checked the “data” folder and the size was showing as 11 MB when the queue size was 1000. So is the queued data stored in some other directory on the disk?

I restarted the broker and after the restart the disk size was showing much lesser number than what I had noted before generating data for 1000 message queue size.

I have also noticed when the broker is started the disk space usage goes up by 100 MB in the 15 minutes and then stops. Is there a reason why this happens?

Please do let me know if you would need more information. I was trying this exercise with HiveMQ Enterprise version 4.5.3 on Ubuntu 18.04.4 LTS

Regards,
Harsha

2

Hi @Harsha,

from experience I say higher than expected disk usage is normal as the underlying persistence frameworks (xodus and rocksDB) keep extra information to work efficient. If you can tell me how big your messages where I will try to reproduce this and confirm my guess (or not).
Update: Please also post your config.xml

I have also noticed when the broker is started the disk space usage goes up by 100 MB in the 15 minutes and then stops. Is there a reason why this happens?

This is due to the statistics logs rocksDB generates, see this comment: RocksDB "LOG" statistics file using unnecessary amount of storage. · Issue #183 · hivemq/hivemq-community-edition · GitHub

Greetings,
Michael from the HiveMQ team

3

Hi Michael,

Thanks for your response.

A single message looks like this. A simple key value pair using NodeJS.
[
{
“timestamp”: “2021-05-26T05:35:24.000Z”,
“value”: -0.54
}
]

We can refer the following site for usage time-series-data-generator - npm

The key value pair is converted to JSON string like this where data is the actual data.

JSON.stringify(data, null, 4)

This is the actual code snippet
const Series = require(“time-series-data-generator”);
const series = new Series();

let d = new Date();
let from = d.setDate(d.getDate() - 1);
from = new Date(from).toISOString();

//const from = "2021-04-20T00:00:00Z";
const until = new Date().toISOString();
const numOfData = 1;
const series = new Series({ type: "random", from, until, numOfData });	
data = series.sin()
console.log(data)
for(var x=0;x<1000;x++){
  client.publish('data/'+client.options.clientId+'/', JSON.stringify(data, null, 4),{qos: 2,retain:false})
}

My config.xml is as follows

<?xml version="1.0"?> 
<listeners>
<tls-websocket-listener>
    <port>8000</port>
    <bind-address>0.0.0.0</bind-address>
    <path>/mqtt</path>
    <proxy-protocol>true</proxy-protocol>
    <tls>
        <keystore>
            <path>./conf/hivemq.jks</path>
            <password>changeme</password>
            <private-key-password>changeme</private-key-password>
        </keystore>
        <client-authentication-mode>REQUIRED</client-authentication-mode>
        <truststore>
            <path>./conf/hivemq-trust-store.jks</path>
            <password>changeme</password>
        </truststore>
    </tls>
</tls-websocket-listener>
</listeners>

<anonymous-usage-statistics>
    <enabled>true</enabled>
</anonymous-usage-statistics>

 <rest-api>
 <enabled>true</enabled>
 <listeners>
     <http>
         <port>8888</port>
         <bind-address>localhost</bind-address>
     </http>
 </listeners>
 </rest-api>

Regards,
Harsha