Dear HiveMQ Community,
Best wishes for the new year ahead!
I recently moved to CE 2020.3 after using CE 2020.2 but am still trying to get a handle on the amount of storage space used up in the publish_payload_store and retained_messages folders under data/persistence.
To help me with this, could you kindly advise -
-
what the difference is between in-memory and file modes? (My searches sadly haven’t uncovered an answer to this.)
-
whether there are any additional parameters that can be used under the retained-messages and publish-paylods configs shown below to limit or discard messages?
in-memory
in-memory
-
what additional parameters could be used to prevent publish_payload_store and retained_messages folders from reaching multiply gigabytes?
-
was support for … parameters (as shown below) only added in CE 2020.3 as per this post on it - https://www.hivemq.com/blog/hivemq-ce-2020-3-released/?
This would explain why my use of these parameters in CE 2020.2 didn’t have any impact on the amount of storage space taken up in the data/persistence folder.
in-memory
60
discard
in-memory
in-memory
<message-flow>
<!-- Changing the incoming message flow -->
<incoming>
<mode>in-memory</mode>
</incoming>
<!-- Changing the outgoing message flow -->
<outgoing>
<mode>in-memory</mode>
</outgoing>
</message-flow>
<!-- Changing the retained message to be in-memory -->
<retained-messages>
<mode>in-memory</mode>
</retained-messages>
<!-- Changing the publish payload persistence to be in-memory -->
<publish-payloads>
<mode>in-memory</mode>
</publish-payloads>
<!-- Changing the session attribute persistence to be in-memory -->
<attribute>
<mode>in-memory</mode>
</attribute>
<!-- Changing the client group persistence to be in-memory -->
<client-group>
<mode>in-memory</mode>
</client-group>
Many thanks,
Johann
Hi @thejtway,
All the best for 2021 to you too!
You are mixing up HiveMQ 3 configurations with HiveMQ CE. These two are pretty much completely unrelated software systems and as such configurations for one of them wont work with the other.
Generally you can find documentation for HiveMQ CE in the github project, explicitly in the wiki.
All the available information on the in-memory persistence can be found there as well.
To answer your questions:
- what the difference is between in-memory and file modes?
With in-memory configured, data which is usually stored in a persistent fashion on disk will be kept in memory. This includes data about PUBLISHES, clients, subscriptions and so on.
The most prominent upside is, that an in-memory HiveMQ CE hardly ever needs to access the disc and the file system.
The most prominent downsides are, that data will be lost upon system reboot and that memory consumption will correlate very strongly to deployment size. (Basically needed RAM = C * client * messages).
- whether there are any additional parameters that can be used under the retained-messages and publish-paylods configs shown below to limit or discard messages?
This setting is HiveMQ 3 specific and wont work with HiveMQ CE.
- what additional parameters could be used to prevent publish_payload_store and retained_messages folders from reaching multiply gigabytes?
Basically the only setting required in HiveMQ CE is:
<persistence>
<mode>in-memory</mode>
</persistence>
- was support for … parameters (as shown below) only added in CE 2020.3 as per this post on it - https://www.hivemq.com/blog/hivemq-ce-2020-3-released/ ?
Yes and no. The parameters you used, are not for HiveMQ CE, see answer 2. HiveMQ CE supports in-memory persistence from version CE 2020.3 onward. The setting needed is described in answer 3 or in the linked documentation.
I wish you a great day
Georg
3 Likes
Side note: If you want to use the file mode you want to use the latest HiveMQ CE version 2020.6 as there we fixed a bug that let to unlimited growth in the publish and retained persistence.
See: Disable LOG-Files in data
Kind regards,
Michael
3 Likes
Issue resolved. Thank you both @michael_w @sauroter very much!