Original Exception:
org.rocksdb.RocksDBException: While mkdir if missing: /opt/hivemq/data/persistence/publish_payload_store/040500_R/publish_payload_store_56: No such file or directory
at org.rocksdb.RocksDB.open(Native Method)
at org.rocksdb.RocksDB.open(RocksDB.java:249)
at com.hivemq.persistence.local.rocksdb.RocksDBLocalPersistence.lambda$start$0(RocksDBLocalPersistence.java:171) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Hi Rahul,
Thanks for reporting this error. We would like to attempt to reproduce it. Which steps do you take to reproduce this error?
Thanks,
Dasha from HiveMQ Team
Simply, my publisher is bit faster than the subscriber.
One more issue is coming :
While appending to file: /opt/hivemq/data/persistence/publish_payload_store/040500_R/publish_payload_store_5/000009.log: No space left on device
at org.rocksdb.RocksDB.put(Native Method)
at org.rocksdb.RocksDB.put(RocksDB.java:716)
at com.hivemq.persistence.payload.PublishPayloadRocksDBLocalPersistence.put(PublishPayloadRocksDBLocalPersistence.java:131)
at com.hivemq.persistence.payload.PublishPayloadPersistenceImpl.lambda$add$0(PublishPayloadPersistenceImpl.java:119)
at com.hivemq.persistence.payload.PublishPayloadPersistenceImpl.accessBucket(PublishPayloadPersistenceImpl.java:240)
at com.hivemq.persistence.payload.PublishPayloadPersistenceImpl.add(PublishPayloadPersistenceImpl.java:108)
at com.hivemq.mqtt.services.PublishDistributorImpl.createPublish(PublishDistributorImpl.java:180)
at com.hivemq.mqtt.services.PublishDistributorImpl.queuePublish(PublishDistributorImpl.java:158)
at com.hivemq.mqtt.services.PublishDistributorImpl.handlePublish(PublishDistributorImpl.java:148)
at com.hivemq.mqtt.services.PublishDistributorImpl.sendMessageToSubscriber(PublishDistributorImpl.java:123)
at com.hivemq.mqtt.services.PublishDistributorImpl.distributeToNonSharedSubscribers(PublishDistributorImpl.java:89)
at jdk.internal.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:50)
at com.sun.proxy.$Proxy42.distributeToNonSharedSubscribers(Unknown Source)
at com.hivemq.mqtt.services.InternalPublishServiceImpl.deliverPublish(InternalPublishServiceImpl.java:175)
at com.hivemq.mqtt.services.InternalPublishServiceImpl.handlePublish(InternalPublishServiceImpl.java:150)
at com.hivemq.mqtt.services.InternalPublishServiceImpl.publish(InternalPublishServiceImpl.java:83)
at com.hivemq.mqtt.handler.publish.IncomingPublishService.publishMessage(IncomingPublishService.java:224)
at com.hivemq.mqtt.handler.publish.IncomingPublishService.authorizePublish(IncomingPublishService.java:157)
at com.hivemq.mqtt.handler.publish.IncomingPublishService.processPublish(IncomingPublishService.java:125)
at com.hivemq.extensions.handler.PluginAuthorizerServiceImpl.authorizePublish(PluginAuthorizerServiceImpl.java:121)
Hey Rahul,
have you checked if your disk has space available, so the broker can write its persistent data to it?
Kind regards,
Finn
Itâs a docker broker image we are running & from disk space perspective we have lot of space free.
Configuration on which we are getting these kind of error
No of device :100
Published message : 400k (total time to send these like 1500 seconds)
received message : 234918
I have repeated these configuration many times.
Hi Rahul,
Thanks for sharing this info! If you could also share your Docker file, we would be happy to take further look at your issue.
Kind regards,
Dasha from HiveMQ Team
Hi Diara,
we have configured community version. Please fetch docker image from docker hub.
Tag : 2021.1 (this is the one which we are using )
https://hub.docker.com/r/hivemq/hivemq-ce/tags
Thanks & Regards,
Rahul
Hi Rahul,
So, you are running a HiveMQ CE from the image in the Docker:
docker run -it -p 8000:8000 -p 1883:1883 hivemq/hivemq-ce
Do you set any additional parameters? Do you make changes to the deployment?
Importantly, what test are you running before you are getting the error, that you have reported to us initially?
Thanks,
Dasha from HiveMQ Team
Hi Daria,
Yes, we are running the same command. We have not add any additional parameter.
In above conversation, I have added configuration which we are continuously repeating. Initially, we were having âOriginal Exception:
org.rocksdb.RocksDBException: While mkdir if missing: /opt/hivemq/data/persistence/publish_payload_store/040500_R/publish_payload_store_56: No such file or directoryâ this issue and now we are struggling with âWhile appending to file: /opt/hivemq/data/persistence/publish_payload_store/040500_R/publish_payload_store_5/000009.log: No space left on deviceâ.
Regards,
Rahul
Hi Rahul,
Looking at the âConfiguration on which we are getting these kind of errorâ that you have mentioned above, I figure, you must be running a script, that is connecting 100 clients to the HiveMQ broker, that are publishing 400k messages and there must be clients that are subscribed and consuming the messages. From the description I can also guess, that you are sending 400k messages and 235k messages are consumed before the error âNo space left on deviceâ happens. Please confirm whether I understand your scenario correctly.
Please let me know, which tool(s) you are using to run your scenario, are you using HiveMQ Swarm or some other tool(s)? Can you share your script with us, so we could run it and reproduce your issue?
Thanks,
Dasha from HiveMQ Team
Hi Daria,
You have got almost everything. I just wanted to add we have published 400k messages by 100 client and consumed 235k messages but here we donât know the exact reason about all the rest messages which are not consumed either those messages are being dropped or no space left exception. We have been observing both issues from last couple of days.
In term of the tool, we have created c# based script/publisher which is generating that records.
Thanks & Regards,
Rahul
Great, Rahul!
Now, for us to reproduce the issue it would be helpful, if you have also shared your testing script with us. Also, if there are any configuration, settings that you set (for example, changes in the config.xml or logback.xml, persistent volumes, docker settings) please also share those with us.
Thanks,
Dasha from HiveMQ Team
Hi Daria,
Thanks for reaching out to us !
In term of script, you can use any script for publishing a message like Jmeter or any client tool. There should be one thing that publisher should be faster than the subscriber and that should also run for longer period.
Docker image setting as same as it is available over the docker hub.
Thanks !
Hi Rahul,
Could you please provide an example for me, how you are using the Jmeter for this test?
Thanks,
Dasha from HiveMQ Team
Hi Rahul,
I have discussed your issue with our engineers and they suggested to use the docker system prune command next time you run into the issue with space. Please let me know if that helps.
Thank you,
Kind regards,
Dasha from HiveMQ Team
1 Like