I have crash in my broker

1

please i need help to understand what is the problem here

HIVEMQ_HOME: /opt/hivemq

JAVA_OPTS: -Djava.net.preferIPv4Stack=true --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.nio=ALL-UNNAMED --add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-opens jdk.management/com.sun.management.internal=ALL-UNNAMED --add-exports java.base/jdk.internal.misc=ALL-UNNAMED -Djava.security.egd=file:/dev/./urandom -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9011 -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -XX:+CrashOnOutOfMemoryError -XX:+HeapDumpOnOutOfMemoryError

JAVA_VERSION: 11

2024-07-29 08:40:15,535 INFO - Starting HiveMQ Community Edition Server
2024-07-29 08:40:15,537 INFO - HiveMQ version: 2022.2-SNAPSHOT
2024-07-29 08:40:15,538 INFO - HiveMQ home directory: /opt/hivemq
2024-07-29 08:40:15,539 INFO - Log Configuration was overridden by /opt/hivemq/conf/logback.xml
2024-07-29 08:40:16,080 INFO - This HiveMQ ID is ID01Q
2024-07-29 08:40:20,610 ERROR - Exception on Log recovery. Approved high address = -1145652621
jetbrains.exodus.ExodusException: Can’t read loggable fully, address = 1704042, file = 00000000000.xd
at jetbrains.exodus.log.Log.(Log.java:151)
at jetbrains.exodus.env.Environments.newLogInstance(Environments.java:137)
at jetbrains.exodus.env.Environments.newLogInstance(Environments.java:115)
at jetbrains.exodus.env.Environments.newContextualInstance(Environments.java:99)
at com.hivemq.persistence.local.xodus.XodusLocalPersistence.lambda$start$2(XodusLocalPersistence.java:164)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2024-07-29 08:40:22,694 ERROR - Exception on Log recovery. Approved high address = -947695375
jetbrains.exodus.ExodusException: Can’t read loggable fully, address = 1687552, file = 00000000000.xd
at jetbrains.exodus.log.Log.(Log.java:151)
at jetbrains.exodus.env.Environments.newLogInstance(Environments.java:137)
at jetbrains.exodus.env.Environments.newLogInstance(Environments.java:115)
at jetbrains.exodus.env.Environments.newContextualInstance(Environments.java:99)
at com.hivemq.persistence.local.xodus.XodusLocalPersistence.lambda$start$2(XodusLocalPersistence.java:164)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)

2

I ran into the same issue today. As the log messages does not say exactly which of the “00000000000.xd” is the problem (there are multiple files with the same name in different subdirectories of ./hivemq/data/persistence) I made a backup directory outside of the current “hivemq” path and copied all subdirectories under the persistence directory into the backup directory just created.

Then I jumped into all 5 main-subdirectory until I reached the level where the 64 subdirectories showed up and deleted those 64 directories including all the files.
After that I was able to restart HiveMQ and it was not complaining.

This might not be the smartest idea for a production system but in my case it’s just a test system so I didn’t care!

It would be great if the log would offer more details which is actually the cause of the issue (client_queue, client_session_store, client_session_subscription, publish_payload_store or retained_messages).

I hope this helps.
If anybody as more “secure” ways to recover from this issue please add your knowledge here!

Many thanks!

1 Like
3

Hi @magog

Greetings! We’re thrilled to have you here, showing interest in MQTT and the HiveMQ broker. Welcome to the HiveMQ Community! We’re always happy to see new users.

About your issue: Thanks for sharing your experience. Could you let us know which version of HiveMQ CE you’re using? Is it the latest 2024.7 release, or an earlier version? Also, how did you encounter this issue? Was it right after installing the HiveMQ broker, or did it happen after running it for a while or performing specific tests? It would be great to understand the context better.
Could you also let us know how much space was available on the /hivemq/data volume when you encountered this issue? Additionally, do you recall how much of that space was occupied at the time? This information might help in understanding what led to the problem.

Best,
Dasha from the HiveMQ Team

4

I deleted the data folder and that works for me I do not know why this happened at the first place but now everything is working properly