HiveMQ

Too much consumption at single-writer task

#1

Hi, I’ve made some prossure testing on HiveMQ-CE, I found that single-writer-x thread cost so much CPU and MEM.
The server is 8C 32G ubuntu, I made 10k connections, then subscribe 5k topics, then publish to these topics at the rate of 22k QPS with Qos1. It performs well with 10min, but faded after 10min, I found the cpu idle time is almost to 0%, but another broker moquette works well.
Is it that xodus cannot support the QPS?

jatack
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
22896 root 20 0 15.930g 8.464g 26656 R 35.6 26.9 3:31.59 single-writer-8
22891 root 20 0 15.930g 8.464g 26656 R 35.2 26.9 3:29.03 single-writer-3
22901 root 20 0 15.930g 8.464g 26656 R 34.9 26.9 3:29.40 single-writer-1
22894 root 20 0 15.930g 8.464g 26656 R 33.6 26.9 3:30.63 single-writer-6
22900 root 20 0 15.930g 8.464g 26656 R 33.2 26.9 3:30.84 single-writer-1
22888 root 20 0 15.930g 8.464g 26656 R 32.9 26.9 3:30.61 single-writer-0
22892 root 20 0 15.930g 8.464g 26656 R 32.9 26.9 3:30.92 single-writer-4
22893 root 20 0 15.930g 8.464g 26656 R 32.6 26.9 3:30.24 single-writer-5
22899 root 20 0 15.930g 8.464g 26656 R 32.6 26.9 3:30.16 single-writer-1
22890 root 20 0 15.930g 8.464g 26656 R 31.6 26.9 3:29.70 single-writer-2
22902 root 20 0 15.930g 8.464g 26656 R 31.2 26.9 3:30.56 single-writer-1
22889 root 20 0 15.930g 8.464g 26656 R 30.9 26.9 3:30.93 single-writer-1
22898 root 20 0 15.930g 8.464g 26656 R 30.9 26.9 3:30.24 single-writer-1
22903 root 20 0 15.930g 8.464g 26656 R 30.9 26.9 3:28.89 single-writer-1
22897 root 20 0 15.930g 8.464g 26656 R 30.2 26.9 3:30.25 single-writer-9
22895 root 20 0 15.930g 8.464g 26656 R 29.9 26.9 3:30.15 single-writer-7

#2

here are my test result:
10k connections:

connection
connection887×224 8.25 KB

CPU idle time:
1.jpg
1.jpg927×303 12.3 KB

MEM used:
mem
mem882×267 7.76 KB

the network traffic:
2.jpg
2.jpg871×295 27.5 KB

As we can see from the result, the server seems not be able to support this QPS after about 10 mins

#3

Hi @helloc,

Tank you for starting this thread and adding some additional information.
Just to be clear:

As we can see from the result, the server seems not be able to support this QPS after about 10 mins

This may be true for your test setup on your test machine. Increasing the number of cores on your server will also increase the possible throughput.

Can you please add the following information:

  • What is the message size of your messages?
  • What is the topic structure you are using?
  • What’s the ratio of publisher to subscriber?
  • What are the numbers for incoming and outgoing PUBLISH packets, as seen by the broker?
  • Is there any noteworthy output in the files hivemq.log or event.log?

You mentioned:

If not otherwise specified, Moquette uses in-memory persistence:

#*********************************************************************
#The file for the persistent store, if not specified, use just memory
#no physical persistence
#*********************************************************************
#persistent_store ./moquette_store.h2

HiveMQ CE comes with full file persistence. That means that all of the crucial data like: subscriptions, unfinished message flows, payloads or retained messages are persisted to file and can be migrated between servers or allow recovery without data loss in disaster scenarios.

  • Did you compare moquette in file persistence?
  • Did you investigate what kind of data is persisted in moquette and how well it can recovery from a disaster?

Happy New Year!
Florian

#4

Hi @fraschbi,
Thank you for your reply.

About my test:

  • the payload size is 1KB
  • the topic strutcture like “/topic/{instanceId}/{num}”, where {instanceId} is a 6 byte fixed string, and {num} is an increasing number from 0 to 4999.
  • step1. I fistly made 10k connections and put them into a list, step2. then continuously subscribe the topic above with the first 5k clients, step3. then continuously(infinitely) publish to the 5k topics with clients of all this list. so is it that in this case the ratio of pub/sub will increase as the infinite publishing loop goes?
  • the clients just infinitely publish to the broker, so no certain numbers of packets.
  • sorry I didn’t keep the log file.

About Moquette:
I started to read the source code of Moquette after I posted this topic, and I found that Moquette actually persist the data in memory, so my puzzle solved. Data completeness and performance contradict to each other, it’s not easy for us to get both.
By the way, I really appreciate the code of HiveMQ, I think it’s well designed and written by experienced developers, so I just can’t believe it inferior to Moquette in my test result.

Happy new Year!
helloc