I am not able to consume the records from dropped topic and I have enabled the configuration in the config.xml file.
Can let me know how to consume them?
Also, can we get the dropped messages from the dropped messages topic to Kafka using Kafka-HiveMQ-extension and see that data in Kafka topic?
I have attached my config file for reference and currently I am using HiveMQ version 4.8.1
I have also attached the screenshot of MQTT fx that I used to subscribe and consume data.
Hi @Daria_H ,
** How do you simulate the scenario where a message is dropped (how do you “generate” dropped messages)?**
I have decreased the incoming message size in kafka and sent large volume of data from MQTT fx to hiveMQ
** Do you see in the HiveMQ Control Center that you have messages dropped?**
Yes, I can see that the message has been dropped in the Control center and got the notification for it.
Do you see from the HiveMQ logs that you have messages dropped?
Yes I can see a WARN log stating that the messages has been dropped.
so I tried it with your config and my client that subscribed to $dropped/# got the dropped messages. I produced the dropped messages by having an offline client that subscribes to topic "test:
mqtt sub -t test -q 1 -se 100000 -i sleepyClient
Note: disconnect client so he is offline.
So now I add subscriber for dropped topic:
mqtt sub -t '$dropped/#'
Then I publish three messages:
mqtt pub -t test -q 1 -m "oh no"
mqtt pub -t test -q 1 -m "oh no2"
mqtt pub -t test -q 1 -m "oh no3"
And my dropped subscriber gets “oh no2” and “oh no3”, as “oh no” is the one message still queued for the offline client.
I’m still unclear though how you produce dropped messages, I would love to see the Control Center
notification and the WARN message.
Regarding:
Also, can we get the dropped messages from the dropped messages topic to Kafka using Kafka-HiveMQ-extension and see that data in Kafka topic?
Thanks for the response and it worked for me the way that you have mentioned. After I have tried the ways you have given me, I have few more doubts.
Whether the dropped messages will sent to dropped topic only if the messages are dropped for the below reasons alone?
And if so, how can I configure for messages dropped from Kafka side so that will also be sent to dropped topic ?
I got the following notification in HiveMQ control center if a message was dropped from Kafka Side:
TLDR: What you want to archive is currently not possible.
The dropped topic only works for MQTT messages and not for Kafka records.
That is the reason why you see the notification for dropped Kafka records but don’t get an MQTT message to dropped topic. There is just no way for a consumer in an extension to let HiveMQ know that MQTT message x can be considered dropped.
@michael_w , Thanks for the Clarification. Can you guide me through on how we can recreate the following issues in the local cluster:
And also, what are all the kind of metrices should be considered for monitoring HiveMQ cluster and broker and what kind of major errors will occur in Production?
And also, what are all the kind of metrices should be considered for monitoring HiveMQ cluster
For monitoring you can either look for this one metric: com.hivemq.messages.dropped.count
This one increases each time, when for whatever reason, an MQTT message is dropped
There are also metrics for each specific reason you can find them here: Monitoring :: HiveMQ Documentation
They have the prefix “com.hivemq.messages.dropped.”
what kind of major errors will occur in Production
Not sure I understand the question but I try to answer it:
From experience the queue full metric will occur the most, this could indicate that you have slow consuming subscribers or your expiry for publishes is to big therefore have a lot of old messages in the system.
Can you guide me through on recreating these issue like the way you informed for “Client Message Queue Full” that you guided me in the previous loop.
My question is that, what sort of errors will occur in production and what kind of errors will occur frequently in production that has to be monitored for ?
I am able to see why these issues are created but not able to re-create these issues back again.
Ah now I get it. I’ll only explain those that are easy to reproduce:
Client queue full:
have an offline subscriber client (with qos > 0) and publish more publishes (with qos > 0) as is the configured queue size, basically the example I gave you above. Of course it’s faster to reproduce with a small queue max size:
Another way is to simulate a subscriber that is slow in acking messages or doesn’t ack for PUBLISH messages at all (need qos > 0)
Extension prevented:
Use for example the PublishInboundInterceptor (or PublishOutboundInterceptor) and call for example:
@Override
public void onInboundPublish(final @NotNull PublishInboundInput publishInboundInput, final @NotNull PublishInboundOutput publishInboundOutput) {
publishInboundOutput.preventPublishDelivery();
}
Maximum packet size exceeded:
Well sent a publish packet that is over the configured packet limit use an MQTT 3 client as most MQTT 5 clients prevent the sending of packets of server limit (as is mandatory per MQTT 5 specification)
My question is that, what sort of errors will occur in production and what kind of errors will occur frequently in production that has to be monitored for ?
As this is basically another question please open a new thread for it.
use an MQTT 3 client as most MQTT 5 clients prevent the sending of packets of server limit (as is mandatory per MQTT 5 specification)