ESF-IDF thru A7670 connection to free HiveMQ instance

[ericl]

Hi, I need some pointers to help me out debug a connection issue.

I got a simple setup with and ESP32 and a SimCom A7670 GSM.
I am prototyping using ESP-IDF 5.1.2 and the example tcp_modem_client from the ESP-MODEM component.

What I got tested/validated:

• MQTTconnection to HiveMQ, mosquitto AND eclipeIO test server (non TLS).
• MQTTSconnection (anonymous) to mosquitto and eclipeIO test server.
• MQTTSconnection with MQTT user/pass on flespi ( I had to validate the USER/PASS over TLS).
• connections to my HiveMQ instance using 3rd party MQTTclients (mqttx and mqttspy).

Since I am using a GSM modem I did not try a local install on my own network.

What is not working:

• Connection to my HiveMQ broker using the esp32.

When I trace the VERBOSE output, I can verify that the TLS auth works:

V (6196) sock_dce: 0x3ffba988   39 48 9a b0 13 1b 82 50  75 54 7b be db ea c2 0d  |9H.....PuT{.....|
D (6206) sock_dce: perform_at
V (6206) sock_dce: 0x3ffba958   0a 0d 0a 4f 4b 0d 0a                              |...OK..|
D (6206) mqtt_client: Transport connected to mqtts://xxx.s2.eu.hivemq.cloud:8883
D (6216) sock_dce: perform_at
D (6226) mqtt_client: Sending MQTT CONNECT message, type: 1, id: 0000
D (6226) sock_commands: response 

OK

Then it always fails on the MQTT authentication:

V (6236) dce: 0x3ffbac08   17 03 03 00 48 00 00 00  00 00 00 00 01 43 cf ce  |....H........C..|
V (6246) dce: 0x3ffbac18   b2 6a 84 aa 87 09 99 0b  d6 ee 28 98 b9 a7 e1 c9  |.j........(.....|
V (6256) dce: 0x3ffbac28   69 7a b0 92 a4 f4 b4 e4  41 4b 0d aa 1b 34 55 c5  |iz......AK...4U.|
V (6266) dce: 0x3ffbac38   f8 b0 90 ac 40 50 a2 e9  c2 a5 d3 2d d2 c4 34 52  |....@P.....-..4R|
V (6276) dce: 0x3ffbac48   35 fc 53 0e 22 62 45 a7  27 17 d0 61 2b           |5.S."bE.'..a+|
V (6286) sock_dce: 0x3ffba958   0d 0a 3e                                          |..>|
D (6286) sock_dce: perform_at
D (6296) sock_commands: response 
>
V (6306) sock_dce: 0x3ffba958   0d 0a 4f 4b 0d 0a 0d 0a  2b 43 49 50 53 45 4e 44  |..OK....+CIPSEND|
V (6306) sock_dce: 0x3ffba968   3a 20 30 2c 37 37 2c 37  37 0d 0a                 |: 0,77,77..|
D (6316) sock_dce: perform_at
D (6316) sock_commands: response 
OK

+CIPSEND: 0,77,77

V (6646) sock_dce: 0x3ffba958   0d 0a 2b 43 49 50 52 58  47 45 54 3a 20 31 2c 30  |..+CIPRXGET: 1,0|
V (6646) sock_dce: 0x3ffba968   0d 0a                                             |..|
D (6656) sock_dce: perform_at
D (6656) sock_commands: response 
+CIPRXGET: 1,0

D (6656) sock_commands: Got data on modem!
D (6656) dce: select read: modem data available
V (6666) sock_dce: 0x3ffba958   0d 0a 2b 49 50 43 4c 4f  53 45 3a 20 30 2c 31 0d  |..+IPCLOSE: 0,1.|
D (6666) dce: select read: modem data available
V (6676) sock_dce: 0x3ffba968   0a                                                |.|
D (6686) sock_dce: perform_at
E (6696) mqtt_client: esp_mqtt_handle_transport_read_error: transport_read() error: errno=0
V (6696) sock_dce: 0x3ffba958   0d 0a 2b 43 49 50 52 58  47 45 54 3a 20 32 2c 30  |..+CIPRXGET: 2,0|
D (6706) event: running post MQTT_EVENTS:0 with handler 0x400d9358 and context 0x3ffbc5c4 on loop 0x3ffbc4c4
0x400d9358: mqtt_event_handler(void*, char const*, long, void*) at /home/dweezil/VSCode/modem_tcp_client/main/modem_client.cpp:56

V (6716) sock_dce: 0x3ffba968   2c 35 2c 35 39 0d 0a 17  03 03 00 1c 0d 0a 0d 0a  |,5,59...........|
D (6726) modem_client: Event dispatched from event loop base=MQTT_EVENTS, event_id=0
V (6736) sock_dce: 0x3ffba978   4f 4b 0d 0a                                       |OK..|
I (6736) modem_client: MQTT_EVENT_ERROR
D (6746) sock_dce: perform_at
I (6756) modem_client: Last error code reported from esp-tls: 0x103
D (6756) sock_commands: response 
+CIPRXGET: 2,0,5,59

I (6766) modem_client: Last tls stack error number: 0x0
I (6776) modem_client: Last captured errno : -1 ()
E (6776) mqtt_client: esp_mqtt_connect: mqtt_message_receive() returned -1
E (6786) mqtt_client: MQTT connect failed
V (6796) sock_commands: tcp_close
V (6796) command_lib: generic_command
D (6796) command_lib: generic_command command AT+CIPCLOSE=0

Now, my question is: Does the problem looks like its from the MQTT auth. handshake, does the MQTT config in the ESP needs particular config flags I am missing, or is the issue in the ESP-IDF API/code.

IMO I would validate first that my MQTT config is correct, to enforce my gut feeling that the issue is in the ESP-IDF code…

Any comment is welcome, I’ve been stomped for a few days now on this issue.

Cheers.

[Diego]

Hello @ericl

Welcome to HiveMQ Community! I’m not familiar with ESP-MODEM component but I can try to help you.

To be able to connect to HiveMQ Cloud from those devices you should provide the Let’s Encrypt certificate, can you point me on how the SDK handles the TLS layer?

Kind regards,
Diego from HiveMQ Team

[ericl]

Hi Diego, thanks for the warm welcome.

The esp-modem is only a component from espressif (esp-protocols/components/esp_modem/README.md at master · espressif/esp-protocols · GitHub), from which I took the modem_tcp_client example: esp-protocols/components/esp_modem/examples/modem_tcp_client at master · espressif/esp-protocols · GitHub.

This example provide a quick MQTT client, with a GSM tcp_transport library and a wrapper over MBED_tls.

So, when I configured the code, I added the TLS transport layer, and added a copy of the let’s encrypt certificate (isrgrootx1.pem) loaded in the binary:

in modem_client.cpp
   ...
   esp_mqtt_client_config_t mqtt_config = {};
   mqtt_config.broker.address.port = BROKER_PORT;
   mqtt_config.session.message_retransmit_timeout = 10000;

#ifndef CONFIG_EXAMPLE_CUSTOM_TCP_TRANSPORT
   mqtt_config.broker.address.uri = "mqtts://127.0.0.1";
   dce->start_listening(BROKER_PORT);
#else

   mqtt_config.broker.address.uri = "mqtts://" BROKER_URL;
   mqtt_config.credentials.username = USERNAME;
   mqtt_config.credentials.authentication.password = PASSWORD;

   mqtt_config.broker.verification.certificate = (const char *)isrgrootx1_pem_start;

   esp_transport_handle_t at = esp_transport_at_init(dce.get());
   esp_transport_handle_t ssl = esp_transport_tls_init(at);

   mqtt_config.network.transport = ssl;
#endif
   esp_mqtt_client_handle_t mqtt_client = esp_mqtt_client_init(&mqtt_config);
   esp_mqtt_client_register_event(mqtt_client, static_cast<esp_mqtt_event_id_t>(ESP_EVENT_ANY_ID), mqtt_event_handler, NULL);
   esp_mqtt_client_start(mqtt_client);
   ESP_LOGI(TAG, "%s", mqtt_config.broker.address.uri);
   ...

Loading the certificate in the binary:

the CMakeLists.txt
cmake_minimum_required(VERSION 3.8)

include($ENV{IDF_PATH}/tools/cmake/project.cmake)
project(modem_tcp_client)

target_add_binary_data(modem_tcp_client.elf "main/isrgrootx1.pem" TEXT)

and the code to acces it in modem_client.cpp

#if CONFIG_BROKER_CERTIFICATE_OVERRIDDEN == 1
static const uint8_t isrgrootx1_pem_start[]  = "-----BEGIN CERTIFICATE-----\n" CONFIG_BROKER_CERTIFICATE_OVERRIDE "\n-----END CERTIFICATE-----";
#else
extern const uint8_t isrgrootx1_pem_start[]   asm("_binary_isrgrootx1_pem_start");
#endif
extern const uint8_t isrgrootx1_pem_end[]   asm("_binary_isrgrootx1_pem_end");

If I use the exact same code with the test.mosquitto.org or flespi certificate, I can connect all right.

Even with HIveMQ, it appears to me the TCP_transport connection over TLS is working. I can use the verbose mode and see the certificates going thru:

Then from there all semms good, up to line 636:

Then it send the user/pass for the conneciton and then at line 668 something closes the connection.
I am wondering what might happen on the broker side, but my gut tells me it might be on the esp side. Either a timeout or another issue.

If I could see the logs for the broker, I could tell if the authentication:

1. Has been received,
2. Is accepted or not.

In the meantime, I am tracing the code as best I can (the JTAG is slow on an ESP32), and try to isolate the code which triggers the IPCLOSE. Any help to understand the broker side is welcome.

Cheers.

[Diego]

Hello @ericl

You’re welcome and thanks for the detailed information. HiveMQ Cloud Serverless offer requires the clients to be SNI capable when establishing the TLS connection. This is necessary to differentiate between individual tenants.

Do you know if the MBED TLS implementation on this SDK supports SNI? Quick looking into the .h library used on the wrapper it seems it needs additional configuration to be set in SDK mbedtls/include/mbedtls/ssl.h at development · Mbed-TLS/mbedtls · GitHub

Also, I see in the logs provided a cluster URL yadiyadiyada.s2.eu.hivemq.cloud, is that correct, I’m confused here.

Kind regards,
Diego from HiveMQ Team

[ericl]

Hello Diego, the yadiyadiyada is only to obfuscate my instance (dunno if its better).

Your help is really appreciated. I was looking around last night and figured out the SNI feature might be the culprit. I am still new to debugging TLS connections and Iǘe got a lot to learn.

I will be following your suggestion and make sure I get the SNI host setup properly in my code.

I will get back to you shortly.

Have a great day!

[ericl]

Diego, can you confirm I should use the string “ISRG Root X1” as the SNI common name?

Cheers!

dweezil@Homer:~/VSCode/modem_tcp_client/main$ openssl x509 -in isrgrootx1.pem -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            82:10:cf:b0:d2:40:e3:59:44:63:e0:bb:63:82:8b:00
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, O = Internet Security Research Group, CN = ISRG Root X1
        Validity
            Not Before: Jun  4 11:04:38 2015 GMT
            Not After : Jun  4 11:04:38 2035 GMT
        Subject: C = US, O = Internet Security Research Group, CN = ISRG Root X1
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (4096 bit)
                Modulus:
                    00:ad:e8:24:73:f4:14:37:f3:9b:9e:2b:57:28:1c:
                    87:be:dc:b7:df:38:90:8c:6e:3c:e6:57:a0:78:f7:
                    75:c2:a2:fe:f5:6a:6e:f6:00:4f:28:db:de:68:86:
                    6c:44:93:b6:b1:63:fd:14:12:6b:bf:1f:d2:ea:31:
                    9b:21:7e:d1:33:3c:ba:48:f5:dd:79:df:b3:b8:ff:
                    12:f1:21:9a:4b:c1:8a:86:71:69:4a:66:66:6c:8f:

[Diego]

@ericl

Server Name Indication (SNI) verifies which hostname it’s attempting to connect to at the start of the handshaking process. For HiveMQ Cloud Serverless the hostname is added to the “Client Hello” TLS packet, you can verify it by examining the packet as illustrated in this post. I believe here you should use your cluster URL as SNI Common Name.

Kind regards,
Diego from HiveMQ Team

[ericl]

Good morning.

I’ve been studying the code and found out the current ESP-mqtt client code does not support AT transport (custom transport) in the code that activates the SNI…

I’ve have to use a workaround or work directly with the modem (AT commands).
Thanks a lot for your help .

Cheers

[frarugi87]

Hello @ericl

Were you able to complete the connection to HiveMQ through ESP32 and a A7670?

[frarugi87]

If you need some help with the AT commands, I just published a sum up of what I had to use to publish with the A7670:

Publishing with SIMcom A7670

Hi all Since I lost some time to understand how the module works and I wasn’t able to find resources to have a working example (apart from arduino libraries which handle the SSL communication in software), here is the list of commands that allows me to publish on a private (serverless) HiveMQ instance with the A7670 module. First of all you need to upload the root_ca certificate on the module: AT+CCERTDOWN="isrgrootx1.pem",1939 -----BEGIN CERTIFICATE----- MIIFazCCA1OgAwIBAgIRAIIQz7DSQONZRGPgu…