Getting Started: Raspberry Pi Pico W

IanW6374 · July 24, 2022, 4:05pm

Hi - I am trying to use the HiveMQ Cloud to start getting familiar with MQTT using a Raspberry Pico W as the client device. I was using your recent blog about using Raspberry Pi Pico / Airlift WiFi as a reference although I am using microPython instead of circuitPython.

I can use the HiveMQ public broker service without any issues but when I try to connect to the HiveMQ cloud I get the error message “MQTT connect failed: extra keyword arguments given” which seems to be related to the ssl_params

Snippet of code:-
from umqtt.simple2 import MQTTClient

mqtt_client = MQTTClient(
client_id=secrets[‘mqtt_clientid’],
server=secrets[“mqtt_server”],
port=secrets[“mqtt_port”],
user=secrets[“mqtt_username”],
password=secrets[“mqtt_key”],
ssl=True,
ssl_params={‘keyfile’:‘/server.key’,‘certfile’:‘/server.pem’}

)
mqtt_client.connect()

Any help would be appreciated

Daria_H · August 4, 2022, 12:22pm

Hello @IanW6374 ,

It is nice to see your interest in MQTT and HiveMQ, please welcome to our community!

The error extra keyword arguments given refers to a method from your SSL library. Can you see the error in the traceback, something like this?

Traceback (most recent call last):                                                                                       
TypeError: extra keyword arguments given

Did you try running your code without specifying ssl_params? The issue is that SSL parameters that the library supports might differ between implementations:

Depending on the underlying module implementation in a particular MicroPython port, some or all keyword arguments above may be not supported.
[Reference: ssl – SSL/TLS module — MicroPython 1.19.1 documentation]

I hope this helps,
Dasha from HiveMQ team

FrankD · September 2, 2022, 1:37pm

Hi, trying to do the same thing…

As WiFi is not supported (yet) by CircuitPython, I’m now using MicroPython with from umqtt.simple import MQTTClient (sources: micropython-lib/simple.py at master · micropython/micropython-lib · GitHub).

Two problems:

With ssl=True I get MQTTException: 5
With ssl=False I get IndexError: bytes index out of range in simple.py, line 97

Was anyone else able to connect to HiveMQ Cloud with this or other MicroPython library?
Can the cause of this exception 5 (connection refused) be seen in the logs on HiveMQ Cloud side?

Thanks
Frank

IanW6374 · September 6, 2022, 10:31am

@Daria_H Thanks for your reply

I have fixed the “TypeError: extra keyword arguments given” issue, however, as per FrankD’s post I am now getting a MQTTException: 5 error. Is there any logging on the HIveMQ side that is available to help troubleshoot??

IanW6374 · September 7, 2022, 4:35am

FYI: The certificate verification within the ussl micro-python module is still being developed and hence some of the confusion.

I have attached some code which can be used to test basic ssl functionality with the “sslparams” - All being well you should get “b’HTTP/1.1 200 OK\r\n’” as a response

import usocket as socket
import ussl as ssl

ca_certificate_path = "/certs/catrust-micropython.der"

print('Loading CA Certificate')
with open(ca_certificate_path, 'rb') as f:
    cacert = f.read()
f.close()
print('Obtained CA Certificate')

hostname = 'micropython.org'
port_no = 443
sslparams = {'server_side':False, 'key':None, 'cert':None,'cert_reqs':ssl.CERT_REQUIRED, 'cadata':cacert}

def main(use_stream=True):
    
    sock = socket.socket()
    addr = socket.getaddrinfo(hostname, port_no)[0][-1]
    sock.connect(addr)
    sock = ssl.wrap_socket(sock, **sslparams)
    
    sock.write(b"GET / HTTP/1.0\r\n\r\n")
    print(sock.read(17))
    sock.close()
    

main()

Prolixteriat · September 7, 2022, 11:12am

Hi - I’m also struggling to crack this particular nut …

I’ve tried multiple variations on this theme, but basically the code is as follows:

from umqtt.simple import MQTTClient

with open("/isrgrootx1.pem", 'r') as f:
    cert = f.read()

sslparams = { 'cert': cert }

client = MQTTClient(client_id='temp_log',
                    server='<SERVER>.s1.eu.hivemq.cloud',
                    port=8883,
                    user='<USER>',
                    password='<PASSWORD>',
                    keepalive=3600,
                    ssl=True,
                    ssl_params=sslparams)
client.connect()

Where isrgrootx1.pem has been downloaded from
https://letsencrypt.org/certificates/

I get the error:

  File "/lib/umqtt/simple.py", line 99, in connect
  MQTTException: 5

I’ve tested subscribing and publishing using Mosquitto CLI using the same creds and cert, so I know they are valid.

I’m now out of ideas and so and suggestions or guidance would be much appreciated.

FrankD · September 7, 2022, 11:34am

We’ll probable need a certificate allowed/provided by HiveMQ. Hopefully, they can join this discussion and help us out.

Daria_H · September 8, 2022, 11:27am

Regarding the certificate please refer to this community FAQ: Frequently Asked Questions

I hope this helps,
Dasha from HiveMQ team

FrankD · September 8, 2022, 12:31pm

Thanks Daria, but unfortunately with this certificate still has the same error…

# Connect to HiveMQ
print('Loading CA Certificate')
with open("/isrgrootx1.pem", 'r') as f:
    cert = f.read()
sslparams = {'cert': cert}
print('Loaded certificate: ' + cert[0:55] + '...' + cert[-55:])

print("Connecting to " + secrets["broker"] + " as user " + secrets["mqtt_username"])
client = MQTTClient(client_id="picow",
                    server=secrets["broker"],
                    port=secrets["port"],
                    user=secrets["mqtt_username"],
                    password=secrets["mqtt_key"],
                    keepalive=3600,
                    ssl=True,
                    ssl_params=sslparams)
client.connect()
print('Connected to %s MQTT Broker'%(mqtt_server))

Result, as you can see with the content of the certificate:

Loading CA Certificate
Loaded certificate: -----BEGIN CERTIFICATE-----
MIIFazCCA1OgAwIBAgIRAIIQz7D...Q99b21/+jh5Xos1AnX5iItreGCc=
-----END CERTIFICATE-----
Connecting to f250f4960aed4d5da670e2248e84ff30.s1.eu.hivemq.cloud as user picow-user
b' \x02\x00\x05'
Traceback (most recent call last):
  File "<stdin>", line 74, in <module>
  File "/lib/umqtt/simple.py", line 100, in connect
MQTTException: 5

Having some kind of log or error output in HiveMQ Cloud would really be helpful in investigating these kind of cases…

FrankD · September 8, 2022, 12:51pm

As every developer I’m happy when I see another error message

When adding this:

import ussl
...
ssl_params={'cert_reqs': ussl.CERT_REQUIRED, 'cert': cert}

Another error is thrown:

OSError: (-30336, 'MBEDTLS_ERR_SSL_CA_CHAIN_REQUIRED')

And when further extending the ssl_params to

ssl_params={'cert_reqs': ussl.CERT_REQUIRED, 'cert': cert, 'server_side': True}

It takes a lot longer until (± 60", so looks like some configured timeout) another error is thrown:

OSError: (-29312, 'MBEDTLS_ERR_SSL_CONN_EOF')

This last error apparently means: " the underlying TCP socket was closed (returned an EOF - End Of File - result). It looks like the initial TCP connection was established (“handshake in progress…” line), and then the server closed the socket during the handshake for some reason (maybe something with the parameters offered by the client?). "

IanW6374 · September 8, 2022, 1:11pm

You need the CA certificate which uses the “cacert” rather than “cert” which is used for server side.

See my ssl example code above

FrankD · September 8, 2022, 1:30pm

thanks Ian, can you explain which certificate this is or where to get it?

Or you just mean to use the HiveMQ certificate as cadata like this?

sslparams = {'server_side': False, 'key': None, 'cert': None, 'cert_reqs': ussl.CERT_REQUIRED, 'cadata': cert}
client = MQTTClient(client_id="picow",
                    server=secrets["broker"],
                    port=secrets["port"],
                    user=secrets["mqtt_username"],
                    password=secrets["mqtt_key"],
                    keepalive=3600,
                    ssl=True,
                    ssl_params=sslparams)

unfortunately also results in another error…

Loading HiveMQ Certificate
Loaded HiveMQ certificate: -----BEGIN CERTIFICATE-----
MIIFazCCA1OgAwIBAgIRAIIQz7D...99b21/+jh5Xos1AnX5iItreGCc=
-----END CERTIFICATE-----

Connecting to f250f4960aed4d5da670e2248e84ff30.s1.eu.hivemq.cloud as user picow-user
Traceback (most recent call last):
  File "<stdin>", line 95, in <module>
  File "/lib/umqtt/simple.py", line 61, in connect
ValueError: invalid cert

IanW6374 · September 8, 2022, 1:48pm

Use openssl:
Get certificate

openssl s_client -connect serveraddress.hivemq.cloud:8883 -showcerts < /dev/null 2> /dev/null | sed -n '/BEGIN/,/END/p' > hivemq-com-chain.pem

****Edit the PEM file to remove host certificate ****

****Convert from PEM to DER ****
openssl x509 -in hivemq-com-chain.pem -out hivemq-com-chain.der -outform DER

FrankD · September 9, 2022, 7:34am

Thanks Ian, those commands worked, but I tried with all possible combinations (I think) with or without the isrgrootx1.pem and any of the three certificates in hivemq-com-chain.der but still end up with either connection error 5 or OSError: (-9984, 'MBEDTLS_ERR_X509_CERT_VERIFY_FAILED').

Having no logs at all in HiveMQ Cloud to follow connection attempts and see what is going on to better understand connection issues, is in my opinion a major missing function that would prevent many issues here in the forum.

Prolixteriat · September 9, 2022, 9:03am

Frank - I’m having the same experiences as you, having tried the same things.
Ian - If it’s working for you, would you mind sharing your complete (sanitised) code as an example we could use, please?

As an aside, I’m able to use the Mosquitto test server in authenticated, encrypted mode without any problems - so it appears to me to be something specific to HiveMQ rather than the UMQTT library or my use of it.

IanW6374 · September 9, 2022, 10:08am

I cannot get the HiveMQ MQTT connection to work as I am getting the “MQTTException: 5”, however, I can make a succession TLS connection to the server. I am trying to debug whether it could be an issue with umqtt.simple or ussl.

It is important that you run the latest nightly microPython build as the certificate checking feature is still under development - I am running “rp2-pico-w-20220906-unstable-v1.19.1-375-ge90b85cc9.uf2”

*****
Connect to WiFi Code
*****

ca_certificate_path = "/certs/catrust-micropython.der"
print('Loading CA Certificate')
with open(ca_certificate_path, 'rb') as f:
    cacert = f.read()
print('Obtained CA Certificate')


hostname = 'serverid.s1.eu.hivemq.cloud'
clientid = 'PicoW'
user_name = 'User1'
passw = 'Password1'
port_no = 8883
sslparams = {'server_side':False, 'key':None, 'cert':None, 'cadata':cacert, 'cert_reqs':ssl.CERT_REQUIRED}


client = MQTTClient(client_id=clientid, server=hostname, port=port_no, user=user_name, password=passw, keepalive=3600, ssl=True, ssl_params=sslparams)
client.connect()

IanW6374 · September 9, 2022, 12:00pm

I believe the “MQTTException: 5”. error is caused by an issue with the ussl / usocket module for 2 reasons:-

I have used extensively the same code in Python to send MQTT CONNECT message and I get CONNECT ACK back from HiveMQ
Using the raw MQTT CONNECT data from test 1 on MicroPython I get the same “MQTTException: 5” error message whereas using the same raw MQTT CONNECT in Python I get the “CONNECT ACK”

IanW6374 · September 9, 2022, 12:11pm

Another know issue with certs is that the time/date on the PicoW isn’t synchronised to NTP - Add the following code:-

import ntptime

ntptime.host = "de.pool.ntp.org"
ntptime.settime()

FrankD · September 9, 2022, 12:11pm

Thanks a lot Ian for further investigating this problem!!!

What do you think should be our next move? Do you think you have enough detailed info for a ticket for the ussl/usocket project, or do we need more support/answers from HiveMQ first?

I see that CircuitPython also has progressed and has unstable builds with Wifi support for Pico W, but don’t have time right now to try out…

PS yes, I did add the NTP already without better results

IanW6374 · September 10, 2022, 5:36am

Finally got it working … Note the addition of the “Server_hostname” in the sslparams (If your are interested this has to do with SNI)

sslparams = {'server_side':False, 'key':None, 'cert':None, 'cadata':cacert, 'cert_reqs':ssl.CERT_REQUIRED, 'server_hostname': hostname}

**** Thanks to Carglglz on the MicroPython Forum for providing the fix