HIVE MQ base64 encode

1

I’m currently utilizing HiveMQ Edge and have integrated an HTTPS adapter to make API calls and retrieve data. However, I’ve noticed that the data returned from these calls is encoded in base64.

For instance, the data I’m receiving looks like this: {"timestamp":1709178693978,"value":"data:application/json; charset=utf-8;base64,eyJjb29yZCI6eyJsb24iOi0wLjEyNTcsImxhdCI6NTEuNTA4NX0sIndlYXRoZXIiOlt7ImlkIjo4MDMsIm1haW4iOiJDbG91ZHMiLCJkZXNjcmlwdGlvbiI6ImJyb2tlbiBjbG91ZHMiLCJpY29uIjoiMDRuIn1dLCJiYXNlIjoic3RhdGlvbnMiLCJtYWluIjp7InRlbXAiOjI4My44OSwiZmVlbHNfbGlrZSI6MjgzLjM0LCJ0ZW1wX21pbiI6MjgyLjYyLCJ0ZW1wX21heCI6Mjg0LjksInByZXNzdXJlIjoxMDA2LCJodW1pZGl0eSI6ODl9LCJ2aXNpYmlsaXR5IjoxMDAwMCwid2luZCI6eyJzcGVlZCI6NC42MywiZGVnIjoyMTB9LCJjbG91ZHMiOnsiYWxsIjo3NX0sImR0IjoxNzA5MTc4MzAwLCJzeXMiOnsidHlwZSI6MiwiaWQiOjIwNzU1MzUsImNvdW50cnkiOiJHQiIsInN1bnJpc2UiOjE3MDkxODkyNDIsInN1bnNldCI6MTcwOTIyODMzNn0sInRpbWV6b25lIjowLCJpZCI6MjY0Mzc0MywibmFtZSI6IkxvbmRvbiIsImNvZCI6MjAwfQ=="}

can some one explain his

2

Hi @Nandhu

Warm greetings! It’s truly a pleasure to welcome you to the HiveMQ Community. We’re delighted to have you on board, exploring MQTT and the functionalities of the HiveMQ broker.

Regarding your inquiry about decoding the base64 string, if you’re using Linux, you can effortlessly achieve this by utilizing the base64 --decode command. Allow me to provide an example tailored to your specific base64 string:

# echo 'eyJjb29yZCI6eyJsb24iOi0wLjEyNTcsImxhdCI6NTEuNTA4NX0sIndlYXRoZXIiOlt7ImlkIjo4MDMsIm1haW4iOiJDbG91ZHMiLCJkZXNjcmlwdGlvbiI6ImJyb2tlbiBjbG91ZHMiLCJpY29uIjoiMDRuIn1dLCJiYXNlIjoic3RhdGlvbnMiLCJtYWluIjp7InRlbXAiOjI4My44OSwiZmVlbHNfbGlrZSI6MjgzLjM0LCJ0ZW1wX21pbiI6MjgyLjYyLCJ0ZW1wX21heCI6Mjg0LjksInByZXNzdXJlIjoxMDA2LCJodW1pZGl0eSI6ODl9LCJ2aXNpYmlsaXR5IjoxMDAwMCwid2luZCI6eyJzcGVlZCI6NC42MywiZGVnIjoyMTB9LCJjbG91ZHMiOnsiYWxsIjo3NX0sImR0IjoxNzA5MTc4MzAwLCJzeXMiOnsidHlwZSI6MiwiaWQiOjIwNzU1MzUsImNvdW50cnkiOiJHQiIsInN1bnJpc2UiOjE3MDkxODkyNDIsInN1bnNldCI6MTcwOTIyODMzNn0sInRpbWV6b25lIjowLCJpZCI6MjY0Mzc0MywibmFtZSI6IkxvbmRvbiIsImNvZCI6MjAwfQ==' | base64 -d | jq
{
  "coord": {
    "lon": -0.1257,
    "lat": 51.5085
  },
  "weather": [
    {
      "id": 803,
      "main": "Clouds",
      "description": "broken clouds",
      "icon": "04n"
    }
  ],
  "base": "stations",
  "main": {
    "temp": 283.89,
    "feels_like": 283.34,
    "temp_min": 282.62,
    "temp_max": 284.9,
    "pressure": 1006,
    "humidity": 89
  },
  "visibility": 10000,
  "wind": {
    "speed": 4.63,
    "deg": 210
  },
  "clouds": {
    "all": 75
  },
  "dt": 1709178300,
  "sys": {
    "type": 2,
    "id": 2075535,
    "country": "GB",
    "sunrise": 1709189242,
    "sunset": 1709228336
  },
  "timezone": 0,
  "id": 2643743,
  "name": "London",
  "cod": 200
}

This command will decode the base64 string and present the structured JSON output. If you have any further questions or need additional assistance, feel free to ask. We’re here to help!

Kind regards,
Dasha from HiveMQ Team

3

I’m currently utilizing HiveMQ Edge and HiveMQ Enterprise along with Kafka to collect data. For HiveMQ Edge, I’m employing HTTPS adapters and leveraging an open API to retrieve data, but I’m noticing that the data I receive is encoded in base64. Meanwhile, I’m using Kafka to obtain data and I prefer to receive this data in plaintext.

I’m curious about why the data I receive is encoded in base64.

Additionally, I have a question regarding MQTT’s data transmission capabilities: does MQTT send data in binary or plaintext?

4

Hello @Nandhu

The reason you’re receiving encoded data (Base64) from HiveMQ Edge is to ensure that binary data can be transmitted over HTTPS adapters that are primarily designed to handle text.

Regarding your query about MQTT’s data transmission capabilities: MQTT itself is agnostic to the format of the data payload. It can handle both binary and plaintext data.

Kind regards,
Diego from HiveMQ Team

5

@Daria_H and @Diego. I’m also having this issue, and I apologize, I don’t understand the solution. I want to query the metrics from the prometheus extension and post them to Edge as JSON. I am running 2024.7 as a Windows service. Just like the original poster, the response from the HTTP adapter is encoded.

I created a Data Policy Transform to decode the payload, and parse it. The script is failing with the below error:

2024-10-29 20:06:50,689 ERROR - Failed to execute intermediate function 'fn:f_Metrics:latest' (policyId: 'Metrics', actionId: 'onSuccess' operationId: 'o_transform'):
      - Script execution failed - An error occurred in line '5' at position '158': ReferenceError: atob is not defined

In my script, I attempted to use atob() to decode it. That obviously isn’t working. Do you have another way to decode it? Your solution mentioned inserting some code via linux, but I don’t see how to do that in the HTTP adapter or the Data Hub Transform.

6

Hi @Damon

It’s wonderful to have someone like you who’s enthusiastic about MQTT and the HiveMQ broker. Welcome to the HiveMQ Community! We’re excited to see new users like you.

You can implement a custom function to decode base64 strings. Here’s an example:

function base64Decode(str) {
    const chars = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/=';
    let output = '';
    let buffer = 0;
    let bufferLength = 0;

    for (let i = 0; i < str.length; i++) {
        const char = str.charAt(i);
        const charIndex = chars.indexOf(char);

        if (charIndex === -1) continue;

        buffer = (buffer << 6) | charIndex;
        bufferLength += 6;

        while (bufferLength >= 8) {
            const byte = (buffer >> (bufferLength - 8)) & 0xFF;
            output += String.fromCharCode(byte);
            bufferLength -= 8;
        }
    }

    return output;
}

const decodedPayload = base64Decode(yourEncodedData);

I hope this helps
Best,
Dasha from HiveMQ Team

7

Thanks! I had gotten some help from my solution engineer a little while ago. They had a slightly different approach - same outcome.

function decodeBase64(base64String) {
    const base64Chars = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/';
    let base64 = base64String.replace(/=+$/, '');
    let binaryString = '';
    for (let i = 0; i < base64.length; i++) {
        let binaryChar = base64Chars.indexOf(base64[i]).toString(2);
        binaryChar = binaryChar.padStart(6, '0');
        binaryString += binaryChar;
    }
    let decodedString = '';
    for (let i = 0; i < binaryString.length; i += 8) {
        let byte = binaryString.substring(i, i + 8);
        decodedString += String.fromCharCode(parseInt(byte, 2));
    }
    return decodedString;
}

function transform(publish, context) {
    const timestamp = Date.now()
    
    const messageCoded = publish.payload.value.split(",")[1]
    const message = decodeBase64(messageCoded)
    const metricsRaw = message.split("\n").filter(line => line.startsWith('com_'))
    const metrics = metricsRaw.map((element, index, array) => {
      const metricParts = element.split(" ")
      return {
          id: metricParts[0],
          v: parseFloat(metricParts[1]),
          q: true,
          t: timestamp
      }
    })
  
  publish.payload = {
      timestamp: timestamp,
      values: metrics
  }
  
  console.log(publish.payload)
  return publish
}

Now if I can just figure out why Hive is outputting Date.now() in scientific notation instead of integer like every other runtime environment I’ve used. :slight_smile: