Requesting mqtt data from LoRa server for emoncms node

[bgrattan (Bob)]

Hello,
I presently have a RAK7244 (chirpstack) running LoRa on my LAN which receives temp/hum data from two LHT65 sensors. I would like to add this data to an emonpi which I have running on the same LAN. I have the parts running but, not being a programmer, I don’t know how to integrate the two.

On the emonpi, when I issue the command [LoRa server is 192.168.1.10]:

mosquitto_sub -h 192.168.1.10 -t “application/7/device/#” -d

I get:

{“applicationID”:“7”,“applicationName”:“LHT65”,“deviceName”:“LHT65x”,“devEUI”:“a8404117b18312e9”,“txInfo”:{“frequency”:905100000,“dr”:0},“adr”:true,“fCnt”:1232,“fPort”:2,“data”:“y+wJdQIKAQlZf/8=”,“object”:{“BatV”:3.052,“Ext_sensor”:“Temperature Sensor LHT65”,“Hum_SHT”:“52.2”,“TempF_DS”:“75.07”,“TempF_SHT”:“75.58”}}

I see a JSON format for the data at the end of the string and this is what I need to send to the emonpi as a new node.

I think this can be done with node-red but am having a hard time understanding how to do it.

If anyone has any suggestions or a similar example I could look at then it would be greatly appreciated.

Thanks,
Bob

[icenov (Anthony Vassallo)]

Hi Bob

I’m not familiar with the RAK7244, so my experience might not help, but I have an Adafruit LoRa sending temperature data through mqtt on node-red back to the emonpi.

I run node-red on a separate raspberry pi as I think it was all too much on one system - especially if you want to use the browser interface.

In my case, I have an Adafruit LoRa receiver connected via the rPi serial and read the data packet using a “serial in” node. But I have also used the “mqtt in” and “mqtt out” node and they also work quite well.

The fiddly bit is to parse the json so it is in the right format for emonpi - but there are plenty of examples and you should be able to sort it even without programming background.

I need to go back into my old node-red setup and look for the code to parse the json. From memory, you need to split the json into an array, then select the elements you need. These can then be sent to the emonpi using the other node.

So I suggest you get a working node red platform, and using the browser interface see if you can read the json into a “debug” node. With this working, then add a “function” node that you can parse the json to extract the relevant data. This data then goes to emonpi through an mqtt out node (from memory).

If you get stuck, post what you have done and I’ll try and help - but there are quite a few node-red experts in the community here, so someone will be able to help.

[bgrattan (Bob)]

Thanks for the help, Anthony. Node-Red is somewhat beyond me at this point. On the surface it looks simple but I have to figure out what it is doing. Does it create a piece of code from the flow chart which I then load onto my LoRa server? Sorry, but you can see I’m not a programmer.
Bob

[icenov (Anthony Vassallo)]

Hi Bob

From what you said, the RAK7244 is already broadcasting the sensor data, as you can see it being received on the emonpi.
The challenge in using it directly on the emonpi is to get the data into the right format for emoncms, which might mean reprograming the RAK7244. The node-red option will allow you to capture the data stream and send it to emoncms using some simple code.

I suggest you setup node-red on a different machine initially and use the browser interface to play with the nodes. I use a different rPi, but it can also work on desktop/laptop etc - but it obviously needs to be on 24/7. See Running Node-RED locally : Node-RED

Once you have node-red running and using the browser interface, you should see a flow tab, which might be empty. You drag nodes from the list on the left and drop them onto the canvas - find the mqtt in node - see image:

If you double clik on the node it will need open an input tab where you fill in the IP address of the mqtt server (in your case 192.168.1.10, and append the 1883 port like 192.168.1.10:1883). Then set the topic like “application/7/device/#”
You might need to provide a name and password depending on settings.

This node will then scan your RAK7244 and subscribe to the topic. But you won’t see anything unless you attach a debug node which will show the output in the right hand (need to click on the debug tab) and click deploy. If all is working you will see the same output as you showed in your post.
If you get this far, the next step is to extract the field with your data. There are a few ways of doing this, but I’ll explain how I did it when you get to this point!
Good luck.

[bgrattan (Bob)]

Looks as if I have it. Node-Red is on my emonpi and connected to RAK7244

[icenov (Anthony Vassallo)]

Hi Bob
That’s great. The trick is to now extract the desired fields from this output. It will be easier if you can get the output without the verbose debug - maybe by removing the “-d” from:

192.168.1.10 -t “application/7/device/#” -d

This should just return the string object:

"object":{
      "BatV":3.052,
      "Ext_sensor":“Temperature Sensor LHT65”,
      "Hum_SHT":“52.2”,
      "TempF_DS":“75.07”,
      "TempF_SHT":“75.58”
   }

It seems like you just want the fields with numerical values - i.e. not the Ext_sensor field. It is slightly complicated with quotes on some but not all numerical values, but this can be overcome.

I copied your output as above (i.e. minus the debug) and put it into an inject node, then passed it to a function node with a simple script to split the string on commas and to assign each fragment to a variable. [I’m sure there is a better way of doing this, and maybe one of the node-red/json experts can help here.]

Drag a function node onto the canvas and insert it between the existing nodes, and open the function node. If you paste the following code, this should split the payload using commas, into a list. It then assigns each element to a variable (a-e), and creates another message payload, which is returned.

output = msg.payload.split(",");
var a=output[0]
var b=output[1]
var c=output[2]
var d=output[3]
var e=output[4]
msg = {payload:{"a":a,"b":b,"c":c,"d":d,"e":e}}
return msg;

You can then work on each variable to extract the required data - by adding extra lines of code. You can replace the letters like “a” in the new payload with any string you want. If necessary, each variable can be sent to a separate output, but for now see if you can get output split into the 5 variables.

Edit - Formatted text. Moderator, BT

[bgrattan (Bob)]

Thanks, Anthony, for your help and patience.
I got most of what you explained but am not able to get rid of the extra “stuff” in the payload.
Thus your script gives me:
msg.payload : Object
object
a: "{“applicationID”:“7"”
b: "“applicationName”:“LHT65"”
c: ““deviceName”:“LHT65x””
d: "“devEUI”:“a8404117b18312e9"”
e: ““txInfo”:{“frequency”:903900000”

It must be coming from the json formatting in the codec for the LHT65.

I have given the exact path: application/7/device/a8404117b18312e9/rx with no “-d”
Not sure how to drop the excess data.

CODEC
// Decode decodes an array of bytes into an object.
// - fPort contains the LoRaWAN fPort number
// - bytes is an array of bytes, e.g. [225, 230, 255, 0]
// - variables contains the device variables e.g. {“calibration”: “3.5”} (both the key / value are of type string)
// The function must return an object, e.g. {“temperature”: 22.5}
function Decode(fPort, bytes){
var data = {
//External sensor
Ext_sensor:
{
“0”:“No external sensor”,
“1”:“Temperature Sensor LHT65”,
“4”:“Interrupt Sensor send”,
“5”:“Illumination Sensor”,
“6”:“ADC Sensor”,
“7”:“Interrupt Sensor count”,
}[bytes[6]&0x7F],

   //Battery,units:V
   BatV:((bytes[0]<<8 | bytes[1]) & 0x3FFF)/1000,
   
   //SHT20,temperature,units:
   //TempC_SHT:((bytes[2]<<24>>16 | bytes[3])/100).toFixed(2),
     TempF_SHT:(((bytes[2]<<24>>16 | bytes[3])/100) * 1.8 + 32).toFixed(2),

   
   //SHT20,Humidity,units:%
   Hum_SHT:((bytes[4]<<8 | bytes[5])/10).toFixed(1),
   
   //DS18B20,temperature,units:
   //TempC_DS:
     TempF_DS:
   {
    // "1":((bytes[7]<<24>>16 | bytes[8])/100).toFixed(2),
       "1":(((bytes[7]<<24>>16 | bytes[8])/100) * 1.8 + 32).toFixed(2),
     // Change to "0" to eliminate DS probe

   }[bytes[6]&0xFF],       
   
   //Exti pin level,PA4
   Exti_pin_level:
   {
     "4":bytes[7] ? "High":"Low",
   }[bytes[6]&0x7F], 
   
   //Exit pin status,PA4
   Exti_status:
   {
     "4":bytes[8] ? "True":"False",
   }[bytes[6]&0x7F],    
   
   //BH1750,illumination,units:lux
   ILL_lux:
   {
     "5":bytes[7]<<8 | bytes[8],
   }[bytes[6]&0x7F],  

    //ADC,PA4,units:V
    ADC_V:
   {
     "6":(bytes[7]<<8 | bytes[8])/1000,
   }[bytes[6]&0x7F],  
   
    //Exti count,PA4,units:times
    Exit_count:
    {
      "7":bytes[7]<<8 | bytes[8],
    }[bytes[6]&0x7F],  
    
    //Applicable to working mode 4567,and working mode 467 requires short circuit PA9 and PA10
    No_connect:
    {
      "1":"Sensor no connection",
    }[(bytes[6]&0x80)>>7],  

};
return data;
}

[icenov (Anthony Vassallo)]

Hi Bob
Actually depending on the output of your mqtt node, the solution might be quite simple. I’ve experimented a bit with the output, but when I copy and paste from the forum, the quotation marks get a bit corrupted in that some of the quotes become begin/end pairs in my editor which trips up the node-red function. If I manually replace these quote marks I can turn the string into an object which then makes the extraction of the data simple.

In my node-red, if the mqtt output is an “object” then a few lines of code in the function node can extract the values. Can you try this in your function node:

input = msg.payload
var BatV = input.object.BatV
var Hum_SHT = input.object.Hum_SHT
var TempF_DS= input.object.TempF_DS
var TempF_SHT= input.object.TempF_SHT
msg2 = {payload:{"BatV":BatV, "Hum_SHT":Hum_SHT, "TempF_DS":TempF_DS,"TempF_SHT":TempF_SHT}};
return msg2;

I inject your string as JSON, which is then handled as an object in the function node. As an object we can return the “value” for any “key”. In this case, the keys are BatV etc and the values are eg 3.052 for batV.

You might need to check the output option of the mqtt in node - there is an auto option, but this might need to be changed to a “parsed JSON object”.

As it stands, the battery value is numeric, but the others look to be strings, however I think it is easy to convert to which ever form is required.

If this works, then it is relatively straightforward to send each key:value to an mqtt out node to emoncms. In fact the function node is easily configured for multiple outputs, so each key:value can be sent to a different mqtt node if required.

Let me know how you go…

[Robert.Wall]

@bgrattan
If you’re having trouble with the forum interpreting parts of your post as formatting instructions to the forum software, try surrounding the whole of the code with 3 ‘backticks’ (normally found at the top left of the keyboard) on a line of their own before the code, and 3 more backticks also on a line of their own after the code:

```
code
```

If you don’t want any language markup applied, then add the word ‘text’ after the first 3 backticks, like this:
```text

[bgrattan (Bob)]

Anthony,
I tried adding your code [input = msg.payload
var BatV = input.object.BatV, etc…] to the function node but maybe I’m still doing something wrong. I get the message:

9/25/2021, 10:56:43 AM [node: 5a173b7b.8cbf7c] (http://192.168.1.173:1880/#) 
function : (error)
"TypeError: Cannot read property 'BatV' of undefined"

I’m afraid I’m just treading water here because it’s not clear to me what node-red is trying to do.
The json message from the lora server contains more data than I need. Sorry for the simplistic questions.
Bob

[bgrattan (Bob)]

Thanks, Robert, I’ll try to clean up my act. By the way, your suggestion to remove the ac/dc power jumper on the emonTX seems to have fixed the problem. No lockups for over a month.
Bob

[icenov (Anthony Vassallo)]

Hi Bob
I had the same problem when playing with your output - we’re both learning here!.
This means that the input to the function is not a Javascript object, so the output from the mqtt node is not in the correct form. I’m on my way out for a few hours, but will see if I can find the solution when I get back. Not far from a solution now!

[bgrattan (Bob)]

Take your time, and many thanks. It’s fall here so happy spring!

[icenov (Anthony Vassallo)]

Hi Bob
Looking at your earlier post I can see that the debug output says “string” - so we need to convert it to an object before applying the function code.
This is easy: Drag a “json” node from under the parser category and link the mqtt node to its input. In the JSON node, under action, select “always convert to Javascript object”. If you check its output with a debug node, it should now show it as an object:

msg.payload : Object
{ applicationID: "7", applicationName: "LHT65", deviceName: "LHT65x", devEUI: "a8404117b18312e9", txInfo: object

If this works, and you see the output as an Object, then you can link the function node and start to extract the data. Your debug node after the function node should now show

If you get this far, then the last step in Node-red is to pass this Object to the mqtt out node (drag it from just under the mqtt in node). Inside this node you need to set the server address and topic, and name/password so it can connect with the emonpi.

When you open the node, you will see an input field for the server, but don’t enter the value here - use the pencil box to the right to open the full properties page. Here enter the IP address of your emonpi (192.168.1.10 from your first post), and then port 1883. I think everything else on this tab is Ok. Then select the security tab, and enter the emonpi username and password. If you haven’t changed them from installation, it will be like emonpi/emonpi2016 if I recall correctly. I didn’t change anything on the Messages tab.

After you update (save) and return to the first page, enter the topic like “emon/mqttest” . In my setup, this sends the four key:value pairs in the object to the emonpi which magically turn up as inputs, under “mqttest” or whatever you specified as the emon topic. (Don’t forget to deploy your changes in Node-red).

My setup looks like this:

But you would have your mqtt in node where I use inject (ignore the 2 outputs on the function node…).

It may not work straight away, but let me know how you go.

[bgrattan (Bob)]

Hi Anthony,
Well, I’m making progress but am not quite there yet. I have my setup below which produces the correct object in debug. However, when I move an “mqtt out” into the picture, and set the destination server, it automatically updates the “mqtt in” to the same info. I can’t seem to change one without it affecting the other.
Here’s my setup which may be the problem. My source broker (mqtt in) is the RAK7244 (192,168.1.10) which is sending the topic "application/7/device/+/rx. My destination (mqtt out) is the emonpi (192.168.1.173) which is also the machine on which I am running node-red. In other words, node-red ( installed on the emonpi machine) subscribes to the topic on the RAK7244 and then needs to send it to the emoncms software which is also on the same machine as the node-red software. I hope this is not too confusing to you as it is becoming to me…
Anyway, it looks as if I have the correct format you are looking for and just need to send it to the emoncms software on the emonpi. Screen shot below. Thanks

Bob

[icenov (Anthony Vassallo)]

Hi Bob
That’s strange, as I think it is normal to have node-red and emoncms etc on the same machine. I’ll try and setup another node-red platform and try and work out if there is a problem. In the meantime just check that your mqtt in and out nodes are configured correctly, that your out node is attached to the function out node. Just to be clear - you have an mqtt in node in the image above, but no mqtt out node. There should be an out node attached to the function out…

[bgrattan (Bob)]

Anthony,
Sorry, I’m probably not very clear here. The image I sent yesterday was simply to illustrate how far I had gotten with your instructions to produce the object–it’s working up to this point. After that (not pictured) was the attempt to add the out node after the function. This didn’t work as the out node simply changed to the same settings as my in node and thus would not allow me to set the in node to source (192.168.1.10) and the out node to the emonpi (192.168.1.173 or localhost).
Something else. While fumbling around with this, before getting your offer of help, I had installed this node-red for emoncms :

[Node-RED - Guide | OpenEnergyMonitor](https://guide.openenergymonitor.org/integrations/nodered/)

I see an emon push node on the left for output which I tried after the mqtt out node didn’t work. The error I get is “no node group” Not sure what I need to put here but maybe this is what I’m looking for to complete the transfer.

I may have just complicated things here…

Again, many thanks for your patience and help.
Bob

[bgrattan (Bob)]

Late breaking news! They say if you get enough monkeys with typewriters that one will write a novel…
By playing around with the emoncms push, which defaults to localhost, and adding the write api plus just a name (I used emoncms) it actually sent the data to the “inputs” section of my emoncms gui.
Since I have two devices (LHT65x and LHT65-1 with different devEUIs) I need to see how this works and they may just overwrite each other with my present setup.

[icenov (Anthony Vassallo)]

That sounds like it will actually work! I’ve not used the emoncms nodes, but they look promising. Maybe a bit more tweaking (fiddling!) will get there. I find it very helpful to have emoncms manage all my sensor data as it is then easy to chart, backup, etc from one place.

[bgrattan (Bob)]

Anthony,
It does seem to be working and I greatly appreciate your time and help getting there.
What I want to do is to leave node-red running on the emonpi so it will keep updating the data from the RAK7244 LoRa server.
I have created two parts which I guess makes just one flow. There is probably a more efficient way to do this but this is working for now. Is it possible to have node-red running in the background (daemon?) at bootup and service the flow I have created? I’m sure this is possible but I’m not sure how to save the flow so it will restart/keep running.
I’ve attached a couple of screen shots. Thanks again and cheers!