Requesting mqtt data from LoRa server for emoncms node

[icenov (Anthony Vassallo)]

Hi Bob

From what you said, the RAK7244 is already broadcasting the sensor data, as you can see it being received on the emonpi.
The challenge in using it directly on the emonpi is to get the data into the right format for emoncms, which might mean reprograming the RAK7244. The node-red option will allow you to capture the data stream and send it to emoncms using some simple code.

I suggest you setup node-red on a different machine initially and use the browser interface to play with the nodes. I use a different rPi, but it can also work on desktop/laptop etc - but it obviously needs to be on 24/7. See Running Node-RED locally : Node-RED

Once you have node-red running and using the browser interface, you should see a flow tab, which might be empty. You drag nodes from the list on the left and drop them onto the canvas - find the mqtt in node - see image:

If you double clik on the node it will need open an input tab where you fill in the IP address of the mqtt server (in your case 192.168.1.10, and append the 1883 port like 192.168.1.10:1883). Then set the topic like “application/7/device/#”
You might need to provide a name and password depending on settings.

This node will then scan your RAK7244 and subscribe to the topic. But you won’t see anything unless you attach a debug node which will show the output in the right hand (need to click on the debug tab) and click deploy. If all is working you will see the same output as you showed in your post.
If you get this far, the next step is to extract the field with your data. There are a few ways of doing this, but I’ll explain how I did it when you get to this point!
Good luck.

[bgrattan (Bob)]

Looks as if I have it. Node-Red is on my emonpi and connected to RAK7244

[icenov (Anthony Vassallo)]

Hi Bob
That’s great. The trick is to now extract the desired fields from this output. It will be easier if you can get the output without the verbose debug - maybe by removing the “-d” from:

192.168.1.10 -t “application/7/device/#” -d

This should just return the string object:

"object":{
      "BatV":3.052,
      "Ext_sensor":“Temperature Sensor LHT65”,
      "Hum_SHT":“52.2”,
      "TempF_DS":“75.07”,
      "TempF_SHT":“75.58”
   }

It seems like you just want the fields with numerical values - i.e. not the Ext_sensor field. It is slightly complicated with quotes on some but not all numerical values, but this can be overcome.

I copied your output as above (i.e. minus the debug) and put it into an inject node, then passed it to a function node with a simple script to split the string on commas and to assign each fragment to a variable. [I’m sure there is a better way of doing this, and maybe one of the node-red/json experts can help here.]

Drag a function node onto the canvas and insert it between the existing nodes, and open the function node. If you paste the following code, this should split the payload using commas, into a list. It then assigns each element to a variable (a-e), and creates another message payload, which is returned.

output = msg.payload.split(",");
var a=output[0]
var b=output[1]
var c=output[2]
var d=output[3]
var e=output[4]
msg = {payload:{"a":a,"b":b,"c":c,"d":d,"e":e}}
return msg;

You can then work on each variable to extract the required data - by adding extra lines of code. You can replace the letters like “a” in the new payload with any string you want. If necessary, each variable can be sent to a separate output, but for now see if you can get output split into the 5 variables.

Edit - Formatted text. Moderator, BT

[bgrattan (Bob)]

Thanks, Anthony, for your help and patience.
I got most of what you explained but am not able to get rid of the extra “stuff” in the payload.
Thus your script gives me:
msg.payload : Object
object
a: "{“applicationID”:“7"”
b: "“applicationName”:“LHT65"”
c: ““deviceName”:“LHT65x””
d: "“devEUI”:“a8404117b18312e9"”
e: ““txInfo”:{“frequency”:903900000”

It must be coming from the json formatting in the codec for the LHT65.

I have given the exact path: application/7/device/a8404117b18312e9/rx with no “-d”
Not sure how to drop the excess data.

CODEC
// Decode decodes an array of bytes into an object.
// - fPort contains the LoRaWAN fPort number
// - bytes is an array of bytes, e.g. [225, 230, 255, 0]
// - variables contains the device variables e.g. {“calibration”: “3.5”} (both the key / value are of type string)
// The function must return an object, e.g. {“temperature”: 22.5}
function Decode(fPort, bytes){
var data = {
//External sensor
Ext_sensor:
{
“0”:“No external sensor”,
“1”:“Temperature Sensor LHT65”,
“4”:“Interrupt Sensor send”,
“5”:“Illumination Sensor”,
“6”:“ADC Sensor”,
“7”:“Interrupt Sensor count”,
}[bytes[6]&0x7F],

   //Battery,units:V
   BatV:((bytes[0]<<8 | bytes[1]) & 0x3FFF)/1000,
   
   //SHT20,temperature,units:
   //TempC_SHT:((bytes[2]<<24>>16 | bytes[3])/100).toFixed(2),
     TempF_SHT:(((bytes[2]<<24>>16 | bytes[3])/100) * 1.8 + 32).toFixed(2),

   
   //SHT20,Humidity,units:%
   Hum_SHT:((bytes[4]<<8 | bytes[5])/10).toFixed(1),
   
   //DS18B20,temperature,units:
   //TempC_DS:
     TempF_DS:
   {
    // "1":((bytes[7]<<24>>16 | bytes[8])/100).toFixed(2),
       "1":(((bytes[7]<<24>>16 | bytes[8])/100) * 1.8 + 32).toFixed(2),
     // Change to "0" to eliminate DS probe

   }[bytes[6]&0xFF],       
   
   //Exti pin level,PA4
   Exti_pin_level:
   {
     "4":bytes[7] ? "High":"Low",
   }[bytes[6]&0x7F], 
   
   //Exit pin status,PA4
   Exti_status:
   {
     "4":bytes[8] ? "True":"False",
   }[bytes[6]&0x7F],    
   
   //BH1750,illumination,units:lux
   ILL_lux:
   {
     "5":bytes[7]<<8 | bytes[8],
   }[bytes[6]&0x7F],  

    //ADC,PA4,units:V
    ADC_V:
   {
     "6":(bytes[7]<<8 | bytes[8])/1000,
   }[bytes[6]&0x7F],  
   
    //Exti count,PA4,units:times
    Exit_count:
    {
      "7":bytes[7]<<8 | bytes[8],
    }[bytes[6]&0x7F],  
    
    //Applicable to working mode 4567,and working mode 467 requires short circuit PA9 and PA10
    No_connect:
    {
      "1":"Sensor no connection",
    }[(bytes[6]&0x80)>>7],  

};
return data;
}

[icenov (Anthony Vassallo)]

Hi Bob
Actually depending on the output of your mqtt node, the solution might be quite simple. I’ve experimented a bit with the output, but when I copy and paste from the forum, the quotation marks get a bit corrupted in that some of the quotes become begin/end pairs in my editor which trips up the node-red function. If I manually replace these quote marks I can turn the string into an object which then makes the extraction of the data simple.

In my node-red, if the mqtt output is an “object” then a few lines of code in the function node can extract the values. Can you try this in your function node:

input = msg.payload
var BatV = input.object.BatV
var Hum_SHT = input.object.Hum_SHT
var TempF_DS= input.object.TempF_DS
var TempF_SHT= input.object.TempF_SHT
msg2 = {payload:{"BatV":BatV, "Hum_SHT":Hum_SHT, "TempF_DS":TempF_DS,"TempF_SHT":TempF_SHT}};
return msg2;

I inject your string as JSON, which is then handled as an object in the function node. As an object we can return the “value” for any “key”. In this case, the keys are BatV etc and the values are eg 3.052 for batV.

You might need to check the output option of the mqtt in node - there is an auto option, but this might need to be changed to a “parsed JSON object”.

As it stands, the battery value is numeric, but the others look to be strings, however I think it is easy to convert to which ever form is required.

If this works, then it is relatively straightforward to send each key:value to an mqtt out node to emoncms. In fact the function node is easily configured for multiple outputs, so each key:value can be sent to a different mqtt node if required.

Let me know how you go…

[Robert.Wall]

@bgrattan
If you’re having trouble with the forum interpreting parts of your post as formatting instructions to the forum software, try surrounding the whole of the code with 3 ‘backticks’ (normally found at the top left of the keyboard) on a line of their own before the code, and 3 more backticks also on a line of their own after the code:

```
code
```

If you don’t want any language markup applied, then add the word ‘text’ after the first 3 backticks, like this:
```text

[bgrattan (Bob)]

Anthony,
I tried adding your code [input = msg.payload
var BatV = input.object.BatV, etc…] to the function node but maybe I’m still doing something wrong. I get the message:

9/25/2021, 10:56:43 AM [node: 5a173b7b.8cbf7c] (http://192.168.1.173:1880/#) 
function : (error)
"TypeError: Cannot read property 'BatV' of undefined"

I’m afraid I’m just treading water here because it’s not clear to me what node-red is trying to do.
The json message from the lora server contains more data than I need. Sorry for the simplistic questions.
Bob

[bgrattan (Bob)]

Thanks, Robert, I’ll try to clean up my act. By the way, your suggestion to remove the ac/dc power jumper on the emonTX seems to have fixed the problem. No lockups for over a month.
Bob

[icenov (Anthony Vassallo)]

Hi Bob
I had the same problem when playing with your output - we’re both learning here!.
This means that the input to the function is not a Javascript object, so the output from the mqtt node is not in the correct form. I’m on my way out for a few hours, but will see if I can find the solution when I get back. Not far from a solution now!

[bgrattan (Bob)]

Take your time, and many thanks. It’s fall here so happy spring!

[icenov (Anthony Vassallo)]

Hi Bob
Looking at your earlier post I can see that the debug output says “string” - so we need to convert it to an object before applying the function code.
This is easy: Drag a “json” node from under the parser category and link the mqtt node to its input. In the JSON node, under action, select “always convert to Javascript object”. If you check its output with a debug node, it should now show it as an object:

msg.payload : Object
{ applicationID: "7", applicationName: "LHT65", deviceName: "LHT65x", devEUI: "a8404117b18312e9", txInfo: object

If this works, and you see the output as an Object, then you can link the function node and start to extract the data. Your debug node after the function node should now show

If you get this far, then the last step in Node-red is to pass this Object to the mqtt out node (drag it from just under the mqtt in node). Inside this node you need to set the server address and topic, and name/password so it can connect with the emonpi.

When you open the node, you will see an input field for the server, but don’t enter the value here - use the pencil box to the right to open the full properties page. Here enter the IP address of your emonpi (192.168.1.10 from your first post), and then port 1883. I think everything else on this tab is Ok. Then select the security tab, and enter the emonpi username and password. If you haven’t changed them from installation, it will be like emonpi/emonpi2016 if I recall correctly. I didn’t change anything on the Messages tab.

After you update (save) and return to the first page, enter the topic like “emon/mqttest” . In my setup, this sends the four key:value pairs in the object to the emonpi which magically turn up as inputs, under “mqttest” or whatever you specified as the emon topic. (Don’t forget to deploy your changes in Node-red).

My setup looks like this:

But you would have your mqtt in node where I use inject (ignore the 2 outputs on the function node…).

It may not work straight away, but let me know how you go.

[bgrattan (Bob)]

Hi Anthony,
Well, I’m making progress but am not quite there yet. I have my setup below which produces the correct object in debug. However, when I move an “mqtt out” into the picture, and set the destination server, it automatically updates the “mqtt in” to the same info. I can’t seem to change one without it affecting the other.
Here’s my setup which may be the problem. My source broker (mqtt in) is the RAK7244 (192,168.1.10) which is sending the topic "application/7/device/+/rx. My destination (mqtt out) is the emonpi (192.168.1.173) which is also the machine on which I am running node-red. In other words, node-red ( installed on the emonpi machine) subscribes to the topic on the RAK7244 and then needs to send it to the emoncms software which is also on the same machine as the node-red software. I hope this is not too confusing to you as it is becoming to me…
Anyway, it looks as if I have the correct format you are looking for and just need to send it to the emoncms software on the emonpi. Screen shot below. Thanks

Bob

[icenov (Anthony Vassallo)]

Hi Bob
That’s strange, as I think it is normal to have node-red and emoncms etc on the same machine. I’ll try and setup another node-red platform and try and work out if there is a problem. In the meantime just check that your mqtt in and out nodes are configured correctly, that your out node is attached to the function out node. Just to be clear - you have an mqtt in node in the image above, but no mqtt out node. There should be an out node attached to the function out…

[bgrattan (Bob)]

Anthony,
Sorry, I’m probably not very clear here. The image I sent yesterday was simply to illustrate how far I had gotten with your instructions to produce the object–it’s working up to this point. After that (not pictured) was the attempt to add the out node after the function. This didn’t work as the out node simply changed to the same settings as my in node and thus would not allow me to set the in node to source (192.168.1.10) and the out node to the emonpi (192.168.1.173 or localhost).
Something else. While fumbling around with this, before getting your offer of help, I had installed this node-red for emoncms :

[Node-RED - Guide | OpenEnergyMonitor](https://guide.openenergymonitor.org/integrations/nodered/)

I see an emon push node on the left for output which I tried after the mqtt out node didn’t work. The error I get is “no node group” Not sure what I need to put here but maybe this is what I’m looking for to complete the transfer.

I may have just complicated things here…

Again, many thanks for your patience and help.
Bob

[bgrattan (Bob)]

Late breaking news! They say if you get enough monkeys with typewriters that one will write a novel…
By playing around with the emoncms push, which defaults to localhost, and adding the write api plus just a name (I used emoncms) it actually sent the data to the “inputs” section of my emoncms gui.
Since I have two devices (LHT65x and LHT65-1 with different devEUIs) I need to see how this works and they may just overwrite each other with my present setup.

[icenov (Anthony Vassallo)]

That sounds like it will actually work! I’ve not used the emoncms nodes, but they look promising. Maybe a bit more tweaking (fiddling!) will get there. I find it very helpful to have emoncms manage all my sensor data as it is then easy to chart, backup, etc from one place.

[bgrattan (Bob)]

Anthony,
It does seem to be working and I greatly appreciate your time and help getting there.
What I want to do is to leave node-red running on the emonpi so it will keep updating the data from the RAK7244 LoRa server.
I have created two parts which I guess makes just one flow. There is probably a more efficient way to do this but this is working for now. Is it possible to have node-red running in the background (daemon?) at bootup and service the flow I have created? I’m sure this is possible but I’m not sure how to save the flow so it will restart/keep running.
I’ve attached a couple of screen shots. Thanks again and cheers!

[icenov (Anthony Vassallo)]

That’s excellent - congratulations! Node-red will run in the background while the rPi is running, but depending on how it was installed, it may not start after a reboot. But you can change this with

sudo systemctl enable nodered.service

on the command line. Further info on starting, stopping etc at node-red. These things do occassionally stop, especially when routers decide to change IP addresses, so worth checking that data is arriving. There are also some great tools in node-red if you feel like playing around - e.g. a dashboard where the data can be displayed, graphed etc in near real time. Have fun!

[bgrattan (Bob)]

Hi Anthony,
Sorry to bother you again but I’m not getting any takers on my latest question regarding graph data for the two nodes coming from MQTT.

My two nodes, emoncms and emoncms1, are working fine thanks to your help. However I seem unable to graph the data beyond 1, 6 hours. or a week–the data is there but I can’t plot it.
The problem, I believe, is the input interval. It seemed to set automatically on the emonTX nodes but not for the new ones I just added. The two new nodes create data every 20 minutes but, since their timers aren’t coordinated, the data comes in an irregular manor, 5 minutes, 10 minutes, 15 minutes, etc. Are there some settings you might suggest for me to try? Here are a couple of screen shots. I have a week showing now but two weeks won’t show at least the one week’s data, just a complete blank chart. Thanks again, Bob

[icenov (Anthony Vassallo)]

Hi Bob
If the data isn’t being logged at a regular interval then you should use PHPTIMESRIES instead of PHPFINA. PHPFINA only really logs the value and assumes a time based on a fixed sampling period, while PHPTIMESERIES logs the actual time with each data point (so uses at least double the disk storage space).
Can you try to setup a new feed from your inputs, but select PHPTIMESERIES as the “engine”?