Sync question

I am trying to understand the sync mechanism so that I can select what I feed from the local copy of emoncms to emoncms.org, and possibly in addition upload to adafruit.

In the Sync section of the local emoncms, all the inputs, feeds and dashboards have nothing in them except for the host https://emoncms.org. The username and password are blank.

However data is being fed to emoncms.org. I can see from the emonhub screen (see below) that data is being sent and indeed, looking at emoncms.org I can see that the data is being received. But how can I control what data is being sent since the relevant screen is empty. There must be something else going on.

I should mention that I have recently upgraded to a new Pi and a new version of emoncms and have backed up and restored the system state from the old system.

2022-12-04 20:56:21,111 DEBUG MQTT Publishing: emon/emontx1/power1 218
2022-12-04 20:56:21,112 DEBUG MQTT Publishing: emon/emontx1/power2 0
2022-12-04 20:56:21,114 DEBUG MQTT Publishing: emon/emontx1/power3 0
2022-12-04 20:56:21,115 DEBUG MQTT Publishing: emon/emontx1/power4 0
2022-12-04 20:56:21,116 DEBUG MQTT Publishing: emon/emontx1/vrms 240.61
2022-12-04 20:56:21,117 DEBUG MQTT Publishing: emon/emontx1/temp1 0
2022-12-04 20:56:21,118 INFO MQTT Publishing ‘node’ formatted msg
2022-12-04 20:56:21,118 DEBUG MQTT Publishing: emonhub/rx/10/values 218,0,0,0,240.61,0
2022-12-04 20:56:26,913 DEBUG emoncmsorg Buffer size: 3
2022-12-04 20:56:26,914 INFO emoncmsorg sending: https://emoncms.org/input/bulk.json?sentat=1670187386 (107 bytes of data, 3 frames, uncompressed)
2022-12-04 20:56:27,123 DEBUG emoncmsorg acknowledged receipt with ‘ok’ from

The sync tool, is very muh a one off - it doesn’t run all the time.

Without knowing the details of your setup, it looks like the configuration of EmonHub sends data to emoncms.org via an HTTP interfacer (INFO emoncmsorg sending: https://emoncms.org/input/bulk.json?)and to a local emoncms via MQTT (MQTT Publishing: emon/emontx1/) .

The 2 instances (emoncms.org and the local emoncms) are completely independent.

" The 2 instances (emoncms.org and the local emoncms) are completely independent." Yes, thanks, agreed.

I also agree with you that it looks like EmonHub is sending datat to emoncms.org via HTTP. So now my question is: Where is this configured? I can’t find any configuration in the graphical user interface, so I presume there is a setting or config file somewhere but I cannot find it.

The emoncmsorg part of emonhub.conf has only this

[[emoncmsorg]]
    Type = EmonHubEmoncmsHTTPInterfacer
    [[[init_settings]]]
    [[[runtimesettings]]]
        pubchannels = ToRFM12,
        subchannels = ToEmonCMS,
        url = https://emoncms.org
        apikey = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
        senddata = 1                    # Enable sending data to Emoncms.org
        sendstatus = 1                  # Enable sending WAN IP to Emoncms.org MyIP > https://emoncms.org/myip/list
        sendinterval= 30                # Bulk send interval to Emoncms.org in seconds

So I am still puzzled by how a particular subset of the data arrives at emoncms.org.

thanks for your help.

I have recently upgraded to a new Pi and a new version of emoncms and have backed up and restored the system state from the old system

emonHub is listed in the left menu.

image

How are you puzzled? ‘subset’ of what?

Yes, thanks, on the emonHub tab I can see the log (copy and pasted from in my previous post in this thread Sync question )

I can also edit emonhub.conf from the Edit Config tab, as you say.

What is puzzling me is that only a subset of the data available is uploaded to emoncms.org. To be specific here are all the mqtt messages (obtained from my espwroom32 which samples these from the wifi)
12:08:24.810 → Message arrived on topic: emon/emontx1/power2. Message: 1548
12:08:24.810 → Message arrived on topic: emon/emontx1/power3. Message: 0
12:08:24.842 → Message arrived on topic: emon/emontx1/power4. Message: 0
12:08:24.842 → Message arrived on topic: emon/emontx1/vrms. Message: 239.09
12:08:24.842 → Message arrived on topic: emon/emontx1/temp1. Message: 0
12:08:26.738 → Message arrived on topic: emon/ardAC/Wh. Message: 1820
12:08:26.834 → Message arrived on topic: emon/ardAC/wTempC. Message: 43
12:08:26.834 → Message arrived on topic: emon/ardAC/immersionW. Message: 1320
12:08:35.236 → Message arrived on topic: emon/emontx1/power1. Message: -73

It is only the topic emon/emontx1 messages that arrive at emoncms.org. I would like to send some of the others but I don’t know where this is configured.

thanks again.

Post your config file to see what is happening (please format correctly - Ctl+E).

What do you expect to see?

Are you sending other data seperatly via MQTT? your seeing that data appear in emoncms locally but not on emoncms.org? That is unfortunately currently a limitation, emonHub is just publishing on MQTT, it doesnt pick up MQTT messages and forward that on to emoncms.org

Sorry, Borpin - where do you want me to use Ctrl-E

You are right TrystanLea. The data gets to MQTT via node-red. It is sent on a serial connection from the arduino that I have controlling the immersion heater and measuring its temperature, etc… Node-red sends it to MQTT

So, how would I include this data on emonHub and the packets it sends to emoncms.org, please?

thanks

Unfortunately there is no mechanism for this yet, apart form the slightly cumbersome use of the sync module for a periodic (e.g 1 per day) upload, it’s a real missing feature that it would be good to have a solution for.

You can set up a sync module cron upload e.g

crontab -e

then add the line:

32 1 * * * php /opt/emoncms/modules/sync/sync_upload.php > /dev/null

that will upload your latest feed data to the configured remote server at 1:32am every morning, please select a random time (or times) so that we dont get everyone uploading at the same time :slight_smile:

This is how I upload my heat pump data for the public dashboard e.g Emoncms - app view dashboard

You need to do one initial sync manually of the feeds that you wish to upload from the Setup > Sync > feeds interface. You can be selective here, e.g only upload feeds that you want remote access too… reducing emoncms.org costs

Just use the emoncms Node-RED node to send to emoncms.org node-red-node-emoncms (node) - Node-RED

1 Like

great that sounds easiest!

Yes, thank you, I have installed the node-red emon node

the I am using emoncms Push , “valid Json object” input type, and the input is
{“tempC”: 33.94,“powerW”: 1050,“Wh”: 1568}
which is what I get if I connect a debug node directly to the serial in from the Arduino,

the node is ardAC
the properties are:

the server base url is https://emoncms.org/
the api key is got from my emoncms.org account for read/write

the name is ardAC

but I am getting an error

09/12/2022, 11:30:00node: Emoncms Push
msg : Object
object
payload: “Error decoding JSON string (invalid or too deeply nested)”
port: “/dev/ttyUSB0”
_msgid: “7ffa4afb36e0191e”
urlsent: “https://emoncms.org//input/post?fulljson=”{"tempC": 34.94,"powerW": 1140,"Wh": 53}\r\n"&node=ardAC&apikey=xxxxxxxxxxxx"
topic: “http response”
rc: 200
warning: “ERROR: API Call Failed”

This suggests to me that the data arriving is not the Json {“tempC”: 33.94,“powerW”: 1050,“Wh”: 1568} which is what I get if I connect a debug node directly to the serial in from the Arduino, but it has other things in it beside the payload. I presume the payload is {“tempC”: 33.94,“powerW”: 1050,“Wh”: 1568} but /dev/ttyUSB0 is where it comes from.

Perhaps I am confusing a javascript object with a json. Which one of those is {“tempC”: 33.94,“powerW”: 1050,“Wh”: 1568}

or is there another problem.

If I take that url, clean it up a bit so that it reads:

https://emoncms.org/input/post?fulljson={"tempC": 34.94,"powerW": 1140,"Wh": 53}&node=ardAC&apikey=APIKEY

APIKEY replaced with my test account, that works fine for me.

Is it something to do with the nodered emoncms plugin not forming the URL correctly?

This I suspect is the problem and the quotes round the JSON string itself.

Try not to post your API Key.

Note it says ‘Object’ not string (it is a long time since I looked at the node). To be sure you could pass it into a JSON node first. It should strip out anything odd.