Problem setting up EmonCMS Sync - PHP errors

I’ve added a little error reporting to the feed re-mapping part of the sync module. I wonder if you could retry after updating the module again and copy and paste the first 10 lines of the input import log?

Hi @TrystanLea, where is this input import log?

It looks like it should be at

~/data/emoncms-sync.log

Ah, I found that earlier, but it only had this in it…

SYNC: Starting
SYNC: Home dir: /home/pi
SYNC: Connected to Redis

Seems like I’ve messed something up?

cd ~/sync
git pull

Should bring the sync module up to date shouldn’t it?

I’ve deleted the Input Nodes and run the sync again, the nodes reappear but as before the Process Lists are empty.

Should look more like this…

Looking at the code again, I think the sync module logs to 2 locations, the log I refereed to looks like it is just for the activation of the service, there seems to be an “emonLogger” instance too so there should be some output in the usual /var/log folder, although it isn’t entirely clear to me as it looks like the logging @TrystanLea added earlier is output to the screen or the request reply rather than a log file

Although, if you look at line 255 it also looks like he’s added a line for the processlist too, possibly an omission in the previous commits? So that might explain why your processlists are not being synced. However that would then raise the question, why was the update not pulled in with the git pull? Did you check the output of the git pull? Did it confirm the changes were pulled in ok?

Try the git pull again and if you are still getting the error can you confirm if that line is present in your local copy?

Hi Paul,
I git pulled sync again and there were no updates.

The patches look like they’ve been applied :

As for the log file, in /var/log/ I have -

The only stuff in the emoncms.log is -

pi@emonpi:/var/log $ more emoncms.log
2018-06-05 17:45:30.391|WARN|phpmqtt_input.php|Not connected, retrying connection
2018-06-05 17:45:30.420|WARN|phpmqtt_input.php|Connecting to MQTT server: Connection Accepted.: code: 0
2018-06-05 17:45:40.858|WARN|phpmqtt_input.php|Not connected, retrying connection
2018-06-05 17:45:40.887|WARN|phpmqtt_input.php|Connecting to MQTT server: Connection Accepted.: code: 0
2018-06-05 18:04:56.514|WARN|sync_model.php|remote save
2018-06-05 18:04:56.675|WARN|sync_model.php|{"success":true,"userid":2798,"apikey_write":"xxx","apikey_read":"xxx"}
2018-06-05 20:09:12.951|WARN|sync_model.php|remote save
2018-06-05 20:09:13.131|WARN|sync_model.php|{"success":true,"userid":2798,"apikey_write":"xxx","apikey_read":"xxx"}
2018-06-06 17:12:06.692|WARN|sync_model.php|remote save
2018-06-06 17:12:14.361|WARN|sync_model.php|remote save
2018-06-06 17:12:14.497|WARN|sync_model.php|{"success":true,"userid":2798,"apikey_write":"xxx","apikey_read":"xxx"}
pi@emonpi:/var/log $

But it’s from the day before the new code was loaded, so I’m not sure where this new logging is going?

The import log is shown on the page for importing inputs, it’s the gray box below the download button:

@TrystanLea, When looking at the code I noticed that the feed mapping only uses feed name not a feed name and tag combo.

Wondering if that’s possibly because duplicate feed name are not supported on emoncms.org I tested and found they are supported, so it is very much possible that users have multiple feeds of the same name, which I assume would cause issues when mapping the process list, but also when re-running the sync process as (for example) if you have 2 “solar” feeds, although they might sync ok first run (or at least the first of the 2 should) the process mapping and subsequent re-sync’s will both act on either the first “solar” or the last “solar” depending on how the code is written (I haven’t looked at that yet).

However, it doesn’t look like it is a straight forward case of checking for a tag:feedname combo either, as I was rather surprised to discover it is also possible to have duplicate feednames with the same tag

so some form of checking for a unique tag : feed name combo is required.

This is also going to be an issue for the use of tag:feedname combo id’s in the dash module and in the device module too. I have already experienced some issues with the device module where is appears the tag is sometimes ignored resulting in omitted or incorrect feed creation.

Good point, unique tag:names are enforced at feed creation but not subsequently! I will add this as an issue to correct. I will update the sync module to check for the tag:name combination. I doubt there are a large number of feeds with the same tag:names, I will add a notice to the sync module instructions that all feeds must have unique tag:names

1 Like

And possibly add a note as it may not be obvious to all, that any change in feedname and/or tags must be done at both ends and match before re-running the sync operation, otherwise duplicate feeds might be created and/or input processing might break.

What is the position if input processing is changed between sync’s? is it over written?

Ah, there was no grey logging box, just the white screen.

That’ll be why I didn’t see it. Is it just the sync package which needed updating? Or something else as well?

Am I the only person to have actually used the Sync module in anger? :confused:
Or, am I the only person to attempt to upgrade an older EmonCMS system and use it in anger?

Should I therefore just reinstall from scratch and try from there? :thinking:
But shouldn’t the lastest GIT updates have made them functionally identical?

Not sure what to do now???

I am trying to use sync to create a clone of my emoncms.org setup I have just updated my sync module however when I download the ‘inputs’ I get
“Notice: Uninitialized string offset: 1 in /home/pi/sync/sync-module/sync_controller.php on line 273
Fatal error: Cannot access empty property in /home/pi/sync/sync-module/sync_controller.php on line 273”

All I can determine from this is that this line fails :-

$process_type = $process_list->$process_id[1];

HI Roger,
I’ve not seen that error. The only thing I can suggest is to update the version of the Sync module you have, and made sure you’re running the latest version of EmonCMS.

Hi,

I downloaded the image “emonSD-26Oct17.img” on 11 May and I had already updated Sync today using git pull before my initial post. I have just done a git pull on emonpi then run “./emoncmsupdate” (I guess this last step probably inherently repeats the previous step?). So as far as I can tell I am fully up to date. Maybe this is data dependent, so I would be happy for you to access my data if that would help.

Is there a manual way of getting the “inputs” data from emoncms.org onto my emoncms raspberry pi?

Rog

Thanks @rog, Im not sure how that bug escaped me, I have now fixed it. If you pull in the latest sync module it should work.

I am running raspbian stretch dated April 18, 2018.
I have installed emoncms (OpenEnergyMonitor.org | 9.8.30 | 2018.05.08) on a Pi 3, together with the sync module updated to the latest changes.

From the local server page, I can see the sync module. I see all my feeds/inputs.
I have 3 menu entries on the side: “Inputs”, “Feeds” and “Dashboards”.

My problem: If I try to download a feed, I just get a file with a couple of bytes, basically an empty file.

I checked the emoncms.log file. Everything looks Ok, no error messages, something like this when clicking on the download button:

2018-06-21 17:24:51.782|WARN|sync_model.php|remote save
2018-06-21 17:24:52.248|WARN|sync_model.php|{“success”:true,“userid”:9143,“apikey_write”:“my_write_key”,“apikey_read”:“my_read_key”}

Anything I need to verify to successfully download my data?

Thanks

Walter

I have started the script manually from the sync folder:

sudo php sync_run.php
SYNC: Starting
SYNC: Home dir: /home/pi
SYNC: Connected to Redis

What should I typically get in the console when running the above script?

Thanks

Hello @daturach it sounds like the feeds are not registering in the queue of feeds to download stored in redis ‘sync-queue’. Did you install the service runner as documented in the sync module readme? GitHub - emoncms/sync: In development: Download or upload emoncms feed data between local and remote emoncms server's

Hello @TrystanLea . Thanks for your quick answer. No I haven’t. I thought it was only for the EmonPi users.
Anyway, I will install it and report back.
Thanks again for this great tool.
Walter