Problem setting up EmonCMS Sync - PHP errors

Emoncms
,
1

Following on from the recently introduced billing for cloud EmonCMS, I’ve decided to move some of my non-critical feeds (the ones I’m tinkering with) to my local instance of EmonCMS. It all seemed pretty simple using the Sync module. I’ve updated EmonCMS and installed the Sync Module from GIT using the wiki page, it seems to have installed OK, as the Sync option now appears on the local EmonCMS web page, however clicking it give me -

Warning: Illegal string offset 'content' in /var/www/emoncms/index.php on line 189

Warning: Illegal string offset 'content' in /var/www/emoncms/index.php on line 213

Warning: Illegal string offset 'content' in /var/www/emoncms/index.php on line 219

Warning: Illegal string offset 'route' in /var/www/emoncms/index.php on line 226

Catchable fatal error: Object of class Route could not be converted to string in /var/www/emoncms/index.php on line 226

Any ideas?

2

Those line numbers do not align with the latest version, it looks like you are running a version that date back to early Febuary, can you try updating just in case it is no longer an issue?

I suspect you have v9.8.28? latest is v9.8.30.

It looks like the index.php is not converting the components of the url as an array, but as strings instead. If the issue remains after an update (incl an amin/db update and a browser refresh with ctrl-f5) you could check the browser console (ctrl-f12) for more error messages and check what url is being requested when you select “sync”.

3

Hi Paul,
Now I’m getting somewhere (and for others who may run into these issues) -
You were correct, in that I was indeed running version 9.8.28.
I would have updated this with -

cd /var/www/emoncms && git pull

But the paranoiac in me made me zip a copy of the /var/www/emoncms first, which has handy as it fell over as the system partition was full.

This turned out to be the /var/cache/apt/archives/ folder using 40% of the 4GB parition.

sudo apt-get clean

sorted that out. Then the git pull worked.
Next going to the admin page, I couldn’t log in. This needed the

/var/www/emoncms/settings.php

file updating, see this page - https://github.com/emoncms/emoncms/blob/master/docs/RaspberryPi/general.md

I also needed to enable Redis in two places, low write mode and MQTT to get things working again.
And I had to change the phptimeseries/phpfina/phpfiwa folders to point to the /home/pi/data/ rather than /var/lib/

Next the feedwriter serivce was not started, the only way that worked to start it was -

sudo systemctl restart feedwriter.service

Which finally allowed me to access the Sync webpage and click the Download button 150 times. Excellent.

There was a glitch for a while - in that most of the feeds where green/yellow for up-to-date/slightly-behind, several of them were just white and the download button had disappeared. These sorted themselves out in 5 mins or so.

So, my only problem(s) how is that the Sync page has four tabs, Inputs/Feeds/Dashboards/+New. Sadly all four of them show me the Feeds page. Any ideas how I get the Dashboards to download and see my inputs?

Cheers,
Lee

4

Apologies, I realise I did not push up the last bit of code that I had added to make this work! Its now uploaded so if you update the sync module it will now show the inputs and dashboard interface.

5

Hi Trystan,
I’ve updated the sync module and that’s allowed me to download the feeds and the dashboards, excellent!

However, the feeds are all have a big red “Error” icon next to them. When I edit a feed, it says something like “Feedid 119443 does not exists or was deleted” in a red icon. Is this by design?
I appreciate that 119443 is the Feed ID on EmonCMS.org rather than my local system…

Also, looking on the Dashboards, everything apart from the text is missing?

Thanks for your time,
Lee

6

In the ongoing quest to move most of my data to a a local copy on EmonCMS…

I’ve discovered that I needed to enable the inputs into the local EmonCMS in EmonHub.

Uncomment the URL to the local instance on EmonCMS.
Insert the Write API key from the local My Account tab.

This enables data to flow to the local as well as the remote EmonCMS.
However, I’m not getting this error in the EMonHub log…

Notice:  Undefined index: time in /var/www/emoncms/Modules/process/process_processlist.php on line 288

Notice:  Undefined index: time in /var/www/emoncms/Modules/process/process_processlist.php on line 288

Warning:  Cannot modify header information - headers already sent by (output started at /var/www/emoncms/Modules/process/process_processlist.php:288) in /var/www/emoncms/index.php on line 244

Which is this line I think? But my PHP is weak…

$last_time = $last[‘time’]*1;

No data seems to be coming through to the Input page.
This is turning out be to trickier than I thought… :grinning:

7

Sorry I do not know offhand how that comes about.

There is a check and correction to handle the value not being found when a time:value pair isn’t returned, but the time appears to be left to fail, I do not know if that is by design or why only the value is handled.

I have not seen this error before, despite this being a very common input process, so I am inclined to think you may have unearthed an unusual or unique issue which maybe down to your specific position/use case, but none the less this seems to be a loose end that needs addressing in a more exact way.

But why is there no time:value in your case? I this a new feed or posting to a “synced” feed for the first time?

8

Hi Paul,

I’ve installed the Sync module and managed to make it run. All the inputs and feeds have been created by the Sync module, and clicking the Download icon brings the local data up to date (I can see a load of disk space has been used). The Inputs have the same node number, Key and Description, but the Process List has a red oval which says “Error”. Clicking the Spanner icon, the process list is there, but the Arg says “Feedid xxxxxx does not exist or was deleted”. This is because the Feedid is the one from EmonCMS.org which does not exist on the local install.
I did a test on one of the nodes to change the process list arg to the new local Feedid. This new feed does not get updated, even though I can see the calls to the local install in the EmonHub log.
The data is still being logged to EmonCMS.org just fine.

9

It looks like that is a work in progress!

as is the dashboard syncing

Is the input seen to be updating from the inputs page? Has the red error message gone?

What method are you using to post to the local emoncms? http or mqtt? (ie is this a emonPi type install?)

Which version of emonhub are you using?

Is this a low-write emoncms install? is the feedwriter working?

10

Hi Paul,

Is the input seen to be updating from the inputs page? Has the red error message gone?
I’ve only updated the feeds from one device, the EmonHub log gives those errors, nothing gets updated and the feeds gets further and further out of sync (according to the sync module).

What method are you using to post to the local emoncms? http or mqtt? (ie is this a emonPi type install?)
I’m using http, this was an EmonSD install from the “EmonSD-03May16” image, that’s been brought up to date from GIT, as per the instructions in the wiki.

Which version of emonhub are you using?
Not sure how to tell, the Emoncms Version is low-write 9.8.30 | 2018.05.08

Is this a low-write emoncms install?
Yes, as above.

Is the feedwriter working?
The service is running yes, when it wasn’t running the Sync module didn’t work either. That seems to be working now.

11

Do you mean emonCMS.log?

The emonPi image has used MQTT to post to local emoncms since the emonPi was launched mid-2015

Use command /path/to/emonhub.py --version or check the emonHUB.log following a restart of emonhub.

I don’t really see any connection there.

12

Hi Paul,

No, the EmonHub log, on the EmonCMS/Setup/EmonHub local webpage, in the Log windows-

Screenshot%20from%202018-06-02%2021-42-13
Screenshot%20from%202018-06-02%2021-42-131006×839 145 KB

Odd, I used http, because in was in the (May2016?) EmonHub config file -

[[emoncmslocal]]
Type = EmonHubEmoncmsHTTPInterfacer
[[[init_settings]]]
[[[runtimesettings]]]
pubchannels = ToRFM12,
subchannels = ToEmonCMS,
url = http://localhost/emoncms
apikey = f67a775da7cfe3872c07ec1b7a2af337
senddata = 1 # Enable sending data to Emoncms.org
sendstatus = 1 # Enable sending WAN IP to Emoncms.org MyIP > Emoncms - myip list
sendinterval = 0 # Bulk send interval to Emoncms.org in seconds

I just uncommented the url and added the api key.

EmonHub version -

pi@emonpi:~/emonhub $ /usr/share/emonhub/emonhub.py --version
emonHub emonHub emon-pi variant v2.1.1

As for the feedwriter, I think the Sync module needs it write the data coming from EmonCMS.org to local storage. After I installed the Sync module (and updated it) it wouldn’t work as the feedwriter service wasn’t running.

13

Right, I think I’ve cracked it. The PHP error above is caused by a missing value going to the “Power to kWh” function. The reason that value is missing is because the input going to it doesn’t exist. Obvious really.

The reason behind all of this that the inputs are broken because the feeds IDs have been recreated/renumbered (and the PHP error above is because it’s converting to kWh). Where in the new local input it says to log to feed 120000 (as it does on EmonCMS.org), this doesn’t exist as the feed ID is now 199.

Rather than having to remap all the processes to the newly created (and numbered) feeds, why not just have the same feed IDs on the local system as on the remote system?

Everything would then tie up and “just work” ?

14

ok, I think we’re getting crossed lines here because yes the screenshot you show now is emonhub, but the errors you’ve previously mentioned are PHP errors, so they didn’t come from that screen.

Ok so you are already using http, is the mqtt enabled as well? It looks like it is enabled in the emonhub.log

Is there any more emonhub.log you can show us?

You’re right that make sense now, I thought you meant the feedwriter was affecting the sync module operation directly, yes for the sync module to work, emoncms must be working and able to write to the feeds, the sync module would work just fine on a non-low-write emoncms instance without feedwriter running.

Sort of, it is because of the change of feed ids, but it is not because the input or the input value doesn’t exist. It’s because the feed the input want to save to doesn’t exist (because it’s the old feed id) and there is no “catch” for the case where there is no previous time found.

That is how the usefulscripts/backup script works, but that isn’t ideal either as you cannot guarantee that feedids won’t change at one end or the other over time, it only works once or until changes are made, not as a ongoing “synchronization”, it won’t even work once if the feedids you need are not available. For example I run a server and I couldn’t use the usefulscripts/backup to backup my earliest emoncms.org accounts to my existing server as the feedids would clash, I had to start another emoncms instance just for the stuff I moved off emoncmsorg. I now have 3 servers and emoncms.org accounts that all use the same range of feeids for different accounts.

The sync module, device module and the dashboard module all use tag:feedname cominations in place of feedids in places, so there is a trend moving away from using feedids which is good in some aspects, but since there is no control over the “uniqueness” of the tag:feed combo, so it could come unstuck at some point.

15

Hi Paul,

Those PHP errors were from that webpage EMonHub log.

image
image1171×859 85.3 KB

What else are you after? The rest of the log looks ‘normal’.

I used http as it was in the emonhub config as I mentioned before, I enabled mqtt as I use that for other stuff (status from Sonoff modules & sending mqtt commands).

Yes, you’re right about the ID’s, I’ve looked again at the Dashboards, the dials/widgets are there but are invisible, when you click on them, you can edit them, but it says it’s a new virtual feed instead of the feed ID/name.

OK, I see what you mean about the FeedID vs Tag:Feedname, but the Process List is clearly interested in the Feedid.

image
image1098×660 85.7 KB

If the Feedid was reused from EmonCMS.org, it would all just work, I see your point about backwards compatibility, but it’s got to be easier that retrofitting new Feedids or manually editing Process Lists - I’ll several hundred operations in mine.

16

Thanks for the scren dump, those errors are part of the response from emoncms which is being logged, they are not supposed to be there and they are not errors as reported by emonhub. Check your emoncms settings.php, I think there is a setting about displaying fatal errors that should be false. That might clear up the emonhub.logs a bit.

I was wanting to see more emonhub.log to try and confirm if the data is being passed to emoncms successfully and by what method(s). You say you are using mqtt too for other targets, what is stopping that data also going to emoncms? have you disabled the mqtt_input? or changed the topics?

If the rest of the log “looks normal” there should be some green “updated at” time values on the inputs page. no?

Do you have any inputs with no errors (ie fixed the feed ids) that can update if the inputs are posted?

Having said that, I wonder if the whole input payload is discarded when an input error occurs, I wonder if you need to do all the feedids for a payload to be delivered succesfully, I’m not sure.

17

Hi Paul,

I’ve updated the /var/www/emoncms/settings.php to switch off fatal errors -

// Show all fatal PHP errors
$display_errors = false;

That seems to stopped the PHP errors in the emonhub log.
Speaking of which, here’s some logging…

2018-06-03 17:27:30,678 DEBUG    emoncmsorg Buffer size: 8
2018-06-03 17:27:30,680 INFO     emoncmsorg sending: https://emoncms.org/input/bulk.json?apikey=E-M-O-N-C-M-S-A-P-I-K-E-Y&data=[[1528043222.080234,10,-74,466,0,24473,-74],[1528043227.778881,10,-76,460,0,24553,-75],[1528043233.438892,10,-77,454,0,24458,-74],[1528043234.96174,18,2275,3911,-67],[1528043235.576045,19,1887,3693,-85],[1528043239.197304,10,-53,454,0,24506,-74],[1528043244.805268,8,0,3705,0,-63],[1528043244.917921,10,-70,468,0,24408,-74]]&sentat=1528043250
2018-06-03 17:27:30,697 DEBUG    RFM12Pi    18435 NEW FRAME : OK 10 162 255 174 1 0 0 106 95 (-75)
2018-06-03 17:27:30,700 DEBUG    RFM12Pi    18435 Timestamp : 1528043250.7
2018-06-03 17:27:30,703 DEBUG    RFM12Pi    18435 From Node : 10
2018-06-03 17:27:30,704 DEBUG    RFM12Pi    18435    Values : [-94, 430, 0, 24426]
2018-06-03 17:27:30,706 DEBUG    RFM12Pi    18435      RSSI : -75
2018-06-03 17:27:30,707 DEBUG    RFM12Pi    18435 Sent to channel(start)' : ToEmonCMS
2018-06-03 17:27:30,708 DEBUG    RFM12Pi    18435 Sent to channel(end)' : ToEmonCMS
2018-06-03 17:27:30,796 DEBUG    MQTT       Publishing: emon/emontx/power1 -94
2018-06-03 17:27:30,799 DEBUG    MQTT       Publishing: emon/emontx/power2 430
2018-06-03 17:27:30,801 DEBUG    MQTT       Publishing: emon/emontx/power3 0
2018-06-03 17:27:30,803 DEBUG    MQTT       Publishing: emon/emontx/vrms 24426
2018-06-03 17:27:30,805 DEBUG    MQTT       Publishing: emon/emontx/rssi -75
2018-06-03 17:27:30,808 INFO     MQTT       Publishing: emonhub/rx/10/values -94,430,0,24426,-75
2018-06-03 17:27:30,864 DEBUG    emoncmsorg acknowledged receipt with 'ok' from https://emoncms.org
2018-06-03 17:27:36,392 DEBUG    RFM12Pi    18436 NEW FRAME : OK 10 171 255 206 1 0 0 52 95 (-75)

from the emonhub.log, you can see it http’ing data to emoncms.org, and the ok received back. MQTT being posted out, so NodeRed can see what’s going on.

2018-06-03 18:05:59,331 DEBUG    emoncmslocal Buffer size: 10
2018-06-03 18:05:59,333 INFO     emoncmslocal sending: http://10.0.0.4/emoncms/input/bulk.json?apikey=E-M-O-N-C-M-S-A-P-I-K-E-Y&data=[[1528045533.307757,10,-927,1976,0,24241,-75],[1528045534.260774,8,0,3705,0,-63],[1528045537.844322,18,2275,3911,-63],[1528045538.998239,10,-890,1987,0,24106,-76],[1528045543.318852,19,1906,3693,-83],[1528045544.739604,10,-839,1945,0,24147,-75],[1528045545.490433,17,2556,3754,-84],[1528045546.543556,24,2406,3792,-75],[1528045550.461262,10,971,1416,0,24033,-75],[1528045556.263443,10,-401,1603,0,24037,-74]]&sentat=1528045559
2018-06-03 18:06:00,068 DEBUG    emoncmslocal acknowledged receipt with 'ok' from http://10.0.0.4/emoncms
2018-06-03 18:06:01,949 DEBUG    RFM12Pi    52 NEW FRAME : OK 10 85 252 176 5 0 0 118 94 (-72)

So the input is picked up, but it’s not processed as the the Process List still has errors. Editing the Process List, and replacing the unknown feed ids creates a working feed. Looking at the sync, those corrected processes causes a glitch in the sync where the local feeds are now running ahead of the remote feeds by an ever increasing number of points. I’d have thought they’d have been in step?

Which brings us back to the Feed Name vs the Feed ID. I undrstand the reasoning behind using the Feed Name, but it’s not using that either, so the feed is broken either way?

18

Can you say what errors? If you have removed the “feed doesn’t exist” issues by changing the feedid, what are the new errors?

How are you changing the feed ids?

I assume you are editing the processlist by reelecting the correct feed from those listed, is that correct?

Can you confirm if you had done all the processes in all the input for nodes 10, 8, 18, 19, 17 and 24 ? as (using your example) that might be what is needed for that full payload to be delivered without an exception.

Try setting batchsize = 1 and interval = 0 in the emoncmslocal runtimesettings, this will allow emonhub to post one frame at a time to emoncms ie to just one node at in any one request.

I can’t say I can picture what is nor what should be happening there, I think @TrystanLea needs to assist, I’m beyond my understanding of the sync module now.

19

I’ve now added feed remapping to the inputs download feature in the sync module. You do need to make sure you download the feeds first so that it knows what the local feed id’s it needs to remap to.

If your writing to both your local and remote feeds simultaneously the synchronisation will break. The idea is to write to either the remote or the local installation and then sync up or down the changes from the installation where your writing the data.

At present emoncms.org only supports data download via the sync module and not yet data upload. Im focusing on the download data route for now as it helps those keen to migrate form emoncms.org to a local installation in the short term.

20

Hi chaps, sorry for the delay, I’ve been going around in circles!

@pb66 Yes, if I chop out the incorrect Feeds from the Process Lists and replace them with the new equivalents from the local feeds, the error disappears and it seems to work fine. The problem is I’ve got 150+ feeds and 100’s of process items creating the data. And I can’t change stuff fast enough to not loose data.

@TrystanLea I updated the Sync module, then deleted all the local inputs & feeds.
I synced all the feeds, and brought them up to date
But syncing the inputs now gives blank Process List details back.

Any ideas?

I kind of guessed the synchronisation between remote/local would break things, I guess it needs a carefully controlled transition from remote to local storage.