Updated EmonCMS today now the graphs are all corrupted

PaulM47 · 2022-04-12T14:23:11.006Z

Ok, I’ve done a definitive test using a second Raspberry Pi 3 on a different sub-net but receiving the same wireless data, fortunately I bought a second wireless receiver a while ago. Both have the 21st July 21 SD card image programmed, the first was updated from a good backup yesterday and has been running ok since, not re-booted or updated. I exported the data from this one and imported it into the second so now both essentially were running with the same data, no corrupted graphs.

I re-booted the second with no problems. I exported the CSV data for the power used during the past day and imported it into a spreadsheet, which shows the same graph as the EmonPi. I then ran a full update, logged out and logged back in, result corrupted graphs. I then exported the CSV data again for the power used during the past day and again exported it into a spreadsheet and again it shows the same (corrupted) graph as the EmonPi.

Not sure about the corruptions I saw yesterday on re-boot, but the update certainly caused the problem. I cannot upload the CSV files as I’m restricted as a new user.

These are the graphs …

PPD1793×804 142 KB

PPD2793×801 98.6 KB

PPD3787×797 110 KB

PPD1 is the first Raspberry Pi 3 that’s been running since yesterday. PPD2 & 3 are the second with the same data after import and update, PPD3 has the fixed axes set to +/- 1000 as it now has negative data

I shall export the data from the second Pi, showing the corrupted data, then re-flash the SD card and re-import the data to see if the raw data is corrupted by the update or if it’s only the graphs…

Robert.Wall · 2022-04-12T14:25:48.337Z

PaulM47:

I cannot upload the CSV files as I’m restricted as a new user.

You should be able to now.

PaulM47 · 2022-04-12T14:39:30.119Z

Thanks, here they are…

EmonPowerPastDay1.csv (29.4 KB)
EmonPowerPastDay2.csv (29.2 KB)

The first is un-corrupted, the second is corrupted.

PaulM47 · 2022-04-12T15:07:00.710Z

PaulM47:

I shall export the data from the second Pi, showing the corrupted data, then re-flash the SD card and re-import the data to see if the raw data is corrupted by the update or if it’s only the graphs…

The re-imported data is not showing as corrupted and displays Ok until the latest full update is run, which is good news. It’s looking to me like extracting the data from the feeds to create the graphs is the problem.

It’s not a browser cache problem either as I’ve been very careful to clear the cache when necessary.

TrystanLea · 2022-04-12T15:28:43.351Z

Robert.Wall:

The timebase for the feeds has not changed? The reporting interval of the sensor nodes should be slightly less than the feed timebase, so the emonTH expects a 60 s timebase and so reports about every 55 s, the emonTx expects a 10 s timebase so reports about every 9.9 s.

It sounds to me that the issue will be related to post interval vs feed interval.

If you put a tick in the box for ‘Average’ for each feed in the ‘Feeds in view’ and then click ‘Reload’ it may show the data that is present. If that works then it is an interval related issue. Perhaps some packets of data are being missed…

I do wonder if we should enable averaging by default on data shown in the graph page view as this is likely to be an issue that will trip a lot of people up, it’s got me a few times… The downside is that it does present a slight performance hit as you zoom out to very long time ranges

Robert.Wall · 2022-04-12T15:41:48.432Z

TrystanLea:

The downside is that it does present a slight performance hit as you zoom out to very long time ranges

Taking a sideways off-topic look… What would be the price of a separate but related database of (say) hourly or daily (or both) averages, automatically created, and automatically used for the graphs as appropriate?

For the 10 s feed, an hourly database represents a 360× speed-up, so you’d use that when the graph spanned more than a month or so.

TrystanLea · 2022-04-12T16:07:02.990Z

Robert.Wall:

What would be the price of a separate but related database of (say) hourly or daily (or both) averages, automatically created, and automatically used for the graphs as appropriate?

I did have that running on emoncms.org until the recent update, it is now all done on the fly, it’s a little slower but a lot simpler in terms of the implementation and ease of code maintainability. That implementation did still require ticking the average box to enable it.

The other issue of course with enabling averaging as default is that you don’t want averaging in all cases, e.g looking at cumulative kWh data

But this issue with some feeds showing as empty when there’s an interval mismatch is also annoying, so I think I need to find some way of improving the experience… perhaps showing an alert box if there is a large % of null values returned…?

cyberelectronics · 2022-04-12T18:04:34.658Z

@PaulM47 I had similar issues, like “corrupt” graphs, few weeks ago (no update, just some power outages).
For me the solution was to disable/enable the checkboxes again, in the Graph section.
This fixed all my graph issues.

emonpi_checkbox1920×1146 160 KB

PaulM47 · 2022-04-12T18:19:17.978Z

cyberelectronics:

For me the solution was to disable/enable the checkboxes again, in the Graph section.
This fixed all my graph issues.

Yep, many thanks, that solved it !!

Update - It’s not a solution - opened the second (updated) Pi this morning and the graphs are corrupted again, had to un-tick and re-tick everything again to see them correctly !! The live Pi is still Ok.

PaulM

PaulM47 · 2022-04-14T06:59:44.307Z

TrystanLea:

If you put a tick in the box for ‘Average’ for each feed in the ‘Feeds in view’ and then click ‘Reload’ it may show the data that is present.

This doesn’t correct anything, the only thing that corrects the graphs is un-ticking and re-ticking the selection box on the left, and that only persists until I log out.

As the problem affects all the graphs, and my accumulated power graph takes on the appearance of the daily power graph when the problem arises, I think I’d be looking for a database pointer initialisation being corrupted when logging in, but being corrected when un-ticking and re-ticking the selection boxes. This has only happened sometime in the past couple of months as before the beginning of February there was no problem. I usually run a full update at the end of the month but for various reasons missed the end of February one so can’t narrow it down any further. Am I the only one seeing this problem ??

TrystanLea · 2022-04-14T14:54:51.872Z

Ah ok, sounds like some of the settings for the graph are not being saved correctly, I will see if I can replicate here, thanks.

borpin · 2022-04-14T19:08:30.152Z

PaulM47:

This doesn’t correct anything, the only thing that corrects the graphs is un-ticking and re-ticking the selection box on the left, and that only persists until I log out.

Have you tried creating the graph from scratch and saving it as a new name (as a test)?

TrystanLea:

Ah ok, sounds like some of the settings for the graph are not being saved correctly,

I’ve got this on a V10 system that I have not updated. If you save the graph with a scale, you can never get it back to ‘auto’.

PaulM47 · 2022-05-01T13:14:09.757Z

Any updates on this problem yet ??

I’m just about to run the end of April archives but won’t be updating Emon if it still destroys the graphs

borpin · 2022-05-02T08:31:24.792Z

What Browser/OS are you using?

PaulM47 · 2022-05-03T21:37:58.624Z

Brave, Firefox and Opera, all displays are the same, on Xubuntu.

borpin · 2022-05-04T06:47:07.064Z

Not that then (there have been issues with MacOS and Safari in the past).

Can you try it on a Windows machine?

PaulM47 · 2022-05-04T09:23:44.566Z

Nope, don’t have any Windoze machines other than a Windoze XP VM, and I’m assuming that’s not relevant

borpin · 2022-05-04T12:25:32.769Z

No, and not a distro I use (actually none of my Linux machines have a GUI on them) so cannot test to see if it is distro specific. Quite possibly it is.

PaulM47 · 2022-06-06T15:20:17.466Z

…to pick this one up again. I’ve just opened emonpi to run through my end of month routine and the graphs are corrupted, I assume because I re-booted it a few days ago to update it’s IP address due to change in my DHCP server. I haven’t run any firmware updates since my last comments so it’s still running the last formal software release.

Laboriously unticking and ticking each graph item corrects the problem and shows the graphs correctly.

Any progress through this past month on this one ??

PaulM

PaulM47 · 2022-12-05T19:11:54.181Z

Updated Emon today and the corrupted graph problem appears to have gone, re-booted without the problem occurring

PaulM