Long term, always on Emoncms dashboard stability testing

Hello @pomonabill220 I haven’t made further progress with this since our last exchange Im afraid, so the software should be mostly the same. The changes in the 24July20 image are mostly on the backend of the software stack rather than front end dashboard.

Well I would sure like to find out what is going on.
I ran the dashboard on a Win10 tablet in firefox, and while FF didn’t crash, it stopped running. Everything on the dashboard quit updating.
Took a look at taskmanager, and FF had consume all available memory. It had started using about 350Meg, and grew to about 2Gig. The numbers are approx., but it had used all available memory. I seem to remember this happening before.

I downloaded chromium and am now running the dashboard on my Win10 tablet in Chromium.
We’ll see what happens.

The android tablet still crashes in about 12 hours running the dashboard.
So I conclude that it is not my Emonpi, is not my network, IS my dashboard eating up all available memory after running for a while.

I am having a hard time believing that I am the only one with this problem.
I have seen a few users that have created extensive dashboards, and they don’t seem to be having any problem.

Win10 tablet memory usage is up to 451MB at 1354 on 9/27.

Just a chromium/win 10 tablet update.
Task manager indicates chromium memory usage at 546Meg.
Dashboard is still updating though.

1800 on 9/27
memory usage at 630MB.
still updating.

at 2030
memory at 755Mb
still updating

at 2230
memory at 813Mb
still updating

at 0020
memory at 875Mb
still updating
________ 9/28

at 0800
memory at 1080 Mb
still updating graphs

at 1230
memory at 1450 Mb
graphs are updating very slow

at 1940
memory at 1740Mb
graphs still updating.

---------- 9/29

at 0750
memory at 2021Mb
graphs updating

at 1640 on 9/29, Just found Chromium on the win 10 tablet had crashed running my dashboard.
The error was out of memory! surprise surprise.
it doesn’t look my tablet (Android and Win 10), or my network which is wired or the browser.
I guess I will just have to assume that I am the only one having this problem and no one else can reproduce it.
Too bad!

Hello @pomonabill220, it does seem strange that this is not a more widely reported problem. Perhaps it indicated that there are not that many people running the dashboards as always on displays?

I realise the ‘stability_testing’ branch of the dashboard module was not yet merged into the dashboard master branch. So when you updated to 24July20 you actually went back to an older version to what I think you were testing before.

I have now merged these changes into the master and stable branches, could you run the Emoncms Update tool to pull these changes in and then run your tests again?

This specifically merges in the following change:

Hi Trystan.
I ran the update (full) on my emonpi and rebooted the Pi.
I am now running the same dashboard on the same win10 tablet and at 1200 it started at 223 Mb.
10 minutes later, chromium crashed.
This is the error in event viewer:
The program chrome.exe version 87.0.4276.0 stopped interacting with Windows and was closed. To see if more information about the problem is available, check the problem history in the Security and Maintenance control panel.
Process ID: c50
Start Time: 01d6975cccb64ea0
Termination Time: 4294967295
Application Path: C:\win 10 stuff\chrome-win\chrome-win\chrome.exe
Report Id: 508b4a4b-0174-44ed-99dd-d1f747e932e7
Faulting package full name:
Faulting package-relative application ID:
Hang type: Top level window is idle

There are more errors related to chromium, and I can get them if you need them.

I am going to try firefox next. Started at 1218 with 365 Mb in use.

I rebooted my win10 tablet and my Android tablet just to see and so far… the dashboard on both machines is running.
Task manager in win 10 have chromium running around 430Mb (plus or minus) so we’ll see how it does in the long term.

2 hours later (1622) memory usage is up to 573 Mb. The increase seems alot slower than it was, but I will keep watching.

at 2050 memory usage is up to 794 Mb. so it still is increasing, but windows might cleanup memory… will let it run and see what happens.

at 0940 memory usage is at 1270 Mb
graphs still updating.

at 1444 memory usage is at 1400 Mb
graphs still updating.

at 1920 memory is at 1500 Mb
graphs still updating.

2140 memory is at 1544 Mb
graphs updating.

at 0200 memory is at 2000

at 1100 memory at 2050 Mb

at 1300 memory at 2100 Mb
Graph updates VERY slow… about 2-3 minute delay.
Gauges update correctly… every 3-5 seconds delay.

Well at about 2200, my win10 tablet running my dashboard, crashed the dashboard.
Last I checked, memory with chromium was at 2300 Mb.
The error code that was being displayed in chromium was “out of memory” and task manager indicated chromium was using 51 Mb (after the crash).

So I got about 3 days of continuous dashboard running, but it still eats up memory and doesn’t release it.

I’m not clear what the data you posted refers to. Is it just the Windows Tablet? What about the Android tablet. If android as well, is this an old tablet?

Can you post a screenshot of the dashboard to see what widgets etc you are using?

The first picture is the tablet’s info. NOT a high end tablet but is isn’t an old one either.
The second picture is what I have been running. Chromium in the background is running my dashboard, and task manager is in the foreground with the amount of memory that Chromium is using.

My list that I created above, is ONLY the Win 10 tablet numbers. I did not include the android tablet numbers to avoid confusion.

Each listing gives the time in 24 hour time, that amount of memory that task manager is reporting that Chromium is using, and whether or not the graphs are updating. This has happened in the past where the gauges are updating, but the graphs do not, or there is a LONG delay.

I was going to “log” my android tablet, but i only did the Win 10 tablet.
The android tablet still crashed, possibly due to the same out of memory problem.

I was thinking more of the Android one.

But you mentioned Android in the preamble; hence the question.

I think you are fairly edge case here. Do you have a PC? Does it exhibit the same behaviour?

All I can suggest is try different combinations of widget and see if it is one particular one. Run an empty dashboard - does it still happen?

Memory leaks like this are hard to pin down I believe.

Yes, I was going to run android but changed my mind… sorry.
What I did do is after I posted all this, I restarted Chromium and reloaded my dashboard.
and task manager to monitor memory usage.
NOW 3 hours later, task manager is still reporting about 235 Mb memory usage and doesn’t seem to increase!!! WTF?
I didn’t restart the win10 tablet, just restarted chromium and my dashboard!

I do have a PC but it is running other things, and i didn’t want to interfere with this test. i could run this on a win 7 laptop just to see. I also have a couple of android tablets that I could run.
I just chose the win10 because of task manager displaying memory usage, but the win 7 can do that also, so maybe I will try that.

Well about 5 hours later running my dashboard on the win 10 tablet, memory usage is 248 Mb (at 1905)! don’t know what I did different, but it seems to be behaving!
Will keep it running just to see, but sure would like to know what is different!

Another update 10/4 at 0012 hrs.
memory usage at 255 Mb!! kinda bouncing around up to 259 down to 248 Mb.

10/4 update
at 0905 memory usage at 260 Mb on win10 tablet.
android tablet browser crashed though.

One thing I have noticed on win10 tablet, task manager has a column for Power usage and it is “very high” but the power usage trend is low. Don’t know that that indicates, if anything at all. CPU usage is also about 29% but this is a slow tablet with an Atom processor.
I am getting a Surface Pro 7 with an i5 so I will try this on that tablet.

At 1445 memory is at 276 Mb… looking good but I don’t know what I did different???

WOW!! at 2320 memory is at 284 Mb!!!
Tomorrow I might try exiting chromium and restarting properly and see what memory does.
The difference between that and what has been running now is that chromium had crashed but I just reloaded the page without restarting the browser.
can’t imagine what would be different but we’ll see.

update 10/5 at 1000 Hrs.
memory usage at 300 Mb. very slow increase in memory usage but still running!

It is possible that the cached javascript in your browser for the dashboard finally got refreshed from the “Full Update” you did before the previous tests, which could explain why the previous test didn’t change the behaviour, even though you’d done the update.

That does make sense.
right now, at 2030, memory usage is at 306 Mb.
So what I will do is close the emonpi window, close chromium, reboot win 10.
Then open chromium and re load my dashboard (with windows task manager), and start the memory usage test again.
It would be wonderful if memory usage would perform like it has the last couple of days, and chromium doesn’t crash!

I’ll keep updating.
wish me luck!

Started new test.
Chromium is using 232 Mb (chromium with nothing loaded, no window, uses about 125 Mb just for reference).
this started at 2040 Hrs.

2 hrs later… 2245 Hrs. memory is at a WHOPPING 226 Mb! WOW!
this is getting EXCITING!!!

at 0040 on 10/6 we are screaming along with 232 Mb memory! I can’t STAND it!!!
OH and by the way… the Android tablet is still running after over 12 hours!!! I don’t know if I can handle much more DRAMA!!!
(just a side note…don’t confuse the win 10 tablet with the android tablet).

on 10/6 at 0910 memory usage is at 247 Mb! Still holding and running!

Android tablet still running also.

At 1850 memory usage is only 258 Mb!
Android still running too.

at 2245 memory is at 252 Mb in Win 10 tablet.
Android still running

1 Like

at 2020 memory at 270 Mb and Android tablet is also still running.
It looks like the problem has been solved!
What that was I have no idea, but I know that you Linux guys do know!

at 1848 memory at 291 Mb.
Android tablet still running also.

Don’t know if anyone is reading these reports? shall I keep them updated?

Thanks @pomonabill220 good to hear. You reported after updating following my last post on this thread that the dashboard still crashed and then it stopped crashing, what changed? :slight_smile:

@TrystanLea, to answer for Bill:

To which I responded: