Long term, always on Emoncms dashboard stability testing

I’m trying to help @pomonabill220 again with his ongoing stability issues with emoncms dashboards and I wonder if anyone else has experienced issues with crashing dashboards after 2-7 hours?

It’s been a while since I have personally had an always on dashboard running in my house, I just load emoncms on my phone when I need to now and have an old EmonGLCD in the livingroom giving a live power readout.

Last year we installed a public display for a local hydro scheme in a shop window running a very basic emoncms dashboard including a dial (without tooltips), a couple of feedvalue widgets and an embeded image of the turbine http://emoncms.org/ynniogwen/ynniogwen this is running on a raspberrypi through chromium and has been very reliable, no problems reported at all.

Has anyone else had good or bad experience with the stability of emoncms dashboards for long term always on displays?

A few observations so far:

  • Realtime graphs seem to add a fair bit to the Javascript Heap memory use, almost doubling other graphs and dials on the dashboard that we are testing.
  • Dial tooltip mousemove event listener was getting called on every dial draw resulting in lots of instances of this event listener in the JS heap, we are using this testing branch to remove this feature to test if this improves things https://github.com/emoncms/dashboard/compare/stability_testing (testing ongoing)
  • I’ve made a number of further optimisations to the dial, jguage, battery, feedvalue and other widgets to reduce the draw rate when there is nothing changing Commits · emoncms/dashboard · GitHub (this did not fix the issue for @pomonabill220).
  • jgauge images were being reloaded repeatedly - now fixed dont reload jguage images multiple times · emoncms/dashboard@c9b8652 · GitHub (but dashboard still crashed for Bill after updating to include this fix)
  • Chromium (not Chrome) has significantly lower memory use than Firefox

I was able to replicate a crash using @pomonabill220’s dashboard last year on a cheap tablet here by leaving it overnight but did not get to the bottom of it at the time

2 Likes

If anyone with more experience with memory profiling and finding JS leaks can help find issues that would also be really appreciated. Thanks a lot!

Sorry, I only use Node-RED and Home Assistant for dashboards.

Probably not related but, I only view the dashboard from my mobile phone and had connection issues after 2-7 hrs like you, I would have to reboot the raspberry Pi to make it work again, I fixed mine by putting a better External antenna on the pi as mine sat in a metal cabinet by the power distribution box downstairs in the garage and my wireless router is upstairs.

I would suggest it’s probably worth trying to use a newer version of jQuery before digging too much into other things. I know it’s probably not a small project, but feels pretty overdue (while I understand there’s an element of if it aint broke, don’t fix it…)

1.11.3 is ooold (April 28, 2015). 1.x and 2.x aren’t supported at all and 3.5.0 is current, with 3.0 being quite a few years old too (June 2016)

Many performance and security fixes in the release history between then and now - jQuery - Wikipedia. Things like dropping IE6-8 support in 2.0 (April 2013) giving some significant performance gains.

The jQuery Migrate plugin will probably be useful too

https://jquery.com/upgrade-guide/3.0/
https://jquery.com/upgrade-guide/3.5/

Similarly, upgrading jquery.flot - emoncms/NEWS.md at master · emoncms/emoncms · GitHub says 0.8.3, and 4.2.0 is the latest release - Releases · flot/flot · GitHub

I do know a few very JS orientated people who I can probably poke for profiling JS hints and tips

Thanks @Dimirll @reedy

That’s interesting, I will consider that and note the issues you have created on the repo thanks.
Hopefully it is a straightforward upgrade.

I don’t mind trying to help, but I do note I’m not really a javascript developer.

It should be easy enough to get going though; replace the jquery file with a newer version, and poke around and see if things break

Even if it doesn’t help (much) with the actual issue, it’s probably a worthwile exercise in terms of good general software development practice

1 Like

Getting a little off topic of this thread… But there’s definitely some cleanup, reorg and stuff going to need to happen to make the upgrades cleaner (especially in Lib/flot where numerous files don’t actually belong to flot…)

thanks for the pull requests and issues on the emoncms repo. I think we get the latest version running in a development branch and then see if it makes any difference positive or negative for @pomonabill220.
His latest tests are still crashing , though after a longer time…

It maybe that the whole dashboard module needs a big overhaul, removing the use of iframes, inspecting every widget, moving to a more object based js format rather than all the global scope functions… not something Im currently in the position to do but that’s what Im thinking when I look through the dashboard module code :slight_smile:

That could be really useful, if even just to generate some pointers, perhaps after updating and testing the newer versions of flot and jquery…

Many options to choose from for the next generation of graphs:

1 Like

I know I have been away for a long time, but my Dad just had a bad operation (perforated ulcer at 98!) and I probably am going to loose him, so I am at his house and seeing him every day in rehab. Very sad time for me!

But I have been checking on my dashboard on my tablet and running the one that crashes daily, and so far, it has NOT crashed in days, if not weeks!!!
I did notice that the graphs stopped updating though, and all I had to do to start them up was to drag in the graph and the data updated.
For instance, the graphs had stopped scrolling at 2100 hrs, and I looked at them at about 1000 hrs the next day. Dragged each one to the left and they restarted.
But NO crashing!
The dials and gauges did not stop updating though.

I know this isn’t much info, but I am away for a while longer with my Father, but I do log into the tablet daily to see what it is doing.

2 Likes

Very sad to hear about your Dad, we send our best wishes to you and your family. That sounds like a really difficult time.

Thank you for the update on the dashboard, that’s promising!

I am still away at my Dad’s but I wanted to make a note as to my dashboard.
The tablet has not crashed, yet, but I noticed to graphs still stop updating.
The only way to restart them is to drag in the graph, and that updates the data to be graphed, and the graph restarts, but only for a while. Don’t know yet how long but I will keep an eye on it.

Really wanting my Dad’s nightmare to end. He does want to “go” and I don’t blame him.

1 Like

An update.
My Dad passed on Thursday 5/22. So glad he is out of his pain and in a much better place!
I will miss him though… REAL hard time!

About my tablet. It hasn’t crashed in about a week! BUT the graphs stop updating after about a day, and I have to restart them by dragging in each window, then they will update for another day.

Condolences to you and yours. Losing a parent is indeed a rough ordeal. :cry:

Thank You Bill. At least now he is with his wife. She passed in 2009 and I don’t think he really got over her passing completely, but now he is with her and my sister.
Still real tough!
But thanks for your thoughts!

Sorry to hear @pomonabill220 sending our condolences, that must be really hard!

I have been taking care of Dad’s stuff so I haven’t had a chance to reply, but I tried the new SD image posted, and we are back to tablet crashing every (about) 6 hours.

I swapped out the new image (24July20) back to 17Oct19, but couldn’t restore because the service runner was not running and displayed restarting, but never restarted. The old image did run though without crashing the tablet. It just stopped updating the graphs after several hours… strange!

Since I had restored my settings from 17Oct19 to the 24Jul20, I had a copy of my settings on 24Jul20, so I re-wrote the 17Oct19 image to test the SD card for errors and it wrote fine.

So now I have a card with the '20 image (and my settings) and the '19 image (without my settings).

Then, since the '20 image seems to crash my tablet in about 6 hours, I ran the '19 image, restored from the USB SD my settings from the '20 card, and NOW even the '19 image crashes in about 12 hours.

My thought was since the '19 card was running “pretty good” (graphs still stopped updating, but the tablet didn’t crash), I can run the '19 image and restore my settings from the '20 card, and I should have been back to where I was before the '20 update… NOT!

My tablet still crashes about every 12 hours now.
I can try running the dashboard on another computer to see if the tablet is failing, but I doubt it… we’ll see.

Very frustrating!
The eompi is wired ethernet and the tablet is wireless. I think I will setup a camera to watch the tablet (so I don’t disturb the tablet itself) and time the crashes.
I seem to be the only one with this problem, but it would be nice to see what is happening!

Strange thing happened last night 9/22.
The dashboard didn’t “crash” on the tablet, but the window minimized but still had the dashboard running. This has happened before but not as frequent as the crashing.\

The dashboard that I have running on my Win7 machine has not crashed though, possibly because I run additional tabs in the browser (Firefox), keeping the browser “alive”.

I will setup another tablet running the same dashboard and see if that has problems, or if it only this particular tablet.