Emoncms dashboard crashes after x hours

I think the issue is still the logged in user session timing out, or possibly changing from a admin session to a standard user session.

Can you please try one or both of the following using dash url with an apikey.

  1. Log out of emoncms and then refresh the dash/graph page
  2. Open the dash/graph page in an incognito browser window

I don’t think this is strictly true and therefore I wonder if the transition from a logged in session to a logged out but apikey provided session without refreshing the page is the issue. We already know that the transition from an admiin session to a standard user session can get ugly and do funky stuff.

The page will originally be drawn using the logged in credentials and simply ignore the apikey, when the (standard or admin) session cookie times out, perhaps the page isn’t refreshing to re-authorise via the alternative method (apikey), therefore the content is no longer updated.

I use apikey urls to view dashboards a lot and they don’t timeout ever, but I’m using an earlier emoncms version and I never log in via the same browser as the tv’s displaying the apikey urls (plus I never view a dash long enough to find out if it stops updating via my PC as I do what I need to do and log out, close browser or sleep the PC).

I agree it seems to be something about the session and the APIKey. I don’t know enough about either to be able to dig for the bug.

Yes and it bugs me every day when it happens - such poor UE.

Well making all my feeds and dashboards public didn’t work… damn! I thought we were on to something.
I started both my tablet and laptop in viewing the now public dashboard, at about 9 a.m. this morning, and as far as I can tell when Firefox created it’s crash report on my laptop, Firefox crashed at about 3-4 p.m. 6 hours after starting.

I will try Paul’s suggestion of logging out of emoncms and refreshing the dash page on the laptop, (didn’t work… when I log out, the log in page displays wanting my uID and PASS).
then using an incognito browser window on the tablet’s chrome (still wanted me to login).
i’ll report what happens.

WOOPS forgot the apidkey… will restart the test.

Well this is the address that i used to access the dashboard:
http://192.168.1.103/emoncms/dashboard/view?id=1&readapikey=1323878bcb618cec5f56b0da8293ec49

The api I got from emoncms/my account/read api key
The dashboard loaded ok, and I think I am logged out because when I click on My Account, a log in dialog pops up (windows 7 laptop). I logged out and used the web address with the apikey string to load the dashboard screen, so we’ll see what happens.
I will do the same on the Android tablet just for fun.

It is now 11 p.m. PST 6 a.m. UTC

A strange thing happened when I made my feeds public.
First, I selected all of them, then Edit (the pencil icon), then checked the box Make Public.
This made all the feeds public, BUT now I only have ONE node rather than two. IE my node named Emonpi disappeared, and all my feeds are now under OpenEVSE.
Really doesn’t matter right now as we are trying to get the dashboard to run for more than a few hours, and I don’t want to mess around re-grouping the feeds yet.

Just thought I would add this

@emrys - one for you to look at?

Well it happened again today (6/6).
Started both the Android tablet and the Windows 7 laptop running firefox and monitoring my dashboard, using the readapikey (http://192.168.1.103/emoncms/dashboard/view?id=1&readapikey=1323878bcb618cec5f56b0da8293ec49).
They both ran for about 8 hours as far as I can tell, then the browsers crashed
There was a re load option on the crash screen which reloaded the page with no problem.

Very strange and frustrating!

Wish there was a way to log activity that may give us a clue.

All I can suggest is using the utility tshark and run that on the Pi to watch the http traffic. Search for ‘tshark’ on the community. It is a bit of a learning curve but worth the effort eventually. Example

You could end up with a lot of data to wade through, but possibly the only way to debug this.

is it possible they are crashing at the same time each day?

When did you last update the emonSD?

Were you definitely fully logged out? 35mins after posting you started a new test you posted that you had some strange behaviour when setting feeds to public, which you must be logged in for.

Can you open a incognito browser window and enter that url again and leave that incognito browser open to see what happens, are you testing this on a PC that is permanently awake? It’s ok for the screen to sleep, but if the PC sleeps it will interrupt the connection and need a reload.

I’m not overly familiar with firefox or keeping an android device connected to a page for a prolonged time. I use chrome. Do you have chrome you can test with?

The reason I say above to use incognito mode is because (with chrome, not sure about FF or android) a chrome user status is common to all chrome windows opened by that user, you cannot be logged out in one window whilst logged in in another window unless they are as different chrome users. an incognito window helps keeps things separate.

Something else you could try is opening the developers console (F12 on chrome) and have that running so that you can see the regular data update requests

in the image above each of those data.json?id=100xxx&start… lines is a data request, one for each trace of the graph, each block of 6 requests happens every 30s (multigraph refresh setting)

You can also tail the apache access.log in a separate ssh terminal window

sudo tail -f /var/log/apache2/access.log

here are a couple of lines from the same site and page as the above example (some details fudged)

123.123.123.123 - - [07/Jun/2019:18:49:03 +0100] "GET /feed/data.json?id=100992&start=1559843342708&end=1559929742708&interval=108&skipmissing=1&limitinterval=1&apikey=abcd1234abcd1234abcd1234abcd1234 HTTP/1.1" 200 4841 "https://myserver.com/vis/multigraph?embed=1&mid=100018" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36"
123.123.123.123 - - [07/Jun/2019:18:49:03 +0100] "GET /feed/data.json?id=100993&start=1559843342708&end=1559929742708&interval=108&skipmissing=1&limitinterval=1&apikey=abcd1234abcd1234abcd1234abcd1234 HTTP/1.1" 200 6470 "https://myserver.com/vis/multigraph?embed=1&mid=100018" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36"
123.123.123.123 - - [07/Jun/2019:18:49:03 +0100] "GET /feed/data.json?id=100959&start=1559843342708&end=1559929742708&interval=108&skipmissing=1&limitinterval=1&apikey=abcd1234abcd1234abcd1234abcd1234 HTTP/1.1" 200 6846 "https://myserver.com/vis/multigraph?embed=1&mid=100018" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36"
123.123.123.123 - - [07/Jun/2019:18:49:03 +0100] "GET /feed/data.json?id=100924&start=1559843342708&end=1559929742708&interval=108&skipmissing=1&limitinterval=1&apikey=abcd1234abcd1234abcd1234abcd1234 HTTP/1.1" 200 7442 "https://myserver.com/vis/multigraph?embed=1&mid=100018" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36"
123.123.123.123 - - [07/Jun/2019:18:49:03 +0100] "GET /feed/data.json?id=100958&start=1559843342708&end=1559929742708&interval=108&skipmissing=1&limitinterval=1&apikey=abcd1234abcd1234abcd1234abcd1234 HTTP/1.1" 200 5121 "https://myserver.com/vis/multigraph?embed=1&mid=100018" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36"
123.123.123.123 - - [07/Jun/2019:18:49:03 +0100] "GET /feed/data.json?id=100925&start=1559843342708&end=1559929742708&interval=108&skipmissing=1&limitinterval=1&apikey=abcd1234abcd1234abcd1234abcd1234 HTTP/1.1" 200 7279 "https://myserver.com/vis/multigraph?embed=1&mid=100018" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36"

again these 6 lines appear every 30s in the access.log.

In addition to the above multigraph data updates every 30s (or whatever interval you’ve set) you should also see a line every 5 seconds, you can see them in my dev console image above, they are the /list.json?userid=100002 lines. They too have a corresponding entry in the access.log that reads

123.123.123.123 - - [07/Jun/2019:18:56:51 +0100] "GET /feed/list.json?userid=100002 HTTP/1.1" 200 3522 "https://myserver.com/example" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36"

These requests are to do with the emoncms dash webpage (rather than the multigraph data) updating, It might help you debug this if you knew if these 2 types of requests are still being made after “the crash” and if they are, what response they are getting.

No. I have started both my laptop and tablet several different times per day, and they both crash about 6-8 hours after starting. I am not able to witness when they crash, but in firefox I can see the time stamped crash logs by going to about:crashes, and FF list the logs.

I assume you are asking what version I am running? It is EmonSD-30Oct18

I guess i wasn’t too clear of what I did.

  1. logged into my local emonpi
  2. selected Feeds.
  3. clicked on the checkmark icon for select all
  4. when all feeds were selected, clicked on the pencil icon that appeared
  5. in Edit Feed popup, checked on Make Feed Public (this is when all my feeds were grouped under one feed node).
  6. clicked OK
  7. clicked on my account icon
  8. selected Log Out which brought me back to the log on page, but I closed the browser (NOT just the browser tab).
  9. opened a new FF browser
  10. entered the URL http://192.168.1.103/emoncms/dashboard/view?id=1&readapikey=1323878bcb618cec5f56b0da8293ec49 which opened my dashboard page.

This also crashed 6-8 hours later when I wasn’t home, so I don’t know when it exactly crashed.
I assumed i was still logged out because the blue bar in the dashboard page was different from when I was logged in, and only had My Account / Bookkmarks / Log Out across the top, and when I click on My Account, the log in popup appears, so I am assuming that I am NOT logged in.

I will try the incognito mode.
Yes, the PC is NOT going to sleep/hibernate/etc. and the screen stays on. Same with the Android tablet.
I have tried Chrome and FF on the tablet in the past, as well as on the PC, but I will do it again just to verify/test.

I was going to try wireshark on the PC as I am semi familiar with it’s use, but no where proficient, but when a crash occurs, it might stand out.

No I am asking you when you last updated the emonSD image. emonSD-30Oct18 only tells us what the base image is, it may mean you have not updated since Oct last year, the emonSD-30Oct18 image is constantly under review with ongoing updates, when you last updated tells us roughly what fix’s and features you might have.

and just to be sure, you didn’t log in to emoncms via another browser window on that PC during this time, correct?

This isn’t necessarily true since the page will only update to reflect the logged in status if it’s refreshed. If the page has stopped updating, the status of the nav bar etc cannot be believed as it too won’t be updating.

I have done some further testing about this problem.
First, I ran the full update by clicking on Admin/Full update and let it run to completion. I have the update log if that would help.
Then, I ran my same dashboard using the API in the URL and displaying ID=5 which is my EV dashboard.
I just happened to catch it before it crashed and what I saw was that the graphs had not updated for several hours, but the dials were still active and updating. I did not get a chance to monitor things with wireshark before it crashed.
This gave me an idea.
I created another dashboard with ONLY dials and gauges, so now I have 3 dashboards created. One with ID=1, ID=5 and ID=6:

default
http://192.168.1.103/emoncms/dashboard/view?id=1&readapikey=1323878bcb618cec5f56b0da8293ec49

EV
http://192.168.1.103/emoncms/dashboard/view?id=5&readapikey=1323878bcb618cec5f56b0da8293ec49

only gauges
http://192.168.1.103/emoncms/dashboard/view?id=6&readapikey=1323878bcb618cec5f56b0da8293ec49

I then logged out (account icon/logout), got the login window, and closed my browser.
I reopened the browser and used the “only gauges” url without logging in.
This opened my gauges only page.

IT has been running for 36 HOURS withOUT crashing!
I don’t know if this might be a clue but it seems strange that if the graphs stop updating, the browser crashes both with a windows laptop and an Android tablet, and not at the same time but randomly.

What could this indicate?

Still running 2+ days later!

Sounds like you’ve made some progress.

The next thing to do is to check some other graphs and dashboard configurations to try and pin it down to whether it is only the multigraphs and/or only the auto-update part of multigraph ( i’m pretty sure it will be the auto-updater). Does the realtime graph work ok over a long period?

Yes some progress has been made.
Next, I will create a dashboard with several realtime graphs and see how long they run.
I have created a dashboard with 4 realtime graphs. It has a new ID=7.
All I did is change the ID number in the URL that I have been using with the API key, and being logged out.
So it is:
http://192.168.1.103/emoncms/dashboard/view?id=7&readapikey=1323878bcb618cec5f56b0da8293ec49

I will let everyone know what happens!

Forgot to mention that I have been having this crashing problem before I started using multigraphs.
I was using realtime and built in graphs along with gauges and dials all on the same dashboard.
I will keep trying different combinations though to see what is causing the problem, and keep this updated.

1 Like

My realtime graphs dashboard did not crash and has been running for about 2 days!
Doesn’t seem to be realtime graphs, so I created another dashboard with rawdata graphs and it has an ID=8… IE:
http://192.168.1.103/emoncms/dashboard/view?id=8&readapikey=1323878bcb618cec5f56b0da8293ec49

We’ll see how THIS one runs now, and I will update in a couple of days or so.
This is turning into a treasure hunt! hehe :laughing:

This dashboard with only “raw data graphs” does NOT update! (oops, corrected, was realtime)
I have tried several times to refresh the page (I am using Chrome in incognito mode and NOT logged into my account).
Also tried shutting down Chrome, restarting Chrome and displaying this particular dashboard, and still does NOT update.

I will try changing to my previous dashboard (ID=7) and make sure that one works, just to rule out any new changes like network problems, wireless problems, tablet problems, and let you know!

Well a few hours later with the realtime graphs running (ID=7) and they are updating just fine!
I will let it run today and make sure it doesn’t crash.
Then I will modify my default dashboard (ID=1) with only gauges and realtime graphs, and let that one run to see if it crashes.
It could be the combination of realtime and raw data graphs that makes it crash.

What I need to do also is make a chart of the combinations that I have tried and need to try, and note whether they have crashed or not.

1 Like

ID=7 (realtime graphs) is still running with no crash!
Next, I will clone my ID=1 graph (my “original” dashboard), and use multi graphs + gauges + dials. NO realtime or graphs.
It will be ID=10
http://192.168.1.103/emoncms/dashboard/view?id=10&readapikey=1323878bcb618cec5f56b0da8293ec49

We’ll see how THAT runs and I’ll update the status in a day (or less if it crashes before that).