"device already exists" ajax async errors perhaps?

pb66 · 2021-05-23T17:32:29.793Z

I’ve been plagued by this for what seems like forever, where a box pops up on the inputs page of emoncms telling me it couldn’t create a device since it already existed!!! Aside from not asking it to create a device I already know that said device already exists, but it blocks the page until I click “ok” (even though it isn’t!) and then to add further insult to injury it cycles through every device telling me the same damn thing. Until now it’s been “annoying” that on the occasions it happens, especially when it then goes into a second, third or fourth lap or more!

There are several references to it on the forum and also github issues, both open and previously closed as “fixed”. But although clearing redis and/or clearing the browser cache sometimes clears the problem temporarily, it doesn’t always and sometimes it takes several stabs so I’m not convinced these actions are directly resolving the issue (even temporarily) and that perhaps something just gets nudged along the way.

Last week I set up a RPi 3B+ with a desktop for a specific task and happened to open emoncms in chromium. The inputs page was absolutely unusable in any way, shape or form. The first error box would be there within seconds and continuously loop endlessly through my inputs. I couldn’t even get a button click in to collapse the inputs to get a better picture let alone scroll down the page.

At this point I fired up my usual PC and found that the input page of that same emoncms instance was currently working fine in Chrome, no device exists errors! Convinced it could be a result of the RPi’s limited power I swapped it out for a 4GB RPi4, same sdcard, also in the same location on the same WiFi. That was still “unusable” IMO but it was better, I could at least get the occasional button press in. this convinced me I might be on to something and decided to pursue further rather than giving up.

For info purposes, the emoncms instance I was using is NOT updated, Nor shall it be any time soon as I do not have the time for the ensuing consequences of updating. However!!! I tried the above tests with an emoncms.org account and got exactly the same results (on the Pi3B+ but the Pi4 wasn’t so bad), so I’m not too concerned about the version of emoncms I’m running at this point.

Using the dev mode of chromium I could see the order of events on each page refresh (autorefresh not manual “reload” type refresh) and by blocking the url/path device/create.json I was free of all those annoying boxes, as tempted as I was to stick there, I continued by looking into the code of emoncms/Modules/input/Views/input_view.js and whilst I have no real understanding of Ajax or JS, I came to the conclusion that it probably wasn’t a good idea for the content of an as yet unfulfilled request to be compared to another potentially unfulfilled requests data and therefore edited 2 Ajax calls to be async: false rather than async: true which I understand allowed the code to continue without waiting for a response, as both the device/list.json data and `input/was to be accessed in the line immediately after sending that request I couldn’t see how it could be reliable.

Since editing those lines on my outdated instance I have had no error messages on the Pi4 at all, whilst I do still get them from emoncms.org when reverting back to the RPi3B+ despite the local instance being error free, slow as a glacier, but error free and usable with patience.

The 2 lines I changed were

github.com

emoncms/emoncms/blob/4efbd9e9c43c33573d68783770b8cacf3b7ce335/Modules/input/Views/input_view.js#L835-L835

    
      
          $.ajax({ url: path+"device/list.json", dataType: 'json', async: true, success: function(result) {

github.com

emoncms/emoncms/blob/4efbd9e9c43c33573d68783770b8cacf3b7ce335/Modules/input/Views/input_view.js#L861-L861

    
      
          return $.ajax({ url: path+"input/list.json", dataType: 'json', async: true, success: function(data, textStatus, xhr) {

This issue is definitely way more apparent on the Pi3B+, even emoncms.org is causing errors very frequently. Where the Pi is situated the wifi signal is at it’s limit, so if anyone wants to recreate the issue on demand to try and fix/test, use the least powerful device eg a RPi3B+ (or less) with desktop and if needed maybe shield from wifi to cause a slower connection? I just recalled the Pi4 is 5GHz wifi not 2.4GHz, maybe that helped too?

I’m not a emoncms dev so I have no idea if I’m circumventing another issue, it seems logical to me NOT to try and user data that might not yet be available unless there is a fallback plan ie don’t reset everything if said data isn’t yet available which is what appears to be the case here.

github.com

emoncms/emoncms/blob/4efbd9e9c43c33573d68783770b8cacf3b7ce335/Modules/input/Views/input_view.js#L877-L891

    
      
          // Device does not exist which means this is likely a new system or that the device was deleted
          // There needs to be a corresponding device for every node and so the system needs to recreate the device here
          if (devices[nodeid]==undefined) {
          
          
    devices[nodeid] = {
                  id: false,
                  userid: inputs[z].userid,
                  nodeid: nodeid,
                  name: nodeid,
                  description: "",
                  type: "",
                  devicekey: false,
                  time:false,
                  inputs: []
              }

I read this to basically, if revised data or existing data for confirmation purpuses isn’t yet available obliterate existing data with empty and/or false fields.

I have no doubt using async: false might be slower, but I see no other way, there is no point in the page refreshing faster if that means false errors.

Having written this long post I have just been back to the RPi3b+ with input pages open at my local instance as well as emoncms.org and my local instance is still error free whilst emoncms.org has a error box.

I will raise this on GitHub (with a PR?) for formal consideration but if anyone has any thoughts or any experts can confirm or rule out my findings that would be great.

github.com/emoncms/emoncms

"device already exists" ajax async errors perhaps?

opened 05:38PM - 23 May 21 UTC

pb66

See https://community.openenergymonitor.org/t/device-already-exists-ajax-async-e…rrors-perhaps/17807?u=pb66 Should we be trying to use data that might not yet be available? Should these ajax calls be `async: false` instead? https://github.com/emoncms/emoncms/blob/4efbd9e9c43c33573d68783770b8cacf3b7ce335/Modules/input/Views/input_view.js#L861 https://github.com/emoncms/emoncms/blob/4efbd9e9c43c33573d68783770b8cacf3b7ce335/Modules/input/Views/input_view.js#L835 relates to previous issues #1485 & #1618 ~~[edit - just noticed the device call IS `async: false` already but I have no idea if emoncms.org is running latest emoncms. If it is, then it must be the input call change that is fixing things for me.]~~ Ignore that! I had linked L895 (device/create.json) in error, it should have been L835 (device/list.json) which I've now corrected above and is inline with my original findings.

pb66 · 2021-05-23T17:57:05.555Z

It seems the very latest emoncms code does use async: false in one of those calls to input/list (github leading me astray) assuming emoncms.org has this “fix” already it might just be the device/list call that needs editing. Before submitting a PR I will try some further testing when I can, but I’ve been at this a while and my bellies rumbling!

pb66 · 2021-05-23T18:00:02.916Z

pb66:

assuming emoncms.org has this “fix” already

@TrystanLea can you please confirm it is or perhaps add that fix if it isn’t?

pb66 · 2021-05-23T18:06:13.206Z

Perhaps to retain async for speed we need to only compare locally cached lists of devices and inputs and only update those lists on completion of a successful request rather than trying to use the live request response that simply might not yet be available.

pb66 · 2021-05-23T23:26:07.488Z

Brief update, RPI3B+ with slow wifi is still updating the inputs page of the local emoncms no problem, I have cleared the error messages several times on the emoncms.org inputs page. Plus a brief look at the code loaded via emoncms.org, in dev console shows input_view.js?v=6 seems to still have both the ajax “list” api calls as async: true.

I still need to test with just the one change as per latest emoncms master to confirm the second change is needed, but theoretically, I still think neither “reply” should be accessed directly, or at least not before it is successfully completed.

TrystanLea · 2021-05-29T07:57:19.067Z

Thanks @pb66 sorry for the delay. I will get back to you on this soon

TrystanLea · 2021-05-29T11:07:36.681Z

Thanks for the tip on the connection speed, I was able to use the browser throttling feature to simulate a GPRS connection and replicate the error on emoncms.org.

There was a commit last September to the input view intended to fix an issue to do with the node name being treated as a number rather than a string in the associative array that holds the nodes that was I thought the reason for this error https://github.com/emoncms/emoncms/commit/7124d6e02348b4bea25dab92e36rea263f5fe328fa#diff-d6b93661d8142fe2e234a2a45d76a8d73fca4d4b2bff6888fb76f3b60eeb4ddd

I realise emoncms.org had not been updated with this commit, I have done this now.

Im not 100% that it did fix it, the first retry I got the same error but now a couple of full page refreshes later with throttling enabled I cant seem to get it again…

The async configuration was the same as the main emoncms repository prior to today’s update. All are async true apart from device/create.

TrystanLea · 2021-05-29T11:45:34.891Z

I’ve added a couple of console logs to record the order in which device list and input lists are requested and received on the emoncms.org server input view, if you keep seeing the error can you check that the order matches my screenshot here @pb66 ?

image693×750 48.3 KB