Development: Indexed Inputs

The missing index error would happen if the redis cache of the inputs did not include the index. Calling

http://localhost/emoncms/input/populateindexes.json 

should reload the redis cache to include the index and hopefully sort that error, although I imagine you have already ran this so Im not sure why the inputs did note have the index present…

I have

$display_errors = true;

Your test bulk api call works fine for me.

Is there any chance that you have phpmqtt_input running in the background, or somehow two emoncms instances running on the same database? causing these intermittency issues?

There is no mqtt installed at all on this server

This emoncms instance is on the same host as one of my live instances, they do not share a database table, obviously they both use the same mysql server (different users) and redis too (different prefixes though). They are on separate ip’s, domain names and virtual hosts, they don’t even share ssl cert and the feed data is on different drives, So they are pretty well isolated.

It isn’t really that intermittent, with $display_errors = true on master branch it’s perfectly ok, with $display_errors = false on indexedinputs branch it is pretty solid too, I had noticed some occasional 100 ~120 s Updated times but not recently, so I could put that down to dodgy wifi or something, I certainly cannot reproduce it this afternoon. And with $display_errors = true on indexedinputs branch it is permanently returning errors to every request, but emoncms seems to soldier on for a while but eventually it always fails, it has now come to light that the input page “Updated” times show an issue before the processing actually stops and produces no feed data, it’s as if it’s getting clogged up, perhaps it’s linked to the size of the payloads emonhub is dropping on it since they will progressively get bigger as the errors prevent emonhub from deleting the previously sent data. Or is emoncms buffering something and eventually tripping up, maybe in stages?

It’s certainly not intermittent in the fact it will fail, it’s just the time and behavior prior to failing is currently less predictable.

Ok good to hear, does running

http://localhost/emoncms/input/populateindexes.json 

get rid of the index missing error?

Yes, apparent so!

But I have definitely run that previously and aside from switch branches I have not rebooted, flushed redis or made any changes to the mysql tables so i have no idea why that needs running again.

Having said that, I think I should wait half an hour or so to be sure it doesn’y eventually fail, but the fact there are no errors coming back in the response from emoncms in the emonhub.logs looks very promising as i was getting those errors permanently, even in the initial minutes before it eventually failed.

[edit] is there a reason the populateindexes cannot go inline with the input processing? I’m not keen on a separate script even if it was a successful single shot thing. This will trip up users updating without knowing about this in advance.

I was just looking through the indexinputs PR/diff and noticed the word “index” is always wrapped in backticks in mysql statements as it is a mysql reserved word. Knowing that, would it not be better to avoid using that word and use “idx” or “input_index” or something similar instead? It’s easy to change now (before wide adoption), but down the road, less so, and it may prove problematic, even if edits are made to the code and the backticks forgotten etc. It just seems an unnecessary risk/evil when any word could be used.

Could the handling of this be different for mysql and mariadb or different version numbers or OS’s?

It’s a longshot I know, but I’m running dry on idea’s.

Something else that’s been on my mind that relates to the fact I had to rerun the populateindedex api call is that late last night I ran the db update again as I’m switching branches and testing multiple things here so db update and clearing browser cache are happening very frequently, and on this occasion it told me there was an update to the user table “index” field, I clicked it and got a green “ok” banner.

I know without any shadow of a doubt I have updated the db several times since running the populateindexes api the second time, and many more times since i did it the first time. So I have to wonder why that change is either not sticking indefinitely or at least being perceived as such by the check db code.

Initially I wanted to avoid adding a checking overhead, but I think it should be possible, and likely needed for its adoption.

Yes we could use a different name at this point, might go for your short hand idx.

I will modify to try both of these.

Ok, auto index population, change of index name to indx and phpmqtt_input support are now implemented:

Change of index name to indx:

Autopopulation of index:

phpmqtt_input support:

1 Like

Great, I’ll aim to try the changes in the next couple of days or so, although I won’t be trying the mqtt for quite sometime yet as I have no mqtt setup on this test server.

Do you have any idea when you might implement indexedinputs into the “device module beta”? I’m not so much interested in the device module integration itself right now, but I’m finding it hard to asses both the new input/feed pages and indexedinputs at the same time. Hopefully if the indexing is working behind the scenes, it might just be a case of adding the indexes to the revised inputs page, is it not?

should be, I will take a look.

I’ve added a label to the new input’s page that shows the index, do you think this is sufficient? @pb66:

labelindex

If the inputs are always sorted into index order by default that might suffice to keep it clean.

I have to say I prefer the way the old input page displays them fully to the new input page but understand you may want to minimize the data displayed on a mobile device. There again it would be easy to hide that column on smaller screens as per the other low priority columns do currently.

There is an argument that since users have become used to the way emoncms currently works, it might be of benefit to have both displayed to highlight the difference between input name and index (previously kind of the same thing).

Either way, I think the sort order must be by index by default as the index is literally “the order” although a “sort by” button/dropdown like the users page now has (formatted differently but functionally the same) to reorder by inputid or input name (description? device location?) would be a nice addition to the inputs page as the new interface seems to have lost the reordering functionality previously done by clicking the column header.

It will also be necessary to add the index to the edit functions as we can currently edit the indexes and that is useful when rearranging the source csv, if the default sort order is by index then it also provides users with a backdoor method of defining a custom sort order if they are not using csv inputs (eg mqtt or key:value json).

I know there is some danger with making the indexes editable but that risk is already there currently and I think it will be essential to edit the indexes for some users as the way emoncms applies the new indexes means that if mqtt delivers the devices individual values out of payload order or if indexedinputs is switched on when a named input payload is half completed only the latter half of the inputs will be indexed from 0 and the former half indexed on the next update with higher indexes, so there needs to be a way or applying corrections without resorting to editing mysql tables.

If you are going to add a hover over info box then maybe it should also give us the inputid as that is not shown on the input page either (yes you can open the input process box to view it but if looking for input id 1234 do we want to open every process list until we find it? the hover over would be ideal place for it) and maybe we should be looking for some consistency across the different pages for hover over tect boxes, the feeds page one has changed significantly to a centre aligned list in black, the graph page is still white with a left aligned list whilst your initial inputs hover over is a black array.

Thanks @pb66.

Will continue to consider the visibility.

I think the default ordering is currently by inputid so yes it would be good to have ordering by index. And a sort by feature would be nice I agree.

Editing the index would need a check to ensure the index remains unique which should be straightforward.

Good idea adding the inputid to a hover over, and for consistency across hoverover’s I think its evolving…

1 Like

I’ve commited a couple of further minor changes after testing the process of migrating from a system running master to the indexedinputs branch.

The likely case where the indx field has not yet been created in the database now fails more graciously. There are errors written to the emoncms log. The inputs stop updating but configuration is not broken. Then after running database update, it all works, the indexes are applied and the system keeps recording as normal.

I have this branch now running on our office monitoring system and will update my home system later as a further test.

1 Like

Just switched to the indewxedinputs branch in pursuit of another issue (When on the master branch see Thoughts on the new Input and Feed page layouts - #17 by pb66)

and noticed the new hover-over apparently not working, after a while I realized it only works when directly over the input name, not the whole row, this is in contrast to the feeds page hover-over and was a point of discussion on github. I had thought the point raised by @Paul was a good one, but now I have experienced what that change might be like, I’m not so sure, I was convinced it wasn’t working and therefore believe many users might not happen to find the hover over.

There are some formatting issues too, it is black in your screenshot and white on mine, this is also a contrast to the feeds page

I understand that both the indexedinputs input page hover-over and the device module integration feeds page hover-over are both works in progress so I do expect some changes but they currently couldn’t be more different if we tried. The overall formatting of the hover-overs is outside the topic of this thread I guess and will get discussed elsewhere (Thoughts on the new Input and Feed page layouts perhaps or in one of the many issues/pr’s open on this topic), the content of this hover-over also does need expanding as per the previous posted comments.

Anyway, just thought I’d comment on this thread whilst I was on this branch, I’m switching back now to see if it’s impacted my phantom inputs issue :slight_smile:

Thanks @pb66 will investigate this week.

I wonder if the indexes should actually be defined to make it obvious. It isn’t currently intuitive as there are 2 forces working against this. 1) JSON is not ordinarily “ordered” so by just looking at JSON content you do not know it will be received and processed as an ordered list. 2) For so long we have told users that you can only have indexes or names not both, that there might be no consideration for the order ie users might prefer to put the inputs in alphabetical order for easy reading, not knowing that will define the indexes. Just a thought!

I see your point but I would have thought the most likely template ordering is in index order… Perhaps we start with implicit indexing based on the indexed array and come back to this if it causes confusion.

Will bear your comments on the tooltips in mind as we bring the styles together :slight_smile:

Thanks for testing the indexed inputs branch @pb66 did you see any other issues? my two test systems appear to be working ok, so I think its nearing a good point to merge into the master branch for wider testing, here’s the latest compare between the two branches: Comparing master...indexedinputs · emoncms/emoncms · GitHub

There are issues with re-initialising devices, I’ve raised and issue on the device module repo

I wanted to do some more testing with creating devices etc but I’m getting bogged with issues with the device module and things like those “phantom inputs”, I’m wasting a lot of time chasing issues that are apparently “features”.

I’m currently going to run the indexinputs branch for a while without the “beta” as it’s too many changes to assess/debug in one go and the switching branches backwards and forwards is adding to the problem.

Are there any plans to introduce a version number change any time soon? The changelist is already pretty substantial and it would be useful to still be be able to switch versions if the branches get merged. Setting a version doesn’t necessarily mean it must be merged into stable, the 2 operations can run independently. ie we don’t need to officially “release” every tagged version, we can split a vast number of changes accross multiple versions and then just release (merge) several versions at once to stable eg 9.8.31 to 9.8.35 for example, it could make testing so much easier.

Yes aware that this should have been done a while ago. Will get that updated.