Copy users on private server, including dashboard, inputs, feeds, graphs, etc?

Emoncms
1

Hi Guys,
I have a situation that I hope you can help me with. I have a privately hosted server, to which I have access to the mysql database, etc. I have created a “master” user with all the required inputs, feeds, graphs, dashboard, etc, and enabled my server to be a multi-user one. So, before explaining how I am approaching this, let me first tell you the goal: I want to automatically create accounts for new users as the devices that sends their inputs come online, from where I want to “copy” all the inputs, feeds, graphs and dashboard from my pre-configured master user and make it available to the new user. I want to do this because I have close to 300 inputs, of which close to 250 of them stored in feeds, and to create that manually for 500 users, including graphs, dashboards from scratch, etc is not viable time wise.

So, herewith then a little background of how it works:
I have pre-written software on a raspberry pi that runs a little app at first startup to ask the user for their detail, after which it sends the .json command to the server to create the user. Based on this being a success, the write api key is copied from the response and automatically populated in the software. I do not mind having all the units post the same nodeID, as they will be posting to different accounts, so that’s not an issue.

Then I have a trigger on SQL that monitors the creation of new accounts, which then fires a bunch of SQL commands to copy the relevant data from my “master” user for each table, so effectively the new user ends up with all the inputs, feeds, graphs, dashboard, etc even before he logs in.

And now for the problem (why this does not work): When I created the input processors and feeds for my master user, they got assigned ID’s that increment as new processes, feeds, etc are added. So FeedID 30 might be populated by inputID 27, which is fair and well, but the moment I copy these values for a new user, it means that the new user would be trying to make use of feeds that does not belong to him. Besides that, the moment I copy a feed or input or any other data, a new ID is created for my new user, so the new user might end up having the feed with the same name as FeedID 2019 and inputID 183 for arguments sake, but it was copied with 1:30 as that’s what it was specified as on the master account.

So really this becomes a relational database issue, for which I hope there is a solution. That being said, I have very little Mysql knowledge, which may be exacerbating the problem too.

I have managed to create the inputs and the associated feeds using the .json Api commands, but ultimately that doesn’t help me at all, because the dashboard data is still referring to the feedid’s that belong to the master user, and that’s not just a column to update in the database.

So, after all of that, the real question is, is it possible to have a new user account setup automatically, including the dashboard, inputs, graphs, etc based on a master account (template)?

I really hope someone has tried this with a bit more success than me before…

2

I think @TrystanLea might be able to help you with this.

3

The bulk of this can be managed using the device module. using the device module will allow you to replicate inputs, processing and feeds using the in built emoncms methods so that the mysqy indexing is used correctly.

As for duplicating dashboards and multigraphs, I started out duplicating these by creating blank dashboard(s)/multigraph(s) in the new user account and then coping an existing dash/graph content to those new “blank” entries in the mysql tables using phpmyadmin. Once the content was copied over I could then just open the new dash in edit mode and reselect teach feed for each widget etc. This method I later replaced with dasboard templates that had __KEYWORDS__ in place of each of the feedids and I created a script that queried the feed info for a newly created account (setup using the device module) via a feed api and used that info to establish the feedids and using regex and text manipulation to create the new html and insert that into the mysql tables.

Ideally the next step for me would be to write “app” pages to replace the dashboards so that the same template can be used over and over since the apps are based on feednames not ids directly. I also wonder if the changes to the adshboard module to use feedids exclusively (purely to allow feed names to be repeated) was a bad move and we should look to change the dashboards to use a feedname+tag combo rather than feedid, this would allow dashboards to be directly reusable for multiple account, whilst retaining the ability to duplicate feed names.

The issue you are seeing with the old feeds being used in the processing is due to the absolute feedid references in the input processing, since your new account has new feedids, the processlist that you copied over needs editing. The problem you have though is that “feedid” is just one of several possible values that maybe used in the processing list. Take this processlist for example

11:1000,2:1000,1:1000

This translates to

+input (process type 11)     : "1000"(inputid)
x (process type 2)           : "1000"(value)
log to feed (process type 1) : "1000"(feedid)

So you can see that the 3x “1000” values all relate to different things depending on the process type, so you would need to use the process type to determine if the second value was an input or feed id and then replace it with the correct feed or input id for the new account. The device module takes care of all this for you.

1 Like
4

Yes I would echo @pb66’s comments, the use of the device module and the potential to use a dedicated app for what your after to get around the dashboard feed issue. Here’s an example used for heatpump monitoring: https://github.com/emoncms/app/blob/master/apps/myheatpump.php - that said that is a solution that requires a lot of coding!

5

If your starting off with the device module it might be worth reading through our ongoing development thread here Development: Devices, Inputs and Feeds in emoncms

6

Since we are using a feedname+tag combo in the device module, how difficult would it be to get the dashboard module to also use a feedname+tag combo? This would provide a way of duplicating dashboards across accounts which would fit in with the efforts in both the device module and the group module.

7

I think the original reason for the use of feedid’s was to support the changing of feed names without breaking the dashboard, but I wonder if the dashboard module could support an autoconfig fixed feedname+tag option as well, perhaps if the feedid is not set it could default back to a feedname+tag option if that’s set.

I’ve got a lot of other items on my todo list at the moment and wont likely get on to it soon, so if anyone fancies looking at that please feel free to try and Id be happy to accept a pull request.

8

I think it might be a bit beyond my php skills, even if I did have the spare time right now.

I haven’t even had a chance to look into editing “apps” yet, beyond changing colours in the myelectric app. One issue with the “apps” approach is the templates are not “drop-in” as you need to edit the list of apps else were in the code. Plus we could ideally do with some sub-directories like in the device module data folder so we can keep our own folder/repo of app templates rather than individual files.

1 Like
9

I have opened a couple of issues on github so these points don’t fall off the radar.

10

Thankyou Paul, I wasnt suggesting you did this necessarily, just thought Id mention that I wouldn’t be able to dive in, in the even that there is someone else reading who might fancy the challenge, your effort would not be wasted :slight_smile:

11

Yeh, no problem. It was more for other readers, someone might be wondering why I don’t get on and do it myself, other potential developers might not have a go if they think I’m working on it behind the scenes.

Any spare time I might find I’m hoping to invest in emonhub development and reuniting the different variants :smile:

12

Hi Guys,

Thanks for all the comments! I have been bashing my head against the wall trying to get this done, and the worst part is, I can’t seem to get the device module installed on my hosted server… But from what I read it sounds as if Paul also tried to accomplish the exact same thing already, so i’ll spend some more time trying to get the device module going…

With my very limited Mysql knowledge I have actually managed to get a new account’s details all copied, and luckily I only use the log to feed process, as well as only one process per input, which makes tying them up with names and uiserid’s easier than when there is a lot of processing happening per input. As Paul pointed out, the real issue comes in the form of the dashboards, as that’s where one need to tie the ever changing feed id’s to that of the “master” user/template account, rebuild the entire thing and then post it back to sql, all in one go. NOT EASY!!

But hey, the longer you work at something, the easier it becomes, so i’ll just keep at it UNTIL I manage to get it right… Will update you guys along the way, thanks for the resources and extra stuff to read and ponder about, appreciate it!

13

Hello @centurionsolar, we now have a solution in testing for copying dashboards, if you would be happy to help testing that would be a great help, see forum post: see: https://community.openenergymonitor.org/t/portable-emoncms-dashboards-looking-for-testers

14

Hi Trystan,

I did a quick test on a simple dashboard, and it looks like it’s working a treat! My requirement was slightly different than that, in that I want to achieve the same result, but without any user intervention, IE: The user signs up, and after specifying his chosen credentials and email addy, the rest should happen without any further input from the user.

I managed to get this right, although the approach I use required A LOT of back-end juggling to do it. The solution you provided her is a lot more elegant in my mind, albeit with some tweaking required like manually creating feeds from inputs and ensuring they have the same name and tags.

With a little tweaking this would go a long way towards the common goal of “exporting” and “importing” dashboards between servers (if one can call it that ;)).

Thanks again, thumbs up from my side!

Regards

1 Like