Dropbox-archive for multiple accounts

Emoncms
1

I would like to try using the dropbox automated back-ups (and restores?) it looks like a great tool and I appreciate all the work @Paul has put into it, but there is a significant reason it probably won’t work for me and therefore perhaps others too. I would therefore like to suggest a change in the way I believe it works to be more widely adoptable. I am not asking for this work to be done for me, I am happy to contribute if there is interest.

I know this is primarily written for the home user with one account as is much of the OEM project, but I run many accounts/users on my self-hosted emoncms, looking at the current settings file for the dropbox-archive, it asks for a write apikey (I’m sure you can see where this is heading now), which write apikey would I choose?

I then assume only that users account would get backed up, correct?

Does that mean I need to run this for each user?

Is it even possible to have multiple instances of the script or at least point it at multiple settings files?

If so would that mean I then have global server settings saved with each and every users data?

If I then restore one backup would it leave me with just one user?

Can I do selective or multiple restores to rebuild parts or the whole system?

Since the ability to have one dropbox-archive that can back-up all users would be a great tool for multi account installs I wonder if the code could be changed to suit one or more accounts?

My initial idea is, since the utility is given the mySQL user and password, could the script query the emoncms database’s user table to get a list of write apikey’s (a list of one in single user applications) and cycle through those apikeys ?

If we can get that bit working then at least a full restore of all users is then easily implemented.

Perhaps then we could look at selective restores. So for example if I or one of my clients make a hash of a single account I could restore just that account. I have not looked at the restore scripts yet but again I would be happy to get involved, so that perhaps the current “restore” command works ‘as is’ for the single account user and does a full restore of all accounts for a multi account user, but something like a “restore user99” command would do just that without touching the “globals” like settings.conf, emonhub.conf and node-red etc.

Any thoughts?

2

When I first prepared this module, I simply stopped emonhub etc, and did a straight copy of the data directories, but it quickly became apparent that there were so many different variants of emoncms and OS’s that stopping and restarting the various components was going to be problematic, so instead of copying the data directories, I lifted a chunk out of the usefulscripts/backup script which Trystan wrote, which cycles through the feeds in turn exporting the data. This is what the Write API key is required for.

Installing the module in each of your remote installations could backup data to DropBox(DB) either timed or on demand by using a websocket. The dropbox-archive/backup.php script on first run takes you through the dropbox OAuth 2 process to identify the app and end user, and saves a hidden access token /home/pi/.dropbox_uploader to authenticate future sessions.
If you wanted to backup a number of remotes to just one DB account, each installation should have it’s own unique token, to safeguard other peoples data, as data can also be downloaded using the same token, and treat each installation as a separate DB app.

Obtaining the write apikey from MYSQL is probably a good place to start.

I would of course welcome your skills in moving this forward, and if we can encompass Glyn’s vision of integrating it further into emoncms, so much the better.
I can add you as a collaborator to my git account, if you wish (or move it over to the emoncms git account?? @glyn.hudson )

Paul

3

No, all my remote installs run original** emonhub and report data back to an account on a central server. I have one emoncms instance (for this project) and there can be multiple emonbases (emonhubs) reporting to any one “sites” account and there are several different sites and growing.

**The reason I mention original emonhub is because it buffers, I have the distinct advantage of being able to take down the server for maintenance (back-up/restore etc) for in excess of 24hrs with no loss of data (unless there is a power outage at one of the remote sites).

So I could just shut down the server for a few minutes (allowing no new data in) and back up once a day in the small hours (all uk businesses, so no user activity at night) and that would serve my own purpose but I wondered if the project would benefit from a shared solution, I would like to use your script but without rewriting it just for me.

4

Well I’ve just added you as a collaborator, so that you can push to the repo directly.
Can I suggest you use the ‘dev’ branch?

Paul

5

Thanks Paul, I agree working on a dev branch is preferable at this stage.

6

This is somewhat OT, but is dropbox related:

7

Fortunately, dropbox-archive doesn’t use your dropbox password or login, instead using a OAuth 2.0 secure access token for authentication.
Apart from the security re-assurance, this brings many other benefits, including that you can change your dropbox password or add 2-step verification without making any changes to dropbox-archive, plus you can remotely revoke the token at any time.

I think that the days of using a simple text password have long gone, and wherever possible I use 2 step verification, especially with dropbox.

Paul

8

Bill - thanks for posting the dropbox hack info! I changed my PW after seeing it!

9

YW, Jon.

I wasn’t trying to be alarmist. Just thought I’d pass along what I’d read earlier today.
Glad you found it useful.

10

Bringing this back to the original post… is there any interest in developing this for multiple accounts?

Paul

11

I’m still up for it, but feeling I maybe on my own here.