Migrate multigraphs from emoncms.org to local emoncms installation

jvda · 12 June 2018 20:10

Is there a way to migrate the multigraphs in emoncms.org to local emoncms installation ?

FYI I migrated the data using the backup script (GitHub - emoncms/usefulscripts: Some useful scripts for administering Emoncms accounts) but this didn’t migrate the multigraphs (and the dashboards) that I have created on emoncms.org.

The version of my local emoncms installation = Version 9.8.30 | 2018.05.08

kr
jan

jvda · 19 June 2018 09:45

I have created many multigraphs on emoncms.org which I would like to migrate to my local installation of emoncms. I already migrated the inputs and feed so that should be OK. I still want to migrate the multigraphs.

So is this possible or will this be made possible (@TrystanLea) ?

any feedback would be much appreciated.
Jan

Vster · 20 June 2018 11:56

Can I just ask how you managed to migrate the Process Lists for the Inputs?

jvda · 20 June 2018 15:10

See Cannot see any feed data after executing backup.php to migrate data from emoncms.org to local emoncms - #3 by jvda

Vster · 26 June 2018 22:13

Interesting. My data seems to be stored in /home/pi/data/phpfina etc.
If I wipe those folders and set the ownership to www-data:root the backup from Emncms.org works and all my feed data if pulled down to the Pi.

However, the sync module does then not work, which is a bit of a problem, as I need to work on some stuff while I do the cutover from remote to local.

jvda · 27 June 2018 10:02

Hi Lee,

This thread is about migration of multigraphs.

I guess that for your issue with the sync module: it is better to post a new question on this forum or check that this issue is not yet addressed in another post.

much luck
Jan.

TrystanLea · 27 June 2018 10:26

Hello @jvda, I have not yet implemented migration of multigraphs in the sync module. I will add it as an issue to the sync module.

jvda · 27 June 2018 12:46

Hi @TrystanLea

Thanks for the response.

I can be patient until it is fixed but how do I prevent that in the mean time I won’t be charged for my emoncms.org usage.

FYI: at this moment I am no longer sending any data to emoncms.org and I have backed up all feed data to my local emoncms installation. The only open thing is the replication of the multigraphs.

I guess I can delete all my inputs and feeds on emoncms.org but would that also not have an impacted on the mutligraphs that I have created as they are of course based on those feeds ?

kr
Jan.

TrystanLea · 28 June 2018 20:08

Hello @jvda, the only thing that affects billing on emoncms.org is feeds, so leaving the multigraphs on there for now will be fine. I will update once this is implemented.

jvda · 28 June 2018 21:11

Hi @TrystanLea,

Thanks for the response.

I just did a little test:
I have deleted a feed, but if I consult then a multigraph based on this feed then it appears as undefined:undefined

So would it later still be possible to migrate this multigraph allthough all feeds in the multigraphs will be undefined:undefined.

Note that as billing starts the 1st July and urgent response is needed.

kr
Jan.

pb66 · 28 June 2018 22:50

There are some less well known multigraph api calls that might help you transfer the multigraphs manually.

github.com

emoncms/emoncms/blob/9.8.31/Modules/vis/vis_controller.php#L142-L157

    
      
          /*
          MULTIGRAPH ACTIONS
          */
          
          
else if ($route->format == 'json' && $route->action == 'multigraph')
          {
              if ($route->subaction == 'get') $result = $multigraph->get(get('id'),$session['userid']);
              else if ($route->subaction == 'getlist') $result = $multigraph->getlist($session['userid']);
          
          
    else if ($session['write']) {
                  if ($route->subaction == 'new') $result = $multigraph->create($session['userid']);
                  else if ($route->subaction == 'delete') $result = $multigraph->delete(get('id'),$session['userid']);
                  else if ($route->subaction == 'set') $result = $multigraph->set(get('id'),$session['userid'],get('feedlist'),get('name'));
              }
          
          
}

e.g.

https://emoncms.org/vis/multigraph/getlist.json&apikey=1234abcd1234abcd1234abcd1234abcd

will get you a list of all your multigraphs, although the output might need fixing as there seem to be a lot of escaped characters.

The other api calls should allow you to create multigraphs too, however, as the feed id’s will not be the same you would need to look up the feednames and tags in a emoncms.org feed list and look up those names and tags in a local feed list so that each multigraph config’s feedids were correct before creating the new multigraphs.

Not an easy option by a long way, but it should be possible.

To answer your question, it depends on how @trystanlea implements the syncing as to whether you could still sync them after deleting the underlying feeds, but unfortunatly I don’t think you can, Because the sync module will not be able to determine the feed name and tag from the feedid if it has been deleted, so it won’t be able to reattach it to your local feeds as they will have different feedids.

jvda · 29 June 2018 10:36

Hi Paul, thanks for the response. It makes sense to me.

I have collected the output of

https://emoncms.org/vis/multigraph/getlist.json&apikey=4b1f...15e50
https://emoncms.org/feed/list.json&apikey=4b1....15e50

I also extracted the feed list (see 2nd https request) as this information will be lost once I delete all my feeds on emoncms.org.

I guess this way I have (in theory) at least all information to reconstruct my multigraphs in local emoncms installation (I have already replicated all feeds successfully to local emoncms installation). Am I right ?

I have been looking at emoncms/Modules/vis/multigraph_model.php at 9.8.31 · emoncms/emoncms · GitHub
and it looks like that multigraphs are only encoded by records in the multigraph mysql table - so I guess that I only need to insert the correct records in the multigraph mysql table. Is that a correct assumption ?

kr
Jan

pb66 · 29 June 2018 10:55

Yes most of the configuration is held in one field of the multigraph table in mySQL, much like the apps module.

Picking one of my multigraphs at random, I see this in the “feedlist” field of the multigraph table

[{"id":"100102","tag":"1","name":"L1-solar_power","datatype":"1","left":true,"right":false,"fill":false,"end":0,"skipmissing":false,"lineColour":"ff0000","stacked":true,"autorefresh":"5","showtag":false,"timeWindow":86400000},{"id":"100136","tag":"2","name":"L2-solar_power","datatype":"1","left":true,"right":false,"fill":false,"end":0,"skipmissing":false,"lineColour":"ffff00","stacked":true},{"id":"100170","tag":"3","name":"L3-solar_power","datatype":"1","left":true,"right":false,"fill":false,"end":0,"skipmissing":false,"lineColour":"0080ff","stacked":true},{"id":"100094","tag":"1","name":"L1-solar_current","datatype":"1","left":false,"right":true,"fill":true,"end":0,"skipmissing":false,"lineColour":"ff8080"},{"id":"100128","tag":"2","name":"L2-solar_current","datatype":"1","left":false,"right":true,"fill":true,"end":0,"skipmissing":false,"lineColour":"ffff80"},{"id":"100162","tag":"3","name":"L3-solar_current","datatype":"1","left":false,"right":true,"fill":true,"end":0,"skipmissing":false,"lineColour":"7dbeff"}]

and querying that same multigraph id via the api

https://myserver.com/vis/multigraph/get.json?id=100002

{"name":"3phaseSolar","feedlist":[{"id":"100102","tag":"1","name":"L1-solar_power","datatype":"1","left":true,"right":false,"fill":false,"end":0,"skipmissing":false,"lineColour":"ff0000","stacked":true,"autorefresh":"5","showtag":false,"timeWindow":86400000},{"id":"100136","tag":"2","name":"L2-solar_power","datatype":"1","left":true,"right":false,"fill":false,"end":0,"skipmissing":false,"lineColour":"ffff00","stacked":true},{"id":"100170","tag":"3","name":"L3-solar_power","datatype":"1","left":true,"right":false,"fill":false,"end":0,"skipmissing":false,"lineColour":"0080ff","stacked":true},{"id":"100094","tag":"1","name":"L1-solar_current","datatype":"1","left":false,"right":true,"fill":true,"end":0,"skipmissing":false,"lineColour":"ff8080"},{"id":"100128","tag":"2","name":"L2-solar_current","datatype":"1","left":false,"right":true,"fill":true,"end":0,"skipmissing":false,"lineColour":"ffff80"},{"id":"100162","tag":"3","name":"L3-solar_current","datatype":"1","left":false,"right":true,"fill":true,"end":0,"skipmissing":false,"lineColour":"7dbeff"}]}

It looks very much like the bit you want is the “feedlist” content from that reply to insert in mysql (I haven’t checked it char by char, but it looks pretty similar)

Although you may well be fine with this operation, I usually try to use api’s where I can to avoid directly editing the mySQL because sometimes the system works against you, redis, indexes, live updates etc etc all mean your updated fields might not get seen or get overwritten, but it’s definately worth a try if you are able to swap over all the feed ids.

jvda · 29 June 2018 11:15

Thanks a lot, once I have time (not really soon), I will give it a shot.