Emoncms.org - Bulk data export

Hi,
Is there any way to do a bulk data export from Emoncms? I have decided to move to a self hosted system and would like to download all data held on my account to my local install?
Is there an easy way of doing this? Bulk CSV etc?
Andrew

Hello Andrew, there are two routes for exporting data from emoncms.org and importing into a self-hosted installation.

The first is the sync module, full details are available here: GitHub - emoncms/sync: In development: Download or upload emoncms feed data between local and remote emoncms server's

The other is the backup script available in the usefulscripts repo, documented here: GitHub - emoncms/usefulscripts: Some useful scripts for administering Emoncms accounts

There is an ongoing discussion about the sync module and exporting dashboards on the billing thread Emoncms.org billing

I’ve run emonbase update on my local hardware but I can’t find any reference to a sync module from the admin page.

Where is the sync option supposed to be?

I’m looking here http://192.168.178.60/emoncms/admin/view

Server Information
Emoncms Version low-write 9.8.30 : 2018.05.08
Modules Administration : App v1.1.0 : Backup v1.1.2 : EmonHub Config v1.0.0 : Dashboard v1.0.1 : EventProcesses : Feed : Graph v1.2.0 : Input : postprocess : CoreProcess : Schedule : setup : Time : User : Visualisation : WiFi v1.0.0
Buffer loading…
Writer Daemon is running with sleep 60s
Server OS Linux 4.4.26-v7+
Host emonpi emonpi (127.0.1.1)
Date 2018-05-24 21:01:24 AEST
Uptime 21:01:24 up 8 days, 7:54, 0 users, load average: 1.96, 1.59, 1.31
HTTP Server Apache/2.4.10 (Raspbian) HTTP/1.1 CGI/1.1 80
MySQL Version 5.5.52-0+deb8u1
Host localhost (127.0.0.1)
Date 2018-05-24 21:01:24 (UTC 10:00‌​)
Stats Uptime: 722638 Threads: 5 Questions: 2957 Slow queries: 0 Opens: 68 Flush tables: 1 Open tables: 51 Queries per second avg: 0.004
Redis Version 2.8.17
Host localhost:6379 (127.0.0.1)
Size 106 keys (532.21K)
Uptime 8 days
MQTT Version 1.4.10
Host localhost:1883 (127.0.0.1)
Pi CPU Temp 54.77°C
Release emonSD-07Nov16
File-system Set root file-system temporarily to read-write, (default read-only)
Memory RAM Used: 31.96% Total: 973.11 MB Used: 311.05 MB Free: 662.05 MB
Disk Mount Stats
/ Used: 66.86% Total: 3.33 GB Used: 2.23 GB Free: 964.82 MB
/boot Used: 34.72% Total: 59.95 MB Used: 20.81 MB Free: 39.13 MB
/home/pi/data Used: 8.34% Total: 3.46 GB Used: 295.47 MB Free: 2.99 GB
PHP Version 5.6.27-0+deb8u1 (Zend Version 2.6.0)
Modules apache2handler : bcmath : bz2 : calendar : Core v5.6.27-0+deb8u1 : ctype : curl : date v5.6.27-0+deb8u1 : dba : dio v0.0.4RC4 : dom v20031129 : ereg : exif v1.4 : fileinfo v1.0.5 : filter v0.11.0 : ftp : gettext : hash v1.0 : iconv : json v1.3.6 : libxml : mbstring : mcrypt : mhash : mosquitto v0.3.0 : mysql v1.0 : mysqli v0.1 : openssl : pcre : PDO v1.0.4dev : pdo_mysql v1.0.2 : Phar v2.0.2 : posix : readline v5.6.27-0+deb8u1 : redis v2.2.7 : Reflection : session : shmop : SimpleXML v0.1 : soap : sockets : SPL v0.2 : standard v5.6.27-0+deb8u1 : sysvmsg : sysvsem : sysvshm : tokenizer v0.1 : wddx : xml : xmlreader v0.1 : xmlwriter v0.1 : Zend OPcache v7.0.6-devFE : zip v1.12.5 : zlib v2.0 :

Client Information
HTTP Browser Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/11.1 Safari/605.1.15
Screen Resolution 1440 x 900
Window Size 1440 x 816

Looking at your “Server information” the sync module “appears” to not be installed, there should be an entry “sync” in the Emoncms => Modules section.

It maybe that it’s not fully installed correctly as the sync module gets installed in /home/pi and part of it is supposed to be symlinked to the emoncms/Modules folder.

Can you see any mention of it in the emonpiupdate.log?

I can’t see any reference to ‘Sync’ when I search the log file.

I hate coming across posts that are full of log text, but I can’t think of a better way…

I assumed the sync module would be part of the software update process. Is there some info available to get my EmonBase fully updated ?

I2C LCD NOT DETECTED on either 0x['27', '3f'] ...exiting LCD script
Starting emonPi Update >
via service-runner-update.sh
EUID: 1000
Argument: rfm69pi
Thu 24 May 20:57:00 AEST 2018
#############################################################

emonSD version: emonSD-07Nov16

emonSD base image check passed...continue update

#############################################################
Filesystem is unlocked - Write access
type ' rpi-ro ' to lock
git pull /home/pi/emonpi
* master
  wifiap
On branch master
Your branch is behind 'origin/master' by 46 commits, and can be fast-forwarded.
  (use "git pull" to update your local branch)
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

	modified:   rc.local_jessieminimal

Untracked files:
  (use "git add <file>..." to include in what will be committed)

	1
	hardware/emonpi/emonpi2c/

no changes added to commit (use "git add" and/or "git commit -a")
Updating 31f25b4..f81c282
git pull /home/pi/RFM2Pi
* master
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working directory clean
Already up-to-date.
git pull /home/pi/emonhub
  K0den-wibeee_interface
* emon-pi
  hmm01i-syslogging
On branch emon-pi
Your branch is up-to-date with 'origin/emon-pi'.
nothing to commit, working directory clean
Already up-to-date.
git pull /home/pi/oem_openHab
* master
On branch master
Your branch is up-to-date with 'origin/master'.
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

	modified:   oem.items
	modified:   oem.sitemap

no changes added to commit (use "git add" and/or "git commit -a")
Already up-to-date.
git pull /home/pi/usefulscripts
* master
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working directory clean
git pull /home/pi/huawei-hilink-status
* master
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working directory clean

Running RFM69Pi firmware update:
RFM69Pi Update Started
date
Getting latest RFM69Pi release info from github
Latest RFM69Pi firmware: V1.4.0
downloading latest RFM69Pi firmware from github releases:
https://github.com/openenergymonitor/RFM2Pi/releases/download/1.4.0/firmware.hex
Saving to /home/pi/data/firmware/rfm69pi-1.4.0.hex

=================================
RFM69Pi update started
=================================

Flashing RFM69Pi with V 1.4.0

avrdude-original: Using autoreset DTR on GPIO Pin 7
Flashing RFM69Pi with V 1.4.0  done


Start emonhub update script:

=================================
EmonPi update started
=================================
Running emonhub automatic node addition script
EUID: 1000
EUID: 1000
[[5]]
Node 5 already present
[[6]]
Node 6 already present
[[7]]
Node 7 already present
[[8]]
Node 8 already present
[[9]]
Node 9 already present
[[10]]
Node 10 already present
[[11]]
Node 11 already present
[[12]]
Node 12 already present
[[13]]
Node 13 already present
[[14]]
Node 14 already present
[[19]]
Node 19 already present
[[20]]
Node 20 already present
[[21]]
Node 21 already present
[[22]]
Node 22 already present
[[23]]
Node 23 already present
[[24]]
Node 24 already present
[[25]]
Node 25 already present
[[26]]
Node 26 already present

Start emoncms update:

=================================
Emoncms update started
Update script V1.1.0

Thu 24 May 20:57:27 AEST 2018

#############################################################

emonSD version: emonSD-07Nov16

emonSD base image check pass...continue update

#############################################################
EUID: 1000
Checking cron tab for service runner entry...
service runner crontab entry already installed

current settings.php md5: f9fa74b4e54adb7d760e875e0764955e
Default settings.php md5: f9fa74b4e54adb7d760e875e0764955e

git pull /var/www/emoncms
  dev-mosquitto-php
  master
* stable
  symlinked_modules
On branch stable
Your branch is up-to-date with 'origin/stable'.
nothing to commit, working directory clean
Already up-to-date.

NEW default settings.php md5: f9fa74b4e54adb7d760e875e0764955e
settings.php had NOT been user modifed
No update required to settings.php


git pull /var/www/emoncms/Modules/app
  9.0
* stable
On branch stable
Your branch is up-to-date with 'origin/stable'.
nothing to commit, working directory clean
Already up-to-date.
Your branch is up-to-date with 'origin/stable'.

git pull /var/www/emoncms/Modules/config
  9.0
* stable
On branch stable
Your branch is up-to-date with 'origin/stable'.
nothing to commit, working directory clean
Already up-to-date.
Your branch is up-to-date with 'origin/stable'.

git pull /var/www/emoncms/Modules/wifi
  9.0
* stable
On branch stable
Your branch is up-to-date with 'origin/stable'.
nothing to commit, working directory clean
Already up-to-date.
Your branch is up-to-date with 'origin/stable'.
git pull /var/www/emoncms/Modules/dashboard
On branch stable
Your branch and 'origin/stable' have diverged,
and have 1 and 45 different commits each, respectively.
  (use "git pull" to merge the remote branch into yours)
nothing to commit, working directory clean
Your branch and 'origin/stable' have diverged,
and have 1 and 45 different commits each, respectively.
  (use "git pull" to merge the remote branch into yours)

git pull /var/www/emoncms/Modules/graph
  master
* stable
On branch stable
Your branch is up-to-date with 'origin/stable'.
nothing to commit, working directory clean
Already up-to-date.
Your branch is up-to-date with 'origin/stable'.

git pull /home/pi/postprocess
Your branch is up-to-date with 'remotes/origin/emonpi'.
Already up-to-date.
git pull /home/pi/backup
  master
* stable
On branch stable
Your branch is up-to-date with 'origin/stable'.
nothing to commit, working directory clean
Already up-to-date.
Your branch is up-to-date with 'origin/stable'.
Cannot find emoncms-setup module, please update ~/emonpi repo

update mqtt_input systemd unit file

Update Emoncms database
[]

Restarting Services...
Restarting OpenEnergyMonitor emonHub: emonhub has been restarted ok.
Log is turned off
Restarting feedwriter

Restarting openhab (via systemctl): openhab.service.


set log rotate config owner to root
Restarting Services...


Starting emonPi LCD service..

Filesystem is locked - Read Only access
type ' rpi-rw ' to unlock
Thu 24 May 20:59:06 AEST 2018


...................
emonPi update done

It has been included, but the updater script itself needs to update for the sync module changes to be available.

Looking at your emonpiupdate log, there is a local change that blocked the emonpi repo from being updated

git pull /home/pi/emonpi
* master
  wifiap
On branch master
Your branch is behind 'origin/master' by 46 commits, and can be fast-forwarded.
  (use "git pull" to update your local branch)
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

	modified:   rc.local_jessieminimal

Untracked files:
  (use "git add <file>..." to include in what will be committed)

you need to undo those changes to rc.local_jessieminimal or “git stash” them and run the updater again so that the emonpi repo gets updated, currently your log says it is using v1.1.0 of the emoncms update script

Emoncms update started
Update script V1.1.0

and the latest version that includes the sync install/update code is v1.1.1

emonpi/emoncmsupdate at master ¡ openenergymonitor/emonpi ¡ GitHub.

Thanks Paul, I will see if I can figure out “git stash”.

I’m pretty sure I had been playing with uploading SMA data but that can be discarded.

I’d like to find the simplest way to do a ‘factory restore’ without losing my logged data…

All resolved.

First time I’ve had to use git from the command line.

1 Like

The only way to do that would be to backup your data via the emonpi/emoncms backup module, carry out a full “factory reset” and then import your data.