Hi,
Is there any way to do a bulk data export from Emoncms? I have decided to move to a self hosted system and would like to download all data held on my account to my local install?
Is there an easy way of doing this? Bulk CSV etc?
Andrew
Hello Andrew, there are two routes for exporting data from emoncms.org and importing into a self-hosted installation.
The first is the sync module, full details are available here: GitHub - emoncms/sync: In development: Download or upload emoncms feed data between local and remote emoncms server's
The other is the backup script available in the usefulscripts repo, documented here: GitHub - emoncms/usefulscripts: Some useful scripts for administering Emoncms accounts
There is an ongoing discussion about the sync module and exporting dashboards on the billing thread Emoncms.org billing
Iâve run emonbase update on my local hardware but I canât find any reference to a sync module from the admin page.
Where is the sync option supposed to be?
Iâm looking here http://192.168.178.60/emoncms/admin/view
Server Information
Emoncms Version low-write 9.8.30 : 2018.05.08 Modules Administration : App v1.1.0 : Backup v1.1.2 : EmonHub Config v1.0.0 : Dashboard v1.0.1 : EventProcesses : Feed : Graph v1.2.0 : Input : postprocess : CoreProcess : Schedule : setup : Time : User : Visualisation : WiFi v1.0.0 Buffer loading⌠Writer Daemon is running with sleep 60s Server OS Linux 4.4.26-v7+ Host emonpi emonpi (127.0.1.1) Date 2018-05-24 21:01:24 AEST Uptime 21:01:24 up 8 days, 7:54, 0 users, load average: 1.96, 1.59, 1.31 HTTP Server Apache/2.4.10 (Raspbian) HTTP/1.1 CGI/1.1 80 MySQL Version 5.5.52-0+deb8u1 Host localhost (127.0.0.1) Date 2018-05-24 21:01:24 (UTC 10:00ââ) Stats Uptime: 722638 Threads: 5 Questions: 2957 Slow queries: 0 Opens: 68 Flush tables: 1 Open tables: 51 Queries per second avg: 0.004 Redis Version 2.8.17 Host localhost:6379 (127.0.0.1) Size 106 keys (532.21K) Uptime 8 days MQTT Version 1.4.10 Host localhost:1883 (127.0.0.1) Pi CPU Temp 54.77°C Release emonSD-07Nov16 File-system Set root file-system temporarily to read-write, (default read-only) Memory RAM Used: 31.96% Total: 973.11 MB Used: 311.05 MB Free: 662.05 MB Disk Mount Stats / Used: 66.86% Total: 3.33 GB Used: 2.23 GB Free: 964.82 MB /boot Used: 34.72% Total: 59.95 MB Used: 20.81 MB Free: 39.13 MB /home/pi/data Used: 8.34% Total: 3.46 GB Used: 295.47 MB Free: 2.99 GB PHP Version 5.6.27-0+deb8u1 (Zend Version 2.6.0) Modules apache2handler : bcmath : bz2 : calendar : Core v5.6.27-0+deb8u1 : ctype : curl : date v5.6.27-0+deb8u1 : dba : dio v0.0.4RC4 : dom v20031129 : ereg : exif v1.4 : fileinfo v1.0.5 : filter v0.11.0 : ftp : gettext : hash v1.0 : iconv : json v1.3.6 : libxml : mbstring : mcrypt : mhash : mosquitto v0.3.0 : mysql v1.0 : mysqli v0.1 : openssl : pcre : PDO v1.0.4dev : pdo_mysql v1.0.2 : Phar v2.0.2 : posix : readline v5.6.27-0+deb8u1 : redis v2.2.7 : Reflection : session : shmop : SimpleXML v0.1 : soap : sockets : SPL v0.2 : standard v5.6.27-0+deb8u1 : sysvmsg : sysvsem : sysvshm : tokenizer v0.1 : wddx : xml : xmlreader v0.1 : xmlwriter v0.1 : Zend OPcache v7.0.6-devFE : zip v1.12.5 : zlib v2.0 :
Client Information
HTTP Browser Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/11.1 Safari/605.1.15 Screen Resolution 1440 x 900 Window Size 1440 x 816
Looking at your âServer informationâ the sync module âappearsâ to not be installed, there should be an entry âsyncâ in the Emoncms => Modules section.
It maybe that itâs not fully installed correctly as the sync module gets installed in /home/pi and part of it is supposed to be symlinked to the emoncms/Modules folder.
Can you see any mention of it in the emonpiupdate.log?
I canât see any reference to âSyncâ when I search the log file.
I hate coming across posts that are full of log text, but I canât think of a better wayâŚ
I assumed the sync module would be part of the software update process. Is there some info available to get my EmonBase fully updated ?
I2C LCD NOT DETECTED on either 0x['27', '3f'] ...exiting LCD script
Starting emonPi Update >
via service-runner-update.sh
EUID: 1000
Argument: rfm69pi
Thu 24 May 20:57:00 AEST 2018
#############################################################
emonSD version: emonSD-07Nov16
emonSD base image check passed...continue update
#############################################################
Filesystem is unlocked - Write access
type ' rpi-ro ' to lock
git pull /home/pi/emonpi
* master
wifiap
On branch master
Your branch is behind 'origin/master' by 46 commits, and can be fast-forwarded.
(use "git pull" to update your local branch)
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git checkout -- <file>..." to discard changes in working directory)
modified: rc.local_jessieminimal
Untracked files:
(use "git add <file>..." to include in what will be committed)
1
hardware/emonpi/emonpi2c/
no changes added to commit (use "git add" and/or "git commit -a")
Updating 31f25b4..f81c282
git pull /home/pi/RFM2Pi
* master
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working directory clean
Already up-to-date.
git pull /home/pi/emonhub
K0den-wibeee_interface
* emon-pi
hmm01i-syslogging
On branch emon-pi
Your branch is up-to-date with 'origin/emon-pi'.
nothing to commit, working directory clean
Already up-to-date.
git pull /home/pi/oem_openHab
* master
On branch master
Your branch is up-to-date with 'origin/master'.
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git checkout -- <file>..." to discard changes in working directory)
modified: oem.items
modified: oem.sitemap
no changes added to commit (use "git add" and/or "git commit -a")
Already up-to-date.
git pull /home/pi/usefulscripts
* master
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working directory clean
git pull /home/pi/huawei-hilink-status
* master
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working directory clean
Running RFM69Pi firmware update:
RFM69Pi Update Started
date
Getting latest RFM69Pi release info from github
Latest RFM69Pi firmware: V1.4.0
downloading latest RFM69Pi firmware from github releases:
https://github.com/openenergymonitor/RFM2Pi/releases/download/1.4.0/firmware.hex
Saving to /home/pi/data/firmware/rfm69pi-1.4.0.hex
=================================
RFM69Pi update started
=================================
Flashing RFM69Pi with V 1.4.0
avrdude-original: Using autoreset DTR on GPIO Pin 7
Flashing RFM69Pi with V 1.4.0 done
Start emonhub update script:
=================================
EmonPi update started
=================================
Running emonhub automatic node addition script
EUID: 1000
EUID: 1000
[[5]]
Node 5 already present
[[6]]
Node 6 already present
[[7]]
Node 7 already present
[[8]]
Node 8 already present
[[9]]
Node 9 already present
[[10]]
Node 10 already present
[[11]]
Node 11 already present
[[12]]
Node 12 already present
[[13]]
Node 13 already present
[[14]]
Node 14 already present
[[19]]
Node 19 already present
[[20]]
Node 20 already present
[[21]]
Node 21 already present
[[22]]
Node 22 already present
[[23]]
Node 23 already present
[[24]]
Node 24 already present
[[25]]
Node 25 already present
[[26]]
Node 26 already present
Start emoncms update:
=================================
Emoncms update started
Update script V1.1.0
Thu 24 May 20:57:27 AEST 2018
#############################################################
emonSD version: emonSD-07Nov16
emonSD base image check pass...continue update
#############################################################
EUID: 1000
Checking cron tab for service runner entry...
service runner crontab entry already installed
current settings.php md5: f9fa74b4e54adb7d760e875e0764955e
Default settings.php md5: f9fa74b4e54adb7d760e875e0764955e
git pull /var/www/emoncms
dev-mosquitto-php
master
* stable
symlinked_modules
On branch stable
Your branch is up-to-date with 'origin/stable'.
nothing to commit, working directory clean
Already up-to-date.
NEW default settings.php md5: f9fa74b4e54adb7d760e875e0764955e
settings.php had NOT been user modifed
No update required to settings.php
git pull /var/www/emoncms/Modules/app
9.0
* stable
On branch stable
Your branch is up-to-date with 'origin/stable'.
nothing to commit, working directory clean
Already up-to-date.
Your branch is up-to-date with 'origin/stable'.
git pull /var/www/emoncms/Modules/config
9.0
* stable
On branch stable
Your branch is up-to-date with 'origin/stable'.
nothing to commit, working directory clean
Already up-to-date.
Your branch is up-to-date with 'origin/stable'.
git pull /var/www/emoncms/Modules/wifi
9.0
* stable
On branch stable
Your branch is up-to-date with 'origin/stable'.
nothing to commit, working directory clean
Already up-to-date.
Your branch is up-to-date with 'origin/stable'.
git pull /var/www/emoncms/Modules/dashboard
On branch stable
Your branch and 'origin/stable' have diverged,
and have 1 and 45 different commits each, respectively.
(use "git pull" to merge the remote branch into yours)
nothing to commit, working directory clean
Your branch and 'origin/stable' have diverged,
and have 1 and 45 different commits each, respectively.
(use "git pull" to merge the remote branch into yours)
git pull /var/www/emoncms/Modules/graph
master
* stable
On branch stable
Your branch is up-to-date with 'origin/stable'.
nothing to commit, working directory clean
Already up-to-date.
Your branch is up-to-date with 'origin/stable'.
git pull /home/pi/postprocess
Your branch is up-to-date with 'remotes/origin/emonpi'.
Already up-to-date.
git pull /home/pi/backup
master
* stable
On branch stable
Your branch is up-to-date with 'origin/stable'.
nothing to commit, working directory clean
Already up-to-date.
Your branch is up-to-date with 'origin/stable'.
Cannot find emoncms-setup module, please update ~/emonpi repo
update mqtt_input systemd unit file
Update Emoncms database
[]
Restarting Services...
Restarting OpenEnergyMonitor emonHub: emonhub has been restarted ok.
Log is turned off
Restarting feedwriter
Restarting openhab (via systemctl): openhab.service.
set log rotate config owner to root
Restarting Services...
Starting emonPi LCD service..
Filesystem is locked - Read Only access
type ' rpi-rw ' to unlock
Thu 24 May 20:59:06 AEST 2018
...................
emonPi update done
It has been included, but the updater script itself needs to update for the sync module changes to be available.
Looking at your emonpiupdate log, there is a local change that blocked the emonpi repo from being updated
git pull /home/pi/emonpi
* master
wifiap
On branch master
Your branch is behind 'origin/master' by 46 commits, and can be fast-forwarded.
(use "git pull" to update your local branch)
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git checkout -- <file>..." to discard changes in working directory)
modified: rc.local_jessieminimal
Untracked files:
(use "git add <file>..." to include in what will be committed)
you need to undo those changes to rc.local_jessieminimal or âgit stashâ them and run the updater again so that the emonpi repo gets updated, currently your log says it is using v1.1.0 of the emoncms update script
Emoncms update started
Update script V1.1.0
and the latest version that includes the sync install/update code is v1.1.1
emonpi/emoncmsupdate at master ¡ openenergymonitor/emonpi ¡ GitHub.
Thanks Paul, I will see if I can figure out âgit stashâ.
Iâm pretty sure I had been playing with uploading SMA data but that can be discarded.
Iâd like to find the simplest way to do a âfactory restoreâ without losing my logged dataâŚ
All resolved.
First time Iâve had to use git from the command line.
1 Like
The only way to do that would be to backup your data via the emonpi/emoncms backup module, carry out a full âfactory resetâ and then import your data.