Community
OpenEnergyMonitor

Community

Sync of daily feeds to local EmonPi


(Daniel) #1

I tried to create a backup of my online data at emoncms.org to my local EmonPi version. The Sync module doesn’t show me daily-feeds (e.g. min / max or kWh/d). It looks like that the script only catches feeds with the parameter engine == 5 and daily feeds have the parameter engine == 2 (found in sync_controller.php).

I also tried to make a backup via the usefulscript backup routine but I was unable to access the data afterwards. EmonPi showed me a lot of errors according a missing meta file to the feed.

Is there any possibility to download and integrate daily feeds into a local installation ?


Migrate old data from emoncms.org to localhost server
(Trystan Lea) #2

Im working on it :slight_smile:


(Jd) #3

Hi, I am also trying to do the same but without success…
My local emoncms reports authentication error when entering my emoncms.org credential in the Sync Interface…
It would be nice to be able to recover my 4 years measurements localy !

Thanks


(Daniel) #4

I’ve updated to the recent version of Emonpi (2018.05.08) and it’s working fine. I was able to copy all historic data.


(Jd) #5

Ok great,
I have installed the downloaded emonSD-26Oct17 …
How can I get this new Version as update ?
Could you give me commandos steps please?
Thanks


(Daniel) #6

You can find an EmonPi update button in the menu Setup > Administration.


(Jd) #7

I did the update
When I try to sync Inputs I have following message
using host https://emoncms.org + my account

Authentication failure, username or password incorrect

Log:
2018-06-05 12:19:11.956|WARN|sync_model.php|remote save
2018-06-05 12:19:12.572|WARN|sync_model.php|{“success”:false,“message”:“Incorrect authentication”}
2018-06-05 12:19:27.737|WARN|sync_model.php|remote save
2018-06-05 12:19:28.520|WARN|sync_model.php|{“success”:false,“message”:“Incorrect authentication”}

What I can notice is that the time is not right, this can perhaps affect the https connection …
how do I get the real and right time configured on the pi ?

any other idea ?


(Jd) #8

Don’t know how to force another branch for update… here my update Log…
Notice I have an old pi2 and RFM2pi as RF module

Filesystem is unlocked - Write access
type ' rpi-ro ' to lock
I2C LCD NOT DETECTED on either 0x['27', '3f'] ...exiting LCD script
Starting emonPi Update >
via service-runner-update.sh
Service Runner update script V1.0.0
EUID: 1000
Argument: emonpi
Thu  7 Jun 16:26:16 UTC 2018
#############################################################

emonSD version: emonSD-26Oct17

emonSD base image check passed...continue update

#############################################################
Filesystem is unlocked - Write access
type ' rpi-ro ' to lock
git pull /home/pi/emonpi
* master
  wifiap
On branch master
Your branch is up-to-date with 'origin/master'.
Untracked files:
  (use "git add <file>..." to include in what will be committed)

	1
	hardware/emonpi/emonpi2c/

nothing added to commit but untracked files present (use "git add" to track)
Updating 5c08fc4..223cd0a
Fast-forward
 emoncms-wifiscan.sh  |  3 +++
 emoncms_wifiscan.php | 27 +++++++++++++++++++++++++++
 service-runner       | 29 ++++++++++++++---------------
 3 files changed, 44 insertions(+), 15 deletions(-)
 create mode 100755 emoncms-wifiscan.sh
 create mode 100644 emoncms_wifiscan.php
git pull /home/pi/RFM2Pi
* master
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working directory clean
Already up-to-date.
git pull /home/pi/emonhub
  K0den-wibeee_interface
* emon-pi
  hmm01i-syslogging
On branch emon-pi
Your branch is up-to-date with 'origin/emon-pi'.
nothing to commit, working directory clean
Already up-to-date.
git pull /home/pi/oem_openHab
* master
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working directory clean
Already up-to-date.
git pull /home/pi/usefulscripts
* master
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working directory clean
Already up-to-date.
git pull /home/pi/huawei-hilink-status
* master
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working directory clean
Already up-to-date.

Start emonPi Atmega328 firmware update:

=================================
EmonPi update started
=================================

EUID: 1000

Requirement already up-to-date: paho-mqtt in /usr/local/lib/python2.7/dist-packages
Stopping OpenEnergyMonitor emonHub: emonhub has been stopped ok.
Start ATmega328 serial upload using avrdude with latest.hex
Discrete Sampling
avrdude -c arduino -p ATMEGA328P -P /dev/ttyAMA0 -b 115200 -U flash:w:/home/pi/emonpi/firmware/compiled/latest.hex
avrdude-original: Using autoreset DTR on GPIO Pin 7
Starting OpenEnergyMonitor emonHub: emonhub has been started ok.
[email protected] node_modules/node-red-node-emoncms


Start emonhub update script:

=================================
EmonPi update started
=================================
Running emonhub automatic node addition script
EUID: 1000
EUID: 1000
[[5]]
Node 5 already present
[[6]]
Node 6 already present
[[7]]
Node 7 already present
[[8]]
Node 8 already present
[[9]]
Node 9 already present
[[10]]
Node 10 already present
[[11]]
Node 11 already present
[[12]]
Node 12 already present
[[13]]
Node 13 already present
[[14]]
Node 14 already present
[[19]]
Node 19 already present
[[20]]
Node 20 already present
[[21]]
Node 21 already present
[[22]]
Node 22 already present
[[23]]
Node 23 already present
[[24]]
Node 24 already present
[[25]]
Node 25 already present
[[26]]
Node 26 already present

Start emoncms update:

=================================
Emoncms update started
Emoncms update script V1.1.1

Thu  7 Jun 16:28:34 UTC 2018

#############################################################

emonSD version: emonSD-26Oct17

emonSD base image check pass...continue update

#############################################################
EUID: 1000
Checking cron tab for service runner entry...
service runner crontab entry already installed

current settings.php md5: b895b8f85cf79351470a1f5b14eace91
Default settings.php md5: f9fa74b4e54adb7d760e875e0764955e

git pull /var/www/emoncms
  dev-mosquitto-php
  master
* stable
  symlinked_modules
On branch stable
Your branch is up-to-date with 'origin/stable'.
Untracked files:
  (use "git add <file>..." to include in what will be committed)

	emoncms

nothing added to commit but untracked files present (use "git add" to track)
Already up-to-date.

NEW default settings.php md5: f9fa74b4e54adb7d760e875e0764955e
settings.php HAS been user modified
settings.php not updated


git pull /var/www/emoncms/Modules/app
  9.0
* stable
On branch stable
Your branch is up-to-date with 'origin/stable'.
nothing to commit, working directory clean
Already up-to-date.
Your branch is up-to-date with 'origin/stable'.

git pull /var/www/emoncms/Modules/config
  9.0
* stable
On branch stable
Your branch is up-to-date with 'origin/stable'.
nothing to commit, working directory clean
Already up-to-date.
Your branch is up-to-date with 'origin/stable'.

git pull /var/www/emoncms/Modules/wifi
  9.0
* stable
On branch stable
Your branch is up-to-date with 'origin/stable'.
nothing to commit, working directory clean
Already up-to-date.
Your branch is up-to-date with 'origin/stable'.
git pull /var/www/emoncms/Modules/dashboard
On branch stable
Your branch is up-to-date with 'origin/stable'.
nothing to commit, working directory clean
Already up-to-date.
Your branch is up-to-date with 'origin/stable'.

git pull /var/www/emoncms/Modules/graph
  master
* stable
On branch stable
Your branch is up-to-date with 'origin/stable'.
nothing to commit, working directory clean
Already up-to-date.
Your branch is up-to-date with 'origin/stable'.

git pull /home/pi/postprocess
Your branch is up-to-date with 'remotes/origin/emonpi'.
Updating c827999..7cc1498
Fast-forward
 postprocess-module/postprocess_controller.php | 26 +++++---------------------
 1 file changed, 5 insertions(+), 21 deletions(-)
git pull /home/pi/sync
Your branch is up-to-date with 'origin/master'.
Updating e88a887..adaf7f7
Fast-forward
 sync-module/sync_controller.php | 19 +++++++++++++++++--
 sync-module/sync_model.php      | 15 ---------------
 2 files changed, 17 insertions(+), 17 deletions(-)
git pull /home/pi/backup
  master
* stable
On branch stable
Your branch is up-to-date with 'origin/stable'.
nothing to commit, working directory clean
Already up-to-date.
Your branch is up-to-date with 'origin/stable'.

update mqtt_input systemd unit file

Update Emoncms database
[]

Restarting Services...
Restarting OpenEnergyMonitor emonHub: emonhub has been restarted ok.
Log is turned off
Restarting feedwriter

Restarting openhab (via systemctl): openhab.service.


set log rotate config owner to root
Restarting Services...


Starting emonPi LCD service..

Filesystem is locked - Read Only access
type ' rpi-rw ' to unlock
Thu  7 Jun 16:29:52 UTC 2018


...................
emonPi update done

emonpiupdate.zip (1.9 KB)


(Jd) #9

Has anyone an Idea how I can get an update Version where the Sync Module works ?
What should I change to get another Version that the Stable one when I call update ?

Thanks


(Lee) #10

Hi,
I’m having similar issues.
I updated the Sync module by ssh’ing into the Pi and getting the new version from GitHub.
From memory (I’m not at home)
rpi-rw
cd sync
git pull

That said, I’m still not able to get all the data, dashes and Processes out of EmonCMS.org correctly yet. I think @TrystanLea is still working on it.


(Trystan Lea) #11

The sync module should update ok as part of the emonpi update process, here’s an snippet out of my emonpiupdate.log from my latest call to emonpi update:

git pull /home/pi/postprocess
Your branch is up-to-date with 'remotes/origin/emonpi'.
Already up-to-date.
git pull /home/pi/sync
Your branch is up-to-date with 'origin/master'.
Already up-to-date.

What are you still missing?


(Jd) #12

Hi Tristan,
I pulled all and it says it is uptodate
I stil have the same issue with Authentication Failure …
I don’t know why … Thanks

2018-06-09 10:18:50.461|WARN|phpmqtt_input.php|Connecting to MQTT server: Connection Accepted.: code: 0
2018-06-13 12:21:47.067|WARN|sync_model.php|remote save
2018-06-13 12:21:47.722|WARN|sync_model.php|{"success":false,"message":"Incorrect authentication"}

(Trystan Lea) #13

I know its asking the obvious, are you using the username and password from the emoncms.org account? could you try re-entering them?


(Jd) #14

Yes i tried with emoncms.org user and pwd
i tried to change the pwd.
But no results. Still the same error.


(Jd) #15

Hi,

I finaly managed to sync my local emoncms.
after requesting a new password for my emoncms.org account.
I just noticed that it did not sync the process of each input, I created them again manually.
I also had to set again the feed valus to be used for each dashboard element.

Now I just need to get all my historic data for the last 4 years. I hope using usefulscripts ?


(Jd) #16

I executed the backup script and got all my remote feeds into my local emoncms
but…
When I try to access a feed graph, i have followind error:

Request error
Notice: Undefined offset: 0 in /var/www/emoncms/Modules/feed/feed_model.php on line 557
Notice: Undefined offset: 1 in /var/www/emoncms/Modules/feed/feed_model.php on line 557
Warning: Cannot modify header information - headers already sent by (output started at /var/www/emoncms/Modules/feed/feed_model.php:557) in /var/www/emoncms/index.php on line 247
{“success”:false,“message”:“Error reading meta data feedid=19258”}

@TrystanLea
Could you help me solve this hopefully last problem ?
Thanks


(Jd) #17

Hi
Has someone the same problem with Graphs after Sync beetween local EmonCMS and .org ?
It to have a problem with the imported Feed IDs …

Thanks


(Robert Wall) #18

Your local emonCMS and emoncms.org are different, but if you look at other threads, moves are afoot to bring the two versions closer together. If you watch that/those threads, there might be updates coming that will solve your problem.

It would appear that @TrystanLea didn’t see your July 4th post and question.


(Trystan Lea) #19

@JideyF Did you get the sync module to work in the end, or did you download the data with usefulscripts? The error is resulting from this line of code here which is to do with the redis buffer, I wonder if you could try temporarily turning the redis buffer off in settings.php?

The redis buffer is important for the emonpi as it helps reduce SD card wear increasing livespan so we can hopefully resolve the situation so that you can turn it back on again.


(Jd) #20

Hi Trystan,

I reset my emonCMS.org user passwd and after this I got the sync module to work yes.
All data was synced, feed size are the same on local and remote emonCMS
What has not being synced are the process lists …
So I recreated them and configured them to write to the synced feed

But now I have problems with reading these imported feeds Request error
Notice : Undefined offset: 0 in /var/www/emoncms/Modules/feed/feed_model.php on line 557

Notice : Undefined offset: 1 in /var/www/emoncms/Modules/feed/feed_model.php on line 557

Warning : Cannot modify header information - headers already sent by (output started at /var/www/emoncms/Modules/feed/feed_model.php:557) in /var/www/emoncms/index.php on line 247
{“success”:false,“message”:“Error reading meta data feedid=31570”}

Log says
WARN|PHPFina.php|post_bulk_prepare() failed to fetch meta feedid=233060
.

I tryed to turn redis to false in settings but then I get an error …

settings.php file error

buffered writing requires redis but its disabled, check settings: settings[‘redisbuffer’][‘enabled’], redis_enabled