Community
OpenEnergyMonitor

OpenEnergyMonitor Community

Python based Emoncms backup utility

I’ve put together a small python based backup utility for emoncms, to make downloading feeds from emoncms.org or any remote emoncms server without importing into a local installation of emoncms a bit easier. The tool also includes a script for converting the downloaded phpfina and phptimeseries binary data into CSV data.

This is much faster and more efficient than using the emoncms CSV export api to download all feeds. It’s also desgined to make it relatively easy to download multiple emoncms accounts, as each account is saved in a separate local directory with the same name as the account username.

The tool is available in the emoncms/usefulscripts repository here: usefulscripts/backup_py at master · emoncms/usefulscripts · GitHub


For anyone interested in downloading data from emoncms.org on to a local installation of emoncms I would recommend using either the php script here: usefulscripts/backup at master · emoncms/usefulscripts · GitHub or the emoncms sync module that is available as standard on the emonSD image: GitHub - emoncms/sync: In development: Download or upload emoncms feed data between local and remote emoncms server's

Cheers :slight_smile:

3 Likes

I’ve recently setup a windows PC for the first time in years to use some software that wasnt easily accesible on Linux and so I thought I’d take the opportunity to test using the above emoncms backup utility on windows, it works fine!

Steps to run on Windows

1. I started here with the latest version of python for windows:
Download Python | Python.org

2. I then needed to install the python requests module. Open windows cmd prompt and navigate to your python installation directory:

C:\Users\USERNAME\AppData\Local\Programs\Python\Python310\Scripts>

3. Install requests:

pip install requests

4. I then downloaded the usefulscripts repository as a zip file, exported and moved the directory to a convenient location:
https://github.com/emoncms/usefulscripts/archive/refs/heads/master.zip

5. Opened data_download.py with IDLE editor and entered the account username and password.

6. Run the data_downloader python script:

C:\Users\USERNAME\Desktop\Develop\usefulscripts\backup_py>py data_downloader.py
PHPFina id=108974 downloaded: 61593472 bytes
PHPFina id=108975 downloaded: 61593468 bytes
PHPFina id=108976 downloaded: 61593464 bytes
PHPFina id=108977 downloaded: 61593460 bytes
PHPFina id=108978 downloaded: 59799748 bytes
PHPFina id=108979 downloaded: 61593456 bytes
PHPFina id=108980 downloaded: 61593452 bytes
PHPFina id=109176 downloaded: 5646760 bytes
PHPFina id=109382 downloaded: 59683704 bytes
PHPFina id=109383 downloaded: 61477412 bytes
PHPTimeSeries id=113187 downloaded: 52901001 bytes
PHPFina id=113624 downloaded: 58636316 bytes
PHPFina id=113625 downloaded: 58636312 bytes
PHPFina id=114011 downloaded: 58509176 bytes
PHPFina id=114377 downloaded: 58370448 bytes
PHPFina id=124827 downloaded: 55264928 bytes
PHPFina id=124828 downloaded: 55264924 bytes
PHPFina id=124830 downloaded: 54636052 bytes
PHPFina id=124831 downloaded: 1168364 bytes
PHPFina id=125040 downloaded: 55204392 bytes
PHPFina id=125041 downloaded: 55204392 bytes
PHPFina id=125042 downloaded: 55204388 bytes
PHPFina id=125043 downloaded: 55204384 bytes
PHPFina id=128323 downloaded: 53467016 bytes
PHPFina id=128324 downloaded: 54095800 bytes
PHPFina id=128325 downloaded: 54095800 bytes
PHPFina id=128326 downloaded: 54095640 bytes
PHPFina id=200095 downloaded: 36074400 bytes
PHPFina id=403190 downloaded: 12 bytes
PHPFina id=423737 downloaded: 1792500 bytes
PHPFina id=423738 downloaded: 1792500 bytes

7. Conversion to csv also works fine.

Is there a similar utility for downloading and exposing the data from a local emonpi to a windows (or linux) PC?.

Hello @shallowal you can use this with a emonpi, you just set the host in the script to your emonpi’s ip address and it will work just the same

As this is written in Python3, so should this not be pip3?

Seems not on windows? you have to be in the python scripts directory itself to run pip, at-least without setting up paths manually… (My windows knowledge on this is minimal)

1 Like