Python based Emoncms backup utility

Emoncms
1

I’ve put together a small python based backup utility for emoncms, to make downloading feeds from emoncms.org or any remote emoncms server without importing into a local installation of emoncms a bit easier. The tool also includes a script for converting the downloaded phpfina and phptimeseries binary data into CSV data.

This is much faster and more efficient than using the emoncms CSV export api to download all feeds. It’s also desgined to make it relatively easy to download multiple emoncms accounts, as each account is saved in a separate local directory with the same name as the account username.

The tool is available in the emoncms/usefulscripts repository here: usefulscripts/backup_py at master · emoncms/usefulscripts · GitHub

For anyone interested in downloading data from emoncms.org on to a local installation of emoncms I would recommend using either the php script here: usefulscripts/backup at master · emoncms/usefulscripts · GitHub or the emoncms sync module that is available as standard on the emonSD image: GitHub - emoncms/sync: In development: Download or upload emoncms feed data between local and remote emoncms server's

Cheers :slight_smile:

4 Likes
2

I’ve recently setup a windows PC for the first time in years to use some software that wasnt easily accesible on Linux and so I thought I’d take the opportunity to test using the above emoncms backup utility on windows, it works fine!

Steps to run on Windows

1. I started here with the latest version of python for windows:
Download Python | Python.org

2. I then needed to install the python requests module. Open windows cmd prompt and navigate to your python installation directory:

C:\Users\USERNAME\AppData\Local\Programs\Python\Python310\Scripts>

3. Install requests:

pip install requests

4. I then downloaded the usefulscripts repository as a zip file, exported and moved the directory to a convenient location:
https://github.com/emoncms/usefulscripts/archive/refs/heads/master.zip

5. Opened data_download.py with IDLE editor and entered the account username and password.

6. Run the data_downloader python script:

C:\Users\USERNAME\Desktop\Develop\usefulscripts\backup_py>py data_downloader.py
PHPFina id=108974 downloaded: 61593472 bytes
PHPFina id=108975 downloaded: 61593468 bytes
PHPFina id=108976 downloaded: 61593464 bytes
PHPFina id=108977 downloaded: 61593460 bytes
PHPFina id=108978 downloaded: 59799748 bytes
PHPFina id=108979 downloaded: 61593456 bytes
PHPFina id=108980 downloaded: 61593452 bytes
PHPFina id=109176 downloaded: 5646760 bytes
PHPFina id=109382 downloaded: 59683704 bytes
PHPFina id=109383 downloaded: 61477412 bytes
PHPTimeSeries id=113187 downloaded: 52901001 bytes
PHPFina id=113624 downloaded: 58636316 bytes
PHPFina id=113625 downloaded: 58636312 bytes
PHPFina id=114011 downloaded: 58509176 bytes
PHPFina id=114377 downloaded: 58370448 bytes
PHPFina id=124827 downloaded: 55264928 bytes
PHPFina id=124828 downloaded: 55264924 bytes
PHPFina id=124830 downloaded: 54636052 bytes
PHPFina id=124831 downloaded: 1168364 bytes
PHPFina id=125040 downloaded: 55204392 bytes
PHPFina id=125041 downloaded: 55204392 bytes
PHPFina id=125042 downloaded: 55204388 bytes
PHPFina id=125043 downloaded: 55204384 bytes
PHPFina id=128323 downloaded: 53467016 bytes
PHPFina id=128324 downloaded: 54095800 bytes
PHPFina id=128325 downloaded: 54095800 bytes
PHPFina id=128326 downloaded: 54095640 bytes
PHPFina id=200095 downloaded: 36074400 bytes
PHPFina id=403190 downloaded: 12 bytes
PHPFina id=423737 downloaded: 1792500 bytes
PHPFina id=423738 downloaded: 1792500 bytes

7. Conversion to csv also works fine.

3

Is there a similar utility for downloading and exposing the data from a local emonpi to a windows (or linux) PC?.

4

Hello @shallowal you can use this with a emonpi, you just set the host in the script to your emonpi’s ip address and it will work just the same

5

As this is written in Python3, so should this not be pip3?

6

Seems not on windows? you have to be in the python scripts directory itself to run pip, at-least without setting up paths manually… (My windows knowledge on this is minimal)

1 Like
7

Just ran this script on windows 10 and got the following result:

C:\Users\PC\AppData\Local\Programs\Python\Python310>python .\scripts\usefulscripts-master\usefulscripts-master\backup_py\allans_data_downloader.py
PHPFina id=17 downloaded: 62787112 bytes
PHPFina id=18 downloaded: 62787112 bytes
PHPFina id=19 downloaded: 62787084 bytes
PHPFina id=20 downloaded: 62787084 bytes
PHPFina id=21 downloaded: 62787068 bytes
PHPFina id=22 downloaded: 62787068 bytes
PHPFina id=23 downloaded: 62787052 bytes
PHPTimeSeries id=24 downloaded: 15822 bytes
PHPFina id=27 downloaded: 62719548 bytes
Traceback (most recent call last):
  File "C:\Users\PC\AppData\Local\Programs\Python\Python310\scripts\usefulscripts-master\usefulscripts-master\backup_py\allans_data_downloader.py", line 96, in <module>
    phpfina_download(username+"/phpfina/",f['id'],host,apikey)
  File "C:\Users\PC\AppData\Local\Programs\Python\Python310\scripts\usefulscripts-master\usefulscripts-master\backup_py\allans_data_downloader.py", line 58, in phpfina_download
    phpfina_create_meta(datadir,feedid,meta)
  File "C:\Users\PC\AppData\Local\Programs\Python\Python310\scripts\usefulscripts-master\usefulscripts-master\backup_py\allans_data_downloader.py", line 26, in phpfina_create_meta
    fh.write(struct.pack("I",meta['start_time']))
TypeError: 'bool' object is not subscriptable

C:\Users\PC\AppData\Local\Programs\Python\Python310>

Only 9 of the 28 feeds

8

Thanks @shallowal does it exit at that point again if you repeat? the tool only downloads new data so you can run it again and again without re-downloading what you have already…

The tool probably needs a bit more code to catch errors like this.

9

Different errors, but same result:

C:\Users\PC\AppData\Local\Programs\Python\Python310>python .\scripts\usefulscripts-master\usefulscripts-master\backup_py\allans_data_downloader.py
PHPFina id=17 downloaded: 30272 bytes
PHPFina id=18 downloaded: 30272 bytes
PHPFina id=19 downloaded: 30272 bytes
PHPFina id=20 downloaded: 30272 bytes
PHPFina id=21 downloaded: 30272 bytes
PHPFina id=22 downloaded: 30272 bytes
PHPFina id=23 downloaded: 30272 bytes
PHPTimeSeries id=24 downloaded: 9 bytes
PHPFina id=27 downloaded: 30152 bytes
Traceback (most recent call last):
  File "C:\Users\PC\AppData\Local\Programs\Python\Python310\scripts\usefulscripts-master\usefulscripts-master\backup_py\allans_data_downloader.py", line 96, in <module>
    phpfina_download(username+"/phpfina/",f['id'],host,apikey)
  File "C:\Users\PC\AppData\Local\Programs\Python\Python310\scripts\usefulscripts-master\usefulscripts-master\backup_py\allans_data_downloader.py", line 47, in phpfina_download
    local_meta = phpfina_get_meta(datadir,feedid)
  File "C:\Users\PC\AppData\Local\Programs\Python\Python310\scripts\usefulscripts-master\usefulscripts-master\backup_py\allans_data_downloader.py", line 11, in phpfina_get_meta
    tmp = struct.unpack("IIII",fh.read(16))
struct.error: unpack requires a buffer of 16 bytes

Formated text - Moderator, BT

10

So, it appears that there’s a problem with feed no:35, which is the next feed number. I cant display any info from this feed in the graphs page: Request error: Error reading meta data feedid=35

11

I altered the script to skip feed no:35, and it seemed to process the rest of the feeds correctly. I noticed that the feeds.json info for feed 35 looks different to the other otherwise similar feed types in that there is no start time and interval entries.
{“id”: “27”, “userid”: “1”, “name”: “GridPowerCost”, “datatype”: “1”, “tag”: “Node WillHenge”, “public”: “0”, “size”: “”, “engine”: “5”, “processList”: “”, “unit”: “”, “value”: 0, “time”: 1643158375, “start_time”: 1486359710, “interval”: 10},
{“id”: “35”, “userid”: “1”, “name”: “GridPowerCost-Day”, “datatype”: “1”, “tag”: “Node WillHenge”, “public”: “0”, “size”: “”, “engine”: “5”, “processList”: “”, “unit”: “”, “value”: 18.360871730547, “time”: 1643158375},

I tried pausing the script and adding those 2 parameters to the json file, but to no avail.

1 Like
12

Thanks @shallowal I’ve made a couple of changes to the scripts to catch the errors that you are seeing. I think feed 35 must be empty and that’s what’s causing the error.

13

@TrystanLea I am also having some errors, trying to download data from emoncms.org today, latest version of script from 11 days ago here is the error I see:

PHPTimeSeries id=120660 downloaded: 15894 bytes
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 360, in _error_catcher
    yield
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 669, in read_chunked
    chunk = self._handle_chunk(amt)
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 619, in _handle_chunk
    value = self._fp._safe_read(amt)
  File "/usr/lib/python3.7/http/client.py", line 622, in _safe_read
    raise IncompleteRead(b''.join(s), amt)
http.client.IncompleteRead: IncompleteRead(0 bytes read, 8192 more expected)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/requests/models.py", line 750, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 490, in stream
    for line in self.read_chunked(amt, decode_content=decode_content):
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 694, in read_chunked
    self._original_response.close()
  File "/usr/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 378, in _error_catcher
    raise ProtocolError('Connection broken: %r' % e, e)
urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(0 bytes read, 8192 more expected)', IncompleteRead(0 bytes read, 8192 more expected))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "data_downloader.py", line 99, in <module>
    phpfina_download(username+"/phpfina/",f['id'],host,apikey)
  File "data_downloader.py", line 63, in phpfina_download
    download_size = download_file(datadir+str(feedid)+".dat",host+"/feed/export.json?id="+str(feedid)+"&start="+str(download_start)+"&apikey="+apikey,'ab')
  File "data_downloader.py", line 35, in download_file
    for chunk in r.iter_content(chunk_size=8192): 
  File "/usr/lib/python3/dist-packages/requests/models.py", line 753, in generate
    raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(0 bytes read, 8192 more expected)', IncompleteRead(0 bytes read, 8192 more expected))

Any ideas what could be causing?

I had not realised I was still uploadign to EmonCMS, looks like your going to close my account (seems fair) but thought I would take a backup first.

14

Thanks for trying @whitecitadel I’ve pushed a change to the script that attempts to catch these exceptions, can you try the latest version? it looks like there may have been a connectivity issue part way through the download, hopefully that hasn’t affected the particular feed in question… you can always delete and try the download again

15

Many thanks, will try that and report back - understood about clearing download first, no connection lost because working at home and would have noticed if there was an outage so not sure what caused that error.

16

@TrystanLea New version worked great without error - thanks

1 Like
17

Great to hear, thanks!

18

I’m getting the following error with the latest data_downloader.py (Python v3.5.2 on Ubuntu Linux):

Traceback (most recent call last):
  File "data_downloader.py", line 102, in <module>
    phpfina_download(username+"/phpfina/",f['id'],host,apikey)
  File "data_downloader.py", line 66, in phpfina_download
    download_size = download_file(datadir+str(feedid)+".dat",host+"/feed/export.json?id="+str(feedid)+"&start="+str(download_start)+"&apikey="+apikey,'ab')
  File "data_downloader.py", line 32, in download_file
    with requests.get(url, stream=True) as r:
AttributeError: __exit__

Thanks