Retrieving data from an old SD card with previous emon OS

Emoncms
#1

Have questions now about retrieving data…
(1) how do I decode my log data from backups or an old SD card with an emosOS image so I can copy them to text files for analysis?
(2) Is there a way to download all the data from the online hosted emoncms database without using the graphical interface? I have a lot of points, and extracting the data a month at a time is tedious…

Thank you for any help.

#2

Hello @SteffanCook, downloading data from emoncms.org or another hosted instance is easy if you have a local Pi running the latest emonSD. You can either use the Emoncms sync module available from the setup menu on the emoncms installation running on the Pi or the backup script detailed here emoncms/Backup.md at master · emoncms/emoncms · GitHub.

#3

Hi @TrystanLea.

Thanks for your message. I have been trying out the various ways of importing backup data this morning (detailed Doing a 'Full Update' to the emonPi caused it to 'brick' - #3 by SteffanCook).

What I really wanted though was a way to turn those .DAT files (in the backups) into readable CSV data. I don’t want to use the graphical interface module on emonCMS to pull out the data, I want to get the data from the backup. THe graphical interface is a bit cumbersome (I could kill those calendar UI) and it tends to be limited to a few thousand data sets, and I have tens of thousand size data sets to get out so that means many attempts.

Any straightforward way to decode those .DAT files available?

Cheers. Diolch
Steffan

#4

This page gives an overview of how the data is stored in the phpfina format:
Learn | OpenEnergyMonitor

Its a while since I tried it, but I think this script will read out the contents of a phpfina file as space separated:
usefulscripts/phpfina.php at master · emoncms/usefulscripts · GitHub

You will need to set the directory and feedid at the top before running the script from command line with php:

$ php phpfina.php
1 Like
#5

@TrystanLea - the phpfina.php file does work and it does print out the data values from the .dat files.

For ease of use, could you add some code that puts those $time and $value results into a text file of the same name in the same folder. (sadly I don’t know enough code to write this)

Thanks for any help.

#6

Hello @SteffanCook the following should work on linux:

php phpfina.php > output.csv
1 Like
#7

@TrystanLea - data header information added to .dat file to .csv using phpfina script

Hi again,

I can get the data from the phpfina script however there are no headers for me to identify the data. Some headers or a way to identify the data would be useful.

Can you teach me something to get the header data from the .meta files? (assuming there is any identifying data in those meta files). If you write code I can add it to my clone version of the phpfina script.

Thanks for any help.

#8

Hello @SteffanCook

Yes you could print the start time and interval from the .meta file, just add the following just below fclose($metafile);

print "start time: ".$meta->start_time."\n";
print "start time: ".$meta->interval."\n";

Im not sure that its giving you that much more than you already know from the data print out which includes the timestamps for each data point?

#9

Thanks for the reply @TrystanLea .

Sorry, that’s not what I meant. I need header information to identify the name of the source and the type of data and units.

Is there any identifiying data in the .meta file to label the source of the data, such as EmonTH node ID or feed name? Can I extract that out using the ‘print’ commands above?

A longer explanation:
I guess that when I create a feed in the EmonPi that a database file (.dat) is created for it along with an associated .meta file. I assume the first feed file is called 1, then 2 and on and on. However, when I use the emonPi I never recall data from their feed number but instead from the name label. This means when I go to extract the data from the SD card I cant remember which file number belongs to which feed (for example “was the cold water temperature feed number 7 or is that the boiler room temperature?”). I need some other label to know which long list of numbers belongs to what source and so some header information above the data column would really be a help.

Hope that makes more sense and thanks again for the help.

#10

Aha ok that makes sense, unfortunately this information is held in the emoncms mysql database not in the phpfina meta file. Do you have any way of accessing the old mysql database and exporting it? are you able to get at the /var/lib/mysql files on the SD card?

I have often thought it would have been useful to have included this information in the phpfina meta file itself to allow this kind of data recovery and may consider that in a future update.

#11

Hi @TrystanLea ,

I can still see the mysql folder at /var/lib/mysql. However, I can’t seem to find an easily readable file that will give me the header info. I don’t know anything about mysql as a database method.
If this is the only way forward then I will give mysql a shot. Any ideas though?

As you suggest, it might be better(?) design if the data folders on the SD card actually had all the meta info needed to use the data included somewhere. Relying on mysql info that is kept separately on another partition makes data retrieval more complicated and less likely to be successful.

I don’t think this is a top priority though, but maybe a nice to have feature in the next update. The actual emoncms on the emonOS does make it easy to get the data most of the time. It looks good graphically too and easy to understand. However, I just want a data dump and I just find the data point limits a bit cumbersome to work around.

#12

You can increase the max_datapoint limit if you want, the default setting is here: emoncms/default-settings.ini at master · emoncms/emoncms · GitHub and can be copied over to your settings.ini file in the feed section

#13

Hi @TrystanLea ,

I am trying your idea of increasing the number of points in the settings.ini file.

As a first step though I am having trouble creating a backup. I created an archive backup of a currently active emonPi at my home and want to import that to an emonbase to tinker around with. Sadly the backup is 90MB and I cant update via the browser (is that beyond the limit?). So, I went to the referred page on manual instructions on how to import an archive backup.
Sadly, the install.sh script and the emoncms-export and emoncms-import.sh scripts dont seem to work for me. I think they may need some updating of references to pathway folders and files. I noticed the default.config.cfg file is called config.cfg in the scripts. It would also be useful to have an example of pathways for each part of the config.cfg file.

Do you think you can check and update the scripts if necessary so I can import from a backup archive?
Thanks

#14

Hello @SteffanCook what are the errors that you are seeing when you try to run the scripts manually?

#15

Thanks for your reply.

For install.sh- the first error is

´´´Missing openenergymonitor directory parameter and default: /opt/openenergymonitor not found´´´

Target directory is not a subdirectory of the folder install.sh is in.

For emoncms-export.sh the first error is:
ERROR: Backup ./backup/config.cfg file does not exist

Correct folder but the config.cfg file was not generated by install.sh so obviously it generates an error.

For emoncms-import.sh the first error is;
cat: ./backup/module.json: No such file or directory

Folder is correct, but the subfolder is now maybe called backup-module

#16

ok, could you try running them like this:

/opt/emoncms/modules/backup/emoncms-import.sh
#17

in this case there is an error because the script calls on a wrong folder.
cat: ./opt/emoncms/modules/backup/backup/module.json: No such file or directory
EUID: 1000
Reading ./opt/emoncms/modules/backup/config.cfg…
ERROR: Backup ./opt/emoncms/modules/backup/backup/config.cfg file does not exist

I think nearly all the errors are tree directory structure errors, some folder and not in the nested position they should be. I think I could almost work it out myself… but itś beer o clock so thats a job for another day.

1 Like
#18

Haven’t heard that one in a while! beer_cheer

1 Like
#19

I have found the link I was looking for between Feed Number and name. When you make an archive backup you can find references in the human readable emoncms.sql file that is created at Line 160.

With this I can now use;

As such I probably wont now be looking into the import archive scripts to try and get them to work. This is a shame I think as backup by archive would seem to be the preferable method rather than physically pulling sd cards in and out (which could lead to read/write errors).

Likewise, although I might come back to it, increasing data points on the graphic display in emoncms would have been nice especially as the pi hardware gets more powerful.

#20

Just to put the above in context for others. I doubt I am a typical user of emonpi and as such the default graph and data export on emoncms should work just fine for most. It is lovely that work arounds to my problems exist at all and a thumbs up from me.