OpenEnergyMonitor Community

Running but corrupted emonpi

I have a pi2 running emonpi (low-write 9.8.25 based on the '16 image).
However there seems to be a problem with the sd card: sometimes it will not boot and it gives an ext4 error. Most of the times it will boot…
I tried to backup the data (for a new sd card and a new pi), but whenever i run the script ( it freezes (with ext4 errors) when adding some of the phpfina files to the archive.
I also tried to usb restore it , but this also gives errors and ends up not importing the data (only the feed named etc).
I also tried the sudo fsck.ext2 /dev/sda3 command on the new pi , but it just gave errors and unmounted the usb stick containing the sd card…

Since i would like to restore the historical feeds i was wondering if there is another way to do this and somehow try it feed by feed or when i get a corrupted feed the data that is still good?

Is this an EmonPi, or a raspberry Pi running Emoncms?

Either way, stop using it - you are only making the data corruption worse.

Get a new SD card and put the new image on it (or buy one ready to go from the shop).

If you do not have a Micro SD card to USB reader, get one. There is a USB import script that will work if the card is not too damaged.

Its a pi running the emonpi sd card.
I already have a new pi running on a new sd card with the latest emonpi image, but the backup fails.
I tried to run the backup command on the old pi, but it just crashes it after a while.
I also tried the usb import with the old sd card in an usb-sd dongle but it failed as well. It imported the feeds but not the data, resulting in errors (feed id does not exist)

So i am looking for an alternate way. Maybe copy the phpfina files manually to check which on fails, or something with json?
I have about 60 inputs resulting in 150 feeds with schedules etc, so avoiding a redo of everything and salvaging historical data would be nice :wink:

1 Like

Hello @promy sorry to hear that

could you copy what you see here? I think the drive need to be unmounted before running the command…

if you boot up the pi using the new sd card with latest image, insert the old sd card in a sd card reader and then run:

sudo fsck.ext4 /dev/sda2 
sudo fsck.ext2 /dev/sda3

what do you see?

you could try that, do you have a copy of the import log? does it import the mysql database ok?

@TrystanLea: sda2 is OK on sda3 i get an error “was not cleanly unmounted, check forced”.
Then "error reading block 123132 (Input/output error) while getting next inode from scan. Ignore error?
when i do yes, it askes “Force rewrite?”. when i say yes, it repeats the message but wit the next block 123132 and it keeps increasing the number.
I also tried to copy the phpfina files (after stopping feedwriter manually), 1.dat and 1.meta are ok, but 10.dat crashes the system (it unmounts the sd card…)
I will try if i can copy other files…

The myqsl database is importing ok.
On the new pi the inputs are showing, but with an error where the feed are going, editing that shows the error feed (or feed id) does not exist.

Update: i was able to copy most of the files in the phpfina directory. only 12 files (6 dat and 6 meta files) were unreadable. Making the backup gzip file myself with these files i was able to import it to the new pi :wink: :wink: :wink:

But obviously :wink: , the corrupted feeds were some of my oldest and most interesting datafeeds: main use and solar. The raw input (watts) are still ok, but the kWh feeds were corrupted and as such not imported
Is there any way of telling emoncms to create the kwh data from the old watt data? Or is this only done when a new input is detected?
Another way would be to make a php script that asks for the watt info in the old pi and push it with a data to the new, but there are approx 13 million values per feed (since 2016) so it would take quite some time i guess (1 timestamp at a time…)

update2 : just a thought: would it be possible to make a new phpfina file from the jsons i can pull from the old pi? i’m looking at the source code now, and i think it should be possible, but not sure how…

Good effort @promy!!

Yes see the post processor guide here

great solution! i was able to get everything working again!
Thanks a lot!!