Experience with Oct 2019 SD Image

Getting Started
1

@borpin

Disaster struck at my remotely monitored location a week ago – a power cut – one instance of emonTx/RPi failed to come back up. Having now retrieved the SDHC and USB HDD which was running the system, I discovered the USB HDD was dead and my attempts at recovery failed. So a case of start again – I then discovered a new SD image (Oct 2019) and decided to try it back at my home as a testbed. For what it’s worth, here are my experiences to date.

Burning/validating the image to SDHC took more than 30 mins – but it’s a big image 14.4GB.

I then added a file ssh to boot so I could use SSH but perhaps that was unnecessary.

Fired up an RPi connected by wired Ethernet to my network and did a network scan to find the IP address so I could puTTY in.

Old habits die hard – went to /home/pi/data. It’s all changed – I was lost! Eventually discovered a readme.md and have printed that out to serve as a ‘navigation cheat sheet’ – if that were added to the documentation, it might help many others?

# --------------------------------------------------------------
# Welcome to your emonPi/emonBase
# --------------------------------------------------------------

This image was build using the EmonScripts installation scripts:
https://github.com/openenergymonitor/EmonScripts

You can find the emonSD software stack installed in the following primary locations:

# /opt/openenergymonitor
Contains software installed from github.com/openenergymonitor.
Including: EmonScripts, emonpi, emonhub, RFM2Pi & avrdude

# /opt/emoncms/modules
Contains symlinked emoncms modules installed from github.com/emoncms
Including: demandshaper, sync, backup, postprocess, usefulscripts

# /var/www/emoncms
Contains the emoncms web application and modules installed directly

# /var/opt/emoncms
Contains emoncms feed data, including PHPFina & PHPTimeSeries

Browsed to the IP address and registered an account. Then went to Admin and discovered I was running ver 10.1.9. Belt & braces, I tried a Full Update but that returned an error referring me to a github link which I did not follow up on.

Next I did an Import Backup (sadly the most recent file on my home laptop is from early August). During the import process there was a warning …

mv: cannot move '/opt/openenergymonitor/data/import/emonhub.conf' to '/etc/emonhub/old.emonhub.conf': Permission denied

I ignored this and did a few cursory checks – could see the Feeds and Graphs OK.

Then I tried an Export Backup. This was trouble free. I discovered the backup in /opt/openenergymonitor/data – quite large at circa 127Mb.

Did lsblk which revealed that the SDHC is split:

  • 256Mb for /boot
  • 4Gb for /
  • 10.1Gb for /var/opt/emoncms

This begs the questions:

  • Is 4Gb big enough to accommodate regular (weekly?) backups?
  • Would it be more logical for data exports to reside alongside the time series data in the 10.1Gb?

My power cut damage experience is salutary – I need to think harder about backups:

  • Auto deleting those more than x weeks old
  • Storing them on a largish USB 3.0 flash drive – or even 2 flash drives – quasi RAID?
  • Given I monitor the site remotely using dataplicity – uploading backups to Dropbox? – and there is a dated script on the Forum addressing this

Haven’t yet sorted how to use the imported emoncms.conf

I’ll keep you posted and thx for yr efforts on this

2 Likes
2

Thanks for the feedback and for writing up your experience @johnbanks.

It is now possible to restore via USB sd card reader import method, described here: Backup module usb-import.sh script (beta)

It’s partly intended to get around the issue of needing to store large backups on the same SD card as the rest of the data, and also helps with restoring from a corrupted disk that wont boot to create an export archive. I had the same issue with a power cut the other week. Using fsck to fix the disc prior to import may be needed…

3

@borpin

Pls ignore the following in my prior post …

Yesterday when I first fired up the image my VirginMedia broadband was down and I was doing email using my friendly neighbour as an access point. I failed to realise my home network was not connected to
the internet. A bit ago my VirginMedia broadband came back up. I rebooted ver 10.1.9 and it automatically and successfully updated to ver 10.1.13.

My other comments and questions are still valid.
Thx

4

3 posts were split to a new topic: Automated backing up of EmonCMS

5

Issues caused by power cuts seem to be more common recently. I suspect there may have been a spike and not just a power cut that caused the issue. I always have equipment plugged in via a spike protection plug/lead.

This seems to depend on the card. In testing @TrystanLea saw some significant differnces between cards from different manufacturers.

It certainly has, and for the better.

There is a file system description at GitHub - openenergymonitor/EmonScripts: Emoncms Stack Installation and Update scripts I’ll look at adding a link to that.

@TrystanLea may be in a better position to comment.

More of a question, is it big enough for rootfs? I need to find time to tweak the partition setup.

Perhaps. That is a legacy setup that has been brought across.

Yes.

You could add the folder you store the backup to, to log rotate and just rotate them out, or a shell script to keep the last x. It depends on your backup strategy Day/Week/Month

I moved the backup discussion onto a separate thread so as not to take this OT.

Cheers

6

@borpin

Thx for yr response.

Why burn/validate a 14+Gb image? Would it not be possible to just burn the min needed and then do an auto SDExpand on first boot up? I use SanDisk SDHC’s - are they particularly slow?

The file system description link you provided did not turn me on. Why not go with my suggestion to add
the readme.md contents to the initial description - full frontal rather than just as a link? Once I discovered it, it was very informative to me as a navigation cheat sheet in this new emon world.

Lastly, thx for splitting the posts to a new Backups topic - hopefully that will be productive

Again thx & Happy Christmas!

7

It is just the way the image is created. I think @glyn.hudson was looking at trying to shrink it. It is to do with how the free space is handled in the image file created.

I’ve not seen it take that long, but @TrystanLea did note during testing that some cards took longer. I use SanDisk Ultras and they don’t take that long with Etcher.

What writing program do you use?

Each to their own… :grinning:

A lot of folk (majority I’d suggest) will not actually care or know where various bit were before, so won’t notice the difference. I also wanted to keep the instructions as short and clean as possible, if more info is needed/wanted, then follow the links (which I need to add).

8

@borpin
Just for info …
Did a trial image burn …
SanDisk Ultra SDHC 16GB Class10 using Etcher
Flashing at 13-14 MB/sec took 19 mins
Validating at 27-28 MB/sec took 10 mins
Total = 29 mins
Whether or not the following affected the results, I do not know …
The SDHC had been used previously
I used a Micro SD adapter in the card slot of my laptop - a reasonably upmarket Dell

USB 3.0 card readers are available - would one of these speed things up?

9

@borpin
PS …
Belatedly realised I was not running the latest Etcher so updated from 1.5.53 to 1.5.70 and did another test running on Windows 10 on my laptop.
Again using a previously used SanDisk Ultra SDHC 16GB Class10.
Flashing was at approx 11 MB/sec and took 24 mins
Validating occurred at 26-27 MB/sec and took 10 mins
Total = 34 mins

Could the amount of previous use have an impact?

10

I think there are quite a few variables from the HW you use to do the flash (PC & USB HW, USB adapter, micro SD Card adapter) to the card itself. I have had an occasion where the micro SD adapter caused an issue.

I need to do some work on the partition sizing and I’ll note how long it takes me.

11

@borpin
I’m becoming intrigued …
I’ve ordered a ‘high spec’ Kingston USB 3.0 card reader.
Amazon have SanDisk Ultra SDHC Class10 - £5.09 for 16GB and £4.99 for 32GB - amazing and that’s not a typo.
I’ll keep you posted.

1 Like