Community
OpenEnergyMonitor

Community

Exports – New Oct 2019 Image

@borpin
The new image export script is … /opt/emoncms/modules/backup/emoncms-export.sh which behaves differently to the previous export script - now any previous export.tar.gz files are removed before the new one is created.

And this maybe a necessity given export.tar.gz’s are saved in /opt/openenergymonitor/data which is in /dev/root sized 4GB and has limited space available – 50% in my test installation with no exports saved there.

Newbie comments …
• If the export process fails then any prior export will have diappeared
• Users may wish to use prior cron scheduled exports as part of their backup scheme.

Is there a way to save several exports without filling /dev/root ?

Thx

PS: I’ve changed my location of the export.tar.gz file to /var/opt/emoncms by making the following change in /opt/emoncms/modules/backup/config.cfg …

# Destination location of exported backup .tar.gz - Must be writable
## Mod by JB
## backup_location="/opt/openenergymonitor/data"
backup_location="/var/opt/emoncms"

The new image allocates 4GB to /dev/root whereas /var/opt/emoncms (/dev/mmcblk0p3) is allocated 10GB. And it also seems very logical for exported data to be saved alongside the current live time series data.

Furthermore not content with 10GB for data, I’ve burned the new image to a 32GB SDHC and then expanded /dev/mmcblk0p3 to 25GB as follows …

Do   sudo parted
Then do    print
This will show #3 as the Ext2 data partition which is to be expanded.
Then do   resizepart    and respond with   3
Respond   Yes  to the question about the partition already being mounted 
At the prompt   End? (15.5GB)?  … enter the new ending point in MB …
I used    31600
Verify the change has been made by doing    print
Finish by doing    quit
Finally expand the Ext2 file system to use its newly enlarged partition …
Do   sudo resize2fs /dev/mmcblk0p3
Finally verify all is OK by doing     df -hT

Of course as it stands, this larger partition will only contain the single latest export.tar.gz. So another step is to figure out how to retain more than one of these files.

And I’m hoping my changes will not be affected by future version upgrades.

Not something I was involved in.

True.

Of course - feel free to make a suggestion and create a PR.

I think I have an open GitHub Issue re the backup locations not being consistent as a result of the new script install.

Raise the issue and submit a PR for any suggested changes.

As I have repeatedly said, there needs to be a bit more work on the setup of the partitions post install either by the Scripts on a Rasbian image or post first use of the EmonSD Image.

If you want to contribute a PR feel free.

2 Likes

@borpin
Apologies - from many other Forum posts I had mistakenly concluded that you were involved in crafting the new SDimage.

I’ve taken a look at what’s involved in GitHub Pull Requests via these two links …

https://community.openenergymonitor.org/t/enhancements-to-multigraph-that-id-like-to-give-back-into-the-community-how/5974

https://www.digitalocean.com/community/tutorials/how-to-create-a-pull-request-on-github

Awesome given my skill level and my conclusion - perhaps OTT for simple suggestions/requests ?

A Suggestions/Recommendations section in the Forum might be a reasonable idea ??

I was, but the component parts that make up EmonCMS <> EmonSD and I’ve not really been involved in any changes to the backup module.

Depends if you have a desire to contribute - it is the point of Open Source. I knew nothing about GitHub etc (granted a Software Engineering background), but wanted to fix something, so worked out how to contribute. Made a few blunders along the way but once is unfortunate, twice is careless.

That is what GitHub Issues is used for, if you have a specific enhancement request or bug to report. Please ensure any Issue raised goes in the appropriate repository. Fair enough to discuss things here first though, but more likely to be forgotten about.

I agree that something needs to be done to improve the way the backup module saves archives, saving to /var/opt/emoncms is a good idea…

Ideally the option to save to a separate USB mounted sd card or drive, or sync to network drive. If that was all integrated in the UI it would be really nice… It’s not something I will get time to look at immediately but It would be good to make an item on the development roadmap

There are actually quite a few issues with the backup module currently https://github.com/emoncms/backup/issues. This one from the scripts is also related https://github.com/openenergymonitor/EmonScripts/issues/72.

I’d suggest perhaps, as we are trying to get folk to migrate, it may be an idea to prioritise these? I don’t think they will take much effort to fix.