emonSD next steps

djh · 2019-04-06T09:31:55.684Z

TrystanLea:

Reading:
Filesystem Hierarchy Standard

FWIW, that’s an obsolete version of the spec - fifteen years old. The current version is Filesystem Hierarchy Standard

/opt/package-name for code and move data directories to /var/opt/package-name for data after installation are reasonable places for external products to be placed. If a distro packages a product and releases it, then it will typically put it under /usr with data under /var/lib
So the package should make sure it uses relative, and configurable paths to enable such changes. Ditto for its dependencies on other packages, which may live in different places on different systems. That also allows for multiple versions if the version number is in the path somewhere.

TrystanLea · 2019-04-06T10:28:00.130Z

Thanks @djh that is useful to know, looks like we are on the right track then moving to /opt. I will also move the data directories to /var/opt/emon/phpfina etc (configurable of course) thanks for the tip on that. Any objections?

I still need to look the idea of supporting multiple emoncms installations on one machine as suggested by @pb66 as part of this.

borpin · 2019-04-06T10:42:41.891Z

TrystanLea:

I still need to look the idea of supporting multiple emoncms installations on one machine

I suspect that will simply be a need to configure the initial path layout so it is extensible.

Perhaps just offer a commented out alternative structure for a multi install environment as it is an edge case.

TrystanLea · 2019-04-06T10:47:58.535Z

borpin:

I suspect that will simply be a need to configure the initial path layout so it is extensible

I think that may well be the case, should be straightforward.

TrystanLea · 2019-04-08T15:48:30.417Z

A little more progress on this.

There is now a wifiAP install script, with the access point tested and working.
Default paths changed and configurable to: /opt/emon and emoncms data: /var/opt/emon/phpfina…
changes to emonPiLCD script that make it path flexible merged into master

I’ve run the build script again from the start on a fresh raspbian strech and it works well (a final sudo reboot is required at the end).

Im going to review the remaining steps in post 92 above next, but will need to come back to it after attending to a number of other things first.

TrystanLea · 2019-04-09T07:50:05.147Z

Noting your comment here @borpin for inclusion Unexpected output from git describe - #11 by borpin

TrystanLea · 2019-04-15T14:04:33.582Z

Further progress on the install script:

Merged multienv branch of backup module into master branch to support flexible install path, update install script to pull in master branch.
Multi environment support for backup module by TrystanLea · Pull Request #23 · emoncms/backup · GitHub
Sync module support for flexible install path
- auto detect background process script path · emoncms/sync@c391cd2 · GitHub
Post process module support for flexible install path
flexible path support · emoncms/postprocess@f3f698a · GitHub
Clean up of wifi.sudoers
cleanup wifi sudoers · openenergymonitor/emonpi@4345ccd · GitHub
Removal of setup sudoers
remove setup sudoers, cleanup install & update · openenergymonitor/emonpi@fc0ce0b · GitHub
Support for setting emoncms modules branch in config.ini:
support for setting repo branch in config.ini · openenergymonitor/emonpi@a654a18 · GitHub

I have made a start on documenting the install scripts here:
https://github.com/openenergymonitor/emonpi/blob/master/install/readme.md

Noted at the top are the items left to complete, which I have separated into releases. I hope to complete the first release this week.

Todo 1st release

ext2 data partition, mount /var/opt/emon as ext2?
review log2ram configuration, noting https://community.openenergymonitor.org/t/emonsd-next-steps/10426/29

Todo 2nd release

fix flexible emoncms_core install location (currently /var/www/emoncms symlinked to /var/www/html)
review emoncms logfile location
review /var/www/html/emoncms symlink

TrystanLea · 2019-04-16T09:26:28.900Z

6 posts were split to a new topic: emonSD next steps: filesystem

djh · 2019-04-19T19:40:52.290Z

It’s annoying that after you split the topic, I no longer get notification emails to tell me there’s been any activity. Short of changing discourse to also split the notifications, can I suggest that as a courtesy whenever anybody splits a topic, they post a message to the original topic stating that that is what they have done?

borpin · 2019-04-25T20:53:49.274Z

TrystanLea:

The installation on a stock raspbian stretch image can be started with:

emonSD next steps: filesystem & logrotate

My draft readme for the install process is available in the folder here:

I’m not sure if you are using the latest Raspbian Lite image, but the readme for expanding the file system does not seem to match this image after first boot. I suspect the file system has already been expanded.

Disk /dev/mmcblk0: 14.5 GiB, 15552479232 bytes, 30375936 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x8b1df36f

Device         Boot Start      End  Sectors  Size Id Type
/dev/mmcblk0p1       8192    96042    87851 42.9M  c W95 FAT32 (LBA)
/dev/mmcblk0p2      98304 30375935 30277632 14.4G 83 Linux

Trying to run the install and I fell at the first hurdle.

[edit] decided to run it anyway. Ran fine until here where it wanted some input.

github.com

openenergymonitor/emonpi/blob/cf7cac6bff569d7aa96ac57c1722824d37c1df6c/install/emonsd.sh#L13-L13


      
          rm -r log2ram

rm: remove write-protected regular file 'log2ram/.git/objects/pack/pack-e60fc815bb6beda9136261e26e2b90e3e9bc867e.pack'?

Other than that seems to work . Did an export/import - all OK.

borpin · 2019-04-25T21:09:48.372Z

@TrystanLea, one other point as part of the install, I think this needs a 'default.config.ini` file that can be copied and changed for use but ignored by git.

At the very least a warning to the user that it is using the default config ‘do you want to continue’ type question.

pb66 · 2019-04-25T22:16:24.528Z

borpin:

I suspect the file system has already been expanded.

Raspbian images now resize automatically, If you look at a virgin cmdline.txt there is a entry like init=/some/path/init_resize.sh, if you delete that before putting into a Pi, it will not resize. I think it completes the full resize on the second reboot, but if you add a 3rd partition before that it might interfere with that process. I would need to look closer at it to be sure, I’m working from memory here. You won’t find the init= in cmdline.txt after it has resized though, it gets removed by raspi-config.

borpin · 2019-04-26T07:35:25.812Z

pb66:

Raspbian images now resize automatically

For the install script I think we should work on the basis it has been resized so need to shrink first. Goes back to where we started - do we actually need to do so?

pb66 · 2019-04-26T08:12:17.662Z

borpin:

For the install script I think we should work on the basis it has been resized so need to shrink first.

I understand what you are saying and don’t disagree entirely. But you cannot make any assumptions and start moving partitions around unless you know the resize has run it course or you take control and block, complete or abort that resize.

I don’t know what error Trystan actually saw when he wrote the steps to create 3rd partition

sudo fdisk -l
Note end of last partition (5785599 on standard sd card)
sudo fdisk /dev/mmcblk0
enter: n->p->3
enter: 5785600
enter: default or 7626751
enter: w (write partition to disk)
fails with error, will write at reboot
sudo reboot

but it’s possible that error might have been caused by unsaved changes to the partition table or saved changes not applied by partprobe (or reboot).

It could be a game of chance or bit of wrestle between the install script and the default init_resize.sh script to dictate the partitions unless the 2 are choreographed to play nicely together.

borpin · 2019-04-26T08:16:20.876Z

pb66:

I understand what you are saying and don’t disagree entirely. But you cannot make any assumptions and start moving partitions around unless you know the resize has run it course or you take control and block, complete or abort that resize.

Absolutely.

pb66:

I don’t know what error Trystan actually saw when he wrote

And not sure what Raspbian version (new one recently).

Could you check if init_resize.sh has been deleted to see if it is complete?

Before we put too much effort into this has the the fundamental question been answered; do we need a data partition?

borpin · 2019-04-26T08:31:59.284Z

@pb66, just looking at the logs for log2ram and I see this sometimes; does it mean the emonhub log has rotated at that point or that a rotation happened outside log2ram.

sent 2,049,721 bytes  received 9,727 bytes  4,118,896.00 bytes/sec
total size is 9,422,137  speedup is 4.58
sending incremental file list
auth.log
daemon.log
log2ram.log
syslog
emonhub/
emonhub/emonhub.log
emonhub/emonhub.log.1

Couple of observations (been running overnight);

The comment sending incremental file list is not that clear to me as a log comment - are the subsequent files the ones copied?

The log could do with a date time stamp .

Looking for the rotated logs and cannot find any. Nothing in /var/log.old. I did create that folder so could that be the issue? Nothing in the logs to suggest an error was found.

All the log2ram output is going into daemon.log as well as log2ram.log.

pb66 · 2019-04-26T08:33:41.503Z

borpin:

Could you check if init_resize.sh has been deleted to see if it is complete?

My understanding isn’t in depth enough to know that off hand. Whilst i have had some dealings with it, I only went as deep as I needed. My initial thought is “no” because the deletion of the file is just an indication that the content of that file has been used, not that the changes have applied and stuck.

borpin:

do we need a data partition?

I think Trystan would say yes. Because there is a notable difference in write levels with the ext2 block sizes. Whether that difference in write levels is absolutly necisary for the target life of the sdcard, I think remains undecided. But this is a near zero running cost optimisation so whilst it does mean a more complex install script needs developing, I see no other reason not to do it.

However, even with the reboot required for L2R, I would recommend we establish exactly what’s happening or take control and stop it happening to be sure of a predictable outcome.

I believe if we looked into it we could get in make arrangements and on rebooting the 3rd partion be ready, L2R be ready and the rootfs expanded. It would be good to use what’s already there.

I was looking into redirecting that path in cmdline.txt so that potentially I could drop a wifi config, blank ssh file and a custom build script into the boot partition, edit cmdline.txt to include `init=/boot/custom_build_script.sh and just let the image self build. This disables the in-built resize and initiates the build script with no interaction. I guess you could even do it without the ssh file once you have got it working 100%.

borpin · 2019-04-26T08:58:01.008Z

I also noticed that there is now a warning message on first boot that the WiFi config is disabled as a country code has not been set. Been a while since I booted a Raspbian image so that may not be new.

pb66 · 2019-04-26T09:06:15.932Z

borpin:

The comment sending incremental file list is not that clear to me as a log comment - are the subsequent files the ones copied?

The log could do with a date time stamp .

All you see there is the rsync output redirected to the logfile by L2R unmodified. You should be able to find info about rsync output fairly easily I guess.

Right at the beginning I highlighted L2R is not a finished solution. It’s taken over a year since I first floated the idea of using L2R to get to the point were it gets any consideration at all. I do not maintain the original L2R, but I am willing to develop it for our needs, first we need to establish what they are, then I will look to improve L2R.

I agree timestamped logs are essential. I have already suggested more verbose logs throughout L2R with a user set loglevel. (Look at emonhub for an idea of how I like to do logging.)

We have also discussed L2R providing data to emonhub/emoncms for capture/monitoring eg write stats, logfile sizes, disk usage etc.

borpin:

Looking for the rotated logs and cannot find any. Nothing in /var/log.old . I did create that folder so could that be the issue? Nothing in the logs to suggest an error was found.

Are the permissions and ownership correct on the folder you created? Had you not created the folder, I would have confidently said “in that case no rotations were needed/performed” but cannot now be 100% sure. The folder only gets created automatically IF it doesn’t exist, so I suspect if one does exist, but is not the right permissions, it will fail.

Another thing regarding the logs, logrotate doesn’t actually maintain a log of it’s own. I think it’s output might go to syslog? I have been using redirection in my commandline whilst debugging

sudo logrotate -v /etc/logrotate.conf | sudo tee >> /var/log/logrotate.log

but I would also like to see this output in the L2R logs aswell.

borpin:

All the log2ram output is going into daemon.log as well as log2ram.log .

I wasn’t aware of that because there is no log output from L2R in cp mode and since changing to rsync, I had no time to work on this, spent some time looking into rsync rules because Trystan was having issues, moved onto tackling logrotation because you found it wasn’t working and then the discussion got heavily into the logrotation side.

Now there is some interest in using L2R etc, I will start developing further as and when time allows me to, this is something I could have been doing over the last year had there been any interest, but in true OEM fashion we have gone from zero interest to I want it now with no middle ground. My main aim has to be to get it operational for inclusion before we lose interest again, the rest can happen over time.

I will look into the duplicate logs, but at the moment L2R logs hardly anything once an hour so this is a fairly trivial thing until we get more log output, I think if the L2R logging is overhauled to include timestamps and loglevels etc, this would get addressed then anyway. In the meantime, the duplicated logs mean there IS probably a timestamp available via daemon.log if you need one

pb66 · 2019-04-26T09:14:48.057Z

borpin:

I also noticed that there is now a warning message on first boot that the WiFi config is disabled as a country code has not been set. Been a while since I booted a Raspbian image so that may not be new.

No That has been around a while. Since this is not an existing emonSD image we can use the in-built wifi settings tool to add wifi at boot. If you include a wpa_supplicant.conf file in the boot partition before booting with something like this content

ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
    update_config=1
    country=GB
    network={
            ssid="WIFI_SSID"
            psk="WIFI_PSK"
    }

it will start straight up on wifi, no Ethernet required.

This doesn’t work well on oct18 emonSD as the firstboot script doesn’t wait for a network connection so you don’t get the proper initialisation/updates. images prior to that were read-only so the wpa_supplicant.conf file could’nt be copied into /etc/wpa_supplicant so it didn’t work at all.