Switching emonpi/emoncms logfile locations to /var/log/emoncms

[TrystanLea]

At present the logfile locations for emonSD and emoncms related scripts are typically in two locations:

/var/log/emoncms.log

/home/pi/data/emonpiupdate.log
/home/pi/data/emoncms-sync.log
/home/pi/data/postprocess.log
/home/pi/data/emoncms-export.log
/home/pi/data/emoncms-import.log
...

The use of /home/pi/data arose from placing /var/log in a tmpfs without any persistance. emonpiupdate.log and several others where written to /home/pi/data for persistance.

The new image build scripts that we are developing use log2ram to persist the logs in /var/log and so it makes sense to move these logs back to a suitable location on /var/log.

The proposal here is to move them to:

/var/log/emon

including the emoncms.log which is currently in the parent directory /var/log.

At present the log paths for emonpiupdate, sync & postprocess are all hardcoded and /var/log/emoncms.log is specified in settings.php using the full file path.

The proposal is to introduce a new setting called $log_location in settings.php that defines the log parent directory (e.g /var/log/emon). If the setting is not present it will default to /var/log/emon. The current $log_filepath would be depreciated. The actual name of the logfile e.g emoncms.log would be hardcoded but its location would be configurable.

I’ve implemented and tested this approach as part of a emoncms development branch called log_location, the changes can be seen here: https://github.com/emoncms/emoncms/compare/log_location

There are also associated changes to the sync module:
https://github.com/emoncms/sync/compare/log_location

postprocess module:
https://github.com/emoncms/postprocess/tree/log_location

backup module:
https://github.com/emoncms/backup/tree/log_location

The /var/log/emon folder is autocreated by the emoncms_mqtt and feedwriter services following @pb66’s suggestion https://github.com/emoncms/emoncms/compare/log_location#diff-3ad0e4f84032532b09b41599d1c0eae9R54

If users are not using these services and have not created the /var/log/emon log location emoncms works just fine but no logs will be created, a suitable error message is displayed on the emoncms administration page.

The main issue I can see is that switching to this approach right now will mean that all emoncms logs will move to the /var/log tmpfs partition and so will cease to be persistent on existing images. The main log likely to be of concern is the emonpiupdate.log which should ideally be persisted. This could be solved with a symlink, but the symlink would need to be created at system startup, probably not a big issue.

I know we’ve already discussed much of this @pb66 @borpin but I wanted to post this up here as a full description so that its is clearer, Interested to hear your and others thoughts on it.

[borpin (Brian Orpin)]

@TrystanLea

The /var/log/emon folder is autocreated by the emoncms_mqtt and feedwriter services following @pb66’s suggestion

I must have missed this bit. Why not just use systemd-tmpfiles to do it (I did suggest it earlier in the discussion IIRC) so it is created regardless of those services running?

[TrystanLea]

Thanks @borpin I will look into that option. What do you think about loosing the persistence of emonpiupdate.log on existing systems, should we put in a symlink to persist this log as a temporary measure?

[pb66 (Paul)]

@TrystanLea

The /var/log/emon folder is autocreated by the emoncms_mqtt and feedwriter services following @pb66’s suggestion

Whilst they will indeed also ensure the log folder is created, my main point was that the service-runner service would create the logfolder as that is becoming pretty much a mandatory item. Certainly in the interim whilst the rc.local is still in play, all emonSD images certainly have all of these services, moving on from there, if installed via the installer the service-runner will definitely be installed as standard so that the updater will work, the emoncms_mqtt and feedwriter services will be more optional. Plus L2R will also be managing the folder too.

@borpin

Why not just use systemd-tmpfiles to do it

That’s a tad more involved than the extra couple of lines added to a service, we already have both belt and braces, and a warning message in emoncms, I don’t think we need anything more.

@TrystanLea

The main log likely to be of concern is the emonpiupdate.log which should ideally be persisted. This could be solved with a symlink, but the symlink would need to be created at system startup, probably not a big issue.

This log also needs renaming to be non-emonpi specific and it also needs to either append new logs or rotate at each update.

Before committing to the proposed structure please think about how multi-instance installs might fit in. Whilst I’m sure the installer or updater won’t manage a second instance, I’m concerned that depreciating $log_filepath and using a path to logs (eg /var/log) plus fixed name of emoncms.log, might actually block the ability to (easily) have multiple instances as they might be competing for the same paths/files. Use of the instance name would be useful.

@TrystanLea

What do you think about loosing the persistence of emonpiupdate.log on existing systems, should we put in a symlink to persist this log as a temporary measure?

I’m not a fan, but i see the need if you want to change emoncms to use the new path. I haven’t thought about this too much, but wouldn’t using tee in the updater to log to the 2 locations be a better option? One is in RAM so the disk-write’s won’t increase. that way you could also address the naming and append to the persisted copy so we can see previous updates if needed. Currently when a update isn’t 100% successful, the favourite course of action is to run the updates again (and/or reboot) which seriously hampers debugging as the original log is overwritten.

[TrystanLea]

Sure we could use service-runner for the folder creation instead or in addition to be safe for now.

@pb66

This log also needs renaming to be non-emonpi specific

sure I will change the name.

@pb66

multi-instance installs

each instance could have its own logfolder e.g /var/log/emon1 /var/log/emon2, for a basic install without emoncms_mqtt and feedwriter I think this would work just fine, I think we have other steps to make emoncms_mqtt and feedwriter support multi-instance installs as it is.

@pb66

but wouldn’t using tee in the updater to log to the 2 locations be a better option?

that sounds like a hack that would need to be placed in admin_controller.php?, the symlink could be handled separately to emoncms - if it works…

[pb66 (Paul)]

@TrystanLea

each instance could have its own logfolder e.g /var/log/emon1 /var/log/emon2,

ok! so if the log folder name is the instance name, it should be “emoncms” not “emon” and derived from the instance name in emoncms perhaps. I’m not entirely sold on this as the service-runner logs are for all instances, but /var/log/emon/emoncms and /var/log/emon/emoncms1 just to be able to put the service-runner in /var/log/emon is perhaps a step too far, so this might be the best way regardless.

@TrystanLea

I think we have other steps to make emoncms_mqtt and feedwriter support multi-instance installs as it is.

Yeah no doubt, I’m not trying to get multi-instance up and running now, just to be aware and not block it accidentally would be great at this stage i think.

@TrystanLea

that sounds like a hack that would need to be placed in admin_controller.php?,

I have no idea off-hand, the current update triggers (and the previous ones) are too complex to recall, I have to delve into the code every time I have a question so I try to avoid that area as it is time costly. This is why I prefer the idea of a command line in settings.php, it could have been just edited there. As could the path to an alternative update script, for testing for example. IMO the majority of users will have a specific requirement and perform the same update(s) each time, a single configurable button (command line in settings.php) and a check in the updater to only flash when the pre-defined FW repo has changed would have given both the ultimate flexibility and the simplicity of a single button update. Perhaps you would consider a PR for an optional/additional “single configurable update button”?

[TrystanLea]

@pb66

so this might be the best way regardless

do you mean /var/log/emon? your right that most of the logs in /var/log/emon are actually emoncms so the emoncms name might well make more sense, i was thinking we might move the emonpilcd log there and there may be other emon specific logs that are not strictly emoncms.

service-runner is not currently logging to a dedicated logfile, its logs are going to systemd something that would be good to change.

@pb66

This is why I prefer the idea of a command line in settings.php

Im not quite sure what you mean here? what do you mean by command line in setting.php?

[pb66 (Paul)]

@TrystanLea

i was thinking we might move the emonpilcd log there and there may be other emon specific logs that are not strictly emoncms.

True, and there may well be other logs now or in the future eg emonUpload? but the emonPiLCD and emonUpload aren’t anything to do with emoncms directly, nor are they likely to be(come) multi-instance, they are more akin to service-runner in that aspect. perhaps we need both /var/log/emon and /var/log/<instance> ? one for the emoncms specific logs and one for non-emoncms and non-multi-instance logs?

IMO emonPiLCD should be an emonhub interfacer anyway! That would remove the need for any additional logs, confs or services, it would use emonhub infrastructure and display whatever the user configures it for plus some mandatory pages/functions eg update/reboot and ssh etc.

is there much traffic in the emonPiLCD log? does that one log need to restrict how the emoncms folders are managed across the board? especially as it’s only a proportion of the emonSD users that will have an LCD screen. I know it’s undoubtedly the biggest group currently, but of all the install variations, it is only used in one. Since it’s rarely used, even to debug emonPi’s, perhaps we can come back to it?

@TrystanLea

Im not quite sure what you mean here? what do you mean by command line in setting.php?

I’ll come back to that, perhaps in a different thread?

[TrystanLea]

@pb66

perhaps we need both /var/log/emon and /var/log/<instance> ?

sure, Im happy with that, I will change it over to /var/log/emoncms by default , which can be changed to /var/log/emoncms1 ,2,3 etc as needed.

@pb66

I’ll come back to that, perhaps in a different thread?

yes good idea

[borpin (Brian Orpin)]

@pb66

@borpin

Why not just use systemd-tmpfiles to do it

That’s a tad more involved than the extra couple of lines added to a service, we already have both belt and braces, and a warning message in emoncms, I don’t think we need anything more.

It is one line in a drop in file. Better than multiple sevices trying to do it and raising an error that the folder already exists. Also creates the folder at exactly the right time as the FS is mounted.

[TrystanLea]

@borpin

It is one line in a drop in file.

Thanks @borpin I had a look through the previous discussions on this and cant find an example of how systemd-tmpfiles work, if it is straightforward could you provide a step by step example of how you see them being used in this case?

[borpin (Brian Orpin)]

It was this discussion that made me look at it and realise how elegant it was. @rohanl wrote a script to create the config file; you could use that as a one off but possibly need to be careful of conflicts with L2R.

Better handling of files in tmpfs /var/log

Since /var/log is mounted as tmpfs the contents including all subdirs are deleted on reboot. This is currently handled by having code in /etc/rc.local that recreates the directories and files that are required, and then restarting services. There are several things I don’t like about this: services being restarted unnecessarily The list is liable to get out of date. for instance it creates /var/log/openhab, whereas OpenHAB2 requires /var/log/openhab2 some of the users do not exist until packa…

[TrystanLea]

Ok that’s interesting, I guess the downside is that it is a separate configuration and the inclusion of the logfolder creation in the service script itself is a more self contained solution, it’s worth considering though.

[TrystanLea]

@TrystanLea

The main issue I can see is that switching to this approach right now will mean that all emoncms logs will move to the /var/log tmpfs partition and so will cease to be persistent on existing images. The main log likely to be of concern is the emonpiupdate.log which should ideally be persisted. This could be solved with a symlink, but the symlink would need to be created at system startup, probably not a big issue.

I’ve got this appraoch working, it requires the creation of the /var/log/emoncms directory in rc.local and then lower down in rc.local:

chown -R pi:pi /var/log/emoncms
touch /home/pi/data/emonpiupdate.log
ln -s /home/pi/data/emonpiupdate.log /var/log/emoncms/emonpiupdate.log

touch creates the file if it doesnt exist and leaves it be if it does.

This is only needed for older images of course and the new image rc.local is the default with no changes + log2ram handles persistance.

Once this is in place Im ready to merge the log_location branches of emoncms and modules ready for the new script release: emonSD next steps: filesystem & logrotate - #192 by TrystanLea

[TrystanLea]

These changes are now merged in the emoncms master and stable branches and are included in the latest release Emoncms v10.1.0

The EmonScripts build script has also been updated to use the latest master branches. See: EmonScripts/install at master · openenergymonitor/EmonScripts · GitHub