I’ve just tested, this is a really tidy method to create /var/log tmpfs files.
I’ve created a dev branch and opend a PR on the emonPi repo to develop and test this:
So far, I’ve got your script in the emonPi repo emonpi/update-tmpfiles-config.sh and have tested running it manually to create /usr/lib/tmpfiles.d/emonpi-dynamic.conf. I then removed all the old file creating entires from rc.local and after a restart all the files were re-created and restarting servivces was not requires so startup was faster
As you say, the next step will be for this script to run automatically. Did you investigate either of these methods?
Ah I notice /var/log/emoncms.log was not picked up by the script. I’m guessing this is because it’s not in a folder. Is it possible for tmpfiles to create a file and set permissions?
I haven’t had a chance to look at automatic running of the script yet…
The script only looks for directories. My rationale for that is that most “well behaved” programs will create necessary files with the correct permissions as long as the directory exists. By including files, you end up with lots of unnecessary entries.
The script also checks for a emonpi.conf and will not add dynamic entries for anything in there. I think anything related to packages that are included as part of the standard emonsd should have entries in there, such that emonpi-dynamic.conf is empty unless the user manually adds their own packages (eg in my case openhab2)
If emoncms.log is required at start-up, you can add an entry to emonpi.conf
emonpi.conf should probably be included in /home/pi/emonpi and copied/linked to /usr/lib/tmpfiles.d That way it can be easily updated if required, as part of the standard update process
If a user installed package requires files to be created, I think it’s reasonable to consider that advanced usage, and require the user to create their own tmpfiles.d entries. It’s at least better than the current behaviour where the user has to create all directories as well (eg /var/log/openhab2)
I don’t think this should be limited to emonpi systems. Whilst that is a large proportion of installs, as it is non-destructive, there is no reason not to include it for all to use if they so wish.
AFAICS, you do not actually need the static conf file as the script will add any existing directories when it is run? The key would be to run it after any new install has taken place or prior to shutdown.
There was some discussion a while back on a script to save and restore the log files on a restart. Anyone remember that?
I also disliked the unnecessary restarting of services at boot although my solution wasn’t as nice as tmpfiles.d. Good work!
The file should be delivered into /etc/tmpfiles.d or should be delivered by deb.
While I was hacking, I noticed that if /etc/rc.local wasn’t restarting emonhub, it would fail to send data to emoncms.org because the kernel random number generator didn’t have enough entropy and so the HTTPS connection failed. The solution is obviously for emonhub to retry if it fails, but the error is deep enough in the system libraries that a workaround isn’t obvious and anyway it will surely just be fixed by the next OS upgrade. In the meantime I don’t have a neat solution, so please be aware of this issue.
If we ignore users installing their own packages for now, and just consider the standard emonpi image, I would consider it its own distribution (based on Debian)
From the tmpfiles.d man page:
Files in /etc/tmpfiles.d override files with the same name in /usr/lib/tmpfiles.d and /run/tmpfiles.d. Files in /run/tmpfiles.d override files with the same name in /usr/lib/tmpfiles.d. Packages should
install their configuration files in /usr/lib/tmpfiles.d. Files in /etc/tmpfiles.d are reserved for the local administrator, who may use this logic to override the configuration files installed by vendor
packages.
So, I think /usr/lib is the correct place for the the conf file(s).
Also, in that case, there is no need for the script, we can just ship /usr/lib/tmpfiles.d/emonpi.conf with all the required directories/files to handle /var/log being mounted as tmpfs.
Now, if we want to support (advanced) users installing their own packages, there are a few options:
provide script, and make it run automatically to detect new log files after packages are installed and/or during clean shutdown. In this case, I would consider the script to still be part of the emonpi “distribution” so I’d put the output in /usr/lib/tmpfiles.d
provide script to be run manually. In this case it could just output the config, and user could use it to update their own files in /etc/tmpfiles.d
Another longer term solution would be to provide upstream patches to packages to make them create their own /usr/lib/tmpfiles.d/<package>.conf when /var/log is mounted as tmpfs
I’ve tried log2ram and found it works really well, I have since modified a fork to improve things further by relocating the rotated files destination to disc rather than holding them in RAM.
Simply maintaining an on disk copy of the tmp log files/folders resolves many of the issues we face without overly complicating things.
All the files and folders are present at start up with the right structure,ownership and permission levels without maintaining any lists.
All logfiles are persisted across reboots as they are persisted at shutdown as well as hourly.
Works transparently with installing new software as any newly installed log files/folders will get copied within the hour or it can be triggered manually (although unnecessary if rebooting)
Plus
All the rotated files held on disc rather than in RAM reduces the RAM usage, minimizing any chance of the FS choking and a much longer history can be retained too, no longer would we need to debug within a couple of hours (without rebooting) in hope of finding some usable log entries before they are rotated out or lost in a reboot. The /var/log folder is much tidier too without the rotated logs cluttering the place up.
If there is any interest in reviewing or discussing this option too, We can start another thread.
My emonPi crashed sometime last night, it stopped logging and disconnected from wifi. Since the logs are all temporary, I don’t have any information to help debug this. I also only have about two hours’ history in the logs, which are absolutely full of very verbose debug logging from RFM2Pi and MQTT, and /var/log/syslog and /var/log/daemon are basically identical.
I would like to suggest the following:
Automatically rsync the logs to permanent storage every 5/15 minutes, e.g. from cron.
Restore from permanent storage at startup.
Compress rotated logs.
Disable the debug logging.
Route log messages to either /var/log/syslog or /var/log/daemon, not both.
I could submit a pull request if someone can point me to the correct source code on github, assuming that it’s not a deb package (I dislike debs intensely).
Hi @qris, have a search around on the forum and you will find why it has happened (change emonhub to be run by systemd) and a couple of fixes that can be applied (to rsyslog & logrotate, or to the service file).
The favoured route to sort this out permanently is to use Log2Ram, modify the service file and rework the Logrotate scheme.
I can definitely fix this on my own box now that I have SSH access, but that’s not my point. I think it creates a very bad user experience if most of the emonPis out there are crashing randomly and it’s just written off as a known problem without a solution. So I wanted to fix it for everyone, not just for me.
Log2ram seems controversial so I would not propose a solution that incorporates it right now. I’d favour simple changes to our default configurations to improve matters for everyone. It may make no difference to those who have already installed log2ram, but I’d guess they are the minority.
Agree with the first part but not the second. I think it is now accepted as a problem by @glyn.hudson & @TrystanLea and there is a solution (well a couple actually) it is just a solution has not yet been rolled out.
@borpin please could you point me to any known solution other than log2ram? I’m reluctant to install that one right now because it’s controversial, invasive and I’m an emonPi newbie (with ~20 years as a Unix developer) and do not want to wreck my new emonPi and lose my data.
I’m surprised that the developers are not placing more urgency on implementing some kind of fix or workaround in the standard image. In my opinion the priority of fixing this issue should be much higher than any new feature development.
Too much log data going into /var/log due to emonhub being switched to systemd and the logs not being cleared due to logrotation failing.
The simple and quick fix to the emonhub.log is to revert emonhub to use init.d rather than systemd. That way “normal” operation is resumed. But that breaks the emonhub log viewer in emoncms.
The “proper” way to fix the emonhub log is to correct the ExecStart line in the service unit to use the --logfile parameter in the command as intended. But this not only breaks the emoncms log viwer as above, but will also require further amendments (ie add emonhub to the rc.local folder creation and service restarts) as the service unit doesn’t create the log folder which the init.d does.
As for the logrotation part, there were some fixes rushed through that didn’t work, again this could be easily fixed in the short term by taking a couple of steps back.
The delay is now possibly because an all encompassing final solution is being sought rather than fixing the immediate problem. The delays to finding that total solution is predominantly due to people throwing in other partial solutions and raising concerns about the L2R solution.
We cannot debate L2R for an eternity AND not implement a hotfix in the interim, but that does seem to be the position.
Sorry Brian but that alone will not work as there is no folder created by the service unit. This is why I would not support the change to systemd until we had resolved the rc.local issue.
Maybe adding emonhub to the rc.local might work as a stop gap. But I do not know for sure what the ramifications might be for forcing a restart. It might well be ok, it might not.
Only if the logrotation remains broken.
I would prefer the rotation to be fixed (I know it wasn’t great when it did work, but it is essential to be working otherwise the logs will just grow and grow, rather than stopping within hours, it will stop after a few weeks maybe.
Yes, sorry, I’d forgotten I had added an rsyslog modification to the standard image which must create the right folders for you based on the service user - cannot see how else they are created for the emonhub user.
Just to correct myself here, I just took a look at the source and it seems a check for /var/log/emonhub/emonhub.log is still there ahead of resorting to a journalctl search.
SO! The least invasive route by far is to just revert to the init.d script, the emoncms logs will still work and there will be no need to alter rc.local as the init.d script will create the log folder at start up.
This is not intended to be permanent or even progress! Just reverting to something that previously worked without fault for a very long time, until a more complete solution is rolled out.
Just need to stop and disable the current emonhub service, remove the symlink to the service unit, add a symlink to the init.d and then restart and enable the emonhub service again. No files need editing, just a change of symlink and a few systemctl commands.
I looks like we might easily stop more users pulling in the breaking changes (to emonhub) simply by commenting out these 3 lines in the emonpi updater (AFAICT but cannot be sure).
and it is easy to walk someone through reverting the changes manually, we just need a way of scripting a check for the new service unit to trigger a reversion in the update script until such time as we can re-implement the service unit.
I’m happy to look into doing that if there’s an appetite, but I’m not ploughing time into something that is going to be challenged and debated for weeks on end.
My thinking is since this is temporary, lets settle for a known good solution that has worked flwlessly for years rather than slapping more sticking plasters on it, the move to systemd has not brought any benefits at all, only headaches! so there is nothing lost by taking a temporary step back, there is no point in ploughing time into trying to get emonhub working on systemd ahead of a complete overhaul to the logging system. That effort is better channelled towards the permanent solution than creating fixes that will be redundant once a proper solution is rolled out.
I would of thought so too, I suggested this way back, in the form of a “requirement” for emonhub to make the change to systemd. That’s pretty much why the emonhub systemd change was on hold, then Trystan decided to go ahead anyway and just rely on systemd to handle the stdout, hence we are where we are.
I have no idea if that would definitely work ok, I forget why there was resistance when I said the service unit should create the log folder, it was a long time ago. It would need to create the folder with the right ownership/permissions, but yes that would (if it works) be an easier solution, providing it’s easy to set up and doesn’t cause any issues that need further fixes, again, it might be better to revert to a known good position temporarily until a full solution is rolled out rather than ploughing time into further developing a new temporary fix.
We just need to get on with it rather than debate 101 different ways to do it! If we have rolled back that change as soon as Brian noted the issue, many users would never have been exposed to it. Because of the debate on how to move forward (rather than a swift and temporary step back) many users now have the issue and we now also need to formulate a update procedure to fix or undo something that could have been nipped in the bud before being widely deployed.
Just to action something! I have just created a PR to stop more users falling into the same trap, just until we have a better fix.
It doesn’t fix broken setup’s, but perhaps more importantly right now, it prevents breaking working setups further.
