Is this related to: RAM usage?
Check the Last post.
Since i installed the node-red nodes, i assume that it was the reason of my RAM increase, but now i have some doubts.
Maybe some update change the logging?
Sorry, I missed that. I will answer over there.
Which is most likely why most distos do not include the changes made to the emonSD to go all “systemd only” and also why most softwares still use traditional logging. On a personal note I still prefer the program logs seperate to the deamon logs so a mix of traditional logging and journalctl just for daemon seems the best option, possibly a position also prefered by the major software and distro vendors at this time. Perhaps this will mature and change (as my my own opinion) over time but we do not need to be leading the charge on adopting this. we have so many other things to resolve and so little experience with the cutting edge systemd and journalctl etc.
tmpfs 464M 460K 464M 1% /run
. . .
log2ram 40M 6.6M 34M 17% /var/log
Looking at my (non-sick) Pi I have 460K /run total and 6.6M in /var/log. The 464M and 40M are just max limits, this space is not reserved.
Your sick, loaded and recently rebooted emonpi shows
tmpfs 489M 6.6M 482M 2% /run
. . .
log2ram 40M 1.4M 39M 4% /var/log
which isn’t a vast difference.
The default for emonhub has always been debug (thankfully) and it has been the most useful debugging tool for the whole OEM project for years.
If it is, that is a new position. even when emonhub cannot post to emonhub.log it has always continued to function 100%. I mentioned above that I had seen full logs in the past. Some of these full logs I found in passing when updating something or just doing a health check. The logging had stopped months previously but gone unnoticed as the system continued to work flawlessly. In contrast to that, I have a local Pi that has also filled it’s log partition and whilst emonhub continues to function and emoncms still gets most of the data, my “cron based” nodes have ceased to update, on inspection, the logs were full. It is apparent that those cron entires fail when syslog cannot write to /var/log.
It only takes one non-systemd software to upset that idea, for example emoncms!!!
We still need a ram log to handle even just one log, plus we need the folder structure at boot, plus we need to try and retain those logs beyond a reboot (or hourly rotation).
installing log2ram is ordinarily a one line install and a total solution to all those issues system wide.
What you propose would involve editing each and every install software (not just OEM stuff) to ensure it logged to systemd only and any future installs by the end user will need to comply to, they will not be able to “just install” stuff and it get handled transparently behind the scenes by log2ram.
I have no problem with working towards a total systemd setup if that’s how it pans out, but I have a problem with us forcing the rollout at the cost of more important stuff especially when it creates new issues too, eg emonhub stopping.
The fact Trystan has shown interest in log2ram is potentially the most significant leap forward for this project in years and I am not exaggerating. Installing log2ram is a quick and well rounded solution to many issues, we are still dependent on rc.local for anything to run! Please do not try and block this progress in hope of fully implementing a far more complex 100% systemd solution in the same timescale and expect that to not cause a multitude of new issues. installing log2ram is an immediate and long overdue repair to the existing image, once that is implemented we can indeed explore the advantages of journalctl etc at a more relaxed pace and get it right.
I do not see the aim to use more systemctl and journalctl as a reason to abandon log2ram, we can use both, eventually if journalctl does successfully negate the need for log2ram, it can be dropped then, but I suspect it will be needed for some time to come.
I’m not against using journalctl, but I am concerned about putting all our eggs in a single (new, unknown and cutting edge) basket. This feels like the adoption of MQTT all over again. It doesn’t seem to matter if it works so much, just as long as it is sytemd and journalctl! We are only just getting to grips with MQTT in emoncms now, 4years later! are we in for years of issues with sytemd and journalctl just because we dropped all the alternatives too soon? What ever happened to the value of “tried and tested”?
We have never solved why it (systems stopping) has been happening - still seems most likely cause to me (unsubstantiated).
Did this change when the systemd emonhub service was introduced and the logging mechanism changed? I suspect it may have. Emonhub itself seems to continue to work but other elements of Emoncms grind to a halt.
I’ve seen it here. The EmonPi continues for quite a while once /var/log is full but eventually Emoncms stops responding and recording feed data.
Accepted.
Starter for 10…
Logging Strategy
- Use log2ram to reduce SDCard writes.
- Use journald for daemon services configured to use RAM to reduce SDCard writes (standard setup).
- Use logrotate to limit space consumed in
/var/log. - Use monit to monitor log directory and force specified actions to recover space.
- Do not direct output from
daemon(services) to syslog (modify rsyslog). - Develop mechanism to store specific journald log entries for specified time (such as emonhub).
As Journald is here to stay, I’ll investigate the persistent options for journald. There are various settings to control size and rotation but I am not sure it uses RAM initially, then will flush to disk - it is unclear.
For now perhaps the solution for emonhub is to have a service timer that extracts the emonhub log entries once a day to file and also a service that extracts the log entries immediately before a reboot. Those log files could then be rotated as normal although that does feel a little Heath Robinson.
Are you still running the init.d emonhub?
There’s a stretch backport of the updated systemd that adds the “log to specified file” functionality for any service unit. It may be worth including that in any considerations?
1 Like
On the contrary, it has been solved many times! Not all “it’s stopped” issues are caused by the same thing, we have found the issues with mqtt, feedwriter, emonhub etc and we have found those by checking the logs (emonhub mainly) if the stoppage was due to /var/log being full, the logs would out of date. Not saying it has never happened, but I do not think is could be as prevalent as you suggest
I believe so, I do not think this rate of filling up would have been previously missed.
Perhaps the mqtt and/or feedwriter cannot function for the same reason the recent change stops emonhub since they all use the same systemd and logging strategy?
I don’t agree with any proposed strategy if it means losing emonhub.log (as it was), but thb the jump has been made and I doubt there will be any backtracking so it doesn’t matter what I think, bolting the stable door after . . . doesn’t help anyone!
Absolutely and as each day passes I can see that is less likely to change.
Just a little?
Just to be clear, are you suggesting that all the apache,redis,mariadb,mosquitto (plus anything else that gets installed) is edited on a one by one basis just to move the logs from log2ram to journalctl? Bear in mind both “use RAM to reduce SDCard writes” one works transparently and must be present for non-systemd stuff, the other demands loads of editing.
I think we will agree to disagree. I think it is more prevalent than we think but masked by folk simply rebooting. Perhaps add in an email warning mechanism for when the log folder fills up and memory use exceeds x% (as I think once the folder is full, rsyslog might keep the logs in RAM).
No, just those that already do out of the box although where there is a simple solution (e.g. redis config item syslog-enabled) it could be considered - we fiddle with the logging quite a lot to reduce writes this would just be a different fiddle. Inherently, our fiddles with the stock image is the reason there is a need to fix this i.e. we mount /var/log to tempfs, so limiting the size, without doing sufficient other actions to prevent it filling up and which stock applications do not take into account with their standard logging configuration. Log2ram will only help if the folder does not fill up in the first place AFAICS.
Not necessarily. If a combination of log2ram, logrotate, journald & rsyslog are setup correctly, then any package (and more will go the journald route I believe) installed will just work with that setup. I don’t think we can say simply installing log2ram will solve everything (and I don’t think that is what you are saying).
For instance, logrotate only works if there is enough destination space to rotate to.
I am not wedded to any approach, I just want to discuss and establish a clear strategy to work to in creating an install script that works.
Amen, I have brought this up so many times in the past, but first I was told it isn’t a problem, then when it was, the hourly logrotation was rolled out as a cure.
This has already been suggested by me previously, I also suggested adding /var/log to the admin page alongside the ram and disc usage.
We need to fiddle less, full stop! I am not suggesting log2ram is a total solution (especially as it stands now) but it can remove most if not all of those fiddles. This is why I added the olddir logrotate to the standard L2R (log2ram) to try and fill it out to better suit our needs, a forced rotate via monit sounds like a good move (not totally researched yet).
True, but when logrotate has been running for a while it has files to delete to make room for the new ones so it becomes less of an issue after the retention period is reached after start up. But this is a managable issue anyway. The monit trigger to force a rotate when the ram partition is full can simply check for space first, make space (by deleting older logs) if needed or dump the rotations to /dev/null (or simply delete/empty the logfiles in situ) to stop the system failing. This is where we should be focusing are efforts, in a global policy that works transparently for all logs and promotes logfile retention without impacting the card life. NOT fiddling here there and everywhere, removing logs, reducing loglevels, deleting valuable log data etc etc.
Exactly my point!
I’m not against using journalctl were appropriate, I’m just not wanting us to waste time converting everything to use it when it won’t suit everything and it will introduce more issues. I am always for choice! L2R and/or journalctl, MQTT and/or HTTP, JSON and/or CSV etc etc I always try to reign in the sudden revolt to another option, it only because those trying to force the total rollout of what ever is new and shiny to everyone whether they want it or not, that causes me to argue against that position that makes me appear against it, I’m not, but it can seem that way when I am forced to keep going over the same points again and again to retain CHOICE or slow the rollout in order to get it right.
You are right, it is not what I’m saying.
If only that was the position BEFORE making all the changes! There is no point being so sure that something is right to roll out and then being open to discussion after it doesn’t work out so well and the horse has bolted.
1 Like
Another point about using L2R with journalctl.
If you simply create a folder in /var/log called journal it would cause journalctl to be “persisted” to /var/log, yes that is still in ram, but then L2R would back it up and rotate it as needed just like any other log. Much simpler than “fiddling” with journalctl to get it to periodically persist.
Dave has made a good point about that not potentially catching vital crash logs if L2R hasn’t got time to write to disk, but that could be explored further (and it’s not something we would miss, currently emonSD starts with no previous log data at all), as can doing it directly via journalctl, my point here is that if we were using L2R already, “just add a folder” and we easily and instantly have persisted journals, no edits, no fiddling, no complex stuff and no additional write load to wear the sdcard, job done with one “mkdir”, takes the heat off to allow a more leisurely look into how to better that rather than a hastily rollout of a irreversible complex solution with very little noticeable operational difference.
Yes I’d thought of that. The problem could be the size. As /run/log/journal is quite large (your RAM size effectively), you might run into issues as journald would not expect the size to be limited. But you can manage that by some other journald directives. You might need a L2R write commanded by monit.
It is an alternative approach to restoring the generation of a specific emonhub log file.
I think it would need a bit more than that as I suggest above.
Not suggesting that.
That sounds like it’s making things over complex. Remember regardless of whether the journal is inside or outside /var/log it is still in memory and using the same amount of space, all we perhaps need is to increase the /var/log to accommodate the extra load. BUT that extra size in /var/log is no bigger than a non-persisted “this boot only” journalctl file most of the time as the (default) daily logrotate would move the previous days/boots data off to disk.
The default max size of the journal file is 26Mb whether it’s in /var/log or not, IMO the 50Mb of the emonSD is too small, as I stated elsewhere the 50MB is only a ceiling not a reservation of space so I would set that to perhaps 10% of the total ram, eg 100M on a Pi 3, the image works fine on a 512M ram Zero or Pi B (effectively leaving only 462M for non log data) so I think we can run in 924 (1G minus 100M) without an issue. Especially if using L2R so that whilst the ceiling is higher, the logs held in ram are smaller (no rotated files) and journalctl is the same size (effectively the move to 100M would potentially only rob 24M from the system even when maxed out) because the 26M journal has moved along with the remapped RAM)
Where did you get this figure from?
Yes agree with all you say - my point was that we can’t simply create the folder and assume all will be fine and dandy. There will need to be some other tweaks along the way.
I read it in passing on my travels when looking at persisting journal logs, but the fact i cannot find it now makes me question it’s validity. I’ll have to do more reading up I guess. None the less, regardless of the exact max size, the concept remains the same, if we have journals bigger than that hanging around in RAM we potentially have bigger issues to resolve.
undoutably, but the aim has to be to keep those tweaks to a minimum and try to use stock settings and non-tweaked setups where possible.
Never suggested that it would be!
However, with a well tuned L2R+ to manage the logs globally, simply creating the folder is exactly how easy adding the journal is!
As I stated sometime ago, I expect L2R to be just a starting point to develop a better solution, I have already added the logrotate olddir directive. To that I’ve already mentioned using monit to push a logrotation. My thoughts on this are actually to delete the logrotation cron and to use L2R to trigger the rotation check hourly immediately before doing the backup. This would mean that the log.bak (persisted logs) and log.old (rotated logs) would always align rather than the rotate doing it’s own interval and the L2R another. To this I would add a log2ram --force (or something similar) so that monit could just call log2ram --force rather than logrotate separately, perhaps the --wrirte option could even escalate itself to execute a forced rotation based on the size of /var/log?
Also I think it might be a help to add the “enable journal persistence” to L2R config file so that on a new install the journal folder is created automatically (if configured) and can be switched on or off via L2R to keep the log management in one place for now.
Even the size in ram could be changed to a percentage value so that when installed to a Zero or Pi B the partition is created by L2R as a percent of total ram (eg 10% = 100M’ish on Pi 2 & 3’s and 50M’ish on a Pi A,B or Zero) rather than a defined size in MB (currently 40).
No. Only if the user (or the distro, perhaps because it thinks its users are stick-in-the-mud) has chosen to do such duplication explicitly.
Only if you have chosen to have it in transient storage. I keep arguing that the journal is THE (pardon the emphasis) thing that you want most in persistent storage, to be able to debug crashes.
AIUI, debug level will log less than warning level. I suggest that warning is a sensible default as @TrystanLea has proposed.
BTW, systemd and cron coexist quite happily, but if you want to be PC, you could rewrite cron jobs as systemd timers instead.
No, debug is more verbose for debugging purposes, whilst warning only logs events that are deemed necessary to alert the user of.
Yes! The default setting in journald is ForwardToSyslog=yes. Whether rsyslog is configured to store the messages (by default) is a different question.
Again incorrect, the default is auto so it will only store to RAM and not disk unless it is changed or a /log/var/journal folder is manually created by the user.
See https://www.freedesktop.org/software/systemd/man/journald.conf.html
" SystemMaxFileSize= and RuntimeMaxFileSize= control how large individual journal files may grow at most. … Defaults to one eighth of the values configured with SystemMaxUse= and RuntimeMaxUse="
" SystemMaxUse= and RuntimeMaxUse= control how much disk space the journal may use up at most. … The first pair defaults to 10% … of the size of the respective file system … each value is capped to 4G."
"ForwardToSyslog= , ForwardToKMsg= , ForwardToConsole= , ForwardToWall=
"Control whether log messages received by the journal daemon shall be forwarded to a traditional syslog daemon, to the kernel log buffer (kmsg), to the system console, or sent as wall messages to all logged-in users. These options take boolean arguments. If forwarding to syslog is enabled but nothing reads messages from the socket, forwarding to syslog has no effect. By default, only forwarding to wall is enabled.
We’re agreeing violently. One way or the other, you choose.
Your default must be different to Raspbian default then.
From the man on my system;
CONFIGURATION DIRECTORIES AND PRECEDENCE
The default configuration is defined during compilation, so a configuration file is only needed when it is necessary to
deviate from those defaults. By default, the configuration file in /etc/systemd/ contains commented out entries showing the
defaults as a guide to the administrator. This file can be edited to create local overrides.
The commented out settings are what was set to default when the package was compiled. journald.conf(5) — systemd — Debian testing — Debian Manpages
pi@emonpi:/var/log $ cat /etc/systemd/journald.conf
# This file is part of systemd.
#
# systemd is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation; either version 2.1 of the License, or
# (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See journald.conf(5) for details.
[Journal]
#Storage=auto
#Compress=yes
#Seal=yes
#SplitMode=uid
#SyncIntervalSec=5m
#RateLimitIntervalSec=30s
#RateLimitBurst=1000
#SystemMaxUse=
#SystemKeepFree=
#SystemMaxFileSize=
#SystemMaxFiles=100
#RuntimeMaxUse=
#RuntimeKeepFree=
#RuntimeMaxFileSize=
#RuntimeMaxFiles=100
#MaxRetentionSec=
#MaxFileSec=1month
#ForwardToSyslog=yes
#ForwardToKMsg=no
#ForwardToConsole=no
#ForwardToWall=yes
#TTYPath=/dev/console
#MaxLevelStore=debug
#MaxLevelSyslog=debug
#MaxLevelKMsg=notice
#MaxLevelConsole=info
#MaxLevelWall=emerg
No, we are not.
On the EmonSD there is no folder /var/log/journal, and the Storage= is set to auto so logs are only held, by journald in RAM not on disk. auto will create a /run/log/journal/ folder if it does not exist but will not create a /var/log/journal folder. So the user needs to choose to hold the journal on disk either by changing the storage= to persistant or create the /var/log/journal folder.