No idea yet, but I very much doubt it. I’m not sure we can afford more tham 10M in a 40-50MB partition for one service in a fault condition, remember emonhub is the ONLY log that will behave itself under these circumstances, so it’s own limitation gives more room to other softwares, eg emoncms for example, might run a little longer and make it to the next rotation (but we may have monit to assist with the overall size of /var/log too?)
copytruncate is costly in terms of space in ram, we would need to maintain a greater headroom for copytruncate to work and not cause logrotate to fail.
rotate 5
size 2M
That’s only 10M, not much bigger than the original logs, even 30 x 3M is only 90M of diskspace (for example), that is a month if rotated daily (due to a low log level).
We need to define the whole logrotation picture, not just emonhub.
In your setup, emonhub is rotating it’s own log in /var/log/. These are then rotated by logrotate, and persisted by L2R? How many do you end up with?
Agreed which is why concatenating (instead of simply moving) is no additional cost in terms of writes (the written part is not rewritten) and can then form a set of daily logs.
Yes, but we had not discussed doing that and it does duplicate anything in rsyslog logs for no real purpose. journald should (I think) manage it’s own persistent log (other than a L2R persistence). I’d just pull the information out as above.
Except after a reboot.
Why not? Simple to do with little overhead (as outlined above).
I think we have already agreed to use standard logrotate config for everything else? I think working out the emonhub config would be a good starting point, we can then apply the same to emoncms.log.
Seems to work fine, apache creates multiple instances all writing to the same log file, its worked find so far as far as Im aware.
However, I’m not sure how much of an issue it is for our system. It seems to work fine.
@pb66 suggested renamecopy in post 71 (not sure how to create a link) although I think that uses the same basic mechanism as copytruncate but works across devices.
Personally Im still thinking emonhub set to warning or error logging level is my preference, it will then hardly ever even get to the first logrotate… If you do otherwise need to run it in debug mode, it might be good to record 1-3 days? I guess it depends how long it takes you to notice that there is an issue and pull up the logs.
@TrystanLea, can I suggest that you move the install scripts to a new ‘EmonScripts’ repo, add in the L2R install, plus some initial config and then we could have a common setup (but with different inputs) to be able to comment on?
No, the emonhub direct rotation isn’t going to be in play** the emonhub.log will be rotated by logrotate to log.old just like anyother log. What ever is left in ram is also sync’d to log.bak, as per any other log too.
**Unless under fault conditions where normal rotation has not occured for what ever reason or a race condition, then the emonhub.log will not be allowed to grow above 2x5M due to inbuilt rotation. When normal rotation resumes or is forced, then the logrotation conf will cater for any additional emonhub logs found in ram, before rotating so that everything returns to normal with one log in ram and the orderly chain of rotated files in log.old.
Ok, perhaps? We could look into that, but again, it’s not essential to getting this set up and then we can fiddle with stuff like this.
We have discussed this several times on another thread and it doesn’t duplicate anyrthing, journald will just maintain it’s log in a different place. We can (as discussed many times in that other thread) move to a journald persistence if a direct approach is preferred. This would possibly be best done using a symlink in place of /var/log/journal. But again, this can be fiddled with afterwards it is not essential to get this up and running.
We seem to be sweating the configurable details that can and will be fiddled with again rather than getting the core eco-system in place
I don’t know what that means, do you mean because the’re not persisted? They would be if we chose to persist! I do!
For the reasons given, not needed, not a good idea to make extra work, extra points of failure, extra complication for very little (if any) gain, it works fine as it is if we just stop off loading everything into it and by simply adding persistence, even simple persistence via L2R (even if only for a stopgap until we do something else)
No, I said I was using that rather than using the emonpi logrotation conf.
No unfortunately not, I was making a start on the logrotation conf on Tuesday but as I stated, not reached any conclusions yet. Any spare time I’ve had since has mainly been spent here on the forum discussing.
But we should try and decide what we would like (in general terms not exact sizes and number of rotations). For example, I have already suggested we avoid editing the standard conf files installed by the packages and instead we maintain our own “emonSD” logrotation conf file(s) for use with L2R. The path to this conf would be used in the logrotation commands rather than the /etc/logrotation.conf that must currently be included, logrotate never assumes a conf file.
By doing this it allows us to maintain our own (set of) conf file(s) without fear of a package overwriting during an apt-get upgrade. It also means that non-emonsd installs can be left as is to use the default logrotate conf etc. This means that a switch to or from L2R (eg installing emonSD image to a hdd?) is just a case of changing the logrotation command string as part opf the L2R install/uninstall.
SO, that means we should have a non-L2R type logrotation conf for both emonhub and emoncms etc for “just dropping” into a non-emonsd install.
Assuming we have a full compliment of logrotate conf files at /etc/logrotate.d/ we can then copy all of /etc/logrotate* to a place in our emonsd? repo (not emoncms, nor emonpi) and edit to our own needs using the originals as a starting point.
We will also need some form of a catch all rotation perhaps, so that when users install something new, it will not cause failures due to log filling up. again monit might help here rather than having too loose a catchment like the current emonpi (there are warnings about using wildcards too much in logrotate)
However, when users do install something new, if the package installs a logrotate conf (to the default location) it will be used by non-emonsd installs, but ignored in emonsd images. We maybe need to document or notify (via log or email?) the user to create a custom logrotation conf. the original can be used as a template. The advantage of mimicking the default multi-file conf is that when users add their own logrotations to a new file rather than editing an existing single file, it won’t block future git pulls.
Ah, I understand, the hard coded limit should not be reached. Got it.
Sorry, but yes it does, as journald forwards everything to syslog by default even if journald is set to persistent (or auto with the folder present)
My point was that relying on journald for log entries is flawed under default conditions (i.e. EmonSD/raspbian standard setup) because journald:
does not persist logs by default (lost on reboot even with L2R)
only keeps a fixed amount of logs (in terms of size) so can be quickly be rotated out under fault conditions or where emonhub is using stdout.
Agreed - but what you then go onto say below seems to contradict this. Just add config files - do not change anything.
No I think the config should go in logrotate.d so it simply gets picked up by other packages as well - for instance the olddirs directive (assuming we want all log files to use it). The config should also be specific to the element of emon system not a catch all config file.
This can be managed by a cascade of logrotate configurations.
No we should simply be writing specific config files.
I think that is wrong. We want L2R to manage all log files in /var/log/ as we do logrotate. there seems little point in impacting on the installation of other packages (which is what we do now with the replacement of the default logrotate config file).
yes I thought that was where we had arrived at. That we where not going to modify or replace /etc/logrotate.conf and not going to modify any of the package configurations e.g /etc/logrotate.d/apache2
that would leave us with only needing to add a logrotate.d configuration for emonhub.log and emoncms.log
Interesting suggestion, Id be happy with that. So /var/log/emon/emonhub.log and /var/log/emon/emoncms.log, /var/log/emon/sync.log? rather than further subfolders?
But that is a different issue external to what we’re doing here. It is not because of persisting the logs that the duplication occurs, the persisted log is not a duplicate of the non-persisted log, only one or the other exists. If that log (persisted or not) is duplicated to syslog that is something else entirely.
One might argue that IF we persist the journal, we are writing the same info twice to disk (if it’s all duplicated in syslog) and that should be avoided. BUT if that is the case, how can you say the service log data is not persisted after a reboot if syslog Is being persisted? We simply need to look in syslog rather than journald? surely.
My point is IF we need/want to persist the data held in journald (again?) it can be a simple case of adding a journal folder, it does not cause further duplication, we do not need to be extracting stuff and putting it somewhere else. BUT regardless of whether we do or do not go that route, it doesn’t in anyway effect how we implement L2R or logrotate.
If we leave journald alone, let it forward everything to rsyslog, L2R will persist all logs so the details are saved within syslog (or more likely daemon.log) for our services.
However, in order to easily filter out what we are not interested in (if debugging), by using a set
of rsyslog configuration files, we could easily pull the relevant entries out of syslog (so no duplication) and put them in our own services log file.
Then when debugging, we want the contents of the running log (e.g. emonhub.log) and the emon-services.log (new name) which has any other output from the services. With a logrotate config file for that log, it is then rotated and persisted by L2R with no further interaction - it just happens.
We have mentioned not using copytruncate, how will you remove that without editing ?
We have mentioned rotating by size and that all softwares should not be treated equally, we cannot cap emonhub at 1M just because we have so many other logs, setting a size for each offers greater flexibility.
The olddir is currently set for a global "log,old` folder, so all logs will get dumped in the one folder. If you want to preserve the folder structure you can define olddir in each conf so that olddir for apache2 (for example) is /var/log.old/apache2. Yes we could move the files afterwards instead of changing olddir, but that would be in a post-rotation so still need to edit the conf files.
Some logs eg apache access may not need retaining so those logs could be deleted or truncated so that recent access is held in ram but not kept on disk.
The limits and settings for emonsd may not be ideal for a hdd server, so different versions maybe better.
If you commit to NOT editing any existing files, then you are stuck with never editing any conf’s as you cannot really start “tinkering” with sed etc in the update scripts. A second copy in a repo allows editing via updates without any “tinkering”.
If there was anyway to just stick with the (untouched) defaults I would jump at it, that was the original intention for the 00_olddir conf file, but the more I look at logrotate, the more I think that a custom system wide conf is needed.
This is nothing to do with L2R really. All L2R is actually doing is maintaining the ram folder structure and keeping the tmpfs sync’d so as not to lose logs.
logrotate is the focus for managing what comes out of ram and what gets kept. logrotate maybe writing to disk more than L2R does, depending on setup. It’s in the logrotation config that you manage and fine tune how well the tmpfs and persistence works, and the defaults were not designed for either SDcard or tmpfs logs, I would be very surprised if we couldn’t improve things via the logrotate conf. This is the main conf area for the emonsd logging.
It’s currently just under 2GB of data on a 8GB card.
Though you are correct, the data seems to be on a 4GB partition, with th OS on a separate partition.
I already use a USB stick plugged into the Pi to create the backup files that go off to DropBox as there isn’t enough space to create more than one backup and the process does a lot of writing.
That minsize means it could hog 1M of ram even when it is never called for long periods of time, for the log to get rotated and release that 1M it must satisfy both the >1M and >one week limits so a runaway monit log (don’t know if that’s a thing or not? just using it as a “what if?” ) might not get rotated for up to a week.
I think generally daily is better than weekly and a (max) size per software should be set, plus olddir as a minimum. Don’t use maxsize as that too requires both the time and size limits to be met before rotating. size is the one we want, it ignores the time limits if the size is reached.
daily or 100K, that gives us neat daily logs unless over 100K, thus keeping RAM usage low. This maybe a tad too aggressive for some debugging, but it shows the contrast. If we just “stick to the package defaults” we are allowing the various package maintainers free reign to use our valuable ram space without control. (Unless we just add a global size 1M and have all small logs. But beware because even then if a package was added that had size 50M (unlikely I know) it could happilly use all the ram without rotating as each packages conf overrides the globals (as far as I can make out) and you cannot add a second conf eg logrotate.d/apache2.1 logrotate accepts the first for any given log file and ignores any others.
I have loads of thoughts on this so don’t think I’m sold on any particular approach yet, I just accept we cannot do nothing!
How about this!
We have our own “main” /emon/somerepo/logrotate.conf file that we can call in our logrotate commands and in that file we set some global settings eg a size 1M (or even 100K?) to limit all logs that are not specifically defined eg emonhub would be size 3M (overriding the 100K).
In that custom conf file we could also have
include /emon/somerepo/logrotate.d/
include /etc/logrotate.d/
that would first include any custom logrotation files that we deemed necessary to over ride with custom conf files maintained in an OEM repo. Then it would include any standard default conf files, the “overridden” conf files will be ignored as they are already loaded by the line above, just the remaining confs would be used. If /emon/somerepo/logrotate.d/ happens to be empty, then all the default “per package” confs are used as per your above expectation. But the globals in the “main” custom logrotate.conf will remain in force.
@TrystanLea Going back to copytruncate question, before writing to the emoncms.log, is a check made to see if the file exists?
I think that copytruncate is for when the log writer cannot (or does not) check for the existence of the log file before writing. Without that directive, logrotate moves (renames) the old log file in place initially (instantaneous) and expects the log writer to create a new one if it finds it is missing (or is instructed to create a new file). Thus the writer can only write to an available file. AIUI, copytruncate literally copies all the data then truncates the file so continues to use the original inode. renamecopy does the same (uses same inode).
If you look at the syslog file before and after rotation you will see the original inode now points to the syslog.1 file and syslog has a new inode.
