# emonSD next steps: filesystem & logrotate **URL:** \ **Category:** emonSD\ **Created:** [15 April 2019 15:10 UTC](https://community.openenergymonitor.org/t/emonsd-next-steps-filesystem-logrotate/10693 "2019-04-15T15:10:43Z")\ **Posts on this page:** 20\ **Page:** 9
**Author:** ![TrystanLea](https://community.openenergymonitor.org/user_avatar/community.openenergymonitor.org/trystanlea/32/31_2.png) [@TrystanLea](https://community.openenergymonitor.org/u/TrystanLea)\ **Post date:** [27 May 2019 14:09 UTC](https://community.openenergymonitor.org/t/emonsd-next-steps-filesystem-logrotate/10693/161 "2019-05-27T14:09:18Z")
Do we still need a dedicated emonhub logrotate? e.g of the form: ```auto /var/log/emonhub/emonhub.log { rotate 10 size 3M missingok renamecopy notifempty compress } ``` ---
**Author:** ![TrystanLea](https://community.openenergymonitor.org/user_avatar/community.openenergymonitor.org/trystanlea/32/31_2.png) [@TrystanLea](https://community.openenergymonitor.org/u/TrystanLea)\ **Post date:** [27 May 2019 20:21 UTC](https://community.openenergymonitor.org/t/emonsd-next-steps-filesystem-logrotate/10693/162 "2019-05-27T20:21:17Z")
The default logrotate on stretch, without comments is: ``` weekly rotate 4 create #compress include /etc/logrotate.d /var/log/wtmp { missingok monthly create 0664 root utmp rotate 1 } /var/log/btmp { missingok monthly create 0660 root utmp rotate 1 } ``` do we replace weekly with “size 100k”, replace rotate 4 with rotate 10, keep create and then add: ```auto missingok renamecopy notifempty compress nomail ``` and remove the missingok entries from wtmp and btmp? Result: ```auto size 100k rotate 10 create missingok renamecopy notifempty compress nomail include /etc/logrotate.d /var/log/wtmp { monthly create 0664 root utmp rotate 1 } /var/log/btmp { monthly create 0660 root utmp rotate 1 } ``` ---
**Author:** ![TrystanLea](https://community.openenergymonitor.org/user_avatar/community.openenergymonitor.org/trystanlea/32/31_2.png) [@TrystanLea](https://community.openenergymonitor.org/u/TrystanLea)\ **Post date:** [27 May 2019 20:31 UTC](https://community.openenergymonitor.org/t/emonsd-next-steps-filesystem-logrotate/10693/163 "2019-05-27T20:31:31Z")
I’ve got this setup here to test, using logrotate.CUSTOM being called from cron.hourly/log2ram Is there any reason you use log2ram write over “systemctl reload log2ram” @pb66? ```auto # then we sync /var/log to disk log2ram write ``` ---
**Author:** ![borpin](https://community.openenergymonitor.org/user_avatar/community.openenergymonitor.org/borpin/32/193_2.png) [@borpin](https://community.openenergymonitor.org/u/borpin)\ **Post date:** [28 May 2019 08:00 UTC](https://community.openenergymonitor.org/t/emonsd-next-steps-filesystem-logrotate/10693/164 "2019-05-28T08:00:08Z")
> [@TrystanLea](#): > > do we replace weekly with “size 100k” Too small. Better with a specific emon\* logrotate directive picked up through the `.d` folder. I have not seen **any** issues with the standard setup other than those caused by emon\* entries. Personally I’d leave the standard setup alone and do any changes via override files but YMMV. ---
**Author:** ![TrystanLea](https://community.openenergymonitor.org/user_avatar/community.openenergymonitor.org/trystanlea/32/31_2.png) [@TrystanLea](https://community.openenergymonitor.org/u/TrystanLea)\ **Post date:** [28 May 2019 08:25 UTC](https://community.openenergymonitor.org/t/emonsd-next-steps-filesystem-logrotate/10693/165 "2019-05-28T08:25:22Z")
Thanks @borpin When I try my example above the global defaults do not actually apply until I add log files via fixed or wildcard paths e.g: ``` /var/log/* /var/log/*/* {} ``` If I do this and leave uncommented: ``` include /etc/logrotate.d ``` I get a whole load of duplicate entry errors: ``` error: apache2:1 duplicate log entry for /var/log/apache2/access.log error: apt:1 duplicate log entry for /var/log/apt/term.log ``` Do you or @pb66 know where Im going wrong? > [@borpin](#): > > Too small. Better with a specific emon\* logrotate directive picked up through the `.d` folder. I have not seen **any** issues with the standard setup other than those caused by emon\* entries. Im certainly open to that and that would solve the issue Im having, but I think @pb66 was keen for a global override? ---
**Author:** ![TrystanLea](https://community.openenergymonitor.org/user_avatar/community.openenergymonitor.org/trystanlea/32/31_2.png) [@TrystanLea](https://community.openenergymonitor.org/u/TrystanLea)\ **Post date:** [28 May 2019 10:50 UTC](https://community.openenergymonitor.org/t/emonsd-next-steps-filesystem-logrotate/10693/166 "2019-05-28T10:50:12Z")
The next problem I get trying the standard logrotate.conf but with an emonhub sepecific configuration of the form: ``` /var/log/emonhub/emonhub.log { su root root rotate 10 size 1M notifempty missingok copytruncate } ``` Is that I get the error: ``` error: error opening /var/log/emonhub/emonhub.log.tmp: No such file or directory ``` when I run cron.hourly/log2ram (which runs logrotate) With copytruncate commented, the file does rotate however emonhub stops logging. I assume this is because emonhub has lost the associated file pointer, its still looking for the original file? If I restart emonhub it starts working fine again. The same applies with renamecopy. ---
**Author:** ![borpin](https://community.openenergymonitor.org/user_avatar/community.openenergymonitor.org/borpin/32/193_2.png) [@borpin](https://community.openenergymonitor.org/u/borpin)\ **Post date:** [28 May 2019 11:30 UTC](https://community.openenergymonitor.org/t/emonsd-next-steps-filesystem-logrotate/10693/167 "2019-05-28T11:30:08Z")
> [@TrystanLea](#): > > When I try my example above the global defaults do not actually apply until I add log files via fixed or wildcard paths Remember that logrotate will always use the setting specific to the log file definition if there is one. So setting a ‘global’ size limit of 1M would be overidden by a size setting in the drop in file for that log file if there is one. > [@TrystanLea](#): > > I get a whole load of duplicate entry errors: > > ```auto > error: apache2:1 duplicate log entry for /var/log/apache2/access.log > error: apt:1 duplicate log entry for /var/log/apt/term.log > > ``` Yes because you are trying to rotate it twice! That is why it is strongly advised **not** to use wildcards in logrotate. > [@TrystanLea](#): > > ```auto > su root root > > ``` Are you running logrotate from the command line? Are you using `sudo` to run it? AIUI, Cron runs the commands with elevated priviledges so the `su root root` should not be needed. Perhaps it is how you have set up the permissions on the emonhub log file initially. Try `su emonhub emonhub` as that is the right user & group I think. I found I needed to completely purge logrotate, deleted the logrotate.conf link and then reinstalled it to get it working. > [@TrystanLea](#): > > Im certainly open to that and that would solve the issue Im having, but I think @pb66 was keen for a global override? We don’t necessarily agree on this… My view is that if there is to be a global override, create an override file to do it, rather than editing the standard logrotate.conf (e.g. for olddir). However, I do not see the risk of a.n.other log file overrunning the log folder as being significant. The emon\* entries started this fiasco off. Before the issues caused by emonhub being run as a service the logs were fine. If a log is running away, that is an issue with that package **not** with the emon\* packages. This can easily be mitigated by running logrotate hourly which is the only change I’d make to the standard setup. ---
**Author:** ![TrystanLea](https://community.openenergymonitor.org/user_avatar/community.openenergymonitor.org/trystanlea/32/31_2.png) [@TrystanLea](https://community.openenergymonitor.org/u/TrystanLea)\ **Post date:** [28 May 2019 12:00 UTC](https://community.openenergymonitor.org/t/emonsd-next-steps-filesystem-logrotate/10693/168 "2019-05-28T12:00:33Z")
Thanks @borpin , I cant seem to get your first point to work, it seems that setting size globally gets ignored and not because the services have their own definition of ‘size’. Most dont declare a size setting. It seems that the global settings are just not applied… > [@borpin](#): > > Remember that logrotate will always use the setting specific to the log file definition if there is one. So setting a ‘global’ size limit of 1M would be overidden by a size setting in the drop in file for that log file if there is one. Edit: or perhaps its because there is no logrotate entry for the logs I was watching I will double check. ---
**Author:** ![borpin](https://community.openenergymonitor.org/user_avatar/community.openenergymonitor.org/borpin/32/193_2.png) [@borpin](https://community.openenergymonitor.org/u/borpin)\ **Post date:** [28 May 2019 12:07 UTC](https://community.openenergymonitor.org/t/emonsd-next-steps-filesystem-logrotate/10693/169 "2019-05-28T12:07:27Z")
> [@TrystanLea](#): > > Edit: or perhaps its because there is no logrotate entry for the logs I was watching I will double check. If you use the -d option, the output will tell you what **will** happen without is actually doing anything. ---
**Author:** ![TrystanLea](https://community.openenergymonitor.org/user_avatar/community.openenergymonitor.org/trystanlea/32/31_2.png) [@TrystanLea](https://community.openenergymonitor.org/u/TrystanLea)\ **Post date:** [30 May 2019 17:12 UTC](https://community.openenergymonitor.org/t/emonsd-next-steps-filesystem-logrotate/10693/170 "2019-05-30T17:12:57Z")
I’ve made a little progress on above, the specific emonhub logrotate conf with standard logrotate.conf is now working. But I still cant get a global size setting to apply to other logs. I’ve gone back and tested the current global logrotate wildcard approach and I think it works really well. It seems to cover all bases. Might it be worth having two options one for those who are happy with the current approach and another for those who want a more custom approach, where existing logrotate configuratinos are upheld? **Option 1:** Current logrotate config on emonSD Oct18 (comments removed to shorten): ``` rotate 1 size 1M copytruncate nomail su root root missingok notifempty /var/log/*.log /var/log/*/*.log /var/log/syslog /var/log/messages /var/log/btmp /var/log/* {} ``` This works well rotating all logs at 1MB, emonhub.log continues to log after rotation (changing to renamecopy does not work for emonhub.log). I think it would be worth adding compress and increasing rotate. **Option 2:** Default stretch logrotate.conf (noted above) + specific emon logrotate.d entry e.g: ``` /var/log/emonhub/emonhub.log { rotate 2 size 1M copytruncate su root root missingok notifempty compress create } ``` Works great for emonhub.log, we can have additional configs for the other emoncms logs. It does not however stop other logs from getting too large. **Option 3** : as option 2 but with global “size 1MB” override, I cant get this override to work, the default settings don’t seem to apply. @borpin I think you want option 2? and @pb66 option 1 but called logrotate.CUSTOM? ---
**Author:** ![davidelang](https://community.openenergymonitor.org/user_avatar/community.openenergymonitor.org/davidelang/32/19406_2.png) [@davidelang](https://community.openenergymonitor.org/u/davidelang)\ **Post date:** [30 May 2019 17:20 UTC](https://community.openenergymonitor.org/t/emonsd-next-steps-filesystem-logrotate/10693/171 "2019-05-30T17:20:07Z")
please don’t use copytruncate, it’s guaranteed to loose logs at some point because any logs that are written after the copy and before the truncate will be lost David Lang ---
**Author:** ![TrystanLea](https://community.openenergymonitor.org/user_avatar/community.openenergymonitor.org/trystanlea/32/31_2.png) [@TrystanLea](https://community.openenergymonitor.org/u/TrystanLea)\ **Post date:** [30 May 2019 19:18 UTC](https://community.openenergymonitor.org/t/emonsd-next-steps-filesystem-logrotate/10693/172 "2019-05-30T19:18:28Z")
Thanks @davidelang at the moment we have the issue that emonhub does not continue logging if renamecopy is used, logfile pointer still connected to old file? Im not sure if emonhub is just the outlier there… ---
**Author:** ![pb66](https://community.openenergymonitor.org/user_avatar/community.openenergymonitor.org/pb66/32/27_2.png) [@pb66](https://community.openenergymonitor.org/u/pb66)\ **Post date:** [31 May 2019 00:09 UTC](https://community.openenergymonitor.org/t/emonsd-next-steps-filesystem-logrotate/10693/173 "2019-05-31T00:09:12Z")
> [@TrystanLea](#): > > and @pb66 option 1 but called logrotate.CUSTOM? No, definitely not! Firstly, whilst I was keen to use the logrotate.CUSTOM file name for a custom logrotate.conf, that is only _ **IF** _ we have a custom logrotate.conf, at the time of that discussion, I felt it was most likely going to be necessary. But I would prefer to use the standard conf if possible, I’m just not entirely convinced we can. But I’m less hung up on whether we do or don’t use the standard conf than I am about getting it to work correctly. First we need to find the right configuration, and once we know what rules we **need** we can decide about how that is achieved. Secondly, I disagree with many of the rules in the emonSD logrotate.conf, so even if we do arrive at a custom conf of my liking, it wouldn’t look like the current conf. `notifempty` is pointless in conjunction with no timed rotation and size only (empty is less than 1M!) We must have a timed rotation, not just rely on size, a 999k log file with zero activity could sit there in ram for ever, better it is rotated out after a week/month. The `rotate 1` is definitely no good. I don’t think we need `nomail` or `su root root`. we shouldn’t really be using `compress` without `delaycompress` if wildcards are being used, if a service is still writing to the rotated file when it is compressed it might crash that service, `delaycompress` only compresses some.log.2 onwards and is safer unless you know it’s ok to compress the new rotation. The `renamecopy` directive is something I brought up to be used in conjunction with the `olddir` directive so that files can be moved out of RAM. By default olddir can only be used on the same device (ie the rotations could go to another folder, but still in RAM) Either renamecopy, copy or copytruncate can be used to facilitate moving files across devices using olddir. The `copytruncate` on the emonSD was necessary because the `include` drop-ins were disabled so apache2, mosquitto and rsyslog were unable to trigger the post-rotate scripts that made the services let go of the old log handle and start logging to the new file, this isn’t good practice. The loose wildcards are also a bit of a no no as you’ve discovered with the duplicate logs warnings. This could be less of an issue once we are rotating out of RAM as the rotated files will no longer be present in /var/log. > [@davidelang](#): > > please don’t use copytruncate, it’s guaranteed to loose logs at some point > because any logs that are written after the copy and before the truncate will be > lost It is indeed better to avoid using copytruncate if possible, but it is far better to potentially loose a log message or 2, than loose actual data. copytruncate is a low quality rotate method to be used when it can’t be avoided. > [@TrystanLea](#): > > **Option 3** : as option 2 but with global “size 1MB” override, I cant get this override to work, the default settings don’t seem to apply. Is that a typo? AFAIK the rule should be `size 1M` (not MB) or better still `size 100k`, 1M is way too big for some logs, small size is both less space used in ram and less for log2ram (rsync) to calculate each hour. Regarding emonhub and `copytruncate`, I wasn’t aware that emonhub clung onto the old file handle, it makes sense now I think about it, but this has been masked for years by the fact that emonSD used copytruncate across the board. Apache uses it’s “reload” (aka graceful restart) to release old log file handles and mosquitto seems to respond to a HUP signal by releasing the file handles without a restart, as it stands, emonhub doesn’t yet have that specific facility, I’m not great with “signals” so I’m not 100% sure, but I believe any signal able to make emonhub release the logfile handles will cause it to restart and we definitely do not want that to happen, data in transit could be lost and buffered data too. In time maybe a “reload” could be added to emonhub, not previously needed as it loads any new settings whilst running. So in the short term copytruncate maybe necessary for emonhub. However! I’ve previously also mentioned emonhub’s “5M self-rotation” kicking in if the logs run away and that if there is a emonhub.log.1 present, this is something that needs handling in the logrotation definition, either in the pre or post rotation. So another possible solution in the immediate short term might be to (reduce the in-built rotation to 3M and) only rotate the emonhub.log.1 file, yes it means there will be 2 emonhub.log files in RAM, but it also means that there is no complication with a) the file handle as emonhub will rotate it’s own log (in ram) so we can avoid copytruncate and b) the potential of finding and having to handle 2 emonhub.log files at logrotate time is removed too. so maybe something along the lines of ```bash /var/log/emonhub/emonhub.log { # Dummy definition to ensure emonhub.log is NOT rotated # only rotate if over 10M (not possible with inbuilt rotation) size 10M } /var/log/emonhub/emonhub.log.1 { daily rotate 28 size 3M compress delaycompress create 775 emonhub emonhub missingok notifempty # rotate emonhub log files to an emonhub folder in log.old olddir /var/log.old/emonhub createolddir 775 root emonhub renamecopy sharedscripts postrotate # possibly some concatenation into daily files endscript } ``` The explicit definition for emonhub.log would prevent a wildcard picking it up, I haven’t tested this though. The specific mention of olddir in the emonhub (and emoncms?) drop-in is so that at least our main logs are segregated, unless we edit/replace each drop-in, all the rotated files will land in the same folder. Some of the finer points such as length of retention and whether logs should be concatenated can be played with, the main point is to get it working well, whether we have a week or a years worth of data at this point is irrelevant unless the sdcard fills up. As for the globals, we already need the olddir directive defined in `logrotate.d/00-olddir`, you could just try adding `size 100k` to that as it is loaded first. But any wildcards will need to go in a file prefixed with “ZZZ-something” so it is loaded last, but even then it may cause conflicts with the wtmp and btmp definitions in the stock config and we do need some form of wildcard I imagine as there are several logs that slip through the net. The above **might** allow us to have a stock logrotate.conf but the main things that currently work against that idea is that all the non-emon rotated log files will be dumped in a single folder, so /var/log/mysql/error.log and /var/log/apache2/error.log (for example) will clash as both will want to be just /var/log/error.log.1 after rotation, the only way to avoid this AFAIA is to add the olddir path in the drop-ins as I have with the emonhub one above. Plus the wildcard cannot be placed after the 2 definitions included in the stock conf and TBH the “000” and “ZZZ” prefixes look a tad amateurish and are not very infallible, a user adds “0-something” it will slip in first and likewise with “ZZZZ-something” slipping in after the wildcards. Plus, something esle we need to factor in is that not all installs will be using L2R/tmpfs logs, so the logrotate on a non-sdcard install should be different, the olddir stuff won’t be needed, the size values won’t be needed, the wildcards probably won’t be needed. > [@TrystanLea](#): > > With copytruncate commented, the file does rotate however emonhub stops logging. I assume this is because emonhub has lost the associated file pointer, its still looking for the original file? Are you sure it has stopped logging? It should still be logging to the original file, even if it has moved or changed name (ie is it logging to emonhub.log.1?) as the file descriptor will still be intact, unless the file is compressed or deleted (incl moving across devices). ---
**Author:** ![TrystanLea](https://community.openenergymonitor.org/user_avatar/community.openenergymonitor.org/trystanlea/32/31_2.png) [@TrystanLea](https://community.openenergymonitor.org/u/TrystanLea)\ **Post date:** [31 May 2019 09:00 UTC](https://community.openenergymonitor.org/t/emonsd-next-steps-filesystem-logrotate/10693/174 "2019-05-31T09:00:29Z")
Thanks @pb66 for the detailed response, I will go through it today. I’ve finally solved the default size setting being ignored problem I was having, these SE threads had the answer: [logrotate daily and size? - Server Fault](https://serverfault.com/questions/391538/logrotate-daily-and-size) & [configuration - How to rotate log based on an interval unless log exceeds a certain size? - Server Fault](https://serverfault.com/questions/474941/how-to-rotate-log-based-on-an-interval-unless-log-exceeds-a-certain-size) we need to use maxsize which is now supported in logrotate-3.8.1 (stretch appears to have logrotate 3.11.0 installed) From the SE thread: > As of logrotate 3.8.1, `maxsize` and `timeperiod` are supported together, which would be the ideal solution. See the answer to this post: [How to rotate log based on an interval unless log exceeds a certain size?](https://serverfault.com/questions/474941/how-to-rotate-log-based-on-an-interval-unless-log-exceeds-a-certain-size) Just to clarify, I tried: > [@pb66](#): > > As for the globals, we already need the olddir directive defined in `logrotate.d/00-olddir` , you could just try adding `size 100k` to that as it is loaded first. this did not work, but changing it to maxsize did. ---
**Author:** ![pb66](https://community.openenergymonitor.org/user_avatar/community.openenergymonitor.org/pb66/32/27_2.png) [@pb66](https://community.openenergymonitor.org/u/pb66)\ **Post date:** [31 May 2019 09:55 UTC](https://community.openenergymonitor.org/t/emonsd-next-steps-filesystem-logrotate/10693/175 "2019-05-31T09:55:30Z")
Regards the “error.log” type clashes. We need to consider the potential for other logs to clash if the user installs other software, packages are not designed with our configuration in mind so it is entirely possible for other clashes to crop up unless we specifically define olddir for each subfolder in /var/log(.old), then only the remaining logs would get thrown in together. The existing clash might be alleviated by using a custom log path for emon-apache logs since we are considering using a dedicated vhost, a dedicated log path for that vhost is normal practice and the vhost.conf is the right place to define that path. This wouldn’t totally remove apache/error.log as any non-host specific errors would still go to the original file. Another option is that we could override one or the others drop-in, eg if we added a logrotate.d/0apache.conf drop-in it would get read before the default logrotate/apache.conf, as would adding it to the other custom drop-ins (eg 000-olddir), there would be a message in the logrotate.log about a second definition, but the first read would prevail without any issues being caused by the second. However if we decide to go for something that requires a custom.conf or replacement of some/most/all drop-ins, we may as well embrace it as you can’t be a little pregnant, neding any edits to those files opens the gate and we can edit anything, so as much as we want to avoid it, it would lift any restriction and potentially make life easier, hence why I was _keen_ for a custom solution. Going back to my logrotate.CUSTOM suggestion, it would remove any need for 000-olddir and ZZZ-something dropins, and also put the wildcards after the wtmp/btmp rules, but it also had 2 include lines so that (for example) a custom apache.conf drop in could be maintained in the repo without having to individually symlink it, manually edit/delete the existing drop-in or use a funky name. ```auto # the globals include /path/to/our/custom/drop-ins/ include /etc/logrotate.d/ # wtmp and btmp # wildcards ``` would load our custom apache.conf before the package maintained apache.conf, I’m not necessarily pushing for this, it would be nice not to need it, but I would rather have a robust logrotate than a tidy logrotate if we can’t have both. > [@TrystanLea](#): > > we need to use maxsize which is now supported in logrotate-3.8.1 Fantastic, I’ve just reread the man page as I was convinced _both_ minsize and maxsize were dependent on a timeperiod also being reached. The man page is less than totally clear as both the maxsize and minsize state they differ from the size option as that [size] rotates without regard for time (suggesting both minsize and maxsize do regard time). > maxsize size > Log files are rotated when they grow bigger than size bytes even before the additionally specified time interval (daily, weekly, monthly, or yearly). **The related size option is similar except that it is mutually exclusive with the time interval options, and it causes log files to be rotated without regard for the last rotation time. When maxsize is used, both the size and timestamp of a log file are considered.** > > minsize size > Log files are rotated when they grow bigger than size bytes, but not before the additionally specified time interval (daily, weekly, monthly, or yearly). **The related size option is similar except that it is mutually exclusive with the time interval options, and it causes log files to be rotated without regard for the last rotation time. When minsize is used, both the size and timestamp of a log file are considered.** The key bit of info is the “even before” and that last line should read “When maxsize is used, both the size and timestamp of a log file are considered **independently** ” because whilst the man page isn’t wrong as such, the maxsize option doesn’t actually consider time, it just doesn’t block interval rotations as the size option apparently does. Which, by the way isn’t documented either > size size > Log files are rotated only if they grow bigger than size bytes. If size is followed by k, the size is assumed to be in kilobytes. If the M is used, the size is in megabytes, and if G is used, the size is in gigabytes. So size 100, size 100k, size 100M and size 100G are all valid. Explicitly stating this disables interval rotations would help a great deal there. Glad you got it sorted. ---
**Author:** ![TrystanLea](https://community.openenergymonitor.org/user_avatar/community.openenergymonitor.org/trystanlea/32/31_2.png) [@TrystanLea](https://community.openenergymonitor.org/u/TrystanLea)\ **Post date:** [31 May 2019 10:43 UTC](https://community.openenergymonitor.org/t/emonsd-next-steps-filesystem-logrotate/10693/176 "2019-05-31T10:43:23Z")
> [@pb66](#): > > Apache uses it’s “reload” (aka graceful restart) to release old log file handles and mosquitto seems to respond to a HUP signal by releasing the file handles without a restart, as it stands I think you make a good point here that many of these other logs have additional measures in place to improve the handling of log rotation and it probably not ideal to overwrite all of that with option 1 above. Now that maxsize appears to be working, Im much more inclined to towards seeing how far we can go with an as standard as possible logrotate configuration. I’ve changed logrotate.conf back to standard on my test system (built using install script, uses log2ram and 00\_olddir config) I added another defaults config file to logrotate.d called 00\_defaults containing just: ``` maxsize 100k ``` and added logrotate.d/emonhub with: ``` /var/log/emonhub/emonhub.log { norenamecopy copytruncate su root root } ``` norenamecopy is needed to overwrite the global renamecopy setting in 00\_olddir. copytruncate as above and su root root seems to be a requirement given user emonhub ownership of the emonhub logfile and logfolder (it gives a permissions error without this). That works nicely, rotating the logs if they are above 100k when logrotate is ran, emonhub.log rotates with the rotation moved to /var/log.old. The other logs rotate with renamecopy, great! * * * > [@pb66](#): > > The above **might** allow us to have a stock logrotate.conf but the main things that currently work against that idea is that all the non-emon rotated log files will be dumped in a single folder, so /var/log/mysql/error.log and /var/log/apache2/error.log (for example) will clash as both will want to be just /var/log/error.log.1 after rotation That sounds like a good option. > [@pb66](#): > > So another possible solution in the immediate short term might be to (reduce the in-built rotation to 3M and) only rotate the emonhub.log.1 file, yes it means there will be 2 emonhub.log files in RAM, but it also means that there is no complication with a) the file handle as emonhub will rotate it’s own log (in ram) so we can avoid copytruncate and b) the potential of finding and having to handle 2 emonhub.log files at logrotate time is removed too. I see what your saying, this sounds quite complex to solve, Il try and get my head around your suggestions ---
**Author:** ![TrystanLea](https://community.openenergymonitor.org/user_avatar/community.openenergymonitor.org/trystanlea/32/31_2.png) [@TrystanLea](https://community.openenergymonitor.org/u/TrystanLea)\ **Post date:** [31 May 2019 11:41 UTC](https://community.openenergymonitor.org/t/emonsd-next-steps-filesystem-logrotate/10693/177 "2019-05-31T11:41:58Z")
> [@pb66](#): > > The specific mention of olddir in the emonhub (and emoncms?) drop-in is so that at least our main logs are segregated, unless we edit/replace each drop-in, all the rotated files will land in the same folder. good idea. > [@pb66](#): > > the only way to avoid this AFAIA is to add the olddir path in the drop-ins as I have with the emonhub one above less than ideal as you say. Ideally logrotate would handle replicating the /var/log folder structure in the /var/log.old folder without requiring a whole load of drop ins . > [@pb66](#): > > The existing clash might be alleviated by using a custom log path for emon-apache logs since we are considering using a dedicated vhost, that’s a good idea. Can we place an override perhaps for the non virtual host error log…? > [@pb66](#): > > the logrotate on a non-sdcard install should be different yes good point. ---
**Author:** ![TrystanLea](https://community.openenergymonitor.org/user_avatar/community.openenergymonitor.org/trystanlea/32/31_2.png) [@TrystanLea](https://community.openenergymonitor.org/u/TrystanLea)\ **Post date:** [31 May 2019 14:09 UTC](https://community.openenergymonitor.org/t/emonsd-next-steps-filesystem-logrotate/10693/178 "2019-05-31T14:09:57Z")
Looking through the logs that are currently present it seems that mysql/error.log and apache2/error.log are the only existing conflicts. Its possible to change the log name for apache2 for both the virtual host as you suggested @pb66 and the default error log in the sites-available emoncms.conf file, testing with the following works well: ``` ServerAdmin webmaster@localhost DocumentRoot /var/www/emoncms # Virtual Host specific error log ErrorLog ${APACHE_LOG_DIR}/apache2-emoncms.log # Access log disabled # CustomLog ${APACHE_LOG_DIR}/emoncms-access.log combined Options FollowSymLinks AllowOverride All DirectoryIndex index.php Order allow,deny Allow from all # Default apache2 error log ErrorLog ${APACHE_LOG_DIR}/apache2.log ``` I think this is a good solution that addresses the known conflict with minimal complication. While there is a risk a user might install a package with a log that conflicts with another, the drop in logrotate config option still requires an awareness of this issue when new packages are added. Is it better to add the dropin logrotate overrides or rename the logs assuming that’s possible…? ---
**Author:** ![pb66](https://community.openenergymonitor.org/user_avatar/community.openenergymonitor.org/pb66/32/27_2.png) [@pb66](https://community.openenergymonitor.org/u/pb66)\ **Post date:** [31 May 2019 17:18 UTC](https://community.openenergymonitor.org/t/emonsd-next-steps-filesystem-logrotate/10693/179 "2019-05-31T17:18:25Z")
> [@TrystanLea](#): > > # Virtual Host specific error log > ErrorLog ${APACHE\_LOG\_DIR}/apache2-emoncms.log I tend to use `${APACHE_LOG_DIR}/-error.log` as we can see it’s apache by the subfolder and it will show up with `ls -la /var/log/*/*` along with other instance specific emoncms.logs. However, this doesn’t work with the olddir as \-error.log will not be in an apache subfolder after rotating out of ram. Perhaps ```auto # Virtual Host specific error log ErrorLog /var/log//apache2-error.log ``` might be another option? That way it will be rotated with the emoncms log files into an \ subfolder and we can set our own logrotate rules for the bulk of the apache.logs in the emoncms logrotate drop-in. > [@TrystanLea](#): > > # Access log disabled > # CustomLog ${APACHE\_LOG\_DIR}/emoncms-access.log combined Not sure I agree with this, I see where you’re coming from but again if that was ```auto CustomLog /var/log//apache2-access.log combined ``` It could be just discarded or truncated on rotation so that it isn’t persisted if you feel that strongly about not retaining those logs, but it’s useful debugging info. > [@TrystanLea](#): > > # Default apache2 error log > ErrorLog ${APACHE\_LOG\_DIR}/apache2.log This feels a little like splitting hairs but this isn’t really right. The renaming is fine, although I think it should be apache2-error.log just to be clear what it actually is/was. But the global apache log shouldn’t be changed in the vhost as things could get funky when other vhosts are added (with conflicting globals?) and/or if the site is disabled using a2dissite, the log file location changes back. It should probably be done in a drop-in to /etc/apache2/conf-available (eg rename-logs.conf or similar, avoid using emoncms.conf as it is server-wide not instance level), perhaps if you used a conf.d drop-in you could also add a global `ServerName localhost` to avoid those “unable to determine server name” error messages that crop up. There should probably be an instance specific servername defined in the vhost too, this will help when users want to enable SSL via letsencrypt, the certbot installer is soooo easy to use when the vhost is set up correctly, but you need a degree in networking to do it when it isn’t quite right, so it is definately worth trying to set up letsencrypt via certbot as a test before committing to any vhost, actually, that might be very important to do as certbot may even edit the vhost file, which may then block emoncms git pulls? I think there is an option for it create a seperate \-ssl.conf vhost file based on the existing vhost and enable it, that is probably the better route. You might even (eventually) want to consider adding SSL to the installer, if the vhost configuration is good, then it should be easy to set up SSL by just providing a url that is active and resolves to the server (ie the user needs to set up any port-forwarding or dynamic dns first, duckdns and letsencrypt are a great combo). But at the moment, just laying the foundations and loosely documenting setting up SSL would be a big step. ---
**Author:** ![djh](https://community.openenergymonitor.org/letter_avatar_proxy/v4/letter/d/e36b37/32.png) [@djh](https://community.openenergymonitor.org/u/djh)\ **Post date:** [31 May 2019 20:19 UTC](https://community.openenergymonitor.org/t/emonsd-next-steps-filesystem-logrotate/10693/180 "2019-05-31T20:19:56Z")
Sorry, I haven’t read much of this thread at all but I noticed the bit about emonhub being an outlier. So apologies if what follows is absolute garbage. emonhub seems to use python’s logging facilities. Specifically a RotatingFileHandler. That seems to do it’s own rotation and indeed that seems to be what happens on my emonpi ``` /var/log/emonhub: total 7748 -rw-r--r-- 1 emonhub emonhub 2810792 May 27 02:30 emonhub.log -rw-r--r-- 1 emonhub emonhub 5119934 May 19 08:56 emonhub.log.1 ``` So that behaviour seems to be fine AFAICT, irrespective of what logrotate is doing. As to why logrotate config is having no effect, maybe the python logger keeps the file open? AFAIK logrotate relies on the traditional behaviour where programs write an entry to the log file then close it. It’s slower but simple and more reliable for writing things where you want to be sure of seeing everything. So logrotate can change things around whilst the files are closed and the various programs will pick up the new situation next time they write to the log. I think apache for example is another special that needs to do a graceful restart for proper log rotation. But if python is keeping the log file open, then it won’t notice whatever logrotate does. I’m not sure I’ve read emonhub properly and I don’t fully understand what python does, but maybe that explains why emonhub doesn’t respond to logrotate. But either way, it’s not a problem on my system. [Previous page](https://community.openenergymonitor.org/t/emonsd-next-steps-filesystem-logrotate/10693.md?page=8) [Next page](https://community.openenergymonitor.org/t/emonsd-next-steps-filesystem-logrotate/10693.md?page=10)