emonSD next steps: filesystem & logrotate

Thanks @borpin , I cant seem to get your first point to work, it seems that setting size globally gets ignored and not because the services have their own definition of ‘size’. Most dont declare a size setting. It seems that the global settings are just not applied…

Edit: or perhaps its because there is no logrotate entry for the logs I was watching I will double check.

If you use the -d option, the output will tell you what will happen without is actually doing anything.

1 Like

I’ve made a little progress on above, the specific emonhub logrotate conf with standard logrotate.conf is now working. But I still cant get a global size setting to apply to other logs. I’ve gone back and tested the current global logrotate wildcard approach and I think it works really well. It seems to cover all bases. Might it be worth having two options one for those who are happy with the current approach and another for those who want a more custom approach, where existing logrotate configuratinos are upheld?

Option 1: Current logrotate config on emonSD Oct18 (comments removed to shorten):

rotate 1
size 1M
copytruncate
nomail
su root root
missingok
notifempty

/var/log/*.log /var/log/*/*.log /var/log/syslog /var/log/messages /var/log/btmp /var/log/* {}

This works well rotating all logs at 1MB, emonhub.log continues to log after rotation (changing to renamecopy does not work for emonhub.log). I think it would be worth adding compress and increasing rotate.

Option 2: Default stretch logrotate.conf (noted above) + specific emon logrotate.d entry e.g:

/var/log/emonhub/emonhub.log {
    rotate 2
    size 1M
    copytruncate
    su root root
    missingok
    notifempty
    compress
    create
}

Works great for emonhub.log, we can have additional configs for the other emoncms logs. It does not however stop other logs from getting too large.

Option 3: as option 2 but with global “size 1MB” override, I cant get this override to work, the default settings don’t seem to apply.

@borpin I think you want option 2? and @pb66 option 1 but called logrotate.CUSTOM?

please don’t use copytruncate, it’s guaranteed to loose logs at some point
because any logs that are written after the copy and before the truncate will be
lost

David Lang

Thanks @davidelang at the moment we have the issue that emonhub does not continue logging if renamecopy is used, logfile pointer still connected to old file? Im not sure if emonhub is just the outlier there…

No, definitely not!

Firstly, whilst I was keen to use the logrotate.CUSTOM file name for a custom logrotate.conf, that is only IF we have a custom logrotate.conf, at the time of that discussion, I felt it was most likely going to be necessary. But I would prefer to use the standard conf if possible, I’m just not entirely convinced we can. But I’m less hung up on whether we do or don’t use the standard conf than I am about getting it to work correctly. First we need to find the right configuration, and once we know what rules we need we can decide about how that is achieved.

Secondly, I disagree with many of the rules in the emonSD logrotate.conf, so even if we do arrive at a custom conf of my liking, it wouldn’t look like the current conf.

notifempty is pointless in conjunction with no timed rotation and size only (empty is less than 1M!)

We must have a timed rotation, not just rely on size, a 999k log file with zero activity could sit there in ram for ever, better it is rotated out after a week/month.

The rotate 1 is definitely no good.

I don’t think we need nomail or su root root.

we shouldn’t really be using compress without delaycompress if wildcards are being used, if a service is still writing to the rotated file when it is compressed it might crash that service, delaycompress only compresses some.log.2 onwards and is safer unless you know it’s ok to compress the new rotation.

The renamecopy directive is something I brought up to be used in conjunction with the olddir directive so that files can be moved out of RAM. By default olddir can only be used on the same device (ie the rotations could go to another folder, but still in RAM) Either renamecopy, copy or copytruncate can be used to facilitate moving files across devices using olddir.

The copytruncate on the emonSD was necessary because the include drop-ins were disabled so apache2, mosquitto and rsyslog were unable to trigger the post-rotate scripts that made the services let go of the old log handle and start logging to the new file, this isn’t good practice.

The loose wildcards are also a bit of a no no as you’ve discovered with the duplicate logs warnings. This could be less of an issue once we are rotating out of RAM as the rotated files will no longer be present in /var/log.

It is indeed better to avoid using copytruncate if possible, but it is far better to potentially loose a log message or 2, than loose actual data. copytruncate is a low quality rotate method to be used when it can’t be avoided.

Is that a typo? AFAIK the rule should be size 1M (not MB) or better still size 100k, 1M is way too big for some logs, small size is both less space used in ram and less for log2ram (rsync) to calculate each hour.

Regarding emonhub and copytruncate, I wasn’t aware that emonhub clung onto the old file handle, it makes sense now I think about it, but this has been masked for years by the fact that emonSD used copytruncate across the board.

Apache uses it’s “reload” (aka graceful restart) to release old log file handles and mosquitto seems to respond to a HUP signal by releasing the file handles without a restart, as it stands, emonhub doesn’t yet have that specific facility, I’m not great with “signals” so I’m not 100% sure, but I believe any signal able to make emonhub release the logfile handles will cause it to restart and we definitely do not want that to happen, data in transit could be lost and buffered data too. In time maybe a “reload” could be added to emonhub, not previously needed as it loads any new settings whilst running. So in the short term copytruncate maybe necessary for emonhub.

However! I’ve previously also mentioned emonhub’s “5M self-rotation” kicking in if the logs run away and that if there is a emonhub.log.1 present, this is something that needs handling in the logrotation definition, either in the pre or post rotation. So another possible solution in the immediate short term might be to (reduce the in-built rotation to 3M and) only rotate the emonhub.log.1 file, yes it means there will be 2 emonhub.log files in RAM, but it also means that there is no complication with a) the file handle as emonhub will rotate it’s own log (in ram) so we can avoid copytruncate and b) the potential of finding and having to handle 2 emonhub.log files at logrotate time is removed too.

so maybe something along the lines of

/var/log/emonhub/emonhub.log {
# Dummy definition to ensure emonhub.log is NOT rotated
# only rotate if over 10M (not possible with inbuilt rotation) 
size 10M
}

/var/log/emonhub/emonhub.log.1 {
    daily
    rotate 28
    size 3M
    compress
    delaycompress
    create 775 emonhub emonhub
    missingok
    notifempty

    # rotate emonhub log files to an emonhub folder in log.old
    olddir /var/log.old/emonhub
    createolddir 775 root emonhub
    renamecopy
    
    sharedscripts
    postrotate
        # possibly some concatenation into daily files
    endscript
}

The explicit definition for emonhub.log would prevent a wildcard picking it up, I haven’t tested this though.

The specific mention of olddir in the emonhub (and emoncms?) drop-in is so that at least our main logs are segregated, unless we edit/replace each drop-in, all the rotated files will land in the same folder.

Some of the finer points such as length of retention and whether logs should be concatenated can be played with, the main point is to get it working well, whether we have a week or a years worth of data at this point is irrelevant unless the sdcard fills up.

As for the globals, we already need the olddir directive defined in logrotate.d/00-olddir, you could just try adding size 100k to that as it is loaded first. But any wildcards will need to go in a file prefixed with “ZZZ-something” so it is loaded last, but even then it may cause conflicts with the wtmp and btmp definitions in the stock config and we do need some form of wildcard I imagine as there are several logs that slip through the net.

The above might allow us to have a stock logrotate.conf but the main things that currently work against that idea is that all the non-emon rotated log files will be dumped in a single folder, so /var/log/mysql/error.log and /var/log/apache2/error.log (for example) will clash as both will want to be just /var/log/error.log.1 after rotation, the only way to avoid this AFAIA is to add the olddir path in the drop-ins as I have with the emonhub one above. Plus the wildcard cannot be placed after the 2 definitions included in the stock conf and TBH the “000” and “ZZZ” prefixes look a tad amateurish and are not very infallible, a user adds “0-something” it will slip in first and likewise with “ZZZZ-something” slipping in after the wildcards.

Plus, something esle we need to factor in is that not all installs will be using L2R/tmpfs logs, so the logrotate on a non-sdcard install should be different, the olddir stuff won’t be needed, the size values won’t be needed, the wildcards probably won’t be needed.

Are you sure it has stopped logging? It should still be logging to the original file, even if it has moved or changed name (ie is it logging to emonhub.log.1?) as the file descriptor will still be intact, unless the file is compressed or deleted (incl moving across devices).

Thanks @pb66 for the detailed response, I will go through it today.

I’ve finally solved the default size setting being ignored problem I was having, these SE threads had the answer: logrotate daily and size? - Server Fault & configuration - How to rotate log based on an interval unless log exceeds a certain size? - Server Fault we need to use maxsize which is now supported in logrotate-3.8.1 (stretch appears to have logrotate 3.11.0 installed)

From the SE thread:

As of logrotate 3.8.1, maxsize and timeperiod are supported together, which would be the ideal solution. See the answer to this post: How to rotate log based on an interval unless log exceeds a certain size?

Just to clarify, I tried:

this did not work, but changing it to maxsize did.

1 Like

Regards the “error.log” type clashes. We need to consider the potential for other logs to clash if the user installs other software, packages are not designed with our configuration in mind so it is entirely possible for other clashes to crop up unless we specifically define olddir for each subfolder in /var/log(.old), then only the remaining logs would get thrown in together.

The existing clash might be alleviated by using a custom log path for emon-apache logs since we are considering using a dedicated vhost, a dedicated log path for that vhost is normal practice and the vhost.conf is the right place to define that path. This wouldn’t totally remove apache/error.log as any non-host specific errors would still go to the original file.

Another option is that we could override one or the others drop-in, eg if we added a logrotate.d/0apache.conf drop-in it would get read before the default logrotate/apache.conf, as would adding it to the other custom drop-ins (eg 000-olddir), there would be a message in the logrotate.log about a second definition, but the first read would prevail without any issues being caused by the second.

However if we decide to go for something that requires a custom.conf or replacement of some/most/all drop-ins, we may as well embrace it as you can’t be a little pregnant, neding any edits to those files opens the gate and we can edit anything, so as much as we want to avoid it, it would lift any restriction and potentially make life easier, hence why I was keen for a custom solution.

Going back to my logrotate.CUSTOM suggestion, it would remove any need for 000-olddir and ZZZ-something dropins, and also put the wildcards after the wtmp/btmp rules, but it also had 2 include lines so that (for example) a custom apache.conf drop in could be maintained in the repo without having to individually symlink it, manually edit/delete the existing drop-in or use a funky name.

# the globals

include /path/to/our/custom/drop-ins/
include /etc/logrotate.d/

# wtmp and btmp

# wildcards

would load our custom apache.conf before the package maintained apache.conf, I’m not necessarily pushing for this, it would be nice not to need it, but I would rather have a robust logrotate than a tidy logrotate if we can’t have both.

Fantastic, I’ve just reread the man page as I was convinced both minsize and maxsize were dependent on a timeperiod also being reached. The man page is less than totally clear as both the maxsize and minsize state they differ from the size option as that [size] rotates without regard for time (suggesting both minsize and maxsize do regard time).

maxsize size
Log files are rotated when they grow bigger than size bytes even before the additionally specified time interval (daily, weekly, monthly, or yearly). The related size option is similar except that it is mutually exclusive with the time interval options, and it causes log files to be rotated without regard for the last rotation time. When maxsize is used, both the size and timestamp of a log file are considered.

minsize size
Log files are rotated when they grow bigger than size bytes, but not before the additionally specified time interval (daily, weekly, monthly, or yearly). The related size option is similar except that it is mutually exclusive with the time interval options, and it causes log files to be rotated without regard for the last rotation time. When minsize is used, both the size and timestamp of a log file are considered.

The key bit of info is the “even before” and that last line should read “When maxsize is used, both the size and timestamp of a log file are considered independently” because whilst the man page isn’t wrong as such, the maxsize option doesn’t actually consider time, it just doesn’t block interval rotations as the size option apparently does. Which, by the way isn’t documented either

size size
Log files are rotated only if they grow bigger than size bytes. If size is followed by k, the size is assumed to be in kilobytes. If the M is used, the size is in megabytes, and if G is used, the size is in gigabytes. So size 100, size 100k, size 100M and size 100G are all valid.

Explicitly stating this disables interval rotations would help a great deal there. Glad you got it sorted.

1 Like

I think you make a good point here that many of these other logs have additional measures in place to improve the handling of log rotation and it probably not ideal to overwrite all of that with option 1 above.

Now that maxsize appears to be working, Im much more inclined to towards seeing how far we can go with an as standard as possible logrotate configuration. I’ve changed logrotate.conf back to standard on my test system (built using install script, uses log2ram and 00_olddir config)

I added another defaults config file to logrotate.d called 00_defaults containing just:

maxsize 100k

and added logrotate.d/emonhub with:

/var/log/emonhub/emonhub.log {
    norenamecopy
    copytruncate
    su root root
}

norenamecopy is needed to overwrite the global renamecopy setting in 00_olddir. copytruncate as above and su root root seems to be a requirement given user emonhub ownership of the emonhub logfile and logfolder (it gives a permissions error without this).

That works nicely, rotating the logs if they are above 100k when logrotate is ran, emonhub.log rotates with the rotation moved to /var/log.old. The other logs rotate with renamecopy, great!


That sounds like a good option.

I see what your saying, this sounds quite complex to solve, Il try and get my head around your suggestions

good idea.

less than ideal as you say. Ideally logrotate would handle replicating the /var/log folder structure in the /var/log.old folder without requiring a whole load of drop ins .

that’s a good idea. Can we place an override perhaps for the non virtual host error log…?

yes good point.

Looking through the logs that are currently present it seems that mysql/error.log and apache2/error.log are the only existing conflicts. Its possible to change the log name for apache2 for both the virtual host as you suggested @pb66 and the default error log in the sites-available emoncms.conf file, testing with the following works well:

<VirtualHost *:80>
    ServerAdmin webmaster@localhost
    DocumentRoot /var/www/emoncms

    # Virtual Host specific error log
    ErrorLog ${APACHE_LOG_DIR}/apache2-emoncms.log
    # Access log disabled
    # CustomLog ${APACHE_LOG_DIR}/emoncms-access.log combined

    <Directory /var/www/emoncms>
        Options FollowSymLinks
        AllowOverride All
        DirectoryIndex index.php
        Order allow,deny
        Allow from all
    </Directory>
</VirtualHost>

# Default apache2 error log
ErrorLog ${APACHE_LOG_DIR}/apache2.log

I think this is a good solution that addresses the known conflict with minimal complication.

While there is a risk a user might install a package with a log that conflicts with another, the drop in logrotate config option still requires an awareness of this issue when new packages are added. Is it better to add the dropin logrotate overrides or rename the logs assuming that’s possible…?

I tend to use ${APACHE_LOG_DIR}/<instance>-error.log as we can see it’s apache by the subfolder and it will show up with ls -la /var/log/*/<instance>* along with other instance specific emoncms.logs. However, this doesn’t work with the olddir as <instance>-error.log will not be in an apache subfolder after rotating out of ram.

Perhaps

# Virtual Host specific error log 
ErrorLog /var/log/<instance>/apache2-error.log 

might be another option? That way it will be rotated with the emoncms log files into an <instance> subfolder and we can set our own logrotate rules for the bulk of the apache.logs in the emoncms logrotate drop-in.

Not sure I agree with this, I see where you’re coming from but again if that was

CustomLog /var/log/<instance>/apache2-access.log combined

It could be just discarded or truncated on rotation so that it isn’t persisted if you feel that strongly about not retaining those logs, but it’s useful debugging info.

This feels a little like splitting hairs but this isn’t really right. The renaming is fine, although I think it should be apache2-error.log just to be clear what it actually is/was. But the global apache log shouldn’t be changed in the vhost as things could get funky when other vhosts are added (with conflicting globals?) and/or if the site is disabled using a2dissite, the log file location changes back.

It should probably be done in a drop-in to /etc/apache2/conf-available (eg rename-logs.conf or similar, avoid using emoncms.conf as it is server-wide not instance level), perhaps if you used a conf.d drop-in you could also add a global ServerName localhost to avoid those “unable to determine server name” error messages that crop up.

There should probably be an instance specific servername defined in the vhost too, this will help when users want to enable SSL via letsencrypt, the certbot installer is soooo easy to use when the vhost is set up correctly, but you need a degree in networking to do it when it isn’t quite right, so it is definately worth trying to set up letsencrypt via certbot as a test before committing to any vhost, actually, that might be very important to do as certbot may even edit the vhost file, which may then block emoncms git pulls? I think there is an option for it create a seperate <instance>-ssl.conf vhost file based on the existing vhost and enable it, that is probably the better route.

You might even (eventually) want to consider adding SSL to the installer, if the vhost configuration is good, then it should be easy to set up SSL by just providing a url that is active and resolves to the server (ie the user needs to set up any port-forwarding or dynamic dns first, duckdns and letsencrypt are a great combo). But at the moment, just laying the foundations and loosely documenting setting up SSL would be a big step.

Sorry, I haven’t read much of this thread at all but I noticed the bit about emonhub being an outlier. So apologies if what follows is absolute garbage.

emonhub seems to use python’s logging facilities. Specifically a RotatingFileHandler. That seems to do it’s own rotation and indeed that seems to be what happens on my emonpi

/var/log/emonhub:
total 7748
-rw-r--r-- 1 emonhub emonhub 2810792 May 27 02:30 emonhub.log
-rw-r--r-- 1 emonhub emonhub 5119934 May 19 08:56 emonhub.log.1

So that behaviour seems to be fine AFAICT, irrespective of what logrotate is doing. As to why logrotate config is having no effect, maybe the python logger keeps the file open? AFAIK logrotate relies on the traditional behaviour where programs write an entry to the log file then close it. It’s slower but simple and more reliable for writing things where you want to be sure of seeing everything. So logrotate can change things around whilst the files are closed and the various programs will pick up the new situation next time they write to the log. I think apache for example is another special that needs to do a graceful restart for proper log rotation. But if python is keeping the log file open, then it won’t notice whatever logrotate does.

I’m not sure I’ve read emonhub properly and I don’t fully understand what python does, but maybe that explains why emonhub doesn’t respond to logrotate. But either way, it’s not a problem on my system.

Which logs are you concerned about? Other than the recent issues, for many years there has not been an issue. I think we are trying to fix something that does not need fixing.

But we do have timed rotation. In fact, logrotate, as setup, only triggers to test for rotation on a fixed time schedule and it will then decide on the rotation to be carried out depending on what rule has been reached (size or time). For true SIZE/MAXSIZE rotation you need monit to monitor the SIZE of the log files and trigger the rotation if necessary.

I really dislike olddir as several packages use error.log in a folder and olddir cannot distinguish between them. It is pointless in the emonhub configuration IMHO even with Log2RAM.

I currently use a post rotate command to move out the emonhub file to persistent memory (as I’ve mentioned before) - much more robust IMHO.

/var/log/emonhub/emonhub.log {
        rotate 6
        daily
        size 2M
        maxsize 2M
        missingok
        copytruncate
        notifempty
        postrotate
             mv /var/log/emonhub/emonhub.log.1 /home/pi/data/
             cat /home/pi/data/emonhub.log.1 >> /home/pi/data/emonhub.log
             rm /home/pi/data/emonhub.log.1
        endscript
}

/home/pi/data/emonhub.log {
        su pi pi
        rotate 10
        daily
        compress
        missingok
        notifempty
}

Of course it isn’t working right now as the update buggered up the logrotate setup as usual.

For /var/log

  • rotation number does not matter but just stops it growing if something goes wrong
  • daily - at the very least rotate daily
  • size/maxsize - rotate once this big
  • copytruncate - this seems to work without loss AFAICS.

For /home/pi/data/

  • keeps 10 daily logs compressed.

I’m curious why you do this instead of the simpler

cat /var/log/emonhub/emonhub.log.1 >> /home/pi/data/emonhub.log
rm /var/log/emonhub/emonhub.log.1

which would reduce the writes to the SD card noticeably, would it not?

I wondered about that too. So I ran dstat -d to monitor disk reads/writes while
running the mv and cp commands on two different large files.
(monitored the copy/move in one ssh session, ran the commands in another ssh session)
One file is ~22MB, the other, ~25MB.
The top half of the screenshot shows the result of running cp file1 /opt on the 1st file.
The bottom half shows the result of running mv file2 /opt on the 2nd file.

You may need to install dstat. (sudo apt-get install dstat)

As Brian often says, YMMV.

(you may have to click on the picture to see it in its entirety)


More info on dstat:

Hi Bill,

Sorry, I haven’t checked into dstat yet (I do have it installed apparently) so I might be wide of the mark, but you do appreciate that mv is equivalent to an ln if the two directories are on the same filesystem but is equivalent to a cp if they are on different filesystems?

i.e. it does a few small directory operations if possible and only copies a lot of data blocks if necessary.

So your test suggests to me that the original location of file2 and /opt are in the same filesystem, or that file1 and file2 are radically different in size. Is that right or did I miss something?

Indeed it is. But…

I’d forgotten that the /home/pi/data directory is on its own partition/filesystem. :sunglasses:

Correct, hence the lack of much disk activity.

No, you nailed it. :wink:

Yes it is simpler, no there is no reason not to do it that way and YMMV.

Thanks @pb66. Happy to move the instance apache2 log to /var/log/emoncms as you suggest

/var/log/emoncms/apache2-error.log
/var/log/emoncms/apache2-access.log

The global log would then be fine in the apache2 folder renamed:

/var/log/apache2/apache2-error.log

Yes happy to use conf-available, I tried it before but forgot I needed to run ‘a2enconf’, testing now works a treat, how about, emonsd.conf?

# ServerName
ServerName localhost

# Default apache2 error log
ErrorLog ${APACHE_LOG_DIR}/apache2.log

Noted your comments on SSL, while Im familiar with setting up certbot on online servers, Its not something I’ve done yet on a pi for application on the local network without a domain name but it is something we are interested in.


Thanks @borpin

Sure, the option to enable a global maxsize limit is only a single line, Id be keen to keep it in there as it provides a fallback but Id also be happy for it to be optional.

Looking at all the logs on my system it was only mysql and apache that where conflicting. I can see your suggestion is a neat solution for emonhub.log and potentially emoncms logs but without olddir we would also need post rotate commands for the other logs - that is unless we dont think the other logs are a problem - as you suggest.

Thinking through this, Lets say for example that we have ufw.log or an apache2 access.log logging with a high throughput, at rotation without olddir these would stay on the log2ram tmpfs partition, log2ram would rsync these to its persistent backup. As these files dont change after rotation, there would be nothing to rsync? causing no additional write load? But they would present processor and read load which we can avoid by moving them out of the log2ram system…

Could we alternatively create our own script to copy out the rotated logs and replicate the folder structure to avoid conflicts? this would run after logrotate and before calling log2ram write…