yes I thought that was where we had arrived at. That we where not going to modify or replace /etc/logrotate.conf and not going to modify any of the package configurations e.g /etc/logrotate.d/apache2
that would leave us with only needing to add a logrotate.d configuration for emonhub.log and emoncms.log
Interesting suggestion, Id be happy with that. So /var/log/emon/emonhub.log and /var/log/emon/emoncms.log, /var/log/emon/sync.log? rather than further subfolders?
My thought is separate logs have their own subfolder but I’m not that bothered TBH.
That is the aim, yes 
But that is a different issue external to what we’re doing here. It is not because of persisting the logs that the duplication occurs, the persisted log is not a duplicate of the non-persisted log, only one or the other exists. If that log (persisted or not) is duplicated to syslog that is something else entirely.
One might argue that IF we persist the journal, we are writing the same info twice to disk (if it’s all duplicated in syslog) and that should be avoided. BUT if that is the case, how can you say the service log data is not persisted after a reboot if syslog Is being persisted? We simply need to look in syslog rather than journald? surely.
My point is IF we need/want to persist the data held in journald (again?) it can be a simple case of adding a journal folder, it does not cause further duplication, we do not need to be extracting stuff and putting it somewhere else. BUT regardless of whether we do or do not go that route, it doesn’t in anyway effect how we implement L2R or logrotate.
Oh yes absolutely 
I think we are violently agreeing.
If we leave journald alone, let it forward everything to rsyslog, L2R will persist all logs so the details are saved within syslog (or more likely daemon.log) for our services.
However, in order to easily filter out what we are not interested in (if debugging), by using a set
of rsyslog configuration files, we could easily pull the relevant entries out of syslog (so no duplication) and put them in our own services log file.
Then when debugging, we want the contents of the running log (e.g. emonhub.log) and the emon-services.log (new name) which has any other output from the services. With a logrotate config file for that log, it is then rotated and persisted by L2R with no further interaction - it just happens.
That is true of non-emonsd installs yes!
We have mentioned not using copytruncate, how will you remove that without editing ?
We have mentioned rotating by size and that all softwares should not be treated equally, we cannot cap emonhub at 1M just because we have so many other logs, setting a size for each offers greater flexibility.
The olddir is currently set for a global "log,old` folder, so all logs will get dumped in the one folder. If you want to preserve the folder structure you can define olddir in each conf so that olddir for apache2 (for example) is /var/log.old/apache2. Yes we could move the files afterwards instead of changing olddir, but that would be in a post-rotation so still need to edit the conf files.
Some logs eg apache access may not need retaining so those logs could be deleted or truncated so that recent access is held in ram but not kept on disk.
The limits and settings for emonsd may not be ideal for a hdd server, so different versions maybe better.
If you commit to NOT editing any existing files, then you are stuck with never editing any conf’s as you cannot really start “tinkering” with sed etc in the update scripts. A second copy in a repo allows editing via updates without any “tinkering”.
If there was anyway to just stick with the (untouched) defaults I would jump at it, that was the original intention for the 00_olddir conf file, but the more I look at logrotate, the more I think that a custom system wide conf is needed.
This is nothing to do with L2R really. All L2R is actually doing is maintaining the ram folder structure and keeping the tmpfs sync’d so as not to lose logs.
logrotate is the focus for managing what comes out of ram and what gets kept. logrotate maybe writing to disk more than L2R does, depending on setup. It’s in the logrotation config that you manage and fine tune how well the tmpfs and persistence works, and the defaults were not designed for either SDcard or tmpfs logs, I would be very surprised if we couldn’t improve things via the logrotate conf. This is the main conf area for the emonsd logging.
It’s currently just under 2GB of data on a 8GB card.
Though you are correct, the data seems to be on a 4GB partition, with th OS on a separate partition.
I already use a USB stick plugged into the Pi to create the backup files that go off to DropBox as there isn’t enough space to create more than one backup and the process does a lot of writing.
Just to picka logrotation.conf at random, look at rsyslog
/var/log/syslog
{
rotate 7
daily
missingok
notifempty
delaycompress
compress
postrotate
invoke-rc.d rsyslog rotate > /dev/null
endscript
}
/var/log/mail.info
/var/log/mail.warn
/var/log/mail.err
/var/log/mail.log
/var/log/daemon.log
/var/log/kern.log
/var/log/auth.log
/var/log/user.log
/var/log/lpr.log
/var/log/cron.log
/var/log/debug
/var/log/messages
{
rotate 4
weekly
missingok
notifempty
compress
delaycompress
sharedscripts
postrotate
invoke-rc.d rsyslog rotate > /dev/null
endscript
}
are we ok with daily unlimited by size rotations to syslog which we have seen grow to 25M in a day? likewise daemon.log is weekly uncapped by size.
We can add a global size, but does “one size fit all”?
In contrast, the monit conf is
/var/log/monit.log {
rotate 4
weekly
minsize 1M
missingok
create 640 root adm
notifempty
compress
delaycompress
postrotate
invoke-rc.d monit reload > /dev/null
endscript
}
That minsize means it could hog 1M of ram even when it is never called for long periods of time, for the log to get rotated and release that 1M it must satisfy both the >1M and >one week limits so a runaway monit log (don’t know if that’s a thing or not? just using it as a “what if?” ) might not get rotated for up to a week.
I think generally daily is better than weekly and a (max) size per software should be set, plus olddir as a minimum. Don’t use maxsize as that too requires both the time and size limits to be met before rotating. size is the one we want, it ignores the time limits if the size is reached.
look at mosquitto
/var/log/mosquitto/mosquitto.log {
rotate 7
daily
compress
size 100k
nocreate
missingok
postrotate
/usr/bin/killall -HUP mosquitto
endscript
}
daily or 100K, that gives us neat daily logs unless over 100K, thus keeping RAM usage low. This maybe a tad too aggressive for some debugging, but it shows the contrast. If we just “stick to the package defaults” we are allowing the various package maintainers free reign to use our valuable ram space without control. (Unless we just add a global size 1M and have all small logs. But beware because even then if a package was added that had size 50M (unlikely I know) it could happilly use all the ram without rotating as each packages conf overrides the globals (as far as I can make out) and you cannot add a second conf eg logrotate.d/apache2.1 logrotate accepts the first for any given log file and ignores any others.
I have loads of thoughts on this so don’t think I’m sold on any particular approach yet, I just accept we cannot do nothing!
How about this!
We have our own “main” /emon/somerepo/logrotate.conf file that we can call in our logrotate commands and in that file we set some global settings eg a size 1M (or even 100K?) to limit all logs that are not specifically defined eg emonhub would be size 3M (overriding the 100K).
In that custom conf file we could also have
include /emon/somerepo/logrotate.d/
include /etc/logrotate.d/
that would first include any custom logrotation files that we deemed necessary to over ride with custom conf files maintained in an OEM repo. Then it would include any standard default conf files, the “overridden” conf files will be ignored as they are already loaded by the line above, just the remaining confs would be used. If /emon/somerepo/logrotate.d/ happens to be empty, then all the default “per package” confs are used as per your above expectation. But the globals in the “main” custom logrotate.conf will remain in force.
@TrystanLea Going back to copytruncate question, before writing to the emoncms.log, is a check made to see if the file exists?
I think that copytruncate is for when the log writer cannot (or does not) check for the existence of the log file before writing. Without that directive, logrotate moves (renames) the old log file in place initially (instantaneous) and expects the log writer to create a new one if it finds it is missing (or is instructed to create a new file). Thus the writer can only write to an available file. AIUI, copytruncate literally copies all the data then truncates the file so continues to use the original inode. renamecopy does the same (uses same inode).
If you look at the syslog file before and after rotation you will see the original inode now points to the syslog.1 file and syslog has a new inode.
No I do not agree; leave standard files and base setups alone. You can override any predefined directives with your 00 L2R conf file in the logrotate.d folder. Remember this install is not just for SD Cards. In theory, you can drop in a conf file for emoncms only (no L2R or emonhub) and it will just work.
You could add in a dropin file that has an include statement pointing to include /emon/somerepo/logrotate.d/ but I do not see any advantage.
Packages copy their own conf files to logrotate.d on install so I think we should do the same and not reinvent the wheel.
Looking at DietPi I’m not sure what it has done with the main logrotate conf file (they are allowed to as the OS owner) but what I can see are standard logrotate conf files within the logrotate.d folder that obviously get included by logrotate when called.
No surprise there!
Exactly what I said!!!
You can, but you cannot head off any configs in the per package conf’s that will take priority over globals. So you are setting vague limits that might be adhered to or more likely over ridden by each package.
No, exactly the opposite. Using L2R and a custom logrotation tailored to the use of tmpfs and minimising writes is specific ONLY to emonSD (or at least all sd based installs).
If you are not using sdcard, you don’t need logs in ram, you don’t need L2R and you don’t need a more specific logrotation config.
Correct! But it should be designed for “normal” server usage on a hdd rather than sdcard. The sdcard specific settings will be part of the low-write optimisations, not the general emoncms repo, likewise for emonhub too.
Indeed you could, the advantage of this (over totally stock) is that we can “drop in” better conf files that do not rotate weekly (overridding a possible daily global?) without disturbing the stock setup.
Exactly that, on a non-emonsd setup yes! But the std packaged confs are no good for rotating tmpfs logs.
No-one is reinventing the wheel, but the wheels off a non-sdcard based package are not necessarily the best fit for our tmpfs logging, or at least the tyres need changing for the different terrain.
We could bundle all our changes into a file like 00_olddir BUT, as I’ve explained, that would be too ridged, users cannot add thier own emonsd friendly logrotations within our modified config, or
perhaps a “ZZ_somethingorother” conf file is also required so that we can have closing catch alls?
The default daily logrotation cron is disabled or deleted so the command needs rewriting anyway, might as well be to a custom file rather than trying to manipulate (without editing) the stock logrotate.conf. I can see no problem with using a custom “main” file and with the include /etc/logrotate.d/ it will still use the original confs, unless we decide otherwise.
As are we as the OS modifier’s! Granted, if we don’t make changes to the OS (no logs in ram etc) we should stick to the defaults!
How is that obvious?
logrotate must be called with a path to the “main” config file, in that file you can include all the files in a chosen directory. When logrotate reads the config, it’s just a long string of text, the beginning and the end of that text is the “main” conf and in the middle you have the contents of the “include” directory, if the “main” conf doesn’t point logrotate to that directory, It is none the wiser, the files do not get used. There is no “obviously” about it.
If dietpi doesn’t call logrotate with a conf file path, it won’t run, if the file on that path doesn’t include the logrotate.d folder, it won’t include any of those package maintained files. From what I can see (from a very brief look, so forgive me if I’m wrong) dietpi doesn’t use logrotate when using a tmpfs ram. It looks like that directory is just somewhere for the packages to dump their config files just to be ignored, sound familiar?
@TrystanLea - I stumbled accross this
You’ll notice a ghastly thing! We need to have a
rotatedirective, because otherwise, logrotate will delete our old log files! Imagine my terror when I momentarily forgot that the-doption prevents logrotate from actually taking action, thereby removing my log files from previous days!
Could this be why you lost your logs?
I can see your point @pb66 re potential for modules specific logrotate config potentially filling our limited ram space for log2ram. Looking at the size of the other logfiles, now that we have sorted the reason for the large daemon and syslog logs (redirecting emonhub + emoncms scripts earlier on), and with the apache2 access logs silenced, the other logs are all relatively slowly increasing in size. Looking at their logrotate settings:
daily: apache2, mosquitto, mysql, log2ram, syslog
weekly: redis, all other rsyslog, ufw
monthly: apt, aptitude, dpkg
ufw could see a lot of activity. most are likely to rotate in time…
In theory enforcing a global rotation size limit sounds good to me (specific for emonSD)
log2ram is currently only installed if you have install_emonsd=true here https://github.com/openenergymonitor/emonpi/blob/master/install/config.ini#L69.
I can see that that would work, you are suggesting changing /etc/cron.daily/logrotate:
#!/bin/sh
test -x /usr/sbin/logrotate || exit 0
/usr/sbin/logrotate /etc/custom_logrotate.conf <- custom config file..
I think it was something with the unmodified rsync command and your branch of log2ram, you mentioned you where not using rsync before? your rsync_mods branch works fine though.
I’ve run a emonSD build using the install script, with the rsync_mods branch of log2ram and emonhub logfile change alongside the issues you flagged in the emonpi issue list @borpin. A couple of minor issues fixed and it all completed successfully.
My draft readme for the install process is available in the folder here:
https://github.com/openenergymonitor/emonpi/tree/master/install
I havent automated the data partition creation, Im not sure how to do that. It needs a reboot to continue etc. The ufw rules also need to be applied after a reboot.
On the todo list for the first release:
Yes, but to something more like
##!/bin/sh
#test -x /usr/sbin/logrotate || exit 0
#/usr/sbin/logrotate /etc/custom_logrotate.conf
# logrotate now triggered by log2ram
# see /etc/cron.hourly/log2ram
and /etc/cron.hourly/log2ram would have something like
#!/usr/bin/env sh
# First we rotate using OEM specific config
test -x /usr/sbin/logrotate || exit 0
/usr/sbin/logrotate /emon/somerepo/logrotate.conf
# then we sync /var/log to disk
log2ram write
I have deliberately pointed the command NOT to /etc/logrotate.conf so that it can stay there unadulterated, and the file we are using is easily updated via git. BUT we can use a symlink instead eg
#!/usr/bin/env sh
# First we rotate using OEM specific config
test -x /usr/sbin/logrotate || exit 0
/usr/sbin/logrotate /etc/custom_logrotate.conf
# then we sync /var/log to disk
log2ram write
where /etc/custom_logrotate.conf is just a symlink to /emon/somerepo/logrotate.conf instead. There should also be a symlink to a README to explain the set up to. Shame logrotate doesn’t use a sub-folder. The naming will be very important so that the files show up together. ie
pi@raspberrypi:~ $ ls /etc/logrotate*
/etc/logrotate.conf
/etc/logrotate.d:
00_olddir apt aptitude dpkg log2ram monit netdata rsyslog
should show as
pi@raspberrypi:~ $ ls /etc/logrotate*
/etc/logrotate.conf
/etc/logrotate.CUSTOM
/etc/logrotate.README
/etc/logrotate.d:
00_olddir apt aptitude dpkg log2ram monit netdata rsyslog
or somthing simiar, I just changed the file extn’s as an example, these are links not files, so the extn’s are less important, the fact they stick out and are obvious is more important, and that they get found in a search for logrotate* etc.
I quite like the /etc/logrotate.CUSTOM option now, it would also be very obvious where ever the command is called.
#!/usr/bin/env sh
# First we rotate using OEM specific config
test -x /usr/sbin/logrotate || exit 0
/usr/sbin/logrotate /etc/logrotate.CUSTOM
# then we sync /var/log to disk
log2ram write
Only a sdcard (ie low-write) install needs the data partition, so L2R needs a reboot too, you should just be able to edit fstab and install L2R and the one reboot create all the partitions.
Why is a reboot needed? Or to be done after a rebbot?
sudo ufw allow ??? ???
sudo ufw enable --force
should do it and the changes will stick, no reboot required.
IMO you should create a OEM set of “apps” at /etc/ufw/applications.d that’s what I usually do.
You can have all your ports defined incl mqtt http https ssh etc etc so you can just use the app names.
Just be sure that “SSH” or “22” is one of the ports/apps you define so as not to lock yourself out.
I need to reboot after creating the partition to then format it with the ext2 filesystem and then mount to /var/opt/emon, the install script needs to run after that so that it creates the phpfina etc directories. I think it requires two reboots before starting the automatic script part. But that is a emonSD specific filesystem setup step so Im not too concerned.
Thanks for the ufw tips.