emonSD next steps: filesystem & logrotate

I have loads of thoughts on this so don’t think I’m sold on any particular approach yet, I just accept we cannot do nothing!

How about this!

We have our own “main” /emon/somerepo/logrotate.conf file that we can call in our logrotate commands and in that file we set some global settings eg a size 1M (or even 100K?) to limit all logs that are not specifically defined eg emonhub would be size 3M (overriding the 100K).

In that custom conf file we could also have

include /emon/somerepo/logrotate.d/
include /etc/logrotate.d/

that would first include any custom logrotation files that we deemed necessary to over ride with custom conf files maintained in an OEM repo. Then it would include any standard default conf files, the “overridden” conf files will be ignored as they are already loaded by the line above, just the remaining confs would be used. If /emon/somerepo/logrotate.d/ happens to be empty, then all the default “per package” confs are used as per your above expectation. But the globals in the “main” custom logrotate.conf will remain in force.

@TrystanLea Going back to copytruncate question, before writing to the emoncms.log, is a check made to see if the file exists?

I think that copytruncate is for when the log writer cannot (or does not) check for the existence of the log file before writing. Without that directive, logrotate moves (renames) the old log file in place initially (instantaneous) and expects the log writer to create a new one if it finds it is missing (or is instructed to create a new file). Thus the writer can only write to an available file. AIUI, copytruncate literally copies all the data then truncates the file so continues to use the original inode. renamecopy does the same (uses same inode).

If you look at the syslog file before and after rotation you will see the original inode now points to the syslog.1 file and syslog has a new inode.

1 Like

No I do not agree; leave standard files and base setups alone. You can override any predefined directives with your 00 L2R conf file in the logrotate.d folder. Remember this install is not just for SD Cards. In theory, you can drop in a conf file for emoncms only (no L2R or emonhub) and it will just work.

You could add in a dropin file that has an include statement pointing to include /emon/somerepo/logrotate.d/ but I do not see any advantage.

Packages copy their own conf files to logrotate.d on install so I think we should do the same and not reinvent the wheel.

Looking at DietPi I’m not sure what it has done with the main logrotate conf file (they are allowed to as the OS owner) but what I can see are standard logrotate conf files within the logrotate.d folder that obviously get included by logrotate when called.

No surprise there! :smile:

Exactly what I said!!!

You can, but you cannot head off any configs in the per package conf’s that will take priority over globals. So you are setting vague limits that might be adhered to or more likely over ridden by each package.

No, exactly the opposite. Using L2R and a custom logrotation tailored to the use of tmpfs and minimising writes is specific ONLY to emonSD (or at least all sd based installs).

If you are not using sdcard, you don’t need logs in ram, you don’t need L2R and you don’t need a more specific logrotation config.

Correct! But it should be designed for “normal” server usage on a hdd rather than sdcard. The sdcard specific settings will be part of the low-write optimisations, not the general emoncms repo, likewise for emonhub too.

Indeed you could, the advantage of this (over totally stock) is that we can “drop in” better conf files that do not rotate weekly (overridding a possible daily global?) without disturbing the stock setup.

Exactly that, on a non-emonsd setup yes! But the std packaged confs are no good for rotating tmpfs logs.

No-one is reinventing the wheel, but the wheels off a non-sdcard based package are not necessarily the best fit for our tmpfs logging, or at least the tyres need changing for the different terrain.

We could bundle all our changes into a file like 00_olddir BUT, as I’ve explained, that would be too ridged, users cannot add thier own emonsd friendly logrotations within our modified config, or
perhaps a “ZZ_somethingorother” conf file is also required so that we can have closing catch alls?

The default daily logrotation cron is disabled or deleted so the command needs rewriting anyway, might as well be to a custom file rather than trying to manipulate (without editing) the stock logrotate.conf. I can see no problem with using a custom “main” file and with the include /etc/logrotate.d/ it will still use the original confs, unless we decide otherwise.

As are we as the OS modifier’s! Granted, if we don’t make changes to the OS (no logs in ram etc) we should stick to the defaults!

How is that obvious?

logrotate must be called with a path to the “main” config file, in that file you can include all the files in a chosen directory. When logrotate reads the config, it’s just a long string of text, the beginning and the end of that text is the “main” conf and in the middle you have the contents of the “include” directory, if the “main” conf doesn’t point logrotate to that directory, It is none the wiser, the files do not get used. There is no “obviously” about it.

If dietpi doesn’t call logrotate with a conf file path, it won’t run, if the file on that path doesn’t include the logrotate.d folder, it won’t include any of those package maintained files. From what I can see (from a very brief look, so forgive me if I’m wrong) dietpi doesn’t use logrotate when using a tmpfs ram. It looks like that directory is just somewhere for the packages to dump their config files just to be ignored, sound familiar?

@TrystanLea - I stumbled accross this

You’ll notice a ghastly thing! We need to have a rotate directive, because otherwise, logrotate will delete our old log files! Imagine my terror when I momentarily forgot that the -d option prevents logrotate from actually taking action, thereby removing my log files from previous days!

Could this be why you lost your logs?

I can see your point @pb66 re potential for modules specific logrotate config potentially filling our limited ram space for log2ram. Looking at the size of the other logfiles, now that we have sorted the reason for the large daemon and syslog logs (redirecting emonhub + emoncms scripts earlier on), and with the apache2 access logs silenced, the other logs are all relatively slowly increasing in size. Looking at their logrotate settings:

daily: apache2, mosquitto, mysql, log2ram, syslog
weekly: redis, all other rsyslog, ufw
monthly: apt, aptitude, dpkg

ufw could see a lot of activity. most are likely to rotate in time…

In theory enforcing a global rotation size limit sounds good to me (specific for emonSD)

log2ram is currently only installed if you have install_emonsd=true here https://github.com/openenergymonitor/emonpi/blob/master/install/config.ini#L69.

I can see that that would work, you are suggesting changing /etc/cron.daily/logrotate:

#!/bin/sh
test -x /usr/sbin/logrotate || exit 0
/usr/sbin/logrotate /etc/custom_logrotate.conf <- custom config file..

I think it was something with the unmodified rsync command and your branch of log2ram, you mentioned you where not using rsync before? your rsync_mods branch works fine though.

I’ve run a emonSD build using the install script, with the rsync_mods branch of log2ram and emonhub logfile change alongside the issues you flagged in the emonpi issue list @borpin. A couple of minor issues fixed and it all completed successfully.

My draft readme for the install process is available in the folder here:
https://github.com/openenergymonitor/emonpi/tree/master/install

I havent automated the data partition creation, Im not sure how to do that. It needs a reboot to continue etc. The ufw rules also need to be applied after a reboot.

On the todo list for the first release:

  • finalise emoncms log locations i.e: /var/log/emon/emoncms/sync.log
  • update emoncms & modules with configurable log directory paths for sync/update etc.
  • finalise logrotate config
  • move install and update from emonpi repo to emonscripts or other suitable name (needs to be carefully managed in order to maintain backwards compatibility with existing emonSD based systems that will be looking for these scripts in the emonpi repo).
1 Like

Yes, but to something more like

##!/bin/sh
#test -x /usr/sbin/logrotate || exit 0
#/usr/sbin/logrotate /etc/custom_logrotate.conf

# logrotate now triggered by log2ram
# see /etc/cron.hourly/log2ram

and /etc/cron.hourly/log2ram would have something like

#!/usr/bin/env sh

# First we rotate using OEM specific config
test -x /usr/sbin/logrotate || exit 0
/usr/sbin/logrotate /emon/somerepo/logrotate.conf

# then we sync /var/log to disk
log2ram write

I have deliberately pointed the command NOT to /etc/logrotate.conf so that it can stay there unadulterated, and the file we are using is easily updated via git. BUT we can use a symlink instead eg

#!/usr/bin/env sh

# First we rotate using OEM specific config
test -x /usr/sbin/logrotate || exit 0
/usr/sbin/logrotate /etc/custom_logrotate.conf

# then we sync /var/log to disk
log2ram write

where /etc/custom_logrotate.conf is just a symlink to /emon/somerepo/logrotate.conf instead. There should also be a symlink to a README to explain the set up to. Shame logrotate doesn’t use a sub-folder. The naming will be very important so that the files show up together. ie

pi@raspberrypi:~ $ ls /etc/logrotate*
/etc/logrotate.conf

/etc/logrotate.d:
00_olddir  apt  aptitude  dpkg  log2ram  monit  netdata  rsyslog

should show as

pi@raspberrypi:~ $ ls /etc/logrotate*
/etc/logrotate.conf
/etc/logrotate.CUSTOM 
/etc/logrotate.README

/etc/logrotate.d:
00_olddir  apt  aptitude  dpkg  log2ram  monit  netdata  rsyslog

or somthing simiar, I just changed the file extn’s as an example, these are links not files, so the extn’s are less important, the fact they stick out and are obvious is more important, and that they get found in a search for logrotate* etc.

I quite like the /etc/logrotate.CUSTOM option now, it would also be very obvious where ever the command is called.

#!/usr/bin/env sh

# First we rotate using OEM specific config
test -x /usr/sbin/logrotate || exit 0
/usr/sbin/logrotate /etc/logrotate.CUSTOM

# then we sync /var/log to disk
log2ram write
1 Like

Only a sdcard (ie low-write) install needs the data partition, so L2R needs a reboot too, you should just be able to edit fstab and install L2R and the one reboot create all the partitions.

Why is a reboot needed? Or to be done after a rebbot?

sudo ufw allow ??? ??? 
sudo ufw enable --force

should do it and the changes will stick, no reboot required.

IMO you should create a OEM set of “apps” at /etc/ufw/applications.d that’s what I usually do.

You can have all your ports defined incl mqtt http https ssh etc etc so you can just use the app names.

Just be sure that “SSH” or “22” is one of the ports/apps you define so as not to lock yourself out.

1 Like

I need to reboot after creating the partition to then format it with the ext2 filesystem and then mount to /var/opt/emon, the install script needs to run after that so that it creates the phpfina etc directories. I think it requires two reboots before starting the automatic script part. But that is a emonSD specific filesystem setup step so Im not too concerned.

Thanks for the ufw tips.

Something like this (pseudo code)

if not /data exists and is sdcard   # or could test the hostname or could test for temporary.service etc
    edit fstab to add /data
    install L2R
    printf blah blah blah > temporary.service
    systemctl enable temporary.service
    edit hostname
    reboot
else
   systemctl disable temporary.service
   rm temporary.service
   format /data

continue install

all temporary.service would be is a basic service unit to restart the install script, the second run will land in the “else” and then carry on.

1 Like

In danger of going OT :laughing:, I had some issues when setting up a temp Pi outside of my firewall so there was a website accessible externally when I was playing with SSL.

Using ‘ShieldsUp’ I was surprised that the default UFW setup did not show ports as stealth but as closed (so they existed). To make a machine stealth you need

ufw default deny incoming

Now I may be wrong here and but I did test it…

Personally I then allow in anything from my local (wired) LAN (if someone is on my LAN I’m screwed anyway). This means if it is a purely internal machine you don’t need anything else. You cannot access it from a machine outside your LAN. Opening just the SSH port means you can access SSH from anywhere (other firewalls permitting).

ufw allow in on eth0 to 192.168.1.0/24 from 192.168.1.0/24

So although I can SSH into this machine which is exposed to the internet via a DMZ, when I run the test, only the WWW and WWW secure ports are shown as open.

If already enabled and a rule is changed all you should need is

sudo ufw reload

YMMV.

1 Like

I think that is the default, From the man page

On installation, ufw is disabled with a default incoming policy of deny and a default
outgoing policy of allow, with stateful tracking for NEW connections.

Intriguing. Just checked on the Latest Pi and it is set by default. Odd.

Because we have gone back to using the emonhub.log file created from the script, the logviewer no longer works but the download does :smile:.

[edit]
@pb66 - the olddir /var/log.old is not being created by the install script it appears.

I’m getting some de ja vue here. That’s right it is auto created by the first rotation. Not the install script.

1 Like

Memory of a fish me. Why not just create it at install time? First look at /var/log and it’s like, why is that link red?!

I installed emoncms onto my EmonPi with the install script. Log2Ram has installed but I did not fiddle with creating a new partition first, just left Raspbian (new image) to it.

mount looks like this, though I’m not sure I am reading it correctly - log2ram seems to be on tmpfs - not sure how!

pi@emonpi:/var/log $ sudo mount | column -t
/dev/mmcblk0p2  on  /                           type  ext4        (rw,noatime,data=ordered)
devtmpfs        on  /dev                        type  devtmpfs    (rw,relatime,size=470116k,nr_inodes=117529,mode=755)
sysfs           on  /sys                        type  sysfs       (rw,nosuid,nodev,noexec,relatime)
proc            on  /proc                       type  proc        (rw,relatime)
tmpfs           on  /dev/shm                    type  tmpfs       (rw,nosuid,nodev)
devpts          on  /dev/pts                    type  devpts      (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs           on  /run                        type  tmpfs       (rw,nosuid,nodev,mode=755)
tmpfs           on  /run/lock                   type  tmpfs       (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs           on  /sys/fs/cgroup              type  tmpfs       (ro,nosuid,nodev,noexec,mode=755)
cgroup          on  /sys/fs/cgroup/systemd      type  cgroup      (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd)
cgroup          on  /sys/fs/cgroup/devices      type  cgroup      (rw,nosuid,nodev,noexec,relatime,devices)
cgroup          on  /sys/fs/cgroup/cpuset       type  cgroup      (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup          on  /sys/fs/cgroup/freezer      type  cgroup      (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup          on  /sys/fs/cgroup/net_cls      type  cgroup      (rw,nosuid,nodev,noexec,relatime,net_cls)
cgroup          on  /sys/fs/cgroup/cpu,cpuacct  type  cgroup      (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup          on  /sys/fs/cgroup/blkio        type  cgroup      (rw,nosuid,nodev,noexec,relatime,blkio)
systemd-1       on  /proc/sys/fs/binfmt_misc    type  autofs      (rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct)
debugfs         on  /sys/kernel/debug           type  debugfs     (rw,relatime)
sunrpc          on  /run/rpc_pipefs             type  rpc_pipefs  (rw,relatime)
mqueue          on  /dev/mqueue                 type  mqueue      (rw,relatime)
configfs        on  /sys/kernel/config          type  configfs    (rw,relatime)
/dev/mmcblk0p1  on  /boot                       type  vfat        (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro)
/dev/mmcblk0p2  on  /var/log.bak                type  ext4        (rw,noatime,data=ordered)
log2ram         on  /var/log                    type  tmpfs       (rw,nosuid,nodev,noexec,relatime,size=51200k,mode=755)
tmpfs           on  /run/user/1000              type  tmpfs       (rw,nosuid,nodev,relatime,size=94944k,mode=700,uid=1000,g