emonSD next steps: filesystem & logrotate

emonSD
141

No I do not agree; leave standard files and base setups alone. You can override any predefined directives with your 00 L2R conf file in the logrotate.d folder. Remember this install is not just for SD Cards. In theory, you can drop in a conf file for emoncms only (no L2R or emonhub) and it will just work.

You could add in a dropin file that has an include statement pointing to include /emon/somerepo/logrotate.d/ but I do not see any advantage.

Packages copy their own conf files to logrotate.d on install so I think we should do the same and not reinvent the wheel.

Looking at DietPi I’m not sure what it has done with the main logrotate conf file (they are allowed to as the OS owner) but what I can see are standard logrotate conf files within the logrotate.d folder that obviously get included by logrotate when called.

142

No surprise there! :smile:

Exactly what I said!!!

You can, but you cannot head off any configs in the per package conf’s that will take priority over globals. So you are setting vague limits that might be adhered to or more likely over ridden by each package.

No, exactly the opposite. Using L2R and a custom logrotation tailored to the use of tmpfs and minimising writes is specific ONLY to emonSD (or at least all sd based installs).

If you are not using sdcard, you don’t need logs in ram, you don’t need L2R and you don’t need a more specific logrotation config.

Correct! But it should be designed for “normal” server usage on a hdd rather than sdcard. The sdcard specific settings will be part of the low-write optimisations, not the general emoncms repo, likewise for emonhub too.

Indeed you could, the advantage of this (over totally stock) is that we can “drop in” better conf files that do not rotate weekly (overridding a possible daily global?) without disturbing the stock setup.

Exactly that, on a non-emonsd setup yes! But the std packaged confs are no good for rotating tmpfs logs.

No-one is reinventing the wheel, but the wheels off a non-sdcard based package are not necessarily the best fit for our tmpfs logging, or at least the tyres need changing for the different terrain.

We could bundle all our changes into a file like 00_olddir BUT, as I’ve explained, that would be too ridged, users cannot add thier own emonsd friendly logrotations within our modified config, or
perhaps a “ZZ_somethingorother” conf file is also required so that we can have closing catch alls?

The default daily logrotation cron is disabled or deleted so the command needs rewriting anyway, might as well be to a custom file rather than trying to manipulate (without editing) the stock logrotate.conf. I can see no problem with using a custom “main” file and with the include /etc/logrotate.d/ it will still use the original confs, unless we decide otherwise.

As are we as the OS modifier’s! Granted, if we don’t make changes to the OS (no logs in ram etc) we should stick to the defaults!

How is that obvious?

logrotate must be called with a path to the “main” config file, in that file you can include all the files in a chosen directory. When logrotate reads the config, it’s just a long string of text, the beginning and the end of that text is the “main” conf and in the middle you have the contents of the “include” directory, if the “main” conf doesn’t point logrotate to that directory, It is none the wiser, the files do not get used. There is no “obviously” about it.

If dietpi doesn’t call logrotate with a conf file path, it won’t run, if the file on that path doesn’t include the logrotate.d folder, it won’t include any of those package maintained files. From what I can see (from a very brief look, so forgive me if I’m wrong) dietpi doesn’t use logrotate when using a tmpfs ram. It looks like that directory is just somewhere for the packages to dump their config files just to be ignored, sound familiar?

143

@TrystanLea - I stumbled accross this

You’ll notice a ghastly thing! We need to have a rotate directive, because otherwise, logrotate will delete our old log files! Imagine my terror when I momentarily forgot that the -d option prevents logrotate from actually taking action, thereby removing my log files from previous days!

Could this be why you lost your logs?

144

I can see your point @pb66 re potential for modules specific logrotate config potentially filling our limited ram space for log2ram. Looking at the size of the other logfiles, now that we have sorted the reason for the large daemon and syslog logs (redirecting emonhub + emoncms scripts earlier on), and with the apache2 access logs silenced, the other logs are all relatively slowly increasing in size. Looking at their logrotate settings:

daily: apache2, mosquitto, mysql, log2ram, syslog
weekly: redis, all other rsyslog, ufw
monthly: apt, aptitude, dpkg

ufw could see a lot of activity. most are likely to rotate in time…

In theory enforcing a global rotation size limit sounds good to me (specific for emonSD)

log2ram is currently only installed if you have install_emonsd=true here https://github.com/openenergymonitor/emonpi/blob/master/install/config.ini#L69.

I can see that that would work, you are suggesting changing /etc/cron.daily/logrotate:

#!/bin/sh
test -x /usr/sbin/logrotate || exit 0
/usr/sbin/logrotate /etc/custom_logrotate.conf <- custom config file..
145

I think it was something with the unmodified rsync command and your branch of log2ram, you mentioned you where not using rsync before? your rsync_mods branch works fine though.

146

I’ve run a emonSD build using the install script, with the rsync_mods branch of log2ram and emonhub logfile change alongside the issues you flagged in the emonpi issue list @borpin. A couple of minor issues fixed and it all completed successfully.

My draft readme for the install process is available in the folder here:
https://github.com/openenergymonitor/emonpi/tree/master/install

I havent automated the data partition creation, Im not sure how to do that. It needs a reboot to continue etc. The ufw rules also need to be applied after a reboot.

On the todo list for the first release:

  • finalise emoncms log locations i.e: /var/log/emon/emoncms/sync.log
  • update emoncms & modules with configurable log directory paths for sync/update etc.
  • finalise logrotate config
  • move install and update from emonpi repo to emonscripts or other suitable name (needs to be carefully managed in order to maintain backwards compatibility with existing emonSD based systems that will be looking for these scripts in the emonpi repo).
1 Like
147

Yes, but to something more like

##!/bin/sh
#test -x /usr/sbin/logrotate || exit 0
#/usr/sbin/logrotate /etc/custom_logrotate.conf

# logrotate now triggered by log2ram
# see /etc/cron.hourly/log2ram

and /etc/cron.hourly/log2ram would have something like

#!/usr/bin/env sh

# First we rotate using OEM specific config
test -x /usr/sbin/logrotate || exit 0
/usr/sbin/logrotate /emon/somerepo/logrotate.conf

# then we sync /var/log to disk
log2ram write

I have deliberately pointed the command NOT to /etc/logrotate.conf so that it can stay there unadulterated, and the file we are using is easily updated via git. BUT we can use a symlink instead eg

#!/usr/bin/env sh

# First we rotate using OEM specific config
test -x /usr/sbin/logrotate || exit 0
/usr/sbin/logrotate /etc/custom_logrotate.conf

# then we sync /var/log to disk
log2ram write

where /etc/custom_logrotate.conf is just a symlink to /emon/somerepo/logrotate.conf instead. There should also be a symlink to a README to explain the set up to. Shame logrotate doesn’t use a sub-folder. The naming will be very important so that the files show up together. ie

pi@raspberrypi:~ $ ls /etc/logrotate*
/etc/logrotate.conf

/etc/logrotate.d:
00_olddir  apt  aptitude  dpkg  log2ram  monit  netdata  rsyslog

should show as

pi@raspberrypi:~ $ ls /etc/logrotate*
/etc/logrotate.conf
/etc/logrotate.CUSTOM 
/etc/logrotate.README

/etc/logrotate.d:
00_olddir  apt  aptitude  dpkg  log2ram  monit  netdata  rsyslog

or somthing simiar, I just changed the file extn’s as an example, these are links not files, so the extn’s are less important, the fact they stick out and are obvious is more important, and that they get found in a search for logrotate* etc.

I quite like the /etc/logrotate.CUSTOM option now, it would also be very obvious where ever the command is called.

#!/usr/bin/env sh

# First we rotate using OEM specific config
test -x /usr/sbin/logrotate || exit 0
/usr/sbin/logrotate /etc/logrotate.CUSTOM

# then we sync /var/log to disk
log2ram write
1 Like
148

Only a sdcard (ie low-write) install needs the data partition, so L2R needs a reboot too, you should just be able to edit fstab and install L2R and the one reboot create all the partitions.

Why is a reboot needed? Or to be done after a rebbot?

sudo ufw allow ??? ??? 
sudo ufw enable --force

should do it and the changes will stick, no reboot required.

IMO you should create a OEM set of “apps” at /etc/ufw/applications.d that’s what I usually do.

You can have all your ports defined incl mqtt http https ssh etc etc so you can just use the app names.

149

Just be sure that “SSH” or “22” is one of the ports/apps you define so as not to lock yourself out.

1 Like
150

I need to reboot after creating the partition to then format it with the ext2 filesystem and then mount to /var/opt/emon, the install script needs to run after that so that it creates the phpfina etc directories. I think it requires two reboots before starting the automatic script part. But that is a emonSD specific filesystem setup step so Im not too concerned.

Thanks for the ufw tips.

151

Something like this (pseudo code)

if not /data exists and is sdcard   # or could test the hostname or could test for temporary.service etc
    edit fstab to add /data
    install L2R
    printf blah blah blah > temporary.service
    systemctl enable temporary.service
    edit hostname
    reboot
else
   systemctl disable temporary.service
   rm temporary.service
   format /data

continue install

all temporary.service would be is a basic service unit to restart the install script, the second run will land in the “else” and then carry on.

1 Like
152

In danger of going OT :laughing:, I had some issues when setting up a temp Pi outside of my firewall so there was a website accessible externally when I was playing with SSL.

Using ‘ShieldsUp’ I was surprised that the default UFW setup did not show ports as stealth but as closed (so they existed). To make a machine stealth you need

ufw default deny incoming

Now I may be wrong here and but I did test it…

Personally I then allow in anything from my local (wired) LAN (if someone is on my LAN I’m screwed anyway). This means if it is a purely internal machine you don’t need anything else. You cannot access it from a machine outside your LAN. Opening just the SSH port means you can access SSH from anywhere (other firewalls permitting).

ufw allow in on eth0 to 192.168.1.0/24 from 192.168.1.0/24

So although I can SSH into this machine which is exposed to the internet via a DMZ, when I run the test, only the WWW and WWW secure ports are shown as open.

If already enabled and a rule is changed all you should need is

sudo ufw reload

YMMV.

1 Like
153

I think that is the default, From the man page

On installation, ufw is disabled with a default incoming policy of deny and a default
outgoing policy of allow, with stateful tracking for NEW connections.

154

Intriguing. Just checked on the Latest Pi and it is set by default. Odd.

155

Because we have gone back to using the emonhub.log file created from the script, the logviewer no longer works but the download does :smile:.

[edit]
@pb66 - the olddir /var/log.old is not being created by the install script it appears.

156

I’m getting some de ja vue here. That’s right it is auto created by the first rotation. Not the install script.

1 Like
157

Memory of a fish me. Why not just create it at install time? First look at /var/log and it’s like, why is that link red?!

158

I installed emoncms onto my EmonPi with the install script. Log2Ram has installed but I did not fiddle with creating a new partition first, just left Raspbian (new image) to it.

mount looks like this, though I’m not sure I am reading it correctly - log2ram seems to be on tmpfs - not sure how!

pi@emonpi:/var/log $ sudo mount | column -t
/dev/mmcblk0p2  on  /                           type  ext4        (rw,noatime,data=ordered)
devtmpfs        on  /dev                        type  devtmpfs    (rw,relatime,size=470116k,nr_inodes=117529,mode=755)
sysfs           on  /sys                        type  sysfs       (rw,nosuid,nodev,noexec,relatime)
proc            on  /proc                       type  proc        (rw,relatime)
tmpfs           on  /dev/shm                    type  tmpfs       (rw,nosuid,nodev)
devpts          on  /dev/pts                    type  devpts      (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs           on  /run                        type  tmpfs       (rw,nosuid,nodev,mode=755)
tmpfs           on  /run/lock                   type  tmpfs       (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs           on  /sys/fs/cgroup              type  tmpfs       (ro,nosuid,nodev,noexec,mode=755)
cgroup          on  /sys/fs/cgroup/systemd      type  cgroup      (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd)
cgroup          on  /sys/fs/cgroup/devices      type  cgroup      (rw,nosuid,nodev,noexec,relatime,devices)
cgroup          on  /sys/fs/cgroup/cpuset       type  cgroup      (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup          on  /sys/fs/cgroup/freezer      type  cgroup      (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup          on  /sys/fs/cgroup/net_cls      type  cgroup      (rw,nosuid,nodev,noexec,relatime,net_cls)
cgroup          on  /sys/fs/cgroup/cpu,cpuacct  type  cgroup      (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup          on  /sys/fs/cgroup/blkio        type  cgroup      (rw,nosuid,nodev,noexec,relatime,blkio)
systemd-1       on  /proc/sys/fs/binfmt_misc    type  autofs      (rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct)
debugfs         on  /sys/kernel/debug           type  debugfs     (rw,relatime)
sunrpc          on  /run/rpc_pipefs             type  rpc_pipefs  (rw,relatime)
mqueue          on  /dev/mqueue                 type  mqueue      (rw,relatime)
configfs        on  /sys/kernel/config          type  configfs    (rw,relatime)
/dev/mmcblk0p1  on  /boot                       type  vfat        (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro)
/dev/mmcblk0p2  on  /var/log.bak                type  ext4        (rw,noatime,data=ordered)
log2ram         on  /var/log                    type  tmpfs       (rw,nosuid,nodev,noexec,relatime,size=51200k,mode=755)
tmpfs           on  /run/user/1000              type  tmpfs       (rw,nosuid,nodev,relatime,size=94944k,mode=700,uid=1000,g
159

That’s what L2R does!

That’s why I have been pushing this as a “simple one line solution” to the “rc.local” and “lack of log history”. Take one stock Raspbian image, add L2R and you have the core basis of a low write image. Anything you add eg apache2, mysql etc etc will have it’s logfiles and folders managed, the log message writes from all the various softwares are buffered into hourly writes and all old logs are rotated out to disk. It basically does it all, to a degree, yes we can improve things further by handling the logrotations better and syncronising things better etc. But essentially, just installing L2R via a simple command takes care of the lions share of the low-write optimisations and emonSD issues (rc.local and lost logs). Minimal fuss maximum results, from there we can focus on improving things rather than reinventing the wheel.

2 Likes
160

Picking up where we left off above. Im happy with your suggestion regarding tying the logrotate in with the log2ram command, If I understand correctly:

  1. cron.hourly/logrotate has its commands commented out
  2. cron.hourly/log2ram now uses logrotate.CUSTOM config before calling log2ram write.

It seems that the last step that we have is to decide is what is inside logrotate.CUSTOM

Going by an earlier post above focused more on emonhub.log we discussed number or rotates and max size per file as well as I think renamecopy, are you thinking something like this? @pb66

# logrotate.CUSTOM

rotate 10
size 3M
missingok
renamecopy
notifempty
compress
nomail

include /etc/logrotate.d

Do we call include /etc/logrotate.d before the global overrides? What else do we need?

Edit: you mentioned above possibly a smaller max size for logs not defined e.g 100k, emonhub.log would then override that with “size 3M” but logrotate configs without the setting would override to the global setting of 100k?