emonSD next steps: filesystem & logrotate

[borpin (Brian Orpin)]

@TrystanLea

Works great for emonhub.log, we can have additional configs for the other emoncms logs. It does not however stop other logs from getting too large.

Which logs are you concerned about? Other than the recent issues, for many years there has not been an issue. I think we are trying to fix something that does not need fixing.

@pb66

We must have a timed rotation, not just rely on size, a 999k log file with zero activity could sit there in ram for ever, better it is rotated out after a week/month.

But we do have timed rotation. In fact, logrotate, as setup, only triggers to test for rotation on a fixed time schedule and it will then decide on the rotation to be carried out depending on what rule has been reached (size or time). For true SIZE/MAXSIZE rotation you need monit to monitor the SIZE of the log files and trigger the rotation if necessary.

@pb66

By default olddir can only be used on the same device (ie the rotations could go to another folder, but still in RAM) Either renamecopy, copy or copytruncate can be used to facilitate moving files across devices using olddir.

I really dislike olddir as several packages use error.log in a folder and olddir cannot distinguish between them. It is pointless in the emonhub configuration IMHO even with Log2RAM.

I currently use a post rotate command to move out the emonhub file to persistent memory (as I’ve mentioned before) - much more robust IMHO.

/var/log/emonhub/emonhub.log {
        rotate 6
        daily
        size 2M
        maxsize 2M
        missingok
        copytruncate
        notifempty
        postrotate
             mv /var/log/emonhub/emonhub.log.1 /home/pi/data/
             cat /home/pi/data/emonhub.log.1 >> /home/pi/data/emonhub.log
             rm /home/pi/data/emonhub.log.1
        endscript
}

/home/pi/data/emonhub.log {
        su pi pi
        rotate 10
        daily
        compress
        missingok
        notifempty
}

Of course it isn’t working right now as the update buggered up the logrotate setup as usual.

For /var/log

• rotation number does not matter but just stops it growing if something goes wrong
• daily - at the very least rotate daily
• size/maxsize - rotate once this big
• copytruncate - this seems to work without loss AFAICS.

For /home/pi/data/

• keeps 10 daily logs compressed.

[djh (Dave Howorth)]

@borpin

mv /var/log/emonhub/emonhub.log.1 /home/pi/data/
cat /home/pi/data/emonhub.log.1 >> /home/pi/data/emonhub.log
rm /home/pi/data/emonhub.log.1

I’m curious why you do this instead of the simpler

cat /var/log/emonhub/emonhub.log.1 >> /home/pi/data/emonhub.log
rm /var/log/emonhub/emonhub.log.1

which would reduce the writes to the SD card noticeably, would it not?

[Bill.Thomson]

@djh

which would reduce the writes to the SD card noticeably, would it not?

I wondered about that too. So I ran dstat -d to monitor disk reads/writes while
running the mv and cp commands on two different large files.
(monitored the copy/move in one ssh session, ran the commands in another ssh session)
One file is ~22MB, the other, ~25MB.
The top half of the screenshot shows the result of running cp file1 /opt on the 1st file.
The bottom half shows the result of running mv file2 /opt on the 2nd file.

You may need to install dstat. (sudo apt-get install dstat)

As Brian often says, YMMV.

(you may have to click on the picture to see it in its entirety)

---

More info on dstat:

Linux commands for measuring disk activity

[djh (Dave Howorth)]

Hi Bill,

Sorry, I haven’t checked into dstat yet (I do have it installed apparently) so I might be wide of the mark, but you do appreciate that mv is equivalent to an ln if the two directories are on the same filesystem but is equivalent to a cp if they are on different filesystems?

i.e. it does a few small directory operations if possible and only copies a lot of data blocks if necessary.

So your test suggests to me that the original location of file2 and /opt are in the same filesystem, or that file1 and file2 are radically different in size. Is that right or did I miss something?

[Bill.Thomson]

@djh

mv is equivalent to an ln if the two directories are on the same filesystem but is equivalent to a cp if they are on different filesystems

Indeed it is. But…

@djh

if the two directories are on the same filesystem

I’d forgotten that the /home/pi/data directory is on its own partition/filesystem.

@djh

file2 and /opt are in the same filesystem

Correct, hence the lack of much disk activity.

@djh

did I miss something?

No, you nailed it.

[borpin (Brian Orpin)]

@djh

cat /var/log/emonhub/emonhub.log.1 >> /home/pi/data/emonhub.log
rm /var/log/emonhub/emonhub.log.1

which would reduce the writes to the SD card noticeably, would it not?

Yes it is simpler, no there is no reason not to do it that way and YMMV.

[TrystanLea]

Thanks @pb66. Happy to move the instance apache2 log to /var/log/emoncms as you suggest

/var/log/emoncms/apache2-error.log
/var/log/emoncms/apache2-access.log

The global log would then be fine in the apache2 folder renamed:

/var/log/apache2/apache2-error.log

Yes happy to use conf-available, I tried it before but forgot I needed to run ‘a2enconf’, testing now works a treat, how about, emonsd.conf?

# ServerName
ServerName localhost

# Default apache2 error log
ErrorLog ${APACHE_LOG_DIR}/apache2.log

Noted your comments on SSL, while Im familiar with setting up certbot on online servers, Its not something I’ve done yet on a pi for application on the local network without a domain name but it is something we are interested in.

---

Thanks @borpin

@borpin

Which logs are you concerned about? Other than the recent issues, for many years there has not been an issue. I think we are trying to fix something that does not need fixing.

Sure, the option to enable a global maxsize limit is only a single line, Id be keen to keep it in there as it provides a fallback but Id also be happy for it to be optional.

@borpin

I really dislike olddir as several packages use error.log in a folder and olddir cannot distinguish between them

Looking at all the logs on my system it was only mysql and apache that where conflicting. I can see your suggestion is a neat solution for emonhub.log and potentially emoncms logs but without olddir we would also need post rotate commands for the other logs - that is unless we dont think the other logs are a problem - as you suggest.

Thinking through this, Lets say for example that we have ufw.log or an apache2 access.log logging with a high throughput, at rotation without olddir these would stay on the log2ram tmpfs partition, log2ram would rsync these to its persistent backup. As these files dont change after rotation, there would be nothing to rsync? causing no additional write load? But they would present processor and read load which we can avoid by moving them out of the log2ram system…

Could we alternatively create our own script to copy out the rotated logs and replicate the folder structure to avoid conflicts? this would run after logrotate and before calling log2ram write…

[borpin (Brian Orpin)]

@pb66

But at the moment, just laying the foundations and loosely documenting setting up SSL would be a big step.

@TrystanLea

Noted your comments on SSL, while Im familiar with setting up certbot on online servers, Its not something I’ve done yet on a pi for application on the local network without a domain name but it is something we are interested in.

SSL will need a push as more browsers do not like non HTTPS addressees even on a local LAN. However, you really do need a domain name to do it although not an external website. By using the DNS challenge method all you need is control of your DNS - Cloudflare will do this for free. I’ve written a guide for Lighttpd, but the certbot part is very similar and actually there is an Apache plugin that makes it even easier.

Setting up the Apache configuration so this can easily be enabled is a nobrainer AFAIAC.

[borpin (Brian Orpin)]

@TrystanLea

Could we alternatively create our own script to copy out the rotated logs and replicate the folder structure to avoid conflicts? this would run after logrotate and before calling log2ram write…

I’ll refer to my previous answer. This never was a problem before so I suggest we put it on the back burner. Use L2R to solve folder creation and replication issues and ensure our emon* logs do not run away / rotate correctly.

[TrystanLea]

@TrystanLea

The global log would then be fine in the apache2 folder renamed:

/var/log/apache2/apache2-error.log

realise that we introduce the filename conflict again here if there is also a similar named apach2-error.log in the instance folder.

[pb66 (Paul)]

@TrystanLea

realise that we introduce the filename conflict again here if there is also a similar named apach2-error.log in the instance folder.

It shouldn’t because the instance liog of the same name will be in the instance folder post rotation, only the server-wide error log will be in the root of hdd.old (global olddir).

[TrystanLea]

To recap on steps to the first release of this image build script:

• Merge env_example branch of emonhub into master, verify that update procedure on existing live images apply the relevant change correctly.
• Merge log_location branches of emoncms and modules into master, again verify that update procedure on existing live images apply the relevant change correctly.
• Finalise virtual host config (the virtual host config is already part of the install process and I’ve applied the change to move the defaults to conf-available).
• Finalise logrotate configuration (present ongoing discussion)
• Test/Fix wifi AP, image currently starts with AP enabled.

The install script does currently work to a complete install, it checks out to the relevant emonhub and emoncms development branches. I’ve been itteratively building fresh images as we’ve been working through these discussions. Latest version is here: EmonScripts/install at master · openenergymonitor/EmonScripts · GitHub

[TrystanLea]

@pb66

It shouldn’t because the instance liog of the same name will be in the instance folder post rotation, only the server-wide error log will be in the root of hdd.old (global olddir).

Ok great, I need to add the relvant config for this in the emoncms logrotate conf

[TrystanLea]

Added in, works well move apache2 instance error log to /var/log/emoncms, rotate emoncms l… · openenergymonitor/EmonScripts@3f8865f · GitHub

I’ve also added a specific olddir setting for emonhub: emonhub olddir · openenergymonitor/EmonScripts@28cab81 · GitHub

[borpin (Brian Orpin)]

@TrystanLea

Merge env_example branch of emonhub into master

Should we try and move to having a stable branch of emonhub?

@TrystanLea

Added in, works well

Great. Love to see this closed off.

[TrystanLea]

@borpin

Should we try and move to having a stable branch of emonhub?

Yes i was thinking the same.

[borpin (Brian Orpin)]

On the EmonScripts more generally, might I suggest dropping the ‘EmonSD’ image mechanism and instead use a version number for the Scripts repo and a Stable branch?

Perhaps change emonpiRelease in the Admin page to be a bit more generic and relate it to the script version? Possibly move it out of the ‘Pi’ section?

[TrystanLea]

Good point, yes.

[borpin (Brian Orpin)]

@TrystanLea I’m resurrecting this discussion following an install onto Ubuntu.

On install:

• There is no log rotate setup for emoncms.log
• apache2-error.log is in emoncms folder (as expected).

I suggest:

• There needs to be a conf file installed by the script to ensure the emoncms log gets rotated on non-EmonSD installations.

Suggested configuration

/etc/logrotate.d/emoncms

/var/log/emoncms/*.log {
        rotate 7
        daily
        compress
        copytruncate
        size 100k
        nocreate
        missingok
        notifempty
        delaycompress
}

This file could be created in the scripts Repo (a specific folder for non-SD configurations?) and linked to if the emonSD_pi_env flag is false.

[djh (Dave Howorth)]

@borpin

apache2-error.log is in emoncms folder (as expected).

Ignorant question, but why is this, and why is it expected? An apache error log is nothing to do with emoncms per se. emoncms may be a user of apache but many other things might be too.