emonSD next steps: filesystem & logrotate

Which logs are you concerned about? Other than the recent issues, for many years there has not been an issue. I think we are trying to fix something that does not need fixing.

But we do have timed rotation. In fact, logrotate, as setup, only triggers to test for rotation on a fixed time schedule and it will then decide on the rotation to be carried out depending on what rule has been reached (size or time). For true SIZE/MAXSIZE rotation you need monit to monitor the SIZE of the log files and trigger the rotation if necessary.

I really dislike olddir as several packages use error.log in a folder and olddir cannot distinguish between them. It is pointless in the emonhub configuration IMHO even with Log2RAM.

I currently use a post rotate command to move out the emonhub file to persistent memory (as Iā€™ve mentioned before) - much more robust IMHO.

/var/log/emonhub/emonhub.log {
        rotate 6
        daily
        size 2M
        maxsize 2M
        missingok
        copytruncate
        notifempty
        postrotate
             mv /var/log/emonhub/emonhub.log.1 /home/pi/data/
             cat /home/pi/data/emonhub.log.1 >> /home/pi/data/emonhub.log
             rm /home/pi/data/emonhub.log.1
        endscript
}

/home/pi/data/emonhub.log {
        su pi pi
        rotate 10
        daily
        compress
        missingok
        notifempty
}

Of course it isnā€™t working right now as the update buggered up the logrotate setup as usual.

For /var/log

  • rotation number does not matter but just stops it growing if something goes wrong
  • daily - at the very least rotate daily
  • size/maxsize - rotate once this big
  • copytruncate - this seems to work without loss AFAICS.

For /home/pi/data/

  • keeps 10 daily logs compressed.

Iā€™m curious why you do this instead of the simpler

cat /var/log/emonhub/emonhub.log.1 >> /home/pi/data/emonhub.log
rm /var/log/emonhub/emonhub.log.1

which would reduce the writes to the SD card noticeably, would it not?

I wondered about that too. So I ran dstat -d to monitor disk reads/writes while
running the mv and cp commands on two different large files.
(monitored the copy/move in one ssh session, ran the commands in another ssh session)
One file is ~22MB, the other, ~25MB.
The top half of the screenshot shows the result of running cp file1 /opt on the 1st file.
The bottom half shows the result of running mv file2 /opt on the 2nd file.

You may need to install dstat. (sudo apt-get install dstat)

As Brian often says, YMMV.

(you may have to click on the picture to see it in its entirety)


More info on dstat:

Hi Bill,

Sorry, I havenā€™t checked into dstat yet (I do have it installed apparently) so I might be wide of the mark, but you do appreciate that mv is equivalent to an ln if the two directories are on the same filesystem but is equivalent to a cp if they are on different filesystems?

i.e. it does a few small directory operations if possible and only copies a lot of data blocks if necessary.

So your test suggests to me that the original location of file2 and /opt are in the same filesystem, or that file1 and file2 are radically different in size. Is that right or did I miss something?

Indeed it is. Butā€¦

Iā€™d forgotten that the /home/pi/data directory is on its own partition/filesystem. :sunglasses:

Correct, hence the lack of much disk activity.

No, you nailed it. :wink:

Yes it is simpler, no there is no reason not to do it that way and YMMV.

Thanks @pb66. Happy to move the instance apache2 log to /var/log/emoncms as you suggest

/var/log/emoncms/apache2-error.log
/var/log/emoncms/apache2-access.log

The global log would then be fine in the apache2 folder renamed:

/var/log/apache2/apache2-error.log

Yes happy to use conf-available, I tried it before but forgot I needed to run ā€˜a2enconfā€™, testing now works a treat, how about, emonsd.conf?

# ServerName
ServerName localhost

# Default apache2 error log
ErrorLog ${APACHE_LOG_DIR}/apache2.log

Noted your comments on SSL, while Im familiar with setting up certbot on online servers, Its not something Iā€™ve done yet on a pi for application on the local network without a domain name but it is something we are interested in.


Thanks @borpin

Sure, the option to enable a global maxsize limit is only a single line, Id be keen to keep it in there as it provides a fallback but Id also be happy for it to be optional.

Looking at all the logs on my system it was only mysql and apache that where conflicting. I can see your suggestion is a neat solution for emonhub.log and potentially emoncms logs but without olddir we would also need post rotate commands for the other logs - that is unless we dont think the other logs are a problem - as you suggest.

Thinking through this, Lets say for example that we have ufw.log or an apache2 access.log logging with a high throughput, at rotation without olddir these would stay on the log2ram tmpfs partition, log2ram would rsync these to its persistent backup. As these files dont change after rotation, there would be nothing to rsync? causing no additional write load? But they would present processor and read load which we can avoid by moving them out of the log2ram systemā€¦

Could we alternatively create our own script to copy out the rotated logs and replicate the folder structure to avoid conflicts? this would run after logrotate and before calling log2ram writeā€¦

SSL will need a push as more browsers do not like non HTTPS addressees even on a local LAN. However, you really do need a domain name to do it although not an external website. By using the DNS challenge method all you need is control of your DNS - Cloudflare will do this for free. Iā€™ve written a guide for Lighttpd, but the certbot part is very similar and actually there is an Apache plugin that makes it even easier.

Setting up the Apache configuration so this can easily be enabled is a nobrainer AFAIAC.

1 Like

Iā€™ll refer to my previous answer. This never was a problem before so I suggest we put it on the back burner. Use L2R to solve folder creation and replication issues and ensure our emon* logs do not run away / rotate correctly.

realise that we introduce the filename conflict again here if there is also a similar named apach2-error.log in the instance folder.

It shouldnā€™t because the instance liog of the same name will be in the instance folder post rotation, only the server-wide error log will be in the root of hdd.old (global olddir).

1 Like

To recap on steps to the first release of this image build script:

  • Merge env_example branch of emonhub into master, verify that update procedure on existing live images apply the relevant change correctly.
  • Merge log_location branches of emoncms and modules into master, again verify that update procedure on existing live images apply the relevant change correctly.
  • Finalise virtual host config (the virtual host config is already part of the install process and Iā€™ve applied the change to move the defaults to conf-available).
  • Finalise logrotate configuration (present ongoing discussion)
  • Test/Fix wifi AP, image currently starts with AP enabled.

The install script does currently work to a complete install, it checks out to the relevant emonhub and emoncms development branches. Iā€™ve been itteratively building fresh images as weā€™ve been working through these discussions. Latest version is here: EmonScripts/install at master Ā· openenergymonitor/EmonScripts Ā· GitHub

Ok great, I need to add the relvant config for this in the emoncms logrotate conf

Added in, works well move apache2 instance error log to /var/log/emoncms, rotate emoncms lā€¦ Ā· openenergymonitor/EmonScripts@3f8865f Ā· GitHub

Iā€™ve also added a specific olddir setting for emonhub: emonhub olddir Ā· openenergymonitor/EmonScripts@28cab81 Ā· GitHub

1 Like

Should we try and move to having a stable branch of emonhub?

Great. Love to see this closed off.

1 Like

Yes i was thinking the same.

1 Like

On the EmonScripts more generally, might I suggest dropping the ā€˜EmonSDā€™ image mechanism and instead use a version number for the Scripts repo and a Stable branch?

Perhaps change emonpiRelease in the Admin page to be a bit more generic and relate it to the script version? Possibly move it out of the ā€˜Piā€™ section?

Good point, yes.

1 Like

@TrystanLea Iā€™m resurrecting this discussion following an install onto Ubuntu.

On install:

  • There is no log rotate setup for emoncms.log
  • apache2-error.log is in emoncms folder (as expected).

I suggest:

  • There needs to be a conf file installed by the script to ensure the emoncms log gets rotated on non-EmonSD installations.

Suggested configuration

/etc/logrotate.d/emoncms

/var/log/emoncms/*.log {
        rotate 7
        daily
        compress
        copytruncate
        size 100k
        nocreate
        missingok
        notifempty
        delaycompress
}

This file could be created in the scripts Repo (a specific folder for non-SD configurations?) and linked to if the emonSD_pi_env flag is false.

1 Like

Ignorant question, but why is this, and why is it expected? An apache error log is nothing to do with emoncms per se. emoncms may be a user of apache but many other things might be too.