Has /var/log Filling up Been Fixed?

I’ve read all the posts related to /var/log filling up but it seems (with my limited understanding) this issue has not been resolved. Several ways to ‘fix’ this have been mentioned, but I’ve found none that work yet, unless I missed something, which is quite likely. :grinning:

My Inputs are updating, but I reboot/update so often that RAM/Swap is not filling up.

I’m running 10.0.2 (master) and I hasn’t been resolved in that release. I think this issue’s importance should be raised as the problem of Inputs not updating greatly reduces the robustness of the system and therefore the reliability and trust.

My knowledge is somewhat limited, but I have plenty of time and can offer to help test or document a fix.

Best wishes,

Hi Neil, there are several layers to the “log partition filling up” issue, yesterday Trystan pushed some changes to the repo’s to change the way the emoonhub.log’s are handled, that should crack the reason this issue has escalated dramatically most recently. AFAIA if you update today you should get those changes, so if your up for testing, you could try a new update today and check to see if there is an emonhub.log created

ls -la /var/log/emonhub

You could also just confirm for us the status of the logrotate by just checking the content of these 2 files

cat /etc/cron.hourly/logrotate
cat /etc/logrotate.conf

AFAIA the logrotate was maybe fixed already but that’s unconfirmed, the latest image (which are you running, assuming you are running an emonSD image?) was originally missing the logrotate mods and the images before that somehow got broken along the way, Both Trystan and Glyn have made changes recently but I’m unsure how effective they are as the emonhub log issue hid the logrotate issue a bit.

It would be good to confirm where we are with this at the moment if you are willing.

As for the longer term, there is other stuff in the pipeline to improve the whole logging side of things, the fixes above should just take us back to where the logging has been for the past few years, which has weaknesses but has in almost all cases worked fine most of the time. The future changes will hopefully give us a far more robust and capable setup, but we’re not there yet. Further testing will undoubtedly be useful thank you.

[edit] could you please also post a copy of your update log too?

Hello Paul,

I checked before updating and after updating. There was no difference in the data returned.

Here’s what I got:

pi@emonpi:~ $ ls -la /var/log/emonhub
ls: cannot access '/var/log/emonhub': No such file or directory

pi@emonpi:~ $ cat /etc/cron.hourly/logrotate
#!/bin/sh
test -x /usr/sbin/logrotate || exit 0

/usr/sbin/logrotate -v -s /var/log/logrotate/logrotate.status /etc/logrotate.conf 2>&1 | tee /var/log/logrotate/logrotate.log

pi@emonpi:~ $ cat /etc/logrotate.conf

# emoncms logrotate config

# default is daily run, move cron script to make this happen hourly:

# sudo mv /etc/cron.daily/logrotate /etc/cron.hourly/logrotate

# keep 2 rotations worth of backlogs (size x 2)

rotate 1

# Max size of log file (rotate log file when this size is reached)

size 1M

# Truncate the original log file in place after creating a copy

copytruncate

# Do not mail logfile

nomail

# log owners

su root root

# Don't rotate if empty or notify if missing

missingok

notifempty

# Don't nclude package specefic logrotate config

#include /etc/logrotate.d

# Include package specefic logrotate config

#include /etc/logrotate.d

# Include all logs including 1 level deep subdirectories

/var/log/*.log /var/log/*/*.log /var/log/syslog /var/log/messages /var/log/btmp /var/log/* {}

Please let me know if you need more info or testing.

emonpiupdate.log.zip (4.5 KB)

If you look at the update log near the top, when it tries to update the first repo “emonpi” it fails due to some local changes

git pull /home/pi/emonpi
  enable-ssh
  enable-ssh-button
  gpiozero-enable-ssh
  greeebs-patch-1
* master
  python-systemd-servicerunner
On branch master
Your branch is behind 'origin/master' by 27 commits, and can be fast-forwarded.
  (use "git pull" to update your local branch)

. . . 

error: Your local changes to the following files would be overwritten by merge:
	lcd/emonPiLCD.cfg
Please commit your changes or stash them before you merge.
Aborting

although it does continue through the rest of the updates, it isn’t using the latest updater as that is part of the emonpi repo, ideally, when the emonpi repo fails to update the update routine should abort to avoid landing in some grey area, eg if there was some new code in emoncms or emonhub that also needed a symlink added or a config file edited, that could break a working setup.

In fact you are missing 27 commits, everything committed to the emonpi repo since April 15th which I think is most of the changes to the updater. actually most of them are for the installer that’s not being used yet.

(Diff between your copy and the current version https://github.com/openenergymonitor/emonpi/compare/0c477c895f3eb198bf821c79cd068f396631cf12...master)

Do you know what changes you made to the lcd/emonPiLCD.cfg file? If not use

git diff ~/emonpi/lcd/emonPiLCD.cfg

to find out

Paul,

After updating, emoncms-mqtt has failed. I restarted emonhub, still failed, I rebooted, still failed. My inputs are not updating now.

What to do?

Server Information

Server Information

Services

  • emonhub :- Failed Failed
  • emoncms_mqtt :- Active Running
  • feedwriter :- Active Running - sleep 60s 6 feed points pending write
  • service-runner :- Active Running
  • emonPiLCD :- Active Running
  • redis-server :- Active Running
  • mosquitto :- Active Running

Emoncms

Server

  • OS :- Linux 4.14.71-v7+
  • Host :- emonpi | emonpi | (192.168.1.11)
  • Date :- 2019-05-22 08:41:09 EDT
  • Uptime :- 08:41:09 up 1 min, 1 user, load average: 1.19, 0.57, 0.22

Memory

  • RAM :- Used: 26.02%
    • Total :- 976.74 MB
    • Used :- 254.13 MB
    • Free :- 722.61 MB
  • Swap :- Used: 0.00%
    • Total :- 100 MB
    • Used :- 0 B
    • Free :- 100 MB

Disk

  • / :- Used: 42.13%
    • Total :- 3.81 GB
    • Used :- 1.61 GB
    • Free :- 2.03 GB
  • /home/pi/data :- Used: 1.53%
    • Total :- 10.32 GB
    • Used :- 161.23 MB
    • Free :- 9.63 GB
  • /boot :- Used: 51.69%
    • Total :- 42.52 MB
    • Used :- 21.98 MB
    • Free :- 20.54 MB
  • /mnt/usbdrive :- Used: 0.25%
    • Total :- 28.97 GB
    • Used :- 73.38 MB
    • Free :- 27.4 GB

HTTP

  • Server :- Apache/2.4.25 (Raspbian) HTTP/1.1 CGI/1.1 80

MySQL

  • Version :- 5.5.5-10.1.23-MariaDB-9+deb9u1
  • Host :- localhost:6379 (127.0.0.1)
  • Date :- 2019-05-22 08:41:09 (UTC -04:00‌​)
  • Stats :- Uptime: 106 Threads: 3 Questions: 538 Slow queries: 0 Opens: 26 Flush tables: 1 Open tables: 20 Queries per second avg: 5.075

Redis

  • Version :- 3.2.6
  • Host :- localhost:6379
  • Size :- 143 keys (768.22K)
  • Uptime :- 0 days

MQTT Server

  • Version :- Mosquitto 1.4.10
  • Host :- localhost:1883 (127.0.0.1)

PHP

  • Version :- 7.0.30-0+deb9u1 (Zend Version 3.0.0)
  • Modules :- apache2handler | calendar v7.0.30-0+deb9u1 | Core v7.0.30-0+deb9u1 | ctype v7.0.30-0+deb9u1 | curl v7.0.30-0+deb9u1 | date v7.0.30-0+deb9u1 | dom v20031129 | exif v7.0.30-0+deb9u1 | fileinfo v1.0.5 | filter v7.0.30-0+deb9u1 | ftp v7.0.30-0+deb9u1 | gd v7.0.30-0+deb9u1 | gettext v7.0.30-0+deb9u1 | hash v1.0 | iconv v7.0.30-0+deb9u1 | igbinary v2.0.1 | json v1.4.0 | libxml v7.0.30-0+deb9u1 | mbstring v7.0.30-0+deb9u1 | mcrypt v7.0.30-0+deb9u1 | mosquitto v0.4.0 | mysqli v7.0.30-0+deb9u1 | mysqlnd vmysqlnd 5.0.12-dev - 20150407 - $Id: b5c5906d452ec590732a93b051f3827e02749b83 $ | openssl v7.0.30-0+deb9u1 | pcre v7.0.30-0+deb9u1 | PDO v7.0.30-0+deb9u1 | pdo_mysql v7.0.30-0+deb9u1 | Phar v2.0.2 | posix v7.0.30-0+deb9u1 | readline v7.0.30-0+deb9u1 | redis v4.1.1 | Reflection v7.0.30-0+deb9u1 | session v7.0.30-0+deb9u1 | shmop v7.0.30-0+deb9u1 | SimpleXML v7.0.30-0+deb9u1 | sockets v7.0.30-0+deb9u1 | SPL v7.0.30-0+deb9u1 | standard v7.0.30-0+deb9u1 | sysvmsg v7.0.30-0+deb9u1 | sysvsem v7.0.30-0+deb9u1 | sysvshm v7.0.30-0+deb9u1 | tokenizer v7.0.30-0+deb9u1 | wddx v7.0.30-0+deb9u1 | xml v7.0.30-0+deb9u1 | xmlreader v7.0.30-0+deb9u1 | xmlwriter v7.0.30-0+deb9u1 | xsl v7.0.30-0+deb9u1 | Zend OPcache v7.0.30-0+deb9u1 | zlib v7.0.30-0+deb9u1

Pi

  • Model :- Raspberry Pi 3 Model B+ Rev 1.3 - 1 GB (Sony UK)
  • SoC :- Broadcom BCM2835
  • Serial num. :- 5ABE4EF9
  • Temperature :- 50.46°C - 51.0°C
  • emonpiRelease :- emonSD-30Oct18
  • File-system :- read-write
Client Information

Client Information

HTTP

  • Browser :- Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1 Safari/605.1.15
  • Language :- en-us

Window

  • Size :- 1268 x 798

Screen

  • Resolution :- 1680 x 1050

emonhub.log.zip (2.5 KB)

Oh dear!

I’m guessing you’ve fallen into one of those “grey area” I mentioned above?

The first thing to do is to get the updater to run properly by getting the emonpi repo to update, have you checked the file as suggested above? Do you recall what changes you made? Are they needed?

The current file looks like this

there was a change commited at the end of April to alter the LCD backlight timeout (ref Update EmonpiLCD · openenergymonitor/emonpi@786cd12 · GitHub).

Paul, you’re right on target

I changed ~/emonpi/lcd/emonPiLCD.cfg backlight_timeout = 0 to backlight_timeout = 30 several weeks ago. I’ve changed it back to 0 and performed an update again. My inputs are updating and the emoncms-mqtt failure is now Active: Running.

EmonHub log looks normal now.

I did this again:

pi@emonpi : ~/emonpi/lcd $ ls -la /var/log/emonhub
total 356
drwxr-xr-x 2 emonhub emonhub 60 May 22 09:02 .
drwxr-xr-x 12 root root 540 May 22 09:02
-rw-r–r-- 1 emonhub emonhub 362355 May 22 09:15 emonhub.log

I did not reboot and df -h reveals:

tmpfs 50M 568K 50M 2% /var/log

Looks good to me.

Phew! You’re a genius!

Phew! Sounds good!

is the lcd timeout now been set to 300 by the update?

Could you do the other test mentioned in my first post just to see what logrotate you have now and can I also see your latest update log please?

Yes, the lcd timeout is now set to 300.

Repeated per your first post:

**pi@emonpi** : **~/emonpi/lcd $** cat /etc/cron.hourly/logrotate
#!/bin/sh
test -x /usr/sbin/logrotate || exit 0
/usr/sbin/logrotate -v -s /var/log/logrotate/logrotate.status /etc/logrotate.conf 2>&1 | tee /var/log/logrotate/logrotate.log

And…

**pi@emonpi** : **~/emonpi/lcd $** cat /etc/logrotate.conf

# emoncms logrotate config

# default is daily run, move cron script to make this happen hourly:

# sudo mv /etc/cron.daily/logrotate /etc/cron.hourly/logrotate

# keep 2 rotations worth of backlogs (size x 2)

rotate 1

# Max size of log file (rotate log file when this size is reached)

size 1M

# Truncate the original log file in place after creating a copy

copytruncate

# Do not mail logfile

nomail

# log owners

su root root

# Don't rotate if empty or notify if missing

missingok

notifempty

# Don't nclude package specefic logrotate config

#include /etc/logrotate.d

# Include package specefic logrotate config

#include /etc/logrotate.d

# Include all logs including 1 level deep subdirectories

/var/log/*.log /var/log/*/*.log /var/log/syslog /var/log/messages /var/log/btmp /var/log/* {}

emonpiupdate.log 2.zip (4.3 KB)

Thanks Neil, I shall take a closer look at the logs later, but it seems the logrotate and emonhub.log is as expected so you should be ok now (fingers crossed :smile:)

It’s amazing how ‘worried’ I get when when a system that: (1) I continually tinker with, (2) is not critical in my application and, (3) and is only a hobby, doesn’t work perfectly!

But I’m learning a lot thanks to everyone who contributes here.

Please let me know if you find anything out of whack in the logs.

Thanks

Both your update logs contain this

------------------------------------------
Updating backup module
------------------------------------------
- git branch: stable
- git tags: 1.1.5-10-g3dcb153
- git status:

On branch stable
Your branch is up-to-date with 'origin/stable'.
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

	modified:   config.cfg
	modified:   emoncms-export.sh

this tells us you have some local changes to 2 files that could block the backup module from being updated. It looks like there were not updates to pull in at the time, so it wasn’t a problem, but it’s something to be aware of as it probably will cause a problem at some point.

how is the install behaving now?

Out of interest could you post the results of

df -h
ls -la /var/log

@TrystanLea - If you check Neils update logs they both contain

Installing emoncms logrotate...

Now setting up Logrotate...
Backing up old logrotate configuration...
Linked to new logrotate configuration...
Backing up old logrotate cron job...
mv: cannot stat '/etc/cron.daily/logrotate': No such file or directory
Linked to new logrotate cron job...

Completed


Completed

Running logrotate...

set log rotate config owner to root

the 2 "Backing up old . . . " lines suggest the backed up files are being over written on every update, there no point backing up the original files if they are going to be overwritten again with the content of the symlinked files, ideally this needs to be multi-run safe by making a backup only if a real file is found (not a symlink) as I proposed

I also think the whole update should be aborted if the emonpi repo git pull fails and failures need to be passed to the user somehow. Perhaps a flag “there were errors check your update log” could be bannered across the emoncms gui and or on the emonpi lcd?

@Paul, I did make changes to both files (config.cfg and emoncms-export.sh. My intention was to automate the backup process Emoncms automatic data backup from EmonPi to Remote Computer.

I changed the filename date format in emoncms-export.sh and the backup location in config.cfg. I have not been able to backup to a remote server (yet) but I am auto-backing-up to a USB drive for now. I wanted to backup to the server so that, in turn, that would be backed up to Time Machine and I would have backups going back for years then, without intervention from me. Perhaps I’m being too paranoid though :nerd_face:

I am surprised that changing these files would cause an update failure. But there may be reasons beyond my knowledge.

Log rotation seems to be working properly now, /var/log is not filling up.

I am available for some testing or documentation for any issues that arise so please feel free to ask.

pi@emonpi:~/backup $ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       3.9G  1.7G  2.1G  45% /
devtmpfs        484M     0  484M   0% /dev
tmpfs           489M     0  489M   0% /dev/shm
tmpfs           489M   32M  458M   7% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           489M     0  489M   0% /sys/fs/cgroup
tmpfs            30M   24K   30M   1% /tmp
tmpfs            50M  8.1M   42M  17% /var/log
tmpfs           1.0M     0  1.0M   0% /var/tmp
/dev/mmcblk0p3   11G  162M  9.7G   2% /home/pi/data
/dev/mmcblk0p1   43M   22M   21M  52% /boot
/dev/sda2        29G   75M   28G   1% /mnt/usbdrive
tmpfs            98M     0   98M   0% /run/user/1000

and …

pi@emonpi:~/backup $ ls -la /var/log
total 3600
drwxr-xr-x 12 root      root          580 May 23 16:17 .
drwxr-xr-x 12 root      root         4096 May 18 07:05 ..
drwxr-xr-x  2 root      adm           120 May 24 05:17 apache2
-rw-r-----  1 root      adm        152315 May 24 07:17 auth.log
-rw-r--r--  1 root      root         6758 May 22 08:39 boot.log
-rw-------  1 root      utmp            0 May 22 08:39 btmp
-rw-r-----  1 root      adm        419427 May 24 07:17 daemon.log
-rw-r-----  1 root      adm       1061241 May 23 16:17 daemon.log.1
-rw-r-----  1 root      adm          1265 May 22 08:39 debug
-rw-rw-rw-  1 root      root         2563 May 23 23:30 emoncms.log
drwxr-xr-x  2 emonhub   emonhub        80 May 24 07:17 emonhub
drwxr-xr-x  2 pi        root           60 May 22 08:39 emonpilcd
-rw-r-----  1 root      adm        166594 May 24 07:13 kern.log
drwxr-xr-x  2 pi        pi             80 May 24 07:17 logrotate
-rw-r-----  1 root      adm        103685 May 24 07:13 messages
drwxr-xr-x  2 mosquitto mosquitto      60 May 22 08:39 mosquitto
-rw-rw-rw-  1 root      root            0 May 22 08:39 mqtt_input.log
drwxr-xr-x  2 mysql     adm            60 May 22 08:39 mysql
-rw-rw-rw-  1 root      root            0 May 22 08:39 mysql.log
-rw-rw-rw-  1 root      root            0 May 22 08:39 ntp_update.log
drwxr-xr-x  2 root      root           40 May 22 08:39 openhab
drwxr-xr-x  2 root      root           40 May 22 08:39 openhab2
drwxr-xr-x  2 redis     redis          60 May 22 08:39 redis
-rw-rw-rw-  1 root      root            0 May 22 08:39 service-runner.log
drwxr-xr-x  2 root      root           60 May 22 08:39 supervisor
-rw-r-----  1 root      adm        645955 May 24 07:17 syslog
-rw-r-----  1 root      adm       1088389 May 23 13:17 syslog.1
-rw-r-----  1 root      adm          1668 May 22 08:39 user.log
-rw-rw-r--  1 root      utmp         4992 May 24 06:50 wtmp
pi@emonpi:~/backup $

Basically it’s due to the the way the updates are done via git pulls. If an update (git pull) is initiated on a repository folder it will “pull in the changes” between your existing version and the latest version. If you have made any edits to the files it want s to change, it doesn’t know the priority, do you want to retain your edits (if that’s even possible) or do you want the latest “stock” code. At which point it doesn’t do either, it just tells you there are reasons it didn’t do it so you can “stash” or “commit” your changes or tell git it just overwrite them.

Looking at the logs it does look ok’ish, logrotate is working. But there are a couple of anomalies such several log files not being rotated, could you show us the next level down (ie in the sub-folders)

sudo ls -la /var/log/*/

and also give us a copy of ‘/var/log/logrotate/logrotate.log’ to look at?

Thanks for your help.

@Paul, Thank you for the explanation, that’s one more hole filled in, in my Swiss cheese brain.

Here’s the information you requested:

pi@emonpi:~ $ sudo ls -la /var/log/*/
/var/log/apache2/:
total 1616
drwxr-xr-x  2 root adm      120 May 24 05:17 .
drwxr-xr-x 12 root root     580 May 23 16:17 ..
-rw-r--r--  1 root root  489322 May 24 08:27 access.log
-rw-r--r--  1 root root 1152557 May 24 05:17 access.log.1
-rw-rw-rw-  1 root adm     6648 May 23 21:04 error.log
-rw-r--r--  1 root root       0 May 22 08:39 other_vhosts_access.log

/var/log/emonhub/:
total 1916
drwxr-xr-x  2 emonhub emonhub      80 May 24 08:17 .
drwxr-xr-x 12 root    root        580 May 23 16:17 ..
-rw-r--r--  1 emonhub emonhub  305154 May 24 08:28 emonhub.log
-rw-r--r--  1 emonhub emonhub 1650723 May 24 08:17 emonhub.log.1

/var/log/emonpilcd/:
total 4
drwxr-xr-x  2 pi   root   60 May 22 08:39 .
drwxr-xr-x 12 root root  580 May 23 16:17 ..
-rw-r--r--  1 pi   root 1689 May 22 09:02 emonpilcd.log

/var/log/logrotate/:
total 12
drwxr-xr-x  2 pi   pi     80 May 24 08:25 .
drwxr-xr-x 12 root root  580 May 23 16:17 ..
-rw-r--r--  1 root root 5313 May 24 08:17 logrotate.log
-rw-r--r--  1 root root 1022 May 24 08:17 logrotate.status

/var/log/mosquitto/:
total 4
drwxr-xr-x  2 mosquitto mosquitto   60 May 22 08:39 .
drwxr-xr-x 12 root      root       580 May 23 16:17 ..
-rw-r--r--  1 mosquitto mosquitto 2402 May 22 09:02 mosquitto.log

/var/log/mysql/:
total 4
drwxr-xr-x  2 mysql adm    60 May 22 08:39 .
drwxr-xr-x 12 root  root  580 May 23 16:17 ..
-rw-rw-rw-  1 mysql adm  1569 May 22 08:39 error.log

/var/log/openhab/:
total 0
drwxr-xr-x  2 root root  40 May 22 08:39 .
drwxr-xr-x 12 root root 580 May 23 16:17 ..

/var/log/openhab2/:
total 0
drwxr-xr-x  2 root root  40 May 22 08:39 .
drwxr-xr-x 12 root root 580 May 23 16:17 ..

/var/log/redis/:
total 4
drwxr-xr-x  2 redis redis   60 May 22 08:39 .
drwxr-xr-x 12 root  root   580 May 23 16:17 ..
-rw-rw-rw-  1 redis redis 4095 May 22 08:39 redis-server.log

/var/log/supervisor/:
total 0
drwxr-xr-x  2 root root  60 May 22 08:39 .
drwxr-xr-x 12 root root 580 May 23 16:17 ..
-rw-rw-rw-  1 root root   0 May 22 08:39 supervisord.log

And here’s the file…

logrotate.log.zip (856 Bytes)

1 Like

Is your logrotate setup still “as standard” for an emonSD? Have you applied any of those “fixes” you mentioned?

I’m going to make a couple of comments here just to document some thoughts, these may or may not affect you and/or may or may not get looked into further or addressed by other future development. Some of it has been brought up before and some of it will not be relevant if/when log2ram (and the use of olddir) is introduced, but if that’s not rolled out to old images, some of these things could easily be addressed. I just want to make notes while it’s fresh in my mind, to refer back to.

  • mysql.log doesn’t appear to be used or even needed, it’s created by rc.local and doesn’t appear in any of my (non-emonsd) installs.
  • several logs are just hanging around using up ram. the logrotate.log is full of entries saying “log does not need rotating (log size is below the ‘size’ threshold)” which means at some point there could be several files sat at 990K for days or weeks on end as there is no daily/weekly/monthly alternative to the size limit of 1M.
  • several logs created by rc.local are not needed on the oct18 image as openhab(2) etc aren’t installed and the likes of mqtt_input has been replaced by emoncms_mqtt.
  • the logrotate log folder is the wrong ownership. It’s set in rc.local as pi:pi but don’t know why, both logs in that folder are root:root.
  • “notifempty” setting seems pointless with the “size 1M” setting, effectively a “not if less than 1M” which I believe covers “empty” too.
  • If a weekly/monthly period was left set then logs with no recent movement but have a rotated file sat there, would get rotated eventually (even though they are small/empty) so the rotated log is removed.
  • some log files like boot.log and debug.log are less likely to reach 1M so they are just hogging RAM, these need to be caught by either a lower size (eg 100K) or eventually by a weekly/monthly rotation.
  • emonhub.logs are too short at >1M, the size setting needs to be at least 2-3M for enonhub. Looking at the logs above we can see emonhub.log has generated over 305K in just 11mins, that’s over 1.5M per hour as seen by the emonhub.log.1 file size, essentially the 1M setting limits us to just 2 hours emonhub.log.
  • The command-line in the cron entry diverts stdout and stderr to logrotate.log using tee, this overwrites previous rotations so is of limited use, tee -a would append the output to the log.
  • copytruncate will block rotation if log partition is nearly full, looking at the rotation of emonhub in the log, it shows the steps where emonhub.log is duplicated to create the rotated file and then the original file is truncated, this can only happen if there is room, in this case it would have needed a t least 1.6M of free space to rotate.
  • logrotate.status doesn’t need to be in the log partition on the RW oct18 image and it could be symlinked to /tmp on the previous RO images rather than straying from the default location.
  • A duplicate entry is reported in the log “error: /etc/logrotate.conf:32 duplicate log entry for /var/log/auth.log”. Line 32 of logrotate.conf is the string of wildcards (emoncms/scripts/logger/logrotate.conf at 10.0.2 · emoncms/emoncms · GitHub) so why does it just error for auth.log, why not boot.log (for example).

@Neil - Sorry to ask again, but could you show us the output of

sudo ls -la /etc/logrotate*
sudo ls -la /etc/cron.*/logrotate

I’ve opened a issue tracker for the issues with the update not aborting if the emonpi repo doesn’t pull successfully

@pb66, I didn’t apply any of the fixes that have been discussed, except for deleting the daemon.log to make space, and I did install log2ram but I removed it before really testing it. I hope that didn’t break anything.

Sorry for the delay, here’s the info you requested.

pi@emonpi:~ $ sudo ls -la /etc/logrotate*
lrwxrwxrwx 1 root root   51 May 22 09:02 /etc/logrotate.conf -> /var/www/html/emoncms/scripts/logger/logrotate.conf

/etc/logrotate.d:
total 48
drwxr-xr-x  2 root root 4096 May 18 07:05 .
drwxr-xr-x 99 root root 4096 May 22 09:02 ..
-rw-r--r--  1 root root  433 Jun  2  2018 apache2
-rw-r--r--  1 root root  173 Sep 13  2017 apt
-rw-r--r--  1 root root   79 Apr 18  2017 aptitude
-rw-r--r--  1 root root  232 Jun  9  2015 dpkg
-rw-r--r--  1 root root  419 Jan 14  2017 lighttpd
-rw-r--r--  1 root root  154 Nov  3  2016 mosquitto
-rw-r--r--  1 root root  802 Jun  7  2017 mysql-server
-rw-r--r--  1 root root  124 Jun 14  2018 redis-server
-rw-r--r--  1 root root  515 Jan 18  2017 rsyslog
-rw-r--r--  1 root root  178 Aug  7  2014 ufw

And.

**pi@emonpi** : **~ $** sudo ls -la /etc/cron.*/logrotate
lrwxrwxrwx 1 root root 46 May 22 09:02 /etc/cron.hourly/logrotate -> /var/www/html/emoncms/scripts/logger/logrotate