So is the line: * * * * * /home/pi/emonpi/service-runner >> /var/log/service-runner.log 2>&1 suppose to be in the root crontab sudo crontab -l or in the user pi crontab crontab -l ?
Ah well spotted, I had forgotten that the update script runs as root, therefore crontab -l in the the update script is actually listing the root crontab > sudo crontab -l. This explains a lotā¦
I will get onto this tomorrow.
Iāve also noticed the log file issue this evening, this is a top priority to sort out. I thought we had nailed it with @Andy_Taylor 's help. Obviously not quite there yet
What log is filling up now? auth.log? (Iām not getting affected by this because my Pi is currently running on batteries - so its rebooted every 12h when I change the battery packā¦
Assuming it is auth.log filling up, Iāll do some digging on that in a moment, but it appears it could be due to the bizarre duplication in both the āpiā userās crontab and the ārootā userās crontab; both of these have the same ā* * * * * /home/pi/emonpi/service-runner >> /var/log/service-runner.log 2>&1ā entry more than once.
@glyn.hudson point me at the update script and Iāll take a look if that helpsā¦
It was /var/log/apache2/error.log that contained 27Mb of ādivide by zeroā errors.
Since applying your PR the errors have stopped, however it would be good to figure out why the log was not rotated.
Iāve also noticed that even the log files that seem to be properly rotated /var/log/emonhub/emonhub.log are greater than 1Mb in size which should be the limit set by logrotate.conf. e.g.
ah, it seems my /etc/logrotate.conf was not owned by root which was causing logrotate to exit. Iāve added a check to the update script to set this file ownership to root.
However /var/log/apache2/error.log is still not being considered, I canāt work out why /var/log/*/*.log should include it :
Handling 1 logs
rotating pattern: /var/log/*.log /var/log/*/*.log /var/log/syslog /var/log/messages 1048576 bytes (2 rotations)
empty log files are not rotated, old logs are removed
considering log /var/log/auth.log
log needs rotating
considering log /var/log/daemon.log
log does not need rotating
considering log /var/log/emoncms.log
log does not need rotating
considering log /var/log/kern.log
log does not need rotating
considering log /var/log/mqtt_input.log
log does not need rotating
considering log /var/log/mysql.log
log does not need rotating
considering log /var/log/service-runner.log
log does not need rotating
considering log /var/log/ufw.log
log does not need rotating
considering log /var/log/apache2/error.log
log does not need rotating
considering log /var/log/emonhub/emonhub.log
log does not need rotating
considering log /var/log/emonpilcd/emonpilcd.log
log does not need rotating
considering log /var/log/logrotate/logrotate.log
log does not need rotating
considering log /var/log/mysql/error.log
log does not need rotating
considering log /var/log/openhab/2016_05_10.request.log
log does not need rotating
considering log /var/log/openhab/events.log
log does not need rotating
considering log /var/log/openhab/openhab.log
log does not need rotating
considering log /var/log/redis/redis-server.log
log does not need rotating
considering log /var/log/syslog
log needs rotating
considering log /var/log/messages
log does not need rotating
rotating log /var/log/auth.log, log->rotateCount is 2
dateext suffix '-20160510'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
renaming /var/log/auth.log.2 to /var/log/auth.log.3 (rotatecount 2, logstart 1, i 2),
old log /var/log/auth.log.2 does not exist
renaming /var/log/auth.log.1 to /var/log/auth.log.2 (rotatecount 2, logstart 1, i 1),
old log /var/log/auth.log.1 does not exist
renaming /var/log/auth.log.0 to /var/log/auth.log.1 (rotatecount 2, logstart 1, i 0),
old log /var/log/auth.log.0 does not exist
log /var/log/auth.log.3 doesn't exist -- won't try to dispose of it
copying /var/log/auth.log to /var/log/auth.log.1
truncating /var/log/auth.log
rotating log /var/log/syslog, log->rotateCount is 2
dateext suffix '-20160510'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
renaming /var/log/syslog.2 to /var/log/syslog.3 (rotatecount 2, logstart 1, i 2),
old log /var/log/syslog.2 does not exist
renaming /var/log/syslog.1 to /var/log/syslog.2 (rotatecount 2, logstart 1, i 1),
old log /var/log/syslog.1 does not exist
renaming /var/log/syslog.0 to /var/log/syslog.1 (rotatecount 2, logstart 1, i 0),
old log /var/log/syslog.0 does not exist
log /var/log/syslog.3 doesn't exist -- won't try to dispose of it
copying /var/log/syslog to /var/log/syslog.1
truncating /var/log/syslog
I have just committed to the ādevā emonPi branch some changes to the update script that will look for and remove any root crontab service runner entries.
Changes have been merged into master. Please test to see if this fixed duplicate service runner entries for you.
After running the update the Pi should be restarted to kill the old service runner thatās mistakenly running as root. After reboot you should EUID in the update log displaying 1000 which is the Pi user (non root) EUID.
Remember emonhub will rotate itās own lofiles at ~5mb which is most probably what you saw here, rather than logrotate doing it at the wrong time/size it just wasnāt doing it at all and emonhub eventually did it.
Also doesnāt apache2 set up itās own logrotate configuration at /etc/logrotate.d/apache2 if so that may well be overriding or at least conflicting with your global config
Ah right, that explains things. Logrotate was not running at all due to the permissions error I mentioned above. Iāve added a check to the emonPi update script (currently in the dev branch) to set /etc/logrotate.conf ownership to root which was the cause of of logrotate not running. https://github.com/openenergymonitor/emonpi/blob/dev/emoncmsupdate#L179
In addition to this; Log files can still spill over 1Mb IF they do so in-between the times that log rotate runs (hourly if I remember right).
So - if there is much being written to the log it can still reach 100Mb before the hour and it gets rotated - make sense?
@pb66 the emonpi version of the logrotate.conf excludes those add-ins, so now Glyn has fixed the permissions error that will take care of all the logs properly.
Really? What does $ crontab - l return? Nothing? I would be very surprised if so, since the update would not work without service runner. If no service runner is present in the pi user crontab the update script should add it in.
What do you see after another reboot and another update? Thanks so much for helping me test this.
@glyn.hudson - emonPi update wonāt run from the webpage. Iām not sure what I need to run from the command line. Is it update from the /home/pi/emonpi directory without the sudo?
EDIT: OK - I think I figured it out. I did a touch /tmp/emonpiupdate and got /home/pi/emonpi/update to run. The service-runner line in now in the pi user crontab.
Thatās what I suspected. If you manually add the service runner entry to Pi crontab using $ crontab - e update will then run from the Web page. The question is what removed it?! Try and manually add it then run update to see if it disappears. I canāt recreate this, on my system the Pi crontab service runner entry always stays as it should.
Correct. This is the problem. Looks like Iām not quite there with this. Iāll do some more investigation tomorrow and figure out whatās going on. Thanks again. EUID 1000 is what it should be running as, the Pi (non root) user.