Did you keep a copy of the original?
If so can you try that to see if it is the same?
What is that line # If this fails, check debian.conf!? Have you done that?
There maybe a misunderstanding?
The words “# - I put everything in one block and ~~~~~ etc” are part of the original file /etc/logrotate.d/mysql-server.
I (johnbanks) have changed no files.
/etc/logrotate.d/mysql-server is identical in both my emoncms instances.
Re debian.conf …
There are 2 occurences of debian.conf both of which are different and irrelevant to logrotate.
However it might be a typo for /etc/mysql/debian.cnf …
# Automatically generated for Debian scripts. DO NOT TOUCH!
[client]
host = localhost
user = root
password =
socket = /var/run/mysqld/mysqld.sock
[mysql_upgrade]
host = localhost
user = root
password =
socket = /var/run/mysqld/mysqld.sock
basedir = /usr
And from my prior googling, I recall a comment about mariaDB creating the file debian.cnf.
And also that this problem could be a bug in MariaDB.
This maybe the definitive link which I’ve only just found …
[[MDEV-14367] /etc/logrotate.d/mysql-server failure - error: 'Plugin 'unix_socket' is not loaded' - Jira]
Ahhh! ok.
So this is something unique to the emonSD image?? I didn’t twig that these were identical images (or that they were emonSD) due to the hostnames or nicknames.
Actually just checked one of mine and it’s the same
pb66@test2:~$ cat /etc/logrotate.d/mysql-server
# - I put everything in one block and added sharedscripts, so that mysql gets
# flush-logs'd only once.
# Else the binary logs would automatically increase by n times every day.
# - The error log is obsolete, messages go to syslog now.
/var/log/mysql/mysql.log /var/log/mysql/mysql-slow.log /var/log/mysql/mariadb-slow.log /var/log/mysql/error.log {
daily
rotate 7
missingok
create 640 mysql adm
compress
sharedscripts
postrotate
test -x /usr/bin/mysqladmin || exit 0
if [ -f `my_print_defaults --mysqld | grep -m 1 -oP "pid-file=\K.+$"` ]; then
# If this fails, check debian.conf!
mysqladmin --defaults-file=/etc/mysql/debian.cnf --local flush-error-log \
flush-engine-log flush-general-log flush-slow-log
fi
endscript
}
pb66@test2:~$
I think the “debian.conf” is a typo. the file doesn’t exist and command actually uses debian.cnf directly below. I suspect “I”, whoever “I” is, was referring to the content or existence of debian.cnf rather than a similarly named file, but it isn’t totally clear.
pb66@test2:~$ cat /etc/mysql/debian.cnf
cat: /etc/mysql/debian.cnf: Permission denied
pb66@test2:~$ sudo cat /etc/mysql/debian.cnf
# Automatically generated for Debian scripts. DO NOT TOUCH!
[client]
host = localhost
user = root
password =
socket = /var/run/mysqld/mysqld.sock
[mysql_upgrade]
host = localhost
user = root
password =
socket = /var/run/mysqld/mysqld.sock
basedir = /usr
pb66@test2:~$ sudo cat /etc/mysql/debian.conf
cat: /etc/mysql/debian.conf: No such file or directory
pb66@test2:~$
I wonder if the duplicate content is the issue? I know it says “DO NOT TOUCH!” but is it worth trying an edit?
Do you get any more info if you do a dummy rotate from the command line
sudo logrotate --debug /etc/logrotate.conf
below is the relevant bit of the output I got
rotating pattern: /var/log/mysql/mysql.log /var/log/mysql/mysql-slow.log /var/log/mysql/mariadb-slow.log /var/log/mysql/error.log after 1 days (7 rotations)
empty log files are rotated, old logs are removed
considering log /var/log/mysql/mysql.log
log /var/log/mysql/mysql.log does not exist -- skipping
considering log /var/log/mysql/mysql-slow.log
log /var/log/mysql/mysql-slow.log does not exist -- skipping
considering log /var/log/mysql/mariadb-slow.log
log /var/log/mysql/mariadb-slow.log does not exist -- skipping
considering log /var/log/mysql/error.log
Now: 2019-02-08 19:20
Last rotated at 2019-02-08 06:25
log does not need rotating (log has been rotated at 2019-2-8 6:25, that is not day ago yet)
not running postrotate script, since no logs were rotated
also check the exit status once it has completed
pb66@test2:~$ echo $?
0
anything but a 0 is considered “unsuccessful”
[edit]
A bit of googling and it seems “debug,force and verbose” are the option for properly testing logrotate
sudo logrotate -vdf /etc/logrotate.conf
again here’s the relevant bit from my output
rotating pattern: /var/log/mysql/mysql.log /var/log/mysql/mysql-slow.log /var/log/mysql/mariadb-slow.log /var/log/mysql/error.log forced from command line (7 rotations)
empty log files are rotated, old logs are removed
considering log /var/log/mysql/mysql.log
log /var/log/mysql/mysql.log does not exist -- skipping
considering log /var/log/mysql/mysql-slow.log
log /var/log/mysql/mysql-slow.log does not exist -- skipping
considering log /var/log/mysql/mariadb-slow.log
log /var/log/mysql/mariadb-slow.log does not exist -- skipping
considering log /var/log/mysql/error.log
Now: 2019-02-08 19:30
Last rotated at 2019-02-08 19:29
log needs rotating
rotating log /var/log/mysql/error.log, log->rotateCount is 7
dateext suffix '-20190208'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
renaming /var/log/mysql/error.log.7.gz to /var/log/mysql/error.log.8.gz (rotatecount 7, logstart 1, i 7),
renaming /var/log/mysql/error.log.6.gz to /var/log/mysql/error.log.7.gz (rotatecount 7, logstart 1, i 6),
renaming /var/log/mysql/error.log.5.gz to /var/log/mysql/error.log.6.gz (rotatecount 7, logstart 1, i 5),
renaming /var/log/mysql/error.log.4.gz to /var/log/mysql/error.log.5.gz (rotatecount 7, logstart 1, i 4),
renaming /var/log/mysql/error.log.3.gz to /var/log/mysql/error.log.4.gz (rotatecount 7, logstart 1, i 3),
renaming /var/log/mysql/error.log.2.gz to /var/log/mysql/error.log.3.gz (rotatecount 7, logstart 1, i 2),
renaming /var/log/mysql/error.log.1.gz to /var/log/mysql/error.log.2.gz (rotatecount 7, logstart 1, i 1),
renaming /var/log/mysql/error.log.0.gz to /var/log/mysql/error.log.1.gz (rotatecount 7, logstart 1, i 0),
log /var/log/mysql/error.log.8.gz doesn't exist -- won't try to dispose of it
renaming /var/log/mysql/error.log to /var/log/mysql/error.log.1
creating new /var/log/mysql/error.log mode = 0640 uid = 107 gid = 4
running postrotate script
running script with arg /var/log/mysql/mysql.log /var/log/mysql/mysql-slow.log /var/log/mysql/mariadb-slow.log /var/log/mysql/error.log : "
test -x /usr/bin/mysqladmin || exit 0
if [ -f `my_print_defaults --mysqld | grep -m 1 -oP "pid-file=\K.+$"` ]; then
# If this fails, check debian.conf!
mysqladmin --defaults-file=/etc/mysql/debian.cnf --local flush-error-log \
flush-engine-log flush-general-log flush-slow-log
fi
"
compressing log with: /bin/gzip
1 Like
That case you linked ([MDEV-14367] /etc/logrotate.d/mysql-server failure - error: 'Plugin 'unix_socket' is not loaded' - Jira) suggests the unix_socket plugin gets deactivated when you upgrade to mariaDB 10.2 (I’m still at 10.1). Can you check the staus of your plugins? See below, 3rd from the end, mine is “ACTIVE”.
pb66@test2:~$ sudo mysql
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 30255057
Server version: 10.1.26-MariaDB-0+deb9u1 Debian 9.1
Copyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> SHOW PLUGINS;
+-------------------------------+----------+--------------------+---------+---------+
| Name | Status | Type | Library | License |
+-------------------------------+----------+--------------------+---------+---------+
| binlog | ACTIVE | STORAGE ENGINE | NULL | GPL |
| mysql_native_password | ACTIVE | AUTHENTICATION | NULL | GPL |
| mysql_old_password | ACTIVE | AUTHENTICATION | NULL | GPL |
| wsrep | ACTIVE | STORAGE ENGINE | NULL | GPL |
| CSV | ACTIVE | STORAGE ENGINE | NULL | GPL |
| MEMORY | ACTIVE | STORAGE ENGINE | NULL | GPL |
| MyISAM | ACTIVE | STORAGE ENGINE | NULL | GPL |
| MRG_MyISAM | ACTIVE | STORAGE ENGINE | NULL | GPL |
| CLIENT_STATISTICS | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INDEX_STATISTICS | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| TABLE_STATISTICS | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| USER_STATISTICS | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| Aria | ACTIVE | STORAGE ENGINE | NULL | GPL |
| PERFORMANCE_SCHEMA | ACTIVE | STORAGE ENGINE | NULL | GPL |
| SEQUENCE | ACTIVE | STORAGE ENGINE | NULL | GPL |
| InnoDB | ACTIVE | STORAGE ENGINE | NULL | GPL |
| XTRADB_READ_VIEW | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| XTRADB_INTERNAL_HASH_TABLES | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| XTRADB_RSEG | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_TRX | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_LOCKS | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_LOCK_WAITS | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_CMP | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_CMP_RESET | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_CMPMEM | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_CMPMEM_RESET | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_CMP_PER_INDEX | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_CMP_PER_INDEX_RESET | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_BUFFER_PAGE | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_BUFFER_PAGE_LRU | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_BUFFER_POOL_STATS | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_METRICS | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_FT_DEFAULT_STOPWORD | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_FT_DELETED | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_FT_BEING_DELETED | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_FT_CONFIG | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_FT_INDEX_CACHE | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_FT_INDEX_TABLE | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_SYS_TABLES | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_SYS_TABLESTATS | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_SYS_INDEXES | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_SYS_COLUMNS | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_SYS_FIELDS | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_SYS_FOREIGN | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_SYS_FOREIGN_COLS | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_SYS_TABLESPACES | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_SYS_DATAFILES | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_CHANGED_PAGES | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_MUTEXES | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_SYS_SEMAPHORE_WAITS | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_TABLESPACES_ENCRYPTION | ACTIVE | INFORMATION SCHEMA | NULL | BSD |
| INNODB_TABLESPACES_SCRUBBING | ACTIVE | INFORMATION SCHEMA | NULL | BSD |
| CHANGED_PAGE_BITMAPS | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| unix_socket | ACTIVE | AUTHENTICATION | NULL | GPL |
| FEEDBACK | DISABLED | INFORMATION SCHEMA | NULL | GPL |
| partition | ACTIVE | STORAGE ENGINE | NULL | GPL |
+-------------------------------+----------+--------------------+---------+---------+
56 rows in set (0.03 sec)
MariaDB [(none)]>
1 Like
Paul …
Ref yr 2 most recent responses …
I did …
sudo logrotate -vdf /etc/logrotate.conf
… on my plain vanilla emonSD (no MTA) instance and my emonSD with MTA instance.
The results were similar to yours and in both cases, echo $? returned 0.
And it did not prompt a nagging email from my with MTA instance.
Also I established that my version is …
Server version: 10.1.23-MariaDB-9+deb9u1 Raspbian 9.0
Your use of sudo prompted me to think about permissions …
I modified /etc/cron.daily/logrotate to add sudo …
#!/bin/sh
sudo test -x /usr/sbin/logrotate || exit 0
/usr/sbin/logrotate /etc/logrotate.conf
I did …
sudo nano /etc/crontab
… to modify the execution time for cron.daily so I could quickly see the result.
Followed by
sudo systemctl restart cron.service
It worked! …
No nagging email and a ‘clean’ report in syslog as follows …
Feb 8 23:53:01 watchman CRON[24655]: (root) CMD (test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ))
Bottomline - if you add an MTA to an instance running emonSD and don’t want a daily nagging email then just add sudo to /etc/cron.daily/logrotate … as above.
Thank you very much for yr time on this - it’s been a valuable learning experience for me.
Regards
That is odd as the cron is run as root user, so adding sudo should not make any difference.
The fact you used sudo to open the crontab to edit means it is the root crontab, to be run as root. So why adding sudo should alter anything is a mystery.
Could it be that the log rotation wasn’t due so the post-rotate and impending message were not fired off?
My /etc/cron.daily/logrotate is the same as yours too (although I do not use mail notifications) so this shouldn’t be a emonSD only issue, besides I thought the emonSD (previously?) used hourly log rotations??? I hope we get a build guide for the current image one day!
Paul …
I spoke too soon & too confidently …
This AM I received a nagging email and this in syslog …
Feb 9 06:25:02 watchman liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="311" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Feb 9 06:25:02 watchman sSMTP[28163]: Creating SSL connection to host
Feb 9 06:25:03 watchman sSMTP[28163]: 220 smtp.gmail.com ESMTP z3sm5155032wmi.32 - gsmtp
Feb 9 06:25:03 watchman sSMTP[28163]: EHLO watchman
Feb 9 06:25:03 watchman sSMTP[28163]: 250 SMTPUTF8
Feb 9 06:25:03 watchman sSMTP[28163]: STARTTLS
Feb 9 06:25:03 watchman sSMTP[28163]: 220 2.0.0 Ready to start TLS
Feb 9 06:25:03 watchman sSMTP[28163]: SSL connection using RSA_AES_128_CBC_SHA1
Feb 9 06:25:03 watchman sSMTP[28163]: EHLO watchman
Feb 9 06:25:03 watchman sSMTP[28163]: 250 SMTPUTF8
Feb 9 06:25:03 watchman sSMTP[28163]: AUTH LOGIN
Feb 9 06:25:03 watchman sSMTP[28163]: 334 VXNlcm5hbWU6
Feb 9 06:25:03 watchman sSMTP[28163]: am9obmJhbmtzMDE0N0BnbWFpbC5jb20=
Feb 9 06:25:03 watchman sSMTP[28163]: 334 UGFzc3dvcmQ6
Feb 9 06:25:03 watchman sSMTP[28163]: bWdld3h1eG13b3RzcXpwYw==
Feb 9 06:25:03 watchman sSMTP[28163]: 235 2.7.0 Accepted
Feb 9 06:25:03 watchman sSMTP[28163]: MAIL FROM:<root@watchman>
Feb 9 06:25:03 watchman sSMTP[28163]: 250 2.1.0 OK z3sm5155032wmi.32 - gsmtp
Feb 9 06:25:03 watchman sSMTP[28163]: RCPT TO:<[email protected]>
Feb 9 06:25:03 watchman sSMTP[28163]: 250 2.1.5 OK z3sm5155032wmi.32 - gsmtp
Feb 9 06:25:03 watchman sSMTP[28163]: DATA
Feb 9 06:25:03 watchman sSMTP[28163]: 354 Go ahead z3sm5155032wmi.32 - gsmtp
Feb 9 06:25:03 watchman sSMTP[28163]: Received: by watchman (sSMTP sendmail emulation); Sat, 09 Feb 2019 06:25:02 +0000
Feb 9 06:25:03 watchman sSMTP[28163]: Date: Sat, 09 Feb 2019 06:25:02 +0000
Feb 9 06:25:03 watchman sSMTP[28163]: From: root (Cron Daemon)
Feb 9 06:25:03 watchman sSMTP[28163]: To: root
Feb 9 06:25:03 watchman sSMTP[28163]: Subject: Cron <root@watchman> test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
Feb 9 06:25:03 watchman sSMTP[28163]: MIME-Version: 1.0
Feb 9 06:25:03 watchman sSMTP[28163]: Content-Type: text/plain; charset=UTF-8
Feb 9 06:25:03 watchman sSMTP[28163]: Content-Transfer-Encoding: 8bit
Feb 9 06:25:03 watchman sSMTP[28163]: X-Cron-Env: <SHELL=/bin/sh>
Feb 9 06:25:03 watchman sSMTP[28163]: X-Cron-Env: <PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin>
Feb 9 06:25:03 watchman sSMTP[28163]: X-Cron-Env: <HOME=/root>
Feb 9 06:25:03 watchman sSMTP[28163]: X-Cron-Env: <LOGNAME=root>
Feb 9 06:25:03 watchman sSMTP[28163]:
Feb 9 06:25:03 watchman sSMTP[28163]: /etc/cron.daily/logrotate:
Feb 9 06:25:03 watchman sSMTP[28163]: #007mysqladmin: connect to server at 'localhost' failed
Feb 9 06:25:03 watchman sSMTP[28163]: error: 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2 "No such file or directory")'
Feb 9 06:25:03 watchman sSMTP[28163]: Check that mysqld is running and that the socket: '/var/run/mysqld/mysqld.sock' exists!
Feb 9 06:25:03 watchman sSMTP[28163]: error: error running shared postrotate script for '/var/log/mysql/mysql.log /var/log/mysql/mysql-slow.log /var/log/mysql/mariadb-slow.log /var/log/mysql/error.log '
Feb 9 06:25:03 watchman sSMTP[28163]: run-parts: /etc/cron.daily/logrotate exited with return code 1
Feb 9 06:25:04 watchman sSMTP[28163]: .
Feb 9 06:25:05 watchman sSMTP[28163]: 250 2.0.0 OK 1549693505 z3sm5155032wmi.32 - gsmtp
Feb 9 06:25:05 watchman sSMTP[28163]: QUIT
Feb 9 06:25:05 watchman sSMTP[28163]: 221 2.0.0 closing connection z3sm5155032wmi.32 - gsmtp
Feb 9 06:25:05 watchman sSMTP[28163]: Sent mail for root@watchman (221 2.0.0 closing connection z3sm5155032wmi.32 - gsmtp) uid=0 username=root outbytes=1212
It was a few mins before 9:14, so I amended /etc/crontab to run cron.daily at 9:14 (rather than 6:25) …
pi@watchman:/etc/cron.d $ cat /etc/crontab
# /etc/crontab: system-wide crontab
# Unlike any other crontab you don't have to run the `crontab'
# command to install the new version when you edit this file
# and files in /etc/cron.d. These files also have username fields,
# that none of the other crontabs do.
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
# m h dom mon dow user command
17 * * * * root cd / && run-parts --report /etc/cron.hourly
14 9 * * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
47 6 * * 7 root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly )
52 6 1 * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly )
By 9:16, I had received NO nagging email and syslog revealed …
Feb 9 09:14:02 watchman CRON[29798]: (root) CMD (test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ))
Apparently there was no problem at 9:14. So clearly there’s something additional happening at 6:25 other than just cron.daily running.
Looks like yr most recent comments are correct. I’m currently at a loss …
Hi John, the problem (as I understand it) seems to be that there is an error triggered by the rotating of the mysql log files. I’m unclear what that error is exactly, but the link you supplied seems feasible and at least worth exploring so if you could test to see if the “unix_socket” plugin is active as per my previous post that would be good.
Can you also check to see if /var/run/mysqld/mysqld.sock exists eg
pb66@test2:~$ ls -la /var/run/mysqld/mysqld.sock
srwxrwxrwx 1 mysql mysql 0 Sep 29 11:52 /var/run/mysqld/mysqld.sock
pb66@test2:~$
The probable reason you are not seeing an issue at 9.15 is the same reason you did not see a problem last night (after adding sudo) where basically the mysql logs are not due for rotation because 24hrs has not passed, if you leave the cron as 9.15, I suspect what normally happens at 6.25 will occur at 9.15 tomorrow because >24hrs will have passed, therefore the rotation (and post-rotation script) will run. Any attempt to run the rotation before 6.25 tomorrow will skip rotation (and post-rotation) as it is not due therefore no fault is triggered or registered.
Paul …
Thanks for the 24 hours explanation.
My unix_socket is ACTIVE per below …
| unix_socket | ACTIVE | AUTHENTICATION | NULL | GPL |
However, unlike you, I do not have the required /var/run/mysqld/mysqld.sock
pi@watchman:/var/run/mysqld $ ls -la /var/run/mysqld/mysqld.sock
ls: cannot access '/var/run/mysqld/mysqld.sock': No such file or directory
Lucky googling - I stumbled across this …
EDIT: A really good way of finding the location of the socket file is:
netstat -ln | grep mysql
unix 2 [ ACC ] STREAM LISTENING 22418 /var/lib/mysql/mysql.sock
So I tried it …
pi@watchman:/etc/cron.monthly $ netstat -ln | grep mysql
which gave the following result …
unix 2 [ ACC ] STREAM LISTENING 13879 /home/pi/data/mysql/mysql.sock
So I hunted further - see third from the bottom …
pi@watchman:/etc/cron.monthly $ cd /home/pi/data/mysql
pi@watchman:~/data/mysql $ ls -al
total 110776
drwxrwxrwx 5 mysql mysql 4096 Feb 7 02:24 .
drwxrwxrwx 10 pi pi 4096 Feb 7 02:15 ..
-rw-rw---- 1 mysql mysql 16384 Feb 7 01:15 aria_log.00000001
-rw-rw---- 1 mysql mysql 52 Feb 7 01:15 aria_log_control
-rw-r--r-- 1 mysql mysql 0 Aug 17 15:27 debian-10.1.flag
drwx------ 2 mysql mysql 4096 Feb 6 23:54 emoncms
-rw-rw---- 1 mysql mysql 12582912 Feb 7 02:24 ibdata1
-rw-rw---- 1 mysql mysql 50331648 Feb 7 02:24 ib_logfile0
-rw-rw---- 1 mysql mysql 50331648 Aug 17 15:28 ib_logfile1
-rw-rw---- 1 mysql mysql 0 Aug 17 15:29 multi-master.info
drwx------ 2 mysql mysql 4096 Aug 17 15:28 mysql
srwxrwxrwx 1 mysql mysql 0 Feb 7 02:24 mysql.sock
drwx------ 2 mysql mysql 4096 Aug 17 15:28 performance_schema
-rw-rw---- 1 mysql mysql 24576 Feb 7 02:24 tc.log
In my ignorance I tried to sudo cp from there to /var/run/mysqld/mysqld.sock - but that’s not allowed.
Is there a way to link both?
Oh my word! I’m not sure it should be there, I’m guessing this is due to the previous images being read-only! Again, I wish we had a build guide, this raises all sorts of questions for me! Like is the NEW image actually a NEW image or the old one un-read-only’ed and updated? Surely if the image was built from scratch to not be read-only, the files that previously needed moving to get write access would be left alone? I understand why the mysql tables might be moved it is far from essential to have everything in a separate “writable” /data partition when the whole image is now writable.
I have long maintained that even when we have a RO image (which is no longer the case) any files/folders that are moved should strictly be “moved and symlinked” so that they can also still be found at their original locations, it avoids situations like this.
I do not know if adding a symlink like this will definitely work but worth a try
sudo ln -s /home/pi/data/mysql /var/lib/mysql
actually before you do try the above, just check to see if anything is still at /var/lib/mysql with
sudo ls -la /var/lib/mysql
should be No such file or directory
and after the ln command above it, the same ls command repeated should return the symlink.
Then if you add a trailing forward slash like so
sudo ls -la /var/lib/mysql/
you should see the same output you got from ls within the ~data/mysql folder. If you do get that, the symlink was succesful and you should get a lay in tomorrow (no email notifications at the crack of dawn) fingers crossed.
Paul …
I’ve followed yr instructions and the results were successful/as you predicted …
pi@watchman:/var/run/mysqld $ sudo ls -la /var/lib/mysql
ls: cannot access '/var/lib/mysql': No such file or directory
pi@watchman:/var/run/mysqld $ sudo ln -s /home/pi/data/mysql /var/lib/mysql
pi@watchman:/var/run/mysqld $ sudo ls -la /var/lib/mysql
lrwxrwxrwx 1 root root 19 Feb 9 18:04 /var/lib/mysql -> /home/pi/data/mysql
pi@watchman:/var/run/mysqld $ sudo ls -la /var/lib/mysql/
total 110776
drwxrwxrwx 5 mysql mysql 4096 Feb 7 02:24 .
drwxrwxrwx 10 pi pi 4096 Feb 7 02:15 ..
-rw-rw---- 1 mysql mysql 16384 Feb 7 01:15 aria_log.00000001
-rw-rw---- 1 mysql mysql 52 Feb 7 01:15 aria_log_control
-rw-r--r-- 1 mysql mysql 0 Aug 17 15:27 debian-10.1.flag
drwx------ 2 mysql mysql 4096 Feb 6 23:54 emoncms
-rw-rw---- 1 mysql mysql 12582912 Feb 7 02:24 ibdata1
-rw-rw---- 1 mysql mysql 50331648 Feb 7 02:24 ib_logfile0
-rw-rw---- 1 mysql mysql 50331648 Aug 17 15:28 ib_logfile1
-rw-rw---- 1 mysql mysql 0 Aug 17 15:29 multi-master.info
drwx------ 2 mysql mysql 4096 Aug 17 15:28 mysql
srwxrwxrwx 1 mysql mysql 0 Feb 7 02:24 mysql.sock
drwx------ 2 mysql mysql 4096 Aug 17 15:28 performance_schema
-rw-rw---- 1 mysql mysql 24576 Feb 7 02:24 tc.log
I will let you know what happens tomorrow.
Assuming all goes to plan then - have we identified a flaw in the current emonSD image? - will it be fixed? - will the fix be incorporated in the imminent new 9.9.6 image?
Many many thx
/var/lib/ or did you mean /var/run/?
Paul …
Unfortunately I received a nagging email again this AM with the same as ever text …
/etc/cron.daily/logrotate:
mysqladmin: connect to server at 'localhost' failed
error: 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2 "No such file or directory")'
Check that mysqld is running and that the socket: '/var/run/mysqld/mysqld.sock' exists!
error: error running shared postrotate script for '/var/log/mysql/mysql.log /var/log/mysql/mysql-slow.log /var/log/mysql/mariadb-slow.log /var/log/mysql/error.log '
run-parts: /etc/cron.daily/logrotate exited with return code 1
I did …
pi@watchman:~ $ ls -la /var/run/mysqld
total 4
drwxr-xr-x 2 mysql root 60 Feb 7 02:24 .
drwxr-xr-x 26 root root 940 Feb 10 11:51 ..
-rw-rw---- 1 mysql mysql 4 Feb 7 02:24 mysqld.pid
So the required /var/run/mysqld/mysqld.sock does not exist.
Is it possible just to link the required mysqld.sock file from /home/pi/data/mysql?
Is @borpin’s question above relevant?
Thx
The content of /var/run/ is managed by the system and softwares, whilst we could add a link, I suspect it would disappear on reboot.
In a way, yes, it was when I saw his post and thinking about a reply that I realized it would have been good advice to suggest you reboot or restart mysql.
Here’s my thinking and reasoning, I’m in the middle of something else right now so do not have the time to research for confirmation of my suspicions, but here goes.
The files in /var/run are runtime settings that are not usually tampered with by the user. The fact that the file was found in ~/data made me realize this was most likely due to the whole mysql folder previously being moved to a writable location for the RO FS. (We do not know for sure what is or isn’t done on this image as there is no build guide. hence I suggested checking the original location). I believe that when the log rotete is due it triggers an error due to the missing file in /var/run and that stems from mysql not finding a file at /var/lib as I would assume that if the whole mysql folder was still in /var/lib things might work as intended. Adding the symlink will hopefully make it work as intended. However I didn’t think it through enough and possibly the file in /var/run is only created at start up and a reboot or service restart is required ( I favor the reboot as offhand I do not know how the tempfiles are managed in stretch so I’m not absolutly sure a service restart will be enough without further research).
Having said “reboot over restart” there is another element that may crop up and that some services start up before symlinks can be read. I’m not saying that’s the case here, but I do not know for sure it isn’t, so there is a slim risk that something might not go to plan, something that I would test thoroughly for if I was putting it into an image or guide. So accepting there may still be hiccups, a reboot should set you up for an alert free morning tomorrow.
I see no good reason why the mysql folder isn’t where it should be, so I do not know why it was moved in this new image, or was it just never put back if this isn’t a entirely fresh image???
IMO the correct permanent fix for this (assuming we are on the right trail) is to move the mysql folder out of ~/data back to /var/lib, The preferred method, if we want the emoncms “mysql” in the ~/data folder is to move (and symlink) just the emoncms tables folder from within the mysql folder to ~/data. This symlink will not be used early enough in the boot up to be a concern as it will not be used until emoncms is used (log in or update input/query feed etc). It is not necessary (or wise) to have all the global mysql supporting indexes and files in the ~/data folder, these are needed at bootup and for other things that might use mysql. Bearing in mind the ~/data is itself a symlink, I suspect we should be ok.
I have no idea how well received any suggestion to do that to the new image would be.
Obviously we do not know for sure this is the issue yet, but the fact that the error message is looking in the “normal place” but the standard mysql fs hierarchy has been tinkered with, is too big a coincidence to ignore and IMO undoing the unnecessary and dubious changes (even if just via a symlink) is the best place to start.
We have to assume a standard install will most likely work and rather than adding more links and files and settings etc, a step backwards might be cleaner and leaner. If for some reason I’m not aware of (as we have no build guide) it is essential the mysql folder stays in ~/data AND the symlink back to it original position doesn’t work, we will need to review this position of course.
Paul …
I’ve done a reboot and then …
pi@watchman:~ $ cd /var/run/mysqld
pi@watchman:/var/run/mysqld $ ls -la
total 4
drwxr-xr-x 2 mysql root 60 Feb 10 14:37 .
drwxr-xr-x 26 root root 940 Feb 10 14:38 ..
-rw-rw---- 1 mysql mysql 4 Feb 10 14:37 mysqld.pid
The sock file is still not there.
However yr originally suggested link still seems OK …
pi@watchman:/var/run/mysqld $ sudo ls -la /var/lib/mysql/
total 110776
drwxrwxrwx 5 mysql mysql 4096 Feb 10 14:37 .
drwxrwxrwx 10 pi pi 4096 Feb 7 02:15 ..
-rw-rw---- 1 mysql mysql 16384 Feb 10 14:37 aria_log.00000001
-rw-rw---- 1 mysql mysql 52 Feb 10 14:37 aria_log_control
-rw-r--r-- 1 mysql mysql 0 Aug 17 15:27 debian-10.1.flag
drwx------ 2 mysql mysql 4096 Feb 6 23:54 emoncms
-rw-rw---- 1 mysql mysql 12582912 Feb 10 14:37 ibdata1
-rw-rw---- 1 mysql mysql 50331648 Feb 10 14:37 ib_logfile0
-rw-rw---- 1 mysql mysql 50331648 Aug 17 15:28 ib_logfile1
-rw-rw---- 1 mysql mysql 0 Aug 17 15:29 multi-master.info
drwx------ 2 mysql mysql 4096 Aug 17 15:28 mysql
srwxrwxrwx 1 mysql mysql 0 Feb 10 14:37 mysql.sock
drwx------ 2 mysql mysql 4096 Aug 17 15:28 performance_schema
-rw-rw---- 1 mysql mysql 24576 Feb 10 14:37 tc.log
Let’s see what tomorrow AM brings
Thx
I’m thinking your gonna be disappointed.
When I have more time (and a quiet moment) I will look deeper, for now we could add the symlink manually as you or Brian suggested, or bite the bullet and move the folder back to it’s original location.
Maybe the files in /var/run/mysql are actually added a install time and have gotten deleted in the read-only implementation and it’s subsequent undoing, it’s all guesswork without the build guide.
Maybe I need to blow the dust off my emonPi and download an image to see whats what.
I just took a look at my stock setup and the sock file is NOT in the mysql folder in /var/lib. Confused by what is going on I looked into the old RO emonSD setup and I think I’ve found the culprit!
see https://github.com/emoncms/emoncms/blob/master/docs/RaspberryPi/read-only.md#move-mysql-database
There is a bit that tells you to edit the mysql setting for the sock file location in /etc/mysql/my.cnf. That is currently socket=/home/pi/data/mysql/mysql.sock as follows
Under [mysqld]:
datadir=/home/pi/data/mysql
socket=/home/pi/data/mysql/mysql.sock
Under [client]:
port=3306
socket=/home/pi/data/mysql/mysql.sock
HOWEVER !!!
My /etc/mysql/my.cnf is as follows
# The MariaDB configuration file
#
# The MariaDB/MySQL tools read configuration files in the following order:
# 1. "/etc/mysql/mariadb.cnf" (this file) to set global defaults,
# 2. "/etc/mysql/conf.d/*.cnf" to set global options.
# 3. "/etc/mysql/mariadb.conf.d/*.cnf" to set MariaDB-only options.
# 4. "~/.my.cnf" to set user-specific options.
#
# If the same option is defined multiple times, the last one will apply.
#
# One can use all long options that the program supports.
# Run program with --help to get a list of available options and with
# --print-defaults to see which it would actually understand and use.
#
# This group is read both both by the client and the server
# use it for options that affect everything
#
[client-server]
# Import all .cnf files from configuration directory
!includedir /etc/mysql/conf.d/
!includedir /etc/mysql/mariadb.conf.d/
so off hand I do not know what yours will be or what it should be.
I’m now wondering if you should just give in and edit the command that is looking in the original location to look in the RO emonSD location instead, or you could edit those lines in /etc/mysql/my.cnf (assuming that is found where it should be). The latter would probably be for the best, in case something else goes looking for it, editing the command will not help other programs. Ultimately I think it needs properly reverting, but that has other ramifications too, like editing the emoncms settings.php and being in conflict with any future emonpi updates might cause an issue too.
this is turning out to be a right can of worms, if this “new emonSD image” was new, I would expect it too have some reference to MariaDB too, or at least differ to the mysql install in some way. This is looking like a conversion from mysql to mariadb rather than a fresh install. After some searching i found my sock setting
pb66@test2:~$ cat /etc/mysql/mariadb.conf.d/50-server.cnf
#
# These groups are read by MariaDB server.
# Use it for options that only the server (but not clients) should see
#
# See the examples of server my.cnf files in /usr/share/mysql/
#
# this is read by the standalone daemon and embedded servers
[server]
# this is only for the mysqld standalone daemon
[mysqld]
#
# * Basic Settings
#
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
lc-messages-dir = /usr/share/mysql
skip-external-locking
# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
bind-address = 127.0.0.1
#
# * Fine Tuning
#
key_buffer_size = 16M
max_allowed_packet = 16M
thread_stack = 192K
thread_cache_size = 8
# This replaces the startup script and checks MyISAM tables if needed
# the first time they are touched
myisam_recover_options = BACKUP
#max_connections = 100
#table_cache = 64
#thread_concurrency = 10
#
# * Query Cache Configuration
#
query_cache_limit = 1M
query_cache_size = 16M
#
# * Logging and Replication
#
# Both location gets rotated by the cronjob.
# Be aware that this log type is a performance killer.
# As of 5.1 you can enable the log at runtime!
#general_log_file = /var/log/mysql/mysql.log
#general_log = 1
#
# Error log - should be very few entries.
#
log_error = /var/log/mysql/error.log
#
# Enable the slow query log to see queries with especially long duration
#slow_query_log_file = /var/log/mysql/mariadb-slow.log
#long_query_time = 10
#log_slow_rate_limit = 1000
#log_slow_verbosity = query_plan
#log-queries-not-using-indexes
#
# The following can be used as easy to replay backup logs or for replication.
# note: if you are setting up a replication slave, see README.Debian about
# other settings you may need to change.
#server-id = 1
#log_bin = /var/log/mysql/mysql-bin.log
expire_logs_days = 10
max_binlog_size = 100M
#binlog_do_db = include_database_name
#binlog_ignore_db = exclude_database_name
#
# * InnoDB
#
# InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/.
# Read the manual for more InnoDB related options. There are many!
#
# * Security Features
#
# Read the manual, too, if you want chroot!
# chroot = /var/lib/mysql/
#
# For generating SSL certificates you can use for example the GUI tool "tinyca".
#
# ssl-ca=/etc/mysql/cacert.pem
# ssl-cert=/etc/mysql/server-cert.pem
# ssl-key=/etc/mysql/server-key.pem
#
# Accept only connections using the latest and most secure TLS protocol version.
# ..when MariaDB is compiled with OpenSSL:
# ssl-cipher=TLSv1.2
# ..when MariaDB is compiled with YaSSL (default in Debian):
# ssl=on
#
# * Character sets
#
# MySQL/MariaDB default is Latin1, but in Debian we rather default to the full
# utf8 4-byte character set. See also client.cnf
#
character-set-server = utf8mb4
collation-server = utf8mb4_general_ci
#
# * Unix socket authentication plugin is built-in since 10.0.22-6
#
# Needed so the root database user can authenticate without a password but
# only when running as the unix root user.
#
# Also available for other users if required.
# See https://mariadb.com/kb/en/unix_socket-authentication-plugin/
# this is only for embedded server
[embedded]
# This group is only read by MariaDB servers, not by MySQL.
# If you use the same .cnf file for MySQL and MariaDB,
# you can put MariaDB-only options here
[mariadb]
# This group is only read by MariaDB-10.1 servers.
# If you use the same .cnf file for MariaDB of different versions,
# use this group for options that older servers don't understand
[mariadb-10.1]
pb66@test2:~$
[edit]
By edit the command, i mean edit the /etc/mysql/debian.cnf file to use the same/home/pi/data/mysql/mysql.sock` path, or yes if you want to minimise the straying from emonSD norm (what ever that might be), just adding a symlink as mentioned earlier should do it I guess (assuming it sticks)
sudo ln -s /home/pi/data/mysql/mysql.sock /var/run/mysql/mysql.sock
it might be the easier and least disruptive move, even if not “the right way to do it”.
No at the time I did mean lib but I now think symlinking to /var/run is the easy way out!
1 Like
Paul …
Yesterday I created the link to the sock file (using mysqld rather than mysql) …
sudo ln -s /home/pi/data/mysql/mysql.sock /var/run/mysqld/mysqld.sock
It tried a reboot but it did not survive as you predicted and so I then used crontab -e to add …
@ reboot sleep 10 && sudo ln -s /home/pi/data/mysql/mysql.sock /var/run/mysqld/mysqld.sock
The link was there after I tried a reboot and this AM I did NOT receive a nagging email.
So this appears to be a pragmatic simple solution.
cat /var/log/syslog revealed …
Feb 11 06:25:03 watchman liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="324" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Feb 11 06:25:03 watchman liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="324" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
I don’t know why the message is repeated. In the past this would have been followed by a string of error reports.
Should I now remove the first link you suggested (between /home/pi/data/mysql and /var/lib/mysql)??
If so, should I use unlink /var/lib/mysql ?
As an aside - my other running instance of the emonSD Oct 2018 image (does not have an MTA installed and does not have the linked sock file) revealed the following in syslog …
Feb 11 06:25:02 emonpi-node-13 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="331" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Feb 11 06:25:02 emonpi-node-13 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="331" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Feb 11 06:25:03 emonpi-node-13 CRON[18960]: (CRON) info (No MTA installed, discarding output)
Clearly it still has the logrotate problem
Many thx
Well spotted, I missed the typo, the emonsd specific setting edits in /etc/mysql/my.cnf from /var/run/mysqld/mysqld.sock to /home/pi/data/mysql/mysql.sock should strictly speaking been /home/pi/data/mysql/mysqld.sock but it doesn’t matter much on the whole scale of issues!
Yeah, it was just a test, it assumed no reboots or power cuts overnight. But what you did was fine. We now know with a degree of certainty what went wrong.
My recommendation would be to edit /etc/mysql/my.cnf and put it back to original to avoid the cron job.
Since this was a manual edit rather than an automated edit (via update, git repo or install script/command) it should not get changed at any time (except in the process of fixing this issue perhaps). That will ensure the file is created at reboot and managed in the usual way.
You can if you wish to try and stay close to the emonSD image, but IMO it should have been linked when it was moved. The link isn’t “wrong” or likely to cause an error, it simply means the mysql folder can also be seen at it’s original location.
As expected, yes. Every emonPi/emonBase/emonSD will have this same error, it just isn’t noticed as an MTA isn’t generally used.
@pb66, I think you are right that it is the included files that are causing the problem. I noticed there is also a socket statement in /etc/mysql/mariadb.conf.d/50-mysqld_safe.cnf
From the command line a mysqld --help --verbose gives a value for socket=/home/pi/data/mysql/mysql.sock so the defaults are being overwritten by the values in the my.cnf in some circumstances (clearly).
I wonder if the socket statement should be added to the [client-server] section and possibly a [mysqld_safe] section as well in the my.cnf probably after the includes statements.