Log Rotate Errors

Oh my word! I’m not sure it should be there, I’m guessing this is due to the previous images being read-only! Again, I wish we had a build guide, this raises all sorts of questions for me! Like is the NEW image actually a NEW image or the old one un-read-only’ed and updated? Surely if the image was built from scratch to not be read-only, the files that previously needed moving to get write access would be left alone? I understand why the mysql tables might be moved it is far from essential to have everything in a separate “writable” /data partition when the whole image is now writable.

I have long maintained that even when we have a RO image (which is no longer the case) any files/folders that are moved should strictly be “moved and symlinked” so that they can also still be found at their original locations, it avoids situations like this.

I do not know if adding a symlink like this will definitely work but worth a try

sudo ln -s /home/pi/data/mysql /var/lib/mysql

actually before you do try the above, just check to see if anything is still at /var/lib/mysql with

sudo ls -la /var/lib/mysql

should be No such file or directory

and after the ln command above it, the same ls command repeated should return the symlink.

Then if you add a trailing forward slash like so

sudo ls -la /var/lib/mysql/

you should see the same output you got from ls within the ~data/mysql folder. If you do get that, the symlink was succesful and you should get a lay in tomorrow (no email notifications at the crack of dawn) fingers crossed.

Paul …

I’ve followed yr instructions and the results were successful/as you predicted …

pi@watchman:/var/run/mysqld $ sudo ls -la /var/lib/mysql
ls: cannot access '/var/lib/mysql': No such file or directory

pi@watchman:/var/run/mysqld $ sudo ln -s /home/pi/data/mysql /var/lib/mysql

pi@watchman:/var/run/mysqld $ sudo ls -la /var/lib/mysql
lrwxrwxrwx 1 root root 19 Feb  9 18:04 /var/lib/mysql -> /home/pi/data/mysql

pi@watchman:/var/run/mysqld $ sudo ls -la /var/lib/mysql/
total 110776
drwxrwxrwx  5 mysql mysql     4096 Feb  7 02:24 .
drwxrwxrwx 10 pi    pi        4096 Feb  7 02:15 ..
-rw-rw----  1 mysql mysql    16384 Feb  7 01:15 aria_log.00000001
-rw-rw----  1 mysql mysql       52 Feb  7 01:15 aria_log_control
-rw-r--r--  1 mysql mysql        0 Aug 17 15:27 debian-10.1.flag
drwx------  2 mysql mysql     4096 Feb  6 23:54 emoncms
-rw-rw----  1 mysql mysql 12582912 Feb  7 02:24 ibdata1
-rw-rw----  1 mysql mysql 50331648 Feb  7 02:24 ib_logfile0
-rw-rw----  1 mysql mysql 50331648 Aug 17 15:28 ib_logfile1
-rw-rw----  1 mysql mysql        0 Aug 17 15:29 multi-master.info
drwx------  2 mysql mysql     4096 Aug 17 15:28 mysql
srwxrwxrwx  1 mysql mysql        0 Feb  7 02:24 mysql.sock
drwx------  2 mysql mysql     4096 Aug 17 15:28 performance_schema
-rw-rw----  1 mysql mysql    24576 Feb  7 02:24 tc.log

I will let you know what happens tomorrow.

Assuming all goes to plan then - have we identified a flaw in the current emonSD image? - will it be fixed? - will the fix be incorporated in the imminent new 9.9.6 image?

Many many thx

/var/lib/ or did you mean /var/run/?

Paul …

Unfortunately I received a nagging email again this AM with the same as ever text …

/etc/cron.daily/logrotate:

mysqladmin: connect to server at 'localhost' failed

error: 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2 "No such file or directory")'

Check that mysqld is running and that the socket: '/var/run/mysqld/mysqld.sock' exists!

error: error running shared postrotate script for '/var/log/mysql/mysql.log /var/log/mysql/mysql-slow.log /var/log/mysql/mariadb-slow.log /var/log/mysql/error.log '

run-parts: /etc/cron.daily/logrotate exited with return code 1

I did …

pi@watchman:~ $ ls -la /var/run/mysqld
total 4
drwxr-xr-x  2 mysql root   60 Feb  7 02:24 .
drwxr-xr-x 26 root  root  940 Feb 10 11:51 ..
-rw-rw----  1 mysql mysql   4 Feb  7 02:24 mysqld.pid

So the required /var/run/mysqld/mysqld.sock does not exist.

Is it possible just to link the required mysqld.sock file from /home/pi/data/mysql?

Is @borpin’s question above relevant?

Thx

The content of /var/run/ is managed by the system and softwares, whilst we could add a link, I suspect it would disappear on reboot.

In a way, yes, it was when I saw his post and thinking about a reply that I realized it would have been good advice to suggest you reboot or restart mysql.

Here’s my thinking and reasoning, I’m in the middle of something else right now so do not have the time to research for confirmation of my suspicions, but here goes.

The files in /var/run are runtime settings that are not usually tampered with by the user. The fact that the file was found in ~/data made me realize this was most likely due to the whole mysql folder previously being moved to a writable location for the RO FS. (We do not know for sure what is or isn’t done on this image as there is no build guide. hence I suggested checking the original location). I believe that when the log rotete is due it triggers an error due to the missing file in /var/run and that stems from mysql not finding a file at /var/lib as I would assume that if the whole mysql folder was still in /var/lib things might work as intended. Adding the symlink will hopefully make it work as intended. However I didn’t think it through enough and possibly the file in /var/run is only created at start up and a reboot or service restart is required ( I favor the reboot as offhand I do not know how the tempfiles are managed in stretch so I’m not absolutly sure a service restart will be enough without further research).

Having said “reboot over restart” there is another element that may crop up and that some services start up before symlinks can be read. I’m not saying that’s the case here, but I do not know for sure it isn’t, so there is a slim risk that something might not go to plan, something that I would test thoroughly for if I was putting it into an image or guide. So accepting there may still be hiccups, a reboot should set you up for an alert free morning tomorrow.

I see no good reason why the mysql folder isn’t where it should be, so I do not know why it was moved in this new image, or was it just never put back if this isn’t a entirely fresh image???

IMO the correct permanent fix for this (assuming we are on the right trail) is to move the mysql folder out of ~/data back to /var/lib, The preferred method, if we want the emoncms “mysql” in the ~/data folder is to move (and symlink) just the emoncms tables folder from within the mysql folder to ~/data. This symlink will not be used early enough in the boot up to be a concern as it will not be used until emoncms is used (log in or update input/query feed etc). It is not necessary (or wise) to have all the global mysql supporting indexes and files in the ~/data folder, these are needed at bootup and for other things that might use mysql. Bearing in mind the ~/data is itself a symlink, I suspect we should be ok.

I have no idea how well received any suggestion to do that to the new image would be.

Obviously we do not know for sure this is the issue yet, but the fact that the error message is looking in the “normal place” but the standard mysql fs hierarchy has been tinkered with, is too big a coincidence to ignore and IMO undoing the unnecessary and dubious changes (even if just via a symlink) is the best place to start.

We have to assume a standard install will most likely work and rather than adding more links and files and settings etc, a step backwards might be cleaner and leaner. If for some reason I’m not aware of (as we have no build guide) it is essential the mysql folder stays in ~/data AND the symlink back to it original position doesn’t work, we will need to review this position of course.

Paul …

I’ve done a reboot and then …

pi@watchman:~ $ cd /var/run/mysqld
pi@watchman:/var/run/mysqld $ ls -la
total 4
drwxr-xr-x  2 mysql root   60 Feb 10 14:37 .
drwxr-xr-x 26 root  root  940 Feb 10 14:38 ..
-rw-rw----  1 mysql mysql   4 Feb 10 14:37 mysqld.pid

The sock file is still not there.

However yr originally suggested link still seems OK …

pi@watchman:/var/run/mysqld $ sudo ls -la /var/lib/mysql/
total 110776
drwxrwxrwx  5 mysql mysql     4096 Feb 10 14:37 .
drwxrwxrwx 10 pi    pi        4096 Feb  7 02:15 ..
-rw-rw----  1 mysql mysql    16384 Feb 10 14:37 aria_log.00000001
-rw-rw----  1 mysql mysql       52 Feb 10 14:37 aria_log_control
-rw-r--r--  1 mysql mysql        0 Aug 17 15:27 debian-10.1.flag
drwx------  2 mysql mysql     4096 Feb  6 23:54 emoncms
-rw-rw----  1 mysql mysql 12582912 Feb 10 14:37 ibdata1
-rw-rw----  1 mysql mysql 50331648 Feb 10 14:37 ib_logfile0
-rw-rw----  1 mysql mysql 50331648 Aug 17 15:28 ib_logfile1
-rw-rw----  1 mysql mysql        0 Aug 17 15:29 multi-master.info
drwx------  2 mysql mysql     4096 Aug 17 15:28 mysql
srwxrwxrwx  1 mysql mysql        0 Feb 10 14:37 mysql.sock
drwx------  2 mysql mysql     4096 Aug 17 15:28 performance_schema
-rw-rw----  1 mysql mysql    24576 Feb 10 14:37 tc.log

Let’s see what tomorrow AM brings

Thx

I’m thinking your gonna be disappointed.

When I have more time (and a quiet moment) I will look deeper, for now we could add the symlink manually as you or Brian suggested, or bite the bullet and move the folder back to it’s original location.

Maybe the files in /var/run/mysql are actually added a install time and have gotten deleted in the read-only implementation and it’s subsequent undoing, it’s all guesswork without the build guide.

Maybe I need to blow the dust off my emonPi and download an image to see whats what.

I just took a look at my stock setup and the sock file is NOT in the mysql folder in /var/lib. Confused by what is going on I looked into the old RO emonSD setup and I think I’ve found the culprit!

see https://github.com/emoncms/emoncms/blob/master/docs/RaspberryPi/read-only.md#move-mysql-database

There is a bit that tells you to edit the mysql setting for the sock file location in /etc/mysql/my.cnf. That is currently socket=/home/pi/data/mysql/mysql.sock as follows

Under [mysqld]:
datadir=/home/pi/data/mysql
socket=/home/pi/data/mysql/mysql.sock
Under [client]:
port=3306
socket=/home/pi/data/mysql/mysql.sock

HOWEVER !!!
My /etc/mysql/my.cnf is as follows

# The MariaDB configuration file
#
# The MariaDB/MySQL tools read configuration files in the following order:
# 1. "/etc/mysql/mariadb.cnf" (this file) to set global defaults,
# 2. "/etc/mysql/conf.d/*.cnf" to set global options.
# 3. "/etc/mysql/mariadb.conf.d/*.cnf" to set MariaDB-only options.
# 4. "~/.my.cnf" to set user-specific options.
#
# If the same option is defined multiple times, the last one will apply.
#
# One can use all long options that the program supports.
# Run program with --help to get a list of available options and with
# --print-defaults to see which it would actually understand and use.

#
# This group is read both both by the client and the server
# use it for options that affect everything
#
[client-server]

# Import all .cnf files from configuration directory
!includedir /etc/mysql/conf.d/
!includedir /etc/mysql/mariadb.conf.d/

so off hand I do not know what yours will be or what it should be.

I’m now wondering if you should just give in and edit the command that is looking in the original location to look in the RO emonSD location instead, or you could edit those lines in /etc/mysql/my.cnf (assuming that is found where it should be). The latter would probably be for the best, in case something else goes looking for it, editing the command will not help other programs. Ultimately I think it needs properly reverting, but that has other ramifications too, like editing the emoncms settings.php and being in conflict with any future emonpi updates might cause an issue too.

this is turning out to be a right can of worms, if this “new emonSD image” was new, I would expect it too have some reference to MariaDB too, or at least differ to the mysql install in some way. This is looking like a conversion from mysql to mariadb rather than a fresh install. After some searching i found my sock setting

pb66@test2:~$ cat /etc/mysql/mariadb.conf.d/50-server.cnf
#
# These groups are read by MariaDB server.
# Use it for options that only the server (but not clients) should see
#
# See the examples of server my.cnf files in /usr/share/mysql/
#

# this is read by the standalone daemon and embedded servers
[server]

# this is only for the mysqld standalone daemon
[mysqld]

#
# * Basic Settings
#
user            = mysql
pid-file        = /var/run/mysqld/mysqld.pid
socket          = /var/run/mysqld/mysqld.sock
port            = 3306
basedir         = /usr
datadir         = /var/lib/mysql
tmpdir          = /tmp
lc-messages-dir = /usr/share/mysql
skip-external-locking

# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
bind-address            = 127.0.0.1

#
# * Fine Tuning
#
key_buffer_size         = 16M
max_allowed_packet      = 16M
thread_stack            = 192K
thread_cache_size       = 8
# This replaces the startup script and checks MyISAM tables if needed
# the first time they are touched
myisam_recover_options  = BACKUP
#max_connections        = 100
#table_cache            = 64
#thread_concurrency     = 10

#
# * Query Cache Configuration
#
query_cache_limit       = 1M
query_cache_size        = 16M

#
# * Logging and Replication
#
# Both location gets rotated by the cronjob.
# Be aware that this log type is a performance killer.
# As of 5.1 you can enable the log at runtime!
#general_log_file        = /var/log/mysql/mysql.log
#general_log             = 1
#
# Error log - should be very few entries.
#
log_error = /var/log/mysql/error.log
#
# Enable the slow query log to see queries with especially long duration
#slow_query_log_file    = /var/log/mysql/mariadb-slow.log
#long_query_time = 10
#log_slow_rate_limit    = 1000
#log_slow_verbosity     = query_plan
#log-queries-not-using-indexes
#
# The following can be used as easy to replay backup logs or for replication.
# note: if you are setting up a replication slave, see README.Debian about
#       other settings you may need to change.
#server-id              = 1
#log_bin                        = /var/log/mysql/mysql-bin.log
expire_logs_days        = 10
max_binlog_size   = 100M
#binlog_do_db           = include_database_name
#binlog_ignore_db       = exclude_database_name

#
# * InnoDB
#
# InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/.
# Read the manual for more InnoDB related options. There are many!

#
# * Security Features
#
# Read the manual, too, if you want chroot!
# chroot = /var/lib/mysql/
#
# For generating SSL certificates you can use for example the GUI tool "tinyca".
#
# ssl-ca=/etc/mysql/cacert.pem
# ssl-cert=/etc/mysql/server-cert.pem
# ssl-key=/etc/mysql/server-key.pem
#
# Accept only connections using the latest and most secure TLS protocol version.
# ..when MariaDB is compiled with OpenSSL:
# ssl-cipher=TLSv1.2
# ..when MariaDB is compiled with YaSSL (default in Debian):
# ssl=on

#
# * Character sets
#
# MySQL/MariaDB default is Latin1, but in Debian we rather default to the full
# utf8 4-byte character set. See also client.cnf
#
character-set-server  = utf8mb4
collation-server      = utf8mb4_general_ci

#
# * Unix socket authentication plugin is built-in since 10.0.22-6
#
# Needed so the root database user can authenticate without a password but
# only when running as the unix root user.
#
# Also available for other users if required.
# See https://mariadb.com/kb/en/unix_socket-authentication-plugin/

# this is only for embedded server
[embedded]

# This group is only read by MariaDB servers, not by MySQL.
# If you use the same .cnf file for MySQL and MariaDB,
# you can put MariaDB-only options here
[mariadb]

# This group is only read by MariaDB-10.1 servers.
# If you use the same .cnf file for MariaDB of different versions,
# use this group for options that older servers don't understand
[mariadb-10.1]
pb66@test2:~$

[edit]
By edit the command, i mean edit the /etc/mysql/debian.cnf file to use the same/home/pi/data/mysql/mysql.sock` path, or yes if you want to minimise the straying from emonSD norm (what ever that might be), just adding a symlink as mentioned earlier should do it I guess (assuming it sticks)

sudo ln -s /home/pi/data/mysql/mysql.sock /var/run/mysql/mysql.sock

it might be the easier and least disruptive move, even if not “the right way to do it”.

No at the time I did mean lib but I now think symlinking to /var/run is the easy way out!

1 Like

Paul …

Yesterday I created the link to the sock file (using mysqld rather than mysql) …

sudo ln -s /home/pi/data/mysql/mysql.sock /var/run/mysqld/mysqld.sock

It tried a reboot but it did not survive as you predicted and so I then used crontab -e to add …

@ reboot sleep 10 && sudo ln -s /home/pi/data/mysql/mysql.sock /var/run/mysqld/mysqld.sock

The link was there after I tried a reboot and this AM I did NOT receive a nagging email.
So this appears to be a pragmatic simple solution.

cat /var/log/syslog revealed …

Feb 11 06:25:03 watchman liblogging-stdlog:  [origin software="rsyslogd" swVersion="8.24.0" x-pid="324" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Feb 11 06:25:03 watchman liblogging-stdlog:  [origin software="rsyslogd" swVersion="8.24.0" x-pid="324" x-info="http://www.rsyslog.com"] rsyslogd was HUPed

I don’t know why the message is repeated. In the past this would have been followed by a string of error reports.

Should I now remove the first link you suggested (between /home/pi/data/mysql and /var/lib/mysql)??
If so, should I use unlink /var/lib/mysql ?

As an aside - my other running instance of the emonSD Oct 2018 image (does not have an MTA installed and does not have the linked sock file) revealed the following in syslog …

Feb 11 06:25:02 emonpi-node-13 liblogging-stdlog:  [origin software="rsyslogd" swVersion="8.24.0" x-pid="331" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Feb 11 06:25:02 emonpi-node-13 liblogging-stdlog:  [origin software="rsyslogd" swVersion="8.24.0" x-pid="331" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Feb 11 06:25:03 emonpi-node-13 CRON[18960]: (CRON) info (No MTA installed, discarding output)

Clearly it still has the logrotate problem

Many thx

Well spotted, I missed the typo, the emonsd specific setting edits in /etc/mysql/my.cnf from /var/run/mysqld/mysqld.sock to /home/pi/data/mysql/mysql.sock should strictly speaking been /home/pi/data/mysql/mysqld.sock but it doesn’t matter much on the whole scale of issues!

Yeah, it was just a test, it assumed no reboots or power cuts overnight. But what you did was fine. We now know with a degree of certainty what went wrong.

My recommendation would be to edit /etc/mysql/my.cnf and put it back to original to avoid the cron job.

Since this was a manual edit rather than an automated edit (via update, git repo or install script/command) it should not get changed at any time (except in the process of fixing this issue perhaps). That will ensure the file is created at reboot and managed in the usual way.

You can if you wish to try and stay close to the emonSD image, but IMO it should have been linked when it was moved. The link isn’t “wrong” or likely to cause an error, it simply means the mysql folder can also be seen at it’s original location.

As expected, yes. Every emonPi/emonBase/emonSD will have this same error, it just isn’t noticed as an MTA isn’t generally used.

@pb66, I think you are right that it is the included files that are causing the problem. I noticed there is also a socket statement in /etc/mysql/mariadb.conf.d/50-mysqld_safe.cnf

From the command line a mysqld --help --verbose gives a value for socket=/home/pi/data/mysql/mysql.sock so the defaults are being overwritten by the values in the my.cnf in some circumstances (clearly).

I wonder if the socket statement should be added to the [client-server] section and possibly a [mysqld_safe] section as well in the my.cnf probably after the includes statements.