Something is breaking installations. On my EmonPi I just did an apt update & apt upgrade (note
not dist-upgrade) and the mariadb service will not start.
[email protected]:/var/log $ sudo apt upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages were automatically installed and are no longer required:
Use 'sudo apt autoremove' to remove them.
The following NEW packages will be installed:
The following packages will be upgraded:
mariadb-client-10.1 mariadb-server mariadb-server-10.1 mariadb-server-core-10.1 python3-gpiozero
5 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 14.9 MB of archives.
After this operation, 1,044 kB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://archive.raspberrypi.org/debian stretch/main armhf python3-colorzero all 1.1 [21.5 kB]
Get:2 http://archive.raspberrypi.org/debian stretch/main armhf python3-gpiozero all 1.5.0 [94.3 kB]
Get:3 http://raspbian.mirror.uk.sargasso.net/raspbian stretch/main armhf libconfig-inifiles-perl all 2.94-1 [53.4 kB]
Get:4 http://raspbian.mirror.uk.sargasso.net/raspbian stretch/main armhf mariadb-server-10.1 armhf 10.1.38-0+deb9u1 [4,687 kB]
Get:5 http://raspbian.mirror.uk.sargasso.net/raspbian stretch/main armhf mariadb-client-10.1 armhf 10.1.38-0+deb9u1 [5,321 kB]
Get:6 http://raspbian.mirror.uk.sargasso.net/raspbian stretch/main armhf mariadb-server-core-10.1 armhf 10.1.38-0+deb9u1 [4,661 kB]
Get:7 http://raspbian.mirror.uk.sargasso.net/raspbian stretch/main armhf mariadb-server all 10.1.38-0+deb9u1 [27.3 kB]
Fetched 14.9 MB in 9s (1,519 kB/s)
Reading changelogs... Done
Preconfiguring packages ...
Selecting previously unselected package libconfig-inifiles-perl.
(Reading database ... 47324 files and directories currently installed.)
Preparing to unpack .../0-libconfig-inifiles-perl_2.94-1_all.deb ...
Unpacking libconfig-inifiles-perl (2.94-1) ...
Preparing to unpack .../1-mariadb-server-10.1_10.1.38-0+deb9u1_armhf.deb ...
Unpacking mariadb-server-10.1 (10.1.38-0+deb9u1) over (10.1.23-9+deb9u1) ...
Preparing to unpack .../2-mariadb-client-10.1_10.1.38-0+deb9u1_armhf.deb ...
Unpacking mariadb-client-10.1 (10.1.38-0+deb9u1) over (10.1.23-9+deb9u1) ...
Preparing to unpack .../3-mariadb-server-core-10.1_10.1.38-0+deb9u1_armhf.deb ...
Unpacking mariadb-server-core-10.1 (10.1.38-0+deb9u1) over (10.1.23-9+deb9u1) ...
Preparing to unpack .../4-mariadb-server_10.1.38-0+deb9u1_all.deb ...
Unpacking mariadb-server (10.1.38-0+deb9u1) over (10.1.23-9+deb9u1) ...
Selecting previously unselected package python3-colorzero.
Preparing to unpack .../5-python3-colorzero_1.1_all.deb ...
Unpacking python3-colorzero (1.1) ...
Preparing to unpack .../6-python3-gpiozero_1.5.0_all.deb ...
Unpacking python3-gpiozero (1.5.0) over (1.4.1) ...
Setting up libconfig-inifiles-perl (2.94-1) ...
Setting up mariadb-server-core-10.1 (10.1.38-0+deb9u1) ...
Processing triggers for systemd (232-25+deb9u11) ...
Processing triggers for man-db (188.8.131.52-2) ...
Setting up python3-colorzero (1.1) ...
Setting up python3-gpiozero (1.5.0) ...
Setting up mariadb-client-10.1 (10.1.38-0+deb9u1) ...
Setting up mariadb-server-10.1 (10.1.38-0+deb9u1) ...
Job for mariadb.service failed because the control process exited with error code.
See "systemctl status mariadb.service" and "journalctl -xe" for details.
mariadb.service couldn't start.
Setting up mariadb-server (10.1.38-0+deb9u1) ...
[email protected]:/var/log $ systemctl status mariadb.service
● mariadb.service - MariaDB 10.1.38 database server
Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Thu 2019-05-02 09:15:34 UTC; 39s ago
Process: 21870 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
Process: 21786 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ] && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited,
Process: 21782 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 21779 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
Main PID: 21870 (code=exited, status=1/FAILURE)
Status: "MariaDB server is down"
May 02 09:15:33 emonpi systemd: Starting MariaDB 10.1.38 database server...
May 02 09:15:34 emonpi mysqld: 2019-05-02 9:15:34 1996406576 [Note] /usr/sbin/mysqld (mysqld 10.1.38-MariaDB-0+deb9u1) starting as process 21870 ...
May 02 09:15:34 emonpi systemd: mariadb.service: Main process exited, code=exited, status=1/FAILURE
May 02 09:15:34 emonpi systemd: Failed to start MariaDB 10.1.38 database server.
May 02 09:15:34 emonpi systemd: mariadb.service: Unit entered failed state.
May 02 09:15:34 emonpi systemd: mariadb.service: Failed with result 'exit-code'.
 I do not understand enough about how mysql is setup and what is modified from standard for the emoncms database to run, to be able to determine what is wrong
Have you had a look at this?
Yes, but that is quite an old post on v10.1.18 - before update is 10.1.23, after 10.1.
My gut feeling is it is a permissions issue related to the data being in the
/home/pi/ folder. I just do not understand the setup and mysql enough to do much.
I noticed the error messages were the same, so figured it might be of some help. Oh well…
I have rebuilt an SD card and did the apt upgrade (all continued to be fine); dist-upgrade caused mariadb to fail to start. However changing the setting to
ProtectHome=false fixes it.
What I have noted is there are 2 different failure modes. As seen here -
Mariadb fails after upgrade the original error seen meant the service failed at this command;
Process: 9092 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld
A new install and a dist-upgrade resulted in the service failing later (note process numbers)
Process: 840 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
Process: 757 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ] && systemc
Process: 752 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 750 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
This error matches my experience. The second failure mode can be fixed by the
did get the first error first time, but cannot replicate the error.
Thanks, Brian, I’ll give it a try. First, I’m going to try to back out of my original upgrade, which, if it works, will allow me to save data for the month of April. Does this look like the way to do it? (Found this on the Inet)
sudo cp boot/ . BOOT20190502
sudo cp -v boot.bak/ . boot
Thanks again for all the time and help.
I have no idea if that will work. I have my doubts as you have changed more than the boot partition.
The other setting to try and override is
ProtectSystem. Set it to
Sorry I cannot test this
Here’s some info on how to go about backing out of an update done via
Unfortunately, it appears it’s not a simple process.
I wonder if it is possible to just roll back the
mariadb-server package to the previous version?
If I read this correctly, the end-result should be the same, or very close to the same.
It’s a downgrade, where you can specify the version number you want to downgrade to.
That does look somewhat complicated, for me anyway. As you have read, my MariaDB isn’t running on the original system which contains all my data, minus a day or so. The new system is now running (thanks to Brian’s fix). Is it possible to somehow move my data from the old system (MariaDB not running) to the new system? I have a backup but it’s a month old.
Unfortunately, I can’t assist you with that issue as I don’t run emonCMS.
That’s one for
If you have a USB sdcard reader then you should be able to just mount the old data partition to the new image and move/copy the data and mysql table over to the new image.
I have the SD card mounted as /media/usb0, usb1, usb2. I’m not sure what to copy where. Everything seems to be running now (with the exception of data for April) so I’m hesitant to try something I’m not sure of. Unless it’s a real easy move then I may just sacrifice a month of data.
Fair enough, leave it as it is then. I was just answering your question. TBH I had thought you had a new blank image, the fact you have imported your backup does complicate things a little since you will be trying to overwrite active data. My suggestion was based on just dragging over the old files to replace the new empty not yet active files. The risks have increased a tad due to the extra complexity of trying to stop updating the files we need to replace, whilst we are replacing them. And if only for a month’s data rather than a complete history, yes I doubt it is worth the trouble.
Many thanks for everything Brian, Bill and Paul. I will resist any future Raspbian upgrades unless I see something on the forum that needs attention. I take it anything that can be upgraded from the Admin page should be ok?
Much appreciated all!
I’ve just tested running
sudo apt-get update & sudo apt-get upgrade on a stock emonSD-30Oct18 and mariadb was still running and not effected by any updates. The issue should only effect
dist-upgrade as reported:
Sorry for the post size, but alas, I decided to do a sudo apt-get update & sudo apt-get dist-upgrade and have hit the original emoncms web connection refused that I posted. (Noted that during upgrade ‘mariadb’ seemed to have issues.)
Can’t connect to database, please verify credentials/configuration in settings.php
Error message: Connection refused
Checked settings.php, looked fine. Also tried restoring file from the default. Data still pushing to remote emoncms. Assuming that…
Maridb will fail to start after running
sudo apt-get dist-upgrade to fix edit:
/lib/systemd/system/mariadb.service the value:
I would not recommend running
sudo apt-get dist-upgrade. Instead, running
sudo apt-get update && sudo apt-get upgrade will update all installed packages and apply any security fixes.
This issue is being discussed:
NO PLEASE DO NOT DO THIS!
It is the reason the fault occurred in the first place.
The correct “fix” is to
sudo systemctl edit mariadb.service
then to that blank file add
and save with ctrl-x, y, enter.
Then reload the service and restart
sudo systemctl daemon-reload
sudo systemctl restart mariadb
This will ensure the same thing doesn’t happen again next time there is an upgrade to the default service unit, regardless of the upgrade command you chose.
Thanks, that’s a much better solution. We will try to roll this change out as a fix to emonSD systems so this issue does not affect users in the future
Just tested, that works great. Thanks
The above creates the file:
With the contents:
I’m assuming the file can just be created directly then the service reloaded? This could easily be implemented in an emonSD update to ensure all users will not be effected by running
I still think
dist-upgrade should be avoided unless specifically required.