WARNING - Do not do apt-get dist-upgrade

Emoncms
1

@glyn.hudson @TrystanLea

Something is breaking installations. On my EmonPi I just did an apt update & apt upgrade (note not dist-upgrade) and the mariadb service will not start.

pi@emonpi:/var/log $ sudo apt upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages were automatically installed and are no longer required:
  libarchive13 liblzo2-2
Use 'sudo apt autoremove' to remove them.
The following NEW packages will be installed:
  libconfig-inifiles-perl python3-colorzero
The following packages will be upgraded:
  mariadb-client-10.1 mariadb-server mariadb-server-10.1 mariadb-server-core-10.1 python3-gpiozero
5 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 14.9 MB of archives.
After this operation, 1,044 kB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://archive.raspberrypi.org/debian stretch/main armhf python3-colorzero all 1.1 [21.5 kB]
Get:2 http://archive.raspberrypi.org/debian stretch/main armhf python3-gpiozero all 1.5.0 [94.3 kB]
Get:3 http://raspbian.mirror.uk.sargasso.net/raspbian stretch/main armhf libconfig-inifiles-perl all 2.94-1 [53.4 kB]
Get:4 http://raspbian.mirror.uk.sargasso.net/raspbian stretch/main armhf mariadb-server-10.1 armhf 10.1.38-0+deb9u1 [4,687 kB]
Get:5 http://raspbian.mirror.uk.sargasso.net/raspbian stretch/main armhf mariadb-client-10.1 armhf 10.1.38-0+deb9u1 [5,321 kB]
Get:6 http://raspbian.mirror.uk.sargasso.net/raspbian stretch/main armhf mariadb-server-core-10.1 armhf 10.1.38-0+deb9u1 [4,661 kB]
Get:7 http://raspbian.mirror.uk.sargasso.net/raspbian stretch/main armhf mariadb-server all 10.1.38-0+deb9u1 [27.3 kB]
Fetched 14.9 MB in 9s (1,519 kB/s)
Reading changelogs... Done
Preconfiguring packages ...
Selecting previously unselected package libconfig-inifiles-perl.
(Reading database ... 47324 files and directories currently installed.)
Preparing to unpack .../0-libconfig-inifiles-perl_2.94-1_all.deb ...
Unpacking libconfig-inifiles-perl (2.94-1) ...
Preparing to unpack .../1-mariadb-server-10.1_10.1.38-0+deb9u1_armhf.deb ...
Unpacking mariadb-server-10.1 (10.1.38-0+deb9u1) over (10.1.23-9+deb9u1) ...
Preparing to unpack .../2-mariadb-client-10.1_10.1.38-0+deb9u1_armhf.deb ...
Unpacking mariadb-client-10.1 (10.1.38-0+deb9u1) over (10.1.23-9+deb9u1) ...
Preparing to unpack .../3-mariadb-server-core-10.1_10.1.38-0+deb9u1_armhf.deb ...
Unpacking mariadb-server-core-10.1 (10.1.38-0+deb9u1) over (10.1.23-9+deb9u1) ...
Preparing to unpack .../4-mariadb-server_10.1.38-0+deb9u1_all.deb ...
Unpacking mariadb-server (10.1.38-0+deb9u1) over (10.1.23-9+deb9u1) ...
Selecting previously unselected package python3-colorzero.
Preparing to unpack .../5-python3-colorzero_1.1_all.deb ...
Unpacking python3-colorzero (1.1) ...
Preparing to unpack .../6-python3-gpiozero_1.5.0_all.deb ...
Unpacking python3-gpiozero (1.5.0) over (1.4.1) ...
Setting up libconfig-inifiles-perl (2.94-1) ...
Setting up mariadb-server-core-10.1 (10.1.38-0+deb9u1) ...
Processing triggers for systemd (232-25+deb9u11) ...
Processing triggers for man-db (2.7.6.1-2) ...
Setting up python3-colorzero (1.1) ...
Setting up python3-gpiozero (1.5.0) ...
Setting up mariadb-client-10.1 (10.1.38-0+deb9u1) ...
Setting up mariadb-server-10.1 (10.1.38-0+deb9u1) ...
Job for mariadb.service failed because the control process exited with error code.
See "systemctl status mariadb.service" and "journalctl -xe" for details.
mariadb.service couldn't start.
Setting up mariadb-server (10.1.38-0+deb9u1) ...
pi@emonpi:/var/log $ systemctl status mariadb.service
ā— mariadb.service - MariaDB 10.1.38 database server
   Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Thu 2019-05-02 09:15:34 UTC; 39s ago
     Docs: man:mysqld(8)
           https://mariadb.com/kb/en/library/systemd/
  Process: 21870 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
  Process: 21786 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited,
  Process: 21782 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
  Process: 21779 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
 Main PID: 21870 (code=exited, status=1/FAILURE)
   Status: "MariaDB server is down"

May 02 09:15:33 emonpi systemd[1]: Starting MariaDB 10.1.38 database server...
May 02 09:15:34 emonpi mysqld[21870]: 2019-05-02  9:15:34 1996406576 [Note] /usr/sbin/mysqld (mysqld 10.1.38-MariaDB-0+deb9u1) starting as process 21870 ...
May 02 09:15:34 emonpi systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
May 02 09:15:34 emonpi systemd[1]: Failed to start MariaDB 10.1.38 database server.
May 02 09:15:34 emonpi systemd[1]: mariadb.service: Unit entered failed state.
May 02 09:15:34 emonpi systemd[1]: mariadb.service: Failed with result 'exit-code'.

[edit] I do not understand enough about how mysql is setup and what is modified from standard for the emoncms database to run, to be able to determine what is wrong :frowning:.

1 Like
2

Hi Brian,

Have you had a look at this?

3

Yes, but that is quite an old post on v10.1.18 - before update is 10.1.23, after 10.1.3038.

My gut feeling is it is a permissions issue related to the data being in the /home/pi/ folder. I just do not understand the setup and mysql enough to do much.

4

I noticed the error messages were the same, so figured it might be of some help. Oh wellā€¦

1 Like
5

I have rebuilt an SD card and did the apt upgrade (all continued to be fine); dist-upgrade caused mariadb to fail to start. However changing the setting to ProtectHome=false fixes it.

What I have noted is there are 2 different failure modes. As seen here - Mariadb fails after upgrade the original error seen meant the service failed at this command;

Process: 9092 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld

A new install and a dist-upgrade resulted in the service failing later (note process numbers)

  Process: 840 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
  Process: 757 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemc
  Process: 752 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
  Process: 750 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)

This error matches my experience. The second failure mode can be fixed by the ProtectHome fix.

I did get the first error first time, but cannot replicate the error.

6

Thanks, Brian, Iā€™ll give it a try. First, Iā€™m going to try to back out of my original upgrade, which, if it works, will allow me to save data for the month of April. Does this look like the way to do it? (Found this on the Inet)

cd /
sudo cp boot/. BOOT20190502
sudo cp -v boot.bak/. boot

Thanks again for all the time and help.

Bob

7

I have no idea if that will work. I have my doubts as you have changed more than the boot partition.

The other setting to try and override is ProtectSystem. Set it to off

ProtectSystem=off

Sorry I cannot test this :smile:.

8

Hereā€™s some info on how to go about backing out of an update done via apt-get

Unfortunately, it appears itā€™s not a simple process.

HTH, YMMV.

1 Like
9

I wonder if it is possible to just roll back the mariadb-server package to the previous version?

10

If I read this correctly, the end-result should be the same, or very close to the same.
Itā€™s a downgrade, where you can specify the version number you want to downgrade to.

11

Bill,
That does look somewhat complicated, for me anyway. As you have read, my MariaDB isnā€™t running on the original system which contains all my data, minus a day or so. The new system is now running (thanks to Brianā€™s fix). Is it possible to somehow move my data from the old system (MariaDB not running) to the new system? I have a backup but itā€™s a month old.
Thanks,

Bob

12

Hi Bob,

Unfortunately, I canā€™t assist you with that issue as I donā€™t run emonCMS.

Thatā€™s one for @glyn.hudson

73 OM,

KR6K

13

If you have a USB sdcard reader then you should be able to just mount the old data partition to the new image and move/copy the data and mysql table over to the new image.

1 Like
14

Paul,
I have the SD card mounted as /media/usb0, usb1, usb2. Iā€™m not sure what to copy where. Everything seems to be running now (with the exception of data for April) so Iā€™m hesitant to try something Iā€™m not sure of. Unless itā€™s a real easy move then I may just sacrifice a month of data.

Bob

15

Fair enough, leave it as it is then. I was just answering your question. TBH I had thought you had a new blank image, the fact you have imported your backup does complicate things a little since you will be trying to overwrite active data. My suggestion was based on just dragging over the old files to replace the new empty not yet active files. The risks have increased a tad due to the extra complexity of trying to stop updating the files we need to replace, whilst we are replacing them. And if only for a monthā€™s data rather than a complete history, yes I doubt it is worth the trouble.

16

Many thanks for everything Brian, Bill and Paul. I will resist any future Raspbian upgrades unless I see something on the forum that needs attention. I take it anything that can be upgraded from the Admin page should be ok?
Much appreciated all!

Bob

17

Iā€™ve just tested running sudo apt-get update & sudo apt-get upgrade on a stock emonSD-30Oct18 and mariadb was still running and not effected by any updates. The issue should only effect dist-upgrade as reported:

Maridb will fail to start after running sudo apt-get dist-upgrade to fix edit:

/lib/systemd/system/mariadb.service the value:

ProtectHome=true

to

ProtectHome=false

I would not recommend running sudo apt-get dist-upgrade. Instead, running sudo apt-get update && sudo apt-get upgrade will update all installed packages and apply any security fixes.

This issue is being discussed:

18

NO PLEASE DO NOT DO THIS!

It is the reason the fault occurred in the first place.

The correct ā€œfixā€ is to

sudo systemctl edit mariadb.service

then to that blank file add

[Service]
ProtectHome=false

and save with ctrl-x, y, enter.

Then reload the service and restart

sudo systemctl daemon-reload
sudo systemctl restart mariadb

This will ensure the same thing doesnā€™t happen again next time there is an upgrade to the default service unit, regardless of the upgrade command you chose.

2 Likes
19

Thanks, thatā€™s a much better solution. We will try to roll this change out as a fix to emonSD systems so this issue does not affect users in the future :+1:

20

Just tested, that works great. Thanks

The above creates the file:

/etc/systemd/system/mariadb.service.d/override.conf

With the contents:

[Service]
ProtectHome=false

Iā€™m assuming the file can just be created directly then the service reloaded? This could easily be implemented in an emonSD update to ensure all users will not be effected by running dist-upgrdae.

I still think dist-upgrade should be avoided unless specifically required.