MariaDB does not start after update on November the 5th: Assertion failure in file

Updated November 22, 2019 07:17

Symptoms

MariaDB fails to start with:

# systemctl start mariadb
Job for mariadb.service failed because a fatal signal was delivered causing the control process to dump core. See "systemctl status mariadb.service" and "journalctl -xe" for details.
# systemctl status mariadb.service
● mariadb.service - MariaDB 10.2.28 database server
Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/mariadb.service.d
└─migrated-from-my.cnf-settings.conf
Active: activating (auto-restart) (Result: core-dump) since Wed 2019-11-06 10:58:12 CST; 4s ago
MariaDB 10.1-10.3 has been recently updated to version 10.1.42 / 10.2.28 / 10.3.19:
- on CentOS/RHEL-based distributions
  
  # grep -i Maria /var/log/yum.log
  Nov 06 03:46:20 Updated: MariaDB-common-10.1.42-1.el7.centos.x86_64
  Nov 06 03:46:22 Updated: MariaDB-client-10.1.42-1.el7.centos.x86_64
  Nov 06 03:46:38 Updated: MariaDB-server-10.1.42-1.el7.centos.x86_64
  Nov 06 03:46:38 Updated: MariaDB-shared-10.1.42-1.el7.centos.x86_64
- on Debian/Ubuntu-based distributions
  
  # grep mariadb /var/log/dpkg.log | grep upgrade
  2019-11-06 06:25:34 upgrade mariadb-common:all 10.2.27+maria~xenial 10.2.28+maria~xenial
  2019-11-06 06:25:34 upgrade mariadb-client-core-10.2:amd64 10.2.27+maria~xenial 10.2.28+maria~xenial
  2019-11-06 06:26:05 upgrade mariadb-server-10.2:amd64 10.2.27+maria~xenial 10.2.28+maria~xenial
  2019-11-06 06:26:08 upgrade mariadb-client-10.2:amd64 10.2.27+maria~xenial 10.2.28+maria~xenial
  2019-11-06 06:26:09 upgrade mariadb-server-core-10.2:amd64 10.2.27+maria~xenial 10.2.28+maria~xenial
The following entries can be found in /var/log/mariadb.log or output of "journalctl -u mariadb"

CONFIG_TEXT: InnoDB: Assertion failure in file /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.X.XX/storage/innobase/dict/dict0dict.cc line 1467
InnoDB: Failing assertion: table->can_be_evicted
Plesk interface is not available with the following error in a web-browser:

PLESK_INFO: SQLSTATE[HY000] [2002] No such file or directory

Cause

The issue is caused by the latest MariaDB update and confirmed as MariaDB bug MDEV-20987. The bug has already been fixed in MariaDB versions 10.1.43, 10.2.29, 10.3.20, 10.4.10, which have been released on November the 8th.

This issue affects the following versions of the MariaDB SQL Server:

10.1.42
10.2.28
10.3.19
10.4.9 (Not supported by Plesk)

Resolution

The next packages update is already available, so if the server has been affected, install the lastest MariaDB updates:

Log into Plesk;
Go to Tools & Settings > System Updates;
Search for "MariaDB"
Choose all the packages and click Update:

Alternatively, updates can be installed via CLI:

For CentOS/RHEL

Connect to the server via SSH.
Update MariaDB packages:

# yum -y update MariaDB-server MariaDB-common MariaDB-shared MariaDB-client MariaDB-compat MariaDB-devel

In case the issue persists apply either of solutions from the article How to fix InnoDB corruption cases for the MySQL databases on Plesk for Linux?

For Debian/Ubuntu

Connect to the server via SSH.
Update MariaDB packages:

# apt-get update MariaDB-server MariaDB-common MariaDB-shared MariaDB-client MariaDB-compat MariaDB-devel

In case the issue persists apply either of solutions from the article How to fix InnoDB corruption cases for the MySQL databases on Plesk for Linux?

Comments

22 comments

Mahipal Singh November 06, 2019 06:01

This upgrade caused MySQL crashed and downtime of website. Please test such upgrades at your end before posting them online.

0

Comment actions Permalink
Maxim Krasikov November 06, 2019 06:10

Hello @Mahipal Singh,

Please let me clarify that the MariaDB update was released by the official MariaDB vendor and not by Plesk.
Until MariaDB releases an official patch, please apply the workaround described in the article.

0

Comment actions Permalink
Mahipal Singh November 06, 2019 06:15

Hi Maxim,

Understood the upgrade was released via MariaDB but don't you think it is Plesk team responsibility to test the upgrade first and the solution was posted after 3 hours of upgrading MariaDB rpm package

0

Comment actions Permalink
Maxim Krasikov November 06, 2019 06:47

Hi @Mahipal Singh,

MariaDB update was performed by yum utility which updated the packages from non-OS vendor repositories.

We reacted accordingly after were informed about the occurred behavior.

0

Comment actions Permalink
Roman Bendik November 06, 2019 10:55 (Edited November 07, 2019 09:37)

I think it automatically updated!?, and I lose a few hours on solution!

Information for others is the solution to this error:

The error in Plesk: ERR [panel] SQLSTATE[HY000] [2002] No such file or directory, Can't connect to local MySQL server through socket

Abstract.php:144

0

Comment actions Permalink
Roy Stultiens November 06, 2019 11:11

this caused quite some issues and angry phonecalls this morning...

Did you stop the update from rolling out automatically via Plesk (or which steps can we take to prevent this)?

0

Comment actions Permalink
Jen Kegels November 06, 2019 13:12 (Edited November 06, 2019 13:12)

Yes, thank you! I've been restoring backups all morning...

0

Comment actions Permalink
Learning Curve November 06, 2019 13:15

Either we are quite lucky... or this bug is not applicable to every installation (depending on the setup etc)

In our case; Obsidian / MariaDB 10.3.* There's no trace of the bug... currently!

# grep mariadb /var/log/dpkg.log | grep upgrade
2019-11-06 05:14:15 upgrade mariadb-common:all 1:10.3.18+maria~bionic 1:10.3.19+maria~bionic
2019-11-06 05:14:15 upgrade mariadb-server:all 1:10.3.18+maria~bionic 1:10.3.19+maria~bionic
2019-11-06 05:14:15 upgrade libmariadb3:amd64 1:10.3.18+maria~bionic 1:10.3.19+maria~bionic
2019-11-06 05:14:15 upgrade mariadb-client-core-10.3:amd64 1:10.3.18+maria~bionic 1:10.3.19+maria~bionic
2019-11-06 05:14:16 upgrade mariadb-server-10.3:amd64 1:10.3.18+maria~bionic 1:10.3.19+maria~bionic
2019-11-06 05:14:19 upgrade mariadb-client-10.3:amd64 1:10.3.18+maria~bionic 1:10.3.19+maria~bionic
2019-11-06 05:14:20 upgrade mariadb-server-core-10.3:amd64 1:10.3.18+maria~bionic 1:10.3.19+maria~bionic

and

# systemctl status mariadb
● mariadb.service - MariaDB 10.3.19 database server
Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/mariadb.service.d
└─migrated-from-my.cnf-settings.conf
Active: active (running) since Wed 2019-11-06 05:14:29 GMT; 7h ago
Docs: man:mysqld(8)
https://mariadb.com/kb/en/library/systemd/
Main PID: 10487 (mysqld)
Status: "Taking your SQL requests now..."
Tasks: 439 (limit: 4643)
CGroup: /system.slice/mariadb.service
└─10487 /usr/sbin/mysqld

~~~~ etc

1

Comment actions Permalink
Jan Schumacher - webhoster.ag November 06, 2019 14:06

After downgrade you can use

yum -y install yum-plugin-versionlock

yum versionlock MariaDB*

to block updates for MariaDB Packages.

with

yum versionlock clear

you can remove the lock if an update is available.

0

Comment actions Permalink
Anton Maslov November 06, 2019 15:50

To disable auto-updates until MariDB fix the bug, disable their repository, e.g. for CentOS:

sed -i 's/enabled=1/enabled=0/g' /etc/yum.repos.d/MariaDB.repo

1

Comment actions Permalink
Anthony November 06, 2019 18:23

MariaDB has removed all affected versions from there repository, so it is no longer necessary to disable auto-updates. (reference below)

If we have disabled auto updates, can you please provide the code to re-enable the auto-updates? Thank you.

ref: https://forums.cpanel.net/threads/known-issues-status-page.644133/page-3

0

Comment actions Permalink
Learning Curve November 07, 2019 03:00

Futher to our post re MariaDB 10.3.19 above. All the upgrade releases with the bug (potential bug) were removed by Maria DB thelmselves quite quickly yeterday. This page: https://mariadb.com/kb/en/library/mariadb-10319-release-notes/ shows how it is a 'conditional' bug that's related to certain tables that may already exist. Having checked ourselves, this is why we have not suffered from the bug effects, luckily. We can --force upgrade MariaDB 10.3.19 once the bug-free version is re-released, or just run a normal upgrade to 10.3.20 if that's released instead, but untl then, we (quite fortunately) can continue as normal.

0

Comment actions Permalink
Julian Bonpland Mignaquy November 07, 2019 23:47

Anthony you can re enable auto updates by following this article https://support.plesk.com/hc/en-us/articles/360000283234-How-to-enable-disable-autoupdates-in-Plesk-?source=search

0

Comment actions Permalink
Ceiling Cat November 08, 2019 11:45

Many Thanks for the Help ... Iwas really fu**ed up for a moment when one of my pals said the database is down ... so have I read it right in the comments?

I can now re-enable auto updates? Is that confirmed?

0

Comment actions Permalink
Ivan Postnikov November 08, 2019 12:03

Hello @Ceiling Cat,

According to the information we have, the update, which caused the issue was removed from MariaDB repos. It should be safe now.

1

Comment actions Permalink
Ceiling Cat November 08, 2019 12:33

Oh.. man! You guys are really fast here at the Plesk forums...
Many thanks I will try it .. and if anything unfortunate happens i will post it here...

Because of you all iam Lucky to use Plesk... and of course because the very cool interface ;)

0

Comment actions Permalink
Ivan Postnikov November 08, 2019 12:39

@Ceiling Cat,

I appreciate your feedback, thank you.

0

Comment actions Permalink

Ceiling Cat November 08, 2019 12:53

Well... There was something unfortunate...

And i have to say something Plesk couldnt load the apps list in the Update Settings so i did it Manually with "apt-get update && apt-get upgrade -y"

Oh and Sorry I use Plesk and ubuntu 18.04 in my language: German but it should be readable for you what happened there

mariadb-common (1:10.3.19+maria~bionic) wird eingerichtet ...
mariadb-server-core-10.3 (1:10.3.19+maria~bionic) wird eingerichtet ...
mariadb-client-core-10.3 (1:10.3.19+maria~bionic) wird eingerichtet ...
mariadb-client-10.3 (1:10.3.19+maria~bionic) wird eingerichtet ...
mariadb-server-10.3 (1:10.3.19+maria~bionic) wird eingerichtet ...
/var/lib/dpkg/info/mariadb-server-10.3.postinst: Zeile 56: 27659 Fertig echo "$replace_query"
27660 Abgebrochen (Speicherabzug geschrieben) | $MYSQL_BOOTSTRAP 2>&1
27661 | $ERR_LOGGER
Job for mariadb.service failed because a fatal signal was delivered causing the control process to dump core.
See "systemctl status mariadb.service" and "journalctl -xe" for details.
invoke-rc.d: initscript mysql, action "start" failed.
mariadb.service - MariaDB 10.3.19 database server
Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/mariadb.service.d
└─migrated-from-my.cnf-settings.conf
Active: activating (auto-restart) (Result: core-dump) since Fri 2019-11-08 13:41:41 CET; 11ms ago
Docs: man:mysqld(8)
https://mariadb.com/kb/en/library/systemd/
Process: 27963 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=dumped, signal=ABRT)
Process: 27800 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ] && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited, status=0/SUCCESS)
Process: 27783 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 27770 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
Main PID: 27963 (code=dumped, signal=ABRT)
Tasks: 1 (limit: 4568)
CGroup: /system.slice/mariadb.service

dpkg: Fehler beim Bearbeiten des Paketes mariadb-server (--configure):
Abhängigkeitsprobleme - verbleibt unkonfiguriert
Trigger für man-db (2.8.3-2ubuntu0.1) werden verarbeitet ...
Fehler traten auf beim Bearbeiten von:
mariadb-server-10.3
mariadb-server


E: Sub-process /usr/bin/dpkg returned an error code (1)

So I did the rollback again... It works for now...

Markus November 09, 2019 00:07 (Edited November 09, 2019 00:29)

Hi,

it seems that I was also not affected by this bug. BUT I got another problem during first upgrade from 10.3.18-MariaDB-1:10.3.18+maria~bionic to 10.3.19-MariaDB-1:10.3.19+maria~bionic (today I made the upgrade from 19 to 20):

...

/usr/bin/mysqlcheck: Got error: 1142: SELECT command denied to user 'root'@'localhost' for table 'column_stats' when executing 'CHECK TABLE ... FOR UPGRADE'
Nov 06 16:39:56 server.x.com /etc/mysql/debian-start[17053]: FATAL ERROR: Upgrade failed

Is this related to the bug, or not. What can I do to fix it?

Here are my outputs which seems to be correct:

mariadb.service - MariaDB 10.3.20 database server
Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)

..

Nov 08 20:56:28 server.x.com /etc/mysql/debian-start[29877]: mysql
Nov 08 20:56:28 server.x.com /etc/mysql/debian-start[29877]: /usr/bin/mysqlcheck: Got error: 1142: SELECT command denied to user 'root'@'localhost' for table 'column_stats' when executing 'CHECK TABLE ... FOR UPGRADE'
Nov 08 20:56:28 server.x.com /etc/mysql/debian-start[29877]: FATAL ERROR: Upgrade failed

I don't know if everything is really okay or not? Shall I do this tutorial here or not? What to do with column_stats?

0

Comment actions Permalink
Anzhelika Khapaknysh November 13, 2019 12:14

Hi @Multimedia Pool,

According to the symptoms provided, it doesn't related to the bug.

I'd recommend contacting our Plesk Support Team for assistance.

0

Comment actions Permalink
Alisa Kasyanova November 14, 2019 06:53

@Ceiling Cat
As I can see from the sent logs, buggy 10.3.19 version was installed again.
It should have been removed from the MariaDB repos, though. Try clearing the apt cache to make sure this old version is not cached locally:

# apt-get clean

1

Comment actions Permalink
Ceiling Cat November 14, 2019 16:05 (Edited November 14, 2019 17:31)

Yeah thank you very much 😌 I did the long way two days ago xD
I went back to the ftp site and downloaded the packages but the .20 :)
and then with dpkg like described at the beginning ... but i will remember your tip it is definitely the better and more professional way :D

0

Comment actions Permalink

Please sign in to leave a comment.

Have more questions? Submit a request

Articles in this section

Symptoms

Cause

Resolution

Related articles