Applicable to:
- Plesk for Linux
Symptoms
-
MariaDB fails to start with:
# systemctl start mariadb
Job for mariadb.service failed because a fatal signal was delivered causing the control process to dump core. See "systemctl status mariadb.service" and "journalctl -xe" for details.
# systemctl status mariadb.service
● mariadb.service - MariaDB 10.2.28 database server
Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/mariadb.service.d
└─migrated-from-my.cnf-settings.conf
Active: activating (auto-restart) (Result: core-dump) since Wed 2019-11-06 10:58:12 CST; 4s ago -
MariaDB 10.1-10.3 has been recently updated to version 10.1.42 / 10.2.28 / 10.3.19:
-
on CentOS/RHEL-based distributions
# grep -i Maria /var/log/yum.log
Nov 06 03:46:20 Updated: MariaDB-common-10.1.42-1.el7.centos.x86_64
Nov 06 03:46:22 Updated: MariaDB-client-10.1.42-1.el7.centos.x86_64
Nov 06 03:46:38 Updated: MariaDB-server-10.1.42-1.el7.centos.x86_64
Nov 06 03:46:38 Updated: MariaDB-shared-10.1.42-1.el7.centos.x86_64 -
on Debian/Ubuntu-based distributions
# grep mariadb /var/log/dpkg.log | grep upgrade
2019-11-06 06:25:34 upgrade mariadb-common:all 10.2.27+maria~xenial 10.2.28+maria~xenial
2019-11-06 06:25:34 upgrade mariadb-client-core-10.2:amd64 10.2.27+maria~xenial 10.2.28+maria~xenial
2019-11-06 06:26:05 upgrade mariadb-server-10.2:amd64 10.2.27+maria~xenial 10.2.28+maria~xenial
2019-11-06 06:26:08 upgrade mariadb-client-10.2:amd64 10.2.27+maria~xenial 10.2.28+maria~xenial
2019-11-06 06:26:09 upgrade mariadb-server-core-10.2:amd64 10.2.27+maria~xenial 10.2.28+maria~xenial
-
-
The following entries can be found in
/var/log/mariadb.logor output of "journalctl -u mariadb"CONFIG_TEXT: InnoDB: Assertion failure in file /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.X.XX/storage/innobase/dict/dict0dict.cc line 1467
InnoDB: Failing assertion: table->can_be_evicted -
Plesk interface is not available with the following error in a web-browser:
PLESK_INFO: SQLSTATE[HY000] [2002] No such file or directory
Cause
The issue is caused by the latest MariaDB update and confirmed as MariaDB bug MDEV-20987. The bug has already been fixed in MariaDB versions 10.1.43, 10.2.29, 10.3.20, 10.4.10, which have been released on November the 8th.
This issue affects the following versions of the MariaDB SQL Server:
- 10.1.42
- 10.2.28
- 10.3.19
- 10.4.9 (Not supported by Plesk)
Resolution
The next packages update is already available, so if the server has been affected, install the lastest MariaDB updates:
-
Log into Plesk
-
Go to Tools & Settings > System Updates
-
Search for "MariaDB"
-
Choose all the packages and click Update
Alternatively, updates can be installed via CLI:
-
Connect to the server via SSH
-
Update MariaDB packages:
# yum -y update MariaDB-server MariaDB-common MariaDB-shared MariaDB-client MariaDB-compat MariaDB-devel
In case the issue persists apply either of the solutions from the article How to fix InnoDB corruption cases for the MySQL databases on Plesk for Linux?
-
Connect to the server via SSH.
-
Update MariaDB packages:
# apt-get update MariaDB-server MariaDB-common MariaDB-shared MariaDB-client MariaDB-compat MariaDB-devel
In case the issue persists apply either of the solutions from the article How to fix InnoDB corruption cases for the MySQL databases on Plesk for Linux?
Comments
22 comments
This upgrade caused MySQL crashed and downtime of website. Please test such upgrades at your end before posting them online.
Hello @Mahipal Singh,
Please let me clarify that the MariaDB update was released by the official MariaDB vendor and not by Plesk.
Until MariaDB releases an official patch, please apply the workaround described in the article.
Hi Maxim,
Understood the upgrade was released via MariaDB but don't you think it is Plesk team responsibility to test the upgrade first and the solution was posted after 3 hours of upgrading MariaDB rpm package
Hi @Mahipal Singh,
MariaDB update was performed by yum utility which updated the packages from non-OS vendor repositories.
We reacted accordingly after were informed about the occurred behavior.
I think it automatically updated!?, and I lose a few hours on solution!
Information for others is the solution to this error:
The error in Plesk: ERR [panel] SQLSTATE[HY000] [2002] No such file or directory, Can't connect to local MySQL server through socket
Abstract.php:144
this caused quite some issues and angry phonecalls this morning...
Did you stop the update from rolling out automatically via Plesk (or which steps can we take to prevent this)?
Yes, thank you! I've been restoring backups all morning...
Either we are quite lucky... or this bug is not applicable to every installation (depending on the setup etc)
In our case; Obsidian / MariaDB 10.3.* There's no trace of the bug... currently!
# grep mariadb /var/log/dpkg.log | grep upgrade
2019-11-06 05:14:15 upgrade mariadb-common:all 1:10.3.18+maria~bionic 1:10.3.19+maria~bionic
2019-11-06 05:14:15 upgrade mariadb-server:all 1:10.3.18+maria~bionic 1:10.3.19+maria~bionic
2019-11-06 05:14:15 upgrade libmariadb3:amd64 1:10.3.18+maria~bionic 1:10.3.19+maria~bionic
2019-11-06 05:14:15 upgrade mariadb-client-core-10.3:amd64 1:10.3.18+maria~bionic 1:10.3.19+maria~bionic
2019-11-06 05:14:16 upgrade mariadb-server-10.3:amd64 1:10.3.18+maria~bionic 1:10.3.19+maria~bionic
2019-11-06 05:14:19 upgrade mariadb-client-10.3:amd64 1:10.3.18+maria~bionic 1:10.3.19+maria~bionic
2019-11-06 05:14:20 upgrade mariadb-server-core-10.3:amd64 1:10.3.18+maria~bionic 1:10.3.19+maria~bionic
and
# systemctl status mariadb
● mariadb.service - MariaDB 10.3.19 database server
Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/mariadb.service.d
└─migrated-from-my.cnf-settings.conf
Active: active (running) since Wed 2019-11-06 05:14:29 GMT; 7h ago
Docs: man:mysqld(8)
https://mariadb.com/kb/en/library/systemd/
Main PID: 10487 (mysqld)
Status: "Taking your SQL requests now..."
Tasks: 439 (limit: 4643)
CGroup: /system.slice/mariadb.service
└─10487 /usr/sbin/mysqld
~~~~ etc
After downgrade you can use
yum -y install yum-plugin-versionlock
yum versionlock MariaDB*
to block updates for MariaDB Packages.
with
yum versionlock clear
you can remove the lock if an update is available.
To disable auto-updates until MariDB fix the bug, disable their repository, e.g. for CentOS:
sed -i 's/enabled=1/enabled=0/g' /etc/yum.repos.d/MariaDB.repo
MariaDB has removed all affected versions from there repository, so it is no longer necessary to disable auto-updates. (reference below)
If we have disabled auto updates, can you please provide the code to re-enable the auto-updates? Thank you.
ref: https://forums.cpanel.net/threads/known-issues-status-page.644133/page-3
Futher to our post re MariaDB 10.3.19 above. All the upgrade releases with the bug (potential bug) were removed by Maria DB thelmselves quite quickly yeterday. This page: https://mariadb.com/kb/en/library/mariadb-10319-release-notes/ shows how it is a 'conditional' bug that's related to certain tables that may already exist. Having checked ourselves, this is why we have not suffered from the bug effects, luckily. We can --force upgrade MariaDB 10.3.19 once the bug-free version is re-released, or just run a normal upgrade to 10.3.20 if that's released instead, but untl then, we (quite fortunately) can continue as normal.
Anthony you can re enable auto updates by following this article https://support.plesk.com/hc/en-us/articles/360000283234-How-to-enable-disable-autoupdates-in-Plesk-?source=search
Many Thanks for the Help ... Iwas really fu**ed up for a moment when one of my pals said the database is down ... so have I read it right in the comments?
I can now re-enable auto updates? Is that confirmed?
Hello @Ceiling Cat,
According to the information we have, the update, which caused the issue was removed from MariaDB repos. It should be safe now.
Oh.. man! You guys are really fast here at the Plesk forums...
Many thanks I will try it .. and if anything unfortunate happens i will post it here...
Because of you all iam Lucky to use Plesk... and of course because the very cool interface ;)
@Ceiling Cat,
I appreciate your feedback, thank you.
Well... There was something unfortunate...
And i have to say something Plesk couldnt load the apps list in the Update Settings so i did it Manually with "apt-get update && apt-get upgrade -y"
Oh and Sorry I use Plesk and ubuntu 18.04 in my language: German but it should be readable for you what happened there
So I did the rollback again... It works for now...
Hi,
it seems that I was also not affected by this bug. BUT I got another problem during first upgrade from 10.3.18-MariaDB-1:10.3.18+maria~bionic to 10.3.19-MariaDB-1:10.3.19+maria~bionic (today I made the upgrade from 19 to 20):
...
/usr/bin/mysqlcheck: Got error: 1142: SELECT command denied to user 'root'@'localhost' for table 'column_stats' when executing 'CHECK TABLE ... FOR UPGRADE'
Nov 06 16:39:56 server.x.com /etc/mysql/debian-start[17053]: FATAL ERROR: Upgrade failed
Is this related to the bug, or not. What can I do to fix it?
Here are my outputs which seems to be correct:
mariadb.service - MariaDB 10.3.20 database server
Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
..
Nov 08 20:56:28 server.x.com /etc/mysql/debian-start[29877]: mysql
Nov 08 20:56:28 server.x.com /etc/mysql/debian-start[29877]: /usr/bin/mysqlcheck: Got error: 1142: SELECT command denied to user 'root'@'localhost' for table 'column_stats' when executing 'CHECK TABLE ... FOR UPGRADE'
Nov 08 20:56:28 server.x.com /etc/mysql/debian-start[29877]: FATAL ERROR: Upgrade failed
I don't know if everything is really okay or not? Shall I do this tutorial here or not? What to do with column_stats?
Hi @Multimedia Pool,
According to the symptoms provided, it doesn't related to the bug.
I'd recommend contacting our Plesk Support Team for assistance.
@Ceiling Cat
As I can see from the sent logs, buggy 10.3.19 version was installed again.
It should have been removed from the MariaDB repos, though. Try clearing the apt cache to make sure this old version is not cached locally:
# apt-get clean
Yeah thank you very much 😌 I did the long way two days ago xD
I went back to the ftp site and downloaded the packages but the .20 :)
and then with dpkg like described at the beginning ... but i will remember your tip it is definitely the better and more professional way :D
Please sign in to leave a comment.