Articles in this section

See more

MariaDB does not start after update on November the 5th: Assertion failure in file

Avatar
Alexandr Nikolaenko
Updated
Follow

Symptoms

  • MariaDB fails to start with:

    # systemctl start mariadb
    Job for mariadb.service failed because a fatal signal was delivered causing the control process to dump core. See "systemctl status mariadb.service" and "journalctl -xe" for details.
    # systemctl status mariadb.service
    ● mariadb.service - MariaDB 10.2.28 database server
    Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
    Drop-In: /etc/systemd/system/mariadb.service.d
    └─migrated-from-my.cnf-settings.conf
    Active: activating (auto-restart) (Result: core-dump) since Wed 2019-11-06 10:58:12 CST; 4s ago

  • MariaDB 10.1-10.3 has been recently updated to version 10.1.42 / 10.2.28 / 10.3.19:

    • on CentOS/RHEL-based distributions

      # grep -i Maria /var/log/yum.log
      Nov 06 03:46:20 Updated: MariaDB-common-10.1.42-1.el7.centos.x86_64
      Nov 06 03:46:22 Updated: MariaDB-client-10.1.42-1.el7.centos.x86_64
      Nov 06 03:46:38 Updated: MariaDB-server-10.1.42-1.el7.centos.x86_64
      Nov 06 03:46:38 Updated: MariaDB-shared-10.1.42-1.el7.centos.x86_64

    • on Debian/Ubuntu-based distributions

      # grep mariadb /var/log/dpkg.log | grep upgrade
      2019-11-06 06:25:34 upgrade mariadb-common:all 10.2.27+maria~xenial 10.2.28+maria~xenial
      2019-11-06 06:25:34 upgrade mariadb-client-core-10.2:amd64 10.2.27+maria~xenial 10.2.28+maria~xenial
      2019-11-06 06:26:05 upgrade mariadb-server-10.2:amd64 10.2.27+maria~xenial 10.2.28+maria~xenial
      2019-11-06 06:26:08 upgrade mariadb-client-10.2:amd64 10.2.27+maria~xenial 10.2.28+maria~xenial
      2019-11-06 06:26:09 upgrade mariadb-server-core-10.2:amd64 10.2.27+maria~xenial 10.2.28+maria~xenial

  • The following entries can be found in /var/log/mariadb.log or output of "journalctl -u mariadb"

    CONFIG_TEXT: InnoDB: Assertion failure in file /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.X.XX/storage/innobase/dict/dict0dict.cc line 1467
    InnoDB: Failing assertion: table->can_be_evicted

  • Plesk interface is not available with the following error in a web-browser:

    PLESK_INFO: SQLSTATE[HY000] [2002] No such file or directory

Cause

The issue is caused by the latest MariaDB update and confirmed as MariaDB bug MDEV-20987. The bug has already been fixed in MariaDB versions 10.1.43, 10.2.29, 10.3.20, 10.4.10, which have been released on November the 8th.

This issue affects the following versions of the MariaDB SQL Server:

  • 10.1.42
  • 10.2.28
  • 10.3.19
  • 10.4.9 (Not supported by Plesk)

Resolution

The next packages update is already available, so if the server has been affected, install the lastest MariaDB updates:

  1. Log into Plesk;

  2. Go to Tools & Settings > System Updates;

  3. Search for "MariaDB"

    border_plus_Capture.PNG

  4. Choose all the packages and click Update:

    border_plus_Capture2.PNG

Alternatively, updates can be installed via CLI:

For CentOS/RHEL

  1. Connect to the server via SSH.

  2. Update MariaDB packages:

    # yum -y update MariaDB-server MariaDB-common MariaDB-shared MariaDB-client MariaDB-compat MariaDB-devel

In case the issue persists apply either of solutions from the article How to fix InnoDB corruption cases for the MySQL databases on Plesk for Linux?

For Debian/Ubuntu

  1. Connect to the server via SSH.

  2. Update MariaDB packages:

    # apt-get update MariaDB-server MariaDB-common MariaDB-shared MariaDB-client MariaDB-compat MariaDB-devel

In case the issue persists apply either of solutions from the article How to fix InnoDB corruption cases for the MySQL databases on Plesk for Linux?

Comments

22 comments

Sort by Date Votes
  • Avatar
    Mahipal Singh

    This upgrade caused MySQL crashed and downtime of website. Please test such upgrades at your end before posting them online.

    0
    Comment actions Permalink
  • Avatar
    Maxim Krasikov

    Hello @Mahipal Singh,

    Please let me clarify that the MariaDB update was released by the official MariaDB vendor and not by Plesk.
    Until MariaDB releases an official patch, please apply the workaround described in the article.

    0
    Comment actions Permalink
  • Avatar
    Mahipal Singh

    Hi Maxim,

    Understood the upgrade was released via MariaDB but don't you think it is Plesk team responsibility to test the upgrade first and the solution was posted after 3 hours of upgrading MariaDB rpm package

    0
    Comment actions Permalink
  • Avatar
    Maxim Krasikov

    Hi @Mahipal Singh,

    MariaDB update was performed by yum utility which updated the packages from non-OS vendor repositories. 

    We reacted accordingly after were informed about the occurred behavior.

    0
    Comment actions Permalink
  • Avatar
    Roman Bendik (Edited )

    I think it automatically updated!?, and I lose a few hours on solution!

    Information for others is the solution to this error:

    The error in Plesk: ERR [panel] SQLSTATE[HY000] [2002] No such file or directory, Can't connect to local MySQL server through socket

    Abstract.php:144

    0
    Comment actions Permalink
  • Avatar
    Roy Stultiens

    this caused quite some issues and angry phonecalls this morning...

    Did you stop the update from rolling out automatically via Plesk (or which steps can we take to prevent this)?

    0
    Comment actions Permalink
  • Avatar
    Jen Kegels (Edited )

    Yes, thank you! I've been restoring backups all morning...

     

    0
    Comment actions Permalink
  • Avatar
    Learning Curve

    Either we are quite lucky... or this bug is not applicable to every installation (depending on the setup etc)

    In our case; Obsidian / MariaDB 10.3.* There's no trace of the bug... currently!

    # grep mariadb /var/log/dpkg.log | grep upgrade
    2019-11-06 05:14:15 upgrade mariadb-common:all 1:10.3.18+maria~bionic 1:10.3.19+maria~bionic
    2019-11-06 05:14:15 upgrade mariadb-server:all 1:10.3.18+maria~bionic 1:10.3.19+maria~bionic
    2019-11-06 05:14:15 upgrade libmariadb3:amd64 1:10.3.18+maria~bionic 1:10.3.19+maria~bionic
    2019-11-06 05:14:15 upgrade mariadb-client-core-10.3:amd64 1:10.3.18+maria~bionic 1:10.3.19+maria~bionic
    2019-11-06 05:14:16 upgrade mariadb-server-10.3:amd64 1:10.3.18+maria~bionic 1:10.3.19+maria~bionic
    2019-11-06 05:14:19 upgrade mariadb-client-10.3:amd64 1:10.3.18+maria~bionic 1:10.3.19+maria~bionic
    2019-11-06 05:14:20 upgrade mariadb-server-core-10.3:amd64 1:10.3.18+maria~bionic 1:10.3.19+maria~bionic

    and

    # systemctl status mariadb
    ● mariadb.service - MariaDB 10.3.19 database server
    Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/mariadb.service.d
    └─migrated-from-my.cnf-settings.conf
    Active: active (running) since Wed 2019-11-06 05:14:29 GMT; 7h ago
    Docs: man:mysqld(8)
    https://mariadb.com/kb/en/library/systemd/
    Main PID: 10487 (mysqld)
    Status: "Taking your SQL requests now..."
    Tasks: 439 (limit: 4643)
    CGroup: /system.slice/mariadb.service
    └─10487 /usr/sbin/mysqld

    ~~~~ etc

    1
    Comment actions Permalink
  • Avatar
    Jan Schumacher - webhoster.ag

    After downgrade you can use

    yum -y install yum-plugin-versionlock

    yum versionlock MariaDB*

    to block updates for MariaDB Packages.

    with

    yum versionlock clear

    you can remove the lock if an update is available.

     

    0
    Comment actions Permalink
  • Avatar
    Anton Maslov

    To disable auto-updates until MariDB fix the bug, disable their repository, e.g. for CentOS:

    sed -i 's/enabled=1/enabled=0/g' /etc/yum.repos.d/MariaDB.repo

    1
    Comment actions Permalink
  • Avatar
    Anthony

    MariaDB has removed all affected versions from there repository, so it is no longer necessary to disable auto-updates. (reference below)

    If we have disabled auto updates, can you please provide the code to re-enable the auto-updates?  Thank you.

     

    ref: https://forums.cpanel.net/threads/known-issues-status-page.644133/page-3

    0
    Comment actions Permalink
  • Avatar
    Learning Curve

    Futher to our post re MariaDB 10.3.19 above. All the upgrade releases with the bug (potential bug) were removed by Maria DB thelmselves quite quickly yeterday. This page: https://mariadb.com/kb/en/library/mariadb-10319-release-notes/ shows how it is a 'conditional' bug that's related to certain tables that may already exist. Having checked ourselves, this is why we have not suffered from the bug effects, luckily. We can --force upgrade MariaDB 10.3.19 once the bug-free version is re-released, or just run a normal upgrade to 10.3.20 if that's released instead, but untl then, we (quite fortunately) can continue as normal.

    0
    Comment actions Permalink
  • Avatar
    Julian Bonpland Mignaquy

    Anthony you can re enable auto updates by following this article https://support.plesk.com/hc/en-us/articles/360000283234-How-to-enable-disable-autoupdates-in-Plesk-?source=search

    0
    Comment actions Permalink
  • Avatar
    Ceiling Cat

    Many Thanks for the Help ... Iwas really fu**ed up for a moment when one of my pals said the database is down ... so have I read it right in the comments?

    I can now re-enable auto updates? Is that confirmed?

    0
    Comment actions Permalink
  • Avatar
    Ivan Postnikov

    Hello @Ceiling Cat,

    According to the information we have, the update, which caused the issue was removed from MariaDB repos. It should be safe now.

    1
    Comment actions Permalink
  • Avatar
    Ceiling Cat

    Oh.. man! You guys are really fast here at the Plesk forums...
    Many thanks I will try it .. and if anything unfortunate happens i will post it here...

    Because of you all iam Lucky to use Plesk... and of course because the very cool interface ;)

    0
    Comment actions Permalink
  • Avatar
    Ivan Postnikov

    @Ceiling Cat,

    I appreciate your feedback, thank you.

    0
    Comment actions Permalink
  • Avatar
    Ceiling Cat

    Well... There was something unfortunate...

    And i have to say something Plesk couldnt load the apps list in the Update Settings so i did it Manually with "apt-get update && apt-get upgrade -y"

    Oh and Sorry I use Plesk and ubuntu 18.04 in my language: German but it should be readable for you what happened there

     

    mariadb-common (1:10.3.19+maria~bionic) wird eingerichtet ...
    mariadb-server-core-10.3 (1:10.3.19+maria~bionic) wird eingerichtet ...
    mariadb-client-core-10.3 (1:10.3.19+maria~bionic) wird eingerichtet ...
    mariadb-client-10.3 (1:10.3.19+maria~bionic) wird eingerichtet ...
    mariadb-server-10.3 (1:10.3.19+maria~bionic) wird eingerichtet ...
    /var/lib/dpkg/info/mariadb-server-10.3.postinst: Zeile 56: 27659 Fertig echo "$replace_query"
    27660 Abgebrochen (Speicherabzug geschrieben) | $MYSQL_BOOTSTRAP 2>&1
    27661 | $ERR_LOGGER
    Job for mariadb.service failed because a fatal signal was delivered causing the control process to dump core.
    See "systemctl status mariadb.service" and "journalctl -xe" for details.
    invoke-rc.d: initscript mysql, action "start" failed.
    mariadb.service - MariaDB 10.3.19 database server
    Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/mariadb.service.d
    └─migrated-from-my.cnf-settings.conf
    Active: activating (auto-restart) (Result: core-dump) since Fri 2019-11-08 13:41:41 CET; 11ms ago
    Docs: man:mysqld(8)
    https://mariadb.com/kb/en/library/systemd/
    Process: 27963 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=dumped, signal=ABRT)
    Process: 27800 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ] && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited, status=0/SUCCESS)
    Process: 27783 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
    Process: 27770 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
    Main PID: 27963 (code=dumped, signal=ABRT)
    Tasks: 1 (limit: 4568)
    CGroup: /system.slice/mariadb.service

    dpkg: Fehler beim Bearbeiten des Paketes mariadb-server (--configure):
    Abhängigkeitsprobleme - verbleibt unkonfiguriert
    Trigger für man-db (2.8.3-2ubuntu0.1) werden verarbeitet ...
    Fehler traten auf beim Bearbeiten von:
    mariadb-server-10.3
    mariadb-server


    E: Sub-process /usr/bin/dpkg returned an error code (1)

     

    So I did the rollback again...  It works for now...

     

     

    0
    Comment actions Permalink
  • Avatar
    Markus (Edited )

    Hi,

    it seems that I was also not affected by this bug. BUT I got another problem during first upgrade from 10.3.18-MariaDB-1:10.3.18+maria~bionic  to 10.3.19-MariaDB-1:10.3.19+maria~bionic (today I made the upgrade from 19 to 20):

    ...

    /usr/bin/mysqlcheck: Got error: 1142: SELECT command denied to user 'root'@'localhost' for table 'column_stats' when executing 'CHECK TABLE ... FOR UPGRADE'
    Nov 06 16:39:56 server.x.com /etc/mysql/debian-start[17053]: FATAL ERROR: Upgrade failed

    Is this related to the bug, or not. What can I do to fix it?

    Here are my outputs which seems to be correct:

    mariadb.service - MariaDB 10.3.20 database server
    Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)

    ..

    Nov 08 20:56:28 server.x.com /etc/mysql/debian-start[29877]: mysql
    Nov 08 20:56:28 server.x.com /etc/mysql/debian-start[29877]: /usr/bin/mysqlcheck: Got error: 1142: SELECT command denied to user 'root'@'localhost' for table 'column_stats' when executing 'CHECK TABLE ... FOR UPGRADE'
    Nov 08 20:56:28 server.x.com /etc/mysql/debian-start[29877]: FATAL ERROR: Upgrade failed

    I don't know if everything is really okay or not? Shall I do this tutorial here or not? What to do with column_stats?

     

    0
    Comment actions Permalink
  • Avatar
    Anzhelika Khapaknysh

    Hi @Multimedia Pool,

    According to the symptoms provided, it doesn't related to the bug. 

    I'd recommend contacting our Plesk Support Team for assistance.

    0
    Comment actions Permalink
  • Avatar
    Alisa Kasyanova

    @Ceiling Cat
    As I can see from the sent logs, buggy 10.3.19 version was installed again.
    It should have been removed from the MariaDB repos, though. Try clearing the apt cache to make sure this old version is not cached locally:

    # apt-get clean

    1
    Comment actions Permalink
  • Avatar
    Ceiling Cat (Edited )

    Yeah thank you very much 😌 I did the long way two days ago xD
    I went back to the ftp site and downloaded the packages but the .20 :)
    and then with dpkg like described at the beginning ... but i will remember your tip it is definitely the better and more professional way :D

    0
    Comment actions Permalink

Please sign in to leave a comment.

Have more questions? Submit a request

Related articles