Systemctl fails to start service: Too many levels of symbolic links

Symptoms

• Plesk is running on CentOS 7.7

• There are multiple failed services:

  # systemctl list-units --failed
  UNIT LOAD ACTIVE SUB DESCRIPTION
  ● courier-authdaemon.service loaded active failed Courier-IMAP authentication daemon
  ● courier-imapd.service loaded active failed Courier IMAP server
  ● courier-imaps.service loaded active failed Courier IMAP server with SSL/TLS support
  ● courier-pop3d.service loaded active failed Courier POP3 server
  ● courier-pop3s.service loaded active failed Courier POP3 server with SSL/TLS support
  ● spamassassin.service loaded active failed Spamassassin daemon
  ● systemd-sysctl.service loaded failed failed Apply Kernel Variables
  ● systemd-tmpfiles-setup-dev.service loaded failed failed Create Static Device Nodes in /dev
  ● systemd-vconsole-setup.service loaded failed failed Setup Virtual Console

• Systemctl cannot start the service, for example, SpamAssassin:

  # systemctl start spamassassin.service
  Job for spamassassin.service failed because a timeout was exceeded. See "systemctl status spamassassin.service" and "journalctl -xe" for details.

• Systemctl status shows the service is Activating or Failed, as follows:

  # systemctl status spamassassin.service
  ● spamassassin.service - Spamassassin daemon
  Loaded: loaded (/etc/systemd/system/spamassassin.service; enabled; vendor preset: disabled)
  Active: activating (start) since Wed 2019-09-18 20:47:36 CEST; 16s ago
  Process: 31790 ExecStart=/usr/bin/spamd --pidfile /var/run/spamd.pid $SPAMDOPTIONS
  ...

  # systemctl status courier-imapd
  ● courier-imapd.service - Courier IMAP server
  Loaded: loaded (/usr/lib/systemd/system/courier-imapd.service; enabled; vendor preset: disabled)
  Active: failed (Result: timeout) since Thu 2019-09-19 11:45:33 CEST; 7s ago
  ...

• There might appear the following errors in /var/log/messages file:

  # egrep "Too many levels of symbolic links" /var/log/messages
  systemd-tmpfiles: Adjusting owner and mode for /sys/class/net/lo failed: Too many levels of symbolic links
  systemd-tmpfiles: Adjusting owner and mode for /sys/class/net/venet0 failed: Too many levels of symbolic links
  ...

• Starting an affected service with strace -f might show this message repeatedly until the service timeout is reached:

  # strace -f systemctl start spamassassin
  ...
  [pid 20656] ppoll([{fd=3, events=POLLIN}], 1, NULL, NULL, 8) = 1 ([{fd=3, revents=POLLIN}])
  [pid 20656] recvmsg(3, {msg_name(0)=NULL, msg_iov(1)=[{"l\4\1\1!\2\0\0H\0\0\0\236\0\0\0\1\1o\0008\0\0\0", 24}], msg_controllen=32, [{cmsg_len=28, cmsg_level=SOL_SOCKET, cmsg_type=SCM_CREDENTIALS, {pid=1, uid=0, gid=0}}], msg_flags=MSG_CMSG_CLOEXEC}, MSG_DONTWAIT|MSG_NOSIGNAL|MSG_CMSG_CLOEXEC) = 24
  [pid 20656] recvmsg(3, {msg_name(0)=NULL, msg_iov(1)=[{"/org/freedesktop/systemd1/unit/c"..., 697}], msg_controllen=32, [{cmsg_len=28, cmsg_level=SOL_SOCKET, cmsg_type=SCM_CREDENTIALS, {pid=1, uid=0, gid=0}}], msg_flags=MSG_CMSG_CLOEXEC}, MSG_DONTWAIT|MSG_NOSIGNAL|MSG_CMSG_CLOEXEC) = 697
  [pid 20656] recvmsg(3, 0x7ffd5da336b0, MSG_DONTWAIT|MSG_NOSIGNAL|MSG_CMSG_CLOEXEC) = -1 EAGAIN (Resource temporarily unavailable)
  ...

• Increasing the TimeoutStartSec parameter on the affected service doesn't help.

Cause

The recent CentOS 7.7 update includes an update to systemd-219-67, which changed how systemd opens symbolic links and causes incompatibility issues with old kernels.

Resolution

Due to this issue can affect all the services, update the Kernel to latest and reboot the server.

Note: In case it's a Virtuozzo container, please contact Virtuozzo support team to know if there's an updated version of the kernel for Virtuozzo.

Click on a section to expand

Additional Information

Service Failures after CentOS 7.7 update