Articles in this section

See more

Unable to start Apache: (28)No space left on device: AH00023: Couldn't create the mutex

Avatar
Daniel Yordanov
Updated
Plesk for Linux kb: technical

Applicable to:

  • Plesk for Linux

Symptoms

  • Plesk Obsidian running on a Linux-based operating system

  • Apache cannot be started and enters a failed state by providing one of the following outputs:

    RHEL-based OS:

    # service httpd start
    Starting httpd : [Failed]

    # systemctl start httpd
    Starting httpd : [Failed]

    Debian-based OS:

    # service apache2 start
    Starting apache2 : [Failed]

    # systemctl start apache2
    Starting apache2 : [Failed]

    OR

    Apache reload attempts result in failed state and a hard Apache restart is performed automatically:

    CONFIG_TEXT: Sep 20 17:29:45 example systemd[1]: Reloaded The Apache HTTP Server.
    Sep 20 17:29:50 example systemd[1]: httpd.service: main process exited, code=exited, status=1/FAILURE
    Sep 20 17:29:50 example kill[3280]: kill: cannot find process ""
    Sep 20 17:29:50 example systemd[1]: httpd.service: control process exited, code=exited status=1
    Sep 20 17:29:50 example systemd[1]: Unit httpd.service entered failed state.
    Sep 20 17:29:50 example systemd[1]: httpd.service failed.
    Sep 20 17:29:51 example systemd[1]: Starting The Apache HTTP Server...

  • One or more of the following errors can be found in the Apache error log file /var/log/httpd/error_log on RHEL-based OS /var/log/apache2/error_log on Debian-based OS:

    CONFIG_TEXT: [Mon Oct 07 18:20:01 2013] [error] (28)No space left on device: Cannot create SSLMutex

    CONFIG_TEXT: [Wed Oct 25 02:41:36 2017] [emerg] (28)No space left on device: Couldn't create accept lock (/etc/httpd/logs/accept.lock.44778) (5)

    CONFIG_TEXT: [Wed Oct 25 04:45:05 2017] [emerg] (43)Identifier removed: couldn't grab the accept mutex

    CONFIG_TEXT: [Thu Sep 20 17:39:20.139444 2018] [core:emerg] [pid 4565] (28)No space left on device: AH00023: Couldn't create the fcgid-pipe mutex

    CONFIG_TEXT: [Thu Sep 20 17:39:54.387218 2018] [core:emerg] [pid 9462] (28)No space left on device: AH00023: Couldn't create the rewrite-map mutex

Cause

The semaphore limit set on the Linux OS level is reached:

# ipcs -s | wc -l
32004

While:

# cat /proc/sys/kernel/msgmni
32000

Or:

# cat /proc/sys/kernel/sem
250 24000 32 1024

A lack of inner-process communication resources in the Linux system, such as semaphores or shared memory segments is preventing Apache from starting.

Resolution

Click on a section to expand

If the /etc/sysctl.d/99-sysctl.conf file exists on the server

  1. Login to the server via SSH

  2. Add the following directives to the /etc/sysctl.d/99-sysctl.conf file by using your favorite command-line text editor:

    CONFIG_TEXT: kernel.msgmni = 64000
    kernel.sem = 250 256000 32 1024

  3. Load the new sysctl settings from sysctlby executing the following command:

    # sysctl -p

  4. Check which user exhausted semaphores:

    # ipcs -s | tail -n +4 | head -n -1 | tr -s ' ' | cut -d' ' -f3 | sort | uniq -c | sort -nr
    234 somesoftware
    33 httpd
    4 root

  5. Clean the semaphores left by Apache (the service name is apache2 on a Debian-based server):

    # ipcs -s | tail -n +4 | head -n -1 | tr -s ' ' | cut -d' ' -f2,3 | while read -r id owner; do [[ $owner == "httpd" ]] && ipcrm -s "$id"; done

  6. In case some other software heavily abusing semaphore limit, disable this software, clean its' semaphores and contact software provider for further investigation:

    # ipcs -s | tail -n +4 | head -n -1 | tr -s ' ' | cut -d' ' -f2,3 | while read -r id owner; do [[ $owner == "somesoftware" ]] && ipcrm -s "$id"; done

  7. Start Apache:

    # service httpd start

    OR

    # service apache2 start

If the /etc/sysctl.d/99-sysctl.conf file does not exist on the server
  1. Login to the server via SSH

  2. Increase the limits in sysctl configuration by creating the /etc/sysctl.d/99-zz_plesk_semaphores.conffile:

    # mkdir -p /etc/sysctl.d/
    # touch /etc/sysctl.d/99-zz_plesk_semaphores.conf
    # chmod 755 /etc/sysctl.d/ /etc/sysctl.d/99-zz_plesk_semaphores.conf
    # chown root:root /etc/sysctl.d/ /etc/sysctl.d/99-zz_plesk_semaphores.conf

    Open the newly created /etc/sysctl.d/99-zz_plesk_semaphores.conf configuration file by using your favorite command-line text editor and enter the following values:

    CONFIG_TEXT: kernel.msgmni = 64000
    kernel.sem = 250 256000 32 1024

  3. Load the new sysctl settings from sysctlby executing the following command:

    # sysctl -p

  4. Check which user exhausted semaphores:

    # ipcs -s | tail -n +4 | head -n -1 | tr -s ' ' | cut -d' ' -f3 | sort | uniq -c | sort -nr
    234 somesoftware
    33 httpd
    4 root

  5. Clean semaphores left by Apache (the service name is apache2 on a Debian-based server):

    # ipcs -s | tail -n +4 | head -n -1 | tr -s ' ' | cut -d' ' -f2,3 | while read -r id owner; do [[ $owner == "httpd" ]] && ipcrm -s "$id"; done

    In case some other software heavily abusing semaphore limit, disable this software, clean its' semaphores and contact software provider for further investigation:

    # ipcs -s | tail -n +4 | head -n -1 | tr -s ' ' | cut -d' ' -f2,3 | while read -r id owner; do [[ $owner == "somesoftware" ]] && ipcrm -s "$id"; done

  6. Start Apache:

    # service httpd start

    OR

    # service apache2 start

Comments

0 comments

Please sign in to leave a comment.

Have more questions? Submit a request

Related articles