Unable to start Apache: (28)No space left on device: AH00023: Couldn't create the mutex

Symptoms

• Apache cannot be started:

  # service httpd start
  Starting httpd : [Failed]

• The following error can be found in the Apache error log file /var/log/httpd/error_log on RHEL/CentOS /var/log/apache2/error_log on Debian/Ubuntu:

  CONFIG_TEXT: [Mon Oct 07 18:20:01 2013] [error] (28)No space left on device: Cannot create SSLMutex

  OR:

  CONFIG_TEXT: [Wed Oct 25 02:41:36 2017] [emerg] (28)No space left on device: Couldn't create accept lock (/etc/httpd/logs/accept.lock.44778) (5)

  OR:

  CONFIG_TEXT: [Wed Oct 25 04:45:05 2017] [emerg] (43)Identifier removed: couldn't grab the accept mutex

• Semaphore limit is reached:

  # ipcs -s | wc -l
  32004

  While:

  # cat /proc/sys/kernel/msgmni
  32000

  Or:

  # cat /proc/sys/kernel/sem
  250 24000 32 1024

Cause

Those errors mean that there is a lack of inner-process communication resources in the system, such as semaphores or shared memory segments.

Resolution

1. Login to the server via SSH.

2. Increase the limits in /etc/sysctl.conf:

   CONFIG_TEXT: kernel.msgmni = 64000
   kernel.sem = 250 256000 32 1024

3. Load the new sysctl settings from /etc/sysctl.conf

   # sysctl -p

4. Check which user exhausted semaphores:

   # ipcs -s | awk '{print $3}' | uniq -c
   4 root
   33 httpd
   234 somesoftware

5. Clean semaphores left by Apache:

   # for i in `ipcs -s | awk '/httpd/ {print $2}'`; do (ipcrm -s $i); done

   In case some other software heavily abusing semaphore limit, disable this software, clean its' semaphores and contact software provider for further investigation:

   # for i in `ipcs -s | awk '/httpd/ {print $2}'`; do (ipcrm -s $i); done

6. Start Apache:

   # service httpd start