Applicable to:
- Plesk
Symptoms
-
Apache cannot be started:
# service httpd start
Starting httpd : [Failed]Or:
Apache reload results in failed state and hard Apache restart is performed:
CONFIG_TEXT: Sep 20 17:29:45 example systemd[1]: Reloaded The Apache HTTP Server.
Sep 20 17:29:50 example systemd[1]: httpd.service: main process exited, code=exited, status=1/FAILURE
Sep 20 17:29:50 example kill[3280]: kill: cannot find process ""
Sep 20 17:29:50 example systemd[1]: httpd.service: control process exited, code=exited status=1
Sep 20 17:29:50 example systemd[1]: Unit httpd.service entered failed state.
Sep 20 17:29:50 example systemd[1]: httpd.service failed.
Sep 20 17:29:51 example systemd[1]: Starting The Apache HTTP Server... -
The following error can be found in the Apache error log file
/var/log/httpd/error_logon RHEL/CentOS/var/log/apache2/error_logon Debian/Ubuntu:CONFIG_TEXT: [Mon Oct 07 18:20:01 2013] [error] (28)No space left on device: Cannot create SSLMutex
OR:
CONFIG_TEXT: [Wed Oct 25 02:41:36 2017] [emerg] (28)No space left on device: Couldn't create accept lock (/etc/httpd/logs/accept.lock.44778) (5)
OR:
CONFIG_TEXT: [Wed Oct 25 04:45:05 2017] [emerg] (43)Identifier removed: couldn't grab the accept mutex
OR:
CONFIG_TEXT: [Thu Sep 20 17:39:20.139444 2018] [core:emerg] [pid 4565] (28)No space left on device: AH00023: Couldn't create the fcgid-pipe mutex
OR:
CONFIG_TEXT: [Thu Sep 20 17:39:54.387218 2018] [core:emerg] [pid 9462] (28)No space left on device: AH00023: Couldn't create the rewrite-map mutex
-
Semaphore limit is reached:
# ipcs -s | wc -l
32004While:
# cat /proc/sys/kernel/msgmni
32000Or:
# cat /proc/sys/kernel/sem
250 24000 32 1024
Cause
Lack of inner-process communication resources in the system, such as semaphores or shared memory segments. This behavior is expected to be fixed in next versions of Plesk Obsidian.
Resolution
As a workaround, perform the following steps:
-
Login to the server via SSH.
-
Increase the limits in
sysctlconfiguration:# mkdir -p /etc/sysctl.d/
# touch /etc/sysctl.d/99-zz_plesk_semaphores.conf
# chmod 755 /etc/sysctl.d/ /etc/sysctl.d/99-zz_plesk_semaphores.conf
# chown root:root /etc/sysctl.d/ /etc/sysctl.d/99-zz_plesk_semaphores.confSuch
/etc/sysctl.d/99-zz_plesk_semaphores.confconfiguration file has to contain:CONFIG_TEXT: kernel.msgmni = 64000
kernel.sem = 250 256000 32 1024 -
Load the new sysctl settings from
sysctl# sysctl -p
-
Check which user exhausted semaphores:
# ipcs -s | tail -n +4 | head -n -1 | tr -s ' ' | cut -d' ' -f3 | sort | uniq -c | sort -nr
234 somesoftware
33 httpd
4 root -
Clean semaphores left by Apache:
# ipcs -s | tail -n +4 | head -n -1 | tr -s ' ' | cut -d' ' -f2,3 | while read -r id owner; do [[ $owner == "httpd" ]] && ipcrm -s "$id"; done
In case some other software heavily abusing semaphore limit, disable this software, clean its' semaphores and contact software provider for further investigation:
# ipcs -s | tail -n +4 | head -n -1 | tr -s ' ' | cut -d' ' -f2,3 | while read -r id owner; do [[ $owner == "somesoftware" ]] && ipcrm -s "$id"; done
-
Start Apache:
# service httpd start
Comments
5 comments
Hi, for to release semaphores of other software, you need to use the following line:
for i in `ipcs -s | awk '/httpd/{ next; } {print $2}'`; do (ipcrm -s $i); done
Think the results should be sorted first:
ipcs -s | awk '{print $3}' | uniq -c
1
1 Arrays
1 owner
53 apache
1 root
126 apache
1
ipcs -s | awk '{print $3}' | sort | uniq -c
2
179 apache
1 Arrays
1 owner
1 root
@Adi Pircalabu
Yes, it seems to be helpful. I have updated the article accordingly. Thank you!
For a cleaner output can also run:
ipcs -s | awk '{print $3}' | egrep -v '(Arrays|owner|^$)' | sort | uniq -c
@Adi Pircalabu,
I have modified the commands in the article to exclude headers of the utility (hence the cleaner output), and spawn no awk processes.
Please sign in to leave a comment.