Unable to start Apache: (28)No space left on device: AH00023: Couldn’t create the mutex

Symptoms

• Apache cannot be started:

  # service httpd start
  Starting httpd : [Failed]

  Or:

  Apache reload results in failed state and hard Apache restart is performed:

  Sep 20 17:29:45 example systemd[1]: Reloaded The Apache HTTP Server.
   Sep 20 17:29:50 example systemd[1]: httpd.service: main process exited, code=exited, status=1/FAILURE 
   Sep 20 17:29:50 example kill[3280]: kill: cannot find process "" 
   Sep 20 17:29:50 example systemd[1]: httpd.service: control process exited, code=exited status=1 
   Sep 20 17:29:50 example systemd[1]: Unit httpd.service entered failed state. 
   Sep 20 17:29:50 example systemd[1]: httpd.service failed. 
   Sep 20 17:29:51 example systemd[1]: Starting The Apache HTTP Server...

• The following error can be found in the Apache error log file /var/log/httpd/error_log on RHEL/CentOS /var/log/apache2/error_log on Debian/Ubuntu:

  [Mon Oct 07 18:20:01 2013] [error] (28)No space left on device: Cannot create SSLMutex

  OR:

  [Wed Oct 25 02:41:36 2017] [emerg] (28)No space left on device: Couldn't create accept lock (/etc/httpd/logs/accept.lock.44778) (5)

  OR:

  [Wed Oct 25 04:45:05 2017] [emerg] (43)Identifier removed: couldn't grab the accept mutex

  OR:

  [Thu Sep 20 17:39:20.139444 2018] [core:emerg] [pid 4565] (28)No space left on device: AH00023: Couldn't create the fcgid-pipe mutex

  OR:

  [Thu Sep 20 17:39:54.387218 2018] [core:emerg] [pid 9462] (28)No space left on device: AH00023: Couldn't create the rewrite-map mutex

• Semaphore limit is reached:

  # ipcs -s | wc -l
  32004

  While:

  # cat /proc/sys/kernel/msgmni
  32000

  Or:

  # cat /proc/sys/kernel/sem
  250 24000 32 1024

Cause

Lack of inner-process communication resources in the system, such as semaphores or shared memory segments. This behavior is expected to be fixed in next versions of Plesk Obsidian.

Resolution

As a workaround, perform the following steps:

1. Login to the server via SSH.

2. Increase the limits in sysctl configuration:

   # mkdir -p /etc/sysctl.d/
   # touch /etc/sysctl.d/99-zz_plesk_semaphores.conf
   # chmod 755 /etc/sysctl.d/ /etc/sysctl.d/99-zz_plesk_semaphores.conf
   # chown root:root /etc/sysctl.d/ /etc/sysctl.d/99-zz_plesk_semaphores.conf

   Such /etc/sysctl.d/99-zz_plesk_semaphores.conf configuration file has to contain:

   kernel.msgmni = 64000
    kernel.sem = 250 256000 32 1024

3. Load the new sysctl settings from sysctl

   # sysctl -p

4. Check which user exhausted semaphores:

   # ipcs -s | tail -n +4 | head -n -1 | tr -s ‘ ‘ | cut -d’ ‘ -f3 | sort | uniq -c | sort -nr
   234 somesoftware
   33 httpd
   4 root

5. Clean semaphores left by Apache:

   # ipcs -s | tail -n +4 | head -n -1 | tr -s ‘ ‘ | cut -d’ ‘ -f2,3 | while read -r id owner; do [[ $owner == “httpd” ]] && ipcrm -s “$id”; done

   In case some other software heavily abusing semaphore limit, disable this software, clean its’ semaphores and contact software provider for further investigation:

   # ipcs -s | tail -n +4 | head -n -1 | tr -s ‘ ‘ | cut -d’ ‘ -f2,3 | while read -r id owner; do [[ $owner == “somesoftware” ]] && ipcrm -s “$id”; done

6. Start Apache:

   # service httpd start