Clearing Apache Unused Sempaphore

In UNIX systems, semaphores are a technique for coordinating or synchronizing activities in which multiple process compete for the same operating system resources. In simple terms a semaphore is a type of Inter process communication resource(IPC) used for synchronization and mutual exclusion between any two asynchronous processes.

On a high load web server (mostly Worker MPM), we usually come across an error message-

[emerg] (28)No space left on device: Couldn’t create accept lock /or
[crit] (28)No space left on device: mod_rewrite: could not create rewrite_log_lock Configuration Failed,

or

[error] (28)No space left on device: Cannot create SSLMutex
Configuration Failed

- leading to a situation where Apache refuse to start.

We will come to solution later. First - lets try to understand why this happens.

What Apache is telling here that - "I can't start, as I need to write some things down before I can start, and I have nowhere to write them! And technically what it means is - the system have run out of semaphore arrays. Sometimes it's full with legitimate semaphores at other times it's because some application has leaked semaphores and haven't cleaned them up during the shutdown (which is usually the case when an application segfaults). As a rule, all semaphores that have been created should be cleared. If semaphores are not cleared, they remain in memory until the process that creates them ends. A process can only clear semaphores that it has created. We will be clearing 'only' semaphores which when got stale when our apache went haywire - post which we will also do a apache restart.

Coming back to the error resolution - pls. remember that , If this happens - we need to check three things basically -

1. Check your disk space
2. Review filesystem quotas
3. Clear out your active semaphores

Since the discussion is around how to clear unused semaphore - i will stress on the point No. 3.

We can use ipcs and ipcrm command to tackle this issue.

$ ipcs -s

------ Semaphore Arrays --------
key semid owner perms nsems
0x000ca001 163840 root 666 1
0x000ca016 20086800 nobody 600 1
0x000ca017 20086803 nobody 600 1
0x000ca018 20086804 nobody 600 1
---
---
Only for apache semaphore -

#ipcs -s | grep   In my case apache user 'nobody' so ,  #ipcs -s | grep nobody

Next you have to figure out what are the dead ones and remove them-

# ipcrm -s 20086804


Or, if you see a bunch of them, you can simply fire the below command:
ipcs -s | grep nobody |  perl -e 'while () { @a=split(/\s+/); print `ipcrm -s $a[1]`}'

You may want to increase your available semaphores, and you'll need to tickle your kernel to do so.
Add this to /etc/sysctl.conf:

kernel.msgmni = 1024
kernel.sem = 250 256000 32 1024

And then run sysctl -p to pick up the new changes.

Also,

If you see - 'Cannot create SSLMutex - Configuration Failed' , as a solution along with the above choose a configuration to leave it to SSL Module to pick the "best" semaphore implementation available to it.

SSLMutex file:logs/ssl_mutex

Change it to :

SSLMutex sem

More options can be read here

Hope it helps to have a better understanding.

Cheers!