Here is the URL to the page that gave me the tip that got polyeidus back up. We should be monitoring this for a couple of days.

http://rackerhacker.com/2007/08/24/apache-no-space-left-on-device-couldn...

So, this was really weird, clearing the 125 orphaned semaphores owned by the www user fixed the problem, and Apache started up with no more complaints.

Now, for what happened. In short, we do not know yet. There were no noticed errors before this, and once we hit this problem, the errors we got were:

"[Wed Mar 25 18:30:57 2009] [crit] (28)No space left on device: mod_jk: could not create jk_log_lock
Configuration Failed"

df reported about 9% of the disk being used, the directory had more perms then it needed, and both the root and www user were able to create and write to file in /var/log/httpd without trouble. So that removes the obvious problems, and the obscure problems did gave us only the crumb trail listed here.