[Supervisor-users] Managing php-cgi - Doesn't auto-restart

[Tom Barrett]

Due to my issues with the current version of supervisord, I've built another
dev box to see if I can get things working in principle.

Here is my php-cgi configuration block:

;[fcgi-program:php]
[fcgi-program:php-cgi]
command=/usr/bin/php-cgi
;command=/usr/bin/php-cgi -c /etc/php.ini -b /tmp/php.socket
;command=/path/to/php-cgi
;socket=unix:///tmp/%program_name)s.sock
socket=tcp://127.0.0.1:57093
process_name=%(program_name)s_%(process_num)02d
user=apache
numprocs=2
priority=999
autostart=true
autorestart=true
startsecs=1
startretries=3
exitcodes=0,2
stopsignal=QUIT
stopwaitsecs=10

python -V

Python 2.4.3

supervisorctl status

php-cgi:php-cgi_00 RUNNING pid 18020, uptime 16:20:49
php-cgi:php-cgi_01 RUNNING pid 18019, uptime 16:20:49

It's running on tiny 256Mb slice, where the php-cgi processes die
unexpectedly (possibly due to lack of memory, but I don't know for sure).
The idea being that supervisord would auto restart them if this happens,
however it doesn't. They just end up in status 'EXITED' (to my
recollection).

What is it I am missing to make supervisord restart them? Can I manually
trigger such an event (via kill, perhaps) to test that it works?

[Chris McDonough]

Post by Tom Barrett
Due to my issues with the current version of supervisord, I've built
another dev box to see if I can get things working in principle.
;[fcgi-program:php]
[fcgi-program:php-cgi]
command=/usr/bin/php-cgi
;command=/usr/bin/php-cgi -c /etc/php.ini -b /tmp/php.socket
;command=/path/to/php-cgi
;socket=unix:///tmp/%program_name)s.sock
socket=tcp://127.0.0.1:57093 <http://127.0.0.1:57093>
process_name=%(program_name)s_%(process_num)02d
user=apache
numprocs=2
priority=999
autostart=true
autorestart=true
startsecs=1
startretries=3
exitcodes=0,2
stopsignal=QUIT
stopwaitsecs=10

python -V

Python 2.4.3

supervisorctl status

php-cgi:php-cgi_00 RUNNING pid 18020, uptime 16:20:49
php-cgi:php-cgi_01 RUNNING pid 18019, uptime 16:20:49
It's running on tiny 256Mb slice, where the php-cgi processes die
unexpectedly (possibly due to lack of memory, but I don't know for sure).
The idea being that supervisord would auto restart them if this happens,
however it doesn't. They just end up in status 'EXITED' (to my
recollection).
What is it I am missing to make supervisord restart them? Can I manually
trigger such an event (via kill, perhaps) to test that it works?

Please try running supervisor like so:

supervisord -n -edebug

This will run it in the foreground with the debug log printing to stdout,
including child process output, which might give you a clue.

- C

[Tom Barrett]

Post by Chris McDonough
supervisord -n -edebug
This will run it in the foreground with the debug log printing to stdout,
including child process output, which might give you a clue.

How should I simulate the php-cgi process dying? I don't know what
constitues an 'unexpected quit'. A simple 'kill <pid>' appears to be an
'expected quit'.

[Chris McDonough]

Post by Chris McDonough
supervisord -n -edebug
This will run it in the foreground with the debug log printing to
stdout, including child process output, which might give you a clue.
How should I simulate the php-cgi process dying? I don't know what
constitues an 'unexpected quit'. A simple 'kill <pid>' appears to be an
'expected quit'.

Nobody could know. I'd just let it run for a while. Note that you can run
supervisord in the background but still log at debug level via:

supervisord -edebug

The log message will end up in supervisord.log

- C

[Tom Barrett]

Post by Chris McDonough

Post by Chris McDonough
supervisord -n -edebug
This will run it in the foreground with the debug log printing to
stdout, including child process output, which might give you a clue.
How should I simulate the php-cgi process dying? I don't know what
constitues an 'unexpected quit'. A simple 'kill <pid>' appears to be an
'expected quit'.

Nobody could know. I'd just let it run for a while. Note that you can run
supervisord -edebug
The log message will end up in supervisord.log

So there is no way that I can simulate php-cgi going down to see whether
supervisor actually works? Sorry, but that strikes me as borderline useless.
I have, so far, absolutely no evidence that supervisor is any better than
launching programs from the shell and checking them manually.

How do I poke the beast to make it *do* something? Can I set supervisor to
restart php-cgi no matter why it dies? If not, then I give up.

[Chris McDonough]

Post by Chris McDonough
supervisord -n -edebug
This will run it in the foreground with the debug log printing to
stdout, including child process output, which might give you a clue.
How should I simulate the php-cgi process dying? I don't know
what constitues an 'unexpected quit'. A simple 'kill <pid>'
appears to be an 'expected quit'.
Nobody could know. I'd just let it run for a while. Note that you
supervisord -edebug
The log message will end up in supervisord.log
So there is no way that I can simulate php-cgi going down to see whether
supervisor actually works? Sorry, but that strikes me as borderline
useless. I have, so far, absolutely no evidence that supervisor is any
better than launching programs from the shell and checking them manually.
How do I poke the beast to make it *do* something? Can I set supervisor
to restart php-cgi no matter why it dies? If not, then I give up.

We're trying to figure out the *circumstance* in which the php process dies so
we can teach supervisord to detect it and restart the process *in that
circumstance*.

Since Supervisor reports an "EXITED" status for this process, it means that the
process exited "cleanly" as far as supervisor can tell; it didn't "crash" (at
least in Supervisor's eyes). It means the process exited with a status code
"cleanly" (the process itself called the UNIX system function "exit()").

Some exit codes are "expected". By default, these are exit status codes 0 and
2. When supervisor sees that a process has exited with an "expected" exit
code, by default it does not try to restart the process again. However, if
supervisor notices that the exit code is "unexpected" (not in the list of the
process' "exitcodes" setting, which by default is 0,2), and "autorestart" is
set to "unexpected", it will try to restart the process. It will keep trying
to restart the process until "startretries" number of restarts are attempted;
if it reaches startretries+1 without a successful startup, it gives up and puts
the process into the FATAL status.

You can use brute force by setting "autorestart" to "always" in the process
configuration, which will make Supervisor always restart the process, by god,
no matter what, but it's a heavy hammer, and may have side effects that make it
undesirable. It's an option.

A better option: By looking at the logs from supervisor when the process "dies"
(exits), it will tell you *which* exit code it exited with.. might be 0, might
be 2. Dunno. Take this exit code number out of the "exitcodes" list for the
process, and it will restart as long as "autorestart=unexpected".

And, FTR, indignant proclamations and threats of not using my software aren't
great motivators. Cynicism caused by age prevents that from inducing any
terror or empathy in me.

- C

[Tom Barrett]

Chris,

Apologies if I cam across as antagonistic. I was trying to convey a need for
something more user friend (perhaps I mean 'installer friendly'). I have
read several blog postings by people who have experienced great success with
Supervisor, and as unquanitifiable as those are, they are an excellent
indication.

I could get down to more details on my experience, but I daren't. I want to
use Supervisor - I want something that does what it says on Supervisor's box
(as it were).

I will continue with my feedback and questions, unless you wish me to stop.

The 'always restart' option does appear to be what I was looking for.
Perhaps not something to use in a production environment. But, whilst
testing, to force a situation where I can observe Supervisor's behaviour
during a restart, it would be very useful.

supervisord -edebug -n -c /etc/supervisord.conf

Error: invalid 'autorestart' value 'always'
For help, use /usr/bin/supervisord -h

It does not appear to be documented. Possibly it is not in the version
installed on that box. Or maybe you meant 'true'.

The good news is that I have got a version running 'out of the box' on a
test machine running CentOS 5.3 with Python 2.4.3. Which is a great starting
point. Hopefully I can revisit this in the new year.

[Chris McDonough]

Post by Tom Barrett
Chris,
Apologies if I cam across as antagonistic. I was trying to convey a need
for something more user friend (perhaps I mean 'installer friendly'). I
have read several blog postings by people who have experienced great
success with Supervisor, and as unquanitifiable as those are, they are
an excellent indication.
I could get down to more details on my experience, but I daren't. I want
to use Supervisor - I want something that does what it says on
Supervisor's box (as it were).
I will continue with my feedback and questions, unless you wish me to stop.
The 'always restart' option does appear to be what I was looking for.
Perhaps not something to use in a production environment. But, whilst
testing, to force a situation where I can observe Supervisor's behaviour
during a restart, it would be very useful.

supervisord -edebug -n -c /etc/supervisord.conf

Error: invalid 'autorestart' value 'always'
For help, use /usr/bin/supervisord -h
It does not appear to be documented. Possibly it is not in the version
installed on that box. Or maybe you meant 'true'.
The good news is that I have got a version running 'out of the box' on a
test machine running CentOS 5.3 with Python 2.4.3. Which is a great
starting point. Hopefully I can revisit this in the new year.

If that was a long way of saying thanks, you're welcome. ;-)

- C