[Supervisor-users] Thanks for a great tool --- and my first question

Discussion:

Luc Saffre

2016-07-28 16:49:54 UTC

Hi Supervisor users and developers,

this is my first post to this list, so I'd like to first say thanks for
such a great tool. I discovered it a month ago and start to use it more
and more on my production servers. I also wrote about it in my own
documentation: http://www.lino-framework.org/admin/linod.html

But now I discovered my first problem where I hope for your help.
I have a single configuration file written by myself:

$ cat /etc/supervisor/conf.d/linod_prod.conf
[program:prod_linod]
command=/bin/bash /var/www/vhosts/prod/linod.sh
username = www-data

The mentioned bash script contains:

$ cat /var/www/vhosts/prod/linod.sh
#!/bin/bash
set -e # exit on error
cd /var/www/vhosts/prod
. env/bin/activate
python manage.py linod

The process starts and works well. The problem is that Supervisor does
not correctly stop the process. Here is a console session:

$ ps aux | grep linod
1001 15828 0.0 0.0 8344 880 pts/0 S+ 18:07 0:00 grep linod

$ sudo service supervisor start
Starting supervisor: supervisord.

$ ps aux | grep linod
root 16279 0.0 0.0 19720 1668 ? S 18:31 0:00
/bin/bash /var/www/vhosts/prod/linod.sh
root 16284 58.8 0.8 328736 135368 ? S 18:31 0:02 python
manage.py linod
1001 16309 0.0 0.0 8344 884 pts/0 S+ 18:31 0:00 grep linod

Until here it looks good. I am a bit surprised to see two processes
here, but that might be caused by some magic.

The real problem is that Supervisor obviously does not see that spawned
second process and therefore doesn't terminate it. After every restart I
have one additional process running:

$ sudo service supervisor restart
Restarting supervisor: supervisord.
$ ps aux | grep linod
root 16284 6.0 0.8 328736 135368 ? S 18:31 0:02 python
manage.py linod
root 16322 0.0 0.0 19720 1672 ? S 18:32 0:00
/bin/bash /var/www/vhosts/prod/linod.sh
root 16327 106 0.7 306560 115280 ? R 18:32 0:02 python
manage.py linod
1001 16352 0.0 0.0 8344 884 pts/0 S+ 18:32 0:00 grep linod

Yes, I am not using the latest version. I have a default configuration
on a Debian Wheezy VPS:

$ uname -a
Linux SRV-LX1 3.2.0-4-amd64 #1 SMP Debian 3.2.78-1 x86_64 GNU/Linux

$ apt-cache show supervisor
Package: supervisor
State: installed
Version: 3.0a8-1.1+deb7u1
...

Any ideas?

Luc

Timothy Jones

2016-07-28 17:44:45 UTC

Permalink

I believe that supervisor is killing the 'bash' process, not the
underlying 'python manage.py linod' process.

There are two things you can do:
1) add a 'trap' command to your linod.sh script to catch the TERM signal
that supervisor is second, and in the handler, kill the underlying 'python
manage.py linod' process.
2) call the 'python manage.py linod' from python directly, taking care of
whatever the linod.sh program is doing elsewhere.

For example, if linod.sh just sets some environment variables before
launching 'python manage.py linod', then set them in the supervisor.conf
file (using 'environment' configuration variable). Or you can set these
things in the environment that launches supervisor (and supervisor will,
naturally, inherit them, and so will your child process 'python manage.py
linod').

A second guess might be that your script linod.sh isn't "trap"ping SIGTERM
and passing it along to the 'python manage.py linod' child process.

t
[1] http://supervisord.org/configuration.html?highlight=stopsignal
...........................................................................
.............................
Tim Bailey-Jones | Senior Advisory Systems Engineer, Development
+1 (813) 454.8643 work mobile | (813) 900.8131 personal mobile
Syniverse | We make mobile work.
***@syniverse.com | http://www.syniverse.com
<http://www.syniverse.com/>

Post by Luc Saffre
Hi Supervisor users and developers,
this is my first post to this list, so I'd like to first say thanks for
such a great tool. I discovered it a month ago and start to use it more
and more on my production servers. I also wrote about it in my own
documentation: http://www.lino-framework.org/admin/linod.html
But now I discovered my first problem where I hope for your help.
$ cat /etc/supervisor/conf.d/linod_prod.conf
[program:prod_linod]
command=/bin/bash /var/www/vhosts/prod/linod.sh
username = www-data
$ cat /var/www/vhosts/prod/linod.sh
#!/bin/bash
set -e # exit on error
cd /var/www/vhosts/prod
. env/bin/activate
python manage.py linod
The process starts and works well. The problem is that Supervisor does
$ ps aux | grep linod
1001 15828 0.0 0.0 8344 880 pts/0 S+ 18:07 0:00 grep linod
$ sudo service supervisor start
Starting supervisor: supervisord.
$ ps aux | grep linod
root 16279 0.0 0.0 19720 1668 ? S 18:31 0:00
/bin/bash /var/www/vhosts/prod/linod.sh
root 16284 58.8 0.8 328736 135368 ? S 18:31 0:02 python
manage.py linod
1001 16309 0.0 0.0 8344 884 pts/0 S+ 18:31 0:00 grep linod
Until here it looks good. I am a bit surprised to see two processes
here, but that might be caused by some magic.
The real problem is that Supervisor obviously does not see that spawned
second process and therefore doesn't terminate it. After every restart I
$ sudo service supervisor restart
Restarting supervisor: supervisord.
$ ps aux | grep linod
root 16284 6.0 0.8 328736 135368 ? S 18:31 0:02 python
manage.py linod
root 16322 0.0 0.0 19720 1672 ? S 18:32 0:00
/bin/bash /var/www/vhosts/prod/linod.sh
root 16327 106 0.7 306560 115280 ? R 18:32 0:02 python
manage.py linod
1001 16352 0.0 0.0 8344 884 pts/0 S+ 18:32 0:00 grep linod
Yes, I am not using the latest version. I have a default configuration
$ uname -a
Linux SRV-LX1 3.2.0-4-amd64 #1 SMP Debian 3.2.78-1 x86_64 GNU/Linux
$ apt-cache show supervisor
Package: supervisor
State: installed
Version: 3.0a8-1.1+deb7u1
...
Any ideas?
Luc
_______________________________________________
Supervisor-users mailing list
https://lists.supervisord.org/mailman/listinfo/supervisor-users

Paul Lockaby

2016-07-28 17:47:48 UTC

Permalink

Another option is to call it like this:

exec python manage.py linode

That has the added benefit of passing any error codes from your python script back to supervisor.

-Paul

Post by Timothy Jones
I believe that supervisor is killing the 'bash' process, not the
underlying 'python manage.py linod' process.
1) add a 'trap' command to your linod.sh script to catch the TERM signal
that supervisor is second, and in the handler, kill the underlying 'python
manage.py linod' process.
2) call the 'python manage.py linod' from python directly, taking care of
whatever the linod.sh program is doing elsewhere.
For example, if linod.sh just sets some environment variables before
launching 'python manage.py linod', then set them in the supervisor.conf
file (using 'environment' configuration variable). Or you can set these
things in the environment that launches supervisor (and supervisor will,
naturally, inherit them, and so will your child process 'python manage.py
linod').
A second guess might be that your script linod.sh isn't "trap"ping SIGTERM
and passing it along to the 'python manage.py linod' child process.
t
[1] http://supervisord.org/configuration.html?highlight=stopsignal
...........................................................................
.............................
Tim Bailey-Jones | Senior Advisory Systems Engineer, Development
+1 (813) 454.8643 work mobile | (813) 900.8131 personal mobile
Syniverse | We make mobile work.
<http://www.syniverse.com/>

_______________________________________________
Supervisor-users mailing list
https://lists.supervisord.org/mailman/listinfo/supervisor-users

Brent Tubbs

2016-07-28 18:20:16 UTC

Permalink

You can also use Supervisor's "stopasgroup" config option to have it stop
all child processes at the same time as the parent, instead of relying on
the parent to stop the children.

http://supervisord.org/configuration.html#program-x-section-settings

Post by Paul Lockaby
exec python manage.py linode
That has the added benefit of passing any error codes from your python
script back to supervisor.
-Paul

'python

Post by Timothy Jones
manage.py linod' process.
2) call the 'python manage.py linod' from python directly, taking care

Post by Timothy Jones
whatever the linod.sh program is doing elsewhere.
For example, if linod.sh just sets some environment variables before
launching 'python manage.py linod', then set them in the supervisor.conf
file (using 'environment' configuration variable). Or you can set these
things in the environment that launches supervisor (and supervisor will,
naturally, inherit them, and so will your child process 'python manage.py
linod').
A second guess might be that your script linod.sh isn't "trap"ping

SIGTERM

Post by Timothy Jones
and passing it along to the 'python manage.py linod' child process.
t
[1] http://supervisord.org/configuration.html?highlight=stopsignal

...........................................................................

Post by Timothy Jones
.............................
Tim Bailey-Jones | Senior Advisory Systems Engineer, Development
+1 (813) 454.8643 work mobile | (813) 900.8131 personal mobile
Syniverse | We make mobile work.
<http://www.syniverse.com/>

_______________________________________________
Supervisor-users mailing list
https://lists.supervisord.org/mailman/listinfo/supervisor-users

Timothy Jones

2016-07-28 18:21:01 UTC

Permalink

Thanks, Paul, that's better (and simpler!) than my idea.

t
...........................................................................
.............................
Tim Bailey-Jones | Senior Advisory Systems Engineer, Development
+1 (813) 454.8643 work mobile | (813) 900.8131 personal mobile
Syniverse | We make mobile work.
***@syniverse.com | http://www.syniverse.com
<http://www.syniverse.com/>

Post by Paul Lockaby
exec python manage.py linode
That has the added benefit of passing any error codes from your python
script back to supervisor.
-Paul

On Jul 28, 2016, at 10:44 AM, Timothy Jones
I believe that supervisor is killing the 'bash' process, not the
underlying 'python manage.py linod' process.
1) add a 'trap' command to your linod.sh script to catch the TERM signal
that supervisor is second, and in the handler, kill the underlying 'python
manage.py linod' process.
2) call the 'python manage.py linod' from python directly, taking care of
whatever the linod.sh program is doing elsewhere.
For example, if linod.sh just sets some environment variables before
launching 'python manage.py linod', then set them in the supervisor.conf
file (using 'environment' configuration variable). Or you can set these
things in the environment that launches supervisor (and supervisor will,
naturally, inherit them, and so will your child process 'python manage.py
linod').
A second guess might be that your script linod.sh isn't "trap"ping SIGTERM
and passing it along to the 'python manage.py linod' child process.
t
[1] http://supervisord.org/configuration.html?highlight=stopsignal
.........................................................................
..
.............................
Tim Bailey-Jones | Senior Advisory Systems Engineer, Development
+1 (813) 454.8643 work mobile | (813) 900.8131 personal mobile
Syniverse | We make mobile work.
<http://www.syniverse.com/>

_______________________________________________
Supervisor-users mailing list
https://lists.supervisord.org/mailman/listinfo/supervisor-users

Timothy Jones

2016-07-28 18:22:41 UTC

Permalink

Also, I noticed your child process is still running as 'root', not
'www-data' as intended.

This is probably because you have 'username=www-data' in your
supervisor.conf file instead of 'user=www-data'.

http://supervisord.org/configuration.html?highlight=stopsignal#program-x-se
ction-settings (search for 'user').

'userNAME' is for authentication for the webserver portion of supervisor.

...........................................................................
.............................
Tim Bailey-Jones | Senior Advisory Systems Engineer, Development
+1 (813) 454.8643 work mobile | (813) 900.8131 personal mobile
Syniverse | We make mobile work.
***@syniverse.com | http://www.syniverse.com
<http://www.syniverse.com/>

Luc Saffre

2016-07-28 19:47:27 UTC

Permalink

Thanks to everybody! Paul's suggestion of "exec" helped, and also that
username problem are now solved!
I wrote about all this in my blog:
http://luc.lino-framework.org/blog/2016/0728.html#supervisor-failed-to-terminate-linod

Luc

Post by Timothy Jones
Also, I noticed your child process is still running as 'root', not
'www-data' as intended.
This is probably because you have 'username=www-data' in your
supervisor.conf file instead of 'user=www-data'.
http://supervisord.org/configuration.html?highlight=stopsignal#program-x-se
ction-settings (search for 'user').
'userNAME' is for authentication for the webserver portion of supervisor.
...........................................................................
.............................
Tim Bailey-Jones | Senior Advisory Systems Engineer, Development
+1 (813) 454.8643 work mobile | (813) 900.8131 personal mobile
Syniverse | We make mobile work.
<http://www.syniverse.com/>

Paul Lockaby

2016-07-28 20:06:39 UTC

Permalink

I feel compelled to add a bit of explanation for why it worked, based on what you wrote on your blog post.

Timothy was correct in the cause of your problem. Your bash script would call python so you'd see a process tree like this:

supervisor
\-> bash
\-> python

When supervisor stops bash (with a TERM unless you've specified something different), bash doesn't send the signal to the programs that it is waiting for. This is pretty well explained here:

http://unix.stackexchange.com/questions/146756/forward-sigterm-to-child-in-bash

When you use "exec", your bash script completely replaces itself with the program that you exec. So your process starts like this:

supervisor
\-> bash

But turns into this after the exec call:

supervisor
\-> python

This does mean that after calling exec your bash script is done and disappears so I hope you don't have to do any cleanup after calling exec!

This is exactly the setup we do for our python programs. We have a small bash script that sets up the virtualenv for the program and then it execs python to run that program.

One other thing to be aware of that we ran into is that we were setting PYTHONPATH in our startup script for supervisor because supervisor was running in a non-standard location. This custom PYTHONPATH was being passed to our python programs by supervisor and royally screwing up the environment because we were running supervisor as python2 but our custom programs as python3.

-Paul

Post by Luc Saffre
Thanks to everybody! Paul's suggestion of "exec" helped, and also that
username problem are now solved!
http://luc.lino-framework.org/blog/2016/0728.html#supervisor-failed-to-terminate-linod
Luc

_______________________________________________
Supervisor-users mailing list
https://lists.supervisord.org/mailman/listinfo/supervisor-users

Luc Saffre

2016-07-28 20:58:42 UTC

Permalink

Thanks very much for the valuable explanations!

Luc

Post by Paul Lockaby
I feel compelled to add a bit of explanation for why it worked, based on what you wrote on your blog post.
supervisor
\-> bash
\-> python
http://unix.stackexchange.com/questions/146756/forward-sigterm-to-child-in-bash
supervisor
\-> bash
supervisor
\-> python
This does mean that after calling exec your bash script is done and disappears so I hope you don't have to do any cleanup after calling exec!
This is exactly the setup we do for our python programs. We have a small bash script that sets up the virtualenv for the program and then it execs python to run that program.
One other thing to be aware of that we ran into is that we were setting PYTHONPATH in our startup script for supervisor because supervisor was running in a non-standard location. This custom PYTHONPATH was being passed to our python programs by supervisor and royally screwing up the environment because we were running supervisor as python2 but our custom programs as python3.
-Paul