Joe Walnes
2016-12-14 23:52:38 UTC
Hello supervisor developers - thank you for a great product!
I have been working on a patch that makes it easier to
manage remote instances in a multi-user, multi-server environment. But
before I go further, I'd appreciate some feedback on if it's worthwhile to
other users, or alternate implementations. I'm not tied to it and happy to
try another approach if there is one.
tl;dr -
$ supervisorctl -s ssh://***@remotemachine/path/to/supervisord.sock
== Proposal ==
Allow supervisorctl to connect to remote supervisord processes using unix
socket over ssh (in addition to the existing http and local unix socket
connections).
This would be through an ssh scheme in serverurl:
ssh://[user@]host/path-to-unix-socket
== Rationale ==
Remote management is a useful feature of supervisor, however the inet HTTP
protocol has some drawbacks. Tunneling domain sockets over ssh has the
following benefits over inet HTTP:
1. encrypted (without having to setup additional infrastructure)
2. uses existing ssh key management strategies, rather than shared plain
text password. Users will typically already have an authorized ssh key for
the machine they use, so it "just works" out the box
3. allows for unix based permission control (e.g. multiple users log in
with their own key, and may or may not have access to the socket based on
local file permissions)
4. does not require an additional open listen port (avoid conflicts in
multi user environments, less friction with firewalls, convenient for VPS
style setups)
This would not replace HTTP - it will just be an alternative.
== Changes ==
1. New class: supervisor.xmlrpc.SshUnixStreamHTTPConnection, which will
handle the ssh connection and domain socket tunneling (see below for how
this could work)
2. Update supervisor.xmlrpc.SupervisorTransport.__init__() to check for
'ssh://' url and use above class.
3. Docs
These changes only affect the supervisorctl. No changes to supervisord or
the config format are required.
== Implementation of ssh tunnel ==
A few alternatives...
[Option 1]: OpenSSH 6.7 now natively supports tunneling unix sockets over
the command line with -L. supervisord could launch this as a subprocess and
connect to the local unix (tunneled) socket.
Pros:
- Super easy to implement. Allow regular ssh client to do all the hard work.
Cons:
- Requires OpenSSH 6.7 (release 2014), which is still not widely used. I
just check
However, 6.7 (released 2014) is not widely used. For example, this would
not work on stock MacOS or Ubuntu 14.04.
[Option 2]: Use Paramiko Python SSH library.
Pros:
- Relatively easy to implement.
Cons:
- Adds a dependency. Ideally this should be optional
(setup.py extras_require) so as not to add friction for users who don't
need this feature. If a user attempts to use an ssh:// url without the
dependency, it provide a helpful error msg explaining they need the lib.
[Option 3]: Run a regular remote SSH command with TCP port forwarding (-L),
and invoke a server side script to connect the server side local TCP port
to the unix socket.
Pros:
- Will work with virtually any SSH client. Even old ones.
- Adds no dependencies.
Cons:
- Requires an element of server side execution - adding complexity. It
could automatically send Python over the wire, on-demand. Alternatively it
could be a standard part of the supervisord installation.
[Option 4]: Hybrid. Implement more than one of the above options, and auto
detect the best approach.
My preferred approach is option 2.
== Other considerations ==
* Should supervisortctl (ClientOptions) attempt to normalize the path? e.g.
"ssh://***@host/~/relative.sock". Git ssh urls supports this. This path
would have to be normalized server side, which adds complications.
* Should users be able override the ssh cmd? For example, git allows users
to change the ssh binary or pass custom args using GIT_SSH /
GIT_SSH_COMMAND environment vars.
* Should the path also accept a path to a supervisord.conf file, where it
can read the path to the unix socket. Additionally, should it be optional,
and it can look in the default locations as defined in Options.searchpaths.
e.g. "supervisorctl -s ssh://myhost" is the most minimal form.
Thank you for your consideration. If you like this proposal, I'm happy to
submit a pull request.
-Joe
(@joewalnes in the usual places)
I have been working on a patch that makes it easier to
manage remote instances in a multi-user, multi-server environment. But
before I go further, I'd appreciate some feedback on if it's worthwhile to
other users, or alternate implementations. I'm not tied to it and happy to
try another approach if there is one.
tl;dr -
$ supervisorctl -s ssh://***@remotemachine/path/to/supervisord.sock
== Proposal ==
Allow supervisorctl to connect to remote supervisord processes using unix
socket over ssh (in addition to the existing http and local unix socket
connections).
This would be through an ssh scheme in serverurl:
ssh://[user@]host/path-to-unix-socket
== Rationale ==
Remote management is a useful feature of supervisor, however the inet HTTP
protocol has some drawbacks. Tunneling domain sockets over ssh has the
following benefits over inet HTTP:
1. encrypted (without having to setup additional infrastructure)
2. uses existing ssh key management strategies, rather than shared plain
text password. Users will typically already have an authorized ssh key for
the machine they use, so it "just works" out the box
3. allows for unix based permission control (e.g. multiple users log in
with their own key, and may or may not have access to the socket based on
local file permissions)
4. does not require an additional open listen port (avoid conflicts in
multi user environments, less friction with firewalls, convenient for VPS
style setups)
This would not replace HTTP - it will just be an alternative.
== Changes ==
1. New class: supervisor.xmlrpc.SshUnixStreamHTTPConnection, which will
handle the ssh connection and domain socket tunneling (see below for how
this could work)
2. Update supervisor.xmlrpc.SupervisorTransport.__init__() to check for
'ssh://' url and use above class.
3. Docs
These changes only affect the supervisorctl. No changes to supervisord or
the config format are required.
== Implementation of ssh tunnel ==
A few alternatives...
[Option 1]: OpenSSH 6.7 now natively supports tunneling unix sockets over
the command line with -L. supervisord could launch this as a subprocess and
connect to the local unix (tunneled) socket.
Pros:
- Super easy to implement. Allow regular ssh client to do all the hard work.
Cons:
- Requires OpenSSH 6.7 (release 2014), which is still not widely used. I
just check
However, 6.7 (released 2014) is not widely used. For example, this would
not work on stock MacOS or Ubuntu 14.04.
[Option 2]: Use Paramiko Python SSH library.
Pros:
- Relatively easy to implement.
Cons:
- Adds a dependency. Ideally this should be optional
(setup.py extras_require) so as not to add friction for users who don't
need this feature. If a user attempts to use an ssh:// url without the
dependency, it provide a helpful error msg explaining they need the lib.
[Option 3]: Run a regular remote SSH command with TCP port forwarding (-L),
and invoke a server side script to connect the server side local TCP port
to the unix socket.
Pros:
- Will work with virtually any SSH client. Even old ones.
- Adds no dependencies.
Cons:
- Requires an element of server side execution - adding complexity. It
could automatically send Python over the wire, on-demand. Alternatively it
could be a standard part of the supervisord installation.
[Option 4]: Hybrid. Implement more than one of the above options, and auto
detect the best approach.
My preferred approach is option 2.
== Other considerations ==
* Should supervisortctl (ClientOptions) attempt to normalize the path? e.g.
"ssh://***@host/~/relative.sock". Git ssh urls supports this. This path
would have to be normalized server side, which adds complications.
* Should users be able override the ssh cmd? For example, git allows users
to change the ssh binary or pass custom args using GIT_SSH /
GIT_SSH_COMMAND environment vars.
* Should the path also accept a path to a supervisord.conf file, where it
can read the path to the unix socket. Additionally, should it be optional,
and it can look in the default locations as defined in Options.searchpaths.
e.g. "supervisorctl -s ssh://myhost" is the most minimal form.
Thank you for your consideration. If you like this proposal, I'm happy to
submit a pull request.
-Joe
(@joewalnes in the usual places)