All mestro commands hang indefinitely when using SSH tunnelling

[dangorst1066]

When I try to use any commands (status/start etc), maestro hangs indefinitely. The hanging occurs in bgtunnel.py in _validate_ssh_process

I can see bgtunnel has opened up an SSH tunnel to the docker host, something like this:

152638154 47788 47783   0 10:01am ttys003    0:00.02 /usr/bin/ssh -o BatchMode=yes -i /Users/jbloggs/.ssh/id_rsa -T -p 22 -L 127.0.0.1:57656:127.0.0.1:4243 jbloggs@10.20.30.40

Interestingly, I can also confirm the tunnel is working fine using curl on the command line:

$ curl 127.0.0.1:57656/images/json?all
[{"Created":1406811123,"Id":"be45fb9546dda26f073dfb9bb2cb232377d22e4ff9138280de831c5a49f38f60","ParentId":"bb18aaa8b7a92176b13aea9eba872cb3a9a9301e66c3a3aad04fbd216d1e9baf","RepoTags":["quay.io/somerepo/somesvc:1.0.0"],"Size":0,"VirtualSize":822164249}

Playing with the maestro code, I added in bgtunnel's expect_hello option and set it to false just to confirm that was not a problem (not including that as an option might block some servers from functioning with maestro), but that just results in getting 'host down' almost immediately when doing maestro status

Anyone got any ideas?

Mestro - 0.2.2
Environments - tried Ubuntu Precise and Mac OS X 10.9.2
Python - tried 2.7.3 and 2.7.5
Docker - 0.11

[mpetazzoni]

I can definitely reproduce the problem. It seems to be depending on the target host though, not necessarily on the Maestro/Python versions. Investigating...

[mpetazzoni]

Hmm, I concerned this is going to be hard to fix. Here are my findings. As you mentioned, the problem comes from _validate_ssh_process(), which waits until some output comes from the SSH process (presumably, the prompt), before considering the connection ready.

The expect_hello parameter can be set to false to accommodate for servers that don't serve a prompt on successful auth (as described in jacobsvante/bgtunnel@1608e99), but then the connection is considered to be immediately ready, which isn't necessarily true. Adding a wait time after the call to bgtunnel.open() makes Maestro work in this situation, but that's not a viable or even remotely reliable fix.

When trying the command ran by bgtunnel, I found that the -T option that bgtunnel adds causes the connection to not output the prompt on successful auth. But it doesn't seem like there is a way to override this behavior.

What's weird is that I don't have this problem with all the SSH servers I'm connecting to. For example, when establishing a tunnel to a Ubuntu VM running OpenSSH_6.6.1p1 Ubuntu-2ubuntu2, OpenSSL 1.0.1f 6 Jan 2014, it works fine as is, with expect_hello set to True. But when connecting to some other machine (running 6.2p2), it doesn't work. But these are just two isolated data points, it might not have anything to do with the remote OpenSSH version.

[dangorst1066]

@mpetazzoni - thanks for taking a look at this and raising it with the bgtunnel folks... In the meantime still keen to play with maestro in our environment so I added in a little option to use existing tunnels in the maestro config yaml (let's face it - they're reasonably easy to script up). Not a sustainable solution though, so I'll pass on making a pull request with the patch :)

[dexterbt1]

+1
Was hit by this as well.
I'm running maestro on Ubuntu 14.04, attempting to tunnel to an EC2 instance with Amazon Linux.

[mpetazzoni]

If one of you has some time, can you try with a hacked version of bgtunnel.py that doesn't add the -T option to the SSH command args?

[dexterbt1]

Commented the -T in bgtunnel.py.
Raised this:

Error:Pseudo-terminal will not be allocated because stdin is not a terminal.

[ianblenke]

Ran into this one as as well, running deis 0.10.0 which uses CoreOS 379.3 and docker 1.1.1

[soby]

As a workaround, I believe you can set the Banner line in /etc/ssh/sshd_config to a file with a bit of text in it

[ianblenke]

Unfortunately coreos mounts /usr as read-only, so editing /etc/ssh/sshd_config directly is out of the question.

Thankfully, the files in /etc/ssh are mutable, so I can:

sudo mv /etc/ssh/sshd_config /etc/ssh/sshd_config.old
sudo cp /etc/ssh/sshd_config.old /etc/ssh/sshd_config
sudo bash -c 'echo Banner > /etc/ssh/sshd-banner'
sudo bash -c 'echo "Banner /etc/ssh/sshd-banner" >> /etc/ssh/sshd_config'

However, after doing this, when I run maestro I get an Error:

$ maestro status
Error: Banner

The first line of anything I put in the Banner file appears as an error when running status.

Note that I was able to get this working on another AWS VPC host on the same subnet using Docker port connections directly using the private IP addresses of the coreos host nodes.

It would be nice to be able to use ssh tunneling though, as then I could run maestro remotely.

[gogochan]

It wouldn't be a bad idea to allow expect_hello to be set to False on yaml file.

[x4lldux]

Any progress? I've added a banner but also get the same problem:

$ maestro status
Error: Banner

[corvis]

Hey guys, just struggled with the same problem. SSHD outputs the banner in stderr which causes validation fail on the bgtunel side.

Anyway to workaround this issue you need to setup you server to write something in STDOUT after login. My solution was to add this line:

echo "Deployer connected"

at the end of .bashrc file of the user I use in maestro.
So just login on you server with a user you want to use in maestro file and run the following:

echo "echo \"Deployer connected\"" >> ~/.bashrc

[mpetazzoni]

Thanks for the workaround @corvis.

Unfortunately there isn't much Maestro can do here to help. jacobsvante/bgtunnel#7 tracks the issue already on bgtunnel's side.