{"action":"created","issue":{"url":"https://api.github.com/repos/fish-shell/fish-shell/issues/4745","repository_url":"https://api.github.com/repos/fish-shell/fish-shell","labels_url":"https://api.github.com/repos/fish-shell/fish-shell/issues/4745/labels{/name}","comments_url":"https://api.github.com/repos/fish-shell/fish-shell/issues/4745/comments","events_url":"https://api.github.com/repos/fish-shell/fish-shell/issues/4745/events","html_url":"https://github.com/fish-shell/fish-shell/issues/4745","id":299289304,"number":4745,"title":"`set -gx SHELL fish` causes ssh to fail on mac when tunneling through a bastion server","user":{"login":"b33j0r","id":5395435,"avatar_url":"https://avatars0.githubusercontent.com/u/5395435?v=4","gravatar_id":"","url":"https://api.github.com/users/b33j0r","html_url":"https://github.com/b33j0r","followers_url":"https://api.github.com/users/b33j0r/followers","following_url":"https://api.github.com/users/b33j0r/following{/other_user}","gists_url":"https://api.github.com/users/b33j0r/gists{/gist_id}","starred_url":"https://api.github.com/users/b33j0r/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/b33j0r/subscriptions","organizations_url":"https://api.github.com/users/b33j0r/orgs","repos_url":"https://api.github.com/users/b33j0r/repos","events_url":"https://api.github.com/users/b33j0r/events{/privacy}","received_events_url":"https://api.github.com/users/b33j0r/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-02-22T10:01:48Z","updated_at":"2018-02-24T09:55:51Z","closed_at":null,"author_association":"NONE","body":"```sh\r\n~> ssh notafakedomain.cat\r\nfish: No such file or directory\r\nssh_exchange_identification: Connection closed by remote host\r\n```\r\n\r\n# Background\r\n\r\n**tl;dr:** *looks like some very masked problem with opening non-interactive subshells from commands such as `ssh -W`, when the startup of that shell misbehaves.*\r\n\r\nI devops for a client (mostly a python engineer/architect in other contexts, long story). The infrastructure uses a classic ssh bastion/jump server topology. Important because this issue only breaks my workflow when tunneling ssh via `ProxyCommand` or `ProxyJump`.\r\n\r\nIt's starting a new shell under the hood, and that's failing with ungooglable error messages. Oh, and preventing connections to production servers! Almost forgot 😛\r\n\r\nThe only thing I have found that has changed since it worked very well--which is pretty damn funny to me in retrospect--I added `set SHELL` to my `config.fish`\r\n\r\nJudging from previous conversations I was able to find, the resolution is likely in better education for users + upstream + downstream (#2160, #2405). I tried to SEO the crap out of my title to help my fellow seekers, apologies if that confuses anyone.\r\n\r\n## Environment\r\n\r\nTwo macs, unfortunately with very much identical configurations due to my `dotfiles` being a bit conformist.\r\n\r\n* macOS High Sierra 10.13.3\r\n* fish 2.7.1 (homebrew install)\r\n* had omf installed\r\n\r\nOMF seems unrelated. I uninstalled, reinstalled, etc, and ran the `home=tempdir` experiment. But you'll see that I believe this to be upstream to a certain extent, so it was worthy of mention.\r\n\r\n### `~/.ssh/config`:\r\n\r\nPlease accept this as a working ssh config:\r\n\r\n```\r\nHost bastion\r\n  HostName bastion.cats.blog.awesome\r\n  # other things here, identity, agent forwarding, muxing, etc\r\n\r\nHost service\r\n  HostName service.cats.blog.awesome\r\n  ProxyJump bastion\r\n```\r\n\r\nAs proof that it doesn't matter to the report, I was always able to do this:\r\n\r\n```\r\n~> ssh bastion\r\n# HI WELCOME TO OUR GLORIOUS BASTION SERVER\r\nb33j0r@bastion$ ssh service\r\n# NO PROBLEM, BRO!\r\nb33j0r@service ~>\r\n```\r\n\r\n### `~/.config/fish/config.fish`\r\n\r\n```\r\nset -gx PATH $HOME/bin $HOME/.pyenv/bin $HOME/.pyenv/shims /usr/local/bin /usr/local/sbin /usr/bin /bin /usr/sbin /sbin\r\nset -gx PYENV_ROOT $HOME/.pyenv\r\nset -gx EDITOR vim\r\nset -gx SHELL fish\r\n\r\nstatus --is-interactive; and source (pyenv init -|psub)\r\n\r\ntest -e {$HOME}/.iterm2_shell_integration.fish ; and source {$HOME}/.iterm2_shell_integration.fish\r\n```\r\n\r\n# Reproduce\r\n\r\nI don't know how to simulate this exact situation without having the actual network hosts... off of the top of my head. But if the above makes sense, here's what happens:\r\n\r\n## Failing Command\r\n\r\n```sh\r\n~> ssh service\r\nfish: No such file or directory\r\nssh_exchange_identification: Connection closed by remote host\r\n```\r\n\r\nImagine my surprise at seeing `fish` in that output! Makes more sense now, but it was weird for a long time. Led to a bunch of red-herrings, but that's debugging. Verbose with redacted paths:\r\n\r\n```sh\r\nOpenSSH_7.6p1, LibreSSL 2.6.2\r\ndebug1: Reading configuration data ~/.ssh/config\r\ndebug1: ~/.ssh/config line 1: Applying options for *\r\ndebug1: ~/.ssh/config line 53: Applying options for service\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 48: Applying options for *\r\ndebug1: Executing proxy command: exec ssh -q -W service bastion\r\ndebug1: identity file ~/.ssh/id_rsa type 0\r\ndebug1: key_load_public: No such file or directory\r\ndebug1: identity file ~/.ssh/id_rsa-cert type -1\r\ndebug1: key_load_public: No such file or directory\r\ndebug1: identity file service.pem type -1\r\ndebug1: key_load_public: No such file or directory\r\ndebug1: identity file service.pem-cert type -1\r\ndebug1: Local version string SSH-2.0-OpenSSH_7.6\r\ndebug1: permanently_drop_suid: 501\r\nfish: No such file or directory\r\nssh_exchange_identification: Connection closed by remote host\r\n```\r\n\r\n### Very Important Clues\r\n\r\nWhy is `fish` in this output? I'm obviously not even reaching the login shell. And in this case, `fish` isn't installed on `bastion` (some things are perplexingly out of my control, folks!) So I did this next, on my local host:\r\n\r\n```sh\r\n~> bash\r\nbash-3.2$ ssh service\r\nfish: No such file or directory\r\nssh_exchange_identification: Connection closed by remote host\r\n```\r\n\r\nWe start to outgrow my knowledge of shell and subshell implementation details, but I at least knew to try changing the login shell (although ignorantly, first on the remote hosts lol). This works on my local host without other changes:\r\n\r\n```\r\nchsh -s (which bash)\r\n```\r\n\r\n`<new session>`\r\n\r\n```\r\nugly-bash-prompt$ ssh service\r\n# NO PROBLEM, BRO!\r\nb33j0r@service ~> \r\n```\r\n\r\nSo it started to dawn on me that it had something to do with the startup of the subshell, and probably stdin/stdout/stderr (or terminal emulation things I don't remember from college). But the exact error was being masked, and I realized that I was out of my league on an analytical solution.\r\n\r\nSo I started messing with my fish config to eliminate variables. After a lot of unproductive experiments, I found the \"culprit\":\r\n\r\n### Startling Workaround\r\n\r\n```sh\r\n## `config.fish`\r\n\r\nset -gx PATH $HOME/bin $HOME/.pyenv/bin $HOME/.pyenv/shims /usr/local/bin /usr/local/sbin /usr/bin /bin /usr/sbin /sbin\r\nset -gx PYENV_ROOT $HOME/.pyenv\r\nset -gx EDITOR vim\r\n\r\n## this [indirectly] breaks ssh ProxyJump/ProxyCommand\r\n# set -gx SHELL fish\r\n\r\n## this last part was suspect after I learned of the nature of the problem,\r\n## but turns out to not matter. can be commented, deleted, unguarded...\r\n## no effect on any of the behaviors described in this issue\r\n\r\nif status --is-interactive\r\n\r\n  source (pyenv init -|psub)\r\n  test -e {$HOME}/.iterm2_shell_integration.fish ; and source {$HOME}/.iterm2_shell_integration.fish\r\n\r\nend\r\n```\r\n\r\n### Almost Definitely Related\r\n\r\nI'm very certain that it's not an \"ssh thing,\" but that's how it finally came up for me:\r\n\r\n* #2160\r\n* #2405\r\n\r\nIt's really hard to google, and honestly, requires a lot of experience to debug. I'm a senior-level guy in most of the related areas, and I didn't know what the hell questions to ask for days 😉 To quote the last mention I found:\r\n\r\n> [I feel like we've had this question a few times, although I can't see any direct duplicates - it might be worth adding to the FAQ if I'm not making that up.](https://github.com/fish-shell/fish-shell/issues/2405#issuecomment-286982566)\r\n\r\nTo top it off, a bitch to reproduce... when you're talking about bastion services and VPCs, everything has to be redacted to comfortably share enough information. So if you've read this far, I respect you!\r\n\r\nFinally, if it wasn't clear from above, I don't really mean that `set -gx` or the `SHELL` variable having a value is broken in `fish`. I have just reached the point where:\r\n\r\n1. There have gotta be domain experts who know how to fill in some of these blanks\r\n2. Create new and more recent searchable results for the next poor souls\r\n\r\nIf I only accomplish one of those goals, it will have been worth the time.\r\n\r\nPS: to make my point a little more forcibly about the difficulty of using search engines to find this topic... c'mon, who hasn't tried googling `terminal fish shell` or `shell fish change 256 color` ... imagine trying to search for `fish shell set SHELL fish` 😆 "},"comment":{"url":"https://api.github.com/repos/fish-shell/fish-shell/issues/comments/368216586","html_url":"https://github.com/fish-shell/fish-shell/issues/4745#issuecomment-368216586","issue_url":"https://api.github.com/repos/fish-shell/fish-shell/issues/4745","id":368216586,"user":{"login":"zanchey","id":520741,"avatar_url":"https://avatars1.githubusercontent.com/u/520741?v=4","gravatar_id":"","url":"https://api.github.com/users/zanchey","html_url":"https://github.com/zanchey","followers_url":"https://api.github.com/users/zanchey/followers","following_url":"https://api.github.com/users/zanchey/following{/other_user}","gists_url":"https://api.github.com/users/zanchey/gists{/gist_id}","starred_url":"https://api.github.com/users/zanchey/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zanchey/subscriptions","organizations_url":"https://api.github.com/users/zanchey/orgs","repos_url":"https://api.github.com/users/zanchey/repos","events_url":"https://api.github.com/users/zanchey/events{/privacy}","received_events_url":"https://api.github.com/users/zanchey/received_events","type":"User","site_admin":false},"created_at":"2018-02-24T09:55:51Z","updated_at":"2018-02-24T09:55:51Z","author_association":"MEMBER","body":"It's a local problem.\r\n\r\n```\r\n> tail -n6 .ssh/config\r\nHost bastion\r\n   Hostname localhost\r\n\r\nHost service\r\n   Hostname localhost\r\n   ProxyJump bastion\r\n> env SHELL=fish ssh service\r\nfish: No such file or directory\r\nssh_exchange_identification: Connection closed by remote host\r\n> env SHELL=asdffish ssh service\r\nasdffish: No such file or directory\r\nssh_exchange_identification: Connection closed by remote host\r\n> env SHELL=/usr/local/bin/fish ssh service\r\nPassword:\r\nPassword:\r\nLast login: Sat Feb 24 17:52:28 2018 from ::1\r\n```\r\n\r\nOn Linux, `SHELL=fish` works, but `SHELL=asdffish` doesn't (with a slightly different error: `write: Broken pipe`).\r\n\r\nTry setting your shell to a full path, as apparently SSH doesn't search your $PATH."}}
{"actor":{"display_login":"zanchey"}}