{"action":"created","issue":{"url":"https://api.github.com/repos/cloudfoundry/bosh/issues/1399","repository_url":"https://api.github.com/repos/cloudfoundry/bosh","labels_url":"https://api.github.com/repos/cloudfoundry/bosh/issues/1399/labels{/name}","comments_url":"https://api.github.com/repos/cloudfoundry/bosh/issues/1399/comments","events_url":"https://api.github.com/repos/cloudfoundry/bosh/issues/1399/events","html_url":"https://github.com/cloudfoundry/bosh/issues/1399","id":171440258,"number":1399,"title":"Unable to ssh-in to GCE deployments `Connection closed by foreign host`","user":{"login":"cunnie","id":1020675,"avatar_url":"https://avatars.githubusercontent.com/u/1020675?v=3","gravatar_id":"","url":"https://api.github.com/users/cunnie","html_url":"https://github.com/cunnie","followers_url":"https://api.github.com/users/cunnie/followers","following_url":"https://api.github.com/users/cunnie/following{/other_user}","gists_url":"https://api.github.com/users/cunnie/gists{/gist_id}","starred_url":"https://api.github.com/users/cunnie/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cunnie/subscriptions","organizations_url":"https://api.github.com/users/cunnie/orgs","repos_url":"https://api.github.com/users/cunnie/repos","events_url":"https://api.github.com/users/cunnie/events{/privacy}","received_events_url":"https://api.github.com/users/cunnie/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2016-08-16T15:11:58Z","updated_at":"2016-08-17T17:13:09Z","closed_at":null,"body":"[firstboot.sh](https://github.com/cloudfoundry/bosh/blob/afe7a27e51430f275a7bb1489efc8c5ffca11a9b/stemcell_builder/stages/base_ubuntu_firstboot/assets/root/firstboot.sh#L5) fails intermittently on  GCE, which causes the VM's ssh keys not to be generated, which in turn prevents sshd from allowing inbound connections.\r\n\r\nThe problem is caused by the `dpkg-reconfigure -fnoninteractive -pcritical openssh-server` line, which fails as follows:\r\n\r\n```\r\nCreating SSH2 RSA key; this may take some time ...gethostname: File name too long\r\n```\r\n\r\nThe hostname is often something cryptic, e.g. `vm-0eb85710-a7e5-4079-59d0-7fff6f3bf16d.c.blabbertabber.internal` (a 65-character hostname).\r\n\r\nWithin ~5 seconds, the hostname will be set to a name that's not too long, e.g. `ab12aa79-f2f6-4833-95ad-e86141c2db6c`, at which point `dpkg-reconfigure` will succeed.\r\n\r\n#### I plan to submit a PR to fix by looping repeatedly until the command succeeds\r\n\r\nI have attached the output from specially-instrumented `firstboot.sh`'s\r\n\r\n[firstboot.0.758.txt](https://github.com/cloudfoundry/bosh/files/420576/firstboot.0.758.txt)\r\n\r\n[firstboot.0.sh.txt](https://github.com/cloudfoundry/bosh/files/420580/firstboot.0.sh.txt)\r\n\r\n[firstboot.1.757.txt](https://github.com/cloudfoundry/bosh/files/420575/firstboot.1.757.txt)\r\n\r\n[firstboot.1.sh.txt](https://github.com/cloudfoundry/bosh/files/420581/firstboot.1.sh.txt)\r\n\r\n/cc @evandbrown @cppforlife \r\n\r\n"},"comment":{"url":"https://api.github.com/repos/cloudfoundry/bosh/issues/comments/240480616","html_url":"https://github.com/cloudfoundry/bosh/issues/1399#issuecomment-240480616","issue_url":"https://api.github.com/repos/cloudfoundry/bosh/issues/1399","id":240480616,"user":{"login":"cunnie","id":1020675,"avatar_url":"https://avatars.githubusercontent.com/u/1020675?v=3","gravatar_id":"","url":"https://api.github.com/users/cunnie","html_url":"https://github.com/cunnie","followers_url":"https://api.github.com/users/cunnie/followers","following_url":"https://api.github.com/users/cunnie/following{/other_user}","gists_url":"https://api.github.com/users/cunnie/gists{/gist_id}","starred_url":"https://api.github.com/users/cunnie/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cunnie/subscriptions","organizations_url":"https://api.github.com/users/cunnie/orgs","repos_url":"https://api.github.com/users/cunnie/repos","events_url":"https://api.github.com/users/cunnie/events{/privacy}","received_events_url":"https://api.github.com/users/cunnie/received_events","type":"User","site_admin":false},"created_at":"2016-08-17T17:13:08Z","updated_at":"2016-08-17T17:13:08Z","body":"Timeline:\r\n\r\n| Time | Event |\r\n|------|--------|\r\n| 15:00:10 | `/var/log/syslog` reports dhclient sends ack/receives reply |\r\n| 15:00:10 | audispd reports `/bin/hostname` |\r\n| 15:00:11 | `/var/log/syslog` reports dhclient sets IP addr |\r\n| 15:00:16 | `firstboot.sh` starts |\r\n| 15:00:19 | bosh-agent re-writes `dhclient.conf` |\r\n| 15:00:19 | audispd reports `/bin/hostname` |\r\n| 15:00:19 | `/var/log/syslog` reports dhclient sends ack/receives reply |\r\n| 15:00:24 | ¿bosh-agent? sets short-enough hostname |\r\n\r\n* Built stemcell with modified `/etc/dhcp/dhclient.conf`\r\n* Failure, log file (`/root/firstboot.758`) showed too-long hostname:\r\n```\r\nCreating SSH2 RSA key; this may take some time ...gethostname: File name too long\r\n+ RECONFIGURED=1\r\n+ hostname\r\nvm-95203f50-b480-437b-7eab-793d7521c883.c.blabbertabber.internal\r\n```\r\n* within 8 seconds it received a workable (shorter hostname).\r\n* the timestamp on the log file was 2016-08-17 15:00:24.039833000 (via `stat`)\r\n* the \"bad\" hostname was set at 15:00:16 or earlier\r\n* the dhclient file was re-written by `bosh-agent` at 15:00:19 (`stat /etc/dhcp/dhclient.conf`); this indicates that the bosh-agent's \"new\" file with \"host-name\""}}
{"actor":{"display_login":"cunnie"}}