{"action":"opened","issue":{"url":"https://api.github.com/repos/StanfordBioinformatics/loom/issues/206","repository_url":"https://api.github.com/repos/StanfordBioinformatics/loom","labels_url":"https://api.github.com/repos/StanfordBioinformatics/loom/issues/206/labels{/name}","comments_url":"https://api.github.com/repos/StanfordBioinformatics/loom/issues/206/comments","events_url":"https://api.github.com/repos/StanfordBioinformatics/loom/issues/206/events","html_url":"https://github.com/StanfordBioinformatics/loom/issues/206","id":176780338,"number":206,"title":"Sporadic ssh connection failure","user":{"login":"nhammond","id":1828858,"avatar_url":"https://avatars.githubusercontent.com/u/1828858?v=3","gravatar_id":"","url":"https://api.github.com/users/nhammond","html_url":"https://github.com/nhammond","followers_url":"https://api.github.com/users/nhammond/followers","following_url":"https://api.github.com/users/nhammond/following{/other_user}","gists_url":"https://api.github.com/users/nhammond/gists{/gist_id}","starred_url":"https://api.github.com/users/nhammond/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nhammond/subscriptions","organizations_url":"https://api.github.com/users/nhammond/orgs","repos_url":"https://api.github.com/users/nhammond/repos","events_url":"https://api.github.com/users/nhammond/events{/privacy}","received_events_url":"https://api.github.com/users/nhammond/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2016-09-13T23:53:41Z","updated_at":"2016-09-13T23:53:41Z","closed_at":null,"body":"This error was observed once Sep 13 2016. After connecting to server by ssh, client was unable to connect during loom setup task.\r\n\r\nPLAYBOOK: gcloud_create_server.yml *********************************************\r\n3 plays in /home/nhammond/loom/loomengine/playbooks/gcloud_create_server.yml\r\n\r\nPLAY [Create a new instance for the Loom server.] ******************************\r\n\r\nTASK [Create a boot disk with the same name as the server instance.] ***********\r\ntask path: /home/nhammond/loom/loomengine/playbooks/gcloud_create_server.yml:17\r\n<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: nhammond\r\n<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p \"` echo $HOME/.ansible/tmp/ansible-tmp-1473786793.95-185575085426862 `\" && echo ansible-tmp-1473786793.95-185575085426862=\"` echo $HOME/.ansible/tmp/ansible-tmp-1473786793.95-185575085426862 `\" ) && sleep 0'\r\n<127.0.0.1> PUT /tmp/tmpYF2tXa TO /home/nhammond/.ansible/tmp/ansible-tmp-1473786793.95-185575085426862/gce_pd\r\n<127.0.0.1> EXEC /bin/sh -c 'LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 /home/nhammond/env/bin/python /home/nhammond/.ansible/tmp/ansible-tmp-1473786793.95-185575085426862/gce_pd; rm -rf \"/home/nhammond/.ansible/tmp/ansible-tmp-1473786793.95-185575085426862/\" > /dev/null 2>&1 && sleep 0'\r\nchanged: [localhost] => {\"changed\": true, \"disk_type\": \"pd-ssd\", \"image\": \"centos-7\", \"invocation\": {\"module_args\": {\"credentials_file\": null, \"detach_only\": null, \"disk_type\": \"pd-ssd\", \"image\": \"centos-7\", \"instance_name\": null, \"mode\": \"READ_WRITE\", \"name\": \"t-lm-srvr-101016\", \"pem_file\": null, \"project_id\": null, \"service_account_email\": null, \"size_gb\": \"163\", \"snapshot\": null, \"state\": \"present\", \"zone\": \"us-central1-a\"}, \"module_name\": \"gce_pd\"}, \"name\": \"t-lm-srvr-101016\", \"size_gb\": 163, \"state\": \"present\", \"zone\": \"us-central1-a\"}\r\n\r\nTASK [Boot up a new instance using the just-created boot disk.] ****************\r\ntask path: /home/nhammond/loom/loomengine/playbooks/gcloud_create_server.yml:19\r\n<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: nhammond\r\n<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p \"` echo $HOME/.ansible/tmp/ansible-tmp-1473786798.75-111157350097054 `\" && echo ansible-tmp-1473786798.75-111157350097054=\"` echo $HOME/.ansible/tmp/ansible-tmp-1473786798.75-111157350097054 `\" ) && sleep 0'\r\n<127.0.0.1> PUT /tmp/tmpiM2e4v TO /home/nhammond/.ansible/tmp/ansible-tmp-1473786798.75-111157350097054/gce\r\n<127.0.0.1> EXEC /bin/sh -c 'LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 /home/nhammond/env/bin/python /home/nhammond/.ansible/tmp/ansible-tmp-1473786798.75-111157350097054/gce; rm -rf \"/home/nhammond/.ansible/tmp/ansible-tmp-1473786798.75-111157350097054/\" > /dev/null 2>&1 && sleep 0'\r\nchanged: [localhost] => {\"changed\": true, \"instance_data\": [{\"disks\": [\"t-lm-srvr-101016\"], \"image\": \"centos-7-v20160803\", \"machine_type\": \"n1-standard-1\", \"metadata\": {}, \"name\": \"t-lm-srvr-101016\", \"network\": \"loom-legacy-network\", \"private_ip\": \"10.128.0.15\", \"public_ip\": \"104.198.218.8\", \"status\": \"RUNNING\", \"tags\": [\"loom-server\", \"http-server\", \"https-server\", \"loom-server-uscentral\"], \"zone\": \"us-central1-a\"}], \"invocation\": {\"module_args\": {\"credentials_file\": null, \"disk_auto_delete\": true, \"disks\": [\"t-lm-srvr-101016\"], \"external_ip\": \"ephemeral\", \"image\": \"debian-7\", \"instance_names\": null, \"ip_forward\": false, \"machine_type\": \"n1-standard-1\", \"metadata\": null, \"name\": \"t-lm-srvr-101016\", \"network\": \"loom-legacy-network\", \"pem_file\": null, \"persistent_boot_disk\": false, \"preemptible\": null, \"project_id\": null, \"service_account_email\": null, \"service_account_permissions\": [\"cloud-platform\"], \"state\": \"present\", \"tags\": [\"loom-server\", \"http-server\", \"https-server\", \"loom-server-uscentral\"], \"zone\": \"us-central1-a\"}, \"module_name\": \"gce\"}, \"name\": \"t-lm-srvr-101016\", \"state\": \"present\", \"zone\": \"us-central1-a\"}\r\n\r\nTASK [Wait for SSH port to be available.] **************************************\r\ntask path: /home/nhammond/loom/loomengine/playbooks/gcloud_create_server.yml:22\r\n<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: nhammond\r\n<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p \"` echo $HOME/.ansible/tmp/ansible-tmp-1473786817.78-53739523572642 `\" && echo ansible-tmp-1473786817.78-53739523572642=\"` echo $HOME/.ansible/tmp/ansible-tmp-1473786817.78-53739523572642 `\" ) && sleep 0'\r\n<127.0.0.1> PUT /tmp/tmpcFsC_P TO /home/nhammond/.ansible/tmp/ansible-tmp-1473786817.78-53739523572642/wait_for\r\n<127.0.0.1> EXEC /bin/sh -c 'LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 /home/nhammond/env/bin/python /home/nhammond/.ansible/tmp/ansible-tmp-1473786817.78-53739523572642/wait_for; rm -rf \"/home/nhammond/.ansible/tmp/ansible-tmp-1473786817.78-53739523572642/\" > /dev/null 2>&1 && sleep 0'\r\nok: [localhost] => (item={u'status': u'RUNNING', u'network': u'loom-legacy-network', u'zone': u'us-central1-a', u'tags': [u'loom-server', u'http-server', u'https-server', u'loom-server-uscentral'], u'image': u'centos-7-v20160803', u'disks': [u't-lm-srvr-101016'], u'name': u't-lm-srvr-101016', u'public_ip': u'104.198.218.8', u'private_ip': u'10.128.0.15', u'machine_type': u'n1-standard-1', u'metadata': {}}) => {\"changed\": false, \"elapsed\": 11, \"invocation\": {\"module_args\": {\"connect_timeout\": 5, \"delay\": 10, \"exclude_hosts\": null, \"host\": \"10.128.0.15\", \"path\": null, \"port\": 22, \"search_regex\": \"OpenSSH\", \"state\": \"started\", \"timeout\": 300}, \"module_name\": \"wait_for\"}, \"item\": {\"disks\": [\"t-lm-srvr-101016\"], \"image\": \"centos-7-v20160803\", \"machine_type\": \"n1-standard-1\", \"metadata\": {}, \"name\": \"t-lm-srvr-101016\", \"network\": \"loom-legacy-network\", \"private_ip\": \"10.128.0.15\", \"public_ip\": \"104.198.218.8\", \"status\": \"RUNNING\", \"tags\": [\"loom-server\", \"http-server\", \"https-server\", \"loom-server-uscentral\"], \"zone\": \"us-central1-a\"}, \"path\": null, \"port\": 22, \"search_regex\": \"OpenSSH\", \"state\": \"started\"}\r\n\r\nTASK [Add host to new_instances.] **********************************************\r\ntask path: /home/nhammond/loom/loomengine/playbooks/gcloud_create_server.yml:25\r\ncreating host via 'add_host': hostname=10.128.0.15\r\nchanged: [localhost] => (item={u'status': u'RUNNING', u'network': u'loom-legacy-network', u'zone': u'us-central1-a', u'tags': [u'loom-server', u'http-server', u'https-server', u'loom-server-uscentral'], u'image': u'centos-7-v20160803', u'disks': [u't-lm-srvr-101016'], u'name': u't-lm-srvr-101016', u'public_ip': u'104.198.218.8', u'private_ip': u'10.128.0.15', u'machine_type': u'n1-standard-1', u'metadata': {}}) => {\"add_host\": {\"groups\": [\"new_instances\"], \"host_name\": \"10.128.0.15\", \"host_vars\": {}}, \"changed\": true, \"invocation\": {\"module_args\": {\"groupname\": \"new_instances\", \"hostname\": \"10.128.0.15\"}, \"module_name\": \"add_host\"}, \"item\": {\"disks\": [\"t-lm-srvr-101016\"], \"image\": \"centos-7-v20160803\", \"machine_type\": \"n1-standard-1\", \"metadata\": {}, \"name\": \"t-lm-srvr-101016\", \"network\": \"loom-legacy-network\", \"private_ip\": \"10.128.0.15\", \"public_ip\": \"104.198.218.8\", \"status\": \"RUNNING\", \"tags\": [\"loom-server\", \"http-server\", \"https-server\", \"loom-server-uscentral\"], \"zone\": \"us-central1-a\"}}\r\n\r\nPLAY [Set up the Loom server on the new instance.] *****************************\r\n\r\nTASK [setup] *******************************************************************\r\n<10.128.0.15> ESTABLISH SSH CONNECTION FOR USER: None\r\n<10.128.0.15> SSH: EXEC ssh -C -q -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile=\"/home/nhammond/.ssh/google_compute_engine\"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/nhammond/.ansible/cp/ansible-ssh-%h-%p-%r 10.128.0.15 '/bin/sh -c '\"'\"'( umask 77 && mkdir -p \"` echo $HOME/.ansible/tmp/ansible-tmp-1473786829.26-57935177421940 `\" && echo ansible-tmp-1473786829.26-57935177421940=\"` echo $HOME/.ansible/tmp/ansible-tmp-1473786829.26-57935177421940 `\" ) && sleep 0'\"'\"''\r\nfatal: [10.128.0.15]: UNREACHABLE! => {\"changed\": false, \"msg\": \"Failed to connect to the host via ssh.\", \"unreachable\": true}\r\nto retry, use: --limit @/home/nhammond/loom/loomengine/playbooks/gcloud_create_server.retry\r\n\r\nPLAY RECAP *********************************************************************\r\n10.128.0.15                : ok=0    changed=0    unreachable=1    failed=0   \r\nlocalhost                  : ok=4    changed=3    unreachable=0    failed=0   \r\n\r\nCreating or updating ~/.loom/gce.ini...\r\nCreating deploy settings /home/nhammond/.loom/gcloud_deploy_settings.ini using user settings /home/nhammond/testsettings.ini...\r\nConfiguring SSH keys for gcloud..."}}
{"actor":{"display_login":"nhammond"}}