{"action":"created","issue":{"url":"https://api.github.com/repos/StanfordBioinformatics/loom/issues/238","repository_url":"https://api.github.com/repos/StanfordBioinformatics/loom","labels_url":"https://api.github.com/repos/StanfordBioinformatics/loom/issues/238/labels{/name}","comments_url":"https://api.github.com/repos/StanfordBioinformatics/loom/issues/238/comments","events_url":"https://api.github.com/repos/StanfordBioinformatics/loom/issues/238/events","html_url":"https://github.com/StanfordBioinformatics/loom/issues/238","id":188350035,"number":238,"title":"Failure provisioning worker and creating task run","user":{"login":"isaacliao","id":1683569,"avatar_url":"https://avatars.githubusercontent.com/u/1683569?v=3","gravatar_id":"","url":"https://api.github.com/users/isaacliao","html_url":"https://github.com/isaacliao","followers_url":"https://api.github.com/users/isaacliao/followers","following_url":"https://api.github.com/users/isaacliao/following{/other_user}","gists_url":"https://api.github.com/users/isaacliao/gists{/gist_id}","starred_url":"https://api.github.com/users/isaacliao/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/isaacliao/subscriptions","organizations_url":"https://api.github.com/users/isaacliao/orgs","repos_url":"https://api.github.com/users/isaacliao/repos","events_url":"https://api.github.com/users/isaacliao/events{/privacy}","received_events_url":"https://api.github.com/users/isaacliao/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-11-09T20:54:47Z","updated_at":"2016-11-11T00:35:54Z","closed_at":null,"body":"From loom_master.log:\r\n```\r\nERROR [2016-11-09 20:36:24,388] Failed to provision host.\r\nTraceback (most recent call last):\r\n  File \"/opt/loom/loomengine/master/api/task_manager/cloud.py\", line 121, in _run\r\n    cls._run_playbook(GCLOUD_CREATE_WORKER_PLAYBOOK, playbook_vars, logfile=ansible_logfile)\r\n  File \"/opt/loom/loomengine/master/api/task_manager/cloud.py\", line 169, in _run_playbook\r\n    raise Exception('Nonzero returncode %s for command: %s' % (returncode, ' '.join(cmd)))\r\nException: Nonzero returncode 2 for command: ansible-playbook -vvv --key-file /root/.ssh/google_compute_engine -i /opt/loom/loomengine/utils/gce.py /opt/loom/loomengine/playbooks/gcloud_create_worker.yml --extra-vars {\"docker_full_name\": \"loomengine/loom:customserviceaccounts\", \"gce_ssh_key_file\": \"/root/.ssh/google_compute_engine\", \"boot_disk_type\": \"pd-ssd\", \"task_run_attempt_id\": \"9a345d1c31bb40379528d2bd29efdd70\", \"worker_log_file\": \"/mnt/loomdata/runtime_volumes/9a345d1c31bb40379528d2bd29efdd70/logs/worker.log\", \"remote_user\": \"loom\", \"log_level\": \"INFO\", \"network\": \"development\", \"zone\": \"us-central1-a\", \"gce_credential\": \"/root/.loom/gce_key.json\", \"scratch_disk_mount_point\": \"/mnt/loomdata\", \"scratch_disk_device_path\": \"/dev/disk/by-id/google-isaac-roles-hellostep-9a345d1c31bb40379528d2bd29efdd70-disk\", \"tags\": \"computes-uscentral\", \"boot_disk_size_gb\": \"163\", \"master_url\": \"https://isaac-roles:443\", \"scratch_disk_size_gb\": \"163\", \"docker_tag\": \"customserviceaccounts\", \"scratch_disk_name\": \"isaac-roles-hellostep-9a345d1c31bb40379528d2bd29efdd70-disk\", \"use_internal_ip\": \"True\", \"gce_email\": \"isaac-roles@gbsc-gcp-project-scgs-dev.iam.gserviceaccount.com\", \"scratch_disk_type\": \"pd-ssd\", \"instance_image\": \"centos-7\", \"instance_name\": \"isaac-roles-hellostep-9a345d1c31bb40379528d2bd29efdd70\", \"instance_type\": \"n1-standard-1\", \"subnetwork\": \"computes-uscentral\", \"task_run_docker_image\": \"ubuntu\"}\r\nERROR [2016-11-09 20:36:24,389] Failed to create task run attempt 9a345d1c31bb40379528d2bd29efdd70: post_task_run_attempt_error() takes exactly 3 arguments (2 given)\r\nTraceback (most recent call last):\r\n  File \"/opt/loom/loomengine/master/api/task_manager/cloud.py\", line 63, in _try_run\r\n    cls._run(task_run_attempt_id, requested_resources, environment, worker_name, worker_log_file)\r\n  File \"/opt/loom/loomengine/master/api/task_manager/cloud.py\", line 126, in _run\r\n    'detail': str(e)\r\nTypeError: post_task_run_attempt_error() takes exactly 3 arguments (2 given)\r\n```\r\nUsing this settings file:\r\n```\r\n[gcloud]\r\nDOCKER_TAG = customserviceaccounts\r\nWORKER_USES_SERVER_INTERNAL_IP = True\r\nCLIENT_USES_SERVER_INTERNAL_IP = True\r\n\r\nSERVER_NETWORK = development\r\nSERVER_CUSTOM_SUBNET = computes-uscentral\r\nSERVER_LOCATION = us-central1-a\r\nSERVER_TAGS = computes-uscentral,https-server\r\n\r\nWORKER_NETWORK = development\r\nWORKER_CUSTOM_SUBNET = computes-uscentral\r\nWORKER_LOCATION = us-central1-a\r\nWORKER_TAGS = computes-uscentral\r\n\r\nSERVICE_ACCOUNT_ROLES = [\"roles/compute.instanceAdmin\", \"roles/compute.storageAdmin\", \"roles/logging.logWriter\", \"roles/storage.objectCreator\", \"roles/storage.objectViewer\", \"roles/logging.viewer\", \"roles/iam.serviceAccountActor\"]\r\n```"},"comment":{"url":"https://api.github.com/repos/StanfordBioinformatics/loom/issues/comments/259849795","html_url":"https://github.com/StanfordBioinformatics/loom/issues/238#issuecomment-259849795","issue_url":"https://api.github.com/repos/StanfordBioinformatics/loom/issues/238","id":259849795,"user":{"login":"isaacliao","id":1683569,"avatar_url":"https://avatars.githubusercontent.com/u/1683569?v=3","gravatar_id":"","url":"https://api.github.com/users/isaacliao","html_url":"https://github.com/isaacliao","followers_url":"https://api.github.com/users/isaacliao/followers","following_url":"https://api.github.com/users/isaacliao/following{/other_user}","gists_url":"https://api.github.com/users/isaacliao/gists{/gist_id}","starred_url":"https://api.github.com/users/isaacliao/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/isaacliao/subscriptions","organizations_url":"https://api.github.com/users/isaacliao/orgs","repos_url":"https://api.github.com/users/isaacliao/repos","events_url":"https://api.github.com/users/isaacliao/events{/privacy}","received_events_url":"https://api.github.com/users/isaacliao/received_events","type":"User","site_admin":false},"created_at":"2016-11-11T00:35:54Z","updated_at":"2016-11-11T00:35:54Z","body":"Also seen when using a custom service account and not specifying custom roles:\r\n```\r\nERROR [2016-11-10 23:58:22,967] Failed to provision host.\r\nTraceback (most recent call last):\r\n  File \"/opt/loom/loomengine/master/api/task_manager/cloud.py\", line 121, in _run\r\n    cls._run_playbook(GCLOUD_CREATE_WORKER_PLAYBOOK, playbook_vars, logfile=ansible_logfile)\r\n  File \"/opt/loom/loomengine/master/api/task_manager/cloud.py\", line 169, in _run_playbook\r\n    raise Exception('Nonzero returncode %s for command: %s' % (returncode, ' '.join(cmd)))\r\nException: Nonzero returncode 2 for command: ansible-playbook -vvv --key-file /root/.ssh/google_compute_engine -i /opt/loom/loomengine/utils/gce.py /opt/loom/loomengine/playbooks/gcloud_create_worker.yml --extra-vars {\"docker_full_name\": \"loomengine/loom:customserviceaccounts\", \"gce_ssh_key_file\": \"/root/.ssh/google_compute_engine\", \"boot_disk_type\": \"pd-ssd\", \"task_run_attempt_id\": \"14f1adfdba8f45ac9a454e37283fcc03\", \"worker_log_file\": \"/mnt/loomdata/runtime_volumes/14f1adfdba8f45ac9a454e37283fcc03/logs/worker.log\", \"remote_user\": \"loom\", \"log_level\": \"INFO\", \"network\": \"development\", \"zone\": \"us-central1-a\", \"gce_credential\": \"/root/.loom/gce_key.json\", \"scratch_disk_mount_point\": \"/mnt/loomdata\", \"scratch_disk_device_path\": \"/dev/disk/by-id/google-isaac-custom4-hellostep-14f1adfdba8f45ac9a454e37283fcc03-disk\", \"tags\": \"computes-uscentral\", \"boot_disk_size_gb\": \"163\", \"master_url\": \"https://isaac-custom4:443\", \"scratch_disk_size_gb\": \"163\", \"docker_tag\": \"customserviceaccounts\", \"scratch_disk_name\": \"isaac-custom4-hellostep-14f1adfdba8f45ac9a454e37283fcc03-disk\", \"use_internal_ip\": \"True\", \"gce_email\": \"isaac-custom2@gbsc-gcp-project-scgs-dev.iam.gserviceaccount.com\", \"scratch_disk_type\": \"pd-ssd\", \"instance_image\": \"centos-7\", \"instance_name\": \"isaac-custom4-hellostep-14f1adfdba8f45ac9a454e37283fcc03\", \"instance_type\": \"n1-standard-1\", \"subnetwork\": \"computes-uscentral\", \"task_run_docker_image\": \"ubuntu\"}\r\nERROR [2016-11-10 23:58:22,967] Failed to create task run attempt 14f1adfdba8f45ac9a454e37283fcc03: post_task_run_attempt_error() takes exactly 3 arguments (2 given)\r\nTraceback (most recent call last):\r\n  File \"/opt/loom/loomengine/master/api/task_manager/cloud.py\", line 63, in _try_run\r\n    cls._run(task_run_attempt_id, requested_resources, environment, worker_name, worker_log_file)\r\n  File \"/opt/loom/loomengine/master/api/task_manager/cloud.py\", line 126, in _run\r\n    'detail': str(e)\r\nTypeError: post_task_run_attempt_error() takes exactly 3 arguments (2 given)\r\n```\r\nSettings file:\r\n```\r\n[gcloud]\r\nDOCKER_TAG = customserviceaccounts\r\nWORKER_USES_SERVER_INTERNAL_IP = True\r\nCLIENT_USES_SERVER_INTERNAL_IP = True\r\n \r\nSERVER_NETWORK = development\r\nSERVER_CUSTOM_SUBNET = computes-uscentral\r\nSERVER_LOCATION = us-central1-a\r\nSERVER_TAGS = computes-uscentral,https-server\r\n\r\nWORKER_NETWORK = development\r\nWORKER_CUSTOM_SUBNET = computes-uscentral\r\nWORKER_LOCATION = us-central1-a\r\nWORKER_TAGS = computes-uscentral\r\n\r\nCUSTOM_SERVICE_ACCOUNT_EMAIL = isaac-custom2@gbsc-gcp-project-scgs-dev.iam.gserviceaccount.com\r\n```"}}
{"actor":{"display_login":"isaacliao"}}