{"action":"opened","issue":{"url":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms/issues/400","repository_url":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms","labels_url":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms/issues/400/labels{/name}","comments_url":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms/issues/400/comments","events_url":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms/issues/400/events","html_url":"https://github.com/googlegenomics/gcp-variant-transforms/issues/400","id":374058771,"node_id":"MDU6SXNzdWUzNzQwNTg3NzE=","number":400,"title":"FR: Retry gsutil cp operations","user":{"login":"mbookman","id":5818294,"node_id":"MDQ6VXNlcjU4MTgyOTQ=","avatar_url":"https://avatars2.githubusercontent.com/u/5818294?v=4","gravatar_id":"","url":"https://api.github.com/users/mbookman","html_url":"https://github.com/mbookman","followers_url":"https://api.github.com/users/mbookman/followers","following_url":"https://api.github.com/users/mbookman/following{/other_user}","gists_url":"https://api.github.com/users/mbookman/gists{/gist_id}","starred_url":"https://api.github.com/users/mbookman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mbookman/subscriptions","organizations_url":"https://api.github.com/users/mbookman/orgs","repos_url":"https://api.github.com/users/mbookman/repos","events_url":"https://api.github.com/users/mbookman/events{/privacy}","received_events_url":"https://api.github.com/users/mbookman/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-10-25T17:46:11Z","updated_at":"2018-10-25T17:46:11Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"I am running `vcf_to_bq` with `--run_annotation_pipeline` set.\r\nI have 5103 gzipped VCF files which were produced by the Broad's Joint Genotyping workflow.\r\n\r\nSome of the annotation steps fail due to transient errors that cause `gsutil cp` to fail.\r\nSince the interactions with GCS depend on fetching the Compute Engine service account credentials from the metadata server, the metadata server failures manifest in a few different errors:\r\n\r\n    Failure: Could not reach metadata service: [Errno 111] Connection refused.\r\n    Failure: GCE credentials requested outside a GCE instance.\r\n    ServiceException: 401 Anonymous caller does not have storage.objects.list access to <project-id>\r\n\r\nI suggest putting in wrappers around the `gsutil cp` commands added by vep_runner.py\r\n`gsutil` has built-in retries, but there are some conditions that it considers to be a hard-failure and authentication/permissions errors are hard failures. For the Pipelines API v1alpha2 and for `dsub`, before putting in our own retries, we would often see a \"low\" failure rate (maybe 3%) with jobs due to `gsutil cp` failures. For 100 operations, that meant chasing down 3 failures and retrying; for 1000 operations... 30, etc.\r\n\r\nOnce we put in the automatic retries those types of failures all but went away."}}
{"actor":{"display_login":"mbookman"}}