{"action":"created","issue":{"url":"https://api.github.com/repos/kubeflow/kubeflow/issues/484","repository_url":"https://api.github.com/repos/kubeflow/kubeflow","labels_url":"https://api.github.com/repos/kubeflow/kubeflow/issues/484/labels{/name}","comments_url":"https://api.github.com/repos/kubeflow/kubeflow/issues/484/comments","events_url":"https://api.github.com/repos/kubeflow/kubeflow/issues/484/events","html_url":"https://github.com/kubeflow/kubeflow/issues/484","id":307794707,"number":484,"title":"TF serving GPU e2e test is flaky","user":{"login":"lluunn","id":32310205,"avatar_url":"https://avatars0.githubusercontent.com/u/32310205?v=4","gravatar_id":"","url":"https://api.github.com/users/lluunn","html_url":"https://github.com/lluunn","followers_url":"https://api.github.com/users/lluunn/followers","following_url":"https://api.github.com/users/lluunn/following{/other_user}","gists_url":"https://api.github.com/users/lluunn/gists{/gist_id}","starred_url":"https://api.github.com/users/lluunn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lluunn/subscriptions","organizations_url":"https://api.github.com/users/lluunn/orgs","repos_url":"https://api.github.com/users/lluunn/repos","events_url":"https://api.github.com/users/lluunn/events{/privacy}","received_events_url":"https://api.github.com/users/lluunn/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"lluunn","id":32310205,"avatar_url":"https://avatars0.githubusercontent.com/u/32310205?v=4","gravatar_id":"","url":"https://api.github.com/users/lluunn","html_url":"https://github.com/lluunn","followers_url":"https://api.github.com/users/lluunn/followers","following_url":"https://api.github.com/users/lluunn/following{/other_user}","gists_url":"https://api.github.com/users/lluunn/gists{/gist_id}","starred_url":"https://api.github.com/users/lluunn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lluunn/subscriptions","organizations_url":"https://api.github.com/users/lluunn/orgs","repos_url":"https://api.github.com/users/lluunn/repos","events_url":"https://api.github.com/users/lluunn/events{/privacy}","received_events_url":"https://api.github.com/users/lluunn/received_events","type":"User","site_admin":false},"assignees":[{"login":"lluunn","id":32310205,"avatar_url":"https://avatars0.githubusercontent.com/u/32310205?v=4","gravatar_id":"","url":"https://api.github.com/users/lluunn","html_url":"https://github.com/lluunn","followers_url":"https://api.github.com/users/lluunn/followers","following_url":"https://api.github.com/users/lluunn/following{/other_user}","gists_url":"https://api.github.com/users/lluunn/gists{/gist_id}","starred_url":"https://api.github.com/users/lluunn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lluunn/subscriptions","organizations_url":"https://api.github.com/users/lluunn/orgs","repos_url":"https://api.github.com/users/lluunn/repos","events_url":"https://api.github.com/users/lluunn/events{/privacy}","received_events_url":"https://api.github.com/users/lluunn/received_events","type":"User","site_admin":false}],"milestone":null,"comments":6,"created_at":"2018-03-22T20:03:36Z","updated_at":"2018-04-02T19:23:08Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"Example: #482 the test failed in a doc PR\r\n\r\n**In the step deploy-tf-serving-gpu, it verified deployment is up.**\r\nINFO|2018-03-22T19:11:44|/mnt/test-data-volume/kubeflow-presubmit-tf-serving-image-482-70f83cc-790-8b67/src/kubeflow/testing/py/kubeflow/testing/util.py|263| Deployment inception-gpu in namespace kubeflow-presubmit-tf-serving-image-482-70f83cc-790-8b67 is ready\r\n\r\n**However, in test-tf-serving-gpu step, it failed:**\r\nINFO|2018-03-22T19:11:48|/mnt/test-data-volume/kubeflow-presubmit-tf-serving-image-482-70f83cc-790-8b67/src/kubeflow/kubeflow/testing/test_tf_serving.py|99| prediction failed: AbortionError(code=StatusCode.UNAVAILABLE, details=\"Connect Failed\"). Retrying...\r\nINFO|2018-03-22T19:11:53|/mnt/test-data-volume/kubeflow-presubmit-tf-serving-image-482-70f83cc-790-8b67/src/kubeflow/kubeflow/testing/test_tf_serving.py|99| prediction failed: AbortionError(code=StatusCode.UNAVAILABLE, details=\"Connect Failed\"). Retrying...\r\nINFO|2018-03-22T19:12:04|/mnt/test-data-volume/kubeflow-presubmit-tf-serving-image-482-70f83cc-790-8b67/src/kubeflow/kubeflow/testing/test_tf_serving.py|116| Writing test results to /mnt/test-data-volume/kubeflow-presubmit-tf-serving-image-482-70f83cc-790-8b67/output/artifacts/junit_kubeflow-tf-serving-image-inception-gpu.xml\r\nINFO|2018-03-22T19:12:04|/mnt/test-data-volume/kubeflow-presubmit-tf-serving-image-482-70f83cc-790-8b67/src/kubeflow/testing/py/kubeflow/testing/test_util.py|151| Creating /mnt/test-data-volume/kubeflow-presubmit-tf-serving-image-482-70f83cc-790-8b67/output/artifacts/junit_kubeflow-tf-serving-image-inception-gpu.xml\r\nTraceback (most recent call last):\r\n  File \"/usr/lib/python2.7/runpy.py\", line 174, in _run_module_as_main\r\n    \"__main__\", fname, loader, pkg_name)\r\n  File \"/usr/lib/python2.7/runpy.py\", line 72, in _run_code\r\n    exec code in run_globals\r\n  File \"/mnt/test-data-volume/kubeflow-presubmit-tf-serving-image-482-70f83cc-790-8b67/src/kubeflow/kubeflow/testing/test_tf_serving.py\", line 127, in <module>\r\n    main()\r\n  File \"/mnt/test-data-volume/kubeflow-presubmit-tf-serving-image-482-70f83cc-790-8b67/src/kubeflow/kubeflow/testing/test_tf_serving.py\", line 94, in main\r\n    result = str(stub.Predict(request, 10.0))  # 10 secs timeout\r\n  File \"/usr/local/lib/python2.7/dist-packages/grpc/beta/_client_adaptations.py\", line 309, in __call__\r\n    self._request_serializer, self._response_deserializer)\r\n  File \"/usr/local/lib/python2.7/dist-packages/grpc/beta/_client_adaptations.py\", line 195, in _blocking_unary_unary\r\n    raise _abortion_error(rpc_error_call)\r\ngrpc.framework.interfaces.face.face.AbortionError: AbortionError(code=StatusCode.UNAVAILABLE, details=\"Connect Failed\")\r\n"},"comment":{"url":"https://api.github.com/repos/kubeflow/kubeflow/issues/comments/378017333","html_url":"https://github.com/kubeflow/kubeflow/issues/484#issuecomment-378017333","issue_url":"https://api.github.com/repos/kubeflow/kubeflow/issues/484","id":378017333,"user":{"login":"lluunn","id":32310205,"avatar_url":"https://avatars0.githubusercontent.com/u/32310205?v=4","gravatar_id":"","url":"https://api.github.com/users/lluunn","html_url":"https://github.com/lluunn","followers_url":"https://api.github.com/users/lluunn/followers","following_url":"https://api.github.com/users/lluunn/following{/other_user}","gists_url":"https://api.github.com/users/lluunn/gists{/gist_id}","starred_url":"https://api.github.com/users/lluunn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lluunn/subscriptions","organizations_url":"https://api.github.com/users/lluunn/orgs","repos_url":"https://api.github.com/users/lluunn/repos","events_url":"https://api.github.com/users/lluunn/events{/privacy}","received_events_url":"https://api.github.com/users/lluunn/received_events","type":"User","site_admin":false},"created_at":"2018-04-02T19:23:08Z","updated_at":"2018-04-02T19:23:08Z","author_association":"CONTRIBUTOR","body":"Sample failure: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/kubeflow_kubeflow/543/kubeflow-presubmit/937/\r\n\r\nERROR|2018-03-30T18:47:56|/mnt/test-data-volume/kubeflow-presubmit-tf-serving-image-543-4476009-937-d294/src/kubeflow/testing/py/kubeflow/testing/util.py|266| Timeout waiting for deployment inception-gpu in namespace kubeflow-presubmit-tf-serving-image-543-4476009-937-d294 to be ready\r\n\r\nFrom Stackdriver, I can only see logs from inception-cpu, no inception-gpu\r\n\r\n![screenshot from 2018-04-02 12-22-17](https://user-images.githubusercontent.com/32310205/38211830-8e091c3e-3670-11e8-8e63-873bc3b2835b.png)\r\n"}}
{"actor":{"display_login":"lluunn"}}