{"action":"created","issue":{"url":"https://api.github.com/repos/GoogleCloudPlatform/spark-on-k8s-operator/issues/196","repository_url":"https://api.github.com/repos/GoogleCloudPlatform/spark-on-k8s-operator","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/spark-on-k8s-operator/issues/196/labels{/name}","comments_url":"https://api.github.com/repos/GoogleCloudPlatform/spark-on-k8s-operator/issues/196/comments","events_url":"https://api.github.com/repos/GoogleCloudPlatform/spark-on-k8s-operator/issues/196/events","html_url":"https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/issues/196","id":334203506,"node_id":"MDU6SXNzdWUzMzQyMDM1MDY=","number":196,"title":"ScheduledSparkApplication failing with no logs to debug","user":{"login":"dharmeshkakadia","id":2123078,"node_id":"MDQ6VXNlcjIxMjMwNzg=","avatar_url":"https://avatars0.githubusercontent.com/u/2123078?v=4","gravatar_id":"","url":"https://api.github.com/users/dharmeshkakadia","html_url":"https://github.com/dharmeshkakadia","followers_url":"https://api.github.com/users/dharmeshkakadia/followers","following_url":"https://api.github.com/users/dharmeshkakadia/following{/other_user}","gists_url":"https://api.github.com/users/dharmeshkakadia/gists{/gist_id}","starred_url":"https://api.github.com/users/dharmeshkakadia/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dharmeshkakadia/subscriptions","organizations_url":"https://api.github.com/users/dharmeshkakadia/orgs","repos_url":"https://api.github.com/users/dharmeshkakadia/repos","events_url":"https://api.github.com/users/dharmeshkakadia/events{/privacy}","received_events_url":"https://api.github.com/users/dharmeshkakadia/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2018-06-20T18:43:43Z","updated_at":"2018-07-06T15:56:17Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"Spark operator log is full with\r\n\r\n```\r\nE0620 13:52:27.962182       1 controller.go:138] failed to sync ScheduledSparkApplication \"default/braze-blob-delete\": pods \"braze-blob-delete-1529197207414975848-1529197209364-driver\" not found\r\nW0620 14:08:40.257852       1 reflector.go:341] k8s.io/spark-on-k8s-operator/pkg/controller/sparkapplication/spark_pod_monitor.go:114: watch of *v1.Pod ended with: too old resource version: 1649124 (1649155)\r\nE0620 14:09:07.975880       1 controller.go:138] failed to sync ScheduledSparkApplication \"default/braze-blob-delete\": pods \"braze-blob-delete-1529197207414975848-1529197209364-driver\" not found\r\nE\r\n```\r\nAlso, the failed driver pods are not available so I can't get the logs as well. Here is the describe output for the run:\r\n\r\n```\r\nkubectl describe sparkapplications/braze-blob-delete-1529197207414975848\r\nName:         braze-blob-delete-1529197207414975848\r\nNamespace:    default\r\nLabels:       <none>\r\nAnnotations:  <none>\r\nAPI Version:  sparkoperator.k8s.io/v1alpha1\r\nKind:         SparkApplication\r\nMetadata:\r\n  Cluster Name:\r\n  Creation Timestamp:  2018-06-17T01:00:07Z\r\n  Generation:          0\r\n  Owner References:\r\n    API Version:     sparkoperator.k8s.io/v1alpha1\r\n    Kind:            ScheduledSparkApplication\r\n    Name:            braze-blob-delete\r\n    UID:             00f7668b-6ffc-11e8-bfb0-000d3a1d5635\r\n  Resource Version:  1583436\r\n  Self Link:         /apis/sparkoperator.k8s.io/v1alpha1/namespaces/default/sparkapplications/braze-blob-delete-1529197207414975848\r\n  UID:               c6e6a7e4-71c9-11e8-b63e-000d3a1d5437\r\nSpec:\r\n  Deps:\r\n  Driver:\r\n    Cores:  2\r\n    Labels:\r\n      Version:  2.4.0\r\n    Memory:     2048m\r\n    Secrets:\r\n      Name:           mdldatasecrets-gdpr-blob\r\n      Path:           /mnt/secrets/\r\n      Secret Type:    Generic\r\n    Service Account:  spark\r\n  Executor:\r\n    Cores:      2\r\n    Instances:  20\r\n    Labels:\r\n      Version:            2.4.0\r\n    Memory:               4096m\r\n  Image:                  mdldata.azurecr.io/data/gdpr-blob:latest\r\n  Main Application File:  local:///code/delete_controller_braze.py\r\n  Mode:                   cluster\r\n  Restart Policy:         Never\r\n  Spark Conf:\r\n    Spark . Event Log . Enabled:                                                               true\r\n  Type:                                                                                        Python\r\nStatus:\r\n  App Id:  braze-blob-delete-1529197207414975848-248004312\r\n  Application State:\r\n    Error Message:\r\n    State:          SUBMITTED\r\n  Completion Time:  2018-06-17T06:38:12Z\r\n  Driver Info:\r\n    Pod Name:             braze-blob-delete-1529197207414975848-1529197209364-driver\r\n    Web UI Port:          31318\r\n    Web UI Service Name:  braze-blob-delete-1529197207414975848-248004312-ui-svc\r\n  Executor State:\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 1:   FAILED\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 10:  FAILED\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 11:  FAILED\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 12:  FAILED\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 13:  FAILED\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 14:  FAILED\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 15:  FAILED\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 16:  FAILED\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 17:  FAILED\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 18:  FAILED\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 19:  FAILED\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 2:   FAILED\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 20:  FAILED\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 21:  FAILED\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 22:  FAILED\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 23:  FAILED\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 24:  FAILED\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 25:  FAILED\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 26:  FAILED\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 27:  FAILED\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 28:  FAILED\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 29:  FAILED\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 3:   FAILED\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 30:  FAILED\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 31:  FAILED\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 32:  FAILED\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 33:  FAILED\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 34:  FAILED\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 35:  FAILED\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 36:  FAILED\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 37:  FAILED\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 38:  FAILED\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 39:  FAILED\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 4:   FAILED\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 40:  FAILED\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 41:  PENDING\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 42:  PENDING\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 43:  PENDING\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 44:  PENDING\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 45:  FAILED\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 46:  FAILED\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 47:  PENDING\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 48:  FAILED\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 49:  PENDING\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 5:   FAILED\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 50:  FAILED\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 51:  PENDING\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 52:  PENDING\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 53:  PENDING\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 54:  PENDING\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 55:  FAILED\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 56:  FAILED\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 57:  PENDING\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 58:  FAILED\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 59:  FAILED\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 6:   FAILED\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 60:  FAILED\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 7:   FAILED\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 8:   FAILED\r\n    Braze - Blob - Delete - 1529197207414975848 - 1529197209364 - Exec - 9:   FAILED\r\n  Submission Time:                                                            2018-06-17T01:00:07Z\r\nEvents:                                                                       <none>\r\n```\r\nHow can I get the driver logs?"},"comment":{"url":"https://api.github.com/repos/GoogleCloudPlatform/spark-on-k8s-operator/issues/comments/403074691","html_url":"https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/issues/196#issuecomment-403074691","issue_url":"https://api.github.com/repos/GoogleCloudPlatform/spark-on-k8s-operator/issues/196","id":403074691,"node_id":"MDEyOklzc3VlQ29tbWVudDQwMzA3NDY5MQ==","user":{"login":"liyinan926","id":3709137,"node_id":"MDQ6VXNlcjM3MDkxMzc=","avatar_url":"https://avatars2.githubusercontent.com/u/3709137?v=4","gravatar_id":"","url":"https://api.github.com/users/liyinan926","html_url":"https://github.com/liyinan926","followers_url":"https://api.github.com/users/liyinan926/followers","following_url":"https://api.github.com/users/liyinan926/following{/other_user}","gists_url":"https://api.github.com/users/liyinan926/gists{/gist_id}","starred_url":"https://api.github.com/users/liyinan926/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/liyinan926/subscriptions","organizations_url":"https://api.github.com/users/liyinan926/orgs","repos_url":"https://api.github.com/users/liyinan926/repos","events_url":"https://api.github.com/users/liyinan926/events{/privacy}","received_events_url":"https://api.github.com/users/liyinan926/received_events","type":"User","site_admin":false},"created_at":"2018-07-06T15:56:17Z","updated_at":"2018-07-06T15:56:17Z","author_association":"MEMBER","body":"With a `Replace` concurrency policy, the operator should delete **both** the driver pod and `SparkApplication` instance of old runs that fall out of the maintenance window. It's weird that you could still see the `SparkApplication` instance while the driver pod of the run was gone. Regarding log access, if the driver pod was deleted, you wouldn't have access to its logs anymore. "}}
{"actor":{"display_login":"liyinan926"}}