{"action":"created","issue":{"url":"https://api.github.com/repos/foxish/spark/issues/3","repository_url":"https://api.github.com/repos/foxish/spark","labels_url":"https://api.github.com/repos/foxish/spark/issues/3/labels{/name}","comments_url":"https://api.github.com/repos/foxish/spark/issues/3/comments","events_url":"https://api.github.com/repos/foxish/spark/issues/3/events","html_url":"https://github.com/foxish/spark/issues/3","id":188448078,"number":3,"title":"Kubernetes ThirdPartyResource for tracking Spark Jobs","user":{"login":"foxish","id":906471,"avatar_url":"https://avatars.githubusercontent.com/u/906471?v=3","gravatar_id":"","url":"https://api.github.com/users/foxish","html_url":"https://github.com/foxish","followers_url":"https://api.github.com/users/foxish/followers","following_url":"https://api.github.com/users/foxish/following{/other_user}","gists_url":"https://api.github.com/users/foxish/gists{/gist_id}","starred_url":"https://api.github.com/users/foxish/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/foxish/subscriptions","organizations_url":"https://api.github.com/users/foxish/orgs","repos_url":"https://api.github.com/users/foxish/repos","events_url":"https://api.github.com/users/foxish/events{/privacy}","received_events_url":"https://api.github.com/users/foxish/received_events","type":"User","site_admin":false},"labels":[{"id":435620695,"url":"https://api.github.com/repos/foxish/spark/labels/enhancement","name":"enhancement","color":"84b6eb","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":14,"created_at":"2016-11-10T08:32:31Z","updated_at":"2016-11-10T23:33:18Z","closed_at":null,"body":"**Issues**:\r\n\r\n* We need a way for users to find, monitor status, and kill their spark jobs.\r\n* Other cluster managers provide an interface to view all of a user's spark-jobs in a cluster, along with their state (QUEUED,  SUBMITTED, RUNNING, FINISHED, FAILED, KILLED).\r\n\r\n**Shortcomings of the current method**:\r\n\r\n* It is not possible to get sufficient detail about every spark job just by looking at the pods and their states.\r\n* As the spark framework is currently performing the entire function of a kubernetes controller, it gives us no way to find the state of various running Spark jobs than to implement `spark-submit --status`, or `spark-submit --kill` which would have to encode special logic to find the drivers in a particular namespace and find their status. This would be bad for cluster administrators who deal with kubernetes abstractions, and not application specifics.\r\n* Accessing arbitrary data such as progress from the driver pod, or the names of its associated executors is difficult. \r\n\r\n**Proposed Solution**:\r\n\r\nA [ThirdPartyResource](http://kubernetes.io/docs/user-guide/thirdpartyresources/) to keep track of the execution state (pending/running/failed) of each SparkJob, failure reasons if any, the identity of the driver and executor pods associated, as well as configuration metadata associated with that job (number of executors, memory per executor, etc). \r\n\r\n```\r\nmetadata:\r\n  name: spark-job.kubernetes.io\r\n  labels:\r\n    resource: spark-job\r\n    object: spark\r\napiVersion: extensions/v1beta1\r\nkind: ThirdPartyResource\r\ndescription: \"A resource that manages a spark job\"\r\nversions:\r\n  - name: v1\r\n```\r\n\r\nThe cluster administrator is responsible for creating this resource, which makes a new API endpoint available in Kubernetes. This TPR would enable us to create objects of the kind `SparkJob` and store JSON within them. Each such object would be associated with a single spark job, and would store all the status and metadata associated with it. The driver pod is responsible for the life-cycle of the SparkJob object, from creation till deletion.\r\n\r\nA sample object of the above kind looks like the following:\r\n\r\n```\r\n{\r\n    \"apiVersion\": \"kubernetes.io/v1\",\r\n    \"image\": \"driver-image\",\r\n    \"kind\": \"SparkJob\",\r\n    \"metadata\": {\r\n        \"name\": \"spark-driver-1924\",\r\n        \"namespace\": \"default\",\r\n        \"selfLink\": \"/apis/kubernetes.io/v1/namespaces/default/sparkjobs/spark-driver-1924\",\r\n        \"uid\": \"91022bc2-a71d-11e6-a4be-42010af00002\",\r\n        \"resourceVersion\": \"765519\",\r\n        \"creationTimestamp\": \"2016-11-10T08:13:31Z\"\r\n    },\r\n    \"num-executors\": 10,\r\n    \"state\": \"completed\",\r\n    \"driver-pod\": \"driver-2ds9f\"\r\n    ...\r\n    ...\r\n}\r\n\r\n```\r\n\r\nThe driver pod has complete visibility into progress of the job, and can set the status of its SparkJob object.  The driver can also watch this resource for configuration changes which may be triggered by the user/cluster administrator. Killing a spark-job can be performed by destroying the associated SparkJob object, which will cause the driver pod to terminate its executors and clean up gracefully.\r\n\r\n- It makes the state of each SparkJob available outside Spark itself and provides some visibility to the cluster administrator regarding various running spark jobs in the system.\r\n- This SparkJob object can be consumed by `spark-submit --status`, or a dashboard to display various details about spark jobs in the system. \r\n\r\n**Further thought**:\r\n* What if the driver pods do not exit cleanly (or are force-killed)? Who is responsible for cleaning up the SparkJob object?"},"comment":{"url":"https://api.github.com/repos/foxish/spark/issues/comments/259839479","html_url":"https://github.com/foxish/spark/issues/3#issuecomment-259839479","issue_url":"https://api.github.com/repos/foxish/spark/issues/3","id":259839479,"user":{"login":"foxish","id":906471,"avatar_url":"https://avatars.githubusercontent.com/u/906471?v=3","gravatar_id":"","url":"https://api.github.com/users/foxish","html_url":"https://github.com/foxish","followers_url":"https://api.github.com/users/foxish/followers","following_url":"https://api.github.com/users/foxish/following{/other_user}","gists_url":"https://api.github.com/users/foxish/gists{/gist_id}","starred_url":"https://api.github.com/users/foxish/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/foxish/subscriptions","organizations_url":"https://api.github.com/users/foxish/orgs","repos_url":"https://api.github.com/users/foxish/repos","events_url":"https://api.github.com/users/foxish/events{/privacy}","received_events_url":"https://api.github.com/users/foxish/received_events","type":"User","site_admin":false},"created_at":"2016-11-10T23:33:18Z","updated_at":"2016-11-10T23:33:18Z","body":"A TPR (for example, the one of kind  'SparkJob' in the issue) itself is not namespaced. The third party **objects** which are created of that SparkJob we introduce, however, are namespaced. If the TPR is not created for that cluster, Spark will be unable to create Third Party objects in which we want to store state and as you said, it can detect this and complain about it. "}}
{"actor":{"display_login":"foxish"}}