{"action":"created","issue":{"url":"https://api.github.com/repos/spark-jobserver/spark-jobserver/issues/442","repository_url":"https://api.github.com/repos/spark-jobserver/spark-jobserver","labels_url":"https://api.github.com/repos/spark-jobserver/spark-jobserver/issues/442/labels{/name}","comments_url":"https://api.github.com/repos/spark-jobserver/spark-jobserver/issues/442/comments","events_url":"https://api.github.com/repos/spark-jobserver/spark-jobserver/issues/442/events","html_url":"https://github.com/spark-jobserver/spark-jobserver/pull/442","id":151316999,"number":442,"title":"New SparkJob API","user":{"login":"velvia","id":1062875,"avatar_url":"https://avatars.githubusercontent.com/u/1062875?v=3","gravatar_id":"","url":"https://api.github.com/users/velvia","html_url":"https://github.com/velvia","followers_url":"https://api.github.com/users/velvia/followers","following_url":"https://api.github.com/users/velvia/following{/other_user}","gists_url":"https://api.github.com/users/velvia/gists{/gist_id}","starred_url":"https://api.github.com/users/velvia/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/velvia/subscriptions","organizations_url":"https://api.github.com/users/velvia/orgs","repos_url":"https://api.github.com/users/velvia/repos","events_url":"https://api.github.com/users/velvia/events{/privacy}","received_events_url":"https://api.github.com/users/velvia/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2016-04-27T07:28:33Z","updated_at":"2016-04-27T08:14:24Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/spark-jobserver/spark-jobserver/pulls/442","html_url":"https://github.com/spark-jobserver/spark-jobserver/pull/442","diff_url":"https://github.com/spark-jobserver/spark-jobserver/pull/442.diff","patch_url":"https://github.com/spark-jobserver/spark-jobserver/pull/442.patch"},"body":"Since the beginning of Spark Job Server, the SparkJob API has pretty much remained unchanged.  Over time, however, the API has shown its age.  Some of its issues include:\r\n- Conflation of the job input with the context configuration\r\n- Need to mix in NamedRDDs / NamedObjects\r\n- Unavailability of things like job ID and other environmental things\r\n- Possible need to parse Config twice: once in Validate method, and once in runJob\r\n\r\nThis PR represents a new SparkJob API which is designed to solve all of the above issues.  \r\nExisting jobs can still be compiled, but will not be binary compatible with this new SJS.\r\nThe new API appears under a new namespace, spark.jobserver.api.SparkJob\r\n\r\nStill TODO: Java support is currently broken. "},"comment":{"url":"https://api.github.com/repos/spark-jobserver/spark-jobserver/issues/comments/215007011","html_url":"https://github.com/spark-jobserver/spark-jobserver/pull/442#issuecomment-215007011","issue_url":"https://api.github.com/repos/spark-jobserver/spark-jobserver/issues/442","id":215007011,"user":{"login":"reactormonk","id":49185,"avatar_url":"https://avatars.githubusercontent.com/u/49185?v=3","gravatar_id":"","url":"https://api.github.com/users/reactormonk","html_url":"https://github.com/reactormonk","followers_url":"https://api.github.com/users/reactormonk/followers","following_url":"https://api.github.com/users/reactormonk/following{/other_user}","gists_url":"https://api.github.com/users/reactormonk/gists{/gist_id}","starred_url":"https://api.github.com/users/reactormonk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/reactormonk/subscriptions","organizations_url":"https://api.github.com/users/reactormonk/orgs","repos_url":"https://api.github.com/users/reactormonk/repos","events_url":"https://api.github.com/users/reactormonk/events{/privacy}","received_events_url":"https://api.github.com/users/reactormonk/received_events","type":"User","site_admin":false},"created_at":"2016-04-27T08:14:24Z","updated_at":"2016-04-27T08:14:24Z","body":"I'd go with `Validated` instead of `Or` - seems better from a semantic side imo. http://eed3si9n.com/herding-cats/Validated.html\r\n\r\nOn the Java side - just add a version with lots of `asInstanceOf`, `Object` and add an \"abandon all types ye who enter here\" shield."}}