{"action":"created","issue":{"url":"https://api.github.com/repos/druid-io/druid/issues/2330","repository_url":"https://api.github.com/repos/druid-io/druid","labels_url":"https://api.github.com/repos/druid-io/druid/issues/2330/labels{/name}","comments_url":"https://api.github.com/repos/druid-io/druid/issues/2330/comments","events_url":"https://api.github.com/repos/druid-io/druid/issues/2330/events","html_url":"https://github.com/druid-io/druid/issues/2330","id":128621740,"number":2330,"title":"[Proposal] Allow Spark to act as the execution engine for druid queries","user":{"login":"jisookim0513","id":3820496,"avatar_url":"https://avatars.githubusercontent.com/u/3820496?v=3","gravatar_id":"","url":"https://api.github.com/users/jisookim0513","html_url":"https://github.com/jisookim0513","followers_url":"https://api.github.com/users/jisookim0513/followers","following_url":"https://api.github.com/users/jisookim0513/following{/other_user}","gists_url":"https://api.github.com/users/jisookim0513/gists{/gist_id}","starred_url":"https://api.github.com/users/jisookim0513/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jisookim0513/subscriptions","organizations_url":"https://api.github.com/users/jisookim0513/orgs","repos_url":"https://api.github.com/users/jisookim0513/repos","events_url":"https://api.github.com/users/jisookim0513/events{/privacy}","received_events_url":"https://api.github.com/users/jisookim0513/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/druid-io/druid/labels/Discuss","name":"Discuss","color":"ff00ff"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2016-01-25T19:48:36Z","updated_at":"2016-05-20T20:38:23Z","closed_at":null,"body":"Currently, Druid has a rough time when there is a long query that prevents from other shorter queries to be processed. In order to mitigate the effect of long queries, I'd like to enable Spark to serve as the execution engine for druid queries. I do not have a concrete (or set) design of what I would try to implement, but here are the two approaches I am considering:\r\n\r\n1) Make Spark query historical nodes and merge the results like a broker does.\r\n2) Make Spark download segments from S3 and do all the computations by itself.\r\n\r\nLinks related to this issue/proposal:\r\n\r\nhttps://groups.google.com/forum/#!searchin/druid-development/spark/druid-development/ULdKYZeven4/NIM5ySTbAQAJ\r\n\r\nhttps://groups.google.com/forum/#!msg/druid-development/Jp9Lv0mGFlg/f9m1U9MBBAAJ\r\n\r\nRelated PRs:\r\n\r\n- #2837 refactoring of client segment walker"},"comment":{"url":"https://api.github.com/repos/druid-io/druid/issues/comments/220712703","html_url":"https://github.com/druid-io/druid/issues/2330#issuecomment-220712703","issue_url":"https://api.github.com/repos/druid-io/druid/issues/2330","id":220712703,"user":{"login":"xvrl","id":815147,"avatar_url":"https://avatars.githubusercontent.com/u/815147?v=3","gravatar_id":"","url":"https://api.github.com/users/xvrl","html_url":"https://github.com/xvrl","followers_url":"https://api.github.com/users/xvrl/followers","following_url":"https://api.github.com/users/xvrl/following{/other_user}","gists_url":"https://api.github.com/users/xvrl/gists{/gist_id}","starred_url":"https://api.github.com/users/xvrl/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xvrl/subscriptions","organizations_url":"https://api.github.com/users/xvrl/orgs","repos_url":"https://api.github.com/users/xvrl/repos","events_url":"https://api.github.com/users/xvrl/events{/privacy}","received_events_url":"https://api.github.com/users/xvrl/received_events","type":"User","site_admin":false},"created_at":"2016-05-20T20:38:23Z","updated_at":"2016-05-20T20:38:23Z","body":"@gianm indeed, we started with the first approach since most of that work is a pre-requisite to get the second one working. So far that seems to solve our use-case, so unless we have issues with it, we probably won't go down the second option. "}}
{"actor":{"display_login":"xvrl"}}