{"action":"opened","number":3611,"pull_request":{"url":"https://api.github.com/repos/druid-io/druid/pulls/3611","id":90916036,"html_url":"https://github.com/druid-io/druid/pull/3611","diff_url":"https://github.com/druid-io/druid/pull/3611.diff","patch_url":"https://github.com/druid-io/druid/pull/3611.patch","issue_url":"https://api.github.com/repos/druid-io/druid/issues/3611","number":3611,"state":"open","locked":false,"title":"IndexTask improvements","user":{"login":"dclim","id":11050882,"avatar_url":"https://avatars.githubusercontent.com/u/11050882?v=3","gravatar_id":"","url":"https://api.github.com/users/dclim","html_url":"https://github.com/dclim","followers_url":"https://api.github.com/users/dclim/followers","following_url":"https://api.github.com/users/dclim/following{/other_user}","gists_url":"https://api.github.com/users/dclim/gists{/gist_id}","starred_url":"https://api.github.com/users/dclim/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dclim/subscriptions","organizations_url":"https://api.github.com/users/dclim/orgs","repos_url":"https://api.github.com/users/dclim/repos","events_url":"https://api.github.com/users/dclim/events{/privacy}","received_events_url":"https://api.github.com/users/dclim/received_events","type":"User","site_admin":false},"body":"This PR introduces a number of improvements to the IndexTask (non-Hadoop batch indexer). The primary motivation of improving IndexTask is with the view that it can be used with something like #1998 for automatic merging and splitting of segments for users without Hadoop. This would replace the functionality of the MergeTask which is limited by the fact that it can only operate on NoneShardSpec-type segments. Secondary motivation is to improve the performance/stability of IndexTask for users who don't run a Hadoop cluster.\r\n\r\nMain changes:\r\n- Overall performance: Old implementation iterated through entire data set once to determine intervals, once to determine shards for that interval, and once to generate index for each shard; hence if your segmentGranularity resulted in 12 time intervals and each interval had 3 shards, the *entire dataset* was read from 1 + 12 + 36 times. This has been reduced to either 1 time if `numShards` and `intervals` is provided, otherwise 2 times. Performance gains will vary based on number of intervals/shards, but in all cases are better than the previous implementation and typically significantly better. Performance compared to Hadoop index task running locally is typically equivalent or better. The Appenderator implementation is now being used for index generation.\r\n- ReplayableFirehose: Introduced a new firehose wrapper type that entirely consumes the wrapped firehose into temporary files and then serves read requests from the cached data. This has a few benefits:\r\n  - In the case that we need to read the data more than once, it prevents us from fetching from remote resources multiple times (saves time, potentially money)\r\n  - In the case that we need to read the data more than once, it gives us deterministic results where the wrapped firehose may not\r\n  - In the case that the underlying firehose is unreliable, it allows us to retriably grab the entire dataset first and only begin serving data after we have a full copy (this addresses a number of issues recently reported with the S3 firehose failing in the middle of reading)\r\n  - I did not document the ReplayableFirehose, even though it could theoretically wrap any other firehose because I'm not sure how much use it has outside of the index task. IndexTask wraps any provided firehose in a ReplayableFirehose, although this behavior can be disabled by configuration.\r\n- segmentGranularity `intervals` no longer needs to be explicitly provided but can be determined, which could make things easier for some use cases\r\n- Append support: Users can set the `appendToExisting` flag which will generate segments that are additional partitions added to the latest version of the existing segments, effectively 'appending data' into the datasource. The set of segments to be appended to must be an extendable-type (i.e. they can't be NoneShardSpecs).\r\n- Reworked IndexTuningConfig to have similarly-named parameters to other index task types: If people agree with this change, it is backwards-incompatible and would require users to update their ingestion specs.\r\n\r\nThis PR depends on #3483 being merged and I don't expect tests to pass until that happens.\r\n\r\n\r\n","created_at":"2016-10-26T00:21:27Z","updated_at":"2016-10-26T00:21:27Z","closed_at":null,"merged_at":null,"merge_commit_sha":null,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/druid-io/druid/milestones/21","html_url":"https://github.com/druid-io/druid/milestones/0.9.3","labels_url":"https://api.github.com/repos/druid-io/druid/milestones/21/labels","id":1831000,"number":21,"title":"0.9.3","description":"","creator":{"login":"fjy","id":428325,"avatar_url":"https://avatars.githubusercontent.com/u/428325?v=3","gravatar_id":"","url":"https://api.github.com/users/fjy","html_url":"https://github.com/fjy","followers_url":"https://api.github.com/users/fjy/followers","following_url":"https://api.github.com/users/fjy/following{/other_user}","gists_url":"https://api.github.com/users/fjy/gists{/gist_id}","starred_url":"https://api.github.com/users/fjy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fjy/subscriptions","organizations_url":"https://api.github.com/users/fjy/orgs","repos_url":"https://api.github.com/users/fjy/repos","events_url":"https://api.github.com/users/fjy/events{/privacy}","received_events_url":"https://api.github.com/users/fjy/received_events","type":"User","site_admin":false},"open_issues":50,"closed_issues":28,"state":"open","created_at":"2016-06-15T22:35:58Z","updated_at":"2016-10-26T00:21:27Z","due_on":null,"closed_at":null},"commits_url":"https://api.github.com/repos/druid-io/druid/pulls/3611/commits","review_comments_url":"https://api.github.com/repos/druid-io/druid/pulls/3611/comments","review_comment_url":"https://api.github.com/repos/druid-io/druid/pulls/comments{/number}","comments_url":"https://api.github.com/repos/druid-io/druid/issues/3611/comments","statuses_url":"https://api.github.com/repos/druid-io/druid/statuses/37688ba7411d31218864145af90735246c2fa8b5","head":{"label":"dclim:sans-hadoops","ref":"sans-hadoops","sha":"37688ba7411d31218864145af90735246c2fa8b5","user":{"login":"dclim","id":11050882,"avatar_url":"https://avatars.githubusercontent.com/u/11050882?v=3","gravatar_id":"","url":"https://api.github.com/users/dclim","html_url":"https://github.com/dclim","followers_url":"https://api.github.com/users/dclim/followers","following_url":"https://api.github.com/users/dclim/following{/other_user}","gists_url":"https://api.github.com/users/dclim/gists{/gist_id}","starred_url":"https://api.github.com/users/dclim/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dclim/subscriptions","organizations_url":"https://api.github.com/users/dclim/orgs","repos_url":"https://api.github.com/users/dclim/repos","events_url":"https://api.github.com/users/dclim/events{/privacy}","received_events_url":"https://api.github.com/users/dclim/received_events","type":"User","site_admin":false},"repo":{"id":42546365,"name":"druid","full_name":"dclim/druid","owner":{"login":"dclim","id":11050882,"avatar_url":"https://avatars.githubusercontent.com/u/11050882?v=3","gravatar_id":"","url":"https://api.github.com/users/dclim","html_url":"https://github.com/dclim","followers_url":"https://api.github.com/users/dclim/followers","following_url":"https://api.github.com/users/dclim/following{/other_user}","gists_url":"https://api.github.com/users/dclim/gists{/gist_id}","starred_url":"https://api.github.com/users/dclim/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dclim/subscriptions","organizations_url":"https://api.github.com/users/dclim/orgs","repos_url":"https://api.github.com/users/dclim/repos","events_url":"https://api.github.com/users/dclim/events{/privacy}","received_events_url":"https://api.github.com/users/dclim/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/dclim/druid","description":"Column oriented distributed data store ideal for powering interactive applications","fork":true,"url":"https://api.github.com/repos/dclim/druid","forks_url":"https://api.github.com/repos/dclim/druid/forks","keys_url":"https://api.github.com/repos/dclim/druid/keys{/key_id}","collaborators_url":"https://api.github.com/repos/dclim/druid/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/dclim/druid/teams","hooks_url":"https://api.github.com/repos/dclim/druid/hooks","issue_events_url":"https://api.github.com/repos/dclim/druid/issues/events{/number}","events_url":"https://api.github.com/repos/dclim/druid/events","assignees_url":"https://api.github.com/repos/dclim/druid/assignees{/user}","branches_url":"https://api.github.com/repos/dclim/druid/branches{/branch}","tags_url":"https://api.github.com/repos/dclim/druid/tags","blobs_url":"https://api.github.com/repos/dclim/druid/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/dclim/druid/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/dclim/druid/git/refs{/sha}","trees_url":"https://api.github.com/repos/dclim/druid/git/trees{/sha}","statuses_url":"https://api.github.com/repos/dclim/druid/statuses/{sha}","languages_url":"https://api.github.com/repos/dclim/druid/languages","stargazers_url":"https://api.github.com/repos/dclim/druid/stargazers","contributors_url":"https://api.github.com/repos/dclim/druid/contributors","subscribers_url":"https://api.github.com/repos/dclim/druid/subscribers","subscription_url":"https://api.github.com/repos/dclim/druid/subscription","commits_url":"https://api.github.com/repos/dclim/druid/commits{/sha}","git_commits_url":"https://api.github.com/repos/dclim/druid/git/commits{/sha}","comments_url":"https://api.github.com/repos/dclim/druid/comments{/number}","issue_comment_url":"https://api.github.com/repos/dclim/druid/issues/comments{/number}","contents_url":"https://api.github.com/repos/dclim/druid/contents/{+path}","compare_url":"https://api.github.com/repos/dclim/druid/compare/{base}...{head}","merges_url":"https://api.github.com/repos/dclim/druid/merges","archive_url":"https://api.github.com/repos/dclim/druid/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/dclim/druid/downloads","issues_url":"https://api.github.com/repos/dclim/druid/issues{/number}","pulls_url":"https://api.github.com/repos/dclim/druid/pulls{/number}","milestones_url":"https://api.github.com/repos/dclim/druid/milestones{/number}","notifications_url":"https://api.github.com/repos/dclim/druid/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/dclim/druid/labels{/name}","releases_url":"https://api.github.com/repos/dclim/druid/releases{/id}","deployments_url":"https://api.github.com/repos/dclim/druid/deployments","created_at":"2015-09-15T21:02:07Z","updated_at":"2016-02-01T17:26:20Z","pushed_at":"2016-10-25T23:41:28Z","git_url":"git://github.com/dclim/druid.git","ssh_url":"git@github.com:dclim/druid.git","clone_url":"https://github.com/dclim/druid.git","svn_url":"https://github.com/dclim/druid","homepage":"http://druid.io","size":110376,"stargazers_count":0,"watchers_count":0,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"druid-io:master","ref":"master","sha":"98465f47b54e1472da2c4a5ec0269a60bdc73a8d","user":{"login":"druid-io","id":4203474,"avatar_url":"https://avatars.githubusercontent.com/u/4203474?v=3","gravatar_id":"","url":"https://api.github.com/users/druid-io","html_url":"https://github.com/druid-io","followers_url":"https://api.github.com/users/druid-io/followers","following_url":"https://api.github.com/users/druid-io/following{/other_user}","gists_url":"https://api.github.com/users/druid-io/gists{/gist_id}","starred_url":"https://api.github.com/users/druid-io/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/druid-io/subscriptions","organizations_url":"https://api.github.com/users/druid-io/orgs","repos_url":"https://api.github.com/users/druid-io/repos","events_url":"https://api.github.com/users/druid-io/events{/privacy}","received_events_url":"https://api.github.com/users/druid-io/received_events","type":"Organization","site_admin":false},"repo":{"id":6358188,"name":"druid","full_name":"druid-io/druid","owner":{"login":"druid-io","id":4203474,"avatar_url":"https://avatars.githubusercontent.com/u/4203474?v=3","gravatar_id":"","url":"https://api.github.com/users/druid-io","html_url":"https://github.com/druid-io","followers_url":"https://api.github.com/users/druid-io/followers","following_url":"https://api.github.com/users/druid-io/following{/other_user}","gists_url":"https://api.github.com/users/druid-io/gists{/gist_id}","starred_url":"https://api.github.com/users/druid-io/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/druid-io/subscriptions","organizations_url":"https://api.github.com/users/druid-io/orgs","repos_url":"https://api.github.com/users/druid-io/repos","events_url":"https://api.github.com/users/druid-io/events{/privacy}","received_events_url":"https://api.github.com/users/druid-io/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/druid-io/druid","description":"Column oriented distributed data store ideal for powering interactive applications","fork":false,"url":"https://api.github.com/repos/druid-io/druid","forks_url":"https://api.github.com/repos/druid-io/druid/forks","keys_url":"https://api.github.com/repos/druid-io/druid/keys{/key_id}","collaborators_url":"https://api.github.com/repos/druid-io/druid/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/druid-io/druid/teams","hooks_url":"https://api.github.com/repos/druid-io/druid/hooks","issue_events_url":"https://api.github.com/repos/druid-io/druid/issues/events{/number}","events_url":"https://api.github.com/repos/druid-io/druid/events","assignees_url":"https://api.github.com/repos/druid-io/druid/assignees{/user}","branches_url":"https://api.github.com/repos/druid-io/druid/branches{/branch}","tags_url":"https://api.github.com/repos/druid-io/druid/tags","blobs_url":"https://api.github.com/repos/druid-io/druid/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/druid-io/druid/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/druid-io/druid/git/refs{/sha}","trees_url":"https://api.github.com/repos/druid-io/druid/git/trees{/sha}","statuses_url":"https://api.github.com/repos/druid-io/druid/statuses/{sha}","languages_url":"https://api.github.com/repos/druid-io/druid/languages","stargazers_url":"https://api.github.com/repos/druid-io/druid/stargazers","contributors_url":"https://api.github.com/repos/druid-io/druid/contributors","subscribers_url":"https://api.github.com/repos/druid-io/druid/subscribers","subscription_url":"https://api.github.com/repos/druid-io/druid/subscription","commits_url":"https://api.github.com/repos/druid-io/druid/commits{/sha}","git_commits_url":"https://api.github.com/repos/druid-io/druid/git/commits{/sha}","comments_url":"https://api.github.com/repos/druid-io/druid/comments{/number}","issue_comment_url":"https://api.github.com/repos/druid-io/druid/issues/comments{/number}","contents_url":"https://api.github.com/repos/druid-io/druid/contents/{+path}","compare_url":"https://api.github.com/repos/druid-io/druid/compare/{base}...{head}","merges_url":"https://api.github.com/repos/druid-io/druid/merges","archive_url":"https://api.github.com/repos/druid-io/druid/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/druid-io/druid/downloads","issues_url":"https://api.github.com/repos/druid-io/druid/issues{/number}","pulls_url":"https://api.github.com/repos/druid-io/druid/pulls{/number}","milestones_url":"https://api.github.com/repos/druid-io/druid/milestones{/number}","notifications_url":"https://api.github.com/repos/druid-io/druid/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/druid-io/druid/labels{/name}","releases_url":"https://api.github.com/repos/druid-io/druid/releases{/id}","deployments_url":"https://api.github.com/repos/druid-io/druid/deployments","created_at":"2012-10-23T19:08:07Z","updated_at":"2016-10-25T23:39:38Z","pushed_at":"2016-10-26T00:21:27Z","git_url":"git://github.com/druid-io/druid.git","ssh_url":"git@github.com:druid-io/druid.git","clone_url":"https://github.com/druid-io/druid.git","svn_url":"https://github.com/druid-io/druid","homepage":"http://druid.io","size":158371,"stargazers_count":3869,"watchers_count":3869,"language":"Java","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":910,"mirror_url":null,"open_issues_count":555,"forks":910,"open_issues":555,"watchers":3869,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/druid-io/druid/pulls/3611"},"html":{"href":"https://github.com/druid-io/druid/pull/3611"},"issue":{"href":"https://api.github.com/repos/druid-io/druid/issues/3611"},"comments":{"href":"https://api.github.com/repos/druid-io/druid/issues/3611/comments"},"review_comments":{"href":"https://api.github.com/repos/druid-io/druid/pulls/3611/comments"},"review_comment":{"href":"https://api.github.com/repos/druid-io/druid/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/druid-io/druid/pulls/3611/commits"},"statuses":{"href":"https://api.github.com/repos/druid-io/druid/statuses/37688ba7411d31218864145af90735246c2fa8b5"}},"merged":false,"mergeable":null,"mergeable_state":"unknown","merged_by":null,"comments":0,"review_comments":0,"commits":1,"additions":1513,"deletions":616,"changed_files":15}}
{"actor":{"display_login":"dclim"}}